premiere
/
premiere-libtorrent
2
2
Fork 1
Code Issues Pull Requests Releases Wiki Activity

regenerate html

This commit is contained in:
Arvid Norberg 2013-12-31 16:59:47 +00:00
parent 488f7697c6
commit b656076323
2 changed files with 379 additions and 308 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

252
docs/dht_store.html
View File

@ -56,20 +56,20 @@
<p class="topic-title first">Table of contents</p>
<ul class="simple">
<li><a class="reference internal" href="#terminology" id="id3">terminology</a></li>
<li><a class="reference internal" href="#messages" id="id4">messages</a></li>
<li><a class="reference internal" href="#immutable-items" id="id5">immutable items</a><ul>
<li><a class="reference internal" href="#put-message" id="id6">put message</a></li>
<li><a class="reference internal" href="#get-message" id="id7">get message</a></li>
<li><a class="reference internal" href="#messages" id="id4">messages</a><ul>
<li><a class="reference internal" href="#put-message" id="id5">put message</a></li>
<li><a class="reference internal" href="#get-message" id="id6">get message</a></li>
</ul>
</li>
<li><a class="reference internal" href="#mutable-items" id="id8">mutable items</a><ul>
<li><a class="reference internal" href="#id1" id="id9">put message</a></li>
<li><a class="reference internal" href="#id2" id="id10">get message</a></li>
<li><a class="reference internal" href="#mutable-items" id="id7">mutable items</a><ul>
<li><a class="reference internal" href="#id1" id="id8">put message</a></li>
<li><a class="reference internal" href="#id2" id="id9">get message</a></li>
</ul>
</li>
<li><a class="reference internal" href="#signature-verification" id="id11">signature verification</a></li>
<li><a class="reference internal" href="#expiration" id="id12">expiration</a></li>
<li><a class="reference internal" href="#test-vectors" id="id13">test vectors</a></li>
<li><a class="reference internal" href="#signature-verification" id="id10">signature verification</a></li>
<li><a class="reference internal" href="#expiration" id="id11">expiration</a></li>
<li><a class="reference internal" href="#test-vector" id="id12">test vector</a></li>
<li><a class="reference internal" href="#resources" id="id13">resources</a></li>
</ul>
</div>
<p>This is a proposal for an extension to the BitTorrent DHT to allow
@ -88,41 +88,45 @@ alive.</p>
</div>
<div class="section" id="messages">
<h1>messages</h1>
<p>The proposed new messages <tt class="docutils literal">get</tt> and <tt class="docutils literal">put</tt> are similar to the existing <tt class="docutils literal">get_peers</tt>
and <tt class="docutils literal">announce_peer</tt>.</p>
<p>Responses to <tt class="docutils literal">get</tt> should always include <tt class="docutils literal">nodes</tt> and <tt class="docutils literal">nodes6</tt>. Those fields
have the same semantics as in its <tt class="docutils literal">get_peers</tt> response. It should also include a write token,
<tt class="docutils literal">token</tt>, with the same semantics as int <tt class="docutils literal">get_peers</tt>. The write token MAY be tied
specifically to the key which <tt class="docutils literal">get</tt> requested. i.e. the <tt class="docutils literal">token</tt> can only be used
to store values under that one key.</p>
<p>The <tt class="docutils literal">id</tt> field in these messages has the same semantics as the standard DHT messages,
i.e. the node ID of the node sending the message, to maintain the structure of the DHT
network.</p>
<p>The <tt class="docutils literal">token</tt> field also has the same semantics as the standard DHT message <tt class="docutils literal">get_peers</tt>
and <tt class="docutils literal">announce_peer</tt>, when requesting an item and to write an item respectively.</p>
<p>The <tt class="docutils literal">k</tt> field is the 32 byte curve25519 public key, which the signature
can be authenticated with. When looking up a mutable item, the <tt class="docutils literal">target</tt> field
MUST be the SHA-1 hash of this key.</p>
<p>The distinction between storing mutable and immutable items is the inclusion
of a public key, a sequence number and signature (<tt class="docutils literal">k</tt>, <tt class="docutils literal">seq</tt> and <tt class="docutils literal">sig</tt>).</p>
<p><tt class="docutils literal">get</tt> requests for mutable items and immutable items cannot be distinguished from
eachother. An implementation can either store mutable and immutable items in the same
hash table internally, or in separate ones and potentially do two lookups for <tt class="docutils literal">get</tt>
requests.</p>
<p>The <tt class="docutils literal">v</tt> field is the <em>value</em> to be stored. It is allowed to be any bencoded type (list,
dict, string or integer). When it's being hashed (for verifying its signature or to calculate
its key), its flattened, bencoded, form is used. It is important to use the exact
bencoded representation as it appeared in the message. decoding and then re-encoding
bencoded structures is not necessarily an identity operation.</p>
<p>Storing nodes SHOULD reject <tt class="docutils literal">put</tt> requests where the bencoded form of <tt class="docutils literal">v</tt> is longer
than 1000 bytes.</p>
</div>
<div class="section" id="immutable-items">
<h1>immutable items</h1>
<p>Immutable items are stored under their SHA-1 hash, and since they cannot be modified,
there is no need to authenticate the origin of them. This makes immutable items simple.</p>
<p>A node making a lookup SHOULD verify the data it receives from the network, to verify
that its hash matches the target that was looked up.</p>
<p>The proposed new messages <tt class="docutils literal">get</tt> and <tt class="docutils literal">put</tt> are similar to the existing
<tt class="docutils literal">get_peers</tt> and <tt class="docutils literal">announce_peer</tt>.</p>
<p>Responses to <tt class="docutils literal">get</tt> should always include <tt class="docutils literal">nodes</tt> and <tt class="docutils literal">nodes6</tt>. Those
fields have the same semantics as in its <tt class="docutils literal">get_peers</tt> response. It should also
include a write token, <tt class="docutils literal">token</tt>, with the same semantics as int <tt class="docutils literal">get_peers</tt>.
The write token MAY be tied specifically to the key which <tt class="docutils literal">get</tt> requested.
i.e. the <tt class="docutils literal">token</tt> can only be used to store values under that one key. It may
also be tied to the node ID and IP address of the requesting node.</p>
<p>The <tt class="docutils literal">id</tt> field in these messages has the same semantics as the standard DHT
messages, i.e. the node ID of the node sending the message, to maintain the
structure of the DHT network.</p>
<p>The <tt class="docutils literal">token</tt> field also has the same semantics as the standard DHT message
<tt class="docutils literal">get_peers</tt> and <tt class="docutils literal">announce_peer</tt>, when requesting an item and to write an
item respectively.</p>
<p>The <tt class="docutils literal">k</tt> field is the 32 byte curve25519 public key, which the signature can be
authenticated with. When looking up a mutable item, the <tt class="docutils literal">target</tt> field MUST be
the SHA-1 hash of this key concatenated with the <tt class="docutils literal">salt</tt>, if present.</p>
<p>The distinction between storing mutable and immutable items is the inclusion of
a public key, a sequence number, signature and an optional salt (<tt class="docutils literal">k</tt>, <tt class="docutils literal">seq</tt>,
<tt class="docutils literal">sig</tt> and <tt class="docutils literal">salt</tt>).</p>
<p><tt class="docutils literal">get</tt> requests for mutable items and immutable items cannot be distinguished
from eachother. An implementation can either store mutable and immutable items
in the same hash table internally, or in separate ones and potentially do two
lookups for <tt class="docutils literal">get</tt> requests.</p>
<p>The <tt class="docutils literal">v</tt> field is the <em>value</em> to be stored. It is allowed to be any bencoded
type (list, dict, string or integer). When it's being hashed (for verifying its
signature or to calculate its key), its flattened, bencoded, form is used. It is
important to use the verbatim bencoded representation as it appeared in the
message. decoding and then re-encoding bencoded structures is not necessarily an
identity operation.</p>
<p>Storing nodes MAY reject <tt class="docutils literal">put</tt> requests where the bencoded form of <tt class="docutils literal">v</tt> is
longer than 1000 bytes. In other words, it's not safe to assume storing more
than 1000 bytes will succeed.</p>
<p>immutable items ---------------</p>
<p>Immutable items are stored under their SHA-1 hash, and since they cannot be
modified, there is no need to authenticate the origin of them. This makes
immutable items simple.</p>
<p>A node making a lookup SHOULD verify the data it receives from the network, to
verify that its hash matches the target that was looked up.</p>
<div class="section" id="put-message">
<h2>put message</h2>
<p>Request:</p>
@ -182,17 +186,32 @@ that its hash matches the target that was looked up.</p>
<div class="section" id="mutable-items">
<h1>mutable items</h1>
<p>Mutable items can be updated, without changing their DHT keys. To authenticate
that only the original publisher can update an item, it is signed by a private key
generated by the original publisher. The target ID mutable items are stored under
is the SHA-1 hash of the public key (as it appears in the <tt class="docutils literal">put</tt> message).</p>
that only the original publisher can update an item, it is signed by a private
key generated by the original publisher. The target ID mutable items are stored
under is the SHA-1 hash of the public key (as it appears in the <tt class="docutils literal">put</tt>
message).</p>
<p>In order to avoid a malicious node to overwrite the list head with an old
version, the sequence number <tt class="docutils literal">seq</tt> must be monotonically increasing for each update,
and a node hosting the list node MUST not downgrade a list head from a higher sequence
number to a lower one, only upgrade. The sequence number SHOULD not exceed <tt class="docutils literal">MAX_INT64</tt>,
(i.e. <tt class="docutils literal">0x7fffffffffffffff</tt>. A client MAY reject any message with a sequence number
exceeding this. A client MAY also reject any message with a negative sequence number.</p>
<p>The signature is a 64 byte curve25519 signature of the bencoded sequence
number concatenated with the <tt class="docutils literal">v</tt> key. e.g. something like this:: <tt class="docutils literal">3:seqi4e1:v12:Hello world!</tt>.</p>
version, the sequence number <tt class="docutils literal">seq</tt> must be monotonically increasing for each
update, and a node hosting the list node MUST not downgrade a list head from a
higher sequence number to a lower one, only upgrade. The sequence number SHOULD
not exceed <tt class="docutils literal">MAX_INT64</tt>, (i.e. <tt class="docutils literal">0x7fffffffffffffff</tt>. A client MAY reject any
message with a sequence number exceeding this. A client MAY also reject any
message with a negative sequence number.</p>
<p>The signature is a 64 byte curve25519 signature of the bencoded sequence number
concatenated with the <tt class="docutils literal">v</tt> key. e.g. something like this:</p>
<pre class="literal-block">
3:seqi4e1:v12:Hello world!
</pre>
<p>If the <tt class="docutils literal">salt</tt> key is present and non-empty, the salt string must be included
in what's signed. Note that if <tt class="docutils literal">salt</tt> is specified and an empty string, it is
as if it was not specified and nothing in addition to the sequence number and
the data is signed.</p>
<p>When a salt is included in what is signed, the key <tt class="docutils literal">salt</tt> with the value of
the key is prepended in its bencoded form. For example, if <tt class="docutils literal">salt</tt> is &quot;foobar&quot;,
the buffer to be signed is:</p>
<pre class="literal-block">
4:salt6:foobar3:seqi4e1:v12:Hello world!
</pre>
<div class="section" id="id1">
<h2>put message</h2>
<p>Request:</p>
@ -203,6 +222,7 @@ number concatenated with the <tt class="docutils literal">v</tt> key. e.g. somet
&quot;cas&quot;: <em>&lt;optional 20 byte hash (string)&gt;</em>,
&quot;id&quot;: <em>&lt;20 byte id of sending node (string)&gt;</em>,
&quot;k&quot;: <em>&lt;curve25519 public key (32 bytes string)&gt;</em>,
&quot;salt&quot;: <em>&lt;optional salt to be appended to &quot;k&quot; when hashing (string)&gt;</em>
&quot;seq&quot;: <em>&lt;monotonically increasing sequence number (integer)&gt;</em>,
&quot;sig&quot;: <em>&lt;curve25519 signature (64 bytes string)&gt;</em>,
&quot;token&quot;: <em>&lt;write-token (string)&gt;</em>,
@ -215,22 +235,36 @@ number concatenated with the <tt class="docutils literal">v</tt> key. e.g. somet
</pre>
<p>Storing nodes receiving a <tt class="docutils literal">put</tt> request where <tt class="docutils literal">seq</tt> is lower than or equal
to what's already stored on the node, MUST reject the request. If the sequence
number is equal, and the value is also the same, the node SHOULD reset its timeout
counter.</p>
number is equal, and the value is also the same, the node SHOULD reset its
timeout counter.</p>
<p>If the sequence number in the <tt class="docutils literal">put</tt> message is lower than the sequence number
associated with the currently stored value, the storing node MAY return an error
message with code 302 (see error codes below).</p>
<p>Note that this request does not contain a target hash. The target hash under
which this blob is stored is implied by the <tt class="docutils literal">k</tt> argument. The key is
the SHA-1 hash of the key (<tt class="docutils literal">k</tt>).</p>
which this blob is stored is implied by the <tt class="docutils literal">k</tt> argument. The key is the SHA-1
hash of the key (<tt class="docutils literal">k</tt>).</p>
<p>In order to support a single key being used to store separate items in the DHT,
an optional <tt class="docutils literal">salt</tt> can be specified in the <tt class="docutils literal">put</tt> request of mutable items.
If the salt entry is not present, it can be assumed to be an empty string, and
its semantics should be identical as specifying a salt key with an empty string.
The salt can be any binary string (but probably most conveniently a hash of
something). This string is appended to the key, as specified in the <tt class="docutils literal">k</tt> field,
when calculating the key to store the blob under (i.e. the key <tt class="docutils literal">get</tt> requests
specify to retrieve this data).</p>
<p>This lets a single entity, with a single key, publish any number of unrelated
items, with a single key that readers can verify. This is useful if the
publisher doesn't know ahead of time how many different items are to be
published. It can distribute a single public key for users to authenticate the
published blobs.</p>
<p>The <tt class="docutils literal">cas</tt> field is optional. If present it is interpreted as the sha-1 hash of
the sequence number and <tt class="docutils literal">v</tt> field that is expected to be replaced. The buffer
to hash is the same as the one signed when storing. <tt class="docutils literal">cas</tt> is short for <em>compare
and swap</em>, it has similar semantics as CAS CPU instructions. If specified as part
of the put command, and the current value stored under the public key differs from
the expected value, the store fails. The <tt class="docutils literal">cas</tt> field only applies to mutable puts.
If there is no current value, the <tt class="docutils literal">cas</tt> field SHOULD be ignored, not preventing
the put based on it.</p>
the sequence number, <tt class="docutils literal">v</tt> field and possibly the <tt class="docutils literal">salt</tt> field, that is
expected to be replaced. The buffer to hash is the same as the one signed when
storing. <tt class="docutils literal">cas</tt> is short for <em>compare and swap</em>, it has similar semantics as
CAS CPU instructions. If specified as part of the put command, and the current
value stored under the public key differs from the expected value, the store
fails. The <tt class="docutils literal">cas</tt> field only applies to mutable puts. If there is no current
value, the <tt class="docutils literal">cas</tt> field SHOULD be ignored. A put operation should not be
prevented based on the <tt class="docutils literal">cas</tt> field if no value is currently present.</p>
<p>Response:</p>
<pre class="literal-block">
{
@ -239,11 +273,13 @@ the put based on it.</p>
&quot;y&quot;: &quot;r&quot;,
}
</pre>
<p>If the store fails for any reason an error message is returned instead of the message
template above, i.e. one where &quot;y&quot; is &quot;e&quot; and &quot;e&quot; is a tuple of [error-code, message]).
Failures include where the <tt class="docutils literal">cas</tt> hash mismatches and the sequence number is outdated.</p>
<p>If no <tt class="docutils literal">cas</tt> field is included in the <tt class="docutils literal">put</tt> message, the value of the current <tt class="docutils literal">v</tt>
field should be disregarded when determining whether or not to save the item.</p>
<p>If the store fails for any reason an error message is returned instead of the
message template above, i.e. one where &quot;y&quot; is &quot;e&quot; and &quot;e&quot; is a tuple of
[error-code, message]). Failures include where the <tt class="docutils literal">cas</tt> hash mismatches and
the sequence number is outdated.</p>
<p>If no <tt class="docutils literal">cas</tt> field is included in the <tt class="docutils literal">put</tt> message, the value of the current
<tt class="docutils literal">v</tt> field should be disregarded when determining whether or not to save the
item. (However, the signature, sequence number obviously still should).</p>
<p>The error message (as specified by <a class="reference external" href="http://www.bittorrent.org/beps/bep_0005.html">BEP5</a>) looks like this:</p>
<pre class="literal-block">
{
@ -283,8 +319,9 @@ current.</td>
</tr>
</tbody>
</table>
<p>An implementation MUST emit 301 errors if the cas-hash mismatches. This is
a critical feature in synchronization of multiple agents sharing an immutable item.</p>
<p>An implementation MUST emit 301 errors if the cas-hash mismatches. This is a
critical feature in synchronization of multiple agents sharing an immutable
item.</p>
</div>
<div class="section" id="id2">
<h2>get message</h2>
@ -294,7 +331,7 @@ a critical feature in synchronization of multiple agents sharing an immutable it
&quot;a&quot;:
{
&quot;id&quot;: <em>&lt;20 byte id of sending node (string)&gt;</em>,
&quot;target:&quot; <em>&lt;20 byte SHA-1 hash of public key (string)&gt;</em>
&quot;target:&quot; <em>&lt;20 byte SHA-1 hash of public key and salt (string)&gt;</em>
},
&quot;t&quot;: <em>&lt;transaction-id (string)&gt;</em>,
&quot;y&quot;: &quot;q&quot;,
@ -310,6 +347,7 @@ a critical feature in synchronization of multiple agents sharing an immutable it
&quot;k&quot;: <em>&lt;curve25519 public key (32 bytes string)&gt;</em>,
&quot;nodes&quot;: <em>&lt;IPv4 nodes close to 'target'&gt;</em>,
&quot;nodes6&quot;: <em>&lt;IPv6 nodes close to 'target'&gt;</em>,
&quot;salt&quot;: <em>&lt;optional salt to be appended to &quot;k&quot; when hashing (string)&gt;</em>
&quot;seq&quot;: <em>&lt;monotonically increasing sequence number (integer)&gt;</em>,
&quot;sig&quot;: <em>&lt;curve25519 signature (64 bytes string)&gt;</em>,
&quot;token&quot;: <em>&lt;write-token (string)&gt;</em>,
@ -323,33 +361,65 @@ a critical feature in synchronization of multiple agents sharing an immutable it
</div>
<div class="section" id="signature-verification">
<h1>signature verification</h1>
<p>The signature, private and public keys are in the format as produced by this derivative <a class="reference external" href="https://github.com/nightcracker/ed25519">ed25519 library</a>.</p>
<p>In order to make it maximally difficult to attack the bencoding parser, signing and verification of the
value and sequence number should be done as follows:</p>
<p>In order to make it maximally difficult to attack the bencoding parser, signing
and verification of the value and sequence number should be done as follows:</p>
<ol class="arabic simple">
<li>encode value and sequence number separately</li>
<li>concatenate <em>&quot;3:seqi&quot;</em> <tt class="docutils literal">seq</tt> <em>&quot;e1:v&quot;</em> <tt class="docutils literal">len</tt> <em>&quot;:&quot;</em> and the encoded value.
sequence number 1 of value &quot;Hello World!&quot; would be converted to: &quot;3:seqi1e1:v12:Hello World!&quot;
In this way it is not possible to convince a node that part of the length is actually part of the
sequence number even if the parser contains certain bugs. Furthermore it is not possible to have a
verification failure if a bencoding serializer alters the order of entries in the dictionary.</li>
<li>concatenate (&quot;4:salt&quot; <em>length-of-salt</em> &quot;:&quot; <em>salt</em>) &quot;3:seqi&quot; <em>seq</em>
&quot;e1:v&quot; <em>len</em> &quot;:&quot; and the encoded value.
sequence number 1 of value &quot;Hello World!&quot; would be converted to:
&quot;3:seqi1e1:v12:Hello World!&quot;. In this way it is not possible to convince a
node that part of the length is actually part of the sequence number even if
the parser contains certain bugs. Furthermore it is not possible to have a
verification failure if a bencoding serializer alters the order of entries in
the dictionary. The salt is in parenthesis because it is optional. It is only
prepended if a non-empty salt is specified in the <tt class="docutils literal">put</tt> request.</li>
<li>sign or verify the concatenated string</li>
</ol>
<p>On the storage node, the signature MUST be verified before accepting the store command. The data
MUST be stored under the SHA-1 hash of the public key (as it appears in the bencoded dict).</p>
<p>On the requesting nodes, the key they get back from a <tt class="docutils literal">get</tt> request MUST be verified to hash
to the target ID the lookup was made for, as well as verifying the signature. If any of these fail,
the response SHOULD be considered invalid.</p>
<p>On the storage node, the signature MUST be verified before accepting the store
command. The data MUST be stored under the SHA-1 hash of the public key (as it
appears in the bencoded dict).</p>
<p>On the requesting nodes, the key they get back from a <tt class="docutils literal">get</tt> request MUST be
verified to hash to the target ID the lookup was made for, as well as verifying
the signature. If any of these fail, the response SHOULD be considered invalid.</p>
</div>
<div class="section" id="expiration">
<h1>expiration</h1>
<p>Without re-announcement, these items MAY expire in 2 hours. In order
to keep items alive, they SHOULD be re-announced once an hour.</p>
<p>Any node that's interested in keeping a blob in the DHT alive may announce it. It would simply
repeat the signature for a mutable put without having the private key.</p>
<p>Any node that's interested in keeping a blob in the DHT alive may announce it.
It would simply repeat the signature for a mutable put without having the
private key.</p>
</div>
<div class="section" id="test-vectors">
<h1>test vectors</h1>
<div class="section" id="test-vector">
<h1>test vector</h1>
<p>The buffer being signed:</p>
<pre class="literal-block">
3:seqi1e1:v12:Hello World!
</pre>
<p>public key:</p>
<pre class="literal-block">
77ff84905a91936367c01360803104f92432fcd904a43511876df5cdf3e7e548
</pre>
<p>private key:</p>
<pre class="literal-block">
e06d3183d14159228433ed599221b80bd0a5ce8352e4bdf0262f76786ef1c74d
b7e7a9fea2c0eb269d61e3b38e450a22e754941ac78479d6c54e1faf6037881d
</pre>
<p>signature:</p>
<pre class="literal-block">
305ac8aeb6c9c151fa120f120ea2cfb923564e11552d06a5d856091e5e853cff
1260d3f39e4999684aa92eb73ffd136e6f4f3ecbfda0ce53a1608ecd7ae21f01
</pre>
</div>
<div class="section" id="resources">
<h1>resources</h1>
<p>Libraries that implement curve25519 DSA:</p>
<ul class="simple">
<li><a class="reference external" href="http://nacl.cr.yp.to/">NaCl</a></li>
<li><a class="reference external" href="https://github.com/jedisct1/libsodium">libsodium</a></li>
<li><a class="reference external" href="https://github.com/nightcracker/ed25519">nightcracker's ed25519</a></li>
</ul>
</div>
</div>
<div id="footer">

435
docs/manual-ref.html
View File

@ -55,17 +55,17 @@
<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of contents</p>
<ul class="simple">
<li><a class="reference internal" href="#overview" id="id54">overview</a></li>
<li><a class="reference internal" href="#things-to-keep-in-mind" id="id55">things to keep in mind</a></li>
<li><a class="reference internal" href="#network-primitives" id="id56">network primitives</a></li>
<li><a class="reference internal" href="#exceptions" id="id57">exceptions</a></li>
<li><a class="reference internal" href="#magnet-links" id="id58">magnet links</a></li>
<li><a class="reference internal" href="#queuing" id="id59">queuing</a></li>
<li><a class="reference internal" href="#fast-resume" id="id60">fast resume</a></li>
<li><a class="reference internal" href="#storage-allocation" id="id61">storage allocation</a></li>
<li><a class="reference internal" href="#extensions" id="id62">extensions</a></li>
<li><a class="reference internal" href="#piece-picker" id="id63">piece picker</a></li>
<li><a class="reference internal" href="#ssl-torrents" id="id64">SSL torrents</a></li>
<li><a class="reference internal" href="#overview" id="id52">overview</a></li>
<li><a class="reference internal" href="#things-to-keep-in-mind" id="id53">things to keep in mind</a></li>
<li><a class="reference internal" href="#network-primitives" id="id54">network primitives</a></li>
<li><a class="reference internal" href="#exceptions" id="id55">exceptions</a></li>
<li><a class="reference internal" href="#magnet-links" id="id56">magnet links</a></li>
<li><a class="reference internal" href="#queuing" id="id57">queuing</a></li>
<li><a class="reference internal" href="#fast-resume" id="id58">fast resume</a></li>
<li><a class="reference internal" href="#storage-allocation" id="id59">storage allocation</a></li>
<li><a class="reference internal" href="#extensions" id="id60">extensions</a></li>
<li><a class="reference internal" href="#piece-picker" id="id61">piece picker</a></li>
<li><a class="reference internal" href="#ssl-torrents" id="id62">SSL torrents</a></li>
</ul>
</div>
<div class="section" id="overview">
@ -80,9 +80,11 @@ the <tt class="docutils literal">session</tt>, it contains the main loop that se
</li>
<li><p class="first">start extensions (see <a class="reference external" href="reference-Core.html#add_extension()">add_extension()</a>).</p>
</li>
<li><p class="first">start DHT, LSD, UPnP, NAT-PMP etc (see <a class="reference external" href="reference-Session.html#start_dht()">start_dht()</a>, <a class="reference external" href="reference-Session.html#start_lsd()">start_lsd()</a>, <a class="reference external" href="reference-Session.html#start_upnp()">start_upnp()</a> and <a class="reference external" href="reference-Session.html#start_natpmp()">start_natpmp()</a>).</p>
<li><p class="first">start DHT, LSD, UPnP, NAT-PMP etc (see <a class="reference external" href="reference-Session.html#start_dht()">start_dht()</a>, <a class="reference external" href="reference-Session.html#start_lsd()">start_lsd()</a>, <a class="reference external" href="reference-Session.html#start_upnp()">start_upnp()</a>
and <a class="reference external" href="reference-Session.html#start_natpmp()">start_natpmp()</a>).</p>
</li>
<li><p class="first">parse .torrent-files and add them to the <a class="reference external" href="reference-Session.html#session">session</a> (see <a class="reference external" href="reference-Core.html#torrent_info">torrent_info</a>, <a class="reference external" href="reference-Session.html#async_add_torrent()">async_add_torrent()</a> and <a class="reference external" href="reference-Session.html#add_torrent()">add_torrent()</a>)</p>
<li><p class="first">parse .torrent-files and add them to the <a class="reference external" href="reference-Session.html#session">session</a> (see <a class="reference external" href="reference-Core.html#torrent_info">torrent_info</a>,
<a class="reference external" href="reference-Session.html#async_add_torrent()">async_add_torrent()</a> and <a class="reference external" href="reference-Session.html#add_torrent()">add_torrent()</a>)</p>
</li>
<li><p class="first">main loop (see <a class="reference external" href="reference-Session.html#session">session</a>)</p>
<blockquote>
@ -219,42 +221,45 @@ download, without any .torrent file.</p>
<p>libtorrent supports <em>queuing</em>. Which means it makes sure that a limited number of
torrents are being downloaded at any given time, and once a torrent is completely
downloaded, the next in line is started.</p>
<p>Torrents that are <em>auto managed</em> are subject to the queuing and the active torrents
limits. To make a torrent auto managed, set <tt class="docutils literal">auto_managed</tt> to true when adding the
torrent (see <a class="reference external" href="reference-Session.html#async_add_torrent()">async_add_torrent()</a> and <a class="reference external" href="reference-Session.html#add_torrent()">add_torrent()</a>).</p>
<p>Torrents that are <em>auto managed</em> are subject to the queuing and the active
torrents limits. To make a torrent auto managed, set <tt class="docutils literal">auto_managed</tt> to true
when adding the torrent (see <a class="reference external" href="reference-Session.html#async_add_torrent()">async_add_torrent()</a> and <a class="reference external" href="reference-Session.html#add_torrent()">add_torrent()</a>).</p>
<p>The limits of the number of downloading and seeding torrents are controlled via
<tt class="docutils literal">active_downloads</tt>, <tt class="docutils literal">active_seeds</tt> and <tt class="docutils literal">active_limit</tt> in <a class="reference external" href="reference-Settings.html#session_settings">session_settings</a>.
These limits takes non auto managed torrents into account as well. If there are
more non-auto managed torrents being downloaded than the <tt class="docutils literal">active_downloads</tt>
setting, any auto managed torrents will be queued until torrents are removed so
that the number drops below the limit.</p>
<tt class="docutils literal">active_downloads</tt>, <tt class="docutils literal">active_seeds</tt> and <tt class="docutils literal">active_limit</tt> in
<a class="reference external" href="reference-Settings.html#session_settings">session_settings</a>. These limits takes non auto managed torrents into account as
well. If there are more non-auto managed torrents being downloaded than the
<tt class="docutils literal">active_downloads</tt> setting, any auto managed torrents will be queued until
torrents are removed so that the number drops below the limit.</p>
<p>The default values are 8 active downloads and 5 active seeds.</p>
<p>At a regular interval, torrents are checked if there needs to be any re-ordering of
which torrents are active and which are queued. This interval can be controlled via
<tt class="docutils literal">auto_manage_interval</tt> in <a class="reference external" href="reference-Settings.html#session_settings">session_settings</a>. It defaults to every 30 seconds.</p>
<p>For queuing to work, resume data needs to be saved and restored for all torrents.
See <a class="reference external" href="reference-Core.html#save_resume_data()">save_resume_data()</a>.</p>
<p>At a regular interval, torrents are checked if there needs to be any
re-ordering of which torrents are active and which are queued. This interval
can be controlled via <tt class="docutils literal">auto_manage_interval</tt> in <a class="reference external" href="reference-Settings.html#session_settings">session_settings</a>. It defaults
to every 30 seconds.</p>
<p>For queuing to work, resume data needs to be saved and restored for all
torrents. See <a class="reference external" href="reference-Core.html#save_resume_data()">save_resume_data()</a>.</p>
<div class="section" id="downloading">
<h2>downloading</h2>
<p>Torrents that are currently being downloaded or incomplete (with bytes still to download)
are queued. The torrents in the front of the queue are started to be actively downloaded
and the rest are ordered with regards to their queue position. Any newly added torrent
is placed at the end of the queue. Once a torrent is removed or turns into a seed, its
queue position is -1 and all torrents that used to be after it in the queue, decreases their
position in order to fill the gap.</p>
<p>Torrents that are currently being downloaded or incomplete (with bytes still to
download) are queued. The torrents in the front of the queue are started to be
actively downloaded and the rest are ordered with regards to their queue
position. Any newly added torrent is placed at the end of the queue. Once a
torrent is removed or turns into a seed, its queue position is -1 and all
torrents that used to be after it in the queue, decreases their position in
order to fill the gap.</p>
<p>The queue positions are always in a sequence without any gaps.</p>
<p>Lower queue position means closer to the front of the queue, and will be started sooner than
torrents with higher queue positions.</p>
<p>Lower queue position means closer to the front of the queue, and will be
started sooner than torrents with higher queue positions.</p>
<p>To query a torrent for its position in the queue, or change its position, see:
<a class="reference external" href="reference-Core.html#queue_position()">queue_position()</a>, <a class="reference external" href="reference-Core.html#queue_position_up()">queue_position_up()</a>, <a class="reference external" href="reference-Core.html#queue_position_down()">queue_position_down()</a>, <a class="reference external" href="reference-Core.html#queue_position_top()">queue_position_top()</a> and <a class="reference external" href="reference-Core.html#queue_position_bottom()">queue_position_bottom()</a>.</p>
<a class="reference external" href="reference-Core.html#queue_position()">queue_position()</a>, <a class="reference external" href="reference-Core.html#queue_position_up()">queue_position_up()</a>, <a class="reference external" href="reference-Core.html#queue_position_down()">queue_position_down()</a>,
<a class="reference external" href="reference-Core.html#queue_position_top()">queue_position_top()</a> and <a class="reference external" href="reference-Core.html#queue_position_bottom()">queue_position_bottom()</a>.</p>
</div>
<div class="section" id="seeding">
<h2>seeding</h2>
<p>Auto managed seeding torrents are rotated, so that all of them are allocated a fair
amount of seeding. Torrents with fewer completed <em>seed cycles</em> are prioritized for
seeding. A seed cycle is completed when a torrent meets either the share ratio limit
(uploaded bytes / downloaded bytes), the share time ratio (time seeding / time
downloaing) or seed time limit (time seeded).</p>
<p>Auto managed seeding torrents are rotated, so that all of them are allocated a
fair amount of seeding. Torrents with fewer completed <em>seed cycles</em> are
prioritized for seeding. A seed cycle is completed when a torrent meets either
the share ratio limit (uploaded bytes / downloaded bytes), the share time ratio
(time seeding / time downloaing) or seed time limit (time seeded).</p>
<p>The relevant settings to control these limits are <tt class="docutils literal">share_ratio_limit</tt>,
<tt class="docutils literal">seed_time_ratio_limit</tt> and <tt class="docutils literal">seed_time_limit</tt> in <a class="reference external" href="reference-Settings.html#session_settings">session_settings</a>.</p>
</div>
@ -269,10 +274,11 @@ will not check the piece hashes then, and rely on the information given in the
fast-resume data. The fast-resume data also contains information about which
blocks, in the unfinished pieces, were downloaded, so it will not have to
start from scratch on the partially downloaded pieces.</p>
<p>To use the fast-resume data you simply give it to <a class="reference external" href="reference-Session.html#async_add_torrent()">async_add_torrent()</a> and <a class="reference external" href="reference-Session.html#add_torrent()">add_torrent()</a>, and it
will skip the time consuming checks. It may have to do the checking anyway, if
the fast-resume data is corrupt or doesn't fit the storage for that torrent,
then it will not trust the fast-resume data and just do the checking.</p>
<p>To use the fast-resume data you simply give it to <a class="reference external" href="reference-Session.html#async_add_torrent()">async_add_torrent()</a> and
<a class="reference external" href="reference-Session.html#add_torrent()">add_torrent()</a>, and it will skip the time consuming checks. It may have to do
the checking anyway, if the fast-resume data is corrupt or doesn't fit the
storage for that torrent, then it will not trust the fast-resume data and just
do the checking.</p>
<div class="section" id="file-format">
<h2>file format</h2>
<p>The file format is a bencoded dictionary containing the following fields:</p>
@ -479,30 +485,28 @@ last resume data checkpoint.</td>
<h1>storage allocation</h1>
<p>There are two modes in which storage (files on disk) are allocated in libtorrent.</p>
<ol class="arabic simple">
<li>The traditional <em>full allocation</em> mode, where the entire files are filled up with
zeros before anything is downloaded. libtorrent will look for sparse files support
in the filesystem that is used for storage, and use sparse files or file system
zero fill support if present. This means that on NTFS, full allocation mode will
only allocate storage for the downloaded pieces.</li>
<li>The <em>sparse allocation</em>, sparse files are used, and pieces are downloaded directly
to where they belong. This is the recommended (and default) mode.</li>
<li>The traditional <em>full allocation</em> mode, where the entire files are filled up
with zeros before anything is downloaded. Files are allocated on demand, the
first time anything is written to them. The main benefit of this mode is that
it avoids creating heavily fragmented files.</li>
<li>The <em>sparse allocation</em>, sparse files are used, and pieces are downloaded
directly to where they belong. This is the recommended (and default) mode.</li>
</ol>
<p>In previous versions of libtorrent, a 3rd mode was supported, <em>compact allocation</em>.
Support for this is deprecated and will be removed in future versions of libtorrent.
It's still described in here for completeness.</p>
<p>In previous versions of libtorrent, a 3rd mode was supported, <em>compact
allocation</em>. Support for this is deprecated and will be removed in future
versions of libtorrent. It's still described in here for completeness.</p>
<p>The allocation mode is selected when a torrent is started. It is passed as an
argument to <a class="reference external" href="reference-Session.html#add_torrent()">session::add_torrent()</a> or <a class="reference external" href="reference-Session.html#async_add_torrent()">session::async_add_torrent()</a>.</p>
<p>The decision to use full allocation or compact allocation typically depends on whether
any files have priority 0 and if the filesystem supports sparse files.</p>
<p>The decision to use full allocation or compact allocation typically depends on
whether any files have priority 0 and if the filesystem supports sparse files.</p>
<div class="section" id="sparse-allocation">
<h2>sparse allocation</h2>
<p>On filesystems that supports sparse files, this allocation mode will only use
as much space as has been downloaded.</p>
<p>The main drawback of this mode is that it may create heavily fragmented files.</p>
<blockquote>
<ul class="simple">
<li>It does not require an allocation pass on startup.</li>
<li>It supports skipping files (setting prioirty to 0 to not download).</li>
<li>Fast resume data will remain valid even when file time stamps are out of date.</li>
</ul>
</blockquote>
</div>
@ -510,31 +514,25 @@ as much space as has been downloaded.</p>
<h2>full allocation</h2>
<p>When a torrent is started in full allocation mode, the disk-io thread
will make sure that the entire storage is allocated, and fill any gaps with zeros.
This will be skipped if the filesystem supports sparse files or automatic zero filling.
It will of course still check for existing pieces and fast resume data. The main
drawbacks of this mode are:</p>
<blockquote>
<ul class="simple">
<li>It may take longer to start the torrent, since it will need to fill the files
with zeros on some systems. This delay is linearly dependent on the size of
the download.</li>
<li>The download may occupy unnecessary disk space between download sessions. In case
sparse files are not supported.</li>
<li>Disk caches usually perform extremely poorly with random access to large files
and may slow down a download considerably.</li>
with zeroes. This delay is linear to the size of the download.</li>
<li>The download may occupy unnecessary disk space between download sessions.</li>
<li>Disk caches usually perform poorly with random access to large files
and may slow down the download some.</li>
</ul>
</blockquote>
<p>The benefits of this mode are:</p>
<blockquote>
<ul class="simple">
<li>Downloaded pieces are written directly to their final place in the files and the
total number of disk operations will be fewer and may also play nicer to
<li>Downloaded pieces are written directly to their final place in the files and
the total number of disk operations will be fewer and may also play nicer to
filesystems' file allocation, and reduce fragmentation.</li>
<li>No risk of a download failing because of a full disk during download. Unless
sparse files are being used.</li>
<li>The fast resume data will be more likely to be usable, regardless of crashes or
out of date data, since pieces won't move around.</li>
<li>Can be used with prioritizing files to 0.</li>
<li>No risk of a download failing because of a full disk during download, once
all files have been created.</li>
</ul>
</blockquote>
</div>
@ -545,10 +543,11 @@ out of date data, since pieces won't move around.</li>
<p class="last">Note that support for compact allocation is deprecated in libttorrent, and will
be removed in future versions.</p>
</div>
<p>The compact allocation will only allocate as much storage as it needs to keep the
pieces downloaded so far. This means that pieces will be moved around to be placed
at their final position in the files while downloading (to make sure the completed
download has all its pieces in the correct place). So, the main drawbacks are:</p>
<p>The compact allocation will only allocate as much storage as it needs to keep
the pieces downloaded so far. This means that pieces will be moved around to be
placed at their final position in the files while downloading (to make sure the
completed download has all its pieces in the correct place). So, the main
drawbacks are:</p>
<blockquote>
<ul class="simple">
<li>More disk operations while downloading since pieces are moved around.</li>
@ -561,13 +560,13 @@ download has all its pieces in the correct place). So, the main drawbacks are:</
<ul class="simple">
<li>No startup delay, since the files don't need allocating.</li>
<li>The download will not use unnecessary disk space.</li>
<li>Disk caches perform much better than in full allocation and raises the download
speed limit imposed by the disk.</li>
<li>Disk caches perform much better than in full allocation and raises the
download speed limit imposed by the disk.</li>
<li>Works well on filesystems that don't support sparse files.</li>
</ul>
</blockquote>
<p>The algorithm that is used when allocating pieces and slots isn't very complicated.
For the interested, a description follows.</p>
<p>The algorithm that is used when allocating pieces and slots isn't very
complicated. For the interested, a description follows.</p>
<p>storing a piece:</p>
<ol class="arabic simple">
<li>let <strong>A</strong> be a newly downloaded piece, with index <strong>n</strong>.</li>
@ -594,10 +593,10 @@ contain any piece), return that slot index.</li>
</div>
<div class="section" id="extensions">
<h1>extensions</h1>
<p>These extensions all operates within the <a class="reference external" href="extension_protocol.html">extension protocol</a>. The
name of the extension is the name used in the extension-list packets,
and the payload is the data in the extended message (not counting the
length-prefix, message-id nor extension-id).</p>
<p>These extensions all operates within the <a class="reference external" href="extension_protocol.html">extension protocol</a>. The name of the
extension is the name used in the extension-list packets, and the payload is
the data in the extended message (not counting the length-prefix, message-id
nor extension-id).</p>
<p>Note that since this protocol relies on one of the reserved bits in the
handshake, it may be incompatible with future versions of the mainline
bittorrent client.</p>
@ -605,16 +604,16 @@ bittorrent client.</p>
<div class="section" id="metadata-from-peers">
<h2>metadata from peers</h2>
<p>Extension name: &quot;LT_metadata&quot;</p>
<p>This extension is deprecated in favor of the more widely supported <tt class="docutils literal">ut_metadata</tt>
extension, see <a class="reference external" href="http://bittorrent.org/beps/bep_0009.html">BEP 9</a>.
The point with this extension is that you don't have to distribute the
metadata (.torrent-file) separately. The metadata can be distributed
through the bittorrent swarm. The only thing you need to download such
a torrent is the tracker url and the info-hash of the torrent.</p>
<p>It works by assuming that the initial seeder has the metadata and that
the metadata will propagate through the network as more peers join.</p>
<p>There are three kinds of messages in the metadata extension. These packets
are put as payload to the extension message. The three packets are:</p>
<p>This extension is deprecated in favor of the more widely supported
<tt class="docutils literal">ut_metadata</tt> extension, see <a class="reference external" href="http://bittorrent.org/beps/bep_0009.html">BEP 9</a>. The point with this extension is that
you don't have to distribute the metadata (.torrent-file) separately. The
metadata can be distributed through the bittorrent swarm. The only thing you
need to download such a torrent is the tracker url and the info-hash of the
torrent.</p>
<p>It works by assuming that the initial seeder has the metadata and that the
metadata will propagate through the network as more peers join.</p>
<p>There are three kinds of messages in the metadata extension. These packets are
put as payload to the extension message. The three packets are:</p>
<blockquote>
<ul class="simple">
<li>request metadata</li>
@ -726,12 +725,13 @@ doesn't have any metadata.</td>
<div class="section" id="dont-have">
<h2>dont_have</h2>
<p>Extension name: &quot;lt_dont_have&quot;</p>
<p>The <tt class="docutils literal">dont_have</tt> extension message is used to tell peers that the client no longer
has a specific piece. The extension message should be advertised in the <tt class="docutils literal">m</tt> dictionary
as <tt class="docutils literal">lt_dont_have</tt>. The message format mimics the regular <tt class="docutils literal">HAVE</tt> bittorrent message.</p>
<p>Just like all extension messages, the first 2 bytes in the mssage itself are 20 (the
bittorrent extension message) and the message ID assigned to this extension in the <tt class="docutils literal">m</tt>
dictionary in the handshake.</p>
<p>The <tt class="docutils literal">dont_have</tt> extension message is used to tell peers that the client no
longer has a specific piece. The extension message should be advertised in the
<tt class="docutils literal">m</tt> dictionary as <tt class="docutils literal">lt_dont_have</tt>. The message format mimics the regular
<tt class="docutils literal">HAVE</tt> bittorrent message.</p>
<p>Just like all extension messages, the first 2 bytes in the mssage itself are 20
(the bittorrent extension message) and the message ID assigned to this
extension in the <tt class="docutils literal">m</tt> dictionary in the handshake.</p>
<table border="1" class="docutils">
<colgroup>
<col width="17%" />
@ -752,24 +752,24 @@ has.</td>
</tr>
</tbody>
</table>
<p>The length of this message (including the extension message prefix) is
6 bytes, i.e. one byte longer than the normal <tt class="docutils literal">HAVE</tt> message, because
of the extension message wrapping.</p>
<p>The length of this message (including the extension message prefix) is 6 bytes,
i.e. one byte longer than the normal <tt class="docutils literal">HAVE</tt> message, because of the extension
message wrapping.</p>
</div>
<div class="section" id="http-seeding">
<h2>HTTP seeding</h2>
<p>There are two kinds of HTTP seeding. One with that assumes a smart
(and polite) client and one that assumes a smart server. These
are specified in <a class="reference external" href="http://bittorrent.org/beps/bep_0019.html">BEP 19</a> and <a class="reference external" href="http://bittorrent.org/beps/bep_0017.html">BEP 17</a> respectively.</p>
<p>libtorrent supports both. In the libtorrent source code and API,
BEP 19 urls are typically referred to as <em>url seeds</em> and BEP 17
urls are typically referred to as <em>HTTP seeds</em>.</p>
<p>The libtorrent implementation of <a class="reference external" href="http://bittorrent.org/beps/bep_0019.html">BEP 19</a> assumes that, if the URL ends with a slash
('/'), the filename should be appended to it in order to request pieces from
that file. The way this works is that if the torrent is a single-file torrent,
only that filename is appended. If the torrent is a multi-file torrent, the
torrent's name '/' the file name is appended. This is the same directory
structure that libtorrent will download torrents into.</p>
<p>There are two kinds of HTTP seeding. One with that assumes a smart (and polite)
client and one that assumes a smart server. These are specified in <a class="reference external" href="http://bittorrent.org/beps/bep_0019.html">BEP 19</a>
and <a class="reference external" href="http://bittorrent.org/beps/bep_0017.html">BEP 17</a> respectively.</p>
<p>libtorrent supports both. In the libtorrent source code and API, BEP 19 urls
are typically referred to as <em>url seeds</em> and BEP 17 urls are typically referred
to as <em>HTTP seeds</em>.</p>
<p>The libtorrent implementation of <a class="reference external" href="http://bittorrent.org/beps/bep_0019.html">BEP 19</a> assumes that, if the URL ends with a
slash ('/'), the filename should be appended to it in order to request pieces
from that file. The way this works is that if the torrent is a single-file
torrent, only that filename is appended. If the torrent is a multi-file
torrent, the torrent's name '/' the file name is appended. This is the same
directory structure that libtorrent will download torrents into.</p>
</div>
</div>
<div class="section" id="piece-picker">
@ -788,64 +788,60 @@ structure that libtorrent will download torrents into.</p>
</ul>
<div class="section" id="internal-representation">
<h2>internal representation</h2>
<p>It is optimized by, at all times, keeping a list of pieces ordered
by rarity, randomly shuffled within each rarity class. This list
is organized as a single vector of contigous memory in RAM, for
optimal memory locality and to eliminate heap allocations and frees
when updating rarity of pieces.</p>
<p>Expensive events, like a peer joining or leaving, are evaluated
lazily, since it's cheaper to rebuild the whole list rather than
updating every single piece in it. This means as long as no blocks
are picked, peers joining and leaving is no more costly than a single
peer joining or leaving. Of course the special cases of peers that have
all or no pieces are optimized to not require rebuilding the list.</p>
<p>It is optimized by, at all times, keeping a list of pieces ordered by rarity,
randomly shuffled within each rarity class. This list is organized as a single
vector of contigous memory in RAM, for optimal memory locality and to eliminate
heap allocations and frees when updating rarity of pieces.</p>
<p>Expensive events, like a peer joining or leaving, are evaluated lazily, since
it's cheaper to rebuild the whole list rather than updating every single piece
in it. This means as long as no blocks are picked, peers joining and leaving is
no more costly than a single peer joining or leaving. Of course the special
cases of peers that have all or no pieces are optimized to not require
rebuilding the list.</p>
</div>
<div class="section" id="picker-strategy">
<h2>picker strategy</h2>
<p>The normal mode of the picker is of course <em>rarest first</em>, meaning
pieces that few peers have are preferred to be downloaded over pieces
that more peers have. This is a fundamental algorithm that is the
basis of the performance of bittorrent. However, the user may set the
piece picker into sequential download mode. This mode simply picks
pieces sequentially, always preferring lower piece indices.</p>
<p>When a torrent starts out, picking the rarest pieces means increased
risk that pieces won't be completed early (since there are only a few
peers they can be downloaded from), leading to a delay of having any
piece to offer to other peers. This lack of pieces to trade, delays
the client from getting started into the normal tit-for-tat mode of
bittorrent, and will result in a long ramp-up time. The heuristic to
mitigate this problem is to, for the first few pieces, pick random pieces
rather than rare pieces. The threshold for when to leave this initial
picker mode is determined by <a class="reference external" href="reference-Settings.html#initial_picker_threshold">session_settings::initial_picker_threshold</a>.</p>
<p>The normal mode of the picker is of course <em>rarest first</em>, meaning pieces that
few peers have are preferred to be downloaded over pieces that more peers have.
This is a fundamental algorithm that is the basis of the performance of
bittorrent. However, the user may set the piece picker into sequential download
mode. This mode simply picks pieces sequentially, always preferring lower piece
indices.</p>
<p>When a torrent starts out, picking the rarest pieces means increased risk that
pieces won't be completed early (since there are only a few peers they can be
downloaded from), leading to a delay of having any piece to offer to other
peers. This lack of pieces to trade, delays the client from getting started
into the normal tit-for-tat mode of bittorrent, and will result in a long
ramp-up time. The heuristic to mitigate this problem is to, for the first few
pieces, pick random pieces rather than rare pieces. The threshold for when to
leave this initial picker mode is determined by
session_settings::initial_picker_threshold.</p>
</div>
<div class="section" id="reverse-order">
<h2>reverse order</h2>
<p>An orthogonal setting is <em>reverse order</em>, which is used for <em>snubbed</em>
peers. Snubbed peers are peers that appear very slow, and might have timed
out a piece request. The idea behind this is to make all snubbed peers
more likely to be able to do download blocks from the same piece,
concentrating slow peers on as few pieces as possible. The reverse order
means that the most common pieces are picked, instead of the rarest pieces
(or in the case of sequential download, the last pieces, intead of the first).</p>
</div>
<div class="section" id="parole-mode">
<h2>parole mode</h2>
<p>Peers that have participated in a piece that failed the hash check, may be
put in <em>parole mode</em>. This means we prefer downloading a full piece from this
peer, in order to distinguish which peer is sending corrupt data. Whether to
do this is or not is controlled by <a class="reference external" href="reference-Settings.html#use_parole_mode">session_settings::use_parole_mode</a>.</p>
<p>An orthogonal setting is <em>reverse order</em>, which is used for <em>snubbed</em> peers.
Snubbed peers are peers that appear very slow, and might have timed out a piece
request. The idea behind this is to make all snubbed peers more likely to be
able to do download blocks from the same piece, concentrating slow peers on as
few pieces as possible. The reverse order means that the most common pieces are
picked, instead of the rarest pieces (or in the case of sequential download,
the last pieces, intead of the first).</p>
<p>parole mode -----------</p>
<p>Peers that have participated in a piece that failed the hash check, may be put
in <em>parole mode</em>. This means we prefer downloading a full piece from this
peer, in order to distinguish which peer is sending corrupt data. Whether to do
this is or not is controlled by <a class="reference external" href="reference-Settings.html#use_parole_mode">session_settings::use_parole_mode</a>.</p>
<p>In parole mode, the piece picker prefers picking one whole piece at a time for
a given peer, avoiding picking any blocks from a piece any other peer has
contributed to (since that would defeat the purpose of parole mode).</p>
</div>
<div class="section" id="prioritize-partial-pieces">
<h2>prioritize partial pieces</h2>
<p>This setting determines if partially downloaded or requested pieces should always
be preferred over other pieces. The benefit of doing this is that the number of
partial pieces is minimized (and hence the turn-around time for downloading a block
until it can be uploaded to others is minimized). It also puts less stress on the
disk cache, since fewer partial pieces need to be kept in the cache. Whether or
not to enable this is controlled by <a class="reference external" href="reference-Settings.html#prioritize_partial_pieces">session_settings::prioritize_partial_pieces</a>.</p>
<p>prioritize partial pieces -------------------------</p>
<p>This setting determines if partially downloaded or requested pieces should
always be preferred over other pieces. The benefit of doing this is that the
number of partial pieces is minimized (and hence the turn-around time for
downloading a block until it can be uploaded to others is minimized). It also
puts less stress on the disk cache, since fewer partial pieces need to be kept
in the cache. Whether or not to enable this is controlled by
session_settings::prioritize_partial_pieces.</p>
<p>The main benefit of not prioritizing partial pieces is that the rarest first
algorithm gets to have more influence on which pieces are picked. The picker is
more likely to truly pick the rarest piece, and hence improving the performance
@ -854,18 +850,15 @@ of the swarm.</p>
in the piece picker exceeds the number of peers we're connected to times 1.5.
This is in order to keep the waste of partial pieces to a minimum, but still
prefer rarest pieces.</p>
</div>
<div class="section" id="prefer-whole-pieces">
<h2>prefer whole pieces</h2>
<p>prefer whole pieces -------------------</p>
<p>The <em>prefer whole pieces</em> setting makes the piece picker prefer picking entire
pieces at a time. This is used by web connections (both http seeding
standards), in order to be able to coalesce the small bittorrent requests
to larger HTTP requests. This significantly improves performance when
downloading over HTTP.</p>
<p>It is also used by peers that are downloading faster than a certain
threshold. The main advantage is that these peers will better utilize the
other peer's disk cache, by requesting all blocks in a single piece, from
the same peer.</p>
standards), in order to be able to coalesce the small bittorrent requests to
larger HTTP requests. This significantly improves performance when downloading
over HTTP.</p>
<p>It is also used by peers that are downloading faster than a certain threshold.
The main advantage is that these peers will better utilize the other peer's
disk cache, by requesting all blocks in a single piece, from the same peer.</p>
<p>This threshold is controlled by <a class="reference external" href="reference-Settings.html#whole_pieces_threshold">session_settings::whole_pieces_threshold</a>.</p>
<p><em>TODO: piece affinity by speed category</em>
<em>TODO: piece priorities</em></p>
@ -874,8 +867,8 @@ the same peer.</p>
<div class="section" id="ssl-torrents">
<h1>SSL torrents</h1>
<p>Torrents may have an SSL root (CA) certificate embedded in them. Such torrents
are called <em>SSL torrents</em>. An SSL torrent talks to all bittorrent peers over SSL.
The protocols are layered like this:</p>
are called <em>SSL torrents</em>. An SSL torrent talks to all bittorrent peers over
SSL. The protocols are layered like this:</p>
<pre class="literal-block">
+-----------------------+
| BitTorrent protocol |
@ -887,58 +880,66 @@ The protocols are layered like this:</p>
| | UDP |
+-----------+-----------+
</pre>
<p>During the SSL handshake, both peers need to authenticate by providing a certificate
that is signed by the CA certificate found in the .torrent file. These peer
certificates are expected to be privided to peers through some other means than
bittorrent. Typically by a peer generating a certificate request which is sent to
the publisher of the torrent, and the publisher returning a signed certificate.</p>
<p>In libtorrent, <a class="reference external" href="reference-Core.html#set_ssl_certificate()">set_ssl_certificate()</a> in <a class="reference external" href="reference-Core.html#torrent_handle">torrent_handle</a> is used to tell libtorrent where
to find the peer certificate and the private key for it. When an SSL torrent is loaded,
the <a class="reference external" href="reference-Alerts.html#torrent_need_cert_alert">torrent_need_cert_alert</a> is posted to remind the user to provide a certificate.</p>
<p>A peer connecting to an SSL torrent MUST provide the <em>SNI</em> TLS extension (server name
indication). The server name is the hex encoded info-hash of the torrent to connect to.
This is required for the client accepting the connection to know which certificate to
present.</p>
<p>SSL connections are accepted on a separate socket from normal bittorrent connections. To
pick which port the SSL socket should bind to, set <a class="reference external" href="reference-Settings.html#ssl_listen">session_settings::ssl_listen</a> to a
different port. It defaults to port 4433. This setting is only taken into account when the
normal listen socket is opened (i.e. just changing this setting won't necessarily close
and re-open the SSL socket). To not listen on an SSL socket at all, set <tt class="docutils literal">ssl_listen</tt> to 0.</p>
<p>This feature is only available if libtorrent is build with openssl support (<tt class="docutils literal">TORRENT_USE_OPENSSL</tt>)
and requires at least openSSL version 1.0, since it needs SNI support.</p>
<p>Peer certificates must have at least one <em>SubjectAltName</em> field of type dNSName. At least
one of the fields must <em>exactly</em> match the name of the torrent. This is a byte-by-byte comparison,
the UTF-8 encoding must be identical (i.e. there's no unicode normalization going on). This is
the recommended way of verifying certificates for HTTPS servers according to <a class="reference external" href="http://www.ietf.org/rfc/rfc2818.txt">RFC 2818</a>. Note
the difference that for torrents only <em>dNSName</em> fields are taken into account (not IP address fields).
The most specific (i.e. last) <em>Common Name</em> field is also taken into account if no <em>SubjectAltName</em>
did not match.</p>
<p>If any of these fields contain a single asterisk (&quot;*&quot;), the certificate is considered covering
any torrent, allowing it to be reused for any torrent.</p>
<p>The purpose of matching the torrent name with the fields in the peer certificate is to allow
a publisher to have a single root certificate for all torrents it distributes, and issue
separate peer certificates for each torrent. A peer receiving a certificate will not necessarily
be able to access all torrents published by this root certificate (only if it has a &quot;star cert&quot;).</p>
<p>During the SSL handshake, both peers need to authenticate by providing a
certificate that is signed by the CA certificate found in the .torrent file.
These peer certificates are expected to be privided to peers through some other
means than bittorrent. Typically by a peer generating a certificate request
which is sent to the publisher of the torrent, and the publisher returning a
signed certificate.</p>
<p>In libtorrent, <a class="reference external" href="reference-Core.html#set_ssl_certificate()">set_ssl_certificate()</a> in <a class="reference external" href="reference-Core.html#torrent_handle">torrent_handle</a> is used to tell
libtorrent where to find the peer certificate and the private key for it. When
an SSL torrent is loaded, the <a class="reference external" href="reference-Alerts.html#torrent_need_cert_alert">torrent_need_cert_alert</a> is posted to remind the
user to provide a certificate.</p>
<p>A peer connecting to an SSL torrent MUST provide the <em>SNI</em> TLS extension
(server name indication). The server name is the hex encoded info-hash of the
torrent to connect to. This is required for the client accepting the connection
to know which certificate to present.</p>
<p>SSL connections are accepted on a separate socket from normal bittorrent
connections. To pick which port the SSL socket should bind to, set
<a class="reference external" href="reference-Settings.html#ssl_listen">session_settings::ssl_listen</a> to a different port. It defaults to port 4433.
This setting is only taken into account when the normal listen socket is opened
(i.e. just changing this setting won't necessarily close and re-open the SSL
socket). To not listen on an SSL socket at all, set <tt class="docutils literal">ssl_listen</tt> to 0.</p>
<p>This feature is only available if libtorrent is build with openssl support
(<tt class="docutils literal">TORRENT_USE_OPENSSL</tt>) and requires at least openSSL version 1.0, since it
needs SNI support.</p>
<p>Peer certificates must have at least one <em>SubjectAltName</em> field of type
dNSName. At least one of the fields must <em>exactly</em> match the name of the
torrent. This is a byte-by-byte comparison, the UTF-8 encoding must be
identical (i.e. there's no unicode normalization going on). This is the
recommended way of verifying certificates for HTTPS servers according to <a class="reference external" href="http://www.ietf.org/rfc/rfc2818.txt">RFC
2818</a>. Note the difference that for torrents only <em>dNSName</em> fields are taken
into account (not IP address fields). The most specific (i.e. last) <em>Common
Name</em> field is also taken into account if no <em>SubjectAltName</em> did not match.</p>
<p>If any of these fields contain a single asterisk (&quot;*&quot;), the certificate is
considered covering any torrent, allowing it to be reused for any torrent.</p>
<p>The purpose of matching the torrent name with the fields in the peer
certificate is to allow a publisher to have a single root certificate for all
torrents it distributes, and issue separate peer certificates for each torrent.
A peer receiving a certificate will not necessarily be able to access all
torrents published by this root certificate (only if it has a &quot;star cert&quot;).</p>
<div class="section" id="testing">
<h2>testing</h2>
<p>To test incoming SSL connections to an SSL torrent, one can use the following <em>openssl</em> command:</p>
<p>To test incoming SSL connections to an SSL torrent, one can use the following
<em>openssl</em> command:</p>
<pre class="literal-block">
openssl s_client -cert &lt;peer-certificate&gt;.pem -key &lt;peer-private-key&gt;.pem -CAfile \
&lt;torrent-cert&gt;.pem -debug -connect 127.0.0.1:4433 -tls1 -servername &lt;info-hash&gt;
</pre>
<p>To create a root certificate, the Distinguished Name (<em>DN</em>) is not taken into account
by bittorrent peers. You still need to specify something, but from libtorrent's point of
view, it doesn't matter what it is. libtorrent only makes sure the peer certificates are
signed by the correct root certificate.</p>
<p>One way to create the certificates is to use the <tt class="docutils literal">CA.sh</tt> script that comes with openssl,
like thisi (don't forget to enter a common Name for the certificate):</p>
<p>To create a root certificate, the Distinguished Name (<em>DN</em>) is not taken into
account by bittorrent peers. You still need to specify something, but from
libtorrent's point of view, it doesn't matter what it is. libtorrent only makes
sure the peer certificates are signed by the correct root certificate.</p>
<p>One way to create the certificates is to use the <tt class="docutils literal">CA.sh</tt> script that comes
with openssl, like thisi (don't forget to enter a common Name for the
certificate):</p>
<pre class="literal-block">
CA.sh -newca
CA.sh -newreq
CA.sh -sign
</pre>
<p>The torrent certificate is located in <tt class="docutils literal">./demoCA/private/demoCA/cacert.pem</tt>, this is
the pem file to include in the .torrent file.</p>
<p>The torrent certificate is located in <tt class="docutils literal">./demoCA/private/demoCA/cacert.pem</tt>,
this is the pem file to include in the .torrent file.</p>
<p>The peer's certificate is located in <tt class="docutils literal">./newcert.pem</tt> and the certificate's
private key in <tt class="docutils literal">./newkey.pem</tt>.</p>
</div>