premiere
/
premiere-libtorrent
2
2
Fork 1
Code Issues Pull Requests Releases Wiki Activity

reflow some rst documents to 80 columns

This commit is contained in:
Arvid Norberg 2013-12-31 16:46:39 +00:00
parent 260e97c4e0
commit 488f7697c6
2 changed files with 307 additions and 299 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

187
docs/dht_store.rst
View File

@ -30,54 +30,56 @@ alive.
messages messages
-------- --------
The proposed new messages ``get`` and ``put`` are similar to the existing ``get_peers`` The proposed new messages ``get`` and ``put`` are similar to the existing
and ``announce_peer``. ``get_peers`` and ``announce_peer``.
Responses to ``get`` should always include ``nodes`` and ``nodes6``. Those fields Responses to ``get`` should always include ``nodes`` and ``nodes6``. Those
have the same semantics as in its ``get_peers`` response. It should also include a write token, fields have the same semantics as in its ``get_peers`` response. It should also
``token``, with the same semantics as int ``get_peers``. The write token MAY be tied include a write token, ``token``, with the same semantics as int ``get_peers``.
specifically to the key which ``get`` requested. i.e. the ``token`` can only be used The write token MAY be tied specifically to the key which ``get`` requested.
to store values under that one key. It may also be tied to the node ID and IP i.e. the ``token`` can only be used to store values under that one key. It may
address of the requesting node. also be tied to the node ID and IP address of the requesting node.
The ``id`` field in these messages has the same semantics as the standard DHT messages, The ``id`` field in these messages has the same semantics as the standard DHT
i.e. the node ID of the node sending the message, to maintain the structure of the DHT messages, i.e. the node ID of the node sending the message, to maintain the
network. structure of the DHT network.
The ``token`` field also has the same semantics as the standard DHT message ``get_peers`` The ``token`` field also has the same semantics as the standard DHT message
and ``announce_peer``, when requesting an item and to write an item respectively. ``get_peers`` and ``announce_peer``, when requesting an item and to write an
item respectively.
The ``k`` field is the 32 byte curve25519 public key, which the signature The ``k`` field is the 32 byte curve25519 public key, which the signature can be
can be authenticated with. When looking up a mutable item, the ``target`` field authenticated with. When looking up a mutable item, the ``target`` field MUST be
MUST be the SHA-1 hash of this key concatenated with the ``salt``, if present. the SHA-1 hash of this key concatenated with the ``salt``, if present.
The distinction between storing mutable and immutable items is the inclusion The distinction between storing mutable and immutable items is the inclusion of
of a public key, a sequence number, signature and an optional salt (``k``, a public key, a sequence number, signature and an optional salt (``k``, ``seq``,
``seq``, ``sig`` and ``salt``). ``sig`` and ``salt``).
``get`` requests for mutable items and immutable items cannot be distinguished from ``get`` requests for mutable items and immutable items cannot be distinguished
eachother. An implementation can either store mutable and immutable items in the same from eachother. An implementation can either store mutable and immutable items
hash table internally, or in separate ones and potentially do two lookups for ``get`` in the same hash table internally, or in separate ones and potentially do two
requests. lookups for ``get`` requests.
The ``v`` field is the *value* to be stored. It is allowed to be any bencoded type (list, The ``v`` field is the *value* to be stored. It is allowed to be any bencoded
dict, string or integer). When it's being hashed (for verifying its signature or to calculate type (list, dict, string or integer). When it's being hashed (for verifying its
its key), its flattened, bencoded, form is used. It is important to use the verbatim signature or to calculate its key), its flattened, bencoded, form is used. It is
bencoded representation as it appeared in the message. decoding and then re-encoding important to use the verbatim bencoded representation as it appeared in the
bencoded structures is not necessarily an identity operation. message. decoding and then re-encoding bencoded structures is not necessarily an
identity operation.
Storing nodes MAY reject ``put`` requests where the bencoded form of ``v`` is longer Storing nodes MAY reject ``put`` requests where the bencoded form of ``v`` is
than 1000 bytes. In other words, it's not safe to assume storing more than longer than 1000 bytes. In other words, it's not safe to assume storing more
1000 bytes will succeed. than 1000 bytes will succeed.
immutable items immutable items ---------------
---------------
Immutable items are stored under their SHA-1 hash, and since they cannot be modified, Immutable items are stored under their SHA-1 hash, and since they cannot be
there is no need to authenticate the origin of them. This makes immutable items simple. modified, there is no need to authenticate the origin of them. This makes
immutable items simple.
A node making a lookup SHOULD verify the data it receives from the network, to verify A node making a lookup SHOULD verify the data it receives from the network, to
that its hash matches the target that was looked up. verify that its hash matches the target that was looked up.
put message put message
........... ...........
@ -147,30 +149,32 @@ mutable items
------------- -------------
Mutable items can be updated, without changing their DHT keys. To authenticate Mutable items can be updated, without changing their DHT keys. To authenticate
that only the original publisher can update an item, it is signed by a private key that only the original publisher can update an item, it is signed by a private
generated by the original publisher. The target ID mutable items are stored under key generated by the original publisher. The target ID mutable items are stored
is the SHA-1 hash of the public key (as it appears in the ``put`` message). under is the SHA-1 hash of the public key (as it appears in the ``put``
message).
In order to avoid a malicious node to overwrite the list head with an old In order to avoid a malicious node to overwrite the list head with an old
version, the sequence number ``seq`` must be monotonically increasing for each update, version, the sequence number ``seq`` must be monotonically increasing for each
and a node hosting the list node MUST not downgrade a list head from a higher sequence update, and a node hosting the list node MUST not downgrade a list head from a
number to a lower one, only upgrade. The sequence number SHOULD not exceed ``MAX_INT64``, higher sequence number to a lower one, only upgrade. The sequence number SHOULD
(i.e. ``0x7fffffffffffffff``. A client MAY reject any message with a sequence number not exceed ``MAX_INT64``, (i.e. ``0x7fffffffffffffff``. A client MAY reject any
exceeding this. A client MAY also reject any message with a negative sequence number. message with a sequence number exceeding this. A client MAY also reject any
message with a negative sequence number.
The signature is a 64 byte curve25519 signature of the bencoded sequence The signature is a 64 byte curve25519 signature of the bencoded sequence number
number concatenated with the ``v`` key. e.g. something like this:: concatenated with the ``v`` key. e.g. something like this::
3:seqi4e1:v12:Hello world! 3:seqi4e1:v12:Hello world!
If the ``salt`` key is present and non-empty, the salt string must be included If the ``salt`` key is present and non-empty, the salt string must be included
in what's signed. Note that if ``salt`` is specified and an empty string, it in what's signed. Note that if ``salt`` is specified and an empty string, it is
is as if it was not specified and nothing in addition to the sequence number as if it was not specified and nothing in addition to the sequence number and
and the data is signed. the data is signed.
When a salt is included in what is signed, the key ``salt`` with the value When a salt is included in what is signed, the key ``salt`` with the value of
of the key is prepended in its bencoded form. For example, if ``salt`` is the key is prepended in its bencoded form. For example, if ``salt`` is "foobar",
"foobar", the buffer to be signed is:: the buffer to be signed is::
4:salt6:foobar3:seqi4e1:v12:Hello world! 4:salt6:foobar3:seqi4e1:v12:Hello world!
@ -200,31 +204,31 @@ Request:
Storing nodes receiving a ``put`` request where ``seq`` is lower than or equal Storing nodes receiving a ``put`` request where ``seq`` is lower than or equal
to what's already stored on the node, MUST reject the request. If the sequence to what's already stored on the node, MUST reject the request. If the sequence
number is equal, and the value is also the same, the node SHOULD reset its timeout number is equal, and the value is also the same, the node SHOULD reset its
counter. timeout counter.
If the sequence number in the ``put`` message is lower than the sequence number If the sequence number in the ``put`` message is lower than the sequence number
associated with the currently stored value, the storing node MAY return an error associated with the currently stored value, the storing node MAY return an error
message with code 302 (see error codes below). message with code 302 (see error codes below).
Note that this request does not contain a target hash. The target hash under Note that this request does not contain a target hash. The target hash under
which this blob is stored is implied by the ``k`` argument. The key is which this blob is stored is implied by the ``k`` argument. The key is the SHA-1
the SHA-1 hash of the key (``k``). hash of the key (``k``).
In order to support a single key being used to store separate items in the DHT, In order to support a single key being used to store separate items in the DHT,
an optional ``salt`` can be specified in the ``put`` request of mutable an optional ``salt`` can be specified in the ``put`` request of mutable items.
items. If the salt entry is not present, it can be assumed to be an empty If the salt entry is not present, it can be assumed to be an empty string, and
string, and its semantics should be identical as specifying a salt key its semantics should be identical as specifying a salt key with an empty string.
with an empty string. The salt can be any binary string (but probably most The salt can be any binary string (but probably most conveniently a hash of
conveniently a hash of something). This string is appended to the key, something). This string is appended to the key, as specified in the ``k`` field,
as specified in the ``k`` field, when calculating the key to store the when calculating the key to store the blob under (i.e. the key ``get`` requests
blob under (i.e. the key ``get`` requests specify to retrieve this data). specify to retrieve this data).
This lets a single entity, with a single key, publish any number of unrelated This lets a single entity, with a single key, publish any number of unrelated
items, with a single key that readers can verify. This is useful if the items, with a single key that readers can verify. This is useful if the
publisher doesn't know ahead of time how many different items are to be publisher doesn't know ahead of time how many different items are to be
published. It can distribute a single public key for users to authenticate published. It can distribute a single public key for users to authenticate the
the published blobs. published blobs.
The ``cas`` field is optional. If present it is interpreted as the sha-1 hash of The ``cas`` field is optional. If present it is interpreted as the sha-1 hash of
the sequence number, ``v`` field and possibly the ``salt`` field, that is the sequence number, ``v`` field and possibly the ``salt`` field, that is
@ -246,13 +250,14 @@ Response:
"y": "r", "y": "r",
} }
If the store fails for any reason an error message is returned instead of the message If the store fails for any reason an error message is returned instead of the
template above, i.e. one where "y" is "e" and "e" is a tuple of [error-code, message]). message template above, i.e. one where "y" is "e" and "e" is a tuple of
Failures include where the ``cas`` hash mismatches and the sequence number is outdated. [error-code, message]). Failures include where the ``cas`` hash mismatches and
the sequence number is outdated.
If no ``cas`` field is included in the ``put`` message, the value of the current ``v`` If no ``cas`` field is included in the ``put`` message, the value of the current
field should be disregarded when determining whether or not to save the item. ``v`` field should be disregarded when determining whether or not to save the
(However, the signature, sequence number obviously still should). item. (However, the signature, sequence number obviously still should).
The error message (as specified by BEP5_) looks like this: The error message (as specified by BEP5_) looks like this:
@ -285,8 +290,9 @@ some additional error codes.
| | current. | | | current. |
+------------+-----------------------------+ +------------+-----------------------------+
An implementation MUST emit 301 errors if the cas-hash mismatches. This is An implementation MUST emit 301 errors if the cas-hash mismatches. This is a
a critical feature in synchronization of multiple agents sharing an immutable item. critical feature in synchronization of multiple agents sharing an immutable
item.
get message get message
........... ...........
@ -330,26 +336,28 @@ Response:
signature verification signature verification
---------------------- ----------------------
In order to make it maximally difficult to attack the bencoding parser, signing and verification of the In order to make it maximally difficult to attack the bencoding parser, signing
value and sequence number should be done as follows: and verification of the value and sequence number should be done as follows:
1. encode value and sequence number separately 1. encode value and sequence number separately
2. concatenate ("4:salt" *length-of-salt* ":" *salt*) "3:seqi" *seq* 2. concatenate ("4:salt" *length-of-salt* ":" *salt*) "3:seqi" *seq*
"e1:v" *len* ":" and the encoded value. "e1:v" *len* ":" and the encoded value.
sequence number 1 of value "Hello World!" would be converted to: "3:seqi1e1:v12:Hello World!" sequence number 1 of value "Hello World!" would be converted to:
In this way it is not possible to convince a node that part of the length is actually part of the "3:seqi1e1:v12:Hello World!". In this way it is not possible to convince a
sequence number even if the parser contains certain bugs. Furthermore it is not possible to have a node that part of the length is actually part of the sequence number even if
verification failure if a bencoding serializer alters the order of entries in the dictionary. the parser contains certain bugs. Furthermore it is not possible to have a
The salt is in parenthesis because it is optional. It is only prepended if verification failure if a bencoding serializer alters the order of entries in
a non-empty salt is specified in the ``put`` request. the dictionary. The salt is in parenthesis because it is optional. It is only
prepended if a non-empty salt is specified in the ``put`` request.
3. sign or verify the concatenated string 3. sign or verify the concatenated string
On the storage node, the signature MUST be verified before accepting the store command. The data On the storage node, the signature MUST be verified before accepting the store
MUST be stored under the SHA-1 hash of the public key (as it appears in the bencoded dict). command. The data MUST be stored under the SHA-1 hash of the public key (as it
appears in the bencoded dict).
On the requesting nodes, the key they get back from a ``get`` request MUST be verified to hash On the requesting nodes, the key they get back from a ``get`` request MUST be
to the target ID the lookup was made for, as well as verifying the signature. If any of these fail, verified to hash to the target ID the lookup was made for, as well as verifying
the response SHOULD be considered invalid. the signature. If any of these fail, the response SHOULD be considered invalid.
expiration expiration
---------- ----------
@ -357,8 +365,9 @@ expiration
Without re-announcement, these items MAY expire in 2 hours. In order Without re-announcement, these items MAY expire in 2 hours. In order
to keep items alive, they SHOULD be re-announced once an hour. to keep items alive, they SHOULD be re-announced once an hour.
Any node that's interested in keeping a blob in the DHT alive may announce it. It would simply Any node that's interested in keeping a blob in the DHT alive may announce it.
repeat the signature for a mutable put without having the private key. It would simply repeat the signature for a mutable put without having the
private key.
test vector test vector
----------- -----------

417
docs/manual.rst
View File

@ -20,8 +20,10 @@ The basic usage is as follows:
* construct a session * construct a session
* load session state from settings file (see load_state()) * load session state from settings file (see load_state())
* start extensions (see add_extension()). * start extensions (see add_extension()).
* start DHT, LSD, UPnP, NAT-PMP etc (see start_dht(), start_lsd(), start_upnp() and start_natpmp()). * start DHT, LSD, UPnP, NAT-PMP etc (see start_dht(), start_lsd(), start_upnp()
* parse .torrent-files and add them to the session (see torrent_info, async_add_torrent() and add_torrent()) and start_natpmp()).
* parse .torrent-files and add them to the session (see torrent_info,
async_add_torrent() and add_torrent())
* main loop (see session) * main loop (see session)
* poll for alerts (see wait_for_alert(), pop_alerts()) * poll for alerts (see wait_for_alert(), pop_alerts())
@ -174,52 +176,55 @@ libtorrent supports *queuing*. Which means it makes sure that a limited number o
torrents are being downloaded at any given time, and once a torrent is completely torrents are being downloaded at any given time, and once a torrent is completely
downloaded, the next in line is started. downloaded, the next in line is started.
Torrents that are *auto managed* are subject to the queuing and the active torrents Torrents that are *auto managed* are subject to the queuing and the active
limits. To make a torrent auto managed, set ``auto_managed`` to true when adding the torrents limits. To make a torrent auto managed, set ``auto_managed`` to true
torrent (see async_add_torrent() and add_torrent()). when adding the torrent (see async_add_torrent() and add_torrent()).
The limits of the number of downloading and seeding torrents are controlled via The limits of the number of downloading and seeding torrents are controlled via
``active_downloads``, ``active_seeds`` and ``active_limit`` in session_settings. ``active_downloads``, ``active_seeds`` and ``active_limit`` in
These limits takes non auto managed torrents into account as well. If there are session_settings. These limits takes non auto managed torrents into account as
more non-auto managed torrents being downloaded than the ``active_downloads`` well. If there are more non-auto managed torrents being downloaded than the
setting, any auto managed torrents will be queued until torrents are removed so ``active_downloads`` setting, any auto managed torrents will be queued until
that the number drops below the limit. torrents are removed so that the number drops below the limit.
The default values are 8 active downloads and 5 active seeds. The default values are 8 active downloads and 5 active seeds.
At a regular interval, torrents are checked if there needs to be any re-ordering of At a regular interval, torrents are checked if there needs to be any
which torrents are active and which are queued. This interval can be controlled via re-ordering of which torrents are active and which are queued. This interval
``auto_manage_interval`` in session_settings. It defaults to every 30 seconds. can be controlled via ``auto_manage_interval`` in session_settings. It defaults
to every 30 seconds.
For queuing to work, resume data needs to be saved and restored for all torrents. For queuing to work, resume data needs to be saved and restored for all
See save_resume_data(). torrents. See save_resume_data().
downloading downloading
----------- -----------
Torrents that are currently being downloaded or incomplete (with bytes still to download) Torrents that are currently being downloaded or incomplete (with bytes still to
are queued. The torrents in the front of the queue are started to be actively downloaded download) are queued. The torrents in the front of the queue are started to be
and the rest are ordered with regards to their queue position. Any newly added torrent actively downloaded and the rest are ordered with regards to their queue
is placed at the end of the queue. Once a torrent is removed or turns into a seed, its position. Any newly added torrent is placed at the end of the queue. Once a
queue position is -1 and all torrents that used to be after it in the queue, decreases their torrent is removed or turns into a seed, its queue position is -1 and all
position in order to fill the gap. torrents that used to be after it in the queue, decreases their position in
order to fill the gap.
The queue positions are always in a sequence without any gaps. The queue positions are always in a sequence without any gaps.
Lower queue position means closer to the front of the queue, and will be started sooner than Lower queue position means closer to the front of the queue, and will be
torrents with higher queue positions. started sooner than torrents with higher queue positions.
To query a torrent for its position in the queue, or change its position, see: To query a torrent for its position in the queue, or change its position, see:
queue_position(), queue_position_up(), queue_position_down(), queue_position_top() and queue_position_bottom(). queue_position(), queue_position_up(), queue_position_down(),
queue_position_top() and queue_position_bottom().
seeding seeding
------- -------
Auto managed seeding torrents are rotated, so that all of them are allocated a fair Auto managed seeding torrents are rotated, so that all of them are allocated a
amount of seeding. Torrents with fewer completed *seed cycles* are prioritized for fair amount of seeding. Torrents with fewer completed *seed cycles* are
seeding. A seed cycle is completed when a torrent meets either the share ratio limit prioritized for seeding. A seed cycle is completed when a torrent meets either
(uploaded bytes / downloaded bytes), the share time ratio (time seeding / time the share ratio limit (uploaded bytes / downloaded bytes), the share time ratio
downloaing) or seed time limit (time seeded). (time seeding / time downloaing) or seed time limit (time seeded).
The relevant settings to control these limits are ``share_ratio_limit``, The relevant settings to control these limits are ``share_ratio_limit``,
``seed_time_ratio_limit`` and ``seed_time_limit`` in session_settings. ``seed_time_ratio_limit`` and ``seed_time_limit`` in session_settings.
@ -237,10 +242,11 @@ fast-resume data. The fast-resume data also contains information about which
blocks, in the unfinished pieces, were downloaded, so it will not have to blocks, in the unfinished pieces, were downloaded, so it will not have to
start from scratch on the partially downloaded pieces. start from scratch on the partially downloaded pieces.
To use the fast-resume data you simply give it to async_add_torrent() and add_torrent(), and it To use the fast-resume data you simply give it to async_add_torrent() and
will skip the time consuming checks. It may have to do the checking anyway, if add_torrent(), and it will skip the time consuming checks. It may have to do
the fast-resume data is corrupt or doesn't fit the storage for that torrent, the checking anyway, if the fast-resume data is corrupt or doesn't fit the
then it will not trust the fast-resume data and just do the checking. storage for that torrent, then it will not trust the fast-resume data and just
do the checking.
file format file format
----------- -----------
@ -402,24 +408,23 @@ storage allocation
There are two modes in which storage (files on disk) are allocated in libtorrent. There are two modes in which storage (files on disk) are allocated in libtorrent.
1. The traditional *full allocation* mode, where the entire files are filled up with 1. The traditional *full allocation* mode, where the entire files are filled up
zeros before anything is downloaded. libtorrent will look for sparse files support with zeros before anything is downloaded. Files are allocated on demand, the
in the filesystem that is used for storage, and use sparse files or file system first time anything is written to them. The main benefit of this mode is that
zero fill support if present. This means that on NTFS, full allocation mode will it avoids creating heavily fragmented files.
only allocate storage for the downloaded pieces.
2. The *sparse allocation*, sparse files are used, and pieces are downloaded directly 2. The *sparse allocation*, sparse files are used, and pieces are downloaded
to where they belong. This is the recommended (and default) mode. directly to where they belong. This is the recommended (and default) mode.
In previous versions of libtorrent, a 3rd mode was supported, *compact allocation*. In previous versions of libtorrent, a 3rd mode was supported, *compact
Support for this is deprecated and will be removed in future versions of libtorrent. allocation*. Support for this is deprecated and will be removed in future
It's still described in here for completeness. versions of libtorrent. It's still described in here for completeness.
The allocation mode is selected when a torrent is started. It is passed as an The allocation mode is selected when a torrent is started. It is passed as an
argument to session::add_torrent() or session::async_add_torrent(). argument to session::add_torrent() or session::async_add_torrent().
The decision to use full allocation or compact allocation typically depends on whether The decision to use full allocation or compact allocation typically depends on
any files have priority 0 and if the filesystem supports sparse files. whether any files have priority 0 and if the filesystem supports sparse files.
sparse allocation sparse allocation
----------------- -----------------
@ -427,45 +432,34 @@ sparse allocation
On filesystems that supports sparse files, this allocation mode will only use On filesystems that supports sparse files, this allocation mode will only use
as much space as has been downloaded. as much space as has been downloaded.
The main drawback of this mode is that it may create heavily fragmented files.
* It does not require an allocation pass on startup. * It does not require an allocation pass on startup.
* It supports skipping files (setting prioirty to 0 to not download).
* Fast resume data will remain valid even when file time stamps are out of date.
full allocation full allocation
--------------- ---------------
When a torrent is started in full allocation mode, the disk-io thread When a torrent is started in full allocation mode, the disk-io thread
will make sure that the entire storage is allocated, and fill any gaps with zeros. will make sure that the entire storage is allocated, and fill any gaps with zeros.
This will be skipped if the filesystem supports sparse files or automatic zero filling.
It will of course still check for existing pieces and fast resume data. The main It will of course still check for existing pieces and fast resume data. The main
drawbacks of this mode are: drawbacks of this mode are:
* It may take longer to start the torrent, since it will need to fill the files * It may take longer to start the torrent, since it will need to fill the files
with zeros on some systems. This delay is linearly dependent on the size of with zeroes. This delay is linear to the size of the download.
the download.
* The download may occupy unnecessary disk space between download sessions. In case * The download may occupy unnecessary disk space between download sessions.
sparse files are not supported.
* Disk caches usually perform extremely poorly with random access to large files * Disk caches usually perform poorly with random access to large files
and may slow down a download considerably. and may slow down the download some.
The benefits of this mode are: The benefits of this mode are:
* Downloaded pieces are written directly to their final place in the files and the * Downloaded pieces are written directly to their final place in the files and
total number of disk operations will be fewer and may also play nicer to the total number of disk operations will be fewer and may also play nicer to
filesystems' file allocation, and reduce fragmentation. filesystems' file allocation, and reduce fragmentation.
* No risk of a download failing because of a full disk during download. Unless * No risk of a download failing because of a full disk during download, once
sparse files are being used. all files have been created.
* The fast resume data will be more likely to be usable, regardless of crashes or
out of date data, since pieces won't move around.
* Can be used with prioritizing files to 0.
compact allocation compact allocation
------------------ ------------------
@ -474,10 +468,11 @@ compact allocation
Note that support for compact allocation is deprecated in libttorrent, and will Note that support for compact allocation is deprecated in libttorrent, and will
be removed in future versions. be removed in future versions.
The compact allocation will only allocate as much storage as it needs to keep the The compact allocation will only allocate as much storage as it needs to keep
pieces downloaded so far. This means that pieces will be moved around to be placed the pieces downloaded so far. This means that pieces will be moved around to be
at their final position in the files while downloading (to make sure the completed placed at their final position in the files while downloading (to make sure the
download has all its pieces in the correct place). So, the main drawbacks are: completed download has all its pieces in the correct place). So, the main
drawbacks are:
* More disk operations while downloading since pieces are moved around. * More disk operations while downloading since pieces are moved around.
@ -491,13 +486,13 @@ The benefits though, are:
* The download will not use unnecessary disk space. * The download will not use unnecessary disk space.
* Disk caches perform much better than in full allocation and raises the download * Disk caches perform much better than in full allocation and raises the
speed limit imposed by the disk. download speed limit imposed by the disk.
* Works well on filesystems that don't support sparse files. * Works well on filesystems that don't support sparse files.
The algorithm that is used when allocating pieces and slots isn't very complicated. The algorithm that is used when allocating pieces and slots isn't very
For the interested, a description follows. complicated. For the interested, a description follows.
storing a piece: storing a piece:
@ -521,14 +516,13 @@ allocating a new slot:
5. return **i** as the newly allocated slot. 5. return **i** as the newly allocated slot.
extensions extensions
========== ==========
These extensions all operates within the `extension protocol`_. The These extensions all operates within the `extension protocol`_. The name of the
name of the extension is the name used in the extension-list packets, extension is the name used in the extension-list packets, and the payload is
and the payload is the data in the extended message (not counting the the data in the extended message (not counting the length-prefix, message-id
length-prefix, message-id nor extension-id). nor extension-id).
.. _`extension protocol`: extension_protocol.html .. _`extension protocol`: extension_protocol.html
@ -543,18 +537,18 @@ metadata from peers
Extension name: "LT_metadata" Extension name: "LT_metadata"
This extension is deprecated in favor of the more widely supported ``ut_metadata`` This extension is deprecated in favor of the more widely supported
extension, see `BEP 9`_. ``ut_metadata`` extension, see `BEP 9`_. The point with this extension is that
The point with this extension is that you don't have to distribute the you don't have to distribute the metadata (.torrent-file) separately. The
metadata (.torrent-file) separately. The metadata can be distributed metadata can be distributed through the bittorrent swarm. The only thing you
through the bittorrent swarm. The only thing you need to download such need to download such a torrent is the tracker url and the info-hash of the
a torrent is the tracker url and the info-hash of the torrent. torrent.
It works by assuming that the initial seeder has the metadata and that It works by assuming that the initial seeder has the metadata and that the
the metadata will propagate through the network as more peers join. metadata will propagate through the network as more peers join.
There are three kinds of messages in the metadata extension. These packets There are three kinds of messages in the metadata extension. These packets are
are put as payload to the extension message. The three packets are: put as payload to the extension message. The three packets are:
* request metadata * request metadata
* metadata * metadata
@ -621,13 +615,14 @@ dont_have
Extension name: "lt_dont_have" Extension name: "lt_dont_have"
The ``dont_have`` extension message is used to tell peers that the client no longer The ``dont_have`` extension message is used to tell peers that the client no
has a specific piece. The extension message should be advertised in the ``m`` dictionary longer has a specific piece. The extension message should be advertised in the
as ``lt_dont_have``. The message format mimics the regular ``HAVE`` bittorrent message. ``m`` dictionary as ``lt_dont_have``. The message format mimics the regular
``HAVE`` bittorrent message.
Just like all extension messages, the first 2 bytes in the mssage itself are 20 (the Just like all extension messages, the first 2 bytes in the mssage itself are 20
bittorrent extension message) and the message ID assigned to this extension in the ``m`` (the bittorrent extension message) and the message ID assigned to this
dictionary in the handshake. extension in the ``m`` dictionary in the handshake.
+-----------+---------------+----------------------------------------+ +-----------+---------------+----------------------------------------+
| size | name | description | | size | name | description |
@ -636,27 +631,27 @@ dictionary in the handshake.
| | | has. | | | | has. |
+-----------+---------------+----------------------------------------+ +-----------+---------------+----------------------------------------+
The length of this message (including the extension message prefix) is The length of this message (including the extension message prefix) is 6 bytes,
6 bytes, i.e. one byte longer than the normal ``HAVE`` message, because i.e. one byte longer than the normal ``HAVE`` message, because of the extension
of the extension message wrapping. message wrapping.
HTTP seeding HTTP seeding
------------ ------------
There are two kinds of HTTP seeding. One with that assumes a smart There are two kinds of HTTP seeding. One with that assumes a smart (and polite)
(and polite) client and one that assumes a smart server. These client and one that assumes a smart server. These are specified in `BEP 19`_
are specified in `BEP 19`_ and `BEP 17`_ respectively. and `BEP 17`_ respectively.
libtorrent supports both. In the libtorrent source code and API, libtorrent supports both. In the libtorrent source code and API, BEP 19 urls
BEP 19 urls are typically referred to as *url seeds* and BEP 17 are typically referred to as *url seeds* and BEP 17 urls are typically referred
urls are typically referred to as *HTTP seeds*. to as *HTTP seeds*.
The libtorrent implementation of `BEP 19`_ assumes that, if the URL ends with a slash The libtorrent implementation of `BEP 19`_ assumes that, if the URL ends with a
('/'), the filename should be appended to it in order to request pieces from slash ('/'), the filename should be appended to it in order to request pieces
that file. The way this works is that if the torrent is a single-file torrent, from that file. The way this works is that if the torrent is a single-file
only that filename is appended. If the torrent is a multi-file torrent, the torrent, only that filename is appended. If the torrent is a multi-file
torrent's name '/' the file name is appended. This is the same directory torrent, the torrent's name '/' the file name is appended. This is the same
structure that libtorrent will download torrents into. directory structure that libtorrent will download torrents into.
.. _`BEP 17`: http://bittorrent.org/beps/bep_0017.html .. _`BEP 17`: http://bittorrent.org/beps/bep_0017.html
.. _`BEP 19`: http://bittorrent.org/beps/bep_0019.html .. _`BEP 19`: http://bittorrent.org/beps/bep_0019.html
@ -679,71 +674,69 @@ The piece picker in libtorrent has the following features:
internal representation internal representation
----------------------- -----------------------
It is optimized by, at all times, keeping a list of pieces ordered It is optimized by, at all times, keeping a list of pieces ordered by rarity,
by rarity, randomly shuffled within each rarity class. This list randomly shuffled within each rarity class. This list is organized as a single
is organized as a single vector of contigous memory in RAM, for vector of contigous memory in RAM, for optimal memory locality and to eliminate
optimal memory locality and to eliminate heap allocations and frees heap allocations and frees when updating rarity of pieces.
when updating rarity of pieces.
Expensive events, like a peer joining or leaving, are evaluated Expensive events, like a peer joining or leaving, are evaluated lazily, since
lazily, since it's cheaper to rebuild the whole list rather than it's cheaper to rebuild the whole list rather than updating every single piece
updating every single piece in it. This means as long as no blocks in it. This means as long as no blocks are picked, peers joining and leaving is
are picked, peers joining and leaving is no more costly than a single no more costly than a single peer joining or leaving. Of course the special
peer joining or leaving. Of course the special cases of peers that have cases of peers that have all or no pieces are optimized to not require
all or no pieces are optimized to not require rebuilding the list. rebuilding the list.
picker strategy picker strategy
--------------- ---------------
The normal mode of the picker is of course *rarest first*, meaning The normal mode of the picker is of course *rarest first*, meaning pieces that
pieces that few peers have are preferred to be downloaded over pieces few peers have are preferred to be downloaded over pieces that more peers have.
that more peers have. This is a fundamental algorithm that is the This is a fundamental algorithm that is the basis of the performance of
basis of the performance of bittorrent. However, the user may set the bittorrent. However, the user may set the piece picker into sequential download
piece picker into sequential download mode. This mode simply picks mode. This mode simply picks pieces sequentially, always preferring lower piece
pieces sequentially, always preferring lower piece indices. indices.
When a torrent starts out, picking the rarest pieces means increased When a torrent starts out, picking the rarest pieces means increased risk that
risk that pieces won't be completed early (since there are only a few pieces won't be completed early (since there are only a few peers they can be
peers they can be downloaded from), leading to a delay of having any downloaded from), leading to a delay of having any piece to offer to other
piece to offer to other peers. This lack of pieces to trade, delays peers. This lack of pieces to trade, delays the client from getting started
the client from getting started into the normal tit-for-tat mode of into the normal tit-for-tat mode of bittorrent, and will result in a long
bittorrent, and will result in a long ramp-up time. The heuristic to ramp-up time. The heuristic to mitigate this problem is to, for the first few
mitigate this problem is to, for the first few pieces, pick random pieces pieces, pick random pieces rather than rare pieces. The threshold for when to
rather than rare pieces. The threshold for when to leave this initial leave this initial picker mode is determined by
picker mode is determined by session_settings::initial_picker_threshold. session_settings::initial_picker_threshold.
reverse order reverse order
------------- -------------
An orthogonal setting is *reverse order*, which is used for *snubbed* An orthogonal setting is *reverse order*, which is used for *snubbed* peers.
peers. Snubbed peers are peers that appear very slow, and might have timed Snubbed peers are peers that appear very slow, and might have timed out a piece
out a piece request. The idea behind this is to make all snubbed peers request. The idea behind this is to make all snubbed peers more likely to be
more likely to be able to do download blocks from the same piece, able to do download blocks from the same piece, concentrating slow peers on as
concentrating slow peers on as few pieces as possible. The reverse order few pieces as possible. The reverse order means that the most common pieces are
means that the most common pieces are picked, instead of the rarest pieces picked, instead of the rarest pieces (or in the case of sequential download,
(or in the case of sequential download, the last pieces, intead of the first). the last pieces, intead of the first).
parole mode parole mode -----------
-----------
Peers that have participated in a piece that failed the hash check, may be Peers that have participated in a piece that failed the hash check, may be put
put in *parole mode*. This means we prefer downloading a full piece from this in *parole mode*. This means we prefer downloading a full piece from this
peer, in order to distinguish which peer is sending corrupt data. Whether to peer, in order to distinguish which peer is sending corrupt data. Whether to do
do this is or not is controlled by session_settings::use_parole_mode. this is or not is controlled by session_settings::use_parole_mode.
In parole mode, the piece picker prefers picking one whole piece at a time for In parole mode, the piece picker prefers picking one whole piece at a time for
a given peer, avoiding picking any blocks from a piece any other peer has a given peer, avoiding picking any blocks from a piece any other peer has
contributed to (since that would defeat the purpose of parole mode). contributed to (since that would defeat the purpose of parole mode).
prioritize partial pieces prioritize partial pieces -------------------------
-------------------------
This setting determines if partially downloaded or requested pieces should always This setting determines if partially downloaded or requested pieces should
be preferred over other pieces. The benefit of doing this is that the number of always be preferred over other pieces. The benefit of doing this is that the
partial pieces is minimized (and hence the turn-around time for downloading a block number of partial pieces is minimized (and hence the turn-around time for
until it can be uploaded to others is minimized). It also puts less stress on the downloading a block until it can be uploaded to others is minimized). It also
disk cache, since fewer partial pieces need to be kept in the cache. Whether or puts less stress on the disk cache, since fewer partial pieces need to be kept
not to enable this is controlled by session_settings::prioritize_partial_pieces. in the cache. Whether or not to enable this is controlled by
session_settings::prioritize_partial_pieces.
The main benefit of not prioritizing partial pieces is that the rarest first The main benefit of not prioritizing partial pieces is that the rarest first
algorithm gets to have more influence on which pieces are picked. The picker is algorithm gets to have more influence on which pieces are picked. The picker is
@ -755,19 +748,17 @@ in the piece picker exceeds the number of peers we're connected to times 1.5.
This is in order to keep the waste of partial pieces to a minimum, but still This is in order to keep the waste of partial pieces to a minimum, but still
prefer rarest pieces. prefer rarest pieces.
prefer whole pieces prefer whole pieces -------------------
-------------------
The *prefer whole pieces* setting makes the piece picker prefer picking entire The *prefer whole pieces* setting makes the piece picker prefer picking entire
pieces at a time. This is used by web connections (both http seeding pieces at a time. This is used by web connections (both http seeding
standards), in order to be able to coalesce the small bittorrent requests standards), in order to be able to coalesce the small bittorrent requests to
to larger HTTP requests. This significantly improves performance when larger HTTP requests. This significantly improves performance when downloading
downloading over HTTP. over HTTP.
It is also used by peers that are downloading faster than a certain It is also used by peers that are downloading faster than a certain threshold.
threshold. The main advantage is that these peers will better utilize the The main advantage is that these peers will better utilize the other peer's
other peer's disk cache, by requesting all blocks in a single piece, from disk cache, by requesting all blocks in a single piece, from the same peer.
the same peer.
This threshold is controlled by session_settings::whole_pieces_threshold. This threshold is controlled by session_settings::whole_pieces_threshold.
@ -778,8 +769,8 @@ SSL torrents
============ ============
Torrents may have an SSL root (CA) certificate embedded in them. Such torrents Torrents may have an SSL root (CA) certificate embedded in them. Such torrents
are called *SSL torrents*. An SSL torrent talks to all bittorrent peers over SSL. are called *SSL torrents*. An SSL torrent talks to all bittorrent peers over
The protocols are layered like this:: SSL. The protocols are layered like this::
+-----------------------+ +-----------------------+
| BitTorrent protocol | | BitTorrent protocol |
@ -791,70 +782,78 @@ The protocols are layered like this::
| | UDP | | | UDP |
+-----------+-----------+ +-----------+-----------+
During the SSL handshake, both peers need to authenticate by providing a certificate During the SSL handshake, both peers need to authenticate by providing a
that is signed by the CA certificate found in the .torrent file. These peer certificate that is signed by the CA certificate found in the .torrent file.
certificates are expected to be privided to peers through some other means than These peer certificates are expected to be privided to peers through some other
bittorrent. Typically by a peer generating a certificate request which is sent to means than bittorrent. Typically by a peer generating a certificate request
the publisher of the torrent, and the publisher returning a signed certificate. which is sent to the publisher of the torrent, and the publisher returning a
signed certificate.
In libtorrent, set_ssl_certificate() in torrent_handle is used to tell libtorrent where In libtorrent, set_ssl_certificate() in torrent_handle is used to tell
to find the peer certificate and the private key for it. When an SSL torrent is loaded, libtorrent where to find the peer certificate and the private key for it. When
the torrent_need_cert_alert is posted to remind the user to provide a certificate. an SSL torrent is loaded, the torrent_need_cert_alert is posted to remind the
user to provide a certificate.
A peer connecting to an SSL torrent MUST provide the *SNI* TLS extension (server name A peer connecting to an SSL torrent MUST provide the *SNI* TLS extension
indication). The server name is the hex encoded info-hash of the torrent to connect to. (server name indication). The server name is the hex encoded info-hash of the
This is required for the client accepting the connection to know which certificate to torrent to connect to. This is required for the client accepting the connection
present. to know which certificate to present.
SSL connections are accepted on a separate socket from normal bittorrent connections. To SSL connections are accepted on a separate socket from normal bittorrent
pick which port the SSL socket should bind to, set session_settings::ssl_listen to a connections. To pick which port the SSL socket should bind to, set
different port. It defaults to port 4433. This setting is only taken into account when the session_settings::ssl_listen to a different port. It defaults to port 4433.
normal listen socket is opened (i.e. just changing this setting won't necessarily close This setting is only taken into account when the normal listen socket is opened
and re-open the SSL socket). To not listen on an SSL socket at all, set ``ssl_listen`` to 0. (i.e. just changing this setting won't necessarily close and re-open the SSL
socket). To not listen on an SSL socket at all, set ``ssl_listen`` to 0.
This feature is only available if libtorrent is build with openssl support (``TORRENT_USE_OPENSSL``) This feature is only available if libtorrent is build with openssl support
and requires at least openSSL version 1.0, since it needs SNI support. (``TORRENT_USE_OPENSSL``) and requires at least openSSL version 1.0, since it
needs SNI support.
Peer certificates must have at least one *SubjectAltName* field of type dNSName. At least Peer certificates must have at least one *SubjectAltName* field of type
one of the fields must *exactly* match the name of the torrent. This is a byte-by-byte comparison, dNSName. At least one of the fields must *exactly* match the name of the
the UTF-8 encoding must be identical (i.e. there's no unicode normalization going on). This is torrent. This is a byte-by-byte comparison, the UTF-8 encoding must be
the recommended way of verifying certificates for HTTPS servers according to `RFC 2818`_. Note identical (i.e. there's no unicode normalization going on). This is the
the difference that for torrents only *dNSName* fields are taken into account (not IP address fields). recommended way of verifying certificates for HTTPS servers according to `RFC
The most specific (i.e. last) *Common Name* field is also taken into account if no *SubjectAltName* 2818`_. Note the difference that for torrents only *dNSName* fields are taken
did not match. into account (not IP address fields). The most specific (i.e. last) *Common
Name* field is also taken into account if no *SubjectAltName* did not match.
If any of these fields contain a single asterisk ("*"), the certificate is considered covering If any of these fields contain a single asterisk ("*"), the certificate is
any torrent, allowing it to be reused for any torrent. considered covering any torrent, allowing it to be reused for any torrent.
The purpose of matching the torrent name with the fields in the peer certificate is to allow The purpose of matching the torrent name with the fields in the peer
a publisher to have a single root certificate for all torrents it distributes, and issue certificate is to allow a publisher to have a single root certificate for all
separate peer certificates for each torrent. A peer receiving a certificate will not necessarily torrents it distributes, and issue separate peer certificates for each torrent.
be able to access all torrents published by this root certificate (only if it has a "star cert"). A peer receiving a certificate will not necessarily be able to access all
torrents published by this root certificate (only if it has a "star cert").
.. _`RFC 2818`: http://www.ietf.org/rfc/rfc2818.txt .. _`RFC 2818`: http://www.ietf.org/rfc/rfc2818.txt
testing testing
------- -------
To test incoming SSL connections to an SSL torrent, one can use the following *openssl* command:: To test incoming SSL connections to an SSL torrent, one can use the following
*openssl* command::
openssl s_client -cert <peer-certificate>.pem -key <peer-private-key>.pem -CAfile \ openssl s_client -cert <peer-certificate>.pem -key <peer-private-key>.pem -CAfile \
<torrent-cert>.pem -debug -connect 127.0.0.1:4433 -tls1 -servername <info-hash> <torrent-cert>.pem -debug -connect 127.0.0.1:4433 -tls1 -servername <info-hash>
To create a root certificate, the Distinguished Name (*DN*) is not taken into account To create a root certificate, the Distinguished Name (*DN*) is not taken into
by bittorrent peers. You still need to specify something, but from libtorrent's point of account by bittorrent peers. You still need to specify something, but from
view, it doesn't matter what it is. libtorrent only makes sure the peer certificates are libtorrent's point of view, it doesn't matter what it is. libtorrent only makes
signed by the correct root certificate. sure the peer certificates are signed by the correct root certificate.
One way to create the certificates is to use the ``CA.sh`` script that comes with openssl, One way to create the certificates is to use the ``CA.sh`` script that comes
like thisi (don't forget to enter a common Name for the certificate):: with openssl, like thisi (don't forget to enter a common Name for the
certificate)::
CA.sh -newca CA.sh -newca
CA.sh -newreq CA.sh -newreq
CA.sh -sign CA.sh -sign
The torrent certificate is located in ``./demoCA/private/demoCA/cacert.pem``, this is The torrent certificate is located in ``./demoCA/private/demoCA/cacert.pem``,
the pem file to include in the .torrent file. this is the pem file to include in the .torrent file.
The peer's certificate is located in ``./newcert.pem`` and the certificate's The peer's certificate is located in ``./newcert.pem`` and the certificate's
private key in ``./newkey.pem``. private key in ``./newkey.pem``.