translating error codes

The error_code::message() function will typically return a localized error string, for system errors. That is, errors that belong to the generic or system category.

Errors that belong to the libtorrent error category are not localized however, they are only available in english. In order to translate libtorrent errors, compare the error category of the error_code object against libtorrent::get_libtorrent_category(), and if matches, you know the error code refers to the list above. You can provide your own mapping from error code to string, which is localized. In this case, you cannot rely on error_code::message() to generate your strings.

The numeric values of the errors are part of the API and will stay the same, although new error codes may be appended at the end.

Here's a simple example of how to translate error codes:

std::string error_code_to_string(boost::system::error_code const& ec)
{
        if (ec.category() != libtorrent::get_libtorrent_category())
        {
                return ec.message();
        }
        // the error is a libtorrent error

        int code = ec.value();
        static const char const* swedish[] =
        {
                "inget fel",
                "en fil i torrenten kolliderar med en fil fran en annan torrent",
                "hash check misslyckades",
                "torrentfilen ar inte en dictionary",
                "'info'-nyckeln saknas eller ar korrupt i torrentfilen",
                "'info'-faltet ar inte en dictionary",
                "'piece length' faltet saknas eller ar korrupt i torrentfilen",
                "torrentfilen saknar namnfaltet",
                "ogiltigt namn i torrentfilen (kan vara en attack)",
                // ... more strings here
        };

        // use the default error string in case we don't have it
        // in our translated list
        if (code < 0 || code >= sizeof(swedish)/sizeof(swedish[0]))
                return ec.message();

        return swedish[code];
}

downloading

Torrents that are currently being downloaded or incomplete (with bytes still to download) are queued. The torrents in the front of the queue are started to be actively downloaded and the rest are ordered with regards to their queue position. Any newly added torrent is placed at the end of the queue. Once a torrent is removed or turns into a seed, its queue position is -1 and all torrents that used to be after it in the queue, decreases their position in order to fill the gap.

The queue positions are always in a sequence without any gaps.

Lower queue position means closer to the front of the queue, and will be started sooner than torrents with higher queue positions.

To query a torrent for its position in the queue, or change its position, see: queue_position(), queue_position_up(), queue_position_down(), queue_position_top() and queue_position_bottom().

seeding

Auto managed seeding torrents are rotated, so that all of them are allocated a fair amount of seeding. Torrents with fewer completed seed cycles are prioritized for seeding. A seed cycle is completed when a torrent meets either the share ratio limit (uploaded bytes / downloaded bytes), the share time ratio (time seeding / time downloaing) or seed time limit (time seeded).

The relevant settings to control these limits are share_ratio_limit, seed_time_ratio_limit and seed_time_limit in session_settings.

file format

The file format is a bencoded dictionary containing the following fields:

sparse allocation

On filesystems that supports sparse files, this allocation mode will only use as much space as has been downloaded.

The main drawback of this mode is that it may create heavily fragmented files.

• It does not require an allocation pass on startup.

full allocation

When a torrent is started in full allocation mode, the disk-io thread will make sure that the entire storage is allocated, and fill any gaps with zeros. It will of course still check for existing pieces and fast resume data. The main drawbacks of this mode are:

• It may take longer to start the torrent, since it will need to fill the files with zeroes. This delay is linear to the size of the download.
• The download may occupy unnecessary disk space between download sessions.
• Disk caches usually perform poorly with random access to large files and may slow down the download some.

The benefits of this mode are:

• Downloaded pieces are written directly to their final place in the files and the total number of disk operations will be fewer and may also play nicer to filesystems' file allocation, and reduce fragmentation.
• No risk of a download failing because of a full disk during download, once all files have been created.

compact allocation

The compact allocation will only allocate as much storage as it needs to keep the pieces downloaded so far. This means that pieces will be moved around to be placed at their final position in the files while downloading (to make sure the completed download has all its pieces in the correct place). So, the main drawbacks are:

• More disk operations while downloading since pieces are moved around.
• Potentially more fragmentation in the filesystem.
• Cannot be used while having files with priority 0.

The benefits though, are:

• No startup delay, since the files don't need allocating.
• The download will not use unnecessary disk space.
• Disk caches perform much better than in full allocation and raises the download speed limit imposed by the disk.
• Works well on filesystems that don't support sparse files.

The algorithm that is used when allocating pieces and slots isn't very complicated. For the interested, a description follows.

storing a piece:

1. let A be a newly downloaded piece, with index n.
2. let s be the number of slots allocated in the file we're downloading to. (the number of pieces it has room for).
3. if n >= s then allocate a new slot and put the piece there.
4. if n < s then allocate a new slot, move the data at slot n to the new slot and put A in slot n.

allocating a new slot:

1. if there's an unassigned slot (a slot that doesn't contain any piece), return that slot index.
2. append the new slot at the end of the file (or find an unused slot).
3. let i be the index of newly allocated slot
4. if we have downloaded piece index i already (to slot j) then
   1. move the data at slot j to slot i.
   2. return slot index j as the newly allocated free slot.
5. return i as the newly allocated slot.

metadata from peers

Extension name: "LT_metadata"

This extension is deprecated in favor of the more widely supported ut_metadata extension, see BEP 9. The point with this extension is that you don't have to distribute the metadata (.torrent-file) separately. The metadata can be distributed through the bittorrent swarm. The only thing you need to download such a torrent is the tracker url and the info-hash of the torrent.

It works by assuming that the initial seeder has the metadata and that the metadata will propagate through the network as more peers join.

There are three kinds of messages in the metadata extension. These packets are put as payload to the extension message. The three packets are:

• request metadata
• metadata
• don't have metadata

request metadata:

metadata:

Don't have metadata:

dont_have

Extension name: "lt_dont_have"

The dont_have extension message is used to tell peers that the client no longer has a specific piece. The extension message should be advertised in the m dictionary as lt_dont_have. The message format mimics the regular HAVE bittorrent message.

Just like all extension messages, the first 2 bytes in the mssage itself are 20 (the bittorrent extension message) and the message ID assigned to this extension in the m dictionary in the handshake.

The length of this message (including the extension message prefix) is 6 bytes, i.e. one byte longer than the normal HAVE message, because of the extension message wrapping.

HTTP seeding

There are two kinds of HTTP seeding. One with that assumes a smart (and polite) client and one that assumes a smart server. These are specified in BEP 19 and BEP 17 respectively.

libtorrent supports both. In the libtorrent source code and API, BEP 19 urls are typically referred to as url seeds and BEP 17 urls are typically referred to as HTTP seeds.

The libtorrent implementation of BEP 19 assumes that, if the URL ends with a slash ('/'), the filename should be appended to it in order to request pieces from that file. The way this works is that if the torrent is a single-file torrent, only that filename is appended. If the torrent is a multi-file torrent, the torrent's name '/' the file name is appended. This is the same directory structure that libtorrent will download torrents into.

internal representation

It is optimized by, at all times, keeping a list of pieces ordered by rarity, randomly shuffled within each rarity class. This list is organized as a single vector of contigous memory in RAM, for optimal memory locality and to eliminate heap allocations and frees when updating rarity of pieces.

Expensive events, like a peer joining or leaving, are evaluated lazily, since it's cheaper to rebuild the whole list rather than updating every single piece in it. This means as long as no blocks are picked, peers joining and leaving is no more costly than a single peer joining or leaving. Of course the special cases of peers that have all or no pieces are optimized to not require rebuilding the list.

picker strategy

The normal mode of the picker is of course rarest first, meaning pieces that few peers have are preferred to be downloaded over pieces that more peers have. This is a fundamental algorithm that is the basis of the performance of bittorrent. However, the user may set the piece picker into sequential download mode. This mode simply picks pieces sequentially, always preferring lower piece indices.

When a torrent starts out, picking the rarest pieces means increased risk that pieces won't be completed early (since there are only a few peers they can be downloaded from), leading to a delay of having any piece to offer to other peers. This lack of pieces to trade, delays the client from getting started into the normal tit-for-tat mode of bittorrent, and will result in a long ramp-up time. The heuristic to mitigate this problem is to, for the first few pieces, pick random pieces rather than rare pieces. The threshold for when to leave this initial picker mode is determined by session_settings::initial_picker_threshold.

reverse order

An orthogonal setting is reverse order, which is used for snubbed peers. Snubbed peers are peers that appear very slow, and might have timed out a piece request. The idea behind this is to make all snubbed peers more likely to be able to do download blocks from the same piece, concentrating slow peers on as few pieces as possible. The reverse order means that the most common pieces are picked, instead of the rarest pieces (or in the case of sequential download, the last pieces, intead of the first).

parole mode

Peers that have participated in a piece that failed the hash check, may be put in parole mode. This means we prefer downloading a full piece from this peer, in order to distinguish which peer is sending corrupt data. Whether to do this is or not is controlled by session_settings::use_parole_mode.

In parole mode, the piece picker prefers picking one whole piece at a time for a given peer, avoiding picking any blocks from a piece any other peer has contributed to (since that would defeat the purpose of parole mode).

prioritize partial pieces

This setting determines if partially downloaded or requested pieces should always be preferred over other pieces. The benefit of doing this is that the number of partial pieces is minimized (and hence the turn-around time for downloading a block until it can be uploaded to others is minimized). It also puts less stress on the disk cache, since fewer partial pieces need to be kept in the cache. Whether or not to enable this is controlled by session_settings::prioritize_partial_pieces.

The main benefit of not prioritizing partial pieces is that the rarest first algorithm gets to have more influence on which pieces are picked. The picker is more likely to truly pick the rarest piece, and hence improving the performance of the swarm.

This setting is turned on automatically whenever the number of partial pieces in the piece picker exceeds the number of peers we're connected to times 1.5. This is in order to keep the waste of partial pieces to a minimum, but still prefer rarest pieces.

prefer whole pieces

The prefer whole pieces setting makes the piece picker prefer picking entire pieces at a time. This is used by web connections (both http seeding standards), in order to be able to coalesce the small bittorrent requests to larger HTTP requests. This significantly improves performance when downloading over HTTP.

It is also used by peers that are downloading faster than a certain threshold. The main advantage is that these peers will better utilize the other peer's disk cache, by requesting all blocks in a single piece, from the same peer.

This threshold is controlled by session_settings::whole_pieces_threshold.

TODO: piece affinity by speed category TODO: piece priorities

testing

To test incoming SSL connections to an SSL torrent, one can use the following openssl command:

openssl s_client -cert <peer-certificate>.pem -key <peer-private-key>.pem -CAfile \
   <torrent-cert>.pem -debug -connect 127.0.0.1:4433 -tls1 -servername <info-hash>

To create a root certificate, the Distinguished Name (DN) is not taken into account by bittorrent peers. You still need to specify something, but from libtorrent's point of view, it doesn't matter what it is. libtorrent only makes sure the peer certificates are signed by the correct root certificate.

One way to create the certificates is to use the CA.sh script that comes with openssl, like thisi (don't forget to enter a common Name for the certificate):

CA.sh -newca
CA.sh -newreq
CA.sh -sign

The torrent certificate is located in ./demoCA/private/demoCA/cacert.pem, this is the pem file to include in the .torrent file.

The peer's certificate is located in ./newcert.pem and the certificate's private key in ./newkey.pem.