forked from premiere/premiere-libtorrent
381 lines
23 KiB
HTML
381 lines
23 KiB
HTML
<?xml version="1.0" encoding="utf-8" ?>
|
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
|
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="Docutils 0.11: http://docutils.sourceforge.net/" />
|
|
<title>libtorrent manual</title>
|
|
<meta name="author" content="Arvid Norberg, arvid@rasterbar.com" />
|
|
<link rel="stylesheet" type="text/css" href="base.css" />
|
|
<link rel="stylesheet" type="text/css" href="rst.css" />
|
|
<script type="text/javascript">
|
|
/* <![CDATA[ */
|
|
(function() {
|
|
var s = document.createElement('script'), t = document.getElementsByTagName('script')[0];
|
|
s.type = 'text/javascript';
|
|
s.async = true;
|
|
s.src = 'http://api.flattr.com/js/0.6/load.js?mode=auto';
|
|
t.parentNode.insertBefore(s, t);
|
|
})();
|
|
/* ]]> */
|
|
</script>
|
|
<link rel="stylesheet" href="style.css" type="text/css" />
|
|
<style type="text/css">
|
|
/* Hides from IE-mac \*/
|
|
* html pre { height: 1%; }
|
|
/* End hide from IE-mac */
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<div class="document" id="libtorrent-manual">
|
|
<div id="container">
|
|
<div id="header">
|
|
<div id="orange"></div>
|
|
<div id="logo"></div>
|
|
</div>
|
|
<div id="main">
|
|
<h1 class="title">libtorrent manual</h1>
|
|
<table class="docinfo" frame="void" rules="none">
|
|
<col class="docinfo-name" />
|
|
<col class="docinfo-content" />
|
|
<tbody valign="top">
|
|
<tr><th class="docinfo-name">Author:</th>
|
|
<td>Arvid Norberg, <a class="last reference external" href="mailto:arvid@rasterbar.com">arvid@rasterbar.com</a></td></tr>
|
|
<tr><th class="docinfo-name">Version:</th>
|
|
<td>1.1.0</td></tr>
|
|
</tbody>
|
|
</table>
|
|
<div class="contents topic" id="table-of-contents">
|
|
<p class="topic-title first">Table of contents</p>
|
|
<ul class="simple">
|
|
<li><a class="reference internal" href="#introduction" id="id2">introduction</a></li>
|
|
<li><a class="reference internal" href="#features" id="id3">features</a><ul>
|
|
<li><a class="reference internal" href="#extensions" id="id4">extensions</a></li>
|
|
<li><a class="reference internal" href="#disk-management" id="id5">disk management</a></li>
|
|
<li><a class="reference internal" href="#network" id="id6">network</a></li>
|
|
</ul>
|
|
</li>
|
|
<li><a class="reference internal" href="#highlighted-features" id="id7">highlighted features</a><ul>
|
|
<li><a class="reference internal" href="#disk-caching" id="id8">disk caching</a></li>
|
|
<li><a class="reference internal" href="#high-performance-disk-subsystem" id="id9">high performance disk subsystem</a></li>
|
|
<li><a class="reference internal" href="#network-buffers" id="id10">network buffers</a></li>
|
|
<li><a class="reference internal" href="#piece-picker" id="id11">piece picker</a></li>
|
|
<li><a class="reference internal" href="#share-mode" id="id12">share mode</a></li>
|
|
<li><a class="reference internal" href="#merkle-hash-tree-torrents" id="id13">merkle hash tree torrents</a></li>
|
|
<li><a class="reference internal" href="#customizable-file-storage" id="id14">customizable file storage</a></li>
|
|
<li><a class="reference internal" href="#easy-to-use-api" id="id15">easy to use API</a></li>
|
|
</ul>
|
|
</li>
|
|
<li><a class="reference internal" href="#portability" id="id16">portability</a></li>
|
|
</ul>
|
|
</div>
|
|
<div class="section" id="introduction">
|
|
<h1>introduction</h1>
|
|
<p>libtorrent is a feature complete C++ bittorrent implementation focusing
|
|
on efficiency and scalability. It runs on embedded devices as well as
|
|
desktops. It boasts a well documented library interface that is easy to
|
|
use. It comes with a simple bittorrent client demonstrating the use of
|
|
the library.</p>
|
|
</div>
|
|
<div class="section" id="features">
|
|
<h1>features</h1>
|
|
<p>libtorrent is under active development. It is an ongoing project. Its
|
|
current state supports and includes the following features:</p>
|
|
<div class="section" id="extensions">
|
|
<h2>extensions</h2>
|
|
<ul class="simple">
|
|
<li>plugin interface for implementing custom bittorrent extensions
|
|
without having to modify libtorrent</li>
|
|
<li>supports trackerless torrents (using the Mainline kademlia DHT protocol) with
|
|
some <a class="reference external" href="dht_extensions.html">DHT extensions</a>. <a class="reference external" href="http://bittorrent.org/beps/bep_0005.html">BEP 5</a>.</li>
|
|
<li>supports the bittorrent <a class="reference external" href="extension_protocol.html">extension protocol</a>. See <a class="reference external" href="manual-ref.html#extensions">extensions</a>. <a class="reference external" href="http://bittorrent.org/beps/bep_0010.html">BEP 10</a>.</li>
|
|
<li>supports the uTorrent metadata transfer protocol <a class="reference external" href="http://bittorrent.org/beps/bep_0009.html">BEP 9</a> (i.e. magnet links).</li>
|
|
<li>supports the uTorrent peer exchange protocol (PEX).</li>
|
|
<li>supports local peer discovery (multicasts for peers on the same local network)</li>
|
|
<li>multitracker extension support (supports both strict <a class="reference external" href="http://bittorrent.org/beps/bep_0012.html">BEP 12</a> and the
|
|
uTorrent interpretation).</li>
|
|
<li>tracker scrapes</li>
|
|
<li>supports lt_trackers extension, to exchange trackers between peers</li>
|
|
<li><a class="reference external" href="manual-ref.html#http-seeding">HTTP seeding</a>, as specified in <a class="reference external" href="http://bittorrent.org/beps/bep_0017.html">BEP 17</a> and <a class="reference external" href="http://bittorrent.org/beps/bep_0019.html">BEP 19</a>.</li>
|
|
<li>supports the udp-tracker protocol. (<a class="reference external" href="http://bittorrent.org/beps/bep_0015.html">BEP 15</a>).</li>
|
|
<li>supports the <tt class="docutils literal">no_peer_id=1</tt> extension that will ease the load off trackers.</li>
|
|
<li>supports the <tt class="docutils literal">compact=1</tt> tracker parameter.</li>
|
|
<li>super seeding/initial seeding (<a class="reference external" href="http://bittorrent.org/beps/bep_0016.html">BEP 16</a>).</li>
|
|
<li>private torrents (<a class="reference external" href="http://bittorrent.org/beps/bep_0027.html">BEP 27</a>).</li>
|
|
<li>upload-only extension (<a class="reference external" href="http://bittorrent.org/beps/bep_0021.html">BEP 21</a>).</li>
|
|
<li>support for IPv6, including <a class="reference external" href="http://bittorrent.org/beps/bep_0007.html">BEP 7</a> and <a class="reference external" href="http://bittorrent.org/beps/bep_0024.html">BEP 24</a>.</li>
|
|
<li>support for merkle hash tree torrents. This makes the size of torrent files
|
|
scale well with the size of the content.</li>
|
|
<li>share-mode. This is a special mode torrents can be put in to optimize share
|
|
ratio rather than downloading the torrent.</li>
|
|
</ul>
|
|
</div>
|
|
<div class="section" id="disk-management">
|
|
<h2>disk management</h2>
|
|
<ul class="simple">
|
|
<li>uses a separate disk I/O thread to not have the disk ever block on network or
|
|
client interaction. (see <a class="reference external" href="manualref.html#threads">threads</a>).</li>
|
|
<li>uses asynchronous disk I/O when available (overlapped I/O, kaio, and posix-aio)
|
|
to make optimal use of disk bandwidth capacity</li>
|
|
<li>supports verifying the SHA-1 hash of pieces in multiple threads, to take full
|
|
advantage of multi core machines.</li>
|
|
<li>supports files > 2 gigabytes.</li>
|
|
<li>fast resume support, a way to get rid of the costly piece check at the
|
|
start of a resumed torrent. Saves the storage state, piece_picker state
|
|
as well as all local peers in a separate fast-resume file.</li>
|
|
<li>has an adjustable read and write disk cache for improved disk throughput.</li>
|
|
<li>queues torrents for file check, instead of checking all of them in parallel.</li>
|
|
<li>does not have any requirements on the piece order in a torrent that it
|
|
resumes. This means it can resume a torrent downloaded by any client.</li>
|
|
<li>seed mode, where the files on disk are assumed to be complete, and each
|
|
piece's hash is verified the first time it is requested.</li>
|
|
</ul>
|
|
</div>
|
|
<div class="section" id="network">
|
|
<h2>network</h2>
|
|
<ul class="simple">
|
|
<li>a high quality uTP implementation (<a class="reference external" href="http://bittorrent.org/beps/bep_0029.html">BEP 29</a>). A transport protocol with
|
|
delay based congestion control. See separate <a class="reference external" href="utp.html">article</a>.</li>
|
|
<li>adjusts the length of the request queue depending on download rate.</li>
|
|
<li>serves multiple torrents on a single port and in a single thread</li>
|
|
<li>piece picking on block-level (as opposed to piece-level).
|
|
This means it can download parts of the same piece from different peers.
|
|
It will also prefer to download whole pieces from single peers if the
|
|
download speed is high enough from that particular peer.</li>
|
|
<li>supports http proxies and basic proxy authentication</li>
|
|
<li>supports gzipped tracker-responses</li>
|
|
<li>can limit the upload and download bandwidth usage and the maximum number of
|
|
unchoked peers</li>
|
|
<li>possibility to limit the number of connections.</li>
|
|
<li>delays have messages if there's no other outgoing traffic to the peer, and
|
|
doesn't send have messages to peers that already has the piece. This saves
|
|
bandwidth.</li>
|
|
<li>selective downloading. The ability to select which parts of a torrent you
|
|
want to download.</li>
|
|
<li>ip filter to disallow ip addresses and ip ranges from connecting and
|
|
being connected.</li>
|
|
<li>NAT-PMP and UPnP support (automatic port mapping on routers that supports it)</li>
|
|
<li>implements automatic upload slots, to optimize download rate without spreading
|
|
upload capacity too thin. The number of upload slots is adjusted based on the
|
|
peers' download capacity to work even for connections that are orders of
|
|
magnitude faster than others.</li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="highlighted-features">
|
|
<h1>highlighted features</h1>
|
|
<div class="section" id="disk-caching">
|
|
<h2>disk caching</h2>
|
|
<p>All disk I/O in libtorrent is done asynchronously to the network thread, by the
|
|
disk io thread. When a block is read, the disk io thread reads all subsequent
|
|
blocks from that piece into the read cache, assuming that the peer requesting
|
|
the block will also request more blocks from the same piece. This decreases the
|
|
number of syscalls for reading data. It also decreases delay from seeking.</p>
|
|
<p>Similarly, for write requests, blocks are cached and flushed to disk once one full
|
|
piece is complete or the piece is the least recently updated one when more cache
|
|
space is needed. The cache dynamically allocates space between the write and read
|
|
cache. The write cache is strictly prioritized over the read cache.</p>
|
|
<p>The cache blocks that are in used, are locked into physical memory to avoid it
|
|
being paged out to disk. Allowing the disk cache to be paged out to disk means
|
|
that it would become extremely inefficient to flush it, since it would have to be
|
|
read back into physical memory only to be flushed back out to disk again.</p>
|
|
<p>In order to conserve memory, and system calls, iovec file operations are
|
|
used to flush multiple cache blocks in a single call.</p>
|
|
<p>On low-memory systems, the disk cache can be disabled altogether or set to smaller
|
|
limit, to save memory.</p>
|
|
<p>The disk caching algorithm is configurable between 'LRU' and 'largest contiguous'.
|
|
The largest contiguous algorithm is the default and flushes the largest contiguous
|
|
block of buffers, instead of flushing all blocks belonging to the piece which was
|
|
written to least recently.</p>
|
|
<p>For version 0.15 a lot of work went into optimizing the cache algorithm, trying
|
|
to increase the cache hit rate and utilization. The graph to the left shows the
|
|
memory utilization in 0.14. This cache is a straight forward, fairly naive, implementation.
|
|
Every block read will also read all subsequent blocks in that piece into the cache.
|
|
Whenever we need more space, the entire oldest piece is evicted from the cache. Caching
|
|
writes always takes presedence over the read cache. Whenever a piece is fully downloaded,
|
|
it is flushed to disk.</p>
|
|
<img alt="disk_buffer_before_optimization.png" src="disk_buffer_before_optimization.png" style="width: 49%;" />
|
|
<img alt="disk_buffer.png" src="disk_buffer.png" style="width: 49%;" />
|
|
<p>The left graph shows the problem of evicting entire pieces at a time, and waiting until
|
|
an entire piece is downloaded until flushing it. These graphs were generated for a torrent
|
|
with fairly large pieces. This means that granularity was poor in 0.14, since it only
|
|
dealt with entire pieces. In 0.15, the granularity problem has been fixed by evicting one
|
|
block at a time from the read cache. This maximizes the read cache utilization. The write
|
|
cache is also flushed when a sufficient number of contiguous blocks have been downloaded
|
|
for a piece, which is not tied to the piece size anymore. This way the cache scales a lot
|
|
better with piece sizes.</p>
|
|
<p>The graph to the right shows the same download but with the new optimized disk cache
|
|
algorithm. It clearly shows an increased utilization, which means higher read hit rates
|
|
or smaller caches with maintained hit rate.</p>
|
|
</div>
|
|
<div class="section" id="high-performance-disk-subsystem">
|
|
<h2>high performance disk subsystem</h2>
|
|
<p>In some circumstances, the disk cache may not suffice to provide maximum performance.
|
|
One such example is high performance seeding, to a large number of peers, over a fast
|
|
up-link. In such a case, the amount of RAM may simply not be enough to cache disk
|
|
reads. When there's not enough RAM to cache disk reads, the disk throughput would
|
|
typically degrade to perform as poorly as with no cache at all, with the majority
|
|
of the time spent waiting for the disk head to seek.</p>
|
|
<p>To solve this problem, libtorrent sorts read requests by their physical offset on the
|
|
disk. They are processed by having the disk read head sweep back and forth over the drive.</p>
|
|
<p>This makes libtorrent very suitable for large scale, high-throughput seeding.</p>
|
|
<img alt="disk_access_no_elevator.png" src="disk_access_no_elevator.png" style="width: 49%;" />
|
|
<img alt="disk_access_elevator.png" src="disk_access_elevator.png" style="width: 49%;" />
|
|
<p>These plots illustrates the physical disk offset for reads over time. The left plot
|
|
is of a run where disk operation re-ordering is turned off and the righ is when it's
|
|
turned on. The right one has a relatively smooth sine wave shape whereas the left
|
|
one is more random and involves much longer seeks back and forth over the disk.</p>
|
|
<p>True physical disk offset queries are only supported on newer linux kernels, Mac OS X and
|
|
Windows 2000 and up.</p>
|
|
</div>
|
|
<div class="section" id="network-buffers">
|
|
<h2>network buffers</h2>
|
|
<p>On CPUs with small L2 caches, copying memory can be expensive operations. It is important
|
|
to keep copying to a minimum on such machines. This mostly applies to embedded systems.</p>
|
|
<p>In order to minimize the number of times received data is copied, the receive buffer
|
|
for payload data is received directly into a page aligned disk buffer. If the connection
|
|
is encrypted, the buffer is decrypted in-place. The buffer is then moved into the disk
|
|
cache without being copied. Once all the blocks for a piece have been received, or the
|
|
cache needs to be flushed, all the blocks are passed directly to <tt class="docutils literal">writev()</tt> to flush
|
|
them in a single syscall. This means a single copy into user space memory, and a single
|
|
copy back into kernel memory, as illustrated by this figure:</p>
|
|
<img alt="write_disk_buffers.png" src="write_disk_buffers.png" style="width: 100%;" />
|
|
<p>When seeding and uploading in general, unnecessary copying is avoided by caching blocks
|
|
in aligned buffers, that are copied once into the peer's send buffer. The peer's send buffer
|
|
is not guaranteed to be aligned, even though it is most of the time. The send buffer is
|
|
then encrypted with the peer specific key and chained onto the <tt class="docutils literal">iovec</tt> for sending.
|
|
This means there is one user space copy in order to allow unaligned peer requests and
|
|
peer-specific encryption. This is illustrated by the following figure:</p>
|
|
<img alt="read_disk_buffers.png" src="read_disk_buffers.png" style="width: 100%;" />
|
|
</div>
|
|
<div class="section" id="piece-picker">
|
|
<h2>piece picker</h2>
|
|
<p>The piece picker is a central component in a bittorrent implementation. The piece picker
|
|
in libtorrent is optimized for quickly finding the rarest pieces. It keeps a list of all
|
|
available pieces sorted by rarity, and pieces with the same rarity, shuffled. The rarest
|
|
first mode is the dominant piece picker mode. Other modes are supported as well, and
|
|
used by peers in specific situations.</p>
|
|
<p>The piece picker allows to combine the availability of a piece with a priority. Together
|
|
they determine the sort order of the piece list. Pieces with priority 0 will never be
|
|
picked, which is used for the selective download feature.</p>
|
|
<p>In order to have as few partially finished pieces as possible, peers have an affinity
|
|
towards picking blocks from the same pieces as other peers in the same speed category.
|
|
The speed category is a coarse categorization of peers based on their download rate. This
|
|
makes slow peers pick blocks from the same piece, and fast peers pick from the same piece,
|
|
and hence decreasing the likelihood of slow peers blocking the completion of pieces.</p>
|
|
<p>The piece picker can also be set to download pieces in sequential order.</p>
|
|
</div>
|
|
<div class="section" id="share-mode">
|
|
<h2>share mode</h2>
|
|
<p>The share mode feature in libtorrent is intended for users who are only interested in
|
|
helping out swarms, not downloading the torrents.</p>
|
|
<p>It works by predicting the demand for pieces, and only download pieces if there is enough
|
|
demand. New pieces will only be downloaded once the share ratio has hit a certain target.</p>
|
|
<p>This feature is especially useful when combined with RSS, so that a client can be set up
|
|
to provide additional bandwidth to an entire feed.</p>
|
|
</div>
|
|
<div class="section" id="merkle-hash-tree-torrents">
|
|
<h2>merkle hash tree torrents</h2>
|
|
<p>Merkle hash tree torrents is an extension that lets a torrent file only contain the
|
|
root hash of the hash tree forming the piece hashes. The main benefit of this feature
|
|
is that regardless of how many pieces there is in a torrent, the .torrent file will
|
|
always be the same size. It will only grow with the number of files (since it still
|
|
has to contain the file names).</p>
|
|
<p>With regular torrents, clients have to request multiple blocks for pieces, typically
|
|
from different peers, before the data can be verified against the piece hash. The
|
|
larger the pieces are, the longer it will take to download a complete piece and verify
|
|
it. Before the piece is verified, it cannot be shared with the swarm, which means the
|
|
larger piece sizes, the slower turnaround data has when it is downloaded by peers.
|
|
Since on average the data has to sit around, waiting, in client buffers before it has
|
|
been verified and can be uploaded again.</p>
|
|
<p>Another problem with large piece sizes is that it is harder for a client to pinpoint
|
|
the malicious or buggy peer when a piece fails, and it will take longer to re-download
|
|
it and take more tries before the piece succeeds the larger the pieces are.</p>
|
|
<p>The piece size in regular torrents is a tradeoff between the size of the .torrent file
|
|
itself and the piece size. Often, for files that are 4 GB, the piece size is 2 or 4 MB,
|
|
just to avoid making the .torrent file too big.</p>
|
|
<p>Merkle torrents solves these problems by removing the tradeoff between .torrent size and
|
|
piece size. With merkle torrents, the piece size can be the minimum block size (16 kB),
|
|
which lets peers verify every block of data received from peers, immediately. This
|
|
gives a minimum turnaround time and completely removes the problem of identifying malicious
|
|
peers.</p>
|
|
<img alt="merkle_tree.png" src="merkle_tree.png" />
|
|
<p>The root hash is built by hashing all the piece hashes pair-wise, until they all collapse
|
|
down to the root.</p>
|
|
<img alt="storage.png" class="align-right" src="storage.png" />
|
|
</div>
|
|
<div class="section" id="customizable-file-storage">
|
|
<h2>customizable file storage</h2>
|
|
<p>libtorrent's storage implementation is customizable. That means a special purpose bittorrent
|
|
client can replace the default way to store files on disk.</p>
|
|
<p>When implementing a bittorrent cache, it doesn't matter how the data is stored on disk, as
|
|
long as it can be retrieved and seeded. In that case a new storage class can be implemented
|
|
(inheriting from the <tt class="docutils literal">storage_interface</tt> class) that avoids the unnecessary step of mapping
|
|
slots to files and offsets. The storage can ignore the file boundaries and just store the
|
|
entire torrent in a single file (which will end up being all the files concatenated). The main
|
|
advantage of this, other than a slight cpu performance gain, is that all file operations would
|
|
be page (and sector) aligned. This enables efficient unbuffered I/O, and can potentially
|
|
lead to more efficient read caching (using the built in disk cache rather than relying on the
|
|
operating system's disk cache).</p>
|
|
<p>The storage interface supports operating systems where you can ask for sparse regions
|
|
(such as Windows and Solaris). The advantage of this is that when checking files, the regions
|
|
that are known to be sparse can be skipped, which can reduce the time to check a torrent
|
|
significantly.</p>
|
|
</div>
|
|
<div class="section" id="easy-to-use-api">
|
|
<h2>easy to use API</h2>
|
|
<p>One of the design goals of the libtorrent API is to make common operations simple, but still
|
|
have it possible to do complicated and advanced operations. This is best illustrated by example
|
|
code to implement a simple bittorrent client:</p>
|
|
<pre class="literal-block">
|
|
#include <iostream>
|
|
#include "libtorrent/session.hpp"
|
|
|
|
// usage a.out [torrent-file]
|
|
int main(int argc, char* argv[]) try
|
|
{
|
|
using namespace libtorrent;
|
|
|
|
session s;
|
|
s.listen_on(std::make_pair(6881, 6889));
|
|
add_torrent_params p;
|
|
p.save_path = "./";
|
|
p.ti = new torrent_info(argv[1]);
|
|
s.add_torrent(p);
|
|
|
|
// wait for the user to end
|
|
char a;
|
|
std::cin.unsetf(std::ios_base::skipws);
|
|
std::cin >> a;
|
|
return 0;
|
|
}
|
|
catch (std::exception& e)
|
|
{
|
|
std::cerr << ec.what() << std::endl;
|
|
return 1;
|
|
}
|
|
</pre>
|
|
<p>This client doesn't give the user any status information or progress about the torrent, but
|
|
it is fully functional.</p>
|
|
<p>libtorrent also comes with python bindings for easy access for python developers.</p>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="portability">
|
|
<h1>portability</h1>
|
|
<p>libtorrent runs on most major operating systems, including Windows,
|
|
MacOS X, Linux, BSD and Solaris.
|
|
It uses Boost.Thread, Boost.Filesystem, Boost.Date_time and various other
|
|
boost libraries. At least version 1.46.1 of boost is required.</p>
|
|
<p>libtorrent uses asio, hence it will take full advantage of high performance
|
|
network APIs on the most popular platforms. I/O completion ports on windows,
|
|
epoll on linux and kqueue on MacOS X and BSD.</p>
|
|
<p>libtorrent does not build with the following compilers:</p>
|
|
<ul class="simple">
|
|
<li>GCC 2.95.4</li>
|
|
<li>Visual Studio 6, 7.0, 7.1</li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
</body>
|
|
</html>
|