premiere
/
premiere-libtorrent
2
2
Fork 1
Code Issues Pull Requests Releases Wiki Activity

document piece picker and dont_have extension

This commit is contained in:
Arvid Norberg 2011-08-14 23:14:46 +00:00
parent e89eb6e606
commit 5e92858360
2 changed files with 339 additions and 20 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

214
docs/manual.html
View File

@ -2,25 +2,23 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<script type="text/javascript">
/* <![CDATA[ */
(function() {
var s = document.createElement('script'), t = document.getElementsByTagName('script')[0];
s.type = 'text/javascript';
s.async = true;
s.src = 'http://api.flattr.com/js/0.6/load.js?mode=auto';
t.parentNode.insertBefore(s, t);
})();
/* ]]> */
</script>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" />
<title>libtorrent API Documentation</title>
<meta name="author" content="Arvid Norberg, arvid&#64;rasterbar.com" />
<link rel="stylesheet" type="text/css" href="../../css/base.css" />
<link rel="stylesheet" type="text/css" href="../../css/rst.css" />
<script type="text/javascript">
/* <![CDATA[ */
(function() {
var s = document.createElement('script'), t = document.getElementsByTagName('script')[0];
s.type = 'text/javascript';
s.async = true;
s.src = 'http://api.flattr.com/js/0.6/load.js?mode=auto';
t.parentNode.insertBefore(s, t);
})();
/* ]]> */
</script>
<link rel="stylesheet" href="style.css" type="text/css" />
<style type="text/css">
/* Hides from IE-mac \*/
@ -122,7 +120,7 @@
</ul>
</li>
<li><a class="reference internal" href="#torrent-handle" id="id72">torrent_handle</a><ul>
<li><a class="reference internal" href="#set-piece-deadline" id="id73">set_piece_deadline()</a></li>
<li><a class="reference internal" href="#set-piece-deadline-reset-piece-deadline" id="id73">set_piece_deadline() reset_piece_deadline()</a></li>
<li><a class="reference internal" href="#piece-availability" id="id74">piece_availability()</a></li>
<li><a class="reference internal" href="#piece-priority-prioritize-pieces-piece-priorities" id="id75">piece_priority() prioritize_pieces() piece_priorities()</a></li>
<li><a class="reference internal" href="#file-priority-prioritize-files-file-priorities" id="id76">file_priority() prioritize_files() file_priorities()</a></li>
@ -324,10 +322,20 @@
</li>
<li><a class="reference internal" href="#extensions" id="id247">extensions</a><ul>
<li><a class="reference internal" href="#metadata-from-peers" id="id248">metadata from peers</a></li>
<li><a class="reference internal" href="#http-seeding" id="id249">HTTP seeding</a></li>
<li><a class="reference internal" href="#dont-have" id="id249">dont_have</a></li>
<li><a class="reference internal" href="#http-seeding" id="id250">HTTP seeding</a></li>
</ul>
</li>
<li><a class="reference internal" href="#filename-checks" id="id250">filename checks</a></li>
<li><a class="reference internal" href="#piece-picker" id="id251">piece picker</a><ul>
<li><a class="reference internal" href="#internal-representation" id="id252">internal representation</a></li>
<li><a class="reference internal" href="#picker-strategy" id="id253">picker strategy</a></li>
<li><a class="reference internal" href="#reverse-order" id="id254">reverse order</a></li>
<li><a class="reference internal" href="#parole-mode" id="id255">parole mode</a></li>
<li><a class="reference internal" href="#prioritize-partial-pieces" id="id256">prioritize partial pieces</a></li>
<li><a class="reference internal" href="#prefer-whole-pieces" id="id257">prefer whole pieces</a></li>
</ul>
</li>
<li><a class="reference internal" href="#filename-checks" id="id258">filename checks</a></li>
</ul>
</div>
<div class="section" id="overview">
@ -2296,7 +2304,9 @@ struct torrent_handle
{
query_distributed_copies = 1,
query_accurate_download_counters = 2,
query_last_seen_complete = 4
query_last_seen_complete = 4,
query_pieces = 8,
query_verified_pieces = 16
};
torrent_status status(boost::uint32_t flags = 0xffffffff);
@ -2377,6 +2387,7 @@ struct torrent_handle
enum deadline_flags { alert_when_available = 1 };
void set_piece_deadline(int index, int deadline, int flags = 0) const;
void reset_piece_deadline(int index) const;
void piece_availability(std::vector&lt;int&gt;&amp; avail) const;
void piece_priority(int index, int priority) const;
@ -2426,12 +2437,13 @@ one exception <tt class="docutils literal"><span class="pre">is_valid()</span></
Since the torrents are processed by a background thread, there is no
guarantee that a handle will remain valid between two calls.</p>
</div>
<div class="section" id="set-piece-deadline">
<h2>set_piece_deadline()</h2>
<div class="section" id="set-piece-deadline-reset-piece-deadline">
<h2>set_piece_deadline() reset_piece_deadline()</h2>
<blockquote>
<pre class="literal-block">
enum deadline_flags { alert_when_available = 1 };
void set_piece_deadline(int index, int deadline, int flags = 0) const;
void reset_piece_deadline(int index) const;
</pre>
</blockquote>
<p>This function sets or resets the deadline associated with a specific piece
@ -2447,6 +2459,8 @@ piece has been downloaded, by passing <tt class="docutils literal"><span class="
the <tt class="docutils literal"><span class="pre">alert_when_available</span></tt> flag is set, in which case it will do the same thing
as calling <a class="reference internal" href="#read-piece">read_piece()</a> for <tt class="docutils literal"><span class="pre">index</span></tt>.</p>
<p><tt class="docutils literal"><span class="pre">deadline</span></tt> is the number of milliseconds until this piece should be completed.</p>
<p><tt class="docutils literal"><span class="pre">reset_piece_deadline</span></tt> removes the deadline from the piece. If it hasn't already
been downloaded, it will no longer be considered a priority.</p>
</div>
<div class="section" id="piece-availability">
<h2>piece_availability()</h2>
@ -2656,6 +2670,9 @@ void force_dht_announce() const;
<p><tt class="docutils literal"><span class="pre">force_reannounce()</span></tt> will force this torrent to do another tracker request, to receive new
peers. The second overload of <tt class="docutils literal"><span class="pre">force_reannounce</span></tt> that takes a <tt class="docutils literal"><span class="pre">time_duration</span></tt> as
argument will schedule a reannounce in that amount of time from now.</p>
<p>If the tracker's <tt class="docutils literal"><span class="pre">min_interval</span></tt> has not passed since the last announce, the forced
announce will be scheduled to happen immediately as the <tt class="docutils literal"><span class="pre">min_interval</span></tt> expires. This is
to honor trackers minimum re-announce interval settings.</p>
<p><tt class="docutils literal"><span class="pre">force_dht_announce</span></tt> will announce the torrent to the DHT immediately.</p>
</div>
<div class="section" id="scrape-tracker">
@ -3227,6 +3244,18 @@ if you're not interested in it (and see performance issues), you can filter them
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt><tt class="docutils literal"><span class="pre">query_pieces</span></tt></dt>
<dd><p class="first last">includes <tt class="docutils literal"><span class="pre">pieces</span></tt>.</p>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt><tt class="docutils literal"><span class="pre">query_verified_pieces</span></tt></dt>
<dd><p class="first last">includes <tt class="docutils literal"><span class="pre">verified_pieces</span></tt> (only applies to torrents in <em>seed mode</em>).</p>
</dd>
</dl>
</li>
</ul>
</div>
<div class="section" id="get-download-queue">
@ -3393,6 +3422,8 @@ struct torrent_status
int connect_candidates;
bitfield pieces;
bitfield verified_pieces;
int num_pieces;
size_type total_done;
@ -3557,6 +3588,9 @@ order block). This is supposed to be as low as possible.</p>
<p><tt class="docutils literal"><span class="pre">pieces</span></tt> is the bitmask that represents which pieces we have (set to true) and
the pieces we don't have. It's a pointer and may be set to 0 if the torrent isn't
downloading or seeding.</p>
<p><tt class="docutils literal"><span class="pre">verified_pieces</span></tt> is a bitmask representing which pieces has had their hash
checked. This only applies to torrents in <em>seed mode</em>. If the torrent is not
in seed mode, this bitmask may be empty.</p>
<p><tt class="docutils literal"><span class="pre">num_pieces</span></tt> is the number of pieces that has been downloaded. It is equivalent
to: <tt class="docutils literal"><span class="pre">std::accumulate(pieces-&gt;begin(),</span> <span class="pre">pieces-&gt;end())</span></tt>. So you don't have to
count yourself. This can be used to see if anything has updated since last time
@ -7403,6 +7437,12 @@ the metadata (.torrent file) and it doesn't have it yet.
This happens for magnet links before they have downloaded the
metadata, and also torrents added by URL.</td>
</tr>
<tr><td>110</td>
<td>invalid_dont_have</td>
<td>The peer sent an invalid <tt class="docutils literal"><span class="pre">dont_have</span></tt> message. The dont have
message is an extension to allow peers to advertise that the
no longer has a piece they previously had.</td>
</tr>
</tbody>
</table>
<p>NAT-PMP errors:</p>
@ -8555,7 +8595,9 @@ bittorrent client.</p>
<div class="section" id="metadata-from-peers">
<h2>metadata from peers</h2>
<p>Extension name: &quot;LT_metadata&quot;</p>
<p>The point with this extension is that you don't have to distribute the
<p>This extension is deprecated in favor of the more widely supported <tt class="docutils literal"><span class="pre">ut_metadata</span></tt>
extension, see <a class="reference external" href="http://bittorrent.org/beps/bep_0009.html">BEP 9</a>.
The point with this extension is that you don't have to distribute the
metadata (.torrent-file) separately. The metadata can be distributed
through the bittorrent swarm. The only thing you need to download such
a torrent is the tracker url and the info-hash of the torrent.</p>
@ -8671,6 +8713,39 @@ doesn't have any metadata.</td>
</tbody>
</table>
</div>
<div class="section" id="dont-have">
<h2>dont_have</h2>
<p>Extension name: &quot;lt_dont_have&quot;</p>
<p>The <tt class="docutils literal"><span class="pre">dont_have</span></tt> extension message is used to tell peers that the client no longer
has a specific piece. The extension message should be advertised in the <tt class="docutils literal"><span class="pre">m</span></tt> dictionary
as <tt class="docutils literal"><span class="pre">lt_dont_have</span></tt>. The message format mimics the regular <tt class="docutils literal"><span class="pre">HAVE</span></tt> bittorrent message.</p>
<p>Just like all extension messages, the first 2 bytes in the mssage itself are 20 (the
bittorrent extension message) and the message ID assigned to this extension in the <tt class="docutils literal"><span class="pre">m</span></tt>
dictionary in the handshake.</p>
<table border="1" class="docutils">
<colgroup>
<col width="17%" />
<col width="23%" />
<col width="61%" />
</colgroup>
<thead valign="bottom">
<tr><th class="head">size</th>
<th class="head">name</th>
<th class="head">description</th>
</tr>
</thead>
<tbody valign="top">
<tr><td>uint32_t</td>
<td>piece</td>
<td>index of the piece the peer no longer
has.</td>
</tr>
</tbody>
</table>
<p>The length of this message (including the extension message prefix) is
6 bytes, i.e. one byte longer than the normal <tt class="docutils literal"><span class="pre">HAVE</span></tt> message, because
of the extension message wrapping.</p>
</div>
<div class="section" id="http-seeding">
<h2>HTTP seeding</h2>
<p>There are two kinds of HTTP seeding. One with that assumes a smart
@ -8687,6 +8762,105 @@ torrent's name '/' the file name is appended. This is the same directory
structure that libtorrent will download torrents into.</p>
</div>
</div>
<div class="section" id="piece-picker">
<h1>piece picker</h1>
<p>The piece picker in libtorrent has the following features:</p>
<ul class="simple">
<li>rarest first</li>
<li>sequential download</li>
<li>random pick</li>
<li>reverse order picking</li>
<li>parole mode</li>
<li>prioritize partial pieces</li>
<li>prefer whole pieces</li>
<li>piece affinity by speed category</li>
<li>piece priorities</li>
</ul>
<div class="section" id="internal-representation">
<h2>internal representation</h2>
<p>It is optimized by, at all times, keeping a list of pieces ordered
by rarity, randomly shuffled within each rarity class. This list
is organized as a single vector of contigous memory in RAM, for
optimal memory locality and to eliminate heap allocations and frees
when updating rarity of pieces.</p>
<p>Expensive events, like a peer joining or leaving, are evaluated
lazily, since it's cheaper to rebuild the whole list rather than
updating every single piece in it. This means as long as no blocks
are picked, peers joining and leaving is no more costly than a single
peer joining or leaving. Of course the special cases of peers that have
all or no pieces are optimized to not require rebuilding the list.</p>
</div>
<div class="section" id="picker-strategy">
<h2>picker strategy</h2>
<p>The normal mode of the picker is of course <em>rarest first</em>, meaning
pieces that few peers have are preferred to be downloaded over pieces
that more peers have. This is a fundamental algorithm that is the
basis of the performance of bittorrent. However, the user may set the
piece picker into sequential download mode. This mode simply picks
pieces sequentially, always preferring lower piece indices.</p>
<p>When a torrent starts out, picking the rarest pieces means increased
risk that pieces won't be completed early (since there are only a few
peers they can be downloaded from), leading to a delay of having any
piece to offer to other peers. This lack of pieces to trade, delays
the client from getting started into the normal tit-for-tat mode of
bittorrent, and will result in a long ramp-up time. The heuristic to
mitigate this problem is to, for the first few pieces, pick random pieces
rather than rare pieces. The threshold for when to leave this initial
picker mode is determined by <tt class="docutils literal"><span class="pre">session_settings::initial_picker_threshold</span></tt>.</p>
</div>
<div class="section" id="reverse-order">
<h2>reverse order</h2>
<p>An orthogonal setting is <em>reverse order</em>, which is used for <em>snubbed</em>
peers. Snubbed peers are peers that appear very slow, and might have timed
out a piece request. The idea behind this is to make all snubbed peers
more likely to be able to do download blocks from the same piece,
concentrating slow peers on as few pieces as possible. The reverse order
means that the most common pieces are picked, instead of the rarest pieces
(or in the case of sequential download, the last pieces, intead of the first).</p>
</div>
<div class="section" id="parole-mode">
<h2>parole mode</h2>
<p>Peers that have participated in a piece that failed the hash check, may be
put in <em>parole mode</em>. This means we prefer downloading a full piece from this
peer, in order to distinguish which peer is sending corrupt data. Whether to
do this is or not is controlled by <tt class="docutils literal"><span class="pre">session_settings::use_parole_mode</span></tt>.</p>
<p>In parole mode, the piece picker prefers picking one whole piece at a time for
a given peer, avoiding picking any blocks from a piece any other peer has
contributed to (since that would defeat the purpose of parole mode).</p>
</div>
<div class="section" id="prioritize-partial-pieces">
<h2>prioritize partial pieces</h2>
<p>This setting determines if partially downloaded or requested pieces should always
be preferred over other pieces. The benefit of doing this is that the number of
partial pieces is minimized (and hence the turn-around time for downloading a block
until it can be uploaded to others is minimized). It also puts less stress on the
disk cache, since fewer partial pieces need to be kept in the cache. Whether or
not to enable this is controlled by <tt class="docutils literal"><span class="pre">session_settings::prioritize_partial_pieces</span></tt>.</p>
<p>The main benefit of not prioritizing partial pieces is that the rarest first
algorithm gets to have more influence on which pieces are picked. The picker is
more likely to truly pick the rarest piece, and hence improving the performance
of the swarm.</p>
<p>This setting is turned on automatically whenever the number of partial pieces
in the piece picker exceeds the number of peers we're connected to times 1.5.
This is in order to keep the waste of partial pieces to a minimum, but still
prefer rarest pieces.</p>
</div>
<div class="section" id="prefer-whole-pieces">
<h2>prefer whole pieces</h2>
<p>The <em>prefer whole pieces</em> setting makes the piece picker prefer picking entire
pieces at a time. This is used by web connections (both http seeding
standards), in order to be able to coalesce the small bittorrent requests
to larger HTTP requests. This significantly improves performance when
downloading over HTTP.</p>
<p>It is also used by peers that are downloading faster than a certain
threshold. The main advantage is that these peers will better utilize the
other peer's disk cache, by requesting all blocks in a single piece, from
the same peer.</p>
<p>This threshold is controlled by <tt class="docutils literal"><span class="pre">session_settings::whole_pieces_threshold</span></tt>.</p>
<p><em>TODO: piece affinity by speed category</em>
<em>TODO: piece priorities</em></p>
</div>
</div>
<div class="section" id="filename-checks">
<h1>filename checks</h1>
<p>Boost.Filesystem will by default check all its paths to make sure they conform

145
docs/manual.rst
View File

@ -7597,6 +7597,10 @@ code symbol description
the metadata (.torrent file) and it doesn't have it yet.
This happens for magnet links before they have downloaded the
metadata, and also torrents added by URL.
------ ----------------------------------------- -----------------------------------------------------------------
110 invalid_dont_have The peer sent an invalid ``dont_have`` message. The dont have
message is an extension to allow peers to advertise that the
no longer has a piece they previously had.
====== ========================================= =================================================================
NAT-PMP errors:
@ -8604,6 +8608,8 @@ metadata from peers
Extension name: "LT_metadata"
This extension is deprecated in favor of the more widely supported ``ut_metadata``
extension, see `BEP 9`_.
The point with this extension is that you don't have to distribute the
metadata (.torrent-file) separately. The metadata can be distributed
through the bittorrent swarm. The only thing you need to download such
@ -8673,6 +8679,32 @@ Don't have metadata:
| | | doesn't have any metadata. |
+-----------+---------------+----------------------------------------+
.. _`BEP 9`: http://bittorrent.org/beps/bep_0009.html
dont_have
---------
Extension name: "lt_dont_have"
The ``dont_have`` extension message is used to tell peers that the client no longer
has a specific piece. The extension message should be advertised in the ``m`` dictionary
as ``lt_dont_have``. The message format mimics the regular ``HAVE`` bittorrent message.
Just like all extension messages, the first 2 bytes in the mssage itself are 20 (the
bittorrent extension message) and the message ID assigned to this extension in the ``m``
dictionary in the handshake.
+-----------+---------------+----------------------------------------+
| size | name | description |
+===========+===============+========================================+
| uint32_t | piece | index of the piece the peer no longer |
| | | has. |
+-----------+---------------+----------------------------------------+
The length of this message (including the extension message prefix) is
6 bytes, i.e. one byte longer than the normal ``HAVE`` message, because
of the extension message wrapping.
HTTP seeding
------------
@ -8694,6 +8726,119 @@ structure that libtorrent will download torrents into.
.. _`BEP 17`: http://bittorrent.org/beps/bep_0017.html
.. _`BEP 19`: http://bittorrent.org/beps/bep_0019.html
piece picker
============
The piece picker in libtorrent has the following features:
* rarest first
* sequential download
* random pick
* reverse order picking
* parole mode
* prioritize partial pieces
* prefer whole pieces
* piece affinity by speed category
* piece priorities
internal representation
-----------------------
It is optimized by, at all times, keeping a list of pieces ordered
by rarity, randomly shuffled within each rarity class. This list
is organized as a single vector of contigous memory in RAM, for
optimal memory locality and to eliminate heap allocations and frees
when updating rarity of pieces.
Expensive events, like a peer joining or leaving, are evaluated
lazily, since it's cheaper to rebuild the whole list rather than
updating every single piece in it. This means as long as no blocks
are picked, peers joining and leaving is no more costly than a single
peer joining or leaving. Of course the special cases of peers that have
all or no pieces are optimized to not require rebuilding the list.
picker strategy
---------------
The normal mode of the picker is of course *rarest first*, meaning
pieces that few peers have are preferred to be downloaded over pieces
that more peers have. This is a fundamental algorithm that is the
basis of the performance of bittorrent. However, the user may set the
piece picker into sequential download mode. This mode simply picks
pieces sequentially, always preferring lower piece indices.
When a torrent starts out, picking the rarest pieces means increased
risk that pieces won't be completed early (since there are only a few
peers they can be downloaded from), leading to a delay of having any
piece to offer to other peers. This lack of pieces to trade, delays
the client from getting started into the normal tit-for-tat mode of
bittorrent, and will result in a long ramp-up time. The heuristic to
mitigate this problem is to, for the first few pieces, pick random pieces
rather than rare pieces. The threshold for when to leave this initial
picker mode is determined by ``session_settings::initial_picker_threshold``.
reverse order
-------------
An orthogonal setting is *reverse order*, which is used for *snubbed*
peers. Snubbed peers are peers that appear very slow, and might have timed
out a piece request. The idea behind this is to make all snubbed peers
more likely to be able to do download blocks from the same piece,
concentrating slow peers on as few pieces as possible. The reverse order
means that the most common pieces are picked, instead of the rarest pieces
(or in the case of sequential download, the last pieces, intead of the first).
parole mode
-----------
Peers that have participated in a piece that failed the hash check, may be
put in *parole mode*. This means we prefer downloading a full piece from this
peer, in order to distinguish which peer is sending corrupt data. Whether to
do this is or not is controlled by ``session_settings::use_parole_mode``.
In parole mode, the piece picker prefers picking one whole piece at a time for
a given peer, avoiding picking any blocks from a piece any other peer has
contributed to (since that would defeat the purpose of parole mode).
prioritize partial pieces
-------------------------
This setting determines if partially downloaded or requested pieces should always
be preferred over other pieces. The benefit of doing this is that the number of
partial pieces is minimized (and hence the turn-around time for downloading a block
until it can be uploaded to others is minimized). It also puts less stress on the
disk cache, since fewer partial pieces need to be kept in the cache. Whether or
not to enable this is controlled by ``session_settings::prioritize_partial_pieces``.
The main benefit of not prioritizing partial pieces is that the rarest first
algorithm gets to have more influence on which pieces are picked. The picker is
more likely to truly pick the rarest piece, and hence improving the performance
of the swarm.
This setting is turned on automatically whenever the number of partial pieces
in the piece picker exceeds the number of peers we're connected to times 1.5.
This is in order to keep the waste of partial pieces to a minimum, but still
prefer rarest pieces.
prefer whole pieces
-------------------
The *prefer whole pieces* setting makes the piece picker prefer picking entire
pieces at a time. This is used by web connections (both http seeding
standards), in order to be able to coalesce the small bittorrent requests
to larger HTTP requests. This significantly improves performance when
downloading over HTTP.
It is also used by peers that are downloading faster than a certain
threshold. The main advantage is that these peers will better utilize the
other peer's disk cache, by requesting all blocks in a single piece, from
the same peer.
This threshold is controlled by ``session_settings::whole_pieces_threshold``.
*TODO: piece affinity by speed category*
*TODO: piece priorities*
filename checks
===============