premiere
/
premiere-libtorrent
2
2
Fork 1
Code Issues Pull Requests Releases Wiki Activity

fixed incorrect sorting of pieces in piece_picker

This commit is contained in:
Arvid Norberg 2006-07-16 00:08:50 +00:00
parent 8c6869b4f2
commit 8d31bf442a
6 changed files with 101 additions and 93 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
docs/manual.html
View File

@ -186,8 +186,7 @@ project (including this documentation). The current state includes the
following features:</p>
<ul class="simple">
<li>multitracker extension support (as <a class="reference" href="http://home.elp.rr.com/tur/multitracker-spec.txt">specified by John Hoffman</a>)</li>
<li>serves multiple torrents on a single port and a single thread</li>
<li>supports http proxies and proxy authentication</li>
<li>serves multiple torrents on a single port and in a single thread</li>
<li>gzipped tracker-responses</li>
<li><a class="reference" href="#http-seeding">HTTP seeding</a>, as <a class="reference" href="http://www.getright.com/seedtorrent.html">specified by Michael Burford of GetRight</a>.</li>
<li>piece picking on block-level like in <a class="reference" href="http://azureus.sourceforge.net">Azureus</a> (as opposed to piece-level).
@ -195,6 +194,7 @@ This means it can download parts of the same piece from different peers.
It will also prefer to download whole pieces from single peers if the
download speed is high enough from that particular peer.</li>
<li>queues torrents for file check, instead of checking all of them in parallel.</li>
<li>supports http proxies and proxy authentication</li>
<li>uses separate threads for checking files and for main downloader, with a
fool-proof thread-safe library interface. (i.e. There's no way for the
user to cause a deadlock). (see <a class="reference" href="#threads">threads</a>)</li>
@ -232,7 +232,7 @@ network APIs on the most popular platforms. I/O completion ports on windows,
epoll on linux and kqueue on MacOS X and BSD.</p>
<p>libtorrent has been successfully compiled and tested on:</p>
<ul class="simple">
<li>Windows 2000 vc7.1</li>
<li>Windows 2000 vc7.1, vc8</li>
<li>Linux x86 GCC 3.3, GCC 3.4.2</li>
<li>MacOS X (darwin), (Apple's) GCC 3.3, (Apple's) GCC 4.0</li>
<li>SunOS 5.8 GCC 3.1</li>
@ -1958,6 +1958,8 @@ peers this client is connected to. The fractional part tells the share of
pieces that have more copies than the rarest piece(s). For example: 2.5 would
mean that the rarest pieces have only 2 copies among the peers this torrent is
connected to, and that 50% of all the pieces have more than two copies.</p>
<p>If sequenced download is activated (in <a class="reference" href="#session-settings">session_settings</a>), the distributed
copies will be saturated at the <tt class="docutils literal"><span class="pre">sequenced_download_threshold</span></tt>.</p>
<p><tt class="docutils literal"><span class="pre">block_size</span></tt> is the size of a block, in bytes. A block is a sub piece, it
is the number of bytes that each piece request asks for and the number of
bytes that each bit in the <tt class="docutils literal"><span class="pre">partial_piece_info</span></tt>'s bitset represents
@ -2175,7 +2177,8 @@ actual number of requests depends on the download rate and this number.</p>
random (rarest first) order. It can be used to tweak disk performance in
settings where the random download property is less necessary. For example, if
the threshold is 10, all pieces which 10 or more peers have, will be downloaded
in index order.</p>
in index order. This setting defaults to 100, which means that it is disabled
in practice.</p>
<p><tt class="docutils literal"><span class="pre">max_allowed_in_request_queue</span></tt> is the number of outstanding block requests
a peer is allowed to queue up in the client. If a peer sends more requests
than this (before the first one has been handled) the last request will be

12
docs/manual.rst
View File

@ -28,8 +28,7 @@ project (including this documentation). The current state includes the
following features:
* multitracker extension support (as `specified by John Hoffman`__)
* serves multiple torrents on a single port and a single thread
* supports http proxies and proxy authentication
* serves multiple torrents on a single port and in a single thread
* gzipped tracker-responses
* `HTTP seeding`_, as `specified by Michael Burford of GetRight`__.
* piece picking on block-level like in Azureus_ (as opposed to piece-level).
@ -37,6 +36,7 @@ following features:
It will also prefer to download whole pieces from single peers if the
download speed is high enough from that particular peer.
* queues torrents for file check, instead of checking all of them in parallel.
* supports http proxies and proxy authentication
* uses separate threads for checking files and for main downloader, with a
fool-proof thread-safe library interface. (i.e. There's no way for the
user to cause a deadlock). (see threads_)
@ -85,7 +85,7 @@ epoll on linux and kqueue on MacOS X and BSD.
libtorrent has been successfully compiled and tested on:
* Windows 2000 vc7.1
* Windows 2000 vc7.1, vc8
* Linux x86 GCC 3.3, GCC 3.4.2
* MacOS X (darwin), (Apple's) GCC 3.3, (Apple's) GCC 4.0
* SunOS 5.8 GCC 3.1
@ -1953,6 +1953,9 @@ pieces that have more copies than the rarest piece(s). For example: 2.5 would
mean that the rarest pieces have only 2 copies among the peers this torrent is
connected to, and that 50% of all the pieces have more than two copies.
If sequenced download is activated (in session_settings_), the distributed
copies will be saturated at the ``sequenced_download_threshold``.
``block_size`` is the size of a block, in bytes. A block is a sub piece, it
is the number of bytes that each piece request asks for and the number of
bytes that each bit in the ``partial_piece_info``'s bitset represents
@ -2187,7 +2190,8 @@ actual number of requests depends on the download rate and this number.
random (rarest first) order. It can be used to tweak disk performance in
settings where the random download property is less necessary. For example, if
the threshold is 10, all pieces which 10 or more peers have, will be downloaded
in index order.
in index order. This setting defaults to 100, which means that it is disabled
in practice.
``max_allowed_in_request_queue`` is the number of outstanding block requests
a peer is allowed to queue up in the client. If a peer sends more requests

2
include/libtorrent/piece_picker.hpp
View File

@ -212,7 +212,7 @@ namespace libtorrent
int num_have_filtered() const { return m_num_have_filtered; }
#ifndef NDEBUG
// used in debug mode
void integrity_check(const torrent* t = 0) const;
void check_invariant(const torrent* t = 0) const;
#endif
// functor that compares indices on downloading_pieces

2
include/libtorrent/session_settings.hpp
View File

@ -50,7 +50,7 @@ namespace libtorrent
, tracker_maximum_response_length(1024*1024)
, piece_timeout(120)
, request_queue_time(3.f)
, sequenced_download_threshold(10)
, sequenced_download_threshold(100)
, max_allowed_in_request_queue(250)
, max_out_request_queue(200)
, whole_pieces_threshold(20)

154
src/piece_picker.cpp
View File

@ -35,6 +35,9 @@ POSSIBILITY OF SUCH DAMAGE.
#include <algorithm>
#include <numeric>
// non-standard header, is_sorted()
//#include <algo.h>
#include "libtorrent/piece_picker.hpp"
#ifndef NDEBUG
@ -42,6 +45,9 @@ POSSIBILITY OF SUCH DAMAGE.
#include "libtorrent/torrent.hpp"
#endif
//#define TORRENT_PIECE_PICKER_INVARIANT_CHECK INVARIANT_CHECK
#define TORRENT_PIECE_PICKER_INVARIANT_CHECK
namespace libtorrent
{
@ -85,10 +91,6 @@ namespace libtorrent
std::vector<int> piece_list;
piece_list.reserve(std::count(pieces.begin(), pieces.end(), false));
#ifndef NDEBUG
integrity_check();
#endif
for (std::vector<bool>::const_iterator i = pieces.begin();
i != pieces.end(); ++i)
{
@ -121,10 +123,6 @@ namespace libtorrent
assert(m_piece_map[index].index != piece_pos::we_have_index);
}
#ifndef NDEBUG
integrity_check();
#endif
// if we have fast resume info
// use it
if (!unfinished.empty())
@ -140,17 +138,12 @@ namespace libtorrent
}
}
}
#ifndef NDEBUG
integrity_check();
#endif
}
void piece_picker::set_sequenced_download_threshold(
int sequenced_download_threshold)
{
#ifndef NDEBUG
integrity_check();
#endif
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
if (sequenced_download_threshold == m_sequenced_download_threshold)
return;
@ -168,14 +161,11 @@ namespace libtorrent
move(p.downloading, p.filtered, prev_priority, i->index);
}
}
#ifndef NDEBUG
integrity_check();
#endif
}
#ifndef NDEBUG
void piece_picker::integrity_check(const torrent* t) const
void piece_picker::check_invariant(const torrent* t) const
{
assert(sizeof(piece_pos) == 4);
@ -265,7 +255,10 @@ namespace libtorrent
const std::vector<std::vector<int> >& c_vec = pick_piece_info_vector(i->downloading, i->filtered);
assert(i->priority(m_sequenced_download_threshold) < (int)c_vec.size());
const std::vector<int>& vec = c_vec[i->priority(m_sequenced_download_threshold)];
assert(i->index < vec.size());
if (i->index >= vec.size())
{
assert(false);
}
assert(vec[i->index] == index);
}
@ -339,8 +332,9 @@ namespace libtorrent
{
// the piece should be inserted ordered, not randomly
std::vector<int>& v = dst_vec[priority];
// assert(is_sorted(v.begin(), v.end()/*, std::greater<int>()*/));
std::vector<int>::iterator i = std::lower_bound(v.begin(), v.end()
, index, std::greater<int>());
, index/*, std::greater<int>()*/);
p.index = i - v.begin();
v.insert(i, index);
i = v.begin() + p.index + 1;
@ -349,6 +343,7 @@ namespace libtorrent
++m_piece_map[*i].index;
assert(v[m_piece_map[*i].index] == *i);
}
// assert(is_sorted(v.begin(), v.end()/*, std::greater<int>()*/));
}
else if (dst_vec[priority].size() < 2)
{
@ -394,13 +389,19 @@ namespace libtorrent
piece_pos& p = m_piece_map[index];
int new_priority = p.priority(m_sequenced_download_threshold);
assert(p.downloading != downloading
|| p.filtered != filtered
|| (int)new_priority != priority);
if (p.downloading == downloading
&& p.filtered == filtered
&& new_priority == priority)
{
assert(p.ordered(m_sequenced_download_threshold));
return;
}
std::vector<std::vector<int> >& dst_vec(pick_piece_info_vector(
p.downloading, p.filtered));
assert(&dst_vec != &src_vec || new_priority != priority);
if ((int)dst_vec.size() <= new_priority)
{
dst_vec.resize(new_priority + 1);
@ -411,9 +412,9 @@ namespace libtorrent
{
// the piece should be inserted ordered, not randomly
std::vector<int>& v = dst_vec[new_priority];
// assert(is_sorted(v.begin(), v.end()/*, std::greater<int>()*/));
std::vector<int>::iterator i = std::lower_bound(v.begin(), v.end()
, index, std::greater<int>());
, index/*, std::greater<int>()*/);
p.index = i - v.begin();
v.insert(i, index);
i = v.begin() + p.index + 1;
@ -422,6 +423,7 @@ namespace libtorrent
++m_piece_map[*i].index;
assert(v[m_piece_map[*i].index] == *i);
}
// assert(is_sorted(v.begin(), v.end()/*, std::greater<int>()*/));
}
else if (dst_vec[new_priority].size() < 2)
{
@ -448,25 +450,40 @@ namespace libtorrent
assert(p.index < dst_vec[p.priority(m_sequenced_download_threshold)].size());
assert(dst_vec[p.priority(m_sequenced_download_threshold)][p.index] == index);
// this will remove elem from the source vector without
// preserving order, but the order is random anyway
int replace_index = src_vec[priority][elem_index] = src_vec[priority].back();
if (index != replace_index)
if (priority >= m_sequenced_download_threshold)
{
// update the entry we moved from the back
m_piece_map[replace_index].index = elem_index;
assert((int)src_vec[priority].size() > elem_index);
assert((int)m_piece_map[replace_index].priority(m_sequenced_download_threshold) == priority);
assert((int)m_piece_map[replace_index].index == elem_index);
assert(src_vec[priority][elem_index] == replace_index);
// remove the element from the source vector and preserve the order
std::vector<int>& v = src_vec[priority];
v.erase(v.begin() + elem_index);
for (std::vector<int>::iterator i = v.begin() + elem_index;
i != v.end(); ++i)
{
--m_piece_map[*i].index;
assert(v[m_piece_map[*i].index] == *i);
}
}
else
{
assert((int)src_vec[priority].size() == elem_index+1);
}
// this will remove elem from the source vector without
// preserving order, but the order is random anyway
int replace_index = src_vec[priority][elem_index] = src_vec[priority].back();
if (index != replace_index)
{
// update the entry we moved from the back
m_piece_map[replace_index].index = elem_index;
src_vec[priority].pop_back();
assert((int)src_vec[priority].size() > elem_index);
assert((int)m_piece_map[replace_index].priority(m_sequenced_download_threshold) == priority);
assert((int)m_piece_map[replace_index].index == elem_index);
assert(src_vec[priority][elem_index] == replace_index);
}
else
{
assert((int)src_vec[priority].size() == elem_index+1);
}
src_vec[priority].pop_back();
}
}
void piece_picker::remove(bool downloading, bool filtered, int priority
@ -497,6 +514,7 @@ namespace libtorrent
if (p.ordered(m_sequenced_download_threshold))
{
std::vector<int>& v = src_vec[priority];
// assert(is_sorted(v.begin(), v.end()/*, std::greater<int>()*/));
std::vector<int>::iterator i = v.begin() + elem_index;
v.erase(i);
i = v.begin() + elem_index;
@ -505,6 +523,7 @@ namespace libtorrent
--m_piece_map[*i].index;
assert(v[m_piece_map[*i].index] == *i);
}
// assert(is_sorted(v.begin(), v.end()/*, std::greater<int>()*/));
}
else
{
@ -520,6 +539,8 @@ namespace libtorrent
void piece_picker::restore_piece(int index)
{
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
assert(index >= 0);
assert(index < (int)m_piece_map.size());
@ -536,14 +557,11 @@ namespace libtorrent
piece_pos& p = m_piece_map[index];
if (p.filtered) return;
move(true, p.filtered, p.priority(m_sequenced_download_threshold), p.index);
#ifndef NDEBUG
// integrity_check();
#endif
}
void piece_picker::inc_refcount(int i)
{
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
assert(i >= 0);
assert(i < (int)m_piece_map.size());
@ -571,9 +589,7 @@ namespace libtorrent
void piece_picker::dec_refcount(int i)
{
#ifndef NDEBUG
// integrity_check();
#endif
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
assert(i >= 0);
assert(i < (int)m_piece_map.size());
@ -599,6 +615,7 @@ namespace libtorrent
// be removed from the available piece list.
void piece_picker::we_have(int index)
{
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
assert(index >= 0);
assert(index < (int)m_piece_map.size());
@ -618,20 +635,14 @@ namespace libtorrent
if (info_index == piece_pos::we_have_index) return;
remove(p.downloading, p.filtered, priority, info_index);
p.index = piece_pos::we_have_index;
#ifndef NDEBUG
integrity_check();
#endif
}
void piece_picker::mark_as_filtered(int index)
{
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
assert(index >= 0);
assert(index < (int)m_piece_map.size());
#ifndef NDEBUG
// integrity_check();
#endif
piece_pos& p = m_piece_map[index];
if (p.filtered == 1) return;
@ -646,10 +657,6 @@ namespace libtorrent
{
++m_num_have_filtered;
}
#ifndef NDEBUG
integrity_check();
#endif
}
// this function can be used for pieces that we don't
@ -659,6 +666,7 @@ namespace libtorrent
// be inserted in the available piece list again
void piece_picker::mark_as_unfiltered(int index)
{
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
assert(index >= 0);
assert(index < (int)m_piece_map.size());
@ -676,10 +684,6 @@ namespace libtorrent
--m_num_have_filtered;
assert(m_num_have_filtered >= 0);
}
#ifndef NDEBUG
integrity_check();
#endif
}
bool piece_picker::is_filtered(int index) const
@ -706,13 +710,10 @@ namespace libtorrent
, int num_blocks, bool prefer_whole_pieces
, tcp::endpoint peer) const
{
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
assert(num_blocks > 0);
assert(pieces.size() == m_piece_map.size());
#ifndef NDEBUG
// integrity_check();
#endif
// free refers to pieces that are free to download, no one else
// is downloading them.
// partial is pieces that are partially being downloaded, and
@ -973,9 +974,8 @@ namespace libtorrent
void piece_picker::mark_as_downloading(piece_block block, const tcp::endpoint& peer)
{
#ifndef NDEBUG
// integrity_check();
#endif
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
assert(block.piece_index >= 0);
assert(block.block_index >= 0);
assert(block.piece_index < (int)m_piece_map.size());
@ -1002,16 +1002,12 @@ namespace libtorrent
i->info[block.block_index].peer = peer;
i->requested_blocks[block.block_index] = 1;
}
#ifndef NDEBUG
// integrity_check();
#endif
}
void piece_picker::mark_as_finished(piece_block block, const tcp::endpoint& peer)
{
#ifndef NDEBUG
// integrity_check();
#endif
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
assert(block.piece_index >= 0);
assert(block.block_index >= 0);
assert(block.piece_index < (int)m_piece_map.size());
@ -1041,9 +1037,6 @@ namespace libtorrent
i->requested_blocks[block.block_index] = 1;
i->finished_blocks[block.block_index] = 1;
}
#ifndef NDEBUG
// integrity_check();
#endif
}
/*
void piece_picker::mark_as_finished(piece_block block, const peer_id& peer)
@ -1110,9 +1103,7 @@ namespace libtorrent
void piece_picker::abort_download(piece_block block)
{
#ifndef NDEBUG
// integrity_check();
#endif
TORRENT_PIECE_PICKER_INVARIANT_CHECK;
assert(block.piece_index >= 0);
assert(block.block_index >= 0);
@ -1151,9 +1142,6 @@ namespace libtorrent
piece_pos& p = m_piece_map[block.piece_index];
move(true, p.filtered, p.priority(m_sequenced_download_threshold), p.index);
}
#ifndef NDEBUG
// integrity_check();
#endif
}
int piece_picker::unverified_blocks() const

13
src/session.cpp
View File

@ -996,7 +996,20 @@ namespace libtorrent { namespace detail
assert(m_abort);
m_abort = true;
for (connection_map::iterator i = m_connections.begin();
i != m_connections.end(); ++i)
{
i->second->disconnect();
}
m_connections.clear();
for (connection_map::iterator i = m_half_open.begin();
i != m_half_open.end(); ++i)
{
i->second->disconnect();
}
m_half_open.clear();
m_connection_queue.clear();