merged changes from RC_1_0

This commit is contained in:
Arvid Norberg 2014-09-01 07:16:31 +00:00
parent ddb133e055
commit b1208c9f63
20 changed files with 494 additions and 1982 deletions

View File

@ -21,10 +21,12 @@ DOCS_IMAGES = \
docs/our_delay_base.png \ docs/our_delay_base.png \
docs/our_delay_base_thumb.png \ docs/our_delay_base_thumb.png \
docs/read_disk_buffers.png \ docs/read_disk_buffers.png \
docs/read_disk_buffers.diagram \
docs/session_stats_peers.png \ docs/session_stats_peers.png \
docs/storage.png \ docs/storage.png \
docs/todo.html \ docs/todo.html \
docs/write_disk_buffers.png \ docs/write_disk_buffers.png \
docs/write_disk_buffers.diagram \
docs/ziptorrent_thumb.gif \ docs/ziptorrent_thumb.gif \
docs/disk_access_elevator.png \ docs/disk_access_elevator.png \
docs/disk_access_no_elevator.png\ docs/disk_access_no_elevator.png\
@ -32,8 +34,13 @@ DOCS_IMAGES = \
docs/ip_id_v6.png \ docs/ip_id_v6.png \
docs/hash_distribution.png \ docs/hash_distribution.png \
docs/complete_bit_prefixes.png \ docs/complete_bit_prefixes.png \
docs/troubleshooting.dot \
docs/troubleshooting.png \ docs/troubleshooting.png \
docs/troubleshooting_thumb.png \ docs/troubleshooting_thumb.png \
docs/hacking.diagram \
docs/hacking.png \
docs/utp_stack.diagram \
docs/utp_stack.png \
docs/style.css \ docs/style.css \
docs/base.css \ docs/base.css \
docs/rst.css \ docs/rst.css \
@ -66,6 +73,24 @@ DOCS_PAGES = \
docs/troubleshooting.html \ docs/troubleshooting.html \
docs/udp_tracker_protocol.html \ docs/udp_tracker_protocol.html \
docs/utp.html \ docs/utp.html \
docs/building.rst \
docs/client_test.rst \
docs/contributing.rst \
docs/dht_extensions.rst \
docs/dht_rss.rst \
docs/dht_sec.rst \
docs/dht_store.rst \
docs/examples.rst \
docs/extension_protocol.rst \
docs/features.rst \
docs/index.rst \
docs/manual-ref.rst \
docs/projects.rst \
docs/python_binding.rst \
docs/tuning.rst \
docs/troubleshooting.rst \
docs/udp_tracker_protocol.rst \
docs/utp.rst \
docs/reference-Alerts.html \ docs/reference-Alerts.html \
docs/reference-Bencoding.html \ docs/reference-Bencoding.html \
docs/reference-Core.html \ docs/reference-Core.html \

View File

@ -276,6 +276,7 @@ to provide additional bandwidth to an entire feed.</p>
</div> </div>
<div class="section" id="merkle-hash-tree-torrents"> <div class="section" id="merkle-hash-tree-torrents">
<h2>merkle hash tree torrents</h2> <h2>merkle hash tree torrents</h2>
<img alt="merkle_tree.png" class="align-right" src="merkle_tree.png" />
<p>Merkle hash tree torrents is an extension that lets a torrent file only contain the <p>Merkle hash tree torrents is an extension that lets a torrent file only contain the
root hash of the hash tree forming the piece hashes. The main benefit of this feature root hash of the hash tree forming the piece hashes. The main benefit of this feature
is that regardless of how many pieces there is in a torrent, the .torrent file will is that regardless of how many pieces there is in a torrent, the .torrent file will
@ -299,13 +300,12 @@ piece size. With merkle torrents, the piece size can be the minimum block size (
which lets peers verify every block of data received from peers, immediately. This which lets peers verify every block of data received from peers, immediately. This
gives a minimum turnaround time and completely removes the problem of identifying malicious gives a minimum turnaround time and completely removes the problem of identifying malicious
peers.</p> peers.</p>
<img alt="merkle_tree.png" src="merkle_tree.png" />
<p>The root hash is built by hashing all the piece hashes pair-wise, until they all collapse <p>The root hash is built by hashing all the piece hashes pair-wise, until they all collapse
down to the root.</p> down to the root.</p>
<img alt="storage.png" class="align-right" src="storage.png" />
</div> </div>
<div class="section" id="customizable-file-storage"> <div class="section" id="customizable-file-storage">
<h2>customizable file storage</h2> <h2>customizable file storage</h2>
<img alt="storage.png" class="align-right" src="storage.png" />
<p>libtorrent's storage implementation is customizable. That means a special purpose bittorrent <p>libtorrent's storage implementation is customizable. That means a special purpose bittorrent
client can replace the default way to store files on disk.</p> client can replace the default way to store files on disk.</p>
<p>When implementing a bittorrent cache, it doesn't matter how the data is stored on disk, as <p>When implementing a bittorrent cache, it doesn't matter how the data is stored on disk, as

View File

@ -277,6 +277,9 @@ to provide additional bandwidth to an entire feed.
merkle hash tree torrents merkle hash tree torrents
------------------------- -------------------------
.. image:: merkle_tree.png
:align: right
Merkle hash tree torrents is an extension that lets a torrent file only contain the Merkle hash tree torrents is an extension that lets a torrent file only contain the
root hash of the hash tree forming the piece hashes. The main benefit of this feature root hash of the hash tree forming the piece hashes. The main benefit of this feature
is that regardless of how many pieces there is in a torrent, the .torrent file will is that regardless of how many pieces there is in a torrent, the .torrent file will
@ -305,17 +308,15 @@ which lets peers verify every block of data received from peers, immediately. Th
gives a minimum turnaround time and completely removes the problem of identifying malicious gives a minimum turnaround time and completely removes the problem of identifying malicious
peers. peers.
.. image:: merkle_tree.png
The root hash is built by hashing all the piece hashes pair-wise, until they all collapse The root hash is built by hashing all the piece hashes pair-wise, until they all collapse
down to the root. down to the root.
.. image:: storage.png
:align: right
customizable file storage customizable file storage
------------------------- -------------------------
.. image:: storage.png
:align: right
libtorrent's storage implementation is customizable. That means a special purpose bittorrent libtorrent's storage implementation is customizable. That means a special purpose bittorrent
client can replace the default way to store files on disk. client can replace the default way to store files on disk.

View File

@ -45,6 +45,10 @@ for f in files:
items[-1]['priority'] = 0 items[-1]['priority'] = 0
if line[0] in '0123456789': if line[0] in '0123456789':
items[-1]['priority'] = int(line[0]) items[-1]['priority'] = int(line[0])
if int(line[0]) > 5:
print 'priority too high: ' + line
sys.exit(1)
line = line[1:].strip() line = line[1:].strip()
items[-1]['todo'] = line items[-1]['todo'] = line
prio = items[-1]['priority'] prio = items[-1]['priority']

26
docs/hacking.diagram Normal file
View File

@ -0,0 +1,26 @@
+--------------+ pimpl +--------------+
| cGRE session +----------->| session_impl |
+--------------+ +------+-----+-+
m_torrents[] | |
+---------------------+ | |
| cGRE torrent_handle +-------+ | |
+---------------------+ weak | | |
| | | m_connections[]
| | +---+-------+
| | | |
m_picker v v | v peers we are connected to
+--------------+ +--------++ +-----------------+
| piece_picker |<---+-+ torrent ++ +--+ peer_connection ++
+--------------+ | ++--------+| | ++----------------+|
m_torrent_file | +---------+ | +-----------------+
+-------------------+ | |
| cGRE torrent_info |<---+ | m_socket
+-------------------+ | | +--------------------------+
| +------+->| socket_type (variant) |
+--------+ m_policy | | | (TCP/uTP/SSL/socks5/...) |
| policy |<---------+ | +--------------------------+
+------+-+ v
list of all | m_peers[] +--------------+
peers we +-------------->| policy::peer ++ contains contact information
know of ++-------------+| for peers we're not necessarily
+--------------+ connected to

View File

@ -62,43 +62,8 @@ structure
This is the high level structure of libtorrent. Bold types are part of the public This is the high level structure of libtorrent. Bold types are part of the public
interface: interface:
.. parsed-literal::
+=========+ pimpl +-------------------+ .. image:: hacking.png
| **session** | ---------> | aux::session_impl |
+=========+ +-------------------+
m_torrents[] | |
+================+ | |
| **torrent_handle** | ------+ | |
+================+ | | |
| | | m_connections[]
| | |
| | +---------------------+
m_picker v v |
+--------------+ +---------+---------+-- . . |
| piece_picker | <--+-| torrent | torrent | to |
+--------------+ | +---------+---------+-- . . |
m_torrent_file | | m_connections[] |
+==============+ | | |
| **torrent_info** | <--+ v v
+==============+ | +-----------------+-----------------+-- . .
m_policy | | peer_connection | peer_connection | pe
+--------+ | +-----------------+-----------------+-- . .
| policy | <--------+ | | m_socket
+--------+ | |
| m_peers[] | v
| | +-----------------------+
| | | socket_type (variant) |
v | +-----------------------+
+--------------+ |
| policy::peer | |
+--------------+ |
| policy::peer | |
+--------------+ m_peer_info|
| policy::peer | <----------+
+--------------+
. .
+ - - - - - - -+
session_impl session_impl
------------ ------------

View File

@ -42,7 +42,13 @@ TARGETS = index \
streaming \ streaming \
$(REFERENCE_TARGETS) $(REFERENCE_TARGETS)
FIGURES = read_disk_buffers write_disk_buffers troubleshooting FIGURES = \
read_disk_buffers \
write_disk_buffers \
troubleshooting \
hacking \
utp_stack \
storage
html: $(TARGETS:=.html) $(FIGURES:=.png) todo.html html: $(TARGETS:=.html) $(FIGURES:=.png) todo.html
@ -65,10 +71,6 @@ troubleshooting_thumb.png: troubleshooting.png
convert troubleshooting.png -resize 800x800 troubleshooting_thumb.png convert troubleshooting.png -resize 800x800 troubleshooting_thumb.png
cp $@ $(WEB_PATH)/$@ cp $@ $(WEB_PATH)/$@
troubleshooting.png: troubleshooting.dot
dot troubleshooting.dot -Tpng >troubleshooting.png
cp $@ $(WEB_PATH)/$@
todo.html:gen_todo.py ../src/*.cpp ../include/libtorrent/*.hpp todo.html:gen_todo.py ../src/*.cpp ../include/libtorrent/*.hpp
python gen_todo.py python gen_todo.py
cp $@ $(WEB_PATH)/$@ cp $@ $(WEB_PATH)/$@
@ -94,6 +96,10 @@ $(REFERENCE_TARGETS:=.rst):gen_reference_doc.py ../include/libtorrent/*.hpp ../i
dot -Teps $? >$@ dot -Teps $? >$@
cp $@ $(WEB_PATH)/$@ cp $@ $(WEB_PATH)/$@
%.png:%.diagram
java -jar /opt/local/share/java/ditaa0_9.jar -E $? $@
cp $@ $(WEB_PATH)/$@
clean: clean:
rm -f $(TARGETS:=.html) $(TARGETS:=.pdf) settings.rst todo.html reference*.html reference*.rst rm -f $(TARGETS:=.html) $(TARGETS:=.pdf) settings.rst todo.html reference*.html reference*.rst

View File

@ -0,0 +1,16 @@
copy into peer's encrypt in place
+----------------+ send buffer +-------------+ (no copy) +-------------+
| receive buffer +----------------->| send buffer +--=--------------->| encrypted |
| | | | | send buffer |
+----------------+ +-------------+ +------+------+
^ |
| read() from file write() to socket |
| (copy) (copy) |
---=----|---------------------------------=---------------------------------|--=----
| kernel space |
| v
+-------+-----------+ +---------------+
| kernel page cache | | socket kernel |
| | | buffer |
+-------------------+ +---------------+

View File

@ -1,20 +0,0 @@
digraph uploading {
node [shape=box];
subgraph user_space {
rank=same;
"disk cache" -> "send buffer" [label="copy into peer's send buffer (copy)"]
"send buffer" -> "encrypted send buffer" [label="encrypt in-place (no copy)" style=dashed];
}
subgraph kernel {
rank=same;
"kernel page cache";
"socket kernel buffer"
}
"encrypted send buffer" -> "socket kernel buffer" [label="write() to socket (copy)"];
"kernel page cache" -> "disk cache" [label="read() from file (copy)"]
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 16 KiB

31
docs/storage.diagram Normal file
View File

@ -0,0 +1,31 @@
+--------------------------+
| disk_io_thread |
| (manages piece cache) |
+--------------------------+
^
|
v
+--------------------------+
| piece_manager |
| (maps pieces to slots) |
+--------------------------+
^
|
v customization point
/--------------------------\ +------------------+
| storage |<---->| file_pool |
| cGRE | | open file cache |
| (maps slots to file and | +------------------+
| offset. reads and writes |
| to disk) | +------------------+
| |<-----+ file_storage |
\--------------------------/ | standard slot to |
^ | file mapping |
| +------------------+
v
+--------------------------+
| file |
| (file class reads and |
| writes files) |
+--------------------------+

File diff suppressed because it is too large Load Diff

Binary file not shown.

Before

Width:  |  Height:  |  Size: 30 KiB

After

Width:  |  Height:  |  Size: 21 KiB

View File

@ -39,6 +39,9 @@ li { margin-left: 2.8em; font-size: 92%; }
p, ul, ol, img {margin-bottom: 1em;} p, ul, ol, img {margin-bottom: 1em;}
.align-right {
float: right;
}
.document { .document {
margin-left: 10px; margin-left: 10px;

File diff suppressed because it is too large Load Diff

9
docs/utp_stack.diagram Normal file
View File

@ -0,0 +1,9 @@
+-----------------------+
| BitTorrent protocol |
+-----------------------+
| SSL |
+-----------+-----------+
| TCP | uTP |
| +-----------+
| | UDP |
+-----------+-----------+

View File

@ -0,0 +1,15 @@
decrypt in place move buffer ref.
+----------------+ (no copy) +------------+ (no copy) +------------+
| receive buffer +--=-------------->| plain text +--=--------------->| disk cache |
+----------------+ | buffer | +------+-----+
^ +------------+ |
| read() on socket write() to file |
| (copy) (copy) |
---=----|---------------------------------=---------------------------------|--=----
| kernel space |
| v
+-------+--------+ +-------------------+
| socket kernel | | kernel page cache |
| buffer | | |
+----------------+ +-------------------+

View File

@ -1,20 +0,0 @@
digraph downloading {
label=""
node [shape=box];
subgraph user_space {
rank=same;
"receive buffer" -> "plain text buffer" [label="decrypt in-place (no copy)" style=dashed];
"plain text buffer" -> "disk cache" [label="move buffer reference (no copy)" style=dashed]
}
subgraph kernel {
rank=same;
"socket kernel buffer";
"kernel page cache"
}
"socket kernel buffer" -> "receive buffer" [label="read() on socket (copy)"];
"disk cache" -> "kernel page cache" [label="write() to file (copy)"]
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 15 KiB

View File

@ -1568,7 +1568,7 @@ namespace libtorrent
// specified in session_time // specified in session_time
boost::uint16_t m_last_download; boost::uint16_t m_last_download;
// TODO: 8 bits free here // TODO: There are 8 bits free here
// this is a second count-down to when we should tick the // this is a second count-down to when we should tick the
// storage for this torrent. Ticking the storage is used // storage for this torrent. Ticking the storage is used
@ -1584,7 +1584,7 @@ namespace libtorrent
// specified in session_time // specified in session_time
boost::uint16_t m_last_upload; boost::uint16_t m_last_upload;
// TODO: 8 bits here // TODO: There are 8 bits here
// if this is true, libtorrent may pause and resume // if this is true, libtorrent may pause and resume
// this torrent depending on queuing rules. Torrents // this torrent depending on queuing rules. Torrents