premiere
/
premiere-libtorrent
2
2
Fork 1
Code Issues Pull Requests Releases Wiki Activity

merged changes from RC_1_0

This commit is contained in:
Arvid Norberg 2014-09-01 07:16:31 +00:00
parent ddb133e055
commit b1208c9f63
20 changed files with 494 additions and 1982 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
Makefile.am
View File

@ -21,10 +21,12 @@ DOCS_IMAGES = \
docs/our_delay_base.png \
docs/our_delay_base_thumb.png \
docs/read_disk_buffers.png \
docs/read_disk_buffers.diagram \
docs/session_stats_peers.png \
docs/storage.png \
docs/todo.html \
docs/write_disk_buffers.png \
docs/write_disk_buffers.diagram \
docs/ziptorrent_thumb.gif \
docs/disk_access_elevator.png \
docs/disk_access_no_elevator.png\
@ -32,8 +34,13 @@ DOCS_IMAGES = \
docs/ip_id_v6.png \
docs/hash_distribution.png \
docs/complete_bit_prefixes.png \
docs/troubleshooting.dot \
docs/troubleshooting.png \
docs/troubleshooting_thumb.png \
docs/hacking.diagram \
docs/hacking.png \
docs/utp_stack.diagram \
docs/utp_stack.png \
docs/style.css \
docs/base.css \
docs/rst.css \
@ -66,6 +73,24 @@ DOCS_PAGES = \
docs/troubleshooting.html \
docs/udp_tracker_protocol.html \
docs/utp.html \
docs/building.rst \
docs/client_test.rst \
docs/contributing.rst \
docs/dht_extensions.rst \
docs/dht_rss.rst \
docs/dht_sec.rst \
docs/dht_store.rst \
docs/examples.rst \
docs/extension_protocol.rst \
docs/features.rst \
docs/index.rst \
docs/manual-ref.rst \
docs/projects.rst \
docs/python_binding.rst \
docs/tuning.rst \
docs/troubleshooting.rst \
docs/udp_tracker_protocol.rst \
docs/utp.rst \
docs/reference-Alerts.html \
docs/reference-Bencoding.html \
docs/reference-Core.html \

4
docs/features.html
View File

@ -276,6 +276,7 @@ to provide additional bandwidth to an entire feed.</p>
</div>
<div class="section" id="merkle-hash-tree-torrents">
<h2>merkle hash tree torrents</h2>
<img alt="merkle_tree.png" class="align-right" src="merkle_tree.png" />
<p>Merkle hash tree torrents is an extension that lets a torrent file only contain the
root hash of the hash tree forming the piece hashes. The main benefit of this feature
is that regardless of how many pieces there is in a torrent, the .torrent file will
@ -299,13 +300,12 @@ piece size. With merkle torrents, the piece size can be the minimum block size (
which lets peers verify every block of data received from peers, immediately. This
gives a minimum turnaround time and completely removes the problem of identifying malicious
peers.</p>
<img alt="merkle_tree.png" src="merkle_tree.png" />
<p>The root hash is built by hashing all the piece hashes pair-wise, until they all collapse
down to the root.</p>
<img alt="storage.png" class="align-right" src="storage.png" />
</div>
<div class="section" id="customizable-file-storage">
<h2>customizable file storage</h2>
<img alt="storage.png" class="align-right" src="storage.png" />
<p>libtorrent's storage implementation is customizable. That means a special purpose bittorrent
client can replace the default way to store files on disk.</p>
<p>When implementing a bittorrent cache, it doesn't matter how the data is stored on disk, as

11
docs/features.rst
View File

@ -277,6 +277,9 @@ to provide additional bandwidth to an entire feed.
merkle hash tree torrents
-------------------------
.. image:: merkle_tree.png
:align: right
Merkle hash tree torrents is an extension that lets a torrent file only contain the
root hash of the hash tree forming the piece hashes. The main benefit of this feature
is that regardless of how many pieces there is in a torrent, the .torrent file will
@ -305,17 +308,15 @@ which lets peers verify every block of data received from peers, immediately. Th
gives a minimum turnaround time and completely removes the problem of identifying malicious
peers.
.. image:: merkle_tree.png
The root hash is built by hashing all the piece hashes pair-wise, until they all collapse
down to the root.
.. image:: storage.png
:align: right
customizable file storage
-------------------------
.. image:: storage.png
:align: right
libtorrent's storage implementation is customizable. That means a special purpose bittorrent
client can replace the default way to store files on disk.

4
docs/gen_todo.py
View File

@ -45,6 +45,10 @@ for f in files:
items[-1]['priority'] = 0
if line[0] in '0123456789':
items[-1]['priority'] = int(line[0])
if int(line[0]) > 5:
print 'priority too high: ' + line
sys.exit(1)
line = line[1:].strip()
items[-1]['todo'] = line
prio = items[-1]['priority']

26
docs/hacking.diagram Normal file
View File

@ -0,0 +1,26 @@
+--------------+ pimpl +--------------+
| cGRE session +----------->| session_impl |
+--------------+ +------+-----+-+
m_torrents[] | |
+---------------------+ | |
| cGRE torrent_handle +-------+ | |
+---------------------+ weak | | |
| | | m_connections[]
| | +---+-------+
| | | |
m_picker v v | v peers we are connected to
+--------------+ +--------++ +-----------------+
| piece_picker |<---+-+ torrent ++ +--+ peer_connection ++
+--------------+ | ++--------+| | ++----------------+|
m_torrent_file | +---------+ | +-----------------+
+-------------------+ | |
| cGRE torrent_info |<---+ | m_socket
+-------------------+ | | +--------------------------+
| +------+->| socket_type (variant) |
+--------+ m_policy | | | (TCP/uTP/SSL/socks5/...) |
| policy |<---------+ | +--------------------------+
+------+-+ v
list of all | m_peers[] +--------------+
peers we +-------------->| policy::peer ++ contains contact information
know of ++-------------+| for peers we're not necessarily
+--------------+ connected to

37
docs/hacking.rst
View File

@ -62,43 +62,8 @@ structure
This is the high level structure of libtorrent. Bold types are part of the public
interface:
.. parsed-literal::
+=========+ pimpl +-------------------+
| **session** | ---------> | aux::session_impl |
+=========+ +-------------------+
m_torrents[] | |
+================+ | |
| **torrent_handle** | ------+ | |
+================+ | | |
| | | m_connections[]
| | |
| | +---------------------+
m_picker v v |
+--------------+ +---------+---------+-- . . |
| piece_picker | <--+-| torrent | torrent | to |
+--------------+ | +---------+---------+-- . . |
m_torrent_file | | m_connections[] |
+==============+ | | |
| **torrent_info** | <--+ v v
+==============+ | +-----------------+-----------------+-- . .
m_policy | | peer_connection | peer_connection | pe
+--------+ | +-----------------+-----------------+-- . .
| policy | <--------+ | | m_socket
+--------+ | |
| m_peers[] | v
| | +-----------------------+
| | | socket_type (variant) |
v | +-----------------------+
+--------------+ |
| policy::peer | |
+--------------+ |
| policy::peer | |
+--------------+ m_peer_info|
| policy::peer | <----------+
+--------------+
. .
+ - - - - - - -+
.. image:: hacking.png
session_impl
------------

16
docs/makefile
View File

@ -42,7 +42,13 @@ TARGETS = index \
streaming \
$(REFERENCE_TARGETS)
FIGURES = read_disk_buffers write_disk_buffers troubleshooting
FIGURES = \
read_disk_buffers \
write_disk_buffers \
troubleshooting \
hacking \
utp_stack \
storage
html: $(TARGETS:=.html) $(FIGURES:=.png) todo.html
@ -65,10 +71,6 @@ troubleshooting_thumb.png: troubleshooting.png
convert troubleshooting.png -resize 800x800 troubleshooting_thumb.png
cp $@ $(WEB_PATH)/$@
troubleshooting.png: troubleshooting.dot
dot troubleshooting.dot -Tpng >troubleshooting.png
cp $@ $(WEB_PATH)/$@
todo.html:gen_todo.py ../src/*.cpp ../include/libtorrent/*.hpp
python gen_todo.py
cp $@ $(WEB_PATH)/$@
@ -94,6 +96,10 @@ $(REFERENCE_TARGETS:=.rst):gen_reference_doc.py ../include/libtorrent/*.hpp ../i
dot -Teps $? >$@
cp $@ $(WEB_PATH)/$@
%.png:%.diagram
java -jar /opt/local/share/java/ditaa0_9.jar -E $? $@
cp $@ $(WEB_PATH)/$@
clean:
rm -f $(TARGETS:=.html) $(TARGETS:=.pdf) settings.rst todo.html reference*.html reference*.rst

16
docs/read_disk_buffers.diagram Normal file
View File

@ -0,0 +1,16 @@
copy into peer's encrypt in place
+----------------+ send buffer +-------------+ (no copy) +-------------+
| receive buffer +----------------->| send buffer +--=--------------->| encrypted |
| | | | | send buffer |
+----------------+ +-------------+ +------+------+
^ |
| read() from file write() to socket |
| (copy) (copy) |
---=----|---------------------------------=---------------------------------|--=----
| kernel space |
| v
+-------+-----------+ +---------------+
| kernel page cache | | socket kernel |
| | | buffer |
+-------------------+ +---------------+

20
docs/read_disk_buffers.dot
View File

@ -1,20 +0,0 @@
digraph uploading {
node [shape=box];
subgraph user_space {
rank=same;
"disk cache" -> "send buffer" [label="copy into peer's send buffer (copy)"]
"send buffer" -> "encrypted send buffer" [label="encrypt in-place (no copy)" style=dashed];
}
subgraph kernel {
rank=same;
"kernel page cache";
"socket kernel buffer"
}
"encrypted send buffer" -> "socket kernel buffer" [label="write() to socket (copy)"];
"kernel page cache" -> "disk cache" [label="read() from file (copy)"]
}

BIN
docs/read_disk_buffers.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 16 KiB

31
docs/storage.diagram Normal file
View File

@ -0,0 +1,31 @@
+--------------------------+
| disk_io_thread |
| (manages piece cache) |
+--------------------------+
^
|
v
+--------------------------+
| piece_manager |
| (maps pieces to slots) |
+--------------------------+
^
|
v customization point
/--------------------------\ +------------------+
| storage |<---->| file_pool |
| cGRE | | open file cache |
| (maps slots to file and | +------------------+
| offset. reads and writes |
| to disk) | +------------------+
| |<-----+ file_storage |
\--------------------------/ | standard slot to |
^ | file mapping |
| +------------------+
v
+--------------------------+
| file |
| (file class reads and |
| writes files) |
+--------------------------+

1726
docs/storage.graffle
View File

File diff suppressed because it is too large Load Diff

BIN
docs/storage.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 30 KiB

After

Width:  |  Height:  |  Size: 21 KiB

3
docs/style.css
View File

@ -39,6 +39,9 @@ li { margin-left: 2.8em; font-size: 92%; }
p, ul, ol, img {margin-bottom: 1em;}
.align-right {
float: right;
}
.document {
margin-left: 10px;

509
docs/todo.html
View File

File diff suppressed because it is too large Load Diff

9
docs/utp_stack.diagram Normal file
View File

@ -0,0 +1,9 @@
+-----------------------+
| BitTorrent protocol |
+-----------------------+
| SSL |
+-----------+-----------+
| TCP | uTP |
| +-----------+
| | UDP |
+-----------+-----------+

15
docs/write_disk_buffers.diagram Normal file
View File

@ -0,0 +1,15 @@
decrypt in place move buffer ref.
+----------------+ (no copy) +------------+ (no copy) +------------+
| receive buffer +--=-------------->| plain text +--=--------------->| disk cache |
+----------------+ | buffer | +------+-----+
^ +------------+ |
| read() on socket write() to file |
| (copy) (copy) |
---=----|---------------------------------=---------------------------------|--=----
| kernel space |
| v
+-------+--------+ +-------------------+
| socket kernel | | kernel page cache |
| buffer | | |
+----------------+ +-------------------+

20
docs/write_disk_buffers.dot
View File

@ -1,20 +0,0 @@
digraph downloading {
label=""
node [shape=box];
subgraph user_space {
rank=same;
"receive buffer" -> "plain text buffer" [label="decrypt in-place (no copy)" style=dashed];
"plain text buffer" -> "disk cache" [label="move buffer reference (no copy)" style=dashed]
}
subgraph kernel {
rank=same;
"socket kernel buffer";
"kernel page cache"
}
"socket kernel buffer" -> "receive buffer" [label="read() on socket (copy)"];
"disk cache" -> "kernel page cache" [label="write() to file (copy)"]
}

BIN
docs/write_disk_buffers.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 15 KiB

4
include/libtorrent/torrent.hpp
View File

@ -1568,7 +1568,7 @@ namespace libtorrent
// specified in session_time
boost::uint16_t m_last_download;
// TODO: 8 bits free here
// TODO: There are 8 bits free here
// this is a second count-down to when we should tick the
// storage for this torrent. Ticking the storage is used
@ -1584,7 +1584,7 @@ namespace libtorrent
// specified in session_time
boost::uint16_t m_last_upload;
// TODO: 8 bits here
// TODO: There are 8 bits here
// if this is true, libtorrent may pause and resume
// this torrent depending on queuing rules. Torrents