merged changes from RC_1_0

Makefile.am

@@ -21,10 +21,12 @@ DOCS_IMAGES = \
   docs/our_delay_base.png         \
   docs/our_delay_base_thumb.png   \
   docs/read_disk_buffers.png      \
+  docs/read_disk_buffers.diagram  \
   docs/session_stats_peers.png    \
   docs/storage.png                \
   docs/todo.html                  \
   docs/write_disk_buffers.png     \
+  docs/write_disk_buffers.diagram \
   docs/ziptorrent_thumb.gif       \
   docs/disk_access_elevator.png   \
   docs/disk_access_no_elevator.png\
@@ -32,8 +34,13 @@ DOCS_IMAGES = \
   docs/ip_id_v6.png               \
   docs/hash_distribution.png      \
   docs/complete_bit_prefixes.png  \
+  docs/troubleshooting.dot        \
   docs/troubleshooting.png        \
   docs/troubleshooting_thumb.png  \
+  docs/hacking.diagram            \
+  docs/hacking.png                \
+  docs/utp_stack.diagram          \
+  docs/utp_stack.png              \
   docs/style.css                  \
   docs/base.css                   \
   docs/rst.css                    \
@@ -66,6 +73,24 @@ DOCS_PAGES = \
   docs/troubleshooting.html       \
   docs/udp_tracker_protocol.html  \
   docs/utp.html                   \
+  docs/building.rst               \
+  docs/client_test.rst            \
+  docs/contributing.rst           \
+  docs/dht_extensions.rst         \
+  docs/dht_rss.rst                \
+  docs/dht_sec.rst                \
+  docs/dht_store.rst              \
+  docs/examples.rst               \
+  docs/extension_protocol.rst     \
+  docs/features.rst               \
+  docs/index.rst                  \
+  docs/manual-ref.rst             \
+  docs/projects.rst               \
+  docs/python_binding.rst         \
+  docs/tuning.rst                 \
+  docs/troubleshooting.rst        \
+  docs/udp_tracker_protocol.rst   \
+  docs/utp.rst                    \
   docs/reference-Alerts.html      \
   docs/reference-Bencoding.html   \
   docs/reference-Core.html        \

docs/features.html

@@ -276,6 +276,7 @@ to provide additional bandwidth to an entire feed.</p>
 </div>
 <div class="section" id="merkle-hash-tree-torrents">
 <h2>merkle hash tree torrents</h2>
+<img alt="merkle_tree.png" class="align-right" src="merkle_tree.png" />
 <p>Merkle hash tree torrents is an extension that lets a torrent file only contain the
 root hash of the hash tree forming the piece hashes. The main benefit of this feature
 is that regardless of how many pieces there is in a torrent, the .torrent file will
@@ -299,13 +300,12 @@ piece size. With merkle torrents, the piece size can be the minimum block size (
 which lets peers verify every block of data received from peers, immediately. This
 gives a minimum turnaround time and completely removes the problem of identifying malicious
 peers.</p>
-<img alt="merkle_tree.png" src="merkle_tree.png" />
 <p>The root hash is built by hashing all the piece hashes pair-wise, until they all collapse
 down to the root.</p>
-<img alt="storage.png" class="align-right" src="storage.png" />
 </div>
 <div class="section" id="customizable-file-storage">
 <h2>customizable file storage</h2>
+<img alt="storage.png" class="align-right" src="storage.png" />
 <p>libtorrent's storage implementation is customizable. That means a special purpose bittorrent
 client can replace the default way to store files on disk.</p>
 <p>When implementing a bittorrent cache, it doesn't matter how the data is stored on disk, as

docs/features.rst

@@ -277,6 +277,9 @@ to provide additional bandwidth to an entire feed.
 merkle hash tree torrents
 -------------------------
 
+.. image:: merkle_tree.png
+	:align: right
+
 Merkle hash tree torrents is an extension that lets a torrent file only contain the
 root hash of the hash tree forming the piece hashes. The main benefit of this feature
 is that regardless of how many pieces there is in a torrent, the .torrent file will
@@ -305,17 +308,15 @@ which lets peers verify every block of data received from peers, immediately. Th
 gives a minimum turnaround time and completely removes the problem of identifying malicious
 peers.
 
-.. image:: merkle_tree.png
-
 The root hash is built by hashing all the piece hashes pair-wise, until they all collapse
 down to the root.
 
-.. image:: storage.png
-	:align: right
-
 customizable file storage
 -------------------------
 
+.. image:: storage.png
+	:align: right
+
 libtorrent's storage implementation is customizable. That means a special purpose bittorrent
 client can replace the default way to store files on disk.
 

docs/gen_todo.py

@@ -45,6 +45,10 @@ for f in files:
 			items[-1]['priority'] = 0
 			if line[0] in '0123456789':
 				items[-1]['priority'] = int(line[0])
+				if int(line[0]) > 5:
+					print 'priority too high: ' + line
+					sys.exit(1)
+
 				line = line[1:].strip()
 			items[-1]['todo'] = line
 			prio = items[-1]['priority']

docs/hacking.diagram

@@ -0,0 +1,26 @@
++--------------+  pimpl     +--------------+
+| cGRE session +----------->| session_impl |
++--------------+            +------+-----+-+
+                      m_torrents[] |     |
++---------------------+            |     |
+| cGRE torrent_handle +-------+    |     |
++---------------------+ weak  |    |     |
+                              |    |     | m_connections[]
+                              |    | +---+-------+
+                              |    | |           |
+              m_picker        v    v |           v peers we are connected to
+      +--------------+      +--------++        +-----------------+             
+      | piece_picker |<---+-+ torrent ++    +--+ peer_connection ++            
+      +--------------+    | ++--------+|    |  ++----------------+|            
+           m_torrent_file |  +---------+    |   +-----------------+            
+ +-------------------+    |                 | 
+ | cGRE torrent_info |<---+                 | m_socket                     
+ +-------------------+    |                 |  +--------------------------+
+                          |          +------+->| socket_type (variant)    |
+      +--------+ m_policy |          |         | (TCP/uTP/SSL/socks5/...) |
+      | policy |<---------+          |         +--------------------------+
+      +------+-+                     v      
+ list of all |     m_peers[] +--------------+        
+    peers we +-------------->| policy::peer ++ contains contact information
+     know of                 ++-------------+| for peers we're not necessarily
+                              +--------------+ connected to

docs/hacking.rst

@@ -62,43 +62,8 @@ structure
 This is the high level structure of libtorrent. Bold types are part of the public
 interface:
 
-.. parsed-literal::
 
-	+=========+  pimpl     +-------------------+
+.. image:: hacking.png
-	| **session** | ---------> | aux::session_impl |
-	+=========+            +-------------------+
-	                m_torrents[]  |  |
-	+================+            |  |
-	| **torrent_handle** | ------+    |  |
-	+================+       |    |  |
-	                         |    |  | m_connections[]
-	                         |    |  |
-	                         |    |  +---------------------+
-	         m_picker        v    v                        |
-	 +--------------+      +---------+---------+-- . .     |
-	 | piece_picker | <--+-| torrent | torrent | to        |
-	 +--------------+    | +---------+---------+-- . .     |
-	      m_torrent_file |      | m_connections[]          |
-	 +==============+    |      |                          |
-	 | **torrent_info** | <--+      v                          v
-	 +==============+    |     +-----------------+-----------------+-- . .
-	            m_policy |     | peer_connection | peer_connection | pe
-	 +--------+          |     +-----------------+-----------------+-- . .
-	 | policy | <--------+      |             | m_socket
-	 +--------+                 |             |
-	   | m_peers[]              |             v
-	   |                        |            +-----------------------+
-	   |                        |            | socket_type (variant) |
-	   v                        |            +-----------------------+
-	+--------------+            |
-	| policy::peer |            |
-	+--------------+            |
-	| policy::peer |            |
-	+--------------+ m_peer_info|
-	| policy::peer | <----------+
-	+--------------+
-	.              .
-	+ - - - - - - -+
 
 session_impl
 ------------

docs/makefile

@@ -42,7 +42,13 @@ TARGETS = index \
 	streaming \
 	$(REFERENCE_TARGETS)
 
-FIGURES = read_disk_buffers write_disk_buffers troubleshooting
+FIGURES = \
+	read_disk_buffers \
+	write_disk_buffers \
+	troubleshooting \
+	hacking \
+	utp_stack \
+	storage
 
 html: $(TARGETS:=.html) $(FIGURES:=.png) todo.html 
 
@@ -65,10 +71,6 @@ troubleshooting_thumb.png: troubleshooting.png
 	convert troubleshooting.png -resize 800x800 troubleshooting_thumb.png
 	cp $@ $(WEB_PATH)/$@
 
-troubleshooting.png: troubleshooting.dot
-	dot troubleshooting.dot -Tpng >troubleshooting.png
-	cp $@ $(WEB_PATH)/$@
-
 todo.html:gen_todo.py ../src/*.cpp ../include/libtorrent/*.hpp
 	python gen_todo.py
 	cp $@ $(WEB_PATH)/$@
@@ -94,6 +96,10 @@ $(REFERENCE_TARGETS:=.rst):gen_reference_doc.py ../include/libtorrent/*.hpp ../i
 	dot -Teps $? >$@
 	cp $@ $(WEB_PATH)/$@
 
+%.png:%.diagram
+	java -jar /opt/local/share/java/ditaa0_9.jar -E $? $@
+	cp $@ $(WEB_PATH)/$@
+
 clean:
 	rm -f $(TARGETS:=.html) $(TARGETS:=.pdf) settings.rst todo.html reference*.html reference*.rst
 

docs/read_disk_buffers.diagram

@@ -0,0 +1,16 @@
+
+                   copy into peer's                encrypt in place
++----------------+ send buffer      +-------------+ (no copy)         +-------------+
+| receive buffer +----------------->| send buffer +--=--------------->| encrypted   |
+|                |                  |             |                   | send buffer |
++----------------+                  +-------------+                   +------+------+
+        ^                                                                   |
+        | read() from file                                write() to socket | 
+        | (copy)                                          (copy)            | 
+---=----|---------------------------------=---------------------------------|--=----
+        |                        kernel space                               |
+        |                                                                   v
++-------+-----------+                                               +---------------+
+| kernel page cache |                                               | socket kernel |
+|                   |                                               | buffer        |
++-------------------+                                               +---------------+

docs/read_disk_buffers.dot

@@ -1,20 +0,0 @@
-digraph uploading {
-
-	node [shape=box];
-
-	subgraph user_space {
-		rank=same;
-		"disk cache" -> "send buffer" [label="copy into peer's send buffer (copy)"]
-		"send buffer" -> "encrypted send buffer" [label="encrypt in-place (no copy)" style=dashed];
-	}
-
-	subgraph kernel {
-		rank=same;
-		"kernel page cache";
-		"socket kernel buffer"
-	}
-
-	"encrypted send buffer" -> "socket kernel buffer" [label="write() to socket (copy)"];
-	"kernel page cache" -> "disk cache" [label="read() from file (copy)"]
-}
-

docs/storage.diagram

@@ -0,0 +1,31 @@
++--------------------------+
+|      disk_io_thread      |
+|   (manages piece cache)  |
++--------------------------+
+              ^
+              |
+              v
++--------------------------+
+|      piece_manager       |
+|  (maps pieces to slots)  |
++--------------------------+
+              ^
+              |
+              v customization point
+/--------------------------\      +------------------+
+|          storage         |<---->| file_pool        |
+| cGRE                     |      | open file cache  |
+| (maps slots to file and  |      +------------------+
+| offset. reads and writes |
+| to disk)                 |      +------------------+
+|                          |<-----+ file_storage     |
+\--------------------------/      | standard slot to |
+              ^                   | file mapping     |
+              |                   +------------------+
+              v
++--------------------------+
+|           file           |
+| (file class reads and    |
+| writes files)            |
++--------------------------+
+

docs/style.css

@@ -39,6 +39,9 @@ li { margin-left: 2.8em; font-size: 92%; }
 
 p, ul, ol, img {margin-bottom: 1em;}
 
+.align-right {
+	float: right;
+}
 
 .document {
     margin-left: 10px;

docs/utp_stack.diagram

@@ -0,0 +1,9 @@
++-----------------------+
+| BitTorrent protocol   |
++-----------------------+
+|          SSL          |
++-----------+-----------+
+| TCP       | uTP       |
+|           +-----------+
+|           | UDP       |
++-----------+-----------+

docs/write_disk_buffers.diagram

@@ -0,0 +1,15 @@
+
+                   decrypt in place                move buffer ref.
++----------------+ (no copy)        +------------+ (no copy)         +------------+
+| receive buffer +--=-------------->| plain text +--=--------------->| disk cache |
++----------------+                  | buffer     |                   +------+-----+
+        ^                           +------------+                          |
+        | read() on socket                                 write() to file  | 
+        | (copy)                                           (copy)           | 
+---=----|---------------------------------=---------------------------------|--=----
+        |                        kernel space                               |
+        |                                                                   v
++-------+--------+                                               +-------------------+
+| socket kernel  |                                               | kernel page cache |
+| buffer         |                                               |                   |
++----------------+                                               +-------------------+

docs/write_disk_buffers.dot

@@ -1,20 +0,0 @@
-digraph downloading {
-	label=""
-	node [shape=box];
-
-	subgraph user_space {
-		rank=same;
-		"receive buffer" -> "plain text buffer" [label="decrypt in-place (no copy)" style=dashed];
-		"plain text buffer" -> "disk cache" [label="move buffer reference (no copy)" style=dashed]
-	}
-
-	subgraph kernel {
-		rank=same;
-		"socket kernel buffer";
-		"kernel page cache"
-	}
-
-	"socket kernel buffer" -> "receive buffer" [label="read() on socket (copy)"];
-	"disk cache" -> "kernel page cache" [label="write() to file (copy)"]
-}
-

include/libtorrent/torrent.hpp

@@ -1568,7 +1568,7 @@ namespace libtorrent
 		// specified in session_time
 		boost::uint16_t m_last_download;
 
-		// TODO: 8 bits free here
+		// TODO: There are 8 bits free here
 
 		// this is a second count-down to when we should tick the
 		// storage for this torrent. Ticking the storage is used
@@ -1584,7 +1584,7 @@ namespace libtorrent
 		// specified in session_time
 		boost::uint16_t m_last_upload;
 
-		// TODO: 8 bits here
+		// TODO: There are 8 bits here
 
 		// if this is true, libtorrent may pause and resume
 		// this torrent depending on queuing rules. Torrents