New features that need testing are streaming (set_piece_deadline()), the different choking algorithms (especially the new BitTyrant choker), the disk cache options (such as explicit_cache).
diff --git a/docs/contributing.rst b/docs/contributing.rst index 0634c0d82..cb93797c2 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -21,13 +21,13 @@ enumerated on this page, please contact arvid@libtorrent.org or the `mailing lis 1. Testing This is not just limited to finding bugs and ways to reproduce crashes, but also sub-optimal behavior is certain scenarios and finding ways to reproduce those. Please - report any issue to the bug tracker at `google code`_. + report any issue to the bug tracker at `github`_. New features that need testing are streaming (``set_piece_deadline()``), the different choking algorithms (especially the new BitTyrant choker), the disk cache options (such as ``explicit_cache``). -.. _`google code`: http://code.google.com/p/libtorrent/issues/entry +.. _`github`: https://github.com/arvidn/libtorrent/issues 2. Documentation Finding typos or outdated sections in the documentation. Contributing documentation diff --git a/docs/disk_cache.diagram b/docs/disk_cache.diagram new file mode 100644 index 000000000..da666fbee --- /dev/null +++ b/docs/disk_cache.diagram @@ -0,0 +1,12 @@ + <---- "recently" "frequently" ---> + "used" "used" + + L1 ghost L1 L2 L2 ghost + AAAAAAAAFFFFFFFF FFFFFFFFAAAAAAAA + + cache size + <---------------> + + 2x cache size + <-------------------------------> + diff --git a/docs/hacking.diagram b/docs/hacking.diagram index db00a06d6..02d64968d 100644 --- a/docs/hacking.diagram +++ b/docs/hacking.diagram @@ -1,26 +1,30 @@ -+--------------+ pimpl +--------------+ -| cGRE session +----------->| session_impl | -+--------------+ +------+-----+-+ - m_torrents[] | | -+---------------------+ | | -| cGRE torrent_handle +-------+ | | -+---------------------+ weak | | | - | | | m_connections[] - | | +---+-------+ - | | | | - m_picker v v | v peers we are connected to - +--------------+ +--------++ +-----------------+ - | piece_picker |<---+-+ torrent ++ +--+ peer_connection ++ - +--------------+ | ++--------+| | ++----------------+| - m_torrent_file | +---------+ | +-----------------+ - +-------------------+ | | - | cGRE torrent_info |<---+ | m_socket - +-------------------+ | | +--------------------------+ - m_peer_list | +------+->| socket_type (variant) | - +-----------+ | | | (TCP/uTP/SSL/socks5/...) | - | peer_list |<---------+ | +--------------------------+ - +---------+-+ v - list of all | m_peers[] +--------------+ - peers we +-------------->| torrent_peer ++ contains contact information - know of ++-------------+| for peers we're not necessarily - +--------------+ connected to ++--------------+ "pimpl" +----------------+ +| "session" +--------------------------->| "session_impl" | ++--------------+ +------+-----+---+ + "m_torrents[]" | | ++---------------------+ | | +| "torrent_handle" +--------+ | | ++---------------------+ "weak" | +--------------+ | + | | | "m_connections[]" + | | +-------------+----+ + | | | | + "m_picker" v v | v "peers we are connected to" + +----------------+ +----------++ +-------------------+ + | "piece_picker" |<---+-+ "torrent" ++ +--+ "peer_connection" ++ + +----------------+ | ++----------+| | ++------------------+| + "m_torrent_file" | +-----------+ | +-------------------+ + +-------------------+ | | + | "torrent_info" |<---+ | "m_socket" + +-------------------+ | | +----------------------------+ + | +->| "socket_type (variant)" | + "m_peer_list" v | | "(TCP/uTP/SSL/socks5/...)" | + +--------------+ | +----------------------------+ + | "peer_list" | | + +------------+-+ | "m_peer_info" + "list of all" | "m_peers[]" | "contains contact information" + "peers we" | | "for peers we're not necessarily" + "know of" | v "connected to" + | +----------------+ + +---->| "torrent_peer" ++ + ++---------------+| + +----------------+ diff --git a/docs/hacking.rst b/docs/hacking.rst index bdb626c4e..7a1de5d93 100644 --- a/docs/hacking.rst +++ b/docs/hacking.rst @@ -136,17 +136,9 @@ The disk cache implements *ARC*, Adaptive Replacement Cache. This consists of a 5. volatile read blocks 6. write cache (blocks waiting to be flushed to disk) -.. parsed-literal:: - - <--- recently used frequently used ---> - +--------------+--------------+ +--------------+--------------+ - | L1 **ghost** | L1 | | L2 | L2 **ghost** | - +--------------+--------------+ +--------------+--------------+ - - <---------- cache_size ----------> - - <---------------------- 2 x cache_size ------------------------> +.. image:: disk_cache.png + These LRUs are stored in ``block_cache`` in an array ``m_lru``. The cache algorithm works like this:: diff --git a/docs/makefile b/docs/makefile index 9a02fd262..ff5b5bdc8 100644 --- a/docs/makefile +++ b/docs/makefile @@ -49,12 +49,13 @@ TARGETS = index \ FIGURES = \ read_disk_buffers \ write_disk_buffers \ - troubleshooting \ hacking \ utp_stack \ - storage + storage \ + disk_cache \ + troubleshooting -html: $(TARGETS:=.html) $(FIGURES:=.png) todo.html +html: $(TARGETS:=.html) $(FIGURES:=.png) todo.html pdf: $(TARGETS:=.pdf) $(FIGURES:=.eps) @@ -71,11 +72,11 @@ stats_counters.rst: ../src/session_stats.cpp ../include/libtorrent/performance_c manual.rst: stats_counters.rst touch manual.rst -troubleshooting_thumb.png: troubleshooting.png - convert troubleshooting.png -resize 800x800 troubleshooting_thumb.png -ifneq ($(STAGE),) - cp $@ $(WEB_PATH)/$@ -endif +#troubleshooting_thumb.png: troubleshooting.png +# convert troubleshooting.png -resize 800x800 troubleshooting_thumb.png +#ifneq ($(STAGE),) +# cp $@ $(WEB_PATH)/$@ +#endif todo.html:gen_todo.py ../src/*.cpp ../include/libtorrent/*.hpp python gen_todo.py @@ -111,11 +112,11 @@ ifneq ($(STAGE),) endif %.png:%.diagram - ditaa -E $? $@ + aafigure -o $@ $? ifneq ($(STAGE),) cp $@ $(WEB_PATH)/$@ endif clean: - rm -f $(TARGETS:=.html) $(TARGETS:=.pdf) settings.rst todo.html reference*.html reference*.rst + rm -f $(TARGETS:=.html) $(TARGETS:=.pdf) $(FIGURES:=.png) $(FIGURES:=.eps) settings.rst todo.html reference*.html reference*.rst diff --git a/docs/read_disk_buffers.diagram b/docs/read_disk_buffers.diagram index 6ffa96479..1d04a25dd 100644 --- a/docs/read_disk_buffers.diagram +++ b/docs/read_disk_buffers.diagram @@ -1,16 +1,16 @@ - copy into peer's encrypt in place -+----------------+ send buffer +-------------+ (no copy) +-------------+ -| receive buffer +----------------->| send buffer +--=--------------->| encrypted | -| | | | | send buffer | -+----------------+ +-------------+ +------+------+ + "copy into peer's" "encrypt in place" ++------------------+ "send buffer" +---------------+ "(no copy)" +---------------+ +| "receive buffer" +----------------->| "send buffer" +------------------>| "encrypted" | +| | | | | "send buffer" | ++------------------+ +---------------+ +---------------+ ^ | - | read() from file write() to socket | - | (copy) (copy) | ----=----|---------------------------------=---------------------------------|--=---- - | kernel space | + | "read() from file" "write() to socket" | + | "(copy)" "user space" "(copy)" | +- - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - + | "kernel space" | | v -+-------+-----------+ +---------------+ -| kernel page cache | | socket kernel | -| | | buffer | -+-------------------+ +---------------+ ++-------+-------------+ +-----------------+ +| "kernel page cache" | | "socket kernel" | +| | | "buffer" | ++---------------------+ +-----------------+ diff --git a/docs/storage.diagram b/docs/storage.diagram index 8f0618d7d..895607642 100644 --- a/docs/storage.diagram +++ b/docs/storage.diagram @@ -1,31 +1,31 @@ +--------------------------+ -| disk_io_thread | -| (manages piece cache) | +| "disk_io_thread" | +| "(manages piece cache)" | +--------------------------+ ^ | v +--------------------------+ -| piece_manager | -| (maps pieces to slots) | +| "piece_manager" | +| "(maps pieces to slots)" | +--------------------------+ ^ | - v customization point -/--------------------------\ +------------------+ -| storage |<---->| file_pool | -| cGRE | | open file cache | -| (maps slots to file and | +------------------+ -| offset. reads and writes | -| to disk) | +------------------+ -| |<-----+ file_storage | -\--------------------------/ | standard slot to | - ^ | file mapping | - | +------------------+ + v "customization point" +/----------------------------\ +-------------------+ +| "storage" |<---->| "file_pool" | +| | | "open file cache" | +| "(maps slots to file and" | +-------------------+ +| "offset. reads and writes" | +| "to disk)" | +--------------------+ +| |<-----+ "file_storage" | +\----------------------------/ | "standard slot to" | + ^ | "file mapping" | + | +--------------------+ v +--------------------------+ -| file | -| (file class reads and | -| writes files) | +| "file" | +| "(file class reads and" | +| "writes files)" | +--------------------------+ diff --git a/docs/todo.html b/docs/todo.html index 2dc6a39f8..d336fcaa2 100644 --- a/docs/todo.html +++ b/docs/todo.html @@ -74,7 +74,7 @@ does not support hard links.relevance 3 ../src/session_impl.cpp:2028 port map SSL udp socket here relevance 3 ../src/session_impl.cpp:2030 port map SSL udp socket here relevance 2 ../src/session_impl.cpp:2012 use bind_to_device in udp_socket relevance 2 ../src/session_impl.cpp:2014 use bind_to_device in udp_socket relevance 2 ../src/session_impl.cpp:2034 use bind_to_device in udp_socket relevance 2 ../src/session_impl.cpp:3513 make a list for torrents that want to be announced on the DHT so we don't have to loop over all torrents, just to find the ones that want to announce relevance 2 ../src/storage.cpp:941 is this risky? The upper layer will assume we have the whole file. Perhaps we should verify that at least the size of the file is correct relevance 2 ../src/storage.cpp:948 is this risky? The upper layer will assume we have the whole file. Perhaps we should verify that at least the size of the file is correct relevance 2 ../src/torrent.cpp:4885 abort lookups this torrent has made via the session host resolver interface relevance 2 ../src/torrent.cpp:4890 abort lookups this torrent has made via the session host resolver interface relevance 2 ../src/torrent.cpp:5036 the tracker login feature should probably be deprecated relevance 2 ../src/torrent.cpp:8023 if peer is a really good peer, maybe we shouldn't disconnect it perhaps this logic should be disabled if we have too many idle peers (with some definition of idle) relevance 2 ../src/kademlia/dht_storage.cpp:110 make this configurable in dht_settings relevance 2 ../include/libtorrent/alert_types.hpp:1427 should the alert baseclass have this object instead? relevance 2 ../include/libtorrent/alert_types.hpp:1429 should the alert baseclass have this object instead? relevance 2 ../include/libtorrent/enum_net.hpp:143 this could be done more efficiently by just looking up the interface with the given name, maybe even with if_nametoindex() relevance 2 ../include/libtorrent/enum_net.hpp:150 this could be done more efficiently by just looking up the interface with the given name, maybe even with if_nametoindex() relevance 2 ../include/libtorrent/aux_/session_impl.hpp:1146 the throttling of saving resume data could probably be factored out into a separate class relevance 2 ../include/libtorrent/aux_/session_impl.hpp:1147 the throttling of saving resume data could probably be factored out into a separate class relevance 1 ../src/session_impl.cpp:5306 report the proper address of the router as the source IP of this understanding of our external address, instead of the empty address relevance 1 ../src/session_impl.cpp:5308 report the proper address of the router as the source IP of this understanding of our external address, instead of the empty address relevance 1 ../src/session_impl.cpp:6700 we only need to do this if our global IPv4 address has changed since the DHT (currently) only supports IPv4. Since restarting the DHT is kind of expensive, it would be nice to not do it unnecessarily relevance 1 ../src/session_impl.cpp:6702 we only need to do this if our global IPv4 address has changed since the DHT (currently) only supports IPv4. Since restarting the DHT is kind of expensive, it would be nice to not do it unnecessarily relevance 1 ../src/torrent.cpp:7139 save the send_stats state instead of throwing them away it may pose an issue when downgrading though relevance 1 ../src/torrent.cpp:7144 save the send_stats state instead of throwing them away it may pose an issue when downgrading though relevance 1 ../src/torrent.cpp:8362 should disconnect all peers that have the pieces we have not just seeds. It would be pretty expensive to check all pieces for all peers though relevance 1 ../src/torrent.cpp:8367 should disconnect all peers that have the pieces we have not just seeds. It would be pretty expensive to check all pieces for all peers though relevance 0 ../test/test_ssl.cpp:385 test using a signed certificate with the wrong info-hash in DN relevance 0 ../test/test_ssl.cpp:391 test using a signed certificate with the wrong info-hash in DN relevance 0 ../test/test_ssl.cpp:483 also test using a hash that refers to a valid torrent but that differs from the SNI hash relevance 0 ../test/test_ssl.cpp:489 also test using a hash that refers to a valid torrent but that differs from the SNI hash relevance 0 ../src/session_impl.cpp:2750 should this function take a shared_ptr instead? relevance 0 ../src/session_impl.cpp:2752 should this function take a shared_ptr instead? relevance 0 ../src/session_impl.cpp:3099 have a separate list for these connections, instead of having to loop through all of them relevance 0 ../src/session_impl.cpp:3101 have a separate list for these connections, instead of having to loop through all of them relevance 0 ../src/session_impl.cpp:3132 this should apply to all bandwidth channels relevance 0 ../src/session_impl.cpp:3134 this should apply to all bandwidth channels relevance 0 ../src/session_impl.cpp:3861 use a lower limit than m_settings.connections_limit to allocate the to 10% or so of connection slots for incoming connections relevance 0 ../src/session_impl.cpp:3863 use a lower limit than m_settings.connections_limit to allocate the to 10% or so of connection slots for incoming connections relevance 0 ../src/session_impl.cpp:4013 post a message to have this happen immediately instead of waiting for the next tick relevance 0 ../src/session_impl.cpp:4397 it might be a nice feature here to limit the number of torrents to send in a single update. By just posting the first n torrents, they would nicely be round-robined because the torrent lists are always pushed back. Perhaps the status_update_alert could even have a fixed array of n entries rather than a vector, to further improve memory locality. relevance 0 ../src/session_impl.cpp:4399 it might be a nice feature here to limit the number of torrents to send in a single update. By just posting the first n torrents, they would nicely be round-robined because the torrent lists are always pushed back. Perhaps the status_update_alert could even have a fixed array of n entries rather than a vector, to further improve memory locality. relevance 0 ../src/session_impl.cpp:4612 this logic could probably be less spaghetti looking by being moved to a function with early exits relevance 0 ../src/storage.cpp:751 make this more generic to not just work if files have been renamed, but also if they have been merged into a single file for instance maybe use the same format as .torrent files and reuse some code from torrent_info relevance 0 ../src/storage.cpp:758 make this more generic to not just work if files have been renamed, but also if they have been merged into a single file for instance maybe use the same format as .torrent files and reuse some code from torrent_info relevance 0 ../src/storage.cpp:1082 if everything moves OK, except for the partfile we currently won't update the save path, which breaks things. it would probably make more sense to give up on the partfile relevance 0 ../src/storage.cpp:1089 if everything moves OK, except for the partfile we currently won't update the save path, which breaks things. it would probably make more sense to give up on the partfile relevance 0 ../src/torrent.cpp:3723 add one peer per IP the hostname resolves to relevance 0 ../src/torrent.cpp:3728 add one peer per IP the hostname resolves to relevance 0 ../src/torrent.cpp:4668 update suggest_piece? relevance 0 ../src/torrent.cpp:4812 really, we should just keep the picker around in this case to maintain the availability counters relevance 0 ../src/torrent.cpp:6838 make this more generic to not just work if files have been renamed, but also if they have been merged into a single file for instance maybe use the same format as .torrent files and reuse some code from torrent_info The mapped_files needs to be read both in the network thread and in the disk thread, since they both have their own mapped files structures which are kept in sync relevance 0 ../src/torrent.cpp:6843 make this more generic to not just work if files have been renamed, but also if they have been merged into a single file for instance maybe use the same format as .torrent files and reuse some code from torrent_info The mapped_files needs to be read both in the network thread and in the disk thread, since they both have their own mapped files structures which are kept in sync relevance 0 ../src/torrent.cpp:6976 if this is a merkle torrent and we can't restore the tree, we need to wipe all the bits in the have array, but not necessarily we might want to do a full check to see if we have all the pieces. This is low priority since almost no one uses merkle torrents relevance 0 ../src/torrent.cpp:7202 make this more generic to not just work if files have been renamed, but also if they have been merged into a single file for instance. using file_base relevance 0 ../src/torrent.cpp:9445 add a flag to ignore stats, and only care about resume data for content. For unchanged files, don't trigger a load of the metadata just to save an empty resume data file relevance 0 ../src/torrent.cpp:11057 instead of resorting the whole list, insert the peers directly into the right place relevance 0 ../src/torrent.cpp:11062 instead of resorting the whole list, insert the peers directly into the right place relevance 0 ../include/libtorrent/settings_pack.hpp:1094 deprecate this ``max_rejects`` is the number of piece requests we will reject in a row while a peer is choked before the peer is considered abusive and is disconnected. relevance 0 ../include/libtorrent/settings_pack.hpp:1092 deprecate this ``max_rejects`` is the number of piece requests we will reject in a row while a peer is choked before the peer is considered abusive and is disconnected. relevance 0 ../include/libtorrent/aux_/session_impl.hpp:858 should this be renamed m_outgoing_interfaces? relevance 0 ../include/libtorrent/aux_/session_impl.hpp:859 should this be renamed m_outgoing_interfaces? relevance 0 ../include/libtorrent/aux_/session_impl.hpp:909 replace this by a proper asio timer relevance 0 ../include/libtorrent/aux_/session_impl.hpp:910 replace this by a proper asio timer relevance 0 ../include/libtorrent/aux_/session_impl.hpp:914 replace this by a proper asio timer relevance 0 ../include/libtorrent/aux_/session_impl.hpp:915 replace this by a proper asio timer relevance 0 ../include/libtorrent/aux_/session_impl.hpp:921 replace this by a proper asio timer relevance 0 ../include/libtorrent/aux_/session_impl.hpp:922 replace this by a proper asio timer