Mainline DHT extensions
libtorrent implements a few extensions to the Mainline DHT protocol.
+get_peers response
+libtorrent always responds with nodes to a get_peers request. If it has +peers for the specified info-hash, it will return values as well. This is +because just because some peer announced to us, doesn't mean that we are +among the 8 closest nodes of the info hash. libtorrent also keeps traversing +nodes using get_peers until it has found the 8 closest ones, and then announces +to those nodes.
+client identification
In each DHT packet, an extra key is inserted named "v". This is a string diff --git a/docs/examples.html b/docs/examples.html index baf23b1f5..f7bee5413 100644 --- a/docs/examples.html +++ b/docs/examples.html @@ -36,16 +36,16 @@
Table of contents
-
-
- examples
@@ -54,7 +54,7 @@
examples
Except for the example programs in this manual, there's also a bigger example of a (little bit) more complete client, client_test. There are separate -instructions for how to use it here if you'd like to try it. Note that building +instructions for how to use it here if you'd like to try it. Note that building client_test also requires boost.regex and boost.program_options library.
dump_torrent
diff --git a/docs/extension_protocol.html b/docs/extension_protocol.html index 093f833f8..3d28c25ff 100644 --- a/docs/extension_protocol.html +++ b/docs/extension_protocol.html @@ -36,8 +36,8 @@ +Author: -Arvid Norberg, arvid@rasterbar.com -Ludvig Strigeus, ludde@utorrent.com Arvid Norberg, arvid@rasterbar.com +Ludvig Strigeus, ludde@utorrent.com diff --git a/docs/features.html b/docs/features.html index 87575f6da..8efe85c6f 100644 --- a/docs/features.html +++ b/docs/features.html @@ -36,16 +36,16 @@ +Author: -Arvid Norberg, arvid@rasterbar.com Arvid Norberg, arvid@rasterbar.com Table of contents
@@ -68,26 +68,35 @@ project (including this documentation). The current state includes the following features:- trackerless torrents (using the Mainline kademlia DHT protocol) with -some DHT extensions. -
- support for IPv6 +some DHT extensions. BEP 5. +
- support for IPv6, including BEP 7 and BEP 24.
- NAT-PMP and UPnP support (automatic port mapping on routers that supports it) -
- piece-wise, unordered, incremental file allocation -
- uses separate threads for checking files and for main downloader, with a -fool-proof thread-safe library interface. (i.e. There's no way for the -user to cause a deadlock). (see threads) +
- uses a separate disk I/O thread to not have the disk ever block on network or +client interaction. (see threads). +
- supports the bittorrent extension protocol. See extensions. BEP 10. +
- supports the uTorrent metadata transfer protocol (i.e. magnet links). +
- supports the uTorrent peer exchange protocol (PEX). +
- supports local peer discovery (multicasts for peers on the same local network)
- adjusts the length of the request queue depending on download rate. -
- multitracker extension support (as specified by John Hoffman) +
- has an adjustable read and write disk cache for improved disk throughput. +
- multitracker extension support (supports both strict BEP 12 and the +uTorrent interpretation). +
- tracker scrapes +
- supports lt_trackers extension, to exchange trackers between peers +
- supports both sparse files and compact file allocation (where pieces +are kept consolidated on disk) +
- super seeding/initial seeding (BEP 16).
- supports files > 2 gigabytes.
- serves multiple torrents on a single port and in a single thread
- fast resume support, a way to get rid of the costly piece check at the start of a resumed torrent. Saves the storage state, piece_picker state as well as all local peers in a separate fast-resume file. -
- HTTP seeding, as specified by Michael Burford of GetRight. +
- HTTP seeding, as specified in BEP 17 and BEP 19.
- piece picking on block-level (as opposed to piece-level). This means it can download parts of the same piece from different peers. It will also prefer to download whole pieces from single peers if the download speed is high enough from that particular peer. -
- supports the udp-tracker protocol by Olaf van der Spek. +
- supports the udp-tracker protocol. (BEP 15).
- queues torrents for file check, instead of checking all of them in parallel.
- supports http proxies and basic proxy authentication
- gzipped tracker-responses @@ -96,7 +105,6 @@ unchoked peers
- implements fair trade. User settable trade-ratio, must at least be 1:1, but one can choose to trade 1 for 2 or any other ratio that isn't unfair to the other party. -
- supports an extension protocol. See extensions.
- supports the no_peer_id=1 extension that will ease the load off trackers.
- possibility to limit the number of connections.
- delays have messages if there's no other outgoing traffic to the peer, and @@ -109,14 +117,15 @@ resumes. This means it can resume a torrent downloaded by any client. want to download.
- ip filter to disallow ip addresses and ip ranges from connecting and being connected +
- private torrents (BEP 27).
portability
libtorrent is portable at least among Windows, MacOS X and other UNIX-systems. It uses Boost.Thread, Boost.Filesystem, Boost.Date_time and various other -boost libraries as well as zlib (shipped) and asio (shipped). At least version -1.33.1 of boost is required.
+boost libraries as well as zlib (shipped) and asio (shipped). At least version +1.34.1 of boost is required.Since libtorrent uses asio, it will take full advantage of high performance network APIs on the most popular platforms. I/O completion ports on windows, epoll on linux and kqueue on MacOS X and BSD.
@@ -136,12 +145,12 @@ epoll on linux and kqueue on MacOS X and BSD.license
-libtorrent is released under the BSD-license.
+libtorrent is released under the BSD-license.
This means that you can use the library in your project without having to release its source code. The only requirement is that you give credit to the author of the library by including the libtorrent license in your software or documentation.
-Here's a list of some projects that uses libtorrent.
+Here's a list of some projects that uses libtorrent.
+Author: -Arvid Norberg, arvid@rasterbar.com Arvid Norberg, arvid@rasterbar.com @@ -44,21 +44,21 @@Contents
libtorrent has a plugin interface for implementing extensions to the protocol. @@ -68,7 +68,7 @@ to fit a particular (closed) network.
In short, the plugin interface makes it possible to:
- register extension messages (sent in the extension handshake), see -extensions. +extensions.
- add data and parse data from the extension handshake.
- send extension messages and standard bittorrent messages.
- override or block the handling of standard bittorrent messages. @@ -85,7 +85,7 @@ thread, you cannot use any of the member functions on the internal structures in libtorrent, since those require the mutex to be locked. Futhermore, you would also need to have a mutex on your own shared data within the plugin, to make sure it is not accessed at the same time from the libtorrent thread (through a -callback). See boost thread's mutex. If you need to send out a message from +callback). See boost thread's mutex. If you need to send out a message from another thread, use an internal queue, and do the actual sending in tick().
Table of contents
-
-
- overview -
- high level example -
- add_files -
- set_piece_hashes() -
- file_storage -
- create_torrent
-
-
- create_torrent() -
- generate() +
- overview +
- high level example +
- add_files +
- set_piece_hashes() +
- file_storage
-
+
- add_file() +
+ - create_torrent
+add_file()
++
++void add_file(file_entry const& e); +void add_file(fs::path const& p, size_type size, int flags = 0); +void add_file(fs::wpath const& p, size_type size, int flags = 0); +
+Adds a file to the file storage. The flags argument sets attributes on the file. +The file attributes is an extension and may not work in all bittorrent clients. +The possible arreibutes are:
++pad_file +attribute_hidden +attribute_executable +
+
create_torrent
@@ -199,8 +234,7 @@ public:struct create_torrent { - create_torrent(file_storage& fs, int piece_size); - create_torrent(file_storage& fs); + create_torrent(file_storage& fs, int piece_size = 0, int pad_size_limit = -1); create_torrent(torrent_info const& ti); entry generate() const; @@ -213,28 +247,33 @@ struct create_torrent void add_url_seed(std::string const& url); void add_node(std::pair<std::string, int> const& node); void add_tracker(std::string const& url, int tier = 0); + void set_priv(bool p); int num_pieces() const; int piece_length() const; int piece_size(int i) const; + bool priv() const; };
create_torrent()
--create_torrent(file_storage& fs, int piece_size); -create_torrent(file_storage& fs); +create_torrent(file_storage& fs, int piece_size = 0, int pad_size_limit = -1); create_torrent(torrent_info const& ti);
The contrstructor that does not take a piece_size will calculate -a piece size such that the torrent file is roughly 40 kB.
+The piece_size is the size of each piece in bytes. It must +be a multiple of 16 kiB. If a piece size of 0 is specified, a +piece_size will becalculated such that the torrent file is roughly 40 kB.
+If a pad_size_limit is specified (other than -1), any file larger than +the specified number of bytes will be preceeded by a pad file to align it +with the start od a piece.
The overlad that takes a torrent_info object will make a verbatim copy of its info dictionary (to preserve the info-hash). The copy of the info dictionary will be used by generate(). This means that none of the member functions of create_torrent that affects -the content of the info dictionary (such as set_hash()), will not +the content of the info dictionary (such as set_hash()), will have any affect.
It may be useful to add custom entries to the torrent file before bencoding it and saving it to disk.
set_comment()
++++void set_comment(char const* str); ++
Sets the comment for the torrent. The string str should be utf-8 encoded. +The comment in a torrent file is optional.
+set_creator()
++++void set_creator(char const* str); ++
Sets the creator of the torrent. The string str should be utf-8 encoded. +This is optional.
+set_hash()
++++void set_hash(int index, sha1_hash const& h); ++
This sets the SHA-1 hash for the specified piece (index). You are required +to set the hash for every piece in the torrent before generating it. If you have +the files on disk, you can use the high level convenience function to do this. +See set_piece_hashes().
+add_url_seed()
++++void add_url_seed(std::string const& url); ++
This adds a url seed to the torrent. You can have any number of url seeds. For a +single file torrent, this should be an HTTP url, pointing to a file with identical +content as the file of the torrent. For a multi-file torrent, it should point to +a directory containing a directory with the same name as this torrent, and all the +files of the torrent in it.
+add_node()
++++void add_node(std::pair<std::string, int> const& node); ++
This adds a DHT node to the torrent. This especially useful if you're creating a +tracker less torrent. It can be used by clients to bootstrap their DHT node from. +The node is a hostname and a port number where there is a DHT node running. +You can have any number of DHT nodes in a torrent.
+add_tracker()
++++void add_tracker(std::string const& url, int tier = 0); ++
Adds a tracker to the torrent. This is not strictly required, but most torrents +use a tracker as their main source of peers. The url should be an http:// or udp:// +url to a machine running a bittorrent tracker that accepts announces for this torrent's +info-hash. The tier is the fallback priority of the tracker. All trackers with tier 0 are +tried first (in any order). If all fail, trackers with tier 1 are tried. If all of those +fail, trackers with tier 2 are tried, and so on.
+set_priv() priv()
++++void set_priv(bool p); +bool priv() const; ++
Sets and queries the private flag of the torrent.
+torrent_info
In previous versions of libtorrent, this class was also used for creating torrent files. This functionality has been moved to create_torrent, see -make_torrent.
+make_torrent.The torrent_info has the following synopsis:
class torrent_info @@ -1369,7 +1388,10 @@ public: std::vector<announce_entry> const& trackers() const; file_storage const& files() const; - file_storage& files(); + file_storage const& orig_files() const; + + void rename_file(int index, std::string const& new_filename); + void rename_file(int index, std::wstring const& new_filename); typedef file_storage::iterator file_iterator; typedef file_storage::reverse_iterator reverse_file_iterator; @@ -1390,6 +1412,9 @@ public: bool priv() const; std::vector<std::string> const& url_seeds() const; + void add_url_seed(std::string const& url); + std::vector<std::string> const& http_seeds() const; + void add_http_seed(std::string const& url); size_type total_size() const; int piece_length() const; @@ -1431,7 +1456,7 @@ from the swarm.The constructor that takes a lazy_entry will create a torrent_info object from the information found in the given torrent_file. The lazy_entry represents a tree node in an bencoded file. To load an ordinary .torrent file -into a lazy_entry, use lazy_bdecode(), see bdecode() bencode().
+into a lazy_entry, use lazy_bdecode(), see bdecode() bencode().The version that takes a buffer pointer and a size will decode it as a .torrent file and initialize the torrent_info object for you.
The version that takes a filename will simply load the torrent file and decode it inside @@ -1446,13 +1471,14 @@ void add_tracker(std::string const& url, int tier = 0);
add_tracker() adds a tracker to the announce-list. The tier determines the order in -which the trackers are to be tried. For more information see trackers().
+which the trackers are to be tried. For more information see trackers().files()
+files() orig_files()
file_storage const& file() const; +file_storage const& orig_files() const;
The file_storage object contains the information on how to map the pieces to @@ -1460,6 +1486,9 @@ files. It is separated from the t a storage object needs to be created without having a torrent file. When renaming files in a storage, the storage needs to make its own copy of the file_storage in order to make its mapping differ from the one in the torrent file.
+orig_files() returns the original (unmodified) file storage for this torrent. This +is used by the web server connection, which needs to request files with the original +names.
For more information on the file_storage object, see the separate document on how to create torrents.
The path is the full (relative) path of each file. i.e. if it is a multi-file @@ -1499,12 +1530,10 @@ if the file. In cases where multiple files are mapped into the same file though, the file_base should be set to an offset so that the different regions do not overlap. This is used when mapping "unselected" files into a so-called part file.
-orig_path is set to 0 in case the path element is an exact copy of that -found in the metadata. In case the path in the original metadata was -incorrectly encoded, and had to be fixed in order to be acceptable utf-8, -the original string is preserved in orig_path. The reason to keep it -is to be able to reproduce the info-section exactly, with the correct -info-hash.
+pad_file is set to true for files that are not part of the data of the torrent. +They are just there to make sure the next file is aligned to a particular byte offset +or piece boundry. These files should typically be hidden from an end user. They are +not written to disk.
num_files() file_at()
@@ -1571,19 +1600,22 @@ struct peer_request + size is not allowed to be greater than the file size. file_index must refer to a valid file, i.e. it cannot be >= num_files().url_seeds() add_url_seed()
+url_seeds() add_url_seed() http_seeds() add_http_seed()
-std::vector<std::string> const& url_seeds() const; void add_url_seed(std::string const& url); +std::vector<std::string> const& http_seeds() const; +void add_http_seed(std::string const& url);
If there are any url-seeds in this torrent, url_seeds() will return a -vector of those urls. If you're creating a torrent file, add_url_seed() -adds one url to the list of url-seeds. Currently, the only transport protocol +
If there are any url-seeds or http seeds in this torrent, url_seeds() +and http_seeds() will return a vector of those urls. +add_url_seed() and add_http_seed() adds one url to the list of +url/http seeds. Currently, the only transport protocol supported for the url is http.
-See HTTP seeding for more information.
+See HTTP seeding for more information.
trackers()
@@ -1662,7 +1694,7 @@ char const* hash_for_piece_ptr(unsigned int index) const;hash_for_piece() takes a piece-index and returns the 20-bytes sha1-hash for that piece and info_hash() returns the 20-bytes sha1-hash for the info-section of the -torrent file. For more information on the sha1_hash, see the big_number class. +torrent file. For more information on the sha1_hash, see the big_number class. hash_for_piece_ptr() returns a pointer to the 20 byte sha1 digest for the piece. Note that the string is not null-terminated.
name() returns the name of the torrent.
comment() returns the comment associated with the torrent. If there's no comment, -it will return an empty string. creation_date() returns a boost::posix_time::ptime +it will return an empty string. creation_date() returns a boost::posix_time::ptime object, representing the time when this torrent file was created. If there's no time stamp in the torrent file, this will return a date of January 1:st 1970.
Both the name and the comment is UTF-8 encoded strings.
@@ -1762,6 +1794,10 @@ struct torrent_handle void remove_url_seed(std::string const& url); std::set<std::string> url_seeds() const; + void add_http_seed(std::string const& url); + void remove_http_seed(std::string const& url); + std::set<std::string> http_seeds() const; + void set_ratio(float ratio) const; void set_max_uploads(int max_uploads) const; void set_max_connections(int max_connections) const; @@ -1822,6 +1858,7 @@ struct torrent_handle enum flags_t { overwrite_existing = 1 }; void add_piece(int piece, char const* data, int flags = 0) const; + void read_piece(int piece) const; sha1_hash info_hash() const; @@ -1836,7 +1873,7 @@ valid handle. If you try to perform any operation on an uninitialized handle, it will throw invalid_handle.Warning
-All operations on a torrent_handle may throw invalid_handle +
All operations on a torrent_handle may throw invalid_handle exception, in case the handle is no longer refering to a torrent. There is one exception is_valid() will never throw. Since the torrents are processed by a background thread, there is no @@ -1913,7 +1950,7 @@ void file_progress(std::vector<size_type>& fp);
This function fills in the supplied vector with the the number of bytes downloaded of each file in this torrent. The progress values are ordered the same as the files -in the torrent_info. This operation is not very cheap. Its complexity is O(n + mj). +in the torrent_info. This operation is not very cheap. Its complexity is O(n + mj). Where n is the number of files, m is the number of downloading pieces and j is the number of blocks in a piece.
save_path() returns the path that was given to add_torrent() when this torrent +
save_path() returns the path that was given to add_torrent() when this torrent was started.
Since the data is written asynchronously, you may know that is passed or failed the hash check by waiting for piece_finished_alert or has_failed_alert.
read_piece()
++++void read_piece(int piece) const; ++
This function starts an asynchronous read operation of the specified piece from +this torrent. You must have completed the download of the specified piece before +calling this function.
+When the read operation is completed, it is passed back through an alert, +read_piece_alert. In order to receive this alert, you must enable +alert::storage_notification in your alert mask (see set_alert_mask()).
+Note that if you read multiple pieces, the read operations are not guaranteed to +finish in the same order as you initiated them.
+force_reannounce()
@@ -2019,8 +2072,8 @@ void scrape_tracker() const; tracker for statistics such as total number of incomplete peers, complete peers, number of downloads etc.This request will specifically update the num_complete and num_incomplete fields in -the torrent_status struct once it completes. When it completes, it will generate a -scrape_reply_alert. If it fails, it will generate a scrape_failed_alert.
+the torrent_status struct once it completes. When it completes, it will generate a +scrape_reply_alert. If it fails, it will generate a scrape_failed_alert.
connect_peer()
@@ -2033,8 +2086,8 @@ void connect_peer(asio::ip::tcp::endpoint const& adr, int source = 0) const; torrent. If the peer does not respond, or is not a member of this torrent, it will simply be disconnected. No harm can be done by using this other than an unnecessary connection attempt is made. If the torrent is uninitialized or in queued or checking mode, this -will throw invalid_handle. The second (optional) argument will be bitwised ORed into -the source mask of this peer. Typically this is one of the source flags in peer_info. +will throw invalid_handle. The second (optional) argument will be bitwised ORed into +the source mask of this peer. Typically this is one of the source flags in peer_info. i.e. tracker, pex, dht etc.is_paused() only returns true if the torrent itself is paused. If the torrent is not running because the session is paused, this still returns false. To know if a torrent is active or not, you need to inspect both torrent_handle::is_paused() @@ -2159,8 +2212,8 @@ bool resolve_countries() const;
Sets or gets the flag that derermines if countries should be resolved for the peers of this -torrent. It defaults to false. If it is set to true, the peer_info structure for the peers -in this torrent will have their country member set. See peer_info for more information +torrent. It defaults to false. If it is set to true, the peer_info structure for the peers +in this torrent will have their country member set. See peer_info for more information on how to interpret this field.
is_auto_managed() returns true if this torrent is currently auto managed. auto_managed() changes whether the torrent is auto managed or not. For more info, -see queuing.
+see queuing.has_metadata() set_metadata()
@@ -2195,7 +2248,7 @@ bool set_metadata(char const* buf, int size) const;has_metadata returns true if this torrent has metadata (either it was started from a .torrent file or the metadata has been downloaded). The only scenario where this can return false is when the torrent was started torrent-less (i.e. with just an info-hash and tracker -ip). Note that if the torrent doesn't have metadata, the member get_torrent_info() will +ip). Note that if the torrent doesn't have metadata, the member get_torrent_info() will throw.
set_metadata expects the info section of metadata. i.e. The buffer passed in will be hashed and verified against the info-hash. If it fails, a metadata_failed_alert will be @@ -2230,7 +2283,7 @@ which this tracker is tried. If you want libtorrent to use another list of trackers for this torrent, you can use replace_trackers() which takes a list of the same form as the one returned from trackers() and will replace it. If you want an immediate effect, you have to call -force_reannounce().
+force_reannounce().add_tracker() will look if the specified tracker is already in the set. If it is, it doesn't do anything. If it's not in the current set of trackers, it will insert it in the tier specified in the announce_entry.
@@ -2251,7 +2304,20 @@ paused, queued, checking or seeding. url_seeds() return a set of the url seeds currently in this torrent. Note that urls that fails may be removed automatically from the list. -See HTTP seeding for more information.
+See HTTP seeding for more information.
+add_http_seed() remove_http_seed() http_seeds()
++++void add_http_seed(std::string const& url); +void remove_http_seed(std::string const& url); +std::set<std::string> http_seeds() const; ++
These functions are identical as the *_url_seed() variants, but they +operate on BEP 17 web seeds instead of BEP 19.
+See HTTP seeding for more information.
queue_position() queue_position_up() queue_position_down() queue_position_top() queue_position_bottom()
@@ -2284,7 +2350,7 @@ void use_interface(char const* net_interface) const;use_interface() sets the network interface this torrent will use when it opens outgoing -connections. By default, it uses the same interface as the session uses to listen on. The +connections. By default, it uses the same interface as the session uses to listen on. The parameter must be a string containing an ip-address (either an IPv4 or IPv6 address). If the string does not conform to this format and exception is thrown.
save_resume_data() generates fast-resume data and returns it as an entry. This entry -is suitable for being bencoded. For more information about how fast-resume works, see fast resume.
+save_resume_data() generates fast-resume data and returns it as an entry. This entry +is suitable for being bencoded. For more information about how fast-resume works, see fast resume.
This operation is asynchronous, save_resume_data will return immediately. The resume data -is delivered when it's done through an save_resume_data_alert.
+is delivered when it's done through an save_resume_data_alert.The fast resume data will be empty in the following cases:
@@ -2332,13 +2398,13 @@ is delivered when it's done through an metadata from peers extension) +(see libtorrent's metadata from peers extension)
Note that by the time you receive the fast resume data, it may already be invalid if the torrent is still downloading! The recommended practice is to first pause the session, then generate the -fast resume data, and then close it down. Make sure to not remove_torrent() before you receive -the save_resume_data_alert though. There's no need to pause when saving intermittent resume data.
+fast resume data, and then close it down. Make sure to not remove_torrent() before you receive +the save_resume_data_alert though. There's no need to pause when saving intermittent resume data.Warning
If you pause every torrent individually instead of pausing the session, every torrent @@ -2349,11 +2415,12 @@ will have its paused state saved in the resume data!
It is typically a good idea to save resume data whenever a torrent is completed or paused. In those cases you don't need to pause the torrent or the session, since the torrent will do no more writing to its files. If you save resume data for torrents when they are paused, you can accelerate the -shutdown process by not saving resume data again for paused torrents.
+shutdown process by not saving resume data again for paused torrents. Completed torrents should +have their resume data saved when they complete and on exit, since their statistics might be updated.In full allocation mode the reume data is never invalidated by subsequent writes to the files, since pieces won't move around. This means that you don't need to pause before writing resume data in full or sparse mode. If you don't, however, any data written to -disk after you saved resume data and before the session closed is lost.
+disk after you saved resume data and before the session closed is lost.It also means that if the resume data is out dated, libtorrent will not re-check the files, but assume that it is fairly recent. The assumption is that it's better to loose a little bit than to re-check @@ -2416,8 +2483,8 @@ torrent_status status() const;
status() will return a structure with information about the status of this -torrent. If the torrent_handle is invalid, it will throw invalid_handle exception. -See torrent_status.
+torrent. If the torrent_handle is invalid, it will throw invalid_handle exception. +See torrent_status.