1549 lines
63 KiB
ReStructuredText
1549 lines
63 KiB
ReStructuredText
============================
|
|
libtorrent API Documentation
|
|
============================
|
|
|
|
:Author: Arvid Norberg, arvid@rasterbar.com
|
|
:Version: 1.0.0
|
|
|
|
.. contents:: Table of contents
|
|
:depth: 1
|
|
:backlinks: none
|
|
|
|
overview
|
|
========
|
|
|
|
The interface of libtorrent consists of a few classes. The main class is
|
|
the ``session``, it contains the main loop that serves all torrents.
|
|
|
|
The basic usage is as follows:
|
|
|
|
* construct a session
|
|
* load session state from settings file (see `load_state() save_state()`_)
|
|
* start extensions (see `add_extension()`_).
|
|
* start DHT, LSD, UPnP, NAT-PMP etc (see `start_dht() stop_dht() set_dht_settings() dht_state() is_dht_running()`_
|
|
`start_lsd() stop_lsd()`_, `start_upnp() stop_upnp()`_ and `start_natpmp() stop_natpmp()`_)
|
|
* parse .torrent-files and add them to the session (see `bdecode() bencode()`_ and `async_add_torrent() add_torrent()`_)
|
|
* main loop (see session_)
|
|
|
|
* query the torrent_handles for progress (see torrent_handle_)
|
|
* query the session for information
|
|
* add and remove torrents from the session at run-time
|
|
|
|
* save resume data for all torrent_handles (optional, see
|
|
`save_resume_data()`_)
|
|
* save session state (see `load_state() save_state()`_)
|
|
* destruct session object
|
|
|
|
Each class and function is described in this manual.
|
|
|
|
For a description on how to create torrent files, see make_torrent_.
|
|
|
|
.. _make_torrent: make_torrent.html
|
|
|
|
things to keep in mind
|
|
======================
|
|
|
|
A common problem developers are facing is torrents stopping without explanation.
|
|
Here is a description on which conditions libtorrent will stop your torrents,
|
|
how to find out about it and what to do about it.
|
|
|
|
Make sure to keep track of the paused state, the error state and the upload
|
|
mode of your torrents. By default, torrents are auto-managed, which means
|
|
libtorrent will pause them, unpause them, scrape them and take them out
|
|
of upload-mode automatically.
|
|
|
|
Whenever a torrent encounters a fatal error, it will be stopped, and the
|
|
``torrent_status::error`` will describe the error that caused it. If a torrent
|
|
is auto managed, it is scraped periodically and paused or resumed based on
|
|
the number of downloaders per seed. This will effectively seed torrents that
|
|
are in the greatest need of seeds.
|
|
|
|
If a torrent hits a disk write error, it will be put into upload mode. This
|
|
means it will not download anything, but only upload. The assumption is that
|
|
the write error is caused by a full disk or write permission errors. If the
|
|
torrent is auto-managed, it will periodically be taken out of the upload
|
|
mode, trying to write things to the disk again. This means torrent will recover
|
|
from certain disk errors if the problem is resolved. If the torrent is not
|
|
auto managed, you have to call `set_upload_mode()`_ to turn
|
|
downloading back on again.
|
|
|
|
network primitives
|
|
==================
|
|
|
|
There are a few typedefs in the ``libtorrent`` namespace which pulls
|
|
in network types from the ``asio`` namespace. These are::
|
|
|
|
typedef asio::ip::address address;
|
|
typedef asio::ip::address_v4 address_v4;
|
|
typedef asio::ip::address_v6 address_v6;
|
|
using asio::ip::tcp;
|
|
using asio::ip::udp;
|
|
|
|
These are declared in the ``<libtorrent/socket.hpp>`` header.
|
|
|
|
The ``using`` statements will give easy access to::
|
|
|
|
tcp::endpoint
|
|
udp::endpoint
|
|
|
|
Which are the endpoint types used in libtorrent. An endpoint is an address
|
|
with an associated port.
|
|
|
|
For documentation on these types, please refer to the `asio documentation`_.
|
|
|
|
.. _`asio documentation`: http://asio.sourceforge.net/asio-0.3.8/doc/asio/reference.html
|
|
|
|
session customization
|
|
=====================
|
|
|
|
You have some control over session configuration through the ``session_settings`` object. You
|
|
create it and fill it with your settings and then use ``session::set_settings()``
|
|
to apply them.
|
|
|
|
You have control over proxy and authorization settings and also the user-agent
|
|
that will be sent to the tracker. The user-agent will also be used to identify the
|
|
client with other peers.
|
|
|
|
session_settings
|
|
----------------
|
|
|
|
::
|
|
|
|
struct session_settings
|
|
{
|
|
session_settings();
|
|
int version;
|
|
std::string user_agent;
|
|
int tracker_completion_timeout;
|
|
int tracker_receive_timeout;
|
|
int stop_tracker_timeout;
|
|
int tracker_maximum_response_length;
|
|
|
|
int piece_timeout;
|
|
float request_queue_time;
|
|
int max_allowed_in_request_queue;
|
|
int max_out_request_queue;
|
|
int whole_pieces_threshold;
|
|
int peer_timeout;
|
|
int urlseed_timeout;
|
|
int urlseed_pipeline_size;
|
|
int file_pool_size;
|
|
bool allow_multiple_connections_per_ip;
|
|
int max_failcount;
|
|
int min_reconnect_time;
|
|
int peer_connect_timeout;
|
|
bool ignore_limits_on_local_network;
|
|
int connection_speed;
|
|
bool send_redundant_have;
|
|
bool lazy_bitfields;
|
|
int inactivity_timeout;
|
|
int unchoke_interval;
|
|
int optimistic_unchoke_interval;
|
|
std::string announce_ip;
|
|
int num_want;
|
|
int initial_picker_threshold;
|
|
int allowed_fast_set_size;
|
|
|
|
enum { no_piece_suggestions = 0, suggest_read_cache = 1 };
|
|
int suggest_mode;
|
|
int max_queued_disk_bytes;
|
|
int handshake_timeout;
|
|
bool use_dht_as_fallback;
|
|
bool free_torrent_hashes;
|
|
bool upnp_ignore_nonrouters;
|
|
int send_buffer_watermark;
|
|
int send_buffer_watermark_factor;
|
|
|
|
#ifndef TORRENT_NO_DEPRECATE
|
|
bool auto_upload_slots;
|
|
bool auto_upload_slots_rate_based;
|
|
#endif
|
|
|
|
enum choking_algorithm_t
|
|
{
|
|
fixed_slots_choker,
|
|
auto_expand_choker,
|
|
rate_based_choker,
|
|
bittyrant_choker
|
|
};
|
|
|
|
int choking_algorithm;
|
|
|
|
enum seed_choking_algorithm_t
|
|
{
|
|
round_robin,
|
|
fastest_upload,
|
|
anti_leech
|
|
};
|
|
|
|
int seed_choking_algorithm;
|
|
|
|
bool use_parole_mode;
|
|
int cache_size;
|
|
int cache_buffer_chunk_size;
|
|
int cache_expiry;
|
|
bool use_read_cache;
|
|
bool explicit_read_cache;
|
|
int explicit_cache_interval;
|
|
|
|
enum io_buffer_mode_t
|
|
{
|
|
enable_os_cache = 0,
|
|
disable_os_cache_for_aligned_files = 1,
|
|
disable_os_cache = 2
|
|
};
|
|
int disk_io_write_mode;
|
|
int disk_io_read_mode;
|
|
|
|
std::pair<int, int> outgoing_ports;
|
|
char peer_tos;
|
|
|
|
int active_downloads;
|
|
int active_seeds;
|
|
int active_dht_limit;
|
|
int active_tracker_limit;
|
|
int active_limit;
|
|
bool auto_manage_prefer_seeds;
|
|
bool dont_count_slow_torrents;
|
|
int auto_manage_interval;
|
|
float share_ratio_limit;
|
|
float seed_time_ratio_limit;
|
|
int seed_time_limit;
|
|
int peer_turnover_interval;
|
|
float peer_turnover;
|
|
float peer_turnover_cutoff;
|
|
bool close_redundant_connections;
|
|
|
|
int auto_scrape_interval;
|
|
int auto_scrape_min_interval;
|
|
|
|
int max_peerlist_size;
|
|
|
|
int min_announce_interval;
|
|
|
|
bool prioritize_partial_pieces;
|
|
int auto_manage_startup;
|
|
|
|
bool rate_limit_ip_overhead;
|
|
|
|
bool announce_to_all_trackers;
|
|
bool announce_to_all_tiers;
|
|
|
|
bool prefer_udp_trackers;
|
|
bool strict_super_seeding;
|
|
|
|
int seeding_piece_quota;
|
|
|
|
int max_sparse_regions;
|
|
|
|
bool lock_disk_cache;
|
|
|
|
int max_rejects;
|
|
|
|
int recv_socket_buffer_size;
|
|
int send_socket_buffer_size;
|
|
|
|
bool optimize_hashing_for_speed;
|
|
|
|
int file_checks_delay_per_block;
|
|
|
|
enum disk_cache_algo_t
|
|
{ lru, largest_contiguous, avoid_readback };
|
|
|
|
disk_cache_algo_t disk_cache_algorithm;
|
|
|
|
int read_cache_line_size;
|
|
int write_cache_line_size;
|
|
|
|
int optimistic_disk_retry;
|
|
bool disable_hash_checks;
|
|
|
|
int max_suggest_pieces;
|
|
|
|
bool drop_skipped_requests;
|
|
|
|
bool low_prio_disk;
|
|
int local_service_announce_interval;
|
|
int dht_announce_interval;
|
|
|
|
int udp_tracker_token_expiry;
|
|
bool volatile_read_cache;
|
|
bool guided_read_cache;
|
|
bool default_cache_min_age;
|
|
|
|
int num_optimistic_unchoke_slots;
|
|
bool no_atime_storage;
|
|
int default_est_reciprocation_rate;
|
|
int increase_est_reciprocation_rate;
|
|
int decrease_est_reciprocation_rate;
|
|
bool incoming_starts_queued_torrents;
|
|
bool report_true_downloaded;
|
|
bool strict_end_game_mode;
|
|
|
|
bool broadcast_lsd;
|
|
|
|
bool enable_outgoing_utp;
|
|
bool enable_incoming_utp;
|
|
bool enable_outgoing_tcp;
|
|
bool enable_incoming_tcp;
|
|
int max_pex_peers;
|
|
bool ignore_resume_timestamps;
|
|
bool no_recheck_incomplete_resume;
|
|
bool anonymous_mode;
|
|
bool force_proxy;
|
|
int tick_interval;
|
|
int share_mode_target;
|
|
|
|
int upload_rate_limit;
|
|
int download_rate_limit;
|
|
int local_upload_rate_limit;
|
|
int local_download_rate_limit;
|
|
int dht_upload_rate_limit;
|
|
int unchoke_slots_limit;
|
|
int half_open_limit;
|
|
int connections_limit;
|
|
|
|
int utp_target_delay;
|
|
int utp_gain_factor;
|
|
int utp_min_timeout;
|
|
int utp_syn_resends;
|
|
int utp_num_resends;
|
|
int utp_connect_timeout;
|
|
bool utp_dynamic_sock_buf;
|
|
int utp_loss_multiplier;
|
|
|
|
enum bandwidth_mixed_algo_t
|
|
{
|
|
prefer_tcp = 0,
|
|
peer_proportional = 1
|
|
|
|
};
|
|
int mixed_mode_algorithm;
|
|
bool rate_limit_utp;
|
|
|
|
int listen_queue_size;
|
|
|
|
bool announce_double_nat;
|
|
|
|
int torrent_connect_boost;
|
|
bool seeding_outgoing_connections;
|
|
|
|
bool no_connect_privileged_ports;
|
|
int alert_queue_size;
|
|
int max_metadata_size;
|
|
bool smooth_connects;
|
|
bool always_send_user_agent;
|
|
bool apply_ip_filter_to_trackers;
|
|
int read_job_every;
|
|
bool use_disk_read_ahead;
|
|
bool lock_files;
|
|
|
|
int ssl_listen;
|
|
|
|
int tracker_backoff;
|
|
|
|
bool ban_web_seeds;
|
|
int max_http_recv_buffer_size;
|
|
|
|
bool support_share_mode;
|
|
bool support_merkle_torrents;
|
|
bool report_redundant_bytes;
|
|
std::string handshake_client_version;
|
|
bool use_disk_cache_pool;
|
|
};
|
|
|
|
exceptions
|
|
==========
|
|
|
|
Many functions in libtorrent have two versions, one that throws exceptions on
|
|
errors and one that takes an ``error_code`` reference which is filled with the
|
|
error code on errors.
|
|
|
|
There is one exception class that is used for errors in libtorrent, it is based
|
|
on boost.system's ``error_code`` class to carry the error code.
|
|
|
|
libtorrent_exception
|
|
--------------------
|
|
|
|
::
|
|
|
|
struct libtorrent_exception: std::exception
|
|
{
|
|
libtorrent_exception(error_code const& s);
|
|
virtual const char* what() const throw();
|
|
virtual ~libtorrent_exception() throw() {}
|
|
boost::system::error_code error() const;
|
|
};
|
|
|
|
|
|
error_code
|
|
==========
|
|
|
|
The names of these error codes are declared in then ``libtorrent::errors`` namespace.
|
|
|
|
HTTP errors are reported in the ``libtorrent::http_category``, with error code enums in
|
|
the ``libtorrent::errors`` namespace.
|
|
|
|
====== =========================================
|
|
code symbol
|
|
====== =========================================
|
|
100 cont
|
|
------ -----------------------------------------
|
|
200 ok
|
|
------ -----------------------------------------
|
|
201 created
|
|
------ -----------------------------------------
|
|
202 accepted
|
|
------ -----------------------------------------
|
|
204 no_content
|
|
------ -----------------------------------------
|
|
300 multiple_choices
|
|
------ -----------------------------------------
|
|
301 moved_permanently
|
|
------ -----------------------------------------
|
|
302 moved_temporarily
|
|
------ -----------------------------------------
|
|
304 not_modified
|
|
------ -----------------------------------------
|
|
400 bad_request
|
|
------ -----------------------------------------
|
|
401 unauthorized
|
|
------ -----------------------------------------
|
|
403 forbidden
|
|
------ -----------------------------------------
|
|
404 not_found
|
|
------ -----------------------------------------
|
|
500 internal_server_error
|
|
------ -----------------------------------------
|
|
501 not_implemented
|
|
------ -----------------------------------------
|
|
502 bad_gateway
|
|
------ -----------------------------------------
|
|
503 service_unavailable
|
|
====== =========================================
|
|
|
|
translating error codes
|
|
-----------------------
|
|
|
|
The error_code::message() function will typically return a localized error string,
|
|
for system errors. That is, errors that belong to the generic or system category.
|
|
|
|
Errors that belong to the libtorrent error category are not localized however, they
|
|
are only available in english. In order to translate libtorrent errors, compare the
|
|
error category of the ``error_code`` object against ``libtorrent::get_libtorrent_category()``,
|
|
and if matches, you know the error code refers to the list above. You can provide
|
|
your own mapping from error code to string, which is localized. In this case, you
|
|
cannot rely on ``error_code::message()`` to generate your strings.
|
|
|
|
The numeric values of the errors are part of the API and will stay the same, although
|
|
new error codes may be appended at the end.
|
|
|
|
Here's a simple example of how to translate error codes::
|
|
|
|
std::string error_code_to_string(boost::system::error_code const& ec)
|
|
{
|
|
if (ec.category() != libtorrent::get_libtorrent_category())
|
|
{
|
|
return ec.message();
|
|
}
|
|
// the error is a libtorrent error
|
|
|
|
int code = ec.value();
|
|
static const char const* swedish[] =
|
|
{
|
|
"inget fel",
|
|
"en fil i torrenten kolliderar med en fil fran en annan torrent",
|
|
"hash check misslyckades",
|
|
"torrent filen ar inte en dictionary",
|
|
"'info'-nyckeln saknas eller ar korrupt i torrentfilen",
|
|
"'info'-faltet ar inte en dictionary",
|
|
"'piece length' faltet saknas eller ar korrupt i torrentfilen",
|
|
"torrentfilen saknar namnfaltet",
|
|
"ogiltigt namn i torrentfilen (kan vara en attack)",
|
|
// ... more strings here
|
|
};
|
|
|
|
// use the default error string in case we don't have it
|
|
// in our translated list
|
|
if (code < 0 || code >= sizeof(swedish)/sizeof(swedish[0]))
|
|
return ec.message();
|
|
|
|
return swedish[code];
|
|
}
|
|
|
|
storage_interface
|
|
=================
|
|
|
|
The storage interface is a pure virtual class that can be implemented to
|
|
customize how and where data for a torrent is stored. The default storage
|
|
implementation uses regular files in the filesystem, mapping the files in the
|
|
torrent in the way one would assume a torrent is saved to disk. Implementing
|
|
your own storage interface makes it possible to store all data in RAM, or in
|
|
some optimized order on disk (the order the pieces are received for instance),
|
|
or saving multifile torrents in a single file in order to be able to take
|
|
advantage of optimized disk-I/O.
|
|
|
|
It is also possible to write a thin class that uses the default storage but
|
|
modifies some particular behavior, for instance encrypting the data before
|
|
it's written to disk, and decrypting it when it's read again.
|
|
|
|
The storage interface is based on slots, each slot is 'piece_size' number
|
|
of bytes. All access is done by writing and reading whole or partial
|
|
slots. One slot is one piece in the torrent, but the data in the slot
|
|
does not necessarily correspond to the piece with the same index (in
|
|
compact allocation mode it won't).
|
|
|
|
libtorrent comes with two built-in storage implementations; ``default_storage``
|
|
and ``disabled_storage``. Their constructor functions are called ``default_storage_constructor``
|
|
and ``disabled_storage_constructor`` respectively. The disabled storage does
|
|
just what it sounds like. It throws away data that's written, and it
|
|
reads garbage. It's useful mostly for benchmarking and profiling purpose.
|
|
|
|
|
|
The interface looks like this::
|
|
|
|
struct storage_interface
|
|
{
|
|
virtual bool initialize(bool allocate_files) = 0;
|
|
virtual bool has_any_file() = 0;
|
|
virtual void hint_read(int slot, int offset, int len);
|
|
virtual int readv(file::iovec_t const* bufs, int slot, int offset, int num_bufs) = 0;
|
|
virtual int writev(file::iovec_t const* bufs, int slot, int offset, int num_bufs) = 0;
|
|
virtual int sparse_end(int start) const;
|
|
virtual bool move_storage(fs::path save_path) = 0;
|
|
virtual bool verify_resume_data(lazy_entry const& rd, error_code& error) = 0;
|
|
virtual bool write_resume_data(entry& rd) const = 0;
|
|
virtual bool move_slot(int src_slot, int dst_slot) = 0;
|
|
virtual bool swap_slots(int slot1, int slot2) = 0;
|
|
virtual bool swap_slots3(int slot1, int slot2, int slot3) = 0;
|
|
virtual bool rename_file(int file, std::string const& new_name) = 0;
|
|
virtual bool release_files() = 0;
|
|
virtual bool delete_files() = 0;
|
|
virtual void finalize_file(int index) {}
|
|
virtual ~storage_interface() {}
|
|
|
|
// non virtual functions
|
|
|
|
disk_buffer_pool* disk_pool();
|
|
void set_error(std::string const& file, error_code const& ec) const;
|
|
error_code const& error() const;
|
|
std::string const& error_file() const;
|
|
void clear_error();
|
|
};
|
|
|
|
|
|
initialize()
|
|
------------
|
|
|
|
::
|
|
|
|
bool initialize(bool allocate_files) = 0;
|
|
|
|
This function is called when the storage is to be initialized. The default storage
|
|
will create directories and empty files at this point. If ``allocate_files`` is true,
|
|
it will also ``ftruncate`` all files to their target size.
|
|
|
|
Returning ``true`` indicates an error occurred.
|
|
|
|
has_any_file()
|
|
--------------
|
|
|
|
::
|
|
|
|
virtual bool has_any_file() = 0;
|
|
|
|
This function is called when first checking (or re-checking) the storage for a torrent.
|
|
It should return true if any of the files that is used in this storage exists on disk.
|
|
If so, the storage will be checked for existing pieces before starting the download.
|
|
|
|
hint_read()
|
|
-----------
|
|
|
|
::
|
|
|
|
void hint_read(int slot, int offset, int len);
|
|
|
|
This function is called when a read job is queued. It gives the storage wrapper an
|
|
opportunity to hint the operating system about this coming read. For instance, the
|
|
storage may call ``posix_fadvise(POSIX_FADV_WILLNEED)`` or ``fcntl(F_RDADVISE)``.
|
|
|
|
readv() writev()
|
|
----------------
|
|
|
|
::
|
|
|
|
int readv(file::iovec_t const* buf, int slot, int offset, int num_bufs) = 0;
|
|
int write(const char* buf, int slot, int offset, int size) = 0;
|
|
|
|
These functions should read or write the data in or to the given ``slot`` at the given ``offset``.
|
|
It should read or write ``num_bufs`` buffers sequentially, where the size of each buffer
|
|
is specified in the buffer array ``bufs``. The file::iovec_t type has the following members::
|
|
|
|
struct iovec_t
|
|
{
|
|
void* iov_base;
|
|
size_t iov_len;
|
|
};
|
|
|
|
The return value is the number of bytes actually read or written, or -1 on failure. If
|
|
it returns -1, the error code is expected to be set to
|
|
|
|
Every buffer in ``bufs`` can be assumed to be page aligned and be of a page aligned size,
|
|
except for the last buffer of the torrent. The allocated buffer can be assumed to fit a
|
|
fully page aligned number of bytes though. This is useful when reading and writing the
|
|
last piece of a file in unbuffered mode.
|
|
|
|
The ``offset`` is aligned to 16 kiB boundries *most of the time*, but there are rare
|
|
exceptions when it's not. Specifically if the read cache is disabled/or full and a
|
|
client requests unaligned data, or the file itself is not aligned in the torrent.
|
|
Most clients request aligned data.
|
|
|
|
sparse_end()
|
|
------------
|
|
|
|
::
|
|
|
|
int sparse_end(int start) const;
|
|
|
|
This function is optional. It is supposed to return the first piece, starting at
|
|
``start`` that is fully contained within a data-region on disk (i.e. non-sparse
|
|
region). The purpose of this is to skip parts of files that can be known to contain
|
|
zeros when checking files.
|
|
|
|
move_storage()
|
|
--------------
|
|
|
|
::
|
|
|
|
bool move_storage(fs::path save_path) = 0;
|
|
|
|
This function should move all the files belonging to the storage to the new save_path.
|
|
The default storage moves the single file or the directory of the torrent.
|
|
|
|
Before moving the files, any open file handles may have to be closed, like
|
|
``release_files()``.
|
|
|
|
Returning ``false`` indicates an error occurred.
|
|
|
|
|
|
verify_resume_data()
|
|
--------------------
|
|
|
|
::
|
|
|
|
bool verify_resume_data(lazy_entry const& rd, error_code& error) = 0;
|
|
|
|
This function should verify the resume data ``rd`` with the files
|
|
on disk. If the resume data seems to be up-to-date, return true. If
|
|
not, set ``error`` to a description of what mismatched and return false.
|
|
|
|
The default storage may compare file sizes and time stamps of the files.
|
|
|
|
Returning ``false`` indicates an error occurred.
|
|
|
|
|
|
write_resume_data()
|
|
-------------------
|
|
|
|
::
|
|
|
|
bool write_resume_data(entry& rd) const = 0;
|
|
|
|
This function should fill in resume data, the current state of the
|
|
storage, in ``rd``. The default storage adds file timestamps and
|
|
sizes.
|
|
|
|
Returning ``true`` indicates an error occurred.
|
|
|
|
|
|
move_slot()
|
|
-----------
|
|
|
|
::
|
|
|
|
bool move_slot(int src_slot, int dst_slot) = 0;
|
|
|
|
This function should copy or move the data in slot ``src_slot`` to
|
|
the slot ``dst_slot``. This is only used in compact mode.
|
|
|
|
If the storage caches slots, this could be implemented more
|
|
efficient than reading and writing the data.
|
|
|
|
Returning ``true`` indicates an error occurred.
|
|
|
|
|
|
swap_slots()
|
|
------------
|
|
|
|
::
|
|
|
|
bool swap_slots(int slot1, int slot2) = 0;
|
|
|
|
This function should swap the data in ``slot1`` and ``slot2``. The default
|
|
storage uses a scratch buffer to read the data into, then moving the other
|
|
slot and finally writing back the temporary slot's data
|
|
|
|
This is only used in compact mode.
|
|
|
|
Returning ``true`` indicates an error occurred.
|
|
|
|
|
|
swap_slots3()
|
|
-------------
|
|
|
|
::
|
|
|
|
bool swap_slots3(int slot1, int slot2, int slot3) = 0;
|
|
|
|
This function should do a 3-way swap, or shift of the slots. ``slot1``
|
|
should move to ``slot2``, which should be moved to ``slot3`` which in turn
|
|
should be moved to ``slot1``.
|
|
|
|
This is only used in compact mode.
|
|
|
|
Returning ``true`` indicates an error occurred.
|
|
|
|
|
|
rename_file()
|
|
-------------
|
|
|
|
::
|
|
|
|
bool rename_file(int file, std::string const& new_name) = 0;
|
|
|
|
Rename file with index ``file`` to the thame ``new_name``. If there is an error,
|
|
``true`` should be returned.
|
|
|
|
|
|
release_files()
|
|
---------------
|
|
|
|
::
|
|
|
|
bool release_files() = 0;
|
|
|
|
This function should release all the file handles that it keeps open to files
|
|
belonging to this storage. The default implementation just calls
|
|
``file_pool::release_files(this)``.
|
|
|
|
Returning ``true`` indicates an error occurred.
|
|
|
|
|
|
delete_files()
|
|
--------------
|
|
|
|
::
|
|
|
|
bool delete_files() = 0;
|
|
|
|
This function should delete all files and directories belonging to this storage.
|
|
|
|
Returning ``true`` indicates an error occurred.
|
|
|
|
The ``disk_buffer_pool`` is used to allocate and free disk buffers. It has the
|
|
following members::
|
|
|
|
struct disk_buffer_pool : boost::noncopyable
|
|
{
|
|
char* allocate_buffer(char const* category);
|
|
void free_buffer(char* buf);
|
|
|
|
char* allocate_buffers(int blocks, char const* category);
|
|
void free_buffers(char* buf, int blocks);
|
|
|
|
int block_size() const { return m_block_size; }
|
|
|
|
void release_memory();
|
|
};
|
|
|
|
finalize_file()
|
|
---------------
|
|
|
|
::
|
|
|
|
virtual void finalize_file(int index);
|
|
|
|
This function is called each time a file is completely downloaded. The
|
|
storage implementation can perform last operations on a file. The file will
|
|
not be opened for writing after this.
|
|
|
|
``index`` is the index of the file that completed.
|
|
|
|
On windows the default storage implementation clears the sparse file flag
|
|
on the specified file.
|
|
|
|
example
|
|
-------
|
|
|
|
This is an example storage implementation that stores all pieces in a ``std::map``,
|
|
i.e. in RAM. It's not necessarily very useful in practice, but illustrates the
|
|
basics of implementing a custom storage.
|
|
|
|
::
|
|
|
|
struct temp_storage : storage_interface
|
|
{
|
|
temp_storage(file_storage const& fs) : m_files(fs) {}
|
|
virtual bool initialize(bool allocate_files) { return false; }
|
|
virtual bool has_any_file() { return false; }
|
|
virtual int read(char* buf, int slot, int offset, int size)
|
|
{
|
|
std::map<int, std::vector<char> >::const_iterator i = m_file_data.find(slot);
|
|
if (i == m_file_data.end()) return 0;
|
|
int available = i->second.size() - offset;
|
|
if (available <= 0) return 0;
|
|
if (available > size) available = size;
|
|
memcpy(buf, &i->second[offset], available);
|
|
return available;
|
|
}
|
|
virtual int write(const char* buf, int slot, int offset, int size)
|
|
{
|
|
std::vector<char>& data = m_file_data[slot];
|
|
if (data.size() < offset + size) data.resize(offset + size);
|
|
std::memcpy(&data[offset], buf, size);
|
|
return size;
|
|
}
|
|
virtual bool rename_file(int file, std::string const& new_name)
|
|
{ assert(false); return false; }
|
|
virtual bool move_storage(std::string const& save_path) { return false; }
|
|
virtual bool verify_resume_data(lazy_entry const& rd, error_code& error) { return false; }
|
|
virtual bool write_resume_data(entry& rd) const { return false; }
|
|
virtual bool move_slot(int src_slot, int dst_slot) { assert(false); return false; }
|
|
virtual bool swap_slots(int slot1, int slot2) { assert(false); return false; }
|
|
virtual bool swap_slots3(int slot1, int slot2, int slot3) { assert(false); return false; }
|
|
virtual size_type physical_offset(int slot, int offset)
|
|
{ return slot * m_files.piece_length() + offset; };
|
|
virtual sha1_hash hash_for_slot(int slot, partial_hash& ph, int piece_size)
|
|
{
|
|
int left = piece_size - ph.offset;
|
|
assert(left >= 0);
|
|
if (left > 0)
|
|
{
|
|
std::vector<char>& data = m_file_data[slot];
|
|
// if there are padding files, those blocks will be considered
|
|
// completed even though they haven't been written to the storage.
|
|
// in this case, just extend the piece buffer to its full size
|
|
// and fill it with zeroes.
|
|
if (data.size() < piece_size) data.resize(piece_size, 0);
|
|
ph.h.update(&data[ph.offset], left);
|
|
}
|
|
return ph.h.final();
|
|
}
|
|
virtual bool release_files() { return false; }
|
|
virtual bool delete_files() { return false; }
|
|
|
|
std::map<int, std::vector<char> > m_file_data;
|
|
file_storage m_files;
|
|
};
|
|
|
|
storage_interface* temp_storage_constructor(
|
|
file_storage const& fs, file_storage const* mapped
|
|
, std::string const& path, file_pool& fp
|
|
, std::vector<boost::uint8_t> const& prio)
|
|
{
|
|
return new temp_storage(fs);
|
|
}
|
|
|
|
magnet links
|
|
============
|
|
|
|
Magnet links are URIs that includes an info-hash, a display name and optionally
|
|
a tracker url. The idea behind magnet links is that an end user can click on a
|
|
link in a browser and have it handled by a bittorrent application, to start a
|
|
download, without any .torrent file.
|
|
|
|
The format of the magnet URI is:
|
|
|
|
**magnet:?xt=urn:btih:** *Base32 encoded info-hash* [ **&dn=** *name of download* ] [ **&tr=** *tracker URL* ]*
|
|
|
|
queuing
|
|
=======
|
|
|
|
libtorrent supports *queuing*. Which means it makes sure that a limited number of
|
|
torrents are being downloaded at any given time, and once a torrent is completely
|
|
downloaded, the next in line is started.
|
|
|
|
Torrents that are *auto managed* are subject to the queuing and the active torrents
|
|
limits. To make a torrent auto managed, set ``auto_managed`` to true when adding the
|
|
torrent (see `async_add_torrent() add_torrent()`_).
|
|
|
|
The limits of the number of downloading and seeding torrents are controlled via
|
|
``active_downloads``, ``active_seeds`` and ``active_limit`` in session_settings_.
|
|
These limits takes non auto managed torrents into account as well. If there are
|
|
more non-auto managed torrents being downloaded than the ``active_downloads``
|
|
setting, any auto managed torrents will be queued until torrents are removed so
|
|
that the number drops below the limit.
|
|
|
|
The default values are 8 active downloads and 5 active seeds.
|
|
|
|
At a regular interval, torrents are checked if there needs to be any re-ordering of
|
|
which torrents are active and which are queued. This interval can be controlled via
|
|
``auto_manage_interval`` in session_settings_. It defaults to every 30 seconds.
|
|
|
|
For queuing to work, resume data needs to be saved and restored for all torrents.
|
|
See `save_resume_data()`_.
|
|
|
|
downloading
|
|
-----------
|
|
|
|
Torrents that are currently being downloaded or incomplete (with bytes still to download)
|
|
are queued. The torrents in the front of the queue are started to be actively downloaded
|
|
and the rest are ordered with regards to their queue position. Any newly added torrent
|
|
is placed at the end of the queue. Once a torrent is removed or turns into a seed, its
|
|
queue position is -1 and all torrents that used to be after it in the queue, decreases their
|
|
position in order to fill the gap.
|
|
|
|
The queue positions are always in a sequence without any gaps.
|
|
|
|
Lower queue position means closer to the front of the queue, and will be started sooner than
|
|
torrents with higher queue positions.
|
|
|
|
To query a torrent for its position in the queue, or change its position, see:
|
|
`queue_position() queue_position_up() queue_position_down() queue_position_top() queue_position_bottom()`_.
|
|
|
|
seeding
|
|
-------
|
|
|
|
Auto managed seeding torrents are rotated, so that all of them are allocated a fair
|
|
amount of seeding. Torrents with fewer completed *seed cycles* are prioritized for
|
|
seeding. A seed cycle is completed when a torrent meets either the share ratio limit
|
|
(uploaded bytes / downloaded bytes), the share time ratio (time seeding / time
|
|
downloaing) or seed time limit (time seeded).
|
|
|
|
The relevant settings to control these limits are ``share_ratio_limit``,
|
|
``seed_time_ratio_limit`` and ``seed_time_limit`` in session_settings_.
|
|
|
|
|
|
fast resume
|
|
===========
|
|
|
|
The fast resume mechanism is a way to remember which pieces are downloaded
|
|
and where they are put between sessions. You can generate fast resume data by
|
|
calling `save_resume_data()`_ on torrent_handle_. You can
|
|
then save this data to disk and use it when resuming the torrent. libtorrent
|
|
will not check the piece hashes then, and rely on the information given in the
|
|
fast-resume data. The fast-resume data also contains information about which
|
|
blocks, in the unfinished pieces, were downloaded, so it will not have to
|
|
start from scratch on the partially downloaded pieces.
|
|
|
|
To use the fast-resume data you simply give it to `async_add_torrent() add_torrent()`_, and it
|
|
will skip the time consuming checks. It may have to do the checking anyway, if
|
|
the fast-resume data is corrupt or doesn't fit the storage for that torrent,
|
|
then it will not trust the fast-resume data and just do the checking.
|
|
|
|
file format
|
|
-----------
|
|
|
|
The file format is a bencoded dictionary containing the following fields:
|
|
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``file-format`` | string: "libtorrent resume file" |
|
|
| | |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``file-version`` | integer: 1 |
|
|
| | |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``info-hash`` | string, the info hash of the torrent this data is saved for. |
|
|
| | |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``blocks per piece`` | integer, the number of blocks per piece. Must be: piece_size |
|
|
| | / (16 * 1024). Clamped to be within the range [1, 256]. It |
|
|
| | is the number of blocks per (normal sized) piece. Usually |
|
|
| | each block is 16 * 1024 bytes in size. But if piece size is |
|
|
| | greater than 4 megabytes, the block size will increase. |
|
|
| | |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``pieces`` | A string with piece flags, one character per piece. |
|
|
| | Bit 1 means we have that piece. |
|
|
| | Bit 2 means we have verified that this piece is correct. |
|
|
| | This only applies when the torrent is in seed_mode. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``slots`` | list of integers. The list maps slots to piece indices. It |
|
|
| | tells which piece is on which slot. If piece index is -2 it |
|
|
| | means it is free, that there's no piece there. If it is -1, |
|
|
| | means the slot isn't allocated on disk yet. The pieces have |
|
|
| | to meet the following requirement: |
|
|
| | |
|
|
| | If there's a slot at the position of the piece index, |
|
|
| | the piece must be located in that slot. |
|
|
| | |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``total_uploaded`` | integer. The number of bytes that have been uploaded in |
|
|
| | total for this torrent. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``total_downloaded`` | integer. The number of bytes that have been downloaded in |
|
|
| | total for this torrent. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``active_time`` | integer. The number of seconds this torrent has been active. |
|
|
| | i.e. not paused. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``seeding_time`` | integer. The number of seconds this torrent has been active |
|
|
| | and seeding. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``num_seeds`` | integer. An estimate of the number of seeds on this torrent |
|
|
| | when the resume data was saved. This is scrape data or based |
|
|
| | on the peer list if scrape data is unavailable. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``num_downloaders`` | integer. An estimate of the number of downloaders on this |
|
|
| | torrent when the resume data was last saved. This is used as |
|
|
| | an initial estimate until we acquire up-to-date scrape info. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``upload_rate_limit`` | integer. In case this torrent has a per-torrent upload rate |
|
|
| | limit, this is that limit. In bytes per second. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``download_rate_limit`` | integer. The download rate limit for this torrent in case |
|
|
| | one is set, in bytes per second. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``max_connections`` | integer. The max number of peer connections this torrent |
|
|
| | may have, if a limit is set. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``max_uploads`` | integer. The max number of unchoked peers this torrent may |
|
|
| | have, if a limit is set. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``seed_mode`` | integer. 1 if the torrent is in seed mode, 0 otherwise. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``file_priority`` | list of integers. One entry per file in the torrent. Each |
|
|
| | entry is the priority of the file with the same index. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``piece_priority`` | string of bytes. Each byte is interpreted as an integer and |
|
|
| | is the priority of that piece. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``auto_managed`` | integer. 1 if the torrent is auto managed, otherwise 0. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``sequential_download`` | integer. 1 if the torrent is in sequential download mode, |
|
|
| | 0 otherwise. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``paused`` | integer. 1 if the torrent is paused, 0 otherwise. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``trackers`` | list of lists of strings. The top level list lists all |
|
|
| | tracker tiers. Each second level list is one tier of |
|
|
| | trackers. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``mapped_files`` | list of strings. If any file in the torrent has been |
|
|
| | renamed, this entry contains a list of all the filenames. |
|
|
| | In the same order as in the torrent file. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``url-list`` | list of strings. List of url-seed URLs used by this torrent. |
|
|
| | The urls are expected to be properly encoded and not contain |
|
|
| | any illegal url characters. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``httpseeds`` | list of strings. List of httpseed URLs used by this torrent. |
|
|
| | The urls are expected to be properly encoded and not contain |
|
|
| | any illegal url characters. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``merkle tree`` | string. In case this torrent is a merkle torrent, this is a |
|
|
| | string containing the entire merkle tree, all nodes, |
|
|
| | including the root and all leaves. The tree is not |
|
|
| | necessarily complete, but complete enough to be able to send |
|
|
| | any piece that we have, indicated by the have bitmask. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``peers`` | list of dictionaries. Each dictionary has the following |
|
|
| | layout: |
|
|
| | |
|
|
| | +----------+-----------------------------------------------+ |
|
|
| | | ``ip`` | string, the ip address of the peer. This is | |
|
|
| | | | not a binary representation of the ip | |
|
|
| | | | address, but the string representation. It | |
|
|
| | | | may be an IPv6 string or an IPv4 string. | |
|
|
| | +----------+-----------------------------------------------+ |
|
|
| | | ``port`` | integer, the listen port of the peer | |
|
|
| | +----------+-----------------------------------------------+ |
|
|
| | |
|
|
| | These are the local peers we were connected to when this |
|
|
| | fast-resume data was saved. |
|
|
| | |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``unfinished`` | list of dictionaries. Each dictionary represents an |
|
|
| | piece, and has the following layout: |
|
|
| | |
|
|
| | +-------------+--------------------------------------------+ |
|
|
| | | ``piece`` | integer, the index of the piece this entry | |
|
|
| | | | refers to. | |
|
|
| | +-------------+--------------------------------------------+ |
|
|
| | | ``bitmask`` | string, a binary bitmask representing the | |
|
|
| | | | blocks that have been downloaded in this | |
|
|
| | | | piece. | |
|
|
| | +-------------+--------------------------------------------+ |
|
|
| | | ``adler32`` | The adler32 checksum of the data in the | |
|
|
| | | | blocks specified by ``bitmask``. | |
|
|
| | | | | |
|
|
| | +-------------+--------------------------------------------+ |
|
|
| | |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``file sizes`` | list where each entry corresponds to a file in the file list |
|
|
| | in the metadata. Each entry has a list of two values, the |
|
|
| | first value is the size of the file in bytes, the second |
|
|
| | is the time stamp when the last time someone wrote to it. |
|
|
| | This information is used to compare with the files on disk. |
|
|
| | All the files must match exactly this information in order |
|
|
| | to consider the resume data as current. Otherwise a full |
|
|
| | re-check is issued. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
| ``allocation`` | The allocation mode for the storage. Can be either ``full`` |
|
|
| | or ``compact``. If this is full, the file sizes and |
|
|
| | timestamps are disregarded. Pieces are assumed not to have |
|
|
| | moved around even if the files have been modified after the |
|
|
| | last resume data checkpoint. |
|
|
+--------------------------+--------------------------------------------------------------+
|
|
|
|
storage allocation
|
|
==================
|
|
|
|
There are two modes in which storage (files on disk) are allocated in libtorrent.
|
|
|
|
1. The traditional *full allocation* mode, where the entire files are filled up with
|
|
zeros before anything is downloaded. libtorrent will look for sparse files support
|
|
in the filesystem that is used for storage, and use sparse files or file system
|
|
zero fill support if present. This means that on NTFS, full allocation mode will
|
|
only allocate storage for the downloaded pieces.
|
|
|
|
2. The *sparse allocation*, sparse files are used, and pieces are downloaded directly
|
|
to where they belong. This is the recommended (and default) mode.
|
|
|
|
In previous versions of libtorrent, a 3rd mode was supported, *compact allocation*.
|
|
Support for this is deprecated and will be removed in future versions of libtorrent.
|
|
It's still described in here for completeness.
|
|
|
|
The allocation mode is selected when a torrent is started. It is passed as an
|
|
argument to ``session::add_torrent()`` (see `async_add_torrent() add_torrent()`_).
|
|
|
|
The decision to use full allocation or compact allocation typically depends on whether
|
|
any files have priority 0 and if the filesystem supports sparse files.
|
|
|
|
sparse allocation
|
|
-----------------
|
|
|
|
On filesystems that supports sparse files, this allocation mode will only use
|
|
as much space as has been downloaded.
|
|
|
|
* It does not require an allocation pass on startup.
|
|
|
|
* It supports skipping files (setting prioirty to 0 to not download).
|
|
|
|
* Fast resume data will remain valid even when file time stamps are out of date.
|
|
|
|
|
|
full allocation
|
|
---------------
|
|
|
|
When a torrent is started in full allocation mode, the disk-io thread
|
|
will make sure that the entire storage is allocated, and fill any gaps with zeros.
|
|
This will be skipped if the filesystem supports sparse files or automatic zero filling.
|
|
It will of course still check for existing pieces and fast resume data. The main
|
|
drawbacks of this mode are:
|
|
|
|
* It may take longer to start the torrent, since it will need to fill the files
|
|
with zeros on some systems. This delay is linearly dependent on the size of
|
|
the download.
|
|
|
|
* The download may occupy unnecessary disk space between download sessions. In case
|
|
sparse files are not supported.
|
|
|
|
* Disk caches usually perform extremely poorly with random access to large files
|
|
and may slow down a download considerably.
|
|
|
|
The benefits of this mode are:
|
|
|
|
* Downloaded pieces are written directly to their final place in the files and the
|
|
total number of disk operations will be fewer and may also play nicer to
|
|
filesystems' file allocation, and reduce fragmentation.
|
|
|
|
* No risk of a download failing because of a full disk during download. Unless
|
|
sparse files are being used.
|
|
|
|
* The fast resume data will be more likely to be usable, regardless of crashes or
|
|
out of date data, since pieces won't move around.
|
|
|
|
* Can be used with prioritizing files to 0.
|
|
|
|
compact allocation
|
|
------------------
|
|
|
|
.. note::
|
|
Note that support for compact allocation is deprecated in libttorrent, and will
|
|
be removed in future versions.
|
|
|
|
The compact allocation will only allocate as much storage as it needs to keep the
|
|
pieces downloaded so far. This means that pieces will be moved around to be placed
|
|
at their final position in the files while downloading (to make sure the completed
|
|
download has all its pieces in the correct place). So, the main drawbacks are:
|
|
|
|
* More disk operations while downloading since pieces are moved around.
|
|
|
|
* Potentially more fragmentation in the filesystem.
|
|
|
|
* Cannot be used while having files with priority 0.
|
|
|
|
The benefits though, are:
|
|
|
|
* No startup delay, since the files don't need allocating.
|
|
|
|
* The download will not use unnecessary disk space.
|
|
|
|
* Disk caches perform much better than in full allocation and raises the download
|
|
speed limit imposed by the disk.
|
|
|
|
* Works well on filesystems that don't support sparse files.
|
|
|
|
The algorithm that is used when allocating pieces and slots isn't very complicated.
|
|
For the interested, a description follows.
|
|
|
|
storing a piece:
|
|
|
|
1. let **A** be a newly downloaded piece, with index **n**.
|
|
2. let **s** be the number of slots allocated in the file we're
|
|
downloading to. (the number of pieces it has room for).
|
|
3. if **n** >= **s** then allocate a new slot and put the piece there.
|
|
4. if **n** < **s** then allocate a new slot, move the data at
|
|
slot **n** to the new slot and put **A** in slot **n**.
|
|
|
|
allocating a new slot:
|
|
|
|
1. if there's an unassigned slot (a slot that doesn't
|
|
contain any piece), return that slot index.
|
|
2. append the new slot at the end of the file (or find an unused slot).
|
|
3. let **i** be the index of newly allocated slot
|
|
4. if we have downloaded piece index **i** already (to slot **j**) then
|
|
|
|
1. move the data at slot **j** to slot **i**.
|
|
2. return slot index **j** as the newly allocated free slot.
|
|
|
|
5. return **i** as the newly allocated slot.
|
|
|
|
|
|
extensions
|
|
==========
|
|
|
|
These extensions all operates within the `extension protocol`__. The
|
|
name of the extension is the name used in the extension-list packets,
|
|
and the payload is the data in the extended message (not counting the
|
|
length-prefix, message-id nor extension-id).
|
|
|
|
__ extension_protocol.html
|
|
|
|
Note that since this protocol relies on one of the reserved bits in the
|
|
handshake, it may be incompatible with future versions of the mainline
|
|
bittorrent client.
|
|
|
|
These are the extensions that are currently implemented.
|
|
|
|
metadata from peers
|
|
-------------------
|
|
|
|
Extension name: "LT_metadata"
|
|
|
|
This extension is deprecated in favor of the more widely supported ``ut_metadata``
|
|
extension, see `BEP 9`_.
|
|
The point with this extension is that you don't have to distribute the
|
|
metadata (.torrent-file) separately. The metadata can be distributed
|
|
through the bittorrent swarm. The only thing you need to download such
|
|
a torrent is the tracker url and the info-hash of the torrent.
|
|
|
|
It works by assuming that the initial seeder has the metadata and that
|
|
the metadata will propagate through the network as more peers join.
|
|
|
|
There are three kinds of messages in the metadata extension. These packets
|
|
are put as payload to the extension message. The three packets are:
|
|
|
|
* request metadata
|
|
* metadata
|
|
* don't have metadata
|
|
|
|
request metadata:
|
|
|
|
+-----------+---------------+----------------------------------------+
|
|
| size | name | description |
|
|
+===========+===============+========================================+
|
|
| uint8_t | msg_type | Determines the kind of message this is |
|
|
| | | 0 means 'request metadata' |
|
|
+-----------+---------------+----------------------------------------+
|
|
| uint8_t | start | The start of the metadata block that |
|
|
| | | is requested. It is given in 256:ths |
|
|
| | | of the total size of the metadata, |
|
|
| | | since the requesting client don't know |
|
|
| | | the size of the metadata. |
|
|
+-----------+---------------+----------------------------------------+
|
|
| uint8_t | size | The size of the metadata block that is |
|
|
| | | requested. This is also given in |
|
|
| | | 256:ths of the total size of the |
|
|
| | | metadata. The size is given as size-1. |
|
|
| | | That means that if this field is set |
|
|
| | | 0, the request wants one 256:th of the |
|
|
| | | metadata. |
|
|
+-----------+---------------+----------------------------------------+
|
|
|
|
metadata:
|
|
|
|
+-----------+---------------+----------------------------------------+
|
|
| size | name | description |
|
|
+===========+===============+========================================+
|
|
| uint8_t | msg_type | 1 means 'metadata' |
|
|
+-----------+---------------+----------------------------------------+
|
|
| int32_t | total_size | The total size of the metadata, given |
|
|
| | | in number of bytes. |
|
|
+-----------+---------------+----------------------------------------+
|
|
| int32_t | offset | The offset of where the metadata block |
|
|
| | | in this message belongs in the final |
|
|
| | | metadata. This is given in bytes. |
|
|
+-----------+---------------+----------------------------------------+
|
|
| uint8_t[] | metadata | The actual metadata block. The size of |
|
|
| | | this part is given implicit by the |
|
|
| | | length prefix in the bittorrent |
|
|
| | | protocol packet. |
|
|
+-----------+---------------+----------------------------------------+
|
|
|
|
Don't have metadata:
|
|
|
|
+-----------+---------------+----------------------------------------+
|
|
| size | name | description |
|
|
+===========+===============+========================================+
|
|
| uint8_t | msg_type | 2 means 'I don't have metadata'. |
|
|
| | | This message is sent as a reply to a |
|
|
| | | metadata request if the the client |
|
|
| | | doesn't have any metadata. |
|
|
+-----------+---------------+----------------------------------------+
|
|
|
|
.. _`BEP 9`: http://bittorrent.org/beps/bep_0009.html
|
|
|
|
dont_have
|
|
---------
|
|
|
|
Extension name: "lt_dont_have"
|
|
|
|
The ``dont_have`` extension message is used to tell peers that the client no longer
|
|
has a specific piece. The extension message should be advertised in the ``m`` dictionary
|
|
as ``lt_dont_have``. The message format mimics the regular ``HAVE`` bittorrent message.
|
|
|
|
Just like all extension messages, the first 2 bytes in the mssage itself are 20 (the
|
|
bittorrent extension message) and the message ID assigned to this extension in the ``m``
|
|
dictionary in the handshake.
|
|
|
|
+-----------+---------------+----------------------------------------+
|
|
| size | name | description |
|
|
+===========+===============+========================================+
|
|
| uint32_t | piece | index of the piece the peer no longer |
|
|
| | | has. |
|
|
+-----------+---------------+----------------------------------------+
|
|
|
|
The length of this message (including the extension message prefix) is
|
|
6 bytes, i.e. one byte longer than the normal ``HAVE`` message, because
|
|
of the extension message wrapping.
|
|
|
|
HTTP seeding
|
|
------------
|
|
|
|
There are two kinds of HTTP seeding. One with that assumes a smart
|
|
(and polite) client and one that assumes a smart server. These
|
|
are specified in `BEP 19`_ and `BEP 17`_ respectively.
|
|
|
|
libtorrent supports both. In the libtorrent source code and API,
|
|
BEP 19 urls are typically referred to as *url seeds* and BEP 17
|
|
urls are typically referred to as *HTTP seeds*.
|
|
|
|
The libtorrent implementation of `BEP 19`_ assumes that, if the URL ends with a slash
|
|
('/'), the filename should be appended to it in order to request pieces from
|
|
that file. The way this works is that if the torrent is a single-file torrent,
|
|
only that filename is appended. If the torrent is a multi-file torrent, the
|
|
torrent's name '/' the file name is appended. This is the same directory
|
|
structure that libtorrent will download torrents into.
|
|
|
|
.. _`BEP 17`: http://bittorrent.org/beps/bep_0017.html
|
|
.. _`BEP 19`: http://bittorrent.org/beps/bep_0019.html
|
|
|
|
piece picker
|
|
============
|
|
|
|
The piece picker in libtorrent has the following features:
|
|
|
|
* rarest first
|
|
* sequential download
|
|
* random pick
|
|
* reverse order picking
|
|
* parole mode
|
|
* prioritize partial pieces
|
|
* prefer whole pieces
|
|
* piece affinity by speed category
|
|
* piece priorities
|
|
|
|
internal representation
|
|
-----------------------
|
|
|
|
It is optimized by, at all times, keeping a list of pieces ordered
|
|
by rarity, randomly shuffled within each rarity class. This list
|
|
is organized as a single vector of contigous memory in RAM, for
|
|
optimal memory locality and to eliminate heap allocations and frees
|
|
when updating rarity of pieces.
|
|
|
|
Expensive events, like a peer joining or leaving, are evaluated
|
|
lazily, since it's cheaper to rebuild the whole list rather than
|
|
updating every single piece in it. This means as long as no blocks
|
|
are picked, peers joining and leaving is no more costly than a single
|
|
peer joining or leaving. Of course the special cases of peers that have
|
|
all or no pieces are optimized to not require rebuilding the list.
|
|
|
|
picker strategy
|
|
---------------
|
|
|
|
The normal mode of the picker is of course *rarest first*, meaning
|
|
pieces that few peers have are preferred to be downloaded over pieces
|
|
that more peers have. This is a fundamental algorithm that is the
|
|
basis of the performance of bittorrent. However, the user may set the
|
|
piece picker into sequential download mode. This mode simply picks
|
|
pieces sequentially, always preferring lower piece indices.
|
|
|
|
When a torrent starts out, picking the rarest pieces means increased
|
|
risk that pieces won't be completed early (since there are only a few
|
|
peers they can be downloaded from), leading to a delay of having any
|
|
piece to offer to other peers. This lack of pieces to trade, delays
|
|
the client from getting started into the normal tit-for-tat mode of
|
|
bittorrent, and will result in a long ramp-up time. The heuristic to
|
|
mitigate this problem is to, for the first few pieces, pick random pieces
|
|
rather than rare pieces. The threshold for when to leave this initial
|
|
picker mode is determined by ``session_settings::initial_picker_threshold``.
|
|
|
|
reverse order
|
|
-------------
|
|
|
|
An orthogonal setting is *reverse order*, which is used for *snubbed*
|
|
peers. Snubbed peers are peers that appear very slow, and might have timed
|
|
out a piece request. The idea behind this is to make all snubbed peers
|
|
more likely to be able to do download blocks from the same piece,
|
|
concentrating slow peers on as few pieces as possible. The reverse order
|
|
means that the most common pieces are picked, instead of the rarest pieces
|
|
(or in the case of sequential download, the last pieces, intead of the first).
|
|
|
|
parole mode
|
|
-----------
|
|
|
|
Peers that have participated in a piece that failed the hash check, may be
|
|
put in *parole mode*. This means we prefer downloading a full piece from this
|
|
peer, in order to distinguish which peer is sending corrupt data. Whether to
|
|
do this is or not is controlled by ``session_settings::use_parole_mode``.
|
|
|
|
In parole mode, the piece picker prefers picking one whole piece at a time for
|
|
a given peer, avoiding picking any blocks from a piece any other peer has
|
|
contributed to (since that would defeat the purpose of parole mode).
|
|
|
|
prioritize partial pieces
|
|
-------------------------
|
|
|
|
This setting determines if partially downloaded or requested pieces should always
|
|
be preferred over other pieces. The benefit of doing this is that the number of
|
|
partial pieces is minimized (and hence the turn-around time for downloading a block
|
|
until it can be uploaded to others is minimized). It also puts less stress on the
|
|
disk cache, since fewer partial pieces need to be kept in the cache. Whether or
|
|
not to enable this is controlled by ``session_settings::prioritize_partial_pieces``.
|
|
|
|
The main benefit of not prioritizing partial pieces is that the rarest first
|
|
algorithm gets to have more influence on which pieces are picked. The picker is
|
|
more likely to truly pick the rarest piece, and hence improving the performance
|
|
of the swarm.
|
|
|
|
This setting is turned on automatically whenever the number of partial pieces
|
|
in the piece picker exceeds the number of peers we're connected to times 1.5.
|
|
This is in order to keep the waste of partial pieces to a minimum, but still
|
|
prefer rarest pieces.
|
|
|
|
prefer whole pieces
|
|
-------------------
|
|
|
|
The *prefer whole pieces* setting makes the piece picker prefer picking entire
|
|
pieces at a time. This is used by web connections (both http seeding
|
|
standards), in order to be able to coalesce the small bittorrent requests
|
|
to larger HTTP requests. This significantly improves performance when
|
|
downloading over HTTP.
|
|
|
|
It is also used by peers that are downloading faster than a certain
|
|
threshold. The main advantage is that these peers will better utilize the
|
|
other peer's disk cache, by requesting all blocks in a single piece, from
|
|
the same peer.
|
|
|
|
This threshold is controlled by ``session_settings::whole_pieces_threshold``.
|
|
|
|
*TODO: piece affinity by speed category*
|
|
*TODO: piece priorities*
|
|
|
|
SSL torrents
|
|
============
|
|
|
|
Torrents may have an SSL root (CA) certificate embedded in them. Such torrents
|
|
are called *SSL torrents*. An SSL torrent talks to all bittorrent peers over SSL.
|
|
The protocols are layered like this::
|
|
|
|
+-----------------------+
|
|
| BitTorrent protocol |
|
|
+-----------------------+
|
|
| SSL |
|
|
+-----------+-----------+
|
|
| TCP | uTP |
|
|
| +-----------+
|
|
| | UDP |
|
|
+-----------+-----------+
|
|
|
|
During the SSL handshake, both peers need to authenticate by providing a certificate
|
|
that is signed by the CA certificate found in the .torrent file. These peer
|
|
certificates are expected to be privided to peers through some other means than
|
|
bittorrent. Typically by a peer generating a certificate request which is sent to
|
|
the publisher of the torrent, and the publisher returning a signed certificate.
|
|
|
|
In libtorrent, `set_ssl_certificate()`_ in torrent_handle_ is used to tell libtorrent where
|
|
to find the peer certificate and the private key for it. When an SSL torrent is loaded,
|
|
the torrent_need_cert_alert_ is posted to remind the user to provide a certificate.
|
|
|
|
A peer connecting to an SSL torrent MUST provide the *SNI* TLS extension (server name
|
|
indication). The server name is the hex encoded info-hash of the torrent to connect to.
|
|
This is required for the client accepting the connection to know which certificate to
|
|
present.
|
|
|
|
SSL connections are accepted on a separate socket from normal bittorrent connections. To
|
|
pick which port the SSL socket should bind to, set ``session_settings::ssl_listen`` to a
|
|
different port. It defaults to port 4433. This setting is only taken into account when the
|
|
normal listen socket is opened (i.e. just changing this setting won't necessarily close
|
|
and re-open the SSL socket). To not listen on an SSL socket at all, set ``ssl_listen`` to 0.
|
|
|
|
This feature is only available if libtorrent is build with openssl support (``TORRENT_USE_OPENSSL``)
|
|
and requires at least openSSL version 1.0, since it needs SNI support.
|
|
|
|
Peer certificates must have at least one *SubjectAltName* field of type dNSName. At least
|
|
one of the fields must *exactly* match the name of the torrent. This is a byte-by-byte comparison,
|
|
the UTF-8 encoding must be identical (i.e. there's no unicode normalization going on). This is
|
|
the recommended way of verifying certificates for HTTPS servers according to `RFC 2818`_. Note
|
|
the difference that for torrents only *dNSName* fields are taken into account (not IP address fields).
|
|
The most specific (i.e. last) *Common Name* field is also taken into account if no *SubjectAltName*
|
|
did not match.
|
|
|
|
If any of these fields contain a single asterisk ("*"), the certificate is considered covering
|
|
any torrent, allowing it to be reused for any torrent.
|
|
|
|
The purpose of matching the torrent name with the fields in the peer certificate is to allow
|
|
a publisher to have a single root certificate for all torrents it distributes, and issue
|
|
separate peer certificates for each torrent. A peer receiving a certificate will not necessarily
|
|
be able to access all torrents published by this root certificate (only if it has a "star cert").
|
|
|
|
.. _`RFC 2818`: http://www.ietf.org/rfc/rfc2818.txt
|
|
|
|
testing
|
|
-------
|
|
|
|
To test incoming SSL connections to an SSL torrent, one can use the following *openssl* command::
|
|
|
|
openssl s_client -cert <peer-certificate>.pem -key <peer-private-key>.pem -CAfile <torrent-cert>.pem -debug -connect 127.0.0.1:4433 -tls1 -servername <info-hash>
|
|
|
|
To create a root certificate, the Distinguished Name (*DN*) is not taken into account
|
|
by bittorrent peers. You still need to specify something, but from libtorrent's point of
|
|
view, it doesn't matter what it is. libtorrent only makes sure the peer certificates are
|
|
signed by the correct root certificate.
|
|
|
|
One way to create the certificates is to use the ``CA.sh`` script that comes with openssl, like thisi (don't forget to enter a common Name for the certificate)::
|
|
|
|
CA.sh -newca
|
|
CA.sh -newreq
|
|
CA.sh -sign
|
|
|
|
The torrent certificate is located in ``./demoCA/private/demoCA/cacert.pem``, this is
|
|
the pem file to include in the .torrent file.
|
|
|
|
The peer's certificate is located in ``./newcert.pem`` and the certificate's
|
|
private key in ``./newkey.pem``.
|
|
|