diff --git a/docs/features.html b/docs/features.html index 15a7e1be1..d153b5f61 100644 --- a/docs/features.html +++ b/docs/features.html @@ -81,7 +81,7 @@ without having to modify libtorrent
  • supports trackerless torrents (using the Mainline kademlia DHT protocol) with some DHT extensions. BEP 5.
  • supports the bittorrent extension protocol. See extensions. BEP 10.
    • -
  • supports the uTorrent metadata transfer protocol (i.e. magnet links).
    • +
  • supports the uTorrent metadata transfer protocol BEP 9 (i.e. magnet links).
  • supports the uTorrent peer exchange protocol (PEX).
  • supports local peer discovery (multicasts for peers on the same local network)
  • multitracker extension support (supports both strict BEP 12 and the @@ -167,6 +167,30 @@ read back into physical memory only to be flushed back out to disk again.

    used to flush multiple cache blocks in a single call.

    On low-memory systems, the disk cache can be disabled altogether or set to smaller limit, to save memory.

    +

    The disk caching algorithm is configurable between 'LRU' and 'largest contiguous'. +The largest contiguous algorithm is the default and flushes the largest contiguous +block of buffers, instead of flushing all blocks belonging to the piece which was +written to least recently.

    +

    For version 0.15 a lot of work went into optimizing the cache algorithm, trying +to increase the cache hit rate and utilization. The graph to the left shows the +memory utilization in 0.14. This cache is a straight forward, fairly naive, implementation. +Every block read will also read all subsequent blocks in that piece into the cache. +Whenever we need more space, the entire oldest piece is evicted from the cache. Caching +writes always takes presedence over the read cache. Whenever a piece is fully downloaded, +it is flushed to disk.

    +disk_buffer_before_optimization.png +disk_buffer.png +

    The left graph shows the problem of evicting entire pieces at a time, and waiting until +an entire piece is downloaded until flushing it. These graphs were generated for a torrent +with fairly large pieces. This means that granularity was poor in 0.14, since it only +dealt with entire pieces. In 0.15, the granularity problem has been fixed by evicting one +block at a time from the read cache. This maximizes the read cache utilization. The write +cache is also flushed when a sufficient number of contiguous blocks have been downloaded +for a piece, which is not tied to the piece size anymore. This way the cache scales a lot +better with piece sizes.

    +

    The graph to the right shows the same download but with the new optimized disk cache +algorithm. It clearly shows an increased utilization, which means higher read hit rates +or smaller caches with maintained hit rate.

    network buffers

    diff --git a/docs/manual.html b/docs/manual.html index c9d16b2c8..a84be146b 100644 --- a/docs/manual.html +++ b/docs/manual.html @@ -44,18 +44,19 @@

    Table of contents

    @@ -372,6 +380,12 @@ class session: public boost::noncopyable int upload_rate_limit() const; void set_download_rate_limit(int bytes_per_second); int download_rate_limit() const; + + void set_local_upload_rate_limit(int bytes_per_second); + int local_upload_rate_limit() const; + void set_local_download_rate_limit(int bytes_per_second); + int local_download_rate_limit() const; + void set_max_uploads(int limit); void set_max_connections(int limit); int max_connections() const; @@ -674,18 +688,39 @@ int download_rate_limit() const;

    set_upload_rate_limit() set the maximum number of bytes allowed to be sent to peers per second. This bandwidth is distributed among all the peers. If -you don't want to limit upload rate, you can set this to -1 (the default). +you don't want to limit upload rate, you can set this to 0 (the default). set_download_rate_limit() works the same way but for download rate instead of upload rate. download_rate_limit() and upload_rate_limit() returns the previously set limits.

    +

    A rate limit of 0 means infinite.

    +

    Upload and download rate limits are not applied to peers on the local network +by default. To change that, see session_settings::ignore_limits_on_local_network.

    -
    -

    set_max_uploads() set_max_connections() max_connections()

    +
    +

    set_local_upload_rate_limit() set_local_download_rate_limit() local_upload_rate_limit() local_download_rate_limit()

    +
    +
    +void set_local_upload_rate_limit(int bytes_per_second);
+void set_local_download_rate_limit(int bytes_per_second);
+int local_upload_rate_limit() const;
+int local_download_rate_limit() const;
+
    +
    +

    These rate limits are only used for local peers (peers within the same subnet as +the client itself) and it is only used when session_settings::ignore_limits_on_local_network +is set to true (which it is by default). These rate limits default to unthrottled, +but can be useful in case you want to treat local peers preferentially, but not +quite unthrottled.

    +

    A rate limit of 0 means infinite.

    +
    +
    +

    set_max_uploads() set_max_connections() max_uploads() max_connections()

     void set_max_uploads(int limit);
 void set_max_connections(int limit);
+int max_uploads() const;
 int max_connections() const;
    @@ -694,7 +729,9 @@ and the number of connections opened. The number of connections is set to a hard minimum of at least two connections per torrent, so if you set a too low connections limit, and open too many torrents, the limit will not be met. The number of uploads is at least one per torrent.

    -

    max_connections() returns the current setting.

    +

    max_uploads() and max_connections() returns the current settings.

    +

    The number of unchoke slots may be ignored. In order to make this setting +take effect, disable session_settings::auto_upload_slots_rate_based.

    num_uploads() num_connections()

    @@ -901,6 +938,7 @@ struct cache_status size_type reads; int cache_size; int read_cache_size; + int total_used_buffers; }; @@ -919,6 +957,9 @@ for the read cache.

    cache_size is the number of 16 KiB blocks currently in the disk cache. This includes both read and write cache.

    read_cache_size is the number of 16KiB blocks in the read cache.

    +

    total_used_buffers is the total number of buffers currently in use. +This includes the read/write disk cache as well as send and receive buffers +used in peer connections.

    get_cache_info()

    @@ -1865,6 +1906,7 @@ struct torrent_handle std::set<std::string> http_seeds() const; void set_ratio(float ratio) const; + int max_uploads() const; void set_max_uploads(int max_uploads) const; void set_max_connections(int max_connections) const; int max_connections() const; @@ -2069,7 +2111,8 @@ This will block all other disk IO, and other torrents download and upload rates drop while copying the file.

    Since disk IO is performed in a separate thread, this operation is also asynchronous. Once the operation completes, the storage_moved_alert is generated, with the new -path as the message.

    +path as the message. If the move fails for some reason, storage_moved_failed_alert +is generated instead, containing the error message.

    rename_file()

    @@ -2456,11 +2499,12 @@ sha1_hash info_hash() const;

    info_hash() returns the info-hash for the torrent.

    -
    -

    set_max_uploads() set_max_connections() max_connections()

    +
    +

    set_max_uploads() max_uploads() set_max_connections() max_connections()

     void set_max_uploads(int max_uploads) const;
+int max_uploads() const;
 void set_max_connections(int max_connections) const;
 int max_connections() const;
    @@ -2471,7 +2515,7 @@ torrent. If you set this to -1, there will be no limit.

    connections are used up, incoming connections may be refused or poor connections may be closed. This must be at least 2. The default is unlimited number of connections. If -1 is given to the function, it means unlimited.

    -

    max_connections() returns the current setting.

    +

    max_uploads() and max_connections() returns the current settings.

    save_resume_data()

    @@ -2568,7 +2612,7 @@ while (num_resume_data > 0) }
    -
    +

    status()

    @@ -2594,9 +2638,9 @@ struct partial_piece_info
 {
         int piece_index;
         int blocks_in_piece;
-        block_info blocks[256];
         enum state_t { none, slow, medium, fast };
         state_t piece_state;
+        block_info* blocks;
 };

    piece_index is the index of the piece in question. blocks_in_piece is the @@ -2614,11 +2658,18 @@ struct block_info enum block_state_t { none, requested, writing, finished }; - tcp::endpoint peer; + void set_peer(tcp::endpoint const& ep); + tcp::endpoint peer() const; + + unsigned bytes_progress:15; + unsigned block_size:15; unsigned state:2; unsigned num_peers:14; }; +

    The blocks field points to an array of blocks_in_piece elements. This pointer is +only valid until the next call to get_download_queue() for any torrent in the same session. +They all share the storaga for the block arrays in their session object.

    The block_info array contains data for each individual block in the piece. Each block has a state (state) which is any of:

      @@ -2630,7 +2681,9 @@ a state (state) which

      The peer field is the ip address of the peer this block was downloaded from. num_peers is the number of peers that is currently requesting this block. Typically this is 0 or 1, but at the end of the torrent blocks may be requested by more peers in parallel to -speed things up.

      +speed things up. +bytes_progress is the number of bytes that have been received for this block, and +block_size is the total number of bytes in this block.

    get_peer_info()

    @@ -3268,12 +3321,39 @@ rates seen on this connection. They are given in bytes per second. This number i reset to 0 on reconnect.

    progress is the progress of the peer.

    -
    -

    session_settings

    -

    You have some control over tracker requests through the session_settings object. You +

    +

    session customization

    +

    You have some control over session configuration through the session_settings object. You create it and fill it with your settings and then use session::set_settings() -to apply them. You have control over proxy and authorization settings and also the user-agent -that will be sent to the tracker. The user-agent is a good way to identify your client.

    +to apply them.

    +

    You have control over proxy and authorization settings and also the user-agent +that will be sent to the tracker. The user-agent will also be used to identify the +client with other peers.

    +
    +

    presets

    +

    The default values of the session settings are set for a regular bittorrent client running +on a desktop system. There are functions that can set the session settings to pre set +settings for other environments. These can be used for the basis, and should be tweaked to +fit your needs better.

    +
    +session_settings min_memory_usage();
+session_settings high_performance_seed();
+
    +

    min_memory_usage returns settings that will use the minimal amount of RAM, at the +potential expense of upload and download performance. It adjusts the socket buffer sizes, +disables the disk cache, lowers the send buffer watermarks so that each connection only has +at most one block in use at any one time. It lowers the outstanding blocks send to the disk +I/O thread so that connections only have one block waiting to be flushed to disk at any given +time. It lowers the max number of peers in the peer list for torrents. It performs multiple +smaller reads when it hashes pieces, instead of reading it all into memory before hashing.

    +

    This configuration is inteded to be the starting point for embedded devices. It will +significantly reduce memory usage.

    +

    high_performance_seed returns settings optimized for a seed box, serving many peers +and that doesn't do any downloading. It has a 128 MB disk cache and has a limit of 400 files +in its file pool. It support fast upload rates by allowing large send buffers.

    +
    +
    +

    session_settings

     struct session_settings
 {
@@ -3315,8 +3395,10 @@ struct session_settings
         bool upnp_ignore_nonrouters;
         int send_buffer_watermark;
         bool auto_upload_slots;
+        bool auto_upload_slots_rate_based;
         bool use_parole_mode;
         int cache_size;
+        int cache_buffer_chunk_size;
         int cache_expiry;
         bool use_read_cache;
         bool disk_io_no_buffer;
@@ -3356,6 +3438,21 @@ struct session_settings
         bool lock_disk_cache;
 
         int max_rejects;
+
+        int recv_socket_buffer_size;
+        int send_socket_buffer_size;
+
+        bool optimize_hashing_for_speed;
+
+        int file_checks_delay_per_block;
+
+        enum disk_cache_algo_t
+        { lru, largest_contiguous };
+
+        disk_cache_algo_t disk_cache_algorithm;
+
+        int read_cache_line_size;
+        int write_cache_line_size;
 };

    user_agent this is the client identification to the tracker. @@ -3503,13 +3600,24 @@ slot is opened. If the upload rate has been saturated for an extended period of time, on upload slot is closed. The number of upload slots will never be less than what has been set by session::set_max_uploads(). To query the current number of upload slots, see session_status::allowed_upload_slots.

    +

    When auto_upload_slots_rate_based is set, and auto_upload_slots is set, +the max upload slots setting is ignored and decided completely automatically. +This algorithm is designed to prevent the peer from spreading its upload +capacity too thin.

    use_parole_mode specifies if parole mode should be used. Parole mode means that peers that participate in pieces that fail the hash check are put in a mode where they are only allowed to download whole pieces. If the whole piece a peer in parole mode fails the hash check, it is banned. If a peer participates in a piece that passes the hash check, it is taken out of parole mode.

    -

    cache_size is the disk write cache. It is specified in units of 16 KiB blocks. -It defaults to 512 (= 8 MB).

    +

    cache_size is the disk write and read cache. It is specified in units of +16 KiB blocks. It defaults to 1024 (= 16 MB). Buffers that are part of a peer's +send or receive buffer also count against this limit. Send and receive buffers +will never be denied to be allocated, but they will cause the actual cached blocks +to be flushed or evicted.

    +

    Disk buffers are allocated using a pool allocator, the number of blocks that +are allocated at a time when the pool needs to grow can be specified in +cache_buffer_chunk_size. This defaults to 16 blocks. Lower numbers +saves memory at the expense of more heap allocations. It must be at least 1.

    cache_expiry is the number of seconds from the last cached write to a piece in the write cache, to when it's forcefully flushed to disk. Default is 60 second.

    use_read_cache, is set to true (default), the disk cache is also used to @@ -3577,7 +3685,12 @@ known peers. These peers are not necessarily connected, so this number should be much greater than the maximum number of connected peers. Peers are evicted from the cache when the list grows passed 90% of this limit, and once the size hits the limit, peers are no longer -added to the list.

    +added to the list. If this limit is set to 0, there is no limit on +how many peers we'll keep in the peer list.

    +

    max_paused_peerlist_size is the max peer list size used for torrents +that are paused. This default to the same as max_peerlist_size, but +can be used to save memory for paused torrents, since it's not as +important for them to keep a large peer list.

    min_announce_interval is the minimum allowed announce interval for a tracker. This is specified in seconds, defaults to 5 minutes and is used as a sanity check on what is returned from a tracker. It @@ -3627,6 +3740,42 @@ being swapped out.

    max_rejects is the number of piece requests we will reject in a row while a peer is choked before the peer is considered abusive and is disconnected.

    +

    recv_socket_buffer_size and send_socket_buffer_size specifies +the buffer sizes set on peer sockets. 0 (which is the default) means +the OS default (i.e. don't change the buffer sizes). The socket buffer +sizes are changed using setsockopt() with SOL_SOCKET/SO_RCVBUF and +SO_SNDBUFFER.

    +

    optimize_hashing_for_speed chooses between two ways of reading back +piece data from disk when its complete and needs to be verified against +the piece hash. This happens if some blocks were flushed to the disk +out of order. Everything that is flushed in order is hashed as it goes +along. Optimizing for speed will allocate space to fit all the the +remaingin, unhashed, part of the piece, reads the data into it in a single +call and hashes it. This is the default. If optimizing_hashing_for_speed +is false, a single block will be allocated (16 kB), and the unhashed parts +of the piece are read, one at a time, and hashed in this single block. This +is appropriate on systems that are memory constrained.

    +

    file_checks_delay_per_block is the number of milliseconds to sleep +in between disk read operations when checking torrents. This defaults +to 0, but can be set to higher numbers to slow down the rate at which +data is read from the disk while checking. This may be useful for +background tasks that doesn't matter if they take a bit longer, as long +as they leave disk I/O time for other processes.

    +

    disk_cache_algorithm tells the disk I/O thread which cache flush +algorithm to use. The default (and original) algorithm is LRU. This +flushes the entire piece, in the write cache, that was least recently +written to. This is specified by the session_settings::lru enum +value. session_settings::largest_contiguous will flush the largest +sequences of contiguous blocks from the write cache, regarless of the +piece's last use time.

    +

    read_cache_line_size is the number of blocks to read into the read +cache when a read cache miss occurs. Setting this to 0 is essentially +the same thing as disabling read cache. The number of blocks read +into the read cache is always capped by the piece boundry.

    +

    When a piece in the write cache has write_cache_line_size contiguous +blocks in it, they will be flushed. Setting this to 1 effectively +disables the write cache.

    +

    pe_settings

    @@ -3774,7 +3923,7 @@ public: }; -
    +

    ip_filter()

    @@ -4403,6 +4552,36 @@ struct file_error_alert: torrent_alert
 };
    +
    +

    file_renamed_alert

    +

    This is posted as a response to a torrent_handle::rename_file call, if the rename +operation succeeds.

    +
    +struct file_renamed_alert: torrent_alert
+{
+        // ...
+        std::string name;
+        int index;
+};
+
    +

    The index member refers to the index of the file that was renamed, +name is the new name of the file.

    +
    +
    +

    file_rename_failed_alert

    +

    This is posted as a response to a torrent_handle::rename_file call, if the rename +operation failed.

    +
    +struct file_rename_failed_alert: torrent_alert
+{
+        // ...
+        int index;
+        error_code error;
+};
+
    +

    The index member refers to the index of the file that was supposed to be renamed, +error is the error code returned from the filesystem.

    +

    tracker_announce_alert

    This alert is generated each time a tracker announce is sent (or attempted to be sent). @@ -4691,6 +4870,18 @@ struct storage_moved_alert: torrent_alert };

    +
    +

    storage_moved_failed_alert

    +

    The storage_moved_failed_alert is generated when an attempt to move the storage +(via torrent_handle::move_storage()) fails.

    +
    +struct storage_moved_failed_alert: torrent_alert
+{
+        // ...
+        error_code error;
+};
+
    +

    torrent_paused_alert

    This alert is generated as a response to a torrent_handle::pause request. It is @@ -4942,12 +5133,27 @@ session

    storage_interface

    The storage interface is a pure virtual class that can be implemented to -change the behavior of the actual file storage. The interface looks like -this:

    +customize how and where data for a torrent is stored. The default storage +implementation uses regular files in the filesystem, mapping the files in the +torrent in the way one would assume a torrent is saved to disk. Implementing +your own storage interface makes it possible to store all data in RAM, or in +some optimized order on disk (the order the pieces are received for instance), +or saving multifile torrents in a single file in order to be able to take +advantage of optimized disk-I/O.

    +

    It is also possible to write a thin class that uses the default storage but +modifies some particular behavior, for instance encrypting the data before +it's written to disk, and decrypting it when it's read again.

    +

    The storage interface is based on slots, each slot is 'piece_size' number +of bytes. All access is done by writing and reading whole or partial +slots. One slot is one piece in the torrent, but the data in the slot +does not necessarily correspond to the piece with the same index (in +compact allocation mode it won't).

    +

    The interface looks like this:

     struct storage_interface
 {
         virtual bool initialize(bool allocate_files) = 0;
+        virtual bool has_any_file() = 0;
         virtual int readv(file::iovec_t const* bufs, int slot, int offset, int num_bufs) = 0;
         virtual int writev(file::iovec_t const* bufs, int slot, int offset, int num_bufs) = 0;
         virtual int sparse_end(int start) const;
@@ -4957,7 +5163,6 @@ struct storage_interface
         virtual bool move_slot(int src_slot, int dst_slot) = 0;
         virtual bool swap_slots(int slot1, int slot2) = 0;
         virtual bool swap_slots3(int slot1, int slot2, int slot3) = 0;
-        virtual sha1_hash hash_for_slot(int slot, partial_hash& h, int piece_size) = 0;
         virtual bool rename_file(int file, std::string const& new_name) = 0;
         virtual bool release_files() = 0;
         virtual bool delete_files() = 0;
@@ -4984,6 +5189,17 @@ will create directories and empty files at this point. If ftruncate all files to their target size.
    
 
    Returning true indicates an error occurred.
    +
    +

    has_any_file()

    +
    +
    +virtual bool has_any_file() = 0;
+
    +
    +

    This function is called when first checking (or re-checking) the storage for a torrent. +It should return true if any of the files that is used in this storage exists on disk. +If so, the storage will be checked for existing pieces before starting the download.

    +

    readv() writev()

    @@ -5025,7 +5241,7 @@ int sparse_end(int start) const; region). The purpose of this is to skip parts of files that can be known to contain zeros when checking files.

    -
    +

    move_storage()

    @@ -5102,29 +5318,7 @@ should be moved to slot1This is only used in compact mode.
    
 
    Returning true indicates an error occurred.
    -
    -

    hash_for_slot()

    -
    -
    -sha1_hash hash_for_slot(int slot, partial_hash& h, int piece_size) = 0;
-
    -
    -

    The function should read the remaining bytes of the slot and hash it with the -sha-1 state in partion_hash. The partial_hash struct looks like this:

    -
    -struct partial_hash
-{
-        partial_hash();
-        int offset;
-        hasher h;
-};
-
    -

    offset is the number of bytes in the slot that has already been hashed, and -h is the sha-1 state of that hash. piece_size is the size of the piece -that is stored in the given slot.

    -

    The function should return the hash of the piece stored in the slot.

    -
    -
    +

    rename_file()