libtorrent manual
+| Author: | +Arvid Norberg, arvid@rasterbar.com |
|---|---|
| Version: | +0.15.0 |
Table of contents
+ +tuning libtorrent
+libtorrent expose most constants used in the bittorrent engine for +customization through the session_settings. This makes it possible to +test and tweak the parameters for certain algorithms to make a client +that fits a wide range of needs. From low memory embedded devices to +servers seeding thousands of torrents. The default settings in libtorrent +are tuned for an end-user bittorrent client running on a normal desktop +computer.
+This document describes techniques to benchmark libtorrent performance +and how parameters are likely to affect it.
+reducing memory footprint
+These are things you can do to reduce the memory footprint of libtorrent. You get +some of this by basing your default session_settings on the min_memory_usage() +setting preset function.
+Keep in mind that lowering memory usage will affect performance, always profile +and benchmark your settings to determine if it's worth the trade-off.
+The typical buffer usage of libtorrent, for a single download, with the cache +size set to 256 blocks (256 * 16 kiB = 4 MiB) is:
++read cache: 149.5 (2392 kiB) +write cache: 89.5 (1432 kiB) +receive buffers: 6.5 (104 kiB) +send buffers: 4 (64 kiB) +hash temp: 0.5 (8 kiB) ++
The receive buffers is proportional to the number of connections we make, and is +limited by the total number of connections in the session (default is 200).
+The send buffers is proportional to the number of upload slots that are allowed +in the session. The default is auto configured based on the observed upload rate.
+The read and write cache can be controlled (see section below).
+The "hash temp" entry size depends on whether or not hashing is optimized for +speed or memory usage. In this test run it was optimized for memory usage.
+disable disk cache
+The bulk of the memory libtorrent will use is used for the disk cache. To save +the absolute most amount of memory, you can disable the the cache by setting +session_settings::cache_size to 0. You might want to consider using the cache +but just disable caching read operations. You do this by settings +session_settings::use_read_cache to false. This is the main factor in how much +memory will be used by the client. Keep in mind that you will degrade performance +by disabling the cache. You should benchmark the disk access in order to make an +informed trade-off.
+remove torrents
+Torrents that have been added to libtorrent will inevitably use up memory, event +when it's paused. A paused torrent will not use any peer connection objects or +any send or receive buffers though. Any added torrent holds the entire .torrent +file in memory, it also remembers the entire list of peers that it's heard about +(which can be fairly long unless it's capped). It also retains information about +which blocks and pieces we have on disk, which can be significant for torrents +with many pieces.
+If you need to minimize the memory footprint, consider removing torrents from +the session rather than pausing them. This will likely only make a difference +when you have a very large number of torrents in a session.
+The downside of removing them is that they will no longer be auto-managed. Paused +auto managed torrents are scraped periodically, to determine which torrents are +in the greatest need of seeding, and libtorrent will prioritize to seed those.
+socket buffer sizes
+You can make libtorrent explicitly set the kernel buffer sizes of all its peer +sockets. If you set this to a low number, you may see reduced throughput, especially +for high latency connections. It is however an oportunity to save memory per +connection, and might be worth considering if you have a very large number of +peer connections. This memory will not be visible in your process, this sets +the amount of kernel memory is used for your sockets.
+Change this by setting session_settings::recv_socket_buffer_size and +session_settings::send_socket_buffer_size.
+peer list size
+The default maximum for the peer list is 4000 peers. For IPv4 peers, each peer +entry uses 32 bytes, which ends up using 128 kB per torrent. If seeding 4 popular +torrents, the peer lists alone uses about half a megabyte.
+The default limit is the same for paused torrents as well, so if you have a +large number of paused torrents (that are popular) it will be even more +significant.
+If you're short of memory, you should consider lowering the limit. 500 is probably +enough. You can do this by setting session_settings::max_peerlist_size to +the max number of peers you want in the torrent's peer list.
+You should also lower the same limit but for paused torrents. It might even make sense +to set that even lower, since you only need a few peers to start up while waiting +for the tracker and DHT to give you fresh ones. The max peer list size for paused +torrents is set by session_settings::max_paused_peerlist_size.
+The drawback of lowering this number is that if you end up in a position where +the tracker is down for an extended period of time, your only hope of finding live +peers is to go through your list of all peers you've ever seen. Having a large +peer list will also help increase performance when starting up, since the torrent +can start connecting to peers in parallel with connecting to the tracker.
+send buffer watermark
+The send buffer watermark controls when libtorrent will ask the disk I/O thread +to read blocks from disk, and append it to a peer's send buffer.
+When the send buffer has fewer than or equal number of bytes as +session_settings::send_buffer_watermark, the peer will ask the disk I/O thread +for more data to send. The trade-off here is between wasting memory by having too +much data in the send buffer, and hurting send rate by starving out the socket, +waiting for the disk read operation to complete.
+If your main objective is memory usage and you're not concerned about being able +to achieve high send rates, you can set the watermark to 9 bytes. This will guarantee +that no more than a single (16 kiB) block will be on the send buffer at a time, for +all peers. This is the least amount of memory possible for the send buffer.
+You should benchmark your max send rate when adjusting this setting. If you have +a very fast disk, you are less likely see a performance hit.
+optimize hashing for memory usage
+When libtorrent is doing hash checks of a file, or when it re-reads a piece that +was just completed to verify its hash, there are two options. The default one +is optimized for speed, which allocates buffers for the entire piece, reads in +the whole piece in one read call, then hashes it.
+The second option is to optimize for memory usage instead, where a single buffer +is allocated, and the piece is read one block at a time, hashing it as each +block is read from the file. For low memory environments, this latter approach +is recommended. Change this by settings session_settings::optimize_hashing_for_speed +to false. This will significantly reduce peak memory usage, especially for +torrents with very large pieces.
+reduce executable size
+Compilers generally add a significant number of bytes to executables that make use +of C++ exceptions. By disabling exceptions (-fno-exceptions on GCC), you can +reduce the executable size with up to 45%.
+Also make sure to optimize for size when compiling.
+play nice with the disk
+When checking a torrent, libtorrent will try to read as fast as possible from the disk. +The only thing that might hold it back is a CPU that is slow at calculating SHA-1 hashes, +but typically the file checking is limited by disk read speed. Most operating systems +today do not prioritize disk access based on the importance of the operation, this means +that checking a torrent might delay other disk accesses, such as virtual memory swapping +or just loading file by other (interactive) applications.
+In order to play nicer with the disk, and leave some spare time for it to service other +processes that might be of higher importance to the end-user, you can introduce a sleep +between the disc accesses. This is a direct tradeoff between how fast you can check a +torrent and how soft you will hit the disk.
+You control this by setting the session_settings::file_checks_delay_per_block to greater +than zero. This number is the number of milliseconds to sleep between each read of 16 kiB.
+The sleeps are not necessarily in between each 16 kiB block (it might be read in larger chunks), +but the number will be multiplied by the number of blocks that were read, to maintain the +same semantics.
+