libtorrent

sourceforge page mailing list

Contents

introduction

libtorrent is a C++ library that aims to be a good alternative to all the other bittorrent implementations around. It is a library and not a full featured client, although it comes with a working example client.

The main goals of libtorrent are:

  • to be cpu efficient
  • to be memory efficient
  • to be very easy to use

libtorrent is not finished. It is an ongoing project (including this documentation). The current state includes the following features:

  • multitracker extension support (as described by TheShadow)
  • serves multiple torrents on a single port and a single thread
  • supports http proxies and proxy authentication
  • gzipped tracker-responses
  • piece picking on block-level (as opposed to piece-level) like in Azureus
  • queues torrents for file check, instead of checking all of them in parallel.
  • uses separate threads for checking files and for main downloader, with a fool-proof thread-safe library interface. (i.e. There's no way for the user to cause a deadlock).
  • can limit the upload bandwidth usage

Functions that are yet to be implemented:

  • optimistic unchoke
  • Snubbing
  • end game mode
  • new allocation model
  • fast resume
  • file-level piece priority
  • a good upload speed cap

libtorrent is portable at least among windows, macosx, and UNIX-systems. It uses boost.thread, boost.filesystem boost.date_time and various other boost libraries and zlib.

libtorrent has been successfully compiled and tested on:

  • Cygwin GCC 3.3.1
  • Windows 2000 vc7.1
  • Linux x86 (debian) GCC 3.0

building

To build libtorrent you need boost and bjam installed. Then you can use bjam to build libtorrent.

To make bjam work, you need to set the environment variable BOOST_ROOT to the path where boost is installed (e.g. c:boost_1_30_2 on windows). Then you can just run bjam in the libtorrent directory.

The Jamfile doesn't work yet. On unix-systems you can use the makefile however. You first have to build boost.thread and boost.filesystem. You do this by, in the directory 'boost-1.30.2/tools/build/jam_src' run the build script ./build.sh. This should produce at least one folder with the 'bin' prefix (and the rest of the name describes your platform). Put the files in that folder somewhere in your path.

You can then invoke bjam in the directories 'boost-1.30.2/libs/thread/build' and 'boost-1.30.2/libs/filesystem/build'. That will produce the needed libraries. Put these libraries in the libtorrent root directory. You then have to modify the makefile to use you prefered compiler and to have the correct path to your boost istallation.

Then the makefile should be able to do the rest.

When building (with boost 1.30.2) on linux and solaris however, I found that I had to make the following modifications to the boost.date-time library. In the file: 'boost-1.30.2/boost/date_time/gregorian_calendar.hpp' line 59. Prepend 'boost/date_time/' to the include path.

And the second modification was in the file: 'boost-1.30.2/boost/date_time/microsec_time_clock.hpp' add the following include at the top of the file:

#include "boost/cstdint.hpp"

In developer studio, you may have to set the compiler options "force conformance in for loop scope" and "treat wchar_t as built-in type" to Yes.

TODO: more detailed build instructions.

using

The interface of libtorrent consists of a few classes. The main class is the session, it contains the main loop that serves all torrents.

session

The session class has the following synopsis:

class session: public boost::noncopyable
{
        session(int listen_port, const std::string& fingerprint = std::string());

        torrent_handle add_torrent(const torrent_info& t, const std::string& save_path);
        void remove_torrent(const torrent_handle& h);

        void set_http_settings(const http_settings& settings);
        void set_upload_rate_limit(int bytes_per_second);
};

Once it's created, it will spawn the main thread that will do all the work. The main thread will be idle as long it doesn't have any torrents to participate in. You add torrents through the add_torrent()-function where you give an object representing the information found in the torrent file and the path where you want to save the files. The save_path will be prepended to the directory- structure in the torrent-file.

remove_torrent() will close all peer connections associated with the torrent and tell the tracker that we've stopped participating in the swarm.

If the torrent you are trying to add already exists in the session (is either queued for checking, being checked or downloading) add_torrent() will throw duplicate_torrent which derives from std::exception.

fingerprint is a short string that will be used in the peer_id to identify the client. If the string is longer than 7 characters it will be trimmed down to 7 characters. The default is an empty string.

set_upload_rate_limit() set the maximum number of bytes allowed to be sent to peers per second. This bandwidth is distributed among all the peers. If you don't want to limit upload rate, you can set this to -1 (the default).

The destructor of session will notify all trackers that our torrents has been shut down. If some trackers are down, they will timout. All this before the destructor of session returns. So, it's adviced that any kind of interface (such as windows) are closed before destructing the sessoin object. Because it can take a few second for it to finish. The timeout can be set with set_http_settings().

How to parse a torrent file and create a torrent_info object is described below.

The torrent_handle returned by add_torrent can be used to retrieve information about the torrent's progress, its peers etc. It is also used to abort a torrent.

The constructor takes a listen port as argument, if the given port is busy it will increase the port number by one and try again. If it still fails it will continue increasing the port number until it succeeds or has failed 9 ports. This will change in the future to give more control of the listen-port.

parsing torrent files

The torrent files are bencoded. There are two functions in libtorrent that can encode and decode bencoded data. They are:

template<class InIt> entry bdecode(InIt start, InIt end);
template<class OutIt> void bencode(OutIt out, const entry& e);

The entry class is the internal representation of the bencoded data and it can be used to retreive information, an entry can also be build by the program and given to bencode() to encode it into the OutIt iterator.

The OutIt and InIt are iterators (InputIterator_ and OutputIterator_ respectively). They are templates and are usually instantiated as ostream_iterator_, back_insert_iterator_ or istream_iterator_. These functions will assume that the iterator refers to a character (char). So, if you want to encode entry e into a buffer in memory, you can do it like this:

std::vector<char> buffer;
bencode(std::back_insert_iterator<std::vector<char> >(buf), e);

If you want to decode a torrent file from a buffer in memory, you can do it like this:

std::vector<char> buffer;
// ...
entry e = bdecode(buf.begin(), buf.end());

Or, if you have a raw char buffer:

const char* buf;
// ...
entry e = bdecode(buf, buf + data_size);

Now we just need to know how to retrieve information from the entry.

entry

The entry class represents one node in a bencoded hierarchy. It works as a variant type, it can be either a list, a dictionary (std::map), an integer or a string. This is its synopsis:

class entry
{
public:

        typedef std::map<std::string, entry> dictionary_type;
        typedef std::string string_type;
        typedef std::vector<entry> list_type;
        typedef implementation-defined integer_type;

        enum data_type
        {
                int_t,
                string_t,
                list_t,
                dictionary_t,
                undefined_t
        };

        data_type type() const;

        entry();
        entry(data_type t);
        entry(const entry& e);

        void operator=(const entry& e);

        integer_type& integer()
        const integer_type& integer() const;
        string_type& string();
        const string_type& string() const;
        list_type& list();
        const list_type& list() const;
        dictionary_type& dict();
        const dictionary_type& dict() const;

        void print(std::ostream& os, int indent = 0) const;
};

The integer(), string(), list() and dict() functions are accessorts that return the respecive type. If the entry object isn't of the type you request, the accessor will throw type_error (which derives from std::runtime_error). You can ask an entry for its type through the type() function.

The print() function is there for debug purposes only.

If you want to create an entry you give it the type you want it to have in its constructor, and then use one of the non-const accessors to get a reference which you then can assign the value you want it to have.

The typical code to get info from a torrent file will then look like this:

entry torrent_file;
// ...

const entry::dictionary_type& dict = torrent_file.dict();
entry::dictionary_type::const_iterator i;
i = dict.find("announce");
if (i != dict.end())
{
        std::string tracker_url= i->second.string();
        std::cout << tracker_url << "\n";
}

To make it easier to extract information from a torren file, the class torrent_info exists.

torrent_info

The torrent_info has the following synopsis:

class torrent_info
{
public:

        torrent_info(const entry& torrent_file)

        typedef std::vector>file>::const_iterator file_iterator;
        typedef std::vector<file>::const_reverse_iterator reverse_file_iterator;

        file_iterator begin_files() const;
        file_iterator end_files() const;
        reverse_file_iterator rbegin_files() const;
        reverse_file_iterator rend_files() const;

        std::size_t num_files() const;
        const file& file_at(int index) const;

        const std::vector<announce_entry>& trackers() const;

        int prioritize_tracker(int index);

        entry::integer_type total_size() const;
        entry::integer_type piece_length() const;
        std::size_t num_pieces() const;
        const sha1_hash& info_hash() const;
        const std::stirng& name() const;
        const std::string& comment() const;
        boost::posiz_time::ptime creation_date() const;


        void print(std::ostream& os) const;

        entry::integer_type piece_size(unsigned int index) const;
        const sha1_hash& hash_for_piece(unsigned int index) const;
};

This class will need some explanation. First of all, to get a list of all files in the torrent, you can use begin_files(), end_files(), rbegin_files() and rend_files(). These will give you standard vector iterators with the type file.

struct file
{
        std::string path;
        std::string filename;
        entry::integer_type size;
};

If you need index-access to files you can use the num_files() and file_at() to access files using indices.

The print() function is there for debug purposes only. It will print the info from the torrent file to the given outstream.

name() returns the name of the torrent.

The trackers() function will return a sorted vector of announce_entry. Each announce entry contains a string, which is the tracker url, and a tier index. The tier index is the high-level priority. No matter which trackers that works or not, the ones with lower tier will always be tried before the one with higher tier number.

struct announce_entry
{
        std::string url;
        int tier;
};

The prioritize_tracker() is used internally to move a tracker to the front of its tier group. i.e. It will never be moved pass a tracker with a different tier number. For more information about how multiple trackers are dealt with, see the specification.

total_size(), piece_length() and num_pieces() returns the total number of bytes the torrent-file represents (all the files in it), the number of byte for each piece and the total number of pieces, respectively. The difference between piece_size() and piece_length() is that piece_size() takes the piece index as argument and gives you the exact size of that piece. It will always be the same as piece_length() except in the case of the last piece, which may be smaller.

hash_for_piece() takes a piece-index and returns the 20-bytes sha1-hash for that piece and info_hash() returns the 20-bytes sha1-hash for the info-section of the torrent file. For more information on the sha1_hash, see the big_number class.

comment() returns the comment associated with the torrent. If there's no comment, it will return an empty string. creation_date() returns a boost::posix_time::ptime_ object, representing the time when this torrent file was created. If there's no timestamp in the torrent file, this will return a date of january 1:st 1970.

torrent_handle

You will usually have to store your torrent handles somewhere, since it's the object through which you retrieve infromation about the torrent and aborts the torrent. Its declaration looks like this:

struct torrent_handle
{
        torrent_handle();

        torrent_status status() const;
        void get_download_queue(std::vector<partial_piece_info>& queue);
        void get_peer_info(std::vector<peer_info>& v);
};

The default constructor will initialize the handle to an invalid state. Which means you cannot perform any operation on it, unless you first assign it a valid handle. If you try to perform any operation they will simply return.

status()

status() will return a structure with information about the status of this torrent. It contains the following fields:

struct torrent_status
{
        enum state_t
        {
                invalid_handle,
                queued_for_checking,
                checking_files,
                downloading,
                seeding
        };

        state_t state;
        float progress;
        boost::posix_time::time_duration next_announce;
        std::size_t total_download;
        std::size_t total_upload;
        std::vector<bool> pieces;
        std::size_t total_done;
};

progress is a value in the range [0, 1], that represents the progress of the torrent's current task. It may be checking files or downloading. The torrent's current task is in the state member, it will be one of the following:

invalid_handle This will be the state if you called status on an uninitialized handle (a handle that was constructed with the default constructor).
queued_for_checking The torrent is in the queue for being checked. But there currently is another torrent that are being checked. This torrent will wait for its turn.
checking_files The torrent has not started its download yet, and is currently checking existing files.
downloading The torrent is being downloaded. This is the state most torrents will be in most of the time. The progress meter will tell how much of the files that has been downloaded.
seeding In this state the torrent has finished downloading and is a pure seeder.

next_announce is the time until the torrent will announce itself to the tracker.

total_download and total_upload is the number of bytes downloaded and uploaded to all peers, accumulated, this session only.

pieces is the bitmask that representw which pieces we have (set to true) and the pieces we don't have.

total_done is the total number of bytes of the file(s) that we have.

get_download_queue()

get_download_queue() takes a non-const reference to a vector which it will fill information about pieces that are partially downloaded or not downloaded at all but partially requested. The entry in the vector (partial_piece_info) looks like this:

struct partial_piece_info
{
        enum { max_blocks_per_piece };
        int piece_index;
        int blocks_in_piece;
        std::bitset<max_blocks_per_piece> requested_blocks;
        std::bitset<max_blocks_per_piece> finished_blocks;
        peer_id peer[max_blocks_per_piece];
        int num_downloads[max_blocks_per_piece];
};

piece_index is the index of the piece in question. blocks_in_piece is the number of blocks in this particular piece. This number will be the same for most pieces, but the last piece may have fewer blocks than the standard pieces.

requested_blocks is a bitset with one bit per block in the piece. If a bit is set, it means that that block has been requested, but not necessarily fully downloaded yet. To know from whom the block has been requested, have a look in the peer array. The bit-index in the requested_blocks and finished_blocks correspons to the array-index into peers and num_downloads. The array of peers is contains the id of the peer the piece was requested from. If a piece hasn't been requested (the bit in requested_blocks is not set) the peer array entry will be undefined.

The finished_blocks is a bitset where each bit says if the block is fully downloaded or not. And the num_downloads array says how many times that block has been downloaded. When a piece fails a hash verification, single blocks may be redownloaded to see if the hash teast may pass then.

get_peer_info()

get_peer_info() takes a reference to a vector that will be cleared and filled with one entry for each peer connected to this torrent. Each entry contains information about that particular peer. It contains the following information:

struct peer_info
{
        enum
        {
                interesting = 0x1,
                choked = 0x2,
                remote_interested = 0x4,
                remote_choked = 0x8
        };
        unsigned int flags;
        address ip;
        float up_speed;
        float down_speed;
        unsigned int total_download;
        unsigned int total_upload;
        peer_id id;
        std::vector<bool> pieces;
        int upload_limit;
};

The flags attribute tells you in which state the peer is. It is set to any combination of the four enums above. Where interesting means that we are interested in pieces from this peer. choked means that we have choked this peer. remote_interested and remote_choked means the same thing but that the peer is interested in pieces from us and the peer has choked us.

The ip field is the IP-address to this peer. Its type is a wrapper around the actual address and the port number. See address class.

up_speed and down_speed is the current upload and download speed we have to and from this peer. These figures are updated aproximately once every second.

total_download and total_upload are the total number of bytes downloaded from and uploaded to this peer. These numbers do not include the protocol chatter, but only the payload data.

id is the peer's id as used in the bit torrent protocol. This id can be used to extract 'fingerprints' from the peer. Sometimes it can tell you which client the peer is using.

pieces is a vector of booleans that has as many entries as there are pieces in the torrent. Each boolean tells you if the peer has that piece (if it's set to true) or if the peer miss that piece (set to false).

upload_limit is the number of bytes per second we are allowed to send to this peer every second. It may be -1 if there's no limit.

address

The address class represents a name of a network endpoint (usually referred to as IP-address) and a port number. This is the same thing as a sockaddr_in would contain. Its declaration looks like this:

class address
{
public:
        address();
        address(
                  unsigned char a
                , unsigned char b
                , unsigned char c
                , unsigned char d
                , unsigned short  port);
        address(unsigned int addr, unsigned short port);
        address(const std::string& addr, unsigned short port);
        address(const address& a);
        ~address();

        std::string as_string() const;
        unsigned int ip() const;
        unsigned short port() const;

        bool operator<(const address& a) const;
        bool operator!=(const address& a) const;
        bool operator==(const address& a) const;
};

It is less-than comparable to make it possible to use it as a key in a map. as_string() may block while it does the DNS lookup, it returns a string that points to the address represented by the object.

ip() will return the 32-bit ip-address as an integer. port() returns the port number.

http_settings

You have some control over tracker requests through the http_settings object. You create it and fill it with your settings and the use session::set_http_settings() to apply them. You have control over proxy and authorization settings and also the user-agent that will be sent to the tracker. The user-agent is a good way to identify your client.

struct http_settings
{
        http_settings();
        std::string proxy_ip;
        int proxy_port;
        std::string proxy_login;
        std::string proxy_password;
        std::string user_agent;
        int tracker_timeout;
        int tracker_maximum_response_length;
};

tracker_timeout is the number of seconds the tracker connection will wait until it considers the tracker to have timed-out. Default value is 10 seconds.

tracker_maximum_response_length is the maximum number of bytes in a tracker response. If a response size passes this number it will be rejected and the connection will be closed. On gzipped responses this size is measured on the uncompressed data. So, if you get 20 bytes of gzip response that'll expand to 2 megs, it will be interrupted before the entire response has been uncompressed (given your limit is lower than 2 megs). Default limit is 1 megabyte.

big_number

Both the peer_id and sha1_hash types are typedefs of the class big_number. It represents 20 bytes of data. Its synopsis follows:

class big_number
{
public:
        bool operator==(const big_number& n) const;
        bool operator!=(const big_number& n) const;
        bool operator<(const big_number& n) const;

        const unsigned char* begin() const;
        const unsigned char* end() const;

        unsigned char* begin();
        unsigned char* end();
};

The iterators gives you access to individual bytes.

hasher

This class creates sha1-hashes. Its declaration looks like this:

class hasher
{
public:
        hasher();

        void update(const char* data, unsigned int len);
        sha1_hash final();
        void reset();
};

You use it by first instantiating it, then call update() to feed it with data. i.e. you don't have to keep the entire buffer of which you want to create the hash in memory. You can feed the hasher parts of it at a time. When You have fed the hasher with all the data, you call final() and it will return the sha1-hash of the data.

If you want to reuse the hasher object once you have created a hash, you have to call reset() to reinitialize it.

The sha1-algorithm used was implemented by Steve Reid and released as public domain. For more info, see src/sha1.c.

example usage

dump_torrent

This is an example of a program that will take a torrent-file as a parameter and print information about it to std out:

#include <iostream>
#include <fstream>
#include <iterator>
#include <exception>
#include <vector>
#include <iomanip>

#include "libtorrent/entry.hpp"
#include "libtorrent/bencode.hpp"
#include "libtorrent/session.hpp"
#include "libtorrent/http_settings.hpp"


int main(int argc, char* argv[])
{
        using namespace libtorrent;

        if (argc != 2)
        {
                std::cerr << "usage: dump_torrent torrent-file\n";
                return 1;
        }

        try
        {
                std::ifstream in(argv[1], std::ios_base::binary);
                in.unsetf(std::ios_base::skipws);
                entry e = bdecode(std::istream_iterator<char>(in), std::istream_iterator<char>());
                torrent_info t(e);

                // print info about torrent
                std::cout << "\n\n----- torrent file info -----\n\n";
                std::cout << "trackers:\n";
                for (std::vector<announce_entry>::const_iterator i = t.trackers().begin();
                        i != t.trackers().end();
                        ++i)
                {
                        std::cout << i->tier << ": " << i->url << "\n";
                }

                std::cout << "number of pieces: " << t.num_pieces() << "\n";
                std::cout << "piece length: " << t.piece_length() << "\n";
                std::cout << "files:\n";
                for (torrent_info::file_iterator i = t.begin_files();
                        i != t.end_files();
                        ++i)
                {
                        std::cout << "  " << std::setw(11) << i->size
                        << "  " << i->path << " " << i->filename << "\n";
                }
                
        }
        catch (std::exception& e)
        {
                std::cout << e.what() << "\n";
        }

        return 0;
}

simple client

This is a simple client. It doesn't have much output to keep it simple:

#include <iostream>
#include <fstream>
#include <iterator>
#include <exception>

#include <boost/format.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>

#include "libtorrent/entry.hpp"
#include "libtorrent/bencode.hpp"
#include "libtorrent/session.hpp"
#include "libtorrent/http_settings.hpp"

int main(int argc, char* argv[])
{
        using namespace libtorrent;

        if (argc != 2)
        {
                std::cerr << "usage: ./simple_cient torrent-file\n"
                        "to stop the client, press return.\n";
                return 1;
        }

        try
        {
                session s(6881, "E\x1");

                std::ifstream in(argv[1], std::ios_base::binary);
                in.unsetf(std::ios_base::skipws);
                entry e = bdecode(std::istream_iterator<char>(in), std::istream_iterator<char>());
                torrent_info t(e);
                s.add_torrent(t, "");
                        
                // wait for the user to end
                char a;
                std::cin.unsetf(std::ios_base::skipws);
                std::cin >> a;
        }
        catch (std::exception& e)
        {
                std::cout << e.what() << "\n";
        }
        return 0;
}

Feedback

There's a mailing list.

You can usually find me as hydri in #btports @ irc.freenode.net.

Aknowledgements

Written by Arvid Norberg. Copyright (c) 2003 Arvid Norberg