expanded plugin interface to support session state. improved re-request logic in ut_metadata extension. made max metadata size configurable

ChangeLog

@@ -1,3 +1,4 @@
+	* expanded plugin interface to support session wide states
 	* made the metadata block requesting algorithm more robust against hash check failures
 	* support a separate option to use proxies for peers or not
 	* pausing the session now also pauses checking torrents

docs/libtorrent_plugins.rst

@@ -17,6 +17,8 @@ In short, the plugin interface makes it possible to:
 * add data and parse data from the extension handshake.
 * send extension messages and standard bittorrent messages.
 * override or block the handling of standard bittorrent messages.
+* save and restore state via the session state
+* see all alerts that are posted
 
 .. _extensions: extension_protocol.html
 
@@ -28,14 +30,20 @@ dead locks and race conditions. Since a plugin has access to internal
 structures it is also quite easy to sabotage libtorrent's operation.
 
 All the callbacks in this interface are called with the main libtorrent thread
-mutex locked. And they are always called from the libtorrent main thread. In
+mutex locked. And they are always called from the libtorrent network thread. In
 case portions of your plugin are called from other threads, typically the main
 thread, you cannot use any of the member functions on the internal structures
 in libtorrent, since those require the mutex to be locked. Futhermore, you would
 also need to have a mutex on your own shared data within the plugin, to make
 sure it is not accessed at the same time from the libtorrent thread (through a
 callback). See `boost thread's mutex`_. If you need to send out a message from
-another thread, use an internal queue, and do the actual sending in ``tick()``.
+another thread, it is advised to use an internal queue, and do the actual
+sending in ``tick()``.
+
+Since the plugin interface gives you easy access to internal structures, it
+is not supported as a stable API. Plugins should be considered spcific to a
+specific version of libtorrent. Although, in practice the internals mostly
+don't change that dramatically.
 
 .. _`boost thread's mutex`: http://www.boost.org/doc/html/mutex.html
 
@@ -43,14 +51,15 @@ another thread, use an internal queue, and do the actual sending in ``tick()``.
 plugin interface
 ================
 
-The plugin interface consists of two base classes that the plugin may
+The plugin interface consists of three base classes that the plugin may
-implement. These are called ``torrent_plugin`` and ``peer_plugin``. They are
+implement. These are called ``plugin``, ``torrent_plugin`` and ``peer_plugin``.
-both found in the ``<libtorrent/extensions.hpp>`` header.
+They are found in the ``<libtorrent/extensions.hpp>`` header.
 
-These plugins are instantiated for each torrent and possibly each peer,
+These plugins are instantiated for each session, torrent and possibly each peer,
 respectively.
 
-This is done by passing in a function or function object to
+For plugins that only need per torrent state, it is enough to only implement
+``torrent_plugin`` and pass a constructor function or function object to
 ``session::add_extension()`` or ``torrent_handle::add_extension()`` (if the
 torrent has already been started and you want to hook in the extension at
 run-time).
@@ -69,6 +78,27 @@ for this torrent. If it is a valid pointer (to a class inheriting
 ``torrent_plugin``), it will be associated with this torrent and callbacks
 will be made on torrent events.
 
+For more elaborate plugins which require session wide state, you would
+implement ``plugin``, construct an object (in a ``boost::shared_ptr``) and pass
+it in to ``session::add_extension()``.
+
+plugin
+======
+
+::
+
+	struct plugin
+	{
+		virtual ~plugin();
+		virtual boost::shared_ptr<torrent_plugin> new_torrent(torrent* t, void* user);
+
+		virtual void added(boost::weak_ptr<aux::session_impl> s);
+		virtual void on_alert(alert const* a);
+		virtual void on_tick();
+		virtual void save_state(entry& ent) const;
+		virtual void load_state(lazy_entry const& ent);
+	};
+
 
 torrent_plugin
 ==============

docs/manual.rst

@@ -4244,6 +4244,7 @@ session_settings
 
 		bool no_connect_privileged_ports;
 		int alert_queue_size;
+		int max_metadata_size;
 	};
 
 ``version`` is automatically set to the libtorrent version you're using
@@ -5067,6 +5068,9 @@ using bittorrent swarms for certain DDoS attacks.
 alerts are not popped, the queue will eventually fill up to this level. This
 defaults to 1000.
 
+``max_metadata_size`` is the maximum allowed size (in bytes) to be received
+by the metadata extension, i.e. magnet links. It defaults to 1 MiB.
+
 pe_settings
 ===========
 

include/libtorrent/alert.hpp

@@ -49,6 +49,11 @@ POSSIBILITY OF SUCH DAMAGE.
 #include <boost/preprocessor/repetition/enum_shifted_params.hpp>
 #include <boost/preprocessor/repetition/enum_shifted_binary_params.hpp>
 
+#ifndef TORRENT_DISABLE_EXTENSIONS
+#include <boost/shared_ptr.hpp>
+#include <list>
+#endif
+
 #ifdef _MSC_VER
 #pragma warning(pop)
 #endif
@@ -65,6 +70,10 @@ POSSIBILITY OF SUCH DAMAGE.
 
 namespace libtorrent {
 
+#ifndef TORRENT_DISABLE_EXTENSIONS
+	struct plugin;
+#endif
+
 	class TORRENT_EXPORT alert
 	{
 	public:
@@ -144,11 +153,17 @@ namespace libtorrent {
 			m_alert_mask = m;
 		}
 
+		int alert_mask() const { return m_alert_mask; }
+
 		size_t alert_queue_size_limit() const { return m_queue_size_limit; }
 		size_t set_alert_queue_size_limit(size_t queue_size_limit_);
 
 		void set_dispatch_function(boost::function<void(std::auto_ptr<alert>)> const&);
 
+#ifndef TORRENT_DISABLE_EXTENSIONS
+		void add_extension(boost::shared_ptr<plugin> ext);
+#endif
+
 	private:
 		std::deque<alert*> m_alerts;
 		mutable mutex m_mutex;
@@ -157,6 +172,11 @@ namespace libtorrent {
 		size_t m_queue_size_limit;
 		boost::function<void(std::auto_ptr<alert>)> m_dispatch;
 		io_service& m_ios;
+
+#ifndef TORRENT_DISABLE_EXTENSIONS
+		typedef std::list<boost::shared_ptr<plugin> > ses_extension_list_t;
+		ses_extension_list_t m_ses_extensions;
+#endif
 	};
 
 	struct TORRENT_EXPORT unhandled_alert : std::exception

include/libtorrent/aux_/session_impl.hpp

@@ -101,6 +101,7 @@ POSSIBILITY OF SUCH DAMAGE.
 namespace libtorrent
 {
 
+	struct plugin;
 	class upnp;
 	class natpmp;
 	class lsd;
@@ -135,6 +136,7 @@ namespace libtorrent
 		// this is the link between the main thread and the
 		// thread started to run the main downloader loop
 		struct session_impl: boost::noncopyable, initialize_timer
+			, boost::enable_shared_from_this<session_impl>
 		{
 
 			// the size of each allocation that is chained in the send buffer
@@ -163,6 +165,7 @@ namespace libtorrent
 #ifndef TORRENT_DISABLE_EXTENSIONS
 			void add_extension(boost::function<boost::shared_ptr<torrent_plugin>(
 				torrent*, void*)> ext);
+			void add_ses_extension(boost::shared_ptr<plugin> ext);
 #endif
 #ifdef TORRENT_DEBUG
 			bool has_peer(peer_connection const* p) const
@@ -280,6 +283,7 @@ namespace libtorrent
 			size_t set_alert_queue_size_limit(size_t queue_size_limit_);
 			std::auto_ptr<alert> pop_alert();
 			void set_alert_dispatch(boost::function<void(std::auto_ptr<alert>)> const&);
+			void post_alert(const alert& alert_);
 
 			alert const* wait_for_alert(time_duration max_wait);
 
@@ -901,6 +905,9 @@ namespace libtorrent
 				torrent_plugin>(torrent*, void*)> > extension_list_t;
 
 			extension_list_t m_extensions;
+
+			typedef std::list<boost::shared_ptr<plugin> > ses_extension_list_t;
+			ses_extension_list_t m_ses_extensions;
 #endif
 
 #ifndef TORRENT_DISABLE_GEO_IP

include/libtorrent/extensions.hpp

@@ -39,7 +39,7 @@ POSSIBILITY OF SUCH DAMAGE.
 #pragma warning(push, 1)
 #endif
 
-#include <boost/shared_ptr.hpp>
+#include <boost/weak_ptr.hpp>
 
 #ifdef _MSC_VER
 #pragma warning(pop)
@@ -51,6 +51,8 @@ POSSIBILITY OF SUCH DAMAGE.
 
 namespace libtorrent
 {
+	namespace aux { struct session_impl; }
+
 	struct peer_plugin;
 	class bt_peer_connection;
 	struct peer_request;
@@ -59,6 +61,32 @@ namespace libtorrent
 	struct lazy_entry;
 	struct disk_buffer_holder;
 	struct bitfield;
+	class alert;
+
+	struct TORRENT_EXPORT plugin
+	{
+		virtual ~plugin() {}
+
+		virtual boost::shared_ptr<torrent_plugin> new_torrent(torrent* t, void* user)
+		{ return boost::shared_ptr<torrent_plugin>(); }
+
+		// called when plugin is added to a session
+		virtual void added(boost::weak_ptr<aux::session_impl> s) {}
+
+		// called when an alert is posted
+		// alerts that are filtered are not
+		// posted
+		virtual void on_alert(alert const* a) {}
+
+		// called once per second
+		virtual void on_tick() {}
+
+		// called when saving settings state
+		virtual void save_state(entry& ent) const {}
+
+		// called when loading settings state
+		virtual void load_state(lazy_entry const& ent) {}
+	};
 
 	struct TORRENT_EXPORT torrent_plugin
 	{
@@ -79,11 +107,16 @@ namespace libtorrent
 		// and no other plugins will have their handlers called, and the
 		// default behavior will be skipped
 		virtual bool on_pause() { return false; }
-		virtual bool on_resume() { return false;}
+		virtual bool on_resume() { return false; }
 
 		// this is called when the initial checking of
 		// files is completed.
 		virtual void on_files_checked() {}
+
+		// called when the torrent changes state
+		// the state is one of torrent_status::state_t
+		// enum members
+		virtual void on_state(int s) {}
 	};
 
 	struct TORRENT_EXPORT peer_plugin

include/libtorrent/session.hpp

@@ -66,6 +66,7 @@ POSSIBILITY OF SUCH DAMAGE.
 
 namespace libtorrent
 {
+	struct plugin;
 	struct torrent_plugin;
 	class torrent;
 	struct ip_filter;
@@ -267,6 +268,7 @@ namespace libtorrent
 
 #ifndef TORRENT_DISABLE_EXTENSIONS
 		void add_extension(boost::function<boost::shared_ptr<torrent_plugin>(torrent*, void*)> ext);
+		void add_extension(boost::shared_ptr<plugin> ext);
 #endif
 
 #ifndef TORRENT_DISABLE_GEO_IP

include/libtorrent/session_settings.hpp

@@ -260,6 +260,7 @@ namespace libtorrent
 			, seeding_outgoing_connections(true)
 			, no_connect_privileged_ports(true)
 			, alert_queue_size(1000)
+			, max_metadata_size(1024*1024)
 		{}
 
 		// libtorrent version. Used for forward binary compatibility
@@ -1025,6 +1026,10 @@ namespace libtorrent
 
 		// the max alert queue size
 		int alert_queue_size;
+
+		// the max allowed size for metadata received by the
+		// ut_metadata extension (i.e. magnet links)
+		int max_metadata_size;
 	};
 
 #ifndef TORRENT_DISABLE_DHT

include/libtorrent/torrent.hpp

@@ -238,6 +238,7 @@ namespace libtorrent
 		void clear_error();
 		void set_error(error_code const& ec, std::string const& file);
 		bool has_error() const { return m_error; }
+		error_code error() const { return m_error; }
 
 		void flush_cache();
 		void pause(bool graceful = false);

src/alert.cpp

@@ -39,6 +39,7 @@ POSSIBILITY OF SUCH DAMAGE.
 #include "libtorrent/time.hpp"
 #include "libtorrent/error_code.hpp"
 #include "libtorrent/escape_string.hpp"
+#include "libtorrent/extensions.hpp"
 #include <boost/bind.hpp>
 
 namespace libtorrent {
@@ -394,20 +395,43 @@ namespace libtorrent {
 
 	void alert_manager::post_alert(const alert& alert_)
 	{
+
 		mutex::scoped_lock lock(m_mutex);
 
 		if (m_dispatch)
 		{
 			TORRENT_ASSERT(m_alerts.empty());
 			m_dispatch(std::auto_ptr<alert>(alert_.clone()));
-			return;
+		}
+		else if (m_alerts.size() < m_queue_size_limit || !alert_.discardable())
+		{
+			m_alerts.push_back(alert_.clone().release());
 		}
 
-		if (m_alerts.size() >= m_queue_size_limit && alert_.discardable()) return;
+#ifndef TORRENT_DISABLE_EXTENSIONS
-		m_alerts.push_back(alert_.clone().release());
+		lock.unlock();
-//		m_condition.signal(lock);
+
-//		m_condition.clear(lock);
+		for (ses_extension_list_t::iterator i = m_ses_extensions.begin()
+			, end(m_ses_extensions.end()); i != end; ++i)
+		{
+#ifndef BOOST_NO_EXCEPTIONS
+			try {
+#endif
+			(*i)->on_alert(&alert_);
+#ifndef BOOST_NO_EXCEPTIONS
+			} catch (std::exception&) {}
+#endif
 		}
+#endif
+
+	}
+
+#ifndef TORRENT_DISABLE_EXTENSIONS
+	void alert_manager::add_extension(boost::shared_ptr<plugin> ext)
+	{
+		m_ses_extensions.push_back(ext);
+	}
+#endif
 
 	std::auto_ptr<alert> alert_manager::get()
 	{

src/metadata_transfer.cpp

@@ -421,7 +421,7 @@ namespace libtorrent { namespace
 						<< " ]\n";
 #endif
 
-					if (total_size > 500 * 1024)
+					if (total_size > m_torrent.session().settings().max_metadata_size)
 					{
 						m_pc.disconnect(errors::metadata_too_large, 2);
 						return true;

src/session.cpp

@@ -417,6 +417,11 @@ namespace libtorrent
 	{
 		TORRENT_ASYNC_CALL1(add_extension, ext);
 	}
+
+	void session::add_extension(boost::shared_ptr<plugin> ext)
+	{
+		TORRENT_ASYNC_CALL1(add_ses_extension, ext);
+	}
 #endif
 
 #ifndef TORRENT_DISABLE_GEO_IP

src/session_impl.cpp

@@ -79,6 +79,7 @@ POSSIBILITY OF SUCH DAMAGE.
 #include "libtorrent/peer_info.hpp"
 #include "libtorrent/settings.hpp"
 #include "libtorrent/build_config.hpp"
+#include "libtorrent/extensions.hpp"
 
 #if defined TORRENT_VERBOSE_LOGGING || defined TORRENT_LOGGING || defined TORRENT_ERROR_LOGGING
 #endif
@@ -354,6 +355,7 @@ namespace aux {
 		TORRENT_SETTING(boolean, seeding_outgoing_connections)
 		TORRENT_SETTING(boolean, no_connect_privileged_ports)
 		TORRENT_SETTING(integer, alert_queue_size)
+		TORRENT_SETTING(integer, max_metadata_size)
 	};
 
 #undef TORRENT_SETTING
@@ -956,6 +958,19 @@ namespace aux {
 			}
 		}
 
+#ifndef TORRENT_DISABLE_EXTENSIONS
+		for (ses_extension_list_t::const_iterator i = m_ses_extensions.begin()
+			, end(m_ses_extensions.end()); i != end; ++i)
+		{
+#ifndef BOOST_NO_EXCEPTIONS
+			try {
+#endif
+			(*i)->save_state(*eptr);
+#ifndef BOOST_NO_EXCEPTIONS
+			} catch (std::exception&) {}
+#endif
+		}
+#endif
 	}
 	
 	void session_impl::set_proxy(proxy_settings const& s)
@@ -1047,6 +1062,20 @@ namespace aux {
 				m_feeds.push_back(f);
 			}
 		}
+
+#ifndef TORRENT_DISABLE_EXTENSIONS
+		for (ses_extension_list_t::iterator i = m_ses_extensions.begin()
+			, end(m_ses_extensions.end()); i != end; ++i)
+		{
+#ifndef BOOST_NO_EXCEPTIONS
+			try {
+#endif
+			(*i)->load_state(*e);
+#ifndef BOOST_NO_EXCEPTIONS
+			} catch (std::exception&) {}
+#endif
+		}
+#endif
 	}
 
 #ifndef TORRENT_DISABLE_GEO_IP
@@ -1169,6 +1198,16 @@ namespace aux {
 
 		m_extensions.push_back(ext);
 	}
+
+	void session_impl::add_ses_extension(boost::shared_ptr<plugin> ext)
+	{
+		TORRENT_ASSERT(is_network_thread());
+		TORRENT_ASSERT_VAL(ext, ext);
+
+		m_ses_extensions.push_back(ext);
+		m_alerts.add_extension(ext);
+		ext->added(shared_from_this());
+	}
 #endif
 
 #ifndef TORRENT_DISABLE_DHT
@@ -2393,6 +2432,20 @@ namespace aux {
 			}
 		}
 
+#ifndef TORRENT_DISABLE_EXTENSIONS
+		for (ses_extension_list_t::const_iterator i = m_ses_extensions.begin()
+			, end(m_ses_extensions.end()); i != end; ++i)
+		{
+#ifndef BOOST_NO_EXCEPTIONS
+			try {
+#endif
+			(*i)->on_tick();
+#ifndef BOOST_NO_EXCEPTIONS
+			} catch (std::exception&) {}
+#endif
+		}
+#endif
+
 		// --------------------------------------------------------------
 		// RSS feeds
 		// --------------------------------------------------------------
@@ -3622,6 +3675,13 @@ namespace aux {
 			boost::shared_ptr<torrent_plugin> tp((*i)(torrent_ptr.get(), params.userdata));
 			if (tp) torrent_ptr->add_extension(tp);
 		}
+
+		for (ses_extension_list_t::iterator i = m_ses_extensions.begin()
+			, end(m_ses_extensions.end()); i != end; ++i)
+		{
+			boost::shared_ptr<torrent_plugin> tp((*i)->new_torrent(torrent_ptr.get(), params.userdata));
+			if (tp) torrent_ptr->add_extension(tp);
+		}
 #endif
 
 #ifndef TORRENT_DISABLE_DHT

src/torrent.cpp

@@ -7107,6 +7107,20 @@ namespace libtorrent
 		if (m_ses.m_alerts.should_post<state_changed_alert>())
 			m_ses.m_alerts.post_alert(state_changed_alert(get_handle(), s, (torrent_status::state_t)m_state));
 		m_state = s;
+
+#ifndef TORRENT_DISABLE_EXTENSIONS
+		for (extension_list_t::iterator i = m_extensions.begin()
+			, end(m_extensions.end()); i != end; ++i)
+		{
+#ifndef BOOST_NO_EXCEPTIONS
+			try {
+#endif
+				(*i)->on_state(m_state);
+#ifndef BOOST_NO_EXCEPTIONS
+			} catch (std::exception&) {}
+#endif
+		}
+#endif
 	}
 
 	torrent_status torrent::status(boost::uint32_t flags) const

src/ut_metadata.cpp

@@ -107,6 +107,7 @@ namespace libtorrent { namespace
 
 		// returns a piece of the metadata that
 		// we should request.
+		// returns -1 if we should hold off the request
 		int metadata_request();
 
 		// this is called from the peer_connection for
@@ -148,8 +149,9 @@ namespace libtorrent { namespace
 
 		struct metadata_piece
 		{
-			metadata_piece(): num_requests(0) {}
+			metadata_piece(): num_requests(0), last_request(0) {}
 			int num_requests;
+			time_t last_request;
 			boost::weak_ptr<ut_metadata_peer_plugin> source;
 			bool operator<(metadata_piece const& rhs) const
 			{ return num_requests < rhs.num_requests; }
@@ -372,6 +374,8 @@ namespace libtorrent { namespace
 				&& has_metadata())
 			{
 				int piece = m_tp.metadata_request();
+				if (piece == -1) return;
+
 				m_sent_requests.push_back(piece);
 				write_metadata_packet(0, piece);
 			}
@@ -432,7 +436,13 @@ namespace libtorrent { namespace
 		}
 
 		int piece = i - m_requested_metadata.begin();
+
+		// don't request the same block more than once every 3 seconds
+		time_t now = time(0);
+		if (now - m_requested_metadata[piece].last_request < 3) return -1;
+
 		++m_requested_metadata[piece].num_requests;
+		m_requested_metadata[piece].last_request = now;
 		return piece;
 	}
 
@@ -440,12 +450,17 @@ namespace libtorrent { namespace
 		boost::weak_ptr<ut_metadata_peer_plugin> const& source
 		, char const* buf, int size, int piece, int total_size)
 	{
-		if (m_torrent.valid_metadata()) return false;
+		if (m_torrent.valid_metadata())
+		{
+			m_torrent.add_redundant_bytes(size);
+			return false;
+		}
 		
 		if (!m_metadata)
 		{
 			// verify the total_size
-			if (total_size <= 0 || total_size > 500 * 1024) return false;
+			if (total_size <= 0 || total_size > m_torrent.session().settings().max_metadata_size)
+				return false;
 
 			m_metadata.reset(new char[total_size]);
 			m_requested_metadata.resize(div_round_up(total_size, 16 * 1024));