first implementation of DHT security implementation. tie the node ID to the external IP

This commit is contained in:
Arvid Norberg 2010-12-11 09:38:07 +00:00
parent 6769df7508
commit 29ed03f720
15 changed files with 356 additions and 14 deletions

154
docs/dht_sec.html Normal file
View File

@ -0,0 +1,154 @@
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<script type="text/javascript">
/* <![CDATA[ */
(function() {
var s = document.createElement('script'), t = document.getElementsByTagName('script')[0];
s.type = 'text/javascript';
s.async = true;
s.src = 'http://api.flattr.com/js/0.6/load.js?mode=auto';
t.parentNode.insertBefore(s, t);
})();
/* ]]> */
</script>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" />
<title>BitTorrent DHT security extension</title>
<meta name="author" content="Arvid Norberg, arvid&#64;rasterbar.com" />
<link rel="stylesheet" type="text/css" href="../../css/base.css" />
<link rel="stylesheet" type="text/css" href="../../css/rst.css" />
<link rel="stylesheet" href="style.css" type="text/css" />
<style type="text/css">
/* Hides from IE-mac \*/
* html pre { height: 1%; }
/* End hide from IE-mac */
</style>
</head>
<body>
<div class="document" id="bittorrent-dht-security-extension">
<div id="container">
<div id="headerNav">
<ul>
<li class="first"><a href="/">Home</a></li>
<li><a href="../../products.html">Products</a></li>
<li><a href="../../contact.html">Contact</a></li>
</ul>
</div>
<div id="header">
<h1><span>Rasterbar Software</span></h1>
<h2><span>Software developement and consulting</span></h2>
</div>
<div id="main">
<h1 class="title">BitTorrent DHT security extension</h1>
<table class="docinfo" frame="void" rules="none">
<col class="docinfo-name" />
<col class="docinfo-content" />
<tbody valign="top">
<tr><th class="docinfo-name">Author:</th>
<td>Arvid Norberg, <a class="last reference external" href="mailto:arvid&#64;rasterbar.com">arvid&#64;rasterbar.com</a></td></tr>
<tr><th class="docinfo-name">Version:</th>
<td>Draft</td></tr>
</tbody>
</table>
<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of contents</p>
<ul class="simple">
<li><a class="reference internal" href="#id1" id="id2">BitTorrent DHT security extension</a></li>
<li><a class="reference internal" href="#node-ids" id="id3">node IDs</a></li>
<li><a class="reference internal" href="#bootstrapping" id="id4">bootstrapping</a></li>
<li><a class="reference internal" href="#enforcement" id="id5">enforcement</a></li>
<li><a class="reference internal" href="#backwards-compatibility-and-transition" id="id6">backwards compatibility and transition</a></li>
</ul>
</div>
<div class="section" id="id1">
<h1>BitTorrent DHT security extension</h1>
<p>The purpose of this extension is to make it harder to launch a few
specific attacks against the BitTorrent DHT and also to make it harder
to snoop the network.</p>
<p>Specifically the attack this extension intends to make harder is launching
8 or more DHT nodes which node-IDs selected close to a specific target
info-hash, in order to become the main nodes hosting peers for it. Currently
this is very easy to do and lets the attacker not only see all the traffic
related to this specific info-hash but also block access to it by other
peers.</p>
<p>The proposed guard against this is to enforce restrictions on which node-ID
a node can choose, based on its external IP address.</p>
</div>
<div class="section" id="node-ids">
<h1>node IDs</h1>
<p>The proposed formula for restricting node IDs is that the 4 first bytes of
the node ID MUST match the 4 first bytes of <tt class="docutils literal"><span class="pre">SHA-1(IP_address)</span></tt>. That is,
the raw, big endian, storage of the address, either IPv4 or IPv6, hashed
with SHA-1.</p>
<p>Example:</p>
<blockquote>
An IP address 89.5.5.5 has a big endian byte representation of
<tt class="docutils literal"><span class="pre">0x59</span> <span class="pre">0x05</span> <span class="pre">0x05</span> <span class="pre">0x05</span></tt>. The SHA-1 hash of this byte sequence is
<tt class="docutils literal"><span class="pre">656d41da810a0a6d92fd2f6a8ba3b466e35ab368</span></tt>. The DHT node must choose
a node ID which starts with <tt class="docutils literal"><span class="pre">656d41da</span></tt>.</blockquote>
</div>
<div class="section" id="bootstrapping">
<h1>bootstrapping</h1>
<p>In order to set ones initial node ID, the external IP needs to be known. This
is not a trivial problem. WIth this extension, <em>all</em> DHT requests whose node
ID does not match its IP address MUST be serviced and MUST also include one
extra result value (inside the <tt class="docutils literal"><span class="pre">r</span></tt> dictionary) called <tt class="docutils literal"><span class="pre">ip</span></tt>. The IP field
contains the raw (big endian) byte representation of the external IP address.
This is the same byte sequence passed to SHA-1.</p>
<p>A DHT node which receives an <tt class="docutils literal"><span class="pre">ip</span></tt> result in a request SHOULD consider restarting
its DHT node with a new node ID, taking this IP into account. Since a single node
can not be trusted, there should be some mechanism of determining whether or
not the node has a correct understanding of its external IP or not. This could
be done by voting, or only restart the DHT once at least a certain number of
nodes, from separate searches, tells you your node ID is incorrect.</p>
</div>
<div class="section" id="enforcement">
<h1>enforcement</h1>
<p>Write tokens from peers whose node ID does not match its external IP should be
considered dropped. In other words, a peer that uses a non-matching ID MUST
never be used to store information on, regardless of which request. In the
original DHT specification only <tt class="docutils literal"><span class="pre">announce_peer</span></tt> stores data in the network,
but any future extension which stores data in the network SHOULD use the same
restriction.</p>
<p>Any peer on a local network address is exempt from this node ID verification.
This includes the following IP blocks:</p>
<dl class="docutils">
<dt>10.0.0.0/8</dt>
<dd>reserved for local networks</dd>
<dt>172.16.0.0/12</dt>
<dd>reserved for local networks</dd>
<dt>192.168.0.0/16</dt>
<dd>reserved for local networks</dd>
<dt>169.254.0.0/16</dt>
<dd>reserved for self-assigned IPs</dd>
<dt>127.0.0.0/8</dt>
<dd>reserved for loopback</dd>
</dl>
</div>
<div class="section" id="backwards-compatibility-and-transition">
<h1>backwards compatibility and transition</h1>
<p>During some transition period, this restriction should not be enforced, and
peers whose node ID does not match this formula relative to their external IP
should not be blocked.</p>
<p>Requests from peers whose node ID does not match their external IP should
always be serviced, even after the transition period. The attack this protects
from is storing data on an attacker's node, not servicing an attackers request.</p>
</div>
</div>
<div id="footer">
<span>Copyright &copy; 2005 Rasterbar Software.</span>
</div>
</div>
<script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
</script>
<script type="text/javascript">
_uacct = "UA-1599045-1";
urchinTracker();
</script>
</div>
</body>
</html>

96
docs/dht_sec.rst Normal file
View File

@ -0,0 +1,96 @@
=================================
BitTorrent DHT security extension
=================================
:Author: Arvid Norberg, arvid@rasterbar.com
:Version: Draft
.. contents:: Table of contents
:depth: 2
:backlinks: none
BitTorrent DHT security extension
---------------------------------
The purpose of this extension is to make it harder to launch a few
specific attacks against the BitTorrent DHT and also to make it harder
to snoop the network.
Specifically the attack this extension intends to make harder is launching
8 or more DHT nodes which node-IDs selected close to a specific target
info-hash, in order to become the main nodes hosting peers for it. Currently
this is very easy to do and lets the attacker not only see all the traffic
related to this specific info-hash but also block access to it by other
peers.
The proposed guard against this is to enforce restrictions on which node-ID
a node can choose, based on its external IP address.
node IDs
--------
The proposed formula for restricting node IDs is that the 4 first bytes of
the node ID MUST match the 4 first bytes of ``SHA-1(IP_address)``. That is,
the raw, big endian, storage of the address, either IPv4 or IPv6, hashed
with SHA-1.
Example:
An IP address 89.5.5.5 has a big endian byte representation of
``0x59 0x05 0x05 0x05``. The SHA-1 hash of this byte sequence is
``656d41da810a0a6d92fd2f6a8ba3b466e35ab368``. The DHT node must choose
a node ID which starts with ``656d41da``.
bootstrapping
-------------
In order to set ones initial node ID, the external IP needs to be known. This
is not a trivial problem. WIth this extension, *all* DHT requests whose node
ID does not match its IP address MUST be serviced and MUST also include one
extra result value (inside the ``r`` dictionary) called ``ip``. The IP field
contains the raw (big endian) byte representation of the external IP address.
This is the same byte sequence passed to SHA-1.
A DHT node which receives an ``ip`` result in a request SHOULD consider restarting
its DHT node with a new node ID, taking this IP into account. Since a single node
can not be trusted, there should be some mechanism of determining whether or
not the node has a correct understanding of its external IP or not. This could
be done by voting, or only restart the DHT once at least a certain number of
nodes, from separate searches, tells you your node ID is incorrect.
enforcement
-----------
Write tokens from peers whose node ID does not match its external IP should be
considered dropped. In other words, a peer that uses a non-matching ID MUST
never be used to store information on, regardless of which request. In the
original DHT specification only ``announce_peer`` stores data in the network,
but any future extension which stores data in the network SHOULD use the same
restriction.
Any peer on a local network address is exempt from this node ID verification.
This includes the following IP blocks:
10.0.0.0/8
reserved for local networks
172.16.0.0/12
reserved for local networks
192.168.0.0/16
reserved for local networks
169.254.0.0/16
reserved for self-assigned IPs
127.0.0.0/8
reserved for loopback
backwards compatibility and transition
--------------------------------------
During some transition period, this restriction should not be enforced, and
peers whose node ID does not match this formula relative to their external IP
should not be blocked.
Requests from peers whose node ID does not match their external IP should
always be serviced, even after the transition period. The attack this protects
from is storing data on an attacker's node, not servicing an attackers request.

View File

@ -68,6 +68,7 @@
<li><a class="reference external" href="extension_protocol.html">extensions protocol</a></li>
<li><a class="reference external" href="libtorrent_plugins.html">plugin interface</a></li>
<li><a class="reference external" href="dht_extensions.html">DHT extensions</a></li>
<li><a class="reference external" href="dht_sec.html">DHT security extension</a></li>
<li><a class="reference external" href="udp_tracker_protocol.html">UDP tracker protocol</a></li>
<li><a class="reference external" href="http://www.getright.com/seedtorrent.html">HTTP seed</a></li>
<li><a class="reference external" href="http://bittorrent.org/beps/bep_0012.html">multitracker</a></li>

View File

@ -26,6 +26,7 @@ Extensions
* `extensions protocol`_
* `plugin interface`_
* `DHT extensions`_
* `DHT security extension`_
* `UDP tracker protocol`_
* `HTTP seed`_
* multitracker_
@ -64,6 +65,7 @@ libtorrent
.. _`extensions protocol`: extension_protocol.html
.. _`plugin interface`: libtorrent_plugins.html
.. _`DHT extensions`: dht_extensions.html
.. _`DHT security extension`: dht_sec.html
.. _`UDP tracker protocol`: udp_tracker_protocol.html
.. _`HTTP seed`: http://www.getright.com/seedtorrent.html
.. _multitracker: http://bittorrent.org/beps/bep_0012.html

View File

@ -13,6 +13,7 @@ TARGETS = index \
extension_protocol \
make_torrent \
dht_extensions \
dht_sec \
libtorrent_plugins \
python_binding \
projects \

View File

@ -38,6 +38,7 @@ POSSIBILITY OF SUCH DAMAGE.
#include "libtorrent/config.hpp"
#include "libtorrent/peer_id.hpp"
#include "libtorrent/assert.hpp"
#include "libtorrent/address.hpp"
namespace libtorrent { namespace dht
{
@ -55,7 +56,9 @@ bool TORRENT_EXPORT compare_ref(node_id const& n1, node_id const& n2, node_id co
// usefult for finding out which bucket a node belongs to
int TORRENT_EXPORT distance_exp(node_id const& n1, node_id const& n2);
node_id TORRENT_EXPORT generate_id();
node_id TORRENT_EXPORT generate_id(address const& external_ip = address());
bool TORRENT_EXPORT verify_id(node_id const& nid, address const& source_ip);
} } // namespace libtorrent::dht

View File

@ -46,6 +46,8 @@ POSSIBILITY OF SUCH DAMAGE.
#include "libtorrent/ptime.hpp"
namespace libtorrent { namespace aux { struct session_impl; } }
namespace libtorrent { namespace dht
{
@ -69,7 +71,7 @@ public:
rpc_manager(node_id const& our_id
, routing_table& table, send_fun const& sf
, void* userdata);
, void* userdata, aux::session_impl& ses);
~rpc_manager();
void unreachable(udp::endpoint const& ep);
@ -116,6 +118,7 @@ private:
node_id m_random_number;
int m_allocated_observers;
bool m_destructing;
aux::session_impl& m_ses;
};
} } // namespace libtorrent::dht

View File

@ -45,6 +45,7 @@ namespace libtorrent
TORRENT_EXPORT std::string print_address(address const& addr);
TORRENT_EXPORT std::string print_endpoint(tcp::endpoint const& ep);
TORRENT_EXPORT std::string print_endpoint(udp::endpoint const& ep);
TORRENT_EXPORT std::string address_to_bytes(address const& a);
namespace detail
{

View File

@ -67,7 +67,8 @@ namespace libtorrent
return ((ip & 0xff000000) == 0x0a000000 // 10.x.x.x
|| (ip & 0xfff00000) == 0xac100000 // 172.16.x.x
|| (ip & 0xffff0000) == 0xc0a80000 // 192.168.x.x
|| (ip & 0xffff0000) == 0xa9fe0000); // 169.254.x.x
|| (ip & 0xffff0000) == 0xa9fe0000 // 169.254.x.x
|| (ip & 0xff000000) == 0x7f000000); // 127.x.x.x
}
bool is_loopback(address const& addr)

View File

@ -183,9 +183,9 @@ node_impl::node_impl(libtorrent::aux::session_impl& ses
, node_id nid
, void* userdata)
: m_settings(settings)
, m_id(nid == (node_id::min)() ? generate_id() : nid)
, m_id(nid == (node_id::min)() || !verify_id(nid, ses.external_address()) ? generate_id(ses.external_address()) : nid)
, m_table(m_id, 8, settings)
, m_rpc(m_id, m_table, f, userdata)
, m_rpc(m_id, m_table, f, userdata, ses)
, m_last_tracker_tick(time_now())
, m_ses(ses)
, m_send(f)
@ -702,6 +702,11 @@ void node_impl::incoming_request(msg const& m, entry& e)
entry& reply = e["r"];
m_rpc.add_our_id(reply);
// if this nodes ID doesn't match its IP, tell it what
// its IP is
if (!verify_id(id, m.addr.address()))
reply["ip"] = address_to_bytes(m.addr.address());
if (strcmp(query, "ping") == 0)
{
// we already have 't' and 'id' in the response
@ -848,6 +853,7 @@ void node_impl::incoming_request(msg const& m, entry& e)
++g_announces;
#endif
}
/*
else if (strcmp(query, "find_torrent") == 0)
{
key_desc_t msg_desc[] = {
@ -921,6 +927,7 @@ void node_impl::incoming_request(msg const& m, entry& e)
i->second.publish(msg_keys[2]->string_value(), in_tags, num_tags);
}
*/
else
{
// if we don't recognize the message but there's a

View File

@ -38,6 +38,7 @@ POSSIBILITY OF SUCH DAMAGE.
#include "libtorrent/kademlia/node_id.hpp"
#include "libtorrent/hasher.hpp"
#include "libtorrent/assert.hpp"
#include "libtorrent/broadcast_socket.hpp" // for is_local et.al
namespace libtorrent { namespace dht
{
@ -94,19 +95,52 @@ int distance_exp(node_id const& n1, node_id const& n2)
}
struct static_ { static_() { std::srand(std::time(0)); } } static__;
node_id generate_id()
void hash_address(address const& ip, sha1_hash& h)
{
if (ip.is_v4())
{
address_v4::bytes_type b = ip.to_v4().to_bytes();
h = hasher((char*)&b[0], b.size()).final();
}
else
{
address_v6::bytes_type b = ip.to_v6().to_bytes();
h = hasher((char*)&b[0], b.size()).final();
}
}
// verifies whether a node-id matches the IP it's used from
// returns true if the node-id is OK coming from this source
// and false otherwise.
bool verify_id(node_id const& nid, address const& source_ip)
{
// no need to verify local IPs, they would be incorrect anyway
if (is_local(source_ip)) return true;
node_id h;
hash_address(source_ip, h);
return memcmp(&nid[0], &h[0], 4) == 0;
}
node_id generate_id(address const& external_ip)
{
node_id h;
char random[20];
#ifdef _MSC_VER
std::generate(random, random + 20, &rand);
#else
std::generate(random, random + 20, &std::rand);
#endif
h = hasher(random, 20).final();
hasher h;
h.update(random, 20);
return h.final();
if (!is_local(external_ip))
{
node_id ph;
hash_address(external_ip, ph);
memcpy(&h[0], &ph[0], 4);
}
return h;
}
} } // namespace libtorrent::dht

View File

@ -176,7 +176,7 @@ bool routing_table::need_refresh(node_id& target) const
if (time_now() - i->last_active < minutes(15)) return false;
// generate a random node_id within the given bucket
target = generate_id();
target = generate_id(address());
int num_bits = std::distance(m_buckets.begin(), i) + 1;
node_id mask(0);
for (int i = 0; i < num_bits; ++i) mask[i/8] |= 0x80 >> (i&7);

View File

@ -33,10 +33,14 @@ POSSIBILITY OF SUCH DAMAGE.
#include "libtorrent/pch.hpp"
#include "libtorrent/socket.hpp"
// TODO: it would be nice to not have this dependency here
#include "libtorrent/aux_/session_impl.hpp"
#include <boost/bind.hpp>
#include <libtorrent/io.hpp>
#include <libtorrent/invariant_check.hpp>
#include <libtorrent/kademlia/node_id.hpp> // for generate_id
#include <libtorrent/kademlia/rpc_manager.hpp>
#include <libtorrent/kademlia/logging.hpp>
#include <libtorrent/kademlia/routing_table.hpp>
@ -150,8 +154,6 @@ void observer::timeout()
m_algorithm->failed(observer_ptr(this));
}
node_id generate_id();
enum { observer_size = max3<
sizeof(find_data_observer)
, sizeof(announce_observer)
@ -161,7 +163,7 @@ enum { observer_size = max3<
rpc_manager::rpc_manager(node_id const& our_id
, routing_table& table, send_fun const& sf
, void* userdata)
, void* userdata, aux::session_impl& ses)
: m_pool_allocator(observer_size, 10)
, m_next_transaction_id(std::rand() % max_transaction_id)
, m_send(sf)
@ -172,6 +174,7 @@ rpc_manager::rpc_manager(node_id const& our_id
, m_random_number(generate_id())
, m_allocated_observers(0)
, m_destructing(false)
, m_ses(ses)
{
std::srand(time(0));
@ -339,6 +342,22 @@ bool rpc_manager::incoming(msg const& m, node_id* id)
return false;
}
lazy_entry const* ext_ip = ret_ent->dict_find_string("ip");
if (ext_ip && ext_ip->string_length() == 4)
{
// this node claims we use the wrong node-ID!
address_v4::bytes_type b;
memcpy(&b[0], ext_ip->string_ptr(), 4);
m_ses.set_external_address(address_v4(b));
}
else if (ext_ip && ext_ip->string_length() == 16)
{
// this node claims we use the wrong node-ID!
address_v6::bytes_type b;
memcpy(&b[0], ext_ip->string_ptr(), 16);
m_ses.set_external_address(address_v6(b));
}
#ifdef TORRENT_DHT_VERBOSE_LOGGING
TORRENT_LOG(rpc) << "[" << o->m_algorithm.get() << "] Reply with transaction id: "
<< tid << " from " << m.addr;

View File

@ -4353,6 +4353,12 @@ namespace aux {
m_external_address = ip;
if (m_alerts.should_post<external_ip_alert>())
m_alerts.post_alert(external_ip_alert(ip));
// since we have a new external IP now, we need to
// restart the DHT with a new node ID
#ifndef TORRENT_DISABLE_DHT
start_dht(m_dht_state);
#endif
}
void session_impl::free_disk_buffer(char* buf)

View File

@ -47,6 +47,20 @@ namespace libtorrent
return addr.to_string(ec);
}
std::string address_to_bytes(address const& a)
{
if (a.is_v4())
{
address_v4::bytes_type b = a.to_v4().to_bytes();
return std::string((char*)&b[0], b.size());
}
else
{
address_v6::bytes_type b = a.to_v6().to_bytes();
return std::string((char*)&b[0], b.size());
}
}
std::string print_endpoint(tcp::endpoint const& ep)
{
error_code ec;