premiere
/
premiere-libtorrent
2
2
Fork 1
Code Issues Pull Requests Releases Wiki Activity

add support for mutable put/get functions in DHT

This commit is contained in:
Arvid Norberg 2011-05-25 02:26:07 +00:00
parent b0586eb47e
commit 6fa1827c39
9 changed files with 824 additions and 87 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

287
docs/dht_store.html Normal file
View File

@ -0,0 +1,287 @@
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<script type="text/javascript">
/* <![CDATA[ */
(function() {
var s = document.createElement('script'), t = document.getElementsByTagName('script')[0];
s.type = 'text/javascript';
s.async = true;
s.src = 'http://api.flattr.com/js/0.6/load.js?mode=auto';
t.parentNode.insertBefore(s, t);
})();
/* ]]> */
</script>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" />
<title>BitTorrent extension for arbitrary DHT store</title>
<meta name="author" content="Arvid Norberg, arvid&#64;rasterbar.com" />
<link rel="stylesheet" type="text/css" href="../../css/base.css" />
<link rel="stylesheet" type="text/css" href="../../css/rst.css" />
<link rel="stylesheet" href="style.css" type="text/css" />
<style type="text/css">
/* Hides from IE-mac \*/
* html pre { height: 1%; }
/* End hide from IE-mac */
</style>
</head>
<body>
<div class="document" id="bittorrent-extension-for-arbitrary-dht-store">
<div id="container">
<div id="headerNav">
<ul>
<li class="first"><a href="/">Home</a></li>
<li><a href="../../products.html">Products</a></li>
<li><a href="../../contact.html">Contact</a></li>
</ul>
</div>
<div id="header">
<h1><span>Rasterbar Software</span></h1>
<h2><span>Software developement and consulting</span></h2>
</div>
<div id="main">
<h1 class="title">BitTorrent extension for arbitrary DHT store</h1>
<table class="docinfo" frame="void" rules="none">
<col class="docinfo-name" />
<col class="docinfo-content" />
<tbody valign="top">
<tr><th class="docinfo-name">Author:</th>
<td>Arvid Norberg, <a class="last reference external" href="mailto:arvid&#64;rasterbar.com">arvid&#64;rasterbar.com</a></td></tr>
<tr><th class="docinfo-name">Version:</th>
<td>Draft</td></tr>
</tbody>
</table>
<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of contents</p>
<ul class="simple">
<li><a class="reference internal" href="#terminology" id="id3">terminology</a></li>
<li><a class="reference internal" href="#messages" id="id4">messages</a></li>
<li><a class="reference internal" href="#immutable-items" id="id5">immutable items</a><ul>
<li><a class="reference internal" href="#put-message" id="id6">put message</a></li>
<li><a class="reference internal" href="#get-message" id="id7">get message</a></li>
</ul>
</li>
<li><a class="reference internal" href="#mutable-items" id="id8">mutable items</a><ul>
<li><a class="reference internal" href="#id1" id="id9">put message</a></li>
<li><a class="reference internal" href="#id2" id="id10">get message</a></li>
</ul>
</li>
<li><a class="reference internal" href="#signature-verification" id="id11">signature verification</a></li>
<li><a class="reference internal" href="#expiration" id="id12">expiration</a></li>
<li><a class="reference internal" href="#test-vectors" id="id13">test vectors</a></li>
</ul>
</div>
<p>This is a proposal for an extension to the BitTorrent DHT to allow
storing and retrieving of arbitrary data.</p>
<p>It supports both storing <em>immutable</em> items, where the key is
the SHA-1 hash of the data itself, and <em>mutable</em> items, where
the key is the public key of the key pair used to sign the data.</p>
<p>There are two new proposed messages, <tt class="docutils literal"><span class="pre">put</span></tt> and <tt class="docutils literal"><span class="pre">get</span></tt>.</p>
<div class="section" id="terminology">
<h1>terminology</h1>
<p>In this document, a <em>storage node</em> refers to the node in the DHT to which
an item is being announced and stored on. A <em>subscribing node</em> refers to
a node which makes look-ups in the DHT to find the storage nodes, to
request items from them, and possibly re-announce those items to keep them
alive.</p>
</div>
<div class="section" id="messages">
<h1>messages</h1>
<p>The proposed new messages <tt class="docutils literal"><span class="pre">get</span></tt> and <tt class="docutils literal"><span class="pre">put</span></tt> are similar to the existing <tt class="docutils literal"><span class="pre">get_peers</span></tt>
and <tt class="docutils literal"><span class="pre">announce_peer</span></tt>.</p>
<p>Responses to <tt class="docutils literal"><span class="pre">get</span></tt> should always include <tt class="docutils literal"><span class="pre">nodes</span></tt> and <tt class="docutils literal"><span class="pre">nodes6</span></tt> has the same
semantics as in its <tt class="docutils literal"><span class="pre">get_peers</span></tt> response. It should also include a write token,
<tt class="docutils literal"><span class="pre">token</span></tt>, with the same semantics as <tt class="docutils literal"><span class="pre">get_peers</span></tt>.</p>
<p>The <tt class="docutils literal"><span class="pre">id</span></tt> field in these messages has the same semantics as the standard DHT messages,
i.e. the node ID of the node sending the message, to maintain the structure of the DHT
network.</p>
<p>The <tt class="docutils literal"><span class="pre">token</span></tt> field also has the same semantics as the standard DHT message <tt class="docutils literal"><span class="pre">get_peers</span></tt>
and <tt class="docutils literal"><span class="pre">announce_peer</span></tt>, when requesting an item and to write an item respectively.</p>
<p>The distinction between storing mutable and immutable items is the inclusion
of a public key, a sequence number and signature (<tt class="docutils literal"><span class="pre">k</span></tt>, <tt class="docutils literal"><span class="pre">seq</span></tt> and <tt class="docutils literal"><span class="pre">sig</span></tt>).
The distinction betwewn retrieving a mutable and immutable item is the inclusion of
the public key spill-over (<tt class="docutils literal"><span class="pre">k</span></tt>) in the <tt class="docutils literal"><span class="pre">get</span></tt> request.</p>
<p>The <tt class="docutils literal"><span class="pre">v</span></tt> key is the <em>value</em> to be stored. It is allowed to be any bencoded type (list,
dict, string or integer). When it's being hashed (for verifying its signature or to calculate
its key), its flattened, bencoded, form is used.</p>
<p>Storing nodes are SHOULD reject <tt class="docutils literal"><span class="pre">put</span></tt> requests where the bencoded form of <tt class="docutils literal"><span class="pre">v</span></tt> is longer
than 767 bytes.</p>
</div>
<div class="section" id="immutable-items">
<h1>immutable items</h1>
<p>Immutable items are stored under their SHA-1 hash, and since they cannot be modified,
there is no need to authenticate the origin of them. This makes immutable items simple.</p>
<div class="section" id="put-message">
<h2>put message</h2>
<p>Request:</p>
<pre class="literal-block">
{
&quot;a&quot;:
{
&quot;id&quot;: <em>&lt;20 byte id of sending node (string)&gt;</em>,
&quot;v&quot;: <em>&lt;any bencoded type, whose encoded size &lt; 768&gt;</em>
},
&quot;t&quot;: <em>&lt;transaction-id (string)&gt;</em>,
&quot;y&quot;: &quot;q&quot;,
&quot;q&quot;: &quot;put&quot;
}
</pre>
<p>Response:</p>
<pre class="literal-block">
{
&quot;r&quot;: { &quot;id&quot;: <em>&lt;20 byte id of sending node (string)&gt;</em> },
&quot;t&quot;: <em>&lt;transaction-id (string)&gt;</em>,
&quot;y&quot;: &quot;r&quot;,
}
</pre>
</div>
<div class="section" id="get-message">
<h2>get message</h2>
<p>Request:</p>
<pre class="literal-block">
{
&quot;a&quot;:
{
&quot;id&quot;: <em>&lt;20 byte id of sending node (string)&gt;</em>,
&quot;target&quot;: <em>&lt;SHA-1 hash of item (string)&gt;</em>,
},
&quot;t&quot;: <em>&lt;transaction-id (string)&gt;</em>,
&quot;y&quot;: &quot;q&quot;,
&quot;q&quot;: &quot;get&quot;
}
</pre>
<p>Response:</p>
<pre class="literal-block">
{
&quot;r&quot;:
{
&quot;id&quot;: <em>&lt;20 byte id of sending node (string)&gt;</em>,
&quot;token&quot;: <em>&lt;write token (string)&gt;</em>,
&quot;v&quot;: <em>&lt;any bencoded type whose SHA-1 hash matches 'target'&gt;</em>,
&quot;nodes&quot;: <em>&lt;IPv4 nodes close to 'target'&gt;</em>
&quot;nodes6&quot;: <em>&lt;IPv6 nodes close to 'target'&gt;</em>
},
&quot;t&quot;: <em>&lt;transaction-id&gt;</em>,
&quot;y&quot;: &quot;r&quot;,
}
</pre>
</div>
</div>
<div class="section" id="mutable-items">
<h1>mutable items</h1>
<p>Mutable items can be updated, without changing their DHT keys. To authenticate
that only the original publisher can update an item, it is signed by a private key
generated by the original publisher.</p>
<p>In order to avoid a malicious node to overwrite the list head with an old
version, the sequence number <tt class="docutils literal"><span class="pre">seq</span></tt> must be monotonically increasing for each update,
and a node hosting the list node MUST not downgrade a list head from a higher sequence
number to a lower one, only upgrade.</p>
<p>The signature is a 2048 bit RSA signature of the SHA-1 hash of the bencoded sequence
number and <tt class="docutils literal"><span class="pre">v</span></tt> key. e.g. something like this:: <tt class="docutils literal"><span class="pre">3:seqi4e1:v12:Hello</span> <span class="pre">world!</span></tt>.</p>
<div class="section" id="id1">
<h2>put message</h2>
<p>Request:</p>
<pre class="literal-block">
{
&quot;a&quot;:
{
&quot;id&quot;: <em>&lt;20 byte id of sending node (string)&gt;</em>,
&quot;k&quot;: <em>&lt;RSA-2048 public key (268 bytes string)&gt;</em>,
&quot;seq&quot;: <em>&lt;monotonically increasing sequence number (integer)&gt;</em>,
&quot;sig&quot;: <em>&lt;RSA-2048 signature (256 bytes string)&gt;</em>,
&quot;token&quot;: <em>&lt;write-token (string)&gt;</em>,
&quot;v&quot;: <em>&lt;any bencoded type, whose encoded size &lt; 768&gt;</em>
},
&quot;t&quot;: <em>&lt;transaction-id (string)&gt;</em>,
&quot;y&quot;: &quot;q&quot;,
&quot;q&quot;: &quot;put&quot;
}
</pre>
<p>Storing nodes receiving a <tt class="docutils literal"><span class="pre">put</span></tt> request where <tt class="docutils literal"><span class="pre">seq</span></tt> is lower than what's already
stored on the node, MUST reject the request.</p>
<p>Response:</p>
<pre class="literal-block">
{
&quot;r&quot;: { &quot;id&quot;: <em>&lt;20 byte id of sending node (string)&gt;</em> },
&quot;t&quot;: <em>&lt;transaction-id (string)&gt;</em>,
&quot;y&quot;: &quot;r&quot;,
}
</pre>
</div>
<div class="section" id="id2">
<h2>get message</h2>
<p>Request:</p>
<pre class="literal-block">
{
&quot;r&quot;:
{
&quot;id&quot;: <em>&lt;20 byte id of sending node (string)&gt;</em>,
&quot;target:&quot; <em>&lt;first 20 bytes of public key (string)&gt;</em>,
&quot;k&quot;: <em>&lt;remaining 248 bytes of public key (string)&gt;</em>
},
&quot;t&quot;: <em>&lt;transaction-id (string)&gt;</em>,
&quot;y&quot;: &quot;r&quot;,
&quot;q&quot;: &quot;get&quot;
}
</pre>
<p>Response:</p>
<pre class="literal-block">
{
&quot;r&quot;:
{
&quot;id&quot;: <em>&lt;20 byte id of sending node (string)&gt;</em>,
&quot;k&quot;: <em>&lt;RSA-2048 public key (268 bytes string)&gt;</em>,
&quot;seq&quot;: <em>&lt;monotonically increasing sequence number (integer)&gt;</em>,
&quot;sig&quot;: <em>&lt;RSA-2048 signature (256 bytes string)&gt;</em>,
&quot;token&quot;: <em>&lt;write-token (string)&gt;</em>,
&quot;v&quot;: <em>&lt;any bencoded type, whose encoded size &lt; 768&gt;</em>
},
&quot;t&quot;: <em>&lt;transaction-id (string)&gt;</em>,
&quot;y&quot;: &quot;r&quot;,
}
</pre>
</div>
</div>
<div class="section" id="signature-verification">
<h1>signature verification</h1>
<p>In order to make it maximally difficult to attack the bencoding parser, signing and verification of the
value and sequence number should be done as follows:</p>
<ol class="arabic simple">
<li>encode value and sequence number separately</li>
<li>concatenate &quot;3:seqi&quot; <tt class="docutils literal"><span class="pre">seq</span></tt> &quot;e1:v&quot; and the encoded value.
sequence number 1 of value &quot;Hello World!&quot; would be converted to: 3:seqi1e1:v12:Hello World!
In this way it is not possible to convince a node that part of the length is actually part of the
sequence number even if the parser contains certain bugs. Furthermore it is not possible to have a
verification failure if a bencoding serializer alters the order of entries in the dictionary.</li>
<li>hash the concatenated string with SHA-1</li>
<li>sign or verify the hash digest.</li>
</ol>
</div>
<div class="section" id="expiration">
<h1>expiration</h1>
<p>Without re-announcement, these items MAY expire in 2 hours. In order
to keep items alive, they SHOULD be re-announced once an hour.</p>
<p>Subscriber nodes MAY help out in announcing items the are interested in to the DHT,
to keep them alive.</p>
</div>
<div class="section" id="test-vectors">
<h1>test vectors</h1>
</div>
</div>
<div id="footer">
<span>Copyright &copy; 2005 Rasterbar Software.</span>
</div>
</div>
<script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
</script>
<script type="text/javascript">
_uacct = "UA-1599045-1";
urchinTracker();
</script>
</div>
</body>
</html>

244
docs/dht_store.rst Normal file
View File

@ -0,0 +1,244 @@
============================================
BitTorrent extension for arbitrary DHT store
============================================
:Author: Arvid Norberg, arvid@rasterbar.com
:Version: Draft
.. contents:: Table of contents
:depth: 2
:backlinks: none
This is a proposal for an extension to the BitTorrent DHT to allow
storing and retrieving of arbitrary data.
It supports both storing *immutable* items, where the key is
the SHA-1 hash of the data itself, and *mutable* items, where
the key is the public key of the key pair used to sign the data.
There are two new proposed messages, ``put`` and ``get``.
terminology
-----------
In this document, a *storage node* refers to the node in the DHT to which
an item is being announced and stored on. A *subscribing node* refers to
a node which makes look-ups in the DHT to find the storage nodes, to
request items from them, and possibly re-announce those items to keep them
alive.
messages
--------
The proposed new messages ``get`` and ``put`` are similar to the existing ``get_peers``
and ``announce_peer``.
Responses to ``get`` should always include ``nodes`` and ``nodes6`` has the same
semantics as in its ``get_peers`` response. It should also include a write token,
``token``, with the same semantics as ``get_peers``.
The ``id`` field in these messages has the same semantics as the standard DHT messages,
i.e. the node ID of the node sending the message, to maintain the structure of the DHT
network.
The ``token`` field also has the same semantics as the standard DHT message ``get_peers``
and ``announce_peer``, when requesting an item and to write an item respectively.
The distinction between storing mutable and immutable items is the inclusion
of a public key, a sequence number and signature (``k``, ``seq`` and ``sig``).
The distinction betwewn retrieving a mutable and immutable item is the inclusion of
the public key spill-over (``k``) in the ``get`` request.
The ``v`` key is the *value* to be stored. It is allowed to be any bencoded type (list,
dict, string or integer). When it's being hashed (for verifying its signature or to calculate
its key), its flattened, bencoded, form is used.
Storing nodes are SHOULD reject ``put`` requests where the bencoded form of ``v`` is longer
than 767 bytes.
immutable items
---------------
Immutable items are stored under their SHA-1 hash, and since they cannot be modified,
there is no need to authenticate the origin of them. This makes immutable items simple.
put message
...........
Request:
.. parsed-literal::
{
"a":
{
"id": *<20 byte id of sending node (string)>*,
"v": *<any bencoded type, whose encoded size < 768>*
},
"t": *<transaction-id (string)>*,
"y": "q",
"q": "put"
}
Response:
.. parsed-literal::
{
"r": { "id": *<20 byte id of sending node (string)>* },
"t": *<transaction-id (string)>*,
"y": "r",
}
get message
...........
Request:
.. parsed-literal::
{
"a":
{
"id": *<20 byte id of sending node (string)>*,
"target": *<SHA-1 hash of item (string)>*,
},
"t": *<transaction-id (string)>*,
"y": "q",
"q": "get"
}
Response:
.. parsed-literal::
{
"r":
{
"id": *<20 byte id of sending node (string)>*,
"token": *<write token (string)>*,
"v": *<any bencoded type whose SHA-1 hash matches 'target'>*,
"nodes": *<IPv4 nodes close to 'target'>*
"nodes6": *<IPv6 nodes close to 'target'>*
},
"t": *<transaction-id>*,
"y": "r",
}
mutable items
-------------
Mutable items can be updated, without changing their DHT keys. To authenticate
that only the original publisher can update an item, it is signed by a private key
generated by the original publisher.
In order to avoid a malicious node to overwrite the list head with an old
version, the sequence number ``seq`` must be monotonically increasing for each update,
and a node hosting the list node MUST not downgrade a list head from a higher sequence
number to a lower one, only upgrade.
The signature is a 2048 bit RSA signature of the SHA-1 hash of the bencoded sequence
number and ``v`` key. e.g. something like this:: ``3:seqi4e1:v12:Hello world!``.
put message
...........
Request:
.. parsed-literal::
{
"a":
{
"id": *<20 byte id of sending node (string)>*,
"k": *<RSA-2048 public key (268 bytes string)>*,
"seq": *<monotonically increasing sequence number (integer)>*,
"sig": *<RSA-2048 signature (256 bytes string)>*,
"token": *<write-token (string)>*,
"v": *<any bencoded type, whose encoded size < 768>*
},
"t": *<transaction-id (string)>*,
"y": "q",
"q": "put"
}
Storing nodes receiving a ``put`` request where ``seq`` is lower than what's already
stored on the node, MUST reject the request.
Response:
.. parsed-literal::
{
"r": { "id": *<20 byte id of sending node (string)>* },
"t": *<transaction-id (string)>*,
"y": "r",
}
get message
...........
Request:
.. parsed-literal::
{
"r":
{
"id": *<20 byte id of sending node (string)>*,
"target:" *<first 20 bytes of public key (string)>*,
"k": *<remaining 248 bytes of public key (string)>*
},
"t": *<transaction-id (string)>*,
"y": "r",
"q": "get"
}
Response:
.. parsed-literal::
{
"r":
{
"id": *<20 byte id of sending node (string)>*,
"k": *<RSA-2048 public key (268 bytes string)>*,
"seq": *<monotonically increasing sequence number (integer)>*,
"sig": *<RSA-2048 signature (256 bytes string)>*,
"token": *<write-token (string)>*,
"v": *<any bencoded type, whose encoded size < 768>*
},
"t": *<transaction-id (string)>*,
"y": "r",
}
signature verification
----------------------
In order to make it maximally difficult to attack the bencoding parser, signing and verification of the
value and sequence number should be done as follows:
1. encode value and sequence number separately
2. concatenate "3:seqi" ``seq`` "e1:v" and the encoded value.
sequence number 1 of value "Hello World!" would be converted to: 3:seqi1e1:v12:Hello World!
In this way it is not possible to convince a node that part of the length is actually part of the
sequence number even if the parser contains certain bugs. Furthermore it is not possible to have a
verification failure if a bencoding serializer alters the order of entries in the dictionary.
3. hash the concatenated string with SHA-1
4. sign or verify the hash digest.
expiration
----------
Without re-announcement, these items MAY expire in 2 hours. In order
to keep items alive, they SHOULD be re-announced once an hour.
Subscriber nodes MAY help out in announcing items the are interested in to the DHT,
to keep them alive.
test vectors
------------

1
docs/makefile
View File

@ -5,6 +5,7 @@ WEB_PATH = ~/Documents/rasterbar/web/products/libtorrent
TARGETS = index \
udp_tracker_protocol \
dht_rss \
dht_store \
client_test \
manual \
building \

15
include/libtorrent/kademlia/node.hpp
View File

@ -130,6 +130,19 @@ struct dht_immutable_item
int size;
};
struct dht_mutable_item : dht_immutable_item
{
char sig[256];
int seq;
};
struct rsa_key { char bytes[268]; };
inline bool operator<(rsa_key const& lhs, rsa_key const& rhs)
{
return memcmp(lhs.bytes, rhs.bytes, sizeof(lhs.bytes)) < 0;
}
inline bool operator<(peer_entry const& lhs, peer_entry const& rhs)
{
return lhs.addr.address() == rhs.addr.address()
@ -165,6 +178,7 @@ class node_impl : boost::noncopyable
{
typedef std::map<node_id, torrent_entry> table_t;
typedef std::map<node_id, dht_immutable_item> dht_immutable_table_t;
typedef std::map<rsa_key, dht_mutable_item> dht_mutable_table_t;
public:
typedef boost::function3<void, address, int, address> external_ip_fun;
@ -277,6 +291,7 @@ public:
private:
table_t m_map;
dht_immutable_table_t m_immutable_table;
dht_mutable_table_t m_mutable_table;
ptime m_last_tracker_tick;

5
include/libtorrent/rsa.hpp
View File

@ -34,18 +34,19 @@ POSSIBILITY OF SUCH DAMAGE.
#define TORRENT_SIGN_HPP_INCLUDED
#include "libtorrent/config.hpp"
#include "libtorrent/hasher.hpp"
namespace libtorrent
{
// both of these use SHA-1 as the message digest to be signed/verified
// returns the size of the resulting signature
TORRENT_EXPORT int sign_rsa(char const* data, int data_len
TORRENT_EXPORT int sign_rsa(sha1_hash const& digest
, char const* private_key, int private_len
, char* signature, int sig_len);
// returns true if the signature is valid
TORRENT_EXPORT bool verify_rsa(char const* data, int data_len
TORRENT_EXPORT bool verify_rsa(sha1_hash const& digest
, char const* public_key, int public_len
, char const* signature, int sig_len);

196
src/kademlia/node.cpp
View File

@ -51,6 +51,7 @@ POSSIBILITY OF SUCH DAMAGE.
#include "libtorrent/kademlia/refresh.hpp"
#include "libtorrent/kademlia/find_data.hpp"
#include "libtorrent/rsa.hpp"
namespace libtorrent { namespace dht
{
@ -786,19 +787,31 @@ void node_impl::incoming_request(msg const& m, entry& e)
}
else if (strcmp(query, "put") == 0)
{
// the first 2 entries are for both mutable and
// immutable puts
const static key_desc_t msg_desc[] = {
{"token", lazy_entry::string_t, 0, 0},
{"v", lazy_entry::none_t, 0, 0},
{"seq", lazy_entry::int_t, 0, 0},
// public key
{"k", lazy_entry::string_t, 268, 0},
{"sig", lazy_entry::string_t, 256, 0},
};
// attempt to parse the message
lazy_entry const* msg_keys[2];
lazy_entry const* msg_keys[5];
if (!verify_message(arg_ent, msg_desc, msg_keys, 2, error_string, sizeof(error_string)))
{
incoming_error(e, error_string);
return;
}
bool mutable_put = false;
// is this a mutable put?
if (verify_message(arg_ent, msg_desc, msg_keys, 5, error_string, sizeof(error_string)))
mutable_put = true;
// pointer and length to the whole entry
std::pair<char const*, int> buf = msg_keys[1]->data_section();
if (buf.second > 767 || buf.second <= 0)
@ -807,7 +820,15 @@ void node_impl::incoming_request(msg const& m, entry& e)
return;
}
sha1_hash target = hasher(buf.first, buf.second).final();
sha1_hash target;
if (!mutable_put)
target = hasher(buf.first, buf.second).final();
else
target = sha1_hash(msg_keys[3]->string_ptr());
// fprintf(stderr, "%s PUT target: %s\n"
// , mutable_put ? "mutable":"immutable"
// , to_hex(target.to_string()).c_str());
// verify the write-token. tokens are only valid to write to
// specific target hashes. it must match the one we got a "get" for
@ -817,50 +838,134 @@ void node_impl::incoming_request(msg const& m, entry& e)
return;
}
dht_immutable_table_t::iterator i = m_immutable_table.find(target);
if (i == m_immutable_table.end())
{
// make sure we don't add too many items
if (int(m_immutable_table.size()) >= m_settings.max_dht_items)
{
// delete the least important one (i.e. the one
// the fewest peers are announcing)
dht_immutable_table_t::iterator j = std::min_element(m_immutable_table.begin()
, m_immutable_table.end()
, boost::bind(&dht_immutable_item::num_announcers
, boost::bind(&dht_immutable_table_t::value_type::second, _1)));
TORRENT_ASSERT(j != m_immutable_table.end());
m_immutable_table.erase(j);
}
dht_immutable_item to_add;
to_add.value = (char*)malloc(buf.second);
to_add.size = buf.second;
memcpy(to_add.value, buf.first, buf.second);
boost::tie(i, boost::tuples::ignore) = m_immutable_table.insert(
std::make_pair(target, to_add));
}
dht_immutable_item* f = 0;
dht_immutable_item& f = i->second;
if (!mutable_put)
{
dht_immutable_table_t::iterator i = m_immutable_table.find(target);
if (i == m_immutable_table.end())
{
// make sure we don't add too many items
if (int(m_immutable_table.size()) >= m_settings.max_dht_items)
{
// delete the least important one (i.e. the one
// the fewest peers are announcing)
dht_immutable_table_t::iterator j = std::min_element(m_immutable_table.begin()
, m_immutable_table.end()
, boost::bind(&dht_immutable_item::num_announcers
, boost::bind(&dht_immutable_table_t::value_type::second, _1)));
TORRENT_ASSERT(j != m_immutable_table.end());
free(j->second.value);
m_immutable_table.erase(j);
}
dht_immutable_item to_add;
to_add.value = (char*)malloc(buf.second);
to_add.size = buf.second;
memcpy(to_add.value, buf.first, buf.second);
boost::tie(i, boost::tuples::ignore) = m_immutable_table.insert(
std::make_pair(target, to_add));
}
// fprintf(stderr, "added immutable item (%d)\n", int(m_immutable_table.size()));
f = &i->second;
}
else
{
// mutable put, we must verify the signature
// generate the message digest by merging the sequence number and the
hasher digest;
char seq[20];
int len = snprintf(seq, sizeof(seq), "3:seqi%"PRId64"e1:v", msg_keys[2]->int_value());
digest.update(seq, len);
std::pair<char const*, int> buf = msg_keys[1]->data_section();
digest.update(buf.first, buf.second);
if (!verify_rsa(digest.final(), msg_keys[3]->string_ptr(), msg_keys[3]->string_length()
, msg_keys[4]->string_ptr(), msg_keys[4]->string_length()))
{
incoming_error(e, "invalid signature");
return;
}
rsa_key target;
memcpy(target.bytes, msg_keys[3]->string_ptr(), sizeof(target.bytes));
dht_mutable_table_t::iterator i = m_mutable_table.find(target);
if (i == m_mutable_table.end())
{
// make sure we don't add too many items
if (int(m_mutable_table.size()) >= m_settings.max_dht_items)
{
// delete the least important one (i.e. the one
// the fewest peers are announcing)
dht_mutable_table_t::iterator j = std::min_element(m_mutable_table.begin()
, m_mutable_table.end()
, boost::bind(&dht_immutable_item::num_announcers
, boost::bind(&dht_mutable_table_t::value_type::second, _1)));
TORRENT_ASSERT(j != m_mutable_table.end());
free(j->second.value);
m_mutable_table.erase(j);
}
dht_mutable_item to_add;
to_add.value = (char*)malloc(buf.second);
to_add.size = buf.second;
to_add.seq = msg_keys[2]->int_value();
memcpy(to_add.sig, msg_keys[4]->string_ptr(), sizeof(to_add.sig));
TORRENT_ASSERT(sizeof(to_add.sig) == msg_keys[4]->string_length());
memcpy(to_add.value, buf.first, buf.second);
boost::tie(i, boost::tuples::ignore) = m_mutable_table.insert(
std::make_pair(target, to_add));
// fprintf(stderr, "added mutable item (%d)\n", int(m_mutable_table.size()));
}
else
{
dht_mutable_item* item = &i->second;
if (item->seq > msg_keys[2]->int_value())
{
incoming_error(e, "old sequence number");
return;
}
if (item->seq < msg_keys[2]->int_value())
{
if (item->size != buf.second)
{
free(item->value);
item->value = (char*)malloc(buf.second);
item->size = buf.second;
}
item->seq = msg_keys[2]->int_value();
memcpy(item->sig, msg_keys[4]->string_ptr(), sizeof(item->sig));
TORRENT_ASSERT(sizeof(item->sig) == msg_keys[4]->string_length());
memcpy(item->value, buf.first, buf.second);
}
}
f = &i->second;
}
m_table.node_seen(id, m.addr);
f.last_seen = time_now();
f->last_seen = time_now();
// maybe increase num_announcers if we haven't seen this IP before
sha1_hash iphash;
hash_address(m.addr.address(), iphash);
if (!f.ips.find(iphash))
if (!f->ips.find(iphash))
{
f.ips.set(iphash);
++f.num_announcers;
f->ips.set(iphash);
++f->num_announcers;
}
}
else if (strcmp(query, "get") == 0)
{
key_desc_t msg_desc[] = {
{"target", lazy_entry::string_t, 20, 0},
{"k", lazy_entry::string_t, 0, key_desc_t::optional},
{"k", lazy_entry::string_t, 268-20, key_desc_t::optional},
};
// k is not used for now
@ -875,6 +980,10 @@ void node_impl::incoming_request(msg const& m, entry& e)
sha1_hash target(msg_keys[0]->string_ptr());
// fprintf(stderr, "%s GET target: %s\n"
// , msg_keys[1] ? "mutable":"immutable"
// , to_hex(target.to_string()).c_str());
reply["token"] = generate_token(m.addr, msg_keys[0]->string_ptr());
nodes_t n;
@ -882,11 +991,28 @@ void node_impl::incoming_request(msg const& m, entry& e)
m_table.find_node(target, n, 0);
write_nodes_entry(reply, n);
dht_immutable_table_t::iterator i = m_immutable_table.find(target);
if (i != m_immutable_table.end())
if (msg_keys[1])
{
dht_immutable_item const& f = i->second;
reply["v"] = bdecode(f.value, f.value + f.size);
rsa_key key;
memcpy(key.bytes, msg_keys[0]->string_ptr(), 20);
memcpy(key.bytes + 20, msg_keys[1]->string_ptr(), 268-20);
dht_mutable_table_t::iterator i = m_mutable_table.find(key);
if (i != m_mutable_table.end())
{
dht_mutable_item const& f = i->second;
reply["v"] = bdecode(f.value, f.value + f.size);
reply["seq"] = f.seq;
reply["sig"] = std::string(f.sig, f.sig + 256);
}
}
else
{
dht_immutable_table_t::iterator i = m_immutable_table.find(target);
if (i != m_immutable_table.end())
{
dht_immutable_item const& f = i->second;
reply["v"] = bdecode(f.value, f.value + f.size);
}
}
}
else

12
src/rsa.cpp
View File

@ -45,7 +45,7 @@ namespace libtorrent
{
// returns the size of the resulting signature
int sign_rsa(char const* data, int data_len
int sign_rsa(sha1_hash const& digest
, char const* private_key, int private_len
, char* signature, int sig_len)
{
@ -54,6 +54,7 @@ int sign_rsa(char const* data, int data_len
RSA* priv = 0;
unsigned char const* key = (unsigned char const*)private_key;
priv = d2i_RSAPrivateKey(&priv, &key, private_len);
if (priv == 0) return -1;
if (RSA_size(priv) > sig_len)
{
@ -61,9 +62,6 @@ int sign_rsa(char const* data, int data_len
return -1;
}
// hash the data
sha1_hash digest = hasher(data, data_len).final();
RSA_sign(NID_sha1, &digest[0], 20, (unsigned char*)signature, (unsigned int*)&sig_len, priv);
RSA_free(priv);
@ -72,7 +70,7 @@ int sign_rsa(char const* data, int data_len
}
// returns true if the signature is valid
bool verify_rsa(char const* data, int data_len
bool verify_rsa(sha1_hash const& digest
, char const* public_key, int public_len
, char const* signature, int sig_len)
{
@ -81,9 +79,7 @@ bool verify_rsa(char const* data, int data_len
RSA* pub = 0;
unsigned char const* key = (unsigned char const*)public_key;
pub = d2i_RSAPublicKey(&pub, &key, public_len);
// hash the data
sha1_hash digest = hasher(data, data_len).final();
if (pub == 0) return false;
int ret = RSA_verify(NID_sha1, &digest[0], 20, (unsigned char*)signature, sig_len, pub);

141
test/test_dht.cpp
View File

@ -36,6 +36,7 @@ POSSIBILITY OF SUCH DAMAGE.
#include "libtorrent/kademlia/node.hpp" // for verify_message
#include "libtorrent/bencode.hpp"
#include "libtorrent/socket_io.hpp" // for hash_address
#include "libtorrent/rsa.hpp" // for generate_rsa_keys and sign_rsa
#include <iostream>
#include "test.hpp"
@ -75,9 +76,10 @@ static const std::string no;
void send_dht_msg(node_impl& node, char const* msg, udp::endpoint const& ep
, lazy_entry* reply, char const* t = "10", char const* info_hash = 0
, char const* name = 0, std::string const token = std::string(), int port = 0
, std::string const target = std::string(), entry const* value = 0
, std::string const id = std::string()
, bool scrape = false, bool seed = false)
, char const* target = 0, entry const* value = 0
, bool scrape = false, bool seed = false
, std::string const key = std::string(), std::string const sig = std::string()
, int seq = -1)
{
// we're about to clear out the backing buffer
// for this lazy_entry, so we better clear it now
@ -87,15 +89,18 @@ void send_dht_msg(node_impl& node, char const* msg, udp::endpoint const& ep
e["t"] = t;
e["y"] = "q";
entry::dictionary_type& a = e["a"].dict();
a["id"] = id.empty() ? generate_next().to_string() : id;
if (info_hash) a["info_hash"] = info_hash;
a["id"] = generate_next().to_string();
if (info_hash) a["info_hash"] = std::string(info_hash, 20);
if (name) a["n"] = name;
if (!token.empty()) a["token"] = token;
if (port) a["port"] = port;
if (!target.empty()) a["target"] = target;
if (target) a["target"] = std::string(target, 20);
if (value) a["v"] = *value;
if (!sig.empty()) a["sig"] = sig;
if (!key.empty()) a["k"] = key;
if (scrape) a["scrape"] = 1;
if (seed) a["seed"] = 1;
if (seq >= 0) a["seq"] = seq;
char msg_buf[1500];
int size = bencode(msg_buf, e);
// std::cerr << "sending: " << e << "\n";
@ -135,6 +140,7 @@ struct announce_item
sha1_hash target;
void gen()
{
num_peers = (rand() % 5) + 1;
ent["next"] = next.to_string();
ent["A"] = "a";
ent["B"] = "b";
@ -147,10 +153,10 @@ struct announce_item
}
};
void announce_items(node_impl& node, udp::endpoint const* eps
, node_id const* ids, announce_item const* items, int num_items)
void announce_immutable_items(node_impl& node, udp::endpoint const* eps
, announce_item const* items, int num_items)
{
std::string tokens[1000];
std::string token;
for (int i = 0; i < 1000; ++i)
{
for (int j = 0; j < num_items; ++j)
@ -158,7 +164,7 @@ void announce_items(node_impl& node, udp::endpoint const* eps
if ((i % items[j].num_peers) == 0) continue;
lazy_entry response;
send_dht_msg(node, "get", eps[i], &response, "10", 0
, 0, no, 0, items[j].target.to_string());
, 0, no, 0, (char const*)&items[j].target[0]);
key_desc_t desc[] =
{
@ -172,12 +178,12 @@ void announce_items(node_impl& node, udp::endpoint const* eps
lazy_entry const* parsed[5];
char error_string[200];
fprintf(stderr, "msg: %s\n", print_entry(response).c_str());
// fprintf(stderr, "msg: %s\n", print_entry(response).c_str());
int ret = verify_message(&response, desc, parsed, 5, error_string, sizeof(error_string));
if (ret)
{
TEST_EQUAL(parsed[4]->string_value(), "r");
tokens[i] = parsed[2]->string_value();
token = parsed[2]->string_value();
}
else
{
@ -194,8 +200,7 @@ void announce_items(node_impl& node, udp::endpoint const* eps
}
send_dht_msg(node, "put", eps[i], &response, "10", 0
, 0, tokens[i], 0, items[j].target.to_string(), &items[j].ent
, ids[i].to_string());
, 0, token, 0, (char const*)&items[j].target[0], &items[j].ent);
key_desc_t desc2[] =
{
@ -220,35 +225,24 @@ void announce_items(node_impl& node, udp::endpoint const* eps
for (int j = 0; j < num_items; ++j)
{
lazy_entry response;
send_dht_msg(node, "get", eps[0], &response, "10", 0
, 0, no, 0, items[j].target.to_string(), 0
, ids[0].to_string());
send_dht_msg(node, "get", eps[j], &response, "10", 0
, 0, no, 0, (char const*)&items[j].target[0]);
key_desc_t desc[] =
{
{ "r", lazy_entry::dict_t, 0, key_desc_t::parse_children },
{ "v", lazy_entry::dict_t, 0, key_desc_t::parse_children},
{ "A", lazy_entry::string_t, 1, 0},
{ "B", lazy_entry::string_t, 1, 0},
{ "num_peers", lazy_entry::int_t, 0, key_desc_t::last_child},
{ "v", lazy_entry::dict_t, 0, 0},
{ "id", lazy_entry::string_t, 20, key_desc_t::last_child},
{ "y", lazy_entry::string_t, 1, 0},
};
lazy_entry const* parsed[7];
lazy_entry const* parsed[4];
char error_string[200];
int ret = verify_message(&response, desc, parsed, 7, error_string, sizeof(error_string));
int ret = verify_message(&response, desc, parsed, 4, error_string, sizeof(error_string));
if (ret)
{
TEST_EQUAL(parsed[6]->string_value(), "r");
TEST_EQUAL(parsed[2]->string_value(), "a");
TEST_EQUAL(parsed[3]->string_value(), "b");
items_num.insert(items_num.begin(), parsed[4]->int_value());
}
else
{
fprintf(stderr, "unexpected msg: %s\n", print_entry(response).c_str());
items_num.insert(items_num.begin(), j);
}
}
@ -401,14 +395,14 @@ int test_main()
}
response.clear();
send_dht_msg(node, "announce_peer", source, &response, "10", "01010101010101010101"
, "test", token, 8080, std::string(), 0, std::string(), false, i >= 50);
, "test", token, 8080, 0, 0, false, i >= 50);
response.clear();
}
// ====== get_peers ======
send_dht_msg(node, "get_peers", source, &response, "10", "01010101010101010101"
, 0, std::string(), 0, std::string(), 0, std::string(), true);
, 0, no, 0, 0, 0, true);
dht::key_desc_t peer2_desc[] = {
{"y", lazy_entry::string_t, 1, 0},
@ -463,6 +457,8 @@ int test_main()
test.set(iphash);
}
// these are test vectors from BEP 33
// http://www.bittorrent.org/beps/bep_0033.html
fprintf(stderr, "test.size: %f\n", test.size());
TEST_CHECK(fabs(test.size() - 1224.93f) < 0.001);
fprintf(stderr, "%s\n", to_hex(test.to_string()).c_str());
@ -473,13 +469,9 @@ int test_main()
// ====== put ======
udp::endpoint eps[1000];
node_id ids[1000];
for (int i = 0; i < 1000; ++i)
{
eps[i] = udp::endpoint(rand_v4(), (rand() % 16534) + 1);
ids[i] = generate_next();
}
announce_item items[] =
{
@ -496,7 +488,80 @@ int test_main()
for (int i = 0; i < sizeof(items)/sizeof(items[0]); ++i)
items[i].gen();
announce_items(node, eps, ids, items, sizeof(items)/sizeof(items[0]));
announce_immutable_items(node, eps, items, sizeof(items)/sizeof(items[0]));
// ==== get / put mutable items ===
char private_key[1192];
int private_len = sizeof(private_key);
char public_key[268];
int public_len = sizeof(public_key);
fprintf(stderr, "generating RSA keys\n");
ret = generate_rsa_keys(public_key, &public_len, private_key, &private_len, 2048);
fprintf(stderr, "pub: %d priv:%d\n", public_len, private_len);
TEST_CHECK(ret);
send_dht_msg(node, "get", source, &response, "10", 0
, 0, no, 0, public_key, 0, false, false, std::string(public_key + 20, public_len-20));
key_desc_t desc[] =
{
{ "r", lazy_entry::dict_t, 0, key_desc_t::parse_children },
{ "id", lazy_entry::string_t, 20, 0},
{ "token", lazy_entry::string_t, 0, 0},
{ "ip", lazy_entry::string_t, 0, key_desc_t::optional | key_desc_t::last_child},
{ "y", lazy_entry::string_t, 1, 0},
};
ret = verify_message(&response, desc, parsed, 5, error_string, sizeof(error_string));
if (ret)
{
TEST_EQUAL(parsed[4]->string_value(), "r");
token = parsed[2]->string_value();
}
else
{
fprintf(stderr, " invalid get response: %s\n%s\n"
, error_string, print_entry(response).c_str());
TEST_ERROR(error_string);
}
char signature[256];
int sig_len = sizeof(signature);
char buffer[1024];
int seq = 4;
int pos = snprintf(buffer, sizeof(buffer), "3:seqi%de1:v", seq);
hasher h(buffer, pos);
char* ptr = buffer;
int len = bencode(ptr, items[0].ent);
h.update(buffer, len);
sign_rsa(h.final(), private_key, private_len, signature, sig_len);
send_dht_msg(node, "put", source, &response, "10", 0
, 0, token, 0, 0, &items[0].ent, false, false
, std::string(public_key, public_len)
, std::string(signature, sig_len), seq);
key_desc_t desc2[] =
{
{ "y", lazy_entry::string_t, 1, 0 }
};
ret = verify_message(&response, desc2, parsed, 1, error_string, sizeof(error_string));
if (ret)
{
fprintf(stderr, "put response: %s\n"
, print_entry(response).c_str());
TEST_EQUAL(parsed[0]->string_value(), "r");
}
else
{
fprintf(stderr, " invalid put response: %s\n%s\n"
, error_string, print_entry(response).c_str());
TEST_ERROR(error_string);
}
return 0;
}

10
test/test_primitives.cpp
View File

@ -397,12 +397,13 @@ int test_main()
#if defined TORRENT_USE_OPENSSL
// test sign_rsa and verify_rsa
char private_key[256];
char private_key[1192];
int private_len = sizeof(private_key);
char public_key[512];
char public_key[268];
int public_len = sizeof(public_key);
ret = generate_rsa_keys(public_key, &public_len, private_key, &private_len, 2048);
fprintf(stderr, "keysizes: pub: %d priv: %d\n", public_len, private_len);
TEST_CHECK(ret);
@ -410,12 +411,13 @@ int test_main()
std::generate(test_message, test_message + 1024, &std::rand);
char signature[256];
int sig_len = sign_rsa(test_message, sizeof(test_message)
int sig_len = sign_rsa(hasher(test_message, sizeof(test_message)).final()
, private_key, private_len, signature, sizeof(signature));
TEST_CHECK(sig_len == 256);
ret = verify_rsa(test_message, sizeof(test_message), public_key, public_len, signature, sig_len);
ret = verify_rsa(hasher(test_message, sizeof(test_message)).final()
, public_key, public_len, signature, sig_len);
TEST_CHECK(ret == 1);
#endif