2011-05-25 04:26:07 +02:00
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
< html xmlns = "http://www.w3.org/1999/xhtml" xml:lang = "en" lang = "en" >
< head >
2012-01-16 03:09:07 +01:00
< meta http-equiv = "Content-Type" content = "text/html; charset=utf-8" / >
2013-07-19 18:31:52 +02:00
< meta name = "generator" content = "Docutils 0.10: http://docutils.sourceforge.net/" / >
2012-01-16 03:09:07 +01:00
< title > BitTorrent extension for arbitrary DHT store< / title >
< meta name = "author" content = "Arvid Norberg, arvid@rasterbar.com" / >
< link rel = "stylesheet" type = "text/css" href = "../../css/base.css" / >
< link rel = "stylesheet" type = "text/css" href = "../../css/rst.css" / >
2011-05-25 04:26:07 +02:00
< script type = "text/javascript" >
/* < ![CDATA[ */
(function() {
var s = document.createElement('script'), t = document.getElementsByTagName('script')[0];
s.type = 'text/javascript';
s.async = true;
s.src = 'http://api.flattr.com/js/0.6/load.js?mode=auto';
t.parentNode.insertBefore(s, t);
})();
/* ]]> */
< / script >
< link rel = "stylesheet" href = "style.css" type = "text/css" / >
< style type = "text/css" >
/* Hides from IE-mac \*/
* html pre { height: 1%; }
/* End hide from IE-mac */
< / style >
< / head >
< body >
< div class = "document" id = "bittorrent-extension-for-arbitrary-dht-store" >
< div id = "container" >
< div id = "headerNav" >
< ul >
< li class = "first" > < a href = "/" > Home< / a > < / li >
< li > < a href = "../../products.html" > Products< / a > < / li >
< li > < a href = "../../contact.html" > Contact< / a > < / li >
< / ul >
< / div >
< div id = "header" >
< h1 > < span > Rasterbar Software< / span > < / h1 >
< h2 > < span > Software developement and consulting< / span > < / h2 >
< / div >
< div id = "main" >
< h1 class = "title" > BitTorrent extension for arbitrary DHT store< / h1 >
< table class = "docinfo" frame = "void" rules = "none" >
< col class = "docinfo-name" / >
< col class = "docinfo-content" / >
< tbody valign = "top" >
< tr > < th class = "docinfo-name" > Author:< / th >
< td > Arvid Norberg, < a class = "last reference external" href = "mailto:arvid@rasterbar.com" > arvid@ rasterbar.com< / a > < / td > < / tr >
< tr > < th class = "docinfo-name" > Version:< / th >
2013-01-21 20:13:24 +01:00
< td > 1.0.0< / td > < / tr >
2011-05-25 04:26:07 +02:00
< / tbody >
< / table >
< div class = "contents topic" id = "table-of-contents" >
< p class = "topic-title first" > Table of contents< / p >
< ul class = "simple" >
< li > < a class = "reference internal" href = "#terminology" id = "id3" > terminology< / a > < / li >
< li > < a class = "reference internal" href = "#messages" id = "id4" > messages< / a > < / li >
< li > < a class = "reference internal" href = "#immutable-items" id = "id5" > immutable items< / a > < ul >
< li > < a class = "reference internal" href = "#put-message" id = "id6" > put message< / a > < / li >
< li > < a class = "reference internal" href = "#get-message" id = "id7" > get message< / a > < / li >
< / ul >
< / li >
< li > < a class = "reference internal" href = "#mutable-items" id = "id8" > mutable items< / a > < ul >
< li > < a class = "reference internal" href = "#id1" id = "id9" > put message< / a > < / li >
< li > < a class = "reference internal" href = "#id2" id = "id10" > get message< / a > < / li >
< / ul >
< / li >
< li > < a class = "reference internal" href = "#signature-verification" id = "id11" > signature verification< / a > < / li >
< li > < a class = "reference internal" href = "#expiration" id = "id12" > expiration< / a > < / li >
< li > < a class = "reference internal" href = "#test-vectors" id = "id13" > test vectors< / a > < / li >
< / ul >
< / div >
< p > This is a proposal for an extension to the BitTorrent DHT to allow
storing and retrieving of arbitrary data.< / p >
< p > It supports both storing < em > immutable< / em > items, where the key is
the SHA-1 hash of the data itself, and < em > mutable< / em > items, where
the key is the public key of the key pair used to sign the data.< / p >
2012-11-09 03:52:27 +01:00
< p > There are two new proposed messages, < tt class = "docutils literal" > put< / tt > and < tt class = "docutils literal" > get< / tt > .< / p >
2011-05-25 04:26:07 +02:00
< div class = "section" id = "terminology" >
< h1 > terminology< / h1 >
< p > In this document, a < em > storage node< / em > refers to the node in the DHT to which
an item is being announced and stored on. A < em > subscribing node< / em > refers to
a node which makes look-ups in the DHT to find the storage nodes, to
request items from them, and possibly re-announce those items to keep them
alive.< / p >
< / div >
< div class = "section" id = "messages" >
< h1 > messages< / h1 >
2012-11-09 03:52:27 +01:00
< p > The proposed new messages < tt class = "docutils literal" > get< / tt > and < tt class = "docutils literal" > put< / tt > are similar to the existing < tt class = "docutils literal" > get_peers< / tt >
and < tt class = "docutils literal" > announce_peer< / tt > .< / p >
< p > Responses to < tt class = "docutils literal" > get< / tt > should always include < tt class = "docutils literal" > nodes< / tt > and < tt class = "docutils literal" > nodes6< / tt > has the same
semantics as in its < tt class = "docutils literal" > get_peers< / tt > response. It should also include a write token,
< tt class = "docutils literal" > token< / tt > , with the same semantics as < tt class = "docutils literal" > get_peers< / tt > .< / p >
< p > The < tt class = "docutils literal" > id< / tt > field in these messages has the same semantics as the standard DHT messages,
2011-05-25 04:26:07 +02:00
i.e. the node ID of the node sending the message, to maintain the structure of the DHT
network.< / p >
2012-11-09 03:52:27 +01:00
< p > The < tt class = "docutils literal" > token< / tt > field also has the same semantics as the standard DHT message < tt class = "docutils literal" > get_peers< / tt >
and < tt class = "docutils literal" > announce_peer< / tt > , when requesting an item and to write an item respectively.< / p >
2012-11-16 23:25:39 +01:00
< p > The < tt class = "docutils literal" > k< / tt > field is the PKCS#1 encoded 2048 bit RSA public key, which the signature
can be authenticated with. When looking up a mutable item, the < tt class = "docutils literal" > target< / tt > field
MUST be the SHA-1 hash of this key.< / p >
2011-05-25 04:26:07 +02:00
< p > The distinction between storing mutable and immutable items is the inclusion
2012-11-16 23:25:39 +01:00
of a public key, a sequence number and signature (< tt class = "docutils literal" > k< / tt > , < tt class = "docutils literal" > seq< / tt > and < tt class = "docutils literal" > sig< / tt > ).< / p >
< p > < tt class = "docutils literal" > get< / tt > requests for mutable items and immutable items cannot be distinguished from
eachother. An implementation can either store mutable and immutable items in the same
hash table internally, or in separate ones and potentially do two lookups for < tt class = "docutils literal" > get< / tt >
requests.< / p >
< p > The < tt class = "docutils literal" > v< / tt > field is the < em > value< / em > to be stored. It is allowed to be any bencoded type (list,
2011-05-25 04:26:07 +02:00
dict, string or integer). When it's being hashed (for verifying its signature or to calculate
2013-01-21 20:13:24 +01:00
its key), its flattened, bencoded, form is used. It is important to use the exact
bencoded representation as it appeared in the message. decoding and then re-encoding
bencoded structures is not necessarily an identity operation.< / p >
2012-11-16 23:25:39 +01:00
< p > Storing nodes SHOULD reject < tt class = "docutils literal" > put< / tt > requests where the bencoded form of < tt class = "docutils literal" > v< / tt > is longer
2011-05-25 04:26:07 +02:00
than 767 bytes.< / p >
< / div >
< div class = "section" id = "immutable-items" >
< h1 > immutable items< / h1 >
< p > Immutable items are stored under their SHA-1 hash, and since they cannot be modified,
there is no need to authenticate the origin of them. This makes immutable items simple.< / p >
2013-01-21 20:13:24 +01:00
< p > A node making a lookup SHOULD verify the data it receives from the network, to verify
that its hash matches the target that was looked up.< / p >
2011-05-25 04:26:07 +02:00
< div class = "section" id = "put-message" >
< h2 > put message< / h2 >
< p > Request:< / p >
< pre class = "literal-block" >
{
" a" :
{
" id" : < em > < 20 byte id of sending node (string)> < / em > ,
" v" : < em > < any bencoded type, whose encoded size < 768> < / em >
},
" t" : < em > < transaction-id (string)> < / em > ,
" y" : " q" ,
" q" : " put"
}
< / pre >
< p > Response:< / p >
< pre class = "literal-block" >
{
" r" : { " id" : < em > < 20 byte id of sending node (string)> < / em > },
" t" : < em > < transaction-id (string)> < / em > ,
" y" : " r" ,
}
< / pre >
< / div >
< div class = "section" id = "get-message" >
< h2 > get message< / h2 >
< p > Request:< / p >
< pre class = "literal-block" >
{
" a" :
{
" id" : < em > < 20 byte id of sending node (string)> < / em > ,
" target" : < em > < SHA-1 hash of item (string)> < / em > ,
},
" t" : < em > < transaction-id (string)> < / em > ,
" y" : " q" ,
" q" : " get"
}
< / pre >
< p > Response:< / p >
< pre class = "literal-block" >
{
" r" :
{
" id" : < em > < 20 byte id of sending node (string)> < / em > ,
" token" : < em > < write token (string)> < / em > ,
" v" : < em > < any bencoded type whose SHA-1 hash matches 'target'> < / em > ,
2012-11-09 03:52:27 +01:00
" nodes" : < em > < IPv4 nodes close to 'target'> < / em > ,
2011-05-25 04:26:07 +02:00
" nodes6" : < em > < IPv6 nodes close to 'target'> < / em >
},
" t" : < em > < transaction-id> < / em > ,
" y" : " r" ,
}
< / pre >
< / div >
< / div >
< div class = "section" id = "mutable-items" >
< h1 > mutable items< / h1 >
< p > Mutable items can be updated, without changing their DHT keys. To authenticate
that only the original publisher can update an item, it is signed by a private key
2012-11-16 23:25:39 +01:00
generated by the original publisher. The target ID mutable items are stored under
is the SHA-1 hash of the public key (as it appears in the < tt class = "docutils literal" > put< / tt > message).< / p >
2011-05-25 04:26:07 +02:00
< p > In order to avoid a malicious node to overwrite the list head with an old
2012-11-09 03:52:27 +01:00
version, the sequence number < tt class = "docutils literal" > seq< / tt > must be monotonically increasing for each update,
2011-05-25 04:26:07 +02:00
and a node hosting the list node MUST not downgrade a list head from a higher sequence
2013-01-21 20:13:24 +01:00
number to a lower one, only upgrade. The sequence number SHOULD not exceed < tt class = "docutils literal" > MAX_INT64< / tt > ,
(i.e. < tt class = "docutils literal" > 0x7fffffffffffffff< / tt > . A client MAY reject any message with a sequence number
exceeding this.< / p >
2011-05-25 04:26:07 +02:00
< p > The signature is a 2048 bit RSA signature of the SHA-1 hash of the bencoded sequence
2012-11-09 03:52:27 +01:00
number and < tt class = "docutils literal" > v< / tt > key. e.g. something like this:: < tt class = "docutils literal" > 3:seqi4e1:v12:Hello world!< / tt > .< / p >
2011-05-25 04:26:07 +02:00
< div class = "section" id = "id1" >
< h2 > put message< / h2 >
< p > Request:< / p >
< pre class = "literal-block" >
{
" a" :
{
" id" : < em > < 20 byte id of sending node (string)> < / em > ,
2012-11-16 23:25:39 +01:00
" k" : < em > < RSA-2048 public key (PKCS#1 encoded)> < / em > ,
2011-05-25 04:26:07 +02:00
" seq" : < em > < monotonically increasing sequence number (integer)> < / em > ,
" sig" : < em > < RSA-2048 signature (256 bytes string)> < / em > ,
" token" : < em > < write-token (string)> < / em > ,
" v" : < em > < any bencoded type, whose encoded size < 768> < / em >
},
" t" : < em > < transaction-id (string)> < / em > ,
" y" : " q" ,
" q" : " put"
}
< / pre >
2012-11-09 03:52:27 +01:00
< p > Storing nodes receiving a < tt class = "docutils literal" > put< / tt > request where < tt class = "docutils literal" > seq< / tt > is lower than what's already
2011-05-25 04:26:07 +02:00
stored on the node, MUST reject the request.< / p >
< p > Response:< / p >
< pre class = "literal-block" >
{
" r" : { " id" : < em > < 20 byte id of sending node (string)> < / em > },
" t" : < em > < transaction-id (string)> < / em > ,
" y" : " r" ,
}
< / pre >
< / div >
< div class = "section" id = "id2" >
< h2 > get message< / h2 >
< p > Request:< / p >
< pre class = "literal-block" >
{
2012-11-09 03:52:27 +01:00
" a" :
2011-05-25 04:26:07 +02:00
{
" id" : < em > < 20 byte id of sending node (string)> < / em > ,
2012-11-16 23:25:39 +01:00
" target:" < em > < 20 byte SHA-1 hash of public key (string)> < / em >
2011-05-25 04:26:07 +02:00
},
" t" : < em > < transaction-id (string)> < / em > ,
2012-11-09 03:52:27 +01:00
" y" : " q" ,
2011-05-25 04:26:07 +02:00
" q" : " get"
}
< / pre >
< p > Response:< / p >
< pre class = "literal-block" >
{
" r" :
{
" id" : < em > < 20 byte id of sending node (string)> < / em > ,
" k" : < em > < RSA-2048 public key (268 bytes string)> < / em > ,
2012-11-09 03:52:27 +01:00
" nodes" : < em > < IPv4 nodes close to 'target'> < / em > ,
" nodes6" : < em > < IPv6 nodes close to 'target'> < / em > ,
2011-05-25 04:26:07 +02:00
" seq" : < em > < monotonically increasing sequence number (integer)> < / em > ,
" sig" : < em > < RSA-2048 signature (256 bytes string)> < / em > ,
" token" : < em > < write-token (string)> < / em > ,
" v" : < em > < any bencoded type, whose encoded size < 768> < / em >
},
" t" : < em > < transaction-id (string)> < / em > ,
" y" : " r" ,
}
< / pre >
< / div >
< / div >
< div class = "section" id = "signature-verification" >
< h1 > signature verification< / h1 >
< p > In order to make it maximally difficult to attack the bencoding parser, signing and verification of the
value and sequence number should be done as follows:< / p >
< ol class = "arabic simple" >
< li > encode value and sequence number separately< / li >
2012-11-09 03:52:27 +01:00
< li > concatenate " 3:seqi" < tt class = "docutils literal" > seq< / tt > " e1:v" and the encoded value.
2011-05-25 04:26:07 +02:00
sequence number 1 of value " Hello World!" would be converted to: 3:seqi1e1:v12:Hello World!
In this way it is not possible to convince a node that part of the length is actually part of the
sequence number even if the parser contains certain bugs. Furthermore it is not possible to have a
verification failure if a bencoding serializer alters the order of entries in the dictionary.< / li >
< li > hash the concatenated string with SHA-1< / li >
< li > sign or verify the hash digest.< / li >
< / ol >
2012-11-16 23:25:39 +01:00
< p > On the storage node, the signature MUST be verified before accepting the store command. The data
MUST be stored under the SHA-1 hash of the public key (as it appears in the bencoded dict).< / p >
< p > On the subscribing nodes, the key they get back from a < tt class = "docutils literal" > get< / tt > request MUST be verified to hash
to the target ID the lookup was made for, as well as verifying the signature. If any of these fail,
the response SHOULD be considered invalid.< / p >
2011-05-25 04:26:07 +02:00
< / div >
< div class = "section" id = "expiration" >
< h1 > expiration< / h1 >
< p > Without re-announcement, these items MAY expire in 2 hours. In order
to keep items alive, they SHOULD be re-announced once an hour.< / p >
< p > Subscriber nodes MAY help out in announcing items the are interested in to the DHT,
to keep them alive.< / p >
< / div >
< div class = "section" id = "test-vectors" >
< h1 > test vectors< / h1 >
< / div >
< / div >
< div id = "footer" >
< span > Copyright © 2005 Rasterbar Software.< / span >
< / div >
< / div >
< script src = "http://www.google-analytics.com/urchin.js" type = "text/javascript" >
< / script >
< script type = "text/javascript" >
_uacct = "UA-1599045-1";
urchinTracker();
< / script >
< / div >
< / body >
< / html >
