2011-01-02 04:53:28 +01:00
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
< html xmlns = "http://www.w3.org/1999/xhtml" xml:lang = "en" lang = "en" >
< head >
2012-01-16 03:09:07 +01:00
< meta http-equiv = "Content-Type" content = "text/html; charset=utf-8" / >
2014-05-08 10:58:19 +02:00
< meta name = "generator" content = "Docutils 0.11: http://docutils.sourceforge.net/" / >
2012-01-16 03:09:07 +01:00
< title > BitTorrent extension for DHT RSS feeds< / title >
< meta name = "author" content = "Arvid Norberg, arvid@rasterbar.com" / >
< link rel = "stylesheet" type = "text/css" href = "../../css/base.css" / >
< link rel = "stylesheet" type = "text/css" href = "../../css/rst.css" / >
2011-01-02 04:53:28 +01:00
< script type = "text/javascript" >
/* < ![CDATA[ */
(function() {
var s = document.createElement('script'), t = document.getElementsByTagName('script')[0];
s.type = 'text/javascript';
s.async = true;
s.src = 'http://api.flattr.com/js/0.6/load.js?mode=auto';
t.parentNode.insertBefore(s, t);
})();
/* ]]> */
< / script >
< link rel = "stylesheet" href = "style.css" type = "text/css" / >
< style type = "text/css" >
/* Hides from IE-mac \*/
* html pre { height: 1%; }
/* End hide from IE-mac */
< / style >
< / head >
< body >
< div class = "document" id = "bittorrent-extension-for-dht-rss-feeds" >
< div id = "container" >
< div id = "headerNav" >
< ul >
< li class = "first" > < a href = "/" > Home< / a > < / li >
< li > < a href = "../../products.html" > Products< / a > < / li >
< li > < a href = "../../contact.html" > Contact< / a > < / li >
< / ul >
< / div >
< div id = "header" >
2013-08-04 11:02:19 +02:00
< div id = "orange" > < / div >
< div id = "logo" > < / div >
2011-01-02 04:53:28 +01:00
< / div >
< div id = "main" >
< h1 class = "title" > BitTorrent extension for DHT RSS feeds< / h1 >
< table class = "docinfo" frame = "void" rules = "none" >
< col class = "docinfo-name" / >
< col class = "docinfo-content" / >
< tbody valign = "top" >
< tr > < th class = "docinfo-name" > Author:< / th >
< td > Arvid Norberg, < a class = "last reference external" href = "mailto:arvid@rasterbar.com" > arvid@ rasterbar.com< / a > < / td > < / tr >
< tr > < th class = "docinfo-name" > Version:< / th >
2014-07-06 21:18:00 +02:00
< td > 1.1.0< / td > < / tr >
2011-01-02 04:53:28 +01:00
< / tbody >
< / table >
< div class = "contents topic" id = "table-of-contents" >
< p class = "topic-title first" > Table of contents< / p >
< ul class = "simple" >
2011-01-19 06:57:44 +01:00
< li > < a class = "reference internal" href = "#terminology" id = "id1" > terminology< / a > < / li >
< li > < a class = "reference internal" href = "#linked-lists" id = "id2" > linked lists< / a > < / li >
< li > < a class = "reference internal" href = "#skip-lists" id = "id3" > skip lists< / a > < / li >
< li > < a class = "reference internal" href = "#list-head" id = "id4" > list-head< / a > < / li >
< li > < a class = "reference internal" href = "#messages" id = "id5" > messages< / a > < ul >
< li > < a class = "reference internal" href = "#requesting-items" id = "id6" > requesting items< / a > < / li >
< li > < a class = "reference internal" href = "#request-item-response" id = "id7" > request item response< / a > < / li >
< li > < a class = "reference internal" href = "#announcing-items" id = "id8" > announcing items< / a > < / li >
2011-01-02 04:53:28 +01:00
< / ul >
< / li >
2011-01-19 06:57:44 +01:00
< li > < a class = "reference internal" href = "#re-announcing" id = "id9" > re-announcing< / a > < / li >
< li > < a class = "reference internal" href = "#timeouts" id = "id10" > timeouts< / a > < / li >
< li > < a class = "reference internal" href = "#rss-feeds" id = "id11" > RSS feeds< / a > < ul >
< li > < a class = "reference internal" href = "#example" id = "id12" > example< / a > < / li >
< / ul >
< / li >
< li > < a class = "reference internal" href = "#rss-feed-uri-scheme" id = "id13" > RSS feed URI scheme< / a > < / li >
< li > < a class = "reference internal" href = "#rationale" id = "id14" > rationale< / a > < / li >
2011-01-02 04:53:28 +01:00
< / ul >
< / div >
2014-05-08 10:58:19 +02:00
< p > This proposal has been superseded by the < a class = "reference external" href = "dht_store.html" > dht_put< / a > feature. This may
still be implemented on top of that.< / p >
2011-01-02 04:53:28 +01:00
< p > This is a proposal for an extension to the BitTorrent DHT to allow
for decentralized RSS feed like functionality.< / p >
< p > The intention is to allow the creation of repositories of torrents
where only a single identity has the authority to add new content. For
this repository to be robust against network failures and resilient
to attacks at the source.< / p >
< p > The target ID under which the repository is stored in the DHT, is the
SHA-1 hash of a feed name and the 512 bit public key. This private key
in this pair MUST be used to sign every item stored in the repository.
Every message that contain signed items MUST also include this key, to
allow the receiver to verify the key itself against the target ID as well
as the validity of the signatures of the items. Every recipient of a
message with feed items in it MUST verify both the validity of the public
key against the target ID it is stored under, as well as the validity of
the signatures of each individual item.< / p >
< p > As with normal DHT announces, the write-token mechanism is used to
2011-01-19 06:57:44 +01:00
prevent IP spoof attacks.< / p >
< div class = "section" id = "terminology" >
< h1 > terminology< / h1 >
< p > In this document, a < em > storage node< / em > refers to the node in the DHT to which
an item is being announce. A < em > subscribing node< / em > refers to a node which
makes look ups in the DHT to find the storage nodes, to request items
from them.< / p >
< / div >
< div class = "section" id = "linked-lists" >
< h1 > linked lists< / h1 >
< p > Items are chained together in a geneal singly linked list. A linked
list does not necessarily contain RSS items, and no RSS related items
are mandatory. However, RSS items will be used as examples in this BEP:< / p >
< pre class = "literal-block" >
key = SHA1(name + key)
+---------+
| head | key = SHA1(bencode(item))
| +---------+ +---------+
| | next |--------> | item | key = SHA1(bencode(item))
| | key | | +---------+ +---------+
| | name | | | next |-------> | item |
| | seq | | | key | | +---------+
| | ... | | | ... | | | next |---> 0
| +---------+ | +---------+ | | key |
| sig | | sig | | | ... |
+---------+ +---------+ | +---------+
| sig |
+---------+
< / pre >
2013-01-21 20:13:24 +01:00
< p > The < tt class = "docutils literal" > next< / tt > pointer is at least 20 byte ID in the DHT key space pointing to where the next
2011-01-19 06:57:44 +01:00
item in the list is announced. The list is terminated with an ID of all zeroes.< / p >
< p > The ID an items is announced to is determined by the SHA1 hash of the bencoded representation
of the item iteself. This contains all fields in the item, except the signature.
2013-01-21 20:13:24 +01:00
The only mandatory fields in an item are < tt class = "docutils literal" > next< / tt > , < tt class = "docutils literal" > key< / tt > and < tt class = "docutils literal" > sig< / tt > .< / p >
< p > The < tt class = "docutils literal" > key< / tt > field MUST match the public key of the list head node. The < tt class = "docutils literal" > sig< / tt > field
MUST be the signature of the bencoded representation of < tt class = "docutils literal" > item< / tt > or < tt class = "docutils literal" > head< / tt > (whichever
2011-01-19 06:57:44 +01:00
is included in the message).< / p >
< p > All subscribers MUST verify that the item is announced under the correct DHT key
and MUST verify the signature is valid and MUST verify the public key is the same
as the list-head. If a node fails any of these checks, it must be ignored and the
chain of items considered terminated.< / p >
< p > Each item holds a bencoded dictionary with arbitrary keys, except two mandatory keys:
2013-01-21 20:13:24 +01:00
< tt class = "docutils literal" > next< / tt > and < tt class = "docutils literal" > key< / tt > . The signature < tt class = "docutils literal" > sig< / tt > is transferred outside of this dictionary
2011-01-19 06:57:44 +01:00
and is the signature of all of it. An implementation should stora any arbitrary keys that
are announced to an item, within reasonable restriction such as nesting, size and numeric
range of integers.< / p >
< / div >
< div class = "section" id = "skip-lists" >
< h1 > skip lists< / h1 >
2013-01-21 20:13:24 +01:00
< p > The < tt class = "docutils literal" > next< / tt > key stored in the list head and the items is a string of at least length
2011-01-19 06:57:44 +01:00
20 bytes, it may be any length divisible by 20. Each 20 bytes are the ID of the next
item in the list, the item 2 hops away, 4 hops away, 8 hops away, and so on. For
2013-01-21 20:13:24 +01:00
simplicity, only the first ID (1 hop) in the < tt class = "docutils literal" > next< / tt > field is illustrated above.< / p >
< p > A publisher of an item SHOULD include as many IDs in the < tt class = "docutils literal" > next< / tt > field as the remaining
2011-01-19 06:57:44 +01:00
size of the list warrants, within reason.< / p >
< p > These skip lists allow for parallelized lookups of items and also makes it more efficient
to search for specific items. It also mitigates breaking lists missing some items.< / p >
< p > Figure of the skip list in the first list item:< / p >
< pre class = "literal-block" >
n Item0 Item1 Item2 Item3 Item4 Item5 Item6 Item7 Item8 Item9 Item10
0 O----->
20 O------------>
40 O-------------------------->
60 O------------------------------------------------------>
< / pre >
2013-01-21 20:13:24 +01:00
< p > < em > n< / em > refers to the byte offset into the < tt class = "docutils literal" > next< / tt > field.< / p >
2011-01-19 06:57:44 +01:00
< / div >
< div class = "section" id = "list-head" >
< h1 > list-head< / h1 >
< p > The list head item is special in that it can be updated, without changing its
DHT key. This is required to prepend new items to the linked list. To authenticate
that only the original publisher can update the head, the whole linked list head
is signed. In order to avoid a malicious node to overwrite the list head with an old
2013-01-21 20:13:24 +01:00
version, the sequence number < tt class = "docutils literal" > seq< / tt > must be monotonically increasing for each update,
2011-01-19 06:57:44 +01:00
and a node hosting the list node MUST not downgrade a list head from a higher sequence
number to a lower one, only upgrade.< / p >
< p > The list head's DHT key (which it is announced to) MUST be the SHA1 hash of the name
2013-01-21 20:13:24 +01:00
(< tt class = "docutils literal" > n< / tt > ) and < tt class = "docutils literal" > key< / tt > fields concatenated.< / p >
2011-01-19 06:57:44 +01:00
< p > Any node MUST reject any list head which is announced under any other ID.< / p >
2011-01-02 04:53:28 +01:00
< / div >
< div class = "section" id = "messages" >
< h1 > messages< / h1 >
2011-01-19 06:57:44 +01:00
< p > These are the messages to deal with linked lists.< / p >
2013-01-21 20:13:24 +01:00
< p > The < tt class = "docutils literal" > id< / tt > field in these messages has the same semantics as the standard DHT messages,
2011-01-19 06:57:44 +01:00
i.e. the node ID of the node sending the message, to maintain the structure of the DHT
network.< / p >
2013-01-21 20:13:24 +01:00
< p > The < tt class = "docutils literal" > token< / tt > field also has the same semantics as the standard DHT message < tt class = "docutils literal" > get_peers< / tt >
and < tt class = "docutils literal" > announce_peer< / tt > , when requesting an item and to write an item respectively.< / p >
< p > < tt class = "docutils literal" > nodes< / tt > and < tt class = "docutils literal" > nodes6< / tt > has the same semantics as in its < tt class = "docutils literal" > get_peers< / tt > response.< / p >
2011-01-02 04:53:28 +01:00
< div class = "section" id = "requesting-items" >
2011-01-02 05:01:08 +01:00
< h2 > requesting items< / h2 >
2011-01-19 06:57:44 +01:00
< p > This message can be used to request both a list head and a list item. When requesting
2013-01-21 20:13:24 +01:00
a list head, the < tt class = "docutils literal" > n< / tt > (name) field MUST be specified. When requesting a list item the
< tt class = "docutils literal" > n< / tt > field is not required.< / p >
2011-01-02 04:53:28 +01:00
< pre class = "literal-block" >
{
2011-01-19 06:57:44 +01:00
" a" :
{
" id" : < em > < 20 byte ID of sending node> < / em > ,
" key" : < em > < 64 byte public curve25519 key for this list> < / em > ,
" n" : < em > < list name> < / em >
" target" : < em > < target-id for 'head' or 'item'> < / em >
},
" q" : " get_item" ,
" t" : < em > < transaction-id> < / em > ,
" y" : " q" ,
2011-01-02 04:53:28 +01:00
}
< / pre >
2013-01-21 20:13:24 +01:00
< p > When requesting a list-head the < tt class = "docutils literal" > target< / tt > MUST always be SHA-1(< em > feed_name< / em > + < em > public_key< / em > ).
< tt class = "docutils literal" > target< / tt > is the target node ID the item was written to.< / p >
< p > The < tt class = "docutils literal" > n< / tt > field is the name of the list. If specified, It MUST be UTF-8 encoded string
2011-01-19 06:57:44 +01:00
and it MUST match the name of the feed in the receiving node.< / p >
2011-01-02 04:53:28 +01:00
< / div >
< div class = "section" id = "request-item-response" >
2011-01-02 05:01:08 +01:00
< h2 > request item response< / h2 >
2011-01-19 06:57:44 +01:00
< p > This is the format of a response of a list head:< / p >
< pre class = "literal-block" >
{
" r" :
{
" head" :
{
" key" : < em > < 64 byte public curve25519 key for this list> < / em > ,
" next" : < em > < 20 bytes item ID> < / em > ,
" n" : < em > < name of the linked list> < / em > ,
" seq" : < em > < monotonically increasing sequence number> < / em >
},
" sig" : < em > < curve25519 signature of 'head' entry (in bencoded form)> < / em > ,
" id" : < em > < 20 byte id of sending node> < / em > ,
" token" : < em > < write-token> < / em > ,
" nodes" : < em > < n * compact IPv4-port pair> < / em > ,
" nodes6" : < em > < n * compact IPv6-port pair> < / em >
},
" t" : < em > < transaction-id> < / em > ,
" y" : " r" ,
}
< / pre >
< p > This is the format of a response of a list item:< / p >
2011-01-02 04:53:28 +01:00
< pre class = "literal-block" >
{
2011-01-19 06:57:44 +01:00
" r" :
{
" item" :
{
" key" : < em > < 64 byte public curve25519 key for this list> < / em > ,
" next" : < em > < 20 bytes item ID> < / em > ,
...
},
" sig" : < em > < curve25519 signature of 'item' entry (in bencoded form)> < / em > ,
" id" : < em > < 20 byte id of sending node> < / em > ,
" token" : < em > < write-token> < / em > ,
" nodes" : < em > < n * compact IPv4-port pair> < / em > ,
" nodes6" : < em > < n * compact IPv6-port pair> < / em >
},
" t" : < em > < transaction-id> < / em > ,
" y" : " r" ,
2011-01-02 04:53:28 +01:00
}
< / pre >
2013-01-21 20:13:24 +01:00
< p > A client receiving a < tt class = "docutils literal" > get_item< / tt > response MUST verify the signature in the < tt class = "docutils literal" > sig< / tt >
field against the bencoded representation of the < tt class = "docutils literal" > item< / tt > field, using the < tt class = "docutils literal" > key< / tt > as
the public key. The < tt class = "docutils literal" > key< / tt > MUST match the public key of the feed.< / p >
< p > The < tt class = "docutils literal" > item< / tt > dictionary MAY contain arbitrary keys, and all keys MUST be stored for
2011-01-19 06:57:44 +01:00
items.< / p >
2011-01-02 04:53:28 +01:00
< / div >
< div class = "section" id = "announcing-items" >
2011-01-02 05:01:08 +01:00
< h2 > announcing items< / h2 >
2011-01-19 06:57:44 +01:00
< p > The message format for announcing a list head:< / p >
2011-01-02 04:53:28 +01:00
< pre class = "literal-block" >
{
2011-01-19 06:57:44 +01:00
" a" :
{
" head" :
{
" key" : < em > < 64 byte public curve25519 key for this list> < / em > ,
" next" : < em > < 20 bytes item ID> < / em > ,
" n" : < em > < name of the linked list> < / em > ,
" seq" : < em > < monotonically increasing sequence number> < / em >
},
" sig" : < em > < curve25519 signature of 'head' entry (in bencoded form)> < / em > ,
" id" : < em > < 20 byte node-id of origin node> < / em > ,
" target" : < em > < target-id as derived from public key and name> < / em > ,
" token" : < em > < write-token as obtained by previous request> < / em >
},
" y" : " q" ,
" q" : " announce_item" ,
" t" : < em > < transaction-id> < / em >
2011-01-02 04:53:28 +01:00
}
< / pre >
2011-01-19 06:57:44 +01:00
< p > The message format for announcing a list item:< / p >
< pre class = "literal-block" >
{
" a" :
{
" item" :
{
" key" : < em > < 64 byte public curve25519 key for this list> < / em > ,
" next" : < em > < 20 bytes item ID> < / em > ,
...
},
" sig" : < em > < curve25519 signature of 'item' entry (in bencoded form)> < / em > ,
" id" : < em > < 20 byte node-id of origin node> < / em > ,
" target" : < em > < target-id as derived from item dict> < / em > ,
" token" : < em > < write-token as obtained by previous request> < / em >
},
" y" : " q" ,
" q" : " announce_item" ,
" t" : < em > < transaction-id> < / em >
}
< / pre >
< p > A storage node MAY reject items and heads whose bencoded representation is
greater than 1024 bytes.< / p >
< / div >
< / div >
< div class = "section" id = "re-announcing" >
< h1 > re-announcing< / h1 >
< p > In order to keep feeds alive, subscriber nodes SHOULD help out in announcing
items they have downloaded to the DHT.< / p >
< p > Every subscriber node SHOULD store items in long term storage, across sessions,
in order to keep items alive for as long as possible, with as few sources as possible.< / p >
2011-01-02 04:53:28 +01:00
< p > Subscribers to a feed SHOULD also announce items that they know of, to the feed.
2011-01-19 06:57:44 +01:00
Since a feed may have many subscribers and many items, subscribers should re-announce
items according to the following algorithm.< / p >
< pre class = "literal-block" >
1. pick one random item (< em > i< / em > ) from the local repository (except
items already announced this round)
2. If all items in the local repository have been announced
2.1 terminate
3. look up item < em > i< / em > in the DHT
4. If fewer than 8 nodes returned the item
4.1 announce < em > i< / em > to the DHT
4.2 goto 1
< / pre >
< p > This ensures a balanced load on the DHT while still keeping items alive< / p >
2011-01-02 05:01:08 +01:00
< / div >
2011-01-19 06:57:44 +01:00
< div class = "section" id = "timeouts" >
< h1 > timeouts< / h1 >
< p > Items SHOULD be announced to the DHT every 30 minutes. A storage node MAY time
out an item after 60 minutes of no one announcing it.< / p >
< p > A storing node MAY extend the timeout when it receives a request for it. Since
items are immutable, the data doesn't go stale. Therefore it doesn't matter if
the storing node no longer is in the set of the 8 closest nodes.< / p >
< / div >
< div class = "section" id = "rss-feeds" >
< h1 > RSS feeds< / h1 >
2013-01-21 20:13:24 +01:00
< p > For RSS feeds, following keys are mandatory in the list item's < tt class = "docutils literal" > item< / tt > dictionary.< / p >
2011-01-19 06:57:44 +01:00
< dl class = "docutils" >
< dt > ih< / dt >
< dd > The torrent's info hash< / dd >
< dt > size< / dt >
< dd > The size (in bytes) of all files the torrent< / dd >
< dt > n< / dt >
< dd > name of the torrent< / dd >
< / dl >
2011-01-02 04:53:28 +01:00
< div class = "section" id = "example" >
2011-01-02 05:01:08 +01:00
< h2 > example< / h2 >
2013-01-21 20:13:24 +01:00
< p > This is an example of an < tt class = "docutils literal" > announce_item< / tt > message:< / p >
2011-01-02 04:53:28 +01:00
< pre class = "literal-block" >
{
2011-01-19 06:57:44 +01:00
" a" :
{
" item" :
{
" key" : " 6bc1de5443d1a7c536cdf69433ac4a7163d3c63e2f9c92d
78f6011cf63dbcd5b638bbc2119cdad0c57e4c61bc69ba5e2c08
b918c2db8d1848cf514bd9958d307" ,
" info-hash" : " 7ea94c240691311dc0916a2a91eb7c3db2c6f3e4" ,
" size" : 24315329,
" n" : " my stuff" ,
" next" : " c68f29156404e8e0aas8761ef5236bcagf7f8f2e"
}
" sig" : < em > < signature> < / em >
" id" : " b46989156404e8e0acdb751ef553b210ef77822e" ,
" target" : " b4692ef0005639e86d7165bf378474107bf3a762"
" token" : " 23ba"
},
" y" : " q" ,
" q" : " announce_item" ,
" t" : " a421"
2011-01-02 04:53:28 +01:00
}
< / pre >
2011-01-19 06:57:44 +01:00
< p > Strings are printed in hex for printability, but actual encoding is binary.< / p >
2013-01-21 20:13:24 +01:00
< p > Note that < tt class = "docutils literal" > target< / tt > is in fact SHA1 hash of the same data the signature < tt class = "docutils literal" > sig< / tt >
2011-01-19 06:57:44 +01:00
is the signature of, i.e.:< / p >
< pre class = "literal-block" >
d9:info-hash20:7ea94c240691311dc0916a2a91eb7c3db2c6f3e43:key64:6bc1de5443d1
a7c536cdf69433ac4a7163d3c63e2f9c92d78f6011cf63dbcd5b638bbc2119cdad0c57e4c61
bc69ba5e2c08b918c2db8d1848cf514bd9958d3071:n8:my stuff4:next20:c68f29156404
e8e0aas8761ef5236bcagf7f8f2e4:sizei24315329ee
< / pre >
< p > (note that binary data is printed as hex)< / p >
2011-01-02 04:53:28 +01:00
< / div >
< / div >
2011-01-19 06:57:44 +01:00
< div class = "section" id = "rss-feed-uri-scheme" >
< h1 > RSS feed URI scheme< / h1 >
2011-01-02 04:53:28 +01:00
< p > The proposed URI scheme for DHT feeds is:< / p >
< pre class = "literal-block" >
magnet:?xt=btfd:< em > < base16-curve25519-public-key> < / em > & dn= < em > < feed name> < / em >
< / pre >
< p > Note that a difference from regular torrent magnet links is the < strong > btfd< / strong >
versus < strong > btih< / strong > used in regular magnet links to torrents.< / p >
< p > The < em > feed name< / em > is mandatory since it is used in the request and when
calculating the target ID.< / p >
< / div >
< div class = "section" id = "rationale" >
< h1 > rationale< / h1 >
2011-01-19 06:57:44 +01:00
< p > The reason to use < a class = "reference external" href = "http://cr.yp.to/ecdh.html" > curve25519< / a > instead of, for instance, RSA is compactness. According to
< a class = "reference external" href = "http://cr.yp.to/" > http://cr.yp.to/< / a > , curve25519 is free from patent claims and there are open implementations
in both C and Java.< / p >
2011-01-02 04:53:28 +01:00
< / div >
< / div >
< / body >
< / html >
