This commit adds several aditional Postgres config options, such
as PORT, USER, DB_NAME and similar. It also relocates the .env
file so that it will be picked up even if flodgat isn't run from
the src/ directory.
This commit stores each message received from Redis as a growable string
rather than in a buffer of fixed size. This allows the server to
receive messages of any length.
Previously, the access token needed to be passed via the query string;
with this commit, the token can be passed *either* through the query
string or the Sec-WebSocket-Protocol header.
This was done to correspond to the changes made to the streaming.js
version in [Improve streaming server security](https://github.com/tootsuite/mastodon/pull/10818).
However, I am not sure that it *does* increase security; as explained
at <https://support.ably.io/support/solutions/articles/3000075120-is-it-secure-to-send-the-access-token-as-part-of-the-websocket-url-query-params->,
there is generally no security advantage to passing sensitive information
via websocket headers instead of the query string—the entire connection
is encrypted and is not stored in the browser history, so the typical
reasons to keep sensitive info out of the query string don't apply.
I would welcome any corrections on this/reasons this change improves
security.
This commit revises the code structure to share a single connection
to Redis (with multiple subscriptions on that connection) rather than
mutiple connections (each with one subscription). It also simplifies the code based on that change.
This commit implements a shared stream of data from Redis, which
allows all SSE connections that send the same data to the client
to share a single connection to Redis. (Previously, each client
got their own connection, which would significantly increase the
number of open Redis connections—especially since nearly all clients
will subscribe to `/public`.)
This commit tracks the number of active Pub/Sub streams and adds code to
keep the total number of streams below 400. (When additional users
attempt to connect past that point, the server will wait for an slot
to open up). This prevents "too many open file" panics and makes the
server better behaved in general. However, we may need to revisit it
based on what capacity we want the server to have.
This commit also includes some general refactoring.
This commit tracks the existence of the SSE stream and closes the
connection to the redis pub/sub channel when the stream is closed. This
prevents the number of redis connections from growing over time.
Note, however, that the current code still subscribes to one redis
channel per SSE connection rather than reusing existing subscriptions.
This will need to be fixed in a later PR.
This commit rolls up several days of work that took place in a side repo
which was originally intended to be a test/proof-of-concept repo but
that grew into a full implementation.
This implementation switches to Warp as the framework for the primary
web server (in large part because Warp offers built-in support for
SSE and several other advantages). This implementation relies on
a custom Redis interface built on Tokio (because the current Redis
Rust crate does not support asnyc PubSub). Using a custom interface
should also be faster, since it does not contain logic for anything
other than pubsub—which is all we need.
Finally, I note that the SSE support is currently as feature-complete
as it is possible to be without having yet added the Postgress interface
this means that all of the endpoints are present and are accessible
on localhost. However, none of the endpoints have authentication yet,
and the endpoints that should provide user-specific data (e.g.,
`/api/v1/streaming/user`) are currently hard-coded to provide data for
user 1 (the admin). Other than those limitations, however, the SSE
implementation is feature complete.
Daniel Sockwell