This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128.
This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
This squashed commit makes a fairly significant structural change to significantly reduce Flodgatt's CPU usage.
Flodgatt connects to Redis in a single (green) thread, and then creates a new thread to handle each WebSocket/SSE connection. Previously, each thread was responsible for polling the Redis thread to determine whether it had a message relevant to the connected client. I initially selected this structure both because it was simple and because it minimized memory overhead – no messages are sent to a particular thread unless they are relevant to the client connected to the thread. However, I recently ran some load tests that show this approach to have unacceptable CPU costs when 300+ clients are simultaneously connected.
Accordingly, Flodgatt now uses a different structure: the main Redis thread now announces each incoming message via a watch channel connected to every client thread, and each client thread filters out irrelevant messages. In theory, this could lead to slightly higher memory use, but tests I have run so far have not found a measurable increase. On the other hand, Flodgatt's CPU use is now an order of magnitude lower in tests I've run.
This approach does run a (very slight) risk of dropping messages under extremely heavy load: because a watch channel only stores the most recent message transmitted, if Flodgatt adds a second message before the thread can read the first message, the first message will be overwritten and never transmitted. This seems unlikely to happen in practice, and we can avoid the issue entirely by changing to a broadcast channel when we upgrade to the most recent Tokio version (see #75).
* Tweak release profile & micro optimizations
* Replace std HashMap with hashbrown::HashMap
The hashbrown::HashMap is faster than the std::collections::HashMap,
though it does not protect as well against malicious hash collisions
(e.g., in a DDoS). Since we don't expose the hashing externally,
we should switch to the faster implementation.
* Remove use of last_polled_time [WIP]
This commit stops removing subscriptions based on their last polled
time to test the impact of this change on CPU use. This is a WIP
because it does not yet remove subscriptions in any other way, which
(if deployed in production) would cause a memory leak – memory use
would grow with each new subscription and would never be reduced as
clients end their subscriptions.
* Fix bug with RedisConnection polling freqeuency
* Improve performance of EventStream
This commit changes the EventStream so no longer polls client
WebSocket connections to see if it should clean up the connection.
Instead, it cleans up the connection whenever it attempts to send a
ping or a message through the connection and receives an error
indicating that the client has disconnected. As a result, client
connections aren't cleaned up quite as quickly, but overall sys CPU
time should be dramatically improved.
* Remove empty entries from MsgQueues hashmap
Before this change, entries in the MsgQueue hashmap would remain once
added, even if their value fell to 0. This could lead to a very
slight memory leak/increase, because the hashmap would grow each time
a new user connected and would not decrease again. This is now fixed.
* Bump version and remove unused benchmark
* Add /status API endpoints [WIP]
* Finish /status API endpoints
This PR enables compiling Flodgatt with the `stub_status` feature.
When compiled with `stub_status`, Flodgatt has 3 new API endpoints:
/api/v1/streaming/status, /api/v1/streaming/status/per_timeline, and
/api/v1/streaming/status/queue. The first endpoint lists the total
number of connections, the second lists the number of connections per
timeline, and the third lists the length of the longest queue of
unsent messages (which should be low or zero when Flodgatt is
functioning normally).
Note that the number of _connections_ is not equal to the number of
connected _clients_. If a user is viewing the local timeline, they
would have at least two connections: one for the local timeline, and
one for their user timeline. Other users could have even more
connections.
I decided to make the status endpoints an option you enable at compile
time rather than at run time for three reasons:
* It keeps the API of the default version of Flodgatt 100%
compatible with the Node server's API;
* I don't beleive it's an option Flodgatt adminstrators will want to
toggle on and off frequently.
* Using a compile time option ensures that there is zero runtime
cost when the option is disabled. (The runtime cost should be
negligible either way, but there is value in being 100% sure that
the cost can be eliminated.)
However, I'm happy to make it a runtime option instead if other think
that would be helpful.
* Initial work to support structured errors
* WIP error handling and RedisConn refactor
* WIP for error handling refactor
* Finish substantive work for Redis error handling
* Apply clippy lints
Previously, if a toot's language was `null`, then it would be
permitted regardless of the user's language filter; however, if it
were `""` (the empty string) it was rejected. Now, the empty string
is treated like `null` and the toot is allowed.
* Fix panic on delete events
Previously, the code attempted to check the toot's language regardless
of event types. That caused a panic for `delete` events, which lack a
language.
* WIP implementation of Message refactor
* Major refactor
* Refactor scope managment to use enum
* Use Timeline type instead of String
* Clean up Receiver's use of Timeline
* Make debug output more readable
* Block statuses from blocking users
This commit fixes an issue where a status from A would be displayed on
B's public timelines even when A had B blocked (i.e., it would treat B
as though they were muted rather than blocked for the purpose of
public timelines).
* Fix bug with incorrect parsing of incomming timeline
* Disable outdated tests
* Bump version
* Read user and domain blocks from Postgres
This commit reads the blocks from pg and stores them in the User
struct; it does not yet actually filter the responses. It also does
not update the tests.
* Update tests
* Filter out toots involving blocked/muted users
* Add support for domain blocks
* Update test and bump version
* Add type-safe wrapper types to deployement_cfg
* Before deleting redundnat macros
* Store error messages as data
* Significant progress on type safety
* Add type safety to RedisConfig
* Refactor configuration
* Fix bug with incorrect Host env variable
* Improve logging of REDIS_NAMESPACE
* Update test for Postgres configuration
* Conform Redis config to Postgres changes
Remove Postgres SSL support, which was not working at the moment and
was preventing flogatt from running on servers without openssl.
We should re-enable SSL support at a later time.
* Refactor organazation of SSE
This commit refactors how SSE requests are handled to bring them into
line with how WS requests are handled and increase consistency.
* Add websocket tests
* Bump version to 0.2.0
Bump version and update name from ragequit to flodgatt.
* Add test for non-existant endpoints
* Update documentation for recent changes``
Daniel Sockwell