Commit Graph

152 Commits

Author SHA1 Message Date
Daniel Sockwell daf7d1ae7f
Add tests for polling for multiple messages (#149)
* Add tests for polling for multiple messages

This commit adds a mock Redis interface and adds tests that poll the
mock interface for multiple messages at a time.  These tests test that
Flodgatt is robust against receiving incomplete messages, including if
the message break results in receiving invalid UTF8.

* Remove temporary files
2020-05-07 10:56:11 -04:00
Daniel Sockwell 4de9a94230
Additional Event parsing tests (#148) 2020-04-29 17:58:01 -04:00
Daniel Sockwell d2b9fbe3b2
Fix `channel full` codepath (#147)
A bug was causing the buffer index to not properly update when
handling full channels; this is now fixed.
2020-04-29 17:10:58 -04:00
Daniel Sockwell 2f07cf7d16
Add RedisMsg and Event parsing tests (#146)
* Add RedisMsg tests

* Testing for Event parsing
2020-04-29 16:18:45 -04:00
Daniel Sockwell c9b58bc11b
Hotfix of previous commit (#145) 2020-04-28 14:37:51 -04:00
Daniel Sockwell 66553408fb
Fix edge case bug for long messages (#144) 2020-04-28 12:47:14 -04:00
Daniel Sockwell 4a13412f98
Improve handling of large Redis input (#143)
* Implement faster buffered input

This commit implements a modified ring buffer for input from Redis.
Specifically, Flodgatt now limits the amount of data it fetches from
Redis in one syscall to 8 KiB (two pages on most systems). Flodgatt
will process all complete messages it receives from Redis and then
re-use the same buffer for the next time it retrieves data.  If
Flodgatt received a partial message, it will copy the partial message
to the beginning of the buffer before its next read.

This change has little effect on Flodgatt under light load (because it
was rare for Redis to have more than 8 KiB of messages available at
any one time).  However, my hope is that this will significantly
reduce memory use on the largest instances.

* Improve handling of backpresure

This commit alters how Flodgatt behaves if it receives enough messages
for a single client to fill that clients channel. (Because the clients
regularly send their messages, should only occur if a single client
receives a large number of messages nearly simultaneously; this is
rare, but could occur, especially on large instances).

Previously, Flodgatt would drop messages in the rare case when the
client's channel was full.  Now, Flodgatt will pause the current Redis
poll and yield control back to the client streams, allowing the
clients to empty their channels; Flodgatt will then resume polling
Redis/sending the messages it previously received.  With the approach,
Flodgatt will never drop messages.

However, the risk to this approach is that, by never dropping
messages, Flodgatt does not have any way to reduce the amount of work
it needs to do when under heavy load – it delays the work slightly,
but doesn't reduce it.  What this means is that it would be
*theoretically* possible for Flodgatt to fall increasingly behind, if
it is continuously receiving more messages than it can process.  Due
to how quickly Flodgatt can process messages, though, I suspect this
would only come up if an admin were running Flodgatt in a
*significantly* resource constrained environment, but I wanted to
mention it for the sake of completeness.

This commit also adds a new /status/backpressure endpoint that
displays the current length of the Redis input buffer (which should
typically be low or 0).  Like the other /status endpoints, this
endpoint is only enabled when Flodgatt is compiled with the
`stub_status` feature.
2020-04-27 16:03:05 -04:00
Daniel Sockwell d8b07b4b03
Simplify handling of disconnected clients (#142)
* Simplify handling of disconnected clients

* Improve disconnection error logging
2020-04-24 18:05:36 -04:00
Daniel Sockwell cae01b6e90
Fix redis_namespace bug (#141)
This fixes a bug that would cause Redis commands not to be sent
properly when users had set the REDIS_NAMESPACE environmental
variable.
2020-04-24 14:47:11 -04:00
Daniel Sockwell b18500b884
Resolve memory-use regression (#140)
* Use monotonically increasing channel_id

Using a monotonically increasing channel_id (instead of a Uuid)
reduces memory use under load by ~3%

* Use replace unbounded channels with bounded

This also slightly reduces memory use

* Heap allocate Event

Wrapping the Event struct in an Arc avoids excessive copying and significantly reduces memory use.

* Implement more efficient unsubscribe strategy

* Fix various Clippy lints; bump version

* Update config defaults
2020-04-24 13:23:59 -04:00
Daniel Sockwell 2725439110
Update concurrency primitive. (#139)
* Initial [WIP] implementation

This initial implementation works to send messages but does not yet
handle unsubscribing properly.

* Implement UnboundedSender

* Implement UnboundedChannels for concurrency
2020-04-23 19:28:26 -04:00
Daniel Sockwell 91186fb9f7
Postgres (#137)
* Revise Postgres to use `simple_query`

* Fix bug in logging `ENV` config errors

* Improve parsing of values from Postgres

* Finish Postgres changes
2020-04-23 12:04:30 -04:00
Daniel Sockwell 016f49a2d8
Improve module privacy (#136)
* Adjust module privacy

* Use trait object

* Finish module privacy refactor
2020-04-22 14:38:22 -04:00
Daniel Sockwell 4a456c6e90
Prevent Travis from caching binary objects (#135)
* Prevent Travis from caching binary objects

* Add one more folder to `rm` in .travis.yml
2020-04-21 17:52:06 -04:00
Daniel Sockwell 6d9797ecc0
Remove unnecessary delay from sample load test script (#134) 2020-04-21 17:06:52 -04:00
Daniel Sockwell 1dcddc23de
Update module privacy (#133)
This squashed commit rolls up a series of changes designed to improve
Flodgatt's public API/module boundary.  Specifically, this limits the
number of Items that are exported outside the top-level modules (in some
cases because they were already not needed outside that module due to
the earlier code reorganization and in some cases by using public
re-exports to export particular Items from a private module).

Similarly, this commit moves the `Event` struct to the `response`
module (maintaining privacy for the `Event`'s implementation details)
while re-exporting the `Id` struct that `Event` uses internally at the
top level.

All of these changes are made with the goal of making Flodgatt's code
easier to reason about in isolation, which should both make it easier to
maintain and make it easier for new contributors to make changes without
understanding the entire codebase.  Additionally, having fewer public
modules will make documenting Flodgatt more extensively much easier.
2020-04-21 16:39:31 -04:00
Daniel Sockwell 10fa24c5d3
Benchmark & performance tune (#132)
* Add temporary perf metrics

* Add load testing and tune performance
2020-04-17 17:07:10 -04:00
Daniel Sockwell 37b652ad79
Error handling, pt3 (#131)
* Improve handling of Postgres errors

* Finish error handling improvements

* Remove `format!` calls from hot path
2020-04-14 20:37:49 -04:00
Daniel Sockwell 45f9d4b9fb
Code reorganization (#130)
* Reorganize files

* Refactor main()

* Code reorganization [WIP]

* Reorganize code [WIP]

* Refacto RedisConn [WIP]

* Complete code reorganization
2020-04-13 16:03:06 -04:00
Daniel Sockwell 0eec8f6f7b Remove unneeded dependency 2020-04-10 22:56:19 -04:00
Daniel Sockwell 5d2b0b94e2
Error handling pt2 (#129)
This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128.

This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-10 22:36:03 -04:00
Daniel Sockwell 1657113c58
Stream events via a watch channel (#128)
This squashed commit makes a fairly significant structural change to significantly reduce Flodgatt's CPU usage.

Flodgatt connects to Redis in a single (green) thread, and then creates a new thread to handle each WebSocket/SSE connection. Previously, each thread was responsible for polling the Redis thread to determine whether it had a message relevant to the connected client. I initially selected this structure both because it was simple and because it minimized memory overhead – no messages are sent to a particular thread unless they are relevant to the client connected to the thread. However, I recently ran some load tests that show this approach to have unacceptable CPU costs when 300+ clients are simultaneously connected.

Accordingly, Flodgatt now uses a different structure: the main Redis thread now announces each incoming message via a watch channel connected to every client thread, and each client thread filters out irrelevant messages. In theory, this could lead to slightly higher memory use, but tests I have run so far have not found a measurable increase. On the other hand, Flodgatt's CPU use is now an order of magnitude lower in tests I've run.

This approach does run a (very slight) risk of dropping messages under extremely heavy load: because a watch channel only stores the most recent message transmitted, if Flodgatt adds a second message before the thread can read the first message, the first message will be overwritten and never transmitted. This seems unlikely to happen in practice, and we can avoid the issue entirely by changing to a broadcast channel when we upgrade to the most recent Tokio version (see #75).
2020-04-09 13:32:36 -04:00
Daniel Sockwell fa8b695129
Minor performance tune (#127)
* Tweak release profile & micro optimizations

* Replace std HashMap with hashbrown::HashMap

The hashbrown::HashMap is faster than the std::collections::HashMap,
though it does not protect as well against malicious hash collisions
(e.g., in a DDoS).  Since we don't expose the hashing externally,
we should switch to the faster implementation.
2020-04-08 18:39:52 -04:00
ktagbnaw cd82e2d34c
Fix typo and reword a sentence (#126)
* Fix typo and reword a sentence

* Reword sentence to active voice

Co-authored-by: Daniel Sockwell <daniel@codesections.com>
2020-04-06 17:51:21 -04:00
Daniel Sockwell d23cc40bea
Iowait (#125)
* Remove use of last_polled_time [WIP]

This commit stops removing subscriptions based on their last polled
time to test the impact of this change on CPU use.  This is a WIP
because it does not yet remove subscriptions in any other way, which
(if deployed in production) would cause a memory leak – memory use
would grow with each new subscription and would never be reduced as
clients end their subscriptions.

* Fix bug with RedisConnection polling freqeuency

* Improve performance of EventStream

This commit changes the EventStream so no longer polls client
WebSocket connections to see if it should clean up the connection.
Instead, it cleans up the connection whenever it attempts to send a
ping or a message through the connection and receives an error
indicating that the client has disconnected.  As a result, client
connections aren't cleaned up quite as quickly, but overall sys CPU
time should be dramatically improved.

* Remove empty entries from MsgQueues hashmap

Before this change, entries in the MsgQueue hashmap would remain once
added, even if their value fell to 0.  This could lead to a very
slight memory leak/increase, because the hashmap would grow each time
a new user connected and would not decrease again.  This is now fixed.

* Bump version and remove unused benchmark
2020-04-05 17:54:05 -04:00
Daniel Sockwell d2e0a01baf
Stub status (#124)
* Add /status API endpoints [WIP]

* Finish /status API endpoints

This PR enables compiling Flodgatt with the `stub_status` feature.
When compiled with `stub_status`, Flodgatt has 3 new API endpoints:
/api/v1/streaming/status, /api/v1/streaming/status/per_timeline, and
/api/v1/streaming/status/queue.  The first endpoint lists the total
number of connections, the second lists the number of connections per
timeline, and the third lists the length of the longest queue of
unsent messages (which should be low or zero when Flodgatt is
functioning normally).

Note that the number of _connections_ is not equal to the number of
connected _clients_.  If a user is viewing the local timeline, they
would have at least two connections: one for the local timeline, and
one for their user timeline.  Other users could have even more
connections.

I decided to make the status endpoints an option you enable at compile
time rather than at run time for three reasons:

  * It keeps the API of the default version of Flodgatt 100%
    compatible with the Node server's API;

  * I don't beleive it's an option Flodgatt adminstrators will want to
    toggle on and off frequently.

  * Using a compile time option ensures that there is zero runtime
    cost when the option is disabled.  (The runtime cost should be
    negligible either way, but there is value in being 100% sure that
    the cost can be eliminated.)

However, I'm happy to make it a runtime option instead if other think
that would be helpful.
2020-04-05 10:54:42 -04:00
Daniel Sockwell 6a6537253d
Fix `DATABASE_URL` parsing (#122) 2020-04-03 16:35:32 -04:00
Daniel Sockwell 19792d9484
Handle non conforment events (#117)
* Initial implementation of DynamicEvent

* Restore early Event parsing
2020-04-03 12:41:53 -04:00
Daniel Sockwell d5f079a864
Error handling, pt1 (#115)
* Initial work to support structured errors

* WIP error handling and RedisConn refactor

* WIP for error handling refactor

* Finish substantive work for Redis error handling

* Apply clippy lints
2020-04-01 15:35:24 -04:00
Daniel Sockwell 81b454c88c
Extract tests to separate files (#113)
This very minor change moves tests from their current location in
submodules within the file under test into submodules in separate
files.  This is a slight deviation from the normal Rust convention
(though only very slight, since the module structure remains the
same).  However, it is justified here since the tests are fairly
verbose and including them in the same file was a bit unwieldy.
2020-03-31 09:05:51 -04:00
Daniel Sockwell 5965a514fd
Reorganize code, pt2 (#112)
* Cleanup RedisMsg parsing [WIP]

* Add tests to Redis parsing

* WIP RedisMsg refactor

Committing WIP before trying a different approach

* WIP

* Refactor RedisConn and Receiver

* Finish second reorganization
2020-03-30 18:54:00 -04:00
Daniel Sockwell 0acbde3eee
Reorganize code, pt1 (#110)
* Prevent Reciever from querying postgres

Before this commit, the Receiver would query Postgres for the name
associated with a hashtag when it encountered one not in its cache.
This ensured that the Receiver never encountered a (valid) hashtag id
that it couldn't handle, but caused a extra DB query and made
independent sections of the code more entangled than they need to be.

Now, we pass the relevant tag name to the Receiver when it first
starts managing a new subscription and it adds the tag name to its
cache then.

* Improve module boundary/privacy

* Reorganize Receiver to cut RedisStream

* Fix tests for code reorganization

Note that this change includes testing some private functionality by
exposing it publicly in tests via conditional compilation.  This
doesn't expose that functionality for the benchmarks, so the benchmark
tests do not currently pass without adding a few `pub use`
statements.  This might be worth changing later, but benchmark tests
aren't part of our CI and it's not hard to change when we want to test
performance.

This change also cuts the benchmark tests that were benchmarking old
ways Flodgatt functioned.  Those were useful for comparison purposes,
but have served their purpose – we've firmly moved away from the
older/slower approach.

* Fix Receiver for tests
2020-03-27 12:00:48 -04:00
Daniel Sockwell 2dd9ccbf91
Performance tuning (#108)
* Initial implementation WIP

* Add Event type for faster parsing

* Add tests and benchmarks

* Add additional parsing tests
2020-03-25 17:50:32 -04:00
Daniel Sockwell 8797a47efd
Stop filtering out toots with language set to `""` (#100)
Previously, if a toot's language was `null`, then it would be
permitted regardless of the user's language filter; however, if it
were `""` (the empty string) it was rejected.  Now, the empty string
is treated like `null` and the toot is allowed.
2020-03-20 15:43:50 -04:00
Daniel Sockwell a6b4d968cb
Add support for WHITELIST_MODE (#99)
When the `WHITELIST_MODE` environmental variable is set, Flodgatt
requires users to authenticate with a valid access token before
subscribing to any timelines (even those that are typically public).
2020-03-20 14:42:01 -04:00
Daniel Sockwell eda52c20b1
Add additional info logging (#98) 2020-03-19 20:54:23 -04:00
Daniel Sockwell 8c6b8e87f5
Remove language filter from home timeline (#97)
Previously, the language filter was incorrectly applied to all
`update` messages.  With this change, it is only applied to public
timelines (i.e., the local timeline and the federated timeline) which
is the behavior described in the docs.
2020-03-19 19:40:01 -04:00
Daniel Sockwell 2096e1348b
Add support for announcement event type (#96) 2020-03-19 13:20:48 -04:00
Daniel Sockwell dc6256d521
Fix double-escaping of delete payload (#95) 2020-03-19 12:10:59 -04:00
Daniel Sockwell 8843f18f5f
Fix valid language (#93)
* Fix panic on delete events

Previously, the code attempted to check the toot's language regardless
of event types.  That caused a panic for `delete` events, which lack a
language.

* WIP implementation of Message refactor

* Major refactor

* Refactor scope managment to use enum

* Use Timeline type instead of String

* Clean up Receiver's use of Timeline

* Make debug output more readable

* Block statuses from blocking users

This commit fixes an issue where a status from A would be displayed on
B's public timelines even when A had B blocked (i.e., it would treat B
as though they were muted rather than blocked for the purpose of
public timelines).

* Fix bug with incorrect parsing of incomming timeline

* Disable outdated tests

* Bump version
2020-03-18 20:37:10 -04:00
Daniel Sockwell 440d691b0f
Filter toots based on user and domain blocks (#89)
* Read user and domain blocks from Postgres

This commit reads the blocks from pg and stores them in the User
struct; it does not yet actually filter the responses.  It also does
not update the tests.

* Update tests

* Filter out toots involving blocked/muted users

* Add support for domain blocks

* Update test and bump version
2020-03-12 22:44:31 -04:00
Daniel Sockwell ebe9aeccbc
Hotfix for v0.4.8 (#87)
Fixes a regression in v0.4.8 that caused some JSON output to not be
properly escaped, leading it to not correctly display in the browser.
2020-03-11 13:26:11 -04:00
Daniel Sockwell ed75905fa3
Increase verbosity of debug info (#86)
* Increase verbosity of debug info

* Add email field to User in tests
2020-03-11 11:31:29 -04:00
Daniel Sockwell 405b5e88e5
Update logging (#85)
* Change "Incoming" log msgs from Warn to Info

* Stop logging err when unix socket closed

* Bump version to 0.4.7
2020-01-10 17:56:19 -05:00
Daniel Sockwell ac75cb54af
Postgres connection pool (#84)
* Upgrade rust-postgres library

* Initial postgres connection pool

* Update tests

* s/pg_conn/pg_pool to match reality
2020-01-10 15:45:16 -05:00
Daniel Sockwell 5482be9414
Fix dev dependencies for benchmarks (#82) 2020-01-09 18:17:01 -05:00
Daniel Sockwell 0462267125
Unix sockets (#81)
* Fix unix socket permission issue

* Add support for Unix sockets

* Update README and bump version
2020-01-09 17:54:57 -05:00
Daniel Sockwell 20b9c7b593
Update README with link to default nginx config (#80) 2020-01-08 16:15:18 -05:00
Daniel Sockwell be143016a2
Send empty message as a keepalive ping (#78) 2020-01-08 10:23:34 -05:00
Daniel Sockwell 5b9feda94e Merge branch 'master' of github.com:tootsuite/ragequit 2020-01-08 10:05:34 -05:00