zelo72
/
flodgatt
mirror of https://github.com/mastodon/flodgatt
1
0
Fork 0
Code Issues Releases Wiki Activity
flodgatt/src/parse_client_request/subscription.rs

321 lines
12 KiB
Rust
Raw Normal View History

Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
//! `User` struct and related functionality
// #[cfg(test)]
// mod mock_postgres;
// #[cfg(test)]
// use mock_postgres as postgres;
// #[cfg(not(test))]
use super::postgres::PgPool;
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
use super::query;
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
use super::query::Query;
use crate::err::TimelineErr;
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
use crate::messages::Id;
Minor performance tune (#127) * Tweak release profile & micro optimizations * Replace std HashMap with hashbrown::HashMap The hashbrown::HashMap is faster than the std::collections::HashMap, though it does not protect as well against malicious hash collisions (e.g., in a DDoS). Since we don't expose the hashing externally, we should switch to the faster implementation.
2020-04-09 00:39:52 +02:00
use hashbrown::HashSet;
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
use lru::LruCache;
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
use warp::{filters::BoxedFilter, path, reject::Rejection, Filter};
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
/// Helper macro to match on the first of any of the provided filters
macro_rules! any_of {
($filter:expr, $($other_filter:expr),*) => {
$filter$(.or($other_filter).unify())*.boxed()
};
}
macro_rules! parse_sse_query {
(path => $start:tt $(/ $next:tt)*
endpoint => $endpoint:expr) => {
path!($start $(/ $next)*)
.and(query::Auth::to_filter())
.and(query::Media::to_filter())
.and(query::Hashtag::to_filter())
.and(query::List::to_filter())
.map(
|auth: query::Auth,
media: query::Media,
hashtag: query::Hashtag,
list: query::List| {
Query {
access_token: auth.access_token,
stream: $endpoint.to_string(),
media: media.is_truthy(),
hashtag: hashtag.tag,
list: list.list,
}
},
)
.boxed()
};
}
#[derive(Clone, Debug, PartialEq)]
pub struct Subscription {
pub timeline: Timeline,
pub allowed_langs: HashSet<String>,
pub blocks: Blocks,
pub hashtag_name: Option<String>,
pub access_token: Option<String>,
}
Minor performance tune (#127) * Tweak release profile & micro optimizations * Replace std HashMap with hashbrown::HashMap The hashbrown::HashMap is faster than the std::collections::HashMap, though it does not protect as well against malicious hash collisions (e.g., in a DDoS). Since we don't expose the hashing externally, we should switch to the faster implementation.
2020-04-09 00:39:52 +02:00
#[derive(Clone, Default, Debug, PartialEq)]
pub struct Blocks {
pub blocked_domains: HashSet<String>,
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
pub blocked_users: HashSet<Id>,
pub blocking_users: HashSet<Id>,
Minor performance tune (#127) * Tweak release profile & micro optimizations * Replace std HashMap with hashbrown::HashMap The hashbrown::HashMap is faster than the std::collections::HashMap, though it does not protect as well against malicious hash collisions (e.g., in a DDoS). Since we don't expose the hashing externally, we should switch to the faster implementation.
2020-04-09 00:39:52 +02:00
}
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
impl Default for Subscription {
fn default() -> Self {
Self {
timeline: Timeline(Stream::Unset, Reach::Local, Content::Notification),
allowed_langs: HashSet::new(),
blocks: Blocks::default(),
hashtag_name: None,
access_token: None,
}
}
}
impl Subscription {
pub fn from_ws_request(pg_pool: PgPool, whitelist_mode: bool) -> BoxedFilter<(Subscription,)> {
parse_ws_query()
.and(query::OptionalAccessToken::from_ws_header())
.and_then(Query::update_access_token)
.and_then(move |q| Subscription::from_query(q, pg_pool.clone(), whitelist_mode))
.boxed()
}
pub fn from_sse_query(pg_pool: PgPool, whitelist_mode: bool) -> BoxedFilter<(Subscription,)> {
any_of!(
parse_sse_query!(
path => "api" / "v1" / "streaming" / "user" / "notification"
endpoint => "user:notification" ),
parse_sse_query!(
path => "api" / "v1" / "streaming" / "user"
endpoint => "user"),
parse_sse_query!(
path => "api" / "v1" / "streaming" / "public" / "local"
endpoint => "public:local"),
parse_sse_query!(
path => "api" / "v1" / "streaming" / "public"
endpoint => "public"),
parse_sse_query!(
path => "api" / "v1" / "streaming" / "direct"
endpoint => "direct"),
parse_sse_query!(path => "api" / "v1" / "streaming" / "hashtag" / "local"
endpoint => "hashtag:local"),
parse_sse_query!(path => "api" / "v1" / "streaming" / "hashtag"
endpoint => "hashtag"),
parse_sse_query!(path => "api" / "v1" / "streaming" / "list"
endpoint => "list")
)
// because SSE requests place their `access_token` in the header instead of in a query
// parameter, we need to update our Query if the header has a token
.and(query::OptionalAccessToken::from_sse_header())
.and_then(Query::update_access_token)
.and_then(move |q| Subscription::from_query(q, pg_pool.clone(), whitelist_mode))
.boxed()
}
Extract tests to separate files (#113) This very minor change moves tests from their current location in submodules within the file under test into submodules in separate files. This is a slight deviation from the normal Rust convention (though only very slight, since the module structure remains the same). However, it is justified here since the tests are fairly verbose and including them in the same file was a bit unwieldy.
2020-03-31 15:05:51 +02:00
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
fn from_query(q: Query, pool: PgPool, whitelist_mode: bool) -> Result<Self, Rejection> {
let user = match q.access_token.clone() {
Some(token) => pool.clone().select_user(&token)?,
None if whitelist_mode => Err(warp::reject::custom("Error: Invalid access token"))?,
None => UserData::public(),
};
let timeline = Timeline::from_query_and_user(&q, &user, pool.clone())?;
let hashtag_name = match timeline {
Timeline(Stream::Hashtag(_), _, _) => Some(q.hashtag),
_non_hashtag_timeline => None,
};
Ok(Subscription {
timeline,
allowed_langs: user.allowed_langs,
blocks: Blocks {
blocking_users: pool.clone().select_blocking_users(user.id),
blocked_users: pool.clone().select_blocked_users(user.id),
Error handling, pt1 (#115) * Initial work to support structured errors * WIP error handling and RedisConn refactor * WIP for error handling refactor * Finish substantive work for Redis error handling * Apply clippy lints
2020-04-01 21:35:24 +02:00
blocked_domains: pool.select_blocked_domains(user.id),
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
},
hashtag_name,
access_token: q.access_token,
})
}
}
fn parse_ws_query() -> BoxedFilter<(Query,)> {
path!("api" / "v1" / "streaming")
.and(path::end())
.and(warp::query())
.and(query::Auth::to_filter())
.and(query::Media::to_filter())
.and(query::Hashtag::to_filter())
.and(query::List::to_filter())
.map(
|stream: query::Stream,
auth: query::Auth,
media: query::Media,
hashtag: query::Hashtag,
list: query::List| {
Query {
access_token: auth.access_token,
stream: stream.stream,
media: media.is_truthy(),
hashtag: hashtag.tag,
list: list.list,
}
},
)
.boxed()
}
#[derive(Clone, Debug, Copy, Eq, Hash, PartialEq)]
pub struct Timeline(pub Stream, pub Reach, pub Content);
impl Timeline {
pub fn empty() -> Self {
use {Content::*, Reach::*, Stream::*};
Self(Unset, Local, Notification)
}
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
pub fn to_redis_raw_timeline(&self, hashtag: Option<&String>) -> Result<String, TimelineErr> {
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
use {Content::*, Reach::*, Stream::*};
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
Ok(match self {
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
Timeline(Public, Federated, All) => "timeline:public".into(),
Timeline(Public, Local, All) => "timeline:public:local".into(),
Timeline(Public, Federated, Media) => "timeline:public:media".into(),
Timeline(Public, Local, Media) => "timeline:public:local:media".into(),
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
Timeline(Hashtag(_id), Federated, All) => format!(
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
"timeline:hashtag:{}",
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
hashtag.ok_or_else(|| TimelineErr::MissingHashtag)?
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
),
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
Timeline(Hashtag(_id), Local, All) => format!(
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
"timeline:hashtag:{}:local",
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
hashtag.ok_or_else(|| TimelineErr::MissingHashtag)?
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
),
Timeline(User(id), Federated, All) => format!("timeline:{}", id),
Timeline(User(id), Federated, Notification) => format!("timeline:{}:notification", id),
Timeline(List(id), Federated, All) => format!("timeline:list:{}", id),
Timeline(Direct(id), Federated, All) => format!("timeline:direct:{}", id),
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
Timeline(_one, _two, _three) => Err(TimelineErr::InvalidInput)?,
})
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
}
Reorganize code, pt2 (#112) * Cleanup RedisMsg parsing [WIP] * Add tests to Redis parsing * WIP RedisMsg refactor Committing WIP before trying a different approach * WIP * Refactor RedisConn and Receiver * Finish second reorganization
2020-03-31 00:54:00 +02:00
pub fn from_redis_text(
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
timeline: &str,
cache: &mut LruCache<String, i64>,
) -> Result<Self, TimelineErr> {
Reorganize code, pt2 (#112) * Cleanup RedisMsg parsing [WIP] * Add tests to Redis parsing * WIP RedisMsg refactor Committing WIP before trying a different approach * WIP * Refactor RedisConn and Receiver * Finish second reorganization
2020-03-31 00:54:00 +02:00
let mut id_from_tag = |tag: &str| match cache.get(&tag.to_string()) {
Some(id) => Ok(*id),
None => Err(TimelineErr::InvalidInput), // TODO more specific
};
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
use {Content::*, Reach::*, Stream::*};
Stream events via a watch channel (#128) This squashed commit makes a fairly significant structural change to significantly reduce Flodgatt's CPU usage. Flodgatt connects to Redis in a single (green) thread, and then creates a new thread to handle each WebSocket/SSE connection. Previously, each thread was responsible for polling the Redis thread to determine whether it had a message relevant to the connected client. I initially selected this structure both because it was simple and because it minimized memory overhead – no messages are sent to a particular thread unless they are relevant to the client connected to the thread. However, I recently ran some load tests that show this approach to have unacceptable CPU costs when 300+ clients are simultaneously connected. Accordingly, Flodgatt now uses a different structure: the main Redis thread now announces each incoming message via a watch channel connected to every client thread, and each client thread filters out irrelevant messages. In theory, this could lead to slightly higher memory use, but tests I have run so far have not found a measurable increase. On the other hand, Flodgatt's CPU use is now an order of magnitude lower in tests I've run. This approach does run a (very slight) risk of dropping messages under extremely heavy load: because a watch channel only stores the most recent message transmitted, if Flodgatt adds a second message before the thread can read the first message, the first message will be overwritten and never transmitted. This seems unlikely to happen in practice, and we can avoid the issue entirely by changing to a broadcast channel when we upgrade to the most recent Tokio version (see #75).
2020-04-09 19:32:36 +02:00
Ok(match &timeline.split(':').collect::<Vec<&str>>()[..] {
Reorganize code, pt2 (#112) * Cleanup RedisMsg parsing [WIP] * Add tests to Redis parsing * WIP RedisMsg refactor Committing WIP before trying a different approach * WIP * Refactor RedisConn and Receiver * Finish second reorganization
2020-03-31 00:54:00 +02:00
["public"] => Timeline(Public, Federated, All),
["public", "local"] => Timeline(Public, Local, All),
["public", "media"] => Timeline(Public, Federated, Media),
["public", "local", "media"] => Timeline(Public, Local, Media),
["hashtag", tag] => Timeline(Hashtag(id_from_tag(tag)?), Federated, All),
["hashtag", tag, "local"] => Timeline(Hashtag(id_from_tag(tag)?), Local, All),
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
[id] => Timeline(User(id.parse()?), Federated, All),
[id, "notification"] => Timeline(User(id.parse()?), Federated, Notification),
["list", id] => Timeline(List(id.parse()?), Federated, All),
["direct", id] => Timeline(Direct(id.parse()?), Federated, All),
Reorganize code, pt2 (#112) * Cleanup RedisMsg parsing [WIP] * Add tests to Redis parsing * WIP RedisMsg refactor Committing WIP before trying a different approach * WIP * Refactor RedisConn and Receiver * Finish second reorganization
2020-03-31 00:54:00 +02:00
// Other endpoints don't exist:
[..] => Err(TimelineErr::InvalidInput)?,
})
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
}
Extract tests to separate files (#113) This very minor change moves tests from their current location in submodules within the file under test into submodules in separate files. This is a slight deviation from the normal Rust convention (though only very slight, since the module structure remains the same). However, it is justified here since the tests are fairly verbose and including them in the same file was a bit unwieldy.
2020-03-31 15:05:51 +02:00
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
fn from_query_and_user(q: &Query, user: &UserData, pool: PgPool) -> Result<Self, Rejection> {
use {warp::reject::custom, Content::*, Reach::*, Scope::*, Stream::*};
let id_from_hashtag = || pool.clone().select_hashtag_id(&q.hashtag);
let user_owns_list = || pool.clone().user_owns_list(user.id, q.list);
Ok(match q.stream.as_ref() {
"public" => match q.media {
true => Timeline(Public, Federated, Media),
false => Timeline(Public, Federated, All),
},
"public:local" => match q.media {
true => Timeline(Public, Local, Media),
false => Timeline(Public, Local, All),
},
"public:media" => Timeline(Public, Federated, Media),
"public:local:media" => Timeline(Public, Local, Media),
"hashtag" => Timeline(Hashtag(id_from_hashtag()?), Federated, All),
"hashtag:local" => Timeline(Hashtag(id_from_hashtag()?), Local, All),
"user" => match user.scopes.contains(&Statuses) {
true => Timeline(User(user.id), Federated, All),
false => Err(custom("Error: Missing access token"))?,
},
"user:notification" => match user.scopes.contains(&Statuses) {
true => Timeline(User(user.id), Federated, Notification),
false => Err(custom("Error: Missing access token"))?,
},
"list" => match user.scopes.contains(&Lists) && user_owns_list() {
true => Timeline(List(q.list), Federated, All),
false => Err(warp::reject::custom("Error: Missing access token"))?,
},
"direct" => match user.scopes.contains(&Statuses) {
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
true => Timeline(Direct(*user.id), Federated, All),
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
false => Err(custom("Error: Missing access token"))?,
},
other => {
log::warn!("Request for nonexistent endpoint: `{}`", other);
Err(custom("Error: Nonexistent endpoint"))?
}
})
}
}
Extract tests to separate files (#113) This very minor change moves tests from their current location in submodules within the file under test into submodules in separate files. This is a slight deviation from the normal Rust convention (though only very slight, since the module structure remains the same). However, it is justified here since the tests are fairly verbose and including them in the same file was a bit unwieldy.
2020-03-31 15:05:51 +02:00
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
#[derive(Clone, Debug, Copy, Eq, Hash, PartialEq)]
pub enum Stream {
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
User(Id),
// TODO consider whether List, Direct, and Hashtag should all be `id::Id`s
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
List(i64),
Direct(i64),
Hashtag(i64),
Public,
Unset,
}
Extract tests to separate files (#113) This very minor change moves tests from their current location in submodules within the file under test into submodules in separate files. This is a slight deviation from the normal Rust convention (though only very slight, since the module structure remains the same). However, it is justified here since the tests are fairly verbose and including them in the same file was a bit unwieldy.
2020-03-31 15:05:51 +02:00
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
#[derive(Clone, Debug, Copy, Eq, Hash, PartialEq)]
pub enum Reach {
Local,
Federated,
}
Extract tests to separate files (#113) This very minor change moves tests from their current location in submodules within the file under test into submodules in separate files. This is a slight deviation from the normal Rust convention (though only very slight, since the module structure remains the same). However, it is justified here since the tests are fairly verbose and including them in the same file was a bit unwieldy.
2020-03-31 15:05:51 +02:00
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
#[derive(Clone, Debug, Copy, Eq, Hash, PartialEq)]
pub enum Content {
All,
Media,
Notification,
}
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub enum Scope {
Read,
Statuses,
Notifications,
Lists,
}
pub struct UserData {
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
pub id: Id,
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
pub allowed_langs: HashSet<String>,
pub scopes: HashSet<Scope>,
}
impl UserData {
fn public() -> Self {
Self {
Error handling pt2 (#129) This commit improves error handling in Flodgatt's main request-response loop, including the portions of that loop that were revised in #128. This nearly completes the addition of more explicit error handling, but there will be a smaller part 3 to bring the handling of configuration/Postgres errors into conformity with the style here.
2020-04-11 04:36:03 +02:00
id: Id(-1),
Reorganize code, pt1 (#110) * Prevent Reciever from querying postgres Before this commit, the Receiver would query Postgres for the name associated with a hashtag when it encountered one not in its cache. This ensured that the Receiver never encountered a (valid) hashtag id that it couldn't handle, but caused a extra DB query and made independent sections of the code more entangled than they need to be. Now, we pass the relevant tag name to the Receiver when it first starts managing a new subscription and it adds the tag name to its cache then. * Improve module boundary/privacy * Reorganize Receiver to cut RedisStream * Fix tests for code reorganization Note that this change includes testing some private functionality by exposing it publicly in tests via conditional compilation. This doesn't expose that functionality for the benchmarks, so the benchmark tests do not currently pass without adding a few `pub use` statements. This might be worth changing later, but benchmark tests aren't part of our CI and it's not hard to change when we want to test performance. This change also cuts the benchmark tests that were benchmarking old ways Flodgatt functioned. Those were useful for comparison purposes, but have served their purpose – we've firmly moved away from the older/slower approach. * Fix Receiver for tests
2020-03-27 17:00:48 +01:00
allowed_langs: HashSet::new(),
scopes: HashSet::new(),
}
}
}