From 7f7a24efd3fdb5256190caf37df7a64fe4990890 Mon Sep 17 00:00:00 2001 From: Daniel Jalkut Date: Tue, 5 Dec 2023 07:58:40 -0500 Subject: [PATCH 01/17] Clarify the documenation to reflect that 'description' on media attachments can be null. (#1263) --- content/en/entities/MediaAttachment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/entities/MediaAttachment.md b/content/en/entities/MediaAttachment.md index eadd1bcd..51e3205e 100644 --- a/content/en/entities/MediaAttachment.md +++ b/content/en/entities/MediaAttachment.md @@ -212,7 +212,7 @@ More importantly, there may be another topl-level `focus` Hash object on images ### `description` {#description} **Description:** Alternate text that describes what is in the media attachment, to be used for the visually impaired or when media attachments do not load.\ -**Type:** String\ +**Type:** {{}} String, or null if alternate text was not provided for the media attachment\ **Version history:**\ 2.0.0 - added From 9d83707620445cca3a565ef5e5600e1ff8469f06 Mon Sep 17 00:00:00 2001 From: Petr Kadlec Date: Tue, 5 Dec 2023 15:50:43 +0100 Subject: [PATCH 02/17] Use example.com as example domain (#1227) --- content/en/spec/security.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/content/en/spec/security.md b/content/en/spec/security.md index 1bcdd5d3..f1a20328 100644 --- a/content/en/spec/security.md +++ b/content/en/spec/security.md @@ -16,14 +16,14 @@ menu: For any HTTP request incoming to Mastodon, the Signature header should be attached: ```http -Signature: keyId="https://my-example.com/actor#main-key",headers="(request-target) host date",signature="Y2FiYW...IxNGRiZDk4ZA==" +Signature: keyId="https://my.example.com/actor#main-key",headers="(request-target) host date",signature="Y2FiYW...IxNGRiZDk4ZA==" ``` The three parts of the `Signature:` header can be broken down like so: ```http Signature: - keyId="https://my-example.com/actor#main-key", + keyId="https://my.example.com/actor#main-key", headers="(request-target) host date", signature="Y2FiYW...IxNGRiZDk4ZA==" ``` @@ -32,8 +32,8 @@ The `keyId` should correspond to the actor and the key being used to generate th ```json "publicKey": { - "id": "https://my-example.com/actor#main-key", - "owner": "https://my-example.com/actor", + "id": "https://my.example.com/actor#main-key", + "owner": "https://my.example.com/actor", "publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvXc4vkECU2/CeuSo1wtn\nFoim94Ne1jBMYxTZ9wm2YTdJq1oiZKif06I2fOqDzY/4q/S9uccrE9Bkajv1dnkO\nVm31QjWlhVpSKynVxEWjVBO5Ienue8gND0xvHIuXf87o61poqjEoepvsQFElA5ym\novljWGSA/jpj7ozygUZhCXtaS2W5AD5tnBQUpcO0lhItYPYTjnmzcc4y2NbJV8hz\n2s2G8qKv8fyimE23gY1XrPJg+cRF+g4PqFXujjlJ7MihD9oqtLGxbu7o1cifTn3x\nBfIdPythWu5b4cujNsB3m3awJjVmx+MHQ9SugkSIYXV0Ina77cTNS0M2PYiH1PFR\nTwIDAQAB\n-----END PUBLIC KEY-----\n" }, ``` @@ -68,10 +68,10 @@ GET /users/username/inbox HTTP/1.1 Host: mastodon.example Date: 18 Dec 2019 10:08:46 GMT Accept: application/ld+json; profile="http://www.w3.org/ns/activitystreams" -Signature: keyId="https://my-example.com/actor#main-key",headers="(request-target) host date",signature="Y2FiYW...IxNGRiZDk4ZA==" +Signature: keyId="https://my.example.com/actor#main-key",headers="(request-target) host date",signature="Y2FiYW...IxNGRiZDk4ZA==" ``` -This request is functionally equivalent to saying that `https://my-example.com/actor` is requesting `https://mastodon.example/users/username/inbox` and is proving that they sent this request by signing `(request-target)`, `Host:`, and `Date:` with their private key linked at `keyId`, resulting in the provided `signature`. +This request is functionally equivalent to saying that `https://my.example.com/actor` is requesting `https://mastodon.example/users/username/inbox` and is proving that they sent this request by signing `(request-target)`, `Host:`, and `Date:` with their private key linked at `keyId`, resulting in the provided `signature`. #### Signing POST requests and the Digest header {#digest} @@ -82,12 +82,12 @@ POST /users/username/inbox HTTP/1.1 HOST: mastodon.example Date: 18 Dec 2019 10:08:46 GMT Digest: sha-256=hcK0GZB1BM4R0eenYrj9clYBuyXs/lemt5iWRYmIX0A= -Signature: keyId="https://my-example.com/actor#main-key",headers="(request-target) host date digest",signature="Y2FiYW...IxNGRiZDk4ZA==" +Signature: keyId="https://my.example.com/actor#main-key",headers="(request-target) host date digest",signature="Y2FiYW...IxNGRiZDk4ZA==" Content-Type: application/ld+json; profile="http://www.w3.org/ns/activitystreams" { "@context": "https://www.w3.org/ns/activitystreams", - "actor": "https://example.com/actor", + "actor": "https://my.example.com/actor", "type": "Create", "object": { "type": "Note", @@ -108,12 +108,12 @@ POST /users/username/inbox HTTP/1.1 Host: mastodon.example Date: 18 Dec 2019 10:08:46 GMT Digest: e37e179c75071a291f90a5fd4f848da87b491f1282f7bb8509ef2115b81ee0f4 -Signature: keyId="https://my-example.com/actor#main-key",headers="(request-target) host date digest",signature="Y2FiYW...IxNGRiZDk4ZA==" +Signature: keyId="https://my.example.com/actor#main-key",headers="(request-target) host date digest",signature="Y2FiYW...IxNGRiZDk4ZA==" Content-Type: application/ld+json; profile="http://www.w3.org/ns/activitystreams" { "@context": "https://www.w3.org/ns/activitystreams", - "actor": "https://example.com/actor", + "actor": "https://my.example.com/actor", "type": "Create", "object": { "type": "Note", From 92c64fd62dc4ee2639b176dfbc32cebad9f01a54 Mon Sep 17 00:00:00 2001 From: CSDUMMI <31551856+CSDUMMI@users.noreply.github.com> Date: Tue, 5 Dec 2023 16:05:29 +0100 Subject: [PATCH 03/17] Add documentation for the ONE_CLICK_SSO_LOGIN environment variable (#1300) --- content/en/admin/config.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/content/en/admin/config.md b/content/en/admin/config.md index f8e8aada..2b1d3959 100644 --- a/content/en/admin/config.md +++ b/content/en/admin/config.md @@ -578,6 +578,15 @@ You must serve the files with CORS headers, otherwise some functions of Mastodon #### `OMNIAUTH_ONLY` +#### `ONE_CLICK_SSO_LOGIN` +Enables the `Login or Register` button. +Useful for instances where all authentication takes place using a single +external provider (CAS, SAML or OIDC). + +Enabling this will prevent caching for anonymous sessions. +And, when using OIDC discovery, the identity provider has to be available +before Mastodon starts. + ### LDAP {#ldap} #### `LDAP_ENABLED` From af73402cbef92c926618c0bc319e298b08b4f8d9 Mon Sep 17 00:00:00 2001 From: Elan Hasson <234704+ElanHasson@users.noreply.github.com> Date: Tue, 5 Dec 2023 10:06:17 -0500 Subject: [PATCH 04/17] Update `ES_HOST` to document TLS usage (#1140) * Update `ES_HOST` to document TLS usage * Update elasticsearch.md * Update content/en/admin/config.md * Apply suggestions from code review Co-authored-by: Michael Stanclift --- content/en/admin/config.md | 3 ++- content/en/admin/elasticsearch.md | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/content/en/admin/config.md b/content/en/admin/config.md index 2b1d3959..1e94f2dc 100644 --- a/content/en/admin/config.md +++ b/content/en/admin/config.md @@ -387,7 +387,7 @@ See the [ElasticSearch setup page for details on each setting](../elasticsearch# #### `ES_HOST` -Host of the Elasticsearch server. Defaults to `localhost` +Host of the Elasticsearch server. Defaults to `localhost`. If using TLS, prepend the hostname with `https://`. For example: `https://elastic.example.com`. #### `ES_PORT` @@ -830,3 +830,4 @@ Defaults to `512`. #### `GITHUB_API_TOKEN` Used in a rake task for generating AUTHORS.md from GitHub commit history. + diff --git a/content/en/admin/elasticsearch.md b/content/en/admin/elasticsearch.md index cf33bda3..676561b8 100644 --- a/content/en/admin/elasticsearch.md +++ b/content/en/admin/elasticsearch.md @@ -71,6 +71,8 @@ ES_PRESET= # single_node_cluster, small_cluster or large_cluster # ES_PASS= ``` +_Note_: If using TLS, prepend the hostname with `https://`. For example: `https://elastic.example.com`. + ### Choosing the correct preset The value for `ES_PRESET` depends on the size of your Elasticsearch and will be used to set the number of shards and replica for your indices to the best value for your setup: From 16dde4cfa3c45540f38dcdb3ce5c96a6a5ee8a0b Mon Sep 17 00:00:00 2001 From: Casper Roursgaard Date: Tue, 5 Dec 2023 19:18:58 +0100 Subject: [PATCH 05/17] Fix links to serializers (#1246) Co-authored-by: Casper Roursgaard Christensen --- content/en/entities/Admin_DomainAllow.md | 2 +- content/en/entities/Admin_DomainBlock.md | 2 +- content/en/entities/Admin_EmailDomainBlock.md | 2 +- content/en/entities/Admin_IpBlock.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/entities/Admin_DomainAllow.md b/content/en/entities/Admin_DomainAllow.md index 69ecc1b4..904210c8 100644 --- a/content/en/entities/Admin_DomainAllow.md +++ b/content/en/entities/Admin_DomainAllow.md @@ -53,4 +53,4 @@ aliases: [ {{< page-relref page="methods/admin/domain_allows" caption="admin/domain_allows API methods" >}} -{{< caption-link url="https://github.com/mastodon/mastodon/blob/main/app/serializers/rest/admin/domain_allows_serializer.rb" caption="app/serializers/rest/admin/domain_allows_serializer.rb" >}} \ No newline at end of file +{{< caption-link url="https://github.com/mastodon/mastodon/blob/main/app/serializers/rest/admin/domain_allow_serializer.rb" caption="app/serializers/rest/admin/domain_allow_serializer.rb" >}} \ No newline at end of file diff --git a/content/en/entities/Admin_DomainBlock.md b/content/en/entities/Admin_DomainBlock.md index 069780a1..4b19999a 100644 --- a/content/en/entities/Admin_DomainBlock.md +++ b/content/en/entities/Admin_DomainBlock.md @@ -104,4 +104,4 @@ aliases: [ {{< page-relref page="methods/admin/domain_blocks" caption="admin/domain_blocks API methods" >}} -{{< caption-link url="https://github.com/mastodon/mastodon/blob/main/app/serializers/rest/admin/domain_blocks_serializer.rb" caption="app/serializers/rest/admin/domain_blocks_serializer.rb" >}} \ No newline at end of file +{{< caption-link url="https://github.com/mastodon/mastodon/blob/main/app/serializers/rest/admin/domain_block_serializer.rb" caption="app/serializers/rest/admin/domain_block_serializer.rb" >}} \ No newline at end of file diff --git a/content/en/entities/Admin_EmailDomainBlock.md b/content/en/entities/Admin_EmailDomainBlock.md index 66e4bafe..1eb6f181 100644 --- a/content/en/entities/Admin_EmailDomainBlock.md +++ b/content/en/entities/Admin_EmailDomainBlock.md @@ -118,4 +118,4 @@ aliases: [ {{< page-relref page="methods/admin/email_domain_blocks" caption="admin/email_domain_blocks API methods" >}} -{{< caption-link url="https://github.com/mastodon/mastodon/blob/main/app/serializers/rest/admin/email_domain_blocks_serializer.rb" caption="app/serializers/rest/admin/email_domain_blocks_serializer.rb" >}} \ No newline at end of file +{{< caption-link url="https://github.com/mastodon/mastodon/blob/main/app/serializers/rest/admin/email_domain_block_serializer.rb" caption="app/serializers/rest/admin/email_domain_block_serializer.rb" >}} \ No newline at end of file diff --git a/content/en/entities/Admin_IpBlock.md b/content/en/entities/Admin_IpBlock.md index 53c2a6f8..129ee6c1 100644 --- a/content/en/entities/Admin_IpBlock.md +++ b/content/en/entities/Admin_IpBlock.md @@ -80,4 +80,4 @@ aliases: [ {{< page-relref page="methods/admin/ip_blocks" caption="admin/ip_blocks API methods" >}} -{{< caption-link url="https://github.com/mastodon/mastodon/blob/main/app/serializers/rest/admin/ip_blocks_serializer.rb" caption="app/serializers/rest/admin/ip_blocks_serializer.rb" >}} \ No newline at end of file +{{< caption-link url="https://github.com/mastodon/mastodon/blob/main/app/serializers/rest/admin/ip_block_serializer.rb" caption="app/serializers/rest/admin/ip_block_serializer.rb" >}} \ No newline at end of file From 11ab7b8d7fa7bc6f18757e15e4496a0d56d2040b Mon Sep 17 00:00:00 2001 From: Harmon Date: Tue, 5 Dec 2023 12:19:32 -0600 Subject: [PATCH 06/17] Remove reference to language detection (#1270) Language detection has been removed: https://github.com/mastodon/mastodon/pull/17478 --- content/en/user/preferences.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/user/preferences.md b/content/en/user/preferences.md index 70a2e8a5..32b4f490 100644 --- a/content/en/user/preferences.md +++ b/content/en/user/preferences.md @@ -79,7 +79,7 @@ If you want to see posts that are boosted multiple times be reinserted into your Posts default to public privacy. You can choose to default new posts as unlisted or followers-only instead. For an explanation of post privacy levels, see [Posting to your Mastodon profile > Publishing levels](../posting#privacy). -By default, the language of your posts is automatically detected, but this detection is imprecise and may not be accurate. If you primarily or exclusively post in a certain language, it is a good idea to set that language here. +If you primarily or exclusively post in a certain language, it is a good idea to set that language here. If you often post sensitive media, you can choose to always mark your media as sensitive. From 3c7c5caed1aa3f2035de3eba0130c83790fa379e Mon Sep 17 00:00:00 2001 From: Mahan Date: Tue, 5 Dec 2023 21:50:37 +0330 Subject: [PATCH 07/17] update return empty return types to Empty (#1298) --- content/en/methods/admin/accounts.md | 2 +- content/en/methods/conversations.md | 2 +- content/en/methods/domain_blocks.md | 4 ++-- content/en/methods/emails.md | 2 +- content/en/methods/featured_tags.md | 2 +- content/en/methods/filters.md | 6 +++--- content/en/methods/lists.md | 6 +++--- content/en/methods/notifications.md | 6 +++--- content/en/methods/oauth.md | 2 +- content/en/methods/push.md | 2 +- content/en/methods/scheduled_statuses.md | 2 +- content/en/methods/suggestions.md | 2 +- 12 files changed, 19 insertions(+), 19 deletions(-) diff --git a/content/en/methods/admin/accounts.md b/content/en/methods/admin/accounts.md index 7f53ba32..643a3606 100644 --- a/content/en/methods/admin/accounts.md +++ b/content/en/methods/admin/accounts.md @@ -686,7 +686,7 @@ POST /api/v1/admin/accounts/:id/action HTTP/1.1 Perform an action against an account and log this action in the moderation history. Also resolves any open reports against this account. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `admin:write:accounts`\ **Permissions:** Manage Users, Manage Reports\ **Version history:**\ diff --git a/content/en/methods/conversations.md b/content/en/methods/conversations.md index b7e5cd5d..e5d0de51 100644 --- a/content/en/methods/conversations.md +++ b/content/en/methods/conversations.md @@ -128,7 +128,7 @@ DELETE /api/v1/conversations/:id HTTP/1.1 Removes a conversation from your list of conversations. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:conversations`\ **Version history:**\ 2.6.0 - added diff --git a/content/en/methods/domain_blocks.md b/content/en/methods/domain_blocks.md index 71aa12ab..b61907be 100644 --- a/content/en/methods/domain_blocks.md +++ b/content/en/methods/domain_blocks.md @@ -90,7 +90,7 @@ Block a domain to: - remove all followers from it - prevent following new users from it (but does not remove existing follows) -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:blocks` or `follow`\ **Version:**\ 1.4.0 - added @@ -154,7 +154,7 @@ DELETE /api/v1/domain_blocks HTTP/1.1 Remove a domain block, if it exists in the user's array of blocked domains. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:blocks` or `follow`\ **Version history:**\ 1.4.0 - added diff --git a/content/en/methods/emails.md b/content/en/methods/emails.md index b73db712..baad47aa 100644 --- a/content/en/methods/emails.md +++ b/content/en/methods/emails.md @@ -24,7 +24,7 @@ aliases: [ POST /api/v1/emails/confirmations HTTP/1.1 ``` -**Returns:** Empty object\ +**Returns:** Empty\ **OAuth:** User token issued to the client that created the unconfirmed user\ **Version history:**\ 3.4.0 - added diff --git a/content/en/methods/featured_tags.md b/content/en/methods/featured_tags.md index 793bf62e..15ad5c31 100644 --- a/content/en/methods/featured_tags.md +++ b/content/en/methods/featured_tags.md @@ -133,7 +133,7 @@ DELETE /api/v1/featured_tags/:id HTTP/1.1 Stop promoting a hashtag on your profile. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:accounts`\ **Version history:**\ 3.0.0 - added diff --git a/content/en/methods/filters.md b/content/en/methods/filters.md index 501b394a..657099b7 100644 --- a/content/en/methods/filters.md +++ b/content/en/methods/filters.md @@ -403,7 +403,7 @@ DELETE /api/v2/filters/:id HTTP/1.1 Delete a filter group with the given id. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:filters`\ **Version history:**\ 4.0.0 - added @@ -730,7 +730,7 @@ DELETE /api/v2/filters/keywords/:id HTTP/1.1 Deletes the given filter keyword. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:filters`\ **Version history:**\ 4.0.0 - added @@ -1344,7 +1344,7 @@ If context is not provided properly: DELETE /api/v1/filters/:id HTTP/1.1 ``` -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:filters`\ **Version history:**\ 2.4.3 - added\ diff --git a/content/en/methods/lists.md b/content/en/methods/lists.md index 4ae081ef..46d03a9c 100644 --- a/content/en/methods/lists.md +++ b/content/en/methods/lists.md @@ -299,7 +299,7 @@ If the `replies_policy` is not understood: DELETE /api/v1/lists/:id HTTP/1.1 ``` -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:lists`\ **Version history:**\ 2.1.0 - added @@ -461,7 +461,7 @@ POST /api/v1/lists/:id/accounts HTTP/1.1 Add accounts to the given list. Note that the user must be following these accounts. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:lists`\ **Version history:**\ 2.1.0 - added @@ -530,7 +530,7 @@ DELETE /api/v1/lists/:id/accounts HTTP/1.1 Remove accounts from the given list. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:lists`\ **Version history:**\ 2.1.0 - added diff --git a/content/en/methods/notifications.md b/content/en/methods/notifications.md index 90193341..1dc7ae24 100644 --- a/content/en/methods/notifications.md +++ b/content/en/methods/notifications.md @@ -262,7 +262,7 @@ POST /api/v1/notifications/clear HTTP/1.1 Clear all notifications from the server. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:notifications`\ **Version history:**\ 0.0.0 - added @@ -303,7 +303,7 @@ POST /api/v1/notifications/:id/dismiss HTTP/1.1 Dismiss a single notification from the server. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:notifications`\ **Version history:**\ 1.3.0 - added @@ -350,7 +350,7 @@ POST /api/v1/notifications/dismiss HTTP/1.1 Dismiss a single notification from the server. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:notifications`\ **Version history**:\ 0.0.0 - available\ diff --git a/content/en/methods/oauth.md b/content/en/methods/oauth.md index a2714847..1f03d8b9 100644 --- a/content/en/methods/oauth.md +++ b/content/en/methods/oauth.md @@ -156,7 +156,7 @@ POST /oauth/revoke HTTP/1.1 Revoke an access token to make it no longer valid for use. -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** Public\ **Version history:**\ x.x.x - added diff --git a/content/en/methods/push.md b/content/en/methods/push.md index cac71e9f..12e773cf 100644 --- a/content/en/methods/push.md +++ b/content/en/methods/push.md @@ -298,7 +298,7 @@ DELETE /api/v1/push/subscription HTTP/1.1 Removes the current Web Push API subscription. -**Returns:** none\ +**Returns:** Empty\ **OAuth:** User token + `push`\ **Version history:**\ 2.4.0 - added diff --git a/content/en/methods/scheduled_statuses.md b/content/en/methods/scheduled_statuses.md index 8fa38269..1fe1255b 100644 --- a/content/en/methods/scheduled_statuses.md +++ b/content/en/methods/scheduled_statuses.md @@ -243,7 +243,7 @@ ScheduledStatus is not owned by you or does not exist DELETE /api/v1/scheduled_statuses/:id HTTP/1.1 ``` -**Returns:** empty object\ +**Returns:** Empty\ **OAuth:** User token + `write:statuses`\ **Version history:**\ 2.7.0 - added diff --git a/content/en/methods/suggestions.md b/content/en/methods/suggestions.md index 68744ad9..374f1478 100644 --- a/content/en/methods/suggestions.md +++ b/content/en/methods/suggestions.md @@ -89,7 +89,7 @@ DELETE /api/v1/suggestions/:account_id HTTP/1.1 Remove an account from follow suggestions. -**Returns:** n/a\ +**Returns:** Empty\ **OAuth:** User token + `read`\ **Version history:**\ 2.4.3 - added From 66612d0084587d8fb385f396987b08d31c177e63 Mon Sep 17 00:00:00 2001 From: Mahan Date: Wed, 6 Dec 2023 01:17:25 +0330 Subject: [PATCH 08/17] Add missing type of id. (#1296) --- content/en/methods/accounts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/methods/accounts.md b/content/en/methods/accounts.md index 85204fe9..abb1bb30 100644 --- a/content/en/methods/accounts.md +++ b/content/en/methods/accounts.md @@ -1952,7 +1952,7 @@ Authorization ##### Query parameters id[] -: Array. Check relationships for the provided account IDs. +: Array of String. Check relationships for the provided account IDs. with_suspended : Boolean. Whether relationships should be returned for suspended users, defaults to false. From 472344a48351affda749eff5f3c2397818885047 Mon Sep 17 00:00:00 2001 From: Emelia Smith Date: Wed, 6 Dec 2023 00:03:47 +0100 Subject: [PATCH 09/17] Improve scaling documentation for Streaming (#1335) * Improve scaling documentation for Streaming This reorganizes the documentation a little bit, adds information about the templates systemd files, and some reference numbers for concurrent streaming connections. * Update content/en/admin/scaling.md * Add section on Streaming Server metrics to Scaling.md --- content/en/admin/scaling.md | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-) diff --git a/content/en/admin/scaling.md b/content/en/admin/scaling.md index 15e63854..97f09f82 100644 --- a/content/en/admin/scaling.md +++ b/content/en/admin/scaling.md @@ -36,17 +36,16 @@ The streaming API handles long-lived HTTP and WebSockets connections, through wh - `PORT` controls the port the streaming server will listen on, by default 4000. The `BIND` and `SOCKET` environment variables are also able to be used. - Additionally the shared [database](/admin/config#postgresql) and [redis](/admin/config#redis) environment variables are used. -The streaming API can be use a different subdomain if you want to by setting `STREAMING_API_BASE_URL`, this allows you to have one load balancer for streaming and one for web/API requests. +The streaming API can be use a different subdomain if you want to by setting `STREAMING_API_BASE_URL`, this allows you to have one load balancer for streaming and one for web/API requests. However, this also requires applications to correctly request the streaming URL from the [instance endpoint](/methods/instance/#v2), instead of assuming that it's hosted on the same host as the Web API. -{{< hint style="warning" >}} -Previous versions of Mastodon had a `STREAMING_CLUSTER_NUM` environment variable that made the streaming server use clustering, which started mulitple processes (workers) and used node.js to load balance them. +One process of the streaming server can handle a reasonably high number of connections and throughput, but if you find that a single process isn't handling your instance's load, you can run multiple processes by varying the `PORT` number of each, and then using nginx to load balance traffic to each of those instances. For example, a community of about 50,000 accounts with 10,000-20,000 monthly active accounts, you'll typically have an average concurrent load of about 800-1200 streaming connections. -This interacted with the other settings in ways which made capacity planning difficult, especially when it comes to database connections and CPU resources. By default the streaming server would consume resources on all available CPUs which could cause contention with other software running on that server. Another common issue was that misconfiguring the `STREAMING_CLUSTER_NUM` would exhaust your database connections by opening up a connection pool per cluster worker process, so a `STREAMING_CLUSTER_NUM` of `5` and `DB_POOL` of `10` would potentially consume 50 database connections. +The streaming server also exposes a [Prometheus](https://prometheus.io/) endpoint on `/metrics` with a lot of metrics to help you understand the current load on your mastodon streaming server, some key metrics are: -Now a single streaming server process will only use at maximum `DB_POOL` PostgreSQL connections, and scaling is handled by running more instances of the streaming server. -{{< /hint >}} - -One process can handle a reasonably high number of connections and throughput, but if you find that a single streaming server process isn't handling your instance's load, you can run multiple processes by varying the `PORT` number of each, and then using nginx to load balance traffic to each of those instances. +* `mastodon_streaming_connected_clients`: This is the number of connected clients, tagged by client type (websocket or eventsource) +* `mastodon_streaming_connected_channels`: This is the number of "channels" that are currently subscribed (note that this is much higher than connected clients due to how our internal "system" channels currently work) +* `mastodon_streaming_messages_sent_total`: This is the total number of messages sent to clients since last restart. +* `mastodon_streaming_redis_messages_received_total`: This is the number of messages received from Redis pubsub, and intended to complement [monitoring Redis directly](https://sysdig.com/blog/redis-prometheus/). {{< hint style="info" >}} The more streaming server processes that you run, the more database connections will be consumed on PostgreSQL, so you'll likely want to use PgBouncer, as documented below. @@ -63,6 +62,24 @@ upstream streaming { } ``` +If you're using the distributed systemd files, then you can start up multiple streaming servers with the following commands: + +``` +$ sudo systemctl start mastodon-streaming@4000.service +$ sudo systemctl start mastodon-streaming@4001.service +$ sudo systemctl start mastodon-streaming@4002.service +``` + +By default, `sudo systemctl start mastodon-streaming` starts just one process on port 4000, equivalent to running `sudo systemctl start mastodon-streaming@4000.service`. + +{{< hint style="warning" >}} +Previous versions of Mastodon had a `STREAMING_CLUSTER_NUM` environment variable that made the streaming server use clustering, which started mulitple workers processes and used node.js to load balance them. + +This interacted with the other settings in ways which made capacity planning difficult, especially when it comes to database connections and CPU resources. By default the streaming server would consume resources on all available CPUs which could cause contention with other software running on that server. Another common issue was that misconfiguring the `STREAMING_CLUSTER_NUM` would exhaust your database connections by opening up a connection pool per cluster worker process, so a `STREAMING_CLUSTER_NUM` of `5` and `DB_POOL` of `10` would potentially consume 50 database connections. + +Now a single streaming server process will only use at maximum `DB_POOL` PostgreSQL connections, and scaling is handled by running more instances of the streaming server. +{{< /hint >}} + ### Background processing (Sidekiq) {#sidekiq} Many tasks in Mastodon are delegated to background processing to ensure the HTTP requests are fast, and to prevent HTTP request aborts from affecting the execution of those tasks. Sidekiq is a single process, with a configurable number of threads. From 1f5f33271c47fa32012a921186e99677c8487336 Mon Sep 17 00:00:00 2001 From: Mark Doliner Date: Wed, 6 Dec 2023 06:56:14 -0500 Subject: [PATCH 10/17] Remove unnecessary steps from Ruby instructions. (#1136) The `./configure` and `make` steps appear to be obsolete. --- content/en/admin/install.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/en/admin/install.md b/content/en/admin/install.md index db9ca626..ea5080c7 100644 --- a/content/en/admin/install.md +++ b/content/en/admin/install.md @@ -74,7 +74,6 @@ And proceed to install rbenv and rbenv-build: ```bash git clone https://github.com/rbenv/rbenv.git ~/.rbenv -cd ~/.rbenv && src/configure && make -C src echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc echo 'eval "$(rbenv init -)"' >> ~/.bashrc exec bash From f2b3dfb70a03d7abbd313c683d218985950e286d Mon Sep 17 00:00:00 2001 From: JakePaustian <87162217+JakePaustian@users.noreply.github.com> Date: Wed, 6 Dec 2023 05:57:15 -0600 Subject: [PATCH 11/17] An important parameter is missing from the API doc on editing a post #1239 (#1351) * Update content/en/methods/statuses.md --- content/en/methods/statuses.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/content/en/methods/statuses.md b/content/en/methods/statuses.md index 0640268a..3fc2e88c 100644 --- a/content/en/methods/statuses.md +++ b/content/en/methods/statuses.md @@ -1,3 +1,4 @@ + --- title: statuses API methods description: Publish, interact, and view information about statuses. @@ -1478,6 +1479,9 @@ language media_ids[] : Array of String. Include Attachment IDs to be attached as media. If provided, `status` becomes optional, and `poll` cannot be used. +media_attributes[][] +: Array of String. Each array includes id, description, and focus. + poll[options][] : Array of String. Possible answers to the poll. If provided, `media_ids` cannot be used, and `poll[expires_in]` must be provided. @@ -1852,3 +1856,4 @@ Status does not exist or is private. {{< caption-link url="https://github.com/mastodon/mastodon/blob/main/app/controllers/api/v1/statuses/reblogs_controller.rb" caption="app/controllers/api/v1/statuses/reblogs_controller.rb" >}} {{< caption-link url="https://github.com/mastodon/mastodon/blob/main/app/controllers/api/v1/statuses/sources_controller.rb" caption="app/controllers/api/v1/statuses/sources_controller.rb" >}} + From 00528a70a61ba788f97153b1f5bf6ba5794cc359 Mon Sep 17 00:00:00 2001 From: Christian Date: Thu, 7 Dec 2023 00:09:36 +0100 Subject: [PATCH 12/17] Update tootctl.md (#1243) Add missing line break --- content/en/admin/tootctl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/admin/tootctl.md b/content/en/admin/tootctl.md index cddc70a8..d0976be4 100644 --- a/content/en/admin/tootctl.md +++ b/content/en/admin/tootctl.md @@ -669,7 +669,7 @@ Removes locally cached copies of media attachments, avatars or profile headers f **Version history:**\ 2.5.0 - added\ -2.6.2 - show freed disk space +2.6.2 - show freed disk space\ 4.1.0 - added --prune-profiles, --remove-headers, and --include-follows. From e8f0e640600498fd8b976962b178249873a43f7f Mon Sep 17 00:00:00 2001 From: Daniil Suvorov Date: Thu, 7 Dec 2023 02:10:11 +0300 Subject: [PATCH 13/17] fix(client/intro): Nested parameters JSON format (#1254) --- content/en/client/intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/client/intro.md b/content/en/client/intro.md index 2f14aad2..432f43ea 100644 --- a/content/en/client/intro.md +++ b/content/en/client/intro.md @@ -113,7 +113,7 @@ As JSON, hashes are formatted like so: { "source": { "privacy": "public", - "language", "en" + "language": "en" } } ``` From e6656d3d37b54bf56fdbce997ea8cfc70d355a21 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tantek=20=C3=87elik?= Date: Thu, 7 Dec 2023 03:08:44 -0800 Subject: [PATCH 14/17] fix "ActivityPub" format mention to be AS 2.0 (#1202) There is no "ActivityPub JSON format", ActivityPub uses the Activity Streams 2.0 format https://www.w3.org/TR/activitystreams-core/. --- content/en/user/moving.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/user/moving.md b/content/en/user/moving.md index b27c5ef6..b2a73ef5 100644 --- a/content/en/user/moving.md +++ b/content/en/user/moving.md @@ -13,7 +13,7 @@ menu: At any time you want, you can go to Settings > Export and download a CSV file for your current followed accounts, your currently created lists, your currently blocked accounts, your currently muted accounts, and your currently blocked domains. Your following, blocking, muting, and domain-blocking lists can be imported at Settings > Import, where they can either be merged or overwritten. -Requesting an archive of your posts and media can be done once every 7 days, and can be downloaded in ActivityPub JSON format. Mastodon currently does not support importing posts or media due to technical limitations, but your archive can be viewed by any software that understands how to parse ActivityPub documents. +Requesting an archive of your posts and media can be done once every 7 days, and can be downloaded in Activity Streams 2.0 JSON format. Mastodon currently does not support importing posts or media due to technical limitations, but your archive can be viewed by any software that understands how to parse Activity Streams 2.0 documents. ## Redirecting or moving your profile {#migration} From 44e96e9428db7a24b5ac4a2925141512e001e417 Mon Sep 17 00:00:00 2001 From: ^cypher <37557436+carrotcypher@users.noreply.github.com> Date: Thu, 7 Dec 2023 10:33:36 -0300 Subject: [PATCH 15/17] Proposal for adding minimal documentation for undocumented moderation API (webhook) (#1329) * Update moderation.md --- content/en/admin/moderation.md | 226 +++++++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) diff --git a/content/en/admin/moderation.md b/content/en/admin/moderation.md index ed7c1565..d04bee99 100644 --- a/content/en/admin/moderation.md +++ b/content/en/admin/moderation.md @@ -90,3 +90,229 @@ sudo iptables -I INPUT 1 -m set --match-set spambots src -j DROP ``` Be careful not to lock yourself out of your machine. + +### Webhooks for moderation-level events {#report-events-webhook} + +Webhooks can be created to facilitate automation through the moderation API by notifying applications about system events in real-time. This also enables integrations with chat apps like Discord, IRC and Slack, helping moderator coordination. + +The X-Hub-Signature header adopted from the WebSub spec can be optionally used to verify that the payloads are authentic. + +Events currently supported: + +* account.approved +* account.created +* account.updated +* report.created +* report.updated +* status.created +* status.updated + + + +#### Example payload: + +```json + +{ + "event":"report.created", + "created_at":"2023-10-26T13:34:00.351Z", + "object":{ + "id":"8437", + "action_taken":false, + "action_taken_at":null, + "category":"violation", + "comment":"", + "forwarded":true, + "created_at":"2023-10-26T13:34:00.348Z", + "updated_at":"2023-10-26T13:34:00.348Z", + "account":{ + "id":"123456789", + "username":"bobisaburger", + "domain":null, + "created_at":"2023-07-13T04:39:22.493Z", + "email":"bobisaburger@emailservice.com", + "ip":"12.34.56.78", + "confirmed":true, + "suspended":false, + "silenced":false, + "sensitized":false, + "disabled":false, + "approved":true, + "locale":"en", + "invite_request":"I would love to be a member of your instance!", + "ips":[ + { + "ip":"12.34.56.78", + "used_at":"2023-07-13T04:45:31.835Z" + }, + { + "ip":"98.76.54.32", + "used_at":"2023-07-13T04:39:22.722Z" + } + ], + "account":{ + "id":"123456789", + "username":"bobisaburger", + "acct":"bobisaburger", + "display_name":"bobisaburger", + "locked":false, + "bot":false, + "discoverable":null, + "group":false, + "created_at":"2023-07-13T00:00:00.000Z", + "note":"", + "url":"https://mastodonwebsite/@bobisaburger", + "uri":"https://mastodonwebsite/users/bobisaburger", + "avatar":"https://locationofavatar.com/image.jpg", + "avatar_static":"https://locationofavatar.com/image.jpg", + "header":"locationofheader.com/image.jpg", + "header_static":"locationofheader.com/image.jpg", + "followers_count":100, + "following_count":200, + "statuses_count":9, + "last_status_at":"2023-08-05", + "noindex":true, + "emojis":[ + + ], + "roles":[ + + ], + "fields":[ + + ] + }, + "role":{ + "id":"-99", + "name":"", + "permissions":"65536", + "color":"", + "highlighted":false + } + }, + "target_account":{ + "id":"123454321", + "username":"cheeseperson", + "domain":"someothermastodonsite.com", + "created_at":"2023-08-20T00:00:00.000Z", + "email":null, + "ip":null, + "confirmed":null, + "suspended":false, + "silenced":false, + "sensitized":false, + "disabled":null, + "approved":null, + "locale":null, + "invite_request":null, + "ips":null, + "account":{ + "id":"123454321", + "username":"cheeseperson", + "acct":"cheeseperson@someothermastodonsite.com", + "display_name":"cheeseperson", + "locked":false, + "bot":false, + "discoverable":false, + "group":false, + "created_at":"2023-08-20T00:00:00.000Z", + "note":"", + "url":"https://someothermastodonsite.com/@cheeseperson", + "uri":"https://someothermastodonsite.com/users/cheeseperson", + "avatar":"https://someothermastadonsite.com/avatars/original/missing.png", + "avatar_static":"https://someothermastadonsite.com/avatars/original/missing.png", + "header":"locationofheader.com/image.jpg", + "header_static":"locationofheader.com/image.jpg", + "followers_count":2, + "following_count":2, + "statuses_count":95, + "last_status_at":"2023-10-26", + "emojis":[ + + ], + "fields":[ + + ] + }, + "role":null + }, + "assigned_account":null, + "action_taken_by_account":null, + "statuses":[ + { + "id":"12345678987654321", + "created_at":"2023-10-26T11:29:13.000Z", + "in_reply_to_id":"1918282746465", + "in_reply_to_account_id":"101010101010", + "sensitive":false, + "spoiler_text":"", + "visibility":"public", + "language":"de", + "uri":"https://someothermastodonsite.com/users/cheeseperson/statuses/111301083360371621", + "url":"https://someothermastodonsite.com/@cheeseperson/111301083360371621", + "replies_count":0, + "reblogs_count":0, + "favourites_count":0, + "edited_at":"2023-10-26T11:30:31.000Z", + "content":"

Here is some content

", + "reblog":null, + "account":{ + "id":"123454321", + "username":"cheeseperson", + "acct":"cheeseperson@someothermastodonsite.com", + "display_name":"cheeseperson", + "locked":false, + "bot":false, + "discoverable":false, + "group":false, + "created_at":"2023-08-20T00:00:00.000Z", + "note":"", + "url":"https://someothermastodonsite.com/@cheeseperson", + "uri":"https://someothermastodonsite.com/users/cheeseperson", + "avatar":"https://someothermastadonsite.com/avatars/original/missing.png", + "avatar_static":"https://someothermastadonsite.com/avatars/original/missing.png", + "header":"locationofheader.com/image.jpg", + "header_static":"locationofheader.com/image.jpg", + "followers_count":2, + "following_count":2, + "statuses_count":95, + "last_status_at":"2023-10-26", + "emojis":[ + + ], + "fields":[ + + ] + }, + "media_attachments":[ + + ], + "mentions":[ + { + "id":"101010101010", + "username":"thirdperson", + "url":"https://thirdpersonsinstance.com/@thirdperson", + "acct":"thirdperson@emailwebsite.com" + } + ], + "tags":[ + + ], + "emojis":[ + + ], + "card":null, + "poll":null + } + ], + "rules":[ + { + "id":"2", + "text":"Don't be a meanie!" + } + ] + } +} + + +``` From c2d93f344f4d230f804f3792ab835afdaee64227 Mon Sep 17 00:00:00 2001 From: cucmberium Date: Thu, 7 Dec 2023 22:36:38 +0900 Subject: [PATCH 16/17] Added missing parameter information in Filter (#1312) --- content/en/methods/filters.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/content/en/methods/filters.md b/content/en/methods/filters.md index 657099b7..0ad24a23 100644 --- a/content/en/methods/filters.md +++ b/content/en/methods/filters.md @@ -863,6 +863,11 @@ Add a status filter to the current filter group. Authorization : {{}} Provide this header with `Bearer ` to gain authorized access to this API method. +##### Form data parameters + +status_id +: {{}} String. The status ID to be added to the filter group. + #### Response ##### 200: OK From 0982664688c8607e34942ef8ac8b83104ef54493 Mon Sep 17 00:00:00 2001 From: Isa S Date: Thu, 7 Dec 2023 14:52:29 +0100 Subject: [PATCH 17/17] Add instructions for seperate sidekiq redis (#1170) * Add instructions for seperate sidekiq redis Co-authored-by: Andy Piper --- content/en/admin/scaling.md | 67 +++++++++++++++++++++++++++++++++++-- 1 file changed, 64 insertions(+), 3 deletions(-) diff --git a/content/en/admin/scaling.md b/content/en/admin/scaling.md index 97f09f82..ed54d74f 100644 --- a/content/en/admin/scaling.md +++ b/content/en/admin/scaling.md @@ -304,16 +304,77 @@ Then use `\q` to quit. ## Separate Redis for cache {#redis} -Redis is used widely throughout the application, but some uses are more important than others. Home feeds, list feeds, and Sidekiq queues as well as the streaming API are backed by Redis and that’s important data you wouldn’t want to lose (even though the loss can be survived, unlike the loss of the PostgreSQL database - never lose that!). However, Redis is also used for volatile cache. If you are at a stage of scaling up where you are worried if your Redis can handle everything, you can use a different Redis database for the cache. In the environment, you can specify `CACHE_REDIS_URL` or individual parts like `CACHE_REDIS_HOST`, `CACHE_REDIS_PORT` etc. Unspecified parts fallback to the same values as without the cache prefix. +Redis is used widely throughout the application, but some uses are more important than others. Home feeds, list feeds, and Sidekiq queues as well as the streaming API are backed by Redis and that’s important data you wouldn’t want to lose (even though the loss can be survived, unlike the loss of the PostgreSQL database - never lose that!). However, Redis is also used for volatile cache. If you are at a stage of scaling up where you are worried about whether your Redis can handle everything, you can use a different Redis database for the cache. In the environment, you can specify `CACHE_REDIS_URL` or individual parts like `CACHE_REDIS_HOST`, `CACHE_REDIS_PORT` etc. Unspecified parts fallback to the same values as without the cache prefix. As far as configuring the Redis database goes, basically you can get rid of background saving to disk, since it doesn’t matter if the data gets lost on restart and you can save some disk I/O on that. You can also add a maximum memory limit and a key eviction policy, for that, see this guide: [Using Redis as an LRU cache](https://redis.io/topics/lru-cache) +## Seperate Redis for Sidekiq {#redis-sidekiq} + +Redis is used in Sidekiq to keep track of its locks and queue. Although in general the performance gain is not that big, some instances may benefit from having a seperate Redis instance for Sidekiq. + +In the environment file, you can specify `SIDEKIQ_REDIS_URL` or individual parts like `SIDEKIQ_REDIS_HOST`, `SIDEKIQ_REDIS_PORT` etc. Unspecified parts fallback to the same values as without the `SIDEKIQ_` prefix. + +Creating a seperate Redis instance for Sidekiq is relatively simple: + +Start by making a copy of the default redis systemd service: +```bash +cp /etc/systemd/system/redis.service /etc/systemd/system/redis-sidekiq.service +``` + +In the `redis-sidekiq.service` file, change the following values: +```bash +ExecStart=/usr/bin/redis-server /etc/redis/redis-sidekiq.conf --supervised systemd --daemonize no +PIDFile=/run/redis/redis-server-sidekiq.pid +ReadWritePaths=-/var/lib/redis-sidekiq +Alias=redis-sidekiq.service +``` + +Make a copy of the Redis configuration file for the new Sidekiq Redis instance + +```bash +cp /etc/redis/redis.conf /etc/redis/redis-sidekiq.conf +``` + +In this `redis-sidekiq.conf` file, change the following values: +```bash +port 6479 +pidfile /var/run/redis/redis-server-sidekiq.pid +logfile /var/log/redis/redis-server-sidekiq.log +dir /var/lib/redis-sidekiq +``` + +Before starting the new Redis instance, create a data directory: + +```bash +mkdir /var/lib/redis-sidekiq +chown redis /var/lib/redis-sidekiq +``` + +Start the new Redis instance: + +```bash +systemctl enable --now redis-sidekiq +``` + +Update your environment, add the following line: + +```bash +SIDEKIQ_REDIS_URL=redis://127.0.0.1:6479/ +``` + +Restart Mastodon to use the new Redis instance, make sure to restart both web and Sidekiq (otherwise, one of them will still be working from the wrong instance): + +```bash +systemctl restart mastodon-web.service +systemctl restart redis-sidekiq.service +``` + ## Read-replicas {#read-replicas} To reduce the load on your Postgresql server, you may wish to setup hot streaming replication (read replica). [See this guide for an example](https://cloud.google.com/community/tutorials/setting-up-postgres-hot-standby). You can make use of the replica in Mastodon in these ways: -- The streaming API server does not issue writes at all, so you can connect it straight to the replica. But it’s not querying the database very often anyway so the impact of this is little. -- Use the Makara driver in the web processes, so that writes go to the primary database, while reads go to the replica. Let’s talk about that. +* The streaming API server does not issue writes at all, so you can connect it straight to the replica (it is not querying the database very often anyway, so the impact of this is small). +* Use the Makara driver in the web and Sidekiq processes, so that writes go to the master database, while reads go to the replica. Let’s talk about that. {{< hint style="warning" >}} Read replicas are currently not supported for the Sidekiq processes, and using them will lead to failing jobs and data loss.