Improve Elasticsearch documentation with `ES_PRESET` and security settings (#1279)

* Improve Elasticsearch documentation with `ES_PRESET` and security settings

* Fix small errors, and make the ES configuration no longer optional

content/en/admin/config.md

@@ -367,12 +367,23 @@ Defaults to value of `REDIS_NAMESPACE`.
 
 ### Elasticsearch {#elasticsearch}
 
-{{< page-ref page="admin/optional/elasticsearch" >}}
+{{< page-ref page="admin/elasticsearch" >}}
 
 #### `ES_ENABLED`
 
 If set to `true`, Mastodon will use Elasticsearch for its search functions.
 
+#### `ES_PRESET`
+
+It controls the ElasticSearch indices configuration (number of shards and replica).
+
+Possible values are:
+- `single_node_cluster` (default)
+- `small_cluster`
+- `large_cluster`
+
+See the [ElasticSearch setup page for details on each setting](../elasticsearch#choosing-the-correct-preset).
+
 #### `ES_HOST`
 
 Host of the Elasticsearch server. Defaults to `localhost`

content/en/admin/elasticsearch.md

@@ -1,16 +1,30 @@
 ---
-title: Full-text search
-description: Setting up Elasticsearch to search for statuses authored, favourited, or mentioned in.
+title: Configuring full-text search
+description: Setting up Elasticsearch to search for statuses (authored, favourited, or mentioned), public indexable status, and accounts
 menu:
   docs:
-    weight: 10
-    parent: admin-optional
+    weight: 40
+    parent: admin
 ---
 
-Mastodon supports full-text search when Elasticsearch is available. Mastodon’s full-text search allows logged in users to find results from their own statuses, their mentions, their favourites, and their bookmarks. It deliberately does not allow searching for arbitrary strings in the entire database.
+Mastodon supports full-text search when Elasticsearch is available. It is strongly recommended to configure this feature.
+
+Mastodon’s full-text search allows logged in users to find results from:
+- public statuses from account that opted into appearing in search results
+- their own statuses
+- their mentions
+- their favourites
+- their bookmarks
+- accounts (display name, usernames and bios)
+
+It deliberately does not allow searching for arbitrary strings in the entire database.
 
 ## Installing Elasticsearch {#install}
 
+{{< hint style="info" >}}
+Mastodon is tested with ElasticSearch version 7. It should support OpenSearch, as well as ElectisSearch versions 6 and 8, but those setups are not officially supported.
+{{< /hint >}}
+
 Elasticsearch requires a Java runtime. If you don’t have Java already installed, do it now. Assuming you are logged in as `root`:
 
 ```bash
@@ -35,10 +49,6 @@ apt install elasticsearch
 **Security warning:** By default, Elasticsearch is supposed to bind to localhost only, i.e. be inaccessible from the outside network. You can check which address Elasticsearch binds to by looking at `network.host` within `/etc/elasticsearch/elasticsearch.yml`. Consider that anyone who can access Elasticsearch can access and modify any data within it, as there is no authentication layer. So it’s really important that the access is secured. Having a firewall that only exposes the 22, 80 and 443 ports is advisable, as outlined in the [main installation instructions](../../prerequisites/#install-a-firewall-and-only-whitelist-ssh-http-and-https-ports). If you have a multi-host setup, you must know how to secure internal traffic.
 {{< /hint >}}
 
-{{< hint style="danger" >}}
-**Security warning:** Elasticsearch versions between `2.0` and `2.14.1` are affected by an [exploit](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228) in the `log4j` library. If affected, please refer to the [temporary mitigation](https://github.com/elastic/elasticsearch/issues/81618#issuecomment-991000240) from the Elasticsearch issue tracker.
-{{< /hint >}}
-
 To start Elasticsearch:
 
 ```bash
@@ -54,10 +64,74 @@ Edit `.env.production` to add the following variables:
 ES_ENABLED=true
 ES_HOST=localhost
 ES_PORT=9200
+ES_PRESET= # single_node_cluster, small_cluster or large_cluster
+ES_USER=
+ES_PASS=
 ```
 
+### Choosing the correct preset
+
+The value for `ES_PRESET` depends on the size of your Elasticsearch and will be used to set the number of shards and replica for your indices to the best value for your setup:
+- `single_node_cluster` if you only have one node in your Elasticsearch cluster. Indices will be configured without any replica
+- `small_cluster` if you have less than 6 nodes in your cluster. Indices will be configured with 1 replica
+- `large_cluster` if you have 6 or more nodes in your cluster. Indices will be configured with more shards than with the `small_cluster` setting, to allow them to be distributed over more nodes
+
 If you have multiple Mastodon servers on the same machine, and you are planning to use the same Elasticsearch installation for all of them, make sure that all of them have unique `REDIS_NAMESPACE` in their configurations, to differentiate the indices. If you need to override the prefix of the Elasticsearch indices, you can set `ES_PREFIX` directly.
 
+### Security
+
+By default, Elasticsearch does not handle any authentication and every request is made with full admin permission. We strongly advise you to configure Elasticsearch security features on your cluster.
+
+To configure it, please refer [to the official documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html). It will guide you through:
+- Enabling the security features (`xpack.security.enabled: true`)
+- Creating password for built-in users
+
+Once done, you can create a custom role for Mastodon to connect.
+
+For example (please adapt this snippet to use your Elastic admin password):
+
+```sh
+curl -X POST -u elastic:admin_password "localhost:9200/_security/role/mastodon_full_access?pretty" -H 'Content-Type: application/json' -d'
+{
+  "cluster": ["monitor"],
+  "indices": [{
+    "names": ["*"],
+    "privileges": ["read", "monitor", "write", "manage"]
+  }]
+}
+'
+```
+
+[Elasticsearch documentation for role creation](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-api-put-role.html)
+
+Once the role is created, you can create a user for the Mastodon server to use, and assign it the role.
+
+For example (please adapt this snippet to use your Elastic admin password, and customize your new user `mastodon` user password):
+
+```sh
+curl -X POST -u elastic:admin_password "localhost:9200/_security/user/mastodon?pretty" -H 'Content-Type: application/json' -d'
+{
+  "password" : "l0ng-r4nd0m-p@ssw0rd",
+  "roles" : ["mastodon_full_access"]
+}
+'
+```
+
+[Elasticsearch documentation for user creation](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-api-put-user.html)
+
+Once this is done, you need to configure Mastodon to use the credentials for your newly created user.
+
+In `.env.production`, adjust your configuration:
+
+```bash
+ES_USER=mastodon
+ES_PASS=l0ng-r4nd0m-p@ssw0rd
+```
+
+You are all set, and your Elasticsearch server should be much more secure!
+
+### Populate the indices
+
 After saving the new configuration, restart Mastodon processes for it to take effect:
 
 ```bash
@@ -151,4 +225,3 @@ diff --git a/app/chewy/tags_index.rb b/app/chewy/tags_index.rb
 
        edge_ngram: {
 ```
-

content/en/admin/optional.md

@@ -9,7 +9,6 @@ menu:
 
 Mastodon offers a few optional features that can be used if needed.
 
-- [Full-text search](./elasticsearch/)
 - [Object storage](./object-storage/)
 - [Hidden services](./tor/)
 - [Single Sign On](./sso/)