From 94aa9f71a7cc3afb420f46b2a6b351f3bd5339a8 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Sun, 10 Dec 2023 12:40:20 -0600 Subject: [PATCH] Consistent use of Postgres and PgBouncer (#1353) * fix pgbouncer * pghero --- content/en/admin/config.md | 2 +- content/en/admin/migrating.md | 12 ++++++------ content/en/admin/scaling.md | 24 ++++++++++++------------ content/en/dev/setup.md | 2 +- content/en/entities/Role.md | 2 +- content/zh-cn/admin/migrating.md | 10 +++++----- content/zh-cn/admin/scaling.md | 4 ++-- 7 files changed, 28 insertions(+), 28 deletions(-) diff --git a/content/en/admin/config.md b/content/en/admin/config.md index ed36827a..d036a8eb 100644 --- a/content/en/admin/config.md +++ b/content/en/admin/config.md @@ -315,7 +315,7 @@ Defines how many database connections to pool in the process. This value should #### `DB_SSLMODE` -Postgres's [SSL mode](https://www.postgresql.org/docs/10/libpq-ssl.html). Defaults to `prefer`. +PostgreSQL [SSL mode](https://www.postgresql.org/docs/10/libpq-ssl.html). Defaults to `prefer`. #### `DATABASE_URL` diff --git a/content/en/admin/migrating.md b/content/en/admin/migrating.md index f108daf0..664af51d 100644 --- a/content/en/admin/migrating.md +++ b/content/en/admin/migrating.md @@ -17,7 +17,7 @@ This guide was written with Ubuntu Server in mind; your mileage may vary for oth 1. Set up a new Mastodon server using the [Production Guide]({{< relref "install" >}}) (however, don’t run `mastodon:setup`). 2. Stop Mastodon on the old server (e.g. `systemctl stop 'mastodon-*.service'`). -3. Dump and load the Postgres database using the instructions below. +3. Dump and load the PostgreSQL database using the instructions below. 4. Copy the `system/` files using the instructions below. (Note: if you’re using S3, you can skip this step.) 5. Copy the `.env.production` file. 6. Run `RAILS_ENV=production bundle exec rails assets:precompile` to compile Mastodon @@ -34,18 +34,18 @@ This guide was written with Ubuntu Server in mind; your mileage may vary for oth At a high level, you’ll need to copy over the following: * The `~/live/public/system` directory, which contains user-uploaded images and videos (if using S3, you don’t need this) -* The Postgres database (using [pg_dump](https://www.postgresql.org/docs/9.1/static/backup-dump.html)) +* The PostgreSQL database (using [pg_dump](https://www.postgresql.org/docs/9.1/static/backup-dump.html)) * The `~/live/.env.production` file, which contains server config and secrets Less crucially, you’ll probably also want to copy the following for convenience: * The nginx config (under `/etc/nginx/sites-available/default`) * The systemd config files (`/etc/systemd/system/mastodon-*.service`), which may contain your server tweaks and customizations -* The pgbouncer configuration under `/etc/pgbouncer` (if you’re using it) +* The PgBouncer configuration under `/etc/pgbouncer` (if you’re using it) -### Dump and load Postgres {#dump-and-load-postgres} +### Dump and load PostgreSQL {#dump-and-load-postgresql} -Instead of running `mastodon:setup`, we’re going to create an empty Postgres database using the `template0` database (which is useful when restoring a Postgres dump, [as described in the pg_dump documentation](https://www.postgresql.org/docs/9.1/static/backup-dump.html#BACKUP-DUMP-RESTORE)). +Instead of running `mastodon:setup`, we’re going to create an empty PostgreSQL database using the `template0` database (which is useful when restoring a PostgreSQL dump, [as described in the pg_dump documentation](https://www.postgresql.org/docs/9.1/static/backup-dump.html#BACKUP-DUMP-RESTORE)). Run this as the `mastodon` user on your old system: @@ -80,7 +80,7 @@ You’ll want to re-run this if any of the files on the old server change. You should also copy over the `.env.production` file, which contains secrets. -Optionally, you may copy over the nginx, systemd, and pgbouncer config files, or rewrite them from scratch. +Optionally, you may copy over the nginx, systemd, and PgBouncer config files, or rewrite them from scratch. ### During migration {#during-migration} diff --git a/content/en/admin/scaling.md b/content/en/admin/scaling.md index 0c323e0e..13071415 100644 --- a/content/en/admin/scaling.md +++ b/content/en/admin/scaling.md @@ -123,13 +123,13 @@ As a solution, it is possible to start different Sidekiq processes for the queue **Make sure you only have one `scheduler` queue running!!** -## Transaction pooling with pgBouncer {#pgbouncer} +## Transaction pooling with PgBouncer {#pgbouncer} ### Why you might need PgBouncer {#pgbouncer-why} -If you start running out of available Postgres connections (the default is 100) then you may find PgBouncer to be a good solution. This document describes some common gotchas as well as good configuration defaults for Mastodon. +If you start running out of available PostgreSQL connections (the default is 100) then you may find PgBouncer to be a good solution. This document describes some common gotchas as well as good configuration defaults for Mastodon. -User roles with `DevOps` permissions in Mastodon can monitor the current usage of Postgres connections through the PgHero link in the Administration view. Generally, the number of connections open is equal to the total threads in Puma, Sidekiq, and the streaming API combined. +User roles with `DevOps` permissions in Mastodon can monitor the current usage of PostgreSQL connections through the PgHero link in the Administration view. Generally, the number of connections open is equal to the total threads in Puma, Sidekiq, and the streaming API combined. ### Installing PgBouncer {#pgbouncer-install} @@ -143,7 +143,7 @@ sudo apt install pgbouncer #### Setting a password {#pgbouncer-password} -First off, if your `mastodon` user in Postgres is set up without a password, you will need to set a password. +First off, if your `mastodon` user in PostgreSQL is set up without a password, you will need to set a password. Here’s how you might reset the password: @@ -193,7 +193,7 @@ In both cases, the password is just `password`. Edit `/etc/pgbouncer/pgbouncer.ini` -Add a line under `[databases]` listing the Postgres databases you want to connect to. Here we’ll just have PgBouncer use the same username/password and database name to connect to the underlying Postgres database: +Add a line under `[databases]` listing the PostgreSQL databases you want to connect to. Here we’ll just have PgBouncer use the same username/password and database name to connect to the underlying PostgreSQL database: ```text [databases] @@ -219,13 +219,13 @@ Make sure the `pgbouncer` user is an admin: admin_users = pgbouncer ``` -Mastodon requires a different pooling mode than the default session-based one. Specifically, it needs a transaction-based pooling mode. This means that a Postgres connection is established at the start of a transaction and terminated upon its completion. Therefore, it's essential to change the `pool_mode` setting from `session` to `transaction`: +Mastodon requires a different pooling mode than the default session-based one. Specifically, it needs a transaction-based pooling mode. This means that a PostgreSQL connection is established at the start of a transaction and terminated upon its completion. Therefore, it's essential to change the `pool_mode` setting from `session` to `transaction`: ```ini pool_mode = transaction ``` -Next up, `max_client_conn` defines how many connections PgBouncer itself will accept, and `default_pool_size` puts a limit on how many Postgres connections will be opened under the hood. (In PgHero the number of connections reported will correspond to `default_pool_size` because it has no knowledge of PgBouncer.) +Next up, `max_client_conn` defines how many connections PgBouncer itself will accept, and `default_pool_size` puts a limit on how many PostgreSQL connections will be opened under the hood. (In PgHero the number of connections reported will correspond to `default_pool_size` because it has no knowledge of PgBouncer.) The defaults are fine to start, and you can always increase them later: @@ -234,7 +234,7 @@ max_client_conn = 100 default_pool_size = 20 ``` -Don’t forget to reload or restart pgbouncer after making your changes: +Don’t forget to reload or restart PgBouncer after making your changes: ```bash sudo systemctl reload pgbouncer @@ -242,7 +242,7 @@ sudo systemctl reload pgbouncer #### Debugging that it all works {#pgbouncer-debug} -You should be able to connect to PgBouncer just like you would with Postgres: +You should be able to connect to PgBouncer just like you would with PostgreSQL: ```bash psql -p 6432 -U mastodon mastodon_production @@ -266,7 +266,7 @@ PREPARED_STATEMENTS=false Since we’re using transaction-based pooling, we can’t use prepared statements. -Next up, configure Mastodon to use port 6432 (PgBouncer) instead of 5432 (Postgres) and you should be good to go: +Next up, configure Mastodon to use port 6432 (PgBouncer) instead of 5432 (PostgreSQL) and you should be good to go: ```bash DB_HOST=localhost @@ -277,7 +277,7 @@ DB_PORT=6432 ``` {{< hint style="warning" >}} -You cannot use pgBouncer to perform `db:migrate` tasks. But this is easy to work around. If your postgres and pgbouncer are on the same host, it can be as simple as defining `DB_PORT=5432` together with `RAILS_ENV=production` when calling the task, for example: `RAILS_ENV=production DB_PORT=5432 bundle exec rails db:migrate` (you can specify `DB_HOST` too if it’s different, etc) +You cannot use PgBouncer to perform `db:migrate` tasks. But this is easy to work around. If your PostgreSQL and PgBouncer are on the same host, it can be as simple as defining `DB_PORT=5432` together with `RAILS_ENV=production` when calling the task, for example: `RAILS_ENV=production DB_PORT=5432 bundle exec rails db:migrate` (you can specify `DB_HOST` too if it’s different, etc) {{< /hint >}} #### Administering PgBouncer {#pgbouncer-admin} @@ -402,7 +402,7 @@ production: url: postgresql://db_user:db_password@db_host:db_port/db_name ``` -Make sure the URLs point to wherever your PostgreSQL servers are. You can add multiple replicas. You could have a locally installed pgBouncer with a configuration to connect to two different servers based on the database name, e.g. “mastodon” going to the primary, “mastodon_replica” going to the replica, so in the file above both URLs would point to the local pgBouncer with the same user, password, host and port, but different database name. There are many possibilities how this could be set up! For more information on Makara, [see their documentation](https://github.com/taskrabbit/makara#databaseyml). +Make sure the URLs point to wherever your PostgreSQL servers are. You can add multiple replicas. You could have a locally installed PgBouncer with a configuration to connect to two different servers based on the database name, e.g. “mastodon” going to the primary, “mastodon_replica” going to the replica, so in the file above both URLs would point to the local PgBouncer with the same user, password, host and port, but different database name. There are many possibilities how this could be set up! For more information on Makara, [see their documentation](https://github.com/taskrabbit/makara#databaseyml). {{< hint style="warning" >}} Make sure the sidekiq processes run with the stock `config/database.yml` to avoid failing jobs and data loss! diff --git a/content/en/dev/setup.md b/content/en/dev/setup.md index 9477bda7..4898a863 100644 --- a/content/en/dev/setup.md +++ b/content/en/dev/setup.md @@ -53,7 +53,7 @@ bundle install yarn install ``` -In the development environment, Mastodon will use PostgreSQL as the currently signed-in Linux user using the `ident` method. Ensure that you have created a Postgres user and database for your current signed-in user: +In the development environment, Mastodon will use PostgreSQL as the currently signed-in Linux user using the `ident` method. Ensure that you have created a PostgreSQL user and database for your current signed-in user: ```sh sudo -u postgres createuser $YOUR_USERNAME_HERE --createdb diff --git a/content/en/entities/Role.md b/content/en/entities/Role.md index 679fe02a..0d8edb95 100644 --- a/content/en/entities/Role.md +++ b/content/en/entities/Role.md @@ -69,7 +69,7 @@ To determine the permissions available to a certain role, convert the `permissio : **Administrator**. Users with this permission bypass all permissions. 0x2 -: **Devops**. Allows users to access Sidekiq and pgHero dashboards. +: **Devops**. Allows users to access Sidekiq and PgHero dashboards. 0x4 : **View Audit Log**. Allows users to see history of admin actions. diff --git a/content/zh-cn/admin/migrating.md b/content/zh-cn/admin/migrating.md index 11732a3b..c1910695 100644 --- a/content/zh-cn/admin/migrating.md +++ b/content/zh-cn/admin/migrating.md @@ -17,7 +17,7 @@ menu: 1. 依照[产品指南]({{< relref "install" >}})安装新的Mastodon服务器（切记，不要运行 `mastodon:setup`）。 2. 停止旧服务器上的Mastodon（`systemctl stop 'mastodon-*.service'`）。 -3. 依照如下指示，导出并导入Postgres数据库。 +3. 依照如下指示，导出并导入PostgreSQL数据库。 4. 依照如下指示，复制 `system/` 目录下文件。（注意：如果你使用S3存储，你可以跳过此步）。 5. 复制 `.env.production` 文件。 6. 运行 `RAILS_ENV=production bundle exec rails assets:precompile` 编译 Mastodon。 @@ -34,18 +34,18 @@ menu: 你必须需要复制如下内容： * `~/live/public/system`目录，里面包含了用户上传的图片与视频（如果使用S3，可跳过此步） -* Postgres数据库（使用[pg_dump](https://www.postgresql.org/docs/9.1/static/backup-dump.html)） +* PostgreSQL数据库（使用[pg_dump](https://www.postgresql.org/docs/9.1/static/backup-dump.html)） * `~/live/.env.production`文件，里面包含了服务器配置与密钥 不太重要的部分，为了方便起见，你也可以复制如下内容： * nginx配置文件（位于`/etc/nginx/sites-available/default`） * systemd配置文件（`/etc/systemd/system/mastodon-*.service`），里面可能包括一些你服务器的调优与个性化 -* pgbouncer配置文件，位于 `/etc/pgbouncer` （如果你使用pgbouncer的话） +* PgBouncer配置文件，位于 `/etc/pgbouncer` （如果你使用PgBouncer的话） -### 导出并导入Postgres数据库 {#dump-and-load-postgres} +### 导出并导入PostgreSQL数据库 {#dump-and-load-postgresql} -不要运行`mastodon:setup`，而是创建一个名为`template0`的空白Postgres数据库（当导入Postgres导出文件时，这是很有用的，参见[pg_dump文档](https://www.postgresql.org/docs/9.1/static/backup-dump.html#BACKUP-DUMP-RESTORE)）。 +不要运行`mastodon:setup`，而是创建一个名为`template0`的空白PostgreSQL数据库（当导入PostgreSQL导出文件时，这是很有用的，参见[pg_dump文档](https://www.postgresql.org/docs/9.1/static/backup-dump.html#BACKUP-DUMP-RESTORE)）。 在你的旧系统，使用`mastodon`用户运行如下命令： diff --git a/content/zh-cn/admin/scaling.md b/content/zh-cn/admin/scaling.md index e9e43d06..fe39a8e9 100644 --- a/content/zh-cn/admin/scaling.md +++ b/content/zh-cn/admin/scaling.md @@ -101,7 +101,7 @@ sudo apt install pgbouncer #### 设置密码 {#pgbouncer-password} 首先，如果你的Postgres中`mastodon`帐户没有设置密码的话，你需要设置一个密码。 -First off, if your `mastodon` user in Postgres is set up without a password, you will need to set a password. +First off, if your `mastodon` user in PostgreSQL is set up without a password, you will need to set a password. 下面是如何重置密码： @@ -212,7 +212,7 @@ PREPARED_STATEMENTS=false 因为我们使用基于事务（transaction-based）的连接池，我们不能使用参数化查询（prepared statement）。 -接下来，配置Mastodon使用6432端口（PgBouncer）而不是5432端口（Postgres）就可以了： +接下来，配置Mastodon使用6432端口（PgBouncer）而不是5432端口（PostgreSQL）就可以了： ```bash DB_HOST=localhost