Consistent use of Postgres and PgBouncer (#1353)

* fix pgbouncer
* pghero
This commit is contained in:
Michael Stanclift 2023-12-10 12:40:20 -06:00 committed by GitHub
parent 921fb57ff4
commit 94aa9f71a7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 28 additions and 28 deletions

View File

@ -315,7 +315,7 @@ Defines how many database connections to pool in the process. This value should
#### `DB_SSLMODE`
Postgres's [SSL mode](https://www.postgresql.org/docs/10/libpq-ssl.html). Defaults to `prefer`.
PostgreSQL [SSL mode](https://www.postgresql.org/docs/10/libpq-ssl.html). Defaults to `prefer`.
#### `DATABASE_URL`

View File

@ -17,7 +17,7 @@ This guide was written with Ubuntu Server in mind; your mileage may vary for oth
1. Set up a new Mastodon server using the [Production Guide]({{< relref "install" >}}) (however, dont run `mastodon:setup`).
2. Stop Mastodon on the old server (e.g. `systemctl stop 'mastodon-*.service'`).
3. Dump and load the Postgres database using the instructions below.
3. Dump and load the PostgreSQL database using the instructions below.
4. Copy the `system/` files using the instructions below. (Note: if youre using S3, you can skip this step.)
5. Copy the `.env.production` file.
6. Run `RAILS_ENV=production bundle exec rails assets:precompile` to compile Mastodon
@ -34,18 +34,18 @@ This guide was written with Ubuntu Server in mind; your mileage may vary for oth
At a high level, youll need to copy over the following:
* The `~/live/public/system` directory, which contains user-uploaded images and videos (if using S3, you dont need this)
* The Postgres database (using [pg_dump](https://www.postgresql.org/docs/9.1/static/backup-dump.html))
* The PostgreSQL database (using [pg_dump](https://www.postgresql.org/docs/9.1/static/backup-dump.html))
* The `~/live/.env.production` file, which contains server config and secrets
Less crucially, youll probably also want to copy the following for convenience:
* The nginx config (under `/etc/nginx/sites-available/default`)
* The systemd config files (`/etc/systemd/system/mastodon-*.service`), which may contain your server tweaks and customizations
* The pgbouncer configuration under `/etc/pgbouncer` (if youre using it)
* The PgBouncer configuration under `/etc/pgbouncer` (if youre using it)
### Dump and load Postgres {#dump-and-load-postgres}
### Dump and load PostgreSQL {#dump-and-load-postgresql}
Instead of running `mastodon:setup`, were going to create an empty Postgres database using the `template0` database (which is useful when restoring a Postgres dump, [as described in the pg_dump documentation](https://www.postgresql.org/docs/9.1/static/backup-dump.html#BACKUP-DUMP-RESTORE)).
Instead of running `mastodon:setup`, were going to create an empty PostgreSQL database using the `template0` database (which is useful when restoring a PostgreSQL dump, [as described in the pg_dump documentation](https://www.postgresql.org/docs/9.1/static/backup-dump.html#BACKUP-DUMP-RESTORE)).
Run this as the `mastodon` user on your old system:
@ -80,7 +80,7 @@ Youll want to re-run this if any of the files on the old server change.
You should also copy over the `.env.production` file, which contains secrets.
Optionally, you may copy over the nginx, systemd, and pgbouncer config files, or rewrite them from scratch.
Optionally, you may copy over the nginx, systemd, and PgBouncer config files, or rewrite them from scratch.
### During migration {#during-migration}

View File

@ -123,13 +123,13 @@ As a solution, it is possible to start different Sidekiq processes for the queue
**Make sure you only have one `scheduler` queue running!!**
## Transaction pooling with pgBouncer {#pgbouncer}
## Transaction pooling with PgBouncer {#pgbouncer}
### Why you might need PgBouncer {#pgbouncer-why}
If you start running out of available Postgres connections (the default is 100) then you may find PgBouncer to be a good solution. This document describes some common gotchas as well as good configuration defaults for Mastodon.
If you start running out of available PostgreSQL connections (the default is 100) then you may find PgBouncer to be a good solution. This document describes some common gotchas as well as good configuration defaults for Mastodon.
User roles with `DevOps` permissions in Mastodon can monitor the current usage of Postgres connections through the PgHero link in the Administration view. Generally, the number of connections open is equal to the total threads in Puma, Sidekiq, and the streaming API combined.
User roles with `DevOps` permissions in Mastodon can monitor the current usage of PostgreSQL connections through the PgHero link in the Administration view. Generally, the number of connections open is equal to the total threads in Puma, Sidekiq, and the streaming API combined.
### Installing PgBouncer {#pgbouncer-install}
@ -143,7 +143,7 @@ sudo apt install pgbouncer
#### Setting a password {#pgbouncer-password}
First off, if your `mastodon` user in Postgres is set up without a password, you will need to set a password.
First off, if your `mastodon` user in PostgreSQL is set up without a password, you will need to set a password.
Heres how you might reset the password:
@ -193,7 +193,7 @@ In both cases, the password is just `password`.
Edit `/etc/pgbouncer/pgbouncer.ini`
Add a line under `[databases]` listing the Postgres databases you want to connect to. Here well just have PgBouncer use the same username/password and database name to connect to the underlying Postgres database:
Add a line under `[databases]` listing the PostgreSQL databases you want to connect to. Here well just have PgBouncer use the same username/password and database name to connect to the underlying PostgreSQL database:
```text
[databases]
@ -219,13 +219,13 @@ Make sure the `pgbouncer` user is an admin:
admin_users = pgbouncer
```
Mastodon requires a different pooling mode than the default session-based one. Specifically, it needs a transaction-based pooling mode. This means that a Postgres connection is established at the start of a transaction and terminated upon its completion. Therefore, it's essential to change the `pool_mode` setting from `session` to `transaction`:
Mastodon requires a different pooling mode than the default session-based one. Specifically, it needs a transaction-based pooling mode. This means that a PostgreSQL connection is established at the start of a transaction and terminated upon its completion. Therefore, it's essential to change the `pool_mode` setting from `session` to `transaction`:
```ini
pool_mode = transaction
```
Next up, `max_client_conn` defines how many connections PgBouncer itself will accept, and `default_pool_size` puts a limit on how many Postgres connections will be opened under the hood. (In PgHero the number of connections reported will correspond to `default_pool_size` because it has no knowledge of PgBouncer.)
Next up, `max_client_conn` defines how many connections PgBouncer itself will accept, and `default_pool_size` puts a limit on how many PostgreSQL connections will be opened under the hood. (In PgHero the number of connections reported will correspond to `default_pool_size` because it has no knowledge of PgBouncer.)
The defaults are fine to start, and you can always increase them later:
@ -234,7 +234,7 @@ max_client_conn = 100
default_pool_size = 20
```
Dont forget to reload or restart pgbouncer after making your changes:
Dont forget to reload or restart PgBouncer after making your changes:
```bash
sudo systemctl reload pgbouncer
@ -242,7 +242,7 @@ sudo systemctl reload pgbouncer
#### Debugging that it all works {#pgbouncer-debug}
You should be able to connect to PgBouncer just like you would with Postgres:
You should be able to connect to PgBouncer just like you would with PostgreSQL:
```bash
psql -p 6432 -U mastodon mastodon_production
@ -266,7 +266,7 @@ PREPARED_STATEMENTS=false
Since were using transaction-based pooling, we cant use prepared statements.
Next up, configure Mastodon to use port 6432 (PgBouncer) instead of 5432 (Postgres) and you should be good to go:
Next up, configure Mastodon to use port 6432 (PgBouncer) instead of 5432 (PostgreSQL) and you should be good to go:
```bash
DB_HOST=localhost
@ -277,7 +277,7 @@ DB_PORT=6432
```
{{< hint style="warning" >}}
You cannot use pgBouncer to perform `db:migrate` tasks. But this is easy to work around. If your postgres and pgbouncer are on the same host, it can be as simple as defining `DB_PORT=5432` together with `RAILS_ENV=production` when calling the task, for example: `RAILS_ENV=production DB_PORT=5432 bundle exec rails db:migrate` (you can specify `DB_HOST` too if its different, etc)
You cannot use PgBouncer to perform `db:migrate` tasks. But this is easy to work around. If your PostgreSQL and PgBouncer are on the same host, it can be as simple as defining `DB_PORT=5432` together with `RAILS_ENV=production` when calling the task, for example: `RAILS_ENV=production DB_PORT=5432 bundle exec rails db:migrate` (you can specify `DB_HOST` too if its different, etc)
{{< /hint >}}
#### Administering PgBouncer {#pgbouncer-admin}
@ -402,7 +402,7 @@ production:
url: postgresql://db_user:db_password@db_host:db_port/db_name
```
Make sure the URLs point to wherever your PostgreSQL servers are. You can add multiple replicas. You could have a locally installed pgBouncer with a configuration to connect to two different servers based on the database name, e.g. “mastodon” going to the primary, “mastodon_replica” going to the replica, so in the file above both URLs would point to the local pgBouncer with the same user, password, host and port, but different database name. There are many possibilities how this could be set up! For more information on Makara, [see their documentation](https://github.com/taskrabbit/makara#databaseyml).
Make sure the URLs point to wherever your PostgreSQL servers are. You can add multiple replicas. You could have a locally installed PgBouncer with a configuration to connect to two different servers based on the database name, e.g. “mastodon” going to the primary, “mastodon_replica” going to the replica, so in the file above both URLs would point to the local PgBouncer with the same user, password, host and port, but different database name. There are many possibilities how this could be set up! For more information on Makara, [see their documentation](https://github.com/taskrabbit/makara#databaseyml).
{{< hint style="warning" >}}
Make sure the sidekiq processes run with the stock `config/database.yml` to avoid failing jobs and data loss!

View File

@ -53,7 +53,7 @@ bundle install
yarn install
```
In the development environment, Mastodon will use PostgreSQL as the currently signed-in Linux user using the `ident` method. Ensure that you have created a Postgres user and database for your current signed-in user:
In the development environment, Mastodon will use PostgreSQL as the currently signed-in Linux user using the `ident` method. Ensure that you have created a PostgreSQL user and database for your current signed-in user:
```sh
sudo -u postgres createuser $YOUR_USERNAME_HERE --createdb

View File

@ -69,7 +69,7 @@ To determine the permissions available to a certain role, convert the `permissio
: **Administrator**. Users with this permission bypass all permissions.
0x2
: **Devops**. Allows users to access Sidekiq and pgHero dashboards.
: **Devops**. Allows users to access Sidekiq and PgHero dashboards.
0x4
: **View Audit Log**. Allows users to see history of admin actions.

View File

@ -17,7 +17,7 @@ menu:
1. 依照[产品指南]({{< relref "install" >}})安装新的Mastodon服务器切记不要运行 `mastodon:setup`)。
2. 停止旧服务器上的Mastodon`systemctl stop 'mastodon-*.service'`)。
3. 依照如下指示导出并导入Postgres数据库。
3. 依照如下指示导出并导入PostgreSQL数据库。
4. 依照如下指示,复制 `system/` 目录下文件。注意如果你使用S3存储你可以跳过此步
5. 复制 `.env.production` 文件。
6. 运行 `RAILS_ENV=production bundle exec rails assets:precompile` 编译 Mastodon。
@ -34,18 +34,18 @@ menu:
你必须需要复制如下内容:
* `~/live/public/system`目录里面包含了用户上传的图片与视频如果使用S3可跳过此步
* Postgres数据库(使用[pg_dump](https://www.postgresql.org/docs/9.1/static/backup-dump.html)
* PostgreSQL数据库(使用[pg_dump](https://www.postgresql.org/docs/9.1/static/backup-dump.html)
* `~/live/.env.production`文件,里面包含了服务器配置与密钥
不太重要的部分,为了方便起见,你也可以复制如下内容:
* nginx配置文件位于`/etc/nginx/sites-available/default`
* systemd配置文件`/etc/systemd/system/mastodon-*.service`),里面可能包括一些你服务器的调优与个性化
* pgbouncer配置文件位于 `/etc/pgbouncer` 如果你使用pgbouncer的话
* PgBouncer配置文件位于 `/etc/pgbouncer` 如果你使用PgBouncer的话
### 导出并导入Postgres数据库 {#dump-and-load-postgres}
### 导出并导入PostgreSQL数据库 {#dump-and-load-postgresql}
不要运行`mastodon:setup`,而是创建一个名为`template0`的空白Postgres数据库当导入Postgres导出文件时,这是很有用的,参见[pg_dump文档](https://www.postgresql.org/docs/9.1/static/backup-dump.html#BACKUP-DUMP-RESTORE))。
不要运行`mastodon:setup`,而是创建一个名为`template0`的空白PostgreSQL数据库当导入PostgreSQL导出文件时,这是很有用的,参见[pg_dump文档](https://www.postgresql.org/docs/9.1/static/backup-dump.html#BACKUP-DUMP-RESTORE))。
在你的旧系统,使用`mastodon`用户运行如下命令:

View File

@ -101,7 +101,7 @@ sudo apt install pgbouncer
#### 设置密码 {#pgbouncer-password}
首先如果你的Postgres中`mastodon`帐户没有设置密码的话,你需要设置一个密码。
First off, if your `mastodon` user in Postgres is set up without a password, you will need to set a password.
First off, if your `mastodon` user in PostgreSQL is set up without a password, you will need to set a password.
下面是如何重置密码:
@ -212,7 +212,7 @@ PREPARED_STATEMENTS=false
因为我们使用基于事务transaction-based的连接池我们不能使用参数化查询prepared statement
接下来配置Mastodon使用6432端口PgBouncer而不是5432端口Postgres)就可以了:
接下来配置Mastodon使用6432端口PgBouncer而不是5432端口PostgreSQL)就可以了:
```bash
DB_HOST=localhost