PgBouncer Guide ==== The following guide explains how to use [PgBouncer](https://pgbouncer.github.io/) as an efficient connection pooler on top of Postgres. For a bit of background, you might read ["Scaling Mastodon"](https://medium.com/@Gargron/scaling-mastodon-1becde463090) which briefly describes this approach. Why you might need PgBouncer ---- If you start running out of available Postgres connections (the default is 100) then you may find PgBouncer to be a good solution. This document describes some common gotchas as well as good configuration defaults for Mastodon. Note that you can check "PgHero" in the administration view to see how many Postgres connections are currently being used. Installing PgBouncer ----- On Ubuntu: sudo apt install pgbouncer Restarting: sudo service pgbouncer restart (Note that this guide assumes you aren't using Docker.) Configuring PgBouncer ----- ### Setting a password First off, if your `mastodon` user in Postgres is set up wthout a password, you will need to set a password. There seems to be no way to use PgBouncer with an empty password. Here's how you might reset the password: psql -p 5432 -U mastodon mastodon_production -w Then: ALTER USER "mastodon" WITH PASSWORD "password"; Then `\q` to quit. ### Configuring PgBouncer PgBouncer has two config files: `pgbouncer.ini` and `userlist.txt` both in `/etc/pgbouncer/`. The first contains the configuration, whereas the second just contains a list of usernames and passwords. #### Configuring userlist.txt Add the `mastodon` user to the `userlist.txt`: "mastodon" "md599dff7ae01ae2dc33ae052264bf22bf4" Here we're using the md5 scheme, where the md5 password is just the md5sum of `username + password` with the string `md5` prepended. For instance, to derive the hash for user `mastodon` with password `password`, you can do: ```bash # ubuntu, debian, etc. echo -n "mastodonpassword" | md5sum # macOS, openBSD, etc. md5 -s "mastodonpassword" ``` Then just add `md5` to the beginning of that. You'll also want to create a `pgbouncer` admin user to log in to the PgBouncer admin database. So here's a sample `userlist.txt`: ``` "mastodon" "md599dff7ae01ae2dc33ae052264bf22bf4" "pgbouncer" "md509c950063a6cf1b217ee831d0f4c6771" ``` In both cases the password is just `password`. #### Configuring pgbouncer.ini Add a line under `[databases]` listing the Postgres databases you want to connect to. Here we'll just have PgBouncer use the same username/password and database name to connect to the underlying Postgres database: ```ini [databases] mastodon_production = host=127.0.0.1 port=5432 dbname=mastodon_production user=mastodon password=password ``` The `listen_addr` and `listen_port` tells PgBouncer which address/port to accept connections. The defaults are fine: ```ini listen_addr = 127.0.0.1 listen_port = 6432 ``` Put `md5` as the `auth_type` (assuming you're using the md5 format in `userlist.txt`): ```ini auth_type = md5 ``` Make sure the `pgbouncer` user is an admin: ```ini admin_users = pgbouncer ``` **This next part is very important!** The default pooling mode is session-based, but for Mastodon we want transaction-based. In other words, a Postgres connection is created when a transaction is created and dropped when the transaction is done. So you'll want to change the `pool_mode` from `session` to `transaction`: ```ini pool_mode = transaction ``` Next up, `max_client_conn` defines how many connections PgBouncer itself will accept, and `default_pool_size` puts a limit on how many Postgres connections will be opened under the hood. (In PgHero the number of connections reported will correspond to `default_pool_size` because it has no knowledge of PgBouncer.) The defaults are fine to start, and you can always increase them later: ```ini max_client_conn = 100 default_pool_size = 20 ``` ### Debugging that it all works You should be able to connect to PgBouncer just like you would with Postgres: psql -p 6432 -U mastodon mastodon_production And then use your password to log in. You can also check the PgBouncer logs like so: tail -f /var/log/postgresql/pgbouncer.log ### Configuring Mastodon to talk to PgBouncer In your `.env.production` file, first off make sure that this is set: ```bash PREPARED_STATEMENTS=false ``` Since we're using transaction-based pooling, we can't use prepared statements. Next up, configure Mastodon to use port 6432 (PgBouncer) instead of 5432 (Postgres) and you should be good to go: ```bash DB_HOST=localhost DB_USER=mastodon DB_NAME=mastodon_production DB_PASS=password DB_PORT=6432 ``` ### Administering PgBouncer The easiest way to reboot is: sudo service pgbouncer restart But if you've set up a PgBouncer admin user, you can also connect as the admin: psql -p 6432 -U pgbouncer pgbouncer And then do: RELOAD; Then use `\q` to quit. ### Resources - ["Scaling Mastodon"](https://medium.com/@Gargron/scaling-mastodon-1becde463090) - [PgBouncer documentation](https://pgbouncer.github.io/) - [Connection Pooling in PostgreSQL using pgbouncer](https://www.slideshare.net/sameerkasi200x/5th-pugs-meetupjuly2014pgbouncer) - [PgBouncer on the Postgres wiki](https://wiki.postgresql.org/wiki/PgBouncer)