docs: add troubleshooting, maintenance, various improvements and fixes

Signed-off-by: strawberry <strawberry@puppygock.gay>
This commit is contained in:
strawberry 2024-05-08 21:26:05 -04:00 committed by June
parent 3504e6e724
commit adbe9268ce
12 changed files with 223 additions and 169 deletions

View file

@ -52,6 +52,8 @@ conduwuit's website uses [`mdbook`][mdbook] and deployed via CI using GitHub Pag
To build the documentation using Nix, run: `bin/nix-build-and-cache just .#book`
The output of the mdbook generation is in `result/`. mdbooks can be opened in your browser from the individual HTML files without any web server needed.
### Inclusivity and Diversity
All **MUST** code and write with inclusivity and diversity in mind. See the [following page by Google on writing inclusive code and documentation](https://developers.google.com/style/inclusive-documentation).

3
debian/README.md vendored
View file

@ -1,5 +1,4 @@
conduwuit for Debian
==================
# conduwuit for Debian
Installation
------------

View file

@ -1,13 +1,18 @@
[Unit]
Description=conduwuit Matrix homeserver
Documentation=https://conduwuit.puppyirl.gay/
After=network-online.target
[Service]
DynamicUser=yes
User=_conduwuit
Group=_conduwuit
User=conduwuit
Group=conduwuit
Type=notify
Environment="CONDUWUIT_CONFIG=/etc/conduwuit/conduwuit.toml"
ExecStart=/usr/sbin/conduwuit
AmbientCapabilities=
CapabilityBoundingSet=
@ -39,14 +44,11 @@ SystemCallArchitectures=native
SystemCallFilter=@system-service @resources
SystemCallFilter=~@clock @debug @module @mount @reboot @swap @cpu-emulation @obsolete @timer @chown @setuid @privileged @keyring @ipc
SystemCallErrorNumber=EPERM
StateDirectory=matrix-conduit
StateDirectory=conduwuit
RuntimeDirectory=conduit
RuntimeDirectoryMode=0750
Environment="CONDUIT_CONFIG=/etc/conduwuit/conduwuit.toml"
ExecStart=/usr/sbin/conduwuit
Restart=on-failure
RestartSec=5

8
debian/postinst vendored
View file

@ -7,21 +7,21 @@ CONDUWUIT_DATABASE_PATH=/var/lib/conduwuit/
case "$1" in
configure)
# Create the `_conduwuit` user if it does not exist yet.
if ! getent passwd _conduwuit > /dev/null ; then
# Create the `conduwuit` user if it does not exist yet.
if ! getent passwd conduwuit > /dev/null ; then
echo 'Adding system user for the conduwuit Matrix homeserver' 1>&2
adduser --system --group --quiet \
--home "$CONDUWUIT_DATABASE_PATH" \
--disabled-login \
--shell "/usr/sbin/nologin" \
--force-badname \
_conduwuit
conduwuit
fi
# Create the database path if it does not exist yet and fix up ownership
# and permissions.
mkdir -p "$CONDUWUIT_DATABASE_PATH"
chown _conduwuit:_conduwuit -R "$CONDUWUIT_DATABASE_PATH"
chown conduwuit:conduwuit -R "$CONDUWUIT_DATABASE_PATH"
chmod 700 "$CONDUWUIT_DATABASE_PATH"
;;
esac

View file

@ -2,15 +2,17 @@
- [Introduction](introduction.md)
- [Differences from upstream Conduit](differences.md)
- [Example configuration](configuration.md)
- [Deploying](deploying.md)
- [Generic](deploying/generic.md)
- [Debian](deploying/debian.md)
- [Docker](deploying/docker.md)
- [NixOS](deploying/nixos.md)
- [Docker](deploying/docker.md)
- [Arch Linux](deploying/arch-linux.md)
- [Debian](deploying/debian.md)
- [TURN](turn.md)
- [Appservices](appservices.md)
- [Maintenance](maintenance.md)
- [Troubleshooting](troubleshooting.md)
- [Development](development.md)
- [Testing](development/testing.md)
- [Contributing](contributing.md)
- [Testing](development/testing.md)

View file

@ -0,0 +1,8 @@
# conduwuit for Arch Linux
Currently conduwuit is only on the Arch User Repository (AUR).
The conduwuit AUR packages are community maintained and are not maintained by conduwuit development team, but the AUR package maintainers are in the Matrix room. Please attempt to verify your AUR package's PKGBUILD file looks fine before asking for support.
- [conduwuit](https://aur.archlinux.org/packages/conduwuit) - latest tagged conduwuit
- [conduwuit-git](https://aur.archlinux.org/packages/conduwuit-git) - latest git conduwuit from `main` branch

View file

@ -51,9 +51,9 @@ docker run -d -p 8448:6167 \
or you can use [docker compose](#docker-compose).
The `-d` flag lets the container run in detached mode. You may supply an optional `conduwuit.toml` config file, an example can be found [here](../configuration.md).
The `-d` flag lets the container run in detached mode. You may supply an optional `conduwuit.toml` config file, the example config can be found [here](../configuration.md).
You can pass in different env vars to change config values on the fly. You can even configure conduwuit completely by using env vars. For an overview of possible
values, please take a look at the `docker-compose.yml` file.
values, please take a look at the [`docker-compose.yml`](docker-compose.yml) file.
If you just want to test conduwuit for a short time, you can use the `--rm` flag, which will clean up everything related to your container after you stop it.
@ -107,92 +107,7 @@ either expose ports `443` and `8448` or serve two endpoints `.well-known/matrix/
With the service `well-known` we use a single `nginx` container that will serve those two files.
So...step by step:
1. Copy [`docker-compose.for-traefik.yml`](docker-compose.for-traefik.yml) (or
[`docker-compose.with-traefik.yml`](docker-compose.with-traefik.yml)) and [`docker-compose.override.yml`](docker-compose.override.yml) from the repository and remove `.for-traefik` (or `.with-traefik`) from the filename.
2. Open both files and modify/adjust them to your needs. Meaning, change the `CONDUIT_SERVER_NAME` and the volume host mappings according to your needs.
3. Create the `conduwuit.toml` config file, an example can be found [here](../configuration.md), or set `CONDUIT_CONFIG=""` and configure conduwuit per env vars.
4. Uncomment the `element-web` service if you want to host your own Element Web Client and create a `element_config.json`.
5. Create the files needed by the `well-known` service.
- `./nginx/matrix.conf` (relative to the compose file, you can change this, but then also need to change the volume mapping)
```nginx
server {
server_name <SUBDOMAIN>.<DOMAIN>;
listen 80 default_server;
location /.well-known/matrix/server {
return 200 '{"m.server": "<SUBDOMAIN>.<DOMAIN>:443"}';
types { } default_type "application/json; charset=utf-8";
}
location /.well-known/matrix/client {
return 200 '{"m.homeserver": {"base_url": "https://<SUBDOMAIN>.<DOMAIN>"}}';
types { } default_type "application/json; charset=utf-8";
add_header "Access-Control-Allow-Origin" *;
}
location / {
return 404;
}
}
```
6. Run `docker compose up -d`
7. Connect to your homeserver with your preferred client and create a user. You should do this immediately after starting Conduit, because the first created user is the admin.
## Voice communication
In order to make or receive calls, a TURN server is required. conduwuit suggests using [Coturn](https://github.com/coturn/coturn) for this purpose, which is also available as a Docker image. Before proceeding with the software installation, it is essential to have the necessary configurations in place.
### Configuration
Create a configuration file called `coturn.conf` containing:
```conf
use-auth-secret
static-auth-secret=<a secret key>
realm=<your server domain>
```
A common way to generate a suitable alphanumeric secret key is by using `pwgen -s 64 1`.
These same values need to be set in conduwuit. You can either modify conduwuit.toml to include these lines:
```
turn_uris = ["turn:<your server domain>?transport=udp", "turn:<your server domain>?transport=tcp"]
turn_secret = "<secret key from coturn configuration>"
```
or append the following to the docker environment variables dependig on which configuration method you used earlier:
```yml
CONDUIT_TURN_URIS: '["turn:<your server domain>?transport=udp", "turn:<your server domain>?transport=tcp"]'
CONDUIT_TURN_SECRET: "<secret key from coturn configuration>"
```
Restart Conduit to apply these changes.
### Run
Run the [Coturn](https://hub.docker.com/r/coturn/coturn) image using
```bash
docker run -d --network=host -v $(pwd)/coturn.conf:/etc/coturn/turnserver.conf coturn/coturn
```
or docker-compose. For the latter, paste the following section into a file called `docker-compose.yml`
and run `docker compose up -d` in the same directory.
```yml
version: 3
services:
turn:
container_name: coturn-server
image: docker.io/coturn/coturn
restart: unless-stopped
network_mode: "host"
volumes:
- ./coturn.conf:/etc/coturn/turnserver.conf
```
To understand why the host networking mode is used and explore alternative configuration options, please visit the following link: https://github.com/coturn/coturn/blob/master/docker/coturn/README.md.
For security recommendations see Synapse's [Coturn documentation](https://github.com/matrix-org/synapse/blob/develop/docs/setup/turn/coturn.md#configuration).
See the [TURN](../turn.md) page.

View file

@ -1,7 +1,5 @@
# Generic deployment documentation
### Please note that this documentation is not fully representative of conduwuit at the moment. Assume majority of it is outdated.
> ## Getting help
>
> If you run into any problems while setting up conduwuit, ask us
@ -13,23 +11,16 @@ You may simply download the binary that fits your machine. Run `uname -m` to see
Prebuilt binaries can be downloaded from the latest tagged release [here](https://github.com/girlbossceo/conduwuit/releases/latest).
Alternatively, you may compile the binary yourself. First, install any dependencies:
The latest tagged release also includes the Debian packages.
```bash
# Debian
$ sudo apt install libclang-dev build-essential
Alternatively, you may compile the binary yourself. We recommend using [Lix](https://lix.systems) to build conduwuit as this has the most guaranteed
reproducibiltiy and easiest to get a build environment and output going.
# RHEL
$ sudo dnf install clang
```
Then, `cd` into the source tree of conduwuit and run:
```bash
$ cargo build --release
```
Otherwise, follow standard Rust project build guides (installing git and cloning the repo, getting the Rust toolchain via rustup, installing LLVM toolchain + libclang, installing liburing for io_uring and RocksDB, etc).
## Adding a conduwuit user
While conduwuit can run as any user it is usually better to use dedicated users for different services. This also allows
While conduwuit can run as any user it is better to use dedicated users for different services. This also allows
you to make sure that the file permissions are correctly set up.
In Debian or RHEL, you can use this command to create a conduwuit user:
@ -38,6 +29,12 @@ In Debian or RHEL, you can use this command to create a conduwuit user:
sudo adduser --system conduwuit --group --disabled-login --no-create-home
```
For distros without `adduser`:
```bash
sudo useradd -r --shell /usr/bin/nologin --no-create-home conduwuit
```
## Forwarding ports in the firewall or the router
conduwuit uses the ports 443 and 8448 both of which need to be open in the firewall.
@ -46,45 +43,18 @@ If conduwuit runs behind a router or in a container and has a different public I
## Setting up a systemd service
Now we'll set up a systemd service for conduwuit, so it's easy to start/stop conduwuit and set it to autostart when your
server reboots. Simply paste the default systemd service you can find below into
`/etc/systemd/system/conduwuit.service`.
```systemd
[Unit]
Description=conduwuit Matrix Server
After=network.target
[Service]
Environment="CONDUWUIT_CONFIG=/etc/conduwuit/conduwuit.toml"
User=conduwuit
Group=conduwuit
RuntimeDirectory=conduwuit
RuntimeDirectoryMode=0750
Restart=always
ExecStart=/usr/local/bin/conduwuit
[Install]
WantedBy=multi-user.target
```
Finally, run
```bash
$ sudo systemctl daemon-reload
```
The systemd unit for conduwuit can be found [here](../../debian/conduwuit.service). You may need to change the `ExecStart=` path to where you placed the conduwuit binary.
## Creating the conduwuit configuration file
Now we need to create the conduwuit's config file in `/etc/conduwuit/conduwuit.toml`. Paste this in **and take a moment
to read it. You need to change at least the server name.**
RocksDB (`rocksdb`) is the only supported database backend. SQLite only exists for historical reasons and is not recommended. Any performance issues, storage issues, database issues, etc will not be assisted if using SQLite and you will be asked to migrate to RocksDB first.
Now we need to create the conduwuit's config file in `/etc/conduwuit/conduwuit.toml`. The example config can be found at [conduwuit-example.toml](../configuration.md).**Please take a moment to read it. You need to change at least the server name.**
See the following example config at [conduwuit-example.toml](../configuration.md)
RocksDB (`rocksdb`) is the only supported database backend. SQLite only exists for historical reasons and is not recommended. Any performance issues, storage issues, database issues, etc will not be assisted if using SQLite and you will be asked to migrate to RocksDB first.
## Setting the correct file permissions
As we are using a conduwuit specific user we need to allow it to read the config. To do that you can run this command on
If you are using a dedicated user for conduwuit, you will need to allow it to read the config. To do that you can run this command on
Debian or RHEL:
```bash
@ -102,7 +72,7 @@ sudo chmod 700 /var/lib/conduwuit/
## Setting up the Reverse Proxy
Refer to the documentation or various guides online of your chosen reverse proxy software. A Caddy example will be provided as this is the recommended reverse proxy for new users and is very trivial.
Refer to the documentation or various guides online of your chosen reverse proxy software. A [Caddy](https://caddyserver.com/) example will be provided as this is the recommended reverse proxy for new users and is very trivial to use (handles TLS, reverse proxy headers, etc transparently with proper defaults).
### Caddy
@ -118,10 +88,10 @@ your.server.name, your.server.name:8448 {
}
```
That's it! Just start or enable the service and you're set.
That's it! Just start and enable the service and you're set.
```bash
$ sudo systemctl enable caddy
$ sudo systemctl enable --now caddy
```
## You're done!

63
docs/maintenance.md Normal file
View file

@ -0,0 +1,63 @@
# Maintaining your conduwuit setup
## Moderation
conduwuit has moderation through admin room commands. "binary commands" (medium priority) and an admin API (low priority) is planned. Some moderation-related config options are available in the example config such as "global ACLs" and blocking media requests to certain servers. See the example config for the moderation config options under the "Moderation / Privacy / Security" section.
conduwuit has moderation admin commands for:
- managing room aliases (`!admin rooms alias`)
- managing room directory (`!admin rooms directory`)
- managing room banning/blocking and user removal (`!admin rooms moderation`)
- managing user accounts (`!admin users`)
- fetching `/.well-known/matrix/support` from servers (`!admin federation`)
- blocking incoming federation for certain rooms (not the same as room banning) (`!admin federation`)
- deleting media (see [the media section](#media))
Any commands with `-list` in them will require a codeblock in the message with each object being newline delimited. An example of doing this is:
````
!admin rooms moderation ban-list-of-rooms
```
!roomid1:server.name
!roomid2:server.name
!roomid3:server.name
```
````
## Database
If using RocksDB, there's very little you need to do. Compaction is ran automatically based on various defined thresholds tuned for conduwuit to be high performance with the least I/O amplifcation or overhead. Manually running compaction is not recommended, or compaction via a timer. RocksDB is built with io_uring support via liburing for async read I/O.
Some RocksDB settings can be adjusted such as the compression method chosen. See the RocksDB section in the [example config](configuration.md). btrfs users may benefit from disabling compression on RocksDB if CoW is in use.
RocksDB troubleshooting can be found [in the RocksDB section of troubleshooting](troubleshooting.md).
## Backups
Currently only RocksDB supports online backups. If you'd like to backup your database online without any downtime, see the `!admin server` command for the backup commands and the `database_backup_path` config options in the example config. Please note that the format of the database backup is not the exact same. This is unfortunately a bad design choice by Facebook as we are using the database backup engine API from RocksDB, however the data is still there and can still be joined together.
To restore a backup from an online RocksDB backup:
- shutdown conduwuit
- create a new directory for merging together the data
- in the online backup created, copy all `.sst` files in `$DATABASE_BACKUP_PATH/shared_checksum` to your new directory
- trim all the strings so instead of `######_sxxxxxxxxx.sst`, it reads `######.sst`. A way of doing this with sed and bash is `for file in *.sst; do mv "$file" "$(echo "$file" | sed 's/_s.*/.sst/')"; done`
- copy all the files in `$DATABASE_BACKUP_PATH/1` to your new directory
- set your `database_path` config option to your new directory, or replace your old one with the new one you crafted
- start up conduwuit again and it should open as normal
If you'd like to do an offline backup, shutdown conduwuit and copy your `database_path` directory elsewhere. This can be restored with no modifications needed.
Backing up media is also just copying the `media/` directory from your database directory.
## Media
Media still needs various work, however conduwuit implements media deletion via:
- MXC URI
- Delete list of MXC URIs
- Delete remote media in the past `N` seconds/minutes
See the `!admin media` command for further information. All media in conduwuit is stored at `$DATABASE_DIR/media`. This will be configurable soon.
If you are finding yourself needing extensive granular control over media, we recommend looking into [Matrix Media Repo](https://github.com/t2bot/matrix-media-repo). conduwuit intends to implement various utilities for media, but MMR is dedicated to extensive media management.
Built-in S3 support is also planned, but for now using a "S3 filesystem" on `media/` works. conduwuit also sends a `Cache-Control` header of 1 year and immutable for all media requests (download and thumbnail) to reduce unnecessary media requests from browsers.

62
docs/troubleshooting.md Normal file
View file

@ -0,0 +1,62 @@
# Troubleshooting conduwuit
> ## Docker users ⚠️
>
> Docker is extremely UX unfriendly. Because of this, a ton of issues or support is actually Docker support, not conduwuit support. We also cannot document the ever-growing list of Docker issues here.
>
> If you intend on asking for support and you are using Docker, **PLEASE** triple validate your issues are **NOT** because you have a misconfiguration in your Docker setup.
>
> If there are things like Compose file issues or Dockerhub image issues, those can still be mentioned as long as they're something we can fix.
## Rocksdb / database issues
#### Direct IO
Some filesystems may not like RocksDB using [Direct IO](https://github.com/facebook/rocksdb/wiki/Direct-IO). Direct IO is for non-buffered I/O which improves conduwuit performance, but at least FUSE is a filesystem potentially known to not like this. See the [example config](configuration.md) for disabling it if needed. Issues from Direct IO on unsupported filesystems are usually shown as startup errors.
#### Database corruption
If your database is corrupted and is failing to start (e.g. checksum mismatch), it may be recoverable but careful steps must be taken, and there is no guarantee it may be recoverable.
RocksDB has the following recovery modes:
- `TolerateCorruptedTailRecords`
- `AbsoluteConsistency`
- `PointInTime`
- `SkipAnyCorruptedRecord`
By default, conduwuit uses `TolerateCorruptedTailRecords` as generally these may be due to bad federation and we can re-fetch the correct data over federation. The RocksDB default is `PointInTime` which will attempt to restore a "snapshot" of the data when it was last known to be good. This data can be either a few seconds old, or multiple minutes prior. `PointInTime` may not be suitable for default usage due to clients and servers possibly not being able to handle sudden "backwards time travels", and `AbsoluteConsistency` may be too strict.
`AbsoluteConsistency` will fail to start the database if any sign of corruption is detected. `SkipAnyCorruptedRecord` will skip all forms of corruption unless it forbids the database from opening (e.g. too severe). Usage of `SkipAnyCorruptedRecord` voids any support as this may cause more damage and/or leave your database in a permanently inconsistent state, but it may do something if `PointInTime` does not work as a last ditch effort.
With this in mind:
- First start conduwuit with the `PointInTime` recovery method. See the [example config](configuration.md) for how to do this using `rocksdb_recovery_mode`
- If your database successfully opens, clients are recommended to clear their client cache to account for the rollback
- Leave your conduwuit running in `PointInTime` for at least 30-60 minutes so as much possible corruption is restored
- If all goes will, you should be able to restore back to using `TolerateCorruptedTailRecords` and you have successfully recovered your database
## Media
#### "File name too long"
If you are running into the "file name is too long" OS error for media requests, your filesystem cannot handle file name lengths >=255 characters. This is unfortuntely due to Conduit (upstream) using base64 for file name keys which is very problematic for some filesystems as the base64 input is untrusted and long file names or specific inputs can cause this. If you would like to avoid this, you may build conduwuit yourself with the `sha256_media` feature. **This will lose database compatibility with upstream**.
## Debugging
Note that users should not really be debugging things. If you find yourself debugging and find the issue, please let us know and/or how we can fix it. Various debug commands can be found in `!admin debug`.
#### Debug/Trace log level
conduwuit builds without debug or trace log levels by default for at least performance reasons. This may change in the future and/or binaries providing such configurations may be provided. If you need to access debug/trace log levels, you will need to build without the `release_max_log_level` feature.
#### Changing log level dynamically
conduwuit supports changing the tracing log environment filter on-the-fly using the admin command `!admin debug change-log-level`. This accepts a string **without quotes** the same format as the `log` config option.
#### Pinging servers
conduwuit can ping other servers using `!admin debug ping`. This takes a server name and goes through the server discovery process and queries `/_matrix/federation/v1/version`. Errors are outputted.
#### Allocator memory stats
If using jemalloc (for now) and built with jemallocator's `stats` feature, you can see conduwuit's jemalloc memory stats by using `!admin debug memory-stats`

View file

@ -1,25 +1,55 @@
# Setting up TURN/STURN
## General instructions
In order to make or receive calls, a TURN server is required. conduwuit suggests using [Coturn](https://github.com/coturn/coturn) for this purpose, which is also available as a Docker image.
* It is assumed you have a [Coturn server](https://github.com/coturn/coturn) up and running. See [Synapse reference implementation](https://github.com/matrix-org/synapse/blob/develop/docs/turn-howto.md).
### Configuration
## Edit/Add a few settings to your existing conduit.toml
Create a configuration file called `coturn.conf` containing:
```conf
use-auth-secret
static-auth-secret=<a secret key>
realm=<your server domain>
```
A common way to generate a suitable alphanumeric secret key is by using `pwgen -s 64 1`.
These same values need to be set in conduwuit. You can either modify conduwuit.toml to include these lines:
```
# Refer to your Coturn settings.
# `your.turn.url` has to match the REALM setting of your Coturn as well as `transport`.
turn_uris = ["turn:your.turn.url?transport=udp", "turn:your.turn.url?transport=tcp"]
# static-auth-secret of your turnserver
turn_secret = "ADD SECRET HERE"
# If you have your TURN server configured to use a username and password
# you can provide these information too. In this case comment out `turn_secret above`!
#turn_username = ""
#turn_password = ""
turn_uris = ["turn:<your server domain>?transport=udp", "turn:<your server domain>?transport=tcp"]
turn_secret = "<secret key from coturn configuration>"
```
## Apply settings
or append the following to the docker environment variables dependig on which configuration method you used earlier:
Restart Conduit.
```yml
CONDUIT_TURN_URIS: '["turn:<your server domain>?transport=udp", "turn:<your server domain>?transport=tcp"]'
CONDUIT_TURN_SECRET: "<secret key from coturn configuration>"
```
Restart conduwuit to apply these changes.
### Run
Run the [Coturn](https://hub.docker.com/r/coturn/coturn) image using
```bash
docker run -d --network=host -v $(pwd)/coturn.conf:/etc/coturn/turnserver.conf coturn/coturn
```
or docker-compose. For the latter, paste the following section into a file called `docker-compose.yml`
and run `docker compose up -d` in the same directory.
```yml
version: 3
services:
turn:
container_name: coturn-server
image: docker.io/coturn/coturn
restart: unless-stopped
network_mode: "host"
volumes:
- ./coturn.conf:/etc/coturn/turnserver.conf
```
To understand why the host networking mode is used and explore alternative configuration options, please visit [Coturn's Docker documentation](https://github.com/coturn/coturn/blob/master/docker/coturn/README.md).
For security recommendations see Synapse's [Coturn documentation](https://element-hq.github.io/synapse/latest/turn-howto.html).

View file

@ -14,6 +14,7 @@ stdenv.mkDerivation {
include = [
"book.toml"
"conduwuit-example.toml"
"CONTRIBUTING.md"
"README.md"
"debian/README.md"
"docs"