# Productier Disaster Recovery Runbook

## Scope

This runbook covers backup and restore of:

- PostgreSQL data (`postgres` service)
- Object storage data (`rustfs` bucket configured by `S3_BUCKET`)

It assumes the production compose stack and env setup in:

- `infra/docker-compose.prod.yml`
- `.env.production`

## Preconditions

1. Validate production env:

```bash
npm run check:prod-env
```

2. Ensure production services are running:

```bash
npm run ops:deploy
```

If deployment is already running and you only want readiness validation:

```bash
npm run ops:preflight
```

## Backup

Create a timestamped backup:

```bash
npm run ops:backup
```

Output directory format:

`backups/<UTC timestamp>/`

Expected files:

- `postgres.sql.gz`
- `s3/` (synced object data)
- `checksums.sha256`
- `metadata.json`

Verify backup:

```bash
bash scripts/ops/verify-backup.sh backups/<timestamp>
```

Run full scheduled-style backup flow (backup + verify + prune):

```bash
npm run ops:backup:job
```

## Restore

Restore is destructive and requires explicit safety flags.

```bash
FORCE=1 RESET_DB=1 RESTORE_S3=1 \
bash scripts/ops/restore-prod.sh .env.production backups/<timestamp>
```

Flags:

- `FORCE=1`: required; otherwise restore exits
- `RESET_DB=1`: recommended; drops and recreates schema before import
- `RESTORE_S3=1`: restore object storage (default on)

## Restore drill (non-destructive)

Run drill against latest backup:

```bash
npm run ops:restore:drill
```

Run full isolated staging drill (temporary compose project + teardown):

```bash
npm run ops:drill:staging
```

Run drill against specific backup:

```bash
bash scripts/ops/restore-drill.sh .env.production backups/<timestamp>
```

Drill behavior:

- Imports DB dump into temporary drill database
- Runs sanity check (`public` table count)
- Optionally syncs backup objects into a temporary drill bucket
- Drops temporary drill DB and bucket by default

## Post-restore checks

1. API health:

```bash
curl -sS https://<PUBLIC_DOMAIN>/v1/health
```

2. Auth health:

```bash
curl -sS https://<PUBLIC_DOMAIN>/api/auth/get-session
```

3. Manual smoke:

- sign in
- open board/calendar/notes/mail
- download at least one attachment

Automated smoke script:

```bash
npm run ops:smoke
```

The smoke script checks:

- public homepage response
- `/v1/health` payload
- security response headers
- HTTP->HTTPS redirect behavior

## Cadence recommendations

- Daily backup (off-peak)
- Weekly restore drill in staging
- Keep at least 14 daily restore points and 8 weekly restore points

## Automation (systemd)

Template files:

- `infra/systemd/productier-backup.service`
- `infra/systemd/productier-backup.timer`

Example install on Linux host:

```bash
sudo cp infra/systemd/productier-backup.service /etc/systemd/system/
sudo cp infra/systemd/productier-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now productier-backup.timer
sudo systemctl status productier-backup.timer
```

Run backup manually through systemd:

```bash
sudo systemctl start productier-backup.service
sudo journalctl -u productier-backup.service -n 200 --no-pager
```

Retention is controlled by `BACKUP_KEEP_COUNT` in `productier-backup.service`.

Alerting:

- `OPS_ALERT_WEBHOOK_URL`
- `OPS_ALERT_WEBHOOK_BEARER_TOKEN`
- `OPS_NOTIFY_ON_SUCCESS`
- `OPS_ALERT_TIMEOUT_SECONDS`

These variables can be set in `/opt/productier/.env.production` and are loaded by `productier-backup.service`.

Restore drill automation:

- `infra/systemd/productier-restore-drill.service`
- `infra/systemd/productier-restore-drill.timer`

Example install:

```bash
sudo cp infra/systemd/productier-restore-drill.service /etc/systemd/system/
sudo cp infra/systemd/productier-restore-drill.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now productier-restore-drill.timer
sudo systemctl status productier-restore-drill.timer
```