Primora/ROADMAP.md

# Primora Roadmap

Primora is a self-hosted backend platform — the open, ownable alternative to
Supabase and Appwrite. One binary deployment gives you auth, Postgres-backed
APIs, object storage, document collections, API keys, an audit log, database
tooling, observability, and integrations — all behind one dashboard.

Design docs: [project_backend.md](project_backend.md) · [project_frontend.md](project_frontend.md)

---

## Phase 0 — Foundation (done)

- SolidJS dashboard, dark-first design system, demo mode
- Go + Gin API, sqlc, migrations, local object storage
- Better Auth service (email/password + OAuth), JWT via JWKS
- Docker Compose stack, nginx proxy, OpenAPI-generated client
- MIT license, CONTRIBUTING, SECURITY, rewritten README/QUICK_START

## Phase 1 — Surface area & release plumbing ✅

Everything an operator touches that isn't in the box yet.

- **`primora` CLI** (TypeScript, `apps/cli`) — login, org/project context,
  projects, buckets, objects up/download, keys, audit tail. `cac` for parsing,
  `@clack/prompts` for interactive UX, `picocolors` for colour. Wraps the
  generated `@primora/api-client` directly — one contract, zero duplication.
  Ships as `@primora/cli` on npm (`npx primora …`) and as standalone binaries
  via `bun build --compile` in the release pipeline (the opencode pattern).
- **`@primora/mcp` server** (`apps/mcp`) — MCP stdio server exposing
  `primora_*` tools (list/create projects, keys, audit query, storage ops) so
  agents can operate the platform directly. Ported from IMS's `apps/mcp`.
- **Auth users panel** — Supabase-style `Authentication → Users` view backed by
  the auth service (list users, disable, reset). Requires a small admin
  read endpoint on the auth service.
- **Release pipeline** — semver tags, CHANGELOG.md, GitHub Actions: build
  matrix → GitHub Release + ghcr.io images (`primora/backend`,
  `primora/auth`, `primora/frontend`, `primora/cli`).
- **Responsive pass** — sidebar already collapses; audit every page at
  390px for table/card overflow.

## Phase 2 — Databases (DBX)

The "any database" story, via [DBX](https://github.com/t8y2/dbx) — 90+ drivers
in a 25 MB MCP server. Primora does not embed DBX; the Go backend keeps
`@dbx-app/mcp-server` as a persistent stdio subprocess and exposes its tools
as REST (pattern already proven in IMS's `internal/app/dbx.go`).

- **Connection registry** ✅ — per-project saved connections (Postgres, MySQL,
  SQLite, Redis, MongoDB, …). Credentials stored server-side, never sent to
  the browser. `core.db_connections` table + endpoints.
- **Databases page** ✅ — connection list → database/schema/table browser →
  column/type/key inspector.
- **SQL editor** ✅ — query editor with ⌘/Ctrl+Enter, result grid, row detail.
- **Redis console** ✅ — command input + output history for redis-type
  connections.
- **Visual overview** ✅ — schema graph: tables as nodes, foreign keys as edges.
- **First-party connections** ✅ — the platform's own Postgres and Dragonfly
  auto-register as connections on every project, so "put Postgres in it and
  it runs Postgres" works out of the box.
- **DB-to-DB links** — DBX data transfer between two saved connections
  (e.g. Postgres → Dragonfly cache warm), plus documented patterns for
  pairing them.

## Phase 3 — Observability (IMS merge) ✅

IMS (`~/Desktop/PROG+HTML/IMS`) is discontinued and folded in — same stack
(Gin + pgx + Postgres), ~2k lines to port.

- **Ingest API** — `POST /api/v1/ingest` authenticated by project `prm_` API keys
  (`X-Primora-Key` or `X-API-Key`); event types: error, metric, log, heartbeat,
  event; component auto-registration; single event or ≤500-event batch;
  permissive CORS so browser SDKs can post directly.
- **Issues** — error groups by fingerprint (count, first/last seen, severity,
  stack payload) — the Firebase Crashlytics-shaped view.
- **Telemetry views** — health overview (totals, activity histogram, component
  health), issues, events, metric series (avg/p50/p95/max), Integrate tab;
  live updates over SSE (`GET …/telemetry/stream`, `?api_key=`/`?token=` for
  EventSource auth).
- **`@primora/client` SDK** — port of `@ims/client`: queued batching,
  `keepalive` flush, `installAuto()` for browser error capture.
- **Agent tooling** — `primora_list_issues`, `primora_list_events`,
  `primora_telemetry_stats`, `primora_metric_series` in `apps/mcp`.
- Migration note in IMS repo README → archive, point at Primora.

## Phase 4 — Integrations ✅

- ~~**Connector framework**~~ — `core.integrations` (migration 00005): type,
  base URL, AES-256-GCM-encrypted credentials (`PRIMORA_ENCRYPTION_KEY`),
  health check (`POST …/test`). `has_credentials` only — secrets never
  serialized.
- ~~**Rybbit connector**~~ — self-hosted URL + API key;
  `GET …/integrations/:id/analytics` normalizes overview/series/top
  pages/referrers server-side. Integrations page carries the per-connector
  analytics panel. Read-only, no tracking proxy.
- ~~**Webhooks**~~ — `core.webhooks` + `core.webhook_deliveries`; events
  `issue.created` (new fingerprint on ingest), `deploy.marker`
  (`POST …/deploy-markers`), `webhook.test`. HMAC-SHA256 signature header
  (`X-Primora-Signature`), in-process dispatcher with backoff retries, HTTPS
  for public targets (HTTP allowed for private/self-hosted sinks).

## Phase 5 — Distribution ✅

- ~~**Desktop app**~~ — `apps/desktop`, a Tauri 2 shell (Linux/Windows/macOS).
  Thin client: asks for the deployment URL on first run, renders the same
  SolidJS bundle, keeps a system-tray presence (Show / Change Server / Quit,
  close-to-tray, single-instance). No embedded stack — self-hosted model
  preserved.
- ~~**Phone**~~ — PWA shipped: `manifest.webmanifest` + Workbox service worker
  precache the shell (`/api`, `/auth`, `/mailpit` stay `NetworkOnly`), plus an
  in-app install banner on `beforeinstallprompt`. Native shells via Capacitor
  remain deferred until push notifications are a real requirement.
- ~~**Versioning**~~ — `v0.x` minor-per-phase until the API surface stabilizes.
  `v0.5.0` tagged and pushed; the release pipeline now also ships desktop
  bundles: tag → GitHub Release notes + ghcr.io images + CLI/MCP binaries +
  desktop installers.
- **Docs site** — deferred on purpose. README stays canonical; the OpenAPI
  contract is the machine-readable source. The API still changes every phase —
  a docs site now would document a moving target. Revisit when `v1.0.0` lands.

Follow-ups:

- Verify the macOS/Windows desktop bundles produced by the `v0.5.0` CI run
  (host-verified on Linux only).
- Desktop bundle signing + auto-updater — unsigned artifacts today; add when
  distribution matures.

## Phase 6 — Production hardening (v0.6.0)

The v1 feature surface is essentially complete; what remains is the
difference between "works" and "safe to run other people's data". Roughly
three-quarters of the distance is travelled — this phase is the rest.

Foundations already in place: liveness/readiness probes with dependency
checks, per-key and per-user rate limiting, CORS locked to `PUBLIC_URL`,
AES-256-GCM credential encryption (`PRIMORA_ENCRYPTION_KEY` enforced in
production), audit log on mutations, org/project roles, SMTP + Resend in
the auth service, migrations on boot, single-command compose deploy,
release pipeline with images + binaries + desktop bundles.

### Security

- **Graceful shutdown** — `router.Run()` drops in-flight requests on every
  deploy. Switch to `http.Server` + `signal.NotifyContext` with a ~10s
  drain.
- **Security headers** — add `X-Frame-Options`, `Referrer-Policy`,
  `Permissions-Policy`, and a baseline CSP at the nginx/frontend layer.
- **TLS story** — shipped nginx vhost is HTTP-only. Document a reverse
  proxy (Caddy/Traefik) in front for production; optionally ship a
  TLS-ready vhost with cert volume mounts. Don't embed a cert manager.
- **Scoped API keys** — `prm_…` keys are full-project-access today. Add a
  `scopes` column (`ingest`, `read`, `write`, `admin`) and enforce in the
  auth middleware.
- **Dev-surface isolation** — mailpit is proxied at `/mailpit/` in the
  single compose file. Move dev services to `docker-compose.dev.yml` so a
  production deploy cannot expose it.
- **Auth brute-force** — verify Better Auth's built-in throttling covers
  login/signup; add IP backoff if it doesn't.
- **Setup ergonomics** — `setup.sh` should generate
  `PRIMORA_ENCRYPTION_KEY` (`openssl rand -hex 32`) rather than leaving it
  to the operator.

### Reliability & operations

- **Backups + restore runbook** — `pg_dump` of `core.*`, storage-root
  archive, cron/timer example, and a documented restore path. The restore
  must be tested, not just the dump.
- **Retention** — telemetry events, audit log, and webhook deliveries grow
  unbounded. Per-project retention settings + a sweeper job.
- **Platform metrics export** — internal counters exist (surfaced in
  readiness); expose `/metrics` for Prometheus scraping.
- **Production compose profile** — healthchecks on every service, restart
  policies, pinned versions for dependencies (dragonfly is on `latest`).
- **Upgrade runbook** — today it's `git pull && compose up -d`. Document
  the safe order (backup → pull → migrate-on-boot → verify) and rollback.

### Correctness

- **Pagination audit** — every list endpoint must paginate; verify the
  heaviest (audit, events, webhook deliveries) can't return unbounded
  rows.
- **DB-to-DB links** — the one open Phase 2 item: DBX transfer between two
  saved connections.
- **External object storage** — files live under `BACKEND_STORAGE_ROOT`
  on local disk. Acceptable for single-node self-hosting; document the
  constraint or add an S3-compatible backend behind the storage service.

### Testing

- **E2E smoke in CI** — `ci.yml` runs unit tests; add a compose-up job
  exercising health, signup/login, project create, and ingest → issue
  created. Six Go test files + one frontend file is thin for a platform.
- **Migration tests** — goose up/down against a scratch database in CI.
- **Load sanity** — one `k6`/`vegeta` script against ingest + a hot read
  path; publish numbers, don't promise scale.

### Distribution

- **arm64 images** — releases build amd64 only; ARM VPS/Pi can't pull. Add
  `platforms: linux/amd64,linux/arm64` to the buildx steps.
- Carried forward from Phase 5 follow-ups: macOS/Windows bundle
  verification, signing + auto-updater.
- Optional: SBOM + provenance attestation on release images.

### Docs

- **Self-host guide** — TLS, SMTP, backups, upgrade, and env reference in
  one place. `scripts/verify-production-ready.sh` already points at
  `DEPLOYMENT_GUIDE.md` / `PRODUCTION_READINESS.md`, which don't exist —
  write them or fix the script.

**v1.0 bar:** every Security item done, a restore test actually run, E2E
smoke green in CI, and the upgrade runbook followed once on a real
upgrade. Docs site revisited here, not before.

---

## What is deliberately out of scope

- Managed/hosted tier — the product is the self-hosted thing.
- Realtime subscriptions, edge functions, generated REST/GraphQL per-table
  (Supabase PostgREST-style). Revisit after Phase 3 if wanted — Collections
  already cover the common case.
- Embedding DBX's desktop UI — we use its MCP server as a library, not its app.