From 17e429b86b9f5f25f20ab774dd3a72827d5cd4f9 Mon Sep 17 00:00:00 2001 From: Tomas Dvorak Date: Thu, 8 Oct 2026 14:31:39 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20ROADMAP2=20=E2=80=94=20dflow=20successo?= =?UTF-8?q?r=20plan=20and=20phased=20build?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deep analysis of dflow (~43 queue processors, ~30 collections, ~180 server actions) mapped against Containr, sectioned into phases A–G. Phase A (agent spine) is in progress — this commit adds PATs. Authored By: TDvorak --- ROADMAP2.md | 396 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 396 insertions(+) create mode 100644 ROADMAP2.md diff --git a/ROADMAP2.md b/ROADMAP2.md new file mode 100644 index 0000000..f96396b --- /dev/null +++ b/ROADMAP2.md @@ -0,0 +1,396 @@ +# Containr — Roadmap 2: The dflow Convergence + +Last verified: **2026-10-08** — deep pass over the full dflow codebase +(`~/Desktop/PROG+HTML/dflow`, community fork of dflow-sh/dflow, frozen for +preservation) against Containr `main` (`15b5b0e`). + +Goal: make Containr the superior successor — absorb everything worth having +from dflow, then go past it and past Railway. Same power for humans and +agents: the UI is the friendly face, the CLI and MCP are the machine face, +all three sit on the same API. **Branding stays Containr/Vertice** — +white-labeling is an admin option, not an identity change. + +--- + +## 1. What dflow actually is (verified) + +Next.js 15 + PayloadCMS 3 + MongoDB + Redis/BullMQ. Runtime model: every +host is a remote server driven **over SSH** with Dokku auto-installed as the +per-app container manager. Async work = 43 BullMQ queue processors +(`src/queues/**`), live logs via Redis pub/sub → SSE endpoint, persisted to +deployment records. ~30 collections, ~180 server actions, 11 dashboard +routes per service (general, deployments, logs, environment, domains, +volumes, proxy, scaling, backups, settings). + +### Full capability matrix — dflow → Containr + +Legend: ✅ done · 🟡 partial/different · ❌ missing · ➖ not applicable (dokku/SSH-specific) + +#### Deployment pipeline +| dflow capability | Containr status | Action | +|---|---|---| +| Queued deployments, serialized per server, cancellable while queued | ❌ goroutine fire-and-forget; `CancelDeployment` exists in engine but is **never routed** | Add deploy queue + cancel route | +| Deployment states `queued → building → success/failed`, live SSE logs + persisted record | 🟡 states exist (`building→running`); WS streams builds; no queued state | Extend status + stream | +| Redeploy cache modes (`ps:rebuild` vs fresh pull); webhook pushes force `no-cache` | ❌ always rebuilds | Add `cache` flag on deploy/redeploy | +| Cancel queued deployment from UI | ❌ (engine fn unrouted) | Route it | +| Per-service job ordering | ❌ concurrent deploys of one service can race | Serialize per service | + +#### Service configuration +| dflow capability | Containr status | Action | +|---|---|---| +| Multi-domain per service + default flag + cert type | ❌ single `services.domain` | `service_domains` table | +| Cert types: letsencrypt / none / Cloudflare Origin / **CF-for-SaaS custom hostnames** | 🟡 Traefik ACME only | LE default; CF custom hostnames deferred (only needed if hosting tenant domains) | +| DNS preflight check (`checkDNSConfigAction`) | ❌ | `GET /services/:id/domains/check` | +| Volumes per service + dangling-volume browser/attach/delete | ❌ engine supports `VolumeMounts`; no API/UI | `services.volumes` jsonb + Volumes tab | +| Resource **limits and reservations**, per-server defaults | 🟡 `cpu_limit`/`memory_limit` int columns (unclear units) | explicit string units + reserves + instance defaults | +| Pre-deploy capacity check (dokku binary interceptor) | ❌ | Node capacity check pre-deploy/scale | +| Maintenance mode toggle | ❌ | Traefik middleware label | +| Basic-auth gate per service (`toggleHttpAuth`) | ❌ | Traefik basic-auth middleware | +| Raw proxy config override (`setServiceNginxConfig`) | ❌ | Allowlisted `traefik_labels` jsonb | +| Service clone (full config copy) | ❌ | `POST /services/:id/clone` | +| Move service between projects (`SwitchServiceProjectDialog`) | ❌ | `PUT /services/:id {project_id}` | +| Env var save → hot-apply + optional restart (`noRestart`) | ❌ rows rewritten; containers keep stale env until manual redeploy | Apply→redeploy flow | +| Static-site builder; Heroku buildpacks | 🟡 railpack/nixpacks/dockerfile auto-detect, no user override | `services.builder` enum + static builder | +| Docker registry accounts (private registries) | ❌ | `registry_credentials` per service/instance | + +#### Databases +| dflow capability | Containr status | Action | +|---|---|---| +| Managed DBs (pg/mysql/mariadb/mongo/redis/clickhouse via dokku plugins) | ✅ plus dragonfly | — | +| **External DB registration** (Neon/Atlas/RDS conn details + test) | ❌ | `provider=external` + test endpoint | +| Internal backups + restore (same-type enforced) | ✅ | — | +| Scheduled backups | ✅ cron field | — | +| **External S3 backups** (auth/schedule/unschedule ops) | ❌ | `backup_targets` + shipper | +| **Cross-server DB migration** (export → sftp hop → import) | ❌ | `POST /databases/:id/migrate` | +| Public DB port expose/unexpose | ❌ loopback only | opt-in TCP expose | +| DB → service env linking (`dokku link`) | ✅ Bind action injects `DATABASE_URL` | — | + +#### Servers / infrastructure +| dflow capability | Containr status | Action | +|---|---|---| +| Attach server = paste IP + SSH key; Dokku auto-installs | 🟡 agents exist, manual install | **SSH bootstrap**: one SSH session installs agent, never needed again | +| Tailscale/Headscale, NetBird, ZeroTier enroll + mesh connect | ❌ | Agent mesh-IP detection first; auth-key enroll optional | +| AWS EC2 + Hetzner provisioning w/ cloud-init | ❌ | IaC (ROADMAP §2.2): Terraform Proxmox/AWS + Hetzner module | +| Security groups (AWS SG rules, sync) | ❌ | Under IaC provider modules | +| Server-level global/wildcard domain (`updateServerDomain`) | ❌ | `nodes.default_domain` | +| Scheduled server cleanup (dokku cleanup → docker prune) | ❌ | Agent `prune` command + schedule | +| Queue flush/inspect per server (`flushServerQueues`, `getServerQueues`) | ➖ no queue yet | Ops page shows jobs once queue exists | +| Arbitrary SSH command on server (`executeCommand`) | ➖ exec is container-scoped | Node terminal via agent (PTY) | +| Server onboarding wizard, reset, cleanup-on-delete | 🟡 token issue/revoke only | Onboarding flow + `nodes add` UX | +| Ansible playbooks (stored, runnable per server, exec history) | ➖ | **Defer** — security surface; SSH bootstrap covers provisioning | +| Per-node resource dashboards | 🟡 heartbeat telemetry exists | Node detail page | + +#### Platform / org +| dflow capability | Containr status | Action | +|---|---|---| +| Multi-tenant organisations + custom RBAC (per-resource CRUD, create-limits, read-scope) | 🟡 `is_admin` + `project_members` | Defer full RBAC; design PAT scopes role-compatible | +| Team invite links/emails | 🟡 manual user create only | `POST /invites` → link join | +| User API keys → full REST API | ❌ **the big gap** — session/JWT only | PATs (Phase A) | +| Outbound webhooks (collection events, HMAC, custom headers) | ❌ | `webhooks` table + delivery log | +| Banners (global + per-tenant, dismissible) | ❌ | `banners` table + top bar | +| White-label (logo/favicon/title/OG/theme colors) | ❌ | `app_settings.branding` — **Containr remains the default brand** | +| Activity feed (icon/severity/category/metadata, 90-day TTL job) | 🟡 `audit_logs` coarser | Enrich + project feed UI | +| Admin metrics dashboard (users/servers/queued/failed counts) | 🟡 `/admin` exists | Extend | +| **Impersonate user** | ❌ | Admin support tool | +| In-app docs (content-collections markdown) | ✅ `/docs` | — | +| Dynamic client-side filter framework (535-line `filter.utils`) | 🟡 audit page filters only | Reuse pattern for tables | +| Ops "bubble" (queue monitor, terminal, sync) | ❌ | Operations page | + +#### Agent surface +| dflow capability | Containr status | Action | +|---|---|---| +| Payload REST/GraphQL auto-API + per-user API keys | ❌ | PATs + OpenAPI (better: typed, documented) | +| CLI | ❌ | none in dflow — **Containr CLI is a genuine leap past it** | +| MCP | ❌ | none — same leap | +| Railway migrator script | ❌ | `containr migrate railway` | +| GitHub App **manifest flow** (instance self-provisions the App) | 🟡 env-configured App | `POST /git/github-app/manifest` flow | +| fork-sync webhook deploy trigger | ❌ | ~20 lines on receiver | +| Azure DevOps provider | ❌ | Optional (dflow does token-clone only) | + +--- + +## 2. The spine: agent-first platform + +*Agent and user on the same level.* Concretely: + +- Every operation reachable by a **personal access token**, not just + sessions. +- Every operation in the **CLI** with `--json`, stable exit codes, `--yes`. +- Every operation in an **MCP server** (stdio first) so Devin/Claude Code/ + Cursor/Codex drive Containr natively. +- `openapi.yaml` stays the single source of truth; UI, CLI, MCP are thin + clients. + +### Phase A — Agent surface (do this FIRST; everything after inherits it) + +- [ ] **Personal access tokens.** `user_tokens` table — `id`, `user_id`, + `name`, `key_prefix` (`cnp_…`), `key_hash` (sha256), `scopes` + (`read`/`write`/`admin`, later resource-scoped), `expires_at`, + `last_used_at`. Model on `agent_auth_tokens` (raw shown once, hash stored). + `POST/GET/DELETE /api/v1/user/tokens`; `middleware.Auth` accepts + `Bearer cnp_…`; Settings → Access tokens UI with one-time display + + copyable CLI/MCP config snippets. +- [ ] **CLI v2.** Current `cmd/cli` is auth + projects only. Full client: + - `auth login [--token] [--url] [--profile]` (multi-instance profiles). + - Resources: `projects services deploy logs exec variables databases + cron templates nodes scaling ha security gateway notifications + webhooks domains volumes backups admin`. + - `containr up` / `deploy` — cwd→service create-or-update→watch + (the `railway up` equivalent; single most important command). + - `containr link` (bind cwd→service), `containr open`, `containr run`, + `containr shell` (PTY once exec upgrade lands), `completion`. + - Universal `--json`, `-o yaml`, `--watch`, `--yes`; exit codes + 0/1/2/3/4 (ok/err/not-found/unauthorized/conflict). + - GoReleaser → GH releases + `install.sh`; `version` command. + - Conventions from docsync/envdiff/cix/repolint. +- [ ] **MCP server** (`cmd/mcp`, same module, stdio). Tools mirror the + CLI: `list_projects create_service deploy get_logs set_variable + restart rollback create_database trigger_cron deploy_template exec + get_metrics security_scan …`. Auth: `CONTAINR_URL` + `CONTAINR_TOKEN` + or shared `~/.containr/config`. `containr mcp serve` + install snippets + for Devin/Claude/Cursor. +- [ ] **API agent ergonomics.** `Idempotency-Key` on mutating endpoints; + uniform `{error, code, details}` envelope (audit bare-string returns); + cursor pagination on growing lists; name-or-id lookup everywhere. +- [ ] **Agent docs.** Root `AGENTS.md`, `/docs` serves `llms.txt` + + `llms-full.txt`, MCP tool list generated from OpenAPI where possible. + +**Gate**: an agent holding only a PAT registers a project, deploys from a +git repo, streams logs, sets vars, rolls back — zero UI. Same via CLI +for a human. + +--- + +## 3. Phase B — Deployment pipeline maturity + +dflow's one real architectural edge. Containr deploys are fire-and-forget +goroutines; this fixes correctness *and* unlocks the ops surface. + +- [ ] **Deploy job queue.** Lightweight internal queue (Postgres + `deployment_queue` table + single worker loop — no new infra; Redis + exists but don't add BullMQ-style sprawl). Serialize per service; + optional per-node serialization. Status gains `queued`. +- [ ] **Cancel deployments.** Route `POST /deployments/:id/cancel` → + engine `CancelDeployment` (exists, unrouted). Queued jobs cancel + cleanly; building jobs cancel context where safe. +- [ ] **Cache modes.** `?cache=full` on redeploy → rebuild from existing + image layers vs clean pull/checkout. Webhook-triggered deploys default + to no-cache (dflow convention — prevents stale-layer surprises). +- [ ] **Queued→building→running status + persisted step logs.** Already + mostly there; add `queued`, stream over existing `/ws`. +- [ ] **Env var apply → redeploy.** Saving variables asks "restart to + apply?" (or `--apply` flag in CLI); backend triggers redeploy honoring + running state. Currently rows update but containers keep stale env — + silent-drift bug class, not just UX. +- [ ] **Encrypt variables at rest.** `environment_variables.value` is + plaintext today (verified). Envelope-encrypt `is_secret` values + (AES-GCM, key from `JWT_SECRET` or dedicated `SECRETS_KEY`); dflow + encrypts every var, we encrypt secrets — migration = transparent + encrypt-on-write + decrypt-on-read. + +## 4. Phase C — Service configuration parity + +- [ ] **Volumes on services** — `services.volumes` jsonb + `[{volume|host_path, container_path, read_only}]`; engine accepts + `VolumeMounts` already. Volumes tab + orphan-volume listing + (`docker volume ls -f dangling`) with delete/attach. +- [ ] **Multi-domain** — `service_domains(service_id, domain, is_default, + cert_type, status, last_checked_at)`. Domains tab: add/remove/ + set-default/DNS-status badge. Keep `services.domain` as derived + default for compat. +- [ ] **DNS preflight** — `GET /services/:id/domains/check` resolves and + compares vs edge IP → `ok|wrong-target|pending`. +- [ ] **Maintenance mode + basic-auth gate** — Traefik middleware labels; + no proxy code needed. +- [ ] **Routing overrides** — `services.traefik_labels` jsonb with + allowlist validation (middlewares, headers, redirects). +- [ ] **Builder override** — `services.builder` enum + `auto|railpack|nixpacks|dockerfile|static`; fix hardcoded `nixpacks` + at `deployments.go:406`. Static builder = build + nginx sidecar. +- [ ] **Resource limits + reserves** — adopt dflow-style explicit units + (`"0.5"`/`"512m"`); add `cpu_reserve`/`memory_reserve` → + `NanoCPUs`/`Memory`/`MemoryReservation`; instance defaults in + `app_settings`. +- [ ] **Pre-deploy capacity check** — sum reservations vs node capacity + (agent telemetry); warn or block per policy flag. +- [ ] **Service clone** + **move between projects**. +- [ ] **Private registries** — `registry_credentials` + (server/username/secret, encrypted); `docker login` at pull time; + per-service registry link. +- [ ] **Service detail tabs to match dflow's surface**: add Domains, + Volumes, Proxy/access-controls, Backups sections to the service page + (Cron/Previews/Scaling already exist). + +## 5. Phase D — Data layer maturity + +- [ ] **External DB registration** — `provider='managed'|'external'`, + conn fields, `POST /databases/test-connection`, same bind flow. +- [ ] **Public DB port expose** — opt-in Traefik TCP route or host port; + default stays loopback. +- [ ] **Offsite backups** — `backup_targets` (S3-compatible: + endpoint/bucket/keys encrypted); per-DB destination; ship archives via + utility container + s3 SDK; restore-from-S3. +- [ ] **Cross-node DB migration** — `POST /databases/:id/migrate + {target_node_id}`: dump → stream transfer → provision → restore → + re-point bound services. Needs ≥2 nodes or local↔node. +- [ ] **Backup import** — upload archive → new restore point (pairs with + migrators). + +## 6. Phase E — Templates v2 (absorbs compose-catalog spec + dflow graph templates) + +- [ ] **Template = ordered service graph** — `services[]` entries with + `{name, type, source, build, ports, volumes, variables, depends_on}`; + deploy order from dependency refs. +- [ ] **Expressions** — `${{service.KEY}}` (already the runtime ref + syntax — reuse), `{{secret(len,"charset")}}`, `{{random:int}}`, + `{{VAR:-default}}`; `POST /templates/:id/plan` returns the resolved + graph without deploying (agents love this). +- [ ] **Compose import** — `POST /templates/import/compose` (compose-go + parser → graph), `POST /templates/import/github` (compose file from + repo path). Implements `docs/superpowers/specs/2026-04-14-*`. +- [ ] **Graph deploy pipeline** — creates project + services in + dependency order, resolves vars, triggers builds; defined + partial-failure policy. +- [ ] **Visual template builder** — canvas mode: drop git/image/db + nodes, wire refs, set order, save/publish. Reuses existing canvas + primitives (dflow's `templates/compose` is exactly this). +- [ ] **Catalog growth** — convert `templates/*.md` (20 app guides) into + real graph templates; screenshots + badges per the old spec. + +**Gate**: paste n8n compose → 3-service project with generated secrets +and wired refs, one click or one CLI call. + +## 7. Phase F — Nodes & infrastructure + +- [ ] **SSH bootstrap install** — `containr nodes add --ssh user@host` + + UI flow: one SSH session runs agent install (binary + systemd + + enroll token), SSH closed forever. dflow onboarding UX, agent-model + security. +- [ ] **Mesh awareness** — agent detects + reports tailscale0/netbird0/ + zt0 IPs at register; backend stores + displays. Optional + `TAILSCALE_AUTH_KEY` enroll-at-provision later. +- [ ] **Node detail page** — placed services, telemetry history, agent + version + self-upgrade, **drain/cordon**, remove. +- [ ] **Node housekeeping** — agent `prune` command (docker system + prune, bounded) + optional schedule (`setServerAutoCleanup` analog — + disk pressure is the #1 small-VPS killer). +- [ ] **Multi-node scheduling** — wire `scheduling_rules` (table exists): + pin/spread/affinity, prefer-lowest-usage; deploy engine routes to + `services.node_id` via agent commands (agent already runs containers). +- [ ] **Build-on-node** — agents currently `docker run` prebuilt images + only; extend agent to pull registry images (build stays central) — + private registry creds flow through Phase C. +- [ ] **IaC provisioning (ROADMAP §2.2, unchanged)** — Terraform/ + OpenTofu, `infra_connections`, Proxmox + AWS modules, + Hetzner module + (dflow's second provider), cloud-init → auto-enroll. +- [ ] **Global/wildcard domain per node** (`nodes.default_domain`) — + auto-domains for services on that node. + +## 8. Phase G — Platform polish + +- [ ] **Outbound webhooks** — `webhooks(url, secret, events[], headers, + enabled)` + `webhook_deliveries` log; HMAC `X-Containr-Signature`; + events: `deployment.* service.* database.* security.* agent.*`. +- [ ] **Activity feed** — enrich `audit_logs` with `severity category + icon label metadata`; per-project timeline + global feed; TTL job. +- [ ] **Banners** — `banners(title, body, level, scope, active)`; top + bar in UI + surfaced to CLI on `auth status`. +- [ ] **White-label** — `app_settings.branding`: product name, logos, + favicon, accent override, docs/support links. `GET /api/v1/branding` + public + cached. **Containr/Vertice remains default;** this is for + self-hosters who rebrand their instance. +- [ ] **Operations page** — unified view of running/queued/failed jobs + (deploys, builds, backups, cron, scans, agent commands) — dflow's + bubble panel as a real page; flush/stuck-job tools once queue exists. +- [ ] **Interactive terminal** — upgrade exec to WS PTY (`xterm.js`); + `nodes/:id/terminal` via agent (dflow's wetty without the extra + container). +- [ ] **Team invites** — `POST /invites` → signed link → join flow; + replaces manual-only user creation. +- [ ] **Impersonate user** — admin support tool, audit-logged. +- [ ] **GitHub App manifest flow** — `POST /git/github-app/manifest` + returns manifest + redirect; instance self-provisions its App + (dflow pattern — removes the env-setup barrier). +- [ ] **fork-sync trigger** — accept repo-sync events on webhook + receiver. +- [ ] **Dynamic filter framework** — port dflow's `filter.utils` + pattern (typed filter configs → URL state → client filtering) for + audit logs, deployments, notifications tables. +- [ ] **Notification producers round-out** — backups, agent offline, + upgrade available, security findings. +- [ ] **Custom roles/RBAC** — defer until demanded; PAT scopes designed + forward-compatible now. + +## 9. Phase H — Migration & ecosystem + +- [ ] **Railway migrator** — `containr migrate railway --token --project + --dry-run`: Railway GraphQL → services/vars/domains → recreate via + PAT. Landing-page pitch: "off Railway in one command." +- [ ] **dflow importer** — `containr migrate dflow` via Payload REST + export or Mongo dump: servers→nodes, projects, services(+vars/ + domains/volumes), databases, templates. Onboards dflow refugees. +- [ ] **`containr import compose`** — Phase-E parser straight to a + project (no template intermediate). +- [ ] **Template registry** — optional `CONTAINR_TEMPLATE_REGISTRY_URL` + fetch/publish; bundled catalog works offline (dflow's self-contained + principle — nothing phones home). + +## 10. Carried-over debt (ROADMAP.md items still open) + +- [ ] **Gateway traffic path** — mount `/g/:slug/*` → key → rate-limit → + `gateway/proxy.go` → metrics. Management API exists; proxy unwired. +- [ ] **Preview environments runtime** — real build+deploy+subdomain+TTL + (bookkeeping only now). Pairs with clone + multi-domain. +- [ ] **Environments** — env switcher, per-env vars/domains, promote. +- [ ] **sqlc port** — ~140 raw `database/sql` sites → `sqlc/queries/`. +- [ ] **Tests** — canvas/wizard frontend coverage, deployment/template/ + webhook table tests, Playwright e2e golden path. +- [ ] **First-run wizard** — admin → GitHub App (manifest flow from + Phase G makes this self-serve) → Tunnel → done. +- [ ] **Self-upgrade loop verification**, docs freshness, v1.0.0 release. + +## 11. Deliberately not imported from dflow + +- Dokku-over-SSH runtime — our Docker API + agent model is strictly + better. +- Per-server-named BullMQ queues — our queue stays a single ordered + table. +- Payload/Mongo/Next.js stack, multi-tenant plugin machinery. +- Ansible playbook runner — arbitrary code-on-node is a security + surface; SSH bootstrap + agent commands cover the real needs. + Revisit only with a concrete demand. +- `isAdmin`-only collection access — our owner/admin/public model is + richer already. +- Netdata install flow — our agent telemetry + optional Beszel covers + monitoring; don't ship a third dashboard stack. + +## 12. Recommended order + +``` +A (agent surface) ──► B (deploy pipeline) ──► C/D (service+data depth) + │ │ + ▼ ▼ + CLI/MCP dogfood every feature E (templates v2) + │ + F (nodes/infra) ──► H (migrators) + │ + G (polish) + debt +``` + +Why this order: PATs+CLI+MCP first = every later feature ships +user-and-agent-visible at once. The queue (B) is prerequisite for +the ops surface and correct concurrency. Templates v2 needs volumes, +multi-domain, clone. Migrators last — they're launch marketing, not +foundation. + +## 13. Design rules + +- **API is the contract.** UI, CLI, MCP are thin clients over + `openapi.yaml`. A feature isn't done until all three reach it. +- **Everything scriptable.** JSON in/out, `--json`/`--yes`, uniform + pagination, name-or-id addressing. +- **Same powers, same position.** Nothing user-facing that isn't + API-facing; agent-visible state (tokens, jobs) visible to users too. +- **dflow is a donor, not a dependency.** Port semantics, not code. +- **Containr branding is the default**; white-label is instance config. +- **Self-hosted first** — no required external service, no phone-home.