feat(scheduling): multi-node service placement + remote lifecycle (#54)

* feat(scheduling): multi-node service placement + remote container lifecycle

services.node_id pins a service to a node agent: explicit id, "auto"
(least-loaded online node), or "local"/empty for the local Docker host.

- DeploymentEngine gains a NodeRunner seam; ReconcileService routes
  pinned services to the agent's command queue (create_container with
  image/env/command/ports/volumes/labels/resource limits), tracked in
  container_instances with deterministic per-replica ids.
- Stop/start/restart/delete fan out to the pinned node — no more leaked
  remote containers on service delete.
- Agent create_container is idempotent (rm -f first) and honors restart
  policy, labels, memory/cpus, published+ephemeral ports, volumes, env.
- GET /nodes/options exposes id/name/status to authed users; full agent
  management stays admin-only.
- CLI services create/update --node, MCP node_id docs, OpenAPI
  node_id/node_name + /nodes/options, service-detail Placement section.

Ceilings: remote nodes run registry-pulled images only — git builds,
Traefik domains, and remote sleep detection stay local until
build-on-node and per-node ingress land.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(scheduling): node cordon/drain + schedulable gate

- node_agents.schedulable column; "auto" placement and explicit pins skip
  cordoned nodes; already-pinned services keep running (reconcile is
  unaffected — cordon is not eviction).
- POST /agents/:id/cordon|uncordon|drain (admin). Drain cordons, removes
  every service container via the agent queue, and clears the services'
  pins so they redeploy on the local host. Requires the node online —
  an offline node can't acknowledge removals.
- CLI nodes cordon|uncordon|drain (drain confirm-gated), MCP
  containr_nodes_cordon|uncordon|drain, OpenAPI paths + schedulable
  field, node-detail page cordon badge + drain/cordon controls.
- Fix: re-registering an agent no longer resets auto_prune.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(scheduling): remote registry pulls + post-run health check

- Dispatch payload carries the project owner's registry credentials for
  the image host; the agent docker-logins for the pull and logs out
  after, so private images deploy to remote nodes.
- Completed command payloads get their registry password scrubbed —
  secrets don't linger in agent_commands rows.
- Agent verifies the container is still running ~1.5s after docker run;
  crash-loops now fail the deployment with the container's last log
  lines instead of reporting "running" optimistically.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Tomáš DvořákandDevin authored and GitHub committed 2026-10-09 16:19:12 +02:00
1 parent 401daf69aa
commit bc93c83cf9
28 files changed
+1565 -48

No files matched your search

+142
View File
@@ -5080,6 +5080,34 @@ paths:
$ref: '#/components/schemas/MessageResponse'
# Node agents — management (JWT-authenticated)
/nodes/options:
get:
tags:
- Agents
summary: Node options for service placement
description: Minimal id/name/status list of registered node agents — available to any authenticated user for the service placement picker. Full agent data is admin-only under /agents.
security:
- bearerAuth: []
responses:
'200':
description: Node options
content:
application/json:
schema:
type: object
properties:
nodes:
type: array
items:
type: object
properties:
id:
type: string
name:
type: string
status:
type: string
/agents:
get:
tags:
@@ -5378,6 +5406,103 @@ paths:
schema:
$ref: '#/components/schemas/Error'
/agents/{id}/cordon:
post:
tags:
- Agents
summary: Cordon a node
description: >-
Sets `schedulable=false`: new explicit pins are rejected and `auto`
placement skips the node. Already-pinned services keep running and can
still reconcile — drain to evict them.
security:
- bearerAuth: []
parameters:
- $ref: '#/components/parameters/AgentIdParam'
responses:
'200':
description: Node cordoned
content:
application/json:
schema:
type: object
properties:
id: { type: string }
schedulable: { type: boolean }
'404':
description: Agent not found
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
/agents/{id}/uncordon:
post:
tags:
- Agents
summary: Uncordon a node
description: Re-opens the node for placement (`schedulable=true`).
security:
- bearerAuth: []
parameters:
- $ref: '#/components/parameters/AgentIdParam'
responses:
'200':
description: Node uncordoned
content:
application/json:
schema:
type: object
properties:
id: { type: string }
schedulable: { type: boolean }
'404':
description: Agent not found
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
/agents/{id}/drain:
post:
tags:
- Agents
summary: Drain a node
description: >-
Cordons the node, removes every service container via the agent
command queue, and clears the affected services' `node_id` pins —
they fall back to the local host on next deploy and can be re-pinned
elsewhere. Requires the node to be online so it can acknowledge the
removals.
security:
- bearerAuth: []
parameters:
- $ref: '#/components/parameters/AgentIdParam'
responses:
'200':
description: Node drained
content:
application/json:
schema:
type: object
properties:
id: { type: string }
schedulable: { type: boolean }
services_drained: { type: integer }
services_unpinned: { type: integer }
'404':
description: Agent not found
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
'409':
description: Node is not online
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
/agents/{id}/metrics:
get:
tags:
@@ -6843,6 +6968,12 @@ components:
sleep_idle_minutes:
type: integer
description: Idle timeout in minutes before sleeping (1-1440, default 15)
node_id:
type: string
description: Node agent id this service is pinned to; empty/null = local Docker host. 'auto' on create/update resolves to the least-loaded online agent. Remote nodes run registry-pulled images only (no git builds, no Traefik domains).
node_name:
type: string
description: Display name of the pinned node agent (read-only)
public_url:
type: string
description: Computed public URL (published port or domain)
@@ -6934,6 +7065,9 @@ components:
static_dir:
type: string
description: Output directory for the static builder (default dist)
node_id:
type: string
description: Node agent id to pin to, 'auto' = least-loaded online agent; omit/empty = local host
UpdateServiceRequest:
type: object
@@ -7020,6 +7154,9 @@ components:
sleep_idle_minutes:
type: integer
description: Idle timeout in minutes before sleeping (1-1440)
node_id:
type: string
description: Node agent id to pin to, 'auto' = least-loaded online agent, 'local' = run on the Containr host (clears the pin)
ServiceDomain:
type: object
@@ -8900,6 +9037,11 @@ components:
auto_prune:
type: boolean
description: Enqueue a bounded docker prune on this node once per day.
schedulable:
type: boolean
description: >-
False on a cordoned/drained node — `auto` placement skips it and
new explicit pins are rejected.
last_heartbeat:
type: string
format: date-time