feat(scheduling): multi-node service placement + remote lifecycle (#54)

* feat(scheduling): multi-node service placement + remote container lifecycle services.node_id pins a service to a node agent: explicit id, "auto" (least-loaded online node), or "local"/empty for the local Docker host. - DeploymentEngine gains a NodeRunner seam; ReconcileService routes pinned services to the agent's command queue (create_container with image/env/command/ports/volumes/labels/resource limits), tracked in container_instances with deterministic per-replica ids. - Stop/start/restart/delete fan out to the pinned node — no more leaked remote containers on service delete. - Agent create_container is idempotent (rm -f first) and honors restart policy, labels, memory/cpus, published+ephemeral ports, volumes, env. - GET /nodes/options exposes id/name/status to authed users; full agent management stays admin-only. - CLI services create/update --node, MCP node_id docs, OpenAPI node_id/node_name + /nodes/options, service-detail Placement section. Ceilings: remote nodes run registry-pulled images only — git builds, Traefik domains, and remote sleep detection stay local until build-on-node and per-node ingress land. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(scheduling): node cordon/drain + schedulable gate - node_agents.schedulable column; "auto" placement and explicit pins skip cordoned nodes; already-pinned services keep running (reconcile is unaffected — cordon is not eviction). - POST /agents/:id/cordon|uncordon|drain (admin). Drain cordons, removes every service container via the agent queue, and clears the services' pins so they redeploy on the local host. Requires the node online — an offline node can't acknowledge removals. - CLI nodes cordon|uncordon|drain (drain confirm-gated), MCP containr_nodes_cordon|uncordon|drain, OpenAPI paths + schedulable field, node-detail page cordon badge + drain/cordon controls. - Fix: re-registering an agent no longer resets auto_prune. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(scheduling): remote registry pulls + post-run health check - Dispatch payload carries the project owner's registry credentials for the image host; the agent docker-logins for the pull and logs out after, so private images deploy to remote nodes. - Completed command payloads get their registry password scrubbed — secrets don't linger in agent_commands rows. - Agent verifies the container is still running ~1.5s after docker run; crash-loops now fail the deployment with the container's last log lines instead of reporting "running" optimistically. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 15:51:26 +00:00 · 2026-10-09 16:19:12 +02:00
1 parent 401daf69aa
commit bc93c83cf9
@@ -2,7 +2,9 @@ import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
 import { useMemo, useState } from 'react';
 import { Link, useNavigate, useParams } from 'react-router-dom';
 import {
+  cordonAgent,
  deleteAgent,
+  drainAgent,
  getAgent,
  getAgentMetrics,
  listAgentCommands,
@@ -43,6 +45,7 @@ export function NodeDetailPage() {
  const [rename, setRename] = useState<string | null>(null);
  const [confirmDelete, setConfirmDelete] = useState(false);
  const [confirmVolumes, setConfirmVolumes] = useState(false);
+  const [confirmDrain, setConfirmDrain] = useState(false);

  const agentQuery = useQuery({
    queryKey: ['node-agent', id],
@@ -86,6 +89,14 @@ export function NodeDetailPage() {
    mutationFn: () => deleteAgent(id),
    onSuccess: () => navigate('/usage'),
  });
+  const cordonMutation = useMutation({
+    mutationFn: (cordoned: boolean) => cordonAgent(id, cordoned),
+    onSuccess: invalidate,
+  });
+  const drainMutation = useMutation({
+    mutationFn: () => drainAgent(id),
+    onSuccess: () => { setConfirmDrain(false); invalidate(); },
+  });

  const agent = agentQuery.data;
  const metrics = useMemo(() => metricsQuery.data ?? [], [metricsQuery.data]);
@@ -168,6 +179,11 @@ export function NodeDetailPage() {
                : 'border-[var(--error)]/40 text-[var(--error)]'}`}>
                {agent.status}
              </span>
+              {!agent.schedulable && (
+                <span className="v-mono text-[11px] px-2 py-1 rounded-md border border-[var(--warning)]/40 text-[var(--warning)]">
+                  cordoned
+                </span>
+              )}
              {isAdmin && (
                <>
                  <label className="flex items-center gap-1.5 v-mono text-[11px] text-[var(--text-tertiary)] cursor-pointer" title="Daily automatic docker prune">
@@ -179,6 +195,26 @@ export function NodeDetailPage() {
                    />
                    auto-prune
                  </label>
+                  <button
+                    onClick={() => cordonMutation.mutate(agent.schedulable)}
+                    disabled={cordonMutation.isPending}
+                    title={agent.schedulable
+                      ? 'Block new placements — running replicas stay'
+                      : 'Re-open for placement'}
+                    className="v-mono text-[11px] px-3 py-1.5 rounded-md border border-[var(--border-subtle)] text-[var(--text-secondary)] hover:text-[var(--text-primary)] disabled:opacity-50"
+                  >
+                    {agent.schedulable ? 'cordon' : 'uncordon'}
+                  </button>
+                  <button
+                    onClick={() => setConfirmDrain(true)}
+                    disabled={drainMutation.isPending || !online}
+                    title={online
+                      ? 'Cordon + remove all service containers + unpin services (they redeploy locally)'
+                      : 'Node must be online to drain'}
+                    className="v-mono text-[11px] px-3 py-1.5 rounded-md border border-[var(--warning)]/40 text-[var(--warning)] hover:bg-[var(--warning)]/10 disabled:opacity-50"
+                  >
+                    drain
+                  </button>
                  <button
                    onClick={() => setConfirmVolumes(true)}
                    disabled={pruneMutation.isPending}
@@ -302,6 +338,17 @@ export function NodeDetailPage() {
          pending={pruneMutation.isPending}
        />
      )}
+      {confirmDrain && (
+        <ConfirmModal
+          title={`Drain ${agent.name}?`}
+          body="Every service container on this node is removed and the affected services are unpinned — they redeploy on the local host until re-pinned elsewhere. The node stays cordoned."
+          confirmLabel="drain node"
+          danger
+          onCancel={() => setConfirmDrain(false)}
+          onConfirm={() => drainMutation.mutate()}
+          pending={drainMutation.isPending}
+        />
+      )}
      {confirmDelete && (
        <ConfirmModal
          title={`Remove ${agent.name}?`}
@@ -48,6 +48,7 @@ import {
  cloneService,
  moveService,
  listProjects,
+  listNodeOptions,
 } from '@/lib/api-client';
 import { getDemoProjectById, getDemoServiceById, getDemoCronJobsByService } from '@/lib/demo-data';
 import { useDemoMode } from '@/lib/demo-mode';
@@ -466,6 +467,7 @@ export function ServiceDetailPage() {
    onSuccess: () => {
      setNetworkForm(null);
      setSleepDraft(null);
+      setNodeDraft(null);
      invalidateService();
    },
  });
@@ -485,6 +487,13 @@ export function ServiceDetailPage() {
  const [domainChecks, setDomainChecks] = useState<Record<string, DomainCheckResult> | null>(null);
  const [accessForm, setAccessForm] = useState<{ maintenance: boolean; basicAuth: string } | null>(null);
  const [sleepDraft, setSleepDraft] = useState<{ id: string; enabled: boolean; minutes: string } | null>(null);
+  const [nodeDraft, setNodeDraft] = useState<string | null>(null);
+  const nodesQuery = useQuery({
+    queryKey: ['node-options'],
+    queryFn: listNodeOptions,
+    enabled: !isDemoMode,
+    staleTime: 30_000,
+  });
  const [buildForm, setBuildForm] = useState<{
    builder: string; cpuReserve: string; memoryReserve: string; staticCmd: string; staticDir: string;
  } | null>(null);
@@ -2384,6 +2393,58 @@ export function ServiceDetailPage() {
              </div>
            )}

+            {!isDemoMode && (nodesQuery.data?.length ?? 0) > 0 && (
+              <div className="mt-6 pt-6 border-t border-[var(--border-subtle)]">
+                <div className="mb-3">
+                  <h3 className="text-sm font-medium text-[var(--text-primary)]">Placement</h3>
+                  <p className="text-xs text-[var(--text-tertiary)] mt-0.5">
+                    Which node runs this service's containers. Remote nodes pull the image
+                    from its registry — git builds and domains stay on the local host.
+                  </p>
+                </div>
+                <div className="flex items-center gap-3 text-xs">
+                  <select
+                    value={nodeDraft ?? (service.nodeId || 'local')}
+                    onChange={(e) => setNodeDraft(e.target.value)}
+                    className="px-2 py-1.5 rounded-[var(--radius-md)] bg-[var(--surface-muted)] border border-[var(--border-subtle)] text-sm text-[var(--text-primary)]"
+                  >
+                    <option value="local">This server (local Docker)</option>
+                    <option value="auto">Auto — least-loaded online node</option>
+                    {(nodesQuery.data ?? []).map((n) => (
+                      <option key={n.id} value={n.id}>
+                        {n.name}{n.status !== 'online' ? ' (offline)' : ''}
+                      </option>
+                    ))}
+                  </select>
+                  {nodeDraft !== null && (
+                    <>
+                      <button
+                        type="button"
+                        disabled={updateServiceMutation.isPending}
+                        onClick={() => updateServiceMutation.mutate({ node_id: nodeDraft })}
+                        className="px-3 py-1.5 rounded-[var(--radius-md)] bg-[var(--accent-primary)] text-[var(--accent-on)] text-xs font-medium disabled:opacity-50"
+                      >
+                        Save
+                      </button>
+                      <button
+                        type="button"
+                        onClick={() => setNodeDraft(null)}
+                        className="text-[var(--text-tertiary)] hover:text-[var(--text-primary)]"
+                      >
+                        Cancel
+                      </button>
+                    </>
+                  )}
+                  {service.nodeName && nodeDraft === null && (
+                    <span className="text-[var(--text-tertiary)]">running on {service.nodeName}</span>
+                  )}
+                </div>
+                {updateServiceMutation.isError && (
+                  <p className="mt-2 text-xs text-[var(--error)]">{(updateServiceMutation.error as Error).message}</p>
+                )}
+              </div>
+            )}
+
            {!isDemoMode && (
              <div className="mt-6 pt-6 border-t border-[var(--border-subtle)]">
                <div className="flex items-center justify-between mb-4">