Dash/docs/specs/2026-10-03-external-monitoring.md

External Monitoring Stack — Design

Date: 2026-10-03 Status: approved in chat 2026-10-03 — implementing

1. Goal

Monitor a homelab (one Proxmox host with LXCs, one always-on Windows 11 dev box, an unmanaged Ugreen CM834 switch, a router) with the collection layer living outside Dash. Dash consumes data for display and owns up/down alerting. Trigger: an unmanaged switch hung overnight with no way to know when or what failed.

2. Lane split — why two engines

Two engines were requested. They earn their place only if lanes don't overlap:

Tool	Lane	Scope
Beszel (hub + agents)	The two physical machines: PVE host, Win11	Agent push, Windows support, instant host tiles, standalone alerts
Prometheus + prometheus-pve-exporter	Everything scrapable: all LXCs/VMs via the PVE API (no guest agents), Dash's own /api/metrics	System of record, PromQL
Grafana	Graphing front for Prometheus	Panel iframes for Dash

Deliberately absent: node_exporter, windows_exporter — Beszel already owns the physical hosts. Two agents total, not two per machine.

3. Dash integration — zero code changes

Existing primitives cover the whole display layer:

• JSON-path widget → Beszel PocketBase API and Prometheus /api/v1/query (data.result[0].value[1] shape already walkable; header config field carries Authorization: Bearer …)
• Embed widget → Grafana panel URLs (anonymous Viewer + embedding enabled on the LAN) and the Beszel UI
• Monitors → ping/TCP/HTTP checks + notification transports + auto-incidents = the "something died at 03:12" alarm the user lacked
• Service links → per-console deep links

4. The switch problem

The Ugreen CM834 is unmanaged: no IP, no SNMP, no telemetry. Root cause is unobservable. What monitoring delivers instead:

• Monitors on router IP, PVE host, and one service per LXC — all reached through the switch. When it hangs, behind-switch targets fail together while router-side checks stay green: a timestamped "it was the switch" signature plus an alert to ntfy/Telegram.
• Plausible mitigation (not proven): disable EEE on host NICs — ethtool --set-eee <nic> eee off. Cheap-switch EEE hangs are a known failure mode.

5. Deliverables

infra/monitoring/
  compose.yaml            # prometheus + grafana + pve-exporter + beszel-hub
  prometheus.yml          # scrape config: pve (via exporter) + dash /api/metrics
  pve.yml.example         # PVE API token config (copy → pve.yml, gitignored)
  grafana/datasources.yaml  # auto-provisions the Prometheus source
docs/monitoring.md        # deploy guide + Dash widget/monitor recipes

No application code. No schema changes. Stack runs anywhere — an LXC on the Proxmox host is the suggested home.

6. Security notes

• Grafana anonymous Viewer + allow_embedding are set for LAN iframe use; both are called out in the guide and removable.
• pve.yml carries an API token — pve.yml.example is committed, the real file is documented as gitignored-adjacent (lives next to the compose, outside the repo if preferred).
• Widget secrets already live in item config JSON per the original spec.

7. Verification

• docker compose -f infra/monitoring/compose.yaml config valid
• promtool check config prometheus.yml (dockerized promtool) valid
• Docs-only otherwise; repo gate (typecheck/lint/build/go tests) unchanged