mirror of
https://github.com/Dvorinka/Dash.git
synced 2026-10-28 15:51:27 +00:00
Tomas DvorakandDevin
f2aa9ceba8
Homelab monitoring design (docs/specs/2026-10-03-external-monitoring.md): Beszel agents own the two physical hosts, prometheus-pve-exporter covers every LXC/VM via the PVE API with no guest agents, and Grafana supplies graphs. Dash needs zero code changes — JSON-path widgets already carry an Authorization header and can walk Prometheus/Beszel JSON, embed widgets iframe Grafana panels, and monitors + notification transports provide the alerting layer for unmanaged hardware like the CM834 switch. infra/monitoring/ ships compose.yaml, prometheus.yml, pve.yml.example (gitignored real file), and Grafana datasource provisioning. docs/monitoring.md documents deployment and widget recipes. Authored By: TDvorak <[email protected]> Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
120 lines
4.7 KiB
Markdown
120 lines
4.7 KiB
Markdown
# Monitoring your infrastructure
|
|
|
|
Dash displays and alerts; it does not collect metrics beyond what
|
|
`dash-agent` pushes. For real infrastructure monitoring, run the reference
|
|
stack in [`infra/monitoring/`](../infra/monitoring/) next to Dash — it works
|
|
on any host. A small Debian LXC on the Proxmox box is a good home.
|
|
|
|
## What watches what
|
|
|
|
| Layer | Tool | Covers |
|
|
|---|---|---|
|
|
| Physical hosts | Beszel hub + agents | Proxmox host, Windows 11 box — CPU/mem/disk/net/temps |
|
|
| Cluster + apps | Prometheus + pve-exporter | Every LXC/VM via the PVE API (no guest agents), Dash's own `/api/metrics` |
|
|
| Dashboards | Grafana | Graphs over Prometheus, embedded into Dash via iframes |
|
|
| Up/down + alerts | Dash monitors | Router, hosts, services — with notification transports and incidents |
|
|
|
|
Two agents total — Beszel deliberately owns only the physical machines; the
|
|
PVE API already knows everything about the guests.
|
|
|
|
## 1. Deploy the stack
|
|
|
|
```sh
|
|
cd infra/monitoring
|
|
cp pve.yml.example pve.yml # fill in the PVE API token (see below)
|
|
# edit prometheus.yml: replace PVE_HOST and DASH_HOST with real addresses
|
|
docker compose up -d
|
|
```
|
|
|
|
UIs: Prometheus `:9090`, Grafana `:3001`, Beszel `:8090`, pve-exporter `:9221`.
|
|
|
|
### Proxmox API token (for pve-exporter)
|
|
|
|
On the PVE host:
|
|
|
|
```sh
|
|
pveum user add dashmon@pve --comment "read-only monitoring"
|
|
pveum aclmod / -user dashmon@pve -role PVEAuditor
|
|
```
|
|
|
|
Then **Datacenter → Permissions → API Tokens → Add** a token named `dash`
|
|
for `dashmon@pve`, uncheck *Privilege Separation*, copy the secret into
|
|
`pve.yml`. Token auth means no password in the file.
|
|
|
|
### Beszel agents (physical hosts)
|
|
|
|
1. Open Beszel `http://<hub>:8090`, create the admin account.
|
|
2. **Add System** → the hub generates the exact install command with the
|
|
key embedded — binary/systemd for the PVE host, Windows build for the
|
|
dev box (run it under NSSM or Task Scheduler for boot persistence).
|
|
3. Agents listen on `:45876`; the hub connects to them. On the same LAN
|
|
this needs no extra plumbing.
|
|
|
|
## 2. Show it in Dash — no code required
|
|
|
|
### Live values → JSON value widget
|
|
|
|
Add item → widget type **JSON value**:
|
|
|
|
**Beszel** — fields are version-dependent; browse the API to pick a path:
|
|
|
|
- URL: `http://<beszel>:8090/api/collections/systems/records`
|
|
- Header: `Authorization: Bearer <token>` — mint a superuser token/key in
|
|
the PocketBase admin at `http://<beszel>:8090/_/`
|
|
- Path: e.g. `items[0].info.cpu` (verify in the JSON response)
|
|
|
|
**Prometheus** — instant queries return JSON:
|
|
|
|
- URL: `http://<prometheus>:9090/api/v1/query?query=pve_cpu_usage_ratio`
|
|
- Path: `data.result[0].value[1]`
|
|
- Useful queries: `pve_cpu_usage_ratio`, `pve_memory_usage_ratio`,
|
|
`pve_disk_usage_ratio`, `pve_up`
|
|
|
|
### Graphs → Embed widget
|
|
|
|
Grafana dashboard → share a panel → **Embed** gives a `d-solo` URL. Paste it
|
|
into a Dash **Embed** widget (append `&kiosk` for chrome-free panels). The
|
|
shipped compose enables anonymous Viewer + embedding, so iframes render
|
|
without login on the LAN — remove the `GF_AUTH_ANONYMOUS_*` and
|
|
`GF_SECURITY_ALLOW_EMBEDDING` env lines if you'd rather require sign-in, and
|
|
use Grafana *public dashboard* links instead.
|
|
|
|
The whole Beszel UI also iframes cleanly if you want its charts in a tile.
|
|
|
|
## 3. Alert on outages → Dash monitors
|
|
|
|
The point of the exercise: when the unmanaged switch hangs, everything
|
|
*behind* it drops at once while router-side checks stay green. That
|
|
co-failure pattern — timestamped, pushed to your phone — is as close to
|
|
"the switch died" as an unmanaged box allows.
|
|
|
|
Recommended monitors:
|
|
|
|
| Target | Type | Why |
|
|
|---|---|---|
|
|
| Router / gateway IP | Ping | Upstream reference point |
|
|
| Proxmox host | Ping + HTTP `:8006` | Behind the switch |
|
|
| Each LXC's service | HTTP/TCP | Per-app liveness |
|
|
| Win11 box | TCP `:3389` (RDP) | Behind the switch |
|
|
| Prometheus/Grafana/Beszel | HTTP | Watch the watchers |
|
|
|
|
Settings → Notifications: configure ntfy/Telegram/etc. once and every
|
|
down-transition alerts you with an incident record.
|
|
|
|
**Possible switch fix, unproven:** cheap-switch hangs are often EEE
|
|
(energy-efficient ethernet) negotiation bugs. `ethtool --set-eee <nic> eee
|
|
off` on the host NICs is a reasonable mitigation. Failing that, the usual
|
|
suspects are heat and the power brick.
|
|
|
|
## 4. Files
|
|
|
|
```
|
|
infra/monitoring/compose.yaml stack definition
|
|
infra/monitoring/prometheus.yml scrape config (edit hostnames)
|
|
infra/monitoring/pve.yml.example PVE token template → copy to pve.yml
|
|
infra/monitoring/grafana/datasources.yaml Prometheus source, auto-provisioned
|
|
```
|
|
|
|
`pve.yml` holds a credential and is gitignored. Stack design rationale:
|
|
[`docs/specs/2026-10-03-external-monitoring.md`](specs/2026-10-03-external-monitoring.md).
|