fix: recover from 2026-07-12 power-event fallout + efficiency cleanup
- gitea: LevelDB queues -> local /var/lib/gitea-queues (stale NFS lock, 7,972-restart loop) - semaphore-db: data -> local /var/lib/semaphore-db (stale ibdata1 NFS lock, 975 restarts) - sonarr: config -> local /var/lib/sonarr (SQLite-on-NFS migration crash) - nzbhydra2: mem_limit 768M -> 1G (was pinned at limit, unhealthy) - jellyfin: deploy pending VAAPI device passthrough (i965/renderD128) - openvas: disable Watchtower (latest major pull forced pg_upgrade crash-loop) - onyx: fully decommissioned (down since ~June 8 unnoticed) - remove duplicate monitoring/traefik/guacamole stacks from .100, failover guacamole from bart - synapse: raise rc_message (alertmanager receiver was 429-limited; alerts silently dropped for weeks) - prometheus: UniFi probe 8443 -> 11443, solo.hobbs.farm via 401-tolerant module, drop onyx target - frigate restart cron: unwritable /var/log path meant it never ran; now 4x daily to ~glyptic - CLAUDE.md: document all of the above + AdGuard broken/unconfigured since 2026-05-17 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -31,31 +31,33 @@ hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, ha
|
|||||||
├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
|
├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
|
||||||
├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops])
|
├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops])
|
||||||
├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090)
|
├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090)
|
||||||
├── [COMPOSE] AdGuard Home (:3000, :53)
|
├── [COMPOSE] AdGuard Home (:3000, :53) -- **BROKEN since 2026-05-17**: conf dir empty, stuck in first-run wizard, DNS not answering. No backup found; needs manual reconfigure via :3000/install.html.
|
||||||
├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
|
├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
|
||||||
├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
|
├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
|
||||||
├── [COMPOSE] Farm Agent (:8889)
|
├── [COMPOSE] Farm Agent (:8889)
|
||||||
├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm.
|
├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm.
|
||||||
├── [COMPOSE] LinkedStorm (:8300 pipeline, :8310 api, cron no-port) -- whole-home audio assistant. VAD+Whisper STT+speaker-ID. Deployed 2026-05-28. Public: linkedstorm.hobbs.farm -> :8310.
|
├── [COMPOSE] LinkedStorm (:8300 pipeline, :8310 api, cron no-port) -- whole-home audio assistant. VAD+Whisper STT+speaker-ID. Deployed 2026-05-28. Public: linkedstorm.hobbs.farm -> :8310.
|
||||||
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
|
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
|
||||||
|
├── [COMPOSE] litellm (:4000?) + litellm-db -- LLM proxy (see Known Issue #15)
|
||||||
|
├── [COMPOSE] synapse-admin -- Synapse admin UI
|
||||||
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
|
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
|
||||||
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
|
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
|
||||||
├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (daily 4am)
|
├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (4x daily 4am/10am/4pm/10pm, log ~/frigate-restart.log)
|
||||||
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)
|
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)
|
||||||
|
|
||||||
bart (15GB RAM, i5-4570T, HDD)
|
bart (15GB RAM, i5-4570T, HDD)
|
||||||
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr)
|
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr) -- Sonarr config on LOCAL /var/lib/sonarr (2026-07-13, SQLite-on-NFS crash)
|
||||||
├── [COMPOSE] Onyx (:3005) -- redeployed 2026-05-20, all services :latest, Watchtower disabled on all 10
|
├── [REMOVED] Onyx -- fully decommissioned 2026-07-13 (had been down since ~2026-06-08 unnoticed; containers, volumes, images removed)
|
||||||
├── [COMPOSE] OpenVAS (vulnerability scanner)
|
├── [COMPOSE] OpenVAS (vulnerability scanner) -- Watchtower disabled 2026-07-13 after :latest major pull forced pg_upgrade crash-loop
|
||||||
├── [COMPOSE] Farm Agent (:8889)
|
├── [COMPOSE] Farm Agent (:8889)
|
||||||
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
|
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
|
||||||
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
|
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
|
||||||
├── [COMPOSE] Watchtower
|
├── [COMPOSE] Watchtower
|
||||||
|
|
||||||
hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
|
hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
|
||||||
├── [COMPOSE] Unifi + unifi-db (MongoDB)
|
├── [COMPOSE] UniFi OS Server (container `unifi-os-server`, HTTPS on host :11443)
|
||||||
├── [COMPOSE] Gitea + gitea-db (PostgreSQL)
|
├── [COMPOSE] Gitea + gitea-db (PostgreSQL) -- LevelDB queues on LOCAL /var/lib/gitea-queues (2026-07-13, stale NFS lock caused 7,972-restart loop)
|
||||||
├── [COMPOSE] Semaphore + semaphore-db (MySQL)
|
├── [COMPOSE] Semaphore + semaphore-db (MySQL, LOCAL /var/lib/semaphore-db since 2026-07-13)
|
||||||
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
|
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
|
||||||
├── [COMPOSE] Vaultwarden, Actual Budget, MCPO, Trivy
|
├── [COMPOSE] Vaultwarden, Actual Budget, MCPO, Trivy
|
||||||
├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- workflow automation
|
├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- workflow automation
|
||||||
@@ -175,7 +177,7 @@ compose/ # Standalone Compose definitions (deployed per-node)
|
|||||||
|
|
||||||
**hf-pdocker-02 (192.168.86.100):**
|
**hf-pdocker-02 (192.168.86.100):**
|
||||||
- Farm Agent: :8889
|
- Farm Agent: :8889
|
||||||
- Unifi: :8443
|
- UniFi OS Server: :11443 (HTTPS, container 443)
|
||||||
- Gitea: :3003 (web), :2222 (SSH)
|
- Gitea: :3003 (web), :2222 (SSH)
|
||||||
- Semaphore: :3004
|
- Semaphore: :3004
|
||||||
- Wiki.js: :3002
|
- Wiki.js: :3002
|
||||||
@@ -299,7 +301,12 @@ All persistent data on NAS at `/mnt/docker-data/`:
|
|||||||
9. **Guacamole schema**: App image pinned to `guacamole/guacamole:1.6.0` + `guacd:1.6.0` with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset.
|
9. **Guacamole schema**: App image pinned to `guacamole/guacamole:1.6.0` + `guacd:1.6.0` with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset.
|
||||||
10. **Gitea image**: Pinned to `gitea/gitea:1.26.1` with Watchtower disabled (2026-05-20). Watchtower previously pulled broken `:latest` images causing missing git binary errors.
|
10. **Gitea image**: Pinned to `gitea/gitea:1.26.1` with Watchtower disabled (2026-05-20). Watchtower previously pulled broken `:latest` images causing missing git binary errors.
|
||||||
11. **Home Assistant reverse proxy**: Config at `/mnt/docker-data/configs/homeassistant/configuration.yaml` (no hyphen). Must have `http: use_x_forwarded_for: true` + `trusted_proxies: [172.16.0.0/12, 127.0.0.1]` or proxied requests return 400.
|
11. **Home Assistant reverse proxy**: Config at `/mnt/docker-data/configs/homeassistant/configuration.yaml` (no hyphen). Must have `http: use_x_forwarded_for: true` + `trusted_proxies: [172.16.0.0/12, 127.0.0.1]` or proxied requests return 400.
|
||||||
12. **hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11)**: Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) fully removed (replaced by FRIDAY Voice on .150). Frigate is the main consumer. **2026-07-11:** Frigate's built-in AI features (`semantic_search: large`, `face_recognition`, `lpr`, `classification.bird`) were found *enabled* (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config `/mnt/docker-data/configs/frigate/config/config.yml` (backup `.bak.20260711-aitrim`). Recognition is instead handled externally by clip-service + frigate-notify v2. NOTE: recording ffmpeg processes still leak 4-14GB over hours (daily 4am restart-frigate cron mitigates); `memswap_limit` is 16G.
|
12. **hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11)**: Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) fully removed (replaced by FRIDAY Voice on .150). Frigate is the main consumer. **2026-07-11:** Frigate's built-in AI features (`semantic_search: large`, `face_recognition`, `lpr`, `classification.bird`) were found *enabled* (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config `/mnt/docker-data/configs/frigate/config/config.yml` (backup `.bak.20260711-aitrim`). Recognition is instead handled externally by clip-service + frigate-notify v2. **2026-07-12 — WHOLE-NODE CRASH:** .192 froze hard (~14:15, sshd/HA/DNS all dead, required hard reboot). Cause: Frigate `memswap_limit` had drifted to **16G** vs `mem_limit` 12G, letting it spill ~4GB into host swap; with the ffmpeg leak pinning it at 12G all day (cgroup OOM every ~45min), the host swap-thrashed to death. **Fix: node compose `memswap_limit: 12G` (= mem_limit) so Frigate can never use host swap** — a hit now trims its own ffmpeg (contained cgroup OOM), not the host. Repo already had 12G; node had drifted (never redeployed). NOTE: recording ffmpeg still leaks ~0.8GB/h atop ~6GB baseline (fills 12G in ~7.5h); twice-daily 4am+4pm restart-frigate cron mitigates but doesn't fully beat it — consider every-6h restart + fixing broken cameras (gate_camera, garage_2). Always keep `memswap_limit == mem_limit` on this node.
|
||||||
13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`.
|
13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`.
|
||||||
14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`.
|
14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`.
|
||||||
15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11).
|
15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11).
|
||||||
|
16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr` on bart). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.**
|
||||||
|
17. **Synapse rate-limited Alertmanager notifications**: default `rc_message` (0.2 msg/s) made every matrix-alertmanager-receiver send fail with M_LIMIT_EXCEEDED 429 — **no alerts reached Matrix for weeks** while outages accumulated (Onyx down 5 weeks, AdGuard dead 2 months, unnoticed). Fixed 2026-07-13: `rc_message: {per_second: 5, burst_count: 50}` in homeserver.yaml (backup `.bak-20260713-ratelimit`).
|
||||||
|
18. **AdGuard Home unconfigured since 2026-05-17**: its conf dir (`/mnt/docker-data/configs/adguardhome/conf`) is empty — container runs the first-launch wizard and serves NO DNS. No AdGuardHome.yaml backup exists anywhere. LAN DNS has been coming from elsewhere (router) since May. Needs manual re-setup at http://192.168.86.192:3000.
|
||||||
|
19. **Duplicate failover stacks removed from .100 and bart (2026-07-13)**: .100 ran a second monitoring stack (prometheus/grafana/alertmanager/influxdb/blackbox), an idle traefik (shared acme.json with .192 — ACME conflict risk), and a stray guacamole; bart ran a failover guacamole. All removed (~2GB RAM reclaimed). Compose dirs kept for reference; data volumes untouched.
|
||||||
|
20. **UniFi OS Server probe port**: `unifi-os-server` on .100 serves HTTPS on host port **11443** (container 443), not 8443. Blackbox probe fixed 2026-07-13. solo.hobbs.farm returns 401 (Traefik basicAuth) when healthy — probed via `http_2xx_or_401` module.
|
||||||
|
|||||||
@@ -6,11 +6,17 @@ services:
|
|||||||
environment:
|
environment:
|
||||||
- PUID=1000
|
- PUID=1000
|
||||||
- PGID=1000
|
- PGID=1000
|
||||||
|
- LIBVA_DRIVER_NAME=i965 # Haswell HD 4600 uses legacy i965 (iHD is Broadwell+)
|
||||||
volumes:
|
volumes:
|
||||||
- /mnt/docker-data/configs/jellyfin:/config
|
- /mnt/docker-data/configs/jellyfin:/config
|
||||||
- /mnt/nas-media:/media
|
- /mnt/nas-media:/media
|
||||||
ports:
|
ports:
|
||||||
- "8096:8096"
|
- "8096:8096"
|
||||||
|
devices:
|
||||||
|
- /dev/dri:/dev/dri # Intel iGPU for VAAPI hardware transcoding
|
||||||
|
group_add:
|
||||||
|
- "993" # render
|
||||||
|
- "44" # video
|
||||||
mem_limit: 2G
|
mem_limit: 2G
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "curl -f http://localhost:8096/health || exit 1"]
|
test: ["CMD-SHELL", "curl -f http://localhost:8096/health || exit 1"]
|
||||||
@@ -33,7 +39,9 @@ services:
|
|||||||
- PGID=1000
|
- PGID=1000
|
||||||
- TZ=America/Chicago
|
- TZ=America/Chicago
|
||||||
volumes:
|
volumes:
|
||||||
- /mnt/docker-data/configs/sonarr:/config
|
# LOCAL disk, not NFS: SQLite-on-NFS locking crashed Sonarr's DB migration
|
||||||
|
# after the 2026-07-12 power event ("waiting for user intervention")
|
||||||
|
- /var/lib/sonarr:/config
|
||||||
- /mnt/nas-media:/media
|
- /mnt/nas-media:/media
|
||||||
ports:
|
ports:
|
||||||
- "8989:8989"
|
- "8989:8989"
|
||||||
@@ -114,7 +122,7 @@ services:
|
|||||||
- /mnt/docker-data/configs/nzbhydra:/config
|
- /mnt/docker-data/configs/nzbhydra:/config
|
||||||
ports:
|
ports:
|
||||||
- "5076:5076"
|
- "5076:5076"
|
||||||
mem_limit: 768M
|
mem_limit: 1G
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "curl -f http://localhost:5076/ || exit 1"]
|
test: ["CMD-SHELL", "curl -f http://localhost:5076/ || exit 1"]
|
||||||
interval: 30s
|
interval: 30s
|
||||||
|
|||||||
@@ -1,231 +0,0 @@
|
|||||||
services:
|
|
||||||
api_server:
|
|
||||||
image: onyxdotapp/onyx-backend:latest
|
|
||||||
command: >
|
|
||||||
/bin/sh -c "alembic upgrade head &&
|
|
||||||
echo Starting Onyx API Server &&
|
|
||||||
uvicorn onyx.main:app --host 0.0.0.0 --port 8080"
|
|
||||||
depends_on:
|
|
||||||
relational_db:
|
|
||||||
condition: service_healthy
|
|
||||||
index:
|
|
||||||
condition: service_healthy
|
|
||||||
cache:
|
|
||||||
condition: service_healthy
|
|
||||||
restart: "no"
|
|
||||||
mem_limit: 2G
|
|
||||||
memswap_limit: 2G
|
|
||||||
env_file:
|
|
||||||
- .env
|
|
||||||
environment:
|
|
||||||
- AUTH_TYPE=basic
|
|
||||||
- DISABLE_TELEMETRY=true
|
|
||||||
- POSTGRES_HOST=relational_db
|
|
||||||
- VESPA_HOST=index
|
|
||||||
- REDIS_HOST=cache
|
|
||||||
- MODEL_SERVER_HOST=inference_model_server
|
|
||||||
- INDEXING_MODEL_SERVER_HOST=indexing_model_server
|
|
||||||
- OPENSEARCH_HOST=opensearch
|
|
||||||
- ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false
|
|
||||||
- FILE_STORE_BACKEND=postgres
|
|
||||||
extra_hosts:
|
|
||||||
- "host.docker.internal:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- onyx_file_connector:/home/file_connector_storage
|
|
||||||
networks:
|
|
||||||
- onyx
|
|
||||||
labels:
|
|
||||||
com.centurylinklabs.watchtower.enable: "false"
|
|
||||||
|
|
||||||
background:
|
|
||||||
image: onyxdotapp/onyx-backend:latest
|
|
||||||
command: >
|
|
||||||
/bin/sh -c "/usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf"
|
|
||||||
depends_on:
|
|
||||||
relational_db:
|
|
||||||
condition: service_healthy
|
|
||||||
index:
|
|
||||||
condition: service_healthy
|
|
||||||
cache:
|
|
||||||
condition: service_healthy
|
|
||||||
restart: "no"
|
|
||||||
mem_limit: 3G
|
|
||||||
memswap_limit: 3G
|
|
||||||
env_file:
|
|
||||||
- .env
|
|
||||||
environment:
|
|
||||||
- AUTH_TYPE=basic
|
|
||||||
- DISABLE_TELEMETRY=true
|
|
||||||
- POSTGRES_HOST=relational_db
|
|
||||||
- VESPA_HOST=index
|
|
||||||
- REDIS_HOST=cache
|
|
||||||
- MODEL_SERVER_HOST=inference_model_server
|
|
||||||
- INDEXING_MODEL_SERVER_HOST=indexing_model_server
|
|
||||||
- OPENSEARCH_HOST=opensearch
|
|
||||||
- ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false
|
|
||||||
- FILE_STORE_BACKEND=postgres
|
|
||||||
extra_hosts:
|
|
||||||
- "host.docker.internal:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- onyx_file_connector:/home/file_connector_storage
|
|
||||||
networks:
|
|
||||||
- onyx
|
|
||||||
labels:
|
|
||||||
com.centurylinklabs.watchtower.enable: "false"
|
|
||||||
|
|
||||||
web_server:
|
|
||||||
image: onyxdotapp/onyx-web-server:latest
|
|
||||||
depends_on:
|
|
||||||
- api_server
|
|
||||||
restart: "no"
|
|
||||||
mem_limit: 512M
|
|
||||||
memswap_limit: 512M
|
|
||||||
environment:
|
|
||||||
- INTERNAL_URL=http://api_server:8080
|
|
||||||
- WEB_DOMAIN=https://onyx.hobbs.farm
|
|
||||||
- NEXT_PUBLIC_DISABLE_STREAMING=false
|
|
||||||
- AUTH_TYPE=basic
|
|
||||||
networks:
|
|
||||||
- onyx
|
|
||||||
labels:
|
|
||||||
com.centurylinklabs.watchtower.enable: "false"
|
|
||||||
|
|
||||||
inference_model_server:
|
|
||||||
image: onyxdotapp/onyx-model-server:latest
|
|
||||||
command: >
|
|
||||||
/bin/sh -c "exec uvicorn model_server.main:app --host 0.0.0.0 --port 9000"
|
|
||||||
restart: "no"
|
|
||||||
mem_limit: 2G
|
|
||||||
memswap_limit: 2G
|
|
||||||
environment:
|
|
||||||
- MIN_THREADS_ML_MODELS=1
|
|
||||||
- LOG_LEVEL=info
|
|
||||||
volumes:
|
|
||||||
- onyx_model_cache_huggingface:/root/.cache/huggingface/
|
|
||||||
networks:
|
|
||||||
- onyx
|
|
||||||
labels:
|
|
||||||
com.centurylinklabs.watchtower.enable: "false"
|
|
||||||
|
|
||||||
indexing_model_server:
|
|
||||||
image: onyxdotapp/onyx-model-server:latest
|
|
||||||
command: >
|
|
||||||
/bin/sh -c "exec uvicorn model_server.main:app --host 0.0.0.0 --port 9000"
|
|
||||||
restart: "no"
|
|
||||||
mem_limit: 2G
|
|
||||||
memswap_limit: 2G
|
|
||||||
environment:
|
|
||||||
- MIN_THREADS_ML_MODELS=1
|
|
||||||
- INDEXING_ONLY=true
|
|
||||||
- LOG_LEVEL=info
|
|
||||||
volumes:
|
|
||||||
- onyx_model_cache_huggingface:/root/.cache/huggingface/
|
|
||||||
networks:
|
|
||||||
- onyx
|
|
||||||
labels:
|
|
||||||
com.centurylinklabs.watchtower.enable: "false"
|
|
||||||
|
|
||||||
relational_db:
|
|
||||||
image: postgres:15.2-alpine
|
|
||||||
restart: "no"
|
|
||||||
mem_limit: 512M
|
|
||||||
memswap_limit: 512M
|
|
||||||
environment:
|
|
||||||
- POSTGRES_USER=postgres
|
|
||||||
- POSTGRES_PASSWORD=onyx_postgres_pw
|
|
||||||
volumes:
|
|
||||||
- onyx_db:/var/lib/postgresql/data
|
|
||||||
networks:
|
|
||||||
- onyx
|
|
||||||
healthcheck:
|
|
||||||
test: ["CMD-SHELL", "pg_isready -U postgres"]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 5
|
|
||||||
labels:
|
|
||||||
com.centurylinklabs.watchtower.enable: "false"
|
|
||||||
|
|
||||||
index:
|
|
||||||
image: vespaengine/vespa:8.609.39
|
|
||||||
restart: "no"
|
|
||||||
mem_limit: 4G
|
|
||||||
memswap_limit: 4G
|
|
||||||
volumes:
|
|
||||||
- onyx_vespa:/opt/vespa/var
|
|
||||||
networks:
|
|
||||||
- onyx
|
|
||||||
healthcheck:
|
|
||||||
test: ["CMD-SHELL", "curl -sf http://localhost:19071/state/v1/health || exit 1"]
|
|
||||||
interval: 30s
|
|
||||||
timeout: 10s
|
|
||||||
retries: 10
|
|
||||||
start_period: 120s
|
|
||||||
labels:
|
|
||||||
com.centurylinklabs.watchtower.enable: "false"
|
|
||||||
|
|
||||||
opensearch:
|
|
||||||
image: opensearchproject/opensearch:2.11.1
|
|
||||||
restart: "no"
|
|
||||||
mem_limit: 2G
|
|
||||||
memswap_limit: 2G
|
|
||||||
environment:
|
|
||||||
- discovery.type=single-node
|
|
||||||
- plugins.security.disabled=true
|
|
||||||
- OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g
|
|
||||||
- DISABLE_INSTALL_DEMO_CONFIG=true
|
|
||||||
volumes:
|
|
||||||
- onyx_opensearch:/usr/share/opensearch/data
|
|
||||||
networks:
|
|
||||||
- onyx
|
|
||||||
healthcheck:
|
|
||||||
test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health || exit 1"]
|
|
||||||
interval: 30s
|
|
||||||
timeout: 10s
|
|
||||||
retries: 10
|
|
||||||
start_period: 120s
|
|
||||||
labels:
|
|
||||||
com.centurylinklabs.watchtower.enable: "false"
|
|
||||||
|
|
||||||
cache:
|
|
||||||
image: redis:7.4-alpine
|
|
||||||
restart: "no"
|
|
||||||
mem_limit: 256M
|
|
||||||
memswap_limit: 256M
|
|
||||||
command: redis-server --maxmemory 128mb --maxmemory-policy allkeys-lru
|
|
||||||
networks:
|
|
||||||
- onyx
|
|
||||||
healthcheck:
|
|
||||||
test: ["CMD", "redis-cli", "ping"]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 5
|
|
||||||
labels:
|
|
||||||
com.centurylinklabs.watchtower.enable: "false"
|
|
||||||
|
|
||||||
nginx:
|
|
||||||
image: nginx:1.25.5-alpine
|
|
||||||
restart: "no"
|
|
||||||
mem_limit: 128M
|
|
||||||
memswap_limit: 128M
|
|
||||||
depends_on:
|
|
||||||
- api_server
|
|
||||||
- web_server
|
|
||||||
ports:
|
|
||||||
- "3005:80"
|
|
||||||
volumes:
|
|
||||||
- ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
|
|
||||||
networks:
|
|
||||||
- onyx
|
|
||||||
labels:
|
|
||||||
com.centurylinklabs.watchtower.enable: "false"
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
onyx_file_connector:
|
|
||||||
onyx_model_cache_huggingface:
|
|
||||||
onyx_db:
|
|
||||||
onyx_vespa:
|
|
||||||
onyx_opensearch:
|
|
||||||
|
|
||||||
networks:
|
|
||||||
onyx:
|
|
||||||
driver: bridge
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
server {
|
|
||||||
listen 80;
|
|
||||||
server_name _;
|
|
||||||
|
|
||||||
# Use Docker's internal DNS with a short TTL so container IP changes
|
|
||||||
# (e.g. after docker compose up -d for a single service) are picked up
|
|
||||||
# without requiring an nginx reload.
|
|
||||||
resolver 127.0.0.11 valid=10s;
|
|
||||||
|
|
||||||
client_max_body_size 100M;
|
|
||||||
|
|
||||||
location /api/ {
|
|
||||||
set $api_upstream http://api_server:8080;
|
|
||||||
rewrite ^/api/(.*)$ /$1 break;
|
|
||||||
proxy_pass $api_upstream;
|
|
||||||
proxy_set_header Host $host;
|
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
||||||
proxy_set_header X-Forwarded-Proto $scheme;
|
|
||||||
proxy_read_timeout 300s;
|
|
||||||
proxy_connect_timeout 75s;
|
|
||||||
proxy_buffering off;
|
|
||||||
proxy_cache off;
|
|
||||||
}
|
|
||||||
|
|
||||||
location / {
|
|
||||||
set $web_upstream http://web_server:3000;
|
|
||||||
proxy_pass $web_upstream;
|
|
||||||
proxy_set_header Host $host;
|
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
||||||
proxy_set_header X-Forwarded-Proto $scheme;
|
|
||||||
proxy_read_timeout 300s;
|
|
||||||
proxy_connect_timeout 75s;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -3,6 +3,10 @@ services:
|
|||||||
image: immauss/openvas:latest
|
image: immauss/openvas:latest
|
||||||
container_name: openvas
|
container_name: openvas
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
labels:
|
||||||
|
# Watchtower pulled a new major that forced a pg_upgrade crash-loop
|
||||||
|
# (212 restarts, 2026-07-13); update OpenVAS deliberately instead
|
||||||
|
- "com.centurylinklabs.watchtower.enable=false"
|
||||||
environment:
|
environment:
|
||||||
- PASSWORD=${OPENVAS_PASSWORD}
|
- PASSWORD=${OPENVAS_PASSWORD}
|
||||||
- USERNAME=${OPENVAS_USERNAME}
|
- USERNAME=${OPENVAS_USERNAME}
|
||||||
|
|||||||
@@ -19,6 +19,9 @@ services:
|
|||||||
- GITEA__server__SSH_PORT=2222
|
- GITEA__server__SSH_PORT=2222
|
||||||
volumes:
|
volumes:
|
||||||
- /mnt/docker-data/configs/gitea:/data
|
- /mnt/docker-data/configs/gitea:/data
|
||||||
|
# LOCAL disk for LevelDB queues: stale NFS lock after 2026-07-12 power event
|
||||||
|
# caused a 28h crash loop (7,972 restarts)
|
||||||
|
- /var/lib/gitea-queues:/data/gitea/queues
|
||||||
- /etc/timezone:/etc/timezone:ro
|
- /etc/timezone:/etc/timezone:ro
|
||||||
- /etc/localtime:/etc/localtime:ro
|
- /etc/localtime:/etc/localtime:ro
|
||||||
depends_on:
|
depends_on:
|
||||||
|
|||||||
@@ -53,7 +53,9 @@ services:
|
|||||||
- MYSQL_USER=semaphore
|
- MYSQL_USER=semaphore
|
||||||
- MYSQL_PASSWORD=${SEMAPHORE_DB_PASSWORD}
|
- MYSQL_PASSWORD=${SEMAPHORE_DB_PASSWORD}
|
||||||
volumes:
|
volumes:
|
||||||
- /mnt/docker-data/databases/semaphore-db:/var/lib/mysql
|
# LOCAL disk, not NFS: MySQL on NFS suffers stale-lock and InnoDB corruption
|
||||||
|
# (2026-07-11 corruption, 2026-07-13 stale ibdata1 lock after power event)
|
||||||
|
- /var/lib/semaphore-db:/var/lib/mysql
|
||||||
networks:
|
networks:
|
||||||
- semaphore-internal
|
- semaphore-internal
|
||||||
labels:
|
labels:
|
||||||
|
|||||||
Reference in New Issue
Block a user