docs: matrix auth inventory and MAS QR sign-in plan

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 14:57:55 -05:00
parent bfb9b03a26
commit 986f219135
2 changed files with 709 additions and 0 deletions
@@ -0,0 +1,668 @@
 # QR Sign-In via Matrix Authentication Service — Implementation Plan
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Enable "sign in with QR code" (MSC4108) for Matrix accounts on hobbs.farm, which requires deploying Matrix Authentication Service (MAS) as the homeserver's auth provider.
 **Architecture:** QR sign-in is an OAuth 2.0 device grant, not a password login. Synapse 1.159.0 refuses to start with `msc4108_enabled: true` unless `matrix_authentication_service.enabled` is also true. MAS requires PostgreSQL, and — critically — **`syn2mas` can only read a Synapse whose own database is PostgreSQL**. Synapse here is on SQLite, so this is a three-stage chain: port Synapse to PostgreSQL, deploy MAS and migrate the accounts into it, then turn on the QR MSCs.
 **Tech Stack:** Synapse 1.159.0 (SQLite → PostgreSQL 16), Matrix Authentication Service (Docker, ghcr.io/element-hq), PostgreSQL 16-alpine ×2, Traefik, Cloudflare DNS, Docker Compose on hf-pdocker-01.
 ## Why this is three migrations, not a config flag
 Verified in the running image's own source (`synapse/config/experimental.py`):
 ```python
 if (self.msc4108_enabled or self.msc4108_delegation_endpoint is not None) and not auth_delegated:
    raise ConfigError("MSC4108 requires matrix_authentication_service to be enabled", ...)
 ```
 The same guard exists for MSC4388 (the QR out-of-band channel). And from the MAS docs:
 > "It is worth noting that MAS currently only supports PostgreSQL as a database backend."
 > "The migration tool only supports reading from PostgreSQL for the Synapse database as well."
 Synapse on this node currently runs `database: {name: sqlite3, database: /data/homeserver.db}`.
 ## Global Constraints
 - Synapse image stays pinned at `ghcr.io/element-hq/synapse:v1.159.0` (**`v` prefix required**; `:1.159.0` 404s). Do not bump during this work.
 - Synapse ≥1.143 requires **PostgreSQL 14+**. Use `postgres:16-alpine`.
 - Synapse's PostgreSQL database MUST be created with `ENCODING 'UTF8' LC_COLLATE 'C' LC_CTYPE 'C'` — Synapse refuses any other collation.
 - MAS requires its **own separate, blank** PostgreSQL database. Do not share Synapse's.
 - All lock-prone databases go on **LOCAL disk**, never NFS (CLAUDE.md Known Issue #16). Use `/var/lib/synapse-db` and `/var/lib/mas-db`.
 - On hf-pdocker-01, **`memswap_limit` must equal `mem_limit`** on every container (Known Issue #12 — a drift here crashed the whole node on 2026-07-12).
 - Node headroom: ~8GB available of 23GB, Frigate is the main consumer. Budget ≤600MB total for the new containers.
 - **hobbs.farm has no wildcard DNS record** (memory `hobbs-farm-dns`). Traefik uses DNS-01, so a certificate will be issued for a hostname with **no A record** — a valid cert does NOT mean the name resolves. Any new public hostname needs an explicit Cloudflare A record → `98.160.103.126`, DNS-only (grey cloud).
 - Watchtower does not run on .192; pin every image tag explicitly.
 - Synapse data lives at `/var/lib/synapse` on local NVMe. The NFS copy at `/mnt/docker-data/configs/matrix/data` is a stale backup — **never remount it**.
 - The mandatory post-change checklist in CLAUDE.md applies (CLAUDE.md, MEMORY.md, Prometheus probes, wiki, Farm Manager links).
 ## Blast Radius
 Authentication for **all 19 accounts** moves to MAS. Known consumers that must keep working:
 | Account | How it authenticates | Migrates via |
 |---|---|---|
 | `@jeremy` | password `bigdog01` + 5 Element devices (2× Element X Android, 2× Element Desktop) | password hash + devices |
 | `@frigate` | long-lived access token `syt_ZnJpZ2F0…` in `frigate-notify/.env` | access token |
 | `@alertbot` | long-lived access token `syt_YWxlcnRib3Q…` in `matrix-receiver-config.yml` | access token |
 | `@margaret` | openclaw.json contains **both** `accessToken` and `password` | both |
 | `@spirit` | openclaw config (path differs from margaret's — resolve in Task 1) | TBD in Task 1 |
 | `@clippyauto` | n8n credential on .100, polls `/sync` every ~2s | TBD in Task 1 |
 | `@fry`, `@hermes`, `@julie`, `@pluto`, `@testbot`, `@admin1`, `@adminuser` | unknown | TBD in Task 1 |
 MAS exposes a compatibility layer proxying `/_matrix/client/*/login`, so password-based bots are expected to keep working — **this is the single biggest assumption in the plan and Task 1 exists to test it, not trust it.**
 ## File Structure
 - Create `compose/hf-pdocker-01/mas/docker-compose.yml` — MAS + `mas-db` PostgreSQL. One file, one responsibility: authentication.
 - Modify `compose/hf-pdocker-01/matrix/docker-compose.yml` — re-add a `synapse-db` service (PostgreSQL 16, local disk). Note this partly reverses the 2026-08-20 removal of the *empty* postgres; this one will actually hold data.
 - Create `/var/lib/synapse/homeserver-postgres.yaml` on the node — port-target config for `synapse_port_db` (node-only, not in repo — contains DB credentials).
 - Create `/mnt/docker-data/configs/mas/config.yaml` on the node — MAS config (node-only, contains signing keys and secrets; back up to Vaultwarden).
 - Modify `/mnt/docker-data/traefik/dynamic/temp-routes.yml` — add the `auth.hobbs.farm` router + service. (Yes, the Matrix, Home Assistant and Guacamole routes all live in a file called `temp-routes.yml`. It is load-bearing despite the name.)
 - Modify `CLAUDE.md`, `MEMORY.md` — post-change checklist.
 ---
 ### Task 1: Auth inventory and full backup
 **Files:**
 - Create: `/var/lib/synapse.bak-premas/` (node), `docs/superpowers/plans/mas-auth-inventory.md` (repo)
 **Interfaces:**
 - Produces: a complete table of every account → auth method → config file location, used by Tasks 7 and 10 to verify nothing broke.
 - [ ] **Step 1: Enumerate every device and its last-seen IP, to know what "still working" means later**
 ```bash
 ssh glyptic@192.168.86.192 'docker exec synapse python3 -c "
 import sqlite3
 c=sqlite3.connect(\"file:/var/lib/synapse/homeserver.db?mode=ro\",uri=True)
 for u,d,n,ip in c.execute(\"select user_id,device_id,display_name,last_seen_ip from devices order by user_id\"):
    print(u,d,n,ip)
 "'
 ```
 Expected: rows for `@jeremy` (5 devices), plus bot devices for margaret/spirit/clippyauto/frigate/alertbot/fry.
 - [ ] **Step 2: Locate spirit's and clippyauto's credentials (unresolved in planning)**
 ```bash
 ssh glyptic@192.168.86.192 'docker inspect spirit --format "{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}"; docker exec spirit sh -c "ls -la /home/glyptic/.openclaw/ 2>/dev/null || find / -name openclaw.json -maxdepth 6 2>/dev/null | head"'
 ssh glyptic@192.168.86.100 'docker exec n8n sh -c "printenv | grep -i matrix" 2>/dev/null; echo "---"; docker exec n8n-db psql -U n8n -tc "select name, type from credentials_entity;" 2>/dev/null | head'
 ```
 Expected: a config path for spirit and a credential type for n8n. Record whether each uses an **access token** or a **password** — access tokens migrate untouched; passwords depend on the MAS compat layer.
 - [ ] **Step 3: Write the inventory file**
 Record one row per account: user id, auth method, exact file path holding the credential, and how to verify it post-migration. Save as `docs/superpowers/plans/mas-auth-inventory.md`.
 - [ ] **Step 4: Take a cold, complete backup**
 ```bash
 ssh glyptic@192.168.86.192 '
 cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose stop synapse
 sudo rsync -a /var/lib/synapse/ /var/lib/synapse.bak-premas/
 sudo python3 -c "
 import sqlite3;c=sqlite3.connect(\"/var/lib/synapse.bak-premas/homeserver.db\")
 print(\"integrity:\", c.execute(\"pragma integrity_check\").fetchone()[0])
 print(\"users:\", c.execute(\"select count(*) from users\").fetchone()[0])
 print(\"events:\", c.execute(\"select count(*) from events\").fetchone()[0])
 "
 docker compose start synapse'
 ```
 Expected: `integrity: ok`, 19 users, ~93,884 events. **Do not proceed if integrity is anything but `ok`.**
 - [ ] **Step 5: Commit the inventory**
 ```bash
 git add docs/superpowers/plans/mas-auth-inventory.md
 git commit -m "docs: matrix auth inventory ahead of MAS migration"
 ```
 ---
 ### Task 2: PostgreSQL for Synapse (blank, correct collation)
 **Files:**
 - Modify: `compose/hf-pdocker-01/matrix/docker-compose.yml`
 **Interfaces:**
 - Produces: `synapse-db` reachable at `synapse-db:5432` on the `matrix-internal` network, database `synapse`, user `synapse`, C collation. Task 3 ports data into it.
 - [ ] **Step 1: Add the service to the compose file**
 ```yaml
  synapse-db:
    image: postgres:16-alpine
    container_name: synapse-db
    restart: unless-stopped
    environment:
      - POSTGRES_USER=synapse
      - POSTGRES_PASSWORD=<generate: openssl rand -base64 24>
      - POSTGRES_DB=synapse
      - POSTGRES_INITDB_ARGS=--encoding=UTF-8 --lc-collate=C --lc-ctype=C
    volumes:
      - /var/lib/synapse-db:/var/lib/postgresql/data   # LOCAL disk - Known Issue #16
    networks:
      - matrix-internal
    mem_limit: 512M
    memswap_limit: 512M          # MUST equal mem_limit - Known Issue #12
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U synapse -d synapse"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
    logging:
      driver: "json-file"
      options: {max-size: "50m", max-file: "3"}
 ```
 Note: **no `user: "1024:100"`** — that workaround is only for NFS root_squash. This volume is local.
 - [ ] **Step 2: Deploy and verify collation**
 ```bash
 scp compose/hf-pdocker-01/matrix/docker-compose.yml glyptic@192.168.86.192:/mnt/docker-data/compose/hf-pdocker-01/matrix/
 ssh glyptic@192.168.86.192 'cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose up -d synapse-db && sleep 20 && docker exec synapse-db psql -U synapse -d synapse -c "show lc_collate;"'
 ```
 Expected: `lc_collate` = `C`. **If it says anything else, delete `/var/lib/synapse-db` and redo** — Synapse will refuse to start otherwise, and fixing it later means re-porting.
 - [ ] **Step 3: Commit**
 ```bash
 git add compose/hf-pdocker-01/matrix/docker-compose.yml
 git commit -m "feat: add postgres for synapse (local disk, C collation) ahead of MAS"
 ```
 ---
 ### Task 3: Port Synapse SQLite → PostgreSQL
 **Files:**
 - Create: `/var/lib/synapse/homeserver-postgres.yaml` (node only)
 - Modify: `/var/lib/synapse/homeserver.yaml` (node only)
 **Interfaces:**
 - Consumes: `synapse-db` from Task 2.
 - Produces: a Synapse running on PostgreSQL — the precondition `syn2mas` requires in Task 7.
 - [ ] **Step 1: Write the port-target config**
 Copy `/var/lib/synapse/homeserver.yaml` to `homeserver-postgres.yaml` and replace the `database:` block with:
 ```yaml
 database:
  name: psycopg2
  args:
    user: synapse
    password: <the password from Task 2>
    dbname: synapse
    host: synapse-db
    cp_min: 5
    cp_max: 10
 ```
 - [ ] **Step 2: Stop Synapse (downtime starts)**
 ```bash
 ssh glyptic@192.168.86.192 'cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose stop synapse'
 ```
 - [ ] **Step 3: Run the port**
 ```bash
 ssh glyptic@192.168.86.192 'docker run --rm --network matrix_matrix-internal \
  -v /var/lib/synapse:/data ghcr.io/element-hq/synapse:v1.159.0 \
  synapse_port_db --sqlite-database /data/homeserver.db --postgres-config /data/homeserver-postgres.yaml'
 ```
 Expected: a progress table per-table, ending in `Success!`. Budget 10–40 minutes for ~94k events on this hardware. If it fails partway, drop and recreate the database before retrying — **the port tool is not idempotent against a partially-filled database.**
 - [ ] **Step 4: Verify row counts match the source**
 ```bash
 ssh glyptic@192.168.86.192 'docker exec synapse-db psql -U synapse -d synapse -tc "select (select count(*) from users) as users, (select count(*) from events) as events;"'
 ```
 Expected: `19 | 93884` (or higher for events if the server ran between backup and port). **Must not be lower.**
 - [ ] **Step 5: Cut Synapse over**
 ```bash
 ssh glyptic@192.168.86.192 'sudo cp /var/lib/synapse/homeserver.yaml /var/lib/synapse/homeserver.yaml.bak-presql
 sudo cp /var/lib/synapse/homeserver-postgres.yaml /var/lib/synapse/homeserver.yaml
 cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose start synapse'
 ```
 - [ ] **Step 6: Verify the service on PostgreSQL**
 ```bash
 ssh glyptic@192.168.86.192 '
 for i in $(seq 1 30); do [ "$(docker inspect synapse --format "{{.State.Health.Status}}")" = healthy ] && echo healthy && break; sleep 10; done
 curl -s http://localhost:8008/_synapse/admin/v1/server_version; echo
 curl -s -X POST http://localhost:8008/_matrix/client/v3/login -H "Content-Type: application/json" \
  -d "{\"type\":\"m.login.password\",\"identifier\":{\"type\":\"m.id.user\",\"user\":\"jeremy\"},\"password\":\"bigdog01\"}" | head -c 80
 docker logs synapse --since 5m 2>&1 | grep -cE " ERROR "'
 ```
 Expected: `healthy`, version `1.159.0`, an `access_token` in the login response, `0` errors. Log out that verification session afterwards (`POST /_matrix/client/v3/logout`).
 - [ ] **Step 7: Confirm bots reconnected before declaring the task done**
 ```bash
 ssh glyptic@192.168.86.192 'sleep 60; docker logs synapse --since 60s 2>&1 | grep -oE "\{@[a-z0-9]+:hobbs.farm\}" | sort | uniq -c'
 ```
 Expected: clippyauto, margaret, spirit, fry all present. **Rollback if not:** restore `homeserver.yaml.bak-presql` and restart — SQLite file is untouched and still authoritative until Task 7.
 - [ ] **Step 8: Commit**
 ```bash
 git commit --allow-empty -m "chore: synapse ported from sqlite to postgres (node config, not in repo)"
 ```
 ---
 ### Task 4: PostgreSQL for MAS
 **Files:**
 - Create: `compose/hf-pdocker-01/mas/docker-compose.yml`
 **Interfaces:**
 - Produces: `mas-db:5432`, database `mas`, user `mas`, on a new `mas` bridge network that also carries the MAS container.
 - [ ] **Step 1: Create the compose file with the database service only**
 Same shape as Task 2 Step 1, but: `container_name: mas-db`, `POSTGRES_USER=mas`, `POSTGRES_DB=mas`, volume `/var/lib/mas-db:/var/lib/postgresql/data`, `mem_limit: 256M`, `memswap_limit: 256M`, network `mas`. Default collation is fine — the C-collation requirement is Synapse's, not MAS's.
 - [ ] **Step 2: Deploy and verify**
 ```bash
 ssh glyptic@192.168.86.192 'cd /mnt/docker-data/compose/hf-pdocker-01/mas && docker compose up -d mas-db && sleep 20 && docker exec mas-db psql -U mas -d mas -tc "select 1;"'
 ```
 Expected: `1`.
 - [ ] **Step 3: Commit**
 ```bash
 git add compose/hf-pdocker-01/mas/docker-compose.yml
 git commit -m "feat: add mas-db postgres for matrix authentication service"
 ```
 ---
 ### Task 5: Generate MAS config and deploy the service
 **Files:**
 - Create: `/mnt/docker-data/configs/mas/config.yaml` (node only — contains secrets)
 - Modify: `compose/hf-pdocker-01/mas/docker-compose.yml`
 **Interfaces:**
 - Consumes: `mas-db` from Task 4.
 - Produces: MAS listening on host port **8020** (verified free on .192: 8008/8010/8080/8082/8085/8086/8090/8123/8181/8199/8300/8310 are taken).
 - [ ] **Step 1: Generate a config seeded from the Synapse config**
 ```bash
 ssh glyptic@192.168.86.192 'sudo mkdir -p /mnt/docker-data/configs/mas && docker run --rm \
  -v /var/lib/synapse:/synapse:ro -v /mnt/docker-data/configs/mas:/out \
  ghcr.io/element-hq/matrix-authentication-service:latest \
  config generate --synapse-config /synapse/homeserver.yaml --output /out/config.yaml'
 ```
 Expected: `config.yaml` written, containing generated signing keys and a `matrix.secret` shared secret.
 - [ ] **Step 2: Set the database URI, public base, and password hashing scheme**
 Edit `/mnt/docker-data/configs/mas/config.yaml`:
 ```yaml
 database:
  uri: postgresql://mas:<password>@mas-db/mas
 http:
  public_base: https://auth.hobbs.farm
  listeners:
    - name: web
      resources: [{name: discovery}, {name: human}, {name: oauth}, {name: compat}, {name: graphql}, {name: assets}]
      binds: [{address: "[::]:8080"}]
 passwords:
  enabled: true
  schemes:
    - version: 1
      algorithm: bcrypt
      unicode_normalization: true    # REQUIRED for imported Synapse hashes
    - version: 2
      algorithm: argon2id
 ```
 The `unicode_normalization: true` on the bcrypt v1 scheme is what makes migrated Synapse password hashes verify. Without it, **every migrated password silently fails**.
 - [ ] **Step 3: Add the MAS service to compose**
 ```yaml
  mas:
    image: ghcr.io/element-hq/matrix-authentication-service:latest   # pin to the digest after first pull
    container_name: mas
    restart: unless-stopped
    command: ["server", "--config", "/config/config.yaml"]
    volumes:
      - /mnt/docker-data/configs/mas:/config:ro
    ports:
      - "8020:8080"
    networks: [mas, matrix-internal]
    depends_on:
      mas-db: {condition: service_healthy}
    mem_limit: 256M
    memswap_limit: 256M
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
 ```
 MAS needs to reach Synapse's admin API and Synapse needs to reach MAS, hence both networks.
 - [ ] **Step 4: Run migrations and start**
 ```bash
 ssh glyptic@192.168.86.192 'cd /mnt/docker-data/compose/hf-pdocker-01/mas && docker compose run --rm mas database migrate --config /config/config.yaml && docker compose up -d mas && sleep 15 && curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8020/health'
 ```
 Expected: migrations apply cleanly, health endpoint returns `200`.
 - [ ] **Step 5: Store the MAS secrets in Vaultwarden and commit**
 The generated `config.yaml` holds signing keys that cannot be regenerated without invalidating sessions. Store a copy in Vaultwarden (note: the vault master password is currently unknown — see memory `vault-master-password-stale`; resolve that first or keep an encrypted copy on the NAS).
 ```bash
 git add compose/hf-pdocker-01/mas/docker-compose.yml
 git commit -m "feat: deploy matrix authentication service on :8020"
 ```
 ---
 ### Task 6: syn2mas check and dry-run
 **Interfaces:**
 - Consumes: Synapse-on-PostgreSQL (Task 3), MAS + DB (Tasks 4–5).
 - Produces: a clean dry-run, the go/no-go gate for Task 7.
 - [ ] **Step 1: Run the checker**
 ```bash
 ssh glyptic@192.168.86.192 'docker exec mas mas-cli syn2mas check --config /config/config.yaml --synapse-config /synapse/homeserver.yaml'
 ```
 (Mount `/var/lib/synapse` read-only into the MAS container as `/synapse` for this — add it in Task 5 Step 3 if not already present.)
 Expected: no blocking findings. Known blockers it looks for: custom password provider modules, SAML/LDAP providers. This deployment has neither.
 - [ ] **Step 2: Dry run**
 ```bash
 ssh glyptic@192.168.86.192 'docker exec mas mas-cli syn2mas migrate --config /config/config.yaml --synapse-config /synapse/homeserver.yaml --dry-run'
 ```
 Expected: a summary of users, password hashes, devices, and access tokens it *would* import. Cross-check the user count against **19** and the device count against Task 1 Step 1. **Do not proceed to Task 7 if the counts disagree.**
 - [ ] **Step 3: Record the dry-run output in the inventory doc and commit**
 ```bash
 git add docs/superpowers/plans/mas-auth-inventory.md
 git commit -m "docs: syn2mas dry-run results"
 ```
 ---
 ### Task 7: Migrate accounts into MAS and delegate Synapse auth
 **Files:**
 - Modify: `/var/lib/synapse/homeserver.yaml` (node only)
 **Interfaces:**
 - Produces: Synapse delegating all authentication to MAS — the precondition MSC4108 requires.
 **This is the point of no return.** The MAS docs state migration "is not easily reversible and requires careful backup procedures before proceeding." Rollback means restoring both databases from backup.
 - [ ] **Step 1: Fresh backup of both databases immediately before migrating**
 ```bash
 ssh glyptic@192.168.86.192 '
 cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose stop synapse
 docker exec synapse-db pg_dump -U synapse synapse | gzip > /tmp/synapse-premas.sql.gz
 docker exec mas-db pg_dump -U mas mas | gzip > /tmp/mas-premigrate.sql.gz
 sudo mv /tmp/*.gz /var/lib/synapse.bak-premas/ && ls -la /var/lib/synapse.bak-premas/*.gz'
 ```
 Expected: two non-empty `.gz` files.
 - [ ] **Step 2: Run the migration (Synapse stays stopped)**
 ```bash
 ssh glyptic@192.168.86.192 'docker exec mas mas-cli syn2mas migrate --config /config/config.yaml --synapse-config /synapse/homeserver.yaml'
 ```
 Expected: completes with an import summary matching the dry-run.
 - [ ] **Step 3: Point Synapse at MAS**
 Add to `/var/lib/synapse/homeserver.yaml`:
 ```yaml
 matrix_authentication_service:
  enabled: true
  endpoint: http://mas:8080/
  secret: <the matrix.secret value from /mnt/docker-data/configs/mas/config.yaml>
 ```
 - [ ] **Step 4: Start Synapse and verify delegation is live**
 ```bash
 ssh glyptic@192.168.86.192 '
 cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose start synapse
 for i in $(seq 1 30); do [ "$(docker inspect synapse --format "{{.State.Health.Status}}")" = healthy ] && echo healthy && break; sleep 10; done
 curl -s -o /dev/null -w "auth_metadata: %{http_code}\n" http://localhost:8008/_matrix/client/unstable/org.matrix.msc2965/auth_metadata'
 ```
 Expected: `healthy`, and auth_metadata now returns **200** (it returns 404 today).
 - [ ] **Step 5: Verify every account from the Task 1 inventory still authenticates**
 ```bash
 ssh glyptic@192.168.86.192 '
 echo "--- password login via MAS compat layer:"
 curl -s -X POST http://localhost:8008/_matrix/client/v3/login -H "Content-Type: application/json" \
  -d "{\"type\":\"m.login.password\",\"identifier\":{\"type\":\"m.id.user\",\"user\":\"jeremy\"},\"password\":\"bigdog01\"}" | head -c 100; echo
 echo "--- migrated access token (frigate):"
 curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8008/_matrix/client/v3/account/whoami -H "Authorization: Bearer syt_ZnJpZ2F0<rest-from-.env>"
 echo "--- bots syncing:"
 sleep 90; docker logs synapse --since 90s 2>&1 | grep -oE "\{@[a-z0-9]+:hobbs.farm\}" | sort | uniq -c'
 ```
 Expected: an `access_token` for jeremy, `200` for the frigate token, and margaret/spirit/clippyauto/fry all syncing. **If password login fails, the bcrypt `unicode_normalization: true` setting from Task 5 Step 2 is the first thing to check.**
 - [ ] **Step 6: Commit**
 ```bash
 git commit --allow-empty -m "feat: synapse auth delegated to MAS (node config)"
 ```
 ---
 ### Task 8: Publish MAS and add DNS
 **Files:**
 - Modify: `/mnt/docker-data/traefik/dynamic/temp-routes.yml` (confirmed: this is the file holding the `mess`, `wellknown`, `yoda` and `solo` routers)
 - External: Cloudflare DNS
 - [ ] **Step 1: Add the Cloudflare A record FIRST**
 `auth.hobbs.farm` → `98.160.103.126`, **DNS-only (grey cloud)**. There is no wildcard record on this zone, and Traefik's DNS-01 challenge will happily issue a valid certificate for a hostname that does not resolve — so a working cert proves nothing here (memory `hobbs-farm-dns`).
 Verify from outside the LAN before continuing:
 ```bash
 getent hosts auth.hobbs.farm
 ```
 Expected: `98.160.103.126`.
 - [ ] **Step 2: Add the Traefik router and service**
 ```yaml
    auth:
      rule: "Host(`auth.hobbs.farm`)"
      service: auth
      entryPoints: [websecure]
      middlewares: [security-headers]
      tls: {certResolver: letsencrypt}
 ```
 ```yaml
    auth:
      loadBalancer:
        servers:
          - url: "http://192.168.86.192:8020"
 ```
 Do **not** add basicAuth — clients must reach this unauthenticated to log in.
 - [ ] **Step 3: Verify publicly**
 ```bash
 curl -s -o /dev/null -w "%{http_code}\n" https://auth.hobbs.farm/health
 curl -s https://mess.hobbs.farm/_matrix/client/unstable/org.matrix.msc2965/auth_metadata | head -c 300
 ```
 Expected: `200`, and auth metadata advertising `https://auth.hobbs.farm` endpoints.
 - [ ] **Step 4: Confirm `.well-known` needs no change (verify, do not assume)**
 Current contents, served from `/mnt/docker-data/configs/matrix/wellknown/` via `matrix-wellknown` on :8010:
 ```
 client -> {"m.homeserver": {"base_url": "https://mess.hobbs.farm"}}
 server -> {"m.server": "mess.hobbs.farm:443"}
 ```
 Under MAS, clients discover the auth server from Synapse's `auth_metadata` endpoint, not from `.well-known`, so **both files should stay as they are**. Verify a client can complete discovery end to end:
 ```bash
 curl -s https://hobbs.farm/.well-known/matrix/client
 curl -s https://mess.hobbs.farm/_matrix/client/unstable/org.matrix.msc2965/auth_metadata | head -c 200
 ```
 Expected: `.well-known` unchanged and still pointing at `mess.hobbs.farm`; auth_metadata returning the `auth.hobbs.farm` issuer. If a client cannot find the auth server, revisit `http.public_base` in the MAS config (Task 5 Step 2) before touching `.well-known`.
 - [ ] **Step 5: Add a Prometheus blackbox probe for the new endpoint**
 Add `https://auth.hobbs.farm/health` to the blackbox target list in `/mnt/docker-data/configs/prometheus/prometheus.yml`, then hot-reload:
 ```bash
 ssh glyptic@192.168.86.192 'curl -s -X POST http://localhost:9095/-/reload && sleep 5 && curl -s "http://localhost:9095/api/v1/query?query=probe_success" | grep -o "auth.hobbs.farm"'
 ```
 Expected: the target appears and probes `1`. **Auth being down now means nobody can log in — this probe is not optional.**
 - [ ] **Step 6: Commit**
 ```bash
 git add compose/ && git commit -m "feat: publish MAS at auth.hobbs.farm with blackbox probe"
 ```
 ---
 ### Task 9: Enable the QR MSCs
 **Files:**
 - Modify: `/var/lib/synapse/homeserver.yaml` (node only)
 - [ ] **Step 1: Enable both MSCs**
 ```yaml
 experimental_features:
  msc4108_enabled: true
  msc4388_mode: authenticated
 ```
 `msc4388_mode: authenticated` requires the rendezvous channel user to be logged in — prefer it over `open`, which would let unauthenticated parties open rendezvous sessions on an internet-facing server.
 - [ ] **Step 2: Restart and confirm Synapse actually starts**
 ```bash
 ssh glyptic@192.168.86.192 'docker restart synapse && sleep 45 && docker logs synapse --since 2m 2>&1 | grep -iE "ConfigError|MSC4108|MSC4388" | head'
 ```
 Expected: **no `ConfigError`**. If you see `MSC4108 requires matrix_authentication_service to be enabled`, Task 7 Step 3 did not take effect.
 - [ ] **Step 3: Verify the feature is advertised**
 ```bash
 ssh glyptic@192.168.86.192 'curl -s http://localhost:8008/_matrix/client/versions | python3 -c "
 import sys,json; uf=json.load(sys.stdin)[\"unstable_features\"]
 print(\"msc4108:\", uf.get(\"org.matrix.msc4108\"))
 "'
 ```
 Expected: `msc4108: True` (it is `False` today).
 - [ ] **Step 4: Commit**
 ```bash
 git commit --allow-empty -m "feat: enable MSC4108/MSC4388 QR sign-in"
 ```
 ---
 ### Task 10: End-to-end QR sign-in test and documentation
 - [ ] **Step 1: Test the actual feature**
 On an already-signed-in Element X Android device: Settings → Sessions → "Sign in with QR code" / "Link new device". On a second client, choose "Sign in with QR code" and scan.
 Expected: the new device signs in and receives E2EE keys **without** the password being typed. This is the deliverable — if it does not work here, the feature is not done regardless of what the config says.
 - [ ] **Step 2: Re-verify the full account inventory one more time**
 Re-run Task 7 Step 5. Expected: unchanged — all bots syncing, all tokens valid.
 - [ ] **Step 3: Complete the mandatory post-change checklist (CLAUDE.md)**
 - CLAUDE.md: MAS + two PostgreSQL containers on .192, port 8020, `auth.hobbs.farm`, Synapse now on PostgreSQL (supersedes Known Issues #24/#25 wording), new known issue for the MAS dependency — **auth is now a hard dependency for every login, including bots**
 - MEMORY.md: new entry for the MAS migration; update `matrix-account` (the reset recipe changes — user management moves to MAS)
 - Prometheus: done in Task 8 Step 4
 - Farm Manager links page: add `auth.hobbs.farm` to `serverLinks` in `/opt/farm-manager/server/static/app.js`, then **rebuild the image** (links are baked in, not mounted) and redeploy
 - Wiki: run the documentation-keeper agent
 - [ ] **Step 4: Retire the backups once stable**
 Leave `/var/lib/synapse.bak-premas/`, `/var/lib/synapse.bak-20260820-pre1159/`, and the SQLite file in place for **at least a week** of normal use. Only then delete.
 - [ ] **Step 5: Final commit**
 ```bash
 git add -A && git commit -m "docs: MAS QR sign-in deployment complete"
 ```
 ---
 ## Rollback Summary
 | Failed at | Recovery |
 |---|---|
 | Task 3 (port to PG) | Restore `homeserver.yaml.bak-presql`; SQLite file untouched and still authoritative |
 | Task 7 (syn2mas) | Restore `synapse-premas.sql.gz` + `mas-premigrate.sql.gz`; remove `matrix_authentication_service` block |
 | Task 9 (MSCs) | Remove `experimental_features` block; restart — no data implications |
 ## Open Questions for the Operator
 1. **Element Desktop compatibility.** Your two Element Desktop sessions are on the legacy auth stack. OIDC-native support in Element Web/Desktop has historically lagged Element X. Worth confirming before migrating, since a broken desktop client is a worse day than no QR code.
 2. **Is `auth.hobbs.farm` the hostname you want?** It becomes permanent — it is baked into session state, and changing `public_base` later invalidates sessions.
 3. **Vaultwarden is currently locked out** (memory `vault-master-password-stale`). MAS generates signing keys that must be backed up somewhere durable. Resolve the vault access first, or agree on an alternative.
 4. **Node memory.** .192 sits at ~14/23GB with Frigate as the main consumer. This adds ~1GB across three containers (Synapse's PG, MAS's PG, MAS). If that is too tight, the MAS stack could go on .100 instead — it only needs network reachability to Synapse, not co-location.
@@ -0,0 +1,41 @@
 # Matrix Auth Inventory — baseline before MAS migration
 Captured 2026-08-20 from `/var/lib/synapse/homeserver.db` (SQLite, pre-migration).
 **Totals:** 20 users (7 deactivated), **65 devices**, **54 access tokens**, 9 e2e device key entries.
 ## Per-account baseline
 | Account | Devices | Tokens | Password hash | Credential location | Auth style |
 |---|---|---|---|---|---|
 | `@jeremy` | 12 (incl. cross-signing keys) | 3 | yes | user-typed (`bigdog01`) | password + Element sessions |
 | `@clippyauto` | **17** | **17** | yes | n8n credential store (encrypted, `/home/node/.n8n/database.sqlite` on .100) | **repeated password login** — new device per login |
 | `@pluto` | **15** | **15** | yes | unknown (OpenClaw Gateway devices) | **repeated password login** |
 | `@julie` | 5 | 2 | yes | Element X Android | password |
 | `@spirit` | 2 | 3 | yes | `/mnt/infrastructure/spirit/openclaw.json` → key `accessToken` only | **access token** |
 | `@margaret` | 2 | 2 | yes | `/mnt/infrastructure/margaret/openclaw.json` → keys `accessToken` **and** `password` | token + password |
 | `@frigate` | 2 | 2 | yes | `compose/hf-pdocker-01/frigate-notify/.env` → `MATRIX_ACCESS_TOKEN=syt_ZnJpZ2F0…` | **access token** |
 | `@alertbot` | 2 | 2 | yes | `/mnt/docker-data/configs/alertmanager/matrix-receiver-config.yml` → `token: "syt_YWxlcnRib3Q…"` | **access token** |
 | `@hermes` | 2 | 2 | yes | unknown | unknown |
 | `@fry` | 1 | 1 | yes | unknown | unknown |
 | `@testbot` | 1 | 1 | yes | `TESTBOT_CI` device | unknown |
 | `@admin1`, `@adminuser` | 2 each | 2 each | yes | unknown | admin accounts |
 | `@aaron`, `@bob`, `@clippy`, `@travis`, `@tempadmin`, `@resetadmin`, `@resetadmin2` | 0 | 0 | none | — | deactivated |
 ## Risk read
 - **Access-token accounts** (`spirit`, `frigate`, `alertbot`) are the safest: syn2mas imports access tokens, so they should continue uninterrupted.
 - **`@clippyauto` and `@pluto` are the canaries.** The 1:1 device-to-token ratio and high counts mean they authenticate with a password on each connect, creating a fresh device every time. Post-migration they exercise the MAS **compatibility layer** (`/_matrix/client/*/login` proxied to MAS). If the bcrypt `unicode_normalization: true` scheme is misconfigured, these break first and loudest — `@clippyauto` polls every ~2s, so failure is visible within seconds.
 - `@jeremy`'s 12 devices include cross-signing keys (master/self_signing/user_signing), not just sessions. Cross-signing state must survive or Element will prompt to re-verify.
 ## Post-migration verification command
 ```bash
 ssh glyptic@192.168.86.192 'docker exec synapse-db psql -U synapse -d synapse -tc "
 select u.name,
        (select count(*) from devices d where d.user_id=u.name) dev,
        (select count(*) from access_tokens a where a.user_id=u.name) tok
 from users u where u.deactivated=0 order by u.name;"'
 ```
 Compare against the table above. Device counts may **grow** (new logins) but must not shrink for token-based accounts.