Server Architecture

Docker Compose stacks managed by the host-level openmates server CLI: a core FastAPI/Directus/Vault stack, an isolated upload server, and an isolated preview server.

Why This Exists

• One api container hosts all app skills in-process (OPE-342) — every backend/apps/{name}/ folder is loaded via importlib at startup, no per-app containers
• Celery workers (app-ai-worker, app-images-worker, app-pdf-worker, task-worker, user-init-worker, user-tasks-worker, core-worker, workflow-worker, task-scheduler) run their own queues for long-running, parallelizable, or autoscaled work — they earn their RAM
• Infrastructure services (cache, vault, monitoring) are co-located in the same Compose stack
• The preview server runs on a separate VM for security isolation (blocks SSRF, prevents hotlinking)
• Image-mode server operations are owned by the CLI, not the web UI. The CLI packages runtime templates, creates pre-update backups, applies service-scoped updates, manages host-level Caddyfile drift, and owns post-update/periodic runtime verification.

CLI-Managed Roles

Role	Compose source	Data-bearing	Health check	Purpose
core	backend/core/docker-compose.selfhost.yml	Yes	http://localhost:8000/v1/health	API, Directus, Postgres, Vault, cache, workers, optional web app
upload	backend/upload/docker-compose.selfhost.yml	Yes	http://localhost:8000/health	Isolated uploads service, local Vault, ClamAV, admin sidecar
preview	backend/preview/docker-compose.selfhost.yml	No product data	http://localhost:8080/health	Isolated image/favicon/metadata proxy with cache

Core profiles are resolved by the CLI:

Profile	Observability services
minimal	none
standard	OpenObserve and Promtail
production	OpenObserve, Promtail, Prometheus, and cAdvisor

Alertmanager is opt-in via --with-alerts. Production backend-only operation should use service filtering such as --exclude webapp when the official web app is hosted separately.

How It Works

Core Services

Defined in docker-compose.yml:

Container	Image / Build	Purpose
api	Custom (FastAPI)	Core REST API, WebSocket server
task-worker	Custom (Celery)	Latency-sensitive authentication and billing email tasks
user-init-worker	Custom (Celery)	Startup cache warming and account-deletion bootstrap tasks isolated from persistence backlog
user-tasks-worker	Custom (Celery)	AI Task admission and stale-dispatch reconciliation isolated from persistence backlog
core-worker	Custom (Celery)	Persistence, deletion, health, and reminder tasks
task-scheduler	Custom (Celery Beat)	Scheduled/periodic task dispatch
cms	directus/directus:11.5	Directus CMS for data management
cms-database	postgres:13-alpine	PostgreSQL database
cms-setup	Custom	Schema migration on startup (runs once)
cache	dragonflydb/dragonfly	Redis-compatible cache (Dragonfly)
vault	hashicorp/vault:1.19	Secret management, transit encryption
vault-setup	Custom	Vault initialization (runs once)
core-admin-sidecar	Custom	Admin utilities (health checks, scripts)

App Containers (Workers Only — In-Process Skills since OPE-342)

There are no per-app sync API containers. The 20 app-{name} Uvicorn containers that used to host one skill class each were removed in OPE-342: they burned ~2.6 GiB of idle RAM, added ~10 ms per skill call (HTTP serialization), required a 60-line docker-compose.yml entry per new app, and provided none of their claimed scaling/isolation benefits.

The api container now loads every backend/apps/{name}/app.yml via filesystem scan at startup and resolves each skill class_path via importlib. Skills are dispatched in-process via the SkillRegistry (backend/core/api/app/services/skill_registry.py) — see app-skills.md.

Only the Celery worker containers remain — they have real, queue-driven workloads:

Worker	Queues	Why containerized
app-ai-worker	app_ai	LLM streaming pipeline, distinct memory profile
app-images-worker	app_images	GPU/CPU-heavy image generation
app-pdf-worker	app_pdf	PDF rendering with pymupdf/reportlab
task-worker	email	Authentication, billing, and notification email delivery isolated from backlog-prone tasks
user-init-worker	user_init	Cache warming isolated so long-running persistence tasks cannot block startup sync
user-tasks-worker	user_tasks	AI Task capacity refill and stale-dispatch reconciliation isolated from persistence backlog
core-worker	persistence, health_check, …	Infrastructure and persistence tasks
workflow-worker	workflow	Manual workflow runs isolated from persistence and scheduled-trigger backlog
task-scheduler	(Celery beat)	Periodic task dispatch

Workers also build their own SkillRegistry instance in init_worker_process() so they can dispatch skills without HTTPing back to api.

Monitoring Stack

Container	Image	Purpose
openobserve	zinclabs/openobserve:v0.70.0-rc3	Log aggregation, metrics (replaces Loki+Grafana)
prometheus	prom/prometheus:v3.2.1	Metrics collection
alertmanager	prom/alertmanager:v0.31.1	Alert routing
cadvisor	cadvisor:v0.47.2	Container resource metrics
promtail	grafana/promtail:3.4.2	Log shipping to OpenObserve

Grafana and a backup-service are defined but commented out.

Host Runtime Contract

The host CLI invokes backend/scripts/runtime_health_verifier.py inside the role’s Python container after Compose readiness and every five minutes thereafter. This is an internal container-exec contract, not a public HTTP endpoint. Independent checks run concurrently after required service and HTTP baselines pass, with a single 60-second global deadline.

Core chat plumbing is tested with a dedicated provider-free Celery task routed to app_ai; it exercises dispatch, worker execution, result transport, and Redis without model inference or user records. Celery Beat writes a no-spend scheduler heartbeat through the health_check queue. Database, cache, Vault, upload antivirus, and preview renderer checks use role-specific operational probes.

The host owns incident state and notification delivery so API/Celery failure cannot suppress stale detection. It installs separate systemd monitor and watchdog timers, stores mode-0600 atomic state under <install>/.openmates/runtime-health/, and sends independently retried email, Discord, or signed webhook events. Generic webhook delivery pins a validated public address, rejects redirects/private networks, and includes replay-bounded HMAC headers.

Billing authority comes only from OPENMATES_DEPLOYMENT_MODE. official_cloud additionally requires agreeing overlay, environment, package-import, hosting-domain, and encrypted-domain witnesses. Any missing or conflicting witness selects self_host before billing imports or secret reads. Self-host inventories contain no billing check IDs; official-cloud checks are read-only and cannot create payment-provider resources.

Preview Server

Runs on a separate VM at preview.openmates.org. See docker-compose.preview.yml.

Endpoints:

• GET /api/v1/image – fetch, resize, cache external images (disk-based LRU, 10GB, 7-day TTL)
• GET /api/v1/favicon – fetch and cache favicons (tries /favicon.ico, falls back to Google Favicon Service)
• POST /api/v1/metadata – extract Open Graph / HTML metadata (24-hour cache TTL)
• GET /health, GET /health/detailed – health checks

Security: referer validation, SSRF protection (blocks private IPs), content-type validation, optional API key auth. 4 uvicorn workers by default.

Self-hosted option: uncomment the preview service in the core docker-compose.yml.

Edge Cases

• Docker network isolation: app containers communicate via internal network only; not exposed publicly
• Vault token management: vault-setup runs once on startup; api, task-worker, user-init-worker, and core-worker wait for it via depends_on: service_completed_successfully
• Cache as Dragonfly: drop-in Redis replacement with better memory efficiency; same protocol

Related Docs

• Security Architecture – Vault integration, encryption
• Apps Architecture – app skill execution model