compose-farm/docs/architecture.md

icon

icon
lucide/layers

Architecture

This document explains how Compose Farm works under the hood.

Design Philosophy

Compose Farm follows three core principles:

1. KISS - Keep it simple. It's a thin wrapper around docker compose over SSH.
2. YAGNI - No orchestration, no service discovery, no health checks until needed.
3. Zero changes - Your existing compose files work unchanged.

High-Level Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Compose Farm CLI                         │
│                                                                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐ │
│  │  Config  │  │  State   │  │Operations│  │   Executor       │ │
│  │  Parser  │  │ Tracker  │  │  Logic   │  │  (SSH/Local)     │ │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────────┬─────────┘ │
└───────┼─────────────┼─────────────┼─────────────────┼───────────┘
        │             │             │                 │
        ▼             ▼             ▼                 ▼
┌───────────────────────────────────────────────────────────────┐
│                         SSH / Local                            │
└───────────────────────────────────────────────────────────────┘
        │                                             │
        ▼                                             ▼
┌───────────────┐                           ┌───────────────┐
│   Host: nuc   │                           │   Host: hp    │
│               │                           │               │
│ docker compose│                           │ docker compose│
│    up -d      │                           │    up -d      │
└───────────────┘                           └───────────────┘

Core Components

Configuration (src/compose_farm/config.py)

Pydantic models for YAML configuration:

• Config - Root configuration with compose_dir, hosts, services
• HostConfig - Host address and SSH user
• ServiceConfig - Service-to-host mappings

Key features:

• Validation with Pydantic
• Multi-host service expansion (all → list of hosts)
• YAML loading with sensible defaults

State Tracking (src/compose_farm/state.py)

Tracks deployment state in compose-farm-state.yaml (stored alongside the config file):

deployed:
  plex: nuc
  sonarr: nuc

Used for:

• Detecting migrations (service moved to different host)
• Identifying orphans (services removed from config)
• cf ps status display

Operations (src/compose_farm/operations.py)

Business logic for service operations:

• up - Start service, handle migration if needed
• down - Stop service
• preflight checks - Verify mounts, networks exist before operations
• discover - Find running services on hosts
• migrate - Down on old host, up on new host

Executor (src/compose_farm/executor.py)

SSH and local command execution:

• Hybrid SSH approach: asyncssh for parallel streaming, native ssh -t for raw mode
• Parallel by default: Multiple services via asyncio.gather
• Streaming output: Real-time stdout/stderr with [service] prefix
• Local detection: Skips SSH when target matches local machine IP

CLI (src/compose_farm/cli/)

Typer-based CLI with subcommand modules:

cli/
├── app.py          # Shared Typer app, version callback
├── common.py       # Shared helpers, options, progress utilities
├── config.py       # config subcommand (init, show, path, validate, edit, symlink)
├── lifecycle.py    # up, down, pull, restart, update, apply
├── management.py   # refresh, check, init-network, traefik-file
├── monitoring.py   # logs, ps, stats
├── ssh.py          # SSH key management (setup, status, keygen)
└── web.py          # Web UI server command

Command Flow

cf up plex

1. Load configuration
   └─► Parse compose-farm.yaml
   └─► Validate service exists

2. Check state
   └─► Load state.yaml
   └─► Is plex already running?
   └─► Is it on a different host? (migration needed)

3. Pre-flight checks
   └─► SSH to target host
   └─► Check compose file exists
   └─► Check required mounts exist
   └─► Check required networks exist

4. Execute migration (if needed)
   └─► SSH to old host
   └─► Run: docker compose down

5. Start service
   └─► SSH to target host
   └─► cd /opt/compose/plex
   └─► Run: docker compose up -d

6. Update state
   └─► Write new state to state.yaml

7. Generate Traefik config (if configured)
   └─► Regenerate traefik file-provider

cf apply

1. Load configuration and state

2. Compute diff
   ├─► Orphans: in state, not in config
   ├─► Migrations: in both, different host
   └─► Missing: in config, not in state

3. Stop orphans
   └─► For each orphan: cf down

4. Migrate services
   └─► For each migration: down old, up new

5. Start missing
   └─► For each missing: cf up

6. Update state

SSH Execution

Parallel Streaming (asyncssh)

For most operations, Compose Farm uses asyncssh:

async def run_command(host, command):
    async with asyncssh.connect(host) as conn:
        result = await conn.run(command)
        return result.stdout, result.stderr

Multiple services run concurrently via asyncio.gather.

Raw Mode (native ssh)

For commands needing PTY (progress bars, interactive):

ssh -t user@host "docker compose pull"

Local Detection

When target host IP matches local machine:

if is_local(host_address):
    # Run locally, no SSH
    subprocess.run(command)
else:
    # SSH to remote
    ssh.run(command)

State Management

State File

Location: compose-farm-state.yaml (stored alongside the config file)

deployed:
  plex: nuc
  sonarr: nuc

Image digests are stored separately in dockerfarm-log.toml (also in the config directory).

State Transitions

Config Change          State Change           Action
─────────────────────────────────────────────────────
Add service           Missing                 cf up
Remove service        Orphaned                cf down
Change host           Migration               down old, up new
No change             No change               none (or refresh)

cf refresh

Syncs state with reality by querying Docker on each host:

docker ps --format '{{.Names}}'

Updates state.yaml to match what's actually running.

Compose File Discovery

For each service, Compose Farm looks for compose files in:

{compose_dir}/{service}/
├── compose.yaml         # preferred
├── compose.yml
├── docker-compose.yml
└── docker-compose.yaml

First match wins.

Traefik Integration

Label Extraction

Compose Farm parses Traefik labels from compose files:

services:
  plex:
    labels:
      - traefik.enable=true
      - traefik.http.routers.plex.rule=Host(`plex.example.com`)
      - traefik.http.services.plex.loadbalancer.server.port=32400

File Provider Generation

Converts labels to Traefik file-provider YAML:

http:
  routers:
    plex:
      rule: Host(`plex.example.com`)
      service: plex
  services:
    plex:
      loadBalancer:
        servers:
          - url: http://192.168.1.10:32400

Variable Resolution

Supports ${VAR} and ${VAR:-default} from:

1. Service's .env file
2. Current environment

Error Handling

Pre-flight Failures

Before any operation, Compose Farm checks:

• SSH connectivity
• Compose file existence
• Required mounts
• Required networks

If checks fail, operation aborts with clear error.

Partial Failures

When operating on multiple services:

• Each service is independent
• Failures are logged, but other services continue
• Exit code reflects overall success/failure

Performance Considerations

Parallel Execution

Services are started/stopped in parallel:

await asyncio.gather(*[
    up_service(service) for service in services
])

SSH Multiplexing

For repeated connections to the same host, SSH reuses connections.

Caching

• Config is parsed once per command
• State is loaded once, written once
• Host discovery results are cached during command

Web UI Architecture

┌─────────────────────────────────────────────────────────────┐
│                         Web UI                               │
│                                                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │   FastAPI   │  │    Jinja    │  │       HTMX          │  │
│  │   Backend   │  │  Templates  │  │   Dynamic Updates   │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
│                                                             │
│  Pattern: Custom events, not hx-swap-oob                    │
│  Elements trigger on: cf:refresh from:body                  │
└─────────────────────────────────────────────────────────────┘

Icons use Lucide. Add new icons as macros in web/templates/partials/icons.html.