Compare commits

...

188 Commits

Author SHA1 Message Date
Bas Nijholt
920b593d5f Fix mypy error: add type annotation for proc variable 2025-12-17 00:17:20 -08:00
Bas Nijholt
27d9b08ce2 Add -f shorthand for --full in apply command 2025-12-17 00:10:16 -08:00
Bas Nijholt
700cdacb4d Add 'a' alias for apply command (cf a = cf apply) 2025-12-17 00:09:45 -08:00
Bas Nijholt
3c7a532704 Add comments explaining lazy imports for startup performance 2025-12-17 00:08:28 -08:00
Bas Nijholt
6048f37ad5 Lazy import pydantic for faster CLI startup
- Create paths.py module with lightweight path utilities (no pydantic)
- Move Config imports to TYPE_CHECKING blocks in CLI modules
- Lazy import load_config only when needed

Combined with asyncssh lazy loading, cf --help now starts in ~150ms
instead of ~500ms (70% faster).
2025-12-17 00:07:15 -08:00
Bas Nijholt
f18952633f Lazy import asyncssh for faster CLI startup
Move asyncssh import inside _run_ssh_command() so it's only loaded
when actually executing SSH commands. This cuts CLI import time
from 414ms to 200ms (52% faster).

cf --help now starts in ~260ms instead of ~500ms.
2025-12-16 23:59:37 -08:00
Bas Nijholt
437257e631 Add cf config symlink command
Creates a symlink from ~/.config/compose-farm/compose-farm.yaml to a
local config file using absolute paths (avoids broken relative symlinks).

Usage:
  cf config symlink                     # Link to ./compose-farm.yaml
  cf config symlink /path/to/config.yaml  # Link to specific file

State files are automatically stored next to the actual config (not
the symlink) since config_path.resolve() follows symlinks.
2025-12-16 23:56:58 -08:00
Bas Nijholt
c720170f26 Add --full flag to apply command
When --full is passed, apply also runs 'docker compose up' on all
services (not just missing/migrating ones) to pick up any config
changes (compose file, .env, etc).

- cf apply          # Fast: state reconciliation only
- cf apply --full   # Thorough: also refresh all running services
2025-12-16 23:54:19 -08:00
Bas Nijholt
d9c03d6509 Feature apply as the hero command in README
- Update intro and NOTE block to lead with cf apply
- Rewrite "How It Works" to show declarative workflow first
- Move apply to top of command table (bolded)
- Reorder examples to show apply as "the main command"
- Update automation bullet to highlight one-command reconciliation
2025-12-16 23:49:49 -08:00
Bas Nijholt
3b7066711f Merge pull request #12 from basnijholt/feature/orphaned-services
Add apply command and refactor CLI for clearer UX
2025-12-16 23:34:34 -08:00
Bas Nijholt
6a630c40a1 Update apply command description to include starting missing services 2025-12-16 23:32:27 -08:00
Bas Nijholt
9f9c042b66 Remove up --migrate flag in favor of apply
Simplifies CLI by having one clear reconciliation command:
- cf up <service>  = start specific services (auto-migrates if needed)
- cf apply         = full reconcile (stop orphans + migrate + start missing)

The --migrate flag was redundant with 'apply --no-orphans'.
2025-12-16 23:27:19 -08:00
github-actions[bot]
2a6d7d0b85 Update README.md 2025-12-17 07:21:38 +00:00
Bas Nijholt
6d813ccd84 Merge af9c760fb8 into affed2edcf 2025-12-17 07:21:23 +00:00
Bas Nijholt
af9c760fb8 Add missing service detection to apply command
Previously, apply only handled:
1. Stopping orphans (in state, not in config)
2. Migrating services (in state, wrong host)

Now it also handles:
3. Starting missing services (in config, not in state)

This fixes the case where a service was stopped as an orphan, then
re-added to config - apply would say "nothing to do" instead of
starting it.

Added get_services_not_in_state() to state.py and updated tests.
2025-12-16 23:21:09 -08:00
Bas Nijholt
90656b05e3 Add tests for apply command and down --orphaned flag
Tests cover:
- apply: nothing to do, dry-run preview, migrations, orphan cleanup, --no-orphans
- down --orphaned: no orphans, stops services, error on invalid combinations

Lifecycle.py coverage improved from 20% to 61%.
2025-12-16 23:15:46 -08:00
github-actions[bot]
d7a3d4e8c7 Update README.md 2025-12-17 07:10:52 +00:00
Bas Nijholt
35f0b8bf99 Merge be6b391121 into affed2edcf 2025-12-16 23:10:36 -08:00
Bas Nijholt
be6b391121 Refactor CLI commands for clearer UX
Separate "read state from reality" from "write config to reality":
- Rename `sync` to `refresh` (updates local state from running services)
- Add `apply` command (makes reality match config: migrate + stop orphans)
- Add `down --orphaned` flag (stops services removed from config)
- Modify `up --migrate` to only handle migrations (not orphans)

The new mental model:
- `refresh` = Reality → State (discover what's running)
- `apply` = Config → Reality (reconcile: migrate services + stop orphans)

Also extract private helper functions for reporting to match codebase style.
2025-12-16 23:06:42 -08:00
Bas Nijholt
7f56ba6a41 Add orphaned service detection and cleanup
When services are removed from config but still tracked in state,
`cf up --migrate` now stops them automatically. This makes the
config truly declarative - comment out a service, run migrate,
and it stops.

Changes:
- Add get_orphaned_services() to state.py for detecting orphans
- Add stop_orphaned_services() to operations.py for cleanup
- Update lifecycle.py to call stop_orphaned_services on --migrate
- Refactor _report_orphaned_services to use shared function
- Rename "missing_from_config" to "unmanaged" for clarity
- Add tests for get_orphaned_services
- Only remove from state on successful down (not on failure)
2025-12-16 22:53:26 -08:00
Bas Nijholt
4b3d7a861e Fix migration and update for services with buildable images
Use `pull --ignore-buildable` to skip images that have `build:` defined
in the compose file, preventing pull failures for locally-built images
like gitea-runner-custom. The build step then handles these images.
2025-12-16 19:42:24 -08:00
Bas Nijholt
affed2edcf Refactor operations.py into smaller helpers
- Add PreflightResult NamedTuple with .ok property
- Extract _run_compose_step to handle raw output and interrupts
- Extract _up_single_service for single-host migration logic
- Deduplicate pull/build handling in _migrate_service
- Add was_running check to only rollback if service was running
2025-12-16 17:08:25 -08:00
Bas Nijholt
34642e8b8e Rollback to old host when migration up fails
When up fails on the target host after migration has already stopped
the service on the old host, attempt to restart on the old host as a
rollback. This prevents services from being left down after a failed
migration attempt.
2025-12-16 16:59:09 -08:00
Bas Nijholt
4c8b6c5209 Add init-network hint when network is missing 2025-12-16 16:22:06 -08:00
Bas Nijholt
2b38ed28c0 Skip traefik regeneration when all services fail
Don't update traefik config if all services in the operation failed.
This prevents adding routes for services that aren't actually running.
2025-12-16 16:21:09 -08:00
Bas Nijholt
26b57895ce Clean up orphaned containers when migration up fails
If `up` fails after migration (when we've already run `down` on the
old host), run `down` on the target host to clean up any containers
that were created but couldn't start (e.g., due to missing devices).

This prevents orphaned containers from lingering on the failed host.
2025-12-16 16:16:09 -08:00
Bas Nijholt
367da13fae Fix path existence check for permission denied
Use stat instead of test -e to distinguish "no such file" from
"permission denied". If stat fails with permission denied, the
path exists (just not accessible), so report it as existing.

Fixes false "missing path" errors for directories with restricted
permissions like /mnt/data/immich/library.
2025-12-16 15:08:38 -08:00
Bas Nijholt
d6ecd42559 Consolidate service requirement checks into shared function
Move check_service_requirements() to operations.py as a public function
that verifies paths, networks, and devices exist on a target host. Both
CLI check command and pre-flight migration checks now use this shared
function, eliminating duplication. Also adds device checking to the
check command output.
2025-12-16 14:53:59 -08:00
Bas Nijholt
233c33fa52 Add device checking to pre-flight migration checks
Services with devices: mappings (e.g., /dev/dri for GPU acceleration)
now have those devices verified on the target host before migration.
This prevents the scenario where a service is stopped on the old host
but fails to start on the new host due to missing devices.

Adds parse_devices() to extract host device paths from compose files.
2025-12-16 14:35:52 -08:00
Bas Nijholt
43974c5743 Abort on first Ctrl+C during migrations
Detect when SSH subprocess is killed by signal (exit code < 0 or 255)
and treat it as an interrupt. This allows single Ctrl+C to abort the
entire operation instead of requiring two presses.
2025-12-16 14:31:33 -08:00
Bas Nijholt
cf94a62f37 docs: Clarify pull/build comments in migration 2025-12-16 14:26:48 -08:00
Bas Nijholt
81b4074827 Pre-build Dockerfile services during migration
After pulling images, also run build for services with Dockerfiles.
This ensures build-based services have their images ready before
stopping the old service, minimizing downtime.

If build fails, abort the migration and leave the service running
on the old host.

Extract _migrate_service helper to reduce function complexity.
2025-12-16 14:17:19 -08:00
Bas Nijholt
455657c8df Abort migration if pre-pull fails
If pulling images on the target host fails (e.g., rate limit),
abort the migration and leave the service running on the old host.
This prevents downtime when Docker Hub rate limits are hit.
2025-12-16 14:14:35 -08:00
Bas Nijholt
ee5a92788a Pre-pull images during migration to reduce downtime
When migrating a service to a new host, pull images on the target
host before stopping the service on the old host. This minimizes
downtime since images are cached when the up command runs.

Migration flow:
1. Pull images on new host (service still running on old)
2. Down on old host
3. Up on new host (fast, images already pulled)
2025-12-16 14:12:53 -08:00
Bas Nijholt
2ba396a419 docs: Move Compose Farm to first column in comparison table 2025-12-16 13:48:40 -08:00
Bas Nijholt
7144d58160 build: Include LICENSE file in package distribution 2025-12-16 13:37:15 -08:00
Bas Nijholt
279fa2e5ef Create LICENSE 2025-12-16 13:36:35 -08:00
Bas Nijholt
dbe0b8b597 docs: Add app.py to CLAUDE.md architecture diagram 2025-12-16 13:14:51 -08:00
Bas Nijholt
b7315d255a refactor: Split CLI into modular subpackage (#11) 2025-12-16 13:08:08 -08:00
renovate[bot]
f003d2931f ⬆️ Update actions/checkout action to v6 (#5)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2025-12-16 12:19:45 -08:00
renovate[bot]
6f7c557065 ⬆️ Update actions/setup-python action to v6 (#6)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2025-12-16 12:18:34 -08:00
renovate[bot]
ecb6ee46b1 ⬆️ Update astral-sh/setup-uv action to v7 (#8)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2025-12-16 12:18:28 -08:00
renovate[bot]
354967010f ⬆️ Update redis Docker tag to v8 (#9)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2025-12-16 12:18:22 -08:00
github-actions[bot]
57122f31a3 Update README.md 2025-12-16 20:01:03 +00:00
Bas Nijholt
cbbcec0d14 Add config subcommand for managing configuration files (#10) 2025-12-16 12:00:44 -08:00
Bas Nijholt
de38c35b8a docs: Add one-liner showing manual equivalent 2025-12-16 11:19:56 -08:00
github-actions[bot]
def996ddf4 Update README.md 2025-12-16 19:14:07 +00:00
Bas Nijholt
790e32e96b Fix test_load_config_not_found for CF_CONFIG env var 2025-12-16 11:13:44 -08:00
Bas Nijholt
fd75c4d87f Add CLI --help output to README 2025-12-16 11:12:43 -08:00
Bas Nijholt
411a99cbc4 Wait for PyPI propagation before Docker build
Also add Python 3.14 to classifiers.
2025-12-16 11:04:35 -08:00
Bas Nijholt
d2c6ab72b2 Add CF_CONFIG env var for simpler Docker workflow
Config search order is now:
1. --config CLI option
2. CF_CONFIG environment variable
3. ./compose-farm.yaml
4. ~/.config/compose-farm/compose-farm.yaml

Docker workflow simplified: mount compose_dir once, set CF_CONFIG
to config file within it. No more symlink issues or multiple mounts.
2025-12-16 10:12:55 -08:00
Bas Nijholt
3656584eda Friendly error when config path is a directory
Docker creates empty directories for missing file mounts,
causing confusing IsADirectoryError tracebacks. Now shows
a clear message explaining the likely cause.
2025-12-16 09:49:40 -08:00
Bas Nijholt
8be370098d Use env vars for docker-compose.yml mounts
- CF_CONFIG_DIR: config directory (default: ~/.config/compose-farm)
- CF_COMPOSE_DIR: compose directory (default: /opt/compose)

Mounts preserve paths so compose_dir in config works correctly.
2025-12-16 09:49:34 -08:00
Bas Nijholt
45057cb6df feat: Add docker-compose.yml for easier Docker usage
Example compose file that mounts SSH agent and config.
Users uncomment the compose_dir mount for their setup.
2025-12-16 09:40:18 -08:00
Bas Nijholt
3f24484d60 fix: Fix VERSION expansion in Dockerfile 2025-12-16 09:24:46 -08:00
Bas Nijholt
b6d50a22b4 fix: Wait for PyPI upload before building Docker image
Use workflow_run trigger to wait for "Upload Python Package" workflow
to complete successfully before building the Docker image. This ensures
the version is available on PyPI when uv tries to install it.
2025-12-16 09:21:35 -08:00
Bas Nijholt
8a658210e1 docs: Add Docker installation instructions with SSH agent 2025-12-16 09:16:43 -08:00
Bas Nijholt
583aaaa080 feat: Add Docker image and GitHub workflow
- Dockerfile using ghcr.io/astral-sh/uv:python3.14-alpine
- Installs compose-farm via uv tool install
- Includes openssh-client for remote host connections
- GitHub workflow builds and pushes to ghcr.io on release
- Supports manual workflow dispatch with version input
- Tags: semver (x.y.z, x.y, x) and latest
2025-12-16 09:11:09 -08:00
Bas Nijholt
22ca4f64e8 docs: Add command quick-reference table to Usage section 2025-12-16 08:30:15 -08:00
Bas Nijholt
32e798fcaa chore: Remove obsolete PLAN.md
The traefik-file feature described in this planning document has been
fully implemented. All open questions have been resolved.
2025-12-15 23:27:27 -08:00
Bas Nijholt
ced81c8b50 refactor: Make internal CLI symbols private
Rename module-internal type aliases, TypeVar, and constants with _ prefix:
- _T, _ServicesArg, _AllOption, _ConfigOption, _LogPathOption, _HostOption
- _MISSING_PATH_PREVIEW_LIMIT
- _DEFAULT_NETWORK_NAME, _DEFAULT_NETWORK_SUBNET, _DEFAULT_NETWORK_GATEWAY

These are only used within cli.py and should not be part of the public API.
2025-12-15 20:57:41 -08:00
Bas Nijholt
7ec4b71101 refactor: Remove unnecessary console aliasing in executor
Import console and err_console directly instead of aliasing to
_console and _err_console. Rename inner function variable to
'out' to avoid shadowing the module-level console import.
2025-12-15 20:36:39 -08:00
Bas Nijholt
94aa58d380 refactor: Make internal constants and classes private
Rename module-internal constants and classes with _ prefix:
- compose.py: SINGLE_PART, PUBLISHED_TARGET_PARTS, HOST_PUBLISHED_PARTS, MIN_VOLUME_PARTS
- logs.py: DIGEST_HEX_LENGTH
- traefik.py: LIST_VALUE_KEYS, MIN_ROUTER_PARTS, MIN_SERVICE_LABEL_PARTS,
  TraefikServiceSource, TRAEFIK_CONFIG_HEADER

These items are only used within their respective modules and should
not be part of the public API.
2025-12-15 20:33:48 -08:00
Bas Nijholt
f8d88e6f97 refactor: Remove run_compose_multi_host and rename report_preflight_failures to _report_preflight_failures
Eliminate the public run_compose_multi_host helper, which was a thin wrapper around the internal _run_sequential_commands_multi_host function, and mark the preflight failure reporting function as internal by prefixing it with an underscore.
Updated all internal calls accordingly.
2025-12-15 20:27:02 -08:00
Bas Nijholt
a95f6309b0 Remove dead code and make internal APIs public
Remove functions that were replaced by _with_progress variants in cli.py:
- discover_running_services, check_mounts_on_configured_hosts,
  check_networks_on_configured_hosts, _check_resources from operations.py
- snapshot_services from logs.py
- get_service_hosts from state.py

Make previously private functions public (remove underscore prefix):
- is_local in executor.py
- isoformat, collect_service_entries, load_existing_entries,
  merge_entries, write_toml in logs.py
- load_env, interpolate, parse_ports in compose.py

Update tests to use renamed public functions.
2025-12-15 20:19:28 -08:00
Bas Nijholt
502de018af docs: Add high availability row to comparison table 2025-12-15 19:51:57 -08:00
Bas Nijholt
a3e8daad33 docs: refine comparison table in README 2025-12-15 16:06:17 -08:00
Bas Nijholt
78a2f65c94 docs: Move comparison link after declarative setup line 2025-12-15 15:48:15 -08:00
Bas Nijholt
1689a6833a docs: Link to comparison section from Why Compose Farm 2025-12-15 15:46:26 -08:00
Bas Nijholt
6d2f32eadf docs: Add feature comparison table with emojis 2025-12-15 15:44:16 -08:00
Bas Nijholt
c549dd50c9 docs: Move comparison section to end, simplify format 2025-12-15 15:41:09 -08:00
Bas Nijholt
82312e9421 docs: add comparison with alternatives to README 2025-12-15 15:37:08 -08:00
Bas Nijholt
e13b367188 docs: Add shields to README 2025-12-15 15:31:30 -08:00
Bas Nijholt
d73049cc1b docs: Add declarative philosophy to Why Compose Farm 2025-12-15 15:17:04 -08:00
Bas Nijholt
4373b23cd3 docs: Simplify xkcd explanation, lead with simplicity 2025-12-15 14:54:29 -08:00
Bas Nijholt
73eb6ccf41 docs: Center xkcd image 2025-12-15 14:52:57 -08:00
Bas Nijholt
6ca48d0d56 docs: Add console.py to CLAUDE.md architecture 2025-12-15 14:52:40 -08:00
Bas Nijholt
b82599005e docs: Add xkcd reference and clarify this is not a new standard 2025-12-15 14:37:33 -08:00
Bas Nijholt
b044053674 docs: Emphasize zero changes required to compose files 2025-12-15 14:19:52 -08:00
Bas Nijholt
e4f03bcd94 docs: Clarify autokuma demonstrates multi-host feature 2025-12-15 14:14:47 -08:00
Bas Nijholt
ac3797912f Add AutoKuma labels to example services 2025-12-15 14:14:07 -08:00
Bas Nijholt
429a1f6e7e docs: Fix outdated .env instructions in examples README 2025-12-15 14:13:09 -08:00
Bas Nijholt
fab20e0796 Add header comment to generated traefik file-provider config
Includes repository link and explanation of what the file does.
Header is added automatically by render_traefik_config().
2025-12-15 14:11:49 -08:00
Bas Nijholt
1bc6baa0b0 Add realistic traefik file-provider example 2025-12-15 14:10:00 -08:00
Bas Nijholt
996e0748f8 style: Simplify compose-farm.yaml comments 2025-12-15 14:08:50 -08:00
Bas Nijholt
ca46fdfaa4 Replace trivial examples with real-world services
- traefik: Reverse proxy with Let's Encrypt DNS challenge
- mealie: Single container with resource limits
- uptime-kuma: Monitoring with Docker socket and user mapping
- paperless-ngx: Multi-container stack (Redis + SQLite)
- autokuma: Multi-host service (runs on all hosts)

Each example demonstrates dual Traefik routes:
- HTTPS (websecure): Custom domain with Let's Encrypt TLS
- HTTP (web): .local domain for LAN access without TLS

Includes compose-farm.yaml with multi-host config and
compose-farm-state.yaml showing deployed state.
2025-12-15 14:08:14 -08:00
Bas Nijholt
b480797e5b Add XDG_CONFIG_HOME support for config paths
Respect the XDG_CONFIG_HOME environment variable when looking for
config files and log paths. Falls back to ~/.config if not set.
2025-12-15 13:06:59 -08:00
Bas Nijholt
c47fdf847e Use _progress_bar helper for all progress bars
Standardize UI by using the same progress bar configuration
everywhere. Removes unused TaskProgressColumn import.
2025-12-15 13:03:50 -08:00
Bas Nijholt
3ca9562013 Consolidate console instances and progress bar patterns
- Add console.py module with shared Console instances
- Add _progress_bar() helper to reduce progress bar boilerplate
- Update cli.py, executor.py, operations.py to use shared console
2025-12-15 12:56:23 -08:00
Bas Nijholt
3104d5de28 Refactor state.py with context manager and cli.py with helper to reduce duplication 2025-12-15 11:10:44 -08:00
Bas Nijholt
fd141cbc8c Refactor executor and operations to eliminate code duplication 2025-12-15 11:07:58 -08:00
Bas Nijholt
aa0c15b6b3 Add project metadata to pyproject.toml
Add license, maintainers, keywords, classifiers, and project URLs
for better discoverability on PyPI.
2025-12-15 11:07:13 -08:00
Bas Nijholt
4630a3e551 Merge pull request #4 from basnijholt/feature/multi-host-services
Add multi-host service support
2025-12-15 10:58:36 -08:00
Bas Nijholt
b70d5c52f1 fix: Use strict=True in zip() for equal-length lists 2025-12-15 10:56:47 -08:00
Bas Nijholt
5d8635ba7b ci: Use prek in CI instead of separate ruff/mypy commands
prek is a faster, Rust-based alternative to pre-commit.
Also updates ruff-pre-commit to v0.14.9 to match project version.
2025-12-15 10:53:34 -08:00
Bas Nijholt
27dad9d9d5 style: Format cli.py 2025-12-15 10:51:51 -08:00
Bas Nijholt
abb4417b15 Add orphaned service detection in check command
Warns when services are in state but not in config. These are
services that were removed from config but may still be running.
Also refactors remote checks into helper function.
2025-12-15 10:43:10 -08:00
Bas Nijholt
388cca5591 Add summary output showing succeeded/failed service counts 2025-12-15 10:38:36 -08:00
Bas Nijholt
8aa019e25f fix: Move imports to top-level in test file 2025-12-15 10:29:29 -08:00
Bas Nijholt
e4061cfbde Fix down command for multi-host services
The result.service for multi-host services is 'svc@host' format.
Extract base service name before removing from state.
2025-12-15 10:09:15 -08:00
Bas Nijholt
9a1f20e2d4 Add per-host control and partial state tracking
- Track partial success: if some hosts succeed, state reflects
  only the hosts that actually started
- Add --host flag to up/down: operate on a specific host only
  - `cf up autokuma --host nuc` starts only on nuc
  - `cf down autokuma --host nuc` stops only on nuc
- Add state helpers: add_service_to_host, remove_service_from_host
- Validate --host is allowed for the service's configured hosts
2025-12-15 09:51:49 -08:00
Bas Nijholt
3b45736729 Validate multi-host config edge cases
- Block 'all' as a host name (reserved keyword)
- Reject empty host lists
- Reject duplicate hosts in explicit lists
2025-12-15 09:46:29 -08:00
Bas Nijholt
1d88fa450a docs: Explain why multi-host services are needed 2025-12-15 09:37:41 -08:00
Bas Nijholt
31ee6be163 Add multi-host service support
Allows services to run on multiple hosts using the "all" keyword
or an explicit list of hosts:

  services:
    autokuma: all              # Run on all configured hosts
    dozzle: [host1, host2]     # Run on specific hosts

Key changes:
- config.py: Added get_hosts() and is_multi_host() methods
- executor.py: Added run_compose_multi_host() for parallel execution
- operations.py: Added _up_multi_host_service() with pre-flight checks
- state.py: Track list of hosts for multi-host services
- cli.py: Updated discovery, stats, and check for multi-host
- README.md: Added documentation

Multi-host services:
- Run on all target hosts in parallel
- Skip migration logic (always run everywhere)
- Show [service@host] prefix in output
- Track all running hosts in state
2025-12-15 09:14:25 -08:00
Bas Nijholt
096a2ca5f4 sync: Remove extra spacing (transient bar already leaves one) 2025-12-14 20:41:38 -08:00
Bas Nijholt
fb04f6f64d sync: Add spacing before capturing progress bar 2025-12-14 20:39:02 -08:00
Bas Nijholt
d8e54aa347 check: Fix double newline spacing in output 2025-12-14 20:35:25 -08:00
Bas Nijholt
b2b6b421ba check: Add spacing before mounts/networks progress bar 2025-12-14 20:33:36 -08:00
Bas Nijholt
c6b35f02f0 check: Add spacing before SSH progress bar 2025-12-14 20:32:16 -08:00
Bas Nijholt
7e43b0a6b8 check: Fix spacing after transient progress bar 2025-12-14 20:31:20 -08:00
Bas Nijholt
2915b287ba check: Add SSH connectivity check as first remote step
- Check SSH connectivity to all remote hosts before mount/network checks
- Skip local hosts (no SSH needed)
- Show progress bar during SSH checks
- Report unreachable hosts with clear error messages
- Add newline spacing for better output formatting
2025-12-14 20:30:36 -08:00
Bas Nijholt
ae561db0c9 check: Add progress bar and parallelize mount/network checks
- Parallelize mount and network checking for all services
- Add Rich progress bar showing count, elapsed time, and service name
- Move all inline imports to top-level (contextlib, datetime, logs)
- Also sort state file entries alphabetically for consistency
2025-12-14 20:24:54 -08:00
Bas Nijholt
2d132747c4 sync: Enhance progress bars with count and elapsed time
Show "Discovering ━━━━ 32/65 • 0:00:05 • service-name" format with:
- M of N complete count
- Elapsed time
- Current service name
2025-12-14 20:15:39 -08:00
Bas Nijholt
2848163a04 sync: Add progress bars and parallelize operations
- Parallelize service discovery across all services
- Parallelize image digest capture
- Show transient Rich progress bars during both operations
- Significantly faster sync due to concurrent SSH connections
2025-12-14 20:13:42 -08:00
Bas Nijholt
76aa6e11d2 logs: Make --all and --host mutually exclusive
These options conflict conceptually - --all means all services across
all hosts, while --host means all services on a specific host.
2025-12-14 20:10:28 -08:00
Bas Nijholt
d377df15b4 logs: Add --host filter and contextual --tail default
- Add --host/-H option to filter logs to services on a specific host
- Default --tail to 20 lines when showing multiple services (--all, --host, or >1 service)
- Default to 100 lines for single service
- Add tests for contextual default and host filtering
2025-12-14 20:04:40 -08:00
Bas Nijholt
334c17cc28 logs: Use contextual default for --tail option
Default to 20 lines when --all is specified (quick overview),
100 lines otherwise (detailed view for specific services).
2025-12-14 19:59:12 -08:00
Bas Nijholt
f148b5bd3a docs: Add TrueNAS NFS root squash configuration guide
Explains how to set maproot_user/maproot_group to root/wheel
in TrueNAS to disable root squash, allowing Docker containers
running as root to write to NFS-mounted volumes.
2025-12-14 17:24:46 -08:00
Bas Nijholt
54af649d76 Make stats progress bar transient 2025-12-14 15:31:00 -08:00
Bas Nijholt
f6e5a5fa56 Add progress bar when querying hosts in stats --live 2025-12-14 15:30:37 -08:00
Bas Nijholt
01aa24d0db style: Add borders to stats summary table 2025-12-14 15:25:17 -08:00
Bas Nijholt
3e702ef72e Add stats command for overview of hosts and services
Shows hosts table (address, configured/running services) and summary
(total hosts, services, compose files on disk, pending migrations).
Use --live to query Docker for actual container counts.
2025-12-14 15:24:30 -08:00
Bas Nijholt
a31218f7e5 docs: Remove trivial progress counter detail 2025-12-14 15:16:45 -08:00
Bas Nijholt
5decb3ed95 docs: Add --migrate flag and hybrid SSH approach 2025-12-14 15:14:56 -08:00
Bas Nijholt
da61436fbb Use native ssh for raw mode, asyncssh for streaming
- Raw mode uses subprocess with `ssh -t` for proper TTY handling
- Progress bars now render correctly on remote hosts
- asyncssh still used for non-raw parallel streaming with prefixes
- Remove redundant header prints (operations.py handles them)
2025-12-14 15:12:48 -08:00
Bas Nijholt
b6025af0c8 Fix newline after raw output to prevent line mixing 2025-12-14 14:49:33 -08:00
Bas Nijholt
ab914677c4 Add progress counter [n/total] to up command 2025-12-14 14:48:48 -08:00
Bas Nijholt
c0b421f812 Add --migrate flag to up command
Automatically detects services where state differs from config
and migrates only those. Usage: cf up --migrate or cf up -m
2025-12-14 14:47:43 -08:00
Bas Nijholt
2a446c800f Always use raw output for up command
- Print service header before raw output (local and SSH)
- up command always uses raw=True since migrations are sequential
- Gives clean progress bar output without per-line prefixes
2025-12-14 14:44:53 -08:00
Bas Nijholt
dc541c0298 test: Skip shell-dependent tests on Windows/Mac 2025-12-14 14:28:31 -08:00
Bas Nijholt
4d9b8b5ba4 docs: Add TrueNAS NFS crossmnt workaround
Documents how to access child ZFS datasets over NFS by injecting
the crossmnt option into /etc/exports. Includes Python script and
setup instructions for cron-based persistence.
2025-12-14 14:11:10 -08:00
Bas Nijholt
566a07d3a4 Refactor: separate concerns into dedicated modules
- Extract compose.py from traefik.py for generic compose parsing
  (env loading, interpolation, ports, volumes, networks)
- Rename ssh.py to executor.py for clarity
- Extract operations.py from cli.py for business logic
  (up_services, discover_running_services, preflight checks)
- Update CLAUDE.md with new architecture diagram
- Add docs/dev/future-improvements.md for low-priority items

CLI is now a thin layer that delegates to operations module.
All 70 tests pass.
2025-12-14 12:49:24 -08:00
Bas Nijholt
921ce6f13a Add raw output mode for single-service operations
When operating on a single service, pass output directly to
stdout/stderr instead of prefixing each line with [service].
This enables proper handling of \r progress bars during
docker pull, up, etc.
2025-12-14 12:15:36 -08:00
Bas Nijholt
708e09a8cc Show target host when starting services 2025-12-14 12:09:07 -08:00
Bas Nijholt
04154b84f6 Add tests for network and path checking
- test_traefik: Tests for parse_external_networks()
- test_ssh: Tests for check_paths_exist() and check_networks_exist()
2025-12-14 12:08:35 -08:00
Bas Nijholt
2bc9b09e58 Add Docker network validation and init-network command
- check: Validates external networks exist on configured hosts
- up: Pre-flight check blocks if networks missing on target host
- init-network: Creates Docker network with consistent subnet/gateway
  across hosts (default: mynetwork 172.20.0.0/16)

Networks defined as `external: true` in compose files are now
checked before starting or migrating services.
2025-12-14 12:06:36 -08:00
Bas Nijholt
16d517dcd0 docs: Update README and CLAUDE.md for redesigned check command 2025-12-14 10:56:04 -08:00
Bas Nijholt
5e8d09b010 Redesign check command: unified validation + host compatibility
Merged check-mounts into check command. Now provides:
- Config validation (compose files exist)
- Traefik label validation
- Mount path validation (SSH-based)
- Host compatibility matrix when checking specific services

Usage:
  cf check              # Full validation of all services
  cf check --local      # Skip SSH mount checks (fast)
  cf check jellyfin     # Check service + show which hosts can run it

Removed standalone check-mounts command (merged into check).
2025-12-14 10:43:34 -08:00
Bas Nijholt
6fc3535449 Add pre-flight mount check before migration
When migrating a service to a new host, check that all required volume
mount paths exist on the target host BEFORE running down on the old host.
This prevents failed migrations where the service is stopped but can't
start on the new host due to missing NFS mounts.
2025-12-14 10:30:56 -08:00
Bas Nijholt
9158dba0ce Add check-mounts command to verify NFS paths exist
New command to verify volume mount paths exist on target hosts before
migration. Parses bind mounts from compose files and SSHs to hosts to
check each path exists.

- check_paths_exist() in ssh.py: batch check multiple paths efficiently
- parse_host_volumes() in traefik.py: extract bind mount paths from compose
- check-mounts command in cli.py: groups by host, reports missing paths

Usage: cf check-mounts plex jellyfin
       cf check-mounts --all
2025-12-14 10:25:26 -08:00
Bas Nijholt
7b2c431ca3 fix: Change whoami example port to 18082 to avoid conflicts 2025-12-14 09:46:20 -08:00
Bas Nijholt
9deb460cfc Add Traefik example to examples directory
- traefik/docker-compose.yml: Traefik with docker and file providers
- whoami/docker-compose.yml: Test service with Traefik labels
- Updated compose-farm.yaml with traefik_file auto-regeneration
- Updated README.md with Traefik usage instructions
2025-12-14 09:44:03 -08:00
Bas Nijholt
2ce6f2473b docs: Add Traefik config options to example 2025-12-14 01:19:13 -08:00
Bas Nijholt
04d8444168 docs: Use consistent server-1/server-2 naming in example config 2025-12-14 01:18:50 -08:00
Bas Nijholt
b539c4ba76 docs: Update CLAUDE.md with all modules and commands 2025-12-14 01:17:30 -08:00
Bas Nijholt
473bc089c7 docs: Use consistent server-1/server-2 naming throughout 2025-12-14 01:15:46 -08:00
Bas Nijholt
50f405eb77 docs: Use uv tool install for CLI tools 2025-12-14 01:14:12 -08:00
Bas Nijholt
fd0d3bcbcf docs: Use clearer host names in NFS example 2025-12-14 01:13:58 -08:00
Bas Nijholt
f2e8ab0387 docs: Recommend uv for installation 2025-12-14 01:13:24 -08:00
Bas Nijholt
dfbf2748c7 docs: Reorganize README for better flow 2025-12-14 01:12:09 -08:00
Bas Nijholt
57b0ba5916 CSS for logo 2025-12-14 00:59:59 -08:00
Bas Nijholt
e668fb0faf Add logo 2025-12-14 00:58:58 -08:00
Bas Nijholt
2702203cb5 fix: Handle non-string address in getaddrinfo result 2025-12-14 00:55:11 -08:00
Bas Nijholt
27f17a2451 Remove unused PortMapping.protocol field 2025-12-14 00:52:47 -08:00
Bas Nijholt
98c2492d21 docs: Add cf alias and check command to README 2025-12-14 00:41:26 -08:00
Bas Nijholt
04339cbb9a Group CLI commands into Lifecycle, Monitoring, Configuration 2025-12-14 00:37:18 -08:00
Bas Nijholt
cdb3b1d257 Show friendly error when config file not found
Instead of a Python traceback, display a clean error message with
the red ✗ symbol when the config file cannot be found.
2025-12-14 00:31:36 -08:00
Bas Nijholt
0913769729 Fix check command to validate all services with check_all flag 2025-12-14 00:23:23 -08:00
Bas Nijholt
3a1d5b77b5 Add traefik port validation to check command 2025-12-14 00:19:17 -08:00
Bas Nijholt
e12002ce86 Add test for network_mode: service:X port lookup 2025-12-14 00:03:11 -08:00
Bas Nijholt
676a6fe72d Support network_mode: service:X for port lookup in traefik config 2025-12-14 00:02:07 -08:00
Bas Nijholt
f29f8938fe Add -h as alias for --help 2025-12-13 23:56:33 -08:00
Bas Nijholt
4c0e147786 Escape log output to prevent Rich markup errors 2025-12-13 23:55:44 -08:00
Bas Nijholt
cba61118de Add cf alias for compose-farm command 2025-12-13 23:54:00 -08:00
Bas Nijholt
32dc6b3665 Skip empty lines in streaming output 2025-12-13 23:50:35 -08:00
Bas Nijholt
7d98e664e9 Auto-detect local IPs to skip SSH when on target host 2025-12-13 23:48:28 -08:00
Bas Nijholt
6763403700 Fix duplicate prefix before traefik config message 2025-12-13 23:46:41 -08:00
Bas Nijholt
feb0e13bfd Add check command to find missing services 2025-12-13 23:43:47 -08:00
Bas Nijholt
b86f6d190f Add Rich styling to CLI output
- Service names in cyan, host names in magenta
- Success checkmarks, warning/error symbols
- Colored sync diff indicators (+/-/~)
- Unicode arrows for migrations
2025-12-13 23:40:07 -08:00
Bas Nijholt
5ed15b5445 docs: Add Docker Swarm overlay network notes 2025-12-13 23:16:09 -08:00
Bas Nijholt
761b6dd2d1 Rename state file to compose-farm-state.yaml (not hidden) 2025-12-13 23:01:40 -08:00
Bas Nijholt
e86c2b6d47 docs: Simplify Traefik port requirement note 2025-12-13 22:59:50 -08:00
basnijholt
9353b74c35 chore(docs): update TOC 2025-12-14 06:58:15 +00:00
Bas Nijholt
b7e8e0f3a9 docs: Add limitations and best practices section
Documents cross-host networking limitations:
- Docker DNS doesn't work across hosts
- Dependent services should stay in same compose file
- Ports must be published for cross-host communication
2025-12-13 22:58:01 -08:00
Bas Nijholt
b6c02587bc Rename traefik_host to traefik_service
Instead of specifying the host directly, specify the service name
that runs Traefik. The host is then looked up from the services
mapping, avoiding redundancy.
2025-12-13 22:43:33 -08:00
Bas Nijholt
d412c42ca4 Store state file alongside config file
State is now stored at .compose-farm-state.yaml in the same
directory as the config file. This allows multiple compose-farm
setups with independent state.

State functions now require a Config parameter to locate the
state file via config.get_state_path().
2025-12-13 22:38:11 -08:00
Bas Nijholt
13e0adbbb9 Add traefik_host config to skip local services
When traefik_host is set, services on that host are skipped in
file-provider generation since Traefik's docker provider handles
them directly. This allows running compose-farm from any host
while still generating correct file-provider config.
2025-12-13 22:34:20 -08:00
Bas Nijholt
68c41eb37c Improve missing ports warning message
Replace technical "L3 reachability" phrasing with actionable
guidance: "Add a ports: mapping for cross-host routing."
2025-12-13 22:29:20 -08:00
Bas Nijholt
8af088bb5d Add traefik_file config for auto-regeneration
When traefik_file is set in config, compose-farm automatically
regenerates the Traefik file-provider config after up, down,
restart, and update commands. Eliminates the need to manually
run traefik-file after service changes.
2025-12-13 22:24:29 -08:00
Bas Nijholt
1308eeca12 fix: Skip local services in traefik-file generation
Local services (localhost, local, 127.0.0.1) are handled by Traefik's
docker provider directly. Generating file-provider entries for them
creates conflicting routes with broken localhost URLs (since Traefik
runs in a container where localhost is isolated).

Now traefik-file only generates config for remote services.
2025-12-13 19:51:57 -08:00
Bas Nijholt
a66a68f395 docs: Clarify no merge commits to main rule 2025-12-13 19:44:30 -08:00
Bas Nijholt
6ea25c862e docs: Add traefik directory merging instructions 2025-12-13 19:41:16 -08:00
Bas Nijholt
280524b546 docs: Use GitHub admonition for TL;DR 2025-12-13 19:34:31 -08:00
Bas Nijholt
db9360771b docs: Shorten TL;DR 2025-12-13 19:33:28 -08:00
Bas Nijholt
c7590ed0b7 docs: Move TOC below TL;DR 2025-12-13 19:32:41 -08:00
Bas Nijholt
bb563b9d4b docs: Add TL;DR to README 2025-12-13 19:31:05 -08:00
Bas Nijholt
fe160ee116 fix: Move traefik import to top-level 2025-12-13 17:07:29 -08:00
Bas Nijholt
4c7f49414f docs: Update README for sync command and auto-migration
- Replace snapshot with sync command
- Add auto-migration documentation
- Update compose file naming convention
2025-12-13 16:55:07 -08:00
65 changed files with 6651 additions and 1287 deletions

View File

@@ -16,10 +16,10 @@ jobs:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}
@@ -39,10 +39,10 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
- name: Set up Python
run: uv python install 3.12
@@ -50,11 +50,5 @@ jobs:
- name: Install dependencies
run: uv sync --all-extras --dev
- name: Run ruff check
run: uv run ruff check .
- name: Run ruff format check
run: uv run ruff format --check .
- name: Run mypy
run: uv run mypy src/compose_farm
- name: Run pre-commit (via prek)
uses: j178/prek-action@v1

92
.github/workflows/docker.yml vendored Normal file
View File

@@ -0,0 +1,92 @@
name: Build and Push Docker Image
on:
workflow_run:
workflows: ["Upload Python Package"]
types: [completed]
workflow_dispatch:
inputs:
version:
description: 'Version to build (leave empty for latest)'
required: false
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-push:
runs-on: ubuntu-latest
# Only run if PyPI upload succeeded (or manual dispatch)
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
permissions:
contents: read
packages: write
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract version
id: version
run: |
if [ "${{ github.event_name }}" = "workflow_run" ]; then
# Get version from the tag that triggered the release
VERSION="${{ github.event.workflow_run.head_branch }}"
# Strip 'v' prefix if present
VERSION="${VERSION#v}"
elif [ -n "${{ github.event.inputs.version }}" ]; then
VERSION="${{ github.event.inputs.version }}"
else
VERSION=""
fi
echo "version=$VERSION" >> $GITHUB_OUTPUT
- name: Wait for PyPI
if: steps.version.outputs.version != ''
run: |
VERSION="${{ steps.version.outputs.version }}"
echo "Waiting for compose-farm==$VERSION on PyPI..."
for i in {1..30}; do
if curl -sf "https://pypi.org/pypi/compose-farm/$VERSION/json" > /dev/null; then
echo "✓ Version $VERSION available on PyPI"
exit 0
fi
echo "Attempt $i: not yet available, waiting 10s..."
sleep 10
done
echo "✗ Timeout waiting for PyPI"
exit 1
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=semver,pattern={{version}},value=v${{ steps.version.outputs.version }}
type=semver,pattern={{major}}.{{minor}},value=v${{ steps.version.outputs.version }}
type=semver,pattern={{major}},value=v${{ steps.version.outputs.version }}
type=raw,value=latest
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
build-args: |
VERSION=${{ steps.version.outputs.version }}
cache-from: type=gha
cache-to: type=gha,mode=max

View File

@@ -13,9 +13,9 @@ jobs:
permissions:
id-token: write
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
- name: Build
run: uv build
- name: Publish package distributions to PyPI

View File

@@ -11,16 +11,16 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4
uses: actions/checkout@v6
with:
persist-credentials: false
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v6
- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
- name: Run markdown-code-runner
env:

View File

@@ -10,7 +10,7 @@ repos:
- id: debug-statements
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.8.4
rev: v0.14.9
hooks:
- id: ruff
args: [--fix]

View File

@@ -10,19 +10,34 @@
```
compose_farm/
├── config.py # Pydantic models, YAML loading
├── ssh.py # asyncssh execution, streaming
└── cli.py # Typer commands
├── cli/ # CLI subpackage
│ ├── __init__.py # Imports modules to trigger command registration
│ ├── app.py # Shared Typer app instance, version callback
│ ├── common.py # Shared helpers, options, progress bar utilities
│ ├── config.py # Config subcommand (init, show, path, validate, edit)
│ ├── lifecycle.py # up, down, pull, restart, update, apply commands
│ ├── management.py # refresh, check, init-network, traefik-file commands
│ └── monitoring.py # logs, ps, stats commands
├── config.py # Pydantic models, YAML loading
├── compose.py # Compose file parsing (.env, ports, volumes, networks)
├── console.py # Shared Rich console instances
├── executor.py # SSH/local command execution, streaming output
├── operations.py # Business logic (up, migrate, discover, preflight checks)
├── state.py # Deployment state tracking (which service on which host)
├── logs.py # Image digest snapshots (dockerfarm-log.toml)
└── traefik.py # Traefik file-provider config generation from labels
```
## Key Design Decisions
1. **asyncssh over Paramiko/Fabric**: Native async support, built-in streaming
1. **Hybrid SSH approach**: asyncssh for parallel streaming with prefixes; native `ssh -t` for raw mode (progress bars)
2. **Parallel by default**: Multiple services run concurrently via `asyncio.gather`
3. **Streaming output**: Real-time stdout/stderr with `[service]` prefix
3. **Streaming output**: Real-time stdout/stderr with `[service]` prefix using Rich
4. **SSH key auth only**: Uses ssh-agent, no password handling (YAGNI)
5. **NFS assumption**: Compose files at same path on all hosts
6. **Local execution**: When host is `localhost`/`local`, skip SSH and run locally
6. **Local IP auto-detection**: Skips SSH when target host matches local machine's IP
7. **State tracking**: Tracks where services are deployed for auto-migration
8. **Pre-flight checks**: Verifies NFS mounts and Docker networks exist before starting/migrating
## Communication Notes
@@ -31,17 +46,26 @@ compose_farm/
## Git Safety
- Never amend commits.
- Never merge into a branch; prefer fast-forward or rebase as directed.
- **NEVER merge anything into main.** Always commit directly or use fast-forward/rebase.
- Never force push.
## Commands Quick Reference
| Command | Docker Compose Equivalent |
|---------|--------------------------|
| `up` | `docker compose up -d` |
| `down` | `docker compose down` |
| `pull` | `docker compose pull` |
| `restart` | `down` + `up -d` |
CLI available as `cf` or `compose-farm`.
| Command | Description |
|---------|-------------|
| `up` | Start services (`docker compose up -d`), auto-migrates if host changed |
| `down` | Stop services (`docker compose down`). Use `--orphaned` to stop services removed from config |
| `pull` | Pull latest images |
| `restart` | `down` + `up -d` |
| `update` | `pull` + `down` + `up -d` |
| `logs` | `docker compose logs` |
| `ps` | `docker compose ps` |
| `apply` | Make reality match config: migrate services + stop orphans. Use `--dry-run` to preview |
| `logs` | Show service logs |
| `ps` | Show status of all services |
| `stats` | Show overview (hosts, services, pending migrations; `--live` for container counts) |
| `refresh` | Update state from reality: discover running services, capture image digests |
| `check` | Validate config, traefik labels, mounts, networks; show host compatibility |
| `init-network` | Create Docker network on hosts with consistent subnet/gateway |
| `traefik-file` | Generate Traefik file-provider config from compose labels |
| `config` | Manage config files (init, show, path, validate, edit) |

16
Dockerfile Normal file
View File

@@ -0,0 +1,16 @@
# syntax=docker/dockerfile:1
FROM ghcr.io/astral-sh/uv:python3.14-alpine
# Install SSH client (required for remote host connections)
RUN apk add --no-cache openssh-client
# Install compose-farm from PyPI
ARG VERSION
RUN uv tool install compose-farm${VERSION:+==$VERSION}
# Add uv tool bin to PATH
ENV PATH="/root/.local/bin:$PATH"
# Default entrypoint
ENTRYPOINT ["cf"]
CMD ["--help"]

21
LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 Bas Nijholt
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

35
PLAN.md
View File

@@ -1,35 +0,0 @@
# Compose Farm Traefik Multihost Ingress Plan
## Goal
Generate a Traefik file-provider fragment from existing docker-compose Traefik labels (no config duplication) so a single front-door Traefik on 192.168.1.66 with wildcard `*.lab.mydomain.org` can route to services running on other hosts. Keep the current simplicity (SSH + docker compose); no Swarm/K8s.
## Requirements
- Traefik stays on main host; keep current `dynamic.yml` and Docker provider for local containers.
- Add a watched directory provider (any path works) and load a generated fragment (e.g., `compose-farm.generated.yml`).
- No edits to compose files: reuse existing `traefik.*` labels as the single source of truth; Compose Farm only reads them.
- Generator infers routing from labels and reachability from `ports:` mappings; prefer host-published ports so Traefik can reach services across hosts. Upstreams point to `<host address>:<published host port>`; warn if no published port is found.
- Only minimal data in `compose-farm.yaml`: hosts map and service→host mapping (already present).
- No new orchestration/discovery layers; respect KISS/YAGNI/DRY.
## Non-Goals
- No Swarm/Kubernetes adoption.
- No global Docker provider across hosts.
- No health checks/service discovery layer.
## Current State (Dec 2025)
- Compose Farm: Typer CLI wrapping `docker compose` over SSH; config in `compose-farm.yaml`; parallel by default; snapshot/log tooling present.
- Traefik: single instance on 192.168.1.66, wildcard `*.lab.mydomain.org`, Docker provider for local services, file provider via `dynamic.yml` already in use.
## Proposed Implementation Steps
1) Add generator command: `compose-farm traefik-file --output <path>`.
2) Resolve per-service host from `compose-farm.yaml`; read compose file at `{compose_dir}/{service}/docker-compose.yml`.
3) Parse `traefik.*` labels to build routers/services/middlewares as in compose; map container port to published host port (from `ports:`) to form upstream URLs with host address.
4) Emit file-provider YAML to the watched directory (recommended default: `/mnt/data/traefik/dynamic.d/compose-farm.generated.yml`, but user chooses via `--output`).
5) Warnings: if no published port is found, warn that cross-host reachability requires L3 reachability to container IPs.
6) Tests: label parsing, port mapping, YAML render; scenario with published port; scenario without published port.
7) Docs: update README/CLAUDE to describe directory provider flags and the generator workflow; note that compose files remain unchanged.
## Open Questions
- How to derive target host address: use `hosts.<name>.address` verbatim, or allow override per service? (Default: use host address.)
- Should we support multiple hosts/backends per service for LB/HA? (Start with single server.)
- Where to store generated file by default? (Default to user-specified `--output`; maybe fallback to `./compose-farm-traefik.yml`.)

454
README.md
View File

@@ -1,57 +1,163 @@
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
- [Compose Farm](#compose-farm)
- [Why Compose Farm?](#why-compose-farm)
- [Key Assumption: Shared Storage](#key-assumption-shared-storage)
- [Installation](#installation)
- [Configuration](#configuration)
- [Usage](#usage)
- [Traefik Multihost Ingress (File Provider)](#traefik-multihost-ingress-file-provider)
- [Requirements](#requirements)
- [How It Works](#how-it-works)
- [License](#license)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
# Compose Farm
[![PyPI](https://img.shields.io/pypi/v/compose-farm)](https://pypi.org/project/compose-farm/)
[![Python](https://img.shields.io/pypi/pyversions/compose-farm)](https://pypi.org/project/compose-farm/)
[![License](https://img.shields.io/github/license/basnijholt/compose-farm)](LICENSE)
[![GitHub stars](https://img.shields.io/github/stars/basnijholt/compose-farm)](https://github.com/basnijholt/compose-farm/stargazers)
<img src="http://files.nijho.lt/compose-farm.png" align="right" style="width: 300px;" />
A minimal CLI tool to run Docker Compose commands across multiple hosts via SSH.
> [!NOTE]
> Run `docker compose` commands across multiple hosts via SSH. One YAML maps services to hosts. Run `cf apply` and reality matches your config—services start, migrate, or stop as needed. No Kubernetes, no Swarm, no magic.
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
- [Why Compose Farm?](#why-compose-farm)
- [How It Works](#how-it-works)
- [Requirements](#requirements)
- [Limitations & Best Practices](#limitations--best-practices)
- [What breaks when you move a service](#what-breaks-when-you-move-a-service)
- [Best practices](#best-practices)
- [What Compose Farm doesn't do](#what-compose-farm-doesnt-do)
- [Installation](#installation)
- [Configuration](#configuration)
- [Multi-Host Services](#multi-host-services)
- [Config Command](#config-command)
- [Usage](#usage)
- [Auto-Migration](#auto-migration)
- [Traefik Multihost Ingress (File Provider)](#traefik-multihost-ingress-file-provider)
- [Comparison with Alternatives](#comparison-with-alternatives)
- [License](#license)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## Why Compose Farm?
I run 100+ Docker Compose stacks on an LXC container that frequently runs out of memory. I needed a way to distribute services across multiple machines without the complexity of:
I used to run 100+ Docker Compose stacks on a single machine that kept running out of memory. I needed a way to distribute services across multiple machines without the complexity of:
- **Kubernetes**: Overkill for my use case. I don't need pods, services, ingress controllers, or YAML manifests 10x the size of my compose files.
- **Docker Swarm**: Effectively in maintenance mode—no longer being invested in by Docker.
**Compose Farm is intentionally simple**: one YAML config mapping services to hosts, and a CLI that runs `docker compose` commands over SSH. That's it.
Both require changes to your compose files. **Compose Farm requires zero changes**—your existing `docker-compose.yml` files work as-is.
## Key Assumption: Shared Storage
I also wanted a declarative setup—one config file that defines where everything runs. Change the config, run `cf apply`, and everything reconciles—services start, migrate, or stop as needed. See [Comparison with Alternatives](#comparison-with-alternatives) for how this compares to other approaches.
Compose Farm assumes **all your compose files are accessible at the same path on all hosts**. This is typically achieved via:
<p align="center">
<a href="https://xkcd.com/927/">
<img src="https://imgs.xkcd.com/comics/standards.png" alt="xkcd: Standards" width="400" />
</a>
</p>
Before you say it—no, this is not a new standard. I changed nothing about my existing setup. When I added more hosts, I just mounted my drives at the same paths, and everything worked. You can do all of this manually today—SSH into a host and run `docker compose up`.
Compose Farm just automates what you'd do by hand:
- Runs `docker compose` commands over SSH
- Tracks which service runs on which host
- **One command (`cf apply`) to reconcile everything**—start missing services, migrate moved ones, stop removed ones
- Generates Traefik file-provider config for cross-host routing
**It's a convenience wrapper, not a new paradigm.**
## How It Works
**The declarative way** — run `cf apply` and reality matches your config:
1. Compose Farm compares your config to what's actually running
2. Services in config but not running? **Starts them**
3. Services on the wrong host? **Migrates them** (stops on old host, starts on new)
4. Services running but removed from config? **Stops them**
**Under the hood** — each service operation is just SSH + docker compose:
1. Look up which host runs the service (e.g., `plex``server-1`)
2. SSH to `server-1` (or run locally if `localhost`)
3. Execute `docker compose -f /opt/compose/plex/docker-compose.yml up -d`
4. Stream output back with `[plex]` prefix
That's it. No orchestration, no service discovery, no magic.
## Requirements
- Python 3.11+ (we recommend [uv](https://docs.astral.sh/uv/) for installation)
- SSH key-based authentication to your hosts (uses ssh-agent)
- Docker and Docker Compose installed on all target hosts
- **Shared storage**: All compose files must be accessible at the same path on all hosts
- **Docker networks**: External networks must exist on all hosts (use `cf init-network` to create)
Compose Farm assumes your compose files are accessible at the same path on all hosts. This is typically achieved via:
- **NFS mount** (e.g., `/opt/compose` mounted from a NAS)
- **Synced folders** (e.g., Syncthing, rsync)
- **Shared filesystem** (e.g., GlusterFS, Ceph)
```
# Example: NFS mount on all hosts
nas:/volume1/compose → /opt/compose (on nas01)
nas:/volume1/compose → /opt/compose (on nas02)
nas:/volume1/compose → /opt/compose (on nas03)
# Example: NFS mount on all Docker hosts
nas:/volume1/compose → /opt/compose (on server-1)
nas:/volume1/compose → /opt/compose (on server-2)
nas:/volume1/compose → /opt/compose (on server-3)
```
Compose Farm simply runs `docker compose -f /opt/compose/{service}/docker-compose.yml` on the appropriate host—it doesn't copy or sync files.
## Limitations & Best Practices
Compose Farm moves containers between hosts but **does not provide cross-host networking**. Docker's internal DNS and networks don't span hosts.
### What breaks when you move a service
- **Docker DNS** - `http://redis:6379` won't resolve from another host
- **Docker networks** - Containers can't reach each other via network names
- **Environment variables** - `DATABASE_URL=postgres://db:5432` stops working
### Best practices
1. **Keep dependent services together** - If an app needs a database, redis, or worker, keep them in the same compose file on the same host
2. **Only migrate standalone services** - Services that don't talk to other containers (or only talk to external APIs) are safe to move
3. **Expose ports for cross-host communication** - If services must communicate across hosts, publish ports and use IP addresses instead of container names:
```yaml
# Instead of: DATABASE_URL=postgres://db:5432
# Use: DATABASE_URL=postgres://192.168.1.66:5432
```
This includes Traefik routing—containers need published ports for the file-provider to reach them
### What Compose Farm doesn't do
- No overlay networking (use Docker Swarm or Kubernetes for that)
- No service discovery across hosts
- No automatic dependency tracking between compose files
If you need containers on different hosts to communicate seamlessly, you need Docker Swarm, Kubernetes, or a service mesh—which adds the complexity Compose Farm is designed to avoid.
## Installation
```bash
pip install compose-farm
uv tool install compose-farm
# or
uv pip install compose-farm
pip install compose-farm
```
<details><summary>🐳 Docker</summary>
Using the provided `docker-compose.yml`:
```bash
docker compose run --rm cf up --all
```
Or directly:
```bash
docker run --rm \
-v $SSH_AUTH_SOCK:/ssh-agent -e SSH_AUTH_SOCK=/ssh-agent \
-v ./compose-farm.yaml:/root/.config/compose-farm/compose-farm.yaml:ro \
ghcr.io/basnijholt/compose-farm up --all
```
</details>
## Configuration
Create `~/.config/compose-farm/compose-farm.yaml` (or `./compose-farm.yaml` in your working directory):
@@ -60,57 +166,223 @@ Create `~/.config/compose-farm/compose-farm.yaml` (or `./compose-farm.yaml` in y
compose_dir: /opt/compose # Must be the same path on all hosts
hosts:
nas01:
server-1:
address: 192.168.1.10
user: docker
nas02:
server-2:
address: 192.168.1.11
# user defaults to current user
local: localhost # Run locally without SSH
services:
plex: nas01
jellyfin: nas02
sonarr: nas01
plex: server-1
jellyfin: server-2
sonarr: server-1
radarr: local # Runs on the machine where you invoke compose-farm
# Multi-host services (run on multiple/all hosts)
autokuma: all # Runs on ALL configured hosts
dozzle: [server-1, server-2] # Explicit list of hosts
```
Compose files are expected at `{compose_dir}/{service}/docker-compose.yml`.
Compose files are expected at `{compose_dir}/{service}/compose.yaml` (also supports `compose.yml`, `docker-compose.yml`, `docker-compose.yaml`).
### Multi-Host Services
Some services need to run on every host. This is typically required for tools that access **host-local resources** like the Docker socket (`/var/run/docker.sock`), which cannot be accessed remotely without security risks.
Common use cases:
- **AutoKuma** - auto-creates Uptime Kuma monitors from container labels (needs local Docker socket)
- **Dozzle** - real-time log viewer (needs local Docker socket)
- **Promtail/Alloy** - log shipping agents (needs local Docker socket and log files)
- **node-exporter** - Prometheus host metrics (needs access to host /proc, /sys)
This is the same pattern as Docker Swarm's `deploy.mode: global`.
Use the `all` keyword or an explicit list:
```yaml
services:
# Run on all configured hosts
autokuma: all
dozzle: all
# Run on specific hosts
node-exporter: [server-1, server-2, server-3]
```
When you run `cf up autokuma`, it starts the service on all hosts in parallel. Multi-host services:
- Are excluded from migration logic (they always run everywhere)
- Show output with `[service@host]` prefix for each host
- Track all running hosts in state
### Config Command
Compose Farm includes a `config` subcommand to help manage configuration files:
```bash
cf config init # Create a new config file with documented example
cf config show # Display current config with syntax highlighting
cf config path # Print the config file path (useful for scripting)
cf config validate # Validate config syntax and schema
cf config edit # Open config in $EDITOR
```
Use `cf config init` to get started with a fully documented template.
## Usage
```bash
# Start services
compose-farm up plex jellyfin
compose-farm up --all
The CLI is available as both `compose-farm` and the shorter `cf` alias.
# Stop services
compose-farm down plex
| Command | Description |
|---------|-------------|
| **`cf apply`** | **Make reality match config (start + migrate + stop orphans)** |
| `cf up <svc>` | Start service (auto-migrates if host changed) |
| `cf down <svc>` | Stop service |
| `cf restart <svc>` | down + up |
| `cf update <svc>` | pull + down + up |
| `cf pull <svc>` | Pull latest images |
| `cf logs -f <svc>` | Follow logs |
| `cf ps` | Show status of all services |
| `cf refresh` | Update state from running services |
| `cf check` | Validate config, mounts, networks |
| `cf init-network` | Create Docker network on hosts |
| `cf traefik-file` | Generate Traefik file-provider config |
| `cf config <cmd>` | Manage config files (init, show, path, validate, edit) |
All commands support `--all` to operate on all services.
Each command replaces: look up host → SSH → find compose file → run `ssh host "cd /opt/compose/plex && docker compose up -d"`.
```bash
# The main command: make reality match your config
cf apply # start missing + migrate + stop orphans
cf apply --dry-run # preview what would change
cf apply --no-orphans # skip stopping orphaned services
cf apply --full # also refresh all services (picks up config changes)
# Or operate on individual services
cf up plex jellyfin # start services (auto-migrates if host changed)
cf up --all
cf down plex # stop services
cf down --orphaned # stop services removed from config
# Pull latest images
compose-farm pull --all
cf pull --all
# Restart (down + up)
compose-farm restart plex
cf restart plex
# Update (pull + down + up) - the end-to-end update command
compose-farm update --all
cf update --all
# Capture image digests to a TOML log (per service or all)
compose-farm snapshot plex
compose-farm snapshot --all # writes ~/.config/compose-farm/dockerfarm-log.toml
# Update state from reality (discovers running services + captures digests)
cf refresh # updates state.yaml and dockerfarm-log.toml
cf refresh --dry-run # preview without writing
# Validate config, traefik labels, mounts, and networks
cf check # full validation (includes SSH checks)
cf check --local # fast validation (skip SSH)
cf check jellyfin # check service + show which hosts can run it
# Create Docker network on new hosts (before migrating services)
cf init-network nuc hp # create mynetwork on specific hosts
cf init-network # create on all hosts
# View logs
compose-farm logs plex
compose-farm logs -f plex # follow
cf logs plex
cf logs -f plex # follow
# Show status
compose-farm ps
cf ps
```
<details>
<summary>See the output of <code>cf --help</code></summary>
<!-- CODE:BASH:START -->
<!-- echo '```yaml' -->
<!-- export NO_COLOR=1 -->
<!-- export TERM=dumb -->
<!-- export TERMINAL_WIDTH=90 -->
<!-- cf --help -->
<!-- echo '```' -->
<!-- CODE:END -->
<!-- OUTPUT:START -->
<!-- ⚠️ This content is auto-generated by `markdown-code-runner`. -->
```yaml
Usage: cf [OPTIONS] COMMAND [ARGS]...
Compose Farm - run docker compose commands across multiple hosts
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --version -v Show version and exit │
│ --install-completion Install completion for the current shell. │
│ --show-completion Show completion for the current shell, to │
│ copy it or customize the installation. │
│ --help -h Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Lifecycle ──────────────────────────────────────────────────────────────────╮
│ up Start services (docker compose up -d). Auto-migrates if host │
│ changed. │
│ down Stop services (docker compose down). │
│ pull Pull latest images (docker compose pull). │
│ restart Restart services (down + up). │
│ update Update services (pull + build + down + up). │
│ apply Make reality match config (start, migrate, stop as needed). │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Configuration ──────────────────────────────────────────────────────────────╮
│ traefik-file Generate a Traefik file-provider fragment from compose │
│ Traefik labels. │
│ refresh Update local state from running services. │
│ check Validate configuration, traefik labels, mounts, and networks. │
│ init-network Create Docker network on hosts with consistent settings. │
│ config Manage compose-farm configuration files. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Monitoring ─────────────────────────────────────────────────────────────────╮
│ logs Show service logs. │
│ ps Show status of all services. │
│ stats Show overview statistics for hosts and services. │
╰──────────────────────────────────────────────────────────────────────────────╯
```
<!-- OUTPUT:END -->
</details>
### Auto-Migration
When you change a service's host assignment in config and run `up`, Compose Farm automatically:
1. Checks that required mounts and networks exist on the new host (aborts if missing)
2. Runs `down` on the old host
3. Runs `up -d` on the new host
4. Updates state tracking
Use `cf apply` to automatically reconcile all services—it finds and migrates services on wrong hosts, stops orphaned services, and starts missing services.
```yaml
# Before: plex runs on server-1
services:
plex: server-1
# After: change to server-2, then run `cf up plex`
services:
plex: server-2 # Compose Farm will migrate automatically
```
**Orphaned services**: When you remove (or comment out) a service from config, it becomes "orphaned"—tracked in state but no longer in config. Use these commands to handle orphans:
- `cf apply` — Migrate services AND stop orphans (the full reconcile)
- `cf down --orphaned` — Only stop orphaned services
- `cf apply --dry-run` — Preview what would change before applying
This makes the config truly declarative: comment out a service, run `cf apply`, and it stops.
## Traefik Multihost Ingress (File Provider)
If you run a single Traefik instance on one frontdoor host and want it to route to
If you run a single Traefik instance on one "frontdoor" host and want it to route to
Compose Farm services on other hosts, Compose Farm can generate a Traefik fileprovider
fragment from your existing compose labels.
@@ -119,11 +391,11 @@ fragment from your existing compose labels.
- Your `docker-compose.yml` remains the source of truth. Put normal `traefik.*` labels on
the container you want exposed.
- Labels and port specs may use `${VAR}` / `${VAR:-default}`; Compose Farm resolves these
using the stacks `.env` file and your current environment, just like Docker Compose.
using the stack's `.env` file and your current environment, just like Docker Compose.
- Publish a host port for that container (via `ports:`). The generator prefers
hostpublished ports so Traefik can reach the service across hosts; if none are found,
it warns and youd need L3 reachability to container IPs.
- If a router label doesnt specify `traefik.http.routers.<name>.service` and theres only
it warns and you'd need L3 reachability to container IPs.
- If a router label doesn't specify `traefik.http.routers.<name>.service` and there's only
one Traefik service defined on that container, Compose Farm wires the router to it.
- `compose-farm.yaml` stays unchanged: just `hosts` and `services: service → host`.
@@ -156,28 +428,82 @@ providers:
**Generate the fragment**
```bash
compose-farm traefik-file --output /mnt/data/traefik/dynamic.d/compose-farm.generated.yml
cf traefik-file --all --output /mnt/data/traefik/dynamic.d/compose-farm.yml
```
Rerun this after changing Traefik labels, moving a service to another host, or changing
published ports.
## Requirements
**Auto-regeneration**
- Python 3.11+
- SSH key-based authentication to your hosts (uses ssh-agent)
- Docker and Docker Compose installed on all target hosts
- **Shared storage**: All compose files at the same path on all hosts (NFS, Syncthing, etc.)
To automatically regenerate the Traefik config after `up`, `down`, `restart`, or `update`,
add `traefik_file` to your config:
## How It Works
```yaml
compose_dir: /opt/compose
traefik_file: /opt/traefik/dynamic.d/compose-farm.yml # auto-regenerate on up/down/restart/update
traefik_service: traefik # skip services on same host (docker provider handles them)
1. You run `compose-farm up plex`
2. Compose Farm looks up which host runs `plex` (e.g., `nas01`)
3. It SSHs to `nas01` (or runs locally if `localhost`)
4. It executes `docker compose -f /opt/compose/plex/docker-compose.yml up -d`
5. Output is streamed back with `[plex]` prefix
hosts:
# ...
services:
traefik: server-1 # Traefik runs here
plex: server-2 # Services on other hosts get file-provider entries
# ...
```
That's it. No orchestration, no service discovery, no magic.
The `traefik_service` option specifies which service runs Traefik. Services on the same host
are skipped in the file-provider config since Traefik's docker provider handles them directly.
Now `cf up plex` will update the Traefik config automatically—no separate
`traefik-file` command needed.
**Combining with existing config**
If you already have a `dynamic.yml` with manual routes, middlewares, etc., move it into the
directory and Traefik will merge all files:
```bash
mkdir -p /opt/traefik/dynamic.d
mv /opt/traefik/dynamic.yml /opt/traefik/dynamic.d/manual.yml
cf traefik-file --all -o /opt/traefik/dynamic.d/compose-farm.yml
```
Update your Traefik config to use directory watching instead of a single file:
```yaml
# Before
- --providers.file.filename=/dynamic.yml
# After
- --providers.file.directory=/dynamic.d
- --providers.file.watch=true
```
## Comparison with Alternatives
There are many ways to run containers on multiple hosts. Here is where Compose Farm sits:
| | Compose Farm | Docker Contexts | K8s / Swarm | Ansible / Terraform | Portainer / Coolify |
|---|:---:|:---:|:---:|:---:|:---:|
| No compose rewrites | ✅ | ✅ | ❌ | ✅ | ✅ |
| Version controlled | ✅ | ✅ | ✅ | ✅ | ❌ |
| State tracking | ✅ | ❌ | ✅ | ✅ | ✅ |
| Auto-migration | ✅ | ❌ | ✅ | ❌ | ❌ |
| Interactive CLI | ✅ | ❌ | ❌ | ❌ | ❌ |
| Parallel execution | ✅ | ❌ | ✅ | ✅ | ✅ |
| Agentless | ✅ | ✅ | ❌ | ✅ | ❌ |
| High availability | ❌ | ❌ | ✅ | ❌ | ❌ |
**Docker Contexts** — You can use `docker context create remote ssh://...` and `docker compose --context remote up`. But it's manual: you must remember which host runs which service, there's no global view, no parallel execution, and no auto-migration.
**Kubernetes / Docker Swarm** — Full orchestration that abstracts away the hardware. But they require cluster initialization, separate control planes, and often rewriting compose files. They introduce complexity (consensus, overlay networks) unnecessary for static "pet" servers.
**Ansible / Terraform** — Infrastructure-as-Code tools that can SSH in and deploy containers. But they're push-based configuration management, not interactive CLIs. Great for setting up state, clumsy for day-to-day operations like `cf logs -f` or quickly restarting a service.
**Portainer / Coolify** — Web-based management UIs. But they're UI-first and often require agents on your servers. Compose Farm is CLI-first and agentless.
**Compose Farm is the middle ground:** a robust CLI that productizes the manual SSH pattern. You get the "cluster feel" (unified commands, state tracking) without the "cluster cost" (complexity, agents, control planes).
## License

View File

@@ -3,23 +3,28 @@
compose_dir: /opt/compose
# Optional: Auto-regenerate Traefik file-provider config after up/down/restart/update
traefik_file: /opt/traefik/dynamic.d/compose-farm.yml
traefik_service: traefik # Skip services on same host (docker provider handles them)
hosts:
# Full form with all options
nas01:
server-1:
address: 192.168.1.10
user: docker
port: 22
# Short form (just address, user defaults to current user)
nas02: 192.168.1.11
server-2: 192.168.1.11
# Local execution (no SSH)
local: localhost
services:
# Map service names to hosts
# Compose file expected at: {compose_dir}/{service}/docker-compose.yml
plex: nas01
jellyfin: nas02
sonarr: nas01
radarr: nas02
# Compose file expected at: {compose_dir}/{service}/compose.yaml
traefik: server-1 # Traefik runs here
plex: server-2 # Services on other hosts get file-provider entries
jellyfin: server-2
sonarr: server-1
radarr: local

11
docker-compose.yml Normal file
View File

@@ -0,0 +1,11 @@
services:
cf:
image: ghcr.io/basnijholt/compose-farm:latest
volumes:
- ${SSH_AUTH_SOCK}:/ssh-agent:ro
# Compose directory (contains compose files AND compose-farm.yaml config)
- ${CF_COMPOSE_DIR:-/opt/compose}:${CF_COMPOSE_DIR:-/opt/compose}
environment:
- SSH_AUTH_SOCK=/ssh-agent
# Config file path (state stored alongside it)
- CF_CONFIG=${CF_COMPOSE_DIR:-/opt/compose}/compose-farm.yaml

View File

@@ -0,0 +1,90 @@
# Docker Swarm Overlay Networks with Compose Farm
Notes from testing Docker Swarm's attachable overlay networks as a way to get cross-host container networking while still using `docker compose`.
## The Idea
Docker Swarm overlay networks can be made "attachable", allowing regular `docker compose` containers (not just swarm services) to join them. This would give us:
- Cross-host Docker DNS (containers find each other by name)
- No need to publish ports for inter-container communication
- Keep using `docker compose up` instead of `docker stack deploy`
## Setup Steps
```bash
# On manager node
docker swarm init --advertise-addr <manager-ip>
# On worker nodes (use token from init output)
docker swarm join --token <token> <manager-ip>:2377
# Create attachable overlay network (on manager)
docker network create --driver overlay --attachable my-network
# In compose files, add the network
networks:
my-network:
external: true
```
## Required Ports
Docker Swarm requires these ports open **bidirectionally** between all nodes:
| Port | Protocol | Purpose |
|------|----------|---------|
| 2377 | TCP | Cluster management |
| 7946 | TCP + UDP | Node communication |
| 4789 | UDP | Overlay network traffic (VXLAN) |
## Test Results (2024-12-13)
- docker-debian (192.168.1.66) as manager
- dev-lxc (192.168.1.167) as worker
### What worked
- Swarm init and join
- Overlay network creation
- Nodes showed as Ready
### What failed
- Container on dev-lxc couldn't attach to overlay network
- Error: `attaching to network failed... context deadline exceeded`
- Cause: Port 7946 blocked from docker-debian → dev-lxc
### Root cause
Firewall on dev-lxc wasn't configured to allow swarm ports. Opening these ports requires sudo access on each node.
## Conclusion
Docker Swarm overlay networks are **not plug-and-play**. Requirements:
1. Swarm init/join on all nodes
2. Firewall rules on all nodes (needs sudo/root)
3. All nodes must have bidirectional connectivity on 3 ports
For a simpler alternative, consider:
- **Tailscale**: VPN mesh, containers use host's Tailscale IP
- **Host networking + published ports**: What compose-farm does today
- **Keep dependent services together**: Avoid cross-host networking entirely
## Future Work
If we decide to support overlay networks:
1. Add a `compose-farm network create` command that:
- Initializes swarm if needed
- Creates attachable overlay network
- Documents required firewall rules
2. Add network config to compose-farm.yaml:
```yaml
overlay_network: compose-farm-net
```
3. Auto-inject network into compose files (or document manual setup)

View File

@@ -0,0 +1,128 @@
# Future Improvements
Low-priority improvements identified during code review. These are not currently causing issues but could be addressed if they become pain points.
## 1. State Module Efficiency (LOW)
**Current:** Every state operation reads and writes the entire file.
```python
def set_service_host(config, service, host):
state = load_state(config) # Read file
state[service] = host
save_state(config, state) # Write file
```
**Impact:** With 87 services, this is fine. With 1000+, it would be slow.
**Potential fix:** Add batch operations:
```python
def update_state(config, updates: dict[str, str | None]) -> None:
"""Batch update: set services to hosts, None means remove."""
state = load_state(config)
for service, host in updates.items():
if host is None:
state.pop(service, None)
else:
state[service] = host
save_state(config, state)
```
**When to do:** Only if state operations become noticeably slow.
---
## 2. Remote-Aware Compose Path Resolution (LOW)
**Current:** `config.get_compose_path()` checks if files exist on the local filesystem:
```python
def get_compose_path(self, service: str) -> Path:
for filename in ("compose.yaml", "compose.yml", ...):
candidate = service_dir / filename
if candidate.exists(): # Local check!
return candidate
```
**Why this works:** NFS/shared storage means local = remote.
**Why it could break:** If running compose-farm from a machine without the NFS mount, it returns `compose.yaml` (the default) even if `docker-compose.yml` exists on the remote host.
**Potential fix:** Query the remote host for file existence, or accept this limitation and document it.
**When to do:** Only if users need to run compose-farm from non-NFS machines.
---
## 3. Add Integration Tests for CLI Commands (MEDIUM)
**Current:** No integration tests for the actual CLI commands. Tests cover the underlying functions but not the Typer commands themselves.
**Potential fix:** Add integration tests using `CliRunner` from Typer:
```python
from typer.testing import CliRunner
from compose_farm.cli import app
runner = CliRunner()
def test_check_command_validates_config():
result = runner.invoke(app, ["check", "--local"])
assert result.exit_code == 0
```
**When to do:** When CLI behavior becomes complex enough to warrant dedicated testing.
---
## 4. Add Tests for operations.py (MEDIUM)
**Current:** Operations module has 30% coverage. Most logic is tested indirectly through test_sync.py.
**Potential fix:** Add dedicated tests for:
- `up_services()` with migration scenarios
- `preflight_check()`
- `check_host_compatibility()`
**When to do:** When adding new operations or modifying migration logic.
---
## 5. Consider Structured Logging (LOW)
**Current:** Operations print directly to console using Rich. This couples the operations module to the Rich library.
**Potential fix:** Use Python's logging module with a custom Rich handler:
```python
import logging
logger = logging.getLogger(__name__)
# In operations:
logger.info("Migrating %s from %s to %s", service, old_host, new_host)
# In cli.py - configure Rich handler:
from rich.logging import RichHandler
logging.basicConfig(handlers=[RichHandler()])
```
**Benefits:**
- Operations become testable without capturing stdout
- Logs can be redirected to files
- Log levels provide filtering
**When to do:** Only if console output coupling becomes a problem for testing or extensibility.
---
## Design Decisions to Keep
These patterns are working well and should be preserved:
1. **asyncio + asyncssh** - Solid async foundation
2. **Pydantic models** - Clean validation
3. **Rich for output** - Good UX
4. **Test structure** - Good coverage
5. **Module separation** - cli/operations/executor/compose pattern
6. **KISS principle** - Don't over-engineer

169
docs/truenas-nested-nfs.md Normal file
View File

@@ -0,0 +1,169 @@
# TrueNAS NFS: Accessing Child ZFS Datasets
When NFS-exporting a parent ZFS dataset on TrueNAS, child datasets appear as **empty directories** to NFS clients. This document explains the problem and provides a workaround.
## The Problem
TrueNAS structures storage as ZFS datasets. A common pattern is:
```
tank/data <- parent dataset (NFS exported)
tank/data/app1 <- child dataset
tank/data/app2 <- child dataset
```
When you create an NFS share for `tank/data`, clients mount it and see the `app1/` and `app2/` directories—but they're empty. This happens because each ZFS dataset is a separate filesystem, and NFS doesn't traverse into child filesystems by default.
## The Solution: `crossmnt`
The NFS `crossmnt` export option tells the server to allow clients to traverse into child filesystems. However, TrueNAS doesn't expose this option in the UI.
### Workaround Script
This Python script injects `crossmnt` into `/etc/exports`:
```python
#!/usr/bin/env python3
"""
Add crossmnt to TrueNAS NFS exports for child dataset visibility.
Usage: fix-nfs-crossmnt.py /mnt/pool/dataset
Setup:
1. scp fix-nfs-crossmnt.py root@truenas.local:/root/
2. chmod +x /root/fix-nfs-crossmnt.py
3. Test: /root/fix-nfs-crossmnt.py /mnt/pool/dataset
4. Add cron job: TrueNAS UI > System > Advanced > Cron Jobs
Command: /root/fix-nfs-crossmnt.py /mnt/pool/dataset
Schedule: */5 * * * *
"""
import re
import subprocess
import sys
from pathlib import Path
EXPORTS_FILE = Path("/etc/exports")
def main():
if len(sys.argv) != 2:
print(f"Usage: {sys.argv[0]} /mnt/pool/dataset", file=sys.stderr)
return 1
export_path = sys.argv[1]
content = EXPORTS_FILE.read_text()
if f'"{export_path}"' not in content:
print(f"ERROR: {export_path} not found in {EXPORTS_FILE}", file=sys.stderr)
return 1
lines = content.splitlines()
result = []
in_block = False
modified = False
for line in lines:
if f'"{export_path}"' in line:
in_block = True
elif line.startswith('"'):
in_block = False
if in_block and line[:1] in (" ", "\t") and "crossmnt" not in line:
line = re.sub(r"\)(\\\s*)?$", r",crossmnt)\1", line)
modified = True
result.append(line)
if not modified:
return 0 # Already applied
EXPORTS_FILE.write_text("\n".join(result) + "\n")
subprocess.run(["exportfs", "-ra"], check=True)
print(f"Added crossmnt to {export_path}")
return 0
if __name__ == "__main__":
sys.exit(main())
```
## Setup Instructions
### 1. Copy the script to TrueNAS
```bash
scp fix-nfs-crossmnt.py root@truenas.local:/root/
ssh root@truenas.local chmod +x /root/fix-nfs-crossmnt.py
```
### 2. Test manually
```bash
ssh root@truenas.local
# Run the script
/root/fix-nfs-crossmnt.py /mnt/tank/data
# Verify crossmnt was added
cat /etc/exports
```
You should see `,crossmnt` added to the client options:
```
"/mnt/tank/data"\
192.168.1.10(sec=sys,rw,no_subtree_check,crossmnt)\
192.168.1.11(sec=sys,rw,no_subtree_check,crossmnt)
```
### 3. Verify on NFS client
```bash
# Before: empty directory
ls /mnt/data/app1/
# (nothing)
# After: actual contents visible
ls /mnt/data/app1/
# config.yaml data/ logs/
```
### 4. Make it persistent
TrueNAS regenerates `/etc/exports` when you modify NFS shares in the UI. To survive this, set up a cron job:
1. Go to **TrueNAS UI → System → Advanced → Cron Jobs → Add**
2. Configure:
- **Description:** Fix NFS crossmnt
- **Command:** `/root/fix-nfs-crossmnt.py /mnt/tank/data`
- **Run As User:** root
- **Schedule:** `*/5 * * * *` (every 5 minutes)
- **Enabled:** checked
3. Save
The script is idempotent—it only modifies the file if `crossmnt` is missing, and skips the write entirely if already applied.
## How It Works
1. Parses `/etc/exports` to find the specified export block
2. Adds `,crossmnt` before the closing `)` on each client line
3. Writes the file only if changes were made
4. Runs `exportfs -ra` to reload the NFS configuration
## Why Not Use SMB Instead?
SMB handles child datasets seamlessly, but:
- NFS is simpler for Linux-to-Linux with matching UIDs
- SMB requires more complex permission mapping for Docker volumes
- Many existing setups already use NFS
## Related Links
- [TrueNAS Forum: Add crossmnt option to NFS exports](https://forums.truenas.com/t/add-crossmnt-option-to-nfs-exports/10573)
- [exports(5) man page](https://man7.org/linux/man-pages/man5/exports.5.html) - see `crossmnt` option
## Tested On
- TrueNAS SCALE 24.10

View File

@@ -0,0 +1,65 @@
# TrueNAS NFS: Disabling Root Squash
When running Docker containers on NFS-mounted storage, containers that run as root will fail to write files unless root squash is disabled. This document explains the problem and solution.
## The Problem
By default, NFS uses "root squash" which maps the root user (UID 0) on clients to `nobody` on the server. This is a security feature to prevent remote root users from having root access to the NFS server's files.
However, many Docker containers run as root internally. When these containers try to write to NFS-mounted volumes, the writes fail with "Permission denied" because the NFS server sees them as `nobody`, not `root`.
Example error in container logs:
```
System.UnauthorizedAccessException: Access to the path '/data' is denied.
Error: EACCES: permission denied, mkdir '/app/data'
```
## The Solution
In TrueNAS, configure the NFS share to map remote root to local root:
### TrueNAS SCALE UI
1. Go to **Shares → NFS**
2. Edit your share
3. Under **Advanced Options**:
- **Maproot User**: `root`
- **Maproot Group**: `wheel`
4. Save
### Result in /etc/exports
```
"/mnt/pool/data"\
192.168.1.25(sec=sys,rw,no_root_squash,no_subtree_check)\
192.168.1.26(sec=sys,rw,no_root_squash,no_subtree_check)
```
The `no_root_squash` option means remote root is treated as root on the server.
## Why `wheel`?
On FreeBSD/TrueNAS, the root user's primary group is `wheel` (GID 0), not `root` like on Linux. So `root:wheel` = `0:0`.
## Security Considerations
Disabling root squash means any machine that can mount the NFS share has full root access to those files. This is acceptable when:
- The NFS clients are on a trusted private network
- Only known hosts (by IP) are allowed to mount the share
- The data isn't security-critical
For home lab setups with Docker containers, this is typically fine.
## Alternative: Run Containers as Non-Root
If you prefer to keep root squash enabled, you can run containers as a non-root user:
1. **LinuxServer.io images**: Set `PUID=1000` and `PGID=1000` environment variables
2. **Other images**: Add `user: "1000:1000"` to the compose service
However, not all containers support running as non-root (they may need to bind to privileged ports, create system directories, etc.).
## Tested On
- TrueNAS SCALE 24.10

View File

@@ -1,42 +1,171 @@
# Compose Farm Examples
This folder contains example Docker Compose services for testing Compose Farm locally.
Real-world examples demonstrating compose-farm patterns for multi-host Docker deployments.
## Services
| Service | Type | Demonstrates |
|---------|------|--------------|
| [traefik](traefik/) | Infrastructure | Reverse proxy, Let's Encrypt, file-provider |
| [mealie](mealie/) | Single container | Traefik labels, resource limits, environment vars |
| [uptime-kuma](uptime-kuma/) | Single container | Docker socket, user mapping, custom DNS |
| [paperless-ngx](paperless-ngx/) | Multi-container | Redis + App stack (SQLite) |
| [autokuma](autokuma/) | Multi-host | Demonstrates `all` keyword (runs on every host) |
## Key Patterns
### External Network
All services connect to a shared external network for inter-service communication:
```yaml
networks:
mynetwork:
external: true
```
Create it on each host with consistent settings:
```bash
compose-farm init-network --network mynetwork --subnet 172.20.0.0/16
```
### Traefik Labels (Dual Routes)
Services expose two routes for different access patterns:
1. **HTTPS route** (`websecure` entrypoint): For your custom domain with Let's Encrypt TLS
2. **HTTP route** (`web` entrypoint): For `.local` domains on your LAN (no TLS needed)
This pattern allows accessing services via:
- `https://mealie.example.com` - from anywhere, with TLS
- `http://mealie.local` - from your local network, no TLS overhead
```yaml
labels:
# HTTPS route for custom domain (e.g., mealie.example.com)
- traefik.enable=true
- traefik.http.routers.myapp.rule=Host(`myapp.${DOMAIN}`)
- traefik.http.routers.myapp.entrypoints=websecure
- traefik.http.services.myapp.loadbalancer.server.port=8080
# HTTP route for .local domain (e.g., myapp.local)
- traefik.http.routers.myapp-local.rule=Host(`myapp.local`)
- traefik.http.routers.myapp-local.entrypoints=web
```
> **Note:** `.local` domains require local DNS (e.g., Pi-hole, Technitium) to resolve to your Traefik host.
### Environment Variables
Each service has a `.env` file for secrets and domain configuration.
Edit these files to set your domain and credentials:
```bash
# Example: set your domain
echo "DOMAIN=example.com" > mealie/.env
```
Variables like `${DOMAIN}` are substituted at runtime by Docker Compose.
### NFS Volume Mounts
All data is stored on shared NFS storage at `/mnt/data/`:
```yaml
volumes:
- /mnt/data/myapp:/app/data
```
This allows services to migrate between hosts without data loss.
### Multi-Host Services
Services that need to run on every host (e.g., monitoring agents):
```yaml
# In compose-farm.yaml
services:
autokuma: all # Runs on every configured host
```
### Multi-Container Stacks
Database-backed apps with multiple services:
```yaml
services:
redis:
image: redis:7
app:
depends_on:
- redis
```
> **NFS + PostgreSQL Warning:** PostgreSQL should NOT run on NFS storage due to
> fsync and file locking issues. Use SQLite (safe for single-writer on NFS) or
> keep PostgreSQL data on local volumes (non-migratable).
### AutoKuma Labels (Optional)
The autokuma example demonstrates compose-farm's **multi-host feature** - running the same service on all hosts using the `all` keyword. AutoKuma itself is not part of compose-farm; it's just a good example because it needs to run on every host to monitor local Docker containers.
[AutoKuma](https://github.com/BigBoot/AutoKuma) automatically creates Uptime Kuma monitors from Docker labels:
```yaml
labels:
- kuma.myapp.http.name=My App
- kuma.myapp.http.url=https://myapp.${DOMAIN}
```
## Quick Start
```bash
cd examples
# Check status of all services
# 1. Create the shared network on all hosts
compose-farm init-network
# 2. Start Traefik first (the reverse proxy)
compose-farm up traefik
# 3. Start other services
compose-farm up mealie uptime-kuma
# 4. Check status
compose-farm ps
# Pull images
compose-farm pull --all
# 5. Generate Traefik file-provider config for cross-host routing
compose-farm traefik-file --all
# Start hello-world (runs and exits)
compose-farm up hello
# 6. View logs
compose-farm logs mealie
# Start nginx (stays running)
compose-farm up nginx
# Check nginx is running
curl localhost:8080
# View logs
compose-farm logs nginx
# Stop nginx
compose-farm down nginx
# Update all (pull + restart)
compose-farm update --all
# 7. Stop everything
compose-farm down --all
```
## Services
## Configuration
- **hello**: Simple hello-world container (exits immediately)
- **nginx**: Nginx web server on port 8080
The `compose-farm.yaml` shows a multi-host setup:
## Config
- **primary** (192.168.1.10): Runs Traefik and heavy services
- **secondary** (192.168.1.11): Runs lighter services
- **autokuma**: Runs on ALL hosts to monitor local containers
The `compose-farm.yaml` in this directory configures both services to run locally (no SSH).
When Traefik runs on `primary` and a service runs on `secondary`, compose-farm
automatically generates file-provider config so Traefik can route to it.
## Traefik File-Provider
When services run on different hosts than Traefik, use `traefik-file` to generate routing config:
```bash
# Generate config for all services
compose-farm traefik-file --all -o traefik/dynamic.d/compose-farm.yml
# Or configure auto-generation in compose-farm.yaml:
traefik_file: /opt/stacks/traefik/dynamic.d/compose-farm.yml
traefik_service: traefik
```
With `traefik_file` configured, compose-farm automatically regenerates the config after `up`, `down`, `restart`, and `update` commands.

4
examples/autokuma/.env Normal file
View File

@@ -0,0 +1,4 @@
# Copy to .env and fill in your values
DOMAIN=example.com
UPTIME_KUMA_USERNAME=admin
UPTIME_KUMA_PASSWORD=your-uptime-kuma-password

View File

@@ -0,0 +1,31 @@
# AutoKuma - Automatic Uptime Kuma monitor creation from Docker labels
# Demonstrates: Multi-host service (runs on ALL hosts)
#
# This service monitors Docker containers on each host and automatically
# creates Uptime Kuma monitors based on container labels.
#
# In compose-farm.yaml, configure as:
# autokuma: all
#
# This runs the same container on every host, so each host's local
# Docker socket is monitored.
name: autokuma
services:
autokuma:
image: ghcr.io/bigboot/autokuma:latest
container_name: autokuma
restart: unless-stopped
environment:
# Connect to your Uptime Kuma instance
AUTOKUMA__KUMA__URL: https://uptime.${DOMAIN}
AUTOKUMA__KUMA__USERNAME: ${UPTIME_KUMA_USERNAME}
AUTOKUMA__KUMA__PASSWORD: ${UPTIME_KUMA_PASSWORD}
# Tag for auto-created monitors
AUTOKUMA__TAG__NAME: autokuma
AUTOKUMA__TAG__COLOR: "#10B981"
volumes:
# Access local Docker socket to discover containers
- /var/run/docker.sock:/var/run/docker.sock:ro
# Custom DNS for resolving internal domains
dns:
- 192.168.1.1 # Your local DNS server

View File

@@ -0,0 +1,9 @@
deployed:
autokuma:
- primary
- secondary
- local
mealie: secondary
paperless-ngx: primary
traefik: primary
uptime-kuma: secondary

View File

@@ -1,11 +1,40 @@
# Example Compose Farm config for local testing
# Run from the examples directory: cd examples && compose-farm ps
# Example Compose Farm configuration
# Demonstrates a multi-host setup with NFS shared storage
#
# To test locally: Update the host addresses and run from the examples directory
compose_dir: .
compose_dir: /opt/stacks/compose-farm/examples
# Auto-regenerate Traefik file-provider config after up/down/restart/update
traefik_file: /opt/stacks/compose-farm/examples/traefik/dynamic.d/compose-farm.yml
traefik_service: traefik # Skip Traefik's host in file-provider (docker provider handles it)
hosts:
# Primary server - runs Traefik and most services
# Full form with all options
primary:
address: 192.168.1.10
user: deploy
port: 22
# Secondary server - runs some services for load distribution
# Short form (user defaults to current user, port defaults to 22)
secondary: 192.168.1.11
# Local execution (no SSH) - for testing or when running on the host itself
local: localhost
services:
hello: local
nginx: local
# Infrastructure (runs on primary where Traefik is)
traefik: primary
# Multi-host services (runs on ALL hosts)
# AutoKuma monitors Docker containers on each host
autokuma: all
# Primary server services
paperless-ngx: primary
# Secondary server services (distributed for performance)
mealie: secondary
uptime-kuma: secondary

View File

@@ -1,4 +0,0 @@
services:
hello:
image: hello-world
container_name: sdc-hello

2
examples/mealie/.env Normal file
View File

@@ -0,0 +1,2 @@
# Copy to .env and fill in your values
DOMAIN=example.com

View File

@@ -0,0 +1,47 @@
# Mealie - Recipe manager
# Simple single-container service with Traefik labels
#
# Demonstrates:
# - HTTPS route: mealie.${DOMAIN} (e.g., mealie.example.com) with Let's Encrypt
# - HTTP route: mealie.local for LAN access without TLS
# - External network, resource limits, environment variables
name: mealie
services:
mealie:
image: ghcr.io/mealie-recipes/mealie:latest
container_name: mealie
restart: unless-stopped
networks:
- mynetwork
ports:
- "9925:9000"
deploy:
resources:
limits:
memory: 1000M
volumes:
- /mnt/data/mealie:/app/data
environment:
ALLOW_SIGNUP: "false"
PUID: 1000
PGID: 1000
TZ: America/Los_Angeles
MAX_WORKERS: 1
WEB_CONCURRENCY: 1
BASE_URL: https://mealie.${DOMAIN}
labels:
# HTTPS route: mealie.example.com (requires DOMAIN in .env)
- traefik.enable=true
- traefik.http.routers.mealie.rule=Host(`mealie.${DOMAIN}`)
- traefik.http.routers.mealie.entrypoints=websecure
- traefik.http.services.mealie.loadbalancer.server.port=9000
# HTTP route: mealie.local (for LAN access, no TLS)
- traefik.http.routers.mealie-local.rule=Host(`mealie.local`)
- traefik.http.routers.mealie-local.entrypoints=web
# AutoKuma: automatically create Uptime Kuma monitor
- kuma.mealie.http.name=Mealie
- kuma.mealie.http.url=https://mealie.${DOMAIN}
networks:
mynetwork:
external: true

View File

@@ -1,6 +0,0 @@
services:
nginx:
image: nginx:alpine
container_name: sdc-nginx
ports:
- "8080:80"

View File

@@ -0,0 +1,3 @@
# Copy to .env and fill in your values
DOMAIN=example.com
PAPERLESS_SECRET_KEY=change-me-to-a-random-string

View File

@@ -0,0 +1,60 @@
# Paperless-ngx - Document management system
#
# Demonstrates:
# - HTTPS route: paperless.${DOMAIN} (e.g., paperless.example.com) with Let's Encrypt
# - HTTP route: paperless.local for LAN access without TLS
# - Multi-container stack (Redis + App with SQLite)
#
# NOTE: This example uses SQLite (the default) instead of PostgreSQL.
# PostgreSQL should NOT be used with NFS storage due to fsync/locking issues.
# If you need PostgreSQL, use local volumes for the database.
name: paperless-ngx
services:
redis:
image: redis:8
container_name: paperless-redis
restart: unless-stopped
networks:
- mynetwork
volumes:
- /mnt/data/paperless/redis:/data
paperless:
image: ghcr.io/paperless-ngx/paperless-ngx:latest
container_name: paperless
restart: unless-stopped
depends_on:
- redis
networks:
- mynetwork
ports:
- "8000:8000"
volumes:
# SQLite database stored here (safe on NFS for single-writer)
- /mnt/data/paperless/data:/usr/src/paperless/data
- /mnt/data/paperless/media:/usr/src/paperless/media
- /mnt/data/paperless/export:/usr/src/paperless/export
- /mnt/data/paperless/consume:/usr/src/paperless/consume
environment:
PAPERLESS_REDIS: redis://redis:6379
PAPERLESS_URL: https://paperless.${DOMAIN}
PAPERLESS_SECRET_KEY: ${PAPERLESS_SECRET_KEY}
USERMAP_UID: 1000
USERMAP_GID: 1000
labels:
# HTTPS route: paperless.example.com (requires DOMAIN in .env)
- traefik.enable=true
- traefik.http.routers.paperless.rule=Host(`paperless.${DOMAIN}`)
- traefik.http.routers.paperless.entrypoints=websecure
- traefik.http.services.paperless.loadbalancer.server.port=8000
- traefik.docker.network=mynetwork
# HTTP route: paperless.local (for LAN access, no TLS)
- traefik.http.routers.paperless-local.rule=Host(`paperless.local`)
- traefik.http.routers.paperless-local.entrypoints=web
# AutoKuma: automatically create Uptime Kuma monitor
- kuma.paperless.http.name=Paperless
- kuma.paperless.http.url=https://paperless.${DOMAIN}
networks:
mynetwork:
external: true

5
examples/traefik/.env Normal file
View File

@@ -0,0 +1,5 @@
# Copy to .env and fill in your values
DOMAIN=example.com
ACME_EMAIL=you@example.com
CF_API_EMAIL=you@example.com
CF_API_KEY=your-cloudflare-api-key

View File

@@ -0,0 +1,58 @@
# Traefik reverse proxy with Let's Encrypt and file-provider support
# This is the foundation service - other services route through it
#
# Entrypoints:
# - web (port 80): HTTP for .local domains (no TLS needed on LAN)
# - websecure (port 443): HTTPS with Let's Encrypt for custom domains
name: traefik
services:
traefik:
image: traefik:v3.2
container_name: traefik
command:
- --api.dashboard=true
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --providers.docker.network=mynetwork
# File provider for routing to services on other hosts
- --providers.file.directory=/dynamic.d
- --providers.file.watch=true
# HTTP entrypoint for .local domains (LAN access, no TLS)
- --entrypoints.web.address=:80
# HTTPS entrypoint for custom domains (with Let's Encrypt TLS)
- --entrypoints.websecure.address=:443
- --entrypoints.websecure.asDefault=true
- --entrypoints.websecure.http.tls.certresolver=letsencrypt
# Let's Encrypt DNS challenge (using Cloudflare as example)
- --certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL}
- --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
- --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
- --certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=1.1.1.1:53
environment:
# Cloudflare API token for DNS challenge
CF_API_EMAIL: ${CF_API_EMAIL}
CF_API_KEY: ${CF_API_KEY}
restart: unless-stopped
ports:
- "80:80"
- "443:443"
- "8080:8080" # Dashboard
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /mnt/data/traefik/letsencrypt:/letsencrypt
- ./dynamic.d:/dynamic.d:ro
networks:
- mynetwork
labels:
- traefik.enable=true
# Dashboard accessible at traefik.yourdomain.com
- traefik.http.routers.traefik.rule=Host(`traefik.${DOMAIN}`)
- traefik.http.routers.traefik.entrypoints=websecure
- traefik.http.routers.traefik.service=api@internal
# AutoKuma: automatically create Uptime Kuma monitor
- kuma.traefik.http.name=Traefik
- kuma.traefik.http.url=https://traefik.${DOMAIN}
networks:
mynetwork:
external: true

View File

@@ -0,0 +1,40 @@
# Auto-generated by compose-farm
# https://github.com/basnijholt/compose-farm
#
# This file routes traffic to services running on hosts other than Traefik's host.
# Services on Traefik's host use the Docker provider directly.
#
# Regenerate with: compose-farm traefik-file --all -o <this-file>
# Or configure traefik_file in compose-farm.yaml for automatic updates.
http:
routers:
mealie:
rule: Host(`mealie.example.com`)
entrypoints:
- websecure
service: mealie
mealie-local:
rule: Host(`mealie.local`)
entrypoints:
- web
service: mealie
uptime:
rule: Host(`uptime.example.com`)
entrypoints:
- websecure
service: uptime
uptime-local:
rule: Host(`uptime.local`)
entrypoints:
- web
service: uptime
services:
mealie:
loadbalancer:
servers:
- url: http://192.168.1.11:9925
uptime:
loadbalancer:
servers:
- url: http://192.168.1.11:3001

View File

@@ -0,0 +1,2 @@
# Copy to .env and fill in your values
DOMAIN=example.com

View File

@@ -0,0 +1,43 @@
# Uptime Kuma - Monitoring dashboard
#
# Demonstrates:
# - HTTPS route: uptime.${DOMAIN} (e.g., uptime.example.com) with Let's Encrypt
# - HTTP route: uptime.local for LAN access without TLS
# - Docker socket access, user mapping for NFS, custom DNS
name: uptime-kuma
services:
uptime-kuma:
image: louislam/uptime-kuma:2
container_name: uptime-kuma
restart: unless-stopped
# Run as non-root user (important for NFS volumes)
user: "1000:1000"
networks:
- mynetwork
ports:
- "3001:3001"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /mnt/data/uptime-kuma:/app/data
environment:
PUID: 1000
PGID: 1000
# Custom DNS for internal domain resolution
dns:
- 192.168.1.1 # Your local DNS server
labels:
# HTTPS route: uptime.example.com (requires DOMAIN in .env)
- traefik.enable=true
- traefik.http.routers.uptime.rule=Host(`uptime.${DOMAIN}`)
- traefik.http.routers.uptime.entrypoints=websecure
- traefik.http.services.uptime.loadbalancer.server.port=3001
# HTTP route: uptime.local (for LAN access, no TLS)
- traefik.http.routers.uptime-local.rule=Host(`uptime.local`)
- traefik.http.routers.uptime-local.entrypoints=web
# AutoKuma: automatically create Uptime Kuma monitor
- kuma.uptime.http.name=Uptime Kuma
- kuma.uptime.http.url=https://uptime.${DOMAIN}
networks:
mynetwork:
external: true

View File

@@ -3,19 +3,61 @@ name = "compose-farm"
dynamic = ["version"]
description = "Compose Farm - run docker compose commands across multiple hosts"
readme = "README.md"
license = "MIT"
license-files = ["LICENSE"]
authors = [
{ name = "Bas Nijholt", email = "bas@nijho.lt" }
]
maintainers = [
{ name = "Bas Nijholt", email = "bas@nijho.lt" }
]
requires-python = ">=3.11"
keywords = [
"docker",
"docker-compose",
"ssh",
"devops",
"deployment",
"container",
"orchestration",
"multi-host",
"homelab",
"self-hosted",
]
classifiers = [
"Development Status :: 4 - Beta",
"Environment :: Console",
"Intended Audience :: Developers",
"Intended Audience :: System Administrators",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Topic :: System :: Systems Administration",
"Topic :: Utilities",
"Typing :: Typed",
]
dependencies = [
"typer>=0.9.0",
"pydantic>=2.0.0",
"asyncssh>=2.14.0",
"pyyaml>=6.0",
"rich>=13.0.0",
]
[project.urls]
Homepage = "https://github.com/basnijholt/compose-farm"
Repository = "https://github.com/basnijholt/compose-farm"
Documentation = "https://github.com/basnijholt/compose-farm#readme"
Issues = "https://github.com/basnijholt/compose-farm/issues"
Changelog = "https://github.com/basnijholt/compose-farm/releases"
[project.scripts]
compose-farm = "compose_farm.cli:app"
cf = "compose_farm.cli:app"
[build-system]
requires = ["hatchling", "hatch-vcs"]

81
reddit-post.md Normal file
View File

@@ -0,0 +1,81 @@
# Title options
- Multi-host Docker Compose without Kubernetes or file changes
- I built a CLI to run Docker Compose across hosts. Zero changes to your files.
---
# I made a CLI to run Docker Compose across multiple hosts without Kubernetes or Swarm
I've been running 100+ Docker Compose stacks on a single machine, and it kept running out of memory. I needed to spread services across multiple hosts, but:
- **Kubernetes** felt like overkill. I don't need pods, ingress controllers, or 10x more YAML.
- **Docker Swarm** is basically in maintenance mode.
- Both require rewriting my compose files.
So I built **Compose Farm**, a simple CLI that runs `docker compose` commands over SSH. No agents, no cluster setup, no changes to your existing compose files.
## How it works
One YAML file maps services to hosts:
```yaml
compose_dir: /opt/stacks
hosts:
nuc: 192.168.1.10
hp: 192.168.1.11
services:
plex: nuc
jellyfin: hp
sonarr: nuc
radarr: nuc
```
Then just:
```bash
cf up plex # runs on nuc via SSH
cf up --all # starts everything on their assigned hosts
cf logs -f plex # streams logs
cf ps # shows status across all hosts
```
## Auto-migration
Change a service's host in the config and run `cf up`. It stops the service on the old host and starts it on the new one. No manual SSH needed.
```yaml
# Before
plex: nuc
# After (just change this)
plex: hp
```
```bash
cf up plex # migrates automatically
```
## Requirements
- SSH key auth to your hosts
- Same paths on all hosts (I use NFS from my NAS)
- That's it. No agents, no daemons.
## What it doesn't do
- No high availability (if a host goes down, services don't auto-migrate)
- No overlay networking (containers on different hosts can't talk via Docker DNS)
- No service discovery
- No health checks or automatic restarts
It's a convenience wrapper around `docker compose` + SSH. If you need failover or cross-host container networking, you probably do need Swarm or Kubernetes.
## Links
- GitHub: https://github.com/basnijholt/compose-farm
- Install: `uv tool install compose-farm` or `pip install compose-farm`
Built this in 4 days because I was mass-SSHing into machines like a caveman. Happy to answer questions or take feedback!

View File

@@ -1,380 +0,0 @@
"""CLI interface using Typer."""
from __future__ import annotations
import asyncio
from pathlib import Path
from typing import TYPE_CHECKING, Annotated, TypeVar
import typer
import yaml
from . import __version__
from .config import Config, load_config
from .logs import snapshot_services
from .ssh import (
CommandResult,
check_service_running,
run_compose,
run_compose_on_host,
run_on_services,
run_sequential_on_services,
)
from .state import get_service_host, load_state, remove_service, save_state, set_service_host
if TYPE_CHECKING:
from collections.abc import Coroutine
T = TypeVar("T")
def _version_callback(value: bool) -> None:
"""Print version and exit."""
if value:
typer.echo(f"compose-farm {__version__}")
raise typer.Exit
app = typer.Typer(
name="compose-farm",
help="Compose Farm - run docker compose commands across multiple hosts",
no_args_is_help=True,
)
@app.callback()
def main(
version: Annotated[
bool,
typer.Option(
"--version",
"-v",
help="Show version and exit",
callback=_version_callback,
is_eager=True,
),
] = False,
) -> None:
"""Compose Farm - run docker compose commands across multiple hosts."""
def _get_services(
services: list[str],
all_services: bool,
config_path: Path | None,
) -> tuple[list[str], Config]:
"""Resolve service list and load config."""
config = load_config(config_path)
if all_services:
return list(config.services.keys()), config
if not services:
typer.echo("Error: Specify services or use --all", err=True)
raise typer.Exit(1)
return list(services), config
def _run_async(coro: Coroutine[None, None, T]) -> T:
"""Run async coroutine."""
return asyncio.run(coro)
def _report_results(results: list[CommandResult]) -> None:
"""Report command results and exit with appropriate code."""
failed = [r for r in results if not r.success]
if failed:
for r in failed:
typer.echo(f"[{r.service}] Failed with exit code {r.exit_code}", err=True)
raise typer.Exit(1)
ServicesArg = Annotated[
list[str] | None,
typer.Argument(help="Services to operate on"),
]
AllOption = Annotated[
bool,
typer.Option("--all", "-a", help="Run on all services"),
]
ConfigOption = Annotated[
Path | None,
typer.Option("--config", "-c", help="Path to config file"),
]
LogPathOption = Annotated[
Path | None,
typer.Option("--log-path", "-l", help="Path to Dockerfarm TOML log"),
]
async def _up_with_migration(
cfg: Config,
services: list[str],
) -> list[CommandResult]:
"""Start services with automatic migration if host changed."""
results: list[CommandResult] = []
for service in services:
target_host = cfg.services[service]
current_host = get_service_host(service)
# If service is deployed elsewhere, migrate it
if current_host and current_host != target_host:
if current_host in cfg.hosts:
typer.echo(f"[{service}] Migrating from {current_host} to {target_host}...")
down_result = await run_compose_on_host(cfg, service, current_host, "down")
if not down_result.success:
results.append(down_result)
continue
else:
typer.echo(
f"[{service}] Warning: was on {current_host} (not in config), skipping down",
err=True,
)
# Start on target host
up_result = await run_compose(cfg, service, "up -d")
results.append(up_result)
# Update state on success
if up_result.success:
set_service_host(service, target_host)
return results
@app.command()
def up(
services: ServicesArg = None,
all_services: AllOption = False,
config: ConfigOption = None,
) -> None:
"""Start services (docker compose up -d). Auto-migrates if host changed."""
svc_list, cfg = _get_services(services or [], all_services, config)
results = _run_async(_up_with_migration(cfg, svc_list))
_report_results(results)
@app.command()
def down(
services: ServicesArg = None,
all_services: AllOption = False,
config: ConfigOption = None,
) -> None:
"""Stop services (docker compose down)."""
svc_list, cfg = _get_services(services or [], all_services, config)
results = _run_async(run_on_services(cfg, svc_list, "down"))
# Remove from state on success
for result in results:
if result.success:
remove_service(result.service)
_report_results(results)
@app.command()
def pull(
services: ServicesArg = None,
all_services: AllOption = False,
config: ConfigOption = None,
) -> None:
"""Pull latest images (docker compose pull)."""
svc_list, cfg = _get_services(services or [], all_services, config)
results = _run_async(run_on_services(cfg, svc_list, "pull"))
_report_results(results)
@app.command()
def restart(
services: ServicesArg = None,
all_services: AllOption = False,
config: ConfigOption = None,
) -> None:
"""Restart services (down + up)."""
svc_list, cfg = _get_services(services or [], all_services, config)
results = _run_async(run_sequential_on_services(cfg, svc_list, ["down", "up -d"]))
_report_results(results)
@app.command()
def update(
services: ServicesArg = None,
all_services: AllOption = False,
config: ConfigOption = None,
) -> None:
"""Update services (pull + down + up)."""
svc_list, cfg = _get_services(services or [], all_services, config)
results = _run_async(run_sequential_on_services(cfg, svc_list, ["pull", "down", "up -d"]))
_report_results(results)
@app.command()
def logs(
services: ServicesArg = None,
all_services: AllOption = False,
follow: Annotated[bool, typer.Option("--follow", "-f", help="Follow logs")] = False,
tail: Annotated[int, typer.Option("--tail", "-n", help="Number of lines")] = 100,
config: ConfigOption = None,
) -> None:
"""Show service logs."""
svc_list, cfg = _get_services(services or [], all_services, config)
cmd = f"logs --tail {tail}"
if follow:
cmd += " -f"
results = _run_async(run_on_services(cfg, svc_list, cmd))
_report_results(results)
@app.command()
def ps(
config: ConfigOption = None,
) -> None:
"""Show status of all services."""
cfg = load_config(config)
results = _run_async(run_on_services(cfg, list(cfg.services.keys()), "ps"))
_report_results(results)
@app.command("traefik-file")
def traefik_file(
services: ServicesArg = None,
all_services: AllOption = False,
output: Annotated[
Path | None,
typer.Option(
"--output",
"-o",
help="Write Traefik file-provider YAML to this path (stdout if omitted)",
),
] = None,
config: ConfigOption = None,
) -> None:
"""Generate a Traefik file-provider fragment from compose Traefik labels."""
from .traefik import generate_traefik_config
svc_list, cfg = _get_services(services or [], all_services, config)
try:
dynamic, warnings = generate_traefik_config(cfg, svc_list)
except (FileNotFoundError, ValueError) as exc:
typer.echo(str(exc), err=True)
raise typer.Exit(1) from exc
rendered = yaml.safe_dump(dynamic, sort_keys=False)
if output:
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(rendered)
typer.echo(f"Traefik config written to {output}")
else:
typer.echo(rendered)
for warning in warnings:
typer.echo(warning, err=True)
async def _discover_running_services(cfg: Config) -> dict[str, str]:
"""Discover which services are running on which hosts.
Returns a dict mapping service names to host names for running services.
"""
discovered: dict[str, str] = {}
for service, assigned_host in cfg.services.items():
# Check assigned host first (most common case)
if await check_service_running(cfg, service, assigned_host):
discovered[service] = assigned_host
continue
# Check other hosts in case service was migrated but state is stale
for host_name in cfg.hosts:
if host_name == assigned_host:
continue
if await check_service_running(cfg, service, host_name):
discovered[service] = host_name
break
return discovered
def _report_sync_changes(
added: list[str],
removed: list[str],
changed: list[tuple[str, str, str]],
discovered: dict[str, str],
current_state: dict[str, str],
) -> None:
"""Report sync changes to the user."""
if added:
typer.echo(f"\nNew services found ({len(added)}):")
for service in sorted(added):
typer.echo(f" + {service} on {discovered[service]}")
if changed:
typer.echo(f"\nServices on different hosts ({len(changed)}):")
for service, old_host, new_host in sorted(changed):
typer.echo(f" ~ {service}: {old_host} -> {new_host}")
if removed:
typer.echo(f"\nServices no longer running ({len(removed)}):")
for service in sorted(removed):
typer.echo(f" - {service} (was on {current_state[service]})")
@app.command()
def sync(
config: ConfigOption = None,
log_path: LogPathOption = None,
dry_run: Annotated[
bool,
typer.Option("--dry-run", "-n", help="Show what would be synced without writing"),
] = False,
) -> None:
"""Sync local state with running services.
Discovers which services are running on which hosts, updates the state
file, and captures image digests. Combines service discovery with
image snapshot into a single command.
"""
cfg = load_config(config)
current_state = load_state()
typer.echo("Discovering running services...")
discovered = _run_async(_discover_running_services(cfg))
# Calculate changes
added = [s for s in discovered if s not in current_state]
removed = [s for s in current_state if s not in discovered]
changed = [
(s, current_state[s], discovered[s])
for s in discovered
if s in current_state and current_state[s] != discovered[s]
]
# Report state changes
state_changed = bool(added or removed or changed)
if state_changed:
_report_sync_changes(added, removed, changed, discovered, current_state)
else:
typer.echo("State is already in sync.")
if dry_run:
typer.echo("\n(dry-run: no changes made)")
return
# Update state file
if state_changed:
save_state(discovered)
typer.echo(f"\nState updated: {len(discovered)} services tracked.")
# Capture image digests for running services
if discovered:
typer.echo("\nCapturing image digests...")
try:
path = _run_async(snapshot_services(cfg, list(discovered.keys()), log_path=log_path))
typer.echo(f"Digests written to {path}")
except RuntimeError as exc:
typer.echo(f"Warning: {exc}", err=True)
if __name__ == "__main__":
app()

View File

@@ -0,0 +1,19 @@
"""CLI interface using Typer."""
from __future__ import annotations
# Import command modules to trigger registration via @app.command() decorators
from compose_farm.cli import (
config, # noqa: F401
lifecycle, # noqa: F401
management, # noqa: F401
monitoring, # noqa: F401
)
# Import the shared app instance
from compose_farm.cli.app import app
__all__ = ["app"]
if __name__ == "__main__":
app()

View File

@@ -0,0 +1,42 @@
"""Shared Typer app instance."""
from __future__ import annotations
from typing import Annotated
import typer
from compose_farm import __version__
__all__ = ["app"]
def _version_callback(value: bool) -> None:
"""Print version and exit."""
if value:
typer.echo(f"compose-farm {__version__}")
raise typer.Exit
app = typer.Typer(
name="compose-farm",
help="Compose Farm - run docker compose commands across multiple hosts",
no_args_is_help=True,
context_settings={"help_option_names": ["-h", "--help"]},
)
@app.callback()
def main(
version: Annotated[
bool,
typer.Option(
"--version",
"-v",
help="Show version and exit",
callback=_version_callback,
is_eager=True,
),
] = False,
) -> None:
"""Compose Farm - run docker compose commands across multiple hosts."""

View File

@@ -0,0 +1,226 @@
"""Shared CLI helpers, options, and utilities."""
from __future__ import annotations
import asyncio
import contextlib
from pathlib import Path
from typing import TYPE_CHECKING, Annotated, TypeVar
import typer
from rich.progress import (
BarColumn,
MofNCompleteColumn,
Progress,
SpinnerColumn,
TaskID,
TextColumn,
TimeElapsedColumn,
)
from compose_farm.console import console, err_console
if TYPE_CHECKING:
from collections.abc import Callable, Coroutine, Generator
from compose_farm.config import Config
from compose_farm.executor import CommandResult
_T = TypeVar("_T")
# --- Shared CLI Options ---
ServicesArg = Annotated[
list[str] | None,
typer.Argument(help="Services to operate on"),
]
AllOption = Annotated[
bool,
typer.Option("--all", "-a", help="Run on all services"),
]
ConfigOption = Annotated[
Path | None,
typer.Option("--config", "-c", help="Path to config file"),
]
LogPathOption = Annotated[
Path | None,
typer.Option("--log-path", "-l", help="Path to Dockerfarm TOML log"),
]
HostOption = Annotated[
str | None,
typer.Option("--host", "-H", help="Filter to services on this host"),
]
# --- Constants (internal) ---
_MISSING_PATH_PREVIEW_LIMIT = 2
_STATS_PREVIEW_LIMIT = 3 # Max number of pending migrations to show by name
@contextlib.contextmanager
def progress_bar(label: str, total: int) -> Generator[tuple[Progress, TaskID], None, None]:
"""Create a standardized progress bar with consistent styling.
Yields (progress, task_id). Use progress.update(task_id, advance=1, description=...)
to advance.
"""
with Progress(
SpinnerColumn(),
TextColumn(f"[bold blue]{label}[/]"),
BarColumn(),
MofNCompleteColumn(),
TextColumn(""),
TimeElapsedColumn(),
TextColumn(""),
TextColumn("[progress.description]{task.description}"),
console=console,
transient=True,
) as progress:
task_id = progress.add_task("", total=total)
yield progress, task_id
def load_config_or_exit(config_path: Path | None) -> Config:
"""Load config or exit with a friendly error message."""
# Lazy import: pydantic adds ~50ms to startup, only load when actually needed
from compose_farm.config import load_config # noqa: PLC0415
try:
return load_config(config_path)
except FileNotFoundError as e:
err_console.print(f"[red]✗[/] {e}")
raise typer.Exit(1) from e
def get_services(
services: list[str],
all_services: bool,
config_path: Path | None,
) -> tuple[list[str], Config]:
"""Resolve service list and load config."""
config = load_config_or_exit(config_path)
if all_services:
return list(config.services.keys()), config
if not services:
err_console.print("[red]✗[/] Specify services or use --all")
raise typer.Exit(1)
return list(services), config
def run_async(coro: Coroutine[None, None, _T]) -> _T:
"""Run async coroutine."""
try:
return asyncio.run(coro)
except KeyboardInterrupt:
console.print("\n[yellow]Interrupted[/]")
raise typer.Exit(130) from None # Standard exit code for SIGINT
def report_results(results: list[CommandResult]) -> None:
"""Report command results and exit with appropriate code."""
succeeded = [r for r in results if r.success]
failed = [r for r in results if not r.success]
# Always print summary when there are multiple results
if len(results) > 1:
console.print() # Blank line before summary
if failed:
for r in failed:
err_console.print(
f"[red]✗[/] [cyan]{r.service}[/] failed with exit code {r.exit_code}"
)
console.print()
console.print(
f"[green]✓[/] {len(succeeded)}/{len(results)} services succeeded, "
f"[red]✗[/] {len(failed)} failed"
)
else:
console.print(f"[green]✓[/] All {len(results)} services succeeded")
elif failed:
# Single service failed
r = failed[0]
err_console.print(f"[red]✗[/] [cyan]{r.service}[/] failed with exit code {r.exit_code}")
if failed:
raise typer.Exit(1)
def maybe_regenerate_traefik(
cfg: Config,
results: list[CommandResult] | None = None,
) -> None:
"""Regenerate traefik config if traefik_file is configured.
If results are provided, skips regeneration if all services failed.
"""
if cfg.traefik_file is None:
return
# Skip if all services failed
if results and not any(r.success for r in results):
return
# Lazy import: traefik/yaml adds startup time, only load when traefik_file is configured
from compose_farm.traefik import ( # noqa: PLC0415
generate_traefik_config,
render_traefik_config,
)
try:
dynamic, warnings = generate_traefik_config(cfg, list(cfg.services.keys()))
new_content = render_traefik_config(dynamic)
# Check if content changed
old_content = ""
if cfg.traefik_file.exists():
old_content = cfg.traefik_file.read_text()
if new_content != old_content:
cfg.traefik_file.parent.mkdir(parents=True, exist_ok=True)
cfg.traefik_file.write_text(new_content)
console.print() # Ensure we're on a new line after streaming output
console.print(f"[green]✓[/] Traefik config updated: {cfg.traefik_file}")
for warning in warnings:
err_console.print(f"[yellow]![/] {warning}")
except (FileNotFoundError, ValueError) as exc:
err_console.print(f"[yellow]![/] Failed to update traefik config: {exc}")
def validate_host_for_service(cfg: Config, service: str, host: str) -> None:
"""Validate that a host is valid for a service."""
if host not in cfg.hosts:
err_console.print(f"[red]✗[/] Host '{host}' not found in config")
raise typer.Exit(1)
allowed_hosts = cfg.get_hosts(service)
if host not in allowed_hosts:
err_console.print(
f"[red]✗[/] Service '{service}' is not configured for host '{host}' "
f"(configured: {', '.join(allowed_hosts)})"
)
raise typer.Exit(1)
def run_host_operation(
cfg: Config,
svc_list: list[str],
host: str,
command: str,
action_verb: str,
state_callback: Callable[[Config, str, str], None],
) -> None:
"""Run an operation on a specific host for multiple services."""
from compose_farm.executor import run_compose_on_host # noqa: PLC0415
results: list[CommandResult] = []
for service in svc_list:
validate_host_for_service(cfg, service, host)
console.print(f"[cyan]\\[{service}][/] {action_verb} on [magenta]{host}[/]...")
result = run_async(run_compose_on_host(cfg, service, host, command, raw=True))
print() # Newline after raw output
results.append(result)
if result.success:
state_callback(cfg, service, host)
maybe_regenerate_traefik(cfg, results)
report_results(results)

View File

@@ -0,0 +1,323 @@
"""Configuration management commands for compose-farm."""
from __future__ import annotations
import os
import platform
import shlex
import shutil
import subprocess
from importlib import resources
from pathlib import Path
from typing import Annotated
import typer
from compose_farm.cli.app import app
from compose_farm.console import console, err_console
from compose_farm.paths import config_search_paths, default_config_path
config_app = typer.Typer(
name="config",
help="Manage compose-farm configuration files.",
no_args_is_help=True,
)
# --- CLI Options (same pattern as cli.py) ---
_PathOption = Annotated[
Path | None,
typer.Option("--path", "-p", help="Path to config file. Uses auto-detection if not specified."),
]
_ForceOption = Annotated[
bool,
typer.Option("--force", "-f", help="Overwrite existing config without confirmation."),
]
_RawOption = Annotated[
bool,
typer.Option("--raw", "-r", help="Output raw file contents (for copy-paste)."),
]
def _get_editor() -> str:
"""Get the user's preferred editor.
Checks $EDITOR, then $VISUAL, then falls back to platform defaults.
"""
for env_var in ("EDITOR", "VISUAL"):
editor = os.environ.get(env_var)
if editor:
return editor
if platform.system() == "Windows":
return "notepad"
# Try common editors on Unix-like systems
for editor in ("nano", "vim", "vi"):
if shutil.which(editor):
return editor
return "vi"
def _generate_template() -> str:
"""Generate a config template with documented schema."""
try:
template_file = resources.files("compose_farm") / "example-config.yaml"
return template_file.read_text(encoding="utf-8")
except FileNotFoundError as e:
err_console.print("[red]Example config template is missing from the package.[/red]")
err_console.print("Reinstall compose-farm or report this issue.")
raise typer.Exit(1) from e
def _get_config_file(path: Path | None) -> Path | None:
"""Resolve config path, or auto-detect from standard locations."""
if path:
return path.expanduser().resolve()
# Check environment variable
if env_path := os.environ.get("CF_CONFIG"):
p = Path(env_path)
if p.exists():
return p.resolve()
# Check standard locations
for p in config_search_paths():
if p.exists():
return p.resolve()
return None
@config_app.command("init")
def config_init(
path: _PathOption = None,
force: _ForceOption = False,
) -> None:
"""Create a new config file with documented example.
The generated config file serves as a template showing all available
options with explanatory comments.
"""
target_path = (path.expanduser().resolve() if path else None) or default_config_path()
if target_path.exists() and not force:
console.print(
f"[bold yellow]Config file already exists at:[/bold yellow] [cyan]{target_path}[/cyan]",
)
if not typer.confirm("Overwrite existing config file?"):
console.print("[dim]Aborted.[/dim]")
raise typer.Exit(0)
# Create parent directories
target_path.parent.mkdir(parents=True, exist_ok=True)
# Generate and write template
template_content = _generate_template()
target_path.write_text(template_content, encoding="utf-8")
console.print(f"[green]✓[/] Config file created at: {target_path}")
console.print("\n[dim]Edit the file to customize your settings:[/dim]")
console.print(" [cyan]cf config edit[/cyan]")
@config_app.command("edit")
def config_edit(
path: _PathOption = None,
) -> None:
"""Open the config file in your default editor.
The editor is determined by: $EDITOR > $VISUAL > platform default.
"""
config_file = _get_config_file(path)
if config_file is None:
console.print("[yellow]No config file found.[/yellow]")
console.print("\nRun [bold cyan]cf config init[/bold cyan] to create one.")
console.print("\nSearched locations:")
for p in config_search_paths():
console.print(f" - {p}")
raise typer.Exit(1)
if not config_file.exists():
console.print("[yellow]Config file not found.[/yellow]")
console.print(f"\nProvided path does not exist: [cyan]{config_file}[/cyan]")
console.print("\nRun [bold cyan]cf config init[/bold cyan] to create one.")
raise typer.Exit(1)
editor = _get_editor()
console.print(f"[dim]Opening {config_file} with {editor}...[/dim]")
try:
editor_cmd = shlex.split(editor, posix=os.name != "nt")
except ValueError as e:
err_console.print("[red]Invalid editor command. Check $EDITOR/$VISUAL.[/red]")
raise typer.Exit(1) from e
if not editor_cmd:
err_console.print("[red]Editor command is empty.[/red]")
raise typer.Exit(1)
try:
subprocess.run([*editor_cmd, str(config_file)], check=True)
except FileNotFoundError:
err_console.print(f"[red]Editor '{editor_cmd[0]}' not found.[/red]")
err_console.print("Set $EDITOR environment variable to your preferred editor.")
raise typer.Exit(1) from None
except subprocess.CalledProcessError as e:
err_console.print(f"[red]Editor exited with error code {e.returncode}[/red]")
raise typer.Exit(e.returncode) from None
@config_app.command("show")
def config_show(
path: _PathOption = None,
raw: _RawOption = False,
) -> None:
"""Display the config file location and contents."""
config_file = _get_config_file(path)
if config_file is None:
console.print("[yellow]No config file found.[/yellow]")
console.print("\nSearched locations:")
for p in config_search_paths():
status = "[green]exists[/green]" if p.exists() else "[dim]not found[/dim]"
console.print(f" - {p} ({status})")
console.print("\nRun [bold cyan]cf config init[/bold cyan] to create one.")
raise typer.Exit(0)
if not config_file.exists():
console.print("[yellow]Config file not found.[/yellow]")
console.print(f"\nProvided path does not exist: [cyan]{config_file}[/cyan]")
console.print("\nRun [bold cyan]cf config init[/bold cyan] to create one.")
raise typer.Exit(1)
content = config_file.read_text(encoding="utf-8")
if raw:
print(content, end="")
return
from rich.syntax import Syntax # noqa: PLC0415
console.print(f"[bold green]Config file:[/bold green] [cyan]{config_file}[/cyan]")
console.print()
syntax = Syntax(content, "yaml", theme="monokai", line_numbers=True, word_wrap=True)
console.print(syntax)
console.print()
console.print("[dim]Tip: Use -r for copy-paste friendly output[/dim]")
@config_app.command("path")
def config_path(
path: _PathOption = None,
) -> None:
"""Print the config file path (useful for scripting)."""
config_file = _get_config_file(path)
if config_file is None:
console.print("[yellow]No config file found.[/yellow]")
console.print("\nSearched locations:")
for p in config_search_paths():
status = "[green]exists[/green]" if p.exists() else "[dim]not found[/dim]"
console.print(f" - {p} ({status})")
raise typer.Exit(1)
# Just print the path for easy piping
print(config_file)
@config_app.command("validate")
def config_validate(
path: _PathOption = None,
) -> None:
"""Validate the config file syntax and schema."""
config_file = _get_config_file(path)
if config_file is None:
err_console.print("[red]✗[/] No config file found")
raise typer.Exit(1)
# Lazy import: pydantic adds ~50ms to startup, only load when actually needed
from compose_farm.config import load_config # noqa: PLC0415
try:
cfg = load_config(config_file)
except FileNotFoundError as e:
err_console.print(f"[red]✗[/] {e}")
raise typer.Exit(1) from e
except Exception as e:
err_console.print(f"[red]✗[/] Invalid config: {e}")
raise typer.Exit(1) from e
console.print(f"[green]✓[/] Valid config: {config_file}")
console.print(f" Hosts: {len(cfg.hosts)}")
console.print(f" Services: {len(cfg.services)}")
@config_app.command("symlink")
def config_symlink(
target: Annotated[
Path | None,
typer.Argument(help="Config file to link to. Defaults to ./compose-farm.yaml"),
] = None,
force: _ForceOption = False,
) -> None:
"""Create a symlink from the default config location to a config file.
This makes a local config file discoverable globally without copying.
Always uses absolute paths to avoid broken symlinks.
Examples:
cf config symlink # Link to ./compose-farm.yaml
cf config symlink /opt/compose/config.yaml # Link to specific file
"""
# Default to compose-farm.yaml in current directory
target_path = (target or Path("compose-farm.yaml")).expanduser().resolve()
if not target_path.exists():
err_console.print(f"[red]✗[/] Target config file not found: {target_path}")
raise typer.Exit(1)
if not target_path.is_file():
err_console.print(f"[red]✗[/] Target is not a file: {target_path}")
raise typer.Exit(1)
symlink_path = default_config_path()
# Check if symlink location already exists
if symlink_path.exists() or symlink_path.is_symlink():
if symlink_path.is_symlink():
current_target = symlink_path.resolve() if symlink_path.exists() else None
if current_target == target_path:
console.print(f"[green]✓[/] Symlink already points to: {target_path}")
return
# Update existing symlink
if not force:
existing = symlink_path.readlink()
console.print(f"[yellow]Symlink exists:[/] {symlink_path} -> {existing}")
if not typer.confirm(f"Update to point to {target_path}?"):
console.print("[dim]Aborted.[/dim]")
raise typer.Exit(0)
symlink_path.unlink()
else:
# Regular file exists
err_console.print(f"[red]✗[/] A regular file exists at: {symlink_path}")
err_console.print(" Back it up or remove it first, then retry.")
raise typer.Exit(1)
# Create parent directories
symlink_path.parent.mkdir(parents=True, exist_ok=True)
# Create symlink with absolute path
symlink_path.symlink_to(target_path)
console.print("[green]✓[/] Created symlink:")
console.print(f" {symlink_path}")
console.print(f" -> {target_path}")
# Register config subcommand on the shared app
app.add_typer(config_app, name="config", rich_help_panel="Configuration")

View File

@@ -0,0 +1,299 @@
"""Lifecycle commands: up, down, pull, restart, update, apply."""
from __future__ import annotations
from typing import TYPE_CHECKING, Annotated
import typer
if TYPE_CHECKING:
from compose_farm.config import Config
from compose_farm.cli.app import app
from compose_farm.cli.common import (
AllOption,
ConfigOption,
HostOption,
ServicesArg,
get_services,
load_config_or_exit,
maybe_regenerate_traefik,
report_results,
run_async,
run_host_operation,
)
from compose_farm.console import console, err_console
from compose_farm.executor import run_on_services, run_sequential_on_services
from compose_farm.operations import stop_orphaned_services, up_services
from compose_farm.state import (
add_service_to_host,
get_orphaned_services,
get_service_host,
get_services_needing_migration,
get_services_not_in_state,
remove_service,
remove_service_from_host,
)
@app.command(rich_help_panel="Lifecycle")
def up(
services: ServicesArg = None,
all_services: AllOption = False,
host: HostOption = None,
config: ConfigOption = None,
) -> None:
"""Start services (docker compose up -d). Auto-migrates if host changed."""
svc_list, cfg = get_services(services or [], all_services, config)
# Per-host operation: run on specific host only
if host:
run_host_operation(cfg, svc_list, host, "up -d", "Starting", add_service_to_host)
return
# Normal operation: use up_services with migration logic
results = run_async(up_services(cfg, svc_list, raw=True))
maybe_regenerate_traefik(cfg, results)
report_results(results)
@app.command(rich_help_panel="Lifecycle")
def down(
services: ServicesArg = None,
all_services: AllOption = False,
orphaned: Annotated[
bool,
typer.Option(
"--orphaned", help="Stop orphaned services (in state but removed from config)"
),
] = False,
host: HostOption = None,
config: ConfigOption = None,
) -> None:
"""Stop services (docker compose down)."""
# Handle --orphaned flag
if orphaned:
if services or all_services or host:
err_console.print("[red]✗[/] Cannot use --orphaned with services, --all, or --host")
raise typer.Exit(1)
cfg = load_config_or_exit(config)
orphaned_services = get_orphaned_services(cfg)
if not orphaned_services:
console.print("[green]✓[/] No orphaned services to stop")
return
console.print(
f"[yellow]Stopping {len(orphaned_services)} orphaned service(s):[/] "
f"{', '.join(orphaned_services.keys())}"
)
results = run_async(stop_orphaned_services(cfg))
report_results(results)
return
svc_list, cfg = get_services(services or [], all_services, config)
# Per-host operation: run on specific host only
if host:
run_host_operation(cfg, svc_list, host, "down", "Stopping", remove_service_from_host)
return
# Normal operation
raw = len(svc_list) == 1
results = run_async(run_on_services(cfg, svc_list, "down", raw=raw))
# Remove from state on success
# For multi-host services, result.service is "svc@host", extract base name
removed_services: set[str] = set()
for result in results:
if result.success:
base_service = result.service.split("@")[0]
if base_service not in removed_services:
remove_service(cfg, base_service)
removed_services.add(base_service)
maybe_regenerate_traefik(cfg, results)
report_results(results)
@app.command(rich_help_panel="Lifecycle")
def pull(
services: ServicesArg = None,
all_services: AllOption = False,
config: ConfigOption = None,
) -> None:
"""Pull latest images (docker compose pull)."""
svc_list, cfg = get_services(services or [], all_services, config)
raw = len(svc_list) == 1
results = run_async(run_on_services(cfg, svc_list, "pull", raw=raw))
report_results(results)
@app.command(rich_help_panel="Lifecycle")
def restart(
services: ServicesArg = None,
all_services: AllOption = False,
config: ConfigOption = None,
) -> None:
"""Restart services (down + up)."""
svc_list, cfg = get_services(services or [], all_services, config)
raw = len(svc_list) == 1
results = run_async(run_sequential_on_services(cfg, svc_list, ["down", "up -d"], raw=raw))
maybe_regenerate_traefik(cfg, results)
report_results(results)
@app.command(rich_help_panel="Lifecycle")
def update(
services: ServicesArg = None,
all_services: AllOption = False,
config: ConfigOption = None,
) -> None:
"""Update services (pull + build + down + up)."""
svc_list, cfg = get_services(services or [], all_services, config)
raw = len(svc_list) == 1
results = run_async(
run_sequential_on_services(
cfg, svc_list, ["pull --ignore-buildable", "build", "down", "up -d"], raw=raw
)
)
maybe_regenerate_traefik(cfg, results)
report_results(results)
def _format_host(host: str | list[str]) -> str:
"""Format a host value for display."""
if isinstance(host, list):
return ", ".join(host)
return host
def _report_pending_migrations(cfg: Config, migrations: list[str]) -> None:
"""Report services that need migration."""
console.print(f"[cyan]Services to migrate ({len(migrations)}):[/]")
for svc in migrations:
current = get_service_host(cfg, svc)
target = cfg.get_hosts(svc)[0]
console.print(f" [cyan]{svc}[/]: [magenta]{current}[/] → [magenta]{target}[/]")
def _report_pending_orphans(orphaned: dict[str, str | list[str]]) -> None:
"""Report orphaned services that will be stopped."""
console.print(f"[yellow]Orphaned services to stop ({len(orphaned)}):[/]")
for svc, hosts in orphaned.items():
console.print(f" [cyan]{svc}[/] on [magenta]{_format_host(hosts)}[/]")
def _report_pending_starts(cfg: Config, missing: list[str]) -> None:
"""Report services that will be started."""
console.print(f"[green]Services to start ({len(missing)}):[/]")
for svc in missing:
target = _format_host(cfg.get_hosts(svc))
console.print(f" [cyan]{svc}[/] on [magenta]{target}[/]")
def _report_pending_refresh(cfg: Config, to_refresh: list[str]) -> None:
"""Report services that will be refreshed."""
console.print(f"[blue]Services to refresh ({len(to_refresh)}):[/]")
for svc in to_refresh:
target = _format_host(cfg.get_hosts(svc))
console.print(f" [cyan]{svc}[/] on [magenta]{target}[/]")
@app.command(rich_help_panel="Lifecycle")
def apply(
dry_run: Annotated[
bool,
typer.Option("--dry-run", "-n", help="Show what would change without executing"),
] = False,
no_orphans: Annotated[
bool,
typer.Option("--no-orphans", help="Only migrate, don't stop orphaned services"),
] = False,
full: Annotated[
bool,
typer.Option("--full", "-f", help="Also run up on all services to apply config changes"),
] = False,
config: ConfigOption = None,
) -> None:
"""Make reality match config (start, migrate, stop as needed).
This is the "reconcile" command that ensures running services match your
config file. It will:
1. Stop orphaned services (in state but removed from config)
2. Migrate services on wrong host (host in state ≠ host in config)
3. Start missing services (in config but not in state)
Use --dry-run to preview changes before applying.
Use --no-orphans to only migrate/start without stopping orphaned services.
Use --full to also run 'up' on all services (picks up compose/env changes).
"""
cfg = load_config_or_exit(config)
orphaned = get_orphaned_services(cfg)
migrations = get_services_needing_migration(cfg)
missing = get_services_not_in_state(cfg)
# For --full: refresh all services not already being started/migrated
handled = set(migrations) | set(missing)
to_refresh = [svc for svc in cfg.services if svc not in handled] if full else []
has_orphans = bool(orphaned) and not no_orphans
has_migrations = bool(migrations)
has_missing = bool(missing)
has_refresh = bool(to_refresh)
if not has_orphans and not has_migrations and not has_missing and not has_refresh:
console.print("[green]✓[/] Nothing to apply - reality matches config")
return
# Report what will be done
if has_orphans:
_report_pending_orphans(orphaned)
if has_migrations:
_report_pending_migrations(cfg, migrations)
if has_missing:
_report_pending_starts(cfg, missing)
if has_refresh:
_report_pending_refresh(cfg, to_refresh)
if dry_run:
console.print("\n[dim](dry-run: no changes made)[/]")
return
# Execute changes
console.print()
all_results = []
# 1. Stop orphaned services first
if has_orphans:
console.print("[yellow]Stopping orphaned services...[/]")
all_results.extend(run_async(stop_orphaned_services(cfg)))
# 2. Migrate services on wrong host
if has_migrations:
console.print("[cyan]Migrating services...[/]")
migrate_results = run_async(up_services(cfg, migrations, raw=True))
all_results.extend(migrate_results)
maybe_regenerate_traefik(cfg, migrate_results)
# 3. Start missing services (reuse up_services which handles state updates)
if has_missing:
console.print("[green]Starting missing services...[/]")
start_results = run_async(up_services(cfg, missing, raw=True))
all_results.extend(start_results)
maybe_regenerate_traefik(cfg, start_results)
# 4. Refresh remaining services (--full: run up to apply config changes)
if has_refresh:
console.print("[blue]Refreshing services...[/]")
refresh_results = run_async(up_services(cfg, to_refresh, raw=True))
all_results.extend(refresh_results)
maybe_regenerate_traefik(cfg, refresh_results)
report_results(all_results)
# Alias: cf a = cf apply
app.command("a", hidden=True)(apply)

View File

@@ -0,0 +1,658 @@
"""Management commands: sync, check, init-network, traefik-file."""
from __future__ import annotations
import asyncio
from datetime import UTC, datetime
from pathlib import Path # noqa: TC003
from typing import TYPE_CHECKING, Annotated
import typer
from rich.progress import Progress, TaskID # noqa: TC002
from compose_farm.cli.app import app
from compose_farm.cli.common import (
_MISSING_PATH_PREVIEW_LIMIT,
AllOption,
ConfigOption,
LogPathOption,
ServicesArg,
get_services,
load_config_or_exit,
progress_bar,
run_async,
)
if TYPE_CHECKING:
from compose_farm.config import Config
from compose_farm.console import console, err_console
from compose_farm.executor import (
CommandResult,
check_service_running,
is_local,
run_command,
)
from compose_farm.logs import (
DEFAULT_LOG_PATH,
SnapshotEntry,
collect_service_entries,
isoformat,
load_existing_entries,
merge_entries,
write_toml,
)
from compose_farm.operations import check_host_compatibility, check_service_requirements
from compose_farm.state import get_orphaned_services, load_state, save_state
from compose_farm.traefik import generate_traefik_config, render_traefik_config
# --- Sync helpers ---
def _discover_services(cfg: Config) -> dict[str, str | list[str]]:
"""Discover running services with a progress bar."""
async def check_service(service: str) -> tuple[str, str | list[str] | None]:
"""Check where a service is running.
For multi-host services, returns list of hosts where running.
For single-host, returns single host name or None.
"""
assigned_hosts = cfg.get_hosts(service)
if cfg.is_multi_host(service):
# Multi-host: find all hosts where running (check in parallel)
checks = await asyncio.gather(
*[check_service_running(cfg, service, h) for h in assigned_hosts]
)
running_hosts = [
h for h, running in zip(assigned_hosts, checks, strict=True) if running
]
return service, running_hosts if running_hosts else None
# Single-host: check assigned host first
assigned_host = assigned_hosts[0]
if await check_service_running(cfg, service, assigned_host):
return service, assigned_host
# Check other hosts
for host_name in cfg.hosts:
if host_name == assigned_host:
continue
if await check_service_running(cfg, service, host_name):
return service, host_name
return service, None
async def gather_with_progress(
progress: Progress, task_id: TaskID
) -> dict[str, str | list[str]]:
services = list(cfg.services.keys())
tasks = [asyncio.create_task(check_service(s)) for s in services]
discovered: dict[str, str | list[str]] = {}
for coro in asyncio.as_completed(tasks):
service, host = await coro
if host is not None:
discovered[service] = host
progress.update(task_id, advance=1, description=f"[cyan]{service}[/]")
return discovered
with progress_bar("Discovering", len(cfg.services)) as (progress, task_id):
return asyncio.run(gather_with_progress(progress, task_id))
def _snapshot_services(
cfg: Config,
services: list[str],
log_path: Path | None,
) -> Path:
"""Capture image digests with a progress bar."""
async def collect_service(service: str, now: datetime) -> list[SnapshotEntry]:
try:
return await collect_service_entries(cfg, service, now=now)
except RuntimeError:
return []
async def gather_with_progress(
progress: Progress, task_id: TaskID, now: datetime, svc_list: list[str]
) -> list[SnapshotEntry]:
# Map tasks to service names so we can update description
task_to_service = {asyncio.create_task(collect_service(s, now)): s for s in svc_list}
all_entries: list[SnapshotEntry] = []
for coro in asyncio.as_completed(list(task_to_service.keys())):
entries = await coro
all_entries.extend(entries)
# Find which service just completed (by checking done tasks)
for t, svc in task_to_service.items():
if t.done() and not hasattr(t, "_reported"):
t._reported = True # type: ignore[attr-defined]
progress.update(task_id, advance=1, description=f"[cyan]{svc}[/]")
break
return all_entries
effective_log_path = log_path or DEFAULT_LOG_PATH
now_dt = datetime.now(UTC)
now_iso = isoformat(now_dt)
with progress_bar("Capturing", len(services)) as (progress, task_id):
snapshot_entries = asyncio.run(gather_with_progress(progress, task_id, now_dt, services))
if not snapshot_entries:
msg = "No image digests were captured"
raise RuntimeError(msg)
existing_entries = load_existing_entries(effective_log_path)
merged_entries = merge_entries(existing_entries, snapshot_entries, now_iso=now_iso)
meta = {"generated_at": now_iso, "compose_dir": str(cfg.compose_dir)}
write_toml(effective_log_path, meta=meta, entries=merged_entries)
return effective_log_path
def _format_host(host: str | list[str]) -> str:
"""Format a host value for display."""
if isinstance(host, list):
return ", ".join(host)
return host
def _report_sync_changes(
added: list[str],
removed: list[str],
changed: list[tuple[str, str | list[str], str | list[str]]],
discovered: dict[str, str | list[str]],
current_state: dict[str, str | list[str]],
) -> None:
"""Report sync changes to the user."""
if added:
console.print(f"\nNew services found ({len(added)}):")
for service in sorted(added):
host_str = _format_host(discovered[service])
console.print(f" [green]+[/] [cyan]{service}[/] on [magenta]{host_str}[/]")
if changed:
console.print(f"\nServices on different hosts ({len(changed)}):")
for service, old_host, new_host in sorted(changed):
old_str = _format_host(old_host)
new_str = _format_host(new_host)
console.print(
f" [yellow]~[/] [cyan]{service}[/]: [magenta]{old_str}[/] → [magenta]{new_str}[/]"
)
if removed:
console.print(f"\nServices no longer running ({len(removed)}):")
for service in sorted(removed):
host_str = _format_host(current_state[service])
console.print(f" [red]-[/] [cyan]{service}[/] (was on [magenta]{host_str}[/])")
# --- Check helpers ---
def _check_ssh_connectivity(cfg: Config) -> list[str]:
"""Check SSH connectivity to all hosts. Returns list of unreachable hosts."""
# Filter out local hosts - no SSH needed
remote_hosts = [h for h in cfg.hosts if not is_local(cfg.hosts[h])]
if not remote_hosts:
return []
console.print() # Spacing before progress bar
async def check_host(host_name: str) -> tuple[str, bool]:
host = cfg.hosts[host_name]
result = await run_command(host, "echo ok", host_name, stream=False)
return host_name, result.success
async def gather_with_progress(progress: Progress, task_id: TaskID) -> list[str]:
tasks = [asyncio.create_task(check_host(h)) for h in remote_hosts]
unreachable: list[str] = []
for coro in asyncio.as_completed(tasks):
host_name, success = await coro
if not success:
unreachable.append(host_name)
progress.update(task_id, advance=1, description=f"[cyan]{host_name}[/]")
return unreachable
with progress_bar("Checking SSH connectivity", len(remote_hosts)) as (progress, task_id):
return asyncio.run(gather_with_progress(progress, task_id))
def _check_service_requirements(
cfg: Config,
services: list[str],
) -> tuple[list[tuple[str, str, str]], list[tuple[str, str, str]], list[tuple[str, str, str]]]:
"""Check mounts, networks, and devices for all services with a progress bar.
Returns (mount_errors, network_errors, device_errors) where each is a list of
(service, host, missing_item) tuples.
"""
async def check_service(
service: str,
) -> tuple[
str,
list[tuple[str, str, str]],
list[tuple[str, str, str]],
list[tuple[str, str, str]],
]:
"""Check requirements for a single service on all its hosts."""
host_names = cfg.get_hosts(service)
mount_errors: list[tuple[str, str, str]] = []
network_errors: list[tuple[str, str, str]] = []
device_errors: list[tuple[str, str, str]] = []
for host_name in host_names:
missing_paths, missing_nets, missing_devs = await check_service_requirements(
cfg, service, host_name
)
mount_errors.extend((service, host_name, p) for p in missing_paths)
network_errors.extend((service, host_name, n) for n in missing_nets)
device_errors.extend((service, host_name, d) for d in missing_devs)
return service, mount_errors, network_errors, device_errors
async def gather_with_progress(
progress: Progress, task_id: TaskID
) -> tuple[list[tuple[str, str, str]], list[tuple[str, str, str]], list[tuple[str, str, str]]]:
tasks = [asyncio.create_task(check_service(s)) for s in services]
all_mount_errors: list[tuple[str, str, str]] = []
all_network_errors: list[tuple[str, str, str]] = []
all_device_errors: list[tuple[str, str, str]] = []
for coro in asyncio.as_completed(tasks):
service, mount_errs, net_errs, dev_errs = await coro
all_mount_errors.extend(mount_errs)
all_network_errors.extend(net_errs)
all_device_errors.extend(dev_errs)
progress.update(task_id, advance=1, description=f"[cyan]{service}[/]")
return all_mount_errors, all_network_errors, all_device_errors
with progress_bar("Checking requirements", len(services)) as (progress, task_id):
return asyncio.run(gather_with_progress(progress, task_id))
def _report_config_status(cfg: Config) -> bool:
"""Check and report config vs disk status. Returns True if errors found."""
configured = set(cfg.services.keys())
on_disk = cfg.discover_compose_dirs()
unmanaged = sorted(on_disk - configured)
missing_from_disk = sorted(configured - on_disk)
if unmanaged:
console.print(f"\n[yellow]Unmanaged[/] (on disk but not in config, {len(unmanaged)}):")
for name in unmanaged:
console.print(f" [yellow]+[/] [cyan]{name}[/]")
if missing_from_disk:
console.print(f"\n[red]In config but no compose file[/] ({len(missing_from_disk)}):")
for name in missing_from_disk:
console.print(f" [red]-[/] [cyan]{name}[/]")
if not unmanaged and not missing_from_disk:
console.print("[green]✓[/] Config matches disk")
return bool(missing_from_disk)
def _report_orphaned_services(cfg: Config) -> bool:
"""Check for services in state but not in config. Returns True if orphans found."""
orphaned = get_orphaned_services(cfg)
if orphaned:
console.print("\n[yellow]Orphaned services[/] (in state but not in config):")
console.print(
"[dim]Run 'cf apply' to stop them, or 'cf down --orphaned' for just orphans.[/]"
)
for name, hosts in sorted(orphaned.items()):
host_str = ", ".join(hosts) if isinstance(hosts, list) else hosts
console.print(f" [yellow]![/] [cyan]{name}[/] on [magenta]{host_str}[/]")
return True
return False
def _report_traefik_status(cfg: Config, services: list[str]) -> None:
"""Check and report traefik label status."""
try:
_, warnings = generate_traefik_config(cfg, services, check_all=True)
except (FileNotFoundError, ValueError):
return
if warnings:
console.print(f"\n[yellow]Traefik issues[/] ({len(warnings)}):")
for warning in warnings:
console.print(f" [yellow]![/] {warning}")
else:
console.print("[green]✓[/] Traefik labels valid")
def _report_mount_errors(mount_errors: list[tuple[str, str, str]]) -> None:
"""Report mount errors grouped by service."""
by_service: dict[str, list[tuple[str, str]]] = {}
for svc, host, path in mount_errors:
by_service.setdefault(svc, []).append((host, path))
console.print(f"[red]Missing mounts[/] ({len(mount_errors)}):")
for svc, items in sorted(by_service.items()):
host = items[0][0]
paths = [p for _, p in items]
console.print(f" [cyan]{svc}[/] on [magenta]{host}[/]:")
for path in paths:
console.print(f" [red]✗[/] {path}")
def _report_network_errors(network_errors: list[tuple[str, str, str]]) -> None:
"""Report network errors grouped by service."""
by_service: dict[str, list[tuple[str, str]]] = {}
for svc, host, net in network_errors:
by_service.setdefault(svc, []).append((host, net))
console.print(f"[red]Missing networks[/] ({len(network_errors)}):")
for svc, items in sorted(by_service.items()):
host = items[0][0]
networks = [n for _, n in items]
console.print(f" [cyan]{svc}[/] on [magenta]{host}[/]:")
for net in networks:
console.print(f" [red]✗[/] {net}")
def _report_device_errors(device_errors: list[tuple[str, str, str]]) -> None:
"""Report device errors grouped by service."""
by_service: dict[str, list[tuple[str, str]]] = {}
for svc, host, dev in device_errors:
by_service.setdefault(svc, []).append((host, dev))
console.print(f"[red]Missing devices[/] ({len(device_errors)}):")
for svc, items in sorted(by_service.items()):
host = items[0][0]
devices = [d for _, d in items]
console.print(f" [cyan]{svc}[/] on [magenta]{host}[/]:")
for dev in devices:
console.print(f" [red]✗[/] {dev}")
def _report_ssh_status(unreachable_hosts: list[str]) -> bool:
"""Report SSH connectivity status. Returns True if there are errors."""
if unreachable_hosts:
console.print(f"[red]Unreachable hosts[/] ({len(unreachable_hosts)}):")
for host in sorted(unreachable_hosts):
console.print(f" [red]✗[/] [magenta]{host}[/]")
return True
console.print("[green]✓[/] All hosts reachable")
return False
def _report_host_compatibility(
compat: dict[str, tuple[int, int, list[str]]],
assigned_hosts: list[str],
) -> None:
"""Report host compatibility for a service."""
for host_name, (found, total, missing) in sorted(compat.items()):
is_assigned = host_name in assigned_hosts
marker = " [dim](assigned)[/]" if is_assigned else ""
if found == total:
console.print(f" [green]✓[/] [magenta]{host_name}[/] {found}/{total}{marker}")
else:
preview = ", ".join(missing[:_MISSING_PATH_PREVIEW_LIMIT])
if len(missing) > _MISSING_PATH_PREVIEW_LIMIT:
preview += f", +{len(missing) - _MISSING_PATH_PREVIEW_LIMIT} more"
console.print(
f" [red]✗[/] [magenta]{host_name}[/] {found}/{total} "
f"[dim](missing: {preview})[/]{marker}"
)
def _run_remote_checks(cfg: Config, svc_list: list[str], *, show_host_compat: bool) -> bool:
"""Run SSH-based checks for mounts, networks, and host compatibility.
Returns True if any errors were found.
"""
has_errors = False
# Check SSH connectivity first
if _report_ssh_status(_check_ssh_connectivity(cfg)):
has_errors = True
console.print() # Spacing before mounts/networks check
# Check mounts, networks, and devices
mount_errors, network_errors, device_errors = _check_service_requirements(cfg, svc_list)
if mount_errors:
_report_mount_errors(mount_errors)
has_errors = True
if network_errors:
_report_network_errors(network_errors)
has_errors = True
if device_errors:
_report_device_errors(device_errors)
has_errors = True
if not mount_errors and not network_errors and not device_errors:
console.print("[green]✓[/] All mounts, networks, and devices exist")
if show_host_compat:
for service in svc_list:
console.print(f"\n[bold]Host compatibility for[/] [cyan]{service}[/]:")
compat = run_async(check_host_compatibility(cfg, service))
assigned_hosts = cfg.get_hosts(service)
_report_host_compatibility(compat, assigned_hosts)
return has_errors
# Default network settings for cross-host Docker networking
_DEFAULT_NETWORK_NAME = "mynetwork"
_DEFAULT_NETWORK_SUBNET = "172.20.0.0/16"
_DEFAULT_NETWORK_GATEWAY = "172.20.0.1"
@app.command("traefik-file", rich_help_panel="Configuration")
def traefik_file(
services: ServicesArg = None,
all_services: AllOption = False,
output: Annotated[
Path | None,
typer.Option(
"--output",
"-o",
help="Write Traefik file-provider YAML to this path (stdout if omitted)",
),
] = None,
config: ConfigOption = None,
) -> None:
"""Generate a Traefik file-provider fragment from compose Traefik labels."""
svc_list, cfg = get_services(services or [], all_services, config)
try:
dynamic, warnings = generate_traefik_config(cfg, svc_list)
except (FileNotFoundError, ValueError) as exc:
err_console.print(f"[red]✗[/] {exc}")
raise typer.Exit(1) from exc
rendered = render_traefik_config(dynamic)
if output:
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(rendered)
console.print(f"[green]✓[/] Traefik config written to {output}")
else:
console.print(rendered)
for warning in warnings:
err_console.print(f"[yellow]![/] {warning}")
@app.command(rich_help_panel="Configuration")
def refresh(
config: ConfigOption = None,
log_path: LogPathOption = None,
dry_run: Annotated[
bool,
typer.Option("--dry-run", "-n", help="Show what would change without writing"),
] = False,
) -> None:
"""Update local state from running services.
Discovers which services are running on which hosts, updates the state
file, and captures image digests. This is a read operation - it updates
your local state to match reality, not the other way around.
Use 'cf apply' to make reality match your config (stop orphans, migrate).
"""
cfg = load_config_or_exit(config)
current_state = load_state(cfg)
discovered = _discover_services(cfg)
# Calculate changes
added = [s for s in discovered if s not in current_state]
removed = [s for s in current_state if s not in discovered]
changed = [
(s, current_state[s], discovered[s])
for s in discovered
if s in current_state and current_state[s] != discovered[s]
]
# Report state changes
state_changed = bool(added or removed or changed)
if state_changed:
_report_sync_changes(added, removed, changed, discovered, current_state)
else:
console.print("[green]✓[/] State is already in sync.")
if dry_run:
console.print("\n[dim](dry-run: no changes made)[/]")
return
# Update state file
if state_changed:
save_state(cfg, discovered)
console.print(f"\n[green]✓[/] State updated: {len(discovered)} services tracked.")
# Capture image digests for running services
if discovered:
try:
path = _snapshot_services(cfg, list(discovered.keys()), log_path)
console.print(f"[green]✓[/] Digests written to {path}")
except RuntimeError as exc:
err_console.print(f"[yellow]![/] {exc}")
@app.command(rich_help_panel="Configuration")
def check(
services: ServicesArg = None,
local: Annotated[
bool,
typer.Option("--local", help="Skip SSH-based checks (faster)"),
] = False,
config: ConfigOption = None,
) -> None:
"""Validate configuration, traefik labels, mounts, and networks.
Without arguments: validates all services against configured hosts.
With service arguments: validates specific services and shows host compatibility.
Use --local to skip SSH-based checks for faster validation.
"""
cfg = load_config_or_exit(config)
# Determine which services to check and whether to show host compatibility
if services:
svc_list = list(services)
invalid = [s for s in svc_list if s not in cfg.services]
if invalid:
for svc in invalid:
err_console.print(f"[red]✗[/] Service '{svc}' not found in config")
raise typer.Exit(1)
show_host_compat = True
else:
svc_list = list(cfg.services.keys())
show_host_compat = False
# Run checks
has_errors = _report_config_status(cfg)
_report_traefik_status(cfg, svc_list)
if not local and _run_remote_checks(cfg, svc_list, show_host_compat=show_host_compat):
has_errors = True
# Check for orphaned services (in state but removed from config)
if _report_orphaned_services(cfg):
has_errors = True
if has_errors:
raise typer.Exit(1)
@app.command("init-network", rich_help_panel="Configuration")
def init_network(
hosts: Annotated[
list[str] | None,
typer.Argument(help="Hosts to create network on (default: all)"),
] = None,
network: Annotated[
str,
typer.Option("--network", "-n", help="Network name"),
] = _DEFAULT_NETWORK_NAME,
subnet: Annotated[
str,
typer.Option("--subnet", "-s", help="Network subnet"),
] = _DEFAULT_NETWORK_SUBNET,
gateway: Annotated[
str,
typer.Option("--gateway", "-g", help="Network gateway"),
] = _DEFAULT_NETWORK_GATEWAY,
config: ConfigOption = None,
) -> None:
"""Create Docker network on hosts with consistent settings.
Creates an external Docker network that services can use for cross-host
communication. Uses the same subnet/gateway on all hosts to ensure
consistent networking.
"""
cfg = load_config_or_exit(config)
target_hosts = list(hosts) if hosts else list(cfg.hosts.keys())
invalid = [h for h in target_hosts if h not in cfg.hosts]
if invalid:
for h in invalid:
err_console.print(f"[red]✗[/] Host '{h}' not found in config")
raise typer.Exit(1)
async def create_network_on_host(host_name: str) -> CommandResult:
host = cfg.hosts[host_name]
# Check if network already exists
check_cmd = f"docker network inspect '{network}' >/dev/null 2>&1"
check_result = await run_command(host, check_cmd, host_name, stream=False)
if check_result.success:
console.print(f"[cyan]\\[{host_name}][/] Network '{network}' already exists")
return CommandResult(service=host_name, exit_code=0, success=True)
# Create the network
create_cmd = (
f"docker network create "
f"--driver bridge "
f"--subnet '{subnet}' "
f"--gateway '{gateway}' "
f"'{network}'"
)
result = await run_command(host, create_cmd, host_name, stream=False)
if result.success:
console.print(f"[cyan]\\[{host_name}][/] [green]✓[/] Created network '{network}'")
else:
err_console.print(
f"[cyan]\\[{host_name}][/] [red]✗[/] Failed to create network: "
f"{result.stderr.strip()}"
)
return result
async def run_all() -> list[CommandResult]:
return await asyncio.gather(*[create_network_on_host(h) for h in target_hosts])
results = run_async(run_all())
failed = [r for r in results if not r.success]
if failed:
raise typer.Exit(1)

View File

@@ -0,0 +1,236 @@
"""Monitoring commands: logs, ps, stats."""
from __future__ import annotations
import asyncio
import contextlib
from typing import TYPE_CHECKING, Annotated
import typer
from rich.progress import Progress, TaskID # noqa: TC002
from rich.table import Table
from compose_farm.cli.app import app
from compose_farm.cli.common import (
_STATS_PREVIEW_LIMIT,
AllOption,
ConfigOption,
HostOption,
ServicesArg,
get_services,
load_config_or_exit,
progress_bar,
report_results,
run_async,
)
from compose_farm.console import console, err_console
from compose_farm.executor import run_command, run_on_services
from compose_farm.state import get_services_needing_migration, load_state
if TYPE_CHECKING:
from collections.abc import Mapping
from compose_farm.config import Config
def _group_services_by_host(
services: dict[str, str | list[str]],
hosts: Mapping[str, object],
all_hosts: list[str] | None = None,
) -> dict[str, list[str]]:
"""Group services by their assigned host(s).
For multi-host services (list or "all"), the service appears in multiple host lists.
"""
by_host: dict[str, list[str]] = {h: [] for h in hosts}
for service, host_value in services.items():
if isinstance(host_value, list):
# Explicit list of hosts
for host_name in host_value:
if host_name in by_host:
by_host[host_name].append(service)
elif host_value == "all" and all_hosts:
# "all" keyword - add to all hosts
for host_name in all_hosts:
if host_name in by_host:
by_host[host_name].append(service)
elif host_value in by_host:
# Single host
by_host[host_value].append(service)
return by_host
def _get_container_counts(cfg: Config) -> dict[str, int]:
"""Get container counts from all hosts with a progress bar."""
async def get_count(host_name: str) -> tuple[str, int]:
host = cfg.hosts[host_name]
result = await run_command(host, "docker ps -q | wc -l", host_name, stream=False)
count = 0
if result.success:
with contextlib.suppress(ValueError):
count = int(result.stdout.strip())
return host_name, count
async def gather_with_progress(progress: Progress, task_id: TaskID) -> dict[str, int]:
hosts = list(cfg.hosts.keys())
tasks = [asyncio.create_task(get_count(h)) for h in hosts]
results: dict[str, int] = {}
for coro in asyncio.as_completed(tasks):
host_name, count = await coro
results[host_name] = count
progress.update(task_id, advance=1, description=f"[cyan]{host_name}[/]")
return results
with progress_bar("Querying hosts", len(cfg.hosts)) as (progress, task_id):
return asyncio.run(gather_with_progress(progress, task_id))
def _build_host_table(
cfg: Config,
services_by_host: dict[str, list[str]],
running_by_host: dict[str, list[str]],
container_counts: dict[str, int],
*,
show_containers: bool,
) -> Table:
"""Build the hosts table."""
table = Table(title="Hosts", show_header=True, header_style="bold cyan")
table.add_column("Host", style="magenta")
table.add_column("Address")
table.add_column("Configured", justify="right")
table.add_column("Running", justify="right")
if show_containers:
table.add_column("Containers", justify="right")
for host_name in sorted(cfg.hosts.keys()):
host = cfg.hosts[host_name]
configured = len(services_by_host[host_name])
running = len(running_by_host[host_name])
row = [
host_name,
host.address,
str(configured),
str(running) if running > 0 else "[dim]0[/]",
]
if show_containers:
count = container_counts.get(host_name, 0)
row.append(str(count) if count > 0 else "[dim]0[/]")
table.add_row(*row)
return table
def _build_summary_table(
cfg: Config, state: dict[str, str | list[str]], pending: list[str]
) -> Table:
"""Build the summary table."""
on_disk = cfg.discover_compose_dirs()
table = Table(title="Summary", show_header=False)
table.add_column("Label", style="dim")
table.add_column("Value", style="bold")
table.add_row("Total hosts", str(len(cfg.hosts)))
table.add_row("Services (configured)", str(len(cfg.services)))
table.add_row("Services (tracked)", str(len(state)))
table.add_row("Compose files on disk", str(len(on_disk)))
if pending:
preview = ", ".join(pending[:_STATS_PREVIEW_LIMIT])
suffix = "..." if len(pending) > _STATS_PREVIEW_LIMIT else ""
table.add_row("Pending migrations", f"[yellow]{len(pending)}[/] ({preview}{suffix})")
else:
table.add_row("Pending migrations", "[green]0[/]")
return table
# --- Command functions ---
@app.command(rich_help_panel="Monitoring")
def logs(
services: ServicesArg = None,
all_services: AllOption = False,
host: HostOption = None,
follow: Annotated[bool, typer.Option("--follow", "-f", help="Follow logs")] = False,
tail: Annotated[
int | None,
typer.Option("--tail", "-n", help="Number of lines (default: 20 for --all, 100 otherwise)"),
] = None,
config: ConfigOption = None,
) -> None:
"""Show service logs."""
if all_services and host is not None:
err_console.print("[red]✗[/] Cannot use --all and --host together")
raise typer.Exit(1)
cfg = load_config_or_exit(config)
# Determine service list based on options
if host is not None:
if host not in cfg.hosts:
err_console.print(f"[red]✗[/] Host '{host}' not found in config")
raise typer.Exit(1)
# Include services where host is in the list of configured hosts
svc_list = [s for s in cfg.services if host in cfg.get_hosts(s)]
if not svc_list:
err_console.print(f"[yellow]![/] No services configured for host '{host}'")
return
else:
svc_list, cfg = get_services(services or [], all_services, config)
# Default to fewer lines when showing multiple services
many_services = all_services or host is not None or len(svc_list) > 1
effective_tail = tail if tail is not None else (20 if many_services else 100)
cmd = f"logs --tail {effective_tail}"
if follow:
cmd += " -f"
results = run_async(run_on_services(cfg, svc_list, cmd))
report_results(results)
@app.command(rich_help_panel="Monitoring")
def ps(
config: ConfigOption = None,
) -> None:
"""Show status of all services."""
cfg = load_config_or_exit(config)
results = run_async(run_on_services(cfg, list(cfg.services.keys()), "ps"))
report_results(results)
@app.command(rich_help_panel="Monitoring")
def stats(
live: Annotated[
bool,
typer.Option("--live", "-l", help="Query Docker for live container stats"),
] = False,
config: ConfigOption = None,
) -> None:
"""Show overview statistics for hosts and services.
Without --live: Shows config/state info (hosts, services, pending migrations).
With --live: Also queries Docker on each host for container counts.
"""
cfg = load_config_or_exit(config)
state = load_state(cfg)
pending = get_services_needing_migration(cfg)
all_hosts = list(cfg.hosts.keys())
services_by_host = _group_services_by_host(cfg.services, cfg.hosts, all_hosts)
running_by_host = _group_services_by_host(state, cfg.hosts, all_hosts)
container_counts: dict[str, int] = {}
if live:
container_counts = _get_container_counts(cfg)
host_table = _build_host_table(
cfg, services_by_host, running_by_host, container_counts, show_containers=live
)
console.print(host_table)
console.print()
console.print(_build_summary_table(cfg, state, pending))

331
src/compose_farm/compose.py Normal file
View File

@@ -0,0 +1,331 @@
"""Compose file parsing utilities.
Handles .env loading, variable interpolation, port/volume/network extraction.
"""
from __future__ import annotations
import os
import re
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
import yaml
if TYPE_CHECKING:
from pathlib import Path
from .config import Config
# Port parsing constants
_SINGLE_PART = 1
_PUBLISHED_TARGET_PARTS = 2
_HOST_PUBLISHED_PARTS = 3
_MIN_VOLUME_PARTS = 2
_VAR_PATTERN = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)(?::-(.*?))?\}")
@dataclass(frozen=True)
class PortMapping:
"""Port mapping for a compose service."""
target: int
published: int | None
def _load_env(compose_path: Path) -> dict[str, str]:
"""Load environment variables for compose interpolation.
Reads from .env file in the same directory as compose file,
then overlays current environment variables.
"""
env: dict[str, str] = {}
env_path = compose_path.parent / ".env"
if env_path.exists():
for line in env_path.read_text().splitlines():
stripped = line.strip()
if not stripped or stripped.startswith("#") or "=" not in stripped:
continue
key, value = stripped.split("=", 1)
key = key.strip()
value = value.strip()
if (value.startswith('"') and value.endswith('"')) or (
value.startswith("'") and value.endswith("'")
):
value = value[1:-1]
env[key] = value
env.update({k: v for k, v in os.environ.items() if isinstance(v, str)})
return env
def _interpolate(value: str, env: dict[str, str]) -> str:
"""Perform ${VAR} and ${VAR:-default} interpolation."""
def replace(match: re.Match[str]) -> str:
var = match.group(1)
default = match.group(2)
resolved = env.get(var)
if resolved:
return resolved
return default or ""
return _VAR_PATTERN.sub(replace, value)
def _parse_ports(raw: Any, env: dict[str, str]) -> list[PortMapping]: # noqa: PLR0912
"""Parse port specifications from compose file.
Handles string formats like "8080", "8080:80", "0.0.0.0:8080:80",
and dict formats with target/published keys.
"""
if raw is None:
return []
mappings: list[PortMapping] = []
items = raw if isinstance(raw, list) else [raw]
for item in items:
if isinstance(item, str):
interpolated = _interpolate(item, env)
port_spec, _, _ = interpolated.partition("/")
parts = port_spec.split(":")
published: int | None = None
target: int | None = None
if len(parts) == _SINGLE_PART and parts[0].isdigit():
target = int(parts[0])
elif (
len(parts) == _PUBLISHED_TARGET_PARTS and parts[0].isdigit() and parts[1].isdigit()
):
published = int(parts[0])
target = int(parts[1])
elif (
len(parts) == _HOST_PUBLISHED_PARTS and parts[-2].isdigit() and parts[-1].isdigit()
):
published = int(parts[-2])
target = int(parts[-1])
if target is not None:
mappings.append(PortMapping(target=target, published=published))
elif isinstance(item, dict):
target_raw = item.get("target")
if isinstance(target_raw, str):
target_raw = _interpolate(target_raw, env)
if target_raw is None:
continue
try:
target_val = int(str(target_raw))
except (TypeError, ValueError):
continue
published_raw = item.get("published")
if isinstance(published_raw, str):
published_raw = _interpolate(published_raw, env)
published_val: int | None
try:
published_val = int(str(published_raw)) if published_raw is not None else None
except (TypeError, ValueError):
published_val = None
mappings.append(PortMapping(target=target_val, published=published_val))
return mappings
def _resolve_host_path(host_path: str, compose_dir: Path) -> str | None:
"""Resolve a host path from volume mount, returning None for named volumes."""
if host_path.startswith("/"):
return host_path
if host_path.startswith(("./", "../")):
return str((compose_dir / host_path).resolve())
return None # Named volume
def _parse_volume_item(
item: str | dict[str, Any],
env: dict[str, str],
compose_dir: Path,
) -> str | None:
"""Parse a single volume item and return host path if it's a bind mount."""
if isinstance(item, str):
interpolated = _interpolate(item, env)
parts = interpolated.split(":")
if len(parts) >= _MIN_VOLUME_PARTS:
return _resolve_host_path(parts[0], compose_dir)
elif isinstance(item, dict) and item.get("type") == "bind":
source = item.get("source")
if source:
interpolated = _interpolate(str(source), env)
return _resolve_host_path(interpolated, compose_dir)
return None
def parse_host_volumes(config: Config, service: str) -> list[str]:
"""Extract host bind mount paths from a service's compose file.
Returns a list of absolute host paths used as volume mounts.
Skips named volumes and resolves relative paths.
"""
compose_path = config.get_compose_path(service)
if not compose_path.exists():
return []
env = _load_env(compose_path)
compose_data = yaml.safe_load(compose_path.read_text()) or {}
raw_services = compose_data.get("services", {})
if not isinstance(raw_services, dict):
return []
paths: list[str] = []
compose_dir = compose_path.parent
for definition in raw_services.values():
if not isinstance(definition, dict):
continue
volumes = definition.get("volumes")
if not volumes:
continue
items = volumes if isinstance(volumes, list) else [volumes]
for item in items:
host_path = _parse_volume_item(item, env, compose_dir)
if host_path:
paths.append(host_path)
# Return unique paths, preserving order
seen: set[str] = set()
unique: list[str] = []
for p in paths:
if p not in seen:
seen.add(p)
unique.append(p)
return unique
def parse_devices(config: Config, service: str) -> list[str]:
"""Extract host device paths from a service's compose file.
Returns a list of host device paths (e.g., /dev/dri, /dev/dri/renderD128).
"""
compose_path = config.get_compose_path(service)
if not compose_path.exists():
return []
env = _load_env(compose_path)
compose_data = yaml.safe_load(compose_path.read_text()) or {}
raw_services = compose_data.get("services", {})
if not isinstance(raw_services, dict):
return []
devices: list[str] = []
for definition in raw_services.values():
if not isinstance(definition, dict):
continue
device_list = definition.get("devices")
if not device_list or not isinstance(device_list, list):
continue
for item in device_list:
if not isinstance(item, str):
continue
interpolated = _interpolate(item, env)
# Format: host_path:container_path[:options]
parts = interpolated.split(":")
if parts:
host_path = parts[0]
if host_path.startswith("/dev/"):
devices.append(host_path)
# Return unique devices, preserving order
seen: set[str] = set()
unique: list[str] = []
for d in devices:
if d not in seen:
seen.add(d)
unique.append(d)
return unique
def parse_external_networks(config: Config, service: str) -> list[str]:
"""Extract external network names from a service's compose file.
Returns a list of network names marked as external: true.
"""
compose_path = config.get_compose_path(service)
if not compose_path.exists():
return []
compose_data = yaml.safe_load(compose_path.read_text()) or {}
networks = compose_data.get("networks", {})
if not isinstance(networks, dict):
return []
external_networks: list[str] = []
for name, definition in networks.items():
if isinstance(definition, dict) and definition.get("external") is True:
external_networks.append(name)
return external_networks
def load_compose_services(
config: Config,
stack: str,
) -> tuple[dict[str, Any], dict[str, str], str]:
"""Load services from a compose file with environment interpolation.
Returns (services_dict, env_dict, host_address).
"""
compose_path = config.get_compose_path(stack)
if not compose_path.exists():
message = f"[{stack}] Compose file not found: {compose_path}"
raise FileNotFoundError(message)
env = _load_env(compose_path)
compose_data = yaml.safe_load(compose_path.read_text()) or {}
raw_services = compose_data.get("services", {})
if not isinstance(raw_services, dict):
return {}, env, config.get_host(stack).address
return raw_services, env, config.get_host(stack).address
def normalize_labels(raw: Any, env: dict[str, str]) -> dict[str, str]:
"""Normalize labels from list or dict format, with interpolation."""
if raw is None:
return {}
if isinstance(raw, dict):
return {
_interpolate(str(k), env): _interpolate(str(v), env)
for k, v in raw.items()
if k is not None
}
if isinstance(raw, list):
labels: dict[str, str] = {}
for item in raw:
if not isinstance(item, str) or "=" not in item:
continue
key_raw, value_raw = item.split("=", 1)
key = _interpolate(key_raw.strip(), env)
value = _interpolate(value_raw.strip(), env)
labels[key] = value
return labels
return {}
def get_ports_for_service(
definition: dict[str, Any],
all_services: dict[str, Any],
env: dict[str, str],
) -> list[PortMapping]:
"""Get ports for a service, following network_mode: service:X if present."""
network_mode = definition.get("network_mode", "")
if isinstance(network_mode, str) and network_mode.startswith("service:"):
# Service uses another service's network - get ports from that service
ref_service = network_mode[len("service:") :]
if ref_service in all_services:
ref_def = all_services[ref_service]
if isinstance(ref_def, dict):
return _parse_ports(ref_def.get("ports"), env)
return _parse_ports(definition.get("ports"), env)

View File

@@ -3,11 +3,14 @@
from __future__ import annotations
import getpass
import os
from pathlib import Path
import yaml
from pydantic import BaseModel, Field, model_validator
from .paths import xdg_config_home
class Host(BaseModel):
"""SSH host configuration."""
@@ -22,23 +25,70 @@ class Config(BaseModel):
compose_dir: Path = Path("/opt/compose")
hosts: dict[str, Host]
services: dict[str, str] # service_name -> host_name
services: dict[str, str | list[str]] # service_name -> host_name or list of hosts
traefik_file: Path | None = None # Auto-regenerate traefik config after up/down
traefik_service: str | None = None # Service name for Traefik (skip its host in file-provider)
config_path: Path = Path() # Set by load_config()
def get_state_path(self) -> Path:
"""Get the state file path (stored alongside config)."""
return self.config_path.parent / "compose-farm-state.yaml"
@model_validator(mode="after")
def validate_service_hosts(self) -> Config:
"""Ensure all services reference valid hosts."""
for service, host_name in self.services.items():
if host_name not in self.hosts:
msg = f"Service '{service}' references unknown host '{host_name}'"
raise ValueError(msg)
def validate_hosts_and_services(self) -> Config:
"""Validate host names and service configurations."""
# "all" is reserved keyword, cannot be used as host name
if "all" in self.hosts:
msg = "'all' is a reserved keyword and cannot be used as a host name"
raise ValueError(msg)
for service, host_value in self.services.items():
# Validate list configurations
if isinstance(host_value, list):
if not host_value:
msg = f"Service '{service}' has empty host list"
raise ValueError(msg)
if len(host_value) != len(set(host_value)):
msg = f"Service '{service}' has duplicate hosts in list"
raise ValueError(msg)
# Validate all referenced hosts exist
host_names = self.get_hosts(service)
for host_name in host_names:
if host_name not in self.hosts:
msg = f"Service '{service}' references unknown host '{host_name}'"
raise ValueError(msg)
return self
def get_host(self, service: str) -> Host:
"""Get host config for a service."""
def get_hosts(self, service: str) -> list[str]:
"""Get list of host names for a service.
Supports:
- Single host: "truenas-debian" -> ["truenas-debian"]
- All hosts: "all" -> list of all configured hosts
- Explicit list: ["host1", "host2"] -> ["host1", "host2"]
"""
if service not in self.services:
msg = f"Unknown service: {service}"
raise ValueError(msg)
return self.hosts[self.services[service]]
host_value = self.services[service]
if isinstance(host_value, list):
return host_value
if host_value == "all":
return list(self.hosts.keys())
return [host_value]
def is_multi_host(self, service: str) -> bool:
"""Check if a service runs on multiple hosts."""
return len(self.get_hosts(service)) > 1
def get_host(self, service: str) -> Host:
"""Get host config for a service (first host if multi-host)."""
if service not in self.services:
msg = f"Unknown service: {service}"
raise ValueError(msg)
host_names = self.get_hosts(service)
return self.hosts[host_names[0]]
def get_compose_path(self, service: str) -> Path:
"""Get compose file path for a service.
@@ -58,6 +108,25 @@ class Config(BaseModel):
# Default to compose.yaml if none exist (will error later)
return service_dir / "compose.yaml"
def discover_compose_dirs(self) -> set[str]:
"""Find all directories in compose_dir that contain a compose file."""
compose_filenames = {
"compose.yaml",
"compose.yml",
"docker-compose.yml",
"docker-compose.yaml",
}
found: set[str] = set()
if not self.compose_dir.exists():
return found
for subdir in self.compose_dir.iterdir():
if subdir.is_dir():
for filename in compose_filenames:
if (subdir / filename).exists():
found.add(subdir.name)
break
return found
def _parse_hosts(raw_hosts: dict[str, str | dict[str, str | int]]) -> dict[str, Host]:
"""Parse hosts from config, handling both simple and full forms."""
@@ -76,17 +145,20 @@ def load_config(path: Path | None = None) -> Config:
"""Load configuration from YAML file.
Search order:
1. Explicit path if provided
2. ./compose-farm.yaml
3. ~/.config/compose-farm/compose-farm.yaml
1. Explicit path if provided via --config
2. CF_CONFIG environment variable
3. ./compose-farm.yaml
4. $XDG_CONFIG_HOME/compose-farm/compose-farm.yaml (defaults to ~/.config)
"""
search_paths = [
Path("compose-farm.yaml"),
Path.home() / ".config" / "compose-farm" / "compose-farm.yaml",
xdg_config_home() / "compose-farm" / "compose-farm.yaml",
]
if path:
config_path = path
elif env_path := os.environ.get("CF_CONFIG"):
config_path = Path(env_path)
else:
config_path = None
for p in search_paths:
@@ -98,10 +170,18 @@ def load_config(path: Path | None = None) -> Config:
msg = f"Config file not found. Searched: {', '.join(str(p) for p in search_paths)}"
raise FileNotFoundError(msg)
if config_path.is_dir():
msg = (
f"Config path is a directory, not a file: {config_path}\n"
"This often happens when Docker creates an empty directory for a missing mount."
)
raise FileNotFoundError(msg)
with config_path.open() as f:
raw = yaml.safe_load(f)
# Parse hosts with flexible format support
raw["hosts"] = _parse_hosts(raw.get("hosts", {}))
raw["config_path"] = config_path.resolve()
return Config(**raw)

View File

@@ -0,0 +1,6 @@
"""Shared console instances for consistent output styling."""
from rich.console import Console
console = Console(highlight=False)
err_console = Console(stderr=True, highlight=False)

View File

@@ -0,0 +1,89 @@
# Compose Farm configuration
# Documentation: https://github.com/basnijholt/compose-farm
#
# This file configures compose-farm to manage Docker Compose services
# across multiple hosts via SSH.
#
# Place this file at:
# - ./compose-farm.yaml (current directory)
# - ~/.config/compose-farm/compose-farm.yaml
# - Or specify with: cf --config /path/to/config.yaml
# - Or set CF_CONFIG environment variable
# ------------------------------------------------------------------------------
# compose_dir: Directory containing service subdirectories with compose files
# ------------------------------------------------------------------------------
# Each subdirectory should contain a compose.yaml (or docker-compose.yml).
# This path must be the same on all hosts (NFS mount recommended).
#
compose_dir: /opt/compose
# ------------------------------------------------------------------------------
# hosts: SSH connection details for each host
# ------------------------------------------------------------------------------
# Simple form:
# hostname: ip-or-fqdn
#
# Full form:
# hostname:
# address: ip-or-fqdn
# user: ssh-username # default: current user
# port: 22 # default: 22
#
# Note: "all" is a reserved keyword and cannot be used as a host name.
#
hosts:
# Example: simple form (uses current user, port 22)
server1: 192.168.1.10
# Example: full form with explicit user
server2:
address: 192.168.1.20
user: admin
# Example: full form with custom port
server3:
address: 192.168.1.30
user: root
port: 2222
# ------------------------------------------------------------------------------
# services: Map service names to their target host(s)
# ------------------------------------------------------------------------------
# Each service name must match a subdirectory in compose_dir.
#
# Single host:
# service-name: hostname
#
# Multiple hosts (explicit list):
# service-name: [host1, host2]
#
# All hosts:
# service-name: all
#
services:
# Example: service runs on a single host
nginx: server1
postgres: server2
# Example: service runs on multiple specific hosts
# prometheus: [server1, server2]
# Example: service runs on ALL hosts (e.g., monitoring agents)
# node-exporter: all
# ------------------------------------------------------------------------------
# traefik_file: (optional) Auto-generate Traefik file-provider config
# ------------------------------------------------------------------------------
# When set, compose-farm automatically regenerates this file after
# up/down/restart/update commands. Traefik watches this file for changes.
#
# traefik_file: /opt/compose/traefik/dynamic.d/compose-farm.yml
# ------------------------------------------------------------------------------
# traefik_service: (optional) Service name running Traefik
# ------------------------------------------------------------------------------
# When generating traefik_file, services on the same host as Traefik are
# skipped (they're handled by Traefik's Docker provider directly).
#
# traefik_service: traefik

View File

@@ -0,0 +1,461 @@
"""Command execution via SSH or locally."""
from __future__ import annotations
import asyncio
import socket
import subprocess
from dataclasses import dataclass
from functools import lru_cache
from typing import TYPE_CHECKING, Any
from rich.markup import escape
from .console import console, err_console
if TYPE_CHECKING:
from collections.abc import Callable
from .config import Config, Host
LOCAL_ADDRESSES = frozenset({"local", "localhost", "127.0.0.1", "::1"})
_DEFAULT_SSH_PORT = 22
@lru_cache(maxsize=1)
def _get_local_ips() -> frozenset[str]:
"""Get all IP addresses of the current machine."""
ips: set[str] = set()
try:
hostname = socket.gethostname()
# Get all addresses for hostname
for info in socket.getaddrinfo(hostname, None):
addr = info[4][0]
if isinstance(addr, str):
ips.add(addr)
# Also try getting the default outbound IP
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
s.connect(("8.8.8.8", 80))
ips.add(s.getsockname()[0])
except OSError:
pass
return frozenset(ips)
@dataclass
class CommandResult:
"""Result of a command execution."""
service: str
exit_code: int
success: bool
stdout: str = ""
stderr: str = ""
# SSH returns 255 when connection is closed unexpectedly (e.g., Ctrl+C)
_SSH_CONNECTION_CLOSED = 255
@property
def interrupted(self) -> bool:
"""Check if command was killed by SIGINT (Ctrl+C)."""
# Negative exit codes indicate signal termination; -2 = SIGINT
return self.exit_code < 0 or self.exit_code == self._SSH_CONNECTION_CLOSED
def is_local(host: Host) -> bool:
"""Check if host should run locally (no SSH)."""
addr = host.address.lower()
if addr in LOCAL_ADDRESSES:
return True
# Check if address matches any of this machine's IPs
return addr in _get_local_ips()
async def _run_local_command(
command: str,
service: str,
*,
stream: bool = True,
raw: bool = False,
) -> CommandResult:
"""Run a command locally with streaming output."""
try:
if raw:
# Run with inherited stdout/stderr for proper \r handling
proc = await asyncio.create_subprocess_shell(
command,
stdout=None, # Inherit
stderr=None, # Inherit
)
await proc.wait()
return CommandResult(
service=service,
exit_code=proc.returncode or 0,
success=proc.returncode == 0,
)
proc = await asyncio.create_subprocess_shell(
command,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
if stream and proc.stdout and proc.stderr:
async def read_stream(
reader: asyncio.StreamReader,
prefix: str,
*,
is_stderr: bool = False,
) -> None:
out = err_console if is_stderr else console
while True:
line = await reader.readline()
if not line:
break
text = line.decode()
if text.strip(): # Skip empty lines
out.print(f"[cyan]\\[{prefix}][/] {escape(text)}", end="")
await asyncio.gather(
read_stream(proc.stdout, service),
read_stream(proc.stderr, service, is_stderr=True),
)
stdout_data = b""
stderr_data = b""
if not stream:
stdout_data, stderr_data = await proc.communicate()
else:
await proc.wait()
return CommandResult(
service=service,
exit_code=proc.returncode or 0,
success=proc.returncode == 0,
stdout=stdout_data.decode() if stdout_data else "",
stderr=stderr_data.decode() if stderr_data else "",
)
except OSError as e:
err_console.print(f"[cyan]\\[{service}][/] [red]Local error:[/] {e}")
return CommandResult(service=service, exit_code=1, success=False)
async def _run_ssh_command(
host: Host,
command: str,
service: str,
*,
stream: bool = True,
raw: bool = False,
) -> CommandResult:
"""Run a command on a remote host via SSH with streaming output."""
if raw:
# Use native ssh with TTY for proper progress bar rendering
ssh_args = ["ssh", "-t"]
if host.port != _DEFAULT_SSH_PORT:
ssh_args.extend(["-p", str(host.port)])
ssh_args.extend([f"{host.user}@{host.address}", command])
# Run in thread to avoid blocking the event loop
result = await asyncio.to_thread(subprocess.run, ssh_args, check=False)
return CommandResult(
service=service,
exit_code=result.returncode,
success=result.returncode == 0,
)
import asyncssh # noqa: PLC0415 - lazy import for faster CLI startup
proc: asyncssh.SSHClientProcess[Any]
try:
async with asyncssh.connect( # noqa: SIM117 - conn needed before create_process
host.address,
port=host.port,
username=host.user,
known_hosts=None,
) as conn:
async with conn.create_process(command) as proc:
if stream:
async def read_stream(
reader: Any,
prefix: str,
*,
is_stderr: bool = False,
) -> None:
out = err_console if is_stderr else console
async for line in reader:
if line.strip(): # Skip empty lines
out.print(f"[cyan]\\[{prefix}][/] {escape(line)}", end="")
await asyncio.gather(
read_stream(proc.stdout, service),
read_stream(proc.stderr, service, is_stderr=True),
)
stdout_data = ""
stderr_data = ""
if not stream:
stdout_data = await proc.stdout.read()
stderr_data = await proc.stderr.read()
await proc.wait()
return CommandResult(
service=service,
exit_code=proc.exit_status or 0,
success=proc.exit_status == 0,
stdout=stdout_data,
stderr=stderr_data,
)
except (OSError, asyncssh.Error) as e:
err_console.print(f"[cyan]\\[{service}][/] [red]SSH error:[/] {e}")
return CommandResult(service=service, exit_code=1, success=False)
async def run_command(
host: Host,
command: str,
service: str,
*,
stream: bool = True,
raw: bool = False,
) -> CommandResult:
"""Run a command on a host (locally or via SSH)."""
if is_local(host):
return await _run_local_command(command, service, stream=stream, raw=raw)
return await _run_ssh_command(host, command, service, stream=stream, raw=raw)
async def run_compose(
config: Config,
service: str,
compose_cmd: str,
*,
stream: bool = True,
raw: bool = False,
) -> CommandResult:
"""Run a docker compose command for a service."""
host = config.get_host(service)
compose_path = config.get_compose_path(service)
command = f"docker compose -f {compose_path} {compose_cmd}"
return await run_command(host, command, service, stream=stream, raw=raw)
async def run_compose_on_host(
config: Config,
service: str,
host_name: str,
compose_cmd: str,
*,
stream: bool = True,
raw: bool = False,
) -> CommandResult:
"""Run a docker compose command for a service on a specific host.
Used for migration - running 'down' on the old host before 'up' on new host.
"""
host = config.hosts[host_name]
compose_path = config.get_compose_path(service)
command = f"docker compose -f {compose_path} {compose_cmd}"
return await run_command(host, command, service, stream=stream, raw=raw)
async def run_on_services(
config: Config,
services: list[str],
compose_cmd: str,
*,
stream: bool = True,
raw: bool = False,
) -> list[CommandResult]:
"""Run a docker compose command on multiple services in parallel.
For multi-host services, runs on all configured hosts.
Note: raw=True only makes sense for single-service operations.
"""
return await run_sequential_on_services(config, services, [compose_cmd], stream=stream, raw=raw)
async def _run_sequential_commands(
config: Config,
service: str,
commands: list[str],
*,
stream: bool = True,
raw: bool = False,
) -> CommandResult:
"""Run multiple compose commands sequentially for a service."""
for cmd in commands:
result = await run_compose(config, service, cmd, stream=stream, raw=raw)
if not result.success:
return result
return CommandResult(service=service, exit_code=0, success=True)
async def _run_sequential_commands_multi_host(
config: Config,
service: str,
commands: list[str],
*,
stream: bool = True,
raw: bool = False,
) -> list[CommandResult]:
"""Run multiple compose commands sequentially for a multi-host service.
Commands are run sequentially, but each command runs on all hosts in parallel.
"""
host_names = config.get_hosts(service)
compose_path = config.get_compose_path(service)
final_results: list[CommandResult] = []
for cmd in commands:
command = f"docker compose -f {compose_path} {cmd}"
tasks = []
for host_name in host_names:
host = config.hosts[host_name]
label = f"{service}@{host_name}" if len(host_names) > 1 else service
tasks.append(run_command(host, command, label, stream=stream, raw=raw))
results = await asyncio.gather(*tasks)
final_results = list(results)
# Check if any failed
if any(not r.success for r in results):
return final_results
return final_results
async def run_sequential_on_services(
config: Config,
services: list[str],
commands: list[str],
*,
stream: bool = True,
raw: bool = False,
) -> list[CommandResult]:
"""Run sequential commands on multiple services in parallel.
For multi-host services, runs on all configured hosts.
Note: raw=True only makes sense for single-service operations.
"""
# Separate multi-host and single-host services for type-safe gathering
multi_host_tasks = []
single_host_tasks = []
for service in services:
if config.is_multi_host(service):
multi_host_tasks.append(
_run_sequential_commands_multi_host(
config, service, commands, stream=stream, raw=raw
)
)
else:
single_host_tasks.append(
_run_sequential_commands(config, service, commands, stream=stream, raw=raw)
)
# Gather results separately to maintain type safety
flat_results: list[CommandResult] = []
if multi_host_tasks:
multi_results = await asyncio.gather(*multi_host_tasks)
for result_list in multi_results:
flat_results.extend(result_list)
if single_host_tasks:
single_results = await asyncio.gather(*single_host_tasks)
flat_results.extend(single_results)
return flat_results
async def check_service_running(
config: Config,
service: str,
host_name: str,
) -> bool:
"""Check if a service has running containers on a specific host."""
host = config.hosts[host_name]
compose_path = config.get_compose_path(service)
# Use ps --status running to check for running containers
command = f"docker compose -f {compose_path} ps --status running -q"
result = await run_command(host, command, service, stream=False)
# If command succeeded and has output, containers are running
return result.success and bool(result.stdout.strip())
async def _batch_check_existence(
config: Config,
host_name: str,
items: list[str],
cmd_template: Callable[[str], str],
context: str,
) -> dict[str, bool]:
"""Check existence of multiple items on a host using a command template."""
if not items:
return {}
host = config.hosts[host_name]
checks = []
for item in items:
escaped = item.replace("'", "'\\''")
checks.append(cmd_template(escaped))
command = "; ".join(checks)
result = await run_command(host, command, context, stream=False)
exists: dict[str, bool] = dict.fromkeys(items, False)
for raw_line in result.stdout.splitlines():
line = raw_line.strip()
if line.startswith("Y:"):
exists[line[2:]] = True
elif line.startswith("N:"):
exists[line[2:]] = False
return exists
async def check_paths_exist(
config: Config,
host_name: str,
paths: list[str],
) -> dict[str, bool]:
"""Check if multiple paths exist on a specific host.
Returns a dict mapping path -> exists.
Handles permission denied as "exists" (path is there, just not accessible).
"""
# Only report missing if stat says "No such file", otherwise assume exists
# (handles permission denied correctly - path exists, just not accessible)
return await _batch_check_existence(
config,
host_name,
paths,
lambda esc: f"stat '{esc}' 2>&1 | grep -q 'No such file' && echo 'N:{esc}' || echo 'Y:{esc}'",
"mount-check",
)
async def check_networks_exist(
config: Config,
host_name: str,
networks: list[str],
) -> dict[str, bool]:
"""Check if Docker networks exist on a specific host.
Returns a dict mapping network_name -> exists.
"""
return await _batch_check_existence(
config,
host_name,
networks,
lambda esc: (
f"docker network inspect '{esc}' >/dev/null 2>&1 && echo 'Y:{esc}' || echo 'N:{esc}'"
),
"network-check",
)

View File

@@ -6,20 +6,21 @@ import json
import tomllib
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import TYPE_CHECKING, Any
from .ssh import run_compose
from .executor import run_compose
from .paths import xdg_config_home
if TYPE_CHECKING:
from collections.abc import Awaitable, Callable, Iterable
from pathlib import Path
from .config import Config
from .ssh import CommandResult
from .executor import CommandResult
DEFAULT_LOG_PATH = Path.home() / ".config" / "compose-farm" / "dockerfarm-log.toml"
DIGEST_HEX_LENGTH = 64
DEFAULT_LOG_PATH = xdg_config_home() / "compose-farm" / "dockerfarm-log.toml"
_DIGEST_HEX_LENGTH = 64
@dataclass(frozen=True)
@@ -46,7 +47,8 @@ class SnapshotEntry:
}
def _isoformat(dt: datetime) -> str:
def isoformat(dt: datetime) -> str:
"""Format a datetime as an ISO 8601 string with Z suffix for UTC."""
return dt.astimezone(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z")
@@ -95,13 +97,13 @@ def _extract_image_fields(record: dict[str, Any]) -> tuple[str, str]:
or ""
)
if digest and not digest.startswith("sha256:") and len(digest) == DIGEST_HEX_LENGTH:
if digest and not digest.startswith("sha256:") and len(digest) == _DIGEST_HEX_LENGTH:
digest = f"sha256:{digest}"
return image, digest
async def _collect_service_entries(
async def collect_service_entries(
config: Config,
service: str,
*,
@@ -116,7 +118,8 @@ async def _collect_service_entries(
raise RuntimeError(error)
records = _parse_images_output(result.stdout)
host_name = config.services[service]
# Use first host for snapshots (multi-host services use same images on all hosts)
host_name = config.get_hosts(service)[0]
compose_path = config.get_compose_path(service)
entries: list[SnapshotEntry] = []
@@ -137,19 +140,21 @@ async def _collect_service_entries(
return entries
def _load_existing_entries(log_path: Path) -> list[dict[str, str]]:
def load_existing_entries(log_path: Path) -> list[dict[str, str]]:
"""Load existing snapshot entries from a TOML log file."""
if not log_path.exists():
return []
data = tomllib.loads(log_path.read_text())
return list(data.get("entries", []))
def _merge_entries(
def merge_entries(
existing: Iterable[dict[str, str]],
new_entries: Iterable[SnapshotEntry],
*,
now_iso: str,
) -> list[dict[str, str]]:
"""Merge new snapshot entries with existing ones, preserving first_seen timestamps."""
merged: dict[tuple[str, str, str], dict[str, str]] = {
(e["service"], e["host"], e["digest"]): dict(e) for e in existing
}
@@ -162,7 +167,8 @@ def _merge_entries(
return list(merged.values())
def _write_toml(log_path: Path, *, meta: dict[str, str], entries: list[dict[str, str]]) -> None:
def write_toml(log_path: Path, *, meta: dict[str, str], entries: list[dict[str, str]]) -> None:
"""Write snapshot entries to a TOML log file."""
lines: list[str] = ["[meta]"]
lines.extend(f'{key} = "{_escape(meta[key])}"' for key in sorted(meta))
@@ -187,45 +193,3 @@ def _write_toml(log_path: Path, *, meta: dict[str, str], entries: list[dict[str,
content = "\n".join(lines).rstrip() + "\n"
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text(content)
async def snapshot_services(
config: Config,
services: list[str],
*,
log_path: Path | None = None,
now: datetime | None = None,
run_compose_fn: Callable[..., Awaitable[CommandResult]] = run_compose,
) -> Path:
"""Capture current image digests for services and write them to a TOML log.
- Preserves the earliest `first_seen` per (service, host, digest)
- Updates `last_seen` for digests observed in this snapshot
- Leaves untouched digests that were not part of this run (history is kept)
"""
if not services:
error = "No services specified for snapshot"
raise RuntimeError(error)
log_path = log_path or DEFAULT_LOG_PATH
now_dt = now or datetime.now(UTC)
now_iso = _isoformat(now_dt)
existing_entries = _load_existing_entries(log_path)
snapshot_entries: list[SnapshotEntry] = []
for service in services:
snapshot_entries.extend(
await _collect_service_entries(
config, service, now=now_dt, run_compose_fn=run_compose_fn
)
)
if not snapshot_entries:
error = "No image digests were captured"
raise RuntimeError(error)
merged_entries = _merge_entries(existing_entries, snapshot_entries, now_iso=now_iso)
meta = {"generated_at": now_iso, "compose_dir": str(config.compose_dir)}
_write_toml(log_path, meta=meta, entries=merged_entries)
return log_path

View File

@@ -0,0 +1,414 @@
"""High-level operations for compose-farm.
Contains the business logic for up, down, sync, check, and migration operations.
CLI commands are thin wrappers around these functions.
"""
from __future__ import annotations
import asyncio
from typing import TYPE_CHECKING, NamedTuple
from .compose import parse_devices, parse_external_networks, parse_host_volumes
from .console import console, err_console
from .executor import (
CommandResult,
check_networks_exist,
check_paths_exist,
check_service_running,
run_command,
run_compose,
run_compose_on_host,
)
from .state import (
get_orphaned_services,
get_service_host,
remove_service,
set_multi_host_service,
set_service_host,
)
if TYPE_CHECKING:
from .config import Config
class OperationInterruptedError(Exception):
"""Raised when a command is interrupted by Ctrl+C."""
class PreflightResult(NamedTuple):
"""Result of pre-flight checks for a service on a host."""
missing_paths: list[str]
missing_networks: list[str]
missing_devices: list[str]
@property
def ok(self) -> bool:
"""Return True if all checks passed."""
return not (self.missing_paths or self.missing_networks or self.missing_devices)
async def _run_compose_step(
cfg: Config,
service: str,
command: str,
*,
raw: bool,
host: str | None = None,
) -> CommandResult:
"""Run a compose command, handle raw output newline, and check for interrupts."""
if host:
result = await run_compose_on_host(cfg, service, host, command, raw=raw)
else:
result = await run_compose(cfg, service, command, raw=raw)
if raw:
print() # Ensure newline after raw output
if result.interrupted:
raise OperationInterruptedError
return result
def get_service_paths(cfg: Config, service: str) -> list[str]:
"""Get all required paths for a service (compose_dir + volumes)."""
paths = [str(cfg.compose_dir)]
paths.extend(parse_host_volumes(cfg, service))
return paths
async def check_service_requirements(
cfg: Config,
service: str,
host_name: str,
) -> PreflightResult:
"""Check if a service can run on a specific host.
Verifies that all required paths (volumes), networks, and devices exist.
"""
# Check mount paths
paths = get_service_paths(cfg, service)
path_exists = await check_paths_exist(cfg, host_name, paths)
missing_paths = [p for p, found in path_exists.items() if not found]
# Check external networks
networks = parse_external_networks(cfg, service)
missing_networks: list[str] = []
if networks:
net_exists = await check_networks_exist(cfg, host_name, networks)
missing_networks = [n for n, found in net_exists.items() if not found]
# Check devices
devices = parse_devices(cfg, service)
missing_devices: list[str] = []
if devices:
dev_exists = await check_paths_exist(cfg, host_name, devices)
missing_devices = [d for d, found in dev_exists.items() if not found]
return PreflightResult(missing_paths, missing_networks, missing_devices)
async def _cleanup_and_rollback(
cfg: Config,
service: str,
target_host: str,
current_host: str,
prefix: str,
*,
was_running: bool,
raw: bool = False,
) -> None:
"""Clean up failed start and attempt rollback to old host if it was running."""
err_console.print(
f"{prefix} [yellow]![/] Cleaning up failed start on [magenta]{target_host}[/]"
)
await run_compose(cfg, service, "down", raw=raw)
if not was_running:
err_console.print(
f"{prefix} [dim]Service was not running on [magenta]{current_host}[/], skipping rollback[/]"
)
return
err_console.print(f"{prefix} [yellow]![/] Rolling back to [magenta]{current_host}[/]...")
rollback_result = await run_compose_on_host(cfg, service, current_host, "up -d", raw=raw)
if rollback_result.success:
console.print(f"{prefix} [green]✓[/] Rollback succeeded on [magenta]{current_host}[/]")
else:
err_console.print(f"{prefix} [red]✗[/] Rollback failed - service is down")
def _report_preflight_failures(
service: str,
target_host: str,
preflight: PreflightResult,
) -> None:
"""Report pre-flight check failures."""
err_console.print(
f"[cyan]\\[{service}][/] [red]✗[/] Cannot start on [magenta]{target_host}[/]:"
)
for path in preflight.missing_paths:
err_console.print(f" [red]✗[/] missing path: {path}")
for net in preflight.missing_networks:
err_console.print(f" [red]✗[/] missing network: {net}")
if preflight.missing_networks:
err_console.print(f" [dim]hint: cf init-network {target_host}[/]")
for dev in preflight.missing_devices:
err_console.print(f" [red]✗[/] missing device: {dev}")
async def _up_multi_host_service(
cfg: Config,
service: str,
prefix: str,
*,
raw: bool = False,
) -> list[CommandResult]:
"""Start a multi-host service on all configured hosts."""
host_names = cfg.get_hosts(service)
results: list[CommandResult] = []
compose_path = cfg.get_compose_path(service)
command = f"docker compose -f {compose_path} up -d"
# Pre-flight checks on all hosts
for host_name in host_names:
preflight = await check_service_requirements(cfg, service, host_name)
if not preflight.ok:
_report_preflight_failures(service, host_name, preflight)
results.append(
CommandResult(service=f"{service}@{host_name}", exit_code=1, success=False)
)
return results
# Start on all hosts
hosts_str = ", ".join(f"[magenta]{h}[/]" for h in host_names)
console.print(f"{prefix} Starting on {hosts_str}...")
succeeded_hosts: list[str] = []
for host_name in host_names:
host = cfg.hosts[host_name]
label = f"{service}@{host_name}"
result = await run_command(host, command, label, stream=not raw, raw=raw)
if raw:
print() # Ensure newline after raw output
results.append(result)
if result.success:
succeeded_hosts.append(host_name)
# Update state with hosts that succeeded (partial success is tracked)
if succeeded_hosts:
set_multi_host_service(cfg, service, succeeded_hosts)
return results
async def _migrate_service(
cfg: Config,
service: str,
current_host: str,
target_host: str,
prefix: str,
*,
raw: bool = False,
) -> CommandResult | None:
"""Migrate a service from current_host to target_host.
Pre-pulls/builds images on target, then stops service on current host.
Returns failure result if migration prep fails, None on success.
"""
console.print(
f"{prefix} Migrating from [magenta]{current_host}[/] → [magenta]{target_host}[/]..."
)
# Prepare images on target host before stopping old service to minimize downtime.
# Pull handles image-based services; build handles Dockerfile-based services.
# --ignore-buildable makes pull skip images that have build: defined.
for cmd, label in [("pull --ignore-buildable", "Pull"), ("build", "Build")]:
result = await _run_compose_step(cfg, service, cmd, raw=raw)
if not result.success:
err_console.print(
f"{prefix} [red]✗[/] {label} failed on [magenta]{target_host}[/], "
"leaving service on current host"
)
return result
# Stop on current host
down_result = await _run_compose_step(cfg, service, "down", raw=raw, host=current_host)
return down_result if not down_result.success else None
async def _up_single_service(
cfg: Config,
service: str,
prefix: str,
*,
raw: bool,
) -> CommandResult:
"""Start a single-host service with migration support."""
target_host = cfg.get_hosts(service)[0]
current_host = get_service_host(cfg, service)
# Pre-flight check: verify paths, networks, and devices exist on target
preflight = await check_service_requirements(cfg, service, target_host)
if not preflight.ok:
_report_preflight_failures(service, target_host, preflight)
return CommandResult(service=service, exit_code=1, success=False)
# If service is deployed elsewhere, migrate it
did_migration = False
was_running = False
if current_host and current_host != target_host:
if current_host in cfg.hosts:
was_running = await check_service_running(cfg, service, current_host)
failure = await _migrate_service(
cfg, service, current_host, target_host, prefix, raw=raw
)
if failure:
return failure
did_migration = True
else:
err_console.print(
f"{prefix} [yellow]![/] was on "
f"[magenta]{current_host}[/] (not in config), skipping down"
)
# Start on target host
console.print(f"{prefix} Starting on [magenta]{target_host}[/]...")
up_result = await _run_compose_step(cfg, service, "up -d", raw=raw)
# Update state on success, or rollback on failure
if up_result.success:
set_service_host(cfg, service, target_host)
elif did_migration and current_host:
await _cleanup_and_rollback(
cfg,
service,
target_host,
current_host,
prefix,
was_running=was_running,
raw=raw,
)
return up_result
async def up_services(
cfg: Config,
services: list[str],
*,
raw: bool = False,
) -> list[CommandResult]:
"""Start services with automatic migration if host changed."""
results: list[CommandResult] = []
total = len(services)
try:
for idx, service in enumerate(services, 1):
prefix = f"[dim][{idx}/{total}][/] [cyan]\\[{service}][/]"
if cfg.is_multi_host(service):
results.extend(await _up_multi_host_service(cfg, service, prefix, raw=raw))
else:
results.append(await _up_single_service(cfg, service, prefix, raw=raw))
except OperationInterruptedError:
raise KeyboardInterrupt from None
return results
async def check_host_compatibility(
cfg: Config,
service: str,
) -> dict[str, tuple[int, int, list[str]]]:
"""Check which hosts can run a service based on paths, networks, and devices.
Returns dict of host_name -> (found_count, total_count, missing_items).
"""
# Get total requirements count
paths = get_service_paths(cfg, service)
networks = parse_external_networks(cfg, service)
devices = parse_devices(cfg, service)
total = len(paths) + len(networks) + len(devices)
results: dict[str, tuple[int, int, list[str]]] = {}
for host_name in cfg.hosts:
preflight = await check_service_requirements(cfg, service, host_name)
all_missing = (
preflight.missing_paths + preflight.missing_networks + preflight.missing_devices
)
found = total - len(all_missing)
results[host_name] = (found, total, all_missing)
return results
async def stop_orphaned_services(cfg: Config) -> list[CommandResult]:
"""Stop orphaned services (in state but not in config).
Runs docker compose down on each service on its tracked host(s).
Only removes from state on successful stop.
Returns list of CommandResults for each service@host.
"""
orphaned = get_orphaned_services(cfg)
if not orphaned:
return []
results: list[CommandResult] = []
tasks: list[tuple[str, str, asyncio.Task[CommandResult]]] = []
# Build list of (service, host, task) for all orphaned services
for service, hosts in orphaned.items():
host_list = hosts if isinstance(hosts, list) else [hosts]
for host in host_list:
# Skip hosts no longer in config
if host not in cfg.hosts:
console.print(
f" [yellow]![/] {service}@{host}: host no longer in config, skipping"
)
results.append(
CommandResult(
service=f"{service}@{host}",
exit_code=1,
success=False,
stderr="host no longer in config",
)
)
continue
coro = run_compose_on_host(cfg, service, host, "down")
tasks.append((service, host, asyncio.create_task(coro)))
# Run all down commands in parallel
if tasks:
for service, host, task in tasks:
try:
result = await task
results.append(result)
if result.success:
console.print(f" [green]✓[/] {service}@{host}: stopped")
else:
console.print(f" [red]✗[/] {service}@{host}: {result.stderr or 'failed'}")
except Exception as e:
console.print(f" [red]✗[/] {service}@{host}: {e}")
results.append(
CommandResult(
service=f"{service}@{host}",
exit_code=1,
success=False,
stderr=str(e),
)
)
# Remove from state only for services where ALL hosts succeeded
for service, hosts in orphaned.items():
host_list = hosts if isinstance(hosts, list) else [hosts]
all_succeeded = all(
r.success
for r in results
if r.service.startswith(f"{service}@") or r.service == service
)
if all_succeeded:
remove_service(cfg, service)
return results

21
src/compose_farm/paths.py Normal file
View File

@@ -0,0 +1,21 @@
"""Path utilities - lightweight module with no heavy dependencies."""
from __future__ import annotations
import os
from pathlib import Path
def xdg_config_home() -> Path:
"""Get XDG config directory, respecting XDG_CONFIG_HOME env var."""
return Path(os.environ.get("XDG_CONFIG_HOME", Path.home() / ".config"))
def default_config_path() -> Path:
"""Get the default user config path."""
return xdg_config_home() / "compose-farm" / "compose-farm.yaml"
def config_search_paths() -> list[Path]:
"""Get search paths for config files."""
return [Path("compose-farm.yaml"), default_config_path()]

View File

@@ -1,244 +0,0 @@
"""Command execution via SSH or locally."""
from __future__ import annotations
import asyncio
import sys
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
import asyncssh
if TYPE_CHECKING:
from .config import Config, Host
LOCAL_ADDRESSES = frozenset({"local", "localhost", "127.0.0.1", "::1"})
@dataclass
class CommandResult:
"""Result of a command execution."""
service: str
exit_code: int
success: bool
stdout: str = ""
stderr: str = ""
def _is_local(host: Host) -> bool:
"""Check if host should run locally (no SSH)."""
return host.address.lower() in LOCAL_ADDRESSES
async def _run_local_command(
command: str,
service: str,
*,
stream: bool = True,
) -> CommandResult:
"""Run a command locally with streaming output."""
try:
proc = await asyncio.create_subprocess_shell(
command,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
if stream and proc.stdout and proc.stderr:
async def read_stream(
reader: asyncio.StreamReader,
prefix: str,
*,
is_stderr: bool = False,
) -> None:
output = sys.stderr if is_stderr else sys.stdout
while True:
line = await reader.readline()
if not line:
break
print(f"[{prefix}] {line.decode()}", end="", file=output, flush=True)
await asyncio.gather(
read_stream(proc.stdout, service),
read_stream(proc.stderr, service, is_stderr=True),
)
stdout_data = b""
stderr_data = b""
if not stream:
stdout_data, stderr_data = await proc.communicate()
else:
await proc.wait()
return CommandResult(
service=service,
exit_code=proc.returncode or 0,
success=proc.returncode == 0,
stdout=stdout_data.decode() if stdout_data else "",
stderr=stderr_data.decode() if stderr_data else "",
)
except OSError as e:
print(f"[{service}] Local error: {e}", file=sys.stderr)
return CommandResult(service=service, exit_code=1, success=False)
async def _run_ssh_command(
host: Host,
command: str,
service: str,
*,
stream: bool = True,
) -> CommandResult:
"""Run a command on a remote host via SSH with streaming output."""
proc: asyncssh.SSHClientProcess[Any]
try:
async with (
asyncssh.connect(
host.address,
port=host.port,
username=host.user,
known_hosts=None,
) as conn,
conn.create_process(command) as proc,
):
if stream:
async def read_stream(
reader: Any,
prefix: str,
*,
is_stderr: bool = False,
) -> None:
output = sys.stderr if is_stderr else sys.stdout
async for line in reader:
print(f"[{prefix}] {line}", end="", file=output, flush=True)
await asyncio.gather(
read_stream(proc.stdout, service),
read_stream(proc.stderr, service, is_stderr=True),
)
stdout_data = ""
stderr_data = ""
if not stream:
stdout_data = await proc.stdout.read()
stderr_data = await proc.stderr.read()
await proc.wait()
return CommandResult(
service=service,
exit_code=proc.exit_status or 0,
success=proc.exit_status == 0,
stdout=stdout_data,
stderr=stderr_data,
)
except (OSError, asyncssh.Error) as e:
print(f"[{service}] SSH error: {e}", file=sys.stderr)
return CommandResult(service=service, exit_code=1, success=False)
async def run_command(
host: Host,
command: str,
service: str,
*,
stream: bool = True,
) -> CommandResult:
"""Run a command on a host (locally or via SSH)."""
if _is_local(host):
return await _run_local_command(command, service, stream=stream)
return await _run_ssh_command(host, command, service, stream=stream)
async def run_compose(
config: Config,
service: str,
compose_cmd: str,
*,
stream: bool = True,
) -> CommandResult:
"""Run a docker compose command for a service."""
host = config.get_host(service)
compose_path = config.get_compose_path(service)
command = f"docker compose -f {compose_path} {compose_cmd}"
return await run_command(host, command, service, stream=stream)
async def run_compose_on_host(
config: Config,
service: str,
host_name: str,
compose_cmd: str,
*,
stream: bool = True,
) -> CommandResult:
"""Run a docker compose command for a service on a specific host.
Used for migration - running 'down' on the old host before 'up' on new host.
"""
host = config.hosts[host_name]
compose_path = config.get_compose_path(service)
command = f"docker compose -f {compose_path} {compose_cmd}"
return await run_command(host, command, service, stream=stream)
async def run_on_services(
config: Config,
services: list[str],
compose_cmd: str,
*,
stream: bool = True,
) -> list[CommandResult]:
"""Run a docker compose command on multiple services in parallel."""
tasks = [run_compose(config, service, compose_cmd, stream=stream) for service in services]
return await asyncio.gather(*tasks)
async def run_sequential_commands(
config: Config,
service: str,
commands: list[str],
*,
stream: bool = True,
) -> CommandResult:
"""Run multiple compose commands sequentially for a service."""
for cmd in commands:
result = await run_compose(config, service, cmd, stream=stream)
if not result.success:
return result
return CommandResult(service=service, exit_code=0, success=True)
async def run_sequential_on_services(
config: Config,
services: list[str],
commands: list[str],
*,
stream: bool = True,
) -> list[CommandResult]:
"""Run sequential commands on multiple services in parallel."""
tasks = [
run_sequential_commands(config, service, commands, stream=stream) for service in services
]
return await asyncio.gather(*tasks)
async def check_service_running(
config: Config,
service: str,
host_name: str,
) -> bool:
"""Check if a service has running containers on a specific host."""
host = config.hosts[host_name]
compose_path = config.get_compose_path(service)
# Use ps --status running to check for running containers
command = f"docker compose -f {compose_path} ps --status running -q"
result = await run_command(host, command, service, stream=False)
# If command succeeded and has output, containers are running
return result.success and bool(result.stdout.strip())

View File

@@ -2,57 +2,165 @@
from __future__ import annotations
from pathlib import Path
from typing import Any
import contextlib
from typing import TYPE_CHECKING, Any
import yaml
if TYPE_CHECKING:
from collections.abc import Generator
def _get_state_path() -> Path:
"""Get the path to the state file."""
state_dir = Path.home() / ".config" / "compose-farm"
state_dir.mkdir(parents=True, exist_ok=True)
return state_dir / "state.yaml"
from .config import Config
def load_state() -> dict[str, str]:
def load_state(config: Config) -> dict[str, str | list[str]]:
"""Load the current deployment state.
Returns a dict mapping service names to host names.
Returns a dict mapping service names to host name(s).
Multi-host services store a list of hosts.
"""
state_path = _get_state_path()
state_path = config.get_state_path()
if not state_path.exists():
return {}
with state_path.open() as f:
data: dict[str, Any] = yaml.safe_load(f) or {}
deployed: dict[str, str] = data.get("deployed", {})
deployed: dict[str, str | list[str]] = data.get("deployed", {})
return deployed
def save_state(deployed: dict[str, str]) -> None:
def _sorted_dict(d: dict[str, str | list[str]]) -> dict[str, str | list[str]]:
"""Return a dictionary sorted by keys."""
return dict(sorted(d.items(), key=lambda item: item[0]))
def save_state(config: Config, deployed: dict[str, str | list[str]]) -> None:
"""Save the deployment state."""
state_path = _get_state_path()
state_path = config.get_state_path()
with state_path.open("w") as f:
yaml.safe_dump({"deployed": deployed}, f, sort_keys=False)
yaml.safe_dump({"deployed": _sorted_dict(deployed)}, f, sort_keys=False)
def get_service_host(service: str) -> str | None:
"""Get the host where a service is currently deployed."""
state = load_state()
return state.get(service)
@contextlib.contextmanager
def _modify_state(config: Config) -> Generator[dict[str, str | list[str]], None, None]:
"""Context manager to load, modify, and save state."""
state = load_state(config)
yield state
save_state(config, state)
def set_service_host(service: str, host: str) -> None:
def get_service_host(config: Config, service: str) -> str | None:
"""Get the host where a service is currently deployed.
For multi-host services, returns the first host or None.
"""
state = load_state(config)
value = state.get(service)
if value is None:
return None
if isinstance(value, list):
return value[0] if value else None
return value
def set_service_host(config: Config, service: str, host: str) -> None:
"""Record that a service is deployed on a host."""
state = load_state()
state[service] = host
save_state(state)
with _modify_state(config) as state:
state[service] = host
def remove_service(service: str) -> None:
def set_multi_host_service(config: Config, service: str, hosts: list[str]) -> None:
"""Record that a multi-host service is deployed on multiple hosts."""
with _modify_state(config) as state:
state[service] = hosts
def remove_service(config: Config, service: str) -> None:
"""Remove a service from the state (after down)."""
state = load_state()
state.pop(service, None)
save_state(state)
with _modify_state(config) as state:
state.pop(service, None)
def add_service_to_host(config: Config, service: str, host: str) -> None:
"""Add a specific host to a service's state.
For multi-host services, adds the host to the list if not present.
For single-host services, sets the host.
"""
with _modify_state(config) as state:
current = state.get(service)
if config.is_multi_host(service):
# Multi-host: add to list if not present
if isinstance(current, list):
if host not in current:
state[service] = [*current, host]
else:
state[service] = [host]
else:
# Single-host: just set it
state[service] = host
def remove_service_from_host(config: Config, service: str, host: str) -> None:
"""Remove a specific host from a service's state.
For multi-host services, removes just that host from the list.
For single-host services, removes the service entirely if host matches.
"""
with _modify_state(config) as state:
current = state.get(service)
if current is None:
return
if isinstance(current, list):
# Multi-host: remove this host from list
remaining = [h for h in current if h != host]
if remaining:
state[service] = remaining
else:
state.pop(service, None)
elif current == host:
# Single-host: remove if matches
state.pop(service, None)
def get_services_needing_migration(config: Config) -> list[str]:
"""Get services where current host differs from configured host.
Multi-host services are never considered for migration.
"""
needs_migration = []
for service in config.services:
# Skip multi-host services
if config.is_multi_host(service):
continue
configured_host = config.get_hosts(service)[0]
current_host = get_service_host(config, service)
if current_host and current_host != configured_host:
needs_migration.append(service)
return needs_migration
def get_orphaned_services(config: Config) -> dict[str, str | list[str]]:
"""Get services that are in state but not in config.
These are services that were previously deployed but have been
removed from the config file (e.g., commented out).
Returns a dict mapping service name to host(s) where it's deployed.
"""
state = load_state(config)
return {service: hosts for service, hosts in state.items() if service not in config.services}
def get_services_not_in_state(config: Config) -> list[str]:
"""Get services that are in config but not in state.
These are services that should be running but aren't tracked
(e.g., newly added to config, or previously stopped as orphans).
"""
state = load_state(config)
return [service for service in config.services if service not in state]

View File

@@ -8,30 +8,25 @@ use host-published ports for cross-host reachability.
from __future__ import annotations
import os
import re
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
import yaml
if TYPE_CHECKING:
from pathlib import Path
from .compose import (
PortMapping,
get_ports_for_service,
load_compose_services,
normalize_labels,
)
from .executor import LOCAL_ADDRESSES
if TYPE_CHECKING:
from .config import Config
@dataclass(frozen=True)
class PortMapping:
"""Port mapping for a compose service."""
target: int
published: int | None
protocol: str | None = None
@dataclass
class TraefikServiceSource:
class _TraefikServiceSource:
"""Source information to build an upstream for a Traefik service."""
traefik_service: str
@@ -43,129 +38,9 @@ class TraefikServiceSource:
scheme: str | None = None
LIST_VALUE_KEYS = {"entrypoints", "middlewares"}
SINGLE_PART = 1
PUBLISHED_TARGET_PARTS = 2
HOST_PUBLISHED_PARTS = 3
MIN_ROUTER_PARTS = 3
MIN_SERVICE_LABEL_PARTS = 6
_VAR_PATTERN = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)(?::-(.*?))?\}")
def _load_env(compose_path: Path) -> dict[str, str]:
"""Load environment variables for compose interpolation."""
env: dict[str, str] = {}
env_path = compose_path.parent / ".env"
if env_path.exists():
for line in env_path.read_text().splitlines():
stripped = line.strip()
if not stripped or stripped.startswith("#") or "=" not in stripped:
continue
key, value = stripped.split("=", 1)
key = key.strip()
value = value.strip()
if (value.startswith('"') and value.endswith('"')) or (
value.startswith("'") and value.endswith("'")
):
value = value[1:-1]
env[key] = value
env.update({k: v for k, v in os.environ.items() if isinstance(v, str)})
return env
def _interpolate(value: str, env: dict[str, str]) -> str:
"""Perform a minimal `${VAR}`/`${VAR:-default}` interpolation."""
def replace(match: re.Match[str]) -> str:
var = match.group(1)
default = match.group(2)
resolved = env.get(var)
if resolved:
return resolved
return default or ""
return _VAR_PATTERN.sub(replace, value)
def _normalize_labels(raw: Any, env: dict[str, str]) -> dict[str, str]:
if raw is None:
return {}
if isinstance(raw, dict):
return {
_interpolate(str(k), env): _interpolate(str(v), env)
for k, v in raw.items()
if k is not None
}
if isinstance(raw, list):
labels: dict[str, str] = {}
for item in raw:
if not isinstance(item, str) or "=" not in item:
continue
key_raw, value_raw = item.split("=", 1)
key = _interpolate(key_raw.strip(), env)
value = _interpolate(value_raw.strip(), env)
labels[key] = value
return labels
return {}
def _parse_ports(raw: Any, env: dict[str, str]) -> list[PortMapping]: # noqa: PLR0912
if raw is None:
return []
mappings: list[PortMapping] = []
items = raw if isinstance(raw, list) else [raw]
for item in items:
if isinstance(item, str):
interpolated = _interpolate(item, env)
port_spec, _, protocol = interpolated.partition("/")
parts = port_spec.split(":")
published: int | None = None
target: int | None = None
if len(parts) == SINGLE_PART and parts[0].isdigit():
target = int(parts[0])
elif len(parts) == PUBLISHED_TARGET_PARTS and parts[0].isdigit() and parts[1].isdigit():
published = int(parts[0])
target = int(parts[1])
elif len(parts) == HOST_PUBLISHED_PARTS and parts[-2].isdigit() and parts[-1].isdigit():
published = int(parts[-2])
target = int(parts[-1])
if target is not None:
mappings.append(
PortMapping(target=target, published=published, protocol=protocol or None)
)
elif isinstance(item, dict):
target_raw = item.get("target")
if isinstance(target_raw, str):
target_raw = _interpolate(target_raw, env)
if target_raw is None:
continue
try:
target_val = int(str(target_raw))
except (TypeError, ValueError):
continue
published_raw = item.get("published")
if isinstance(published_raw, str):
published_raw = _interpolate(published_raw, env)
published_val: int | None
try:
published_val = int(str(published_raw)) if published_raw is not None else None
except (TypeError, ValueError):
published_val = None
protocol_val = item.get("protocol")
mappings.append(
PortMapping(
target=target_val,
published=published_val,
protocol=str(protocol_val) if protocol_val else None,
)
)
return mappings
_LIST_VALUE_KEYS = {"entrypoints", "middlewares"}
_MIN_ROUTER_PARTS = 3
_MIN_SERVICE_LABEL_PARTS = 6
def _parse_value(key: str, raw_value: str) -> Any:
@@ -176,7 +51,7 @@ def _parse_value(key: str, raw_value: str) -> Any:
if value.isdigit():
return int(value)
last_segment = key.rsplit(".", 1)[-1]
if last_segment in LIST_VALUE_KEYS:
if last_segment in _LIST_VALUE_KEYS:
parts = [v.strip() for v in value.split(",")] if "," in value else [value]
return [part for part in parts if part]
return value
@@ -227,7 +102,7 @@ def _insert(root: dict[str, Any], key_path: list[str], value: Any) -> None: # n
current = container_list[list_index]
def _resolve_published_port(source: TraefikServiceSource) -> tuple[int | None, str | None]:
def _resolve_published_port(source: _TraefikServiceSource) -> tuple[int | None, str | None]:
"""Resolve host-published port for a Traefik service.
Returns (published_port, warning_message).
@@ -263,23 +138,9 @@ def _resolve_published_port(source: TraefikServiceSource) -> tuple[int | None, s
)
def _load_stack(config: Config, stack: str) -> tuple[dict[str, Any], dict[str, str], str]:
compose_path = config.get_compose_path(stack)
if not compose_path.exists():
message = f"[{stack}] Compose file not found: {compose_path}"
raise FileNotFoundError(message)
env = _load_env(compose_path)
compose_data = yaml.safe_load(compose_path.read_text()) or {}
raw_services = compose_data.get("services", {})
if not isinstance(raw_services, dict):
return {}, env, config.get_host(stack).address
return raw_services, env, config.get_host(stack).address
def _finalize_http_services(
dynamic: dict[str, Any],
sources: dict[str, TraefikServiceSource],
sources: dict[str, _TraefikServiceSource],
warnings: list[str],
) -> None:
for traefik_service, source in sources.items():
@@ -289,8 +150,8 @@ def _finalize_http_services(
if published_port is None:
warnings.append(
f"[{source.stack}/{source.compose_service}] "
f"No host-published port found for Traefik service '{traefik_service}'. "
"Traefik will require L3 reachability to container IPs."
f"No published port found for Traefik service '{traefik_service}'. "
"Add a ports: mapping (e.g., '8080:8080') for cross-host routing."
)
continue
@@ -350,7 +211,7 @@ def _process_router_label(
if not key_without_prefix.startswith("http.routers."):
return
router_parts = key_without_prefix.split(".")
if len(router_parts) < MIN_ROUTER_PARTS:
if len(router_parts) < _MIN_ROUTER_PARTS:
return
router_name = router_parts[2]
router_remainder = router_parts[3:]
@@ -368,12 +229,12 @@ def _process_service_label(
host_address: str,
ports: list[PortMapping],
service_names: set[str],
sources: dict[str, TraefikServiceSource],
sources: dict[str, _TraefikServiceSource],
) -> None:
if not key_without_prefix.startswith("http.services."):
return
parts = key_without_prefix.split(".")
if len(parts) < MIN_SERVICE_LABEL_PARTS:
if len(parts) < _MIN_SERVICE_LABEL_PARTS:
return
traefik_service = parts[2]
service_names.add(traefik_service)
@@ -381,7 +242,7 @@ def _process_service_label(
source = sources.get(traefik_service)
if source is None:
source = TraefikServiceSource(
source = _TraefikServiceSource(
traefik_service=traefik_service,
stack=stack,
compose_service=compose_service,
@@ -402,20 +263,21 @@ def _process_service_labels(
stack: str,
compose_service: str,
definition: dict[str, Any],
all_services: dict[str, Any],
host_address: str,
env: dict[str, str],
dynamic: dict[str, Any],
sources: dict[str, TraefikServiceSource],
sources: dict[str, _TraefikServiceSource],
warnings: list[str],
) -> None:
labels = _normalize_labels(definition.get("labels"), env)
labels = normalize_labels(definition.get("labels"), env)
if not labels:
return
enable_raw = labels.get("traefik.enable")
if enable_raw is not None and _parse_value("enable", enable_raw) is False:
return
ports = _parse_ports(definition.get("ports"), env)
ports = get_ports_for_service(definition, all_services, env)
routers: dict[str, bool] = {}
service_names: set[str] = set()
@@ -450,17 +312,41 @@ def _process_service_labels(
def generate_traefik_config(
config: Config,
services: list[str],
*,
check_all: bool = False,
) -> tuple[dict[str, Any], list[str]]:
"""Generate Traefik dynamic config from compose labels.
Args:
config: The compose-farm config.
services: List of service names to process.
check_all: If True, check all services for warnings (ignore host filtering).
Used by the check command to validate all traefik labels.
Returns (config_dict, warnings).
"""
dynamic: dict[str, Any] = {}
warnings: list[str] = []
sources: dict[str, TraefikServiceSource] = {}
sources: dict[str, _TraefikServiceSource] = {}
# Determine Traefik's host from service assignment
traefik_host = None
if config.traefik_service and not check_all:
traefik_host = config.services.get(config.traefik_service)
for stack in services:
raw_services, env, host_address = _load_stack(config, stack)
raw_services, env, host_address = load_compose_services(config, stack)
stack_host = config.services.get(stack)
# Skip services on Traefik's host - docker provider handles them directly
# (unless check_all is True, for validation purposes)
if not check_all:
if host_address.lower() in LOCAL_ADDRESSES:
continue
if traefik_host and stack_host == traefik_host:
continue
for compose_service, definition in raw_services.items():
if not isinstance(definition, dict):
continue
@@ -468,6 +354,7 @@ def generate_traefik_config(
stack,
compose_service,
definition,
raw_services,
host_address,
env,
dynamic,
@@ -477,3 +364,22 @@ def generate_traefik_config(
_finalize_http_services(dynamic, sources, warnings)
return dynamic, warnings
_TRAEFIK_CONFIG_HEADER = """\
# Auto-generated by compose-farm
# https://github.com/basnijholt/compose-farm
#
# This file routes traffic to services running on hosts other than Traefik's host.
# Services on Traefik's host use the Docker provider directly.
#
# Regenerate with: compose-farm traefik-file --all -o <this-file>
# Or configure traefik_file in compose-farm.yaml for automatic updates.
"""
def render_traefik_config(dynamic: dict[str, Any]) -> str:
"""Render Traefik dynamic config as YAML with a header comment."""
body = yaml.safe_dump(dynamic, sort_keys=False)
return _TRAEFIK_CONFIG_HEADER + body

426
tests/test_cli_lifecycle.py Normal file
View File

@@ -0,0 +1,426 @@
"""Tests for CLI lifecycle commands (apply, down --orphaned)."""
from pathlib import Path
from unittest.mock import patch
import pytest
import typer
from compose_farm.cli.lifecycle import apply, down
from compose_farm.config import Config, Host
from compose_farm.executor import CommandResult
def _make_config(tmp_path: Path, services: dict[str, str] | None = None) -> Config:
"""Create a minimal config for testing."""
compose_dir = tmp_path / "compose"
compose_dir.mkdir()
svc_dict = services or {"svc1": "host1", "svc2": "host2"}
for svc in svc_dict:
svc_dir = compose_dir / svc
svc_dir.mkdir()
(svc_dir / "docker-compose.yml").write_text("services: {}\n")
config_path = tmp_path / "compose-farm.yaml"
config_path.write_text("")
return Config(
compose_dir=compose_dir,
hosts={"host1": Host(address="localhost"), "host2": Host(address="localhost")},
services=svc_dict,
config_path=config_path,
)
def _make_result(service: str, success: bool = True) -> CommandResult:
"""Create a command result."""
return CommandResult(
service=service,
exit_code=0 if success else 1,
success=success,
stdout="",
stderr="",
)
class TestApplyCommand:
"""Tests for the apply command."""
def test_apply_nothing_to_do(self, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
"""When no migrations, orphans, or missing services, prints success message."""
cfg = _make_config(tmp_path)
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.lifecycle.get_orphaned_services", return_value={}),
patch("compose_farm.cli.lifecycle.get_services_needing_migration", return_value=[]),
patch("compose_farm.cli.lifecycle.get_services_not_in_state", return_value=[]),
):
apply(dry_run=False, no_orphans=False, full=False, config=None)
captured = capsys.readouterr()
assert "Nothing to apply" in captured.out
def test_apply_dry_run_shows_preview(
self, tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Dry run shows what would be done without executing."""
cfg = _make_config(tmp_path)
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch(
"compose_farm.cli.lifecycle.get_orphaned_services",
return_value={"old-svc": "host1"},
),
patch(
"compose_farm.cli.lifecycle.get_services_needing_migration",
return_value=["svc1"],
),
patch("compose_farm.cli.lifecycle.get_services_not_in_state", return_value=[]),
patch("compose_farm.cli.lifecycle.get_service_host", return_value="host1"),
patch("compose_farm.cli.lifecycle.stop_orphaned_services") as mock_stop,
patch("compose_farm.cli.lifecycle.up_services") as mock_up,
):
apply(dry_run=True, no_orphans=False, full=False, config=None)
captured = capsys.readouterr()
assert "Services to migrate" in captured.out
assert "svc1" in captured.out
assert "Orphaned services to stop" in captured.out
assert "old-svc" in captured.out
assert "dry-run" in captured.out
# Should not have called the actual operations
mock_stop.assert_not_called()
mock_up.assert_not_called()
def test_apply_executes_migrations(self, tmp_path: Path) -> None:
"""Apply runs migrations when services need migration."""
cfg = _make_config(tmp_path)
mock_results = [_make_result("svc1")]
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.lifecycle.get_orphaned_services", return_value={}),
patch(
"compose_farm.cli.lifecycle.get_services_needing_migration",
return_value=["svc1"],
),
patch("compose_farm.cli.lifecycle.get_services_not_in_state", return_value=[]),
patch("compose_farm.cli.lifecycle.get_service_host", return_value="host1"),
patch(
"compose_farm.cli.lifecycle.run_async",
return_value=mock_results,
),
patch("compose_farm.cli.lifecycle.up_services") as mock_up,
patch("compose_farm.cli.lifecycle.maybe_regenerate_traefik"),
patch("compose_farm.cli.lifecycle.report_results"),
):
apply(dry_run=False, no_orphans=False, full=False, config=None)
mock_up.assert_called_once()
call_args = mock_up.call_args
assert call_args[0][1] == ["svc1"] # services list
def test_apply_executes_orphan_cleanup(self, tmp_path: Path) -> None:
"""Apply stops orphaned services."""
cfg = _make_config(tmp_path)
mock_results = [_make_result("old-svc@host1")]
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch(
"compose_farm.cli.lifecycle.get_orphaned_services",
return_value={"old-svc": "host1"},
),
patch("compose_farm.cli.lifecycle.get_services_needing_migration", return_value=[]),
patch("compose_farm.cli.lifecycle.get_services_not_in_state", return_value=[]),
patch(
"compose_farm.cli.lifecycle.run_async",
return_value=mock_results,
),
patch("compose_farm.cli.lifecycle.stop_orphaned_services") as mock_stop,
patch("compose_farm.cli.lifecycle.report_results"),
):
apply(dry_run=False, no_orphans=False, full=False, config=None)
mock_stop.assert_called_once_with(cfg)
def test_apply_no_orphans_skips_orphan_cleanup(
self, tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""--no-orphans flag skips orphan cleanup."""
cfg = _make_config(tmp_path)
mock_results = [_make_result("svc1")]
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch(
"compose_farm.cli.lifecycle.get_orphaned_services",
return_value={"old-svc": "host1"},
),
patch(
"compose_farm.cli.lifecycle.get_services_needing_migration",
return_value=["svc1"],
),
patch("compose_farm.cli.lifecycle.get_services_not_in_state", return_value=[]),
patch("compose_farm.cli.lifecycle.get_service_host", return_value="host1"),
patch(
"compose_farm.cli.lifecycle.run_async",
return_value=mock_results,
),
patch("compose_farm.cli.lifecycle.up_services") as mock_up,
patch("compose_farm.cli.lifecycle.stop_orphaned_services") as mock_stop,
patch("compose_farm.cli.lifecycle.maybe_regenerate_traefik"),
patch("compose_farm.cli.lifecycle.report_results"),
):
apply(dry_run=False, no_orphans=True, full=False, config=None)
# Should run migrations but not orphan cleanup
mock_up.assert_called_once()
mock_stop.assert_not_called()
# Orphans should not appear in output
captured = capsys.readouterr()
assert "old-svc" not in captured.out
def test_apply_no_orphans_nothing_to_do(
self, tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""--no-orphans with only orphans means nothing to do."""
cfg = _make_config(tmp_path)
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch(
"compose_farm.cli.lifecycle.get_orphaned_services",
return_value={"old-svc": "host1"},
),
patch("compose_farm.cli.lifecycle.get_services_needing_migration", return_value=[]),
patch("compose_farm.cli.lifecycle.get_services_not_in_state", return_value=[]),
):
apply(dry_run=False, no_orphans=True, full=False, config=None)
captured = capsys.readouterr()
assert "Nothing to apply" in captured.out
def test_apply_starts_missing_services(self, tmp_path: Path) -> None:
"""Apply starts services that are in config but not in state."""
cfg = _make_config(tmp_path)
mock_results = [_make_result("svc1")]
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.lifecycle.get_orphaned_services", return_value={}),
patch("compose_farm.cli.lifecycle.get_services_needing_migration", return_value=[]),
patch(
"compose_farm.cli.lifecycle.get_services_not_in_state",
return_value=["svc1"],
),
patch(
"compose_farm.cli.lifecycle.run_async",
return_value=mock_results,
),
patch("compose_farm.cli.lifecycle.up_services") as mock_up,
patch("compose_farm.cli.lifecycle.maybe_regenerate_traefik"),
patch("compose_farm.cli.lifecycle.report_results"),
):
apply(dry_run=False, no_orphans=False, full=False, config=None)
mock_up.assert_called_once()
call_args = mock_up.call_args
assert call_args[0][1] == ["svc1"]
def test_apply_dry_run_shows_missing_services(
self, tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Dry run shows services that would be started."""
cfg = _make_config(tmp_path)
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.lifecycle.get_orphaned_services", return_value={}),
patch("compose_farm.cli.lifecycle.get_services_needing_migration", return_value=[]),
patch(
"compose_farm.cli.lifecycle.get_services_not_in_state",
return_value=["svc1"],
),
):
apply(dry_run=True, no_orphans=False, full=False, config=None)
captured = capsys.readouterr()
assert "Services to start" in captured.out
assert "svc1" in captured.out
assert "dry-run" in captured.out
def test_apply_full_refreshes_all_services(self, tmp_path: Path) -> None:
"""--full runs up on all services to pick up config changes."""
cfg = _make_config(tmp_path)
mock_results = [_make_result("svc1"), _make_result("svc2")]
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.lifecycle.get_orphaned_services", return_value={}),
patch("compose_farm.cli.lifecycle.get_services_needing_migration", return_value=[]),
patch("compose_farm.cli.lifecycle.get_services_not_in_state", return_value=[]),
patch(
"compose_farm.cli.lifecycle.run_async",
return_value=mock_results,
),
patch("compose_farm.cli.lifecycle.up_services") as mock_up,
patch("compose_farm.cli.lifecycle.maybe_regenerate_traefik"),
patch("compose_farm.cli.lifecycle.report_results"),
):
apply(dry_run=False, no_orphans=False, full=True, config=None)
mock_up.assert_called_once()
call_args = mock_up.call_args
# Should refresh all services in config
assert set(call_args[0][1]) == {"svc1", "svc2"}
def test_apply_full_dry_run_shows_refresh(
self, tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""--full --dry-run shows services that would be refreshed."""
cfg = _make_config(tmp_path)
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.lifecycle.get_orphaned_services", return_value={}),
patch("compose_farm.cli.lifecycle.get_services_needing_migration", return_value=[]),
patch("compose_farm.cli.lifecycle.get_services_not_in_state", return_value=[]),
):
apply(dry_run=True, no_orphans=False, full=True, config=None)
captured = capsys.readouterr()
assert "Services to refresh" in captured.out
assert "svc1" in captured.out
assert "svc2" in captured.out
assert "dry-run" in captured.out
def test_apply_full_excludes_already_handled_services(self, tmp_path: Path) -> None:
"""--full doesn't double-process services that are migrating or starting."""
cfg = _make_config(tmp_path, {"svc1": "host1", "svc2": "host2", "svc3": "host1"})
mock_results = [_make_result("svc1"), _make_result("svc3")]
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.lifecycle.get_orphaned_services", return_value={}),
patch(
"compose_farm.cli.lifecycle.get_services_needing_migration",
return_value=["svc1"],
),
patch(
"compose_farm.cli.lifecycle.get_services_not_in_state",
return_value=["svc2"],
),
patch("compose_farm.cli.lifecycle.get_service_host", return_value="host2"),
patch(
"compose_farm.cli.lifecycle.run_async",
return_value=mock_results,
),
patch("compose_farm.cli.lifecycle.up_services") as mock_up,
patch("compose_farm.cli.lifecycle.maybe_regenerate_traefik"),
patch("compose_farm.cli.lifecycle.report_results"),
):
apply(dry_run=False, no_orphans=False, full=True, config=None)
# up_services should be called 3 times: migrate, start, refresh
assert mock_up.call_count == 3
# Get the third call (refresh) and check it only has svc3
refresh_call = mock_up.call_args_list[2]
assert refresh_call[0][1] == ["svc3"]
class TestDownOrphaned:
"""Tests for down --orphaned flag."""
def test_down_orphaned_no_orphans(
self, tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""When no orphans exist, prints success message."""
cfg = _make_config(tmp_path)
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.lifecycle.get_orphaned_services", return_value={}),
):
down(
services=None,
all_services=False,
orphaned=True,
host=None,
config=None,
)
captured = capsys.readouterr()
assert "No orphaned services to stop" in captured.out
def test_down_orphaned_stops_services(self, tmp_path: Path) -> None:
"""--orphaned stops orphaned services."""
cfg = _make_config(tmp_path)
mock_results = [_make_result("old-svc@host1")]
with (
patch("compose_farm.cli.lifecycle.load_config_or_exit", return_value=cfg),
patch(
"compose_farm.cli.lifecycle.get_orphaned_services",
return_value={"old-svc": "host1"},
),
patch(
"compose_farm.cli.lifecycle.run_async",
return_value=mock_results,
),
patch("compose_farm.cli.lifecycle.stop_orphaned_services") as mock_stop,
patch("compose_farm.cli.lifecycle.report_results"),
):
down(
services=None,
all_services=False,
orphaned=True,
host=None,
config=None,
)
mock_stop.assert_called_once_with(cfg)
def test_down_orphaned_with_services_errors(self) -> None:
"""--orphaned cannot be combined with service arguments."""
with pytest.raises(typer.Exit) as exc_info:
down(
services=["svc1"],
all_services=False,
orphaned=True,
host=None,
config=None,
)
assert exc_info.value.exit_code == 1
def test_down_orphaned_with_all_errors(self) -> None:
"""--orphaned cannot be combined with --all."""
with pytest.raises(typer.Exit) as exc_info:
down(
services=None,
all_services=True,
orphaned=True,
host=None,
config=None,
)
assert exc_info.value.exit_code == 1
def test_down_orphaned_with_host_errors(self) -> None:
"""--orphaned cannot be combined with --host."""
with pytest.raises(typer.Exit) as exc_info:
down(
services=None,
all_services=False,
orphaned=True,
host="host1",
config=None,
)
assert exc_info.value.exit_code == 1

207
tests/test_cli_logs.py Normal file
View File

@@ -0,0 +1,207 @@
"""Tests for CLI logs command."""
from collections.abc import Coroutine
from pathlib import Path
from typing import Any
from unittest.mock import patch
import pytest
import typer
from compose_farm.cli.monitoring import logs
from compose_farm.config import Config, Host
from compose_farm.executor import CommandResult
def _make_config(tmp_path: Path) -> Config:
"""Create a minimal config for testing."""
compose_dir = tmp_path / "compose"
compose_dir.mkdir()
for svc in ("svc1", "svc2", "svc3"):
svc_dir = compose_dir / svc
svc_dir.mkdir()
(svc_dir / "docker-compose.yml").write_text("services: {}\n")
return Config(
compose_dir=compose_dir,
hosts={"local": Host(address="localhost"), "remote": Host(address="192.168.1.10")},
services={"svc1": "local", "svc2": "local", "svc3": "remote"},
)
def _make_result(service: str) -> CommandResult:
"""Create a successful command result."""
return CommandResult(service=service, exit_code=0, success=True, stdout="", stderr="")
def _mock_run_async_factory(
services: list[str],
) -> tuple[Any, list[CommandResult]]:
"""Create a mock run_async that returns results for given services."""
results = [_make_result(s) for s in services]
def mock_run_async(_coro: Coroutine[Any, Any, Any]) -> list[CommandResult]:
return results
return mock_run_async, results
class TestLogsContextualDefault:
"""Tests for logs --tail contextual default behavior."""
def test_logs_all_services_defaults_to_20(self, tmp_path: Path) -> None:
"""When --all is specified, default tail should be 20."""
cfg = _make_config(tmp_path)
mock_run_async, _ = _mock_run_async_factory(["svc1", "svc2", "svc3"])
with (
patch("compose_farm.cli.monitoring.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.common.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.monitoring.run_async", side_effect=mock_run_async),
patch("compose_farm.cli.monitoring.run_on_services") as mock_run,
):
mock_run.return_value = None
logs(services=None, all_services=True, host=None, follow=False, tail=None, config=None)
mock_run.assert_called_once()
call_args = mock_run.call_args
assert call_args[0][2] == "logs --tail 20"
def test_logs_single_service_defaults_to_100(self, tmp_path: Path) -> None:
"""When specific services are specified, default tail should be 100."""
cfg = _make_config(tmp_path)
mock_run_async, _ = _mock_run_async_factory(["svc1"])
with (
patch("compose_farm.cli.monitoring.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.common.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.monitoring.run_async", side_effect=mock_run_async),
patch("compose_farm.cli.monitoring.run_on_services") as mock_run,
):
logs(
services=["svc1"],
all_services=False,
host=None,
follow=False,
tail=None,
config=None,
)
mock_run.assert_called_once()
call_args = mock_run.call_args
assert call_args[0][2] == "logs --tail 100"
def test_logs_explicit_tail_overrides_default(self, tmp_path: Path) -> None:
"""When --tail is explicitly provided, it should override the default."""
cfg = _make_config(tmp_path)
mock_run_async, _ = _mock_run_async_factory(["svc1", "svc2", "svc3"])
with (
patch("compose_farm.cli.monitoring.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.common.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.monitoring.run_async", side_effect=mock_run_async),
patch("compose_farm.cli.monitoring.run_on_services") as mock_run,
):
logs(
services=None,
all_services=True,
host=None,
follow=False,
tail=50,
config=None,
)
mock_run.assert_called_once()
call_args = mock_run.call_args
assert call_args[0][2] == "logs --tail 50"
def test_logs_follow_appends_flag(self, tmp_path: Path) -> None:
"""When --follow is specified, -f should be appended to command."""
cfg = _make_config(tmp_path)
mock_run_async, _ = _mock_run_async_factory(["svc1"])
with (
patch("compose_farm.cli.monitoring.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.common.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.monitoring.run_async", side_effect=mock_run_async),
patch("compose_farm.cli.monitoring.run_on_services") as mock_run,
):
logs(
services=["svc1"],
all_services=False,
host=None,
follow=True,
tail=None,
config=None,
)
mock_run.assert_called_once()
call_args = mock_run.call_args
assert call_args[0][2] == "logs --tail 100 -f"
class TestLogsHostFilter:
"""Tests for logs --host filter behavior."""
def test_logs_host_filter_selects_services_on_host(self, tmp_path: Path) -> None:
"""When --host is specified, only services on that host are included."""
cfg = _make_config(tmp_path)
mock_run_async, _ = _mock_run_async_factory(["svc1", "svc2"])
with (
patch("compose_farm.cli.monitoring.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.monitoring.run_async", side_effect=mock_run_async),
patch("compose_farm.cli.monitoring.run_on_services") as mock_run,
):
logs(
services=None,
all_services=False,
host="local",
follow=False,
tail=None,
config=None,
)
mock_run.assert_called_once()
call_args = mock_run.call_args
# svc1 and svc2 are on "local", svc3 is on "remote"
assert set(call_args[0][1]) == {"svc1", "svc2"}
def test_logs_host_filter_defaults_to_20_lines(self, tmp_path: Path) -> None:
"""When --host is specified, default tail should be 20 (multiple services)."""
cfg = _make_config(tmp_path)
mock_run_async, _ = _mock_run_async_factory(["svc1", "svc2"])
with (
patch("compose_farm.cli.monitoring.load_config_or_exit", return_value=cfg),
patch("compose_farm.cli.monitoring.run_async", side_effect=mock_run_async),
patch("compose_farm.cli.monitoring.run_on_services") as mock_run,
):
logs(
services=None,
all_services=False,
host="local",
follow=False,
tail=None,
config=None,
)
mock_run.assert_called_once()
call_args = mock_run.call_args
assert call_args[0][2] == "logs --tail 20"
def test_logs_all_and_host_mutually_exclusive(self) -> None:
"""Using --all and --host together should error."""
# No config mock needed - error is raised before config is loaded
with pytest.raises(typer.Exit) as exc_info:
logs(
services=None,
all_services=True,
host="local",
follow=False,
tail=None,
config=None,
)
assert exc_info.value.exit_code == 1

View File

@@ -128,6 +128,8 @@ class TestLoadConfig:
def test_load_config_not_found(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.chdir(tmp_path)
monkeypatch.delenv("CF_CONFIG", raising=False)
monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "empty_config"))
with pytest.raises(FileNotFoundError, match="Config file not found"):
load_config()

230
tests/test_config_cmd.py Normal file
View File

@@ -0,0 +1,230 @@
"""Tests for config command module."""
from pathlib import Path
from typing import Any
import pytest
import yaml
from typer.testing import CliRunner
from compose_farm.cli import app
from compose_farm.cli.config import (
_generate_template,
_get_config_file,
_get_editor,
)
@pytest.fixture
def runner() -> CliRunner:
return CliRunner()
@pytest.fixture
def valid_config_data() -> dict[str, Any]:
return {
"compose_dir": "/opt/compose",
"hosts": {"server1": "192.168.1.10"},
"services": {"nginx": "server1"},
}
class TestGetEditor:
"""Tests for _get_editor function."""
def test_uses_editor_env(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("EDITOR", "code")
monkeypatch.delenv("VISUAL", raising=False)
assert _get_editor() == "code"
def test_uses_visual_env(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("EDITOR", raising=False)
monkeypatch.setenv("VISUAL", "subl")
assert _get_editor() == "subl"
def test_editor_takes_precedence(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("EDITOR", "vim")
monkeypatch.setenv("VISUAL", "code")
assert _get_editor() == "vim"
class TestGetConfigFile:
"""Tests for _get_config_file function."""
def test_explicit_path(self, tmp_path: Path) -> None:
config_file = tmp_path / "my-config.yaml"
config_file.touch()
result = _get_config_file(config_file)
assert result == config_file.resolve()
def test_cf_config_env(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
config_file = tmp_path / "env-config.yaml"
config_file.touch()
monkeypatch.setenv("CF_CONFIG", str(config_file))
result = _get_config_file(None)
assert result == config_file.resolve()
def test_returns_none_when_not_found(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.chdir(tmp_path)
monkeypatch.delenv("CF_CONFIG", raising=False)
# Set XDG_CONFIG_HOME to a nonexistent path - config_search_paths() will
# now return paths that don't exist
monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "nonexistent"))
result = _get_config_file(None)
assert result is None
class TestGenerateTemplate:
"""Tests for _generate_template function."""
def test_generates_valid_yaml(self) -> None:
template = _generate_template()
# Should be valid YAML
data = yaml.safe_load(template)
assert "compose_dir" in data
assert "hosts" in data
assert "services" in data
def test_has_documentation_comments(self) -> None:
template = _generate_template()
assert "# Compose Farm configuration" in template
assert "hosts:" in template
assert "services:" in template
class TestConfigInit:
"""Tests for cf config init command."""
def test_init_creates_file(
self, runner: CliRunner, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.delenv("CF_CONFIG", raising=False)
config_file = tmp_path / "new-config.yaml"
result = runner.invoke(app, ["config", "init", "-p", str(config_file)])
assert result.exit_code == 0
assert config_file.exists()
assert "Config file created" in result.stdout
def test_init_force_overwrites(
self, runner: CliRunner, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.delenv("CF_CONFIG", raising=False)
config_file = tmp_path / "existing.yaml"
config_file.write_text("old content")
result = runner.invoke(app, ["config", "init", "-p", str(config_file), "-f"])
assert result.exit_code == 0
content = config_file.read_text()
assert "old content" not in content
assert "compose_dir" in content
def test_init_prompts_on_existing(
self, runner: CliRunner, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.delenv("CF_CONFIG", raising=False)
config_file = tmp_path / "existing.yaml"
config_file.write_text("old content")
result = runner.invoke(app, ["config", "init", "-p", str(config_file)], input="n\n")
assert result.exit_code == 0
assert "Aborted" in result.stdout
assert config_file.read_text() == "old content"
class TestConfigPath:
"""Tests for cf config path command."""
def test_path_shows_config(
self,
runner: CliRunner,
tmp_path: Path,
valid_config_data: dict[str, Any],
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.chdir(tmp_path)
monkeypatch.delenv("CF_CONFIG", raising=False)
config_file = tmp_path / "compose-farm.yaml"
config_file.write_text(yaml.dump(valid_config_data))
result = runner.invoke(app, ["config", "path"])
assert result.exit_code == 0
assert str(config_file) in result.stdout
def test_path_with_explicit_path(self, runner: CliRunner, tmp_path: Path) -> None:
# When explicitly provided, path is returned even if file doesn't exist
nonexistent = tmp_path / "nonexistent.yaml"
result = runner.invoke(app, ["config", "path", "-p", str(nonexistent)])
assert result.exit_code == 0
assert str(nonexistent) in result.stdout
class TestConfigShow:
"""Tests for cf config show command."""
def test_show_displays_content(
self,
runner: CliRunner,
tmp_path: Path,
valid_config_data: dict[str, Any],
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.chdir(tmp_path)
monkeypatch.delenv("CF_CONFIG", raising=False)
config_file = tmp_path / "compose-farm.yaml"
config_file.write_text(yaml.dump(valid_config_data))
result = runner.invoke(app, ["config", "show"])
assert result.exit_code == 0
assert "Config file:" in result.stdout
def test_show_raw_output(
self,
runner: CliRunner,
tmp_path: Path,
valid_config_data: dict[str, Any],
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.chdir(tmp_path)
monkeypatch.delenv("CF_CONFIG", raising=False)
config_file = tmp_path / "compose-farm.yaml"
content = yaml.dump(valid_config_data)
config_file.write_text(content)
result = runner.invoke(app, ["config", "show", "-r"])
assert result.exit_code == 0
assert content in result.stdout
class TestConfigValidate:
"""Tests for cf config validate command."""
def test_validate_valid_config(
self,
runner: CliRunner,
tmp_path: Path,
valid_config_data: dict[str, Any],
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.chdir(tmp_path)
monkeypatch.delenv("CF_CONFIG", raising=False)
config_file = tmp_path / "compose-farm.yaml"
config_file.write_text(yaml.dump(valid_config_data))
result = runner.invoke(app, ["config", "validate"])
assert result.exit_code == 0
assert "Valid config" in result.stdout
assert "Hosts: 1" in result.stdout
assert "Services: 1" in result.stdout
def test_validate_invalid_config(self, runner: CliRunner, tmp_path: Path) -> None:
config_file = tmp_path / "invalid.yaml"
config_file.write_text("invalid: [yaml: content")
result = runner.invoke(app, ["config", "validate", "-p", str(config_file)])
assert result.exit_code == 1
# Error goes to stderr (captured in output when using CliRunner)
output = result.stdout + (result.stderr or "")
assert "Invalid config" in output or "" in output
def test_validate_missing_config(self, runner: CliRunner, tmp_path: Path) -> None:
nonexistent = tmp_path / "nonexistent.yaml"
result = runner.invoke(app, ["config", "validate", "-p", str(nonexistent)])
assert result.exit_code == 1
# Error goes to stderr
output = result.stdout + (result.stderr or "")
assert "Config file not found" in output or "not found" in output.lower()

241
tests/test_executor.py Normal file
View File

@@ -0,0 +1,241 @@
"""Tests for executor module."""
import sys
from pathlib import Path
import pytest
from compose_farm.config import Config, Host
from compose_farm.executor import (
CommandResult,
_run_local_command,
check_networks_exist,
check_paths_exist,
is_local,
run_command,
run_compose,
run_on_services,
)
# These tests run actual shell commands that only work on Linux
linux_only = pytest.mark.skipif(sys.platform != "linux", reason="Linux-only shell commands")
class TestIsLocal:
"""Tests for is_local function."""
@pytest.mark.parametrize(
"address",
["local", "localhost", "127.0.0.1", "::1", "LOCAL", "LOCALHOST"],
)
def test_local_addresses(self, address: str) -> None:
host = Host(address=address)
assert is_local(host) is True
@pytest.mark.parametrize(
"address",
["192.168.1.10", "nas01.local", "10.0.0.1", "example.com"],
)
def test_remote_addresses(self, address: str) -> None:
host = Host(address=address)
assert is_local(host) is False
class TestRunLocalCommand:
"""Tests for local command execution."""
async def test_run_local_command_success(self) -> None:
result = await _run_local_command("echo hello", "test-service")
assert result.success is True
assert result.exit_code == 0
assert result.service == "test-service"
async def test_run_local_command_failure(self) -> None:
result = await _run_local_command("exit 1", "test-service")
assert result.success is False
assert result.exit_code == 1
async def test_run_local_command_not_found(self) -> None:
result = await _run_local_command("nonexistent_command_xyz", "test-service")
assert result.success is False
assert result.exit_code != 0
async def test_run_local_command_captures_output(self) -> None:
result = await _run_local_command("echo hello", "test-service", stream=False)
assert "hello" in result.stdout
class TestRunCommand:
"""Tests for run_command dispatcher."""
async def test_run_command_local(self) -> None:
host = Host(address="localhost")
result = await run_command(host, "echo test", "test-service")
assert result.success is True
async def test_run_command_result_structure(self) -> None:
host = Host(address="local")
result = await run_command(host, "true", "my-service")
assert isinstance(result, CommandResult)
assert result.service == "my-service"
assert result.exit_code == 0
assert result.success is True
class TestRunCompose:
"""Tests for compose command execution."""
async def test_run_compose_builds_correct_command(self, tmp_path: Path) -> None:
# Create a minimal compose file
compose_dir = tmp_path / "compose"
service_dir = compose_dir / "test-service"
service_dir.mkdir(parents=True)
compose_file = service_dir / "docker-compose.yml"
compose_file.write_text("services: {}")
config = Config(
compose_dir=compose_dir,
hosts={"local": Host(address="localhost")},
services={"test-service": "local"},
)
# This will fail because docker compose isn't running,
# but we can verify the command structure works
result = await run_compose(config, "test-service", "config", stream=False)
# Command may fail due to no docker, but structure is correct
assert result.service == "test-service"
class TestRunOnServices:
"""Tests for parallel service execution."""
async def test_run_on_services_parallel(self) -> None:
config = Config(
compose_dir=Path("/tmp"),
hosts={"local": Host(address="localhost")},
services={"svc1": "local", "svc2": "local"},
)
# Use a simple command that will work without docker
# We'll test the parallelism structure
results = await run_on_services(config, ["svc1", "svc2"], "version", stream=False)
assert len(results) == 2
assert results[0].service == "svc1"
assert results[1].service == "svc2"
@linux_only
class TestCheckPathsExist:
"""Tests for check_paths_exist function (uses 'test -e' shell command)."""
async def test_check_existing_paths(self, tmp_path: Path) -> None:
"""Check paths that exist."""
config = Config(
compose_dir=tmp_path,
hosts={"local": Host(address="localhost")},
services={},
)
# Create test paths
(tmp_path / "dir1").mkdir()
(tmp_path / "file1").touch()
result = await check_paths_exist(
config, "local", [str(tmp_path / "dir1"), str(tmp_path / "file1")]
)
assert result[str(tmp_path / "dir1")] is True
assert result[str(tmp_path / "file1")] is True
async def test_check_missing_paths(self, tmp_path: Path) -> None:
"""Check paths that don't exist."""
config = Config(
compose_dir=tmp_path,
hosts={"local": Host(address="localhost")},
services={},
)
result = await check_paths_exist(
config, "local", [str(tmp_path / "missing1"), str(tmp_path / "missing2")]
)
assert result[str(tmp_path / "missing1")] is False
assert result[str(tmp_path / "missing2")] is False
async def test_check_mixed_paths(self, tmp_path: Path) -> None:
"""Check mix of existing and missing paths."""
config = Config(
compose_dir=tmp_path,
hosts={"local": Host(address="localhost")},
services={},
)
(tmp_path / "exists").mkdir()
result = await check_paths_exist(
config, "local", [str(tmp_path / "exists"), str(tmp_path / "missing")]
)
assert result[str(tmp_path / "exists")] is True
assert result[str(tmp_path / "missing")] is False
async def test_check_empty_paths(self, tmp_path: Path) -> None:
"""Empty path list returns empty dict."""
config = Config(
compose_dir=tmp_path,
hosts={"local": Host(address="localhost")},
services={},
)
result = await check_paths_exist(config, "local", [])
assert result == {}
@linux_only
class TestCheckNetworksExist:
"""Tests for check_networks_exist function (requires Docker)."""
async def test_check_bridge_network_exists(self, tmp_path: Path) -> None:
"""The 'bridge' network always exists on Docker hosts."""
config = Config(
compose_dir=tmp_path,
hosts={"local": Host(address="localhost")},
services={},
)
result = await check_networks_exist(config, "local", ["bridge"])
assert result["bridge"] is True
async def test_check_nonexistent_network(self, tmp_path: Path) -> None:
"""Check a network that doesn't exist."""
config = Config(
compose_dir=tmp_path,
hosts={"local": Host(address="localhost")},
services={},
)
result = await check_networks_exist(config, "local", ["nonexistent_network_xyz_123"])
assert result["nonexistent_network_xyz_123"] is False
async def test_check_mixed_networks(self, tmp_path: Path) -> None:
"""Check mix of existing and non-existing networks."""
config = Config(
compose_dir=tmp_path,
hosts={"local": Host(address="localhost")},
services={},
)
result = await check_networks_exist(
config, "local", ["bridge", "nonexistent_network_xyz_123"]
)
assert result["bridge"] is True
assert result["nonexistent_network_xyz_123"] is False
async def test_check_empty_networks(self, tmp_path: Path) -> None:
"""Empty network list returns empty dict."""
config = Config(
compose_dir=tmp_path,
hosts={"local": Host(address="localhost")},
services={},
)
result = await check_networks_exist(config, "local", [])
assert result == {}

View File

@@ -8,8 +8,15 @@ from pathlib import Path
import pytest
from compose_farm.config import Config, Host
from compose_farm.logs import _parse_images_output, snapshot_services
from compose_farm.ssh import CommandResult
from compose_farm.executor import CommandResult
from compose_farm.logs import (
_parse_images_output,
collect_service_entries,
isoformat,
load_existing_entries,
merge_entries,
write_toml,
)
def test_parse_images_output_handles_list_and_lines() -> None:
@@ -55,26 +62,29 @@ async def test_snapshot_preserves_first_seen(tmp_path: Path) -> None:
log_path = tmp_path / "dockerfarm-log.toml"
# First snapshot
first_time = datetime(2025, 1, 1, tzinfo=UTC)
await snapshot_services(
config,
["svc"],
log_path=log_path,
now=first_time,
run_compose_fn=fake_run_compose,
first_entries = await collect_service_entries(
config, "svc", now=first_time, run_compose_fn=fake_run_compose
)
first_iso = isoformat(first_time)
merged = merge_entries([], first_entries, now_iso=first_iso)
meta = {"generated_at": first_iso, "compose_dir": str(config.compose_dir)}
write_toml(log_path, meta=meta, entries=merged)
after_first = tomllib.loads(log_path.read_text())
first_seen = after_first["entries"][0]["first_seen"]
# Second snapshot
second_time = datetime(2025, 2, 1, tzinfo=UTC)
await snapshot_services(
config,
["svc"],
log_path=log_path,
now=second_time,
run_compose_fn=fake_run_compose,
second_entries = await collect_service_entries(
config, "svc", now=second_time, run_compose_fn=fake_run_compose
)
second_iso = isoformat(second_time)
existing = load_existing_entries(log_path)
merged = merge_entries(existing, second_entries, now_iso=second_iso)
meta = {"generated_at": second_iso, "compose_dir": str(config.compose_dir)}
write_toml(log_path, meta=meta, entries=merged)
after_second = tomllib.loads(log_path.read_text())
entry = after_second["entries"][0]

111
tests/test_operations.py Normal file
View File

@@ -0,0 +1,111 @@
"""Tests for operations module."""
from __future__ import annotations
import inspect
from pathlib import Path # noqa: TC003
from unittest.mock import patch
import pytest
from compose_farm.cli import lifecycle
from compose_farm.config import Config, Host
from compose_farm.executor import CommandResult
from compose_farm.operations import _migrate_service
@pytest.fixture
def basic_config(tmp_path: Path) -> Config:
"""Create a basic test config."""
compose_dir = tmp_path / "compose"
service_dir = compose_dir / "test-service"
service_dir.mkdir(parents=True)
(service_dir / "docker-compose.yml").write_text("services: {}")
return Config(
compose_dir=compose_dir,
hosts={
"host1": Host(address="localhost"),
"host2": Host(address="localhost"),
},
services={"test-service": "host2"},
)
class TestMigrationCommands:
"""Tests for migration command sequence."""
@pytest.fixture
def config(self, tmp_path: Path) -> Config:
"""Create a test config."""
compose_dir = tmp_path / "compose"
service_dir = compose_dir / "test-service"
service_dir.mkdir(parents=True)
(service_dir / "docker-compose.yml").write_text("services: {}")
return Config(
compose_dir=compose_dir,
hosts={
"host1": Host(address="localhost"),
"host2": Host(address="localhost"),
},
services={"test-service": "host2"},
)
async def test_migration_uses_pull_ignore_buildable(self, config: Config) -> None:
"""Migration should use 'pull --ignore-buildable' to skip buildable images."""
commands_called: list[str] = []
async def mock_run_compose_step(
cfg: Config, # noqa: ARG001
service: str,
command: str,
*,
raw: bool, # noqa: ARG001
host: str | None = None, # noqa: ARG001
) -> CommandResult:
commands_called.append(command)
return CommandResult(
service=service,
exit_code=0,
success=True,
)
with patch(
"compose_farm.operations._run_compose_step",
side_effect=mock_run_compose_step,
):
await _migrate_service(
config,
"test-service",
current_host="host1",
target_host="host2",
prefix="[test]",
raw=False,
)
# Migration should call pull with --ignore-buildable, then build, then down
assert "pull --ignore-buildable" in commands_called
assert "build" in commands_called
assert "down" in commands_called
# pull should come before build
pull_idx = commands_called.index("pull --ignore-buildable")
build_idx = commands_called.index("build")
assert pull_idx < build_idx
class TestUpdateCommandSequence:
"""Tests for update command sequence."""
def test_update_command_sequence_includes_build(self) -> None:
"""Update command should use pull --ignore-buildable and build."""
# This is a static check of the command sequence in lifecycle.py
# The actual command sequence is defined in the update function
source = inspect.getsource(lifecycle.update)
# Verify the command sequence includes pull --ignore-buildable
assert "pull --ignore-buildable" in source
# Verify build is included
assert '"build"' in source or "'build'" in source
# Verify the sequence is pull, build, down, up
assert "down" in source
assert "up -d" in source

View File

@@ -1,16 +1,15 @@
"""Tests for sync command and related functions."""
from pathlib import Path
from typing import Any
from unittest.mock import AsyncMock, patch
import pytest
from compose_farm import cli as cli_module
from compose_farm import ssh as ssh_module
from compose_farm import executor as executor_module
from compose_farm import state as state_module
from compose_farm.cli import management as cli_management_module
from compose_farm.config import Config, Host
from compose_farm.ssh import CommandResult, check_service_running
from compose_farm.executor import CommandResult, check_service_running
@pytest.fixture
@@ -58,7 +57,7 @@ class TestCheckServiceRunning:
@pytest.mark.asyncio
async def test_service_running(self, mock_config: Config) -> None:
"""Returns True when service has running containers."""
with patch.object(ssh_module, "run_command", new_callable=AsyncMock) as mock_run:
with patch.object(executor_module, "run_command", new_callable=AsyncMock) as mock_run:
mock_run.return_value = CommandResult(
service="plex",
exit_code=0,
@@ -71,7 +70,7 @@ class TestCheckServiceRunning:
@pytest.mark.asyncio
async def test_service_not_running(self, mock_config: Config) -> None:
"""Returns False when service has no running containers."""
with patch.object(ssh_module, "run_command", new_callable=AsyncMock) as mock_run:
with patch.object(executor_module, "run_command", new_callable=AsyncMock) as mock_run:
mock_run.return_value = CommandResult(
service="plex",
exit_code=0,
@@ -84,7 +83,7 @@ class TestCheckServiceRunning:
@pytest.mark.asyncio
async def test_command_failed(self, mock_config: Config) -> None:
"""Returns False when command fails."""
with patch.object(ssh_module, "run_command", new_callable=AsyncMock) as mock_run:
with patch.object(executor_module, "run_command", new_callable=AsyncMock) as mock_run:
mock_run.return_value = CommandResult(
service="plex",
exit_code=1,
@@ -94,48 +93,12 @@ class TestCheckServiceRunning:
assert result is False
class TestDiscoverRunningServices:
"""Tests for _discover_running_services function."""
@pytest.mark.asyncio
async def test_discovers_on_assigned_host(self, mock_config: Config) -> None:
"""Discovers service running on its assigned host."""
with patch.object(
cli_module, "check_service_running", new_callable=AsyncMock
) as mock_check:
# plex running on nas01, jellyfin not running, sonarr on nas02
async def check_side_effect(_cfg: Any, service: str, host: str) -> bool:
return (service == "plex" and host == "nas01") or (
service == "sonarr" and host == "nas02"
)
mock_check.side_effect = check_side_effect
result = await cli_module._discover_running_services(mock_config)
assert result == {"plex": "nas01", "sonarr": "nas02"}
@pytest.mark.asyncio
async def test_discovers_on_different_host(self, mock_config: Config) -> None:
"""Discovers service running on non-assigned host (after migration)."""
with patch.object(
cli_module, "check_service_running", new_callable=AsyncMock
) as mock_check:
# plex migrated to nas02
async def check_side_effect(_cfg: Any, service: str, host: str) -> bool:
return service == "plex" and host == "nas02"
mock_check.side_effect = check_side_effect
result = await cli_module._discover_running_services(mock_config)
assert result == {"plex": "nas02"}
class TestReportSyncChanges:
"""Tests for _report_sync_changes function."""
def test_reports_added(self, capsys: pytest.CaptureFixture[str]) -> None:
"""Reports newly discovered services."""
cli_module._report_sync_changes(
cli_management_module._report_sync_changes(
added=["plex", "jellyfin"],
removed=[],
changed=[],
@@ -149,7 +112,7 @@ class TestReportSyncChanges:
def test_reports_removed(self, capsys: pytest.CaptureFixture[str]) -> None:
"""Reports services that are no longer running."""
cli_module._report_sync_changes(
cli_management_module._report_sync_changes(
added=[],
removed=["sonarr"],
changed=[],
@@ -162,7 +125,7 @@ class TestReportSyncChanges:
def test_reports_changed(self, capsys: pytest.CaptureFixture[str]) -> None:
"""Reports services that moved to a different host."""
cli_module._report_sync_changes(
cli_management_module._report_sync_changes(
added=[],
removed=[],
changed=[("plex", "nas01", "nas02")],
@@ -171,4 +134,4 @@ class TestReportSyncChanges:
)
captured = capsys.readouterr()
assert "Services on different hosts (1)" in captured.out
assert "~ plex: nas01 -> nas02" in captured.out
assert "~ plex: nas01 nas02" in captured.out

View File

@@ -1,118 +0,0 @@
"""Tests for ssh module."""
from pathlib import Path
import pytest
from compose_farm.config import Config, Host
from compose_farm.ssh import (
CommandResult,
_is_local,
_run_local_command,
run_command,
run_compose,
run_on_services,
)
class TestIsLocal:
"""Tests for _is_local function."""
@pytest.mark.parametrize(
"address",
["local", "localhost", "127.0.0.1", "::1", "LOCAL", "LOCALHOST"],
)
def test_local_addresses(self, address: str) -> None:
host = Host(address=address)
assert _is_local(host) is True
@pytest.mark.parametrize(
"address",
["192.168.1.10", "nas01.local", "10.0.0.1", "example.com"],
)
def test_remote_addresses(self, address: str) -> None:
host = Host(address=address)
assert _is_local(host) is False
class TestRunLocalCommand:
"""Tests for local command execution."""
async def test_run_local_command_success(self) -> None:
result = await _run_local_command("echo hello", "test-service")
assert result.success is True
assert result.exit_code == 0
assert result.service == "test-service"
async def test_run_local_command_failure(self) -> None:
result = await _run_local_command("exit 1", "test-service")
assert result.success is False
assert result.exit_code == 1
async def test_run_local_command_not_found(self) -> None:
result = await _run_local_command("nonexistent_command_xyz", "test-service")
assert result.success is False
assert result.exit_code != 0
async def test_run_local_command_captures_output(self) -> None:
result = await _run_local_command("echo hello", "test-service", stream=False)
assert "hello" in result.stdout
class TestRunCommand:
"""Tests for run_command dispatcher."""
async def test_run_command_local(self) -> None:
host = Host(address="localhost")
result = await run_command(host, "echo test", "test-service")
assert result.success is True
async def test_run_command_result_structure(self) -> None:
host = Host(address="local")
result = await run_command(host, "true", "my-service")
assert isinstance(result, CommandResult)
assert result.service == "my-service"
assert result.exit_code == 0
assert result.success is True
class TestRunCompose:
"""Tests for compose command execution."""
async def test_run_compose_builds_correct_command(self, tmp_path: Path) -> None:
# Create a minimal compose file
compose_dir = tmp_path / "compose"
service_dir = compose_dir / "test-service"
service_dir.mkdir(parents=True)
compose_file = service_dir / "docker-compose.yml"
compose_file.write_text("services: {}")
config = Config(
compose_dir=compose_dir,
hosts={"local": Host(address="localhost")},
services={"test-service": "local"},
)
# This will fail because docker compose isn't running,
# but we can verify the command structure works
result = await run_compose(config, "test-service", "config", stream=False)
# Command may fail due to no docker, but structure is correct
assert result.service == "test-service"
class TestRunOnServices:
"""Tests for parallel service execution."""
async def test_run_on_services_parallel(self) -> None:
config = Config(
compose_dir=Path("/tmp"),
hosts={"local": Host(address="localhost")},
services={"svc1": "local", "svc2": "local"},
)
# Use a simple command that will work without docker
# We'll test the parallelism structure
results = await run_on_services(config, ["svc1", "svc2"], "version", stream=False)
assert len(results) == 2
assert results[0].service == "svc1"
assert results[1].service == "svc2"

View File

@@ -4,9 +4,11 @@ from pathlib import Path
import pytest
from compose_farm import state as state_module
from compose_farm.config import Config, Host
from compose_farm.state import (
get_orphaned_services,
get_service_host,
get_services_not_in_state,
load_state,
remove_service,
save_state,
@@ -15,52 +17,51 @@ from compose_farm.state import (
@pytest.fixture
def state_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""Create a temporary state directory and patch _get_state_path."""
state_path = tmp_path / ".config" / "compose-farm"
state_path.mkdir(parents=True)
def mock_get_state_path() -> Path:
return state_path / "state.yaml"
monkeypatch.setattr(state_module, "_get_state_path", mock_get_state_path)
return state_path
def config(tmp_path: Path) -> Config:
"""Create a config with a temporary config path for state storage."""
config_path = tmp_path / "compose-farm.yaml"
config_path.write_text("") # Create empty file
return Config(
compose_dir=tmp_path / "compose",
hosts={"nas01": Host(address="192.168.1.10")},
services={"plex": "nas01"},
config_path=config_path,
)
class TestLoadState:
"""Tests for load_state function."""
def test_load_state_empty(self, state_dir: Path) -> None:
def test_load_state_empty(self, config: Config) -> None:
"""Returns empty dict when state file doesn't exist."""
_ = state_dir # Fixture activates the mock
result = load_state()
result = load_state(config)
assert result == {}
def test_load_state_with_data(self, state_dir: Path) -> None:
def test_load_state_with_data(self, config: Config) -> None:
"""Loads existing state from file."""
state_file = state_dir / "state.yaml"
state_file = config.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n jellyfin: nas02\n")
result = load_state()
result = load_state(config)
assert result == {"plex": "nas01", "jellyfin": "nas02"}
def test_load_state_empty_file(self, state_dir: Path) -> None:
def test_load_state_empty_file(self, config: Config) -> None:
"""Returns empty dict for empty file."""
state_file = state_dir / "state.yaml"
state_file = config.get_state_path()
state_file.write_text("")
result = load_state()
result = load_state(config)
assert result == {}
class TestSaveState:
"""Tests for save_state function."""
def test_save_state(self, state_dir: Path) -> None:
def test_save_state(self, config: Config) -> None:
"""Saves state to file."""
save_state({"plex": "nas01", "jellyfin": "nas02"})
save_state(config, {"plex": "nas01", "jellyfin": "nas02"})
state_file = state_dir / "state.yaml"
state_file = config.get_state_path()
assert state_file.exists()
content = state_file.read_text()
assert "plex: nas01" in content
@@ -70,65 +71,171 @@ class TestSaveState:
class TestGetServiceHost:
"""Tests for get_service_host function."""
def test_get_existing_service(self, state_dir: Path) -> None:
def test_get_existing_service(self, config: Config) -> None:
"""Returns host for existing service."""
state_file = state_dir / "state.yaml"
state_file = config.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n")
host = get_service_host("plex")
host = get_service_host(config, "plex")
assert host == "nas01"
def test_get_nonexistent_service(self, state_dir: Path) -> None:
def test_get_nonexistent_service(self, config: Config) -> None:
"""Returns None for service not in state."""
state_file = state_dir / "state.yaml"
state_file = config.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n")
host = get_service_host("unknown")
host = get_service_host(config, "unknown")
assert host is None
class TestSetServiceHost:
"""Tests for set_service_host function."""
def test_set_new_service(self, state_dir: Path) -> None:
def test_set_new_service(self, config: Config) -> None:
"""Adds new service to state."""
_ = state_dir # Fixture activates the mock
set_service_host("plex", "nas01")
set_service_host(config, "plex", "nas01")
result = load_state()
result = load_state(config)
assert result["plex"] == "nas01"
def test_update_existing_service(self, state_dir: Path) -> None:
def test_update_existing_service(self, config: Config) -> None:
"""Updates host for existing service."""
state_file = state_dir / "state.yaml"
state_file = config.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n")
set_service_host("plex", "nas02")
set_service_host(config, "plex", "nas02")
result = load_state()
result = load_state(config)
assert result["plex"] == "nas02"
class TestRemoveService:
"""Tests for remove_service function."""
def test_remove_existing_service(self, state_dir: Path) -> None:
def test_remove_existing_service(self, config: Config) -> None:
"""Removes service from state."""
state_file = state_dir / "state.yaml"
state_file = config.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n jellyfin: nas02\n")
remove_service("plex")
remove_service(config, "plex")
result = load_state()
result = load_state(config)
assert "plex" not in result
assert result["jellyfin"] == "nas02"
def test_remove_nonexistent_service(self, state_dir: Path) -> None:
def test_remove_nonexistent_service(self, config: Config) -> None:
"""Removing nonexistent service doesn't error."""
state_file = state_dir / "state.yaml"
state_file = config.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n")
remove_service("unknown") # Should not raise
remove_service(config, "unknown") # Should not raise
result = load_state()
result = load_state(config)
assert result["plex"] == "nas01"
class TestGetOrphanedServices:
"""Tests for get_orphaned_services function."""
def test_no_orphans(self, config: Config) -> None:
"""Returns empty dict when all services in state are in config."""
state_file = config.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n")
result = get_orphaned_services(config)
assert result == {}
def test_finds_orphaned_service(self, config: Config) -> None:
"""Returns services in state but not in config."""
state_file = config.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n jellyfin: nas02\n")
result = get_orphaned_services(config)
# plex is in config, jellyfin is not
assert result == {"jellyfin": "nas02"}
def test_finds_orphaned_multi_host_service(self, config: Config) -> None:
"""Returns multi-host orphaned services with host list."""
state_file = config.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n dozzle:\n - nas01\n - nas02\n")
result = get_orphaned_services(config)
assert result == {"dozzle": ["nas01", "nas02"]}
def test_empty_state(self, config: Config) -> None:
"""Returns empty dict when state is empty."""
result = get_orphaned_services(config)
assert result == {}
def test_all_orphaned(self, tmp_path: Path) -> None:
"""Returns all services when none are in config."""
config_path = tmp_path / "compose-farm.yaml"
config_path.write_text("")
cfg = Config(
compose_dir=tmp_path / "compose",
hosts={"nas01": Host(address="192.168.1.10")},
services={}, # No services in config
config_path=config_path,
)
state_file = cfg.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n jellyfin: nas02\n")
result = get_orphaned_services(cfg)
assert result == {"plex": "nas01", "jellyfin": "nas02"}
class TestGetServicesNotInState:
"""Tests for get_services_not_in_state function."""
def test_all_in_state(self, config: Config) -> None:
"""Returns empty list when all services are in state."""
state_file = config.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n")
result = get_services_not_in_state(config)
assert result == []
def test_finds_missing_service(self, tmp_path: Path) -> None:
"""Returns services in config but not in state."""
config_path = tmp_path / "compose-farm.yaml"
config_path.write_text("")
cfg = Config(
compose_dir=tmp_path / "compose",
hosts={"nas01": Host(address="192.168.1.10")},
services={"plex": "nas01", "jellyfin": "nas01"},
config_path=config_path,
)
state_file = cfg.get_state_path()
state_file.write_text("deployed:\n plex: nas01\n")
result = get_services_not_in_state(cfg)
assert result == ["jellyfin"]
def test_empty_state(self, tmp_path: Path) -> None:
"""Returns all services when state is empty."""
config_path = tmp_path / "compose-farm.yaml"
config_path.write_text("")
cfg = Config(
compose_dir=tmp_path / "compose",
hosts={"nas01": Host(address="192.168.1.10")},
services={"plex": "nas01", "jellyfin": "nas01"},
config_path=config_path,
)
result = get_services_not_in_state(cfg)
assert set(result) == {"plex", "jellyfin"}
def test_empty_config(self, config: Config) -> None:
"""Returns empty list when config has no services."""
# config fixture has plex: nas01, but we need empty config
config_path = config.config_path
config_path.write_text("")
cfg = Config(
compose_dir=config.compose_dir,
hosts={"nas01": Host(address="192.168.1.10")},
services={},
config_path=config_path,
)
result = get_services_not_in_state(cfg)
assert result == []

View File

@@ -4,6 +4,7 @@ from pathlib import Path
import yaml
from compose_farm.compose import parse_external_networks
from compose_farm.config import Config, Host
from compose_farm.traefik import generate_traefik_config
@@ -76,7 +77,7 @@ def test_generate_traefik_config_without_published_port_warns(tmp_path: Path) ->
dynamic, warnings = generate_traefik_config(cfg, ["app"])
assert dynamic["http"]["routers"]["app"]["rule"] == "Host(`app.lab.mydomain.org`)"
assert any("No host-published port found" in warning for warning in warnings)
assert any("No published port found" in warning for warning in warnings)
def test_generate_interpolates_env_and_infers_router_service(tmp_path: Path) -> None:
@@ -193,3 +194,145 @@ def test_generate_skips_services_with_enable_false(tmp_path: Path) -> None:
assert dynamic == {}
assert warnings == []
def test_generate_follows_network_mode_service_for_ports(tmp_path: Path) -> None:
"""Services using network_mode: service:X should use ports from service X."""
cfg = Config(
compose_dir=tmp_path,
hosts={"nas01": Host(address="192.168.1.10")},
services={"vpn-stack": "nas01"},
)
compose_path = tmp_path / "vpn-stack" / "docker-compose.yml"
_write_compose(
compose_path,
{
"services": {
"vpn": {
"image": "gluetun",
"ports": ["5080:5080", "9696:9696"],
},
"qbittorrent": {
"image": "qbittorrent",
"network_mode": "service:vpn",
"labels": [
"traefik.enable=true",
"traefik.http.routers.torrent.rule=Host(`torrent.example.com`)",
"traefik.http.services.torrent.loadbalancer.server.port=5080",
],
},
"prowlarr": {
"image": "prowlarr",
"network_mode": "service:vpn",
"labels": [
"traefik.enable=true",
"traefik.http.routers.prowlarr.rule=Host(`prowlarr.example.com`)",
"traefik.http.services.prowlarr.loadbalancer.server.port=9696",
],
},
}
},
)
dynamic, warnings = generate_traefik_config(cfg, ["vpn-stack"])
assert warnings == []
# Both services should get their ports from the vpn service
torrent_servers = dynamic["http"]["services"]["torrent"]["loadbalancer"]["servers"]
assert torrent_servers == [{"url": "http://192.168.1.10:5080"}]
prowlarr_servers = dynamic["http"]["services"]["prowlarr"]["loadbalancer"]["servers"]
assert prowlarr_servers == [{"url": "http://192.168.1.10:9696"}]
def test_parse_external_networks_single(tmp_path: Path) -> None:
"""Extract a single external network from compose file."""
cfg = Config(
compose_dir=tmp_path,
hosts={"host1": Host(address="192.168.1.10")},
services={"app": "host1"},
)
compose_path = tmp_path / "app" / "compose.yaml"
_write_compose(
compose_path,
{
"services": {"app": {"image": "nginx"}},
"networks": {"mynetwork": {"external": True}},
},
)
networks = parse_external_networks(cfg, "app")
assert networks == ["mynetwork"]
def test_parse_external_networks_multiple(tmp_path: Path) -> None:
"""Extract multiple external networks from compose file."""
cfg = Config(
compose_dir=tmp_path,
hosts={"host1": Host(address="192.168.1.10")},
services={"app": "host1"},
)
compose_path = tmp_path / "app" / "compose.yaml"
_write_compose(
compose_path,
{
"services": {"app": {"image": "nginx"}},
"networks": {
"frontend": {"external": True},
"backend": {"external": True},
"internal": {"driver": "bridge"}, # not external
},
},
)
networks = parse_external_networks(cfg, "app")
assert set(networks) == {"frontend", "backend"}
def test_parse_external_networks_none(tmp_path: Path) -> None:
"""No external networks returns empty list."""
cfg = Config(
compose_dir=tmp_path,
hosts={"host1": Host(address="192.168.1.10")},
services={"app": "host1"},
)
compose_path = tmp_path / "app" / "compose.yaml"
_write_compose(
compose_path,
{
"services": {"app": {"image": "nginx"}},
"networks": {"internal": {"driver": "bridge"}},
},
)
networks = parse_external_networks(cfg, "app")
assert networks == []
def test_parse_external_networks_no_networks_section(tmp_path: Path) -> None:
"""No networks section returns empty list."""
cfg = Config(
compose_dir=tmp_path,
hosts={"host1": Host(address="192.168.1.10")},
services={"app": "host1"},
)
compose_path = tmp_path / "app" / "compose.yaml"
_write_compose(
compose_path,
{"services": {"app": {"image": "nginx"}}},
)
networks = parse_external_networks(cfg, "app")
assert networks == []
def test_parse_external_networks_missing_compose(tmp_path: Path) -> None:
"""Missing compose file returns empty list."""
cfg = Config(
compose_dir=tmp_path,
hosts={"host1": Host(address="192.168.1.10")},
services={"app": "host1"},
)
# Don't create compose file
networks = parse_external_networks(cfg, "app")
assert networks == []

2
uv.lock generated
View File

@@ -131,6 +131,7 @@ dependencies = [
{ name = "asyncssh" },
{ name = "pydantic" },
{ name = "pyyaml" },
{ name = "rich" },
{ name = "typer" },
]
@@ -151,6 +152,7 @@ requires-dist = [
{ name = "asyncssh", specifier = ">=2.14.0" },
{ name = "pydantic", specifier = ">=2.0.0" },
{ name = "pyyaml", specifier = ">=6.0" },
{ name = "rich", specifier = ">=13.0.0" },
{ name = "typer", specifier = ">=0.9.0" },
]