ci: inject cache tarballs instead of directory trees

* perf(tests): cache go and pnpm stores across integration image builds

Add BuildKit cache mounts for GOCACHE/GOMODCACHE and the pnpm store to the
integration Dockerfiles, and build the image via the docker CLI (docker-py,
used by testcontainers' DockerImage, does not support BuildKit). Embed the
go build command directly so Makefile changes do not invalidate the build
layer, and pin HOME/GOCACHE/GOMODCACHE/PNPM_HOME explicitly so cache-mount
targets match tool defaults by contract. The with-web node stage fetches
dependencies from the lockfile before the source copy, so frontend edits
only re-run the offline install and build.

* feat(tests): add --clean flag to prune buildkit cache mounts

The go and pnpm caches introduced for the integration image build survive
--teardown since they belong to the docker builder, not to any container.
--clean runs docker builder prune with a type=exec.cachemount filter at
session start, forcing the next image build to start cold. Documented in
the integration testing guide.

* feat(tests): add --rebuild flag to refresh the signoz container under --reuse

--reuse keeps the running signoz container, so backend source changes are
never picked up without tearing down the whole stack. --rebuild deletes the
cached signoz container and recreates it from the current sources (an
incremental image build), while databases, mocks and migrations stay reused.
Requires --reuse; combining with --teardown or --clean is a usage error.

* chore(tests): prune comments to non-obvious constraints

* docs(tests): make py-test-setup rebuild signoz and audit the integration guide

py-test-setup now passes --rebuild so re-running it after backend changes
transparently swaps in a signoz container built from the current sources.
The integration guide documents the iteration loop and fixes stale content:
option defaults (clickhouse 25.12.5, schema migrator v0.144.6), the
nonexistent --zookeeper-version option, Zookeeper vs ClickHouse Keeper, the
e2e doc path, and the lint toolchain (ruff).

* docs(tests): wire --rebuild into the e2e setup flow

The e2e bootstrap shares the signoz fixture, so --rebuild already applies;
with --with-web it also picks up frontend changes since the image bakes the
built frontend in. The setup command now passes --rebuild, and the guide
documents the iteration loop, the --rebuild/--clean flags, ClickHouse Keeper
instead of Zookeeper, and the corrected integration doc path.

* docs(tests): qualify --rebuild workflow for suites with custom signoz variants

make py-test-setup only rebuilds the default signoz instance; suites that
create their own via create_signoz(cache_key=...) keep a separately cached
container. Passing --rebuild on the suite run itself rebuilds every variant
that run instantiates.

* docs(tests): describe --clean behaviour instead of its exact command

Keeps the docs from drifting if the prune invocation behind --clean changes.

.github/workflows/e2eci.yaml

@@ -75,6 +75,30 @@ jobs:
           docker rm pw
           echo "PLAYWRIGHT_BROWSERS_PATH=$RUNNER_TEMP/ms-playwright" >> "$GITHUB_ENV"
           cd tests/e2e && pnpm playwright install-deps ${{ matrix.project }}
+      # Restore-only: the cacheci workflow owns cache saves. Seeds the
+      # BuildKit cache mounts so the in-test image build is incremental.
+      - name: restore
+        id: restore
+        uses: actions/cache/restore@v4
+        with:
+          path: ${{ runner.temp }}/cacheci
+          key: tests-primary
+          restore-keys: |
+            tests-secondary
+      - name: inject
+        if: steps.restore.outputs.cache-matched-key != ''
+        run: |
+          cat > "$RUNNER_TEMP/inject.Dockerfile" <<'EOF'
+          FROM busybox:1.37
+          RUN --mount=type=cache,target=/root/.cache/go-build \
+              --mount=type=cache,target=/go/pkg/mod \
+              --mount=type=cache,target=/pnpm/store \
+              --mount=type=bind,target=/restored \
+              tar -xf /restored/go-build.tar -C /root/.cache/go-build && \
+              tar -xf /restored/go-mod.tar -C /go/pkg/mod && \
+              tar -xf /restored/pnpm-store.tar -C /pnpm/store
+          EOF
+          docker build -f "$RUNNER_TEMP/inject.Dockerfile" "$RUNNER_TEMP/cacheci"
       - name: bring-up-stack
         run: |
           cd tests && \

.github/workflows/integrationci.yaml

@@ -110,6 +110,30 @@ jobs:
           sudo mv chromedriver-linux64/chromedriver /usr/local/bin/chromedriver
           chromedriver -version
           google-chrome-stable --version
+      # Restore-only: the cacheci workflow owns cache saves. Seeds the
+      # BuildKit cache mounts so the in-test image build is incremental.
+      - name: restore
+        id: restore
+        uses: actions/cache/restore@v4
+        with:
+          path: ${{ runner.temp }}/cacheci
+          key: tests-primary
+          restore-keys: |
+            tests-secondary
+      - name: inject
+        if: steps.restore.outputs.cache-matched-key != ''
+        run: |
+          cat > "$RUNNER_TEMP/inject.Dockerfile" <<'EOF'
+          FROM busybox:1.37
+          RUN --mount=type=cache,target=/root/.cache/go-build \
+              --mount=type=cache,target=/go/pkg/mod \
+              --mount=type=cache,target=/pnpm/store \
+              --mount=type=bind,target=/restored \
+              tar -xf /restored/go-build.tar -C /root/.cache/go-build && \
+              tar -xf /restored/go-mod.tar -C /go/pkg/mod && \
+              tar -xf /restored/pnpm-store.tar -C /pnpm/store
+          EOF
+          docker build -f "$RUNNER_TEMP/inject.Dockerfile" "$RUNNER_TEMP/cacheci"
       - name: run
         run: |
           cd tests && \

Makefile

@@ -209,8 +209,8 @@ py-lint: ## Run ruff check across the shared tests project
 	@cd tests && uv run ruff check --fix .
 
 .PHONY: py-test-setup
-py-test-setup: ## Bring up the shared SigNoz backend used by integration and e2e tests
-	@cd tests && uv run pytest --basetemp=./tmp/ -vv --reuse --capture=no integration/bootstrap/setup.py::test_setup
+py-test-setup: ## Bring up the shared SigNoz backend used by integration and e2e tests, rebuilding signoz from the current sources
+	@cd tests && uv run pytest --basetemp=./tmp/ -vv --reuse --rebuild --capture=no integration/bootstrap/setup.py::test_setup
 
 .PHONY: py-test-teardown
 py-test-teardown: ## Tear down the shared SigNoz backend

cmd/enterprise/Dockerfile.integration

@@ -4,9 +4,13 @@ ARG OS="linux"
 ARG TARGETARCH
 ARG ZEUSURL
 
+# HOME comes from the build user, not the image config; declare it so the
+# /root paths below trace to it.
+ENV HOME=/root
+
 # This path is important for stacktraces
 WORKDIR $GOPATH/src/github.com/signoz/signoz
-WORKDIR /root
+WORKDIR $HOME
 
 RUN set -eux; \
 	apt-get update; \
@@ -14,23 +18,36 @@ RUN set -eux; \
 		g++ \
 		gcc \
 		libc6-dev \
-		make \
 		pkg-config \
 	; \
 	rm -rf /var/lib/apt/lists/*
 
+# Keep the literal cache-mount targets below in sync with these. The caches
+# are shared with Dockerfile.with-web.integration (same target paths).
+ENV GOCACHE=$HOME/.cache/go-build
+ENV GOMODCACHE=$GOPATH/pkg/mod
+
 COPY go.mod go.sum ./
 
-RUN go mod download
+RUN --mount=type=cache,target=/go/pkg/mod \
+	go mod download
 
 COPY ./cmd/ ./cmd/
 COPY ./ee/ ./ee/
 COPY ./pkg/ ./pkg/
 COPY ./templates /root/templates
 
-COPY Makefile Makefile
-RUN TARGET_DIR=/root ARCHS=${TARGETARCH} ZEUS_URL=${ZEUSURL} LICENSE_URL=${ZEUSURL}/api/v1 make go-build-enterprise-race
-RUN mv /root/linux-${TARGETARCH}/signoz /root/signoz
+# Invoked directly instead of via make so Makefile changes don't invalidate
+# this layer; the Makefile's git-derived ldflags resolve to empty in here
+# anyway (.git is dockerignored).
+RUN --mount=type=cache,target=/go/pkg/mod \
+	--mount=type=cache,target=/root/.cache/go-build \
+	GOARCH=${TARGETARCH} GOOS=${OS} go build -C ./cmd/enterprise -race -tags timetzdata -o /root/signoz \
+	-ldflags "-s -w \
+		-X github.com/SigNoz/signoz/pkg/version.version=integration \
+		-X github.com/SigNoz/signoz/pkg/version.variant=enterprise \
+		-X github.com/SigNoz/signoz/ee/zeus.url=${ZEUSURL} \
+		-X github.com/SigNoz/signoz/ee/zeus.deprecatedURL=${ZEUSURL}/api/v1"
 
 RUN chmod 755 /root /root/signoz
 

cmd/enterprise/Dockerfile.with-web.integration

@@ -1,10 +1,23 @@
 FROM node:22-bookworm AS build
 
 WORKDIR /opt/
-COPY ./frontend/ ./
+
+# HOME comes from the build user, not the image config.
+ENV HOME=/root
+# pnpm's store lives at $PNPM_HOME/store — a dedicated directory pnpm
+# manages. Keep the literal cache-mount targets below in sync.
+ENV PNPM_HOME=/pnpm
 ENV NODE_OPTIONS=--max-old-space-size=8192
+
 RUN CI=1 npm i -g pnpm@10
-RUN CI=1 pnpm install
+
+# pnpm fetch resolves from the lockfile alone and runs no lifecycle scripts;
+# the repo's postinstall needs source files that are not copied yet.
+COPY ./frontend/package.json ./frontend/pnpm-lock.yaml ./frontend/pnpm-workspace.yaml ./
+RUN --mount=type=cache,target=/pnpm/store CI=1 pnpm fetch
+
+COPY ./frontend/ ./
+RUN --mount=type=cache,target=/pnpm/store CI=1 pnpm install --offline
 RUN CI=1 pnpm build
 
 FROM golang:1.25-bookworm
@@ -13,9 +26,13 @@ ARG OS="linux"
 ARG TARGETARCH
 ARG ZEUSURL
 
+# HOME comes from the build user, not the image config; declare it so the
+# /root paths below trace to it.
+ENV HOME=/root
+
 # This path is important for stacktraces
 WORKDIR $GOPATH/src/github.com/signoz/signoz
-WORKDIR /root
+WORKDIR $HOME
 
 RUN set -eux; \
 	apt-get update; \
@@ -23,23 +40,36 @@ RUN set -eux; \
 		g++ \
 		gcc \
 		libc6-dev \
-		make \
 		pkg-config \
 	; \
 	rm -rf /var/lib/apt/lists/*
 
+# Keep the literal cache-mount targets below in sync with these. The caches
+# are shared with Dockerfile.integration (same target paths).
+ENV GOCACHE=$HOME/.cache/go-build
+ENV GOMODCACHE=$GOPATH/pkg/mod
+
 COPY go.mod go.sum ./
 
-RUN go mod download
+RUN --mount=type=cache,target=/go/pkg/mod \
+	go mod download
 
 COPY ./cmd/ ./cmd/
 COPY ./ee/ ./ee/
 COPY ./pkg/ ./pkg/
 COPY ./templates /root/templates
 
-COPY Makefile Makefile
-RUN TARGET_DIR=/root ARCHS=${TARGETARCH} ZEUS_URL=${ZEUSURL} LICENSE_URL=${ZEUSURL}/api/v1 make go-build-enterprise-race
-RUN mv /root/linux-${TARGETARCH}/signoz /root/signoz
+# Invoked directly instead of via make so Makefile changes don't invalidate
+# this layer; the Makefile's git-derived ldflags resolve to empty in here
+# anyway (.git is dockerignored).
+RUN --mount=type=cache,target=/go/pkg/mod \
+	--mount=type=cache,target=/root/.cache/go-build \
+	GOARCH=${TARGETARCH} GOOS=${OS} go build -C ./cmd/enterprise -race -tags timetzdata -o /root/signoz \
+	-ldflags "-s -w \
+		-X github.com/SigNoz/signoz/pkg/version.version=integration \
+		-X github.com/SigNoz/signoz/pkg/version.variant=enterprise \
+		-X github.com/SigNoz/signoz/ee/zeus.url=${ZEUSURL} \
+		-X github.com/SigNoz/signoz/ee/zeus.deprecatedURL=${ZEUSURL}/api/v1"
 
 COPY --from=build /opt/build ./web/
 

docs/contributing/tests/e2e.md

@@ -30,11 +30,11 @@ yarn install:browsers   # one-time Playwright browser install
 
 ### Starting the Test Environment
 
-To spin up the backend stack (SigNoz, ClickHouse, Postgres, Zookeeper, Zeus mock, gateway mock, seeder, migrator-with-web) and keep it running:
+To spin up the backend stack (SigNoz, ClickHouse, Postgres, ClickHouse Keeper, Zeus mock, gateway mock, seeder, migrator-with-web) and keep it running:
 
 ```bash
 cd tests
-uv run pytest --basetemp=./tmp/ -vv --reuse --with-web \
+uv run pytest --basetemp=./tmp/ -vv --reuse --rebuild --with-web \
   e2e/bootstrap/setup.py::test_setup
 ```
 
@@ -45,8 +45,13 @@ This command will:
 - Start the HTTP seeder container (`tests/seeder/` — exposing `/telemetry/{traces,logs,metrics}` POST + DELETE)
 - Write backend coordinates to `tests/e2e/.env.local` (loaded by `playwright.config.ts` via dotenv)
 - Keep containers running via the `--reuse` flag
+- Rebuild the SigNoz container from the current sources via the `--rebuild` flag
 
-The `--with-web` flag builds the frontend into the SigNoz container — required for E2E. The build takes ~4 mins on a cold start.
+The `--with-web` flag builds the frontend into the SigNoz container — required for E2E. The build takes ~4 mins on a cold start; later builds are incremental.
+
+### Rebuilding After Source Changes
+
+The `--with-web` image bakes the built frontend in, so neither backend nor frontend changes are picked up while `--reuse` keeps the container running. `--rebuild` fixes that for both: it kills the SigNoz container, rebuilds the image incrementally (go build cache + pnpm store — a frontend-only change rebuilds in about a minute), and starts a fresh one while databases, mocks, migrations, and the seeder stay reused. The setup command above passes it, so the iteration loop is: change code → re-run the setup command → re-run your specs. `--rebuild` requires `--reuse` and cannot be combined with `--teardown` or `--clean`.
 
 ### Stopping the Test Environment
 
@@ -281,13 +286,16 @@ The full `playwright.config.ts` is the source of truth. Common things to tweak:
 The same pytest flags integration tests expose work here, since E2E reuses the shared fixture graph:
 
 - `--reuse` — keep containers warm between runs (required for all iteration).
+- `--rebuild` — recreate the SigNoz container from the current sources (backend and, with `--with-web`, frontend) while the rest of the stack stays up. Requires `--reuse`.
 - `--teardown` — tear everything down.
+- `--clean` — prune the docker build caches, forcing the next image build to start cold.
 - `--with-web` — build the frontend into the SigNoz container. **Required for E2E**; integration tests don't need it.
-- `--sqlstore-provider`, `--postgres-version`, `--clickhouse-version`, etc. — see `docs/contributing/integration.md`.
+- `--sqlstore-provider`, `--postgres-version`, `--clickhouse-version`, etc. — see `docs/contributing/tests/integration.md`.
 
 ## What should I remember?
 
-- **Always use the `--reuse` flag** when setting up the E2E stack. `--with-web` adds a ~4 min frontend build; you only want to pay that once.
+- **Always use the `--reuse` flag** when setting up the E2E stack. `--with-web` adds a ~4 min frontend build on a cold start; later builds are incremental.
+- **Changed backend or frontend code? Re-run the setup command** — it passes `--rebuild`, swapping the SigNoz container for one built from your current sources while the rest of the stack stays up.
 - **Don't teardown before setup.** `--reuse` correctly handles partially-set-up state, so chaining teardown → setup wastes time.
 - **Prefer UI-driven flows.** Playwright captures BE requests in the trace; a parallel `fetch` probe is almost always redundant. Drop to `page.request.*` only when the UI can't reach what you need.
 - **Use `page.waitForResponse` on UI clicks** to assert BE contracts — it still exercises the UI trigger path.

docs/contributing/tests/integration.md

@@ -37,13 +37,34 @@ make py-test-setup
 Under the hood this runs, from `tests/`:
 
 ```bash
-uv run pytest --basetemp=./tmp/ -vv --reuse integration/bootstrap/setup.py::test_setup
+uv run pytest --basetemp=./tmp/ -vv --reuse --rebuild --capture=no integration/bootstrap/setup.py::test_setup
 ```
 
 This command will:
-- Start all required services (ClickHouse, PostgreSQL, Zookeeper, SigNoz, Zeus mock, gateway mock)
+- Start all required services (ClickHouse, PostgreSQL, ClickHouse Keeper, SigNoz, Zeus mock, gateway mock)
 - Register an admin user
 - Keep containers running via the `--reuse` flag
+- Rebuild the SigNoz container from the current sources via the `--rebuild` flag
+
+### Rebuilding After Source Changes
+
+`--reuse` keeps the running SigNoz container, which means backend source changes are not picked up. `--rebuild` fixes exactly that: it kills the existing SigNoz container, rebuilds the image (incremental — only changed packages recompile thanks to the build cache), and starts a fresh one, while everything else (databases, mocks, migrations) stays reused. `make py-test-setup` passes it by default, so the iteration loop is simply:
+
+```bash
+make py-test-setup      # (re)build signoz from your current sources
+uv run pytest --basetemp=./tmp/ -vv --reuse integration/tests/<suite>/
+# ... edit backend code or tests ...
+make py-test-setup      # pick up the backend changes
+uv run pytest --basetemp=./tmp/ -vv --reuse integration/tests/<suite>/
+```
+
+The same applies to the e2e stack. `--rebuild` requires `--reuse` and cannot be combined with `--teardown` or `--clean`.
+
+Some suites define their own SigNoz variant in a suite-local `conftest.py` (`create_signoz(..., cache_key=...)` — e.g. `basepath`, `metricreduction`, `querier_json_body`). Those containers are not touched by `make py-test-setup`, which only rebuilds the default instance. For such suites, pass `--rebuild` on the suite run itself — it rebuilds every SigNoz variant the run instantiates:
+
+```bash
+uv run pytest --basetemp=./tmp/ -vv --reuse --rebuild integration/tests/<suite>/
+```
 
 ### Stopping the Test Environment
 
@@ -56,11 +77,21 @@ make py-test-teardown
 Which runs:
 
 ```bash
-uv run pytest --basetemp=./tmp/ -vv --teardown integration/bootstrap/setup.py::test_teardown
+uv run pytest --basetemp=./tmp/ -vv --teardown --capture=no integration/bootstrap/setup.py::test_teardown
 ```
 
 This destroys the running integration test setup and cleans up resources.
 
+### Cleaning the Image Build Cache
+
+The `signoz:integration` image build keeps its Go build and module caches in BuildKit cache mounts, so rebuilds only recompile what changed. These caches survive `--teardown` (they belong to the Docker builder, not to any container). If a cache ever needs to be nuked — suspected corruption, disk pressure, or to force a genuinely cold build — pass the `--clean` flag:
+
+```bash
+uv run pytest --basetemp=./tmp/ -vv --teardown --clean integration/bootstrap/setup.py::test_teardown
+```
+
+`--clean` prunes the docker build artifacts backing the incremental image build at session start, so the next build starts from a clean slate. Images and regular layer cache stay intact, but note the pruning is host-wide — it clears build caches for other projects too, not just SigNoz's. The flag composes with any invocation — passing it on a normal `--reuse` run simply makes the next image build start cold (~3–4 minutes instead of seconds).
+
 ## Understanding the Integration Test Framework
 
 Python and pytest form the foundation of the integration testing framework. Testcontainers are used to spin up disposable integration environments. WireMock is used to spin up **test doubles** of external services (Zeus cloud API, gateway, etc.).
@@ -99,7 +130,7 @@ tests/
 │       ├── passwordauthn/
 │       ├── querier/
 │       └── ...
-└── e2e/                     # Playwright suite (see docs/contributing/e2e.md)
+└── e2e/                     # Playwright suite (see docs/contributing/tests/e2e.md)
 ```
 
 Each test suite follows these principles:
@@ -224,9 +255,9 @@ Tests can be configured using pytest options:
 - `--sqlstore-provider` — Choose the SQL store provider (default: `postgres`)
 - `--sqlite-mode` — SQLite journal mode: `delete` or `wal` (default: `delete`). Only relevant when `--sqlstore-provider=sqlite`.
 - `--postgres-version` — PostgreSQL version (default: `15`)
-- `--clickhouse-version` — ClickHouse version (default: `25.5.6`)
-- `--zookeeper-version` — Zookeeper version (default: `3.7.1`)
-- `--schema-migrator-version` — SigNoz schema migrator version (default: `v0.144.2`)
+- `--clickhouse-version` — ClickHouse version, also used for ClickHouse Keeper (default: `25.12.5`)
+- `--schema-migrator-version` — SigNoz schema migrator version (default: `v0.144.6`)
+- `--with-web` — Build the frontend into the SigNoz image (required for e2e)
 
 Example:
 
@@ -239,6 +270,7 @@ uv run pytest --basetemp=./tmp/ -vv --reuse \
 ## What should I remember?
 
 - **Always use the `--reuse` flag** when setting up the environment or running tests to keep containers warm. Without it every run rebuilds the stack (~4 mins).
+- **Changed backend code? Re-run `make py-test-setup`** — it passes `--rebuild`, swapping the SigNoz container for one built from your current sources while the rest of the stack stays up.
 - **Use the `--teardown` flag** only when cleaning up — mixing `--teardown` with `--reuse` is a contradiction.
 - **Do not pre-emptively teardown before setup.** If the stack is partially up, `--reuse` picks up from wherever it is. `make py-test-teardown` then `make py-test-setup` wastes minutes.
 - **Follow the naming convention** with two-digit numeric prefixes (`01_`, `02_`) for ordered test execution within a suite.
@@ -247,5 +279,5 @@ uv run pytest --basetemp=./tmp/ -vv --reuse \
 - **Use descriptive test names** that clearly indicate what is being tested.
 - **Leverage fixtures** for common setup. The shared fixture package is at `tests/fixtures/` — reuse before adding new ones.
 - **Test both success and failure scenarios** (4xx / 5xx paths) to ensure robust functionality.
-- **Run `make py-fmt` and `make py-lint` before committing** Python changes — black + isort + autoflake + pylint.
+- **Run `make py-fmt` and `make py-lint` before committing** Python changes — ruff format + ruff check.
 - **`--sqlite-mode=wal` does not work on macOS.** The integration test environment runs SigNoz inside a Linux container with the SQLite database file mounted from the macOS host. WAL mode requires shared memory between connections, and connections crossing the VM boundary (macOS host ↔ Linux container) cannot share the WAL index, resulting in `SQLITE_IOERR_SHORT_READ`. WAL mode is tested in CI on Linux only.

tests/conftest.py

@@ -1,3 +1,5 @@
+import subprocess
+
 import pytest
 
 pytest_plugins = [
@@ -32,6 +34,22 @@ pytest_plugins = [
 ]
 
 
+def pytest_configure(config: pytest.Config):
+    if config.getoption("--rebuild"):
+        if not config.getoption("--reuse"):
+            raise pytest.UsageError("--rebuild requires --reuse: it replaces the signoz container within an environment that is being reused.")
+        if config.getoption("--teardown"):
+            raise pytest.UsageError("--rebuild cannot be combined with --teardown.")
+        if config.getoption("--clean"):
+            raise pytest.UsageError("--rebuild cannot be combined with --clean: --clean forces a cold build, which defeats the purpose of --rebuild.")
+
+
+def pytest_sessionstart(session: pytest.Session):
+    if session.config.getoption("--clean"):
+        # The type filter removes only cache mounts, leaving images and layer cache intact.
+        subprocess.run(["docker", "builder", "prune", "--force", "--filter", "type=exec.cachemount"], check=True)
+
+
 def pytest_addoption(parser: pytest.Parser):
     parser.addoption(
         "--reuse",
@@ -45,6 +63,18 @@ def pytest_addoption(parser: pytest.Parser):
         default=False,
         help="Teardown environment. Run pytest --basetemp=./tmp/ -vv --teardown src/bootstrap/setup::test_teardown to teardown your local dev environment.",
     )
+    parser.addoption(
+        "--rebuild",
+        action="store_true",
+        default=False,
+        help="Rebuild the signoz container from the current sources while reusing the rest of the stack (databases, mocks, migrations). Only meaningful together with --reuse: pytest --basetemp=./tmp/ -vv --reuse --rebuild integration/bootstrap/setup.py::test_setup.",
+    )
+    parser.addoption(
+        "--clean",
+        action="store_true",
+        default=False,
+        help="Prune the BuildKit cache mounts (go build and module caches) used by the signoz image build, forcing the next build to start cold. Combine with --teardown to reset everything: pytest --basetemp=./tmp/ -vv --teardown --clean integration/bootstrap/setup.py::test_teardown.",
+    )
     parser.addoption(
         "--with-web",
         action="store_true",

tests/fixtures/reuse.py

@@ -41,6 +41,7 @@ def wrap(  # pylint: disable=too-many-arguments,too-many-positional-arguments
     create: Callable[[], T],
     delete: Callable[[T], None],
     restore: Callable[[dict], T],
+    rebuild: bool = False,
 ) -> T:
     """
     Wraps a resource creation and cleanup process with reuse and teardown options.
@@ -51,6 +52,7 @@ def wrap(  # pylint: disable=too-many-arguments,too-many-positional-arguments
     - create: function to create the resource
     - delete: function to delete the resource
     - restore: function to restore resource from cache
+    - rebuild: under --reuse, delete the cached resource and recreate it instead of restoring it
     """
     resource = empty()
 
@@ -58,8 +60,13 @@ def wrap(  # pylint: disable=too-many-arguments,too-many-positional-arguments
         existing_resource = pytestconfig.cache.get(key, None)
         if existing_resource:
             assert isinstance(existing_resource, dict)
-            logger.info("Reusing existing %s(%s)", key, existing_resource)
-            return restore(existing_resource)
+            if rebuild:
+                logger.info("Rebuilding %s(%s), removing the existing one", key, existing_resource)
+                delete(restore(existing_resource))
+                pytestconfig.cache.set(key, None)
+            else:
+                logger.info("Reusing existing %s(%s)", key, existing_resource)
+                return restore(existing_resource)
 
     if not teardown(request):
         resource = create()

tests/fixtures/signoz.py

@@ -1,4 +1,6 @@
+import os
 import platform
+import subprocess
 import time
 from http import HTTPStatus
 from os import path
@@ -8,7 +10,6 @@ import docker.errors
 import pytest
 import requests
 from testcontainers.core.container import DockerContainer, Network
-from testcontainers.core.image import DockerImage
 
 from fixtures import reuse, types
 from fixtures.logger import setup_logger
@@ -50,17 +51,27 @@ def create_signoz(
 
         # Docker build context is the repo root — one up from pytest's
         # rootdir (tests/).
-        self = DockerImage(
-            path=str(pytestconfig.rootpath.parent),
-            dockerfile_path=dockerfile_path,
-            tag="signoz:integration",
-            buildargs={
-                "TARGETARCH": arch,
-                "ZEUSURL": zeus.container_configs["8080"].base(),
-            },
-        )
+        context = pytestconfig.rootpath.parent
 
-        self.build()
+        # The docker CLI is required: the Dockerfiles use BuildKit cache
+        # mounts, which docker-py does not support.
+        subprocess.run(
+            [
+                "docker",
+                "build",
+                "--file",
+                str(context / dockerfile_path),
+                "--tag",
+                "signoz:integration",
+                "--build-arg",
+                f"TARGETARCH={arch}",
+                "--build-arg",
+                f"ZEUSURL={zeus.container_configs['8080'].base()}",
+                str(context),
+            ],
+            check=True,
+            env=os.environ | {"DOCKER_BUILDKIT": "1"},
+        )
 
         env = (
             {
@@ -200,6 +211,7 @@ def create_signoz(
         create=create,
         delete=delete,
         restore=restore,
+        rebuild=pytestconfig.getoption("--rebuild"),
     )