mirror of
https://github.com/SigNoz/signoz.git
synced 2026-08-07 21:50:39 +01:00
5c0dfe2ad1
> **Stack** (review in order; each PR's diff is against its predecessor): > 1. #12323 `v2-read-path` — v2 native read path (leaf package) > 2. #12324 `v2-wiring` — wiring, shadow/pin rollout machinery, dual-leg conformance > 3. #12325 `v2-transpiler` — PromQL→ClickHouse transpiler + classification golden > 4. #12093 `issue-4293` — the /prometheus API move (breaking slice, last) ### What The performance half of the v2 provider: an allowlist compiler (`classify`/`rewrite`) that evaluates proven PromQL shapes entirely inside ClickHouse on the `timeSeries*ToGrid` aggregate functions (CH ≥ 25.6), so one row per output series comes back instead of every raw sample. Everything not provably equivalent falls back to the engine over the PR-1 querier; a transpilable subtree under a non-transpilable node runs hybrid (subtree materialized as synthetic series, engine on top). `TryExecuteRange` slots into the PR-2 serve/shadow paths (until now engine-only) through the new `prometheus.RangeExecutor` capability interface — the provider stays unexported and pkg/querier keeps holding `prometheus.Prometheus`; providers without the capability (v1) simply never transpile. The capability folds into the main interface once v1 is removed. Highlights (docs/contributing/prometheus.md carries the full correctness story): - Range functions map to verified grid aggregates; `increase` is `rate × range` exactly (same extrapolated delta, factor algebra). - Instant selectors reproduce stale-marker shadowing with a three-aggregate compare — skipping stale rows in WHERE would resurrect the sample the marker buried. - `*_over_time` at range = k·step aggregates whole step buckets (`groupArrayInsertAt` + slide) — no per-window fan-out, no prefix-sum differencing. - **Window-sliver filtering** (the headline perf commit, folded here): when the window is narrower than the step, only window/step of the timeline can influence any grid point; a lattice predicate in WHERE cuts the aggregate's input by the coverage ratio — measured 74s/28GiB → 16s/4.3GiB on a 36k-series 1w rate, and a 2.67B-sample case that exceeded 150GiB now completes in 19s/17GiB. Over sliver-filtered rows the last-style gates lift (instant selectors and `last_over_time` transpile at window < step), and disjoint-window `*_over_time` forms drop the divisibility gate. - Scalar-op pipelines apply in Go, slot by slot — same float64 ops, same order the AST dictates. Two guards land with it: - **Classification golden** (`classification_golden_test.go` + `testdata/classification_golden.json`): freezes the route (full/hybrid(n)/fallback + reason) of every conformance-corpus expression, one line each — 317 expressions: 132 full, 39 hybrid, 146 fallback. The test also requires each expression to route the same on every corpus grid; if a classifier change ever makes the route grid-dependent, the test fails and the key must grow. Routing is its own correctness surface — silently falling back costs the pushdown, silently transpiling an unproven shape risks wrong numbers; both now show up in review as a golden diff, with the corpus suite's v2 leg judging the numbers. - **Workload coverage reporter** (`TestClassifyCorpus`, env-gated): classifies a JSON-lines corpus of real dashboard/alert queries and buckets fallbacks by reason, to steer future allowlist work. **What the dual-leg suite caught on its first transpiled run** (evidence the PR-2 guard works, worth stating in review): - The classifier read a duration expression's offset (`x offset step()`) as zero and transpiled it — offset expressions parse *without* the experimental-parser flag, so they reach production. 20 corpus cases served silently wrong numbers. Fixed by refusing `OriginalOffsetExpr` / `RangeExpr` / `StepExpr` at classification (engine evaluates them exactly); regression cases added, golden regenerated (30 routings flipped to fallback). - Name-drop assembly treated temporally-disjoint same-labelset twins as separate series: `-{job="api"}` spanning `http_requests`/`http_errors` returned a 400 the engine would not raise, and hybrid `-metric_a or -metric_b` returned duplicate `{}` series. The engine's actual rule is: assemble the matrix by labelset, merging elements that never share an evaluation timestamp; error only on a same-timestamp conflict. Both the full-plan path (`mergeSameLabelsetSeries`) and the hybrid post-strip path (`mergeMatrixByLabelset`) now reproduce it, with unit tests pinning the corpus scenarios. - 12 remaining divergences, all one class, recorded in `known_divergences_v2.json` with causes: the engine aggregates with Kahan compensated summation (sum, sum_over_time) and an overflow-free incremental mean (avg); ClickHouse's `sumForEach`/`avgForEach`/`arraySum` are naive, so ±1e100 cancellation returns 0/residue and near-max-float64 `avg` overflows to ±Inf. Burn-down note: `sumKahanForEach` for the cancellation class; the overflow class needs an incremental-mean aggregate ClickHouse doesn't have. ### Alternatives considered and discarded - **General PromQL→SQL translation.** An allowlist inverts the failure mode: an overlooked construct becomes a fallback instead of a wrong number. Every shape on the list was validated slot-for-slot against the vendored engine on live data before entering it. - **ClickHouse's own PromQL dialect** (ClickHouse#57545, `dialect='promql'`). Emits the same grid functions, but currently covers only rate/irate/delta/idelta/last_over_time, has no fallback engine, and ties us to their TimeSeries table engine. We use the same primitives with our own classifier and our own exactness gates. - **Prefix-sum differencing for `*_over_time` windows.** Large-minus-large cancellation drifts past the shadow tolerance on counter-sized values; direct per-slot combination of at most W bucket partials adds the way the engine adds. - **Fanning each sample into every window that covers it.** Multiplies rows by W — billions of rows for a long range over a short step; the bucketed form's row count is series × buckets, the size of the output. - **Handling staleness by filtering stale rows in WHERE (instant units).** Resurrects the older real sample the marker was written to bury; hence the last-overall vs last-non-stale timestamp comparison. - **Transpiling @-modifier and default-resolution subqueries.** Their evaluation grid depends on server runtime settings the transpiler cannot see; they stay on the (exact) engine path. ### Test plan - `go test ./pkg/prometheus/clickhouseprometheusv2` — transpiler unit tests (SQL forms, classification, scalar ops, subquery grids), golden. - `pytest integration/tests/promqlconformance/` — the v2 leg now exercises transpiled serving for every routable corpus case; ledger unchanged (empty). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Pandey <vibhupandey28@gmail.com>