feat(promql): transpile allowlisted query shapes to ClickHouse grid statements

An allowlist compiler (classify/rewrite) evaluates proven PromQL shapes
entirely inside ClickHouse on the timeSeries*ToGrid aggregate functions
(CH >= 25.6): one row per output series comes back instead of every raw
sample. Everything not provably equivalent falls back to the engine over
the native querier; transpilable subtrees under non-transpilable nodes run
hybrid (materialized as synthetic series, engine on top). TryExecuteRange
slots into the serve/shadow paths, which until now ran engine-only.

Window-sliver filtering folded in: when the window is narrower than the
step, a lattice predicate admits only the samples any grid point can see -
measured 74s/28GiB -> 16s/4.3GiB on a 36k-series 1w rate, and a
2.67B-sample 1w case that died at 150GiB completes in 19s/17GiB. Over
slivered rows the last-style gates lift and disjoint over_time forms drop
the divisibility gate.

The classification golden freezes the routing decision (full/hybrid/
fallback + reason) for every conformance-corpus expression: silently
falling back costs the pushdown, silently transpiling an unproven shape
risks wrong numbers - both now surface in review as a golden diff.

The dual-leg conformance suite already earned its keep on its first
transpiled run:

- It caught the classifier reading a duration expression's offset
  (x offset step()) as zero - offset expressions parse WITHOUT the
  experimental-parser flag, so they reach production. Such selectors are
  now refused and the engine evaluates them exactly.
- It caught name-drop assembly treating temporally-disjoint same-labelset
  twins as separate series (-{job="api"} spanning http_requests and
  http_errors 400'd; hybrid -metric_a or -metric_b returned duplicate {}
  series). Both paths now merge by labelset slot-wise, raising the
  engine's duplicate error only on a same-timestamp conflict - the
  engine's actual rule.
- The 12 remaining divergences are one class, recorded with causes in
  known_divergences_v2.json (the swap scorecard): the engine sums with
  Kahan compensation and an overflow-free incremental mean, ClickHouse's
  sum/avg/arraySum are naive - visible only at 1e100-class cancellation
  and near-max-float overflow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/contributing/prometheus.md

@@ -1,123 +1,377 @@
 # PromQL Serving — clickhouseprometheusv2
 
-This document is the subsystem context for `pkg/prometheus/clickhouseprometheusv2`,
-the second-generation ClickHouse-backed Prometheus provider. It explains why the
-package exists, the correctness constraints that shaped it, and how each fetch
-reduction is proven not to change results. Any change to the provider must keep
-these invariants; if a change would violate one, it must be flagged and
-discussed.
+This document gives the context for `pkg/prometheus/clickhouseprometheusv2`.
+This package is the second-generation ClickHouse-backed Prometheus provider.
+The document tells you why the package exists. It tells you the correctness
+rules that shaped it. It shows how we prove that each construct does not
+change results. Keep these invariants when you change the provider. If your
+change breaks an invariant, flag it and discuss it first.
 
 ---
 
 ## Why a second provider
 
-The v1 provider (`pkg/prometheus/clickhouseprometheus`) serves the promql engine
-through the remote-read protobuf adapter: every raw sample of a query's union
-window is fetched, serialized, and handed to the engine. The cost is a function
-of ingested data, not of the question asked — which is how a dashboard of PromQL
-panels can take an instance down.
+The v1 provider (`pkg/prometheus/clickhouseprometheus`) serves the promql
+engine through the remote-read protobuf adapter. It fetches every raw sample
+of a query's union window. It serializes all of them and gives them to the
+engine. The cost follows the ingested data, not the question. This is how a
+dashboard of PromQL panels can take an instance down.
 
-In v2 the stock promql engine evaluates over a native `storage.Querier`: no
-translation layer, per-selector fetch windows, and fetch reductions that are
-provably invisible to the engine.
+In v2, each query runs in one of two ways. The classifier decides per query:
 
-**The core constraint: every reduction either preserves engine semantics exactly
-or is not performed.** A PromQL result that differs from upstream Prometheus is
-a lost user. The conformance suite
+- **Transpiled**: ClickHouse evaluates the query. Only final (or near-final)
+  per-group grid arrays come back. The statements use the
+  `timeSeries*ToGrid` aggregate functions. The supported ClickHouse floor is
+  25.6 or later, so these functions are assumed available.
+- **Engine**: the stock promql engine evaluates over this package's native
+  `storage.Querier`. Every shape that does not transpile takes this path.
+
+**The core rule: a PromQL result that differs from upstream Prometheus is a
+lost user. A construct that cannot reproduce engine semantics exactly falls
+back. It does not approximate.** The conformance suite
 (`tests/integration/tests/promqlconformance/`) replays Prometheus' own test
-corpus against both providers and is the arbiter.
+corpus against both providers. It is the arbiter. The classification golden
+(`testdata/classification_golden.json`) freezes the route of each corpus
+expression. The rest of this document is the PromQL-to-SQL story. That
+mapping is where correctness is won or lost.
 
 ---
 
+## The evaluation model the SQL must reproduce
+
+A PromQL range query is an instant query evaluated at each grid point
+`t_i = start + i*step`, for `i = 0..(end-start)/step`. At each `t_i`:
+
+- An instant selector resolves to the latest sample in the left-open
+  lookback window `(t_i - lookback, t_i]`. If that latest sample is a stale
+  marker, the selector resolves to nothing. Older real samples in the window
+  do not change this.
+- A range selector `[r]` collects every sample in `(t_i - r, t_i]`. Stale
+  markers are excluded.
+- `offset d` shifts both windows to `(t_i - d - w, t_i - d]`.
+
+The transpilation invariant follows from this model. Each transpiled
+construct produces one array per output series. The array has exactly one
+slot per grid point. Slot `i` holds the value at `t_i`. NULL means absent.
+This makes composition correct, not only convenient. The engine evaluates
+these operators independently per `t_i`. A representation that gets every
+slot right gets the whole query right. Spatial aggregation over arrays is
+sound because it combines values that belong to the same `t_i` by
+construction. Scan time maps slot `i` back to `t_i = start + i*step`
+(`toMatrix`). The sections below fill those slots with exactly the numbers
+the engine computes. We validated each equivalence against the vendored
+engine on live data before its shape entered the allowlist. An unproven
+shape stays on the engine path.
+
+## Classification: finding what a statement can answer
+
+`classify` walks the parsed AST and looks for "core units". A core unit is a
+maximal subtree of this shape:
+
+	[agg by/without (...)] [fn(] selector[range] [offset d] [)] [op scalar]...
+
+`classifyCore` peels that chain from the outside in. It takes an optional
+sum/min/max/avg/count aggregation. It then takes one allowlisted function or
+a bare instant selector. It then takes the selector with its offset. On the
+way out, it collects number-literal arithmetic, comparisons (including
+`bool`), and unary minus into a scalar-op pipeline. A node qualifies only if
+its type, arguments, and children are in the proven set. This is an
+allowlist. An overlooked construct becomes a fallback, not a wrong number.
+
+Three unit kinds come out. Each kind has its own SQL form:
+
+- `unitRange`: rate, irate, increase, delta, idelta over a range selector.
+- `unitInstant`: instant vector selection, bare or comparison-filtered.
+- `unitOverTime`: avg/min/max/sum/count/last `_over_time`.
+
+If the whole tree is one unit, the plan is "full". The statement's rows are
+the query result. Otherwise, `rewrite` cuts out each maximal unit and puts a
+synthetic selector `__signoz_transpiled_N__` in its place. The engine then
+runs the rewritten expression over the units' materialized results. This is
+a "hybrid" plan. `histogram_quantile`, `topk`, `or`/`and`/`unless`, and
+vector matching keep exact engine semantics. Their expensive inputs were
+aggregated server-side.
+
+Classification refuses a shape when it cannot guarantee exact semantics
+server-side:
+
+- The `@` modifier, anywhere.
+- Default-resolution subqueries. Their resolution is a server runtime
+  setting that the transpiler cannot see.
+- Duration expressions (`offset step()`, `[range()]`, ...), anywhere. The
+  engine resolves them into the selector's static fields only at evaluation
+  time. At classification time those fields hold zero values. A transpile
+  would silently use the wrong offset or range.
+- Steps or ranges that are not whole seconds. The grid functions take
+  whole-second parameters.
+- Grouping by `__name__`, or matching on it, in hybrid plans. The synthetic
+  name would leak into results.
+- Name-keeping units in hybrid plans. Bare and comparison-filtered instant
+  selectors and `last_over_time` keep their real `__name__` (`keepsName`).
+  Substitution would replace that name. These units transpile only as full
+  plans.
+- Every function outside the allowlist: changes, resets,
+  quantile_over_time, absent, native-histogram functions, and more.
+
+Units inside a fixed-resolution subquery evaluate on the subquery's own
+grid, not the query grid. That grid is the set of epoch-aligned multiples of
+the resolution strictly after `outerStart - offset - range`, ending at
+`outer end - offset`. This is the exact derivation the engine uses. A grid
+shifted by one step changes which samples every window sees.
+
+## From one unit to one statement
+
+`buildUnitSQL` renders each unit as one statement. For
+`sum by (pod) (rate(m{job="api"}[5m]))` the skeleton is:
+
+	SELECT g0, sumForEach(grid) AS grid FROM (
+	    SELECT any(series.g0) AS g0,
+	           timeSeriesRateToGrid(<start>, <end>, <step>, <range>)(fromUnixTimestamp64Milli(unix_milli), value) AS grid
+	    FROM signoz_metrics.distributed_samples_v4 AS points
+	    INNER JOIN (
+	        SELECT fingerprint, JSONExtractString(labels, 'pod') AS g0
+	        FROM signoz_metrics.time_series_v4
+	        WHERE <series predicates>
+	        GROUP BY fingerprint, g0
+	    ) AS series ON points.fingerprint = series.fingerprint
+	    WHERE metric_name = ? AND temporality IN ['Cumulative', 'Unspecified']
+	      AND unix_milli > <start - range> AND unix_milli <= <end>
+	      AND bitAnd(flags, 1) = 0
+	    GROUP BY points.fingerprint
+	) GROUP BY g0
+	SETTINGS allow_experimental_ts_to_grid_aggregate_function = 1
+
+Read it from the inside out.
+
+**The time window** is the selector's semantics, verbatim. Strict `>` on the
+lower bound and `<=` on the upper bound is the left-open `(t - w, t]` rule.
+The offset shifts the whole window. `bitAnd(flags, 1) = 0` drops stale
+markers. PromQL excludes them from range vectors.
+
+**The inner GROUP BY** computes one grid array per series.
+`timeSeriesRateToGrid(start, end, step, range)` is a parametric aggregate.
+It takes (timestamp, value) pairs and produces `Array(Nullable(Float64))`
+with one slot per grid point. It is correct because it implements the
+engine's `extrapolatedRate`, decision for decision: counter resets, the
+zero-point clamp, the extrapolation thresholds, the two-samples rule, and
+the left-open window. We verified this: we fed identical samples to both and
+compared slot for slot. The only observed difference is the last bit.
+ClickHouse's C++ and Go round the same formula differently. That is the
+floating-point floor, not a semantic gap. irate/delta/idelta map to their
+own `timeSeries*ToGrid` functions, with the same verification. `increase`
+has no function of its own. We emit
+`arrayMap(x -> x * <range seconds>, <rate expr>)`. This is exact by
+definition: `extrapolatedRate` computes the same extrapolated delta for both
+and divides by the range only when `isRate`. The multiplication reverses it
+exactly. The grid parameters render as literals, not bound args. They are
+aggregate-function parameters. The experimental gate rides as a SETTINGS
+clause on the statement itself, so telemetrystore hooks cannot remove it.
+
+The group key is functionally dependent on the fingerprint: one fingerprint
+is the hash of one labelset. So the inner query groups by the fingerprint
+alone and reads the key columns with `any()`. This is exact, and it makes
+the per-row hash key smaller.
+
+**The join** gives each series its group key, in one of two forms.
+`by (...)` extracts each listed label as a plain column
+(`JSONExtractString(labels, 'pod') AS g0`) and groups on the columns. The
+projection is a known short list, and the label names live in Go. To build,
+sort, and stringify every label pair per row would be waste. This is correct
+because column-tuple equality is label-set equality on the projection. An
+extracted `''` means the label is absent. That is Prometheus semantics for
+`by()` over missing labels. The empties are skipped when the columns turn
+back into labels. `without` and no-aggregation project a label set that
+varies per series. They get the canonical key: `toJSONString` of the sorted
+[label, value] pairs that the unit projects. `without` excludes the listed
+labels plus `__name__`. No-aggregation keeps everything; the name comes off
+in Go, per the engine's name-dropping rules. Here the sort is load-bearing.
+Stored JSON key order is not canonical across fingerprints. Two orderings of
+the same labels must land in one group. Empty values are filtered for the
+same absent-label reason. The same string parses back into the output label
+set (`labelsFromGroupKey`).
+
+**The outer GROUP BY** is the spatial aggregation. sum/min/max/avg/count
+by/without become the `-ForEach` combinators. Element-wise aggregation over
+grid arrays is the engine's per-`t_i` aggregation: slot `i` of every input
+array refers to the same `t_i`. The combinators skip NULLs. That is the
+engine aggregating only the series present at `t_i`. An index where every
+series is absent stays NULL. Two edges need explicit handling. First,
+`countForEach` wraps in a map of 0 back to NULL. A count over an all-absent
+index is an absent point, not 0. Second, a unit without aggregation still
+passes through `maxForEach`. That is the identity for the common
+one-fingerprint group. It is a deterministic NULL-skipping merge when a
+regex `__name__` selector collapses distinct metrics onto one projected
+label set. One caveat is inherent: the summation order over series differs
+from the engine's. Spatial aggregates can differ in the last ULP. Float
+addition is not associative. No ordering reproduces the engine's result
+bit-exactly from inside a GROUP BY.
+
+## Instant selectors: staleness needs two aggregates
+
+`unitInstant` uses window = lookback. It must reproduce the shadowing rule:
+the point is absent when the latest in-window sample is a stale marker.
+`timeSeriesLastToGrid` alone cannot express that. To skip stale rows in
+WHERE would resurrect the older real sample that the marker buried. So stale
+rows stay in the scan for this kind only. The grid expression compares three
+aggregates per slot:
+
+	arrayMap((tall, tok, vok) -> if(tall IS NULL OR tok IS NULL OR tall != tok, NULL, vok),
+	         timeSeriesLastToGrid(...)(ts, toFloat64(unix_milli)),                           -- last sample overall
+	         timeSeriesLastToGridIf(...)(ts, toFloat64(unix_milli), bitAnd(flags, 1) = 0),   -- last non-stale, its timestamp
+	         timeSeriesLastToGridIf(...)(ts, value, bitAnd(flags, 1) = 0))                   -- last non-stale, its value
+
+This is correct by cases on a slot's window. No samples at all: both
+timestamp aggregates are NULL, so the slot is NULL. That is absent, as the
+engine says. Latest sample non-stale: it is the latest overall and the
+latest non-stale. The timestamps agree. The slot takes its value. That is
+the engine's pick. Latest sample stale: the last-overall timestamp is the
+marker's. The last-non-stale timestamp is older, or NULL when the window
+holds only markers. They disagree. The slot is NULL. The marker shadows,
+exactly as the engine's rule says. Timestamps are unique per series (ingest
+dedups). So timestamp equality identifies "the same sample" without
+ambiguity. We probed the `-If` combinator against these experimental
+aggregates before we trusted it.
+
+## Windowed *_over_time: whole buckets instead of a grid function
+
+avg/min/max/sum/count `_over_time` aggregate every raw sample in the window.
+No `timeSeries*ToGrid` function computes them. (`last_over_time` is the
+exception. The last sample of a range vector is exactly
+`timeSeriesLastToGrid`. PromQL excludes stale markers from range vectors; we
+exclude them in WHERE.) These shapes transpile only when the range is a
+whole multiple of the step. Then the window needs no per-sample fan-out.
+With `W = range/step`, the window `(t_k - range, t_k]` is exactly the union
+of W step buckets. Both are left-open on the same boundaries. So bucket
+membership fully determines window membership. Each sample lands in exactly
+one bucket:
+
+	intDiv(unix_milli - <start> + <range> - 1, <step>)
+
+This is `ceil((ts - start)/step)` shifted by W-1, so the earliest in-window
+sample sits at 0. Slot k's window is buckets in `[k, k+W-1]`. The
+alternative fans each sample into all W windows that cover it. That
+multiplies rows by W. For a long range over a short step, that is a row
+explosion measured in billions. The bucketed form's row count is
+series × buckets: the size of the output, for any W.
+
+Each series aggregates in one group. The `-Resample` combinator
+(`sumResample`, `countResample`) holds the dense per-bucket partials inside
+one group state: a bucket count, plus the function's value aggregate (sum
+for sum/avg, min, max). An earlier form grouped by (series, bucket) and
+assembled with `groupArrayInsertAt`. At scale that made 37M hash groups, and
+per-thread partials scaled memory with the thread count. The slide then
+combines each slot's at-most-W bucket partials by direct aggregation
+(`arraySum(arraySlice(...))`). Window sums are added the way the engine adds
+them. There is no prefix-sum differencing: its large-minus-large
+cancellation would drift past the shadow tolerance on counter-sized values.
+This is correct per slot because the bucket union is the exact window
+multiset, and avg/min/max/sum/count are order-insensitive on a multiset
+(sum/avg up to summation order; see the float caveat above). A slot with
+zero window count is absent. min/max filter their slices on the bucket
+counts. An empty bucket's default can never look like a value: a real sample
+can legitimately be +Inf.
+
+Two shapes fall back to the engine path, which is exact: a range that does
+not divide the step, and a window wider than `maxWindowBuckets` buckets (the
+slide costs W combines per slot). A range narrower than the step needs
+neither gate: the windows are pairwise disjoint, one bucket per slot, no
+slide. That form is exact only together with the window-sliver predicate
+below.
+
+## Scalar ops, full plans, hybrid plans
+
+The scalar-op pipeline runs in Go on the returned arrays
+(`applyScalarOps`), slot by slot. Arithmetic operators compute. Comparisons
+filter: the slot keeps the vector-side value or becomes NULL. Under `bool`
+they return 0/1. This is trivially correct. It is the same float64 operation
+the engine applies, to the same slot value, in the same operator order the
+AST dictates. Go instead of another SQL layer changes where, not what.
+
+A full plan's arrays map straight to the result matrix. A hybrid plan
+materializes each unit's arrays as synthetic series under its
+`__signoz_transpiled_N__` name. The engine evaluates the rewritten
+expression over a storage that serves synthetic names from memory and
+everything else live. Substitution is sound because a unit's output is a
+plain instant vector to the engine: same values at same timestamps, under a
+different name. The name cannot matter. Plans that group by or match on
+`__name__` were refused at classification. Name-keeping units are never
+substituted. One subtlety makes it exact: we write stale markers at absent
+grid points. Without them, the engine's lookback would resurrect a point
+from up to `lookback` earlier. The marker encodes "absent here" the way the
+engine itself encodes it. Units evaluate concurrently. Each unit is one
+series lookup plus one grid statement. A step of 0 is an instant query: a
+single evaluation at `end`.
+
+A note on the window sliver: when the window is narrower than the step, the
+grid windows cover only `window/step` of the timeline. A sample in a gap
+belongs to no window. It cannot move any grid point, but the grid aggregate
+would buffer it. A WHERE predicate keeps only the in-window rows:
+`positiveModulo(selStart - unix_milli, step) < window`, with the scan capped
+at the last grid point. The lattice anchors at the selector start, because
+the end can sit off-lattice on unaligned grids. This cut a 36k-series
+one-week rate from 74s/28GiB to 16s/4.3GiB on fleet data. Over slivered
+rows, `timeSeriesLastToGrid`'s window widening is harmless, so instant
+selectors and `last_over_time` transpile at window < step too.
+
 ## Series lookup
 
-Matchers resolve to series once per selector (`selectSeries`) against the series
-tables, which hold one row per (fingerprint, bucket) at 1h/6h/1d/1w
-granularities. Table selection and window rounding delegate to the shared
-metrics schema package (`pkg/telemetryschema/metricstelemetryschema`); the
-window start rounds down to the bucket boundary so a window beginning mid-bucket
-still matches the bucket's row.
+Both paths resolve matchers the same way, once per selector
+(`selectSeries`). The series tables hold one row per (fingerprint, bucket)
+at 1h/6h/1d/1w granularities. The shared schema package
+(`pkg/telemetryschema/metricstelemetryschema`) picks the table whose bucket
+fits the window. It rounds the window start down to the bucket boundary, so
+a window that begins mid-bucket still matches the bucket's row. How matchers
+become SQL, and why regexes are anchored, is documented at
+`applySeriesConditions`. Empty-valued labels come off at this boundary. An
+empty value means "label absent" in Prometheus, but stored attribute JSON
+can carry them.
 
-How matchers become SQL is documented at `applySeriesConditions`. The rules that
-carry semantics:
+## The engine path
 
-- `__name__` matchers (all four types) translate to the `metric_name` column.
-- Every other matcher becomes a `JSONExtractString` condition on the labels
-  column. An equality matcher against `""` matches series *without* the label,
-  mirroring PromQL, because `JSONExtractString` returns `""` for missing keys.
-- Regexes are anchored (`^(?:...)$`) before they reach `match()`: PromQL
-  matchers match the whole value, ClickHouse `match()` searches for a
-  substring.
-- The series-lookup upper bound is inclusive (`unix_milli <= end`) because the
-  exporter floors registration rows to the bucket start: a series first
-  registered in the bucket beginning exactly at `end` would otherwise be
-  invisible while its samples are in range.
-
-Empty-valued labels come off at this boundary: an empty value means "label
-absent" in Prometheus, but stored attribute JSON can carry them.
-
----
-
-## Sample fetch
-
-Samples are fetched per selector using the engine's per-selector hints, not the
-query-wide union window — `foo / foo offset 1d` reads two narrow windows
-instead of the widest one twice.
-
-**Last-sample-per-step reduction.** Instant selectors of subquery-free queries
-fetch only the last sample per step bucket. The engine resolves an instant
-selector at each grid timestamp `t` to the latest sample in the left-open
-lookback window `(t − lookback, t]`. Buckets anchor at the selector's first
-evaluation timestamp — recovered from the hints as
-`hints.Start + lookback − 1ms`, the inverse of how the engine derives
-`hints.Start` — so bucket boundaries coincide with evaluation timestamps, and a
-non-final sample of a bucket can never be the latest sample in
-`(t − lookback, t]` for any grid `t`. Real timestamps are preserved, so the
-engine's own lookback and staleness handling stay exact.
-
-Range selectors always fetch raw — every sample feeds the range function. The
-subquery-free proof travels in the context as `prometheus.QueryTraits`, because
-subquery selectors evaluate at the subquery's step while the hints carry the
-top-level step; call sites that do not attach traits get the conservative raw
-fetch.
-
-**Row assembly** maps stale flags to the engine's `StaleNaN` and merges series
-with identical label sets (`sortAndMerge`) — the engine assumes storages never
-emit duplicates. Duplicate timestamps pass through as stored: uniqueness is
-ingest's job, and v1 feeds them to the engine as-is over the same data.
-
-**The fingerprint filter is a shard-local semi-join.** The samples query
-restricts to the matched series by re-running the series predicates as an
-`IN (SELECT fingerprint FROM <local series table> ...)` subquery, not a GLOBAL
-broadcast of the matched set. ClickHouse materializes the subquery's set per
-shard before the scan, so it still engages the fingerprint primary-key column.
-Because the subquery re-executes the predicates after the lookup ran, it can
-match series registered in between; sample rows whose fingerprint the lookup
-never saw are skipped — the lookup is the read snapshot.
-
----
+Queries that do not transpile run in the stock engine over this package's
+`storage.Querier`. This is still not the v1 path. Samples are fetched per
+selector with the engine's per-selector hints, not the query-wide union
+window. So `foo / foo offset 1d` reads two narrow windows, not the widest
+one twice. Instant selectors of subquery-free queries fetch only the last
+sample per step bucket (`lastSamplePerStep`). Buckets anchor at the
+selector's first evaluation timestamp. The code recovers it from the hints
+as `hints.Start + lookback - 1ms`, the inverse of how the engine derives
+`hints.Start`. Bucket boundaries then coincide with evaluation timestamps.
+A non-final sample of a bucket can never be the latest sample in
+`(t - lookback, t]` for any grid `t`. Real timestamps are preserved, so the
+engine's own lookback and staleness handling stay exact. Range selectors
+always fetch raw: every sample feeds the range function. The subquery-free
+proof travels in the context as `prometheus.QueryTraits`. Subquery selectors
+evaluate at the subquery's step, while the hints carry the top-level step.
+Row assembly maps stale flags to the engine's StaleNaN. It merges series
+with identical label sets (`sortAndMerge`): the engine assumes storages
+never emit duplicates.
 
 ## Sharding
 
-`samples_v4` and `time_series_v4` (and all their rollups) shard on the same key
-— `cityHash64(env, temporality, metric_name, fingerprint)` — so a series'
-samples and catalog rows live on the same shard. The semi-join above exploits
-that: each shard filters by its own series rows, which are exactly the series
-of that shard's samples.
-
-The temporality filter on every samples statement
-(`temporality IN ['Cumulative', 'Unspecified']`) is a semantic no-op — the
-matched fingerprints already come from those temporalities — that engages the
-leading samples primary-key column.
-
-Delta-temporality series stay invisible to PromQL exactly as they are in v1:
-the rollout gate is parity with v1, and a Delta stream fed to `rate()`
+`samples_v4` and `time_series_v4` (and all their rollups) shard on the same
+key: `cityHash64(env, temporality, metric_name, fingerprint)`. So a series'
+samples and catalog rows live on the same shard. The transpiled statement
+exploits that. The distributed samples table at the top-level FROM makes
+ClickHouse rewrite the whole inner query per shard. The join against the
+shard-local series table and the per-series grid aggregation run next to
+the data. The initiator only merges aggregate states and applies the
+spatial `-ForEach` step. This is the same layout as the telemetrymetrics
+statement builder. The group-key join alone restricts the transpiled scan
+to the matched series. The engine path's samples fetch restricts by the
+same predicates as a shard-local semi-join, not a GLOBAL broadcast of the
+matched set. The temporality filter on every samples statement is a
+semantic no-op: the matched fingerprints already come from those
+temporalities. It engages the leading samples primary-key column.
+Delta-temporality series stay invisible to PromQL here, exactly as in v1.
+The rollout gate is parity with v1. To make Delta visible is its own change
+with its own semantics to design. A Delta stream fed to `rate()`
 as-if-cumulative would be wrong, not just new.
 
----
-
 ## Observability
 
 Every statement carries a `log_comment` with
-`code.namespace=clickhouse-prometheus-v2` and `code.function.name` naming the
-call site, so this provider's work is attributable in `system.query_log`.
+`code.namespace=clickhouse-prometheus-v2` and `code.function.name` naming
+the call site (`selectSeries`, `selectSamples`, `transpiledUnit`,
+`LabelValues`, `LabelNames`). This provider's work is attributable in
+`system.query_log` without guessing from query text.

pkg/prometheus/clickhouseprometheusv2/classification_golden_test.go

@@ -0,0 +1,162 @@
+package clickhouseprometheusv2
+
+import (
+	"encoding/json"
+	"flag"
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"testing"
+
+	"github.com/prometheus/prometheus/promql/parser"
+	"github.com/stretchr/testify/require"
+)
+
+var updateGolden = flag.Bool("update", false, "rewrite the classification golden file")
+
+const goldenFile = "testdata/classification_golden.json"
+
+// corpusFile is the conformance corpus that the integration suite replays.
+// The golden freezes the route of every expression in it.
+const corpusFile = "../../../tests/integration/testdata/promqltestcorpus/corpus.json"
+
+// TestClassificationGolden freezes the route of every conformance-corpus
+// expression: "full", "hybrid(<units>)", or "fallback: <reason>". The route
+// is a correctness surface of its own. A change that silently sends a shape
+// to the engine loses the pushdown. A change that silently transpiles an
+// unproven shape risks wrong numbers. Both must show as a diff of this file.
+// The corpus suite's clickhousev2 leg then judges the numbers.
+//
+// The golden keys on the expression alone. The corpus evaluates each
+// expression on several grids, and the test requires the route to be the
+// same on all of them. If a classifier change ever makes the route depend
+// on the grid, this test fails and the key must grow.
+//
+// Regenerate after an intended classifier change:
+//
+//	go test ./pkg/prometheus/clickhouseprometheusv2 -run TestClassificationGolden -update
+func TestClassificationGolden(t *testing.T) {
+	raw, err := os.ReadFile(corpusFile)
+	require.NoError(t, err)
+
+	var corpus struct {
+		Cases []struct {
+			Expr    string `json:"expr"`
+			StartMs int64  `json:"start_ms"`
+			EndMs   int64  `json:"end_ms"`
+			StepMs  int64  `json:"step_ms"`
+		} `json:"cases"`
+	}
+	require.NoError(t, json.Unmarshal(raw, &corpus))
+	require.NotEmpty(t, corpus.Cases)
+
+	promParser := parser.NewParser(parser.Options{})
+	routes := map[string]string{}
+	for _, c := range corpus.Cases {
+		expr, err := promParser.ParseExpr(c.Expr)
+		require.NoError(t, err, "corpus expression must parse: %q", c.Expr)
+
+		var route string
+		plan, ok := classify(expr, gridContext{startMs: c.StartMs, endMs: c.EndMs, stepMs: c.StepMs})
+		switch {
+		case ok && plan.full:
+			route = "full"
+		case ok:
+			route = fmt.Sprintf("hybrid(%d)", len(plan.units))
+		default:
+			route = "fallback: " + fallbackShape(expr)
+		}
+
+		if prev, seen := routes[c.Expr]; seen {
+			require.Equal(t, prev, route,
+				"route differs between grids for %q — the golden key must grow to include the grid", c.Expr)
+			continue
+		}
+		routes[c.Expr] = route
+	}
+
+	// json.MarshalIndent sorts map keys: the file is deterministic.
+	got, err := json.MarshalIndent(routes, "", "  ")
+	require.NoError(t, err)
+	got = append(got, '\n')
+
+	if *updateGolden {
+		require.NoError(t, os.MkdirAll(filepath.Dir(goldenFile), 0o755))
+		require.NoError(t, os.WriteFile(goldenFile, got, 0o644))
+		return
+	}
+
+	want, err := os.ReadFile(goldenFile)
+	require.NoError(t, err, "golden missing — generate it with -update")
+	require.Equal(t, string(want), string(got),
+		"classification route changed; if intended, regenerate with -update and explain the diff in review")
+}
+
+// fallbackShape buckets a non-transpilable query by why it stays on the engine
+// path, to separate "already served well" (instant selectors on the last-sample-per-step
+// path) from genuine compiler gaps.
+func fallbackShape(expr parser.Expr) string {
+	var hasMatrix, hasSubquery, hasAt, hasDurationExpr, overTime bool
+	rangeFns := map[string]bool{"rate": true, "increase": true, "delta": true, "irate": true, "idelta": true}
+	var unsupportedFns []string
+	parser.Inspect(expr, func(node parser.Node, _ []parser.Node) error {
+		switch n := node.(type) {
+		case *parser.MatrixSelector:
+			hasMatrix = true
+			if n.RangeExpr != nil {
+				hasDurationExpr = true
+			}
+		case *parser.SubqueryExpr:
+			hasSubquery = true
+			if n.RangeExpr != nil || n.StepExpr != nil || n.OriginalOffsetExpr != nil {
+				hasDurationExpr = true
+			}
+		case *parser.VectorSelector:
+			if n.Timestamp != nil || n.StartOrEnd != 0 {
+				hasAt = true
+			}
+			if n.OriginalOffsetExpr != nil {
+				hasDurationExpr = true
+			}
+		case *parser.Call:
+			if strings.HasSuffix(n.Func.Name, "_over_time") {
+				overTime = true
+			} else if !rangeFns[n.Func.Name] {
+				unsupportedFns = append(unsupportedFns, n.Func.Name)
+			}
+		}
+		return nil
+	})
+
+	switch {
+	case hasDurationExpr:
+		return "duration expression (resolved only at evaluation time)"
+	case hasSubquery:
+		return "subquery"
+	case hasAt:
+		return "@ modifier"
+	case overTime:
+		return "*_over_time range function"
+	case !hasMatrix:
+		return "instant-selector shape (last-sample-per-step engine path)"
+	case len(unsupportedFns) > 0:
+		return fmt.Sprintf("range shape with unsupported function(s): %s", strings.Join(dedupe(unsupportedFns), ",")) //nolint:makezero
+	default:
+		return "other range shape"
+	}
+}
+
+func dedupe(in []string) []string {
+	seen := map[string]bool{}
+	var out []string
+	for _, s := range in {
+		if !seen[s] {
+			seen[s] = true
+			out = append(out, s)
+		}
+	}
+	sort.Strings(out)
+	return out
+}

pkg/prometheus/clickhouseprometheusv2/provider.go

@@ -2,27 +2,33 @@ package clickhouseprometheusv2
 
 import (
 	"context"
+	"time"
 
 	"github.com/SigNoz/signoz/pkg/factory"
 	"github.com/SigNoz/signoz/pkg/prometheus"
 	"github.com/SigNoz/signoz/pkg/telemetrystore"
+	"github.com/prometheus/prometheus/promql"
 	"github.com/prometheus/prometheus/storage"
 )
 
-// provider ties the package together: its own engine and parser, and the
-// ClickHouse client behind the native storage.Querier. It stays unexported:
-// callers hold the prometheus.Prometheus interface, which is the boundary
-// between the two provider implementations.
+// provider ties the package together: its own engine and parser, the
+// ClickHouse client behind the native storage.Querier, and the transpiler
+// executor. It stays unexported. Callers hold the prometheus.Prometheus
+// interface, which is the boundary between the two provider
+// implementations. They reach the transpiler only through the
+// prometheus.RangeExecutor capability.
 type provider struct {
 	settings factory.ScopedProviderSettings
 	engine   *prometheus.Engine
 	parser   prometheus.Parser
 	client   *client
+	executor *executor
 }
 
 var (
 	_ prometheus.Prometheus        = (*provider)(nil)
 	_ prometheus.StatementCapturer = (*provider)(nil)
+	_ prometheus.RangeExecutor     = (*provider)(nil)
 )
 
 func NewFactory(telemetryStore telemetrystore.TelemetryStore) factory.ProviderFactory[prometheus.Prometheus, prometheus.Config] {
@@ -43,9 +49,17 @@ func New(_ context.Context, providerSettings factory.ProviderSettings, config pr
 		engine:   engine,
 		parser:   parser,
 		client:   client,
+		executor: &executor{client: client, engine: engine, parser: parser},
 	}, nil
 }
 
+// TryExecuteRange evaluates transpilable query shapes directly in ClickHouse
+// (see transpiler.go). ok=false means the shape is not transpilable and the
+// caller should evaluate through Engine over Storage instead.
+func (p *provider) TryExecuteRange(ctx context.Context, query string, start, end time.Time, step time.Duration) (promql.Matrix, bool, error) {
+	return p.executor.TryExecuteRange(ctx, query, start, end, step)
+}
+
 func (p *provider) Engine() *prometheus.Engine {
 	return p.engine
 }

pkg/prometheus/clickhouseprometheusv2/testdata/classification_golden.json

@@ -0,0 +1,319 @@
+{
+  "(metric1_total offset 2) ^ 2": "full",
+  "-metric_a or -metric_b": "hybrid(2)",
+  "-metric_total": "full",
+  "-{job=\"api\"}": "full",
+  "10 atan2 20": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "10 atan2 NaN": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "AVG(http_requests) BY (job)": "full",
+  "COUNT(http_requests) BY (job)": "full",
+  "MAX(http_requests) BY (job)": "full",
+  "MIN(http_requests) BY (job)": "full",
+  "SUM BY (group) (((http_requests{job=\"api-server\"})))": "full",
+  "SUM BY (group) (http_requests{job=\"api-server\"})": "full",
+  "SUM(http_requests)": "full",
+  "SUM(http_requests) BY (job)": "full",
+  "SUM(http_requests) BY (job, group)": "full",
+  "SUM(http_requests) BY (job, nonexistent)": "full",
+  "SUM(http_requests{instance=\"0\"}) BY(job)": "full",
+  "abs(-1 * http_requests{group=\"production\",job=\"api-server\"})": "hybrid(1)",
+  "acos(trig - 10.1)": "hybrid(1)",
+  "acosh(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "asin(trig - 10.1)": "hybrid(1)",
+  "asinh(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "atan(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "atanh(trig - 10.1)": "hybrid(1)",
+  "avg by (group) (data{test=\"nan\"})": "full",
+  "avg by (group) (data{test=\"neg_inf\"})": "full",
+  "avg by (group) (data{test=\"pos_inf\"})": "full",
+  "avg by (group) (http_requests{job=\"api-server\"})": "full",
+  "avg(data)": "full",
+  "avg(data{test=\"-big\"})": "full",
+  "avg(data{test=\"-inf\"})": "full",
+  "avg(data{test=\"-inf2\"})": "full",
+  "avg(data{test=\"-inf3\"})": "full",
+  "avg(data{test=\"big\"})": "full",
+  "avg(data{test=\"bigzero\"})": "full",
+  "avg(data{test=\"inf\"})": "full",
+  "avg(data{test=\"inf2\"})": "full",
+  "avg(data{test=\"inf3\"})": "full",
+  "avg(data{test=\"inf_inf\"})": "full",
+  "avg(data{test=\"nan\"})": "full",
+  "avg(data{test=\"ten\"})": "full",
+  "avg(foo) - 52": "full",
+  "avg(foo) == 52": "full",
+  "avg(topk(10, foo)) - 52": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "avg(topk(10, foo)) == 52": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "avg(topk(11, foo)) - 52": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "avg(topk(11, foo)) == 52": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "avg(topk(8, foo)) - 52": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "avg(topk(8, foo)) == 52": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "avg(topk(9, foo)) - 52": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "avg(topk(9, foo)) == 52": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "avg_over_time(foo[100s]) - 52": "full",
+  "avg_over_time(foo[100s]) == 52": "full",
+  "avg_over_time(foo[110s]) - 52": "full",
+  "avg_over_time(foo[110s]) == 52": "full",
+  "avg_over_time(foo[120s]) - 52": "full",
+  "avg_over_time(foo[120s]) == 52": "full",
+  "avg_over_time(foo[130s]) - 52": "full",
+  "avg_over_time(foo[130s]) == 52": "full",
+  "avg_over_time(metric10[1m])": "full",
+  "avg_over_time(metric11[1m])": "full",
+  "avg_over_time(metric1[1m])": "full",
+  "avg_over_time(metric2[1m])": "full",
+  "avg_over_time(metric3[1m])": "full",
+  "avg_over_time(metric4[1m])": "full",
+  "avg_over_time(metric5[1m])": "full",
+  "avg_over_time(metric6[1m])": "full",
+  "avg_over_time(metric7[1m])": "full",
+  "avg_over_time(metric8[1m])": "full",
+  "avg_over_time(metric9[1m])": "full",
+  "avg_over_time(metric[2m])": "full",
+  "avg_over_time(rate(http_requests_total[1m])[1m:1s])": "hybrid(1)",
+  "ceil(0.004 * http_requests{group=\"production\",job=\"api-server\"})": "hybrid(1)",
+  "changes(http_requests[1800])": "fallback: range shape with unsupported function(s): changes",
+  "changes(http_requests[30m])": "fallback: range shape with unsupported function(s): changes",
+  "changes(metric[1m])": "fallback: range shape with unsupported function(s): changes",
+  "changes(metric[5m])": "fallback: range shape with unsupported function(s): changes",
+  "changes(x[20m])": "fallback: range shape with unsupported function(s): changes",
+  "clamp(metric_total, 0, 100)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "cos(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "cosh(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "count by (group) (http_requests{job=\"api-server\"})": "full",
+  "count by(namespace, pod, cpu) (node_cpu_seconds_total{cpu=~\".*\",job=\"node-exporter\",mode=\"idle\",namespace=\"observability\",pod=\"node-exporter-l454v\"}) * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:{namespace=\"observability\",pod=\"node-exporter-l454v\"}": "hybrid(1)",
+  "count_over_time(metric1_total[range()])": "fallback: duration expression (resolved only at evaluation time)",
+  "count_over_time(metric1_total[step()])": "fallback: duration expression (resolved only at evaluation time)",
+  "count_over_time(metric[10])": "full",
+  "count_over_time(metric[10s])": "full",
+  "count_over_time(metric[1m])": "full",
+  "count_over_time(metric[1s])": "full",
+  "count_over_time(metric[20])": "full",
+  "count_over_time(metric[20s])": "full",
+  "deg(trig - 10)": "hybrid(1)",
+  "deg(trig - 20)": "hybrid(1)",
+  "deg(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "delta(metric[1m])": "full",
+  "floor(0.004 * http_requests{group=\"production\",job=\"api-server\"})": "hybrid(1)",
+  "foo \u003e 2 or bar": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "http_requests_total{foo!=\"bar\", job=\"api-server\"}": "full",
+  "http_requests_total{foo!=\"bar\"}": "full",
+  "http_requests_total{foo!~\"bar\", job=\"api-server\", instance=\"1\", x!=\"y\", z=\"\", group!=\"\"}": "full",
+  "http_requests_total{foo!~\"bar\", job=\"api-server\"}": "full",
+  "http_requests_total{group!=\"canary\"}": "full",
+  "http_requests_total{group=\"production\",job=\"api-server\"} offset 5m": "full",
+  "http_requests_total{group=\"production\",job=~\"api-.+\"}": "full",
+  "http_requests_total{job!~\"api-.+\",group!=\"canary\"}": "full",
+  "http_requests_total{job=~\".+-server\",group!=\"canary\"}": "full",
+  "increase(http_requests_total[100m])": "full",
+  "increase(http_requests_total[30m])": "full",
+  "increase(http_requests_total[50m])": "full",
+  "increase(metric[1m])": "full",
+  "increase(metric[5m])": "full",
+  "label_join(series, \"idx\", \",\", \"label\", \"label\")": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "label_replace((((testmetric))), ((\"dst\")), ((\"value-$1\")), ((\"src\")), ((\"non-matching-regex\")))": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "label_replace(series, \"idx\", \"replaced\", \"idx\", \".*\")": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "label_replace(sum by (__name__) (rate(metric_total{env=\"2\"}[5m])), \"__name__\", \"$1\", \"__name__\", \"(.+)\")": "fallback: range shape with unsupported function(s): label_replace",
+  "label_replace(testmetric, \"dst\", \"\", \"dst\", \".*\")": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "label_replace(testmetric, \"dst\", \"$1-value-$2\", \"src\", \"(.*)-value-(.*)\")": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "label_replace(testmetric, \"dst\", \"destination-value-$1\", \"src\", \"source-value-(.*)\")": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "label_replace(testmetric, \"dst\", \"destination-value-$1\", \"src\", \"value-(.*)\")": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "label_replace(testmetric, \"dst\", \"value-$1\", \"nonexistent-src\", \"(.*)\")": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "label_replace(testmetric, \"dst\", \"value-$1\", \"nonexistent-src\", \"source-value-(.*)\")": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "label_replace(testmetric, \"dst\", \"value-$1\", \"src\", \"non-matching-regex\")": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "last_over_time(metric_total{env=\"1\"}[10m])": "full",
+  "max_over_time(metric_total{env=\"1\"}[10m])": "full",
+  "metric": "full",
+  "metric1 offset 15m or metric2 offset 45m": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "metric1_total offset +min(step(), 1s)^0": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset -(min(step(), 1s))+8000": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset -min(step(), 1s)+8000": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset -min(step(), 1s)^0": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset -step()*2": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset 100 + 2": "full",
+  "metric1_total offset 2 ^ 2": "full",
+  "metric1_total offset STEP()": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset max(3s,min(step(), 1s))+8000": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset min(range(), 8s)": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset min(step(), 1s)": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset min(step(), 1s)+8000": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset min(step(), 1s)^0": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset range()": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset step()": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset step()*0": "fallback: duration expression (resolved only at evaluation time)",
+  "metric1_total offset step()^0": "fallback: duration expression (resolved only at evaluation time)",
+  "metricA + ignoring() metricB": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "metricA + metricB": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "metric_total * 2": "full",
+  "metric_total + another_metric_total": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "metric_total \u003c= another_metric_total": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "metric_total \u003c= bool another_metric_total": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "metric_total{env=\"1\"}": "full",
+  "min_over_time(metric_total[10s])": "full",
+  "min_over_time(metric_total[15s:10s])": "fallback: subquery",
+  "min_over_time(rate(metric_total[5m])[20m:1m])": "hybrid(1)",
+  "node_cpu % 2": "full",
+  "node_cpu * 2": "full",
+  "node_cpu * ignoring (role, mode) group_left (role) node_role": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "node_cpu * on (instance) group_left (role) node_role": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "node_cpu + 2": "full",
+  "node_cpu + on(dummy) group_left(foo) random*0": "hybrid(1)",
+  "node_cpu - 2": "full",
+  "node_cpu / 2": "full",
+  "node_cpu / ignoring (mode) group_left sum without (mode)(node_cpu)": "hybrid(1)",
+  "node_cpu / ignoring (mode) group_left(dummy) sum without (mode)(node_cpu)": "hybrid(1)",
+  "node_cpu / on (instance) group_left sum by (instance,job)(node_cpu)": "hybrid(1)",
+  "node_cpu \u003e on(job, instance) group_left(target) (threshold or on (job, instance) (sum by (job, instance)(node_cpu) * 0 + 1))": "hybrid(1)",
+  "node_cpu \u003e on(job, instance) group_left(target) threshold": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "node_cpu ^ 2": "full",
+  "node_role * ignoring (role) group_right (role) node_var": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "node_role * on (instance) group_right (role) node_var": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "node_var * ignoring (role) group_left (role) node_role": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "node_var * on (instance) group_left (role) node_role": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "other + fill": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "present_over_time(http_requests_total[10m])": "fallback: *_over_time range function",
+  "present_over_time(http_requests_total[16m])": "fallback: *_over_time range function",
+  "present_over_time(http_requests_total[5m])": "fallback: *_over_time range function",
+  "present_over_time(http_requests_total[6m])": "fallback: *_over_time range function",
+  "present_over_time(httpd_handshake_failures_total[1m])": "fallback: *_over_time range function",
+  "present_over_time(httpd_log_lines_total[30s])": "fallback: *_over_time range function",
+  "present_over_time(rate(http_requests_total[5m])[5m:1m])": "hybrid(1)",
+  "present_over_time({instance=\"127.0.0.1\"}[5m:5s])": "fallback: subquery",
+  "present_over_time({instance=\"127.0.0.1\"}[5m])": "fallback: *_over_time range function",
+  "present_over_time({job=\"grok\"}[20m])": "fallback: *_over_time range function",
+  "present_over_time({job=\"ingress\"}[4m])": "fallback: *_over_time range function",
+  "rad(trig - 10)": "hybrid(1)",
+  "rad(trig - 20)": "hybrid(1)",
+  "rad(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "random + on() metricA": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "rate(calculate_rate_offset_total[10m] offset 5m)": "full",
+  "rate(calculate_rate_window_total[50m])": "full",
+  "rate(http_requests_total[1m])": "full",
+  "rate(http_requests_total[40s]) - rate(http_requests_total[1m] offset 10000s)": "hybrid(2)",
+  "rate(http_requests_total{group=~\"((?i)PRO).*\"}[1m])": "full",
+  "rate(http_requests_total{group=~\"(?i:PRO).*\"}[1m])": "full",
+  "rate(http_requests_total{group=~\"(?i:PRODUCTION)\"}[1m])": "full",
+  "rate(http_requests_total{group=~\".*((?i)DUC).*\"}[1m])": "full",
+  "rate(http_requests_total{group=~\".*((?i)TION)\"}[1m])": "full",
+  "rate(http_requests_total{group=~\".*(?i:C).*\"}[1m])": "full",
+  "rate(http_requests_total{group=~\".*(?i:DUC).*\"}[1m])": "full",
+  "rate(http_requests_total{group=~\".*(?i:TION)\"}[1m])": "full",
+  "rate(http_requests_total{group=~\".*(?i:TION).*?\"}[1m])": "full",
+  "rate(http_requests_total{group=~\".*?(?i:PRO).*\"}[1m])": "full",
+  "rate(http_requests_total{group=~\".*ry\", instance=\"1\"}[1m])": "full",
+  "rate(http_requests_total{group=~\"pro.*\"}[1m:10s])": "fallback: subquery",
+  "rate(http_requests_total{group=~\"pro.*\"}[1m])": "full",
+  "rate(http_requests_total{instance!=\"3\"}[1m] offset 10000s)": "full",
+  "rate(metric_total[1m1s:10s])": "fallback: subquery",
+  "rate(metric_total[1m500ms:10s])": "fallback: subquery",
+  "rate(metric_total[1m])": "full",
+  "rate(metric_total[20s:10s])": "fallback: subquery",
+  "rate(metric_total[20s:5s])": "fallback: subquery",
+  "rate(metric_total{env=\"1\"}[10m])": "full",
+  "rate(sum_over_time((metric1_total+metric2_total+metric3_total)[30s:10s])[30s:10s])": "fallback: subquery",
+  "rate(sum_over_time(metric1_total[30s:10s])[50s:10s])": "fallback: subquery",
+  "rate(sum_over_time(metric2_total[30s:10s])[50s:10s])": "fallback: subquery",
+  "rate(sum_over_time(metric3_total[30s:10s])[50s:10s])": "fallback: subquery",
+  "rate(testcounter_reset_end_total[5m])": "full",
+  "rate(testcounter_reset_end_total[6m])": "full",
+  "rate(testcounter_reset_middle_total[50m])": "full",
+  "rate(testcounter_zero_cutoff_total[20m])": "full",
+  "requests * 2": "full",
+  "resets(metric[1m])": "fallback: range shape with unsupported function(s): resets",
+  "resets(metric[5m])": "fallback: range shape with unsupported function(s): resets",
+  "round(-1 * (0.004 * http_requests{group=\"production\",job=\"api-server\"}))": "hybrid(1)",
+  "round(-1 * (0.005 * http_requests{group=\"production\",job=\"api-server\"}))": "hybrid(1)",
+  "round(-1 * (1 + 0.005 * http_requests{group=\"production\",job=\"api-server\"}))": "hybrid(1)",
+  "round(-1 * (5.2 + 0.0005 * http_requests{group=\"production\",job=\"api-server\"}), 0.1)": "hybrid(1)",
+  "round(0.0005 * http_requests{group=\"production\",job=\"api-server\"}, 0.1)": "hybrid(1)",
+  "round(0.004 * http_requests{group=\"production\",job=\"api-server\"})": "hybrid(1)",
+  "round(0.005 * http_requests{group=\"production\",job=\"api-server\"})": "hybrid(1)",
+  "round(0.025 * http_requests{group=\"production\",job=\"api-server\"}, 5)": "hybrid(1)",
+  "round(0.045 * http_requests{group=\"production\",job=\"api-server\"}, 5)": "hybrid(1)",
+  "round(1 + 0.005 * http_requests{group=\"production\",job=\"api-server\"})": "hybrid(1)",
+  "round(2.1 + 0.0005 * http_requests{group=\"production\",job=\"api-server\"}, 0.1)": "hybrid(1)",
+  "round(5.2 + 0.0005 * http_requests{group=\"production\",job=\"api-server\"}, 0.1)": "hybrid(1)",
+  "round(metric_total)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "sin(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "sinh(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stddev (series)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stddev by (instance)(http_requests)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stddev by (label) (series)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stddev(http_requests)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stddev(series)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stddev_over_time(metric[1m])": "fallback: *_over_time range function",
+  "stdvar (series)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stdvar by (instance)(http_requests)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stdvar by (label) (series)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stdvar(http_requests)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stdvar(series)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "stdvar_over_time(metric[1m])": "fallback: *_over_time range function",
+  "sum by () (http_requests{job=\"api-server\"})": "full",
+  "sum by (__name__) (metric_total{env=\"1\"} or rate(metric_total{env=\"2\"}[5m]))": "fallback: other range shape",
+  "sum by (__name__) (metric_total{env=\"1\"})": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "sum by (__name__) (metric_total{env=\"3\"} or rate(metric_total{env=\"2\"}[5m]))": "fallback: other range shape",
+  "sum by (__name__) (rate(metric_total{env=\"2\"}[5m]) or metric_total{env=\"1\"})": "fallback: other range shape",
+  "sum by (__name__) (rate(metric_total{env=\"2\"}[5m]))": "fallback: other range shape",
+  "sum by (__name__) (rate(metric_total{env=\"3\"}[5m]) or metric_total{env=\"1\"})": "fallback: other range shape",
+  "sum by (__name__, env) (metric_total{env=\"1\"})": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "sum by (group) (data{test=\"nan\"})": "full",
+  "sum by (group) (data{test=\"neg_inf\"})": "full",
+  "sum by (group) (data{test=\"pos_inf\"})": "full",
+  "sum by (group) (http_requests{job=\"api-server\"})": "full",
+  "sum by (mode, job)(node_cpu) / on (job) group_left sum by (job)(node_cpu)": "hybrid(2)",
+  "sum without () (http_requests{job=\"api-server\",group=\"production\"})": "full",
+  "sum without (instance) (http_requests{job=\"api-server\"} or foo)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "sum without (instance) (http_requests{job=\"api-server\"})": "full",
+  "sum without (instance)(node_cpu) / ignoring (mode) group_left sum without (instance, mode)(node_cpu)": "hybrid(2)",
+  "sum(data{test=\"inf_inf\"})": "full",
+  "sum(data{test=\"ten\"})": "full",
+  "sum(http_requests) by (job) + min(http_requests) by (job) + max(http_requests) by (job) + avg(http_requests) by (job)": "hybrid(4)",
+  "sum(http_requests{job=\"api-server\"})": "full",
+  "sum(label_grouping_test) by (a, b)": "full",
+  "sum(sum by (group) (http_requests{job=\"api-server\"})) by (job)": "hybrid(1)",
+  "sum(sum by (mode, job)(node_cpu) / on (job) group_left sum by (job)(node_cpu))": "hybrid(2)",
+  "sum(sum without (instance)(node_cpu) / ignoring (mode) group_left sum without (instance, mode)(node_cpu))": "hybrid(2)",
+  "sum_over_time((metric1_total)[30:10] offset 3)": "fallback: subquery",
+  "sum_over_time((metric1_total)[30:10] offset 3s)": "fallback: subquery",
+  "sum_over_time((metric1_total)[30:10s] offset 3s)": "fallback: subquery",
+  "sum_over_time((metric1_total)[30s:10s] offset 3s)": "fallback: subquery",
+  "sum_over_time(bar[30s])": "full",
+  "sum_over_time(metric1_total[30:10] offset 3)": "fallback: subquery",
+  "sum_over_time(metric1_total[30s:10s] offset 10s)": "fallback: subquery",
+  "sum_over_time(metric1_total[30s:10s] offset 3s)": "fallback: subquery",
+  "sum_over_time(metric1_total[30s:10s] offset 5s)": "fallback: subquery",
+  "sum_over_time(metric1_total[30s:10s] offset 7s)": "fallback: subquery",
+  "sum_over_time(metric1_total[30s:10s] offset 9s)": "fallback: subquery",
+  "sum_over_time(metric1_total[30s:10s])": "fallback: subquery",
+  "sum_over_time(metric1_total[30s:5s])": "fallback: subquery",
+  "sum_over_time(metric[1000ms])": "full",
+  "sum_over_time(metric[1001ms])": "fallback: *_over_time range function",
+  "sum_over_time(metric[1002ms])": "fallback: *_over_time range function",
+  "sum_over_time(metric[1003ms])": "fallback: *_over_time range function",
+  "sum_over_time(metric[2000ms])": "full",
+  "sum_over_time(metric[2001ms])": "fallback: *_over_time range function",
+  "sum_over_time(metric[2002ms])": "fallback: *_over_time range function",
+  "sum_over_time(metric[2003ms])": "fallback: *_over_time range function",
+  "sum_over_time(metric[2m])": "full",
+  "sum_over_time(metric[3000ms])": "full",
+  "sum_over_time(metric[3001ms])": "fallback: *_over_time range function",
+  "sum_over_time(metric[3002ms])": "fallback: *_over_time range function",
+  "sum_over_time(metric[3003ms])": "fallback: *_over_time range function",
+  "sum_over_time(metric_total[50s:10s])": "fallback: subquery",
+  "sum_over_time(metric_total[50s:5s])": "fallback: subquery",
+  "sum_over_time(metric_total[60s:10s])": "fallback: subquery",
+  "tan(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "tanh(trig)": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "test_total \u003c bool test_smaller": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "test_total \u003c test_smaller": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "test_total \u003e bool test_smaller": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "test_total \u003e test_smaller": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "testmetric": "full",
+  "topk(10, sum by (__name__, env) (metric_total{env=\"1\"}))": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "topk(10, sum by (__name__, env) (rate(metric_total{env=\"1\"}[10m])))": "fallback: other range shape",
+  "trigy atan2 trigNaN": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "trigy atan2 trigx": "fallback: instant-selector shape (last-sample-per-step engine path)",
+  "x{y=\"testvalue\"}": "full",
+  "{__name__=~\".+\"}": "full",
+  "{job=~\".+-server\", job!~\"api-.+\"}": "full"
+}

pkg/prometheus/clickhouseprometheusv2/transpiler.go

@@ -0,0 +1,493 @@
+package clickhouseprometheusv2
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/prometheus/prometheus/model/labels"
+	"github.com/prometheus/prometheus/promql/parser"
+)
+
+// The transpiler turns PromQL subtrees into single ClickHouse statements.
+// The statements use the timeSeries*ToGrid aggregate functions (ClickHouse
+// 25.6 or later). We verified their semantics against this repo's vendored
+// engine: exact extrapolatedRate behavior with counter resets, the counter
+// zero-point clamp, the 1.1x-average extrapolation threshold, left-open
+// windows, the two-samples rule, stale-marker shadowing, and millisecond
+// grid starts. Sample rows never leave ClickHouse. One row per output
+// series comes back. It holds the whole grid as an array.
+//
+// Scope (the allowlist): an optional sum/min/max/avg/count by/without
+// aggregation over a core unit, plus number-literal arithmetic, comparisons
+// and unary minus on top. A core unit is a rate/increase/delta/irate/idelta
+// range selection, an instant vector selection, or an
+// avg/min/max/sum/count/last _over_time window. Units inside
+// fixed-resolution subqueries evaluate on the subquery's own grid. Every
+// other shape falls back to the engine over this package's querier. When a
+// transpilable subtree sits under a non-transpilable node, the plan is
+// hybrid: ClickHouse computes the subtree's grids, and the engine reads
+// them as synthetic series (see transpiler_exec.go). See
+// docs/contributing/prometheus.md for the fallback list and the reason for
+// each entry.
+
+// rangeFn is a transpilable range-vector function.
+type rangeFn string
+
+const (
+	fnRate     rangeFn = "rate"
+	fnIncrease rangeFn = "increase"
+	fnDelta    rangeFn = "delta"
+	fnIRate    rangeFn = "irate"
+	fnIDelta   rangeFn = "idelta"
+)
+
+var gridFunction = map[rangeFn]string{
+	fnRate:     "timeSeriesRateToGrid",
+	fnIncrease: "timeSeriesRateToGrid", // increase == rate * range seconds, exactly (same factor algebra)
+	fnDelta:    "timeSeriesDeltaToGrid",
+	fnIRate:    "timeSeriesInstantRateToGrid",
+	fnIDelta:   "timeSeriesInstantDeltaToGrid",
+}
+
+// scalarOp is one number-literal arithmetic or comparison applied to a
+// compiled vector, evaluated in Go during assembly with the same float64
+// operations the engine uses.
+type scalarOp struct {
+	op           parser.ItemType
+	scalar       float64
+	scalarOnLeft bool
+	returnBool   bool
+}
+
+// isComparison reports whether the op is a filtering/bool comparison, which
+// preserves the metric name (arithmetic drops it).
+func (o scalarOp) isComparison() bool {
+	return o.op.IsComparisonOperator()
+}
+
+// unitKind is the selector shape at the bottom of a core unit.
+type unitKind int
+
+const (
+	// unitRange: rate/increase/delta/irate/idelta over a matrix selector.
+	unitRange unitKind = iota
+	// unitInstant: a plain vector selector resolved per grid point with
+	// lookback and stale-marker shadowing.
+	unitInstant
+	// unitOverTime: avg/min/max/sum/count/last_over_time over a matrix
+	// selector (aggregation over the window's samples, stale rows excluded).
+	unitOverTime
+)
+
+// coreUnit is one transpilable subtree: selector [-> range function] ->
+// optional aggregation -> scalar op pipeline.
+type coreUnit struct {
+	kind     unitKind
+	matchers []*labels.Matcher
+	offsetMs int64
+	fn       rangeFn // unitRange
+	overFn   string  // unitOverTime: avg|min|max|sum|count|last
+	rangeMs  int64   // unitRange/unitOverTime window
+
+	hasAgg   bool
+	aggOp    parser.ItemType // SUM MIN MAX AVG COUNT
+	by       bool
+	grouping []string
+
+	ops []scalarOp
+}
+
+// keepsName reports whether the unit's output series keep their real
+// __name__. Bare and comparison-filtered instant selectors keep it, and so
+// does last_over_time: they return the raw sample, name included. Range
+// functions, the other *_over_time functions, aggregations, arithmetic, and
+// bool comparisons all drop it. A bool comparison returns 0/1, not the
+// sample, so the engine drops the name there too. A unit that keeps the
+// name cannot become a synthetic series in a hybrid plan: the synthetic
+// name would replace the real one. It transpiles fine as a full plan, where
+// assembly emits the real names.
+func (u *coreUnit) keepsName() bool {
+	nameKeepingSelector := u.kind == unitInstant || (u.kind == unitOverTime && u.overFn == "last")
+	if !nameKeepingSelector || u.hasAgg {
+		return false
+	}
+	for _, op := range u.ops {
+		if !op.isComparison() || op.returnBool {
+			return false
+		}
+	}
+	return true
+}
+
+// gridContext is the evaluation grid a unit computes on. The query grid for
+// top-level units; for units inside subqueries, the subquery's own grid:
+// epoch-aligned multiples of its resolution covering the subquery window,
+// exactly as the engine derives it (engine.go, *parser.SubqueryExpr case).
+type gridContext struct {
+	startMs int64
+	endMs   int64
+	stepMs  int64
+}
+
+// subqueryGrid derives the inner grid for a subquery evaluated on outer:
+// interval S, end = outer end − offset, start = first multiple of S strictly
+// greater than outer start − offset − range.
+func subqueryGrid(outer gridContext, rangeMs, stepMs, offsetMs int64) gridContext {
+	lower := outer.startMs - offsetMs - rangeMs
+	start := stepMs * (lower / stepMs)
+	if start <= lower {
+		start += stepMs
+	}
+	return gridContext{startMs: start, endMs: outer.endMs - offsetMs, stepMs: stepMs}
+}
+
+// transpiledUnit is a coreUnit scheduled for execution, named for hybrid
+// substitution, carrying the grid it evaluates on.
+type transpiledUnit struct {
+	core coreUnit
+	name string // __signoz_transpiled_<n>__
+	grid gridContext
+}
+
+// transpilePlan is the outcome of classifying a query.
+type transpilePlan struct {
+	units []*transpiledUnit
+	grid  gridContext // the query's top-level grid
+	// full is set when the entire query is units[0]; otherwise rewritten
+	// holds the query with each unit replaced by a synthetic selector, to be
+	// evaluated by the engine over a hybrid storage.
+	full      bool
+	rewritten string
+}
+
+const syntheticNamePrefix = "__signoz_transpiled_"
+
+func syntheticName(i int) string {
+	return fmt.Sprintf("%s%d__", syntheticNamePrefix, i)
+}
+
+// classifyCore matches a subtree against the transpilable core shape.
+// stepMs gates second-granularity: the grid functions take whole-second step
+// and window parameters (grid *starts* are millisecond-precise).
+func classifyCore(node parser.Expr, stepMs int64) (*coreUnit, bool) {
+	unit := &coreUnit{}
+
+	expr := node
+	// Peel scalar ops and parens off the top, outermost first; ops apply in
+	// evaluation order, so prepend while peeling.
+	for {
+		switch n := expr.(type) {
+		case *parser.ParenExpr:
+			expr = n.Expr
+			continue
+		case *parser.UnaryExpr:
+			if n.Op != parser.SUB {
+				expr = n.Expr // unary '+' is a no-op
+				continue
+			}
+			// -x == -1 * x for every float64 (incl. NaN and signed zero).
+			unit.ops = append([]scalarOp{{op: parser.MUL, scalar: -1}}, unit.ops...)
+			expr = n.Expr
+			continue
+		case *parser.StepInvariantExpr:
+			// @-pinned expressions evaluate on a different grid.
+			return nil, false
+		case *parser.BinaryExpr:
+			lit, litOnLeft, ok := numberLiteralSide(n)
+			if !ok {
+				return nil, false
+			}
+			if !n.Op.IsOperator() && !n.Op.IsComparisonOperator() {
+				return nil, false
+			}
+			if n.Op == parser.ATAN2 {
+				// atan2 is arithmetic in PromQL but rarely used; keep the
+				// allowlist tight.
+				return nil, false
+			}
+			returnBool := n.ReturnBool
+			unit.ops = append([]scalarOp{{op: n.Op, scalar: lit, scalarOnLeft: litOnLeft, returnBool: returnBool}}, unit.ops...)
+			if litOnLeft {
+				expr = n.RHS
+			} else {
+				expr = n.LHS
+			}
+			continue
+		}
+		break
+	}
+
+	// Optional aggregation.
+	if agg, ok := expr.(*parser.AggregateExpr); ok {
+		switch agg.Op {
+		case parser.SUM, parser.MIN, parser.MAX, parser.AVG, parser.COUNT:
+		default:
+			return nil, false
+		}
+		for _, g := range agg.Grouping {
+			if g == metricNameLabel {
+				// by(__name__)/without(__name__) over synthetic or compiled
+				// output needs name bookkeeping the compiler doesn't do.
+				return nil, false
+			}
+		}
+		unit.hasAgg = true
+		unit.aggOp = agg.Op
+		unit.by = !agg.Without
+		unit.grouping = agg.Grouping
+		expr = agg.Expr
+		for {
+			if p, ok := expr.(*parser.ParenExpr); ok {
+				expr = p.Expr
+				continue
+			}
+			break
+		}
+	}
+
+	// The grid functions take whole-second steps; stepMs == 0 is an instant
+	// query (single-point grid).
+	if stepMs < 0 || stepMs%1000 != 0 {
+		return nil, false
+	}
+
+	// Bare instant selector: resolved per grid point with lookback and
+	// stale-marker shadowing (see compiler_sql.go).
+	if vs, ok := expr.(*parser.VectorSelector); ok {
+		// A duration expression (offset step(), offset range()*2, ...) is
+		// resolved into OriginalOffset only at evaluation time; at
+		// classification time the field still holds its zero value, so
+		// transpiling would silently use the wrong offset.
+		if vs.Timestamp != nil || vs.StartOrEnd != 0 || vs.Anchored || vs.Smoothed || vs.OriginalOffsetExpr != nil {
+			return nil, false
+		}
+		offsetMs := vs.OriginalOffset.Milliseconds()
+		if offsetMs < 0 {
+			return nil, false
+		}
+		unit.kind = unitInstant
+		unit.offsetMs = offsetMs
+		unit.matchers = vs.LabelMatchers
+		return unit, true
+	}
+
+	// Range or *_over_time function over a plain matrix selector.
+	call, ok := expr.(*parser.Call)
+	if !ok {
+		return nil, false
+	}
+	var fn rangeFn
+	var overFn string
+	switch call.Func.Name {
+	case "rate":
+		fn = fnRate
+	case "increase":
+		fn = fnIncrease
+	case "delta":
+		fn = fnDelta
+	case "irate":
+		fn = fnIRate
+	case "idelta":
+		fn = fnIDelta
+	case "avg_over_time", "min_over_time", "max_over_time", "sum_over_time", "count_over_time", "last_over_time":
+		overFn = strings.TrimSuffix(call.Func.Name, "_over_time")
+	default:
+		return nil, false
+	}
+	if len(call.Args) != 1 {
+		return nil, false
+	}
+	ms, ok := call.Args[0].(*parser.MatrixSelector)
+	if !ok {
+		return nil, false
+	}
+	vs, ok := ms.VectorSelector.(*parser.VectorSelector)
+	if !ok {
+		return nil, false
+	}
+	// Duration expressions resolve at evaluation time (see the instant
+	// selector case above); Range/OriginalOffset would be read as zero here.
+	if vs.Timestamp != nil || vs.StartOrEnd != 0 || vs.Anchored || vs.Smoothed || vs.OriginalOffsetExpr != nil || ms.RangeExpr != nil {
+		return nil, false
+	}
+
+	rangeMs := ms.Range.Milliseconds()
+	offsetMs := vs.OriginalOffset.Milliseconds()
+	if rangeMs <= 0 || rangeMs%1000 != 0 || offsetMs < 0 {
+		return nil, false
+	}
+
+	if overFn != "" {
+		unit.kind = unitOverTime
+		unit.overFn = overFn
+	} else {
+		unit.kind = unitRange
+		unit.fn = fn
+	}
+	unit.rangeMs = rangeMs
+	unit.offsetMs = offsetMs
+	unit.matchers = vs.LabelMatchers
+	return unit, true
+}
+
+// numberLiteralSide returns the number literal on one side of a binary
+// expression (peeling parens and unary minus), and which side it is on.
+func numberLiteralSide(b *parser.BinaryExpr) (float64, bool, bool) {
+	if v, ok := literalValue(b.LHS); ok {
+		return v, true, true
+	}
+	if v, ok := literalValue(b.RHS); ok {
+		return v, false, true
+	}
+	return 0, false, false
+}
+
+func literalValue(e parser.Expr) (float64, bool) {
+	neg := false
+	for {
+		switch n := e.(type) {
+		case *parser.ParenExpr:
+			e = n.Expr
+			continue
+		case *parser.StepInvariantExpr:
+			e = n.Expr
+			continue
+		case *parser.UnaryExpr:
+			if n.Op == parser.SUB {
+				neg = !neg
+			}
+			e = n.Expr
+			continue
+		case *parser.NumberLiteral:
+			if neg {
+				return -n.Val, true
+			}
+			return n.Val, true
+		default:
+			return 0, false
+		}
+	}
+}
+
+// classify builds the compile plan for a query: full when the root is a core
+// unit, hybrid when core units sit strictly below the root (including inside
+// fixed-resolution subqueries, computed on the subquery grid), none
+// otherwise.
+func classify(root parser.Expr, grid gridContext) (*transpilePlan, bool) {
+	if unit, ok := classifyCore(root, grid.stepMs); ok {
+		return &transpilePlan{
+			units: []*transpiledUnit{{core: *unit, name: syntheticName(0), grid: grid}},
+			grid:  grid,
+			full:  true,
+		}, true
+	}
+
+	plan := &transpilePlan{grid: grid}
+	rewritten := rewrite(root, grid, plan, false)
+	if len(plan.units) == 0 {
+		return nil, false
+	}
+	plan.rewritten = rewritten.String()
+	return plan, true
+}
+
+// rewrite walks top-down replacing maximal transpilable subtrees with synthetic
+// vector selectors. nameSensitive marks scopes where an ancestor's semantics
+// depend on __name__ (grouping or vector matching on it): synthetic series
+// carry a synthetic __name__, so substitution there would change results.
+// Fixed-resolution subqueries recurse with the subquery's own grid; scopes
+// whose evaluation grid is unknowable (@-pinned, default-resolution
+// subqueries) are not entered.
+func rewrite(node parser.Expr, grid gridContext, plan *transpilePlan, nameSensitive bool) parser.Expr {
+	if node == nil {
+		return nil
+	}
+
+	if !nameSensitive {
+		// Units whose output keeps the real __name__ (bare instant selectors)
+		// cannot be substituted: the synthetic name would replace it in the
+		// engine's output. They still compile as full plans.
+		if unit, ok := classifyCore(node, grid.stepMs); ok && !unit.keepsName() {
+			cu := &transpiledUnit{core: *unit, name: syntheticName(len(plan.units)), grid: grid}
+			plan.units = append(plan.units, cu)
+			return &parser.VectorSelector{
+				Name: cu.name,
+				LabelMatchers: []*labels.Matcher{
+					labels.MustNewMatcher(labels.MatchEqual, metricNameLabel, cu.name),
+				},
+				PosRange: node.PositionRange(),
+			}
+		}
+	}
+
+	switch n := node.(type) {
+	case *parser.ParenExpr:
+		n.Expr = rewrite(n.Expr, grid, plan, nameSensitive)
+	case *parser.UnaryExpr:
+		n.Expr = rewrite(n.Expr, grid, plan, nameSensitive)
+	case *parser.AggregateExpr:
+		sensitive := nameSensitive || groupingUsesName(n.Grouping)
+		n.Expr = rewrite(n.Expr, grid, plan, sensitive)
+		// n.Param is a scalar/string; nothing transpilable inside for our core.
+	case *parser.Call:
+		for i, arg := range n.Args {
+			n.Args[i] = rewrite(arg, grid, plan, nameSensitive)
+		}
+	case *parser.BinaryExpr:
+		sensitive := nameSensitive || vectorMatchingUsesName(n.VectorMatching)
+		n.LHS = rewrite(n.LHS, grid, plan, sensitive)
+		n.RHS = rewrite(n.RHS, grid, plan, sensitive)
+	case *parser.SubqueryExpr:
+		// The alert-smoothing idiom fn_over_time((expr)[R:S]) dominates real
+		// rule fleets; inner units evaluate on the subquery grid, and the
+		// engine does the smoothing over the synthetic series. Requires an
+		// explicit whole-second resolution (S == 0 needs the engine's
+		// default-interval function) and no @ pinning.
+		stepMs := n.Step.Milliseconds()
+		rangeMs := n.Range.Milliseconds()
+		offsetMs := n.OriginalOffset.Milliseconds()
+		if n.Timestamp == nil && n.StartOrEnd == 0 &&
+			n.RangeExpr == nil && n.StepExpr == nil && n.OriginalOffsetExpr == nil &&
+			stepMs > 0 && stepMs%1000 == 0 && rangeMs%1000 == 0 && offsetMs >= 0 {
+			inner := subqueryGrid(grid, rangeMs, stepMs, offsetMs)
+			n.Expr = rewrite(n.Expr, inner, plan, nameSensitive)
+		}
+	case *parser.StepInvariantExpr, *parser.MatrixSelector,
+		*parser.VectorSelector, *parser.NumberLiteral, *parser.StringLiteral:
+		// Leaves, or scopes substitution must not enter.
+	}
+	return node
+}
+
+func groupingUsesName(grouping []string) bool {
+	for _, g := range grouping {
+		if g == metricNameLabel {
+			return true
+		}
+	}
+	return false
+}
+
+func vectorMatchingUsesName(vm *parser.VectorMatching) bool {
+	if vm == nil {
+		return false
+	}
+	for _, l := range append(append([]string{}, vm.MatchingLabels...), vm.Include...) {
+		if l == metricNameLabel {
+			return true
+		}
+	}
+	// Default (all-labels) matching ignores __name__, and by()/ignoring()
+	// lists were checked above.
+	return false
+}
+
+// isSyntheticSelector reports whether matchers target a compiled unit.
+func isSyntheticSelector(matchers []*labels.Matcher) (string, bool) {
+	for _, m := range matchers {
+		if m.Name == metricNameLabel && m.Type == labels.MatchEqual && strings.HasPrefix(m.Value, syntheticNamePrefix) {
+			return m.Value, true
+		}
+	}
+	return "", false
+}

pkg/prometheus/clickhouseprometheusv2/transpiler_exec.go

@@ -0,0 +1,551 @@
+package clickhouseprometheusv2
+
+import (
+	"context"
+	"encoding/json"
+	"math"
+	"sort"
+	"time"
+
+	"github.com/SigNoz/signoz/pkg/errors"
+	"github.com/SigNoz/signoz/pkg/prometheus"
+	"github.com/prometheus/prometheus/model/labels"
+	promValue "github.com/prometheus/prometheus/model/value"
+	"github.com/prometheus/prometheus/promql"
+	"github.com/prometheus/prometheus/promql/parser"
+	"github.com/prometheus/prometheus/storage"
+	"golang.org/x/sync/errgroup"
+)
+
+// executor evaluates transpilable PromQL directly in ClickHouse. It falls
+// back (ok=false) when the query shape or the step does not qualify. The
+// timeSeries*ToGrid functions are assumed available: the supported
+// ClickHouse floor is 25.6 or later.
+type executor struct {
+	client *client
+	engine *prometheus.Engine
+	parser prometheus.Parser
+}
+
+// maxWindowBuckets caps range/step for the windowed *_over_time form.
+// Every grid slot combines that many bucket partials. The fleet's windows
+// sit well under the cap ([1m]..[17m] at 30-60s steps). Anything larger is
+// a long-range query whose step a dashboard scales up anyway. The engine
+// path serves the rest.
+const maxWindowBuckets = 64
+
+// TryExecuteRange transpiles and runs the query in ClickHouse when its
+// shape is in the allowlist. ok=false means "not transpilable" and carries
+// no error. The caller then runs the engine path.
+func (e *executor) TryExecuteRange(ctx context.Context, qs string, start, end time.Time, step time.Duration) (promql.Matrix, bool, error) {
+	expr, err := e.parser.ParseExpr(qs)
+	if err != nil {
+		// Let the engine path produce the (enhanced) parse error.
+		return nil, false, nil
+	}
+
+	plan, ok := classify(expr, queryGrid(start, end, step))
+	if !ok {
+		return nil, false, nil
+	}
+
+	// timeSeriesLastToGrid widens its window to max(window, step). We
+	// probed this: a sample aged (window, step] still fills the slot. The
+	// rate/delta family enforces the window strictly. The Last-style kinds
+	// used to fall back when window < step because of that widening. The
+	// window-sliver filter (see samplesConditions) makes the widening
+	// harmless there: samples exist only inside (t_k - window, t_k]
+	// slivers, so the widened window intersected with the data IS the
+	// lookback window. If a future ClickHouse stops widening, the
+	// unwidened window is the sliver too. Correct either way. A
+	// non-positive window still falls back: the sliver argument needs a
+	// real window to filter to.
+	//
+	// The windowed *_over_time form gates only the range >= step regime.
+	// It decomposes the window into whole step buckets (see windowedInner).
+	// That is exact only when the range is a multiple of the step. The
+	// per-slot slide costs range/step bucket combines; maxWindowBuckets
+	// bounds it, so a long-range short-step query cannot turn the slide
+	// into the bottleneck. range < step needs neither gate: the windows
+	// are disjoint slivers, aggregated one slot each, with no slide. Every
+	// miss falls back to the engine path, which is exact.
+	for _, unit := range plan.units {
+		stepMs := unit.grid.stepMs
+		if stepMs == 0 {
+			stepMs = 1000
+		}
+		switch {
+		case unit.core.kind == unitInstant || (unit.core.kind == unitOverTime && unit.core.overFn == "last"):
+			windowMs := unit.core.rangeMs
+			if unit.core.kind == unitInstant {
+				windowMs = e.client.lookbackMs
+			}
+			if windowMs <= 0 {
+				return nil, false, nil
+			}
+		case unit.core.kind == unitOverTime:
+			if unit.core.rangeMs < unit.grid.stepMs {
+				// Disjoint slivers: no divisibility or width requirement.
+				continue
+			}
+			if unit.core.rangeMs%stepMs != 0 || unit.core.rangeMs/stepMs > maxWindowBuckets {
+				return nil, false, nil
+			}
+		}
+	}
+
+	// Evaluate every unit concurrently on its own grid (the query grid, or a
+	// subquery grid); each is one series lookup plus one grid query.
+	results := make([][]transpiledSeries, len(plan.units))
+	eg, egCtx := errgroup.WithContext(ctx)
+	for i, unit := range plan.units {
+		eg.Go(func() error {
+			res, err := e.executeUnit(egCtx, &unit.core, unit.grid)
+			if err != nil {
+				return err
+			}
+			results[i] = res
+			return nil
+		})
+	}
+	if err := eg.Wait(); err != nil {
+		return nil, true, err
+	}
+
+	if plan.full {
+		g := plan.units[0].grid
+		return toMatrix(results[0], g.startMs, g.stepMs), true, nil
+	}
+
+	matrix, err := e.executeHybrid(ctx, plan, results)
+	if err != nil {
+		return nil, true, err
+	}
+	return matrix, true, nil
+}
+
+// queryGrid derives the top-level evaluation grid; step 0 is an instant
+// query: a single evaluation at end, whatever start was.
+func queryGrid(start, end time.Time, step time.Duration) gridContext {
+	startMs, endMs, stepMs := start.UnixMilli(), end.UnixMilli(), step.Milliseconds()
+	if stepMs == 0 {
+		startMs = endMs
+	}
+	return gridContext{startMs: startMs, endMs: endMs, stepMs: stepMs}
+}
+
+// transpiledSeries is one output series of a unit: projected labels and one
+// value pointer per grid point (nil = absent).
+type transpiledSeries struct {
+	lset   labels.Labels
+	values []*float64
+}
+
+// executeUnit runs one core unit on its grid: the series lookup, then the
+// single grid query, then the scalar-op pipeline.
+func (e *executor) executeUnit(ctx context.Context, unit *coreUnit, grid gridContext) ([]transpiledSeries, error) {
+	startMs, endMs, stepMs := grid.startMs, grid.endMs, grid.stepMs
+	windowMs := unit.rangeMs
+	if unit.kind == unitInstant {
+		windowMs = e.client.lookbackMs
+	}
+	dataStart := startMs - unit.offsetMs - windowMs
+	dataEnd := endMs - unit.offsetMs
+
+	seriesQuery, seriesArgs, err := buildSeriesQuery(dataStart, dataEnd, unit.matchers)
+	if err != nil {
+		return nil, err
+	}
+	lookup, err := e.client.selectSeries(ctx, seriesQuery, seriesArgs)
+	if err != nil {
+		return nil, err
+	}
+	if len(lookup.fingerprints) == 0 {
+		return nil, nil
+	}
+
+	query, args, err := buildUnitSQL(unit, lookup.metricNames, dataStart, dataEnd, startMs, endMs, stepMs, e.client.lookbackMs)
+	if err != nil {
+		return nil, err
+	}
+
+	rows, err := e.client.telemetryStore.ClickhouseDB().Query(e.client.withContext(ctx, "transpiledUnit"), query, args...)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+
+	// Name-dropping units keep __name__ in the SQL group key, so distinct
+	// metrics never merge server-side. The name comes off here. Two
+	// metrics can then share a labelset. The engine merges their samples
+	// into one series when they never overlap in time. It raises the
+	// duplicate-labelset error only when two samples land on the same
+	// evaluation timestamp. mergeSameLabelsetSeries reproduces exactly
+	// that.
+	stripName := !unit.hasAgg && !unit.keepsName()
+
+	// by (...) units return one plain column per grouped label. Everything
+	// else returns the single canonical JSON key (see groupKeyColumns).
+	keyNames := groupKeyColumns(unit)
+	keyVals := make([]string, max(len(keyNames), 1))
+	targets := make([]any, 0, len(keyVals)+1)
+	for i := range keyVals {
+		targets = append(targets, &keyVals[i])
+	}
+	var gridValues []*float64
+	targets = append(targets, &gridValues)
+
+	var out []transpiledSeries
+	for rows.Next() {
+		if err := rows.Scan(targets...); err != nil {
+			return nil, err
+		}
+		var lset labels.Labels
+		if keyNames != nil {
+			builder := labels.NewScratchBuilder(len(keyNames))
+			for i, name := range keyNames {
+				// An empty extracted value is the label being absent.
+				if keyVals[i] != "" {
+					builder.Add(name, keyVals[i])
+				}
+			}
+			builder.Sort()
+			lset = builder.Labels()
+		} else {
+			lset, err = labelsFromGroupKey(keyVals[0])
+			if err != nil {
+				return nil, err
+			}
+		}
+		if stripName {
+			lset = labels.NewBuilder(lset).Del(metricNameLabel).Labels()
+		}
+		values := make([]*float64, len(gridValues))
+		copy(values, gridValues)
+		applyScalarOps(unit.ops, values)
+		out = append(out, transpiledSeries{lset: lset, values: values})
+	}
+	if err := rows.Err(); err != nil {
+		return nil, err
+	}
+
+	if stripName {
+		if out, err = mergeSameLabelsetSeries(out); err != nil {
+			return nil, err
+		}
+	}
+	sort.Slice(out, func(i, j int) bool { return labels.Compare(out[i].lset, out[j].lset) < 0 })
+	return out, nil
+}
+
+// mergeSameLabelsetSeries combines series that a name strip left with
+// identical labelsets, slot by slot. The engine assembles its result matrix
+// by labelset. Post-strip twins whose points interleave in time are one
+// series to it. Two values on the same evaluation timestamp are its
+// duplicate-labelset error. v1 errors there too, so to silently pick one
+// value would be a divergence.
+func mergeSameLabelsetSeries(in []transpiledSeries) ([]transpiledSeries, error) {
+	index := make(map[uint64]int, len(in))
+	out := in[:0]
+	for _, s := range in {
+		hash := s.lset.Hash()
+		idx, ok := index[hash]
+		if ok && labels.Equal(out[idx].lset, s.lset) {
+			dst := out[idx].values
+			for k, v := range s.values {
+				if v == nil {
+					continue
+				}
+				if dst[k] != nil {
+					return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "vector cannot contain metrics with the same labelset")
+				}
+				dst[k] = v
+			}
+			continue
+		}
+		index[hash] = len(out)
+		out = append(out, s)
+	}
+	return out, nil
+}
+
+// labelsFromGroupKey parses the toJSONString'd sorted [key, value] pairs.
+func labelsFromGroupKey(gkey string) (labels.Labels, error) {
+	var pairs [][]string
+	if err := json.Unmarshal([]byte(gkey), &pairs); err != nil {
+		return labels.EmptyLabels(), errors.WrapInternalf(err, errors.CodeInternal, "malformed compiled group key %q", gkey)
+	}
+	builder := labels.NewScratchBuilder(len(pairs))
+	for _, p := range pairs {
+		if len(p) != 2 {
+			return labels.EmptyLabels(), errors.NewInternalf(errors.CodeInternal, "malformed compiled group key pair %q", gkey)
+		}
+		builder.Add(p[0], p[1])
+	}
+	builder.Sort()
+	return builder.Labels(), nil
+}
+
+// applyScalarOps applies the number-literal op pipeline in place, with the
+// same float64 arithmetic and comparison-filter semantics as the engine.
+func applyScalarOps(ops []scalarOp, values []*float64) {
+	for _, op := range ops {
+		for i, v := range values {
+			if v == nil {
+				continue
+			}
+			lhs, rhs := *v, op.scalar
+			if op.scalarOnLeft {
+				lhs, rhs = op.scalar, *v
+			}
+			switch op.op {
+			case parser.ADD:
+				res := lhs + rhs
+				values[i] = &res
+			case parser.SUB:
+				res := lhs - rhs
+				values[i] = &res
+			case parser.MUL:
+				res := lhs * rhs
+				values[i] = &res
+			case parser.DIV:
+				res := lhs / rhs
+				values[i] = &res
+			case parser.MOD:
+				res := math.Mod(lhs, rhs)
+				values[i] = &res
+			case parser.POW:
+				res := math.Pow(lhs, rhs)
+				values[i] = &res
+			default:
+				keep := compare(op.op, lhs, rhs)
+				switch {
+				case op.returnBool:
+					res := 0.0
+					if keep {
+						res = 1.0
+					}
+					values[i] = &res
+				case keep:
+					// Filter comparisons keep the vector-side value.
+					vec := *v
+					values[i] = &vec
+				default:
+					values[i] = nil
+				}
+			}
+		}
+	}
+}
+
+func compare(op parser.ItemType, lhs, rhs float64) bool {
+	switch op {
+	case parser.EQLC:
+		return lhs == rhs
+	case parser.NEQ:
+		return lhs != rhs
+	case parser.GTR:
+		return lhs > rhs
+	case parser.LSS:
+		return lhs < rhs
+	case parser.GTE:
+		return lhs >= rhs
+	case parser.LTE:
+		return lhs <= rhs
+	}
+	return false
+}
+
+// toMatrix converts a unit result to a promql matrix on the query grid.
+func toMatrix(series []transpiledSeries, startMs, stepMs int64) promql.Matrix {
+	matrix := make(promql.Matrix, 0, len(series))
+	for _, s := range series {
+		var floats []promql.FPoint
+		for i, v := range s.values {
+			if v == nil {
+				continue
+			}
+			floats = append(floats, promql.FPoint{T: startMs + int64(i)*stepMs, F: *v})
+		}
+		if len(floats) == 0 {
+			continue
+		}
+		matrix = append(matrix, promql.Series{Metric: s.lset, Floats: floats})
+	}
+	return matrix
+}
+
+// executeHybrid substitutes each unit's grids into the engine as synthetic
+// series. It evaluates the rewritten query over a storage that serves
+// synthetic selectors from memory and everything else from the live
+// querier. Absent grid points become stale markers, so the engine's
+// lookback cannot resurrect the previous grid point. Each unit's synthetic
+// samples sit on its own grid: the query grid, or the subquery grid for
+// units inside subqueries.
+func (e *executor) executeHybrid(ctx context.Context, plan *transpilePlan, results [][]transpiledSeries) (promql.Matrix, error) {
+	synthetic := make(map[string][]*series, len(plan.units))
+	staleMarker := math.Float64frombits(promValue.StaleNaN)
+
+	queryGrid := plan.grid
+
+	for i, unit := range plan.units {
+		g := unit.grid
+		gridLen := 1
+		if g.stepMs > 0 {
+			gridLen = int((g.endMs-g.startMs)/g.stepMs) + 1
+		}
+		list := make([]*series, 0, len(results[i]))
+		for _, cs := range results[i] {
+			builder := labels.NewBuilder(cs.lset)
+			builder.Set(metricNameLabel, unit.name)
+			s := &series{lset: builder.Labels()}
+			s.ts = make([]int64, 0, gridLen)
+			s.vs = make([]float64, 0, gridLen)
+			for idx := 0; idx < gridLen; idx++ {
+				t := g.startMs + int64(idx)*g.stepMs
+				var v float64
+				if idx < len(cs.values) && cs.values[idx] != nil {
+					v = *cs.values[idx]
+				} else {
+					v = staleMarker
+				}
+				s.ts = append(s.ts, t)
+				s.vs = append(s.vs, v)
+			}
+			list = append(list, s)
+		}
+		synthetic[unit.name] = list
+	}
+
+	hybrid := &hybridQueryable{client: e.client, synthetic: synthetic}
+
+	var qry promql.Query
+	var err error
+	if queryGrid.stepMs == 0 {
+		qry, err = e.engine.NewInstantQuery(ctx, hybrid, nil, plan.rewritten, time.UnixMilli(queryGrid.endMs))
+	} else {
+		qry, err = e.engine.NewRangeQuery(ctx, hybrid, nil, plan.rewritten, time.UnixMilli(queryGrid.startMs), time.UnixMilli(queryGrid.endMs), time.Duration(queryGrid.stepMs)*time.Millisecond)
+	}
+	if err != nil {
+		return nil, err
+	}
+	defer qry.Close()
+
+	res := qry.Exec(ctx)
+	if res.Err != nil {
+		return nil, res.Err
+	}
+
+	matrix, err := resultToMatrix(res)
+	if err != nil {
+		return nil, err
+	}
+
+	// Deep-copy before Close returns the result's slices to the engine pool,
+	// and drop the synthetic __name__ that filter comparisons preserve.
+	out := make(promql.Matrix, 0, len(matrix))
+	for _, s := range matrix {
+		lset := s.Metric
+		if name := lset.Get(metricNameLabel); len(name) >= len(syntheticNamePrefix) && name[:len(syntheticNamePrefix)] == syntheticNamePrefix {
+			builder := labels.NewBuilder(lset)
+			builder.Del(metricNameLabel)
+			lset = builder.Labels()
+		}
+		floats := make([]promql.FPoint, len(s.Floats))
+		copy(floats, s.Floats)
+		out = append(out, promql.Series{Metric: lset.Copy(), Floats: floats})
+	}
+	// The strip can leave twins: two units' outputs that only their
+	// synthetic names told apart (e.g. -metric_a or -metric_b, both {}
+	// once real names are dropped). The engine assembles its matrix by
+	// labelset. It merges such temporally-disjoint elements into one
+	// series. Reproduce that, with its duplicate error on same-timestamp
+	// overlap.
+	out, err = mergeMatrixByLabelset(out)
+	if err != nil {
+		return nil, err
+	}
+	sort.Slice(out, func(i, j int) bool { return labels.Compare(out[i].Metric, out[j].Metric) < 0 })
+	return out, nil
+}
+
+// mergeMatrixByLabelset merges series that share a labelset. It interleaves
+// their points in timestamp order. A timestamp present in both is the
+// engine's duplicate-labelset error.
+func mergeMatrixByLabelset(matrix promql.Matrix) (promql.Matrix, error) {
+	index := make(map[uint64]int, len(matrix))
+	out := matrix[:0]
+	for _, s := range matrix {
+		hash := s.Metric.Hash()
+		idx, ok := index[hash]
+		if ok && labels.Equal(out[idx].Metric, s.Metric) {
+			merged := make([]promql.FPoint, 0, len(out[idx].Floats)+len(s.Floats))
+			a, b := out[idx].Floats, s.Floats
+			for len(a) > 0 && len(b) > 0 {
+				switch {
+				case a[0].T < b[0].T:
+					merged, a = append(merged, a[0]), a[1:]
+				case b[0].T < a[0].T:
+					merged, b = append(merged, b[0]), b[1:]
+				default:
+					return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "vector cannot contain metrics with the same labelset")
+				}
+			}
+			out[idx].Floats = append(append(merged, a...), b...)
+			continue
+		}
+		index[hash] = len(out)
+		out = append(out, s)
+	}
+	return out, nil
+}
+
+func resultToMatrix(res *promql.Result) (promql.Matrix, error) {
+	switch v := res.Value.(type) {
+	case promql.Matrix:
+		return v, nil
+	case promql.Vector:
+		matrix := make(promql.Matrix, 0, len(v))
+		for _, s := range v {
+			matrix = append(matrix, promql.Series{Metric: s.Metric, Floats: []promql.FPoint{{T: s.T, F: s.F}}})
+		}
+		return matrix, nil
+	case promql.Scalar:
+		return promql.Matrix{{Metric: labels.EmptyLabels(), Floats: []promql.FPoint{{T: v.T, F: v.V}}}}, nil
+	default:
+		return nil, errors.NewInternalf(errors.CodeInternal, "unexpected hybrid result type %T", res.Value)
+	}
+}
+
+// hybridQueryable serves synthetic (compiled) selectors from memory and
+// everything else from the live storage.
+type hybridQueryable struct {
+	client    *client
+	synthetic map[string][]*series
+}
+
+func (h *hybridQueryable) Querier(mint, maxt int64) (storage.Querier, error) {
+	return &hybridQuerier{
+		querier:   querier{mint: mint, maxt: maxt, client: h.client},
+		synthetic: h.synthetic,
+	}, nil
+}
+
+type hybridQuerier struct {
+	querier
+	synthetic map[string][]*series
+}
+
+func (h *hybridQuerier) Select(ctx context.Context, sortSeries bool, hints *storage.SelectHints, matchers ...*labels.Matcher) storage.SeriesSet {
+	if name, ok := isSyntheticSelector(matchers); ok {
+		list := h.synthetic[name]
+		if sortSeries {
+			sorted := make([]*series, len(list))
+			copy(sorted, list)
+			sort.Slice(sorted, func(i, j int) bool { return labels.Compare(sorted[i].lset, sorted[j].lset) < 0 })
+			list = sorted
+		}
+		return newSeriesSet(list)
+	}
+	return h.querier.Select(ctx, sortSeries, hints, matchers...)
+}

pkg/prometheus/clickhouseprometheusv2/transpiler_sql.go

@@ -0,0 +1,415 @@
+package clickhouseprometheusv2
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
+	"github.com/huandu/go-sqlbuilder"
+)
+
+// experimental gate for the timeSeries*ToGrid aggregate functions; attached
+// as a SETTINGS clause so telemetrystore hooks cannot clobber it.
+const gridFunctionsSetting = "SETTINGS allow_experimental_ts_to_grid_aggregate_function = 1"
+
+var aggForEach = map[string]string{
+	"sum":   "sumForEach",
+	"min":   "minForEach",
+	"max":   "maxForEach",
+	"avg":   "avgForEach",
+	"count": "countForEach",
+}
+
+// buildUnitSQL renders the single ClickHouse statement that evaluates one
+// core unit over the [startMs, endMs] / stepMs grid. The inner level
+// computes per-series grids with a timeSeries*ToGrid aggregate, or with a
+// windowed aggregation for *_over_time. The outer level is the spatial
+// aggregation: -ForEach combinators grouped by the projected group key.
+//
+// The heavy level runs on the shards. The top-level FROM is the distributed
+// samples table. The group-key join partner is a subquery on the
+// shard-local time series table. So the shard rewrite executes the join and
+// the per-series aggregation next to the data. Fingerprint co-locality
+// makes this complete: samples and series shard on the same key. The
+// initiator only merges the per-series states and applies the spatial
+// -ForEach step. This is the same layout as the telemetrymetrics statement
+// builder. The windowed *_over_time form shares the frame but holds
+// per-bucket partials inside each series group (see windowedInner).
+//
+// The offset shifts the selector's data window. The grid indices map 1:1
+// onto the query grid: output ts = startMs + i*stepMs. Grid parameters
+// render as literals. They are aggregate-function parameters, not bindable
+// values.
+//
+// Statements nest builder-rendered SQL as text. So the returned args must
+// follow the position of each fragment in the final statement: ClickHouse
+// binds ? placeholders by position. A JOIN renders before WHERE, so a
+// joined subquery's args come before the outer query's condition args.
+//
+// Row shape: the group-key columns (see groupKeyColumns), then grid
+// Array(Nullable(Float64)). A NULL grid point is an absent point, the
+// engine's "no value here". The -ForEach combinators preserve it: an index
+// where every series is NULL aggregates to NULL, and countForEach's 0 maps
+// back to NULL.
+func buildUnitSQL(unit *coreUnit, metricNames []string, dataStart, dataEnd int64, startMs, endMs, stepMs, lookbackMs int64) (string, []any, error) {
+	selStart := startMs - unit.offsetMs
+	selEnd := endMs - unit.offsetMs
+	stepSec := stepMs / 1000
+	if stepSec == 0 {
+		// Instant query: start == end, so the grid has one point for any
+		// positive step.
+		stepSec = 1
+	}
+	windowMs := unit.rangeMs
+	if unit.kind == unitInstant {
+		windowMs = lookbackMs
+	}
+	windowSec := windowMs / 1000
+
+	adjustedTsStartU, _, _, localTsTable := metricstelemetryschema.WhichTSTableToUse(uint64(dataStart), uint64(dataEnd), false, nil)
+	adjustedTsStart := int64(adjustedTsStartU)
+	keyNames := groupKeyColumns(unit)
+
+	// seriesSub computes fingerprint -> group key columns. It reads the
+	// local series table when it rides inside the shard-rewritten samples
+	// query, and the distributed one when it joins at the initiator
+	// (windowed form).
+	seriesSub := func(table string) (string, []any, error) {
+		sub := sqlbuilder.NewSelectBuilder()
+		selects := []string{"fingerprint"}
+		if keyNames == nil {
+			selects = append(selects, groupKeyExpr(unit)+" AS gkey")
+		} else {
+			// by (...) grouping extracts exactly the listed labels as plain
+			// columns: no reason to build, sort and stringify every label
+			// pair per row when the projection is a known short list and
+			// the label names live in Go anyway.
+			for i, name := range keyNames {
+				selects = append(selects, fmt.Sprintf("JSONExtractString(labels, %s) AS g%d", sub.Var(name), i))
+			}
+		}
+		sub.Select(selects...)
+		sub.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, table))
+		if err := applySeriesConditions(sub, adjustedTsStart, dataEnd, unit.matchers); err != nil {
+			return "", nil, err
+		}
+		sub.GroupBy(append([]string{"fingerprint"}, keyColumnAliases(keyNames)...)...)
+		q, args := sub.BuildWithFlavor(sqlbuilder.ClickHouse)
+		return q, args, nil
+	}
+
+	// samplesConditions adds the samples-side WHERE. The group-key join
+	// restricts to the matched series; no fingerprint condition is added
+	// here.
+	samplesConditions := func(sb *sqlbuilder.SelectBuilder, excludeStale bool) {
+		switch len(metricNames) {
+		case 0:
+			// No name constraint derivable; correct but unable to use the
+			// metric_name primary-key prefix.
+		case 1:
+			sb.Where(sb.EQ("metric_name", metricNames[0]))
+		default:
+			sb.Where(sb.In("metric_name", sqlbuilder.List(metricNames)))
+		}
+		// temporality precedes metric_name in the samples primary key; the
+		// fingerprints already come from these temporalities, so this only
+		// helps granule pruning.
+		sb.Where("temporality IN ['Cumulative', 'Unspecified']")
+		// When the window is narrower than the step, the grid windows
+		// (t_k − window, t_k] cover only window/step of the timeline. A
+		// sample in a gap belongs to no window. It cannot move any grid
+		// point, but the grid aggregate buffers every row it is fed. This
+		// predicate keeps only the in-window rows. It cut a 36k-series
+		// one-week rate from 74s/28GiB to 16s/4.3GiB on fleet data: the
+		// read stays the same, and the aggregate input shrinks by the
+		// coverage ratio. The lattice anchors at selStart, because the end
+		// can sit off-lattice on unaligned grids. positiveModulo is
+		// necessary because samples above selStart make the dividend
+		// negative. The upper bound tightens to the last grid point: rows
+		// past it are equally windowless. When window >= step, the windows
+		// tile the timeline, and the plain bounds stay.
+		sliver := stepMs > 0 && windowMs > 0 && windowMs < stepMs
+		upper := selEnd
+		if sliver {
+			upper = selStart + (selEnd-selStart)/stepMs*stepMs
+		}
+		// Left-open window: a sample exactly at the window's lower boundary
+		// is never used (range selectors and lookback are both left-open).
+		sb.Where(sb.GT("unix_milli", selStart-windowMs), sb.LTE("unix_milli", upper))
+		if sliver {
+			sb.Where(fmt.Sprintf("positiveModulo(%s - unix_milli, %s) < %s",
+				sb.Var(selStart), sb.Var(stepMs), sb.Var(windowMs)))
+		}
+		if excludeStale {
+			// PromQL excludes stale markers from range vectors. Instant
+			// selectors need the stale rows for shadowing instead.
+			sb.Where("bitAnd(flags, 1) = 0")
+		}
+	}
+
+	keyCols := keyColumnAliases(keyNames)
+
+	// joinedInner builds the shard-side SELECT for the single-pass kinds:
+	// grid expression per (fingerprint, group key), group-key join against
+	// the local series table.
+	joinedInner := func(gridExpr string, excludeStale bool) (string, []any, error) {
+		seriesSQL, seriesArgs, err := seriesSub(localTsTable)
+		if err != nil {
+			return "", nil, err
+		}
+		sb := sqlbuilder.NewSelectBuilder()
+		selects := make([]string, 0, len(keyCols)+1)
+		// A fingerprint is the hash of one labelset, so every group-key
+		// column is functionally dependent on it: any() is exact, and
+		// grouping by the fingerprint alone spares hashing the joined
+		// string per sample row — measured -10-13% on a 1.9B-row rate.
+		for _, col := range keyCols {
+			selects = append(selects, fmt.Sprintf("any(series.%s) AS %s", col, col))
+		}
+		sb.Select(append(selects, gridExpr+" AS grid")...)
+		sb.From(fmt.Sprintf("%s.%s AS points", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4TableName))
+		sb.JoinWithOption(sqlbuilder.InnerJoin, fmt.Sprintf("(%s) AS series", seriesSQL), "points.fingerprint = series.fingerprint")
+		samplesConditions(sb, excludeStale)
+		sb.GroupBy("points.fingerprint")
+		q, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
+		// The join text renders before WHERE: its args come first.
+		return q, append(seriesArgs, args...), nil
+	}
+
+	var inner string
+	var innerArgs []any
+	var err error
+	switch unit.kind {
+	case unitInstant:
+		// Instant selection with stale shadowing: the grid value is the last
+		// non-stale sample in (t-lookback, t], absent when the overall last
+		// sample in that window is a stale marker (verified semantics: the
+		// -If combinator applies to the grid aggregates, and NULL comparisons
+		// make a stale-latest point absent).
+		gridParams := fmt.Sprintf("(fromUnixTimestamp64Milli(%d), fromUnixTimestamp64Milli(%d), %d, %d)", selStart, selEnd, stepSec, windowSec)
+		gridExpr := fmt.Sprintf(
+			"arrayMap((tall, tok, vok) -> if(tall IS NULL OR tok IS NULL OR tall != tok, NULL, vok), timeSeriesLastToGrid%s(fromUnixTimestamp64Milli(unix_milli), toFloat64(unix_milli)), timeSeriesLastToGridIf%s(fromUnixTimestamp64Milli(unix_milli), toFloat64(unix_milli), bitAnd(flags, 1) = 0), timeSeriesLastToGridIf%s(fromUnixTimestamp64Milli(unix_milli), value, bitAnd(flags, 1) = 0))",
+			gridParams, gridParams, gridParams,
+		)
+		inner, innerArgs, err = joinedInner(gridExpr, false)
+	case unitOverTime:
+		if unit.overFn == "last" {
+			// last_over_time == last non-stale sample in the window: the
+			// stale rows are already excluded in WHERE.
+			gridExpr := fmt.Sprintf(
+				"timeSeriesLastToGrid(fromUnixTimestamp64Milli(%d), fromUnixTimestamp64Milli(%d), %d, %d)(fromUnixTimestamp64Milli(unix_milli), value)",
+				selStart, selEnd, stepSec, windowSec,
+			)
+			inner, innerArgs, err = joinedInner(gridExpr, true)
+			break
+		}
+		inner, innerArgs, err = windowedInner(unit, samplesConditions, seriesSub, keyCols, localTsTable, selStart, selEnd, stepMs, windowMs)
+	default: // unitRange
+		gridExpr := fmt.Sprintf(
+			"%s(fromUnixTimestamp64Milli(%d), fromUnixTimestamp64Milli(%d), %d, %d)(fromUnixTimestamp64Milli(unix_milli), value)",
+			gridFunction[unit.fn], selStart, selEnd, stepSec, windowSec,
+		)
+		if unit.fn == fnIncrease {
+			// increase == rate * range-seconds, exactly: extrapolatedRate
+			// divides by the range only when isRate.
+			gridExpr = fmt.Sprintf("arrayMap(x -> x * %d, %s)", windowSec, gridExpr)
+		}
+		inner, innerArgs, err = joinedInner(gridExpr, true)
+	}
+	if err != nil {
+		return "", nil, err
+	}
+
+	spatial := "maxForEach(grid)"
+	switch {
+	case !unit.hasAgg:
+		// Per-series output: one row per (labels-minus-__name__) group.
+		// Distinct fingerprints can collapse onto the same projected label
+		// set only via a regex __name__ selector over metrics with identical
+		// other labels; maxForEach is a deterministic NULL-skipping merge and
+		// the identity for the overwhelmingly common one-fingerprint group.
+	case unit.aggOp.String() == "count":
+		// count over an all-absent index is an absent point, not 0.
+		spatial = "arrayMap(c -> if(c = 0, NULL, toFloat64(c)), countForEach(grid))"
+	default:
+		spatial = fmt.Sprintf("%s(grid)", aggForEach[unit.aggOp.String()])
+	}
+
+	keyList := strings.Join(keyCols, ", ")
+	query := fmt.Sprintf("SELECT %s, %s AS grid FROM (%s) GROUP BY %s %s", keyList, spatial, inner, keyList, gridFunctionsSetting)
+	return query, innerArgs, nil
+}
+
+// groupKeyColumns returns the label names to extract as plain group-key
+// columns, or nil when the unit needs the canonical JSON key instead. Only
+// by (...) grouping qualifies: its projection is a known short list, so
+// extracting each label directly beats building, sorting and stringifying
+// every label pair per row. without and no-aggregation project a label SET
+// that varies per series — there the sorted-JSON key is load-bearing: the
+// sort is what makes two fingerprints with different stored JSON key order
+// land in one group, and the string carries the labels back out.
+func groupKeyColumns(unit *coreUnit) []string {
+	if unit.hasAgg && unit.by && len(unit.grouping) > 0 {
+		return unit.grouping
+	}
+	return nil
+}
+
+// keyColumnAliases names the group-key columns in every SELECT level: g0..gN
+// for direct extraction, the single canonical gkey otherwise.
+func keyColumnAliases(keyNames []string) []string {
+	if keyNames == nil {
+		return []string{"gkey"}
+	}
+	cols := make([]string, len(keyNames))
+	for i := range keyNames {
+		cols[i] = fmt.Sprintf("g%d", i)
+	}
+	return cols
+}
+
+// windowedInner builds the avg/min/max/sum/count _over_time form without
+// fanning samples out. It runs only when the range is a whole multiple of
+// the step (see the transpile gate), because then the window
+// (t_k - range, t_k] is exactly the union of W = range/step step buckets —
+// both are left-open on the same boundaries — so bucket membership fully
+// determines window membership. Fanning each sample into all W windows it
+// covers (ARRAY JOIN) multiplies rows by W, which at long ranges over short
+// steps is a row explosion measured in billions.
+//
+// The bucketing itself is the -Resample combinator: one group per (series,
+// group key) whose state is a fixed array of per-bucket aggregates, updated
+// in place per sample. Grouping by (series, bucket) instead — measured on a
+// 100k-series x 371-bucket workload — creates a 37M-entry hash aggregation
+// whose per-thread partial tables scale memory WITH max_threads (12 -> 48
+// GiB from 2 to 8 threads, dead at 16) and ships one row per group to the
+// initiator; the Resample form carries the same numbers in 100k compact
+// array states, like every other unit kind.
+//
+// The wrapper level slides the window: slot k combines buckets k..k+W-1 by
+// direct aggregation over at most W partials — no prefix-sum tricks, so no
+// large-minus-large cancellation against the engine's directly-summed
+// windows. A slot with zero window count is absent, which also keeps
+// min/max honest: their slices filter on the bucket counts, so an empty
+// bucket's zero-fill can never be mistaken for a value (a real sample can
+// legitimately be 0 or +Inf).
+func windowedInner(unit *coreUnit, samplesConditions func(*sqlbuilder.SelectBuilder, bool), seriesSub func(string) (string, []any, error), keyCols []string, localSeriesTable string, selStart, selEnd, stepMs, windowMs int64) (string, []any, error) {
+	effStepMs := stepMs
+	if effStepMs == 0 {
+		effStepMs = 1000
+	}
+	lastIdx := (selEnd - selStart) / effStepMs
+	gridLen := lastIdx + 1
+	w := windowMs / effStepMs
+	bucketLen := gridLen + w
+
+	// A window narrower than the step makes the windows (t_k - range, t_k]
+	// pairwise disjoint. There is nothing to slide. Each slot reads exactly
+	// its own window's aggregate. This is exact ONLY over sliver-filtered
+	// rows (samplesConditions adds the window<step predicate): the index
+	// below assigns every gap sample to the window above it, and the
+	// filter removes those samples. This needs a real step. Instant
+	// queries carry no sliver filter, so they keep the tiled form and its
+	// gates.
+	disjoint := stepMs > 0 && windowMs < stepMs
+	if disjoint {
+		w = 1
+		bucketLen = gridLen
+	}
+
+	seriesSQL, seriesArgs, err := seriesSub(localSeriesTable)
+	if err != nil {
+		return "", nil, err
+	}
+
+	// Bucket index, shifted so the earliest in-window sample lands at 0:
+	// jj = ceil((ts - selStart)/step) + W - 1, folded into one intDiv.
+	// Slot k's window is then buckets jj in [k, k+W-1]. In the disjoint
+	// form, the same ceil lands each in-window sample directly on its slot
+	// (W = 1). The numerator stays positive: the fetch floor is
+	// selStart - range > selStart - step.
+	jjShift := windowMs
+	if disjoint {
+		jjShift = effStepMs
+	}
+	jj := fmt.Sprintf("intDiv(unix_milli - %d + %d - 1, %d)", selStart, jjShift, effStepMs)
+	buckets := sqlbuilder.NewSelectBuilder()
+	selects := make([]string, 0, len(keyCols)+2)
+	// any() over the group key: exact because the key is functionally
+	// dependent on the fingerprint (see joinedInner).
+	for _, col := range keyCols {
+		selects = append(selects, fmt.Sprintf("any(series.%s) AS %s", col, col))
+	}
+	selects = append(selects, fmt.Sprintf("countResample(0, %d, 1)(value, %s) AS cnts", bucketLen, jj))
+	if unit.overFn != "count" {
+		selects = append(selects, fmt.Sprintf("%sResample(0, %d, 1)(value, %s) AS vals", map[string]string{
+			"avg": "sum",
+			"sum": "sum",
+			"min": "min",
+			"max": "max",
+		}[unit.overFn], bucketLen, jj))
+	}
+	buckets.Select(selects...)
+	buckets.From(fmt.Sprintf("%s.%s AS points", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4TableName))
+	buckets.JoinWithOption(sqlbuilder.InnerJoin, fmt.Sprintf("(%s) AS series", seriesSQL), "points.fingerprint = series.fingerprint")
+	samplesConditions(buckets, true)
+	buckets.GroupBy("points.fingerprint")
+	bucketsSQL, bucketsArgs := buckets.BuildWithFlavor(sqlbuilder.ClickHouse)
+
+	windowCnt := fmt.Sprintf("arraySum(arraySlice(cnts, k + 1, %d))", w)
+	var slot string
+	switch unit.overFn {
+	case "count":
+		slot = fmt.Sprintf("if(%s = 0, NULL, toFloat64(%s))", windowCnt, windowCnt)
+	case "sum":
+		slot = fmt.Sprintf("if(%s = 0, NULL, arraySum(arraySlice(vals, k + 1, %d)))", windowCnt, w)
+	case "avg":
+		slot = fmt.Sprintf("if(%s = 0, NULL, arraySum(arraySlice(vals, k + 1, %d)) / %s)", windowCnt, w, windowCnt)
+	case "min":
+		slot = fmt.Sprintf("if(%s = 0, NULL, arrayMin(arrayFilter((v, c) -> c > 0, arraySlice(vals, k + 1, %d), arraySlice(cnts, k + 1, %d))))", windowCnt, w, w)
+	case "max":
+		slot = fmt.Sprintf("if(%s = 0, NULL, arrayMax(arrayFilter((v, c) -> c > 0, arraySlice(vals, k + 1, %d), arraySlice(cnts, k + 1, %d))))", windowCnt, w, w)
+	}
+
+	keyList := strings.Join(keyCols, ", ")
+	inner := fmt.Sprintf(
+		"SELECT %s, arrayMap(k -> %s, range(toUInt64(%d))) AS grid FROM (%s)",
+		keyList, slot, gridLen, bucketsSQL,
+	)
+	return inner, append(seriesArgs, bucketsArgs...), nil
+}
+
+// groupKeyExpr renders the canonical JSON group key for the units whose
+// projected label SET varies per series (see groupKeyColumns): the sorted
+// [key, value] pairs of the projected labels, JSON-encoded.
+//   - by () with no labels: one constant group;
+//   - without (a, b): keep everything except the listed labels and __name__;
+//   - no aggregation: keep everything including __name__ — even when the
+//     unit drops the name from its OUTPUT, the key must keep it so distinct
+//     metrics never merge in SQL; executeUnit strips the name afterwards and
+//     turns a post-strip collision into the engine's duplicate-labelset
+//     error instead of a silently invented merge.
+func groupKeyExpr(unit *coreUnit) string {
+	// An empty label value means "label absent" in Prometheus; the stored
+	// labels JSON can carry empty attribute values, which must not become
+	// output labels or group keys.
+	pairs := "arraySort(JSONExtractKeysAndValues(labels, 'String'))"
+	if !unit.hasAgg {
+		return fmt.Sprintf("toJSONString(arrayFilter(p -> p.2 != '', %s))", pairs)
+	}
+	if unit.by {
+		// Non-empty by (...) never reaches here; groupKeyColumns extracts
+		// those labels as plain columns instead.
+		return "'[]'"
+	}
+	excluded := append([]string{metricNameLabel}, unit.grouping...)
+	return fmt.Sprintf("toJSONString(arrayFilter(p -> p.2 != '' AND p.1 NOT IN (%s), %s))", quotedList(excluded), pairs)
+}
+
+func quotedList(items []string) string {
+	quoted := make([]string, len(items))
+	for i, s := range items {
+		quoted[i] = "'" + strings.ReplaceAll(s, "'", "\\'") + "'"
+	}
+	return strings.Join(quoted, ", ")
+}

pkg/prometheus/clickhouseprometheusv2/transpiler_test.go

@@ -0,0 +1,698 @@
+package clickhouseprometheusv2
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"github.com/DATA-DOG/go-sqlmock"
+	cmock "github.com/SigNoz/clickhouse-go-mock"
+	"github.com/SigNoz/signoz/pkg/errors"
+	"github.com/SigNoz/signoz/pkg/factory"
+	"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
+	"github.com/SigNoz/signoz/pkg/prometheus"
+	"github.com/SigNoz/signoz/pkg/telemetrystore"
+	"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
+	"github.com/prometheus/prometheus/model/labels"
+	"github.com/prometheus/prometheus/promql"
+	"github.com/prometheus/prometheus/promql/parser"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func newTestClient(t *testing.T) (*client, *telemetrystoretest.Provider) {
+	t.Helper()
+	store := telemetrystoretest.New(telemetrystore.Config{Provider: "clickhouse"}, sqlmock.QueryMatcherRegexp)
+	settings := factory.NewScopedProviderSettings(instrumentationtest.New().ToProviderSettings(), "clickhouseprometheusv2_test")
+	return newClient(settings, store, prometheus.Config{}), store
+}
+
+var seriesCols = []cmock.ColumnType{
+	{Name: "fingerprint", Type: "UInt64"},
+	{Name: "labels", Type: "String"},
+}
+
+func parse(t *testing.T, q string) parser.Expr {
+	t.Helper()
+	expr, err := parser.NewParser(parser.Options{}).ParseExpr(q)
+	require.NoError(t, err)
+	return expr
+}
+
+func TestClassifyFullShapes(t *testing.T) {
+	tests := []struct {
+		name  string
+		query string
+		check func(t *testing.T, u *coreUnit)
+	}{
+		{
+			name:  "sum by rate",
+			query: `sum by (pod) (rate(http_requests_total{job="api"}[5m]))`,
+			check: func(t *testing.T, u *coreUnit) {
+				assert.Equal(t, fnRate, u.fn)
+				assert.Equal(t, int64(300_000), u.rangeMs)
+				assert.True(t, u.hasAgg)
+				assert.True(t, u.by)
+				assert.Equal(t, []string{"pod"}, u.grouping)
+			},
+		},
+		{
+			name:  "bare increase with offset",
+			query: `increase(errors_total[10m] offset 30m)`,
+			check: func(t *testing.T, u *coreUnit) {
+				assert.Equal(t, fnIncrease, u.fn)
+				assert.Equal(t, int64(1_800_000), u.offsetMs)
+				assert.False(t, u.hasAgg)
+			},
+		},
+		{
+			name:  "avg without over delta",
+			query: `avg without (instance) (delta(gauge_metric[15m]))`,
+			check: func(t *testing.T, u *coreUnit) {
+				assert.Equal(t, fnDelta, u.fn)
+				assert.True(t, u.hasAgg)
+				assert.False(t, u.by)
+			},
+		},
+		{
+			name:  "scalar pipeline with comparison",
+			query: `sum(rate(x[5m])) * 100 > 5`,
+			check: func(t *testing.T, u *coreUnit) {
+				require.Len(t, u.ops, 2)
+				assert.Equal(t, parser.ItemType(parser.MUL), u.ops[0].op)
+				assert.Equal(t, 100.0, u.ops[0].scalar)
+				assert.Equal(t, parser.ItemType(parser.GTR), u.ops[1].op)
+			},
+		},
+		{
+			name:  "scalar on left with unary minus",
+			query: `-1 * sum(rate(x[5m]))`,
+			check: func(t *testing.T, u *coreUnit) {
+				require.Len(t, u.ops, 1)
+				assert.True(t, u.ops[0].scalarOnLeft)
+				assert.Equal(t, -1.0, u.ops[0].scalar)
+			},
+		},
+		{
+			name:  "bool comparison",
+			query: `sum(rate(x[5m])) >= bool 0.5`,
+			check: func(t *testing.T, u *coreUnit) {
+				require.Len(t, u.ops, 1)
+				assert.True(t, u.ops[0].returnBool)
+			},
+		},
+		{
+			name:  "irate utf8 name",
+			query: `sum by ("k8s.pod.name") (irate({"k8s.container.cpu.time"}[2m]))`,
+			check: func(t *testing.T, u *coreUnit) {
+				assert.Equal(t, fnIRate, u.fn)
+				assert.Equal(t, []string{"k8s.pod.name"}, u.grouping)
+			},
+		},
+		{
+			name:  "bare instant selector keeps name",
+			query: `up{job="api"}`,
+			check: func(t *testing.T, u *coreUnit) {
+				assert.Equal(t, unitInstant, u.kind)
+				assert.True(t, u.keepsName())
+			},
+		},
+		{
+			name:  "gauge aggregation",
+			query: `sum by (pod) (container_memory offset 5m)`,
+			check: func(t *testing.T, u *coreUnit) {
+				assert.Equal(t, unitInstant, u.kind)
+				assert.Equal(t, int64(300_000), u.offsetMs)
+				assert.True(t, u.hasAgg)
+				assert.False(t, u.keepsName())
+			},
+		},
+		{
+			name:  "gauge comparison keeps name",
+			query: `container_memory > 100`,
+			check: func(t *testing.T, u *coreUnit) {
+				assert.Equal(t, unitInstant, u.kind)
+				assert.True(t, u.keepsName())
+			},
+		},
+		{
+			name:  "gauge arithmetic drops name",
+			query: `container_memory / 1024`,
+			check: func(t *testing.T, u *coreUnit) {
+				assert.Equal(t, unitInstant, u.kind)
+				assert.False(t, u.keepsName())
+			},
+		},
+		{
+			name:  "avg_over_time",
+			query: `max by (node) (avg_over_time(load1[10m]))`,
+			check: func(t *testing.T, u *coreUnit) {
+				assert.Equal(t, unitOverTime, u.kind)
+				assert.Equal(t, "avg", u.overFn)
+				assert.Equal(t, int64(600_000), u.rangeMs)
+			},
+		},
+		{
+			name:  "last_over_time keeps name",
+			query: `last_over_time(load1[10m])`,
+			check: func(t *testing.T, u *coreUnit) {
+				assert.Equal(t, unitOverTime, u.kind)
+				assert.Equal(t, "last", u.overFn)
+				assert.True(t, u.keepsName())
+			},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			plan, ok := classify(parse(t, tt.query), testGrid(60_000))
+			require.True(t, ok, "expected transpilable")
+			require.True(t, plan.full, "expected full compilation")
+			require.Len(t, plan.units, 1)
+			tt.check(t, &plan.units[0].core)
+		})
+	}
+}
+
+func TestClassifyFallbackShapes(t *testing.T) {
+	queries := []struct {
+		name  string
+		query string
+		step  int64
+	}{
+		{"default-resolution subquery", `max_over_time(rate(x[5m])[30m:])`, 60_000},
+		{"at modifier", `sum(rate(x[5m] @ 1609746000))`, 60_000},
+		{"at modifier on gauge", `sum(container_memory @ 1609746000)`, 60_000},
+		{"sub-second step", `sum(rate(x[5m]))`, 500},
+		{"sub-second range", `sum(rate(x[1500ms]))`, 60_000},
+		{"by __name__ full", `sum by (__name__) (rate({__name__=~"a|b"}[5m]))`, 60_000},
+		{"quantile_over_time unsupported", `quantile_over_time(0.9, load1[10m])`, 60_000},
+		// Duration expressions resolve into the selectors' static fields only
+		// at evaluation time; classification reads those fields as zero, so
+		// transpiling would silently use the wrong offset (caught by the
+		// conformance corpus' duration_expression.test cases). Offset
+		// expressions parse without the experimental-parser flag, so they do
+		// reach the transpiler; range-position expressions are rejected at
+		// parse (the RangeExpr/StepExpr guards are defense-in-depth).
+		{"duration expression offset on instant", `x offset step()`, 60_000},
+		{"duration expression offset arithmetic", `x offset -step()*2`, 60_000},
+		{"duration expression offset on range", `sum(rate(x[5m] offset max(3s, step())))`, 60_000},
+		{"duration expression subquery step", `max_over_time(rate(x[5m])[30m:step()])`, 60_000},
+	}
+	for _, tt := range queries {
+		t.Run(tt.name, func(t *testing.T) {
+			_, ok := classify(parse(t, tt.query), testGrid(tt.step))
+			assert.False(t, ok, "expected fallback for %s", tt.query)
+		})
+	}
+}
+
+func TestClassifyHybridShapes(t *testing.T) {
+	tests := []struct {
+		name          string
+		query         string
+		wantUnits     int
+		wantRewritten string
+	}{
+		{
+			name:          "histogram quantile",
+			query:         `histogram_quantile(0.95, sum by (le) (rate(http_bucket[5m])))`,
+			wantUnits:     1,
+			wantRewritten: `histogram_quantile(0.95, __signoz_transpiled_0__)`,
+		},
+		{
+			name:          "topk over compiled",
+			query:         `topk(5, sum by (pod) (rate(x[5m])))`,
+			wantUnits:     1,
+			wantRewritten: `topk(5, __signoz_transpiled_0__)`,
+		},
+		{
+			name:          "ratio of compiled units",
+			query:         `sum(rate(a[5m])) / sum(rate(b[5m]))`,
+			wantUnits:     2,
+			wantRewritten: `__signoz_transpiled_0__ / __signoz_transpiled_1__`,
+		},
+		{
+			name:          "or vector zero",
+			query:         `sum(rate(a[5m])) or vector(0)`,
+			wantUnits:     1,
+			wantRewritten: `__signoz_transpiled_0__ or vector(0)`,
+		},
+		{
+			name:          "quantile agg over compiled rate",
+			query:         `quantile(0.9, rate(x[5m]))`,
+			wantUnits:     1,
+			wantRewritten: `quantile(0.9, __signoz_transpiled_0__)`,
+		},
+		{
+			name:          "non-literal scalar side stays engine-side",
+			query:         `sum(rate(x[5m])) * scalar(y)`,
+			wantUnits:     1,
+			wantRewritten: `__signoz_transpiled_0__ * scalar(y)`,
+		},
+		{
+			name:          "compiled mixed with raw selector",
+			query:         `sum by (pod) (rate(a[5m])) / on (pod) group_left () b`,
+			wantUnits:     1,
+			wantRewritten: `__signoz_transpiled_0__ / on (pod) group_left () b`,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			plan, ok := classify(parse(t, tt.query), testGrid(60_000))
+			require.True(t, ok)
+			assert.False(t, plan.full)
+			assert.Len(t, plan.units, tt.wantUnits)
+			assert.Equal(t, tt.wantRewritten, plan.rewritten)
+		})
+	}
+}
+
+func TestClassifyHybridGuards(t *testing.T) {
+	t.Run("no substitution under on(__name__)", func(t *testing.T) {
+		plan, ok := classify(parse(t, `sum(rate(a[5m])) * on (__name__) b`), testGrid(60_000))
+		_ = plan
+		assert.False(t, ok, "matching on __name__ must not see synthetic names")
+	})
+	t.Run("no substitution inside @-pinned subquery", func(t *testing.T) {
+		_, ok := classify(parse(t, `max_over_time(rate(x[5m])[30m:1m] @ 1609746000)`), testGrid(60_000))
+		assert.False(t, ok)
+	})
+}
+
+// The alert-smoothing idiom: units inside a fixed-resolution subquery
+// evaluate on the subquery grid — epoch-aligned multiples of the resolution,
+// starting strictly after (outer start - range), exactly as the engine
+// derives it.
+func TestClassifySubqueryUnits(t *testing.T) {
+	grid := gridContext{startMs: 1_700_000_030_000, endMs: 1_700_007_200_000, stepMs: 60_000}
+
+	plan, ok := classify(parse(t, `min_over_time((sum by (ns) (increase(x[5m])))[10m:5m]) > 0`), grid)
+	require.True(t, ok)
+	require.False(t, plan.full)
+	require.Len(t, plan.units, 1)
+	assert.Equal(t, `min_over_time(__signoz_transpiled_0__[10m:5m]) > 0`, plan.rewritten)
+
+	unit := plan.units[0]
+	// lower bound = outer start - range = 1_699_999_430_000; first multiple
+	// of 300_000 strictly greater is 1_699_999_500_000.
+	assert.Equal(t, int64(1_699_999_500_000), unit.grid.startMs)
+	assert.Equal(t, grid.endMs, unit.grid.endMs)
+	assert.Equal(t, int64(300_000), unit.grid.stepMs)
+	assert.Equal(t, fnIncrease, unit.core.fn)
+
+	t.Run("subquery offset shifts the grid", func(t *testing.T) {
+		plan, ok := classify(parse(t, `max_over_time((sum(rate(x[5m])))[10m:5m] offset 30m)`), grid)
+		require.True(t, ok)
+		require.Len(t, plan.units, 1)
+		// lower = start - offset - range = 1_699_997_630_000 -> first
+		// multiple of 300_000 above = 1_699_997_700_000; end shifts too.
+		assert.Equal(t, int64(1_699_997_700_000), plan.units[0].grid.startMs)
+		assert.Equal(t, grid.endMs-1_800_000, plan.units[0].grid.endMs)
+	})
+
+	t.Run("mollusk ratio-inside-subquery idiom", func(t *testing.T) {
+		q := `min_over_time(((sum by (a) (rate(m1[5m]))) / (avg by (a) (m2)))[5m:1m])`
+		plan, ok := classify(parse(t, q), grid)
+		require.True(t, ok)
+		// Both sides compile on the subquery grid: the rate side and the
+		// gauge aggregation side; the engine joins them and smooths.
+		require.Len(t, plan.units, 2)
+		assert.Equal(t, int64(60_000), plan.units[0].grid.stepMs)
+		assert.Equal(t, unitInstant, plan.units[1].core.kind)
+		assert.Contains(t, plan.rewritten, `__signoz_transpiled_0__ / __signoz_transpiled_1__`)
+	})
+}
+
+func TestBuildUnitSQL(t *testing.T) {
+	unit := &coreUnit{
+		fn:       fnRate,
+		rangeMs:  300_000,
+		hasAgg:   true,
+		aggOp:    parser.SUM,
+		by:       true,
+		grouping: []string{"pod"},
+		matchers: []*labels.Matcher{mustMatcher(t, labels.MatchEqual, "__name__", "http_requests_total")},
+	}
+	sql, args, err := buildUnitSQL(unit, []string{"http_requests_total"}, 1_699_999_700_000, 1_700_003_600_000, 1_700_000_000_000, 1_700_003_600_000, 60_000, 300_000)
+	require.NoError(t, err)
+
+	assert.Contains(t, sql, "timeSeriesRateToGrid(fromUnixTimestamp64Milli(1700000000000), fromUnixTimestamp64Milli(1700003600000), 60, 300)(fromUnixTimestamp64Milli(unix_milli), value)")
+	assert.Contains(t, sql, "unix_milli > ? AND unix_milli <= ?")
+	assert.Contains(t, sql, "bitAnd(flags, 1) = 0")
+	assert.Contains(t, sql, "sumForEach(grid)")
+	// The group-key join rides inside the shard query: distributed samples
+	// at the top level, the local series table in the join subquery, the
+	// grid aggregation grouped per (fingerprint, group key) shard-side.
+	assert.Contains(t, sql, "FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint,")
+	assert.Contains(t, sql, "FROM signoz_metrics.time_series_v4 WHERE")
+	// The group key is functionally dependent on the fingerprint (one
+	// labelset per fingerprint): any() is exact and the per-row hash key
+	// shrinks to the fingerprint alone.
+	assert.Contains(t, sql, "any(series.g0) AS g0")
+	assert.Contains(t, sql, "GROUP BY points.fingerprint)")
+	// No samples-side fingerprint condition: the group-key join restricts.
+	assert.NotContains(t, sql, "points.fingerprint IN (")
+	// by (pod) extracts the grouped label directly — no per-row JSON
+	// build/sort/stringify for a known projection.
+	assert.Contains(t, sql, "JSONExtractString(labels, ?) AS g0")
+	assert.NotContains(t, sql, "toJSONString")
+	assert.Contains(t, sql, "SETTINGS allow_experimental_ts_to_grid_aggregate_function = 1")
+	// Args follow placeholder order: the joined series subquery renders
+	// before the samples WHERE, and its select list ('pod') renders before
+	// its own conditions.
+	assert.Equal(t, []any{"pod", "http_requests_total", int64(1_699_999_200_000), int64(1_700_003_600_000), "http_requests_total", int64(1_699_999_700_000), int64(1_700_003_600_000)}, args)
+}
+
+func TestBuildUnitSQLIncreaseAndOffset(t *testing.T) {
+	unit := &coreUnit{
+		fn:       fnIncrease,
+		rangeMs:  600_000,
+		offsetMs: 1_800_000,
+		matchers: []*labels.Matcher{mustMatcher(t, labels.MatchEqual, "__name__", "errors_total")},
+	}
+	sql, _, err := buildUnitSQL(unit, nil, 1_699_997_600_000, 1_700_001_800_000, 1_700_000_000_000, 1_700_003_600_000, 60_000, 300_000)
+	require.NoError(t, err)
+
+	// Grid and window shift by the offset; increase multiplies rate by the
+	// range in seconds.
+	assert.Contains(t, sql, "fromUnixTimestamp64Milli(1699998200000), fromUnixTimestamp64Milli(1700001800000)")
+	assert.Contains(t, sql, "arrayMap(x -> x * 600, timeSeriesRateToGrid")
+	assert.Contains(t, sql, "maxForEach(grid)")
+}
+
+func TestBuildUnitSQLWindowSliver(t *testing.T) {
+	// rate[5m] on a 30m grid evaluates only a 5m sliver before each grid
+	// point — samples in the gaps belong to no window and would only be
+	// buffered by the grid aggregate. The WHERE must keep exactly the
+	// in-window rows: positiveModulo anchored at the selector start (end
+	// can sit off-lattice on unaligned grids, and samples above the start
+	// make the plain modulo dividend negative), and the scan capped at the
+	// last grid point — rows past it are equally windowless.
+	unit := &coreUnit{
+		fn:       fnRate,
+		rangeMs:  300_000,
+		hasAgg:   true,
+		aggOp:    parser.SUM,
+		by:       true,
+		grouping: []string{"pod"},
+		matchers: []*labels.Matcher{mustMatcher(t, labels.MatchEqual, "__name__", "http_requests_total")},
+	}
+	sql, args, err := buildUnitSQL(unit, []string{"http_requests_total"}, 1_699_999_700_000, 1_700_003_600_000, 1_700_000_000_000, 1_700_003_600_000, 1_800_000, 300_000)
+	require.NoError(t, err)
+
+	assert.Contains(t, sql, "positiveModulo(? - unix_milli, ?) < ?")
+	assert.Equal(t, []any{"pod", "http_requests_total", int64(1_699_999_200_000), int64(1_700_003_600_000), "http_requests_total", int64(1_699_999_700_000), int64(1_700_003_600_000), int64(1_700_000_000_000), int64(1_800_000), int64(300_000)}, args)
+
+	t.Run("off-lattice end caps the scan at the last grid point", func(t *testing.T) {
+		// end - start = 50m at a 30m step: the only grid points are start
+		// and start+30m; samples in the trailing 20m serve no window.
+		_, args, err := buildUnitSQL(unit, []string{"http_requests_total"}, 1_699_999_700_000, 1_700_003_000_000, 1_700_000_000_000, 1_700_003_000_000, 1_800_000, 300_000)
+		require.NoError(t, err)
+		assert.Contains(t, args, int64(1_700_001_800_000))
+	})
+
+	t.Run("window covering the step keeps plain bounds", func(t *testing.T) {
+		sql, _, err := buildUnitSQL(unit, []string{"http_requests_total"}, 1_699_999_700_000, 1_700_003_600_000, 1_700_000_000_000, 1_700_003_600_000, 60_000, 300_000)
+		require.NoError(t, err)
+		assert.NotContains(t, sql, "positiveModulo")
+	})
+}
+
+func TestBuildUnitSQLWindowedBucketsWithoutFanOut(t *testing.T) {
+	// The window is W = range/step whole buckets, so each sample lands in
+	// exactly one bucket via GROUP BY and the window slides over bucket
+	// partials — fanning samples into every covered window (ARRAY JOIN)
+	// multiplies rows by W, a row explosion at long ranges.
+	unit := &coreUnit{
+		kind:     unitOverTime,
+		overFn:   "avg",
+		rangeMs:  600_000,
+		matchers: []*labels.Matcher{mustMatcher(t, labels.MatchEqual, "__name__", "node_load1")},
+	}
+	sql, _, err := buildUnitSQL(unit, []string{"node_load1"}, 1_699_999_400_000, 1_700_003_600_000, 1_700_000_000_000, 1_700_003_600_000, 60_000, 600_000)
+	require.NoError(t, err)
+
+	assert.NotContains(t, sql, "ARRAY JOIN")
+	// One group per series with fixed per-bucket arrays (-Resample); the
+	// bucket index jj = ceil((ts - start)/step) + W - 1 folded into a single
+	// intDiv. Grouping by (series, bucket) instead measured 37M hash groups
+	// whose per-thread partials scale memory with max_threads.
+	assert.Contains(t, sql, "countResample(0, 71, 1)(value, intDiv(unix_milli - 1700000000000 + 600000 - 1, 60000)) AS cnts")
+	assert.Contains(t, sql, "sumResample(0, 71, 1)(value, intDiv(unix_milli - 1700000000000 + 600000 - 1, 60000)) AS vals")
+	assert.Contains(t, sql, "any(series.gkey) AS gkey")
+	assert.Contains(t, sql, "GROUP BY points.fingerprint)")
+	assert.NotContains(t, sql, "jj) AS jj")
+	assert.Contains(t, sql, "INNER JOIN (SELECT fingerprint,")
+	assert.Contains(t, sql, "FROM signoz_metrics.time_series_v4 WHERE")
+	// Slide: W = 10 buckets per slot, absent when the window count is 0.
+	assert.Contains(t, sql, "arraySum(arraySlice(cnts, k + 1, 10))")
+	assert.Contains(t, sql, "arraySum(arraySlice(vals, k + 1, 10))")
+}
+
+func TestBuildUnitSQLDisjointOverTime(t *testing.T) {
+	// avg_over_time[5m] on a 30m grid: the windows are pairwise disjoint,
+	// so there is no slide — one Resample bucket per grid slot, read
+	// directly. Exact only together with the window-sliver predicate, which
+	// removes the gap samples the ceil index would otherwise assign to the
+	// window above them.
+	unit := &coreUnit{
+		kind:     unitOverTime,
+		overFn:   "avg",
+		rangeMs:  300_000,
+		matchers: []*labels.Matcher{mustMatcher(t, labels.MatchEqual, "__name__", "node_load1")},
+	}
+	sql, _, err := buildUnitSQL(unit, []string{"node_load1"}, 1_699_999_700_000, 1_700_003_600_000, 1_700_000_000_000, 1_700_003_600_000, 1_800_000, 300_000)
+	require.NoError(t, err)
+
+	assert.NotContains(t, sql, "ARRAY JOIN")
+	// gridLen = 3 slots, bucket array the same length — no W tail.
+	assert.Contains(t, sql, "countResample(0, 3, 1)(value, intDiv(unix_milli - 1700000000000 + 1800000 - 1, 1800000)) AS cnts")
+	assert.Contains(t, sql, "sumResample(0, 3, 1)(value, intDiv(unix_milli - 1700000000000 + 1800000 - 1, 1800000)) AS vals")
+	// Single-bucket window: the slide degenerates to reading one slot.
+	assert.Contains(t, sql, "arraySum(arraySlice(cnts, k + 1, 1))")
+	// The sliver predicate is the correctness precondition of this form.
+	assert.Contains(t, sql, "positiveModulo(? - unix_milli, ?) < ?")
+}
+
+// TestDisjointWindowLattice brute-forces the disjoint-form arithmetic: a
+// sample survives the sliver predicate exactly when some grid window
+// contains it, and the ceil bucket index then lands it on that window's
+// slot. This is the pure-Go mirror of the SQL expressions — the predicate
+// in samplesConditions and jj in windowedInner — over random lattices,
+// including off-lattice ends and samples beyond the last grid point.
+func TestDisjointWindowLattice(t *testing.T) {
+	rng := func(seed *uint64) int64 {
+		*seed = *seed*6364136223846793005 + 1442695040888963407
+		return int64(*seed >> 33)
+	}
+	seed := uint64(42)
+	for trial := 0; trial < 2000; trial++ {
+		stepMs := 1_000 * (1 + rng(&seed)%3600)
+		windowMs := 1 + rng(&seed)%(stepMs-1) // strictly below the step
+		selStart := 1_700_000_000_000 + rng(&seed)%1_000_000
+		selEnd := selStart + rng(&seed)%(50*stepMs) // end may sit off-lattice
+		lastIdx := (selEnd - selStart) / stepMs
+		upper := selStart + lastIdx*stepMs
+
+		for i := 0; i < 50; i++ {
+			u := selStart - windowMs - stepMs + rng(&seed)%(selEnd-selStart+3*stepMs)
+
+			// Oracle: is u inside any window (t_k - window, t_k]?
+			inWindow := false
+			var slot int64 = -1
+			for k := int64(0); k <= lastIdx; k++ {
+				tk := selStart + k*stepMs
+				if u > tk-windowMs && u <= tk {
+					inWindow = true
+					slot = k
+					break
+				}
+			}
+
+			// The SQL: fetch bounds, then the sliver predicate
+			// positiveModulo(selStart - u, step) < window.
+			kept := u > selStart-windowMs && u <= upper
+			if kept {
+				pmod := (selStart - u) % stepMs
+				if pmod < 0 {
+					pmod += stepMs
+				}
+				kept = pmod < windowMs
+			}
+
+			require.Equal(t, inWindow, kept,
+				"sliver keep mismatch: u=%d selStart=%d step=%d window=%d", u, selStart, stepMs, windowMs)
+			if !kept {
+				continue
+			}
+			// jj = ceil((u - selStart)/step) via one intDiv; numerator is
+			// positive because u > selStart - window > selStart - step.
+			jj := (u - selStart + stepMs - 1) / stepMs
+			require.Equal(t, slot, jj,
+				"slot mismatch: u=%d selStart=%d step=%d window=%d", u, selStart, stepMs, windowMs)
+		}
+	}
+}
+
+func TestTryExecuteRange_WindowedGateFallsBack(t *testing.T) {
+	c, store := newTestClient(t)
+	e := &executor{client: c, parser: prometheus.NewParser()}
+
+	start := time.UnixMilli(1_700_000_000_000)
+	end := time.UnixMilli(1_700_003_600_000)
+
+	// 10m range at 90s step: the window is not a whole number of buckets.
+	_, ok, err := e.TryExecuteRange(context.Background(), `avg_over_time(up[10m])`, start, end, 90*time.Second)
+	require.NoError(t, err)
+	assert.False(t, ok, "range not divisible by step must not transpile")
+
+	// 1d range at 60s step: 1440 bucket combines per slot, over the cap.
+	_, ok, err = e.TryExecuteRange(context.Background(), `avg_over_time(up[1d])`, start, end, time.Minute)
+	require.NoError(t, err)
+	assert.False(t, ok, "range/step above maxWindowBuckets must not transpile")
+
+	// 1m range at 5m step: the windows are disjoint slivers — no
+	// divisibility or width requirement, so this transpiles.
+	store.Mock().ExpectQuery("SELECT fingerprint, any\\(labels\\)").WithArgs("up", int64(1_699_999_200_000), int64(1_700_003_600_000)).WillReturnRows(cmock.NewRows(seriesCols, [][]any{}))
+	_, ok, err = e.TryExecuteRange(context.Background(), `avg_over_time(up[1m])`, start, end, 5*time.Minute)
+	require.NoError(t, err)
+	assert.True(t, ok, "range below step is the disjoint form and must transpile")
+}
+
+func TestApplyScalarOps(t *testing.T) {
+	f := func(v float64) *float64 { return &v }
+
+	t.Run("arithmetic chain", func(t *testing.T) {
+		values := []*float64{f(2), nil, f(4)}
+		applyScalarOps([]scalarOp{{op: parser.MUL, scalar: 100}, {op: parser.ADD, scalar: 1}}, values)
+		require.NotNil(t, values[0])
+		assert.Equal(t, 201.0, *values[0])
+		assert.Nil(t, values[1])
+		assert.Equal(t, 401.0, *values[2])
+	})
+
+	t.Run("comparison filters points", func(t *testing.T) {
+		values := []*float64{f(1), f(10)}
+		applyScalarOps([]scalarOp{{op: parser.GTR, scalar: 5}}, values)
+		assert.Nil(t, values[0])
+		require.NotNil(t, values[1])
+		assert.Equal(t, 10.0, *values[1], "filter comparisons keep the original value")
+	})
+
+	t.Run("bool comparison emits 0/1", func(t *testing.T) {
+		values := []*float64{f(1), f(10)}
+		applyScalarOps([]scalarOp{{op: parser.GTR, scalar: 5, returnBool: true}}, values)
+		assert.Equal(t, 0.0, *values[0])
+		assert.Equal(t, 1.0, *values[1])
+	})
+
+	t.Run("scalar on left division", func(t *testing.T) {
+		values := []*float64{f(4)}
+		applyScalarOps([]scalarOp{{op: parser.DIV, scalar: 100, scalarOnLeft: true}}, values)
+		assert.Equal(t, 25.0, *values[0])
+	})
+}
+
+func TestLabelsFromGroupKey(t *testing.T) {
+	lset, err := labelsFromGroupKey(`[["pod","api-0"],["ns","prod"]]`)
+	require.NoError(t, err)
+	assert.Equal(t, "api-0", lset.Get("pod"))
+	assert.Equal(t, "prod", lset.Get("ns"))
+
+	empty, err := labelsFromGroupKey(`[]`)
+	require.NoError(t, err)
+	assert.True(t, empty.IsEmpty())
+}
+
+// testGrid is a 2h query grid ending on a round timestamp.
+func testGrid(stepMs int64) gridContext {
+	return gridContext{startMs: 1_700_000_000_000, endMs: 1_700_007_200_000, stepMs: stepMs}
+}
+
+// A bool comparison returns 0/1, not the sample, so the engine drops
+// __name__; keeping it would change downstream vector matching.
+func TestKeepsName_BoolComparisonDropsName(t *testing.T) {
+	plan, ok := classify(parse(t, `up > bool 0`), testGrid(60_000))
+	require.True(t, ok)
+	assert.False(t, plan.units[0].core.keepsName())
+
+	plan, ok = classify(parse(t, `up > 0`), testGrid(60_000))
+	require.True(t, ok)
+	assert.True(t, plan.units[0].core.keepsName())
+}
+
+// timeSeriesLastToGrid widens its window to max(window, step) — probed on
+// 25.12 — so Last-style units at window < step must fall back or they would
+// resurrect samples the engine's lookback already dropped.
+func TestTryExecuteRange_LastStyleWindowBelowStepTranspiles(t *testing.T) {
+	// These used to fall back because timeSeriesLastToGrid widens its window
+	// to max(window, step). Over sliver-filtered rows the widening is
+	// harmless — the widened window intersected with the data IS the
+	// lookback window — so the gate is gone and both shapes transpile. The
+	// mock returns no series: the point here is the routing, the value
+	// semantics are the parity suite's job.
+	c, store := newTestClient(t)
+	e := &executor{client: c, parser: prometheus.NewParser()}
+
+	start := time.UnixMilli(1_700_000_000_000)
+	end := time.UnixMilli(1_700_003_600_000)
+
+	store.Mock().ExpectQuery("SELECT fingerprint, any\\(labels\\)").WithArgs("up", int64(1_699_999_200_000), int64(1_700_003_600_000)).WillReturnRows(cmock.NewRows(seriesCols, [][]any{}))
+	_, ok, err := e.TryExecuteRange(context.Background(), `sum by (pod) (up)`, start, end, time.Hour)
+	require.NoError(t, err)
+	assert.True(t, ok, "instant selection at step > lookback must transpile")
+
+	store.Mock().ExpectQuery("SELECT fingerprint, any\\(labels\\)").WithArgs("up", int64(1_699_999_200_000), int64(1_700_003_600_000)).WillReturnRows(cmock.NewRows(seriesCols, [][]any{}))
+	_, ok, err = e.TryExecuteRange(context.Background(), `last_over_time(up[10m])`, start, end, time.Hour)
+	require.NoError(t, err)
+	assert.True(t, ok, "last_over_time at range < step must transpile")
+}
+
+// A nameless selector can span metrics whose series alternate in time (one
+// dies inside the lookback before the other appears); after the name drop
+// the engine merges them into ONE series and errors only when two samples
+// share an evaluation timestamp. Pinned by conformance cases
+// operators.test:994/997 (-{job="api"} over http_requests/http_errors).
+func TestMergeSameLabelsetSeries(t *testing.T) {
+	f := func(v float64) *float64 { return &v }
+	api := labels.FromStrings("job", "api")
+
+	out, err := mergeSameLabelsetSeries([]transpiledSeries{
+		{lset: api, values: []*float64{f(-2), nil}},
+		{lset: api, values: []*float64{nil, f(-4)}},
+		{lset: labels.FromStrings("job", "web"), values: []*float64{f(7), nil}},
+	})
+	require.NoError(t, err)
+	require.Len(t, out, 2)
+	assert.Equal(t, []*float64{f(-2), f(-4)}, out[0].values, "temporally disjoint twins must merge into one series")
+
+	_, err = mergeSameLabelsetSeries([]transpiledSeries{
+		{lset: api, values: []*float64{f(1), nil}},
+		{lset: api, values: []*float64{f(2), nil}},
+	})
+	require.Error(t, err, "two values on one evaluation timestamp is the engine's duplicate error")
+	assert.True(t, errors.Ast(err, errors.TypeInvalidInput))
+}
+
+// Hybrid twin case: stripping the synthetic __name__ can leave two engine
+// output series distinguishable only by those names (-metric_a or -metric_b:
+// both {} once real names are dropped). Pinned by conformance cases
+// name_label_dropping.test:137 and operators.test:1016.
+func TestMergeMatrixByLabelset(t *testing.T) {
+	empty := labels.EmptyLabels()
+
+	out, err := mergeMatrixByLabelset(promql.Matrix{
+		{Metric: empty, Floats: []promql.FPoint{{T: 0, F: -1}}},
+		{Metric: empty, Floats: []promql.FPoint{{T: 600_000, F: -4}}},
+	})
+	require.NoError(t, err)
+	require.Len(t, out, 1)
+	assert.Equal(t, []promql.FPoint{{T: 0, F: -1}, {T: 600_000, F: -4}}, out[0].Floats)
+
+	_, err = mergeMatrixByLabelset(promql.Matrix{
+		{Metric: empty, Floats: []promql.FPoint{{T: 0, F: -1}}},
+		{Metric: empty, Floats: []promql.FPoint{{T: 0, F: -3}}},
+	})
+	require.Error(t, err)
+	assert.True(t, errors.Ast(err, errors.TypeInvalidInput))
+}

pkg/prometheus/prometheus.go

@@ -1,6 +1,9 @@
 package prometheus
 
 import (
+	"context"
+	"time"
+
 	"github.com/prometheus/prometheus/promql"
 	"github.com/prometheus/prometheus/promql/parser"
 	"github.com/prometheus/prometheus/storage"
@@ -41,3 +44,14 @@ type StatementCapturer interface {
 // X-SigNoz-PromQL-Provider request header all use it, so they cannot drift
 // apart.
 const ProviderClickhouseV2 = "clickhousev2"
+
+// RangeExecutor is the optional capability of a provider that can evaluate
+// some range queries entirely inside the datastore. ok=false means the
+// query is not evaluable that way. The caller then runs the engine over the
+// provider's Storage, which is always exact. Only the clickhousev2 provider
+// implements this capability. When that provider is the only one, the
+// capability folds into Prometheus itself, and the engine-vs-datastore
+// decision becomes internal.
+type RangeExecutor interface {
+	TryExecuteRange(ctx context.Context, query string, start, end time.Time, step time.Duration) (promql.Matrix, bool, error)
+}

pkg/querier/promql_query.go

@@ -344,8 +344,8 @@ func (q *promqlQuery) Execute(ctx context.Context) (*qbv5.Result, error) {
 	}
 
 	// Accumulate ClickHouse-side scan stats across every storage query this
-	// evaluation issues: progress options propagate to each ClickHouse query
-	// through the context.
+	// evaluation issues (engine selectors or the compiled executor): progress
+	// options propagate to each ClickHouse query through the context.
 	var statsMu sync.Mutex
 	var rowsScanned, bytesScanned uint64
 	ctx = clickhouse.Context(ctx, clickhouse.WithProgress(func(p *clickhouse.Progress) {
@@ -371,6 +371,23 @@ func (q *promqlQuery) Execute(ctx context.Context) (*qbv5.Result, error) {
 		return q.toResult(matrix, nil, began, &statsMu, &rowsScanned, &bytesScanned), nil
 	}
 
+	// When the serving provider has the RangeExecutor capability
+	// (prometheus::provider: clickhousev2), serve the way the provider is
+	// designed to serve: transpiled when the shape allows. Without this the
+	// override would silently run the engine path only.
+	if re, ok := q.promEngine.(prometheus.RangeExecutor); ok {
+		matrix, served, err := re.TryExecuteRange(ctx, query, time.Unix(0, start), time.Unix(0, end), q.query.Step.Duration)
+		if err != nil {
+			if enhanced := tryEnhancePromQLExecError(err); enhanced != nil {
+				return nil, enhanced
+			}
+			return nil, err
+		}
+		if served {
+			return q.toResult(matrix, nil, began, &statsMu, &rowsScanned, &bytesScanned), nil
+		}
+	}
+
 	qry, err := q.promEngine.Engine().NewRangeQuery(
 		ctx,
 		q.promEngine.Storage(),

pkg/querier/promql_shadow.go

@@ -19,11 +19,11 @@ import (
 const shadowTimeout = 2 * time.Minute
 
 // runShadowCompare executes the query on the clickhousev2 provider exactly
-// as it would serve (the engine over the v2 querier), compares against the
-// served result and logs the outcome. Serving is never affected: this runs
-// after the response, off the request context, and only logs. The mismatch
-// and failure logs are the rollout evidence — serving cuts over to v2 only
-// after they stay clean.
+// as it would serve (transpiled when the shape allows, engine over the v2
+// querier otherwise), compares against the served result and logs the
+// outcome. Serving is never affected: this runs after the response, off the
+// request context, and only logs. The mismatch and failure logs are the
+// rollout evidence — serving cuts over to v2 only after they stay clean.
 func (q *promqlQuery) runShadowCompare(ctx context.Context, query string, startNs, endNs int64, served promql.Matrix, servedIn time.Duration) {
 	defer func() {
 		if r := recover(); r != nil {
@@ -45,7 +45,7 @@ func (q *promqlQuery) runShadowCompare(ctx context.Context, query string, startN
 
 	start, end := time.Unix(0, startNs), time.Unix(0, endNs)
 	began := time.Now()
-	shadow, err := executeOnProvider(ctx, q.opts.shadow, query, start, end, q.query.Step.Duration)
+	shadow, transpiled, err := executeOnProvider(ctx, q.opts.shadow, query, start, end, q.query.Step.Duration)
 	shadowIn := time.Since(began)
 
 	logAttrs := []any{
@@ -53,6 +53,7 @@ func (q *promqlQuery) runShadowCompare(ctx context.Context, query string, startN
 		slog.Int64("start_ms", startNs/int64(time.Millisecond)),
 		slog.Int64("end_ms", endNs/int64(time.Millisecond)),
 		slog.Duration("step", q.query.Step.Duration),
+		slog.Bool("transpiled", transpiled),
 		slog.Duration("served_in", servedIn),
 		slog.Duration("shadow_in", shadowIn),
 	}
@@ -82,29 +83,41 @@ func (q *promqlQuery) runShadowCompare(ctx context.Context, query string, startN
 // serveFromProvider evaluates the query the way the pinned provider would
 // serve it.
 func (q *promqlQuery) serveFromProvider(ctx context.Context, query string, startNs, endNs int64) (promql.Matrix, error) {
-	return executeOnProvider(ctx, q.opts.serve, query, time.Unix(0, startNs), time.Unix(0, endNs), q.query.Step.Duration)
+	matrix, _, err := executeOnProvider(ctx, q.opts.serve, query, time.Unix(0, startNs), time.Unix(0, endNs), q.query.Step.Duration)
+	return matrix, err
 }
 
-// executeOnProvider evaluates the query the way the provider would serve it:
-// the engine over the provider's storage. The returned matrix is an owned
-// copy.
-func executeOnProvider(ctx context.Context, prov prometheus.Prometheus, query string, start, end time.Time, step time.Duration) (promql.Matrix, error) {
+// executeOnProvider evaluates the query the way the provider would serve
+// it: transpiled in the datastore when the provider has the RangeExecutor
+// capability and the shape allows, or the engine over the provider's
+// storage. The returned matrix is an owned copy.
+func executeOnProvider(ctx context.Context, prov prometheus.Prometheus, query string, start, end time.Time, step time.Duration) (promql.Matrix, bool, error) {
+	if re, ok := prov.(prometheus.RangeExecutor); ok {
+		matrix, served, err := re.TryExecuteRange(ctx, query, start, end, step)
+		if err != nil {
+			return nil, true, err
+		}
+		if served {
+			return matrix, true, nil
+		}
+	}
+
 	qry, err := prov.Engine().NewRangeQuery(ctx, prov.Storage(), nil, query, start, end, step)
 	if err != nil {
-		return nil, err
+		return nil, false, err
 	}
 	defer qry.Close()
 
 	res := qry.Exec(ctx)
 	if res.Err != nil {
-		return nil, res.Err
+		return nil, false, res.Err
 	}
 	matrix, err := res.Matrix()
 	if err != nil {
-		return nil, err
+		return nil, false, err
 	}
 	// Close returns the result's sample slices to the engine pool.
-	return copyMatrix(matrix), nil
+	return copyMatrix(matrix), false, nil
 }
 
 func copyMatrix(matrix promql.Matrix) promql.Matrix {

tests/integration/testdata/promqltestcorpus/known_divergences_v2.json

@@ -1,4 +1,17 @@
 {
-  "note": "Divergences of the clickhousev2 provider (pinned via X-SigNoz-PromQL-Provider) from the upstream reference engine, enforced exactly by 01_upstream_corpus.py in both directions. This ledger is the rollout scorecard for the provider swap: the default provider cannot be replaced by clickhousev2 while anything is listed here. Entries must carry the defect's cause and be REMOVED as the provider is fixed.",
-  "divergences": {}
+  "note": "Divergences of the clickhousev2 provider (pinned via X-SigNoz-PromQL-Provider) from the upstream reference engine, enforced exactly by 01_upstream_corpus.py in both directions. This ledger is the rollout scorecard for the provider swap: the default provider cannot be replaced by clickhousev2 while anything is listed here. Entries must carry the defect's cause and be REMOVED as the provider is fixed. Current class: the engine aggregates floats with Kahan compensated summation (sum, sum_over_time) and an overflow-free incremental mean (avg); ClickHouse's sumForEach/avgForEach/arraySum are naive, so extreme-magnitude corpus data (±1e100 cancellation, ±1.8e308 overflow) diverges on transpiled plans. Burn-down candidates: sumKahanForEach for the cancellation class; the overflow class needs an incremental-mean aggregate ClickHouse does not have.",
+  "divergences": {
+    "aggregators.test:651[base]": "avg over near-max-float64 values: engine's incremental mean never forms the overflowing sum; avgForEach sums then divides, overflowing to +Inf",
+    "aggregators.test:651[instant-coarse]": "same as aggregators.test:651[base] on the coarse-step grid variant",
+    "aggregators.test:654[base]": "avg over near-min-float64 values: engine's incremental mean never forms the overflowing sum; avgForEach overflows to -Inf",
+    "aggregators.test:654[instant-coarse]": "same as aggregators.test:654[base] on the coarse-step grid variant",
+    "aggregators.test:687[base]": "sum over {1e100, -1e100, small}: engine uses Kahan compensated summation; sumForEach's naive summation loses the small terms to cancellation and returns 0",
+    "aggregators.test:687[instant-coarse]": "same as aggregators.test:687[base] on the coarse-step grid variant",
+    "aggregators.test:695[base]": "avg over {1e100, -1e100, small}: same Kahan-vs-naive cancellation as aggregators.test:687, divided by count",
+    "aggregators.test:695[instant-coarse]": "same as aggregators.test:695[base] on the coarse-step grid variant",
+    "functions.test:1084[instant-coarse]": "sum_over_time over a window containing ±1e100: the disjoint coarse-step form's arraySum slide is naive summation, cancelling to 0 (the base variant's W>64 shape falls back to the engine and is exact)",
+    "functions.test:1087[instant-coarse]": "avg_over_time, same window and cancellation as functions.test:1084[instant-coarse]",
+    "functions.test:1149[base]": "avg_over_time over ±2.258e220-magnitude samples: engine's Kahan-compensated incremental mean cancels exactly to 0; the bucketed form's naive slide summation leaves a ~1e202 residue",
+    "functions.test:1149[instant-coarse]": "same as functions.test:1149[base] through the disjoint coarse-step form"
+  }
 }