#### Description
- A referenced name in a trace query now resolves to a `LogicalField`
(#12499): one field, addressed by the requested spelling, backed by its
physical member keys. A semantic-convention family
(`deployment.environment.name` / `deployment.environment`) merges into
one expression with current-wins precedence; the response keeps the
requested spelling.
- `FieldMapper` gets one new method, `ExistsFor` (the per-key presence
primitive). `LogicalValueExpr` and `LogicalExistsExpr` build all family
SQL in one place from `FieldFor` and `ExistsFor`; no signal implements
family logic.
- Statement builders prefetch sibling spellings; the metadata store
stays family-blind and autocomplete stays literal. Traces and the
resource filter compile per logical field; logs, metrics, and the other
signals keep their SQL unchanged.
- The `resolve_semconv_families` feature flag (default: disabled) gates
all family behavior. With the flag off, the generated SQL is the same as
main; tests pin this. Part of #6143.
#### Additional Information
- Stack: #12441 (merged) → **#12442** → #12443 → #12444 → #12445 →
#12446 → #12447. This layer bases on main.
- Rollback: turn the flag off; stored telemetry is untouched.
## Pull Request
---
### 📄 Summary
- Add a `type` param to `/api/v1/fields/keys`; for type=builder_ai_query
(flag-gated) the metadata store returns the per-trace aggregate columns
(llm_call_count, input_tokens, …) as
trace-context keys — they're computed at query time, never ingested, so
the attribute scan can't serve them.
- Split `TraceColumn.Orderable` into `Orderable + Filterable`: ORDER BY
uses orderable, the trace-level filter validates against filterable, and
the API only returns keys that are both. `last_activity_time` is
order-only and now rejected in filters with a targeted error.**
- UI note: last_activity_time should be added to client-side list (it's
the default sort).
#### Issues closed by this PR
Part of https://github.com/SigNoz/engineering-pod/issues/5714
---
### ✅ Change Type
_Select all that apply_
- [x] ✨ Feature
- [ ] 🐛 Bug fix
- [ ] ♻️ Refactor
- [ ] 🛠️ Infra / Tooling
- [ ] 🧪 Test-only
---
### 🧪 Testing Strategy
> How was this change validated?
- Tests added/updated: ✅
- Manual verification: ✅
- Edge cases covered: ✅
---
### ⚠️ Risk & Impact Assessment
> What could break? How do we recover?
- Blast radius: None
- Potential regressions:
- Rollback plan:
A filter on a metrics label that isn't in metadata ran silently: the
query fell back to reading the label directly, but nothing told the user
the key was unknown. Removes the `TODO(srikanthccv)` in the metrics
statement builder.
### What
The detection was already written, and already in the right place.
`conditionBuilder.ConditionFor` spots a filter key with no metadata
match, warns, and synthesizes an attribute-context key so the query
still runs — and it only ever sees terms in **key position**, so it
cannot mistake a value or a dashboard variable for a key. That is
exactly what the TODO was waiting for.
Two things hid it:
- `Build` pre-seeded the field-key map with a synthesized entry for
every lexer-derived selector, so `MatchingFieldKeys` always matched and
the missing-key branch was dead code.
- The metrics builder never read `PrepareWhereClause`'s warnings — the
visitor collects them and `unionStatements` merges them, but nothing
ever set them.
So: drop the pre-seeding, and carry the warnings out of
`buildTimeSeriesCTE` onto the statement.
### Notes
- **Generated SQL is unchanged.** The key the condition builder
synthesizes (attribute context, name as written) is what the pre-seeding
was injecting, so `test_missing_key_falls_back_to_labels` still expects
byte-identical SQL and only gains the warning.
- A full-text term routes through `labels`, which is a real column, so
it takes the `isColumn` branch and stays silent — bare-word searches
don't start warning.
- The reduced statement prepares the same filter over the same keys, so
only the main path's warnings go into the union; carrying both would
show each warning twice.
### Testing
- `test_missing_key_falls_back_to_labels` gains the expected warning,
same SQL.
-
`queriermetrics/10_key_resolution.py::test_metrics_filter_unknown_label_matches_nothing`
now asserts the warning instead of asserting silence.
- `queriermetrics/02_warnings.py` already covered the TODO's own example
(`my_tag = $tag`). It passed before only because every warning was
suppressed; it is now a real guard that a value-position variable is not
flagged.
- `queriermetrics` integration suite: 118 passed. `go test
./pkg/statementbuilder/... ./pkg/telemetryschema/...` green. `make
go-lint` and `make py-lint` clean.
A metric label may be named after a column the generated query builds
for itself, and the metrics and meter builders selected group-by columns
under the label's own name — so `group by ts` or `group by value`
produced SQL with two columns of that name, which ClickHouse rejects.
### What
- Metrics and meter now alias group-by columns
`__GROUP_BY_KEY_<i>_<name>`, the scheme the logs and traces statement
builders already use.
- `pkg/querier/consume.go` already strips that prefix on all three read
paths (time-series, scalar, raw), so API responses are unchanged.
- Metrics' `ColumnExpressionFor` now returns the bare expression like
the logs and traces mappers. It was the only one returning an aliased
expression (`expr AS <name>`), which `agg_rewrite.go` splices inside a
function argument — giving `sum(expr AS <name>)` if metrics ever grows
expression aggregations. Callers alias and escape, as logs does.
- The histogram pipeline derives its CTE-side query once — `le` appended
last, plus the existing rate/sum rewrite — instead of mutating the query
and restoring it around the whole pipeline. The final select takes the
original minus `le`, so the remaining keys hold the positions their CTE
aliases were built from.
With a label named `ts`, before:
```sql
SELECT ts, `ts`, multiIf(…) … GROUP BY fingerprint, ts, `ts`
```
and after:
```sql
SELECT ts, `__GROUP_BY_KEY_0_ts`, multiIf(…) …
```
A label named `value` was the quieter case — the spatial CTE selected it
next to the aggregate of the same name:
```sql
SELECT ts, `value`, sum(per_series_value) AS value …
```
### Notes
- Meter comes along because it holds a
`*metricsstatementbuilder.StatementBuilder` and calls the shared
`BuildFinalSelect`; aliasing metrics alone would leave meter ordering by
an alias its own select never produced. `GroupByColumnAlias` /
`GroupByAliases` are exported for it, alongside the `GetKeySelectors` /
`RateTmpl` already shared across that boundary.
- Meter had the identical collision, so this fixes it there too.
### Testing
- `TestGroupByAliasAvoidsColumnCollision` covers `ts`, `value`,
`fingerprint` and an ordinary label, in both the metrics and meter
builders; all three collision cases fail without the change.
- `reduced_test.go` gains `histogram_p99_group_by` and
`gauge_avg_avg_group_by` — the reduced path had no group-by coverage at
all, so neither the aliases in its four CTE builders nor the union's
`ORDER BY` were exercised. The histogram case pins that both `UNION ALL`
branches emit the same columns.
- `test_histogram_count_no_param` pins the `SELECT *` branch, where `le`
stays unaliased so `ORDER BY toFloat64(le)` resolves.
- Both new behaviours were mutation-checked: appending `le` first
instead of last, and returning the bare name from `GroupByColumnAlias`,
each turn the relevant tests red.
- Twelve expected-SQL blobs regenerated across the metrics and meter
statement builder tests — alias-only diffs.
- `go test ./...` green, `make go-lint` clean.
Fixes https://github.com/SigNoz/engineering-pod/issues/5868
#### Description
- `IN` and `NOT IN` now route each value back through the condition
builder with `=` / `!=` instead of assembling the comparisons a second
time, so whatever a builder does for a scalar comparison applies to the
list form too. Applied to logs, traces and audit — the three that
already fanned a list out into per-value comparisons.
- Fixes `body.<path>[*] IN [...]` with `use_json_body` off returning a
**500**. The list shape made the path extract as `Array(String)`, and
ClickHouse refuses to compare an array to a scalar (code 130);
extracting per value reads the field instead. The new case in
`querierlogs/06_json_body.py` fails on `main` and passes here — verified
both ways against a real ClickHouse.
#### Additional Information
- **resourcefilter is deliberately left out**: it asserts the key index
filter (`labels LIKE '%key%'`) once for the whole list, alongside one
value filter per value. A recursed arm derives its own key filter, so
the same predicate would be repeated per value — `(e1 AND kIdx AND l1)
OR (e2 AND kIdx AND l2)` instead of `(e1 OR e2) AND kIdx AND (l1 OR
l2)`. Same rows either way, but no reason to emit the duplicate.
- **telemetrymetadata is deliberately left out**: it applies a
key-existence guard at a single exit, so a recursed arm comes back
already wrapped and the guard would either nest or the case would have
to skip the shared tail — losing the invariant that every condition is
guarded.
- **metrics and rulestatehistory** build a real `sb.In`; there is no
per-value fan-out to delegate to.
- Mixed-type lists are safe: `DataTypeCollisionHandledFieldName`
normalises the whole list before the loop, so `IN ('200', 5)` emits
identical SQL before and after.
- Base of a stack — the follow-ups add index-friendly predicates to `=`,
which `IN` then picks up.
- Replace the search() stub: the logs condition builder fans a case-insensitive
match of the term across every searchable column (log columns, body/body_v2,
attribute + resource maps).
- Grammar: searchCall takes a valueList (parser regenerated), so the scoped form
search('term', body, resource) parses and narrows the fan-out to those contexts.
- The visitor emits FilterOperatorSearch and flags the statement scan-heavy; the
statement builder attaches a CostGuard the querier enforces via EXPLAIN ESTIMATE
against a per-shard budget, cumulative across buckets on the window-list path.
- body_v2 gets its own, lower budget (search_max_scan_rows_json_body): toString()
rebuilds every document and no skip index prunes.
- Logs-only: other signals reject search().
- Unit tests for the per-context fan-out SQL and both budgets; integration suites
run the same matrix over the legacy body and over body_v2.
- Embed statementbuilder.Config into querier.Config with mapstructure ",squash";
keys move to querier.skip_resource_fingerprint.* (env SIGNOZ_QUERIER_*).
- Drop the standalone statementbuilder section and its config factory.
- Pass statementbuilder.Config wholesale into NewLogQueryStatementBuilder.
primus bumped golangci-lint to v2.12.2, whose govet now runs the inline
analyzer and whose sloglint is stricter. CI resolves primus.workflows@main,
so every PR started failing lint the moment that landed.
- reflect.Ptr is a deprecated alias carrying //go:fix inline, so it is now
reflect.Pointer at all five call sites
- metricsstatementbuilder imported golang.org/x/exp/slices, which carries
//go:fix inline pointing at the stdlib; the analyzer cannot inline
generics, so switch the import to stdlib slices as the directive intends
- pkg/instrumentation/loghandler emits OpenTelemetry semantic-convention
attributes (code.filepath, exception.type, ...), which are dotted rather
than snake_case by definition. Renaming them would break every log
consumer, so the keys move to constants in instrumentationtypes, which
already held this kind of key -- and already defined code.function, so
source.go was duplicating it. Six of the seven alias the semconv
constants that define them; exception.code has no OTel equivalent.
sloglint resolves a same-package constant back to its literal but skips
a qualified one, so this needs no exclusion.
CI reported 8 issues but capped at max-same-issues=3, hiding 2 more
reflect.Ptr sites and 4 more sloglint ones.
Separately, TestTimeout/WaitTillNoTimeoutForExcludedPath failed with
"transport connection broken: http: CloseIdleConnections called".
TestTimeout and TestCache issue requests through http.DefaultClient while
a parallel subtest in response_test.go closes an httptest.Server, and
httptest.Server.Close calls http.DefaultTransport.CloseIdleConnections.
Both tests now use their own client and transport, and are closed via
t.Cleanup. That makes Serve return ErrServerClosed on every run, so the
require.NoError wrapping it is dropped -- it could never have held, and
require runs t.FailNow off the test goroutine anyway. Bare Serve in a
goroutine matches routerweb and render tests.
These fired at info on every query build across the audit, logs and traces
builders. The behavior is settled, so drop them to debug and remove the
TODOs that asked for exactly this.
Remove the statementbuilder.Builders aggregate. The querier chain (querier.New,
signozquerier.NewFactory, NewQuerierProviderFactories) now takes the six per-signal
statement builders individually, and signoz.go::newQueryStack returns them directly
via a plain multi-return. The statementbuilder package is now config-only.
Remove each <signal>statementbuilder/new.go and fold its NewFactory into that
package's statement_builder.go (traces' NewOperatorFactory into
trace_operator_statement_builder.go), giving the layout: struct -> NewFactory ->
New<X>QueryStatementBuilder.
Split the five telemetry<signal> packages, which mixed three concerns, into
three per-signal layers with a cycle-free dependency direction:
- telemetryschema/<signal>telemetryschema — primitives (const + table
selection, field_mapper, condition_builder, trace helpers); leaf layer.
- statementbuilder/<signal>statementbuilder — SQL generation. The parent
statementbuilder package is contract-only (the Builders bundle + Config);
each sub-package exposes a factory.ProviderFactory[..., statementbuilder.Config]
whose New internalizes FieldMapper/ConditionBuilder/AggExprRewriter and reads
SkipResourceFingerprint. Traces exposes two factories (query + operator).
telemetryresourcefilter moves here as statementbuilder/resourcefilter.
- telemetrymetadata — key/value resolution; NewTelemetryMetaStore collapses
from 24 args to (settings, telemetrystore, flagger), sourcing table names
from the schema constants.
Centralize query-stack assembly in signoz.go via newQueryStack: build the
single metadata store, run each per-signal statement-builder factory, assemble
the statementbuilder.Builders bundle, and build the bucket cache — once. This
is the only place that imports the concrete sub-packages (so the edge runs
subs -> parent, cycle-free), and it removes the duplicate metadata store that
signozquerier used to build, leaving signozquerier a thin
querier.New(*statementbuilder.Builders) adapter.
Also:
- statementbuilder.Config owns SkipResourceFingerprint (moved off
querier.Config). YAML key moves querier.skip_resource_fingerprint ->
statementbuilder.skip_resource_fingerprint.
- Querier interface moves into querier.go (interfaces.go removed); BucketCache
-> bucket_cache.go, Handler -> api.go.
- Add pkg/querier/queriertest.MockQuerier.
