#### Description
Stacked on #12715. Makes the list-view ("all fields", empty
`selectFields`) attributes bag read from the `attributes` JSON column,
evolution-aware, so the bag stays complete once we stop dual-writing the
legacy maps.
- The empty-`selectFields` list path was a static, map-only column list
(`attributes_string/number/bool`). It now picks attribute homes by
column evolution for the query window: legacy maps before the JSON
rollout, the `attributes` JSON column after it, both across it — via a
new `MetadataStore.GetColumnEvolutions` and
`tracestelemetryschema.BulkAttributeColumns`, mirroring the per-key
selection.
- `mergeSpanAttributeColumns` merges the natively-decoded JSON column
(relies on #12555's `JSONValue` scan — no `toJSONString`, no second
unmarshal) into the flat bag: nested paths flatten back to dotted keys
and win over the legacy maps on collision, so the response shape is
identical across the rollout with a single decode and no re-marshal.
- Per-key group-by/aggregation is unchanged — it already resolves
JSON-only after the rollout — so those queries keep working once map
dual-write stops.
Follow-up (out of scope, tracked by #12555's known gaps): the
waterfall/flamegraph struct readers need the same `JSONValue` type
before span attributes can go JSON-only; `resources_string` moves to the
`resource` JSON column later.
#### Additional Information
Tests: bulk select across before/after/straddle windows; group-by after
rollout reads JSON not the map; consume flatten/merge (nested
flattening, type preservation, JSON-wins-on-collision).
Assisted-by: Claude Opus 4.8
The semantic convention family as first call citizen revealed that the
current state of the query builder needs a bit refactoring for long term
maintenance.
The `FieldMapper` and `ConditionBuilder` are now one abstraction
`Storage`.
A storage now answers
- what the compiler cannot know i.e one read per field key (the bare
SQL, the membership present or absent, what an absent row reads, and
whether the read keeps its type or filters only).
- the fallback for a key metadata does not report
- its traits
- and one Condition compilation part.
And we introduce a new type to use in the system, `Resolved`
```
// Resolved is what resolution produces for one key: its meanings, and how
// they came to be. It is the only thing the compilers receive. Compile it
// with the operator and value it was resolved with.
type Resolved struct {
Key *telemetrytypes.TelemetryFieldKey
Fields []*telemetrytypes.LogicalField
// FromFallback: the fields came from the storage's fallback, not from
// metadata matches.
FromFallback bool
// Ambiguous: the matches held several interpretations.
Ambiguous bool
// Skipped: the storage contributes nothing for this key.
Skipped bool
Warnings []string
}
```
The prepared SQL has no changes, where it changed, it specifically made
the expression better by removing the redundant part.
- The prepared SQL remains identical with this refactoring
- No changes to integration tests
Assisted-by: Claude Fable 5.1
#### Description
- Every user-controlled field name that reaches generated SQL goes
through the new `pkg/clickhousesql` package (`Identifier`,
`StringLiteral`, `Literal`, `LikePattern`): map reads and `mapContains`,
JSON sub-column paths and the JSON body access plan, labels, fingerprint
labels, materialized column names, select aliases, group-by and order-by
references, the legacy string-body JSONPath, and the raw SQL in the
trace funnel, trace detail and infra monitoring modules. Filter
expressions built from request or telemetry values use
`querybuilder.FilterStringLiteral`. The same package now also renders
dashboard variable values in the querier, LIKE patterns in the metadata
store and label lists in the PromQL transpiler, which each had their own
escaping.
- A `$` followed by a digit, `{` or `?` is written as `\x24`, which
ClickHouse decodes in identifiers and literals. Those are the forms the
tools react to: go-sqlbuilder resolves `$0` in a compiled fragment to
its own WHERE clause and recurses until the stack overflows, and
clickhouse-go rejects a query mixing `$<digits>` with `?` arguments. Any
other `$` stays literal, so materialized column names keep their `$$`
and render exactly as before; a key like `http.2xx` becomes ``
`attribute_string_http$\x242xx` `` instead of failing in the driver.
- Compiled sqlbuilder fragments (Select, GroupBy, OrderBy, raw Where
text) are wrapped with `sqlbuilder.Escape`; the metrics builder escapes
its compiled time-series subquery, which is compiled a second time when
joined.
- The raw statement validator (`ErrIfStatementIsNotValid`,
`LogIfStatementIsNotValid`) moves from
`pkg/querybuilder/clickhouse_sql.go` to
`pkg/clickhousesql/statement.go`. Its `Code*` identifiers drop the
`ClickHouseSQL` prefix; the code strings are unchanged.
- Unit tests round-trip the helpers over hostile names and drive them
through the modules' raw SQL;
`tests/integration/tests/queriercommon/08_field_name_quoting.py` and
`querier_json_body/07_field_name_quoting.py` query such names through
the logs, traces and metrics builders against a real ClickHouse.
#### Additional Information
- `docs/contributing/go/clickhousesql.md` documents the quoting
functions, where `sqlbuilder.Escape` belongs, the `$` rule and the
statement validator; `.claude/rules/go-contrib.md` points at it.
- `pkg/clickhousesql` is a leaf package so `telemetrytypes` (JSON access
plan) and `querybuilder` share one implementation without a cycle.
- For names without special characters the generated SQL is byte
identical.
- Not covered here: the legacy v3/v4 query_range builders and the
`pkg/query-service/utils` quoting helpers (`QuoteEscapedString`,
`QuoteEscapedStringForContains`, `ClickHouseFormattedValue`,
`AddBackTickToFormatTag`), the collector's `JSONSubColumnIndexExpr`, and
aggregation arguments naming a key that contains a backtick (rejected by
the SQL parser, a 500 as before).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Pull Request
---
### 📄 Summary
* Span list (raw) queries can now leverage trace-level `trace.` filter
conditions leading to the trace-level component being qualified with
`__trace_scope`.
* An error is now raised if the span list is ordered by the trace level
key.
The following constraint is known: in case of ordering by `timestamp`,
the querier processes the span list by time bucket thus performing
trace-level aggregates calculation per bucket not within a time window.
The records are kept separately.
#### Issues closed by this PR
Fixes https://github.com/SigNoz/engineering-pod/issues/5976
---
### ✅ Change Type
_Select all that apply_
- [✅ ] ✨ Feature
- [ ] 🐛 Bug fix
- [ ] ♻️ Refactor
- [ ] 🛠️ Infra / Tooling
- [ ] 🧪 Test-only
---
### 🧪 Testing Strategy
> How was this change validated?
- Tests added/updated: ✅
- Manual verification: ✅
- Edge cases covered:
---
### ⚠️ Risk & Impact Assessment
> What could break? How do we recover?
- Blast radius: None
- Potential regressions:
- Rollback plan:
## Pull Request
---
### 📄 Summary
follow-up for #12027. Span-list trace-aggregate filtering ships in
#12122.
Adds `scalar` and `time_series` request types to `builder_ai_query`. The
`trace.` prefix selects the aggregation domain: trace aggregates use a
native CTE pipeline, while span aggregates delegate to the standard
traces builder with the qualification gate applied.
Trace-level filters qualify entire traces across both domains using the
standard filter pipeline. Grouping, `HAVING`, ordering, and limits match
the traces builder, including whole-window ranking for grouped time
series and top-N limits for scalar queries.
`count(trace.trace_id)` counts every AI trace, matching the trace list;
token aggregates average over traces that have token data (standard
`NULL` semantics, same as span-attribute aggregations elsewhere).
Includes SQL golden tests, rewrite unit tests, and integration coverage
for both domains, qualification, grouping, limits, bucketing, variables,
and targeted `400` errors.
#### Issues closed by this PR
Fixes https://github.com/SigNoz/engineering-pod/issues/5602
Fixes https://github.com/SigNoz/engineering-pod/issues/5603
---
### ✅ Change Type
_Select all that apply_
- [x] ✨ Feature
- [ ] 🐛 Bug fix
- [ ] ♻️ Refactor
- [ ] 🛠️ Infra / Tooling
- [ ] 🧪 Test-only
---
### 🧪 Testing Strategy
> How was this change validated?
- Tests added/updated: ✅
- Manual verification:
- Edge cases covered:
---
### ⚠️ Risk & Impact Assessment
> What could break? How do we recover?
- Blast radius: None
- Potential regressions:
- Rollback plan:
---
### 📋 Checklist
- [x] Tests added or explicitly not required
- [x] Manually tested
- [ ] Breaking changes documented
- [ ] Backward compatibility considered
---
## 👀 Notes for Reviewers
Still in testing phase
---
#### Description
- A referenced name in a trace query now resolves to a `LogicalField`
(#12499): one field, addressed by the requested spelling, backed by its
physical member keys. A semantic-convention family
(`deployment.environment.name` / `deployment.environment`) merges into
one expression with current-wins precedence; the response keeps the
requested spelling.
- `FieldMapper` gets one new method, `ExistsFor` (the per-key presence
primitive). `LogicalValueExpr` and `LogicalExistsExpr` build all family
SQL in one place from `FieldFor` and `ExistsFor`; no signal implements
family logic.
- Statement builders prefetch sibling spellings; the metadata store
stays family-blind and autocomplete stays literal. Traces and the
resource filter compile per logical field; logs, metrics, and the other
signals keep their SQL unchanged.
- The `resolve_semconv_families` feature flag (default: disabled) gates
all family behavior. With the flag off, the generated SQL is the same as
main; tests pin this. Part of #6143.
#### Additional Information
- Stack: #12441 (merged) → **#12442** → #12443 → #12444 → #12445 →
#12446 → #12447. This layer bases on main.
- Rollback: turn the flag off; stored telemetry is untouched.
These fired at info on every query build across the audit, logs and traces
builders. The behavior is settled, so drop them to debug and remove the
TODOs that asked for exactly this.
Remove the statementbuilder.Builders aggregate. The querier chain (querier.New,
signozquerier.NewFactory, NewQuerierProviderFactories) now takes the six per-signal
statement builders individually, and signoz.go::newQueryStack returns them directly
via a plain multi-return. The statementbuilder package is now config-only.
Remove each <signal>statementbuilder/new.go and fold its NewFactory into that
package's statement_builder.go (traces' NewOperatorFactory into
trace_operator_statement_builder.go), giving the layout: struct -> NewFactory ->
New<X>QueryStatementBuilder.
Split the five telemetry<signal> packages, which mixed three concerns, into
three per-signal layers with a cycle-free dependency direction:
- telemetryschema/<signal>telemetryschema — primitives (const + table
selection, field_mapper, condition_builder, trace helpers); leaf layer.
- statementbuilder/<signal>statementbuilder — SQL generation. The parent
statementbuilder package is contract-only (the Builders bundle + Config);
each sub-package exposes a factory.ProviderFactory[..., statementbuilder.Config]
whose New internalizes FieldMapper/ConditionBuilder/AggExprRewriter and reads
SkipResourceFingerprint. Traces exposes two factories (query + operator).
telemetryresourcefilter moves here as statementbuilder/resourcefilter.
- telemetrymetadata — key/value resolution; NewTelemetryMetaStore collapses
from 24 args to (settings, telemetrystore, flagger), sourcing table names
from the schema constants.
Centralize query-stack assembly in signoz.go via newQueryStack: build the
single metadata store, run each per-signal statement-builder factory, assemble
the statementbuilder.Builders bundle, and build the bucket cache — once. This
is the only place that imports the concrete sub-packages (so the edge runs
subs -> parent, cycle-free), and it removes the duplicate metadata store that
signozquerier used to build, leaving signozquerier a thin
querier.New(*statementbuilder.Builders) adapter.
Also:
- statementbuilder.Config owns SkipResourceFingerprint (moved off
querier.Config). YAML key moves querier.skip_resource_fingerprint ->
statementbuilder.skip_resource_fingerprint.
- Querier interface moves into querier.go (interfaces.go removed); BucketCache
-> bucket_cache.go, Handler -> api.go.
- Add pkg/querier/queriertest.MockQuerier.
