Compare commits

...

4 Commits

Author SHA1 Message Date
srikanthccv
88a522a638 feat(querybuilder): compile every signal through one storage contract
Each signal had its own copy of name resolution, the guard rule, and the
column renderer, and each copy made its own choices. The semantic
convention families made the differences visible: the same family fold
had to be written seven times, and the guards did not agree.

A storage now answers only what the compiler cannot know: the bare read
of a stored field, its presence test with its absence semantics, the
fallback for a key metadata does not report, its traits, and two
overrides for a storage with its own condition language. Resolution, the
term compiler, and the column renderer live in querybuilder and are
written one time.

- Storage replaces FieldMapper and ConditionBuilder for traces, logs,
  metrics, audit, rule state history, the resource fingerprint, and the
  related-values metadata.
- Absent says how a stored field reads when a row does not carry it,
  and every guard derives from it. A sentinel read takes the exists
  guard on positive operators and in the column stages. A NULL read and
  a table column take none. A declared keyless contract (metrics labels,
  rule state labels) takes none.
- QueryInfo carries the org, the time range, the signal, the queried
  metric, and the query-path flags, evaluated one time per builder stage.
- LogicalField carries ValueMaps, so a family member's stored vocabulary
  translates to the current one in every read.
- The not-found warning fires only when every fallback field is a guess.
  A key under a signal's own context resolves as if it had no context.

Assisted-by: Claude Fable 5.1
Claude-Session: https://claude.ai/code/session_014F37Rwd6xYrr5Abi6JJqJW
2026-09-04 18:31:59 +05:30
srikanthccv
25298585b8 fix(querybuilder): scope map attributes take the default exists guard
Logs and audit store scope attributes in a string map, like attributes, but
exempted the scope context from the default exists guard, so `scope.x = ''`
matched every row without the key. A scope attribute now takes the guard.
scope_name and scope_version are columns and take none.

Assisted-by: Claude Fable 5.1
Claude-Session: https://claude.ai/code/session_014F37Rwd6xYrr5Abi6JJqJW
2026-09-04 10:53:28 +05:30
srikanthccv
608bb9eeee fix(audit): raw select reads NULL for an absent map attribute
Logs and traces render a raw-select map attribute behind its presence test,
so a row without the key shows NULL. Audit rendered the bare map read, which
shows the empty string. Audit now follows the same contract.

Assisted-by: Claude Fable 5.1
Claude-Session: https://claude.ai/code/session_014F37Rwd6xYrr5Abi6JJqJW
2026-09-04 10:53:28 +05:30
srikanthccv
693ea77f90 fix(querybuilder): columns take no guard in the coerced stages
A table column is present in every row. The group-by, order-by, and
aggregation-argument stages wrapped it in its non-empty test, which turned 0
and the empty string into NULL: zero-duration spans left percentiles and
averages, and an empty severity became the NULL group. The stages now read
the column bare. A column among several candidates ends the candidate list,
because no candidate after it can be reached. Traces, logs, and audit.

Assisted-by: Claude Fable 5.1
Claude-Session: https://claude.ai/code/session_014F37Rwd6xYrr5Abi6JJqJW
2026-09-04 10:53:27 +05:30
80 changed files with 2691 additions and 3255 deletions

View File

@@ -54,50 +54,95 @@ The `fieldContexts` map includes aliases (`tag` -> `attribute`, `spanfield` -> `
## The Abstraction Stack
The query pipeline is built from four interfaces that compose vertically. Each layer has a single responsibility. Each layer depends only on the layers below it. This layering is intentional and must be preserved.
The query pipeline has three layers. The generic layer is written one time, in `pkg/querybuilder`. A storage is written one time per signal. The statement builders compose them. Each layer depends only on the layer below it. This layering is intentional and must be preserved.
```
StatementBuilder <- Orchestrates everything into executable SQL
├── AggExprRewriter <- Rewrites aggregation expressions (maps field refs to columns)
├── ConditionBuilder <- Builds WHERE predicates (field + operator + value -> SQL)
└── FieldMapper <- Maps TelemetryFieldKey -> ClickHouse column expression
StatementBuilder <- Composes one query into executable SQL
├── AggExprRewriter <- Rewrites aggregation expressions through the generic layer
├── filter visitor <- Parses the filter expression and compiles it one term at a time
└── querybuilder (generic) <- Resolution, the condition builder, the column expression builder
└── Storage <- What one signal's tables can answer about one field key
```
### FieldMapper
### Storage
**Contract:** Given a `TelemetryFieldKey`, return a ClickHouse column expression that yields the value for that field when used in a SELECT.
**Contract:** `qbtypes.Storage` in `pkg/types/querybuildertypes/querybuildertypesv5/qb.go`. One implementation per signal: traces, logs, metrics, audit, rule state history, the resource fingerprint sub-query, and the related-values metadata.
**Principle:** This is the *only* place where field-to-column translation happens. No other layer should contain knowledge of how fields map to storage. If you need a column expression, go through the FieldMapper.
A storage answers five questions and nothing else:
**Why:** The user says `http.request.method`. ClickHouse might store it as `attributes_string['http.request.method']`, or as a materialized column `` `attribute_string_http$$request$$method` ``, or via a JSON access path in a body column. This variation is entirely contained within the FieldMapper. Everything above it is storage-agnostic.
- `Read(field)`: the bare SQL read of one field key. No alias, no guard, no cast. It honors the materialization and the evolutions the field carries.
- `Exists(field, exists)`: the presence test of one field key, and how the field reads when a row lacks it (`Absent`, below).
- `Fallback(key, operator, value)`: the field keys that could hold a key metadata does not report: column aliases, the type variants of a map read, body paths, and virtual keys that compile to structural predicates (a span search scope, a full-text search over a scope).
- `Traits()`: the storage's part in the resource fingerprint split, whether it supports body functions, what it does with an unknown key, and which contexts mean "this signal's own record".
- `Compile` and `ColumnRead`: two overrides for a storage with its own condition language (the body JSON language in logs, the index hints of the resource fingerprint, the polarity form of the related values, the String-typed labels of metrics). Every other storage returns `querybuilder.SharedCondition` and `querybuilder.DefaultRead`.
### ConditionBuilder
**Principle:** A storage describes its field keys. It never decides a guard, an ambiguity, a warning, or the shape of a fold. Those decisions are derived one time, in the generic layer, from those descriptions.
**Contract:** Given a field key, an operator, and a value, produce a valid SQL predicate for a WHERE clause.
### Absent
**Dependency:** Uses FieldMapper for the left-hand side of the condition.
`Exists` returns the field's `Absent`: what a row without the field reads. It is a property of the read, not of the column. `resource.x::String` reads the empty string for an absent row, the multi-era fold `multiIf(..., NULL)` reads NULL, and a table column always reads a real value. Every guard derives from it:
**Principle:** The ConditionBuilder owns all the complexity of operator semantics, i.e type casting, array operators (`hasAny`/`hasAll` vs `=`), existence checks, and negative operator behavior. This complexity must not leak upward into the StatementBuilder.
| WhenAbsent | Absent row reads | Positive filter | Raw select | Multi-candidate column | Field keys |
|---|---|---|---|---|---|
| `AlwaysPresent` | a real value | no guard | no guard | no branch, ends the candidate list | table columns |
| `AbsentIsSentinel` | `''`, 0, false, and that is not a value | exists guard | exists guard | presence branch | map attributes, cast JSON paths, string families |
| `AbsentIsNull` | NULL | no guard | no guard | presence branch | multi-era folds, body JSON paths, numeric families |
| `AbsentIsValue` | `''`, and that is the keyless contract | no guard | no guard | no presence branch | metrics labels, rule state history labels |
### AggExprRewriter
### The generic layer
**Contract:** Given a user-facing aggregation expression like `sum(duration_nano)`, resolve field references within it and produce valid ClickHouse SQL.
`pkg/querybuilder` needs two inputs, made one time per request:
**Dependency:** Uses FieldMapper to resolve field names within expressions.
- The metadata keys: `keys := metadataStore.GetKeysMulti(...)`, the field keys the metadata store reports for the query's names, as `map[name][]*TelemetryFieldKey`.
- `q := querybuilder.NewQueryInfo(ctx, orgID, fl, signal, metric, startNs, endNs)`: the org and time range every read needs, the signal and the queried metric that family admission needs, and the query-path flags (`FamiliesOn`, `BodyJSONOn`), evaluated one time.
**Principle:** Aggregation expressions are user-authored strings that contain field references. The rewriter parses them, identifies field references, resolves each through the FieldMapper, and reassembles the expression.
The functions, from the outside in:
### StatementBuilder
| Function | Does |
|---|---|
| `PrepareWhereClause(query, opts)` | The filter visitor. Parses the filter grammar and compiles each term through `RejectsBodyFunction`, `Resolve`, and `Condition`. Returns the WHERE clause, the warnings, and the cost-guard flag. |
| `NewAggExprRewriter(settings, fullTextColumn, storage, fl, signal)` | Parses an aggregation expression such as `sum(duration_nano)` and resolves each field reference through `ResolveColumn`. |
| `ResolveColumn(ctx, q, storage, key, target, metadata)` | `Resolve` with `FilterOperatorUnknown`, then `Column`. The column stages (raw select, order by, group by, aggregation arguments) call it. |
| `Resolve(ctx, q, storage, key, operator, value, metadata)` | One requested key to its meanings in this storage (see "A resolved key"). |
| `Condition(ctx, q, storage, resolved, dropResourceFields, operator, value, sb)` | A resolved key to the conditions of one filter term: the split narrows the fields, and each field compiles through `storage.Compile`. |
| `Conditions(...)` | `RejectsBodyFunction`, `Resolve`, and `Condition` in one call, for callers outside the visitor: the related-values metadata, the scoped traces predicate resolver, tests. |
| `Column(ctx, q, storage, resolved, target)` | A resolved key to one bare column expression. The caller aliases. |
| `RejectsBodyFunction(traits, operator)` | Before resolution: a storage without body functions (`has`, `hasAny`, `hasAll`, `hasToken`, `search`) errors; the fingerprint side of a split skips the term, because the main query evaluates it. |
| `SharedCondition(...)` | The `Compile` of every storage without its own condition language: `LogicalValueExpr`, the shared data-type collision cast, `OperatorCondition`, then the guard rule. |
| `OperatorCondition(...)` | The operator switch over an already cast read. A storage with its own cast policy composes with it. |
| `DefaultRead(...)` | The `ColumnRead` of every storage without a target-dependent read: `LogicalValueExpr` as an uncoerced column expression. |
| `LogicalValueExpr(...)`, `LogicalExistsExpr(...)` | The only place family expressions are built. A single-member field reads through its member. A family merges the member reads current-first (`COALESCE(NULLIF(m1, ''), NULLIF(m2, ''), '')` for strings, `multiIf` with a NULL tail for numbers) and ORs the member presence tests. A member with a value map reads through `TransformRead`. |
**Contract:** Given a complete `QueryBuilderQuery`, a time range, and a request type, produces an executable SQL statement.
### A resolved key
**Dependency:** Uses all three abstractions above.
`Resolve` turns one requested key into a `Resolved` value. It is the only thing the condition builder and the column expression builder receive. Compile it with the operator and value it was resolved with: the stage is the operand, and a nil value means a column or presence use.
**Principle:** This is the composition layer. It does not contain field mapping logic, condition building logic, or expression rewriting logic. It orchestrates the other abstractions. If you find storage-specific logic creeping into the StatementBuilder, push it down into the appropriate abstraction.
```go
type Resolved struct {
Key *TelemetryFieldKey // the spelling the request used
Fields []*LogicalField // its meanings in this storage, one per interpretation
FromFallback bool // the fields came from the storage's Fallback, not from metadata matches
Ambiguous bool // the matches held several interpretations
Skipped bool // the storage contributes nothing for this key
Warnings []string // the warnings to surface: ambiguity, not-found
}
```
A `LogicalField` is one meaning: one name, context, and data type, backed by one or more physical members. A family (one field with several spellings) is one logical field with several members, current spelling first. Ambiguity (one name, different fields) is several logical fields.
The resolution order is the same for every storage and every stage:
1. **Own context.** A key under one of the storage's own contexts (`span.x`, `log.x`) looks up as if it had no context. Strict contexts (`resource.`, `attribute.`, `scope.`, `body.`) are honored as written.
2. **Matches.** The metadata keys under the key's spellings, grouped into families when the flag is on. Each combination of context and data type is one interpretation.
3. **Ambiguity.** A filter settles several interpretations by resource over attribute, with a warning. A column stage keeps every interpretation in metadata order and folds them, so a select or a group by sees the value wherever it is.
4. **Intrinsic column first**, bare keys only. A column every row has leads the list, whether metadata reports it or the storage's `Fallback` does. Sentinel-reading fields with a contradicting data type drop. A metadata gap degrades to the correct column, never to a corrupt metadata key.
5. **Fallback.** With no match, the storage's fallback keys for the key. When the storage ignores unknown keys (a side query whose main query owns the error), the key is `Skipped`. Otherwise a key nothing can serve is an error with suggestions. The not-found warning fires only when every fallback key is a guess, that is, none of them is always present.
The condition builder then applies the fingerprint split (`MainOfSplit` drops the resource fields the sub-query serves and keeps fallback keys; `FingerprintOfSplit` keeps resource fields only), compiles each field, and the visitor joins the per-field conditions by the operator's polarity. The column expression builder reads each field through `ColumnRead`, casts for the coerced stages unless the read keeps its type, guards by `Absent`, and renders one candidate bare or several as `multiIf(..., NULL)`. A candidate that is not selectable (`ErrNotSelectable`) drops; the error surfaces only when none remains.
### Invariant: No layer skipping
The StatementBuilder must not call FieldMapper directly to build conditions, it goes through the ConditionBuilder. The AggExprRewriter must not hardcode column names, it goes through the FieldMapper. Skipping layers creates hidden coupling and makes the system fragile to storage changes.
A statement builder must not spell a column or a condition. It calls `ResolveColumn` and the filter visitor. A storage must not decide a guard or an ambiguity. It declares `Absent` and answers the five questions. Skipping layers recreates the per-signal copies the contract removed.
---
@@ -119,14 +164,15 @@ Only additive/counting aggregations (`count`, `count_distinct`, `sum`, `rate`) d
**Enforcement:** `GetQueriesSupportingZeroDefault` determines which queries can default to zero. The `FormulaEvaluator` consumes this via `canDefaultZero`. Changes to aggregation handling must preserve this distinction.
### Constraint: Existence semantics differ for positive vs negative operators
### Constraint: The exists guard derives from the operator and from the field
- **Positive operators** (`=`, `>`, `LIKE`, `IN`, etc.) implicitly assert field existence. `http.method = GET` means "the field exists AND equals GET".
- **Negative operators** (`!=`, `NOT IN`, `NOT LIKE`, etc.) do **not** add an existence check. `http.method != GET` includes records where the field doesn't exist at all.
- **Positive operators** (`=`, `>`, `LIKE`, `IN`, etc.) implicitly assert field existence for a field that reads a sentinel when absent. `http.method = GET` on a map attribute means "the field exists AND equals GET".
- **Negative operators** (`!=`, `NOT IN`, `NOT LIKE`, etc.) never add an existence check. `http.method != GET` includes records where the field doesn't exist at all.
- A field that reads NULL when absent, and a table column, take no guard on any operator: the comparison already excludes the absent row, or there is no absent row.
**Why:** The user's intent with negative operators is ambiguous. Rather than guess, we take the broader interpretation. Users can add an explicit `EXISTS` filter if they want the narrower one. This is documented in `AddDefaultExistsFilter`.
**Why:** The user's intent with negative operators is ambiguous. Rather than guess, we take the broader interpretation. Users can add an explicit `EXISTS` filter if they want the narrower one. The operator side is declared in `AddDefaultExistsFilter`; the field side is the `Absent` a storage returns from `Exists`.
**Consequence:** Any new operator must declare its existence behavior in `AddDefaultExistsFilter`. Do not add operators without considering this.
**Consequence:** Any new operator must declare its existence behavior in `AddDefaultExistsFilter`. Any new read must declare what an absent row reads. Never add a guard by hand in a storage.
### Constraint: Post-processing functions operate on result sets, not in SQL
@@ -188,11 +234,11 @@ The `MetadataStore` interface provides runtime field discovery and type resoluti
The same name can map to multiple `TelemetryFieldKey` variants (different contexts, different types). The metadata store returns *all* variants. Resolution to a single field happens during query building, using the query's signal and any explicit context/type hints from the user.
**Consequence:** Code that calls `GetKey` or `GetKeys` must handle multiple results. Do not assume a name maps to a single field.
**Consequence:** Code that calls `GetKey` or `GetKeys` must handle multiple results. Do not assume a name maps to a single field. `querybuilder.Resolve` is where the variants settle: it returns every interpretation as a `LogicalField`, marks the result `Ambiguous`, and carries the warning.
### Principle: Materialized fields are a performance optimization, not a semantic distinction
A materialized field and its non-materialized equivalent represent the same logical field. The `Materialized` flag tells the FieldMapper to generate a simpler column expression. The user should never need to know whether a field is materialized.
A materialized field and its non-materialized equivalent represent the same logical field. The `Materialized` flag tells the storage's `Read` to generate a simpler column expression. The user should never need to know whether a field is materialized.
### Principle: JSON body fields require access plans
@@ -203,14 +249,14 @@ Fields inside JSON body columns (`body.response.errors[].code`) need pre-compute
## Summary of Inviolable Rules
1. **User-facing types never contain ClickHouse column names or SQL fragments.**
2. **Field-to-column translation only happens in FieldMapper.**
2. **Field-to-column translation only happens in a Storage (`Read`, `Exists`, `Fallback`).**
3. **Normalization happens once at the API boundary, never deeper.**
4. **Historical aliases in fieldContexts and fieldDataTypes must not be removed.**
5. **Formula evaluation stays in Go — do not push it into ClickHouse JOINs.**
6. **Zero-defaulting is aggregation-type-dependent — do not universally default to zero.**
7. **Positive operators imply existence, negative operators do not.**
7. **The exists guard derives from `AddDefaultExistsFilter` and `Absent`; positive operators guard sentinel reads, negative operators never guard.**
8. **Post-processing functions operate on Go result sets, not in SQL.**
9. **All user-facing types reject unknown JSON fields with suggestions.**
10. **Validation rules are gated by request type.**
11. **Query names must be unique within a composite query.**
12. **The four-layer abstraction stack (FieldMapper -> ConditionBuilder -> AggExprRewriter -> StatementBuilder) must not be bypassed or flattened.**
12. **The three-layer abstraction stack (Storage -> querybuilder generic layer -> StatementBuilder) must not be bypassed or flattened. A storage describes its field keys; the generic layer decides.**

View File

@@ -446,14 +446,12 @@ func (m *module) buildFilterClause(ctx context.Context, orgID valuer.UUID, filte
}
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
Logger: m.logger,
FieldMapper: m.fieldMapper,
ConditionBuilder: m.condBuilder,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "metric_name", FieldContext: telemetrytypes.FieldContextMetric},
FieldKeys: keys,
StartNs: querybuilder.ToNanoSecs(uint64(startMillis)),
EndNs: querybuilder.ToNanoSecs(uint64(endMillis)),
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, orgID, m.fl, telemetrytypes.SignalMetrics, nil, querybuilder.ToNanoSecs(uint64(startMillis)), querybuilder.ToNanoSecs(uint64(endMillis))),
Storage: m.storage,
Logger: m.logger,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "metric_name", FieldContext: telemetrytypes.FieldContextMetric},
FieldKeys: keys,
}
whereClause, err := querybuilder.PrepareWhereClause(expression, opts)

View File

@@ -24,8 +24,7 @@ type module struct {
telemetryStore telemetrystore.TelemetryStore
telemetryMetadataStore telemetrytypes.MetadataStore
querier querier.Querier
fieldMapper qbtypes.FieldMapper
condBuilder qbtypes.ConditionBuilder
storage qbtypes.Storage
logger *slog.Logger
config inframonitoring.Config
fl flagger.Flagger
@@ -40,14 +39,11 @@ func NewModule(
providerSettings factory.ProviderSettings,
cfg inframonitoring.Config,
) inframonitoring.Module {
fieldMapper := metricstelemetryschema.NewFieldMapper()
condBuilder := metricstelemetryschema.NewConditionBuilder(fieldMapper)
return &module{
telemetryStore: telemetryStore,
telemetryMetadataStore: telemetryMetadataStore,
querier: querier,
fieldMapper: fieldMapper,
condBuilder: condBuilder,
storage: metricstelemetryschema.NewStorage(),
logger: providerSettings.Logger,
config: cfg,
fl: fl,

View File

@@ -37,8 +37,7 @@ import (
type module struct {
telemetryStore telemetrystore.TelemetryStore
telemetryMetadataStore telemetrytypes.MetadataStore
fieldMapper qbtypes.FieldMapper
condBuilder qbtypes.ConditionBuilder
storage qbtypes.Storage
logger *slog.Logger
cache cache.Cache
ruleStore ruletypes.RuleStore
@@ -49,12 +48,9 @@ type module struct {
// NewModule constructs the metrics module with the provided dependencies.
func NewModule(ts telemetrystore.TelemetryStore, telemetryMetadataStore telemetrytypes.MetadataStore, cache cache.Cache, ruleStore ruletypes.RuleStore, dashboardModule dashboard.Module, fl flagger.Flagger, providerSettings factory.ProviderSettings, cfg metricsexplorer.Config) metricsexplorer.Module {
fieldMapper := metricstelemetryschema.NewFieldMapper()
condBuilder := metricstelemetryschema.NewConditionBuilder(fieldMapper)
return &module{
telemetryStore: ts,
fieldMapper: fieldMapper,
condBuilder: condBuilder,
storage: metricstelemetryschema.NewStorage(),
logger: providerSettings.Logger,
telemetryMetadataStore: telemetryMetadataStore,
cache: cache,
@@ -975,14 +971,12 @@ func (m *module) buildFilterClause(ctx context.Context, orgID valuer.UUID, filte
}
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
Logger: m.logger,
FieldMapper: m.fieldMapper,
ConditionBuilder: m.condBuilder,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "metric_name", FieldContext: telemetrytypes.FieldContextMetric},
FieldKeys: keys,
StartNs: querybuilder.ToNanoSecs(uint64(startMillis)),
EndNs: querybuilder.ToNanoSecs(uint64(endMillis)),
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, orgID, m.fl, telemetrytypes.SignalMetrics, nil, querybuilder.ToNanoSecs(uint64(startMillis)), querybuilder.ToNanoSecs(uint64(endMillis))),
Storage: m.storage,
Logger: m.logger,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "metric_name", FieldContext: telemetrytypes.FieldContextMetric},
FieldKeys: keys,
}
whereClause, err := querybuilder.PrepareWhereClause(expression, opts)

View File

@@ -1,150 +0,0 @@
package implrulestatehistory
import (
"context"
"fmt"
"slices"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
type conditionBuilder struct {
fm qbtypes.FieldMapper
}
func newConditionBuilder(fm qbtypes.FieldMapper) qbtypes.ConditionBuilder {
return &conditionBuilder{fm: fm}
}
// Rule state history has no resource sub-query, so options are unused.
func (c *conditionBuilder) ConditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for rule state history.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}
// Rule state history fields have no family support, so every logical field
// is single-member and flattens losslessly to its physical key.
resolved, warning := querybuilder.ResolveLogicalFields(key, querybuilder.MatchingLogicalFields(ctx, orgID, nil, key, fieldKeys))
keys := querybuilder.SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
if len(keys) == 0 {
return nil, warnings, querybuilder.NewKeyNotFoundError(key.Name)
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
cond, err := c.conditionForKey(ctx, orgID, startNs, endNs, k, operator, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
}
return conds, warnings, nil
}
func (c *conditionBuilder) conditionForKey(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
if operator.IsStringSearchOperator() {
value = querybuilder.FormatValueForContains(value)
}
fieldName, err := c.fm.FieldFor(ctx, orgID, startNs, endNs, key)
if err != nil {
return "", err
}
switch operator {
case qbtypes.FilterOperatorEqual:
return sb.E(fieldName, value), nil
case qbtypes.FilterOperatorNotEqual:
return sb.NE(fieldName, value), nil
case qbtypes.FilterOperatorGreaterThan:
return sb.G(fieldName, value), nil
case qbtypes.FilterOperatorGreaterThanOrEq:
return sb.GE(fieldName, value), nil
case qbtypes.FilterOperatorLessThan:
return sb.LT(fieldName, value), nil
case qbtypes.FilterOperatorLessThanOrEq:
return sb.LE(fieldName, value), nil
case qbtypes.FilterOperatorLike:
return sb.Like(fieldName, value), nil
case qbtypes.FilterOperatorNotLike:
return sb.NotLike(fieldName, value), nil
case qbtypes.FilterOperatorILike:
return sb.ILike(fieldName, value), nil
case qbtypes.FilterOperatorNotILike:
return sb.NotILike(fieldName, value), nil
case qbtypes.FilterOperatorContains:
return sb.ILike(fieldName, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorNotContains:
return sb.NotILike(fieldName, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorRegexp:
return fmt.Sprintf(`match(%s, %s)`, sqlbuilder.Escape(fieldName), sb.Var(value)), nil
case qbtypes.FilterOperatorNotRegexp:
return fmt.Sprintf(`NOT match(%s, %s)`, sqlbuilder.Escape(fieldName), sb.Var(value)), nil
case qbtypes.FilterOperatorBetween:
values, ok := value.([]any)
if !ok || len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.Between(fieldName, values[0], values[1]), nil
case qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok || len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.NotBetween(fieldName, values[0], values[1]), nil
case qbtypes.FilterOperatorIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
return sb.In(fieldName, values), nil
case qbtypes.FilterOperatorNotIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
return sb.NotIn(fieldName, values), nil
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
intrinsic := []string{"rule_id", "rule_name", "overall_state", "overall_state_changed", "state", "state_changed", "unix_milli", "fingerprint", "value"}
if slices.Contains(intrinsic, key.Name) {
return "true", nil
}
if operator == qbtypes.FilterOperatorExists {
return fmt.Sprintf("JSONHas(labels, %s)", sb.Var(key.Name)), nil
}
return fmt.Sprintf("not JSONHas(labels, %s)", sb.Var(key.Name)), nil
}
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported operator: %v", operator)
}

View File

@@ -9,7 +9,6 @@ import (
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
@@ -26,19 +25,15 @@ var ruleStateHistoryColumns = map[string]*schema.Column{
"value": {Name: "value", Type: schema.ColumnTypeFloat64},
}
type fieldMapper struct{}
type storage struct{}
func newFieldMapper() qbtypes.FieldMapper {
return &fieldMapper{}
var _ qbtypes.Storage = (*storage)(nil)
func newStorage() *storage {
return &storage{}
}
// CandidateKeys returns nil: rule-state history has no attribute-map fallback, so a
// context-missing key stays unresolved and the caller errors.
func (m *fieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, _ *telemetrytypes.TelemetryFieldKey, _ any, _ map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
}
func (m *fieldMapper) getColumn(_ context.Context, key *telemetrytypes.TelemetryFieldKey) (*schema.Column, error) { //nolint:unparam
func (m *storage) getColumn(_ context.Context, key *telemetrytypes.TelemetryFieldKey) (*schema.Column, error) { //nolint:unparam
name := strings.TrimSpace(key.Name)
if col, ok := ruleStateHistoryColumns[name]; ok {
return col, nil
@@ -46,7 +41,7 @@ func (m *fieldMapper) getColumn(_ context.Context, key *telemetrytypes.Telemetry
return ruleStateHistoryColumns["labels"], nil
}
func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
func (m *storage) Read(ctx context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
col, err := m.getColumn(ctx, key)
if err != nil {
return "", err
@@ -57,36 +52,39 @@ func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, _, _ uint64,
return col.Name, nil
}
func (m *fieldMapper) ColumnFor(ctx context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
// Exists answers a label inside the JSON with a membership check, with the
// same condition that Read uses for extraction. Absent, a label reads the
// empty string, and that is the keyless contract here, so no query guards
// it; every real column always exists.
func (m *storage) Exists(ctx context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, exists bool) (qbtypes.Existence, error) {
col, err := m.getColumn(ctx, key)
if err != nil {
return nil, err
}
return []*schema.Column{col}, nil
}
// ExistsFor implements the per-key existence primitive of qbtypes.FieldMapper.
// A label inside the JSON gets a membership check, with the same condition
// that FieldFor uses for extraction; every real column always exists.
func (m *fieldMapper) ExistsFor(ctx context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey, exists bool) (string, error) {
col, err := m.getColumn(ctx, key)
if err != nil {
return "", err
return qbtypes.Existence{}, err
}
if col.Name == "labels" && key.Name != "labels" {
pred := fmt.Sprintf("JSONHas(labels, %s)", querybuilder.ClickHouseStringLiteral(key.Name))
if exists {
return pred, nil
predicate := fmt.Sprintf("JSONHas(labels, %s)", querybuilder.ClickHouseStringLiteral(key.Name))
if !exists {
predicate = "not " + predicate
}
return "not " + pred, nil
return qbtypes.Existence{Predicate: predicate, WhenAbsent: qbtypes.AbsentIsValue}, nil
}
return "true", nil
return qbtypes.Existence{Predicate: "true", WhenAbsent: qbtypes.AlwaysPresent}, nil
}
func (m *fieldMapper) ColumnExpressionFor(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, field *telemetrytypes.TelemetryFieldKey, _ telemetrytypes.FieldDataType, _ map[string][]*telemetrytypes.TelemetryFieldKey) (string, error) {
colName, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, field)
if err != nil {
return "", err
}
return fmt.Sprintf("%s AS `%s`", sqlbuilder.Escape(colName), field.Name), nil
// Fallback returns nil: rule-state history has no attribute-map fallback, so
// a key metadata does not hold stays unresolved and the caller errors.
func (m *storage) Fallback(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, qbtypes.FilterOperator, any) ([]*telemetrytypes.LogicalField, error) {
return nil, nil
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{}
}
func (m *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
return querybuilder.SharedCondition(ctx, q, m, logical, operator, value, sb)
}
func (m *storage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
return querybuilder.DefaultRead(ctx, q, m, logical)
}

View File

@@ -25,18 +25,15 @@ const (
type store struct {
telemetryStore telemetrystore.TelemetryStore
telemetryMetadataStore telemetrytypes.MetadataStore
fieldMapper qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
storage qbtypes.Storage
logger *slog.Logger
}
func NewStore(telemetryStore telemetrystore.TelemetryStore, telemetryMetadataStore telemetrytypes.MetadataStore, logger *slog.Logger) rulestatehistorytypes.Store {
fm := newFieldMapper()
return &store{
telemetryStore: telemetryStore,
telemetryMetadataStore: telemetryMetadataStore,
fieldMapper: fm,
conditionBuilder: newConditionBuilder(fm),
storage: newStorage(),
logger: logger,
}
}
@@ -500,15 +497,13 @@ func (s *store) buildFilterClause(ctx context.Context, orgID valuer.UUID, filter
}
opts := querybuilder.FilterExprVisitorOpts{
Logger: s.logger,
FieldMapper: s.fieldMapper,
ConditionBuilder: s.conditionBuilder,
FieldKeys: fieldKeys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels", FieldContext: telemetrytypes.FieldContextAttribute},
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, orgID, nil, telemetrytypes.SignalUnspecified, nil, querybuilder.ToNanoSecs(uint64(startMillis)), querybuilder.ToNanoSecs(uint64(endMillis))),
Storage: s.storage,
Logger: s.logger,
FieldKeys: fieldKeys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels", FieldContext: telemetrytypes.FieldContextAttribute},
}
opts.StartNs = querybuilder.ToNanoSecs(uint64(startMillis))
opts.EndNs = querybuilder.ToNanoSecs(uint64(endMillis))
prepared, err := querybuilder.PrepareWhereClause(expression, opts)
if err != nil {
return nil, err

View File

@@ -17,11 +17,11 @@ import (
)
type aggExprRewriter struct {
logger *slog.Logger
fullTextColumn *telemetrytypes.TelemetryFieldKey
fieldMapper qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
flagger flagger.Flagger
logger *slog.Logger
fullTextColumn *telemetrytypes.TelemetryFieldKey
storage qbtypes.Storage
flagger flagger.Flagger
signal telemetrytypes.Signal
}
var _ qbtypes.AggExprRewriter = (*aggExprRewriter)(nil)
@@ -29,18 +29,18 @@ var _ qbtypes.AggExprRewriter = (*aggExprRewriter)(nil)
func NewAggExprRewriter(
settings factory.ProviderSettings,
fullTextColumn *telemetrytypes.TelemetryFieldKey,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
storage qbtypes.Storage,
fl flagger.Flagger,
signal telemetrytypes.Signal,
) *aggExprRewriter {
set := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/querybuilder/agg_rewrite")
return &aggExprRewriter{
logger: set.Logger(),
fullTextColumn: fullTextColumn,
fieldMapper: fieldMapper,
conditionBuilder: conditionBuilder,
flagger: fl,
logger: set.Logger(),
fullTextColumn: fullTextColumn,
storage: storage,
flagger: fl,
signal: signal,
}
}
@@ -78,18 +78,14 @@ func (r *aggExprRewriter) Rewrite(
return "", nil, errors.NewInternalf(errors.CodeInternal, "no SELECT items for %q", expr)
}
visitor := newExprVisitor(
ctx,
orgID,
startNs,
endNs,
r.logger,
keys,
r.fullTextColumn,
r.fieldMapper,
r.conditionBuilder,
r.flagger,
)
visitor := &exprVisitor{
ctx: ctx,
query: NewQueryInfo(ctx, orgID, r.flagger, r.signal, nil, startNs, endNs),
logger: r.logger,
fieldKeys: keys,
fullTextColumn: r.fullTextColumn,
storage: r.storage,
}
// Rewrite the first select item (our expression)
if err := sel.SelectItems[0].Accept(visitor); err != nil {
return "", nil, err
@@ -130,48 +126,19 @@ func (r *aggExprRewriter) RewriteMulti(
return out, chArgsList, nil
}
// exprVisitor walks FunctionExpr nodes and applies the mappers.
// exprVisitor walks FunctionExpr nodes and resolves and renders their
// arguments.
type exprVisitor struct {
ctx context.Context
orgID valuer.UUID
startNs uint64
endNs uint64
ctx context.Context
query qbtypes.QueryInfo
chparser.DefaultASTVisitor
logger *slog.Logger
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey
fullTextColumn *telemetrytypes.TelemetryFieldKey
fieldMapper qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
flagger flagger.Flagger
Modified bool
chArgs []any
isRate bool
}
func newExprVisitor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
logger *slog.Logger,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
fullTextColumn *telemetrytypes.TelemetryFieldKey,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
fl flagger.Flagger,
) *exprVisitor {
return &exprVisitor{
ctx: ctx,
orgID: orgID,
startNs: startNs,
endNs: endNs,
logger: logger,
fieldKeys: fieldKeys,
fullTextColumn: fullTextColumn,
fieldMapper: fieldMapper,
conditionBuilder: conditionBuilder,
flagger: fl,
}
logger *slog.Logger
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey
fullTextColumn *telemetrytypes.TelemetryFieldKey
storage qbtypes.Storage
Modified bool
chArgs []any
isRate bool
}
// VisitFunctionExpr is invoked for each function call in the AST.
@@ -211,15 +178,12 @@ func (v *exprVisitor) VisitFunctionExpr(fn *chparser.FunctionExpr) error {
whereClause, err := PrepareWhereClause(
origPred,
FilterExprVisitorOpts{
Context: v.ctx,
OrgID: v.orgID,
Logger: v.logger,
FieldKeys: v.fieldKeys,
FieldMapper: v.fieldMapper,
ConditionBuilder: v.conditionBuilder,
FullTextColumn: v.fullTextColumn,
StartNs: v.startNs,
EndNs: v.endNs,
Context: v.ctx,
Query: v.query,
Storage: v.storage,
Logger: v.logger,
FieldKeys: v.fieldKeys,
FullTextColumn: v.fullTextColumn,
},
)
if err != nil {
@@ -244,7 +208,7 @@ func (v *exprVisitor) VisitFunctionExpr(fn *chparser.FunctionExpr) error {
for i := 0; i < len(args)-1; i++ {
origVal := chparser.Format(args[i])
fieldKey := telemetrytypes.GetFieldKeyFromKeyText(origVal)
expr, err := v.fieldMapper.ColumnExpressionFor(v.ctx, v.orgID, v.startNs, v.endNs, &fieldKey, dataType, v.fieldKeys)
expr, err := ResolveColumn(v.ctx, v.query, v.storage, &fieldKey, dataType, v.fieldKeys)
if err != nil {
return errors.WrapInvalidInputf(err, errors.CodeInvalidInput, "failed to get table field name for %q", origVal)
}
@@ -261,7 +225,7 @@ func (v *exprVisitor) VisitFunctionExpr(fn *chparser.FunctionExpr) error {
for i, arg := range args {
orig := chparser.Format(arg)
fieldKey := telemetrytypes.GetFieldKeyFromKeyText(orig)
expr, err := v.fieldMapper.ColumnExpressionFor(v.ctx, v.orgID, v.startNs, v.endNs, &fieldKey, dataType, v.fieldKeys)
expr, err := ResolveColumn(v.ctx, v.query, v.storage, &fieldKey, dataType, v.fieldKeys)
if err != nil {
return err
}

102
pkg/querybuilder/column.go Normal file
View File

@@ -0,0 +1,102 @@
package querybuilder
import (
"context"
"fmt"
"strings"
"github.com/SigNoz/signoz/pkg/errors"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// Column renders a resolved key as one bare column expression; the caller
// aliases. The coerced stages (group by, order, aggregation arguments) cast
// every candidate to the target type. The guard follows Absent: a sentinel
// field reads behind its presence test, so an absent row yields NULL; a column
// every row has reads bare and ends the candidate list; a NULL-reading field
// takes a presence branch only beside other candidates.
func Column(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
resolved qbtypes.Resolved,
target telemetrytypes.FieldDataType,
) (string, error) {
if len(resolved.Fields) == 0 {
return "", NewKeyNotFoundError(resolved.Key.Name, nil)
}
var targetValue any = ""
if target == telemetrytypes.FieldDataTypeFloat64 {
targetValue = 0.0
}
coerced := target != telemetrytypes.FieldDataTypeUnspecified
several := len(resolved.Fields) > 1
branches := make([]string, 0, len(resolved.Fields)*2)
var unreadable error
for _, logical := range resolved.Fields {
read, err := storage.ColumnRead(ctx, q, logical, targetValue)
if errors.Is(err, qbtypes.ErrNotSelectable) {
unreadable = err
continue
}
if err != nil {
return "", err
}
// an array cannot sit inside Nullable or multiIf
if !several && bareRead(logical) {
return read.SQL, nil
}
expr := read.SQL
if coerced && !read.KeepType {
expr, _ = DataTypeCollisionHandledFieldName(logical.Single(), targetValue, expr, qbtypes.FilterOperatorUnknown)
}
// several native shapes share one multiIf, so every branch reads as text
branch := expr
if !coerced && several {
branch, _ = DataTypeCollisionHandledFieldName(logical.Single(), "", expr, qbtypes.FilterOperatorUnknown)
}
existence, err := LogicalExistsExpr(ctx, q, storage, logical, true)
if err != nil {
return "", err
}
switch existence.WhenAbsent {
case qbtypes.AlwaysPresent, qbtypes.AbsentIsValue:
if len(branches) == 0 {
return expr, nil
}
return fmt.Sprintf("multiIf(%s, %s)", strings.Join(branches, ", "), branch), nil
case qbtypes.AbsentIsNull:
if !several {
return expr, nil
}
branches = append(branches, existence.Predicate, branch)
default:
branches = append(branches, existence.Predicate, branch)
}
}
if len(branches) == 0 {
return "", unreadable
}
return fmt.Sprintf("multiIf(%s, NULL)", strings.Join(branches, ", ")), nil
}
func bareRead(logical *telemetrytypes.LogicalField) bool {
key := logical.Single()
return strings.Contains(key.Name, telemetrytypes.ArraySep) ||
strings.Contains(key.Name, telemetrytypes.ArrayAnyIndex) ||
key.FieldDataType.IsArray()
}
// DefaultRead is the ColumnRead every storage without a target-dependent
// read answers with: the merged read for a family, the member's own read
// otherwise.
func DefaultRead(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, logical *telemetrytypes.LogicalField) (qbtypes.ColumnExpr, error) {
expr, err := LogicalValueExpr(ctx, q, storage, logical)
if err != nil {
return qbtypes.ColumnExpr{}, err
}
return qbtypes.ColumnExpr{SQL: expr}, nil
}

View File

@@ -0,0 +1,112 @@
package querybuilder
import (
"context"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
)
// Condition compiles a resolved key into the conditions of one filter term:
// the storage's part in the fingerprint split narrows the fields, and every
// field compiles through the storage's Compile. It returns the per-field
// warnings; the resolution carries its own.
func Condition(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
resolved qbtypes.Resolved,
dropResourceFields bool,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
if resolved.Skipped {
return nil, nil, nil
}
fields := resolved.Fields
switch storage.Traits().Split {
case qbtypes.MainOfSplit:
// the sub-query cannot know fallback keys, so those stay
if dropResourceFields && !resolved.FromFallback {
filtered := make([]*telemetrytypes.LogicalField, 0, len(fields))
for _, logical := range fields {
if logical.FieldContext != telemetrytypes.FieldContextResource {
filtered = append(filtered, logical)
}
}
if len(filtered) == 0 {
return nil, nil, nil
}
fields = filtered
}
case qbtypes.FingerprintOfSplit:
filtered := make([]*telemetrytypes.LogicalField, 0, len(fields))
for _, logical := range fields {
if logical.FieldContext == telemetrytypes.FieldContextResource {
filtered = append(filtered, logical)
}
}
fields = filtered
}
conds := make([]string, 0, len(fields))
var warnings []string
for _, logical := range fields {
compiled, err := storage.Compile(ctx, q, logical, operator, value, sb)
if err != nil {
return nil, nil, err
}
if compiled.Condition != "" {
conds = append(conds, compiled.Condition)
}
warnings = append(warnings, compiled.Warnings...)
}
return conds, warnings, nil
}
// RejectsBodyFunction reports the error a storage without body functions
// returns for one, before resolution; the fingerprint side of a split skips
// the term instead, because the main query evaluates it.
func RejectsBodyFunction(traits qbtypes.Traits, operator qbtypes.FilterOperator) (skip bool, err error) {
if !operator.IsFunctionOperator() && operator != qbtypes.FilterOperatorSearch {
return false, nil
}
if traits.SupportsBodyFunctions {
return false, nil
}
if traits.Split == qbtypes.FingerprintOfSplit {
return true, nil
}
return false, NewFunctionUnsupportedError(operator)
}
// Conditions resolves one key and compiles it: the condition builder for callers
// that do not run the filter visitor. The warnings carry the resolution's
// warnings first.
func Conditions(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
dropResourceFields bool,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
if _, err := RejectsBodyFunction(storage.Traits(), operator); err != nil {
return nil, nil, err
}
resolved, err := Resolve(ctx, q, storage, key, operator, value, fieldKeys)
if err != nil {
return nil, nil, err
}
conds, warnings, err := Condition(ctx, q, storage, resolved, dropResourceFields, operator, value, sb)
if err != nil {
return nil, nil, err
}
return conds, append(resolved.Warnings, warnings...), nil
}

View File

@@ -80,6 +80,11 @@ func ExistsExpression(columns []*schema.Column, key *telemetrytypes.TelemetryFie
return comparison("<>", "0"), nil
}
return comparison("=", "0"), nil
case schema.ColumnTypeEnumArray:
if exists {
return fmt.Sprintf("notEmpty(%s)", fieldExpression), nil
}
return fmt.Sprintf("empty(%s)", fieldExpression), nil
case schema.ColumnTypeEnumMap:
keyType := column.Type.(schema.MapColumnType).KeyType
if _, ok := keyType.(schema.LowCardinalityColumnType); !ok {

View File

@@ -0,0 +1,162 @@
package querybuilder
import (
"context"
"fmt"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
)
// SharedCondition is the Compile of every storage without its own condition
// language: the field's read, the shared data-type collision cast, the
// operator, then the guard rule. A sentinel-reading field takes the exists
// guard on the operators that would otherwise match the sentinel.
func SharedCondition(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
field *telemetrytypes.LogicalField,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (qbtypes.Compiled, error) {
condition, err := sharedOperator(ctx, q, storage, field, operator, value, sb)
if err != nil || condition == "" {
return qbtypes.Compiled{}, err
}
if !operator.AddDefaultExistsFilter() {
return qbtypes.Compiled{Condition: condition}, nil
}
existence, err := LogicalExistsExpr(ctx, q, storage, field, true)
if err != nil {
return qbtypes.Compiled{}, err
}
if existence.WhenAbsent != qbtypes.AbsentIsSentinel {
return qbtypes.Compiled{Condition: condition}, nil
}
return qbtypes.Compiled{Condition: sb.And(condition, sqlbuilder.Escape(existence.Predicate))}, nil
}
// sharedOperator expands a list per item, so each item takes its own cast,
// and casts the read against the operand for everything else.
func sharedOperator(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
field *telemetrytypes.LogicalField,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
switch operator {
case qbtypes.FilterOperatorIn, qbtypes.FilterOperatorNotIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
itemOperator := qbtypes.FilterOperatorEqual
if operator == qbtypes.FilterOperatorNotIn {
itemOperator = qbtypes.FilterOperatorNotEqual
}
conditions := make([]string, 0, len(values))
for _, item := range values {
condition, err := sharedOperator(ctx, q, storage, field, itemOperator, item, sb)
if err != nil {
return "", err
}
conditions = append(conditions, condition)
}
// `=`+OR and `!=`+AND instead of IN and NOT IN, to make use of the index
if operator == qbtypes.FilterOperatorIn {
return sb.Or(conditions...), nil
}
return sb.And(conditions...), nil
}
if operator.IsStringSearchOperator() {
value = FormatValueForContains(value)
}
read, err := LogicalValueExpr(ctx, q, storage, field)
if err != nil {
return "", err
}
// Coercion switches only on the data type, which every member shares, so
// the first member stands in for the field.
read, value = DataTypeCollisionHandledFieldName(field.Single(), value, read, operator)
return OperatorCondition(ctx, q, storage, field, read, operator, value, sb)
}
// OperatorCondition renders one operator over an already cast read. It is
// the shared switch a storage with its own cast policy composes with. A list
// operator is the caller's to expand, so each item takes its own cast.
func OperatorCondition(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
field *telemetrytypes.LogicalField,
read string,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
switch operator {
case qbtypes.FilterOperatorEqual:
return sb.E(read, value), nil
case qbtypes.FilterOperatorNotEqual:
return sb.NE(read, value), nil
case qbtypes.FilterOperatorGreaterThan:
return sb.G(read, value), nil
case qbtypes.FilterOperatorGreaterThanOrEq:
return sb.GE(read, value), nil
case qbtypes.FilterOperatorLessThan:
return sb.LT(read, value), nil
case qbtypes.FilterOperatorLessThanOrEq:
return sb.LE(read, value), nil
case qbtypes.FilterOperatorLike:
return sb.Like(read, value), nil
case qbtypes.FilterOperatorNotLike:
return sb.NotLike(read, value), nil
case qbtypes.FilterOperatorILike:
return sb.ILike(read, value), nil
case qbtypes.FilterOperatorNotILike:
return sb.NotILike(read, value), nil
case qbtypes.FilterOperatorContains:
return sb.ILike(read, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorNotContains:
return sb.NotILike(read, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorRegexp:
// Note: Escape $$ to $$$$ to avoid sqlbuilder interpreting materialized $ signs
// Only needed because we are using sprintf instead of sb.Match (not implemented in sqlbuilder)
return fmt.Sprintf(`match(%s, %s)`, sqlbuilder.Escape(read), sb.Var(value)), nil
case qbtypes.FilterOperatorNotRegexp:
return fmt.Sprintf(`NOT match(%s, %s)`, sqlbuilder.Escape(read), sb.Var(value)), nil
case qbtypes.FilterOperatorBetween:
values, ok := value.([]any)
if !ok || len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.Between(read, values[0], values[1]), nil
case qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok || len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.NotBetween(read, values[0], values[1]), nil
// exists and not exists are key membership checks, so the storage's
// presence test answers them
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
existence, err := LogicalExistsExpr(ctx, q, storage, field, operator == qbtypes.FilterOperatorExists)
if err != nil {
return "", err
}
return sqlbuilder.Escape(existence.Predicate), nil
}
return "", qbtypes.ErrUnsupportedOperator
}

View File

@@ -74,22 +74,14 @@ func WrapAsLogicalFields(requestedName string, keys []*telemetrytypes.TelemetryF
return fields
}
// SingleKeys flattens logical fields to their single members. It is the
// adapter for signals whose fields are single-member by construction (every
// signal without family support); their condition builders keep compiling per
// physical key.
func SingleKeys(fields []*telemetrytypes.LogicalField) []*telemetrytypes.TelemetryFieldKey {
keys := make([]*telemetrytypes.TelemetryFieldKey, 0, len(fields))
for _, field := range fields {
keys = append(keys, field.Single())
// NewKeyNotFoundError builds the error for a key that neither metadata nor
// the storage can serve, with the closest known names as suggestions.
func NewKeyNotFoundError(name string, known []string) error {
err := errors.NewInvalidInputf(errors.CodeInvalidInput, "key `%s` not found", name).WithUrl(KeyNotFoundDocURL)
if len(known) == 0 {
return err
}
return keys
}
// NewKeyNotFoundError builds the error a condition builder returns when a filter term
// references a key it has no matching field key for.
func NewKeyNotFoundError(name string) error {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "key `%s` not found", name).WithUrl(KeyNotFoundDocURL)
return err.WithSuggestions(errors.NewSuggestionsOnLevenshteinDistance(name, errors.NounKeys, known)...)
}
// NewKeyNotFoundWarning is the warning surfaced when a referenced key is absent from

View File

@@ -7,32 +7,26 @@ import (
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
// The two functions below are the only place family expressions are built.
// They compose exclusively from the mapper's per-key primitives (FieldFor,
// ExistsFor), so every member honors its own storage: materialized columns,
// evolutions, and JSON plans ride the member keys, and a signal supports
// families the moment its primitives are correct.
// They compose exclusively from the storage's per-field reads (Read, Exists),
// so every member honors its own storage: materialized columns, evolutions,
// and JSON plans ride the member fields, and a signal supports families the
// moment its reads are correct.
// LogicalValueExpr returns the value expression for a resolved logical field:
// the member's own expression for a single-member field, and a current-first
// merge across the members' expressions for a family.
func LogicalValueExpr(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
fm qbtypes.FieldMapper,
logical *telemetrytypes.LogicalField,
) (string, error) {
// the member's own read for a single-member field, and a current-first merge
// across the members' reads for a family. A member with a value map reads
// in the current vocabulary.
func LogicalValueExpr(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, logical *telemetrytypes.LogicalField) (string, error) {
if !logical.IsFamily() {
return fm.FieldFor(ctx, orgID, tsStart, tsEnd, logical.Single())
return memberRead(ctx, q, storage, logical, 0)
}
memberExprs := make([]string, 0, len(logical.Members))
for _, member := range logical.Members {
expr, err := fm.FieldFor(ctx, orgID, tsStart, tsEnd, member)
for i := range logical.Members {
expr, err := memberRead(ctx, q, storage, logical, i)
if err != nil {
return "", err
}
@@ -42,9 +36,9 @@ func LogicalValueExpr(
if logical.FieldDataType == telemetrytypes.FieldDataTypeString {
// The trailing '' keeps single-key semantics for rows without any
// member: string maps read '' for an absent key, and negative
// operators must keep including such rows (see AddDefaultExistsFilter).
// A NULL tail would drop them: NULL != 'x' evaluates to NULL, and the
// row falls out of the result.
// operators must keep including such rows. A NULL tail would drop
// them: NULL != 'x' evaluates to NULL, and the row falls out of the
// result.
values := make([]string, 0, len(memberExprs))
for _, expr := range memberExprs {
values = append(values, fmt.Sprintf("NULLIF(%s, '')", expr))
@@ -56,41 +50,69 @@ func LogicalValueExpr(
// either type is enabled, this tail must become zero too.
branches := make([]string, 0, len(logical.Members)*2)
for i, member := range logical.Members {
guard, err := fm.ExistsFor(ctx, orgID, tsStart, tsEnd, member, true)
existence, err := storage.Exists(ctx, q, member, true)
if err != nil {
return "", err
}
branches = append(branches, guard, memberExprs[i])
branches = append(branches, existence.Predicate, memberExprs[i])
}
return "multiIf(" + strings.Join(branches, ", ") + ", NULL)", nil
}
func memberRead(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, logical *telemetrytypes.LogicalField, i int) (string, error) {
expr, err := storage.Read(ctx, q, logical.Members[i])
if err != nil {
return "", err
}
if i < len(logical.ValueMaps) && logical.ValueMaps[i] != nil {
expr = TransformRead(expr, logical.ValueMaps[i])
}
return expr, nil
}
// TransformRead brings a member's read into the current vocabulary: a stored
// value maps to its current value, any other value reads as it is.
func TransformRead(read string, valueMap *telemetrytypes.ValueMap) string {
return fmt.Sprintf("transform(%s, %s, %s, %s)", read, clickHouseStringArray(valueMap.Stored), clickHouseStringArray(valueMap.Current), read)
}
func clickHouseStringArray(values []string) string {
items := make([]string, 0, len(values))
for _, value := range values {
items = append(items, ClickHouseStringLiteral(value))
}
return "[" + strings.Join(items, ", ") + "]"
}
// LogicalExistsExpr returns the existence predicate for a resolved logical
// field: the member's own predicate for a single-member field, presence of
// any member for a family.
func LogicalExistsExpr(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
fm qbtypes.FieldMapper,
logical *telemetrytypes.LogicalField,
exists bool,
) (string, error) {
// any member for a family. A family reads through its merge, so its absence
// semantics are the merge's tail, not the members' own.
func LogicalExistsExpr(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, logical *telemetrytypes.LogicalField, exists bool) (qbtypes.Existence, error) {
if !logical.IsFamily() {
return fm.ExistsFor(ctx, orgID, tsStart, tsEnd, logical.Single(), exists)
return storage.Exists(ctx, q, logical.Single(), exists)
}
guards := make([]string, 0, len(logical.Members))
for _, member := range logical.Members {
guard, err := fm.ExistsFor(ctx, orgID, tsStart, tsEnd, member, true)
existence, err := storage.Exists(ctx, q, member, true)
if err != nil {
return "", err
return qbtypes.Existence{}, err
}
guards = append(guards, guard)
guards = append(guards, existence.Predicate)
}
combined := "(" + strings.Join(guards, " OR ") + ")"
if exists {
return combined, nil
if !exists {
combined = "NOT " + combined
}
return "NOT " + combined, nil
return qbtypes.Existence{Predicate: combined, WhenAbsent: familyAbsence(logical)}, nil
}
// familyAbsence is what the merged read yields for a row without any
// member: the sentinel tail of a string family, NULL for the others.
func familyAbsence(logical *telemetrytypes.LogicalField) qbtypes.Absent {
if logical.FieldDataType == telemetrytypes.FieldDataTypeString {
return qbtypes.AbsentIsSentinel
}
return qbtypes.AbsentIsNull
}

View File

@@ -4,39 +4,41 @@ import (
"context"
"testing"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// stubFieldMapper provides just the two per-key primitives the shared
// composition builds on; the remaining FieldMapper methods are unused here.
type stubFieldMapper struct{}
// stubStorage provides the two reads the shared composition builds on.
type stubStorage struct{}
func (stubFieldMapper) FieldFor(_ context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
func (stubStorage) Read(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return "value(" + key.Name + ")", nil
}
func (stubFieldMapper) ExistsFor(_ context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey, exists bool) (string, error) {
func (stubStorage) Exists(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, exists bool) (qbtypes.Existence, error) {
if exists {
return "has(" + key.Name + ")", nil
return qbtypes.Existence{Predicate: "has(" + key.Name + ")", WhenAbsent: qbtypes.AbsentIsSentinel}, nil
}
return "NOT has(" + key.Name + ")", nil
return qbtypes.Existence{Predicate: "NOT has(" + key.Name + ")", WhenAbsent: qbtypes.AbsentIsSentinel}, nil
}
func (stubFieldMapper) ColumnFor(context.Context, valuer.UUID, uint64, uint64, *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
return nil, qbtypes.ErrColumnNotFound
func (stubStorage) Fallback(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, qbtypes.FilterOperator, any) ([]*telemetrytypes.LogicalField, error) {
return nil, nil
}
func (stubFieldMapper) ColumnExpressionFor(context.Context, valuer.UUID, uint64, uint64, *telemetrytypes.TelemetryFieldKey, telemetrytypes.FieldDataType, map[string][]*telemetrytypes.TelemetryFieldKey) (string, error) {
return "", qbtypes.ErrColumnNotFound
func (stubStorage) Traits() qbtypes.Traits {
return qbtypes.Traits{}
}
func (stubFieldMapper) CandidateKeys(context.Context, valuer.UUID, *telemetrytypes.TelemetryFieldKey, any, map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
func (s stubStorage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
return SharedCondition(ctx, q, s, logical, operator, value, sb)
}
func (s stubStorage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
return DefaultRead(ctx, q, s, logical)
}
func stringFamily(names ...string) *telemetrytypes.LogicalField {
@@ -49,13 +51,13 @@ func stringFamily(names ...string) *telemetrytypes.LogicalField {
func TestLogicalValueExprSingleMemberDelegatesToFieldFor(t *testing.T) {
logical := telemetrytypes.SingleLogicalField("a", &telemetrytypes.TelemetryFieldKey{Name: "a"})
expr, err := LogicalValueExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, logical)
expr, err := LogicalValueExpr(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, logical)
require.NoError(t, err)
assert.Equal(t, "value(a)", expr)
}
func TestLogicalValueExprStringFamilyMergesCurrentFirst(t *testing.T) {
expr, err := LogicalValueExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, stringFamily("current", "old"))
expr, err := LogicalValueExpr(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, stringFamily("current", "old"))
require.NoError(t, err)
// The trailing '' preserves keyless-row semantics for negative operators.
assert.Equal(t, "COALESCE(NULLIF(value(current), ''), NULLIF(value(old), ''), '')", expr)
@@ -70,26 +72,26 @@ func TestLogicalValueExprNumericFamilyGuardsEveryMember(t *testing.T) {
{Name: "old", FieldDataType: telemetrytypes.FieldDataTypeNumber},
},
}
expr, err := LogicalValueExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, logical)
expr, err := LogicalValueExpr(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, logical)
require.NoError(t, err)
assert.Equal(t, "multiIf(has(current), value(current), has(old), value(old), NULL)", expr)
}
func TestLogicalExistsExprSingleMemberDelegatesToExistsFor(t *testing.T) {
logical := telemetrytypes.SingleLogicalField("a", &telemetrytypes.TelemetryFieldKey{Name: "a"})
expr, err := LogicalExistsExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, logical, false)
existence, err := LogicalExistsExpr(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, logical, false)
require.NoError(t, err)
assert.Equal(t, "NOT has(a)", expr)
assert.Equal(t, "NOT has(a)", existence.Predicate)
}
func TestLogicalExistsExprFamilyIsAnyMemberPresence(t *testing.T) {
family := stringFamily("current", "old")
expr, err := LogicalExistsExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, family, true)
existence, err := LogicalExistsExpr(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, family, true)
require.NoError(t, err)
assert.Equal(t, "(has(current) OR has(old))", expr)
assert.Equal(t, "(has(current) OR has(old))", existence.Predicate)
expr, err = LogicalExistsExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, family, false)
existence, err = LogicalExistsExpr(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, family, false)
require.NoError(t, err)
assert.Equal(t, "NOT (has(current) OR has(old))", expr)
assert.Equal(t, "NOT (has(current) OR has(old))", existence.Predicate)
}

View File

@@ -48,7 +48,7 @@ func TestFamiliesOffByDefault(t *testing.T) {
}},
}
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, flaggertest.New(t), &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
fields := matchingLogicalFields(false, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
require.Len(t, fields, 1)
assert.False(t, fields[0].IsFamily())
assert.Equal(t, []string{"deployment.environment.name"}, memberNames(fields[0]))
@@ -76,7 +76,7 @@ func TestMatchingLogicalFieldsGroupsFamilyMembers(t *testing.T) {
}
for _, requested := range []string{"deployment.environment.name", "deployment.environment"} {
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, familiesOn(t), &telemetrytypes.TelemetryFieldKey{Name: requested}, fieldKeys)
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: requested}, fieldKeys)
require.Len(t, fields, 1, "a family is one logical field, requested via %s", requested)
logical := fields[0]
assert.Equal(t, requested, logical.Name, "response identity is the requested spelling")
@@ -106,7 +106,7 @@ func TestMatchingLogicalFieldsOrdersMembersByFamilyRank(t *testing.T) {
}},
}
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, familiesOn(t), &telemetrytypes.TelemetryFieldKey{
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{
Name: "deployment.environment.name",
FieldContext: telemetrytypes.FieldContextResource,
}, fieldKeys)
@@ -131,7 +131,7 @@ func TestMatchingLogicalFieldsKeepsLogsLiteral(t *testing.T) {
"deployment.environment": {logsKey("deployment.environment")},
}
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, familiesOn(t), &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
require.Len(t, fields, 1)
assert.False(t, fields[0].IsFamily())
assert.Equal(t, []string{"deployment.environment.name"}, memberNames(fields[0]))
@@ -165,7 +165,7 @@ func TestResolveLogicalFieldsKeepsFamilyThroughAmbiguity(t *testing.T) {
}
requested := &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, familiesOn(t), requested, fieldKeys)
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, requested, fieldKeys)
require.Len(t, fields, 2, "resource family + attribute collision")
resolved, warning := ResolveLogicalFields(requested, fields)
@@ -193,7 +193,7 @@ func TestMatchingLogicalFieldsNeverMergesAcrossDataTypes(t *testing.T) {
}},
}
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, familiesOn(t), &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
require.Len(t, fields, 2)
for _, logical := range fields {
assert.False(t, logical.IsFamily())

202
pkg/querybuilder/resolve.go Normal file
View File

@@ -0,0 +1,202 @@
package querybuilder
import (
"context"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"golang.org/x/exp/maps"
)
// NewQueryInfo binds the context of one query and evaluates the query-path
// flags one time. A nil flagger keeps resolution literal and the log body in
// its legacy column.
func NewQueryInfo(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger, signal telemetrytypes.Signal, metric *telemetrytypes.MetricContext, startNs, endNs uint64) qbtypes.QueryInfo {
q := qbtypes.QueryInfo{
OrgID: orgID,
StartNs: startNs,
EndNs: endNs,
Signal: signal,
Metric: metric,
FamiliesOn: semconvFamiliesEnabled(ctx, orgID, fl),
}
if fl != nil {
q.BodyJSONOn = fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
}
return q
}
// Resolve turns one requested key into its meanings, one time per stage
// entry. The order is the same for every storage and every stage:
//
// 1. matches: the metadata keys under the key's spellings, grouped into
// families when the flag is on; a key under one of the storage's own
// contexts matches as if it had none;
// 2. ambiguity: in a filter, several interpretations settle by the
// resource-over-attribute policy, with a warning; a column stage keeps
// every interpretation in metadata order and folds them;
// 3. intrinsic column first: for a bare key, a column every row has leads, whether
// metadata reports it or the storage's own tables do, and sentinel
// fields of a contradicting type drop;
// 4. fallback: with no match, the storage's fallback keys; the not-found
// warning fires only when every one of them is a guess.
func Resolve(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
) (qbtypes.Resolved, error) {
traits := storage.Traits()
lookup := key
for _, own := range traits.OwnContexts {
if key.FieldContext == own {
bare := *key
bare.FieldContext = telemetrytypes.FieldContextUnspecified
lookup = &bare
break
}
}
matches := matchingLogicalFields(q.FamiliesOn, q.Signal, lookup, fieldKeys)
resolved := qbtypes.Resolved{Key: key, Ambiguous: len(matches) > 1}
fields := matches
if operator != qbtypes.FilterOperatorUnknown {
var warning string
fields, warning = ResolveLogicalFields(key, matches)
if warning != "" {
resolved.Warnings = append(resolved.Warnings, warning)
}
}
if lookup.FieldContext == telemetrytypes.FieldContextUnspecified && len(fields) > 0 {
var err error
fields, err = intrinsicColumnFirst(ctx, q, storage, key, operator, value, fields)
if err != nil {
return qbtypes.Resolved{}, err
}
}
if len(fields) > 0 {
resolved.Fields = fields
return resolved, nil
}
fallback, err := storage.Fallback(ctx, q, key, operator, value)
if err != nil {
return qbtypes.Resolved{}, err
}
if len(fallback) == 0 {
if traits.UnknownKey == qbtypes.IgnoreUnknownKey {
resolved.Skipped = true
return resolved, nil
}
return qbtypes.Resolved{}, NewKeyNotFoundError(key.Name, maps.Keys(fieldKeys))
}
resolved.FromFallback = true
resolved.Fields = fallback
if fallbackIsGuess(ctx, q, storage, fallback) {
resolved.Warnings = append(resolved.Warnings, NewKeyNotFoundWarning(key.Name))
}
return resolved, nil
}
// ResolveColumn resolves one key for a column stage and renders it.
func ResolveColumn(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
key *telemetrytypes.TelemetryFieldKey,
target telemetrytypes.FieldDataType,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
resolved, err := Resolve(ctx, q, storage, key, qbtypes.FilterOperatorUnknown, nil, fieldKeys)
if err != nil {
return "", err
}
return Column(ctx, q, storage, resolved, target)
}
// intrinsicColumnFirst applies rule 3 to a bare key. The column comes from the
// matches when metadata reports it, else from the storage's own fallback,
// so a metadata gap degrades to the correct column and never to a corrupt
// metadata key. A match of a contradicting data type drops.
func intrinsicColumnFirst(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
fields []*telemetrytypes.LogicalField,
) ([]*telemetrytypes.LogicalField, error) {
column := alwaysPresent(ctx, q, storage, fields)
if column == nil {
// a fallback that cannot answer this term has no column for it; its
// error belongs to the no-match path
if fallback, err := storage.Fallback(ctx, q, key, operator, value); err == nil {
column = alwaysPresent(ctx, q, storage, fallback)
}
}
if column == nil {
return fields, nil
}
out := make([]*telemetrytypes.LogicalField, 0, len(fields)+1)
out = append(out, column)
for _, logical := range fields {
if logical == column || sameFact(logical, column) {
continue
}
if dataTypesConsistent(column.FieldDataType, logical.FieldDataType) {
out = append(out, logical)
}
}
return out, nil
}
// alwaysPresent returns the first field every row has. A field the storage
// cannot test for presence is not that field.
func alwaysPresent(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, fields []*telemetrytypes.LogicalField) *telemetrytypes.LogicalField {
for _, logical := range fields {
if logical.IsFamily() {
continue
}
existence, err := storage.Exists(ctx, q, logical.Single(), true)
if err == nil && existence.WhenAbsent == qbtypes.AlwaysPresent {
return logical
}
}
return nil
}
func sameFact(a, b *telemetrytypes.LogicalField) bool {
return !a.IsFamily() && !b.IsFamily() &&
a.FieldContext == b.FieldContext && a.Single().Name == b.Single().Name
}
// dataTypesConsistent reports whether a metadata key's data type can
// describe the same stored value as the column's: an unspecified type
// matches anything, and the numeric kinds match each other.
func dataTypesConsistent(column, entry telemetrytypes.FieldDataType) bool {
if column == telemetrytypes.FieldDataTypeUnspecified || entry == telemetrytypes.FieldDataTypeUnspecified {
return true
}
if column == entry {
return true
}
return isNumber(column) && isNumber(entry)
}
func isNumber(dt telemetrytypes.FieldDataType) bool {
return dt == telemetrytypes.FieldDataTypeInt64 || dt == telemetrytypes.FieldDataTypeFloat64 || dt == telemetrytypes.FieldDataTypeNumber
}
// fallbackIsGuess reports whether every fallback key is a guess. A column
// the storage knows is not one, and its presence means the key was found.
func fallbackIsGuess(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, fields []*telemetrytypes.LogicalField) bool {
return alwaysPresent(ctx, q, storage, fields) == nil
}

View File

@@ -9,12 +9,10 @@ import (
"strings"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
grammar "github.com/SigNoz/signoz/pkg/parser/filterquery/grammar"
"github.com/SigNoz/signoz/pkg/semconv"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/antlr4-go/antlr/v4"
sqlbuilder "github.com/huandu/go-sqlbuilder"
@@ -28,10 +26,8 @@ const stringMatchingOperatorDocURL = "https://signoz.io/docs/userguide/operators
// to convert the parsed filter expressions into ClickHouse WHERE clause.
type filterExpressionVisitor struct {
context context.Context
orgID valuer.UUID
fl flagger.Flagger
fieldMapper qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
query qbtypes.QueryInfo
storage qbtypes.Storage
warnings []string
mainWarnURL string
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey
@@ -44,39 +40,31 @@ type filterExpressionVisitor struct {
variables map[string]qbtypes.VariableItem
keysWithWarnings map[string]bool
startNs uint64
endNs uint64
requiresCostGuard bool
}
type FilterExprVisitorOpts struct {
Context context.Context
OrgID valuer.UUID
// Flagger evaluates the resolve_semconv_families flag during resolution.
// A nil Flagger keeps resolution literal.
Flagger flagger.Flagger
// Query is the request's context with the query-path flags evaluated
// one time; Storage answers the signal's part of every term.
Query qbtypes.QueryInfo
Storage qbtypes.Storage
Logger *slog.Logger
FieldMapper qbtypes.FieldMapper
ConditionBuilder qbtypes.ConditionBuilder
FieldKeys map[string][]*telemetrytypes.TelemetryFieldKey
Builder *sqlbuilder.SelectBuilder
FullTextColumn *telemetrytypes.TelemetryFieldKey
SkipResourceFilter bool
SkipFullTextFilter bool
Variables map[string]qbtypes.VariableItem
StartNs uint64
EndNs uint64
}
// newFilterExpressionVisitor creates a new filterExpressionVisitor.
func newFilterExpressionVisitor(opts FilterExprVisitorOpts) *filterExpressionVisitor {
return &filterExpressionVisitor{
context: opts.Context,
orgID: opts.OrgID,
fl: opts.Flagger,
fieldMapper: opts.FieldMapper,
conditionBuilder: opts.ConditionBuilder,
query: opts.Query,
storage: opts.Storage,
fieldKeys: opts.FieldKeys,
builder: opts.Builder,
fullTextColumn: opts.FullTextColumn,
@@ -84,8 +72,6 @@ func newFilterExpressionVisitor(opts FilterExprVisitorOpts) *filterExpressionVis
skipFullTextFilter: opts.SkipFullTextFilter,
variables: opts.Variables,
keysWithWarnings: make(map[string]bool),
startNs: opts.StartNs,
endNs: opts.EndNs,
}
}
@@ -367,7 +353,7 @@ func (v *filterExpressionVisitor) VisitPrimary(ctx *grammar.PrimaryContext) any
return ErrorConditionLiteral
}
}
conds, ok := v.buildConditions(v.fullTextColumn, []*telemetrytypes.LogicalField{telemetrytypes.SingleLogicalField(v.fullTextColumn.Name, v.fullTextColumn)}, qbtypes.FilterOperatorRegexp, FormatFullTextSearch(searchText))
conds, ok := v.compile(storageKey(v.fullTextColumn), qbtypes.FilterOperatorRegexp, FormatFullTextSearch(searchText))
if !ok {
return ErrorConditionLiteral
}
@@ -386,7 +372,6 @@ func (v *filterExpressionVisitor) VisitPrimary(ctx *grammar.PrimaryContext) any
// VisitComparison handles all comparison operators.
func (v *filterExpressionVisitor) VisitComparison(ctx *grammar.ComparisonContext) any {
key := v.Visit(ctx.Key()).(*telemetrytypes.TelemetryFieldKey)
matching := MatchingLogicalFields(v.context, v.orgID, v.fl, key, v.fieldKeys)
// Handle EXISTS specially
if ctx.EXISTS() != nil {
@@ -395,7 +380,7 @@ func (v *filterExpressionVisitor) VisitComparison(ctx *grammar.ComparisonContext
op = qbtypes.FilterOperatorNotExists
}
conds, ok := v.buildConditions(key, matching, op, nil)
conds, ok := v.buildConditions(key, op, nil)
if !ok {
return ErrorConditionLiteral
}
@@ -468,7 +453,7 @@ func (v *filterExpressionVisitor) VisitComparison(ctx *grammar.ComparisonContext
op = qbtypes.FilterOperatorNotIn
}
conds, ok := v.buildConditions(key, matching, op, values)
conds, ok := v.buildConditions(key, op, values)
if !ok {
return ErrorConditionLiteral
}
@@ -516,7 +501,7 @@ func (v *filterExpressionVisitor) VisitComparison(ctx *grammar.ComparisonContext
return ErrorConditionLiteral
}
conds, ok := v.buildConditions(key, matching, op, []any{value1, value2})
conds, ok := v.buildConditions(key, op, []any{value1, value2})
if !ok {
return ErrorConditionLiteral
}
@@ -600,7 +585,7 @@ func (v *filterExpressionVisitor) VisitComparison(ctx *grammar.ComparisonContext
}
}
conds, ok := v.buildConditions(key, matching, op, value)
conds, ok := v.buildConditions(key, op, value)
if !ok {
return ErrorConditionLiteral
}
@@ -682,7 +667,7 @@ func (v *filterExpressionVisitor) VisitFullText(ctx *grammar.FullTextContext) an
v.errors = append(v.errors, "full text search is not supported")
return ErrorConditionLiteral
}
conds, ok := v.buildConditions(v.fullTextColumn, []*telemetrytypes.LogicalField{telemetrytypes.SingleLogicalField(v.fullTextColumn.Name, v.fullTextColumn)}, qbtypes.FilterOperatorRegexp, FormatFullTextSearch(text))
conds, ok := v.compile(storageKey(v.fullTextColumn), qbtypes.FilterOperatorRegexp, FormatFullTextSearch(text))
if !ok {
return ErrorConditionLiteral
}
@@ -737,7 +722,7 @@ func (v *filterExpressionVisitor) VisitFunctionCall(ctx *grammar.FunctionCallCon
return ErrorConditionLiteral
}
conds, ok := v.buildConditions(key, MatchingLogicalFields(v.context, v.orgID, v.fl, key, v.fieldKeys), operator, value)
conds, ok := v.buildConditions(key, operator, value)
if !ok {
return ErrorConditionLiteral
}
@@ -793,6 +778,12 @@ func normalizeFunctionValue(operator qbtypes.FilterOperator, functionName string
// search term plus optional field-context scopes, ORing one FilterOperatorSearch per
// scope (no scope = keyless, covering every field).
func (v *filterExpressionVisitor) VisitSearchCall(ctx *grammar.SearchCallContext) any {
if skip, err := RejectsBodyFunction(v.storage.Traits(), qbtypes.FilterOperatorSearch); err != nil {
v.recordError(err)
return ErrorConditionLiteral
} else if skip {
return SkipConditionLiteral
}
// Flag scan-heavy so the statement builder attaches the cost guard.
v.requiresCostGuard = true
@@ -835,7 +826,7 @@ func (v *filterExpressionVisitor) VisitSearchCall(ctx *grammar.SearchCallContext
var conds []string
for _, fieldContext := range fieldContexts {
key := telemetrytypes.NewTelemetryFieldKey("", fieldContext, telemetrytypes.FieldDataTypeUnspecified)
scoped, cok := v.buildConditions(key, nil, qbtypes.FilterOperatorSearch, searchText)
scoped, cok := v.compile(storageKey(key), qbtypes.FilterOperatorSearch, searchText)
if !cok {
return ErrorConditionLiteral
}
@@ -927,20 +918,55 @@ func (v *filterExpressionVisitor) VisitKey(ctx *grammar.KeyContext) any {
return &fieldKey
}
// buildConditions invokes the condition builder for a filter term, folding its
// warnings/errors into visitor state; returns false if an error was recorded.
func (v *filterExpressionVisitor) buildConditions(key *telemetrytypes.TelemetryFieldKey, matching []*telemetrytypes.LogicalField, op qbtypes.FilterOperator, value any) ([]string, bool) {
conds, warns, err := v.conditionBuilder.ConditionFor(v.context, v.orgID, v.startNs, v.endNs, key, v.fieldKeys, qbtypes.ConditionBuilderOptions{SkipResourceFilter: v.skipResourceFilter}, op, value, v.builder)
// buildConditions resolves and compiles one filter term, folding its
// warnings and errors into the visitor state; ok is false when an error was
// recorded.
func (v *filterExpressionVisitor) buildConditions(key *telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, value any) ([]string, bool) {
if skip, err := RejectsBodyFunction(v.storage.Traits(), op); err != nil {
v.recordError(err)
return nil, false
} else if skip {
return nil, true
}
resolved, err := Resolve(v.context, v.query, v.storage, key, op, value, v.fieldKeys)
if err != nil {
_, _, _, _, errURL, _ := errors.Unwrapb(err)
assignIfEmpty(&v.mainErrorURL, errURL)
v.errors = append(v.errors, err.Error())
v.recordError(err)
return nil, false
}
v.addWarnings(warns, len(matching) > 1)
v.addWarnings(resolved.Warnings, resolved.Ambiguous)
return v.compile(resolved, op, value)
}
// compile turns a resolved term into its conditions, folding the storage's
// warnings into the visitor state.
func (v *filterExpressionVisitor) compile(resolved qbtypes.Resolved, op qbtypes.FilterOperator, value any) ([]string, bool) {
conds, warns, err := Condition(v.context, v.query, v.storage, resolved, v.skipResourceFilter, op, value, v.builder)
if err != nil {
v.recordError(err)
return nil, false
}
v.addWarnings(warns, resolved.Ambiguous)
return conds, true
}
func (v *filterExpressionVisitor) recordError(err error) {
_, _, _, _, errURL, _ := errors.Unwrapb(err)
assignIfEmpty(&v.mainErrorURL, errURL)
v.errors = append(v.errors, err.Error())
}
// storageKey is a key the storage knows without metadata, resolved as
// itself: the full-text column, or a search() scope that names a set of
// columns. It is a fallback key: the fingerprint sub-query cannot serve it,
// so the main query keeps it when the split runs.
func storageKey(key *telemetrytypes.TelemetryFieldKey) qbtypes.Resolved {
return qbtypes.Resolved{
Key: key,
Fields: []*telemetrytypes.LogicalField{telemetrytypes.SingleLogicalField(key.Name, key)},
FromFallback: true,
}
}
// addWarnings appends de-duplicated warnings to the visitor. ambiguous marks warnings
// from a multi-match key so the field-context doc URL is attached.
func (v *filterExpressionVisitor) addWarnings(warns []string, ambiguous bool) {
@@ -988,15 +1014,14 @@ func assignIfEmpty(s *string, value string) {
// familyMemberNames returns the physical spellings to look up for the
// referenced key: the semantic-convention family members (current-first) when
// the resolve_semconv_families flag is on for the org and the key can resolve
// to traces, else just the requested name. Only trace field mappers understand
// families today; logs and metrics keep the requested spelling until theirs
// land.
func familyMemberNames(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger, field *telemetrytypes.TelemetryFieldKey) []string {
if !semconvFamiliesEnabled(ctx, orgID, fl) {
// families are on and the query can resolve to traces, else just the requested
// name. Only trace field mappers understand families today; logs and metrics
// keep the requested spelling until theirs land.
func familyMemberNames(familiesOn bool, signal telemetrytypes.Signal, field *telemetrytypes.TelemetryFieldKey) []string {
if !familiesOn {
return []string{field.Name}
}
if field.Signal != telemetrytypes.SignalUnspecified && field.Signal != telemetrytypes.SignalTraces {
if signal != telemetrytypes.SignalUnspecified && signal != telemetrytypes.SignalTraces {
return []string{field.Name}
}
return semconv.Members(semconv.KindAttribute, telemetrytypes.FieldKeySelector{
@@ -1006,7 +1031,7 @@ func familyMemberNames(ctx context.Context, orgID valuer.UUID, fl flagger.Flagge
})
}
// MatchingLogicalFields resolves the referenced key against the metadata map
// matchingLogicalFields resolves the referenced key against the metadata map
// into logical fields, honoring any context/data type the user specified.
//
// Physical keys that are members of one semantic-convention family (traces
@@ -1014,16 +1039,15 @@ func familyMemberNames(ctx context.Context, orgID valuer.UUID, fl flagger.Flagge
// identity, members ordered current-first. Every other matching key becomes
// its own single-member logical field. Ambiguity is the length of the
// returned slice: one family is one element and is never ambiguous with
// itself, but the slice can hold several logical fields including several
// itself, but the slice can hold several logical fields, including several
// family fields, one per identity, when the family exists under more than
// one context or data type. Members alias the metadata map entries; nothing
// is copied or mutated.
//
// Family grouping only happens when the resolve_semconv_families flag is on
// for the org. A nil flagger means off: every match then stays a
// single-member logical field.
func MatchingLogicalFields(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger, field *telemetrytypes.TelemetryFieldKey, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.LogicalField {
members := familyMemberNames(ctx, orgID, fl, field)
// Family grouping only happens when families are on for the query. Off,
// every match stays a single-member logical field.
func matchingLogicalFields(familiesOn bool, signal telemetrytypes.Signal, field *telemetrytypes.TelemetryFieldKey, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.LogicalField {
members := familyMemberNames(familiesOn, signal, field)
matches := collectMemberMatches(field, members, fieldKeys)
return groupIntoLogicalFields(field.Name, len(members) > 1, matches)
}

View File

@@ -10,7 +10,6 @@ import (
grammar "github.com/SigNoz/signoz/pkg/parser/filterquery/grammar"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/antlr4-go/antlr/v4"
sqlbuilder "github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/assert"
@@ -590,15 +589,18 @@ func TestVisitKey(t *testing.T) {
// and decides not-found handling. Replay that here against the generic
// builder behavior (error unless the key is ignored). The test maps carry
// no signal, so every logical field is single-member and flattens losslessly.
matching := MatchingLogicalFields(context.Background(), valuer.UUID{}, nil, key, tt.fieldKeys)
matching := matchingLogicalFields(false, telemetrytypes.SignalUnspecified, key, tt.fieldKeys)
resolved, warning := ResolveLogicalFields(key, matching)
keys := SingleKeys(resolved)
keys := make([]*telemetrytypes.TelemetryFieldKey, 0, len(resolved))
for _, logical := range resolved {
keys = append(keys, logical.Single())
}
var gotErrors []string
var gotMainErrURL, gotMainWrnURL string
var gotWarnings []string
if len(keys) == 0 && !tt.ignoreNotFoundKeys {
err := NewKeyNotFoundError(key.Name)
err := NewKeyNotFoundError(key.Name, nil)
gotErrors = append(gotErrors, err.Error())
_, _, _, _, gotMainErrURL, _ = errors.Unwrapb(err)
}
@@ -748,99 +750,67 @@ var visitTestKeys = map[string][]*telemetrytypes.TelemetryFieldKey{
"body": {{Name: "body", FieldContext: telemetrytypes.FieldContextLog, FieldDataType: telemetrytypes.FieldDataTypeString}},
}
type resourceConditionBuilder struct{}
// resourceStorage mirrors the fingerprint storage: only resource keys
// compile, and unknown keys and body functions are skipped.
type resourceStorage struct{}
func (b *resourceConditionBuilder) ConditionFor(
_ context.Context,
_ valuer.UUID,
_ uint64,
_ uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
_ any,
_ *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// mirror the real resource builder: function operators never apply to resources
if operator.IsFunctionOperator() {
return nil, nil, nil
}
resolved, warning := ResolveLogicalFields(key, MatchingLogicalFields(context.Background(), valuer.UUID{}, nil, key, fieldKeys))
keys := SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
var conds []string
for _, k := range keys {
// only resource keys contribute; others (and unknown keys) are ignored
if k.FieldContext != telemetrytypes.FieldContextResource {
continue
}
conds = append(conds, fmt.Sprintf("%s_cond", k.Name))
}
return conds, warnings, nil
func (resourceStorage) Read(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return key.Name, nil
}
type conditionBuilder struct{}
func (resourceStorage) Exists(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, _ bool) (qbtypes.Existence, error) {
return qbtypes.Existence{Predicate: "has(" + key.Name + ")", WhenAbsent: qbtypes.AbsentIsSentinel}, nil
}
func (b *conditionBuilder) ConditionFor(
_ context.Context,
_ valuer.UUID,
_ uint64,
_ uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
options qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
_ any,
_ *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
func (resourceStorage) Fallback(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, qbtypes.FilterOperator, any) ([]*telemetrytypes.LogicalField, error) {
return nil, nil
}
// has/hasAny/hasAll/hasToken only support body fields; mirror the real
// condition builder which now owns this validation and errors for non-body keys.
switch operator {
case qbtypes.FilterOperatorHas, qbtypes.FilterOperatorHasAny, qbtypes.FilterOperatorHasAll, qbtypes.FilterOperatorHasToken:
if key.FieldContext != telemetrytypes.FieldContextBody {
return nil, nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "function supports only body JSON search")
}
return []string{fmt.Sprintf("%s_cond", key.Name)}, nil, nil
}
func (resourceStorage) Traits() qbtypes.Traits {
return qbtypes.Traits{Split: qbtypes.FingerprintOfSplit, UnknownKey: qbtypes.IgnoreUnknownKey}
}
resolved, warning := ResolveLogicalFields(key, MatchingLogicalFields(context.Background(), valuer.UUID{}, nil, key, fieldKeys))
keys := SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
if len(keys) == 0 {
// errors on unknown keys (no IgnoreNotFoundKeys equivalent for this builder)
return nil, warnings, NewKeyNotFoundError(key.Name)
}
func (resourceStorage) Compile(_ context.Context, _ qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ qbtypes.FilterOperator, _ any, _ *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
return qbtypes.Compiled{Condition: fmt.Sprintf("%s_cond", logical.Single().Name)}, nil
}
// A resource sub-query already covers the term; drop resource keys from the main query.
if options.SkipResourceFilter {
filtered := make([]*telemetrytypes.TelemetryFieldKey, 0, len(keys))
for _, k := range keys {
if k.FieldContext != telemetrytypes.FieldContextResource {
filtered = append(filtered, k)
}
}
if len(filtered) == 0 {
return nil, warnings, nil
}
keys = filtered
}
func (s resourceStorage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
return DefaultRead(ctx, q, s, logical)
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
conds = append(conds, fmt.Sprintf("%s_cond", k.Name))
// mainStorage mirrors a main-query storage: body functions apply to body keys
// only, an unknown key errors, and a body path without metadata matches is
// its own fallback.
type mainStorage struct{}
func (mainStorage) Read(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return key.Name, nil
}
func (mainStorage) Exists(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, _ bool) (qbtypes.Existence, error) {
return qbtypes.Existence{Predicate: "has(" + key.Name + ")", WhenAbsent: qbtypes.AbsentIsSentinel}, nil
}
func (mainStorage) Fallback(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, _ qbtypes.FilterOperator, _ any) ([]*telemetrytypes.LogicalField, error) {
if key.FieldContext != telemetrytypes.FieldContextBody {
return nil, nil
}
return conds, warnings, nil
return WrapAsLogicalFields(key.Name, []*telemetrytypes.TelemetryFieldKey{key}), nil
}
func (mainStorage) Traits() qbtypes.Traits {
return qbtypes.Traits{Split: qbtypes.MainOfSplit, SupportsBodyFunctions: true}
}
func (mainStorage) Compile(_ context.Context, _ qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, _ any, _ *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
if operator.IsFunctionOperator() && logical.FieldContext != telemetrytypes.FieldContextBody {
return qbtypes.Compiled{}, errors.NewInvalidInputf(errors.CodeInvalidInput, "function supports only body JSON search")
}
return qbtypes.Compiled{Condition: fmt.Sprintf("%s_cond", logical.Single().Name)}, nil
}
func (s mainStorage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
return DefaultRead(ctx, q, s, logical)
}
// visitComparisonCase is a single test case for the TestVisitComparison_* family.
@@ -879,7 +849,7 @@ func visitComparisonOpts(t *testing.T) (rsbOpts, sbOpts FilterExprVisitorOpts) {
rsbOpts = FilterExprVisitorOpts{
Context: t.Context(),
FieldKeys: visitTestKeys,
ConditionBuilder: &resourceConditionBuilder{},
Storage: &resourceStorage{},
Variables: allVariable,
SkipResourceFilter: false,
SkipFullTextFilter: true,
@@ -887,7 +857,7 @@ func visitComparisonOpts(t *testing.T) (rsbOpts, sbOpts FilterExprVisitorOpts) {
sbOpts = FilterExprVisitorOpts{
Context: t.Context(),
FieldKeys: visitTestKeys,
ConditionBuilder: &conditionBuilder{},
Storage: &mainStorage{},
Variables: allVariable,
SkipResourceFilter: true,
SkipFullTextFilter: false,
@@ -1630,10 +1600,11 @@ func TestVisitComparison_FunctionCalls(t *testing.T) {
wantErrSB: true,
},
{
name: "has on resource key",
expr: "has(x, 'hello')",
wantRSB: "",
wantErrSB: true,
// SB: the resource sub-query covers x, so the main query drops the term.
name: "has on resource key",
expr: "has(x, 'hello')",
wantRSB: "",
wantSB: "",
},
{
// RSB: TrueConditionLiteral stripped by AND; x_cond remains.

View File

@@ -196,7 +196,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -267,7 +267,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
countIf(attribute_string_gen_ai$$request$$model_exists) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -340,7 +340,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -409,7 +409,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -479,7 +479,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -555,7 +555,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -632,7 +632,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -832,7 +832,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `+"`service.name`"+`,
any(multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `+"`service.name`"+`,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -855,12 +855,11 @@ SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000
assert.Empty(t, attrStmt.Warnings)
// span. corrects to the same attribute (identical SQL) but the span-context
// metadata lookup misses, surfacing a key-not-found warning.
// span. is the signal's own context, so it resolves to the same attribute
// (identical SQL) without a warning.
spanStmt := build("span.output_tokens > 100")
assert.Equal(t, renderSQL(t, attrStmt), renderSQL(t, spanStmt))
require.Len(t, spanStmt.Warnings, 1)
assert.Contains(t, spanStmt.Warnings[0], "key `output_tokens` not found in metadata")
assert.Empty(t, spanStmt.Warnings)
// bare spelling is claimed by the aggregate alias
bareStmt := build("output_tokens > 100")
@@ -900,7 +899,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `+"`service.name`"+`,
any(multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `+"`service.name`"+`,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,

View File

@@ -154,7 +154,7 @@ func TestBuild_FullSQL_Scalar_GroupByIntrinsic(t *testing.T) {
assertSQLEqual(t, `
WITH __scoped_traces AS (
SELECT trace_id,
toString(multiIf(name <> '', toString(name), NULL)) AS __GROUP_BY_KEY_0_name,
toString(toString(name)) AS __GROUP_BY_KEY_0_name,
sum(multiIf(mapContains(attributes_number, 'gen_ai.usage.output_tokens'), toFloat64(attributes_number['gen_ai.usage.output_tokens']), NULL)) AS output_tokens
FROM signoz_traces.distributed_signoz_index_v3
WHERE timestamp >= '1747947419000000000'
@@ -718,7 +718,7 @@ SELECT sum(multiIf(mapContains(attributes_number, 'gen_ai.usage.output_tokens'),
FROM signoz_traces.distributed_signoz_index_v3
WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter)
AND trace_id GLOBAL IN (SELECT trace_id FROM __trace_scope)
AND (((mapContains(attributes_string, 'gen_ai.request.model') OR mapContains(attributes_string, 'gen_ai.tool.name') OR mapContains(attributes_string, 'gen_ai.agent.name'))) AND ((multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = 'api' AND multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))
AND (((mapContains(attributes_string, 'gen_ai.request.model') OR mapContains(attributes_string, 'gen_ai.tool.name') OR mapContains(attributes_string, 'gen_ai.agent.name'))) AND (multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = 'api'))
AND timestamp >= '1747947419000000000'
AND timestamp < '1747983448000000000'
AND ts_bucket_start >= 1747945619

View File

@@ -22,11 +22,11 @@ import (
type auditQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
storage qbtypes.Storage
resourceFilterStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation]
aggExprRewriter qbtypes.AggExprRewriter
fullTextColumn *telemetrytypes.TelemetryFieldKey
fl flagger.Flagger
}
var _ qbtypes.StatementBuilder[qbtypes.LogAggregation] = (*auditQueryStatementBuilder)(nil)
@@ -40,11 +40,10 @@ func NewFactory(
return factory.NewProviderFactory(
factory.MustNewName("audit"),
func(_ context.Context, settings factory.ProviderSettings, _ statementbuilder.Config) (qbtypes.StatementBuilder[qbtypes.LogAggregation], error) {
fm := audittelemetryschema.NewFieldMapper()
cb := audittelemetryschema.NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, audittelemetryschema.DefaultFullTextColumn, fm, cb, fl)
storage := audittelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, audittelemetryschema.DefaultFullTextColumn, storage, fl, telemetrytypes.SignalLogs)
return NewAuditQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, audittelemetryschema.DefaultFullTextColumn, fl,
settings, metadataStore, storage, aggExprRewriter, audittelemetryschema.DefaultFullTextColumn, fl,
), nil
},
)
@@ -53,8 +52,7 @@ func NewFactory(
func NewAuditQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
storage qbtypes.Storage,
aggExprRewriter qbtypes.AggExprRewriter,
fullTextColumn *telemetrytypes.TelemetryFieldKey,
flagger flagger.Flagger,
@@ -75,11 +73,11 @@ func NewAuditQueryStatementBuilder(
return &auditQueryStatementBuilder{
logger: auditSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
cb: conditionBuilder,
storage: storage,
resourceFilterStmtBuilder: resourceFilterStmtBuilder,
aggExprRewriter: aggExprRewriter,
fullTextColumn: fullTextColumn,
fl: flagger,
}
}
@@ -228,6 +226,7 @@ func (b *auditQueryStatementBuilder) buildListQuery(
keys map[string][]*telemetrytypes.TelemetryFieldKey,
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
info := b.queryInfo(ctx, orgID, start, end)
var (
cteFragments []string
cteArgs [][]any
@@ -263,11 +262,11 @@ func (b *auditQueryStatementBuilder) buildListQuery(
continue
}
colExpr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &query.SelectFields[index], telemetrytypes.FieldDataTypeUnspecified, keys)
colExpr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &query.SelectFields[index], telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
sb.SelectMore(colExpr)
sb.SelectMore(fmt.Sprintf("%s AS `%s`", sqlbuilder.Escape(colExpr), query.SelectFields[index].Name))
}
}
@@ -279,11 +278,11 @@ func (b *auditQueryStatementBuilder) buildListQuery(
}
for _, orderBy := range query.Order {
colExpr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
colExpr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
sb.OrderBy(fmt.Sprintf("%s %s", colExpr, orderBy.Direction.StringValue()))
sb.OrderBy(fmt.Sprintf("%s %s", sqlbuilder.Escape(colExpr), orderBy.Direction.StringValue()))
}
if query.Limit > 0 {
@@ -340,8 +339,9 @@ func (b *auditQueryStatementBuilder) buildTimeSeriesQuery(
var allGroupByArgs []any
fieldNames := make([]string, 0, len(query.GroupBy))
info := b.queryInfo(ctx, orgID, start, end)
for _, gb := range query.GroupBy {
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -475,8 +475,9 @@ func (b *auditQueryStatementBuilder) buildScalarQuery(
var allGroupByArgs []any
info := b.queryInfo(ctx, orgID, start, end)
for _, gb := range query.GroupBy {
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -566,16 +567,13 @@ func (b *auditQueryStatementBuilder) addFilterCondition(
if query.Filter != nil && query.Filter.Expression != "" {
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: orgID,
Query: b.queryInfo(ctx, orgID, start, end),
Storage: b.storage,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
SkipResourceFilter: true,
FullTextColumn: b.fullTextColumn,
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
@@ -633,3 +631,9 @@ func (b *auditQueryStatementBuilder) maybeAttachResourceFilter(
return fmt.Sprintf("__resource_filter AS (%s)", stmt.Query), stmt.Args, nil
}
// queryInfo binds one request's context; the query-path flags are evaluated
// one time here.
func (b *auditQueryStatementBuilder) queryInfo(ctx context.Context, orgID valuer.UUID, start, end uint64) qbtypes.QueryInfo {
return querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalLogs, nil, start, end)
}

View File

@@ -55,15 +55,13 @@ func newTestAuditStatementBuilder(t *testing.T) *auditQueryStatementBuilder {
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = auditFieldKeyMap()
fm := audittelemetryschema.NewFieldMapper()
cb := audittelemetryschema.NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
storage := audittelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
return NewAuditQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
audittelemetryschema.DefaultFullTextColumn,
fl,

View File

@@ -46,12 +46,11 @@ func TestStatementBuilderGroupByUnknownKey(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.useJSONBody)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
storage := logstelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore, fm, cb, aggExprRewriter,
mockMetadataStore, storage, aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)

View File

@@ -1125,7 +1125,7 @@ func TestJSONStmtBuilder_SelectField(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "SELECT timestamp, id, multiIf((dynamicElement(body_v2.`user.name`, 'String') IS NOT NULL), dynamicElement(body_v2.`user.name`, 'String'), NULL) AS `__SELECT_KEY_0_user.name` FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? LIMIT ?",
Query: "SELECT timestamp, id, dynamicElement(body_v2.`user.name`, 'String') AS `__SELECT_KEY_0_user.name` FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448), 10},
},
},
@@ -1202,7 +1202,7 @@ func TestJSONStmtBuilder_OrderBy(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, scope_name, scope_version, body_v2 as body, attributes_string, attributes_number, attributes_bool, resources_string, scope_string FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? ORDER BY multiIf((dynamicElement(body_v2.`user.name`, 'String') IS NOT NULL), dynamicElement(body_v2.`user.name`, 'String'), NULL) asc LIMIT ?",
Query: "SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, scope_name, scope_version, body_v2 as body, attributes_string, attributes_number, attributes_bool, resources_string, scope_string FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? ORDER BY dynamicElement(body_v2.`user.name`, 'String') asc LIMIT ?",
Args: []any{"1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448), 10},
},
},
@@ -1333,16 +1333,14 @@ func buildJSONTestStatementBuilder(t *testing.T, addIndexes bool) (*logQueryStat
mockMetadataStore := buildTestTelemetryMetadataStore(t, addIndexes)
fl := flaggertest.WithUseJSONBody(t, true)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
storage := logstelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,

View File

@@ -26,14 +26,13 @@ func TestSearchCostGuard(t *testing.T) {
end := uint64(releaseTime.UnixMilli())
fl := flaggertest.WithBooleanFlags(t, map[string]bool{})
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
storage := logstelemetryschema.NewStorage()
store := telemetrytypestest.NewMockMetadataStore()
store.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
sb := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
store, fm, cb, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
store, storage, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{SearchMaxScanRows: 100000, SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
@@ -57,14 +56,13 @@ func TestSearchCostGuardJSONBody(t *testing.T) {
end := uint64(releaseTime.UnixMilli())
fl := flaggertest.WithUseJSONBody(t, true)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
storage := logstelemetryschema.NewStorage()
store := telemetrytypestest.NewMockMetadataStore()
store.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
sb := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
store, fm, cb, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
store, storage, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{
SearchMaxScanRows: 100000,
SearchMaxScanRowsJSONBody: 10000,

View File

@@ -34,8 +34,7 @@ func bodyAliasExpression(bodyJSONEnabled bool) string {
type logQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
storage qbtypes.Storage
resourceFilterResolver *resourcefilter.ResourceFingerprintResolver[qbtypes.LogAggregation]
aggExprRewriter qbtypes.AggExprRewriter
fl flagger.Flagger
@@ -59,11 +58,10 @@ func NewFactory(
return factory.NewProviderFactory(
factory.MustNewName("logs"),
func(_ context.Context, settings factory.ProviderSettings, cfg statementbuilder.Config) (qbtypes.StatementBuilder[qbtypes.LogAggregation], error) {
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, logstelemetryschema.DefaultFullTextColumn, fm, cb, fl)
storage := logstelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, logstelemetryschema.DefaultFullTextColumn, storage, fl, telemetrytypes.SignalLogs)
return NewLogQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, logstelemetryschema.DefaultFullTextColumn,
settings, metadataStore, storage, aggExprRewriter, logstelemetryschema.DefaultFullTextColumn,
fl, telemetryStore, cfg,
), nil
},
@@ -73,8 +71,7 @@ func NewFactory(
func NewLogQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
storage qbtypes.Storage,
aggExprRewriter qbtypes.AggExprRewriter,
fullTextColumn *telemetrytypes.TelemetryFieldKey,
fl flagger.Flagger,
@@ -99,8 +96,7 @@ func NewLogQueryStatementBuilder(
b := &logQueryStatementBuilder{
logger: logsSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
cb: conditionBuilder,
storage: storage,
resourceFilterResolver: resourceFilterResolver,
aggExprRewriter: aggExprRewriter,
fl: fl,
@@ -322,6 +318,7 @@ func (b *logQueryStatementBuilder) buildListQuery(
keys map[string][]*telemetrytypes.TelemetryFieldKey,
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
info := b.queryInfo(ctx, orgID, start, end)
var (
cteFragments []string
@@ -349,7 +346,7 @@ func (b *logQueryStatementBuilder) buildListQuery(
sb.SelectMore(logstelemetryschema.LogsV2SeverityNumberColumn)
sb.SelectMore(logstelemetryschema.LogsV2ScopeNameColumn)
sb.SelectMore(logstelemetryschema.LogsV2ScopeVersionColumn)
sb.SelectMore(bodyAliasExpression(b.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))))
sb.SelectMore(bodyAliasExpression(info.BodyJSONOn))
sb.SelectMore(logstelemetryschema.LogsV2AttributesStringColumn)
sb.SelectMore(logstelemetryschema.LogsV2AttributesNumberColumn)
sb.SelectMore(logstelemetryschema.LogsV2AttributesBoolColumn)
@@ -364,7 +361,7 @@ func (b *logQueryStatementBuilder) buildListQuery(
}
// get column expression for the field - use array index directly to avoid pointer to loop variable
colExpr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &query.SelectFields[index], telemetrytypes.FieldDataTypeUnspecified, keys)
colExpr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &query.SelectFields[index], telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
@@ -383,7 +380,7 @@ func (b *logQueryStatementBuilder) buildListQuery(
// Add order by
for _, orderBy := range query.Order {
colExpr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
colExpr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
@@ -447,13 +444,13 @@ func (b *logQueryStatementBuilder) buildTimeSeriesQuery(
))
// Keep original column expressions so we can build the tuple
bodyJSONEnabled := b.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
info := b.queryInfo(ctx, orgID, start, end)
fieldNames := make([]string, 0, len(query.GroupBy))
for i, gb := range query.GroupBy {
if !bodyJSONEnabled && (strings.Contains(gb.Name, telemetrytypes.ArraySep) || strings.Contains(gb.Name, telemetrytypes.ArrayAnyIndex)) {
if !info.BodyJSONOn && (strings.Contains(gb.Name, telemetrytypes.ArraySep) || strings.Contains(gb.Name, telemetrytypes.ArrayAnyIndex)) {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "Group by/Aggregation isn't available for the Array Paths: %s", gb.Name)
}
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -612,13 +609,13 @@ func (b *logQueryStatementBuilder) buildScalarQuery(
allAggChArgs := []any{}
bodyJSONEnabled := b.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
info := b.queryInfo(ctx, orgID, start, end)
fieldNames := make([]string, 0, len(query.GroupBy))
for i, gb := range query.GroupBy {
if !bodyJSONEnabled && (strings.Contains(gb.Name, telemetrytypes.ArraySep) || strings.Contains(gb.Name, telemetrytypes.ArrayAnyIndex)) {
if !info.BodyJSONOn && (strings.Contains(gb.Name, telemetrytypes.ArraySep) || strings.Contains(gb.Name, telemetrytypes.ArrayAnyIndex)) {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "Group by/Aggregation isn't available for the Array Paths: %s", gb.Name)
}
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -731,16 +728,13 @@ func (b *logQueryStatementBuilder) addFilterCondition(
// add filter expression
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: orgID,
Query: b.queryInfo(ctx, orgID, start, end),
Storage: b.storage,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
SkipResourceFilter: skipResourceFilter,
FullTextColumn: b.fullTextColumn,
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
@@ -831,3 +825,9 @@ func (b *logQueryStatementBuilder) maybeAttachResourceFilter(
sb.Where("resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter)")
return fmt.Sprintf("__resource_filter AS (%s)", stmt.Query), stmt.Args, true, nil
}
// queryInfo binds one request's context; the query-path flags are evaluated
// one time here.
func (b *logQueryStatementBuilder) queryInfo(ctx context.Context, orgID valuer.UUID, start, end uint64) qbtypes.QueryInfo {
return querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalLogs, nil, start, end)
}

View File

@@ -106,7 +106,7 @@ func TestStatementBuilderTimeSeries(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, countDistinct(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (attributes_string['http.method'] = ? AND mapContains(attributes_string, 'http.method'))) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, countDistinct(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (attributes_string['http.method'] = ? AND mapContains(attributes_string, 'http.method'))) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, countDistinct(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR (attributes_string['http.method'] = ? AND mapContains(attributes_string, 'http.method'))) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, countDistinct(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR (attributes_string['http.method'] = ? AND mapContains(attributes_string, 'http.method'))) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"redis-manual", "GET", "1705226400000000000", uint64(1705224600), "1705485600000000000", uint64(1705485600), 10, "redis-manual", "GET", "1705226400000000000", uint64(1705224600), "1705485600000000000", uint64(1705485600)},
},
expectedErr: nil,
@@ -218,16 +218,14 @@ func TestStatementBuilderTimeSeries(t *testing.T) {
mockMetadataStore.KeysMap = keysMap
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
storage := logstelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -356,20 +354,18 @@ func TestStatementBuilderListQuery(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fm := logstelemetryschema.NewFieldMapper(fl)
storage := logstelemetryschema.NewStorage()
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -509,19 +505,17 @@ func TestStatementBuilderListQueryResourceTests(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fm := logstelemetryschema.NewFieldMapper(fl)
storage := logstelemetryschema.NewStorage()
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -579,26 +573,24 @@ func TestStatementBuilderTimeSeriesBodyGroupBy(t *testing.T) {
},
},
},
expectedErrContains: "Operation isn't available for the body column",
expectedErrContains: "field cannot be selected",
},
}
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fm := logstelemetryschema.NewFieldMapper(fl)
storage := logstelemetryschema.NewStorage()
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -684,17 +676,15 @@ func TestStatementBuilderListQueryServiceCollision(t *testing.T) {
ctx := context.Background()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fl := flaggertest.New(t)
fm := logstelemetryschema.NewFieldMapper(fl)
storage := logstelemetryschema.NewStorage()
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMapCollision()
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -908,18 +898,16 @@ func TestAdjustKey(t *testing.T) {
}
fl := flaggertest.New(t)
fm := logstelemetryschema.NewFieldMapper(fl)
storage := logstelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMapCollision()
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -1052,20 +1040,18 @@ func TestStmtBuilderBodyField(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.enableUseJSONBody)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
storage := logstelemetryschema.NewStorage()
// build the key map
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
for _, field := range logstelemetryschema.IntrinsicFields {
f := field
mockMetadataStore.KeysMap[field.Name] = append(mockMetadataStore.KeysMap[field.Name], &f)
}
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -1152,20 +1138,18 @@ func TestStmtBuilderBodyFullTextSearch(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.enableUseJSONBody)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
storage := logstelemetryschema.NewStorage()
// build the key map
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
for _, field := range logstelemetryschema.IntrinsicFields {
f := field
mockMetadataStore.KeysMap[field.Name] = append(mockMetadataStore.KeysMap[field.Name], &f)
}
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -1269,24 +1253,16 @@ func newSkipResourceFingerprintLogsBuilder(
t.Helper()
fl := flaggertest.New(t)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
storage := logstelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC))
aggExprRewriter := querybuilder.NewAggExprRewriter(
instrumentationtest.New().ToProviderSettings(),
logstelemetryschema.DefaultFullTextColumn,
fm,
cb,
fl,
)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), logstelemetryschema.DefaultFullTextColumn, storage, fl, telemetrytypes.SignalLogs)
return NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,

View File

@@ -23,8 +23,7 @@ import (
type meterQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
storage qbtypes.Storage
metricsStatementBuilder *metricsstatementbuilder.StatementBuilder
}
@@ -44,9 +43,7 @@ func NewFactory(
if err != nil {
return nil, err
}
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
return NewMeterQueryStatementBuilder(settings, metadataStore, fm, cb, metricsStatementBuilder), nil
return NewMeterQueryStatementBuilder(settings, metadataStore, metricstelemetryschema.NewStorage(), metricsStatementBuilder), nil
},
)
}
@@ -54,8 +51,7 @@ func NewFactory(
func NewMeterQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
storage qbtypes.Storage,
metricsStatementBuilder *metricsstatementbuilder.StatementBuilder,
) *meterQueryStatementBuilder {
metricsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema")
@@ -63,8 +59,7 @@ func NewMeterQueryStatementBuilder(
return &meterQueryStatementBuilder{
logger: metricsSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
cb: conditionBuilder,
storage: storage,
metricsStatementBuilder: metricsStatementBuilder,
}
}
@@ -150,8 +145,9 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDeltaFastPath(
"toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(%d)) AS ts",
stepSec,
))
info := b.queryInfo(ctx, orgID, start, end, query.Aggregations[0].MetricName)
for i, g := range query.GroupBy {
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
col, err := querybuilder.ResolveColumn(ctx, info, b.storage, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return "", nil, err
}
@@ -176,16 +172,13 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDeltaFastPath(
)
if query.Filter != nil && query.Filter.Expression != "" {
filterWhere, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
StartNs: start,
EndNs: end,
Context: ctx,
Query: info,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
})
if err != nil {
return "", nil, err
@@ -239,8 +232,9 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDelta(
stepSec,
))
info := b.queryInfo(ctx, orgID, start, end, query.Aggregations[0].MetricName)
for i, g := range query.GroupBy {
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
col, err := querybuilder.ResolveColumn(ctx, info, b.storage, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return "", nil, err
}
@@ -268,16 +262,13 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDelta(
if query.Filter != nil && query.Filter.Expression != "" {
filterWhere, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
StartNs: start,
EndNs: end,
Context: ctx,
Query: info,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
})
if err != nil {
return "", nil, err
@@ -317,8 +308,9 @@ func (b *meterQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
"toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(%d)) AS ts",
stepSec,
))
info := b.queryInfo(ctx, orgID, start, end, query.Aggregations[0].MetricName)
for i, g := range query.GroupBy {
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
col, err := querybuilder.ResolveColumn(ctx, info, b.storage, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return "", nil, err
}
@@ -340,16 +332,13 @@ func (b *meterQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
)
if query.Filter != nil && query.Filter.Expression != "" {
filterWhere, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
StartNs: start,
EndNs: end,
Context: ctx,
Query: info,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
})
if err != nil {
return "", nil, err
@@ -428,3 +417,9 @@ func (b *meterQueryStatementBuilder) buildSpatialAggregationCTE(
q, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
return fmt.Sprintf("__spatial_aggregation_cte AS (%s)", q), args, nil
}
// queryInfo binds one request's context. The meter has no flagger, so
// resolution stays literal.
func (b *meterQueryStatementBuilder) queryInfo(ctx context.Context, orgID valuer.UUID, start, end uint64, metricName string) qbtypes.QueryInfo {
return querybuilder.NewQueryInfo(ctx, orgID, nil, telemetrytypes.SignalMetrics, &telemetrytypes.MetricContext{MetricName: metricName}, start, end)
}

View File

@@ -162,8 +162,7 @@ func TestStatementBuilder(t *testing.T) {
},
}
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
storage := metricstelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
if err != nil {
@@ -173,13 +172,12 @@ func TestStatementBuilder(t *testing.T) {
flagger := flaggertest.New(t)
metricStmtBuilder := metricsstatementbuilder.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, fm, cb, flagger)
metricStmtBuilder := metricsstatementbuilder.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, storage, flagger)
statementBuilder := NewMeterQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
metricStmtBuilder,
)
@@ -202,8 +200,7 @@ func TestStatementBuilder(t *testing.T) {
}
func TestGroupByAliasAvoidsColumnCollision(t *testing.T) {
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
storage := metricstelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
require.NoError(t, err)
@@ -214,9 +211,8 @@ func TestGroupByAliasAvoidsColumnCollision(t *testing.T) {
statementBuilder := NewMeterQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
metricsstatementbuilder.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, fm, cb, flagger),
storage,
metricsstatementbuilder.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, storage, flagger),
)
for _, groupBy := range []string{"ts", "value", "fingerprint", "service.name"} {

View File

@@ -144,11 +144,10 @@ func TestReducedStatementBuilder(t *testing.T) {
},
}
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
storage := metricstelemetryschema.NewStorage()
fl, err := flagger.New(context.Background(), instrumentationtest.New().ToProviderSettings(), flagger.Config{}, flagger.MustNewRegistry())
require.NoError(t, err)
sb := NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), telemetrytypestest.NewMockMetadataStore(), fm, cb, fl)
sb := NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), telemetrytypestest.NewMockMetadataStore(), storage, fl)
const start, end = uint64(1747000000000), uint64(1747172800000)

View File

@@ -34,8 +34,7 @@ const (
type StatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
storage qbtypes.Storage
flagger flagger.Flagger
}
@@ -51,9 +50,7 @@ func NewFactory(
return factory.NewProviderFactory(
factory.MustNewName("metrics"),
func(_ context.Context, settings factory.ProviderSettings, _ statementbuilder.Config) (*StatementBuilder, error) {
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
return NewMetricQueryStatementBuilder(settings, metadataStore, fm, cb, fl), nil
return NewMetricQueryStatementBuilder(settings, metadataStore, metricstelemetryschema.NewStorage(), fl), nil
},
)
}
@@ -61,16 +58,14 @@ func NewFactory(
func NewMetricQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
storage qbtypes.Storage,
flagger flagger.Flagger,
) *StatementBuilder {
metricsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema")
return &StatementBuilder{
logger: metricsSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
cb: conditionBuilder,
storage: storage,
flagger: flagger,
}
}
@@ -254,21 +249,19 @@ func (b *StatementBuilder) buildReducedTimeSeriesCTE(
variables map[string]qbtypes.VariableItem,
) (string, []any, error) {
sb := sqlbuilder.NewSelectBuilder()
info := b.queryInfo(ctx, orgID, start, end, query.Aggregations[0].MetricName)
var preparedWhereClause querybuilder.PreparedWhereClause
var err error
if query.Filter != nil && query.Filter.Expression != "" {
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
StartNs: start,
EndNs: end,
Context: ctx,
Query: info,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
})
if err != nil {
return "", nil, err
@@ -278,7 +271,7 @@ func (b *StatementBuilder) buildReducedTimeSeriesCTE(
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedLocalTableName))
sb.Select("fingerprint")
for i, g := range query.GroupBy {
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
col, err := querybuilder.ResolveColumn(ctx, info, b.storage, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return "", nil, err
}
@@ -474,22 +467,20 @@ func (b *StatementBuilder) buildTimeSeriesCTE(
tsTable string,
) (string, []any, []string, error) {
sb := sqlbuilder.NewSelectBuilder()
info := b.queryInfo(ctx, orgID, start, end, query.Aggregations[0].MetricName)
var preparedWhereClause querybuilder.PreparedWhereClause
var err error
if query.Filter != nil && query.Filter.Expression != "" {
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
StartNs: start,
EndNs: end,
Context: ctx,
Query: info,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
})
if err != nil {
return "", nil, nil, err
@@ -500,7 +491,7 @@ func (b *StatementBuilder) buildTimeSeriesCTE(
sb.Select("fingerprint")
for i, g := range query.GroupBy {
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
col, err := querybuilder.ResolveColumn(ctx, info, b.storage, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return "", nil, nil, err
}
@@ -876,3 +867,9 @@ func GroupByAliases(groupBy []qbtypes.GroupByKey) []string {
}
return aliases
}
// queryInfo binds one request's context; the query-path flags are evaluated
// one time here.
func (b *StatementBuilder) queryInfo(ctx context.Context, orgID valuer.UUID, start, end uint64, metricName string) qbtypes.QueryInfo {
return querybuilder.NewQueryInfo(ctx, orgID, b.flagger, telemetrytypes.SignalMetrics, &telemetrytypes.MetricContext{MetricName: metricName}, start, end)
}

View File

@@ -376,7 +376,7 @@ func TestStatementBuilder(t *testing.T) {
expected: qbtypes.Statement{
Query: "WITH __temporal_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_k8s.statefulset.name`, multiIf(row_number() OVER rate_window = 1, nan, (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) < 0, per_series_value / (ts - lagInFrame(ts, 1) OVER rate_window), (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) / (ts - lagInFrame(ts, 1) OVER rate_window)) AS per_series_value FROM (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_k8s.statefulset.name`, max(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'k8s.statefulset.name') AS `__GROUP_BY_KEY_0_k8s.statefulset.name` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) AND JSONExtractString(labels, 'k8s.statefulset.name') = ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_k8s.statefulset.name`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_k8s.statefulset.name` ORDER BY fingerprint, ts) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_k8s.statefulset.name`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_k8s.statefulset.name`) SELECT * FROM __spatial_aggregation_cte ORDER BY `__GROUP_BY_KEY_0_k8s.statefulset.name`, ts",
Args: []any{"signoz_calls_total", uint64(1747936800000), uint64(1747983420000), "cumulative", "my-statefulset", "signoz_calls_total", uint64(1747947360000), uint64(1747983420000), 0},
Warnings: []string{"label `k8s.statefulset.name` not found in metadata; check the label name for typos"},
Warnings: []string{"key `k8s.statefulset.name` not found in metadata; querying the underlying data directly. If this is unexpected, check the key name for typos."},
},
expectedErr: nil,
},
@@ -414,8 +414,7 @@ func TestStatementBuilder(t *testing.T) {
},
}
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
storage := metricstelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
if err != nil {
@@ -438,8 +437,7 @@ func TestStatementBuilder(t *testing.T) {
statementBuilder := NewMetricQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
flagger,
)
@@ -462,8 +460,7 @@ func TestStatementBuilder(t *testing.T) {
}
func TestGroupByAliasAvoidsColumnCollision(t *testing.T) {
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
storage := metricstelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
require.NoError(t, err)
@@ -475,8 +472,7 @@ func TestGroupByAliasAvoidsColumnCollision(t *testing.T) {
statementBuilder := NewMetricQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
fl,
)

View File

@@ -5,27 +5,12 @@ import (
"fmt"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
type defaultConditionBuilder struct {
fm qbtypes.FieldMapper
// fl evaluates the resolve_semconv_families flag during resolution.
// A nil flagger keeps resolution literal.
fl flagger.Flagger
}
var _ qbtypes.ConditionBuilder = (*defaultConditionBuilder)(nil)
func NewConditionBuilder(fm qbtypes.FieldMapper, fl flagger.Flagger) *defaultConditionBuilder {
return &defaultConditionBuilder{fm: fm, fl: fl}
}
func valueForIndexFilter(op qbtypes.FilterOperator, key *telemetrytypes.TelemetryFieldKey, value any) any {
switch v := value.(type) {
case []any:
@@ -112,55 +97,20 @@ func memberPresenceCondition(sb *sqlbuilder.SelectBuilder, column string, member
return sb.And(conditions...)
}
// SkipResourceFilter is not applicable here: the fingerprint table only stores resource attributes.
func (b *defaultConditionBuilder) ConditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
op qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
matches := querybuilder.MatchingLogicalFields(ctx, orgID, b.fl, key, fieldKeys)
// has/hasAny/hasAll/hasToken are logs-body-only functions; they never apply to the
// resource fingerprint table, so skip them (the main query still evaluates them).
if op.IsFunctionOperator() {
return nil, nil, nil
// Compile weaves the index hints of the fingerprint table into each
// operator. It reads the members directly, so a member with a value map must
// translate its operand through StoredValues before the hint is built.
func (b *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, op qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
condition, err := b.conditionForLogicalField(ctx, q, logical, op, value, sb)
if err != nil {
return qbtypes.Compiled{}, err
}
logicalFields, warning := querybuilder.ResolveLogicalFields(key, matches)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
conds := make([]string, 0, len(logicalFields))
for _, logical := range logicalFields {
// the resource fingerprint table only stores resource attributes; fields from
// any other context contribute no condition and are omitted. An empty result
// (including an unknown key) lets the caller skip this filter entirely.
if logical.FieldContext != telemetrytypes.FieldContextResource {
continue
}
cond, err := b.conditionForLogicalField(ctx, orgID, startNs, endNs, logical, op, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
}
return conds, warnings, nil
return qbtypes.Compiled{Condition: condition}, nil
}
func (b *defaultConditionBuilder) conditionForLogicalField(
func (b *storage) conditionForLogicalField(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
q qbtypes.QueryInfo,
logical *telemetrytypes.LogicalField,
op qbtypes.FilterOperator,
value any,
@@ -173,7 +123,7 @@ func (b *defaultConditionBuilder) conditionForLogicalField(
// Every resource-context key maps to the labels column, so any member
// resolves the column for the whole field.
columns, err := b.fm.ColumnFor(ctx, orgID, startNs, endNs, logical.Single())
columns, err := b.getColumn(ctx, q.StartNs, q.EndNs, logical.Single())
if err != nil {
return "", err
}
@@ -191,7 +141,7 @@ func (b *defaultConditionBuilder) conditionForLogicalField(
keyIdxFilter := keyIndexCondition(sb, column.Name, members)
singleValueIndexFilter := valueForIndexFilter(op, members[0], value)
fieldName, err := querybuilder.LogicalValueExpr(ctx, orgID, startNs, endNs, b.fm, logical)
fieldName, err := querybuilder.LogicalValueExpr(ctx, q, b, logical)
if err != nil {
return "", err
}

View File

@@ -2,12 +2,11 @@ package resourcefilter
import (
"context"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -201,13 +200,12 @@ func TestConditionBuilder(t *testing.T) {
},
}
fm := NewFieldMapper()
conditionBuilder := NewConditionBuilder(fm, flaggertest.New(t))
storage := newStorage()
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, _, err := conditionBuilder.ConditionFor(context.Background(), valuer.UUID{}, 0, 0, tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.op, tc.value, sb)
cond, _, err := querybuilder.Conditions(context.Background(), qbtypes.QueryInfo{}, storage, tc.key, tc.op, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {tc.key}}, false, sb)
sb.Where(cond...)
if tc.expectedErr != nil {

View File

@@ -2,6 +2,7 @@ package resourcefilter
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
"github.com/SigNoz/signoz/pkg/flagger"
@@ -18,7 +19,8 @@ import (
// the metadata map as trace resource attributes.
func TestFamilyEqualWidensIndexHintsToAnyMember(t *testing.T) {
cb := NewConditionBuilder(NewFieldMapper(), flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true}))
fl := flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true})
storage := newStorage()
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {{
Name: "deployment.environment.name",
@@ -35,9 +37,7 @@ func TestFamilyEqualWidensIndexHintsToAnyMember(t *testing.T) {
}
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, 0, 0,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "production", sb)
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorEqual, "production", fieldKeys, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -47,7 +47,8 @@ func TestFamilyEqualWidensIndexHintsToAnyMember(t *testing.T) {
}
func TestFamilyNotEqualDropsNegatedValueHint(t *testing.T) {
cb := NewConditionBuilder(NewFieldMapper(), flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true}))
fl := flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true})
storage := newStorage()
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {{
Name: "deployment.environment.name",
@@ -64,9 +65,7 @@ func TestFamilyNotEqualDropsNegatedValueHint(t *testing.T) {
}
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, 0, 0,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorNotEqual, "production", sb)
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorNotEqual, "production", fieldKeys, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -76,7 +75,8 @@ func TestFamilyNotEqualDropsNegatedValueHint(t *testing.T) {
}
func TestFamilyExistsIsAnyMemberPresence(t *testing.T) {
cb := NewConditionBuilder(NewFieldMapper(), flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true}))
fl := flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true})
storage := newStorage()
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {{
Name: "deployment.environment.name",
@@ -93,9 +93,7 @@ func TestFamilyExistsIsAnyMemberPresence(t *testing.T) {
}
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, 0, 0,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorExists, nil, sb)
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorExists, nil, fieldKeys, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -104,9 +102,7 @@ func TestFamilyExistsIsAnyMemberPresence(t *testing.T) {
require.Equal(t, []any{true, true, "%deployment.environment.name%", "%deployment.environment%"}, args)
sb = sqlbuilder.NewSelectBuilder()
conds, _, err = cb.ConditionFor(context.Background(), valuer.UUID{}, 0, 0,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorNotExists, nil, sb)
conds, _, err = querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorNotExists, nil, fieldKeys, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -118,7 +114,8 @@ func TestFamilyExistsIsAnyMemberPresence(t *testing.T) {
// With only one member in metadata the SQL keeps the exact pre-family shape,
// including the negated value hint on !=.
func TestSingleMemberShapesUnchanged(t *testing.T) {
cb := NewConditionBuilder(NewFieldMapper(), flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true}))
fl := flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true})
storage := newStorage()
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {{
Name: "deployment.environment.name",
@@ -129,9 +126,7 @@ func TestSingleMemberShapesUnchanged(t *testing.T) {
}
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, 0, 0,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorNotEqual, "production", sb)
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorNotEqual, "production", fieldKeys, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)

View File

@@ -5,9 +5,9 @@ import (
"fmt"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
var (
@@ -18,21 +18,15 @@ var (
}
)
type defaultFieldMapper struct{}
type storage struct{}
var _ qbtypes.FieldMapper = (*defaultFieldMapper)(nil)
var _ qbtypes.Storage = (*storage)(nil)
// CandidateKeys returns nil: the resource filter has no attribute-map fallback, so a
// context-missing key stays unresolved and the caller errors.
func (m *defaultFieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, _ *telemetrytypes.TelemetryFieldKey, _ any, _ map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
func newStorage() *storage {
return &storage{}
}
func NewFieldMapper() *defaultFieldMapper {
return &defaultFieldMapper{}
}
func (m *defaultFieldMapper) getColumn(
func (m *storage) getColumn(
_ context.Context,
_, _ uint64,
key *telemetrytypes.TelemetryFieldKey,
@@ -46,22 +40,8 @@ func (m *defaultFieldMapper) getColumn(
return nil, qbtypes.ErrColumnNotFound
}
func (m *defaultFieldMapper) ColumnFor(
ctx context.Context,
_ valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
) ([]*schema.Column, error) {
return m.getColumn(ctx, tsStart, tsEnd, key)
}
func (m *defaultFieldMapper) FieldFor(
ctx context.Context,
_ valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
) (string, error) {
columns, err := m.getColumn(ctx, tsStart, tsEnd, key)
func (m *storage) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, q.StartNs, q.EndNs, key)
if err != nil {
return "", err
}
@@ -71,44 +51,38 @@ func (m *defaultFieldMapper) FieldFor(
return columns[0].Name, nil
}
// ExistsFor reports key presence in the fingerprint labels JSON. Only resource
// context keys have a presence notion here; anything else is a real column and
// always present.
func (m *defaultFieldMapper) ExistsFor(
ctx context.Context,
_ valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
exists bool,
) (string, error) {
columns, err := m.getColumn(ctx, tsStart, tsEnd, key)
// Exists reports key presence in the fingerprint labels JSON. Only resource
// context keys have a presence notion here; anything else is a real column
// and always present.
func (m *storage) Exists(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, exists bool) (qbtypes.Existence, error) {
columns, err := m.getColumn(ctx, q.StartNs, q.EndNs, key)
if err != nil {
return "", err
return qbtypes.Existence{}, err
}
if key.FieldContext != telemetrytypes.FieldContextResource {
if exists {
return "true", nil
predicate := "true"
if !exists {
predicate = "false"
}
return "false", nil
return qbtypes.Existence{Predicate: predicate, WhenAbsent: qbtypes.AlwaysPresent}, nil
}
pred := fmt.Sprintf("simpleJSONHas(%s, '%s')", columns[0].Name, key.Name)
if exists {
return pred, nil
predicate := fmt.Sprintf("simpleJSONHas(%s, '%s')", columns[0].Name, key.Name)
if !exists {
predicate = "NOT " + predicate
}
return "NOT " + pred, nil
return qbtypes.Existence{Predicate: predicate, WhenAbsent: qbtypes.AbsentIsSentinel}, nil
}
func (m *defaultFieldMapper) ColumnExpressionFor(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
_ telemetrytypes.FieldDataType,
_ map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
fieldExpression, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
return "", err
}
return fmt.Sprintf("%s AS `%s`", fieldExpression, key.Name), nil
// Fallback returns nil: the fingerprint table holds only what the metadata
// reports, and a term it cannot serve is the main query's to evaluate.
func (m *storage) Fallback(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, qbtypes.FilterOperator, any) ([]*telemetrytypes.LogicalField, error) {
return nil, nil
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{Split: qbtypes.FingerprintOfSplit, UnknownKey: qbtypes.IgnoreUnknownKey}
}
func (m *storage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
return querybuilder.DefaultRead(ctx, q, m, logical)
}

View File

@@ -16,15 +16,14 @@ import (
// resourceFilterStatementBuilder builds resource fingerprint filter CTEs.
type resourceFilterStatementBuilder[T any] struct {
logger *slog.Logger
dbName string
tableName string
fieldMapper qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
metadataStore telemetrytypes.MetadataStore
signal telemetrytypes.Signal
source telemetrytypes.Source
flagger flagger.Flagger
logger *slog.Logger
dbName string
tableName string
storage qbtypes.Storage
metadataStore telemetrytypes.MetadataStore
signal telemetrytypes.Signal
source telemetrytypes.Source
flagger flagger.Flagger
fullTextColumn *telemetrytypes.TelemetryFieldKey
}
@@ -46,19 +45,16 @@ func New[T any](
fl flagger.Flagger,
) *resourceFilterStatementBuilder[T] {
set := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/statementbuilder/resourcefilter")
fm := NewFieldMapper()
cb := NewConditionBuilder(fm, fl)
return &resourceFilterStatementBuilder[T]{
logger: set.Logger(),
dbName: dbName,
tableName: tableName,
fieldMapper: fm,
conditionBuilder: cb,
metadataStore: metadataStore,
signal: signal,
source: source,
flagger: fl,
fullTextColumn: fullTextColumn,
logger: set.Logger(),
dbName: dbName,
tableName: tableName,
storage: newStorage(),
metadataStore: metadataStore,
signal: signal,
source: source,
flagger: fl,
fullTextColumn: fullTextColumn,
}
}
@@ -163,19 +159,15 @@ func (b *resourceFilterStatementBuilder[T]) addConditions(
// warnings would be encountered as part of the main condition already
filterWhereClause, err := querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: orgID,
Flagger: b.flagger,
Query: querybuilder.NewQueryInfo(ctx, orgID, b.flagger, b.signal, nil, start, end),
Storage: b.storage,
Logger: b.logger,
FieldMapper: b.fieldMapper,
ConditionBuilder: b.conditionBuilder,
FieldKeys: keys,
FullTextColumn: b.fullTextColumn,
SkipFullTextFilter: true,
// the resource-filter condition builder ignores keys it can't resolve (and
// the resource-filter storage ignores keys it can't resolve (and
// skips function calls), so no "key not found" error arises here.
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {

View File

@@ -8,7 +8,6 @@ import (
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
// Aggregate renders one column's SQL through the resolvers and lists the attribute
@@ -16,7 +15,7 @@ import (
// constructors below; the zero value is not usable.
type Aggregate struct {
keys []*telemetrytypes.TelemetryFieldKey
render func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (expr string, err error)
render func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (expr string, err error)
}
// IntrinsicSpanKey references an intrinsic span-index field (timestamp, name, …).
@@ -47,15 +46,15 @@ const (
// CountAll renders count().
func CountAll() Aggregate {
return Aggregate{render: func(context.Context, valuer.UUID, uint64, uint64, *columnResolver, *predicateResolver) (string, error) {
return Aggregate{render: func(context.Context, qbtypes.QueryInfo, *columnResolver, *predicateResolver) (string, error) {
return "count()", nil
}}
}
// FieldReduce renders <fn>(<field>) over a field-mapper-resolved column.
func FieldReduce(fn AggFunc, key *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, _ *predicateResolver) (string, error) {
f, err := cols.FieldFor(ctx, orgID, startNs, endNs, key)
return Aggregate{render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, _ *predicateResolver) (string, error) {
f, err := cols.Read(ctx, q, key)
if err != nil {
return "", err
}
@@ -66,12 +65,12 @@ func FieldReduce(fn AggFunc, key *telemetrytypes.TelemetryFieldKey) Aggregate {
// TraceDuration renders the full-trace wall duration: last span end minus first
// span start.
func TraceDuration(tsKey, durationKey *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, _ *predicateResolver) (string, error) {
ts, err := cols.FieldFor(ctx, orgID, startNs, endNs, tsKey)
return Aggregate{render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, _ *predicateResolver) (string, error) {
ts, err := cols.Read(ctx, q, tsKey)
if err != nil {
return "", err
}
dur, err := cols.FieldFor(ctx, orgID, startNs, endNs, durationKey)
dur, err := cols.Read(ctx, q, durationKey)
if err != nil {
return "", err
}
@@ -83,52 +82,52 @@ func TraceDuration(tsKey, durationKey *telemetrytypes.TelemetryFieldKey) Aggrega
// FieldAnyWhere renders anyIf(<field>, <cond>) — the field value from any span
// matching the condition.
func FieldAnyWhere(valueKey, condKey *telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, condValue any) Aggregate {
return Aggregate{render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.FieldFor(ctx, orgID, startNs, endNs, valueKey)
return Aggregate{render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.Read(ctx, q, valueKey)
if err != nil {
return "", err
}
cond, err := preds.ConditionFor(ctx, orgID, startNs, endNs, condKey, op, condValue)
cond, err := preds.ConditionFor(ctx, q, condKey, op, condValue)
return fmt.Sprintf("anyIf(%s, %s)", v, cond), err
}}
}
// AnyValue renders any(<value>) over a metadata-resolved attribute value.
func AnyValue(key *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.FieldDataType) Aggregate {
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{key}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, _ *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, orgID, startNs, endNs, key, dt)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{key}, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, _ *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, q, key, dt)
return fmt.Sprintf("any(%s)", v), err
}}
}
// CountExists renders countIf(<key> EXISTS) — counts spans carrying key.
func CountExists(key *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{key}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, _ *columnResolver, preds *predicateResolver) (string, error) {
cond, err := preds.ExistsFor(ctx, orgID, startNs, endNs, key)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{key}, render: func(ctx context.Context, q qbtypes.QueryInfo, _ *columnResolver, preds *predicateResolver) (string, error) {
cond, err := preds.ExistsFor(ctx, q, key)
return fmt.Sprintf("countIf(%s)", cond), err
}}
}
// CondCount renders countIf(<cond>) over a condition-builder-resolved predicate.
func CondCount(key *telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, value any) Aggregate {
return Aggregate{render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, _ *columnResolver, preds *predicateResolver) (string, error) {
cond, err := preds.ConditionFor(ctx, orgID, startNs, endNs, key, op, value)
return Aggregate{render: func(ctx context.Context, q qbtypes.QueryInfo, _ *columnResolver, preds *predicateResolver) (string, error) {
cond, err := preds.ConditionFor(ctx, q, key, op, value)
return fmt.Sprintf("countIf(%s)", cond), err
}}
}
// Reduce renders <fn>(<value>) over a resolved numeric attribute value.
func Reduce(fn AggFunc, valueKey *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, _ *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, orgID, startNs, endNs, valueKey, telemetrytypes.FieldDataTypeFloat64)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, _ *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, q, valueKey, telemetrytypes.FieldDataTypeFloat64)
return fmt.Sprintf("%s(%s)", fn, v), err
}}
}
// ScopedReduce renders <fn>If(<field>, <gate mask>) over a field-mapper-resolved column.
func ScopedReduce(fn AggFunc, key *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (string, error) {
f, err := cols.FieldFor(ctx, orgID, startNs, endNs, key)
return Aggregate{render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (string, error) {
f, err := cols.Read(ctx, q, key)
if err != nil {
return "", err
}
@@ -139,12 +138,12 @@ func ScopedReduce(fn AggFunc, key *telemetrytypes.TelemetryFieldKey) Aggregate {
// ScopedToKeyColumn renders <fn>If(<field>, <scopeKey> EXISTS) — a span-index field
// aggregated over spans carrying scopeKey (e.g. max LLM latency).
func ScopedToKeyColumn(fn AggFunc, columnKey, scopeKey *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{scopeKey}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (string, error) {
col, err := cols.FieldFor(ctx, orgID, startNs, endNs, columnKey)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{scopeKey}, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (string, error) {
col, err := cols.Read(ctx, q, columnKey)
if err != nil {
return "", err
}
cond, err := preds.ExistsFor(ctx, orgID, startNs, endNs, scopeKey)
cond, err := preds.ExistsFor(ctx, q, scopeKey)
return fmt.Sprintf("%sIf(%s, %s)", fn, col, cond), err
}}
}
@@ -156,28 +155,28 @@ func PickBy(valueKey *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.FieldD
if dir == PickEarliest {
fn = "argMinIf"
}
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, orgID, startNs, endNs, valueKey, dt)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, q, valueKey, dt)
if err != nil {
return "", err
}
order, err := cols.FieldFor(ctx, orgID, startNs, endNs, orderKey)
order, err := cols.Read(ctx, q, orderKey)
if err != nil {
return "", err
}
cond, err := preds.ExistsFor(ctx, orgID, startNs, endNs, valueKey)
cond, err := preds.ExistsFor(ctx, q, valueKey)
return fmt.Sprintf("%s(%s, %s, %s)", fn, v, order, cond), err
}}
}
// UniqCount renders uniqIf(<value>, <value> EXISTS) — distinct count of an attribute.
func UniqCount(valueKey *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.FieldDataType) Aggregate {
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, orgID, startNs, endNs, valueKey, dt)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, q, valueKey, dt)
if err != nil {
return "", err
}
cond, err := preds.ExistsFor(ctx, orgID, startNs, endNs, valueKey)
cond, err := preds.ExistsFor(ctx, q, valueKey)
return fmt.Sprintf("uniqIf(%s, %s)", v, cond), err
}}
}
@@ -185,10 +184,10 @@ func UniqCount(valueKey *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.Fie
// SumOfKeys renders coalesce(sum(<v1>), 0) + coalesce(sum(<v2>), 0) + …; coalesced
// because a key absent from every span sums to NULL and NULL + n = NULL.
func SumOfKeys(dt telemetrytypes.FieldDataType, valueKeys ...*telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{keys: valueKeys, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, _ *predicateResolver) (string, error) {
return Aggregate{keys: valueKeys, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, _ *predicateResolver) (string, error) {
parts := make([]string, 0, len(valueKeys))
for _, k := range valueKeys {
v, err := cols.ValueFor(ctx, orgID, startNs, endNs, k, dt)
v, err := cols.ValueFor(ctx, q, k, dt)
if err != nil {
return "", err
}

View File

@@ -3,35 +3,36 @@ package scopedtracesstatementbuilder
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
// columnResolver resolves keys to bare column/value expressions through the shared
// field mapper. It binds no args, so its expressions embed in any builder; predicates
// (which do bind args) are the predicateResolver's job.
// columnResolver resolves keys to bare column/value expressions through the
// shared storage. It binds no args, so its expressions embed in any builder;
// predicates (which do bind args) are the predicateResolver's job.
type columnResolver struct {
fm qbtypes.FieldMapper
keys map[string][]*telemetrytypes.TelemetryFieldKey
storage qbtypes.Storage
keys map[string][]*telemetrytypes.TelemetryFieldKey
}
func newColumnResolver(fm qbtypes.FieldMapper, keys map[string][]*telemetrytypes.TelemetryFieldKey) *columnResolver {
return &columnResolver{fm: fm, keys: keys}
func newColumnResolver(storage qbtypes.Storage, keys map[string][]*telemetrytypes.TelemetryFieldKey) *columnResolver {
return &columnResolver{storage: storage, keys: keys}
}
func (r *columnResolver) FieldFor(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return r.fm.FieldFor(ctx, orgID, startNs, endNs, key)
// Read returns the bare read of one field key.
func (r *columnResolver) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return r.storage.Read(ctx, q, key)
}
// ValueFor returns the value expression for an attribute key.
func (r *columnResolver) ValueFor(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.FieldDataType) (string, error) {
func (r *columnResolver) ValueFor(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.FieldDataType) (string, error) {
// TODO(nitya): Fix this as this is not correct way
if cands := r.keys[key.Name]; len(cands) > 0 {
key = cands[0]
}
expr, err := r.fm.ColumnExpressionFor(ctx, orgID, startNs, endNs, key, dt, r.keys)
expr, err := querybuilder.ResolveColumn(ctx, q, r.storage, key, dt, r.keys)
if err != nil {
return "", err
}

View File

@@ -3,31 +3,31 @@ package scopedtracesstatementbuilder
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
// predicateResolver resolves key + operator + value to boolean predicates through the
// shared condition builder. Args bind into sb as $n markers, so returned predicates
// can be embedded anywhere in sb; maskExpr is set by the builder after resolveMask
// shared storage. Args bind into sb as $n markers, so returned predicates can be
// embedded anywhere in sb; maskExpr is set by the builder after resolveMask
// (Scoped* aggregates embed it).
type predicateResolver struct {
cb qbtypes.ConditionBuilder
storage qbtypes.Storage
keys map[string][]*telemetrytypes.TelemetryFieldKey
sb *sqlbuilder.SelectBuilder
maskExpr string
}
func newPredicateResolver(cb qbtypes.ConditionBuilder, keys map[string][]*telemetrytypes.TelemetryFieldKey, sb *sqlbuilder.SelectBuilder) *predicateResolver {
return &predicateResolver{cb: cb, keys: keys, sb: sb}
func newPredicateResolver(storage qbtypes.Storage, keys map[string][]*telemetrytypes.TelemetryFieldKey, sb *sqlbuilder.SelectBuilder) *predicateResolver {
return &predicateResolver{storage: storage, keys: keys, sb: sb}
}
// ConditionFor returns a boolean predicate for key via the condition builder
// (materialized column when present, else map access), args bound into sb.
func (r *predicateResolver) ConditionFor(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, value any) (string, error) {
conds, _, err := r.cb.ConditionFor(ctx, orgID, startNs, endNs, key, r.keys, qbtypes.ConditionBuilderOptions{}, op, value, r.sb)
// ConditionFor returns a boolean predicate for key (materialized column when
// present, else map access), args bound into sb.
func (r *predicateResolver) ConditionFor(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, value any) (string, error) {
conds, _, err := querybuilder.Conditions(ctx, q, r.storage, key, op, value, r.keys, false, r.sb)
if err != nil {
return "", err
}
@@ -42,6 +42,6 @@ func (r *predicateResolver) ConditionFor(ctx context.Context, orgID valuer.UUID,
}
// ExistsFor returns the EXISTS predicate for key.
func (r *predicateResolver) ExistsFor(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return r.ConditionFor(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorExists, nil)
func (r *predicateResolver) ExistsFor(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return r.ConditionFor(ctx, q, key, qbtypes.FilterOperatorExists, nil)
}

View File

@@ -31,8 +31,7 @@ var (
type scopedTraceStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
storage qbtypes.Storage
scope TraceScope
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation]
resourceFilterStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation]
@@ -58,9 +57,7 @@ func NewFactory(
if err != nil {
return nil, err
}
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
return NewScopedTraceStatementBuilder(settings, metadataStore, fm, cb, scope, traceStmtBuilder, fl), nil
return NewScopedTraceStatementBuilder(settings, metadataStore, tracestelemetryschema.NewStorage(), scope, traceStmtBuilder, fl), nil
},
)
}
@@ -70,8 +67,7 @@ func NewFactory(
func NewScopedTraceStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
storage qbtypes.Storage,
scope TraceScope,
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation],
fl flagger.Flagger,
@@ -92,8 +88,7 @@ func NewScopedTraceStatementBuilder(
return &scopedTraceStatementBuilder{
logger: scopedSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
cb: conditionBuilder,
storage: storage,
scope: scope,
traceStmtBuilder: traceStmtBuilder,
resourceFilterStmtBuilder: resourceFilterStmtBuilder,
@@ -229,12 +224,13 @@ func (b *scopedTraceStatementBuilder) buildTraceListQuery(
return nil, err
}
matchedSB := sqlbuilder.NewSelectBuilder()
maskExpr, resolved, err := b.resolveFor(ctx, orgID, start, end, keys, matchedSB)
q := b.queryInfo(ctx, orgID, start, end)
maskExpr, resolved, err := b.resolveFor(ctx, q, keys, matchedSB)
if err != nil {
return nil, err
}
enrichSB := sqlbuilder.NewSelectBuilder()
_, enrichResolved, err := b.resolveFor(ctx, orgID, start, end, keys, enrichSB)
_, enrichResolved, err := b.resolveFor(ctx, q, keys, enrichSB)
if err != nil {
return nil, err
}
@@ -350,15 +346,15 @@ func (b *scopedTraceStatementBuilder) resolverFieldKeys() []*telemetrytypes.Tele
// resolveFor renders the gate mask and every scope column with condition args bound
// into sb.
func (b *scopedTraceStatementBuilder) resolveFor(ctx context.Context, orgID valuer.UUID, start, end uint64, keys map[string][]*telemetrytypes.TelemetryFieldKey, sb *sqlbuilder.SelectBuilder) (string, []resolvedColumn, error) {
cols := newColumnResolver(b.fm, keys)
preds := newPredicateResolver(b.cb, keys, sb)
maskExpr, err := b.resolveMask(ctx, orgID, start, end, preds)
func (b *scopedTraceStatementBuilder) resolveFor(ctx context.Context, q qbtypes.QueryInfo, keys map[string][]*telemetrytypes.TelemetryFieldKey, sb *sqlbuilder.SelectBuilder) (string, []resolvedColumn, error) {
cols := newColumnResolver(b.storage, keys)
preds := newPredicateResolver(b.storage, keys, sb)
maskExpr, err := b.resolveMask(ctx, q, preds)
if err != nil {
return "", nil, err
}
preds.maskExpr = maskExpr
resolved, err := b.resolveColumns(ctx, orgID, start, end, cols, preds)
resolved, err := b.resolveColumns(ctx, q, cols, preds)
if err != nil {
return "", nil, err
}
@@ -366,11 +362,11 @@ func (b *scopedTraceStatementBuilder) resolveFor(ctx context.Context, orgID valu
}
// resolveMask builds the per-span in-scope mask: OR of the gate keys' EXISTS predicates.
func (b *scopedTraceStatementBuilder) resolveMask(ctx context.Context, orgID valuer.UUID, start, end uint64, preds *predicateResolver) (string, error) {
func (b *scopedTraceStatementBuilder) resolveMask(ctx context.Context, q qbtypes.QueryInfo, preds *predicateResolver) (string, error) {
fieldKeys := b.scope.FieldKeys
parts := make([]string, 0, len(fieldKeys))
for _, key := range fieldKeys {
e, err := preds.ExistsFor(ctx, orgID, start, end, key)
e, err := preds.ExistsFor(ctx, q, key)
if err != nil {
return "", err
}
@@ -385,10 +381,10 @@ type resolvedColumn struct {
orderable bool
}
func (b *scopedTraceStatementBuilder) resolveColumns(ctx context.Context, orgID valuer.UUID, start, end uint64, cols *columnResolver, preds *predicateResolver) ([]resolvedColumn, error) {
func (b *scopedTraceStatementBuilder) resolveColumns(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) ([]resolvedColumn, error) {
out := make([]resolvedColumn, 0, len(b.scope.Columns))
for _, c := range b.scope.Columns {
expr, err := c.Expr.render(ctx, orgID, start, end, cols, preds)
expr, err := c.Expr.render(ctx, q, cols, preds)
if err != nil {
return nil, err
}
@@ -457,7 +453,7 @@ func (b *scopedTraceStatementBuilder) splitFilter(ctx context.Context, orgID val
}
havingExpr = traceExpr
if strings.TrimSpace(spanExpr) != "" {
pred, warnings, url, err := b.resolveSpanPredicate(ctx, orgID, start, end, spanExpr, keys, variables, sb)
pred, warnings, url, err := b.resolveSpanPredicate(ctx, b.queryInfo(ctx, orgID, start, end), spanExpr, keys, variables, sb)
if err != nil {
return fp, err
}
@@ -486,21 +482,17 @@ func (b *scopedTraceStatementBuilder) splitFilter(ctx context.Context, orgID val
// resolveSpanPredicate resolves a span-level filter expression to a bare boolean
// predicate, args bound into sb; keys must cover the expression's selectors.
func (b *scopedTraceStatementBuilder) resolveSpanPredicate(ctx context.Context, orgID valuer.UUID, start, end uint64, expr string, keys map[string][]*telemetrytypes.TelemetryFieldKey, variables map[string]qbtypes.VariableItem, sb *sqlbuilder.SelectBuilder) (string, []string, string, error) {
func (b *scopedTraceStatementBuilder) resolveSpanPredicate(ctx context.Context, q qbtypes.QueryInfo, expr string, keys map[string][]*telemetrytypes.TelemetryFieldKey, variables map[string]qbtypes.VariableItem, sb *sqlbuilder.SelectBuilder) (string, []string, string, error) {
prepared, err := querybuilder.PrepareWhereClause(expr, querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: orgID,
Flagger: b.fl,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
Builder: sb,
Context: ctx,
Query: q,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
Builder: sb,
// resource conditions are handled by __resource_filter
SkipResourceFilter: true,
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
return "", nil, "", err
@@ -685,3 +677,9 @@ func quoteAlias(alias string) string {
}
return alias
}
// queryInfo binds one request's context; the query-path flags are evaluated
// one time here.
func (b *scopedTraceStatementBuilder) queryInfo(ctx context.Context, orgID valuer.UUID, start, end uint64) qbtypes.QueryInfo {
return querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end)
}

View File

@@ -340,7 +340,7 @@ func (b *scopedTraceStatementBuilder) buildQualifiedStatement(
return nil, nil, err
}
sb := sqlbuilder.NewSelectBuilder()
maskExpr, resolved, err := b.resolveFor(ctx, orgID, start, end, keys, sb)
maskExpr, resolved, err := b.resolveFor(ctx, b.queryInfo(ctx, orgID, start, end), keys, sb)
if err != nil {
return nil, nil, err
}
@@ -475,15 +475,15 @@ func groupBySelectors(groupBy []qbtypes.GroupByKey) []*telemetrytypes.FieldKeySe
return selectors
}
// resolveGroupColumns resolves group-by keys through the field mapper for selection
// resolveGroupColumns resolves group-by keys through the storage for selection
// inside the per-trace scan; keys must cover the group-by selectors.
func (b *scopedTraceStatementBuilder) resolveGroupColumns(ctx context.Context, orgID valuer.UUID, start, end uint64, groupBy []qbtypes.GroupByKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) ([]groupColumn, error) {
func (b *scopedTraceStatementBuilder) resolveGroupColumns(ctx context.Context, q qbtypes.QueryInfo, groupBy []qbtypes.GroupByKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) ([]groupColumn, error) {
if len(groupBy) == 0 {
return nil, nil
}
out := make([]groupColumn, 0, len(groupBy))
for i := range groupBy {
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &groupBy[i].TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := querybuilder.ResolveColumn(ctx, q, b.storage, &groupBy[i].TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -518,12 +518,13 @@ func (b *scopedTraceStatementBuilder) newScanContext(
) (*scanContext, error) {
sc := &scanContext{sb: sqlbuilder.NewSelectBuilder()}
var err error
sc.maskExpr, sc.resolved, err = b.resolveFor(ctx, orgID, start, end, keys, sc.sb)
q := b.queryInfo(ctx, orgID, start, end)
sc.maskExpr, sc.resolved, err = b.resolveFor(ctx, q, keys, sc.sb)
if err != nil {
return nil, err
}
if strings.TrimSpace(spanExpr) != "" {
pred, warns, url, err := b.resolveSpanPredicate(ctx, orgID, start, end, spanExpr, keys, variables, sc.sb)
pred, warns, url, err := b.resolveSpanPredicate(ctx, q, spanExpr, keys, variables, sc.sb)
if err != nil {
return nil, err
}
@@ -596,7 +597,7 @@ func (b *scopedTraceStatementBuilder) buildTraceAggregationQuery(
}
}
groupCols, err := b.resolveGroupColumns(ctx, orgID, start, end, query.GroupBy, keys)
groupCols, err := b.resolveGroupColumns(ctx, b.queryInfo(ctx, orgID, start, end), query.GroupBy, keys)
if err != nil {
return nil, err
}

View File

@@ -8,7 +8,6 @@ import (
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
qbvariables "github.com/SigNoz/signoz/pkg/variables"
"github.com/huandu/go-sqlbuilder"
)
@@ -53,14 +52,15 @@ func (b *scopedTraceStatementBuilder) resolveTraceHaving(ctx context.Context, ex
fieldKeys[alias] = []*telemetrytypes.TelemetryFieldKey{key}
}
cb := &aliasConditionBuilder{allowed: allowed, used: make(map[string]struct{})}
storage := &aliasStorage{allowed: allowed, used: make(map[string]struct{})}
prepared, err := querybuilder.PrepareWhereClause(expr, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Logger: b.logger,
ConditionBuilder: cb,
FieldKeys: fieldKeys,
Variables: variables,
Builder: sb,
Context: ctx,
Query: qbtypes.QueryInfo{Signal: telemetrytypes.SignalTraces},
Storage: storage,
Logger: b.logger,
FieldKeys: fieldKeys,
Variables: variables,
Builder: sb,
})
if err != nil {
return nil, err
@@ -68,37 +68,42 @@ func (b *scopedTraceStatementBuilder) resolveTraceHaving(ctx context.Context, ex
if prepared.IsEmpty() {
return nil, nil //nolint:nilnil
}
return &traceHaving{pred: prepared.Expr, used: cb.used}, nil
return &traceHaving{pred: prepared.Expr, used: storage.used}, nil
}
// aliasConditionBuilder renders filter conditions directly against the per-trace
// aliases, recording the ones it touches; a key resolving to no alias is an error.
type aliasConditionBuilder struct {
// aliasStorage renders filter conditions directly against the per-trace
// aliases, recording the ones it touches. Every alias is a computed column,
// present in every row; a key resolving to no alias is an error.
type aliasStorage struct {
allowed map[string]struct{}
used map[string]struct{}
}
var _ qbtypes.ConditionBuilder = (*aliasConditionBuilder)(nil)
var _ qbtypes.Storage = (*aliasStorage)(nil)
func (c *aliasConditionBuilder) ConditionFor(
_ context.Context,
_ valuer.UUID,
_, _ uint64,
key *telemetrytypes.TelemetryFieldKey,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
op qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
matching := keys[key.Name]
if len(matching) == 0 {
return nil, nil, errors.NewInvalidInputf(errors.CodeInvalidInput,
"aggregate %q cannot be used in a trace-level filter; filterable aggregates: %s",
key.Name, strings.Join(sortedAliases(c.allowed), ", "))
}
alias := matching[0].Name
c.used[alias] = struct{}{}
func (s *aliasStorage) Read(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
s.used[key.Name] = struct{}{}
return quoteAlias(key.Name), nil
}
func (s *aliasStorage) Exists(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, bool) (qbtypes.Existence, error) {
return qbtypes.Existence{Predicate: "true", WhenAbsent: qbtypes.AlwaysPresent}, nil
}
func (s *aliasStorage) Fallback(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, _ qbtypes.FilterOperator, _ any) ([]*telemetrytypes.LogicalField, error) {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput,
"aggregate %q cannot be used in a trace-level filter; filterable aggregates: %s",
key.Name, strings.Join(sortedAliases(s.allowed), ", "))
}
func (s *aliasStorage) Traits() qbtypes.Traits {
return qbtypes.Traits{}
}
// Compile supports the comparison operators only: an aggregate is a number.
func (s *aliasStorage) Compile(_ context.Context, _ qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, op qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
alias := logical.Single().Name
s.used[alias] = struct{}{}
col := quoteAlias(alias)
var cond string
@@ -128,7 +133,7 @@ func (c *aliasConditionBuilder) ConditionFor(
case qbtypes.FilterOperatorBetween, qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok || len(values) != 2 {
return nil, nil, errors.NewInvalidInputf(errors.CodeInvalidInput,
return qbtypes.Compiled{}, errors.NewInvalidInputf(errors.CodeInvalidInput,
"between on trace-level aggregate %q requires exactly two values", alias)
}
if op == qbtypes.FilterOperatorBetween {
@@ -137,8 +142,12 @@ func (c *aliasConditionBuilder) ConditionFor(
cond = sb.NotBetween(col, values[0], values[1])
}
default:
return nil, nil, errors.NewInvalidInputf(errors.CodeInvalidInput,
return qbtypes.Compiled{}, errors.NewInvalidInputf(errors.CodeInvalidInput,
"trace-level aggregate %q supports only comparison operators (=, !=, <, <=, >, >=, in, between)", alias)
}
return []string{cond}, nil, nil
return qbtypes.Compiled{Condition: cond}, nil
}
func (s *aliasStorage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
return querybuilder.DefaultRead(ctx, q, s, logical)
}

View File

@@ -27,8 +27,7 @@ var (
type traceQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
storage qbtypes.Storage
resourceFilterResolver *resourcefilter.ResourceFingerprintResolver[qbtypes.TraceAggregation]
aggExprRewriter qbtypes.AggExprRewriter
fl flagger.Flagger
@@ -54,11 +53,10 @@ func NewFactory(
return factory.NewProviderFactory(
factory.MustNewName("traces"),
func(_ context.Context, settings factory.ProviderSettings, cfg statementbuilder.Config) (qbtypes.StatementBuilder[qbtypes.TraceAggregation], error) {
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, fm, cb, fl)
storage := tracestelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, storage, fl, telemetrytypes.SignalTraces)
return NewTraceQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, telemetryStore, fl,
settings, metadataStore, storage, aggExprRewriter, telemetryStore, fl,
cfg.SkipResourceFingerprint.Enabled, cfg.SkipResourceFingerprint.Threshold,
), nil
},
@@ -68,8 +66,7 @@ func NewFactory(
func NewTraceQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
storage qbtypes.Storage,
aggExprRewriter qbtypes.AggExprRewriter,
telemetryStore telemetrystore.TelemetryStore,
flagger flagger.Flagger,
@@ -94,8 +91,7 @@ func NewTraceQueryStatementBuilder(
return &traceQueryStatementBuilder{
logger: tracesSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
cb: conditionBuilder,
storage: storage,
resourceFilterResolver: resourceFilterResolver,
aggExprRewriter: aggExprRewriter,
fl: flagger,
@@ -382,8 +378,9 @@ func (b *traceQueryStatementBuilder) buildListQuery(
cteArgs = append(cteArgs, scopeArgs...)
}
info := b.queryInfo(ctx, orgID, start, end)
for i, field := range query.SelectFields {
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &field, telemetrytypes.FieldDataTypeUnspecified, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &field, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
@@ -407,7 +404,7 @@ func (b *traceQueryStatementBuilder) buildListQuery(
// Add order by
for _, orderBy := range query.Order {
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
@@ -595,8 +592,9 @@ func (b *traceQueryStatementBuilder) buildTimeSeriesQuery(
// Keep original column expressions so we can build the tuple
fieldNames := make([]string, 0, len(query.GroupBy))
info := b.queryInfo(ctx, orgID, start, end)
for i, gb := range query.GroupBy {
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -758,8 +756,9 @@ func (b *traceQueryStatementBuilder) buildScalarQuery(
allAggChArgs := []any{}
fieldNames := make([]string, 0, len(query.GroupBy))
info := b.queryInfo(ctx, orgID, start, end)
for i, gb := range query.GroupBy {
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -870,16 +869,12 @@ func (b *traceQueryStatementBuilder) addFilterCondition(
// add filter expression
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: orgID,
Flagger: b.fl,
Query: b.queryInfo(ctx, orgID, start, end),
Storage: b.storage,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
SkipResourceFilter: skipResourceFilter,
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
@@ -995,3 +990,9 @@ func (b *traceQueryStatementBuilder) expandRawSelectFields(query qbtypes.QueryBu
query.SelectFields = selectFields
return query
}
// queryInfo binds one request's context; the query-path flags are evaluated
// one time here.
func (b *traceQueryStatementBuilder) queryInfo(ctx context.Context, orgID valuer.UUID, start, end uint64) qbtypes.QueryInfo {
return querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end)
}

View File

@@ -71,7 +71,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -100,7 +100,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (attributes_string['http.request.method'] = ? AND mapContains(attributes_string, 'http.request.method'))) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (attributes_string['http.request.method'] = ? AND mapContains(attributes_string, 'http.request.method'))) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR (attributes_string['http.request.method'] = ? AND mapContains(attributes_string, 'http.request.method'))) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR (attributes_string['http.request.method'] = ? AND mapContains(attributes_string, 'http.request.method'))) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"redis-manual", "GET", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "redis-manual", "GET", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -129,7 +129,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"redis-manual", "redis-manual", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "redis-manual", "redis-manual", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -160,7 +160,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(attribute_string_http$$route <> '', attribute_string_http$$route, NULL)) AS `__GROUP_BY_KEY_0_httpRoute`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_httpRoute` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(attribute_string_http$$route <> '', attribute_string_http$$route, NULL)) AS `__GROUP_BY_KEY_0_httpRoute`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_httpRoute`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_httpRoute` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_httpRoute`",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(attribute_string_http$$route) AS `__GROUP_BY_KEY_0_httpRoute`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_httpRoute` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(attribute_string_http$$route) AS `__GROUP_BY_KEY_0_httpRoute`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_httpRoute`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_httpRoute` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_httpRoute`",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -198,7 +198,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(attribute_string_http$$route <> '', attribute_string_http$$route, NULL)) AS `__GROUP_BY_KEY_0_httpRoute`, toString(multiIf(http_method <> '', http_method, NULL)) AS `__GROUP_BY_KEY_1_httpMethod`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((resource_string_service$$name = ? AND resource_string_service$$name <> '') AND http_method <> '' AND kind_string = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(attribute_string_http$$route <> '', attribute_string_http$$route, NULL)) AS `__GROUP_BY_KEY_0_httpRoute`, toString(multiIf(http_method <> '', http_method, NULL)) AS `__GROUP_BY_KEY_1_httpMethod`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((resource_string_service$$name = ? AND resource_string_service$$name <> '') AND http_method <> '' AND kind_string = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod`",
Query: "WITH __limit_cte AS (SELECT toString(attribute_string_http$$route) AS `__GROUP_BY_KEY_0_httpRoute`, toString(http_method) AS `__GROUP_BY_KEY_1_httpMethod`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (resource_string_service$$name = ? AND http_method <> '' AND kind_string = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(attribute_string_http$$route) AS `__GROUP_BY_KEY_0_httpRoute`, toString(http_method) AS `__GROUP_BY_KEY_1_httpMethod`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (resource_string_service$$name = ? AND http_method <> '' AND kind_string = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod`",
Args: []any{"redis-manual", "Server", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "redis-manual", "Server", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -239,7 +239,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(mapContains(attributes_number, 'metric.max_count'), toFloat64(attributes_number['metric.max_count']), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 desc LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(mapContains(attributes_number, 'metric.max_count'), toFloat64(attributes_number['metric.max_count']), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name` ORDER BY ts desc",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(mapContains(attributes_number, 'metric.max_count'), toFloat64(attributes_number['metric.max_count']), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 desc LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(mapContains(attributes_number, 'metric.max_count'), toFloat64(attributes_number['metric.max_count']), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name` ORDER BY ts desc",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -268,7 +268,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -307,7 +307,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY `__GROUP_BY_KEY_0_service.name` desc LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name` ORDER BY `__GROUP_BY_KEY_0_service.name` desc, ts desc",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY `__GROUP_BY_KEY_0_service.name` desc LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name` ORDER BY `__GROUP_BY_KEY_0_service.name` desc, ts desc",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -338,7 +338,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(response_status_code <> '', response_status_code, NULL)) AS `__GROUP_BY_KEY_0_responseStatusCode`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_responseStatusCode` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(response_status_code <> '', response_status_code, NULL)) AS `__GROUP_BY_KEY_0_responseStatusCode`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_responseStatusCode`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_responseStatusCode` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_responseStatusCode`",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(response_status_code) AS `__GROUP_BY_KEY_0_responseStatusCode`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_responseStatusCode` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(response_status_code) AS `__GROUP_BY_KEY_0_responseStatusCode`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_responseStatusCode`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_responseStatusCode` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_responseStatusCode`",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -369,7 +369,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(response_status_code <> '', response_status_code, NULL)) AS `__GROUP_BY_KEY_0_responseStatusCode`, quantile(0.90)(multiIf(duration_nano <> 0, accurateCastOrNull(duration_nano, 'Float64'), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_responseStatusCode` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(response_status_code <> '', response_status_code, NULL)) AS `__GROUP_BY_KEY_0_responseStatusCode`, quantile(0.90)(multiIf(duration_nano <> 0, accurateCastOrNull(duration_nano, 'Float64'), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_responseStatusCode`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_responseStatusCode` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_responseStatusCode`",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(response_status_code) AS `__GROUP_BY_KEY_0_responseStatusCode`, quantile(0.90)(toFloat64(duration_nano)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_responseStatusCode` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(response_status_code) AS `__GROUP_BY_KEY_0_responseStatusCode`, quantile(0.90)(toFloat64(duration_nano)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_responseStatusCode`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_responseStatusCode` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_responseStatusCode`",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -458,24 +458,22 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"1.0.0", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1.0.0", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
},
}
fl := flaggertest.New(t)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
nil,
fl,
@@ -671,7 +669,7 @@ func TestStatementBuilderListQuery(t *testing.T) {
Limit: 10,
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint) SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, name AS `__SELECT_KEY_3_name`, resource_string_service$$name AS `__SELECT_KEY_4_serviceName`, duration_nano AS `__SELECT_KEY_5_durationNano`, http_method AS `__SELECT_KEY_6_httpMethod`, multiIf(`attribute_string_mixed$$materialization$$key_exists`, toString(`attribute_string_mixed$$materialization$$key`), multiIf(resource.`mixed.materialization.key` IS NOT NULL, resource.`mixed.materialization.key`::String, mapContains(resources_string, 'mixed.materialization.key'), resources_string['mixed.materialization.key'], NULL) IS NOT NULL, toString(multiIf(resource.`mixed.materialization.key` IS NOT NULL, resource.`mixed.materialization.key`::String, mapContains(resources_string, 'mixed.materialization.key'), resources_string['mixed.materialization.key'], NULL)), NULL) AS `__SELECT_KEY_7_mixed.materialization.key` FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint) SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, name AS `__SELECT_KEY_3_name`, resource_string_service$$name AS `__SELECT_KEY_4_serviceName`, duration_nano AS `__SELECT_KEY_5_durationNano`, http_method AS `__SELECT_KEY_6_httpMethod`, multiIf(`attribute_string_mixed$$materialization$$key_exists`, `attribute_string_mixed$$materialization$$key`, multiIf(resource.`mixed.materialization.key` IS NOT NULL, resource.`mixed.materialization.key`::String, mapContains(resources_string, 'mixed.materialization.key'), resources_string['mixed.materialization.key'], NULL) IS NOT NULL, multiIf(resource.`mixed.materialization.key` IS NOT NULL, resource.`mixed.materialization.key`::String, mapContains(resources_string, 'mixed.materialization.key'), resources_string['mixed.materialization.key'], NULL), NULL) AS `__SELECT_KEY_7_mixed.materialization.key` FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
expectedErr: nil,
@@ -758,7 +756,7 @@ func TestStatementBuilderListQuery(t *testing.T) {
Limit: 10,
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint) SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, name AS `__SELECT_KEY_3_name`, resource_string_service$$name AS `__SELECT_KEY_4_serviceName`, duration_nano AS `__SELECT_KEY_5_durationNano`, http_method AS `__SELECT_KEY_6_httpMethod`, multiIf(mapContains(attributes_string, 'non-existent.key'), toString(attributes_string['non-existent.key']), mapContains(attributes_number, 'non-existent.key'), toString(attributes_number['non-existent.key']), mapContains(attributes_bool, 'non-existent.key'), toString(attributes_bool['non-existent.key']), NULL) AS `__SELECT_KEY_7_non-existent.key` FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint) SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, name AS `__SELECT_KEY_3_name`, resource_string_service$$name AS `__SELECT_KEY_4_serviceName`, duration_nano AS `__SELECT_KEY_5_durationNano`, http_method AS `__SELECT_KEY_6_httpMethod`, multiIf(mapContains(attributes_string, 'non-existent.key'), attributes_string['non-existent.key'], mapContains(attributes_number, 'non-existent.key'), toString(attributes_number['non-existent.key']), mapContains(attributes_bool, 'non-existent.key'), toString(attributes_bool['non-existent.key']), NULL) AS `__SELECT_KEY_7_non-existent.key` FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
expectedErr: nil,
@@ -766,17 +764,15 @@ func TestStatementBuilderListQuery(t *testing.T) {
}
fl := flaggertest.New(t)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
nil,
fl,
@@ -930,7 +926,7 @@ func TestStatementBuilderListQueryWithCorruptData(t *testing.T) {
Limit: 10,
},
expected: qbtypes.Statement{
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.version::String <> '', scope.version::String, NULL) AS `__SELECT_KEY_3_scope.version` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.version::String <> '', scope.version::String, scope.attributes.`scope.scope.version` IS NOT NULL, scope.attributes.`scope.scope.version`::String, NULL) AS `__SELECT_KEY_3_scope.version` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
},
@@ -1021,7 +1017,7 @@ func TestStatementBuilderListQueryWithCorruptData(t *testing.T) {
Limit: 10,
},
expected: qbtypes.Statement{
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.name::String <> '', scope.name::String, NULL) AS `__SELECT_KEY_3_scope.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.name::String <> '', scope.name::String, scope.attributes.`scope.scope.name` IS NOT NULL, scope.attributes.`scope.scope.name`::String, NULL) AS `__SELECT_KEY_3_scope.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
},
@@ -1030,20 +1026,18 @@ func TestStatementBuilderListQueryWithCorruptData(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.New(t)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = c.keysMap
if mockMetadataStore.KeysMap == nil {
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
}
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
nil,
fl,
@@ -1080,7 +1074,7 @@ func TestStatementBuilderGroupByResourceEvolution(t *testing.T) {
startMs: 1747947419000, // 2025-05-22 21:56:59 UTC, ~3m before release
endMs: 1747983448000, // 2025-05-23 07:57:28 UTC, ~10h after release
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
},
@@ -1105,17 +1099,15 @@ func TestStatementBuilderGroupByResourceEvolution(t *testing.T) {
}
fl := flaggertest.New(t)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
nil,
fl,
@@ -1202,7 +1194,7 @@ func TestStatementBuilderTraceQuery(t *testing.T) {
Limit: 10,
},
expected: qbtypes.Statement{
Query: "WITH __toe AS (SELECT trace_id FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __toe_duration_sorted AS (SELECT trace_id, duration_nano, resource_string_service$$name as `service.name`, name FROM signoz_traces.distributed_signoz_index_v3 WHERE parent_span_id = '' AND trace_id GLOBAL IN __toe AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? ORDER BY duration_nano DESC LIMIT 1 BY trace_id) SELECT __toe_duration_sorted.`service.name` AS `service.name`, __toe_duration_sorted.name AS `name`, count() AS span_count, __toe_duration_sorted.duration_nano AS `duration_nano`, __toe_duration_sorted.trace_id AS `trace_id` FROM __toe INNER JOIN __toe_duration_sorted ON __toe.trace_id = __toe_duration_sorted.trace_id GROUP BY trace_id, duration_nano, name, `service.name` ORDER BY duration_nano DESC LIMIT 1 BY trace_id LIMIT ? SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Query: "WITH __toe AS (SELECT trace_id FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __toe_duration_sorted AS (SELECT trace_id, duration_nano, resource_string_service$$name as `service.name`, name FROM signoz_traces.distributed_signoz_index_v3 WHERE parent_span_id = '' AND trace_id GLOBAL IN __toe AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? ORDER BY duration_nano DESC LIMIT 1 BY trace_id) SELECT __toe_duration_sorted.`service.name` AS `service.name`, __toe_duration_sorted.name AS `name`, count() AS span_count, __toe_duration_sorted.duration_nano AS `duration_nano`, __toe_duration_sorted.trace_id AS `trace_id` FROM __toe INNER JOIN __toe_duration_sorted ON __toe.trace_id = __toe_duration_sorted.trace_id GROUP BY trace_id, duration_nano, name, `service.name` ORDER BY duration_nano DESC LIMIT 1 BY trace_id LIMIT ? SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Args: []any{"redis-manual", "redis-manual", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
expectedErr: nil,
@@ -1272,17 +1264,15 @@ func TestStatementBuilderTraceQuery(t *testing.T) {
}
fl := flaggertest.New(t)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
nil,
fl,
@@ -1910,21 +1900,17 @@ func newSkipResourceFingerprintBuilder(
t.Helper()
fl := flaggertest.New(t)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(
instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl,
)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
return NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
telemetryStore,
fl,
@@ -1940,17 +1926,15 @@ func TestStatementBuilderGroupByUnseenKey(t *testing.T) {
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
nil,
fl,
@@ -1982,17 +1966,15 @@ func TestStatementBuilderAggregationUnseenKey(t *testing.T) {
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
nil,
fl,
@@ -2057,17 +2039,15 @@ func TestStatementBuilderSemconvFamilies(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureResolveSemconvFamilies.String(): c.flag,
})
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
nil,
fl,

View File

@@ -264,15 +264,11 @@ func (b *traceOperatorCTEBuilder) buildQueryCTE(ctx context.Context, queryName s
query.Filter.Expression,
querybuilder.FilterExprVisitorOpts{
Context: ctx,
OrgID: b.orgID,
Flagger: b.stmtBuilder.fl,
Query: b.queryInfo(ctx),
Storage: b.stmtBuilder.storage,
Logger: b.stmtBuilder.logger,
FieldMapper: b.stmtBuilder.fm,
ConditionBuilder: b.stmtBuilder.cb,
FieldKeys: keys,
SkipResourceFilter: true,
StartNs: b.start,
EndNs: b.end,
},
)
if err != nil {
@@ -488,11 +484,12 @@ func (b *traceOperatorCTEBuilder) buildListQuery(ctx context.Context, selectFrom
}
// Add selectFields since we now have all base table columns
info := b.queryInfo(ctx)
for i, field := range b.operator.SelectFields {
if selectedFields[field.Name] {
continue
}
expr, err := b.stmtBuilder.fm.ColumnExpressionFor(ctx, b.orgID, b.start, b.end, &field, telemetrytypes.FieldDataTypeUnspecified, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.stmtBuilder.storage, &field, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
b.stmtBuilder.logger.WarnContext(ctx, "failed to map select field",
slog.String("field", field.Name), errors.Attr(err))
@@ -513,7 +510,7 @@ func (b *traceOperatorCTEBuilder) buildListQuery(ctx context.Context, selectFrom
// Add order by support
orderApplied := false
for _, orderBy := range b.operator.Order {
expr, err := b.stmtBuilder.fm.ColumnExpressionFor(ctx, b.orgID, b.start, b.end, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.stmtBuilder.storage, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
@@ -631,8 +628,9 @@ func (b *traceOperatorCTEBuilder) buildTimeSeriesQuery(ctx context.Context, sele
int64(b.operator.StepInterval.Seconds()),
))
info := b.queryInfo(ctx)
for _, gb := range b.operator.GroupBy {
expr, err := b.stmtBuilder.fm.ColumnExpressionFor(ctx, b.orgID, b.start, b.end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.stmtBuilder.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, errors.NewInvalidInputf(
errors.CodeInvalidInput,
@@ -727,8 +725,9 @@ func (b *traceOperatorCTEBuilder) buildTraceQuery(ctx context.Context, selectFro
sb := sqlbuilder.NewSelectBuilder()
info := b.queryInfo(ctx)
for _, gb := range b.operator.GroupBy {
expr, err := b.stmtBuilder.fm.ColumnExpressionFor(ctx, b.orgID, b.start, b.end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.stmtBuilder.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, errors.NewInvalidInputf(
errors.CodeInvalidInput,
@@ -853,8 +852,9 @@ func (b *traceOperatorCTEBuilder) buildTraceQuery(ctx context.Context, selectFro
func (b *traceOperatorCTEBuilder) buildScalarQuery(ctx context.Context, selectFromCTE string, keys map[string][]*telemetrytypes.TelemetryFieldKey) (*qbtypes.Statement, error) {
sb := sqlbuilder.NewSelectBuilder()
info := b.queryInfo(ctx)
for _, gb := range b.operator.GroupBy {
expr, err := b.stmtBuilder.fm.ColumnExpressionFor(ctx, b.orgID, b.start, b.end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := querybuilder.ResolveColumn(ctx, info, b.stmtBuilder.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, errors.NewInvalidInputf(
errors.CodeInvalidInput,
@@ -963,3 +963,9 @@ func (b *traceOperatorCTEBuilder) aggOrderBy(k qbtypes.OrderBy) (int, bool) {
}
return 0, false
}
// queryInfo binds the operator's request context; the query-path flags are
// evaluated one time here.
func (b *traceOperatorCTEBuilder) queryInfo(ctx context.Context) qbtypes.QueryInfo {
return querybuilder.NewQueryInfo(ctx, b.orgID, b.stmtBuilder.fl, telemetrytypes.SignalTraces, nil, b.start, b.end)
}

View File

@@ -20,18 +20,17 @@ func newTestTraceOperatorStatementBuilder(t *testing.T) *traceOperatorStatementB
t.Helper()
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
traceStmtBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore, fm, cb, aggExprRewriter, nil, fl, false, 100000,
mockMetadataStore, storage, aggExprRewriter, nil, fl, false, 100000,
)
return NewTraceOperatorStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore, fm, cb, traceStmtBuilder, aggExprRewriter, fl,
mockMetadataStore, storage, traceStmtBuilder, aggExprRewriter, fl,
)
}
@@ -285,7 +284,7 @@ func TestTraceOperatorStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH toDateTime64(1747947419000000000, 9) AS t_from, toDateTime64(1747983448000000000, 9) AS t_to, 1747945619 AS bucket_from, 1747983448 AS bucket_to, all_spans AS (SELECT *, resource_string_service$$name AS `service.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_A AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), A AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_A) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_B AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), B AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_B) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), A_DIR_DESC_B AS (SELECT p.* FROM A AS p INNER JOIN B AS c ON p.trace_id = c.trace_id AND p.span_id = c.parent_span_id) SELECT toStartOfInterval(timestamp, INTERVAL 60 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `service.name`, count() AS __result_0 FROM A_DIR_DESC_B GROUP BY ts, `service.name` ORDER BY ts desc SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Query: "WITH toDateTime64(1747947419000000000, 9) AS t_from, toDateTime64(1747983448000000000, 9) AS t_to, 1747945619 AS bucket_from, 1747983448 AS bucket_to, all_spans AS (SELECT *, resource_string_service$$name AS `service.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_A AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), A AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_A) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_B AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), B AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_B) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), A_DIR_DESC_B AS (SELECT p.* FROM A AS p INNER JOIN B AS c ON p.trace_id = c.trace_id AND p.span_id = c.parent_span_id) SELECT toStartOfInterval(timestamp, INTERVAL 60 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `service.name`, count() AS __result_0 FROM A_DIR_DESC_B GROUP BY ts, `service.name` ORDER BY ts desc SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), "frontend", "%service.name%", "%service.name\":\"frontend%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), "backend", "%service.name%", "%service.name\":\"backend%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -344,7 +343,7 @@ func TestTraceOperatorStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH toDateTime64(1747947419000000000, 9) AS t_from, toDateTime64(1747983448000000000, 9) AS t_to, 1747945619 AS bucket_from, 1747983448 AS bucket_to, all_spans AS (SELECT *, resource_string_service$$name AS `service.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_A AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), A AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_A) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), B AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND toFloat64(response_status_code) < ?), A_AND_B AS (SELECT l.* FROM A AS l INNER JOIN B AS r ON l.trace_id = r.trace_id) SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `service.name`, avg(multiIf(duration_nano <> 0, accurateCastOrNull(duration_nano, 'Float64'), mapContains(attributes_number, 'duration_nano'), toFloat64(attributes_number['duration_nano']), NULL)) AS __result_0 FROM A_AND_B GROUP BY `service.name` ORDER BY __result_0 desc SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Query: "WITH toDateTime64(1747947419000000000, 9) AS t_from, toDateTime64(1747983448000000000, 9) AS t_to, 1747945619 AS bucket_from, 1747983448 AS bucket_to, all_spans AS (SELECT *, resource_string_service$$name AS `service.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_A AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), A AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_A) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), B AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND toFloat64(response_status_code) < ?), A_AND_B AS (SELECT l.* FROM A AS l INNER JOIN B AS r ON l.trace_id = r.trace_id) SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `service.name`, avg(toFloat64(duration_nano)) AS __result_0 FROM A_AND_B GROUP BY `service.name` ORDER BY __result_0 desc SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), "frontend", "%service.name%", "%service.name\":\"frontend%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), float64(400)},
},
expectedErr: nil,

View File

@@ -20,8 +20,7 @@ import (
type traceOperatorStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
storage qbtypes.Storage
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation]
resourceFilterStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation]
aggExprRewriter qbtypes.AggExprRewriter
@@ -42,15 +41,14 @@ func NewOperatorFactory(
return factory.NewProviderFactory(
factory.MustNewName("traceoperator"),
func(_ context.Context, settings factory.ProviderSettings, cfg statementbuilder.Config) (qbtypes.TraceOperatorStatementBuilder, error) {
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, fm, cb, fl)
storage := tracestelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, storage, fl, telemetrytypes.SignalTraces)
traceStmtBuilder := NewTraceQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, telemetryStore, fl,
settings, metadataStore, storage, aggExprRewriter, telemetryStore, fl,
cfg.SkipResourceFingerprint.Enabled, cfg.SkipResourceFingerprint.Threshold,
)
return NewTraceOperatorStatementBuilder(
settings, metadataStore, fm, cb, traceStmtBuilder, aggExprRewriter, fl,
settings, metadataStore, storage, traceStmtBuilder, aggExprRewriter, fl,
), nil
},
)
@@ -59,8 +57,7 @@ func NewOperatorFactory(
func NewTraceOperatorStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
storage qbtypes.Storage,
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation],
aggExprRewriter qbtypes.AggExprRewriter,
flagger flagger.Flagger,
@@ -81,8 +78,7 @@ func NewTraceOperatorStatementBuilder(
return &traceOperatorStatementBuilder{
logger: tracesSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
cb: conditionBuilder,
storage: storage,
traceStmtBuilder: traceStmtBuilder,
resourceFilterStmtBuilder: resourceFilterStmtBuilder,
aggExprRewriter: aggExprRewriter,

View File

@@ -22,8 +22,7 @@ func TestTraceTimeRangeOptimization(t *testing.T) {
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
@@ -40,13 +39,12 @@ func TestTraceTimeRangeOptimization(t *testing.T) {
Signal: telemetrytypes.SignalTraces,
}}
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
storage,
aggExprRewriter,
nil, // telemetryStore is nil - adaptive path is disabled
fl,

View File

@@ -8,61 +8,26 @@ import (
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
type conditionBuilder struct {
fm qbtypes.FieldMapper
// Compile keeps the related-values polarity form for a single key: the key
// must exist to apply the filter, and a row without it answers with the
// operator's polarity. A family compiles through the shared condition.
func (s *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
if logical.IsFamily() {
return querybuilder.SharedCondition(ctx, q, s, logical, operator, value, sb)
}
condition, err := s.conditionForKey(ctx, q, logical.Single(), operator, value, sb)
if err != nil {
return qbtypes.Compiled{}, err
}
return qbtypes.Compiled{Condition: condition}, nil
}
func NewConditionBuilder(fm qbtypes.FieldMapper) *conditionBuilder {
return &conditionBuilder{fm: fm}
}
// Metadata has no resource sub-query, so options are unused.
func (c *conditionBuilder) ConditionFor(
func (s *storage) conditionForKey(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken are logs-body-only; reject to avoid malformed related-values SQL.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}
// an unknown key simply yields no condition rather than an error. Metadata
// fields have no family support, so every logical field is single-member
// and flattens losslessly to its physical key.
resolved, warning := querybuilder.ResolveLogicalFields(key, querybuilder.MatchingLogicalFields(ctx, orgID, nil, key, fieldKeys))
keys := querybuilder.SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
cond, err := c.conditionForKey(ctx, orgID, tsStart, tsEnd, k, operator, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
}
return conds, warnings, nil
}
func (c *conditionBuilder) conditionForKey(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
q qbtypes.QueryInfo,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
@@ -79,13 +44,13 @@ func (c *conditionBuilder) conditionForKey(
value = querybuilder.FormatValueForContains(value)
}
columns, err := c.fm.ColumnFor(ctx, orgID, tsStart, tsEnd, key)
columns, err := s.getColumn(ctx, q.StartNs, q.EndNs, key)
if err != nil {
// if we don't have a column, we can't build a condition for related values
return "", nil
}
fieldExpression, err := c.fm.FieldFor(ctx, orgID, tsStart, tsEnd, key)
fieldExpression, err := s.Read(ctx, q, key)
if err != nil {
// if we don't have a table field name, we can't build a condition for related values
return "", nil

View File

@@ -2,11 +2,11 @@ package telemetrymetadata
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -14,7 +14,7 @@ import (
func TestConditionFor(t *testing.T) {
ctx := context.Background()
conditionBuilder := NewConditionBuilder(NewFieldMapper())
storage := NewStorage()
testCases := []struct {
name string
@@ -198,7 +198,7 @@ func TestConditionFor(t *testing.T) {
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, 0, 0, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
cond, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &tc.key, tc.operator, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, false, sb)
sb.Where(cond...)
if tc.expectedError != nil {

View File

@@ -3,15 +3,11 @@ package telemetrymetadata
import (
"context"
"fmt"
"strings"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"golang.org/x/exp/maps"
)
var (
@@ -27,20 +23,15 @@ var (
}
)
type fieldMapper struct {
type storage struct{}
var _ qbtypes.Storage = (*storage)(nil)
func NewStorage() *storage {
return &storage{}
}
// CandidateKeys returns nil: this mapper has no attribute-map fallback, so a context-missing
// key stays unresolved and the caller errors.
func (m *fieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, _ *telemetrytypes.TelemetryFieldKey, _ any, _ map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
}
func NewFieldMapper() qbtypes.FieldMapper {
return &fieldMapper{}
}
func (m *fieldMapper) getColumn(_ context.Context, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
func (m *storage) getColumn(_ context.Context, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
switch key.FieldContext {
case telemetrytypes.FieldContextResource:
return []*schema.Column{attributeMetadataColumns["resource_attributes"]}, nil
@@ -50,29 +41,8 @@ func (m *fieldMapper) getColumn(_ context.Context, _, _ uint64, key *telemetryty
return nil, qbtypes.ErrColumnNotFound
}
func (m *fieldMapper) ColumnFor(ctx context.Context, _ valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
columns, err := m.getColumn(ctx, tsStart, tsEnd, key)
if err != nil {
return nil, err
}
return columns, nil
}
// ExistsFor implements the per-key existence primitive of qbtypes.FieldMapper.
func (m *fieldMapper) ExistsFor(ctx context.Context, _ valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey, exists bool) (string, error) {
columns, err := m.getColumn(ctx, tsStart, tsEnd, key)
if err != nil {
return "", err
}
pred := fmt.Sprintf("mapContains(%s, '%s')", columns[0].Name, key.Name)
if exists {
return pred, nil
}
return "NOT " + pred, nil
}
func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, startNs, endNs, key)
func (m *storage) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, q.StartNs, q.EndNs, key)
if err != nil {
return "", err
}
@@ -87,48 +57,28 @@ func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, startNs, endN
return columns[0].Name, nil
}
func (m *fieldMapper) ColumnExpressionFor(
ctx context.Context,
orgID valuer.UUID,
startNs, endNs uint64,
field *telemetrytypes.TelemetryFieldKey,
_ telemetrytypes.FieldDataType,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
fieldExpression, err := m.FieldFor(ctx, orgID, startNs, endNs, field)
if errors.Is(err, qbtypes.ErrColumnNotFound) {
// the key didn't have the right context to be added to the query
// we try to use the context we know of
keysForField := keys[field.Name]
if len(keysForField) == 0 {
// is it a static field?
if _, ok := attributeMetadataColumns[field.Name]; ok {
// if it is, attach the column name directly
field.FieldContext = telemetrytypes.FieldContextSpan
fieldExpression, _ = m.FieldFor(ctx, orgID, startNs, endNs, field)
} else {
// - the context is not provided
// - there are not keys for the field
// - it is not a static field
// - the next best thing to do is see if there is a typo
// and suggest a correction
wrappedErr := errors.Wrapf(err, errors.TypeInvalidInput, errors.CodeInvalidInput, "field `%s` not found", field.Name).WithSuggestions(errors.NewSuggestionsOnLevenshteinDistance(field.Name, errors.NounKeys, maps.Keys(keys))...)
return "", wrappedErr
}
} else if len(keysForField) == 1 {
// we have a single key for the field, use it
fieldExpression, _ = m.FieldFor(ctx, orgID, startNs, endNs, keysForField[0])
} else {
// select any non-empty value from the keys
args := []string{}
for _, key := range keysForField {
fieldExpression, _ = m.FieldFor(ctx, orgID, startNs, endNs, key)
args = append(args, fmt.Sprintf("toString(%s) != '', toString(%s)", fieldExpression, fieldExpression))
}
fieldExpression = fmt.Sprintf("multiIf(%s, NULL)", strings.Join(args, ", "))
}
func (m *storage) Exists(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, exists bool) (qbtypes.Existence, error) {
columns, err := m.getColumn(ctx, q.StartNs, q.EndNs, key)
if err != nil {
return qbtypes.Existence{}, err
}
return fmt.Sprintf("%s AS `%s`", sqlbuilder.Escape(fieldExpression), field.Name), nil
predicate := fmt.Sprintf("mapContains(%s, '%s')", columns[0].Name, key.Name)
if !exists {
predicate = "NOT " + predicate
}
return qbtypes.Existence{Predicate: predicate, WhenAbsent: qbtypes.AbsentIsSentinel}, nil
}
// Fallback returns nil: the metadata tables have no attribute synthesis, so
// a key metadata does not hold yields no condition.
func (m *storage) Fallback(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, qbtypes.FilterOperator, any) ([]*telemetrytypes.LogicalField, error) {
return nil, nil
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{UnknownKey: qbtypes.IgnoreUnknownKey}
}
func (m *storage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
return querybuilder.DefaultRead(ctx, q, m, logical)
}

View File

@@ -7,7 +7,6 @@ import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -125,11 +124,11 @@ func TestGetColumn(t *testing.T) {
},
}
fm := NewFieldMapper()
storage := NewStorage()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
col, err := fm.ColumnFor(context.Background(), valuer.UUID{}, 0, 0, &tc.key)
col, err := storage.getColumn(context.Background(), 0, 0, &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
@@ -202,11 +201,11 @@ func TestGetFieldKeyName(t *testing.T) {
},
}
fm := NewFieldMapper()
storage := NewStorage()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
result, err := fm.FieldFor(ctx, valuer.UUID{}, tc.tsStart, tc.tsEnd, &tc.key)
result, err := storage.Read(ctx, qbtypes.QueryInfo{StartNs: tc.tsStart, EndNs: tc.tsEnd}, &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)

View File

@@ -65,8 +65,7 @@ type telemetryMetaStore struct {
relatedMetadataTblName string
columnEvolutionMetadataTblName string
fm qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
storage qbtypes.Storage
fl flagger.Flagger
jsonColumnMetadata map[telemetrytypes.Signal]map[telemetrytypes.FieldContext]telemetrytypes.JSONColumnMetadata
}
@@ -82,9 +81,6 @@ func NewTelemetryMetaStore(
) telemetrytypes.MetadataStore {
metadataSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrymetadata")
fm := NewFieldMapper()
conditionBuilder := NewConditionBuilder(fm)
t := &telemetryMetaStore{
logger: metadataSettings.Logger(),
telemetrystore: telemetrystore,
@@ -117,9 +113,8 @@ func NewTelemetryMetaStore(
},
},
},
fl: fl,
fm: fm,
conditionBuilder: conditionBuilder,
fl: fl,
storage: NewStorage(),
}
return t
@@ -1362,18 +1357,19 @@ func (t *telemetryMetaStore) getRelatedValues(ctx context.Context, orgID valuer.
FieldDataType: fieldValueSelector.FieldDataType,
}
selectColumn, err := t.fm.FieldFor(ctx, orgID, 0, 0, key)
q := querybuilder.NewQueryInfo(ctx, orgID, nil, fieldValueSelector.Signal, nil, 0, 0)
selectColumn, err := t.storage.Read(ctx, q, key)
if err != nil {
// we don't have a explicit column to select from the related metadata table
// so we will select either from resource_attributes or attributes table
// in that order
resourceColumn, _ := t.fm.FieldFor(ctx, orgID, 0, 0, &telemetrytypes.TelemetryFieldKey{
resourceColumn, _ := t.storage.Read(ctx, q, &telemetrytypes.TelemetryFieldKey{
Name: key.Name,
FieldContext: telemetrytypes.FieldContextResource,
FieldDataType: telemetrytypes.FieldDataTypeString,
})
attributeColumn, _ := t.fm.FieldFor(ctx, orgID, 0, 0, &telemetrytypes.TelemetryFieldKey{
attributeColumn, _ := t.storage.Read(ctx, q, &telemetrytypes.TelemetryFieldKey{
Name: key.Name,
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeString,
@@ -1394,11 +1390,11 @@ func (t *telemetryMetaStore) getRelatedValues(ctx context.Context, orgID valuer.
}
whereClause, err := querybuilder.PrepareWhereClause(fieldValueSelector.ExistingQuery, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Logger: t.logger,
FieldMapper: t.fm,
ConditionBuilder: t.conditionBuilder,
FieldKeys: keys,
Context: ctx,
Query: q,
Storage: t.storage,
Logger: t.logger,
FieldKeys: keys,
})
if err != nil {
t.logger.WarnContext(ctx, "error parsing existing query for related values", errors.Attr(err))
@@ -1429,20 +1425,20 @@ func (t *telemetryMetaStore) getRelatedValues(ctx context.Context, orgID valuer.
// search on attributes
key.FieldContext = telemetrytypes.FieldContextAttribute
attrConds, _, err := t.conditionBuilder.ConditionFor(ctx, orgID, 0, 0, key, map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {key}}, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorContains, fieldValueSelector.Value, sb)
attrConds, err := t.containsConditions(ctx, q, key, fieldValueSelector.Value, sb)
if err == nil {
conds = append(conds, attrConds...)
}
// search on resource
key.FieldContext = telemetrytypes.FieldContextResource
resourceConds, _, err := t.conditionBuilder.ConditionFor(ctx, orgID, 0, 0, key, map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {key}}, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorContains, fieldValueSelector.Value, sb)
resourceConds, err := t.containsConditions(ctx, q, key, fieldValueSelector.Value, sb)
if err == nil {
conds = append(conds, resourceConds...)
}
key.FieldContext = origContext
} else {
keyConds, _, err := t.conditionBuilder.ConditionFor(ctx, orgID, 0, 0, key, map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {key}}, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorContains, fieldValueSelector.Value, sb)
keyConds, err := t.containsConditions(ctx, q, key, fieldValueSelector.Value, sb)
if err == nil {
conds = append(conds, keyConds...)
}
@@ -2654,3 +2650,11 @@ func (t *telemetryMetaStore) fetchLastSeenInfoForTable(ctx context.Context, tabl
}
return lastSeenInfo, nil
}
// containsConditions compiles a contains search on one key of the related
// values table; the key is its own metadata.
func (t *telemetryMetaStore) containsConditions(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, value string, sb *sqlbuilder.SelectBuilder) ([]string, error) {
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {key}}
conds, _, err := querybuilder.Conditions(ctx, q, t.storage, key, qbtypes.FilterOperatorContains, value, fieldKeys, false, sb)
return conds, err
}

View File

@@ -1,215 +0,0 @@
package audittelemetryschema
import (
"context"
"fmt"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
type conditionBuilder struct {
fm qbtypes.FieldMapper
}
func NewConditionBuilder(fm qbtypes.FieldMapper) *conditionBuilder {
return &conditionBuilder{fm: fm}
}
func (c *conditionBuilder) conditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs, endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
if operator.IsStringSearchOperator() {
value = querybuilder.FormatValueForContains(value)
}
fieldExpression, err := c.fm.FieldFor(ctx, orgID, startNs, endNs, key)
if err != nil {
return "", err
}
fieldExpression, value = querybuilder.DataTypeCollisionHandledFieldName(key, value, fieldExpression, operator)
switch operator {
case qbtypes.FilterOperatorEqual:
return sb.E(fieldExpression, value), nil
case qbtypes.FilterOperatorNotEqual:
return sb.NE(fieldExpression, value), nil
case qbtypes.FilterOperatorGreaterThan:
return sb.G(fieldExpression, value), nil
case qbtypes.FilterOperatorGreaterThanOrEq:
return sb.GE(fieldExpression, value), nil
case qbtypes.FilterOperatorLessThan:
return sb.LT(fieldExpression, value), nil
case qbtypes.FilterOperatorLessThanOrEq:
return sb.LE(fieldExpression, value), nil
case qbtypes.FilterOperatorLike:
return sb.Like(fieldExpression, value), nil
case qbtypes.FilterOperatorNotLike:
return sb.NotLike(fieldExpression, value), nil
case qbtypes.FilterOperatorILike:
return sb.ILike(fieldExpression, value), nil
case qbtypes.FilterOperatorNotILike:
return sb.NotILike(fieldExpression, value), nil
case qbtypes.FilterOperatorContains:
return sb.ILike(fieldExpression, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorNotContains:
return sb.NotILike(fieldExpression, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorRegexp:
return fmt.Sprintf(`match(%s, %s)`, sqlbuilder.Escape(fieldExpression), sb.Var(value)), nil
case qbtypes.FilterOperatorNotRegexp:
return fmt.Sprintf(`NOT match(%s, %s)`, sqlbuilder.Escape(fieldExpression), sb.Var(value)), nil
case qbtypes.FilterOperatorBetween:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrBetweenValues
}
if len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.Between(fieldExpression, values[0], values[1]), nil
case qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrBetweenValues
}
if len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.NotBetween(fieldExpression, values[0], values[1]), nil
case qbtypes.FilterOperatorIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
conditions := []string{}
for _, value := range values {
cond, err := c.conditionFor(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorEqual, value, sb)
if err != nil {
return "", err
}
conditions = append(conditions, cond)
}
return sb.Or(conditions...), nil
case qbtypes.FilterOperatorNotIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
conditions := []string{}
for _, value := range values {
cond, err := c.conditionFor(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorNotEqual, value, sb)
if err != nil {
return "", err
}
conditions = append(conditions, cond)
}
return sb.And(conditions...), nil
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
columns, err := c.fm.ColumnFor(ctx, orgID, startNs, endNs, key)
if err != nil {
return "", err
}
pred, err := querybuilder.ExistsExpression(columns, key, startNs, endNs, fieldExpression, operator == qbtypes.FilterOperatorExists)
if err != nil {
return "", err
}
return sqlbuilder.Escape(pred), nil
}
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported operator: %v", operator)
}
func (c *conditionBuilder) ConditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
options qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for audit.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}
// Audit fields have no family support, so every logical field is
// single-member and flattens losslessly to its physical key.
resolved, warning := querybuilder.ResolveLogicalFields(key, querybuilder.MatchingLogicalFields(ctx, orgID, nil, key, fieldKeys))
keys := querybuilder.SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
if len(keys) == 0 {
return nil, warnings, querybuilder.NewKeyNotFoundError(key.Name)
}
// Drop resource keys the sub-query already covers; if none remain, skip the term (not an error).
if options.SkipResourceFilter {
filtered := make([]*telemetrytypes.TelemetryFieldKey, 0, len(keys))
for _, k := range keys {
if k.FieldContext != telemetrytypes.FieldContextResource {
filtered = append(filtered, k)
}
}
if len(filtered) == 0 {
return nil, warnings, nil
}
keys = filtered
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
cond, err := c.conditionForKey(ctx, orgID, startNs, endNs, k, operator, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
}
return conds, warnings, nil
}
func (c *conditionBuilder) conditionForKey(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
condition, err := c.conditionFor(ctx, orgID, startNs, endNs, key, operator, value, sb)
if err != nil {
return "", err
}
if key.FieldContext == telemetrytypes.FieldContextLog || key.FieldContext == telemetrytypes.FieldContextScope {
return condition, nil
}
if operator.AddDefaultExistsFilter() {
existsCondition, err := c.conditionFor(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorExists, nil, sb)
if err != nil {
return "", err
}
return sb.And(condition, existsCondition), nil
}
return condition, nil
}

View File

@@ -9,19 +9,18 @@ import (
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"golang.org/x/exp/maps"
)
type fieldMapper struct{}
type storage struct{}
func NewFieldMapper() qbtypes.FieldMapper {
return &fieldMapper{}
var _ qbtypes.Storage = (*storage)(nil)
func NewStorage() *storage {
return &storage{}
}
func (m *fieldMapper) getColumn(_ context.Context, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
func (m *storage) getColumn(_ context.Context, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
switch key.FieldContext {
case telemetrytypes.FieldContextResource:
return []*schema.Column{auditLogColumns["resource"]}, nil
@@ -53,7 +52,7 @@ func (m *fieldMapper) getColumn(_ context.Context, key *telemetrytypes.Telemetry
return nil, qbtypes.ErrColumnNotFound
}
func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
func (m *storage) Read(ctx context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, key)
if err != nil {
return "", err
@@ -93,74 +92,51 @@ func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, _, _ uint64,
return column.Name, nil
}
func (m *fieldMapper) ColumnFor(ctx context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
return m.getColumn(ctx, key)
}
// ExistsFor implements the per-key existence primitive of qbtypes.FieldMapper.
func (m *fieldMapper) ExistsFor(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey, exists bool) (string, error) {
fieldExpression, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
func (m *storage) Exists(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, exists bool) (qbtypes.Existence, error) {
fieldExpression, err := m.Read(ctx, q, key)
if err != nil {
return "", err
return qbtypes.Existence{}, err
}
columns, err := m.getColumn(ctx, key)
if err != nil {
return "", err
return qbtypes.Existence{}, err
}
return querybuilder.ExistsExpression(columns, key, tsStart, tsEnd, fieldExpression, exists)
predicate, err := querybuilder.ExistsExpression(columns, key, q.StartNs, q.EndNs, fieldExpression, exists)
if err != nil {
return qbtypes.Existence{}, err
}
return qbtypes.Existence{Predicate: predicate, WhenAbsent: absentReads(columns[0])}, nil
}
func (m *fieldMapper) ColumnExpressionFor(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
field *telemetrytypes.TelemetryFieldKey,
requiredDataType telemetrytypes.FieldDataType,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
resolved := field
fieldExpression, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, field)
if errors.Is(err, qbtypes.ErrColumnNotFound) {
keysForField := keys[field.Name]
if len(keysForField) == 0 {
if _, ok := auditLogColumns[field.Name]; ok {
field.FieldContext = telemetrytypes.FieldContextLog
fieldExpression, _ = m.FieldFor(ctx, orgID, tsStart, tsEnd, field)
} else {
wrappedErr := errors.Wrapf(err, errors.TypeInvalidInput, errors.CodeInvalidInput, "field `%s` not found", field.Name).WithSuggestions(errors.NewSuggestionsOnLevenshteinDistance(field.Name, errors.NounKeys, maps.Keys(keys))...)
return "", wrappedErr
}
} else {
resolved = keysForField[0]
fieldExpression, _ = m.FieldFor(ctx, orgID, tsStart, tsEnd, keysForField[0])
}
// absentReads tells what a row without the key reads from the column: the
// empty value of a map or of a JSON path (its ::String cast folds NULL to
// the empty string), and a real value from every other column.
func absentReads(column *schema.Column) qbtypes.Absent {
switch column.Type.GetType() {
case schema.ColumnTypeEnumMap, schema.ColumnTypeEnumJSON:
return qbtypes.AbsentIsSentinel
}
// Group-by/order (String) and aggregation (String/Float64): exists-guarded and coerced
// to requiredDataType, returned bare (the caller adds any alias). Raw select
// (Unspecified) returns the aliased column expression.
if requiredDataType != telemetrytypes.FieldDataTypeUnspecified {
var dummyValue any = ""
if requiredDataType == telemetrytypes.FieldDataTypeFloat64 {
dummyValue = 0.0
}
columns, err := m.getColumn(ctx, resolved)
if err != nil {
return "", err
}
guard, err := querybuilder.ExistsExpression(columns, resolved, tsStart, tsEnd, fieldExpression, true)
if err != nil {
return "", err
}
coerced, _ := querybuilder.DataTypeCollisionHandledFieldName(resolved, dummyValue, fieldExpression, qbtypes.FilterOperatorUnknown)
return fmt.Sprintf("multiIf(%s, %s, NULL)", guard, coerced), nil
}
return fmt.Sprintf("%s AS `%s`", sqlbuilder.Escape(fieldExpression), field.Name), nil
return qbtypes.AlwaysPresent
}
// CandidateKeys returns nil: audit has no synthesize-on-unknown-key fallback, so an
// unknown key stays unresolved and the caller errors.
func (m *fieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, _ *telemetrytypes.TelemetryFieldKey, _ any, _ map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
// Fallback answers a column name with the column; audit has no attribute
// synthesis, so any other unknown key stays unresolved and the caller errors.
func (m *storage) Fallback(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, _ qbtypes.FilterOperator, _ any) ([]*telemetrytypes.LogicalField, error) {
if _, ok := auditLogColumns[key.Name]; !ok {
return nil, nil
}
column := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextLog, key.FieldDataType)
return querybuilder.WrapAsLogicalFields(key.Name, []*telemetrytypes.TelemetryFieldKey{column}), nil
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{Split: qbtypes.MainOfSplit}
}
func (m *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
return querybuilder.SharedCondition(ctx, q, m, logical, operator, value, sb)
}
func (m *storage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
return querybuilder.DefaultRead(ctx, q, m, logical)
}

View File

@@ -7,32 +7,17 @@ import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
type conditionBuilder struct {
fm qbtypes.FieldMapper
fl flagger.Flagger
}
var _ qbtypes.ConditionBuilder = (*conditionBuilder)(nil)
func NewConditionBuilder(fm qbtypes.FieldMapper, fl flagger.Flagger) *conditionBuilder {
return &conditionBuilder{fm: fm, fl: fl}
}
// conditionForSearch ORs a case-insensitive match of the search term across the key
// context's searchable columns (unspecified context = every column).
func (c *conditionBuilder) conditionForSearch(
ctx context.Context,
orgID valuer.UUID,
func (s *storage) conditionForSearch(
q qbtypes.QueryInfo,
key *telemetrytypes.TelemetryFieldKey,
value any,
sb *sqlbuilder.SelectBuilder,
@@ -41,7 +26,7 @@ func (c *conditionBuilder) conditionForSearch(
// LOWER(toString(body_v2)) skip index.
term := regexp.QuoteMeta(fmt.Sprintf("%v", value))
useJSONBody := c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
useJSONBody := q.BodyJSONOn
var conditions []string
@@ -91,9 +76,8 @@ func isBodyJSONSearch(key *telemetrytypes.TelemetryFieldKey, columns []*schema.C
// conditionForArrayFunction builds has/hasAny/hasAll over a body JSON path — via the JSON
// access plan (flag on) or legacy typed extraction (flag off).
func (c *conditionBuilder) conditionForArrayFunction(
ctx context.Context,
orgID valuer.UUID,
func (s *storage) conditionForArrayFunction(
q qbtypes.QueryInfo,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
@@ -110,7 +94,7 @@ func (c *conditionBuilder) conditionForArrayFunction(
needle = args[0]
}
if c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
if q.BodyJSONOn {
// JSON access plan: data-type collision handling, nested array paths.
valueType, needle := InferDataType(needle, operator, key)
// A not-found (synthesized) body path carries no metadata plan; build an exhaustive
@@ -184,9 +168,8 @@ func firstTokenSeparator(s string) (string, bool) {
// conditionForHasToken builds a hasToken full-text search over the body column, resolving the
// column from the key name + use_json_body flag.
func (c *conditionBuilder) conditionForHasToken(
ctx context.Context,
orgID valuer.UUID,
func (s *storage) conditionForHasToken(
q qbtypes.QueryInfo,
key *telemetrytypes.TelemetryFieldKey,
value any,
sb *sqlbuilder.SelectBuilder,
@@ -211,7 +194,7 @@ func (c *conditionBuilder) conditionForHasToken(
needleStr, sep, needleStr).WithUrl(hasTokenFunctionDocURL)
}
bodyJSONEnabled := c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
bodyJSONEnabled := q.BodyJSONOn
if !bodyJSONEnabled {
// legacy: token search over the plain body string column only.
@@ -246,10 +229,9 @@ func (c *conditionBuilder) conditionForHasToken(
"function `hasToken` only supports the body field or a body JSON string field as first parameter").WithUrl(hasTokenFunctionDocURL)
}
func (c *conditionBuilder) conditionForResolvedKey(
func (s *storage) conditionForResolvedKey(
ctx context.Context,
orgID valuer.UUID,
startNs, endNs uint64,
q qbtypes.QueryInfo,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
@@ -257,13 +239,13 @@ func (c *conditionBuilder) conditionForResolvedKey(
) (string, error) {
// hasToken resolves from the key name + flag alone (no column resolution), so handle it first.
if operator == qbtypes.FilterOperatorHasToken {
return c.conditionForHasToken(ctx, orgID, key, value, sb)
return s.conditionForHasToken(q, key, value, sb)
}
columns, err := c.fm.ColumnFor(ctx, orgID, startNs, endNs, key)
columns, err := s.getColumn(q, key)
if errors.Is(err, qbtypes.ErrColumnNotFound) && key.FieldContext == telemetrytypes.FieldContextUnspecified {
key = telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextBody, key.FieldDataType)
columns, err = c.fm.ColumnFor(ctx, orgID, startNs, endNs, key)
columns, err = s.getColumn(q, key)
}
if err != nil {
return "", err
@@ -271,12 +253,12 @@ func (c *conditionBuilder) conditionForResolvedKey(
// has/hasAny/hasAll take the body-JSON path, not the normal operator paths.
if operator.IsArrayFunctionOperator() {
return c.conditionForArrayFunction(ctx, orgID, key, operator, value, columns, sb)
return s.conditionForArrayFunction(q, key, operator, value, columns, sb)
}
// TODO(Piyush): Update this to support multiple JSON columns based on evolutions
for _, column := range columns {
if column.Type.GetType() == schema.ColumnTypeEnumJSON && isBodyJSONSearch(key, columns) && c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) && key.Name != messageSubField {
if column.Type.GetType() == schema.ColumnTypeEnumJSON && isBodyJSONSearch(key, columns) && q.BodyJSONOn && key.Name != messageSubField {
valueType, value := InferDataType(value, operator, key)
if len(key.JSONPlan) == 0 {
keyCopy := telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, key.FieldDataType)
@@ -299,13 +281,13 @@ func (c *conditionBuilder) conditionForResolvedKey(
value = querybuilder.FormatValueForContains(value)
}
fieldExpression, err := c.fm.FieldFor(ctx, orgID, startNs, endNs, key)
fieldExpression, err := s.Read(ctx, q, key)
if err != nil {
return "", err
}
// Check if this is a body JSON search (legacy string-body path, JSON flag off).
if isBodyJSONSearch(key, columns) && !c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
if isBodyJSONSearch(key, columns) && !q.BodyJSONOn {
fieldExpression, value = GetBodyJSONKey(ctx, key, operator, value)
}
@@ -356,13 +338,13 @@ func (c *conditionBuilder) conditionForResolvedKey(
return sb.NotILike(fieldExpression, value), nil
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
if isBodyJSONSearch(key, columns) && !c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
if isBodyJSONSearch(key, columns) && !q.BodyJSONOn {
if operator == qbtypes.FilterOperatorExists {
return GetBodyJSONKeyForExists(ctx, key, operator, value), nil
}
return "NOT " + GetBodyJSONKeyForExists(ctx, key, operator, value), nil
}
pred, err := querybuilder.ExistsExpression(columns, key, startNs, endNs, fieldExpression, operator == qbtypes.FilterOperatorExists)
pred, err := querybuilder.ExistsExpression(columns, key, q.StartNs, q.EndNs, fieldExpression, operator == qbtypes.FilterOperatorExists)
if err != nil {
return "", err
}
@@ -410,7 +392,7 @@ func (c *conditionBuilder) conditionForResolvedKey(
// instead of using IN, we use `=` + `OR` to make use of index
conditions := []string{}
for _, value := range values {
cond, err := c.conditionForResolvedKey(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorEqual, value, sb)
cond, err := s.conditionForResolvedKey(ctx, q, key, qbtypes.FilterOperatorEqual, value, sb)
if err != nil {
return "", err
}
@@ -425,7 +407,7 @@ func (c *conditionBuilder) conditionForResolvedKey(
// instead of using NOT IN, we use `!=` + `AND` to make use of index
conditions := []string{}
for _, value := range values {
cond, err := c.conditionForResolvedKey(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorNotEqual, value, sb)
cond, err := s.conditionForResolvedKey(ctx, q, key, qbtypes.FilterOperatorNotEqual, value, sb)
if err != nil {
return "", err
}
@@ -437,132 +419,29 @@ func (c *conditionBuilder) conditionForResolvedKey(
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported operator: %v", operator)
}
// candidateLookupKeys returns the metadata map only for fold-contexts, where CandidateKeys
// would otherwise fold the prefix into the key name. Handing it the map lets a same-named
// key under another context resolve first (as ColumnExpressionFor does). Strict contexts
// (resource/attribute/scope) get nil so their explicit context is always honored.
func candidateLookupKeys(key *telemetrytypes.TelemetryFieldKey, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) map[string][]*telemetrytypes.TelemetryFieldKey {
if key.FieldContext == telemetrytypes.FieldContextLog {
return fieldKeys
}
return nil
}
func (c *conditionBuilder) ConditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
options qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
matches := querybuilder.MatchingLogicalFields(ctx, orgID, nil, key, fieldKeys)
skipResourceFilter := options.SkipResourceFilter
// search() resolves its own (optional) scope; handle it before key resolution.
if operator == qbtypes.FilterOperatorSearch {
return c.conditionForSearch(ctx, orgID, key, value, sb)
}
// Logs fields have no family support yet, so every logical field is
// single-member and flattens losslessly to its physical key.
resolved, warning := querybuilder.ResolveLogicalFields(key, matches)
keys := querybuilder.SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
synthesized := false
if len(keys) == 0 {
_, isIntrinsicColumn := logsV2Columns[key.Name]
switch {
case key.FieldContext == telemetrytypes.FieldContextBody && key.Name == "":
return nil, warnings, errors.NewInvalidInputf(errors.CodeInvalidInput, "missing key for body json search - expected key of the form `body.key` (ex: `body.status`)")
case key.FieldContext == telemetrytypes.FieldContextLog && isIntrinsicColumn:
keys = []*telemetrytypes.TelemetryFieldKey{key}
default:
// Fold-contexts get the metadata map so a same-named key under another context
// wins before the prefix folds into the key name (matching ColumnExpressionFor);
// strict contexts pass nil and stay honored as-is.
keys = c.fm.CandidateKeys(ctx, orgID, key, value, candidateLookupKeys(key, fieldKeys))
if operator.IsFunctionOperator() {
if key.FieldContext != telemetrytypes.FieldContextBody {
// has/hasAny/hasAll/hasToken are body-JSON only
return nil, warnings, querybuilder.NewFunctionUnsupportedError(operator)
}
bodyKeys := make([]*telemetrytypes.TelemetryFieldKey, 0, len(keys))
for _, k := range keys {
if k.FieldContext == telemetrytypes.FieldContextBody {
bodyKeys = append(bodyKeys, k)
}
}
keys = bodyKeys
}
if len(keys) == 0 {
return nil, warnings, querybuilder.NewKeyNotFoundError(key.Name)
}
synthesized = true
warnings = append(warnings, querybuilder.NewKeyNotFoundWarning(key.Name))
}
}
if skipResourceFilter && !synthesized {
filtered := make([]*telemetrytypes.TelemetryFieldKey, 0, len(keys))
for _, k := range keys {
if k.FieldContext != telemetrytypes.FieldContextResource {
filtered = append(filtered, k)
}
}
if len(filtered) == 0 {
return nil, warnings, nil
}
keys = filtered
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
cond, err := c.conditionForKey(ctx, orgID, startNs, endNs, k, operator, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
if w := c.bodyFullTextDefaultWarning(ctx, orgID, startNs, endNs, k, operator); w != "" {
warnings = append(warnings, w)
}
}
return conds, warnings, nil
}
// bodyFullTextDefaultWarning returns the advisory shown when a regexp full-text
// search on `body` resolves to the body.message sub-field (JSON mode), else "". This
// keeps the JSON-vs-legacy decision in the builder rather than the filter visitor.
func (c *conditionBuilder) bodyFullTextDefaultWarning(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey, operator qbtypes.FilterOperator) string {
func (s *storage) bodyFullTextDefaultWarning(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, operator qbtypes.FilterOperator) string {
if operator != qbtypes.FilterOperatorRegexp || key.Name != LogsV2BodyColumn {
return ""
}
if field, err := c.fm.FieldFor(ctx, orgID, startNs, endNs, key); err == nil && field == messageSubColumn {
if field, err := s.Read(ctx, q, key); err == nil && field == messageSubColumn {
return querybuilder.BodyFullTextSearchDefaultWarning
}
return ""
}
func (c *conditionBuilder) conditionForKey(
func (s *storage) conditionForKey(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
q qbtypes.QueryInfo,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
condition, err := c.conditionForResolvedKey(ctx, orgID, startNs, endNs, key, operator, value, sb)
condition, err := s.conditionForResolvedKey(ctx, q, key, operator, value, sb)
if err != nil {
return "", err
}
@@ -570,22 +449,28 @@ func (c *conditionBuilder) conditionForKey(
// Skip adding exists filter for intrinsic fields i.e. Table level log context fields
buildExistCondition := operator.AddDefaultExistsFilter()
switch key.FieldContext {
case telemetrytypes.FieldContextLog, telemetrytypes.FieldContextScope:
case telemetrytypes.FieldContextLog:
// pass; No need to build exist condition for top level columns
// immediately return
return condition, nil
case telemetrytypes.FieldContextScope:
// scope_name and scope_version are columns; a scope attribute lives in
// a map and follows the keyless contract like an attribute
if !s.mapBacked(q, key) {
return condition, nil
}
case telemetrytypes.FieldContextResource, telemetrytypes.FieldContextAttribute:
// build exist condition for resource and attribute fields based on filter operator
case telemetrytypes.FieldContextBody:
// Querying JSON fields already account for Nullability of fields
// so additional exists checks are not needed
if c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
if q.BodyJSONOn {
return condition, nil
}
}
if buildExistCondition {
existsCondition, err := c.conditionForResolvedKey(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorExists, nil, sb)
existsCondition, err := s.conditionForResolvedKey(ctx, q, key, qbtypes.FilterOperatorExists, nil, sb)
if err != nil {
return "", err
}
@@ -594,3 +479,44 @@ func (c *conditionBuilder) conditionForKey(
return condition, nil
}
// mapBacked reports whether the key reads a map column, so a row can lack it.
func (s *storage) mapBacked(q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) bool {
columns, err := s.getColumn(q, key)
return err == nil && len(columns) == 1 && columns[0].Type.GetType() == schema.ColumnTypeEnumMap
}
// Compile keeps the body language: search over a scope, the body functions,
// the body JSON paths, and the body column forms that use its index. Every
// other field compiles through the shared condition.
func (s *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
if operator == qbtypes.FilterOperatorSearch {
conditions, _, err := s.conditionForSearch(q, logical.Single(), value, sb)
if err != nil || len(conditions) == 0 {
return qbtypes.Compiled{}, err
}
return qbtypes.Compiled{Condition: conditions[0]}, nil
}
if logical.IsFamily() || !s.ownLanguage(logical.Single(), operator) {
return querybuilder.SharedCondition(ctx, q, s, logical, operator, value, sb)
}
key := logical.Single()
condition, err := s.conditionForKey(ctx, q, key, operator, value, sb)
if err != nil {
return qbtypes.Compiled{}, err
}
compiled := qbtypes.Compiled{Condition: condition}
if w := s.bodyFullTextDefaultWarning(ctx, q, key, operator); w != "" {
compiled.Warnings = append(compiled.Warnings, w)
}
return compiled, nil
}
// ownLanguage reports whether a term needs the body language: a body path,
// a body function, or the body column itself.
func (s *storage) ownLanguage(key *telemetrytypes.TelemetryFieldKey, operator qbtypes.FilterOperator) bool {
if operator.IsFunctionOperator() || key.FieldContext == telemetrytypes.FieldContextBody {
return true
}
return key.Name == LogsV2BodyColumn && (key.FieldContext == telemetrytypes.FieldContextLog || key.FieldContext == telemetrytypes.FieldContextUnspecified)
}

View File

@@ -2,6 +2,7 @@ package logstelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
"time"
@@ -125,14 +126,13 @@ func TestExistsConditionForWithEvolutions(t *testing.T) {
},
}
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
conditionBuilder := NewConditionBuilder(fm, fl)
storage := NewStorage()
ctx := context.Background()
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, tc.startTs, tc.endTs, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
cond, _, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, tc.startTs, tc.endTs), storage, &tc.key, tc.operator, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, false, sb)
sb.Where(cond...)
if tc.expectedError != nil {
@@ -517,13 +517,12 @@ func TestConditionFor(t *testing.T) {
},
}
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
conditionBuilder := NewConditionBuilder(fm, fl)
storage := NewStorage()
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
tc.key.Evolutions = tc.evolutions
cond, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, 0, 0, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
cond, _, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &tc.key, tc.operator, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, false, sb)
sb.Where(cond...)
if tc.expectedError != nil {
@@ -543,12 +542,12 @@ func TestConditionFor(t *testing.T) {
func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
cb := NewConditionBuilder(NewFieldMapper(fl), fl)
storage := NewStorage()
t.Run("bare intrinsic column resolves to the column, not synthesized attributes", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "severity_text"}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "ERROR", sb)
conds, _, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &key, qbtypes.FilterOperatorEqual, "ERROR", nil, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -560,7 +559,7 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("attribute context", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextAttribute}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &key, qbtypes.FilterOperatorEqual, "v", nil, false, sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 2)
@@ -571,18 +570,22 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("resource context", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextResource}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &key, qbtypes.FilterOperatorEqual, "v", nil, false, sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 2)
assert.Contains(t, conds[0], "mapContains(resources_string, 'custom.key')")
assert.Contains(t, conds[1], "mapContains(resources_string, 'resource.custom.key')")
// the resource read spans both eras and yields NULL for an absent key, so
// it takes no presence guard
sb.Where(conds...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
assert.Contains(t, sql, "resources_string['custom.key']")
assert.Contains(t, sql, "resources_string['resource.custom.key']")
})
t.Run("log context folds to attributes then body", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextLog}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &key, qbtypes.FilterOperatorEqual, "v", nil, false, sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 4)
@@ -626,15 +629,14 @@ func TestConditionForMultipleKeys(t *testing.T) {
}
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
conditionBuilder := NewConditionBuilder(fm, fl)
storage := NewStorage()
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
var err error
for _, key := range tc.keys {
cond, err := conditionBuilder.conditionForResolvedKey(ctx, valuer.UUID{}, 0, 0, &key, tc.operator, tc.value, sb)
cond, err := storage.conditionForResolvedKey(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), &key, tc.operator, tc.value, sb)
sb.Where(cond)
if err != nil {
t.Fatalf("Error getting condition for key %s: %v", key.Name, err)
@@ -886,13 +888,12 @@ func TestConditionForJSONBodySearch(t *testing.T) {
}
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
conditionBuilder := NewConditionBuilder(fm, fl)
storage := NewStorage()
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, err := conditionBuilder.conditionForResolvedKey(ctx, valuer.UUID{}, 0, 0, &tc.key, tc.operator, tc.value, sb)
cond, err := storage.conditionForResolvedKey(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), &tc.key, tc.operator, tc.value, sb)
sb.Where(cond)
if tc.expectedError != nil {
@@ -930,8 +931,7 @@ func TestConditionForBodyIn(t *testing.T) {
}
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
conditionBuilder := NewConditionBuilder(fm, fl)
storage := NewStorage()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
@@ -942,9 +942,7 @@ func TestConditionForBodyIn(t *testing.T) {
}
sb := sqlbuilder.NewSelectBuilder()
sb.Select("1").From("t")
cond, _, err := conditionBuilder.ConditionFor(context.Background(), valuer.UUID{}, 0, 0, &key,
map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {&key}}, qbtypes.ConditionBuilderOptions{},
qbtypes.FilterOperatorIn, tc.values, sb)
cond, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &key, qbtypes.FilterOperatorIn, tc.values, map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {&key}}, false, sb)
require.NoError(t, err)
sb.Where(cond...)

View File

@@ -8,12 +8,9 @@ import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz-otel-collector/utils"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
@@ -63,15 +60,15 @@ var (
}
)
type fieldMapper struct {
fl flagger.Flagger
type storage struct{}
var _ qbtypes.Storage = (*storage)(nil)
func NewStorage() *storage {
return &storage{}
}
func NewFieldMapper(fl flagger.Flagger) qbtypes.FieldMapper {
return &fieldMapper{fl: fl}
}
func (m *fieldMapper) getColumn(ctx context.Context, orgID valuer.UUID, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
func (m *storage) getColumn(q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
switch key.FieldContext {
case telemetrytypes.FieldContextResource:
columns := []*schema.Column{logsV2Columns["resource"], logsV2Columns["resources_string"]}
@@ -95,7 +92,7 @@ func (m *fieldMapper) getColumn(ctx context.Context, orgID valuer.UUID, key *tel
}
case telemetrytypes.FieldContextBody:
// Body context is for JSON body fields. Use body_v2 if feature flag is enabled.
if m.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
if q.BodyJSONOn {
if key.Name == messageSubField {
return []*schema.Column{logsV2Columns[messageSubColumn]}, nil
}
@@ -104,7 +101,7 @@ func (m *fieldMapper) getColumn(ctx context.Context, orgID valuer.UUID, key *tel
// Fall back to legacy body column
return []*schema.Column{logsV2Columns["body"]}, nil
case telemetrytypes.FieldContextLog:
if key.Name == LogsV2BodyColumn && m.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
if key.Name == LogsV2BodyColumn && q.BodyJSONOn {
return []*schema.Column{logsV2Columns[messageSubColumn]}, nil
}
col, ok := logsV2Columns[key.Name]
@@ -117,13 +114,13 @@ func (m *fieldMapper) getColumn(ctx context.Context, orgID valuer.UUID, key *tel
return nil, qbtypes.ErrColumnNotFound
}
func (m *fieldMapper) FieldFor(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, orgID, key)
func (m *storage) Read(_ context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(q, key)
if err != nil {
return "", err
}
newColumns, evolutionsEntries, err := qbtypes.SelectEvolutionsForColumns(columns, key.Evolutions, tsStart, tsEnd)
newColumns, evolutionsEntries, err := qbtypes.SelectEvolutionsForColumns(columns, key.Evolutions, q.StartNs, q.EndNs)
if err != nil {
return "", err
}
@@ -213,191 +210,8 @@ func (m *fieldMapper) FieldFor(ctx context.Context, orgID valuer.UUID, tsStart,
return columns[0].Name, nil
}
func (m *fieldMapper) ColumnFor(ctx context.Context, orgID valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
return m.getColumn(ctx, orgID, key)
}
func (m *fieldMapper) ColumnExpressionFor(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
field *telemetrytypes.TelemetryFieldKey,
requiredDataType telemetrytypes.FieldDataType,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
bodyJSONEnabled := m.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
var candidates []*telemetrytypes.TelemetryFieldKey
switch _, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, field); {
case err == nil:
if field.FieldContext == telemetrytypes.FieldContextBody && !bodyJSONEnabled {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "Operation isn't available for the body column")
}
candidates = []*telemetrytypes.TelemetryFieldKey{field}
case errors.Is(err, qbtypes.ErrColumnNotFound):
if _, ok := logsV2Columns[field.Name]; ok {
field.FieldContext = telemetrytypes.FieldContextLog
candidates = []*telemetrytypes.TelemetryFieldKey{field}
break
}
candidates = keys[field.Name]
if len(candidates) == 0 {
candidates = keys[fmt.Sprintf("%s.%s", field.FieldContext.StringValue(), field.Name)]
}
if len(candidates) == 0 {
// synthesized attribute candidates first, body path last; legacy body
// doesn't support group by/select, so bare keys keep attributes only
for _, key := range m.CandidateKeys(ctx, orgID, field, nil, nil) {
if !bodyJSONEnabled && field.FieldContext == telemetrytypes.FieldContextUnspecified &&
key.FieldContext == telemetrytypes.FieldContextBody {
continue
}
candidates = append(candidates, key)
}
}
if len(candidates) == 0 {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "field `%s` not found", field.Name)
}
default:
return "", err
}
// Group-by/order (String) and aggregation (String/Float64): every candidate is
// exists-guarded and coerced to requiredDataType, in a single multiIf. Raw select
// (Unspecified) keeps the lighter native shape below.
if requiredDataType != telemetrytypes.FieldDataTypeUnspecified {
// arrays cannot sit inside Nullable/multiIf, so a lone array candidate stays bare
if len(candidates) == 1 && (strings.Contains(candidates[0].Name, telemetrytypes.ArraySep) ||
strings.Contains(candidates[0].Name, telemetrytypes.ArrayAnyIndex) ||
candidates[0].FieldDataType.IsArray()) {
return m.FieldFor(ctx, orgID, tsStart, tsEnd, candidates[0])
}
var dummyValue any = ""
if requiredDataType == telemetrytypes.FieldDataTypeFloat64 {
dummyValue = 0.0
}
var stmts []string
for _, key := range candidates {
guard, err := m.ExistsFor(ctx, orgID, tsStart, tsEnd, key, true)
if err != nil {
return "", err
}
var fieldExpression string
if key.FieldContext == telemetrytypes.FieldContextBody && !bodyJSONEnabled {
fieldExpression, _ = GetBodyJSONKey(ctx, key, qbtypes.FilterOperatorUnknown, dummyValue)
} else {
fieldExpression, err = m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
return "", err
}
fieldExpression, _ = querybuilder.DataTypeCollisionHandledFieldName(key, dummyValue, fieldExpression, qbtypes.FilterOperatorUnknown)
}
stmts = append(stmts, guard, fieldExpression)
}
return fmt.Sprintf("multiIf(%s, NULL)", strings.Join(stmts, ", ")), nil
}
if len(candidates) == 1 {
// arrays cannot sit inside Nullable, so array candidates stay bare
if strings.Contains(candidates[0].Name, telemetrytypes.ArraySep) ||
strings.Contains(candidates[0].Name, telemetrytypes.ArrayAnyIndex) ||
candidates[0].FieldDataType.IsArray() {
return m.FieldFor(ctx, orgID, tsStart, tsEnd, candidates[0])
}
if !m.membershipGuarded(ctx, orgID, tsStart, tsEnd, candidates[0]) {
return m.FieldFor(ctx, orgID, tsStart, tsEnd, candidates[0])
}
guard, err := m.ExistsFor(ctx, orgID, tsStart, tsEnd, candidates[0], true)
if err != nil {
return "", err
}
var fieldExpression string
if candidates[0].FieldContext == telemetrytypes.FieldContextBody && !bodyJSONEnabled {
fieldExpression, _ = GetBodyJSONKey(ctx, candidates[0], qbtypes.FilterOperatorUnknown, "")
} else {
fieldExpression, err = m.FieldFor(ctx, orgID, tsStart, tsEnd, candidates[0])
if err != nil {
return "", err
}
}
return fmt.Sprintf("multiIf(%s, %s, NULL)", guard, fieldExpression), nil
}
var stmts []string
for _, key := range candidates {
guard, err := m.ExistsFor(ctx, orgID, tsStart, tsEnd, key, true)
if err != nil {
return "", err
}
stmts = append(stmts, guard)
var fieldExpression string
if key.FieldContext == telemetrytypes.FieldContextBody && !bodyJSONEnabled {
fieldExpression, _ = GetBodyJSONKey(ctx, key, qbtypes.FilterOperatorUnknown, "")
} else {
fieldExpression, err = m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
return "", err
}
fieldExpression, _ = querybuilder.DataTypeCollisionHandledFieldName(key, "", fieldExpression, qbtypes.FilterOperatorUnknown)
}
stmts = append(stmts, fieldExpression)
}
return fmt.Sprintf("multiIf(%s, NULL)", strings.Join(stmts, ", ")), nil
}
func (m *fieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, field *telemetrytypes.TelemetryFieldKey, value any, keys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
if matches := keys[field.Name]; len(matches) > 0 {
return matches
}
if matches := keys[fmt.Sprintf("%s.%s", field.FieldContext.StringValue(), field.Name)]; len(matches) > 0 {
return matches
}
switch field.FieldContext {
case telemetrytypes.FieldContextBody:
if field.Name == "" {
return nil
}
return []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(field.Name, field.FieldContext, field.FieldDataType)}
case telemetrytypes.FieldContextUnspecified:
// a real column wins before synthesis (so adjustKeys is not needed to resolve these)
if _, ok := logsV2Columns[field.Name]; ok {
return []*telemetrytypes.TelemetryFieldKey{{Name: field.Name, FieldContext: telemetrytypes.FieldContextLog}}
}
bodyKey := telemetrytypes.NewTelemetryFieldKey(field.Name, telemetrytypes.FieldContextBody, field.FieldDataType)
if value == nil && bodyKey.FieldDataType == telemetrytypes.FieldDataTypeUnspecified {
bodyKey.FieldDataType = telemetrytypes.FieldDataTypeString
}
return append(querybuilder.SynthesizeKeys(field, value), bodyKey)
case telemetrytypes.FieldContextAttribute, telemetrytypes.FieldContextResource:
// stripped interpretation first, the literal `{context}.{name}` spelling second
literal := telemetrytypes.NewTelemetryFieldKey(field.FieldContext.StringValue()+"."+field.Name, field.FieldContext, field.FieldDataType)
return append(querybuilder.SynthesizeKeys(field, value), querybuilder.SynthesizeKeys(literal, value)...)
case telemetrytypes.FieldContextLog:
if _, ok := logsV2Columns[field.Name]; ok {
return []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(field.Name, field.FieldContext, field.FieldDataType)}
}
stripped := telemetrytypes.NewTelemetryFieldKey(field.Name, telemetrytypes.FieldContextUnspecified, field.FieldDataType)
literal := telemetrytypes.NewTelemetryFieldKey(field.FieldContext.StringValue()+"."+field.Name, telemetrytypes.FieldContextUnspecified, field.FieldDataType)
// attribute candidates first (stripped, then literal), body paths last
candidates := append(querybuilder.SynthesizeKeys(stripped, value), querybuilder.SynthesizeKeys(literal, value)...)
for _, key := range []*telemetrytypes.TelemetryFieldKey{stripped, literal} {
bodyKey := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextBody, key.FieldDataType)
if value == nil && bodyKey.FieldDataType == telemetrytypes.FieldDataTypeUnspecified {
bodyKey.FieldDataType = telemetrytypes.FieldDataTypeString
}
candidates = append(candidates, bodyKey)
}
return candidates
}
return nil
}
// buildFieldForJSON builds the field expression for body JSON fields using arrayConcat pattern.
func (m *fieldMapper) buildFieldForJSON(key *telemetrytypes.TelemetryFieldKey) (string, error) {
func (m *storage) buildFieldForJSON(key *telemetrytypes.TelemetryFieldKey) (string, error) {
plan := key.JSONPlan
if len(plan) == 0 {
if key.KeyNameContainsArray() {
@@ -468,7 +282,7 @@ func (m *fieldMapper) buildFieldForJSON(key *telemetrytypes.TelemetryFieldKey) (
}
// buildArrayConcat builds the arrayConcat pattern directly from the tree structure.
func (m *fieldMapper) buildArrayConcat(plan telemetrytypes.JSONAccessPlan) (string, error) {
func (m *storage) buildArrayConcat(plan telemetrytypes.JSONAccessPlan) (string, error) {
if len(plan) == 0 {
return "", errors.NewInternalf(CodeGroupByPlanEmpty, "group by plan is empty while building arrayConcat")
}
@@ -499,7 +313,7 @@ func (m *fieldMapper) buildArrayConcat(plan telemetrytypes.JSONAccessPlan) (stri
}
// buildArrayMap builds the arrayMap expression for a specific branch, handling all sub-branches.
func (m *fieldMapper) buildArrayMap(currentNode *telemetrytypes.JSONAccessNode, branchType telemetrytypes.JSONAccessBranchType) (string, error) {
func (m *storage) buildArrayMap(currentNode *telemetrytypes.JSONAccessNode, branchType telemetrytypes.JSONAccessBranchType) (string, error) {
if currentNode == nil {
return "", errors.NewInternalf(CodeCurrentNodeNil, "current node is nil while building arrayMap")
}
@@ -553,90 +367,6 @@ func (m *fieldMapper) buildArrayMap(currentNode *telemetrytypes.JSONAccessNode,
return fmt.Sprintf("arrayMap(%s->%s, %s)", currentNode.Alias(), nestedExpr, arrayExpr), nil
}
func (m *fieldMapper) membershipGuarded(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey) bool {
if key.FieldContext == telemetrytypes.FieldContextBody {
return true
}
columns, err := m.getColumn(ctx, orgID, key)
if err != nil {
return false
}
newColumns, _, err := qbtypes.SelectEvolutionsForColumns(columns, key.Evolutions, tsStart, tsEnd)
if err != nil || len(newColumns) != 1 {
return false
}
columnType := newColumns[0].Type.GetType()
return columnType == schema.ColumnTypeEnumMap || columnType == schema.ColumnTypeEnumJSON
}
// ExistsFor implements the per-key existence primitive of qbtypes.FieldMapper.
func (m *fieldMapper) ExistsFor(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
exists bool,
) (string, error) {
columns, err := m.getColumn(ctx, orgID, key)
if errors.Is(err, qbtypes.ErrColumnNotFound) && key.FieldContext == telemetrytypes.FieldContextUnspecified {
bodyKey := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextBody, key.FieldDataType)
columns, err = m.getColumn(ctx, orgID, bodyKey)
}
if err != nil {
return "", err
}
operator := qbtypes.FilterOperatorExists
if !exists {
operator = qbtypes.FilterOperatorNotExists
}
bodyJSONEnabled := m.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
for _, column := range columns {
if column.Type.GetType() == schema.ColumnTypeEnumJSON && isBodyJSONSearch(key, columns) && bodyJSONEnabled && key.Name != messageSubField {
valueType, value := InferDataType(nil, operator, key)
if len(key.JSONPlan) == 0 {
keyCopy := telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, key.FieldDataType)
if err := keyCopy.SetExhaustiveJSONAccessPlan(
telemetrytypes.JSONColumnMetadata{BaseColumn: LogsV2BodyV2Column}, valueType,
); err != nil {
return "", err
}
key = keyCopy
}
sb := sqlbuilder.NewSelectBuilder()
cond, err := NewJSONConditionBuilder(key, valueType).buildJSONCondition(operator, value, sb)
if err != nil {
return "", err
}
sb.Where(cond)
expr, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
expr = strings.TrimPrefix(expr, "WHERE ")
if len(args) > 0 {
expr, err = sqlbuilder.ClickHouse.Interpolate(expr, args)
if err != nil {
return "", err
}
}
return expr, nil
}
}
if isBodyJSONSearch(key, columns) && !bodyJSONEnabled {
if exists {
return GetBodyJSONKeyForExists(ctx, key, operator, nil), nil
}
return "NOT " + GetBodyJSONKeyForExists(ctx, key, operator, nil), nil
}
fieldExpression, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
return "", err
}
return querybuilder.ExistsExpression(columns, key, tsStart, tsEnd, fieldExpression, exists)
}
// searchColumns is the single source of truth for the columns search() fans out across,
// by context; body is body_v2 when useJSONBody, else the legacy body string.
func searchColumns(fieldContext telemetrytypes.FieldContext, useJSONBody bool) []*schema.Column {
@@ -670,3 +400,159 @@ func searchColumns(fieldContext telemetrytypes.FieldContext, useJSONBody bool) [
return columns
}
}
func (m *storage) Exists(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, exists bool) (qbtypes.Existence, error) {
columns, err := m.getColumn(q, key)
if errors.Is(err, qbtypes.ErrColumnNotFound) && key.FieldContext == telemetrytypes.FieldContextUnspecified {
bodyKey := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextBody, key.FieldDataType)
columns, err = m.getColumn(q, bodyKey)
}
if err != nil {
return qbtypes.Existence{}, err
}
operator := qbtypes.FilterOperatorExists
if !exists {
operator = qbtypes.FilterOperatorNotExists
}
for _, column := range columns {
if column.Type.GetType() == schema.ColumnTypeEnumJSON && isBodyJSONSearch(key, columns) && q.BodyJSONOn && key.Name != messageSubField {
valueType, value := InferDataType(nil, operator, key)
if len(key.JSONPlan) == 0 {
keyCopy := telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, key.FieldDataType)
if err := keyCopy.SetExhaustiveJSONAccessPlan(
telemetrytypes.JSONColumnMetadata{BaseColumn: LogsV2BodyV2Column}, valueType,
); err != nil {
return qbtypes.Existence{}, err
}
key = keyCopy
}
sb := sqlbuilder.NewSelectBuilder()
cond, err := NewJSONConditionBuilder(key, valueType).buildJSONCondition(operator, value, sb)
if err != nil {
return qbtypes.Existence{}, err
}
sb.Where(cond)
expr, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
expr = strings.TrimPrefix(expr, "WHERE ")
if len(args) > 0 {
expr, err = sqlbuilder.ClickHouse.Interpolate(expr, args)
if err != nil {
return qbtypes.Existence{}, err
}
}
// a JSON path reads NULL when the row lacks it
return qbtypes.Existence{Predicate: expr, WhenAbsent: qbtypes.AbsentIsNull}, nil
}
}
if isBodyJSONSearch(key, columns) && !q.BodyJSONOn {
predicate := GetBodyJSONKeyForExists(ctx, key, operator, nil)
if !exists {
predicate = "NOT " + predicate
}
return qbtypes.Existence{Predicate: predicate, WhenAbsent: qbtypes.AbsentIsSentinel}, nil
}
fieldExpression, err := m.Read(ctx, q, key)
if err != nil {
return qbtypes.Existence{}, err
}
predicate, err := querybuilder.ExistsExpression(columns, key, q.StartNs, q.EndNs, fieldExpression, exists)
if err != nil {
return qbtypes.Existence{}, err
}
return qbtypes.Existence{Predicate: predicate, WhenAbsent: absentReads(q, key, columns)}, nil
}
// absentReads tells what a row without the key reads: NULL from a multi-era
// read, the empty value from a map or from a JSON path (its ::String cast
// folds NULL to the empty string), and a real value from every other column.
func absentReads(q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, columns []*schema.Column) qbtypes.Absent {
newColumns, _, err := qbtypes.SelectEvolutionsForColumns(columns, key.Evolutions, q.StartNs, q.EndNs)
if err != nil || len(newColumns) != 1 {
return qbtypes.AbsentIsNull
}
switch newColumns[0].Type.GetType() {
case schema.ColumnTypeEnumMap, schema.ColumnTypeEnumJSON:
return qbtypes.AbsentIsSentinel
}
return qbtypes.AlwaysPresent
}
// Fallback answers a key metadata does not report: a column name is the
// column, a body path is itself, and any other name synthesizes its type
// variants under the stripped and the literal spelling, with the body path
// last. A body function narrows the answer to body paths.
func (m *storage) Fallback(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, operator qbtypes.FilterOperator, value any) ([]*telemetrytypes.LogicalField, error) {
if key.FieldContext == telemetrytypes.FieldContextBody && key.Name == "" {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "missing key for body json search - expected key of the form `body.key` (ex: `body.status`)")
}
var keys []*telemetrytypes.TelemetryFieldKey
switch key.FieldContext {
case telemetrytypes.FieldContextBody:
keys = []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, key.FieldDataType)}
case telemetrytypes.FieldContextUnspecified:
if _, ok := logsV2Columns[key.Name]; ok {
keys = []*telemetrytypes.TelemetryFieldKey{{Name: key.Name, FieldContext: telemetrytypes.FieldContextLog}}
break
}
keys = append(querybuilder.SynthesizeKeys(key, value), bodyPath(key.Name, key.FieldDataType, value))
case telemetrytypes.FieldContextAttribute, telemetrytypes.FieldContextResource:
// a context can be a legitimate prefix in user data
literal := telemetrytypes.NewTelemetryFieldKey(key.FieldContext.StringValue()+"."+key.Name, key.FieldContext, key.FieldDataType)
keys = append(querybuilder.SynthesizeKeys(key, value), querybuilder.SynthesizeKeys(literal, value)...)
case telemetrytypes.FieldContextLog:
if _, ok := logsV2Columns[key.Name]; ok {
keys = []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, key.FieldDataType)}
break
}
stripped := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextUnspecified, key.FieldDataType)
literal := telemetrytypes.NewTelemetryFieldKey(key.FieldContext.StringValue()+"."+key.Name, telemetrytypes.FieldContextUnspecified, key.FieldDataType)
keys = append(querybuilder.SynthesizeKeys(stripped, value), querybuilder.SynthesizeKeys(literal, value)...)
for _, spelling := range []*telemetrytypes.TelemetryFieldKey{stripped, literal} {
keys = append(keys, bodyPath(spelling.Name, spelling.FieldDataType, value))
}
}
if operator.IsFunctionOperator() {
if key.FieldContext != telemetrytypes.FieldContextBody {
return nil, querybuilder.NewFunctionUnsupportedError(operator)
}
bodyKeys := make([]*telemetrytypes.TelemetryFieldKey, 0, len(keys))
for _, k := range keys {
if k.FieldContext == telemetrytypes.FieldContextBody {
bodyKeys = append(bodyKeys, k)
}
}
keys = bodyKeys
}
return querybuilder.WrapAsLogicalFields(key.Name, keys), nil
}
// bodyPath is the body JSON path for a name; without an operand to type it,
// a string path.
func bodyPath(name string, dataType telemetrytypes.FieldDataType, value any) *telemetrytypes.TelemetryFieldKey {
bodyKey := telemetrytypes.NewTelemetryFieldKey(name, telemetrytypes.FieldContextBody, dataType)
if value == nil && bodyKey.FieldDataType == telemetrytypes.FieldDataTypeUnspecified {
bodyKey.FieldDataType = telemetrytypes.FieldDataTypeString
}
return bodyKey
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{
Split: qbtypes.MainOfSplit,
SupportsBodyFunctions: true,
OwnContexts: []telemetrytypes.FieldContext{telemetrytypes.FieldContextLog},
}
}
// ColumnRead answers the legacy body column as not selectable: without the
// JSON body, a body path has no column read.
func (m *storage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
if logical.FieldContext == telemetrytypes.FieldContextBody && !q.BodyJSONOn {
return qbtypes.ColumnExpr{}, qbtypes.ErrNotSelectable
}
return querybuilder.DefaultRead(ctx, q, m, logical)
}

View File

@@ -2,6 +2,7 @@ package logstelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
"time"
@@ -15,7 +16,6 @@ import (
)
func TestGetColumn(t *testing.T) {
ctx := context.Background()
testCases := []struct {
name string
@@ -168,11 +168,11 @@ func TestGetColumn(t *testing.T) {
}
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
storage := NewStorage()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
col, err := fm.ColumnFor(ctx, valuer.UUID{}, 0, 0, &tc.key)
col, err := storage.getColumn(querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
@@ -277,8 +277,8 @@ func TestGetFieldKeyName(t *testing.T) {
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
result, err := fm.FieldFor(ctx, valuer.UUID{}, 0, 0, &tc.key)
storage := NewStorage()
result, err := storage.Read(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
@@ -519,13 +519,13 @@ func TestFieldForWithEvolutions(t *testing.T) {
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
storage := NewStorage()
tsStart := uint64(tc.tsStartTime.UnixNano())
tsEnd := uint64(tc.tsEndTime.UnixNano())
tc.key.Evolutions = tc.evolutions
result, err := fm.FieldFor(ctx, valuer.UUID{}, tsStart, tsEnd, tc.key)
result, err := storage.Read(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, tsStart, tsEnd), tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
@@ -585,13 +585,13 @@ func TestFieldForWithMaterialized(t *testing.T) {
}
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
storage := NewStorage()
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
start := uint64(tc.start.UnixNano())
end := uint64(tc.end.UnixNano())
result, err := fm.FieldFor(ctx, valuer.UUID{}, start, end, materializedKey)
result, err := storage.Read(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, start, end), materializedKey)
require.NoError(t, err)
assert.Equal(t, tc.expectedResult, result)
})

View File

@@ -2,6 +2,8 @@ package logstelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"testing"
"time"
@@ -15,19 +17,17 @@ import (
func TestLikeAndILikeWithoutWildcards_Warns(t *testing.T) {
fl := flaggertest.New(t)
ctx := context.Background()
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
storage := NewStorage()
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
Context: ctx,
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0),
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
}
tests := []string{
@@ -53,19 +53,17 @@ func TestLikeAndILikeWithoutWildcards_Warns(t *testing.T) {
// TestLikeAndILikeWithWildcards_NoWarn Tests that LIKE/ILIKE with wildcards do not add warnings.
func TestLikeAndILikeWithWildcards_NoWarn(t *testing.T) {
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
storage := NewStorage()
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn}
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0),
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn}
tests := []string{
"service.name LIKE 'demo-%'",

View File

@@ -3,6 +3,7 @@ package logstelemetryschema
import (
"context"
"fmt"
"github.com/SigNoz/signoz/pkg/valuer"
"testing"
"time"
@@ -17,19 +18,17 @@ import (
// TestFilterExprLogsBodyJSON tests a comprehensive set of query patterns for body JSON search.
func TestFilterExprLogsBodyJSON(t *testing.T) {
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
storage := NewStorage()
// Define a comprehensive set of field keys to support all test cases
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "body"},
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0),
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "body"},
}
testCases := []struct {

View File

@@ -3,6 +3,7 @@ package logstelemetryschema
import (
"context"
"fmt"
"github.com/SigNoz/signoz/pkg/valuer"
"strings"
"testing"
"time"
@@ -37,21 +38,17 @@ func TestFilterExprLogs(t *testing.T) {
fl := flaggertest.New(t)
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
ctx := context.Background()
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
storage := NewStorage()
// Define a comprehensive set of field keys to support all test cases
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
StartNs: uint64(releaseTime.Add(-5 * time.Minute).UnixNano()),
EndNs: uint64(releaseTime.Add(5 * time.Minute).UnixNano()),
Context: ctx,
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, uint64(releaseTime.Add(-5*time.Minute).UnixNano()), uint64(releaseTime.Add(5*time.Minute).UnixNano())),
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
}
testCases := []struct {
@@ -485,7 +482,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "FREETEXT with conditions",
query: "error service.name=authentication",
shouldPass: true,
expectedQuery: "WHERE (match(LOWER(body), LOWER(?)) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedQuery: "WHERE (match(LOWER(body), LOWER(?)) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{"error", "authentication"},
expectedErrorContains: "",
},
@@ -843,7 +840,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Basic equality",
query: "service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedQuery: "WHERE multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -1203,7 +1200,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "IN operator (parentheses)",
query: "service.name IN (\"api\", \"web\", \"auth\")",
shouldPass: true,
expectedQuery: "WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{"api", "web", "auth"},
expectedErrorContains: "",
},
@@ -1211,7 +1208,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "IN operator (parentheses)",
query: "environment IN (\"dev\", \"test\", \"staging\", \"prod\")",
shouldPass: true,
expectedQuery: "WHERE ((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ?) AND multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) IS NOT NULL)",
expectedQuery: "WHERE (multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ?)",
expectedArgs: []any{"dev", "test", "staging", "prod"},
expectedErrorContains: "",
},
@@ -1237,7 +1234,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "IN operator (brackets)",
query: "service.name IN [\"api\", \"web\", \"auth\"]",
shouldPass: true,
expectedQuery: "WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{"api", "web", "auth"},
expectedErrorContains: "",
},
@@ -1245,7 +1242,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "IN operator (brackets)",
query: "environment IN [\"dev\", \"test\", \"staging\", \"prod\"]",
shouldPass: true,
expectedQuery: "WHERE ((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ?) AND multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) IS NOT NULL)",
expectedQuery: "WHERE (multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ?)",
expectedArgs: []any{"dev", "test", "staging", "prod"},
expectedErrorContains: "",
},
@@ -1609,7 +1606,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Explicit AND",
query: "status=200 AND service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{float64(200), "api"},
expectedErrorContains: "",
},
@@ -1643,7 +1640,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Explicit OR",
query: "service.name=\"api\" OR service.name=\"web\"",
shouldPass: true,
expectedQuery: "WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{"api", "web"},
expectedErrorContains: "",
},
@@ -1669,7 +1666,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "NOT with expressions",
query: "NOT service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE NOT ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedQuery: "WHERE NOT (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -1687,7 +1684,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "AND + OR combinations",
query: "status=200 AND (service.name=\"api\" OR service.name=\"web\")",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)))",
expectedArgs: []any{float64(200), "api", "web"},
expectedErrorContains: "",
},
@@ -1713,7 +1710,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "AND + NOT combinations",
query: "status=200 AND NOT service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND NOT ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND NOT (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))",
expectedArgs: []any{float64(200), "api"},
expectedErrorContains: "",
},
@@ -1731,7 +1728,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "OR + NOT combinations",
query: "NOT status=200 OR NOT service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) OR NOT ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))",
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) OR NOT (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))",
expectedArgs: []any{float64(200), "api"},
expectedErrorContains: "",
},
@@ -1749,7 +1746,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "AND + OR + NOT combinations",
query: "status=200 AND (service.name=\"api\" OR NOT duration>1000)",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR NOT ((toFloat64(attributes_number['duration']) > ? AND mapContains(attributes_number, 'duration'))))))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR NOT ((toFloat64(attributes_number['duration']) > ? AND mapContains(attributes_number, 'duration'))))))",
expectedArgs: []any{float64(200), "api", float64(1000)},
expectedErrorContains: "",
},
@@ -1765,7 +1762,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "AND + OR + NOT combinations",
query: "NOT (status=200 AND service.name=\"api\") OR count>0",
shouldPass: true,
expectedQuery: "WHERE (NOT ((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))) OR (toFloat64(attributes_number['count']) > ? AND mapContains(attributes_number, 'count')))",
expectedQuery: "WHERE (NOT ((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))) OR (toFloat64(attributes_number['count']) > ? AND mapContains(attributes_number, 'count')))",
expectedArgs: []any{float64(200), "api", float64(0)},
expectedErrorContains: "",
},
@@ -1775,7 +1772,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Implicit AND",
query: "status=200 service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{float64(200), "api"},
expectedErrorContains: "",
},
@@ -1801,7 +1798,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Mixed implicit/explicit AND",
query: "status=200 AND service.name=\"api\" duration<1000",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) AND (toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND (toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')))",
expectedArgs: []any{float64(200), "api", float64(1000)},
expectedErrorContains: "",
},
@@ -1827,7 +1824,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Simple grouping",
query: "service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedQuery: "WHERE multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -1853,7 +1850,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Nested grouping",
query: "(((service.name=\"api\")))",
shouldPass: true,
expectedQuery: "WHERE ((((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))))",
expectedQuery: "WHERE (((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)))",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -1871,7 +1868,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Complex nested grouping",
query: "(status=200 AND (service.name=\"api\" OR service.name=\"web\"))",
shouldPass: true,
expectedQuery: "WHERE (((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))))",
expectedQuery: "WHERE (((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))))",
expectedArgs: []any{float64(200), "api", "web"},
expectedErrorContains: "",
},
@@ -1897,7 +1894,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Deep nesting",
query: "(((status=200 OR status=201) AND service.name=\"api\") OR ((status=202 OR status=203) AND service.name=\"web\"))",
shouldPass: true,
expectedQuery: "WHERE (((((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')))) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))) OR (((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')))) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))))",
expectedQuery: "WHERE (((((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')))) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)) OR (((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')))) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))))",
expectedArgs: []any{float64(200), float64(201), "api", float64(202), float64(203), "web"},
expectedErrorContains: "",
},
@@ -1905,7 +1902,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Deep nesting",
query: "(count>0 AND ((duration<1000 AND service.name=\"api\") OR (duration<500 AND service.name=\"web\")))",
shouldPass: true,
expectedQuery: "WHERE (((toFloat64(attributes_number['count']) > ? AND mapContains(attributes_number, 'count')) AND (((((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))) OR (((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))))))",
expectedQuery: "WHERE (((toFloat64(attributes_number['count']) > ? AND mapContains(attributes_number, 'count')) AND (((((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)) OR (((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))))))",
expectedArgs: []any{float64(0), float64(1000), "api", float64(500), "web"},
expectedErrorContains: "",
},
@@ -1915,7 +1912,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "String quote styles",
query: "service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedQuery: "WHERE multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -1923,7 +1920,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "String quote styles",
query: "service.name='api'",
shouldPass: true,
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedQuery: "WHERE multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -2083,28 +2080,28 @@ func TestFilterExprLogs(t *testing.T) {
category: "Operator precedence",
query: "NOT status=200 AND service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{float64(200), "api"}, // Should be (NOT status=200) AND service.name="api"
},
{
category: "Operator precedence",
query: "status=200 AND service.name=\"api\" OR service.name=\"web\"",
shouldPass: true,
expectedQuery: "WHERE (((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedQuery: "WHERE (((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{float64(200), "api", "web"}, // Should be (status=200 AND service.name="api") OR service.name="web"
},
{
category: "Operator precedence",
query: "NOT status=200 OR NOT service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) OR NOT ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))",
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) OR NOT (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))",
expectedArgs: []any{float64(200), "api"}, // Should be (NOT status=200) OR (NOT service.name="api")
},
{
category: "Operator precedence",
query: "status=200 OR service.name=\"api\" AND level=\"ERROR\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) AND (attributes_string['level'] = ? AND mapContains(attributes_string, 'level'))))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND (attributes_string['level'] = ? AND mapContains(attributes_string, 'level'))))",
expectedArgs: []any{float64(200), "api", "ERROR"}, // Should be status=200 OR (service.name="api" AND level="ERROR")
},
@@ -2129,7 +2126,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Whitespace patterns",
query: "status=200 AND service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{float64(200), "api"}, // Multiple spaces
},
@@ -2299,7 +2296,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "More common filters",
query: "service.name=\"api\" AND (status>=500 OR duration>1000) AND NOT message CONTAINS \"expected\"",
shouldPass: true,
expectedQuery: "WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) AND (((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['duration']) > ? AND mapContains(attributes_number, 'duration')))) AND NOT ((LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message'))))",
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND (((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['duration']) > ? AND mapContains(attributes_number, 'duration')))) AND NOT ((LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message'))))",
expectedArgs: []any{"api", float64(500), float64(1000), "%expected%"},
},
@@ -2365,7 +2362,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Unusual whitespace",
query: "status = 200 AND service.name = \"api\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedArgs: []any{float64(200), "api"},
},
{
@@ -2426,7 +2423,7 @@ func TestFilterExprLogs(t *testing.T) {
)
`,
shouldPass: true,
expectedQuery: "WHERE ((((((((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) AND (toFloat64(attributes_number['status']) < ? AND mapContains(attributes_number, 'status')))) OR (((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) AND (toFloat64(attributes_number['status']) < ? AND mapContains(attributes_number, 'status')) AND NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))))))) AND ((((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (((multiIf(resource.`service.type` IS NOT NULL, resource.`service.type`::String, mapContains(resources_string, 'service.type'), resources_string['service.type'], NULL) = ? AND multiIf(resource.`service.type` IS NOT NULL, resource.`service.type`::String, mapContains(resources_string, 'service.type'), resources_string['service.type'], NULL) IS NOT NULL) AND NOT ((multiIf(resource.`service.deprecated` IS NOT NULL, resource.`service.deprecated`::String, mapContains(resources_string, 'service.deprecated'), resources_string['service.deprecated'], NULL) = ? AND multiIf(resource.`service.deprecated` IS NOT NULL, resource.`service.deprecated`::String, mapContains(resources_string, 'service.deprecated'), resources_string['service.deprecated'], NULL) IS NOT NULL)))))))) AND (((((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) OR ((toFloat64(attributes_number['duration']) BETWEEN ? AND ? AND mapContains(attributes_number, 'duration'))))) AND ((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) <> ? OR (((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? AND multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) IS NOT NULL) AND (attributes_bool['is_automated_test'] = ? AND mapContains(attributes_bool, 'is_automated_test')))))))) AND NOT ((((((LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message')) OR (LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message')))) AND (attributes_string['severity'] = ? AND mapContains(attributes_string, 'severity'))))))",
expectedQuery: "WHERE ((((((((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) AND (toFloat64(attributes_number['status']) < ? AND mapContains(attributes_number, 'status')))) OR (((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) AND (toFloat64(attributes_number['status']) < ? AND mapContains(attributes_number, 'status')) AND NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))))))) AND (((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) OR ((multiIf(resource.`service.type` IS NOT NULL, resource.`service.type`::String, mapContains(resources_string, 'service.type'), resources_string['service.type'], NULL) = ? AND NOT (multiIf(resource.`service.deprecated` IS NOT NULL, resource.`service.deprecated`::String, mapContains(resources_string, 'service.deprecated'), resources_string['service.deprecated'], NULL) = ?))))))) AND (((((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) OR ((toFloat64(attributes_number['duration']) BETWEEN ? AND ? AND mapContains(attributes_number, 'duration'))))) AND ((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) <> ? OR ((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? AND (attributes_bool['is_automated_test'] = ? AND mapContains(attributes_bool, 'is_automated_test')))))))) AND NOT ((((((LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message')) OR (LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message')))) AND (attributes_string['severity'] = ? AND mapContains(attributes_string, 'severity'))))))",
expectedArgs: []any{
float64(200), float64(300), float64(400), float64(500), float64(404),
"api", "web", "auth",
@@ -2471,8 +2468,7 @@ func TestFilterExprLogs(t *testing.T) {
// TestFilterExprLogs tests a comprehensive set of query patterns for logs search.
func TestFilterExprLogsConflictNegation(t *testing.T) {
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
storage := NewStorage()
// Define a comprehensive set of field keys to support all test cases
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
@@ -2492,12 +2488,11 @@ func TestFilterExprLogsConflictNegation(t *testing.T) {
}
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0),
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
}
testCases := []struct {

View File

@@ -2,6 +2,7 @@ package logstelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/valuer"
"testing"
"time"
@@ -50,8 +51,8 @@ func TestFilterExprSearch(t *testing.T) {
logScope := "(match(LOWER(severity_text), LOWER(?)) OR match(LOWER(trace_id), LOWER(?)) OR match(LOWER(span_id), LOWER(?)))"
resourceScope := "(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(resources_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(resources_string)))"
serviceNameEq := "(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? " +
"AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)"
// the read spans both eras and yields NULL for an absent key, so it takes no presence guard
serviceNameEq := "multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?"
testCases := []struct {
name string
@@ -243,19 +244,15 @@ func TestFilterExprSearch(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureUseJSONBody.String(): tc.jsonBodyEnabled,
})
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
storage := NewStorage()
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: tc.fullTextColumn,
StartNs: tc.startNs,
EndNs: tc.endNs,
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, tc.startNs, tc.endNs),
FieldKeys: keys,
FullTextColumn: tc.fullTextColumn,
}
clause, err := querybuilder.PrepareWhereClause(tc.query, opts)
@@ -282,3 +279,54 @@ func TestFilterExprSearch(t *testing.T) {
})
}
}
// A search scope is a field the resource fingerprint sub-query cannot serve,
// so the main query keeps it when the split runs.
func TestFilterExprSearchResourceScopeUnderSplit(t *testing.T) {
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
resourceScope := "(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(resources_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(resources_string)))"
legacyBody := "match(LOWER(body), LOWER(?))"
testCases := []struct {
name string
query string
expectedQuery string
expectedArgs []any
}{
{
name: "resource scope alone",
query: "search('checkout', resource)",
expectedQuery: "WHERE (" + resourceScope + ")",
expectedArgs: []any{"checkout", "checkout"},
},
{
name: "resource scope in a union",
query: "search('checkout', body, resource)",
expectedQuery: "WHERE ((" + legacyBody + ") OR (" + resourceScope + "))",
expectedArgs: []any{"checkout", "checkout", "checkout"},
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureUseJSONBody.String(): false})
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: NewStorage(),
Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, uint64(releaseTime.Add(-5*time.Minute).UnixNano()), uint64(releaseTime.Add(5*time.Minute).UnixNano())),
FieldKeys: BuildCompleteFieldKeyMap(releaseTime),
FullTextColumn: DefaultFullTextColumn,
SkipResourceFilter: true,
}
clause, err := querybuilder.PrepareWhereClause(tc.query, opts)
require.NoError(t, err)
require.False(t, clause.IsEmpty())
sql, args := clause.WhereClause.BuildWithFlavor(sqlbuilder.ClickHouse)
require.Equal(t, tc.expectedQuery, sql)
require.Equal(t, tc.expectedArgs, args)
})
}
}

View File

@@ -69,7 +69,7 @@ func TestExhaustiveJSONPlan_ConditionBuilder(t *testing.T) {
}
func TestExhaustiveJSONPlan_FieldMapper(t *testing.T) {
m := &fieldMapper{}
m := &storage{}
key := &telemetrytypes.TelemetryFieldKey{
Name: "education[].name",

View File

@@ -3,26 +3,15 @@ package metricstelemetryschema
import (
"context"
"fmt"
"slices"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
type conditionBuilder struct {
fm qbtypes.FieldMapper
}
func NewConditionBuilder(fm qbtypes.FieldMapper) *conditionBuilder {
return &conditionBuilder{fm: fm}
}
// Labels read back as String from the `labels` JSON whatever type the metadata claims, so the
// collision is always String vs the literal; intrinsic columns keep their own type.
func resolveTypeCollisionForFieldName(fieldExpression string, value any) string {
@@ -45,178 +34,61 @@ func resolveTypeCollisionForFieldName(fieldExpression string, value any) string
return fieldExpression
}
func (c *conditionBuilder) conditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
// Compile casts by the operand: labels read back as String from the labels
// JSON whatever type the metadata claims, and intrinsic columns keep their
// own type. The operator switch is the shared one. No label takes a guard:
// an absent label reads the empty string, and that is the keyless contract.
func (s *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
condition, err := s.condition(ctx, q, logical, operator, value, sb)
if err != nil {
return qbtypes.Compiled{}, err
}
return qbtypes.Compiled{Condition: condition}, nil
}
func (s *storage) condition(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (string, error) {
switch operator {
case qbtypes.FilterOperatorIn, qbtypes.FilterOperatorNotIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
itemOperator := qbtypes.FilterOperatorEqual
if operator == qbtypes.FilterOperatorNotIn {
itemOperator = qbtypes.FilterOperatorNotEqual
}
conditions := make([]string, 0, len(values))
for _, item := range values {
condition, err := s.condition(ctx, q, logical, itemOperator, item, sb)
if err != nil {
return "", err
}
conditions = append(conditions, condition)
}
// `=`+OR and `!=`+AND instead of IN and NOT IN, to make use of the index
if operator == qbtypes.FilterOperatorIn {
return sb.Or(conditions...), nil
}
return sb.And(conditions...), nil
}
if operator.IsStringSearchOperator() {
value = querybuilder.FormatValueForContains(value)
}
fieldExpression, err := c.fm.FieldFor(ctx, orgID, startNs, endNs, key)
read, err := querybuilder.LogicalValueExpr(ctx, q, s, logical)
if err != nil {
return "", err
}
// TODO(srikanthccv): use querybuilder.DataTypeCollisionHandledFieldName when metrics schemas are updated
fieldExpression = resolveTypeCollisionForFieldName(fieldExpression, value)
switch operator {
case qbtypes.FilterOperatorEqual:
return sb.E(fieldExpression, value), nil
case qbtypes.FilterOperatorNotEqual:
return sb.NE(fieldExpression, value), nil
case qbtypes.FilterOperatorGreaterThan:
return sb.G(fieldExpression, value), nil
case qbtypes.FilterOperatorGreaterThanOrEq:
return sb.GE(fieldExpression, value), nil
case qbtypes.FilterOperatorLessThan:
return sb.LT(fieldExpression, value), nil
case qbtypes.FilterOperatorLessThanOrEq:
return sb.LE(fieldExpression, value), nil
// like and not like
case qbtypes.FilterOperatorLike:
return sb.Like(fieldExpression, value), nil
case qbtypes.FilterOperatorNotLike:
return sb.NotLike(fieldExpression, value), nil
case qbtypes.FilterOperatorILike:
return sb.ILike(fieldExpression, value), nil
case qbtypes.FilterOperatorNotILike:
return sb.NotILike(fieldExpression, value), nil
case qbtypes.FilterOperatorContains:
return sb.ILike(fieldExpression, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorNotContains:
return sb.NotILike(fieldExpression, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorRegexp:
// Note: Escape $$ to $$$$ to avoid sqlbuilder interpreting materialized $ signs
// Only needed because we are using sprintf instead of sb.Match (not implemented in sqlbuilder)
return fmt.Sprintf(`match(%s, %s)`, sqlbuilder.Escape(fieldExpression), sb.Var(value)), nil
case qbtypes.FilterOperatorNotRegexp:
// Note: Escape $$ to $$$$ to avoid sqlbuilder interpreting materialized $ signs
// Only needed because we are using sprintf instead of sb.Match (not implemented in sqlbuilder)
return fmt.Sprintf(`NOT match(%s, %s)`, sqlbuilder.Escape(fieldExpression), sb.Var(value)), nil
// between and not between
case qbtypes.FilterOperatorBetween:
case qbtypes.FilterOperatorBetween, qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrBetweenValues
}
if len(values) != 2 {
if !ok || len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
// both bounds share one expression, so the lower bound picks the cast
fieldExpression = resolveTypeCollisionForFieldName(fieldExpression, values[0])
return sb.Between(fieldExpression, values[0], values[1]), nil
case qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrBetweenValues
}
if len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
fieldExpression = resolveTypeCollisionForFieldName(fieldExpression, values[0])
return sb.NotBetween(fieldExpression, values[0], values[1]), nil
// in and not in
case qbtypes.FilterOperatorIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
// instead of using IN, we use `=` + `OR` to make use of index
conditions := []string{}
for _, item := range values {
conditions = append(conditions, sb.E(resolveTypeCollisionForFieldName(fieldExpression, item), item))
}
return sb.Or(conditions...), nil
case qbtypes.FilterOperatorNotIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
// instead of using NOT IN, we use `!=` + `AND` to make use of index
conditions := []string{}
for _, item := range values {
conditions = append(conditions, sb.NE(resolveTypeCollisionForFieldName(fieldExpression, item), item))
}
return sb.And(conditions...), nil
// exists and not exists
// in the UI based query builder, `exists` and `not exists` are used for
// key membership checks, so depending on the column type, the condition changes
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
// if the field is intrinsic, it always exists
if slices.Contains(IntrinsicFields, key.Name) {
return "true", nil
}
if operator == qbtypes.FilterOperatorExists {
return fmt.Sprintf("has(JSONExtractKeys(labels), '%s')", key.Name), nil
}
return fmt.Sprintf("not has(JSONExtractKeys(labels), '%s')", key.Name), nil
read = resolveTypeCollisionForFieldName(read, values[0])
default:
read = resolveTypeCollisionForFieldName(read, value)
}
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported operator: %v", operator)
}
// Metrics has no resource sub-query, so options are unused.
func (c *conditionBuilder) ConditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for metrics.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}
// Metric labels have no family support, so every logical field is
// single-member and flattens losslessly to its physical key.
keys := querybuilder.SingleKeys(querybuilder.MatchingLogicalFields(ctx, orgID, nil, key, fieldKeys))
var warnings []string
if len(keys) == 0 {
if _, isColumn := timeSeriesV4Columns[key.Name]; isColumn {
keys = []*telemetrytypes.TelemetryFieldKey{key}
} else {
if len(fieldKeys[key.Name]) == 0 {
warnings = append(warnings, fmt.Sprintf("label `%s` not found in metadata; check the label name for typos", key.Name))
}
keys = []*telemetrytypes.TelemetryFieldKey{
telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextAttribute, key.FieldDataType),
}
if key.FieldContext != telemetrytypes.FieldContextUnspecified {
keys = append(keys, telemetrytypes.NewTelemetryFieldKey(
key.FieldContext.StringValue()+"."+key.Name, telemetrytypes.FieldContextAttribute, key.FieldDataType))
}
}
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
cond, err := c.conditionFor(ctx, orgID, startNs, endNs, k, operator, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
}
return conds, warnings, nil
return querybuilder.OperatorCondition(ctx, q, s, logical, read, operator, value, sb)
}

View File

@@ -2,11 +2,11 @@ package metricstelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -343,13 +343,12 @@ func TestConditionFor(t *testing.T) {
},
}
fm := NewFieldMapper()
conditionBuilder := NewConditionBuilder(fm)
storage := NewStorage()
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, 0, 0, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
cond, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &tc.key, tc.operator, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, false, sb)
sb.Where(cond...)
if tc.expectedError != nil {
@@ -396,15 +395,14 @@ func TestConditionForMultipleKeys(t *testing.T) {
},
}
fm := NewFieldMapper()
conditionBuilder := NewConditionBuilder(fm)
storage := NewStorage()
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
var err error
for _, key := range tc.keys {
cond, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {&key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
cond, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, tc.operator, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {&key}}, false, sb)
sb.Where(cond...)
if err != nil {
t.Fatalf("Error getting condition for key %s: %v", key.Name, err)
@@ -483,13 +481,12 @@ func TestConditionForKeyNotInMetadata(t *testing.T) {
},
}
fm := NewFieldMapper()
conditionBuilder := NewConditionBuilder(fm)
storage := NewStorage()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
cond, warnings, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, 0, 0, &tc.key, tc.fieldKeys, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
cond, warnings, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &tc.key, tc.operator, tc.value, tc.fieldKeys, false, sb)
require.NoError(t, err)
sb.Where(cond...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)

View File

@@ -6,9 +6,9 @@ import (
"slices"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
var (
@@ -35,19 +35,15 @@ var (
}
)
type fieldMapper struct{}
type storage struct{}
// CandidateKeys returns nil: metrics has no attribute-map fallback, so a context-missing
// key stays unresolved and the caller errors.
func (m *fieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, _ *telemetrytypes.TelemetryFieldKey, _ any, _ map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
var _ qbtypes.Storage = (*storage)(nil)
func NewStorage() *storage {
return &storage{}
}
func NewFieldMapper() qbtypes.FieldMapper {
return &fieldMapper{}
}
func (m *fieldMapper) getColumn(_ context.Context, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
func (m *storage) getColumn(_ context.Context, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
switch key.FieldContext {
case telemetrytypes.FieldContextResource, telemetrytypes.FieldContextScope, telemetrytypes.FieldContextAttribute:
@@ -71,8 +67,8 @@ func (m *fieldMapper) getColumn(_ context.Context, _, _ uint64, key *telemetryty
return nil, qbtypes.ErrColumnNotFound
}
func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, startNs, endNs, key)
func (m *storage) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, q.StartNs, q.EndNs, key)
if err != nil {
return "", err
}
@@ -92,29 +88,57 @@ func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, startNs, endN
return columns[0].Name, nil
}
func (m *fieldMapper) ColumnFor(ctx context.Context, _ valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
return m.getColumn(ctx, tsStart, tsEnd, key)
}
// ExistsFor implements the per-key existence primitive of qbtypes.FieldMapper.
// Intrinsic fields always exist; labels are checked for key membership.
func (m *fieldMapper) ExistsFor(_ context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey, exists bool) (string, error) {
// Exists answers an intrinsic column as always present. A label is checked
// for key membership; absent, it reads the empty string, and that is the
// metrics keyless contract, so no query guards it.
func (m *storage) Exists(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, exists bool) (qbtypes.Existence, error) {
if slices.Contains(IntrinsicFields, key.Name) {
return "true", nil
return qbtypes.Existence{Predicate: "true", WhenAbsent: qbtypes.AlwaysPresent}, nil
}
if exists {
return fmt.Sprintf("has(JSONExtractKeys(labels), '%s')", key.Name), nil
predicate := fmt.Sprintf("has(JSONExtractKeys(labels), '%s')", key.Name)
if !exists {
predicate = "not " + predicate
}
return fmt.Sprintf("not has(JSONExtractKeys(labels), '%s')", key.Name), nil
return qbtypes.Existence{Predicate: predicate, WhenAbsent: qbtypes.AbsentIsValue}, nil
}
func (m *fieldMapper) ColumnExpressionFor(
ctx context.Context,
orgID valuer.UUID,
startNs, endNs uint64,
field *telemetrytypes.TelemetryFieldKey,
_ telemetrytypes.FieldDataType,
_ map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
return m.FieldFor(ctx, orgID, startNs, endNs, field)
// Fallback answers every name: a column name is the column, any other name
// is a label, and a context-qualified name is also the label spelled with
// its context, because a context can be a legitimate prefix in user data.
func (m *storage) Fallback(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, _ qbtypes.FilterOperator, _ any) ([]*telemetrytypes.LogicalField, error) {
if _, isColumn := timeSeriesV4Columns[key.Name]; isColumn {
return querybuilder.WrapAsLogicalFields(key.Name, []*telemetrytypes.TelemetryFieldKey{key}), nil
}
keys := []*telemetrytypes.TelemetryFieldKey{
telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextAttribute, key.FieldDataType),
}
if key.FieldContext != telemetrytypes.FieldContextUnspecified {
keys = append(keys, telemetrytypes.NewTelemetryFieldKey(
key.FieldContext.StringValue()+"."+key.Name, telemetrytypes.FieldContextAttribute, key.FieldDataType))
}
return querybuilder.WrapAsLogicalFields(key.Name, keys), nil
}
// Traits: every context addresses the one label bag, so a key under any
// context resolves as if it had none.
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{
OwnContexts: []telemetrytypes.FieldContext{
telemetrytypes.FieldContextResource,
telemetrytypes.FieldContextScope,
telemetrytypes.FieldContextAttribute,
telemetrytypes.FieldContextMetric,
},
}
}
// ColumnRead keeps every read in its native type: a label already reads
// back as String, and coercion adds nothing.
func (m *storage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
read, err := querybuilder.DefaultRead(ctx, q, m, logical)
if err != nil {
return qbtypes.ColumnExpr{}, err
}
read.KeepType = true
return read, nil
}

View File

@@ -7,7 +7,6 @@ import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -120,11 +119,11 @@ func TestGetColumn(t *testing.T) {
},
}
fm := NewFieldMapper()
storage := NewStorage()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
col, err := fm.ColumnFor(ctx, valuer.UUID{}, 0, 0, &tc.key)
col, err := storage.getColumn(ctx, 0, 0, &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
@@ -204,11 +203,11 @@ func TestGetFieldKeyName(t *testing.T) {
},
}
fm := NewFieldMapper()
storage := NewStorage()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
result, err := fm.FieldFor(ctx, valuer.UUID{}, 0, 0, &tc.key)
result, err := storage.Read(ctx, qbtypes.QueryInfo{}, &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)

View File

@@ -1,375 +0,0 @@
package tracestelemetryschema
import (
"context"
"fmt"
"slices"
"strconv"
"strings"
"time"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"golang.org/x/exp/maps"
)
type conditionBuilder struct {
fm qbtypes.FieldMapper
// fl evaluates the resolve_semconv_families flag during resolution.
// A nil flagger keeps resolution literal.
fl flagger.Flagger
}
var _ qbtypes.ConditionBuilder = (*conditionBuilder)(nil)
func NewConditionBuilder(fm qbtypes.FieldMapper, fl flagger.Flagger) *conditionBuilder {
return &conditionBuilder{fm: fm, fl: fl}
}
func (c *conditionBuilder) conditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
logical *telemetrytypes.LogicalField,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
if operator.IsStringSearchOperator() {
value = querybuilder.FormatValueForContains(value)
}
fieldExpression, err := querybuilder.LogicalValueExpr(ctx, orgID, startNs, endNs, c.fm, logical)
if err != nil {
return "", err
}
// TODO(srikanthccv): maybe extend this to every possible attribute
if logical.Name == "duration_nano" || logical.Name == "durationNano" { // QoL improvement
switch v := value.(type) {
case string:
if duration, err := time.ParseDuration(v); err == nil {
value = duration.Nanoseconds()
} else if f, err := strconv.ParseFloat(v, 64); err == nil {
value = int64(f)
} else {
return "", errors.WrapInvalidInputf(err, errors.CodeInvalidInput, "invalid duration value: %s", v)
}
case float64:
value = int64(v)
case float32:
value = int64(v)
}
}
// Coercion switches only on the data type, which every member shares, so
// the first member stands in for the field.
fieldExpression, value = querybuilder.DataTypeCollisionHandledFieldName(logical.Single(), value, fieldExpression, operator)
// regular operators
switch operator {
// regular operators
case qbtypes.FilterOperatorEqual:
return sb.E(fieldExpression, value), nil
case qbtypes.FilterOperatorNotEqual:
return sb.NE(fieldExpression, value), nil
case qbtypes.FilterOperatorGreaterThan:
return sb.G(fieldExpression, value), nil
case qbtypes.FilterOperatorGreaterThanOrEq:
return sb.GE(fieldExpression, value), nil
case qbtypes.FilterOperatorLessThan:
return sb.LT(fieldExpression, value), nil
case qbtypes.FilterOperatorLessThanOrEq:
return sb.LE(fieldExpression, value), nil
// like and not like
case qbtypes.FilterOperatorLike:
return sb.Like(fieldExpression, value), nil
case qbtypes.FilterOperatorNotLike:
return sb.NotLike(fieldExpression, value), nil
case qbtypes.FilterOperatorILike:
return sb.ILike(fieldExpression, value), nil
case qbtypes.FilterOperatorNotILike:
return sb.NotILike(fieldExpression, value), nil
case qbtypes.FilterOperatorContains:
return sb.ILike(fieldExpression, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorNotContains:
return sb.NotILike(fieldExpression, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorRegexp:
// Note: Escape $$ to $$$$ to avoid sqlbuilder interpreting materialized $ signs
// Only needed because we are using sprintf instead of sb.Match (not implemented in sqlbuilder)
return fmt.Sprintf(`match(%s, %s)`, sqlbuilder.Escape(fieldExpression), sb.Var(value)), nil
case qbtypes.FilterOperatorNotRegexp:
// Note: Escape $$ to $$$$ to avoid sqlbuilder interpreting materialized $ signs
// Only needed because we are using sprintf instead of sb.Match (not implemented in sqlbuilder)
return fmt.Sprintf(`NOT match(%s, %s)`, sqlbuilder.Escape(fieldExpression), sb.Var(value)), nil
// between and not between
case qbtypes.FilterOperatorBetween:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrBetweenValues
}
if len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.Between(fieldExpression, values[0], values[1]), nil
case qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrBetweenValues
}
if len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.NotBetween(fieldExpression, values[0], values[1]), nil
// in and not in
case qbtypes.FilterOperatorIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
// instead of using IN, we use `=` + `OR` to make use of index
conditions := []string{}
for _, value := range values {
cond, err := c.conditionFor(ctx, orgID, startNs, endNs, logical, qbtypes.FilterOperatorEqual, value, sb)
if err != nil {
return "", err
}
conditions = append(conditions, cond)
}
return sb.Or(conditions...), nil
case qbtypes.FilterOperatorNotIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
// instead of using NOT IN, we use `!=` + `AND` to make use of index
conditions := []string{}
for _, value := range values {
cond, err := c.conditionFor(ctx, orgID, startNs, endNs, logical, qbtypes.FilterOperatorNotEqual, value, sb)
if err != nil {
return "", err
}
conditions = append(conditions, cond)
}
return sb.And(conditions...), nil
// exists and not exists
// in the query builder, `exists` and `not exists` are used for
// key membership checks, so depending on the column type, the condition changes
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
pred, err := querybuilder.LogicalExistsExpr(ctx, orgID, startNs, endNs, c.fm, logical, operator == qbtypes.FilterOperatorExists)
if err != nil {
return "", err
}
return sqlbuilder.Escape(pred), nil
}
return "", nil
}
// isFoldContext reports whether the context is one CandidateKeys would fold the prefix into
// the key name for (span/trace). These behave like a default context that also addresses
// columns and attributes, unlike strict resource/attribute/scope contexts.
func isFoldContext(fc telemetrytypes.FieldContext) bool {
switch fc {
case telemetrytypes.FieldContextSpan, telemetrytypes.FieldContextTrace:
return true
}
return false
}
// candidateLookupKeys returns the metadata map only for fold-contexts, where CandidateKeys
// would otherwise fold the prefix into the key name. Handing it the map lets a same-named
// key under another context resolve first (as ColumnExpressionFor does). Strict contexts
// (resource/attribute/scope) get nil so their explicit context is always honored.
func candidateLookupKeys(key *telemetrytypes.TelemetryFieldKey, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) map[string][]*telemetrytypes.TelemetryFieldKey {
if isFoldContext(key.FieldContext) {
return fieldKeys
}
return nil
}
// ConditionFor resolves the referenced key to the key(s) to filter on (ResolveKeys, else
// synthesized keys with a warning) and builds one condition per resolved key. fieldKeys is
// the full metadata map; the builder owns key resolution.
func (c *conditionBuilder) ConditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
options qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for traces.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}
matches := querybuilder.MatchingLogicalFields(ctx, orgID, c.fl, key, fieldKeys)
skipResourceFilter := options.SkipResourceFilter
logicalFields, warning := querybuilder.ResolveLogicalFields(key, matches)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
// A bare key that names a real column filters on the column too — first. When metadata
// only knows the name under other contexts, prepend the column and keep metadata matches
// only where their type is consistent with it (a corrupt entry can't degrade the column).
if key.FieldContext == telemetrytypes.FieldContextUnspecified && len(logicalFields) > 0 {
hasColumn := false
for _, logical := range logicalFields {
if logical.FieldContext == telemetrytypes.FieldContextSpan {
hasColumn = true
break
}
}
if !hasColumn {
probe := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextSpan, key.FieldDataType)
if cols, colErr := c.fm.ColumnFor(ctx, orgID, startNs, endNs, probe); colErr == nil && len(cols) > 0 {
combined := make([]*telemetrytypes.LogicalField, 0, len(logicalFields)+1)
combined = append(combined, telemetrytypes.SingleLogicalField(key.Name, probe))
for _, logical := range logicalFields {
if columnMatchesDataType(cols[0], logical.FieldDataType) {
combined = append(combined, logical)
}
}
logicalFields = combined
}
}
}
synthesized := false
if len(logicalFields) == 0 {
// Not in metadata. CandidateKeys resolves it: fold contexts (span/trace) get the
// metadata map so it can honor a real column, correct to a stripped-name metadata
// match, or synthesize; strict contexts pass nil and keep their synthesize path.
logicalFields = querybuilder.WrapAsLogicalFields(key.Name, c.fm.CandidateKeys(ctx, orgID, key, value, candidateLookupKeys(key, fieldKeys)))
if len(logicalFields) == 0 {
return nil, warnings, querybuilder.NewKeyNotFoundError(key.Name)
}
synthesized = true
warnings = append(warnings, querybuilder.NewKeyNotFoundWarning(key.Name))
}
// When a resource sub-query already covers the term, drop resource fields from the main
// query. Synthesized keys are exempt: the sub-query skips keys absent from metadata.
if skipResourceFilter && !synthesized {
filtered := make([]*telemetrytypes.LogicalField, 0, len(logicalFields))
for _, logical := range logicalFields {
if logical.FieldContext != telemetrytypes.FieldContextResource {
filtered = append(filtered, logical)
}
}
if len(filtered) == 0 {
return nil, warnings, nil
}
logicalFields = filtered
}
conds := make([]string, 0, len(logicalFields))
for _, logical := range logicalFields {
cond, err := c.conditionForLogicalField(ctx, orgID, startNs, endNs, logical, operator, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
}
return conds, warnings, nil
}
func (c *conditionBuilder) conditionForLogicalField(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
logical *telemetrytypes.LogicalField,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
if c.isSpanScopeField(logical.Name) {
return c.buildSpanScopeCondition(logical.Single(), operator, value, startNs)
}
condition, err := c.conditionFor(ctx, orgID, startNs, endNs, logical, operator, value, sb)
if err != nil {
return "", err
}
if operator.AddDefaultExistsFilter() {
// skip adding exists filter for intrinsic fields
field, _ := c.fm.FieldFor(ctx, orgID, startNs, endNs, logical.Single())
if slices.Contains(maps.Keys(IntrinsicFields), field) ||
slices.Contains(maps.Keys(IntrinsicFieldsDeprecated), field) ||
slices.Contains(maps.Keys(CalculatedFields), field) ||
slices.Contains(maps.Keys(CalculatedFieldsDeprecated), field) {
return condition, nil
}
existsCondition, err := c.conditionFor(ctx, orgID, startNs, endNs, logical, qbtypes.FilterOperatorExists, nil, sb)
if err != nil {
return "", err
}
return sb.And(condition, existsCondition), nil
}
return condition, nil
}
func (c *conditionBuilder) isSpanScopeField(name string) bool {
keyName := strings.ToLower(name)
return keyName == SpanSearchScopeRoot || keyName == SpanSearchScopeEntryPoint
}
func (c *conditionBuilder) buildSpanScopeCondition(key *telemetrytypes.TelemetryFieldKey, operator qbtypes.FilterOperator, value any, startNs uint64) (string, error) {
if operator != qbtypes.FilterOperatorEqual {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "span scope field %s only supports '=' operator", key.Name)
}
var isTrue bool
switch v := value.(type) {
case bool:
isTrue = v
case string:
isTrue = strings.ToLower(v) == "true"
default:
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "span scope field %s expects boolean value, got %T", key.Name, value)
}
if !isTrue {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "span scope field %s can only be filtered with value 'true'", key.Name)
}
keyName := strings.ToLower(key.Name)
switch keyName {
case SpanSearchScopeRoot:
return "parent_span_id = ''", nil
case SpanSearchScopeEntryPoint:
if startNs > 0 { // only add time filter if it is a valid time, else do not add
startS := int64(startNs / 1_000_000_000)
// Note: Escape $$ to $$$$ to avoid sqlbuilder interpreting materialized $ signs
return sqlbuilder.Escape(fmt.Sprintf("((name, resource_string_service$$name) GLOBAL IN (SELECT DISTINCT name, serviceName from %s.%s WHERE time >= toDateTime(%d))) AND parent_span_id != ''",
DBName, TopLevelOperationsTableName, startS)), nil
}
// Note: Escape $$ to $$$$ to avoid sqlbuilder interpreting materialized $ signs
return sqlbuilder.Escape(fmt.Sprintf("((name, resource_string_service$$name) GLOBAL IN (SELECT DISTINCT name, serviceName from %s.%s)) AND parent_span_id != ''",
DBName, TopLevelOperationsTableName)), nil
default:
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid span search scope: %s", key.Name)
}
}

View File

@@ -2,13 +2,12 @@ package tracestelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
"time"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -289,13 +288,12 @@ func TestConditionFor(t *testing.T) {
},
}
fm := NewFieldMapper(flaggertest.New(t))
conditionBuilder := NewConditionBuilder(fm, flaggertest.New(t))
storage := NewStorage()
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, 1761437108000000000, 1761458708000000000, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
cond, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{StartNs: 1761437108000000000, EndNs: 1761458708000000000}, storage, &tc.key, tc.operator, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, false, sb)
sb.Where(cond...)
if tc.expectedError != nil {
@@ -376,13 +374,12 @@ func TestConditionForResourceWithEvolution(t *testing.T) {
},
}
fm := NewFieldMapper(flaggertest.New(t))
conditionBuilder := NewConditionBuilder(fm, flaggertest.New(t))
storage := NewStorage()
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, tc.tsStart, tc.tsEnd, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, nil, sb)
cond, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{StartNs: tc.tsStart, EndNs: tc.tsEnd}, storage, &tc.key, tc.operator, nil, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, false, sb)
require.NoError(t, err)
sb.Where(cond...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -399,8 +396,7 @@ func TestConditionForResourceWithEvolution(t *testing.T) {
// undeclared (Dynamic) paths and genuinely NULL when absent.
func TestConditionForScopeIntrinsicFields(t *testing.T) {
ctx := context.Background()
fm := NewFieldMapper(flaggertest.New(t))
conditionBuilder := NewConditionBuilder(fm, flaggertest.New(t))
storage := NewStorage()
testCases := []struct {
name string
@@ -464,14 +460,14 @@ func TestConditionForScopeIntrinsicFields(t *testing.T) {
},
operator: qbtypes.FilterOperatorEqual,
value: "io.signoz.checkout",
expectedSQL: "(scope.attributes.`name`::String = ? AND scope.attributes.`name` IS NOT NULL)",
expectedSQL: "scope.attributes.`name`::String = ?",
},
}
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
conds, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, 0, 0, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
conds, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &tc.key, tc.operator, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, false, sb)
require.NoError(t, err)
sb.Where(conds...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -486,8 +482,7 @@ func TestConditionForScopeIntrinsicFields(t *testing.T) {
// user input and queries anyway, emitting a warning instead of failing.
func TestConditionForSynthesizedKeys(t *testing.T) {
ctx := context.Background()
fm := NewFieldMapper(flaggertest.New(t))
cb := NewConditionBuilder(fm, flaggertest.New(t))
storage := NewStorage()
// no metadata matches -> the builder must synthesize from user input
var noMatches map[string][]*telemetrytypes.TelemetryFieldKey
@@ -495,7 +490,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("bare key with string operand -> attribute string", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "error.type"}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "timeout", sb)
conds, warnings, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, "timeout", noMatches, false, sb)
assert.NoError(t, err)
assert.NotEmpty(t, warnings, "a not-found warning should be emitted")
sb.Where(conds...)
@@ -507,7 +502,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("scope context with no metadata -> scope attribute", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.attr", FieldContext: telemetrytypes.FieldContextScope}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, "v", noMatches, false, sb)
assert.NoError(t, err, "an undeclared scope attribute must still be filterable")
assert.NotEmpty(t, warnings)
sb.Where(conds...)
@@ -521,7 +516,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("bare key with number operand -> attribute number", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "http.status"}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorGreaterThan, float64(5), sb)
conds, warnings, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorGreaterThan, float64(5), noMatches, false, sb)
assert.NoError(t, err)
assert.NotEmpty(t, warnings)
sb.Where(conds...)
@@ -532,7 +527,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("bare key with bool operand -> attribute bool", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "sampled"}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, true, sb)
conds, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, true, noMatches, false, sb)
assert.NoError(t, err)
sb.Where(conds...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -542,7 +537,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("exists with no operand fans out across type variants", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "exception.type"}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorExists, nil, sb)
conds, warnings, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorExists, nil, noMatches, false, sb)
assert.NoError(t, err)
assert.NotEmpty(t, warnings)
assert.Len(t, conds, 3, "exists should fan out to string/number/bool")
@@ -556,7 +551,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("qualified data type honored without fanout", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldDataType: telemetrytypes.FieldDataTypeString}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
conds, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, "v", noMatches, false, sb)
assert.NoError(t, err)
assert.Len(t, conds, 1)
sb.Where(conds...)
@@ -567,7 +562,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("bare intrinsic column resolves to the column, not synthesized attributes", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "duration_nano"}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorGreaterThan, float64(100), sb)
conds, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorGreaterThan, float64(100), noMatches, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -580,7 +575,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("bare deprecated alias resolves to the calculated column", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "httpMethod"}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "GET", sb)
conds, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, "GET", noMatches, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -592,7 +587,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("span-context key that is a real column stays the column", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "duration_nano", FieldContext: telemetrytypes.FieldContextSpan}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorGreaterThan, float64(100), sb)
conds, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorGreaterThan, float64(100), noMatches, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -607,9 +602,9 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
keysMap := map[string][]*telemetrytypes.TelemetryFieldKey{
"http.method": {{Name: "http.method", FieldContext: telemetrytypes.FieldContextAttribute, FieldDataType: telemetrytypes.FieldDataTypeString}},
}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, keysMap, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "GET", sb)
conds, warnings, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, "GET", keysMap, false, sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
assert.Empty(t, warnings, "the stripped name matches metadata, not a guess")
require.Len(t, conds, 1)
sb.Where(conds...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -620,7 +615,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("span-context key absent from metadata synthesizes the stripped name", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.attr", FieldContext: telemetrytypes.FieldContextSpan}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, "v", noMatches, false, sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 1)
@@ -633,7 +628,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("qualified resource context honored with literal spelling second", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "k8s.cluster.name", FieldContext: telemetrytypes.FieldContextResource}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "prod", sb)
conds, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, "prod", noMatches, false, sb)
assert.NoError(t, err)
assert.Len(t, conds, 2, "stripped interpretation first, literal `resource.` spelling second")
sb.Where(conds...)
@@ -646,7 +641,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
// the resource sub-query never covered a key absent from metadata
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "deployment.environment", FieldContext: telemetrytypes.FieldContextResource}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{SkipResourceFilter: true}, qbtypes.FilterOperatorEqual, "prod", sb)
conds, warnings, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, "prod", nil, true, sb)
assert.NoError(t, err)
assert.NotEmpty(t, warnings)
assert.Len(t, conds, 2, "the synthesized resource conditions must not be dropped")
@@ -662,7 +657,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
keysMap := map[string][]*telemetrytypes.TelemetryFieldKey{
"service.name": {{Name: "service.name", FieldContext: telemetrytypes.FieldContextResource, FieldDataType: telemetrytypes.FieldDataTypeString}},
}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, keysMap, qbtypes.ConditionBuilderOptions{SkipResourceFilter: true}, qbtypes.FilterOperatorEqual, "redis", sb)
conds, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, "redis", keysMap, true, sb)
assert.NoError(t, err)
assert.Empty(t, conds, "the resource CTE covers metadata-backed keys")
})
@@ -670,7 +665,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("synthesized resource key coerces numeric operand", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "replica.count", FieldContext: telemetrytypes.FieldContextResource}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, float64(3), sb)
conds, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorEqual, float64(3), noMatches, false, sb)
assert.NoError(t, err)
sb.Where(conds...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -681,7 +676,7 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
t.Run("negative operator builds without exists guard (matches-everything semantics)", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "error.type"}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, noMatches, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorNotEqual, "fatal", sb)
conds, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &key, qbtypes.FilterOperatorNotEqual, "fatal", noMatches, false, sb)
assert.NoError(t, err)
sb.Where(conds...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)

View File

@@ -2,6 +2,7 @@ package tracestelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
"time"
@@ -50,14 +51,12 @@ func TestConditionForFamilyMergesMembersCurrentFirst(t *testing.T) {
}},
}
fl := familyFlagOn(t)
cb := NewConditionBuilder(NewFieldMapper(fl), fl)
storage := NewStorage()
// The requested spelling is the old name; precedence must still be
// current-first.
sb := sqlbuilder.NewSelectBuilder()
conds, warnings, err := cb.ConditionFor(context.Background(), valuer.UUID{}, startNs, endNs,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "production", sb)
conds, warnings, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, startNs, endNs), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment"}, qbtypes.FilterOperatorEqual, "production", fieldKeys, false, sb)
require.NoError(t, err)
require.Empty(t, warnings, "a family is one logical field, never ambiguous with itself")
require.Len(t, conds, 1)
@@ -88,12 +87,10 @@ func TestConditionForFamilyNegativeKeepsKeylessRows(t *testing.T) {
}},
}
fl := familyFlagOn(t)
cb := NewConditionBuilder(NewFieldMapper(fl), fl)
storage := NewStorage()
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, startNs, endNs,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorNotEqual, "production", sb)
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, startNs, endNs), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorNotEqual, "production", fieldKeys, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
@@ -125,12 +122,10 @@ func TestConditionForFamilyExists(t *testing.T) {
}},
}
fl := familyFlagOn(t)
cb := NewConditionBuilder(NewFieldMapper(fl), fl)
storage := NewStorage()
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, startNs, endNs,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorNotExists, nil, sb)
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, startNs, endNs), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorNotExists, nil, fieldKeys, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
@@ -162,18 +157,16 @@ func TestConditionForFamilyOffByDefault(t *testing.T) {
}},
}
fl := flaggertest.New(t)
cb := NewConditionBuilder(NewFieldMapper(fl), fl)
storage := NewStorage()
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, startNs, endNs,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "production", sb)
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, startNs, endNs), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorEqual, "production", fieldKeys, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
sql, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
require.Equal(t, "WHERE (multiIf(resource.`deployment.environment.name` IS NOT NULL, resource.`deployment.environment.name`::String, mapContains(resources_string, 'deployment.environment.name'), resources_string['deployment.environment.name'], NULL) = ? AND multiIf(resource.`deployment.environment.name` IS NOT NULL, resource.`deployment.environment.name`::String, mapContains(resources_string, 'deployment.environment.name'), resources_string['deployment.environment.name'], NULL) IS NOT NULL)", sql)
require.Equal(t, "WHERE multiIf(resource.`deployment.environment.name` IS NOT NULL, resource.`deployment.environment.name`::String, mapContains(resources_string, 'deployment.environment.name'), resources_string['deployment.environment.name'], NULL) = ?", sql)
require.Equal(t, []any{"production"}, args)
}
@@ -200,18 +193,16 @@ func TestConditionForSingleMemberIsUnchanged(t *testing.T) {
}
delete(fieldKeys, "deployment.environment")
fl := familyFlagOn(t)
cb := NewConditionBuilder(NewFieldMapper(fl), fl)
storage := NewStorage()
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, startNs, endNs,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "production", sb)
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, startNs, endNs), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorEqual, "production", fieldKeys, false, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
sql, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
require.Equal(t, "WHERE (multiIf(resource.`deployment.environment.name` IS NOT NULL, resource.`deployment.environment.name`::String, mapContains(resources_string, 'deployment.environment.name'), resources_string['deployment.environment.name'], NULL) = ? AND multiIf(resource.`deployment.environment.name` IS NOT NULL, resource.`deployment.environment.name`::String, mapContains(resources_string, 'deployment.environment.name'), resources_string['deployment.environment.name'], NULL) IS NOT NULL)", sql)
require.Equal(t, "WHERE multiIf(resource.`deployment.environment.name` IS NOT NULL, resource.`deployment.environment.name`::String, mapContains(resources_string, 'deployment.environment.name'), resources_string['deployment.environment.name'], NULL) = ?", sql)
require.Equal(t, []any{"production"}, args)
}
@@ -234,10 +225,10 @@ func TestColumnExpressionForFamilyGroupBy(t *testing.T) {
Evolutions: MockEvolutionData(releaseTime),
}},
}
fm := NewFieldMapper(familyFlagOn(t))
fl := familyFlagOn(t)
storage := NewStorage()
expr, err := fm.ColumnExpressionFor(context.Background(), valuer.UUID{}, startNs, endNs,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, telemetrytypes.FieldDataTypeString, fieldKeys)
expr, err := querybuilder.ResolveColumn(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, startNs, endNs), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, telemetrytypes.FieldDataTypeString, fieldKeys)
require.NoError(t, err)
require.Equal(t, "multiIf((multiIf(resource.`deployment.environment.name` IS NOT NULL, resource.`deployment.environment.name`::String, mapContains(resources_string, 'deployment.environment.name'), resources_string['deployment.environment.name'], NULL) IS NOT NULL OR multiIf(resource.`deployment.environment` IS NOT NULL, resource.`deployment.environment`::String, mapContains(resources_string, 'deployment.environment'), resources_string['deployment.environment'], NULL) IS NOT NULL), COALESCE(NULLIF(multiIf(resource.`deployment.environment.name` IS NOT NULL, resource.`deployment.environment.name`::String, mapContains(resources_string, 'deployment.environment.name'), resources_string['deployment.environment.name'], NULL), ''), NULLIF(multiIf(resource.`deployment.environment` IS NOT NULL, resource.`deployment.environment`::String, mapContains(resources_string, 'deployment.environment'), resources_string['deployment.environment'], NULL), ''), ''), NULL)", expr)
}

View File

@@ -7,12 +7,10 @@ import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"golang.org/x/exp/maps"
"github.com/huandu/go-sqlbuilder"
)
var (
@@ -161,19 +159,15 @@ var (
}
)
type fieldMapper struct {
// fl evaluates the resolve_semconv_families flag during resolution.
// A nil flagger keeps resolution literal.
fl flagger.Flagger
type storage struct{}
var _ qbtypes.Storage = (*storage)(nil)
func NewStorage() *storage {
return &storage{}
}
var _ qbtypes.FieldMapper = (*fieldMapper)(nil)
func NewFieldMapper(fl flagger.Flagger) *fieldMapper {
return &fieldMapper{fl: fl}
}
func (m *fieldMapper) getColumn(
func (m *storage) getColumn(
_ context.Context,
_, _ uint64,
key *telemetrytypes.TelemetryFieldKey,
@@ -220,56 +214,9 @@ func (m *fieldMapper) getColumn(
return nil, qbtypes.ErrColumnNotFound
}
func (m *fieldMapper) ColumnFor(
ctx context.Context,
_ valuer.UUID,
startNs, endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
) ([]*schema.Column, error) {
return m.getColumn(ctx, startNs, endNs, key)
}
// FieldFor returns the table field name for the given key if it exists
// otherwise it returns qbtypes.ErrColumnNotFound.
func (m *fieldMapper) FieldFor(
ctx context.Context,
_ valuer.UUID,
startNs, endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
) (string, error) {
// Special handling for span scope fields
if key.FieldContext == telemetrytypes.FieldContextSpan &&
(strings.ToLower(key.Name) == SpanSearchScopeRoot || strings.ToLower(key.Name) == SpanSearchScopeEntryPoint) {
// Return the field name as-is, the condition builder will handle the SQL generation
return key.Name, nil
}
exprs, existExpr, columns, err := m.resolveColumnExprs(ctx, startNs, endNs, key)
if err != nil {
return "", err
}
if len(exprs) == 1 {
return exprs[0], nil
} else if len(exprs) > 1 {
// Ensure existExpr has the same length as exprs
if len(existExpr) != len(exprs) {
return "", errors.New(errors.TypeInternal, errors.CodeInternal, "length of exist exprs doesn't match to that of exprs")
}
finalExprs := []string{}
for i, expr := range exprs {
finalExprs = append(finalExprs, fmt.Sprintf("%s, %s", existExpr[i], expr))
}
return "multiIf(" + strings.Join(finalExprs, ", ") + ", NULL)", nil
}
// should not reach here
return columns[0].Name, nil
}
// resolveColumnExprs resolves key to its per-column value expressions and existence guards
// (after evolution selection); existExprs only carries guards for guardable column types.
func (m *fieldMapper) resolveColumnExprs(
func (m *storage) resolveColumnExprs(
ctx context.Context,
startNs, endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
@@ -353,156 +300,9 @@ func (m *fieldMapper) resolveColumnExprs(
return exprs, existExprs, columns, nil
}
// upgradeToFamilies swaps single-member candidates for their family when the
// metadata map proves membership. Candidate order and every non-family
// candidate stay exactly as the legacy flow produced them; sibling candidates
// of an already-emitted family are dropped rather than duplicated.
func (m *fieldMapper) upgradeToFamilies(ctx context.Context, orgID valuer.UUID, field *telemetrytypes.TelemetryFieldKey, candidates []*telemetrytypes.LogicalField, keys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.LogicalField {
var families []*telemetrytypes.LogicalField
for _, logical := range querybuilder.MatchingLogicalFields(ctx, orgID, m.fl, field, keys) {
if logical.IsFamily() {
families = append(families, logical)
}
}
if len(families) == 0 {
return candidates
}
out := make([]*telemetrytypes.LogicalField, 0, len(candidates))
emitted := make(map[*telemetrytypes.LogicalField]bool)
for _, candidate := range candidates {
var family *telemetrytypes.LogicalField
for _, fam := range families {
if fam.FieldContext != candidate.FieldContext || fam.FieldDataType != candidate.FieldDataType {
continue
}
memberOfFamily := candidate.Single().Name == field.Name
for _, member := range fam.Members {
if member.Name == candidate.Single().Name {
memberOfFamily = true
break
}
}
if memberOfFamily {
family = fam
break
}
}
if family == nil {
out = append(out, candidate)
continue
}
if emitted[family] {
continue
}
emitted[family] = true
out = append(out, family)
}
return out
}
// ColumnExpressionFor returns the bare (unaliased) SQL expression for the field, resolving
// unknown keys via CandidateKeys and wrapping guardable columns with exists-guard multiIfs
// so an absent key yields NULL.
func (m *fieldMapper) ColumnExpressionFor(
ctx context.Context,
orgID valuer.UUID,
startNs, endNs uint64,
field *telemetrytypes.TelemetryFieldKey,
requiredDataType telemetrytypes.FieldDataType,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
// Resolve the candidate logical field(s).
var candidates []*telemetrytypes.LogicalField
switch _, err := m.FieldFor(ctx, orgID, startNs, endNs, field); {
case err == nil:
// Every match from metadata is kept, similar to the filter path.
candidates = querybuilder.MatchingLogicalFields(ctx, orgID, m.fl, field, keys)
if len(candidates) == 0 {
candidates = []*telemetrytypes.LogicalField{telemetrytypes.SingleLogicalField(field.Name, field)}
}
case errors.Is(err, qbtypes.ErrColumnNotFound):
// The legacy candidate flow, unchanged: column (when the bare name is
// one) plus metadata matches, else synthesized type-variant keys. The
// family step below only swaps candidates for their family; it never
// changes candidate order or non-family behavior.
raw := m.CandidateKeys(ctx, orgID, field, nil, keys)
if len(raw) == 0 {
return "", errors.Wrapf(err, errors.TypeInvalidInput, errors.CodeInvalidInput, "field `%s` not found", field.Name).WithSuggestions(errors.NewSuggestionsOnLevenshteinDistance(field.Name, errors.NounKeys, maps.Keys(keys))...)
}
candidates = m.upgradeToFamilies(ctx, orgID, field, querybuilder.WrapAsLogicalFields(field.Name, raw), keys)
default:
return "", err
}
// Group-by/order (String) and aggregation (String/Float64): every candidate is
// exists-guarded and coerced to requiredDataType, in a single multiIf. Raw select
// (Unspecified) keeps the lighter native shape below.
if requiredDataType != telemetrytypes.FieldDataTypeUnspecified {
var dummyValue any = ""
if requiredDataType == telemetrytypes.FieldDataTypeFloat64 {
dummyValue = 0.0
}
stmts := make([]string, 0, len(candidates)*2)
for _, logical := range candidates {
value, err := querybuilder.LogicalValueExpr(ctx, orgID, startNs, endNs, m, logical)
if err != nil {
return "", err
}
guard, err := querybuilder.LogicalExistsExpr(ctx, orgID, startNs, endNs, m, logical, true)
if err != nil {
return "", err
}
coerced := value
// a time column keeps its native type; coercing it would yield seconds
if temporal, err := m.logicalIsTemporal(ctx, startNs, endNs, logical); err != nil {
return "", err
} else if !temporal {
coerced, _ = querybuilder.DataTypeCollisionHandledFieldName(logical.Single(), dummyValue, value, qbtypes.FilterOperatorUnknown)
}
stmts = append(stmts, guard, coerced)
}
return fmt.Sprintf("multiIf(%s, NULL)", strings.Join(stmts, ", ")), nil
}
if len(candidates) == 1 {
logical := candidates[0]
value, err := querybuilder.LogicalValueExpr(ctx, orgID, startNs, endNs, m, logical)
if err != nil {
return "", err
}
exprs, existExprs, _, _ := m.resolveColumnExprs(ctx, startNs, endNs, logical.Single())
if !logical.IsFamily() && len(exprs) == 1 && len(existExprs) == 1 {
guard, err := querybuilder.LogicalExistsExpr(ctx, orgID, startNs, endNs, m, logical, true)
if err != nil {
return "", err
}
return fmt.Sprintf("multiIf(%s, %s, NULL)", guard, value), nil
}
return value, nil
}
// Multiple candidates (collision / synth): multiIf picks the first that exists,
// stringified so branches share a type.
args := make([]string, 0, len(candidates))
for _, logical := range candidates {
value, err := querybuilder.LogicalValueExpr(ctx, orgID, startNs, endNs, m, logical)
if err != nil {
return "", err
}
guard, err := querybuilder.LogicalExistsExpr(ctx, orgID, startNs, endNs, m, logical, true)
if err != nil {
return "", err
}
args = append(args, fmt.Sprintf("%s, toString(%s)", guard, value))
}
return fmt.Sprintf("multiIf(%s, NULL)", strings.Join(args, ", ")), nil
}
// logicalIsTemporal reports whether the logical field resolves to a single time
// column. A family is attribute-backed and never temporal.
func (m *fieldMapper) logicalIsTemporal(ctx context.Context, startNs, endNs uint64, logical *telemetrytypes.LogicalField) (bool, error) {
func (m *storage) logicalIsTemporal(ctx context.Context, startNs, endNs uint64, logical *telemetrytypes.LogicalField) (bool, error) {
if logical.IsFamily() {
return false, nil
}
@@ -511,7 +311,7 @@ func (m *fieldMapper) logicalIsTemporal(ctx context.Context, startNs, endNs uint
// columnIsTemporal reports whether key resolves to a single time column, after evolution
// selection. Multiple columns mean an attribute-map union, which is never temporal.
func (m *fieldMapper) columnIsTemporal(ctx context.Context, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey) (bool, error) {
func (m *storage) columnIsTemporal(ctx context.Context, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey) (bool, error) {
columns, err := m.getColumn(ctx, startNs, endNs, key)
if err != nil {
return false, err
@@ -523,87 +323,6 @@ func (m *fieldMapper) columnIsTemporal(ctx context.Context, startNs, endNs uint6
return len(newColumns) == 1 && querybuilder.ColumnIsTemporal(newColumns[0]), nil
}
// columnMatchesDataType reports whether a metadata field's data type is consistent with a
// column's ClickHouse type. A bare key's column is only unioned with same-named metadata
// keys that could be the same field; a string attribute named `timestamp` is corrupt
// metadata against the DateTime column and must not degrade the intrinsic.
func columnMatchesDataType(col *schema.Column, dt telemetrytypes.FieldDataType) bool {
if dt == telemetrytypes.FieldDataTypeUnspecified {
return true
}
switch col.Type.GetType() {
case schema.ColumnTypeEnumBool:
return dt == telemetrytypes.FieldDataTypeBool
case schema.ColumnTypeEnumDateTime64:
return false
case schema.ColumnTypeEnumUInt64, schema.ColumnTypeEnumUInt32,
schema.ColumnTypeEnumInt8, schema.ColumnTypeEnumInt16,
schema.ColumnTypeEnumFloat64:
return dt == telemetrytypes.FieldDataTypeInt64 ||
dt == telemetrytypes.FieldDataTypeFloat64 ||
dt == telemetrytypes.FieldDataTypeNumber
default: // String, FixedString, LowCardinality(String), …
return dt == telemetrytypes.FieldDataTypeString
}
}
// CandidateKeys resolves a referenced field to the key(s) to query when it isn't already a
// resolved column. A bare key unions the real column with type-consistent same-named
// metadata keys; a forgiving span/trace context is honored as-is, correcting to a
// metadata match for the stripped name when present; strict attribute/resource contexts
// are honored as-is. Falls back to synthesized type-variant keys.
func (m *fieldMapper) CandidateKeys(ctx context.Context, _ valuer.UUID, field *telemetrytypes.TelemetryFieldKey, value any, keys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
// A real column is considered before metadata for bare and forgiving contexts, so a
// same-named corrupt attribute can't shadow the intrinsic/calculated column.
switch field.FieldContext {
case telemetrytypes.FieldContextUnspecified:
// bare key: the column comes first, alongside same-named metadata keys under other
// contexts whose type is consistent with the column (corrupt entries dropped).
probe := telemetrytypes.NewTelemetryFieldKey(field.Name, telemetrytypes.FieldContextSpan, field.FieldDataType)
if cols, err := m.getColumn(ctx, 0, 0, probe); err == nil && len(cols) > 0 {
candidates := []*telemetrytypes.TelemetryFieldKey{probe}
for _, match := range keys[field.Name] {
if match.FieldContext != telemetrytypes.FieldContextSpan &&
columnMatchesDataType(cols[0], match.FieldDataType) {
candidates = append(candidates, match)
}
}
return candidates
}
case telemetrytypes.FieldContextSpan, telemetrytypes.FieldContextTrace:
// forgiving context honored as-is: a real column wins (span.duration_nano -> col).
if _, err := m.getColumn(ctx, 0, 0, field); err == nil {
return []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(field.Name, field.FieldContext, field.FieldDataType)}
}
}
// Metadata match by name, then the literal `{context}.{name}` spelling (a context can be
// a legitimate prefix in user data, e.g. `metric.max_count`). For a forgiving context
// this is the correction step (span.http.method -> attribute http.method).
if matches := keys[field.Name]; len(matches) > 0 {
return matches
}
if matches := keys[fmt.Sprintf("%s.%s", field.FieldContext.StringValue(), field.Name)]; len(matches) > 0 {
return matches
}
// No metadata: synthesize per context.
switch field.FieldContext {
case telemetrytypes.FieldContextUnspecified:
return querybuilder.SynthesizeKeys(field, value)
case telemetrytypes.FieldContextSpan, telemetrytypes.FieldContextTrace:
// honored as-is: the stripped name lives in the attribute maps
stripped := telemetrytypes.NewTelemetryFieldKey(field.Name, telemetrytypes.FieldContextUnspecified, field.FieldDataType)
return querybuilder.SynthesizeKeys(stripped, value)
case telemetrytypes.FieldContextAttribute, telemetrytypes.FieldContextResource, telemetrytypes.FieldContextScope:
// strict context honored as-is: stripped interpretation first, literal spelling second
literal := telemetrytypes.NewTelemetryFieldKey(field.FieldContext.StringValue()+"."+field.Name, field.FieldContext, field.FieldDataType)
return append(querybuilder.SynthesizeKeys(field, value), querybuilder.SynthesizeKeys(literal, value)...)
}
// contexts that don't exist on spans (log, body, …) have nothing to synthesize
return nil
}
// scopeJSONExistsExpression renders the existence predicate for the scope JSON column, the one
// signal-specific case the generic querybuilder.ExistsExpression must not carry.
func scopeJSONExistsExpression(key *telemetrytypes.TelemetryFieldKey, fieldExpression string, exists bool) (string, bool) {
@@ -626,24 +345,154 @@ func scopeJSONExistsExpression(key *telemetrytypes.TelemetryFieldKey, fieldExpre
return path + " IS NULL", true
}
// ExistsFor implements the per-key existence primitive of qbtypes.FieldMapper.
func (m *fieldMapper) ExistsFor(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
exists bool,
) (string, error) {
columns, err := m.getColumn(ctx, tsStart, tsEnd, key)
// Read returns the bare read of one field key. A span scope key has no column; it
// compiles to a structural predicate, so its read is its name.
func (m *storage) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
if isSpanScopeField(key.Name) {
return key.Name, nil
}
exprs, existExpr, columns, err := m.resolveColumnExprs(ctx, q.StartNs, q.EndNs, key)
if err != nil {
return "", err
}
fieldExpression, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if len(exprs) == 1 {
return exprs[0], nil
} else if len(exprs) > 1 {
// Ensure existExpr has the same length as exprs
if len(existExpr) != len(exprs) {
return "", errors.New(errors.TypeInternal, errors.CodeInternal, "length of exist exprs doesn't match to that of exprs")
}
finalExprs := []string{}
for i, expr := range exprs {
finalExprs = append(finalExprs, fmt.Sprintf("%s, %s", existExpr[i], expr))
}
return "multiIf(" + strings.Join(finalExprs, ", ") + ", NULL)", nil
}
// should not reach here
return columns[0].Name, nil
}
func (m *storage) Exists(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, exists bool) (qbtypes.Existence, error) {
if isSpanScopeField(key.Name) {
return qbtypes.Existence{Predicate: "true", WhenAbsent: qbtypes.AlwaysPresent}, nil
}
columns, err := m.getColumn(ctx, q.StartNs, q.EndNs, key)
if err != nil {
return "", err
return qbtypes.Existence{}, err
}
fieldExpression, err := m.Read(ctx, q, key)
if err != nil {
return qbtypes.Existence{}, err
}
whenAbsent, err := absentReads(q, key, columns)
if err != nil {
return qbtypes.Existence{}, err
}
if expr, ok := scopeJSONExistsExpression(key, fieldExpression, exists); ok {
return expr, nil
return qbtypes.Existence{Predicate: expr, WhenAbsent: whenAbsent}, nil
}
return querybuilder.ExistsExpression(columns, key, tsStart, tsEnd, fieldExpression, exists)
predicate, err := querybuilder.ExistsExpression(columns, key, q.StartNs, q.EndNs, fieldExpression, exists)
if err != nil {
return qbtypes.Existence{}, err
}
return qbtypes.Existence{Predicate: predicate, WhenAbsent: whenAbsent}, nil
}
// absentReads tells what a row without the key reads: NULL from a multi-era
// read, the empty value from a map or from a JSON path (its ::String cast
// folds NULL to the empty string), and a real value from every other column.
func absentReads(q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, columns []*schema.Column) (qbtypes.Absent, error) {
newColumns, _, err := qbtypes.SelectEvolutionsForColumns(columns, key.Evolutions, q.StartNs, q.EndNs)
if err != nil {
return qbtypes.AlwaysPresent, err
}
if len(newColumns) > 1 {
return qbtypes.AbsentIsNull, nil
}
if len(newColumns) == 0 {
return qbtypes.AlwaysPresent, nil
}
switch newColumns[0].Type.GetType() {
case schema.ColumnTypeEnumMap, schema.ColumnTypeEnumJSON:
return qbtypes.AbsentIsSentinel, nil
}
return qbtypes.AlwaysPresent, nil
}
// Fallback answers a key metadata does not report. A bare key that names
// a real column is that column; a forgiving span or trace context is honored
// as-is and corrects to the attribute maps when it names no column; a strict
// context synthesizes its type variants under the stripped and the literal
// spelling.
func (m *storage) Fallback(ctx context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, _ qbtypes.FilterOperator, value any) ([]*telemetrytypes.LogicalField, error) {
var keys []*telemetrytypes.TelemetryFieldKey
switch key.FieldContext {
case telemetrytypes.FieldContextUnspecified:
probe := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextSpan, key.FieldDataType)
if _, err := m.getColumn(ctx, 0, 0, probe); err == nil {
keys = []*telemetrytypes.TelemetryFieldKey{probe}
} else {
keys = querybuilder.SynthesizeKeys(key, value)
}
case telemetrytypes.FieldContextSpan, telemetrytypes.FieldContextTrace:
if _, err := m.getColumn(ctx, 0, 0, key); err == nil {
keys = []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, key.FieldDataType)}
} else {
// the stripped name lives in the attribute maps
stripped := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextUnspecified, key.FieldDataType)
keys = querybuilder.SynthesizeKeys(stripped, value)
}
case telemetrytypes.FieldContextAttribute, telemetrytypes.FieldContextResource, telemetrytypes.FieldContextScope:
// a context can be a legitimate prefix in user data
literal := telemetrytypes.NewTelemetryFieldKey(key.FieldContext.StringValue()+"."+key.Name, key.FieldContext, key.FieldDataType)
keys = append(querybuilder.SynthesizeKeys(key, value), querybuilder.SynthesizeKeys(literal, value)...)
}
return querybuilder.WrapAsLogicalFields(key.Name, keys), nil
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{
Split: qbtypes.MainOfSplit,
OwnContexts: []telemetrytypes.FieldContext{telemetrytypes.FieldContextSpan, telemetrytypes.FieldContextTrace},
}
}
// Compile keeps the two trace-specific rules ahead of the shared condition:
// a span scope key compiles to a structural predicate, and a duration
// operand accepts duration syntax.
func (m *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
if isSpanScopeField(logical.Name) {
condition, err := buildSpanScopeCondition(logical.Single(), operator, value, q.StartNs)
if err != nil {
return qbtypes.Compiled{}, err
}
return qbtypes.Compiled{Condition: condition}, nil
}
// TODO(srikanthccv): maybe extend this to every possible attribute
if logical.Name == "duration_nano" || logical.Name == "durationNano" { // QoL improvement
coerced, err := coerceDurationValue(value)
if err != nil {
return qbtypes.Compiled{}, err
}
value = coerced
}
return querybuilder.SharedCondition(ctx, q, m, logical, operator, value, sb)
}
// ColumnRead keeps a time column in its native type; coercing it would
// yield seconds.
func (m *storage) ColumnRead(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ any) (qbtypes.ColumnExpr, error) {
read, err := querybuilder.DefaultRead(ctx, q, m, logical)
if err != nil {
return qbtypes.ColumnExpr{}, err
}
temporal, err := m.logicalIsTemporal(ctx, q.StartNs, q.EndNs, logical)
if err != nil {
return qbtypes.ColumnExpr{}, err
}
read.KeepType = temporal
return read, nil
}

View File

@@ -2,13 +2,12 @@ package tracestelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
"time"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -147,8 +146,8 @@ func TestGetFieldKeyName(t *testing.T) {
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fm := NewFieldMapper(flaggertest.New(t))
result, err := fm.FieldFor(ctx, valuer.UUID{}, uint64(time.Date(2024, 6, 1, 0, 0, 0, 0, time.UTC).UnixNano()), uint64(time.Date(2024, 6, 5, 0, 0, 0, 0, time.UTC).UnixNano()), &tc.key)
storage := NewStorage()
result, err := storage.Read(ctx, qbtypes.QueryInfo{StartNs: uint64(time.Date(2024, 6, 1, 0, 0, 0, 0, time.UTC).UnixNano()), EndNs: uint64(time.Date(2024, 6, 5, 0, 0, 0, 0, time.UTC).UnixNano())}, &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
@@ -235,8 +234,8 @@ func TestFieldForResourceWithEvolution(t *testing.T) {
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fm := NewFieldMapper(flaggertest.New(t))
result, err := fm.FieldFor(ctx, valuer.UUID{}, tc.tsStart, tc.tsEnd, &tc.key)
storage := NewStorage()
result, err := storage.Read(ctx, qbtypes.QueryInfo{StartNs: tc.tsStart, EndNs: tc.tsEnd}, &tc.key)
require.NoError(t, err)
assert.Equal(t, tc.expectedResult, result)
})
@@ -261,25 +260,25 @@ func TestColumnExpressionForTemporalColumn(t *testing.T) {
name: "time column is not coerced under a numeric aggregation",
key: telemetrytypes.TelemetryFieldKey{Name: "timestamp"},
requiredDataType: telemetrytypes.FieldDataTypeFloat64,
expectedResult: "multiIf(timestamp <> toDateTime64(0, 9), timestamp, NULL)",
expectedResult: "timestamp",
},
{
name: "time column is not coerced under a group by",
key: telemetrytypes.TelemetryFieldKey{Name: "timestamp"},
requiredDataType: telemetrytypes.FieldDataTypeString,
expectedResult: "multiIf(timestamp <> toDateTime64(0, 9), timestamp, NULL)",
expectedResult: "timestamp",
},
{
name: "numeric intrinsic keeps its cast and guard",
name: "numeric intrinsic keeps its cast and takes no guard",
key: telemetrytypes.TelemetryFieldKey{Name: "duration_nano"},
requiredDataType: telemetrytypes.FieldDataTypeFloat64,
expectedResult: "multiIf(duration_nano <> 0, accurateCastOrNull(duration_nano, 'Float64'), NULL)",
expectedResult: "accurateCastOrNull(duration_nano, 'Float64')",
},
{
name: "string intrinsic keeps its cast and guard",
name: "string intrinsic keeps its cast and takes no guard",
key: telemetrytypes.TelemetryFieldKey{Name: "name"},
requiredDataType: telemetrytypes.FieldDataTypeFloat64,
expectedResult: "multiIf(name <> '', accurateCastOrNull(name, 'Float64'), NULL)",
expectedResult: "accurateCastOrNull(name, 'Float64')",
},
{
name: "map-backed attribute keeps its exists guard",
@@ -289,14 +288,14 @@ func TestColumnExpressionForTemporalColumn(t *testing.T) {
FieldDataType: telemetrytypes.FieldDataTypeString,
},
requiredDataType: telemetrytypes.FieldDataTypeString,
expectedResult: "multiIf(mapContains(attributes_string, 'user.id'), attributes_string['user.id'], NULL)",
expectedResult: "multiIf(mapContains(attributes_string, 'user.id'), attributes_string['user.id'], mapContains(attributes_string, 'attribute.user.id'), attributes_string['attribute.user.id'], NULL)",
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fm := NewFieldMapper(flaggertest.New(t))
result, err := fm.ColumnExpressionFor(ctx, valuer.UUID{}, tsStart, tsEnd, &tc.key, tc.requiredDataType, nil)
storage := NewStorage()
result, err := querybuilder.ResolveColumn(ctx, qbtypes.QueryInfo{StartNs: tsStart, EndNs: tsEnd}, storage, &tc.key, tc.requiredDataType, nil)
require.NoError(t, err)
assert.Equal(t, tc.expectedResult, result)
})
@@ -323,13 +322,13 @@ func TestColumnExpressionForTimestampAttributeCollision(t *testing.T) {
},
}
fm := NewFieldMapper(flaggertest.New(t))
storage := NewStorage()
t.Run("bare timestamp resolves to the intrinsic column alone", func(t *testing.T) {
bare := telemetrytypes.TelemetryFieldKey{Name: "timestamp"}
result, err := fm.ColumnExpressionFor(ctx, valuer.UUID{}, tsStart, tsEnd, &bare, telemetrytypes.FieldDataTypeFloat64, keys)
result, err := querybuilder.ResolveColumn(ctx, qbtypes.QueryInfo{StartNs: tsStart, EndNs: tsEnd}, storage, &bare, telemetrytypes.FieldDataTypeFloat64, keys)
require.NoError(t, err)
assert.Equal(t, "multiIf(timestamp <> toDateTime64(0, 9), timestamp, NULL)", result)
assert.Equal(t, "timestamp", result)
})
t.Run("explicit attribute context still reaches the attribute", func(t *testing.T) {
@@ -338,7 +337,7 @@ func TestColumnExpressionForTimestampAttributeCollision(t *testing.T) {
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeNumber,
}
result, err := fm.ColumnExpressionFor(ctx, valuer.UUID{}, tsStart, tsEnd, &attr, telemetrytypes.FieldDataTypeFloat64, keys)
result, err := querybuilder.ResolveColumn(ctx, qbtypes.QueryInfo{StartNs: tsStart, EndNs: tsEnd}, storage, &attr, telemetrytypes.FieldDataTypeFloat64, keys)
require.NoError(t, err)
assert.Contains(t, result, "attributes_number['timestamp']")
})
@@ -403,7 +402,7 @@ func TestColumnExpressionForScopeDeclaredPath(t *testing.T) {
name: "attribute prefix reaches the named scope attribute",
key: telemetrytypes.TelemetryFieldKey{Name: "attribute.name", FieldContext: telemetrytypes.FieldContextScope},
keys: withAttr,
expectedResult: "multiIf(scope.attributes.`name` IS NOT NULL, scope.attributes.`name`::String, NULL)",
expectedResult: "multiIf(scope.attributes.`name` IS NOT NULL, scope.attributes.`name`::String, scope.attributes.`scope.attribute.name` IS NOT NULL, scope.attributes.`scope.attribute.name`::String, NULL)",
},
{
// the caller supplied the context, so `scope.` is part of the name rather than a
@@ -412,7 +411,7 @@ func TestColumnExpressionForScopeDeclaredPath(t *testing.T) {
name: "explicit context keeps a scope-prefixed name intact",
key: telemetrytypes.TelemetryFieldKey{Name: "scope.testing.env", FieldContext: telemetrytypes.FieldContextScope},
keys: declaredOnly,
expectedResult: "multiIf(scope.attributes.`scope.testing.env` IS NOT NULL, scope.attributes.`scope.testing.env`::String, NULL)",
expectedResult: "multiIf(scope.attributes.`scope.testing.env` IS NOT NULL, scope.attributes.`scope.testing.env`::String, scope.attributes.`scope.scope.testing.env` IS NOT NULL, scope.attributes.`scope.scope.testing.env`::String, NULL)",
},
{
// metadata knows both homes under this name, so the short spelling coalesces
@@ -420,20 +419,20 @@ func TestColumnExpressionForScopeDeclaredPath(t *testing.T) {
name: "short name coalesces a known scope attribute with the declared path",
key: telemetrytypes.TelemetryFieldKey{Name: "name", FieldContext: telemetrytypes.FieldContextScope},
keys: withAttr,
expectedResult: "multiIf(scope.attributes.`name` IS NOT NULL, toString(scope.attributes.`name`::String), scope.name::String <> '', toString(scope.name::String), NULL)",
expectedResult: "multiIf(scope.attributes.`name` IS NOT NULL, scope.attributes.`name`::String, scope.name::String <> '', scope.name::String, NULL)",
},
{
name: "short version coalesces a known scope attribute with the declared path",
key: telemetrytypes.TelemetryFieldKey{Name: "version", FieldContext: telemetrytypes.FieldContextScope},
keys: withAttr,
expectedResult: "multiIf(scope.attributes.`version` IS NOT NULL, toString(scope.attributes.`version`::String), scope.version::String <> '', toString(scope.version::String), NULL)",
expectedResult: "multiIf(scope.attributes.`version` IS NOT NULL, scope.attributes.`version`::String, scope.version::String <> '', scope.version::String, NULL)",
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fm := NewFieldMapper(flaggertest.New(t))
result, err := fm.ColumnExpressionFor(ctx, valuer.UUID{}, 0, 0, &tc.key, telemetrytypes.FieldDataTypeUnspecified, tc.keys)
storage := NewStorage()
result, err := querybuilder.ResolveColumn(ctx, qbtypes.QueryInfo{}, storage, &tc.key, telemetrytypes.FieldDataTypeUnspecified, tc.keys)
require.NoError(t, err)
assert.Equal(t, tc.expectedResult, result)
})

View File

@@ -0,0 +1,86 @@
package tracestelemetryschema
import (
"fmt"
"strconv"
"strings"
"time"
"github.com/SigNoz/signoz/pkg/errors"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
)
// coerceDurationValue accepts duration syntax and numeric strings for a
// duration operand, item by item for a list.
func coerceDurationValue(value any) (any, error) {
switch v := value.(type) {
case string:
if duration, err := time.ParseDuration(v); err == nil {
return duration.Nanoseconds(), nil
} else if f, err := strconv.ParseFloat(v, 64); err == nil {
return int64(f), nil
} else {
return nil, errors.WrapInvalidInputf(err, errors.CodeInvalidInput, "invalid duration value: %s", v)
}
case float64:
return int64(v), nil
case float32:
return int64(v), nil
case []any:
coerced := make([]any, len(v))
for i, item := range v {
itemValue, err := coerceDurationValue(item)
if err != nil {
return nil, err
}
coerced[i] = itemValue
}
return coerced, nil
}
return value, nil
}
func isSpanScopeField(name string) bool {
keyName := strings.ToLower(name)
return keyName == SpanSearchScopeRoot || keyName == SpanSearchScopeEntryPoint
}
func buildSpanScopeCondition(key *telemetrytypes.TelemetryFieldKey, operator qbtypes.FilterOperator, value any, startNs uint64) (string, error) {
if operator != qbtypes.FilterOperatorEqual {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "span scope field %s only supports '=' operator", key.Name)
}
var isTrue bool
switch v := value.(type) {
case bool:
isTrue = v
case string:
isTrue = strings.ToLower(v) == "true"
default:
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "span scope field %s expects boolean value, got %T", key.Name, value)
}
if !isTrue {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "span scope field %s can only be filtered with value 'true'", key.Name)
}
keyName := strings.ToLower(key.Name)
switch keyName {
case SpanSearchScopeRoot:
return "parent_span_id = ''", nil
case SpanSearchScopeEntryPoint:
if startNs > 0 { // only add time filter if it is a valid time, else do not add
startS := int64(startNs / 1_000_000_000)
// Note: Escape $$ to $$$$ to avoid sqlbuilder interpreting materialized $ signs
return sqlbuilder.Escape(fmt.Sprintf("((name, resource_string_service$$name) GLOBAL IN (SELECT DISTINCT name, serviceName from %s.%s WHERE time >= toDateTime(%d))) AND parent_span_id != ''",
DBName, TopLevelOperationsTableName, startS)), nil
}
// Note: Escape $$ to $$$$ to avoid sqlbuilder interpreting materialized $ signs
return sqlbuilder.Escape(fmt.Sprintf("((name, resource_string_service$$name) GLOBAL IN (SELECT DISTINCT name, serviceName from %s.%s)) AND parent_span_id != ''",
DBName, TopLevelOperationsTableName)), nil
default:
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid span search scope: %s", key.Name)
}
}

View File

@@ -2,7 +2,7 @@ package tracestelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"testing"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
@@ -15,8 +15,7 @@ import (
func TestSpanScopeFilterExpression(t *testing.T) {
// Test that span scope fields work in filter expressions
fm := NewFieldMapper(flaggertest.New(t))
cb := NewConditionBuilder(fm, flaggertest.New(t))
storage := NewStorage()
tests := []struct {
name string
@@ -78,14 +77,11 @@ func TestSpanScopeFilterExpression(t *testing.T) {
}}
whereClause, err := querybuilder.PrepareWhereClause(tt.expression, querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: fieldKeys,
Builder: sb,
StartNs: tt.startNs,
EndNs: 1761458708000000000,
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: qbtypes.QueryInfo{StartNs: tt.startNs, EndNs: 1761458708000000000},
FieldKeys: fieldKeys,
Builder: sb,
})
if tt.expectError {
@@ -129,8 +125,7 @@ func TestSpanScopeWithResourceFilter(t *testing.T) {
// For now, just verify the expression parses correctly
// In a real implementation, we'd need to check that the resource filter
// is properly skipped when span scope fields are present
fm := NewFieldMapper(flaggertest.New(t))
cb := NewConditionBuilder(fm, flaggertest.New(t))
storage := NewStorage()
// Prepare field keys for the test
fieldKeys := make(map[string][]*telemetrytypes.TelemetryFieldKey)
@@ -148,14 +143,11 @@ func TestSpanScopeWithResourceFilter(t *testing.T) {
}}
_, err := querybuilder.PrepareWhereClause(tt.expression, querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: qbtypes.QueryInfo{StartNs: 1761437108000000000, EndNs: 1761458708000000000},
FieldKeys: fieldKeys,
SkipResourceFilter: false, // This would be set by the statement builder
StartNs: 1761437108000000000,
EndNs: 1761458708000000000,
})
assert.NoError(t, err)

View File

@@ -3,7 +3,6 @@ package querybuildertypesv5
import (
"context"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -16,48 +15,202 @@ var (
ErrBetweenValuesType = errors.NewInvalidInputf(errors.CodeInvalidInput, "(not) between operator requires two values of the number type")
ErrInValues = errors.NewInvalidInputf(errors.CodeInvalidInput, "(not) in operator requires a list of values")
ErrUnsupportedOperator = errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported operator")
// ErrNotSelectable: a field the storage can filter on but cannot read as
// a column. The column expression builder drops such a candidate, and errors only
// when none remains.
ErrNotSelectable = errors.NewInvalidInputf(errors.CodeInvalidInput, "field cannot be selected")
)
// FieldMapper is a signal's storage dialect: given a resolved physical key, it
// renders how this signal's tables express that key — its value (FieldFor),
// its presence (ExistsFor), and its backing columns (ColumnFor) — honoring
// materialization and column-generation choices carried on the key.
// The contract between the generic query compiler (pkg/querybuilder) and
// one signal's storage knowledge. The generic side owns resolution, the
// condition builder, and the column expression builder, written once. A storage answers
// only what the generic side cannot know: two reads, one fallback, its
// traits, and two overrides for a storage with its own sub-language.
//
// ColumnExpressionFor and CandidateKeys additionally carry per-signal
// resolution behavior (unknown-name fallback, candidate ordering); they
// predate the resolution layer and are slated to move behind it.
type FieldMapper interface {
// FieldFor returns the field name for the given key.
FieldFor(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey) (string, error)
// ColumnFor returns the column for the given key.
ColumnFor(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error)
// ColumnExpressionFor returns the column expression for the given key. The audit,
// metrics, metadata, and resource-filter mappers return it aliased (`expr AS name`);
// the logs and traces mappers return a bare expression and callers add their own alias.
// requiredDataType selects the mode: Unspecified builds a group-by/select expression;
// a concrete type (String/Float64) builds an aggregation-argument expression coerced to it.
ColumnExpressionFor(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey, requiredDataType telemetrytypes.FieldDataType, keys map[string][]*telemetrytypes.TelemetryFieldKey) (string, error)
// CandidateKeys returns the key(s) to query for a referenced field: metadata matches for
// the name (or `{context}.{name}`) first, else synthesized type-variant keys for sources
// that support it, else nil (caller errors). value is the filter operand, nil otherwise.
CandidateKeys(ctx context.Context, orgID valuer.UUID, field *telemetrytypes.TelemetryFieldKey, value any, keys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey
// ExistsFor returns the existence predicate for a single physical key (negated when
// exists is false), self-contained and arg-free so it can guard column expressions.
// It is the per-member primitive querybuilder.LogicalExistsExpr and the numeric branch
// of querybuilder.LogicalValueExpr compose family expressions from.
ExistsFor(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey, exists bool) (string, error)
// Rules the generic side holds:
// 1. One resolution order for every storage and every stage.
// 2. One guard rule, derived from Absent.
// 3. The not-found warning fires only when every fallback key is a
// guess: none of them is always present.
// 4. The stage is the operand: a nil value means a column or presence
// use.
// 5. Metadata must report a signal's intrinsic columns. The intrinsic-column
// step also consults the always-present Fallback keys, so a metadata
// gap degrades to the correct column, never to a corrupt metadata
// entry.
// QueryInfo is the query's context as one value: the org and time range
// every read needs, the signal and queried metric that family admission
// needs, and the query-path flags evaluated one time per request. The
// generic flows read FamiliesOn; only the logs storage reads BodyJSONOn.
type QueryInfo struct {
OrgID valuer.UUID
StartNs uint64
EndNs uint64
Signal telemetrytypes.Signal
Metric *telemetrytypes.MetricContext
FamiliesOn bool
BodyJSONOn bool
}
// ConditionBuilder builds the conditions for a filter term. The builder owns key resolution:
// the visitor hands it the raw key + full metadata map + options, not a pre-matched key list.
type ConditionBuilder interface {
ConditionFor(ctx context.Context, orgID valuer.UUID, startNs uint64, endNs uint64, key *telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey, options ConditionBuilderOptions, operator FilterOperator, value any, sb *sqlbuilder.SelectBuilder) ([]string, []string, error)
// Absent is how a field key reads for a row that does not carry it, with
// the signal's keyless contract for that reading. It describes the
// storage, never a per-query choice. Every guard derives from it.
type Absent int
const (
// AlwaysPresent: every row reads a real value. Table columns. No guard
// anywhere; no presence branch in a multi-candidate column.
AlwaysPresent Absent = iota
// AbsentIsSentinel: an absent key reads a sentinel (an empty string, 0,
// false), and the sentinel is not a legitimate value. The field takes
// the exists guard on positive operators and in every column stage, and
// a presence branch in a multi-candidate column. Map-backed attributes.
AbsentIsSentinel
// AbsentIsNull: an absent path reads NULL natively. No condition
// guard; a presence branch in a multi-candidate column only. JSON
// paths.
AbsentIsNull
// AbsentIsValue: an absent key reads a sentinel, and the sentinel is
// the signal's declared keyless contract (label = '' matches rows
// without the label). No guard anywhere. Folds over such fields merge
// by sentinel (NULLIF), never by presence branch, so a presence-branch
// fold (a numeric family) is invalid for them. Metrics labels, rule
// state history labels.
AbsentIsValue
)
// Existence is a field's presence test together with its absence semantics.
type Existence struct {
// Predicate is self-contained and arg-free, so it can sit inside
// guards.
Predicate string
WhenAbsent Absent
}
type ConditionBuilderOptions struct {
// SkipResourceFilter drops the resource context from the candidate set.
SkipResourceFilter bool
// FingerprintSplit is a signal's part in the resource fingerprint split.
type FingerprintSplit int
const (
// NoSplit: the signal has no resource sub-query.
NoSplit FingerprintSplit = iota
// MainOfSplit: a fingerprint sub-query covers the resource fields the
// metadata knows; when it runs, the main query drops them and keeps only
// fallback keys, which the sub-query cannot know.
MainOfSplit
// FingerprintOfSplit: the signal is the sub-query and stores resource
// attributes only. Terms it cannot serve are skipped; the main query
// evaluates them.
FingerprintOfSplit
)
// UnknownKey is what a signal does with a key that neither metadata nor
// its fallback can serve.
type UnknownKey int
const (
// ErrorOnUnknownKey: the term is an error, with suggestions from the
// metadata. Main queries.
ErrorOnUnknownKey UnknownKey = iota
// IgnoreUnknownKey: the term contributes nothing. Side queries whose
// main query owns the error.
IgnoreUnknownKey
)
// Traits are the per-signal capabilities the generic flows read as data.
type Traits struct {
Split FingerprintSplit
// SupportsBodyFunctions: has/hasAny/hasAll/hasToken/search. A
// FingerprintOfSplit signal skips them; every other signal without
// them rejects them before resolution.
SupportsBodyFunctions bool
UnknownKey UnknownKey
// OwnContexts are the contexts that mean "this signal's own record": a
// key under them addresses columns and attributes alike, so the
// resolver matches metadata as if no context was given. Strict contexts
// (resource, attribute, scope, body) are honored as written. Body is
// never one.
OwnContexts []telemetrytypes.FieldContext
}
// ColumnExpr is one field's uncoerced column read, with the one thing the
// renderer cannot derive: whether the native type must survive the stage
// coercion (a time column would collapse to seconds).
type ColumnExpr struct {
SQL string
KeepType bool
}
// Compiled is one field's condition with any warning that came with it. An
// empty Condition contributes nothing.
type Compiled struct {
Condition string
Warnings []string
}
// Storage is what the query compiler knows about one signal's tables: two
// reads it cannot spell itself, one fallback it cannot guess, the traits it
// branches on, and the two compile steps a storage with its own sub-language
// overrides. Resolution, the condition builder, and the column expression builder live in
// querybuilder and are written once.
type Storage interface {
// Read returns the bare SQL read of one field key: no alias, no guard,
// no cast. It honors the materialization and evolution choices the field
// carries.
Read(ctx context.Context, q QueryInfo, field *telemetrytypes.TelemetryFieldKey) (string, error)
// Exists returns the presence test of one field (negated when exists is
// false), and how the field reads when the row lacks it. Virtual keys (a
// span scope) answer AlwaysPresent with the predicate true.
Exists(ctx context.Context, q QueryInfo, field *telemetrytypes.TelemetryFieldKey, exists bool) (Existence, error)
// Fallback returns the fields that could hold a key metadata does not
// report: column aliases, type-variant map reads, body paths, and
// virtual keys that compile to structural predicates (a span search
// scope, a full-text search over a scope). value is the filter operand
// when the term has one and nil otherwise; a filter's fallback may
// narrow the variants by it.
Fallback(ctx context.Context, q QueryInfo, key *telemetrytypes.TelemetryFieldKey, operator FilterOperator, value any) ([]*telemetrytypes.LogicalField, error)
Traits() Traits
// Compile compiles one resolved field into a condition. Every storage
// without its own condition language returns
// querybuilder.SharedCondition. Logs (the body JSON language), the
// resource fingerprint (index hints woven into each operator), the
// related-values metadata (its polarity form), and metrics (labels read
// back as String, so the cast follows the operand) compose the same
// pieces themselves. A Compile that reads a member directly, not through
// querybuilder.LogicalValueExpr, applies the member's ValueMap itself:
// on the read, or through StoredValues on the operand. Pattern operands
// (LIKE, REGEXP, CONTAINS) have no stored translation: transform the
// read, or drop that member's operand-side use (an index hint), never
// emit an untranslated one.
Compile(ctx context.Context, q QueryInfo, field *telemetrytypes.LogicalField, operator FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (Compiled, error)
// ColumnRead returns the uncoerced column read of one resolved field.
// Every storage without a read that depends on the target type returns
// querybuilder.DefaultRead; logs (the legacy body read) and traces (time
// columns) answer their own. target carries the stage's type as a value.
ColumnRead(ctx context.Context, q QueryInfo, field *telemetrytypes.LogicalField, target any) (ColumnExpr, error)
}
// Resolved is what resolution produces for one key: its meanings, and how
// they came to be. It is the only thing the compilers receive. Compile it
// with the operator and value it was resolved with.
type Resolved struct {
Key *telemetrytypes.TelemetryFieldKey
Fields []*telemetrytypes.LogicalField
// FromFallback: the fields came from the storage's fallback, not from
// metadata matches.
FromFallback bool
// Ambiguous: the matches held several interpretations.
Ambiguous bool
// Skipped: the storage contributes nothing for this key.
Skipped bool
Warnings []string
}
type AggExprRewriter interface {
// Rewrite rewrites the aggregation expression to be used in the query.
Rewrite(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, expr string, rateInterval uint64, keys map[string][]*telemetrytypes.TelemetryFieldKey) (string, []any, error)

View File

@@ -35,8 +35,46 @@ type LogicalField struct {
// Members are the physical keys that store this field, in current-first
// order. Each member has its own physical data (Materialized,
// Evolutions, JSONPlan, ...). A per-member accessor does not need data
// from the other members.
// from the other members. The members are shared metadata keys and are
// never changed here.
Members []*TelemetryFieldKey
// ValueMaps is index-aligned with Members: the map that brings a
// member's stored values into the current member's vocabulary, nil
// when the vocabularies are the same. Resolution sets it on this
// wrapper; the reads apply it.
ValueMaps []*ValueMap
}
// ValueMap translates the values one family member stores into the current
// member's vocabulary. Stored and Current are index-aligned, and several
// stored values can map to one current value. A map never holds a sentinel
// (an empty string, 0, false) on either side: a translated sentinel would
// turn absent rows into values, so the generator rejects it. Three readers
// apply it: the family folds in querybuilder, a storage's own Compile, and
// the metadata values suggestions, which read members outside Compile and
// carry the same obligation.
type ValueMap struct {
Stored []string
Current []string
}
// StoredValues returns the values a member stores for one current value:
// the value itself when no map covers it.
func (m *ValueMap) StoredValues(current string) []string {
if m == nil {
return []string{current}
}
var stored []string
for i, c := range m.Current {
if c == current {
stored = append(stored, m.Stored[i])
}
}
if len(stored) == 0 {
return []string{current}
}
return stored
}
// SingleLogicalField makes a logical field that has one physical key.

View File

@@ -209,7 +209,7 @@ def test_metrics_filter_unknown_label_matches_nothing(
)
assert response.status_code == HTTPStatus.OK, response.text
assert querier.get_scalar_table_data(response.json()) == []
assert [w["message"] for w in querier.get_all_warnings(response.json())] == ["label `does_not_exist_label` not found in metadata; check the label name for typos"]
assert [w["message"] for w in querier.get_all_warnings(response.json())] == ["key `does_not_exist_label` not found in metadata; querying the underlying data directly. If this is unexpected, check the key name for typos."]
def test_metrics_full_text_filter_does_not_error(