Compare commits

..

52 Commits

Author SHA1 Message Date
Nikhil Soni
0d7350b2f1 test(queriertraces): cover scope-prefixed attribute resolution
Add a scope attribute whose own name carries a scope. prefix (scope.prefixed)
and a filter case for it. This exercises the targeted scope.<name> metadata
selector: without it the metadata fetch never surfaces a scope.-prefixed key
name and the filter resolves to nothing.

Assisted-by: Claude Opus 4.8
2026-08-19 22:41:58 +05:30
Nikhil Soni
79bfcc987b chore: regenerate openapi specs for scope field context
Add scope to the FieldContext enum in the generated OpenAPI spec and frontend
client (follows enabling FieldContextScope in the enum). Map the new scope value
to undefined in the alert-history suggestion Record so the exhaustive mapping
stays complete.

Assisted-by: Claude Opus 4.8
2026-08-19 19:28:52 +05:30
Nikhil Soni
8bad55bcb3 test(queriertraces): inline the query window in the scope-filter test
Replace the undefined _query_window(now) call with the inline start/end-ms
computation used by the other tests in the suite, fixing the ruff F821 lint
failure. Per the suite's pytest conventions, test modules avoid `_`-prefixed
helpers and inline such logic.

Assisted-by: Claude Opus 4.8
2026-08-19 19:25:27 +05:30
Nikhil Soni
cb05e09a85 refactor(traces): extract lookupIntrinsicOrCalculatedField as a package helper
Move the intrinsic/calculated field lookup out of adjustTraceKey into a
package-level function, consistent with the other helpers in this file.

Assisted-by: Claude Opus 4.8
2026-08-19 19:17:28 +05:30
Nikhil Soni
3bc493175e refactor(traces): resolve intrinsics by context-qualified name in adjustTraceKey
Look up the intrinsic/calculated field by the key's <context>.<name> first, then
its bare name. The qualified lookup lets a scope-context key match the declared
scope.name / scope.version intrinsic instead of the span field that shares its
short name (e.g. {name, scope} is the scope's name, not the span name column),
replacing the scope-specific guard with a generic name resolution.

Assisted-by: Claude Opus 4.8
2026-08-19 19:11:49 +05:30
Nikhil Soni
9f0bbf2209 refactor(traces): treat scope.name/version as intrinsics in adjustTraceKey
Resolve scope keys through the normal intrinsic path instead of skipping
adjustTraceKey. scope.name / scope.version are matched by their full name as
scope-context intrinsics (context preserved, no override), while a scope-context
key is guarded from binding to a span-context intrinsic that shares only its
short name (e.g. {name, scope} must not become the span name column). The
context-blind span<->attribute remapping of legacy fields is untouched.

Assisted-by: Claude Opus 4.8
2026-08-19 19:05:23 +05:30
Nikhil Soni
4c98b1e5b3 refactor(traces): synthesize scope attribute only under scope context
Drop the scope-attribute candidate from the bare (unspecified) and span/trace
synthesize paths in CandidateKeys; a scope attribute is synthesized only for an
explicit scope context, matching main's per-context synthesis. Unqualified keys
absent from metadata no longer probe scope.attributes. Revert the affected
condition/statement builder test expectations accordingly.

Assisted-by: Claude Opus 4.8
2026-08-19 19:00:51 +05:30
Nikhil Soni
1cb343dbd8 refactor(traces): simplify scope resolution to match main's field-mapper structure
Resolve scope keys through a dedicated ColumnExpressionFor branch
(MatchingLogicalFields + ResolveLogicalFields, CandidateKeys fallback) alongside
main's FieldFor switch for non-scope, dropping the resolveReferencedField
refactor. This restores main's non-scope behavior (accurateCastOrNull coercion,
single-candidate attribute resolution) that the refactor had unintentionally
changed, and keeps scope short-name/declared-path/attribute-union handling.

adjustTraceKey uses a single scope early-return; FieldContextScope is enabled in
the field-context enum. query_to_keys still emits a targeted scope.<name>
selector so scope.-prefixed attributes resolve reliably (test added).

Assisted-by: Claude Opus 4.8
2026-08-19 18:37:12 +05:30
Nikhil Soni
82b2331116 Merge remote-tracking branch 'origin/main' into ns/scope 2026-08-19 17:15:15 +05:30
Nikhil Soni
cb2e8f5d35 fix(traces): resolve short scope name to the declared scope path
A scope-context key whose short name matches a declared scope path (e.g.
{name, scope}) must resolve to that declared path (scope.name), not the span
name column and not an undeclared scope attribute. The adjustTraceKey guard
keeps such a key in scope context, and the field mapper's no-metadata synth now
maps the short name to the declared scope path. With a same-named scope
attribute also in metadata, the two union as before.

Assisted-by: Claude Opus 4.8
2026-08-19 11:52:22 +05:30
Nikhil Soni
d8d4c477c2 fix(traces): resolve scope-context intrinsic collision in adjustTraceKey
A scope-context key that shares its name with a span intrinsic (e.g. the
scope's name addressed as {name, scope}) was resolved against the span 'name'
column when no metadata matched. Guard the intrinsic/calculated lookup in
adjustTraceKey so a scope-context key never binds to a non-scope intrinsic;
the intentionally context-blind span<->attribute remapping of legacy fields is
left untouched. This supersedes the AdjustKey change in collision.go, which is
reverted to keep the shared query builder generic.

Assisted-by: Claude Opus 4.8
2026-08-18 20:08:16 +05:30
Nikhil Soni
54c31332bb test: cover scope select union of declared path and same-named attribute
Add ColumnExpressionFor cases for scope names that collide with a declared
scope path: a short name under scope context binds to the declared path, unions
a same-named scope attribute when present, and the full scope.<x> spelling under
explicit scope context addresses the declared path alone. Add the matching
statement-builder list-query case asserting the full union SELECT.

Assisted-by: Claude Opus 4.8
2026-08-18 19:49:05 +05:30
Nikhil Soni
050f5405e9 refactor: simplify the key resolution 2026-08-18 18:53:05 +05:30
Nikhil Soni
dd97cc3bf0 Merge remote-tracking branch 'origin/main' into ns/scope
# Conflicts:
#	pkg/telemetryschema/tracestelemetryschema/condition_builder.go
#	pkg/telemetryschema/tracestelemetryschema/field_mapper.go
#	pkg/telemetryschema/tracestelemetryschema/test_data.go
2026-08-18 17:31:54 +05:30
Nikhil Soni
e3d39386f9 chore: remove evolution logic from scope exists 2026-08-18 17:09:03 +05:30
Nikhil Soni
3205551b63 refactor: move scope exist expr to signal specific module 2026-08-18 14:55:38 +05:30
Nikhil Soni
7aafd63d11 fix: revert normz. fix due to large blast radius 2026-08-18 12:17:31 +05:30
Nikhil Soni
933c009093 fix: normalization consistant and wrong context override removed 2026-08-18 12:16:04 +05:30
Nikhil Soni
cff401ad79 fix: mark instrinsic only when given context doesn't exist 2026-08-17 19:35:54 +05:30
Nikhil Soni
abefa35fde fix: handle exists expr after recent refactor 2026-08-11 22:17:14 +05:30
Nikhil Soni
5c5a4a7a3f Merge remote-tracking branch 'origin/main' into ns/scope 2026-08-11 17:34:30 +05:30
Nikhil Soni
883e9492d6 Merge remote-tracking branch 'origin/main' into ns/scope 2026-08-10 15:01:20 +05:30
Nikhil Soni
ab3b88966e Merge remote-tracking branch 'origin/main' into ns/scope
# Conflicts:
#	pkg/telemetryschema/tracestelemetryschema/field_mapper.go
#	tests/conftest.py
#	tests/integration/tests/querier/04_traces.py
2026-08-05 16:44:41 +05:30
Nikhil Soni
08ebc37109 fix: handle fixed scope.name and scope.version fields to work without scope 2026-07-02 19:04:41 +05:30
Nikhil Soni
aa5a1c5e62 chore: update collector version 2026-07-02 14:46:07 +05:30
Nikhil Soni
7b34a47ac5 chore: run formatting on tests 2026-07-02 14:41:33 +05:30
Nikhil Soni
b4b2d7bb66 test: add test to show cross context matching 2026-06-24 18:34:38 +05:30
Nikhil Soni
e16416475b refactor: drop unused fields 2026-06-24 18:07:47 +05:30
Nikhil Soni
0ea7c1ae6e test: add more cases for scope name 2026-06-24 18:05:50 +05:30
Nikhil Soni
a023c8ed4a test: add integration test for scope fields 2026-06-24 15:25:17 +05:30
Nikhil Soni
a73ae62cd1 Merge remote-tracking branch 'origin/main' into ns/scope 2026-06-24 13:02:18 +05:30
Nikhil Soni
ec6fb58052 chore: add more tests 2026-06-24 12:37:42 +05:30
Nikhil Soni
d3d13eb7ff fix: remove handling of normalized properties for scope
Otherwise it will be impossible to query if scope attribute also
exists with same name - name and version
2026-05-21 11:19:22 +05:30
Nikhil Soni
782de2b210 fix: use correct error type for internal issues 2026-05-20 15:32:26 +05:30
Nikhil Soni
d3c38693f3 fix: allow 'scope.' prefix for keys with other context 2026-05-20 15:30:25 +05:30
Nikhil Soni
8791df3697 fix: avoid removing context prefix to support attr with prefix 2026-05-19 19:14:23 +05:30
Nikhil Soni
eb719c3d0d fix: use key selector with context prefix 2026-05-19 18:56:00 +05:30
Nikhil Soni
f10435c210 Merge remote-tracking branch 'origin' into ns/scope 2026-05-19 13:55:36 +05:30
Nikhil Soni
f3f1e9cb59 chore: add tests for denormalized field name as well 2026-05-19 13:55:25 +05:30
Nikhil Soni
d0370ce3ef fix: handle fields with included context for scope (select clause) 2026-05-14 17:02:56 +05:30
Nikhil Soni
d169761e65 Merge remote-tracking branch 'origin/main' into ns/scope 2026-05-14 11:50:33 +05:30
Nikhil Soni
87864ef5d4 chore: remove duplicates from .gitignore 2026-05-11 15:45:32 +05:30
Nikhil Soni
2e0bc8998e chore: use name as key name for scope instead of scope.name 2026-05-11 15:40:45 +05:30
Nikhil Soni
7e1f4aa50d Merge remote-tracking branch 'origin/main' into ns/scope 2026-05-11 14:27:19 +05:30
Nikhil Soni
35da39247c Merge branch 'main' into ns/scope 2026-05-07 17:41:11 +05:30
Nikhil Soni
ceccc47a34 fix: fix test for case without resource filter 2026-05-07 16:04:03 +05:30
Nikhil Soni
23da5e22ec Merge branch 'main' into ns/scope 2026-05-07 13:27:34 +05:30
Nikhil Soni
4c1b479149 chore: add tests for scope fields 2026-04-28 20:27:10 +05:30
Nikhil Soni
f72204a8b2 refactor: simplify field mapper for scope 2026-04-28 20:26:37 +05:30
Nikhil Soni
deb3f385fa chore: remove underscore version of scope fields 2026-04-23 10:26:55 +05:30
Nikhil Soni
77ce5f86b1 fix: use scope as json field instead with name and version 2026-04-23 01:15:02 +05:30
Nikhil Soni
ff211de441 feat: add support for scope fields in traces 2026-04-14 10:45:08 +05:30
16 changed files with 712 additions and 543 deletions

View File

@@ -8807,6 +8807,7 @@ components:
- span
- trace
- resource
- scope
- attribute
- body
- ""

View File

@@ -3492,6 +3492,7 @@ export enum TelemetrytypesFieldContextDTO {
span = 'span',
trace = 'trace',
resource = 'resource',
scope = 'scope',
attribute = 'attribute',
body = 'body',
'' = '',

View File

@@ -10,6 +10,7 @@ const fieldContextToSuggestionMap: Record<
[TelemetrytypesFieldContextDTO.attribute]: 'attribute',
// no maps for the following values on suggestion context
[TelemetrytypesFieldContextDTO.trace]: undefined,
[TelemetrytypesFieldContextDTO.scope]: undefined,
[TelemetrytypesFieldContextDTO.body]: undefined,
[TelemetrytypesFieldContextDTO.metric]: undefined,
[TelemetrytypesFieldContextDTO.log]: undefined,

View File

@@ -147,11 +147,6 @@ func AdjustKey(key *telemetrytypes.TelemetryFieldKey, keys map[string][]*telemet
// So we can safely override the context and data type
actions = append(actions, fmt.Sprintf("Overriding key: %s to %s", key, intrinsicOrCalculatedField))
// Adopt the canonical name of the field it resolved to, the same way the metadata
// path below does. This is a no-op when the caller looked the field up by the key's
// own name, and carries the qualified name for fields registered under one
// (`name` with scope context -> `scope.name`).
key.Name = intrinsicOrCalculatedField.Name
key.OverrideMetadataFrom(intrinsicOrCalculatedField)
return actions

View File

@@ -57,10 +57,6 @@ func QueryStringToKeysSelectors(query string) []*telemetrytypes.FieldKeySelector
})
}
// A scope attribute whose flattened name begins with `scope.` (e.g. an OTel
// scope attribute nested under `scope`) is indistinguishable from the `scope.`
// context prefix after Normalize strips it. Also fetch the metadata key under
// its full `scope.`-prefixed name so resolution can find it.
// todo(tushar): consider reverting changes done to this method in below PR to avoid scope specific checks
// https://github.com/SigNoz/signoz/issues/11374
if key.FieldContext == telemetrytypes.FieldContextScope {

View File

@@ -73,10 +73,28 @@ func TestQueryToKeys(t *testing.T) {
},
},
{
// A scope reference also fetches its full `scope.`-prefixed name so a scope
// attribute whose flattened name begins with `scope.` (e.g. `scope.prefixed`)
// is discoverable after Normalize strips the prefix.
query: `scope.prefixed = 'local'`,
query: `scope.version = '1.0.0'`,
expectedKeys: []telemetrytypes.FieldKeySelector{
{
Name: "version",
Signal: telemetrytypes.SignalUnspecified,
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
{
Name: "scope.version",
Signal: telemetrytypes.SignalUnspecified,
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
},
},
{
// A scope attribute whose own name carries a `scope.` prefix. `scope.prefixed`
// normalizes to {prefixed, scope}; the second selector re-adds the prefix so the
// metadata fetch can target the attribute's exact key `scope.prefixed` rather than
// relying on the broad `%prefixed%` match.
query: `scope.prefixed = 'x'`,
expectedKeys: []telemetrytypes.FieldKeySelector{
{
Name: "prefixed",

View File

@@ -259,23 +259,6 @@ func adjustTraceKeys(keys map[string][]*telemetrytypes.TelemetryFieldKey, query
return actions
}
// intrinsicLookupName returns the name under which a key is registered in the intrinsic and
// calculated field tables. Span-context intrinsics are registered bare (`name`, `duration_nano`)
// while scope intrinsics are registered fully qualified (`scope.name`, `scope.version`), so a
// scope key must be looked up qualified — bare lookup would match the span intrinsic of the same
// name. A scope key that misses stays qualified rather than falling back to the bare name: a
// scope attribute named `duration_nano` is not the span `duration_nano` column.
func intrinsicLookupName(key *telemetrytypes.TelemetryFieldKey) string {
if key.FieldContext != telemetrytypes.FieldContextScope {
return key.Name
}
prefix := telemetrytypes.FieldContextScope.StringValue() + "."
if strings.HasPrefix(key.Name, prefix) {
return key.Name
}
return prefix + key.Name
}
// adjustTraceKey resolves a single TelemetryFieldKey against the keys map.
func adjustTraceKey(key *telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) []string {
@@ -286,22 +269,14 @@ func adjustTraceKey(key *telemetrytypes.TelemetryFieldKey, keys map[string][]*te
For example: trace_id (intrinsic), response_status_code (calculated).
*/
lookupName := intrinsicLookupName(key)
// Resolve against the context-qualified name first, then the bare name since that can be instrinsic field e.g. scope.name.
var isIntrinsicOrCalculatedField bool
var intrinsicOrCalculatedField telemetrytypes.TelemetryFieldKey
if _, ok := tracestelemetryschema.IntrinsicFields[lookupName]; ok {
isIntrinsicOrCalculatedField = true
intrinsicOrCalculatedField = tracestelemetryschema.IntrinsicFields[lookupName]
} else if _, ok := tracestelemetryschema.CalculatedFields[lookupName]; ok {
isIntrinsicOrCalculatedField = true
intrinsicOrCalculatedField = tracestelemetryschema.CalculatedFields[lookupName]
} else if _, ok := tracestelemetryschema.IntrinsicFieldsDeprecated[lookupName]; ok {
isIntrinsicOrCalculatedField = true
intrinsicOrCalculatedField = tracestelemetryschema.IntrinsicFieldsDeprecated[lookupName]
} else if _, ok := tracestelemetryschema.CalculatedFieldsDeprecated[lookupName]; ok {
isIntrinsicOrCalculatedField = true
intrinsicOrCalculatedField = tracestelemetryschema.CalculatedFieldsDeprecated[lookupName]
if key.FieldContext != telemetrytypes.FieldContextUnspecified {
intrinsicOrCalculatedField, isIntrinsicOrCalculatedField = lookupIntrinsicOrCalculatedField(key.FieldContext.StringValue() + "." + key.Name)
}
if !isIntrinsicOrCalculatedField {
intrinsicOrCalculatedField, isIntrinsicOrCalculatedField = lookupIntrinsicOrCalculatedField(key.Name)
}
if isIntrinsicOrCalculatedField {
@@ -313,6 +288,24 @@ func adjustTraceKey(key *telemetrytypes.TelemetryFieldKey, keys map[string][]*te
return actions
}
// lookupIntrinsicOrCalculatedField returns the intrinsic or calculated field registered under
// name, across the current and deprecated tables.
func lookupIntrinsicOrCalculatedField(name string) (telemetrytypes.TelemetryFieldKey, bool) {
if f, ok := tracestelemetryschema.IntrinsicFields[name]; ok {
return f, true
}
if f, ok := tracestelemetryschema.CalculatedFields[name]; ok {
return f, true
}
if f, ok := tracestelemetryschema.IntrinsicFieldsDeprecated[name]; ok {
return f, true
}
if f, ok := tracestelemetryschema.CalculatedFieldsDeprecated[name]; ok {
return f, true
}
return telemetrytypes.TelemetryFieldKey{}, false
}
// buildListQuery builds a query for list panel type.
func (b *traceQueryStatementBuilder) buildListQuery(
ctx context.Context,

View File

@@ -374,6 +374,94 @@ func TestStatementBuilder(t *testing.T) {
},
expectedErr: nil,
},
{
name: "scope.name filter and group by",
requestType: qbtypes.RequestTypeTimeSeries,
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Aggregations: []qbtypes.TraceAggregation{
{
Expression: "count()",
},
},
Filter: &qbtypes.Filter{
Expression: "scope.name = 'opentelemetry-io'",
},
Limit: 10,
GroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "scope.name",
FieldContext: telemetrytypes.FieldContextScope,
},
},
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(scope.name::String <> '', scope.name::String, NULL)) AS `__GROUP_BY_KEY_0_scope.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.name::String = ? AND scope.name::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_scope.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(scope.name::String <> '', scope.name::String, NULL)) AS `__GROUP_BY_KEY_0_scope.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.name::String = ? AND scope.name::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_scope.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_scope.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_scope.name`",
Args: []any{"opentelemetry-io", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "opentelemetry-io", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
},
{
name: "scope.version filter with scope.name group by",
requestType: qbtypes.RequestTypeTimeSeries,
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Aggregations: []qbtypes.TraceAggregation{
{
Expression: "count()",
},
},
Filter: &qbtypes.Filter{
Expression: "scope.version = '1.0.0'",
},
Limit: 10,
GroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "scope.name",
FieldContext: telemetrytypes.FieldContextScope,
},
},
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(scope.name::String <> '', scope.name::String, NULL)) AS `__GROUP_BY_KEY_0_scope.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_scope.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(scope.name::String <> '', scope.name::String, NULL)) AS `__GROUP_BY_KEY_0_scope.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_scope.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_scope.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_scope.name`",
Args: []any{"1.0.0", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1.0.0", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
},
{
name: "scope.version filter only (no scope field in group by)",
requestType: qbtypes.RequestTypeTimeSeries,
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Aggregations: []qbtypes.TraceAggregation{
{
Expression: "count()",
},
},
Filter: &qbtypes.Filter{
Expression: "scope.version = '1.0.0'",
},
Limit: 10,
GroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "service.name",
},
},
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"1.0.0", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1.0.0", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
},
}
fl := flaggertest.New(t)
@@ -675,107 +763,6 @@ func TestStatementBuilderListQuery(t *testing.T) {
},
expectedErr: nil,
},
{
name: "List query selecting and filtering scope fields",
requestType: qbtypes.RequestTypeRaw,
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Filter: &qbtypes.Filter{
Expression: "scope.name = 'otelcol'",
},
Limit: 10,
SelectFields: []telemetrytypes.TelemetryFieldKey{
{
Name: "scope.name",
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
{
Name: "telemetry.sdk.language",
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
},
},
},
expected: qbtypes.Statement{
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.name::String <> '', scope.name::String, NULL) AS `__SELECT_KEY_3_scope.name`, multiIf(scope.attributes.`telemetry.sdk.language` IS NOT NULL, scope.attributes.`telemetry.sdk.language`::String, NULL) AS `__SELECT_KEY_4_telemetry.sdk.language` FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.name::String = ? AND scope.name::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"otelcol", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
expectedErr: nil,
},
{
// Short scope names (`name`/`version`) collide with span intrinsics; adjustTraceKeys
// must keep them in scope and resolve the declared paths, not the span `name` column.
name: "List query selecting short scope declared names",
requestType: qbtypes.RequestTypeRaw,
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Limit: 10,
SelectFields: []telemetrytypes.TelemetryFieldKey{
{
Name: "name",
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
},
{
Name: "version",
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
},
},
},
expected: qbtypes.Statement{
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.name::String <> '', scope.name::String, NULL) AS `__SELECT_KEY_3_scope.name`, multiIf(scope.version::String <> '', scope.version::String, NULL) AS `__SELECT_KEY_4_scope.version` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
expectedErr: nil,
},
{
// `scope.name` resolves to the declared path; the scope attribute of the same name
// stays reachable by addressing the attributes home explicitly.
name: "List query selecting declared scope path and its shadowed attribute",
requestType: qbtypes.RequestTypeRaw,
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Limit: 10,
SelectFields: []telemetrytypes.TelemetryFieldKey{
{Name: "name", Signal: telemetrytypes.SignalTraces, FieldContext: telemetrytypes.FieldContextScope},
{Name: "attributes.name", Signal: telemetrytypes.SignalTraces, FieldContext: telemetrytypes.FieldContextScope},
},
},
expected: qbtypes.Statement{
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.name::String <> '', scope.name::String, NULL) AS `__SELECT_KEY_3_scope.name`, multiIf(scope.attributes.`name` IS NOT NULL, scope.attributes.`name`::String, NULL) AS `__SELECT_KEY_4_attributes.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
expectedErr: nil,
},
{
// A scope attribute may share its name with a span intrinsic. It must resolve to the
// scope attribute, never to the span column of that name.
name: "List query selecting scope attribute colliding with span intrinsic",
requestType: qbtypes.RequestTypeRaw,
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Limit: 10,
SelectFields: []telemetrytypes.TelemetryFieldKey{
{
Name: "duration_nano",
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
},
},
},
expected: qbtypes.Statement{
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.attributes.`duration_nano` IS NOT NULL, scope.attributes.`duration_nano`::String, NULL) AS `__SELECT_KEY_3_duration_nano` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
expectedErr: nil,
},
}
fl := flaggertest.New(t)
@@ -901,6 +888,111 @@ func TestStatementBuilderListQueryWithCorruptData(t *testing.T) {
},
expectedErr: nil,
},
{
name: "List query with scope filter only (no scope in select or group by)",
requestType: qbtypes.RequestTypeRaw,
keysMap: map[string][]*telemetrytypes.TelemetryFieldKey{
"scope.version": {
{
Name: "scope.version",
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
},
},
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Filter: &qbtypes.Filter{
Expression: "scope.version = '1.0.0'",
},
Limit: 10,
},
expected: qbtypes.Statement{
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, trace_state AS `__SELECT_KEY_3_trace_state`, parent_span_id AS `__SELECT_KEY_4_parent_span_id`, flags AS `__SELECT_KEY_5_flags`, name AS `__SELECT_KEY_6_name`, kind AS `__SELECT_KEY_7_kind`, kind_string AS `__SELECT_KEY_8_kind_string`, duration_nano AS `__SELECT_KEY_9_duration_nano`, status_code AS `__SELECT_KEY_10_status_code`, status_message AS `__SELECT_KEY_11_status_message`, status_code_string AS `__SELECT_KEY_12_status_code_string`, events AS `__SELECT_KEY_13_events`, links AS `__SELECT_KEY_14_links`, response_status_code AS `__SELECT_KEY_15_response_status_code`, external_http_url AS `__SELECT_KEY_16_external_http_url`, http_url AS `__SELECT_KEY_17_http_url`, external_http_method AS `__SELECT_KEY_18_external_http_method`, http_method AS `__SELECT_KEY_19_http_method`, http_host AS `__SELECT_KEY_20_http_host`, db_name AS `__SELECT_KEY_21_db_name`, db_operation AS `__SELECT_KEY_22_db_operation`, has_error AS `__SELECT_KEY_23_has_error`, is_remote AS `__SELECT_KEY_24_is_remote`, attributes_string, attributes_number, attributes_bool, resources_string FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1.0.0", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
},
{
// Regression test: scope.version in selectFields with no metadata (isColumn=true filters it out)
// must still produce scope.version::String, not scope.attributes.version::String
name: "scope.version in selectFields only, no metadata (intrinsic field fallback)",
requestType: qbtypes.RequestTypeRaw,
keysMap: map[string][]*telemetrytypes.TelemetryFieldKey{},
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Filter: &qbtypes.Filter{},
SelectFields: []telemetrytypes.TelemetryFieldKey{
{Name: "scope.version", FieldContext: telemetrytypes.FieldContextUnspecified},
},
Limit: 10,
},
expected: qbtypes.Statement{
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.version::String <> '', scope.version::String, NULL) AS `__SELECT_KEY_3_scope.version` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
},
{
// A scope-context key whose name matches a declared scope path resolves to that
// declared path (scope.name), not the span `name` column and not an undeclared
// scope attribute, even with no metadata.
name: "scope-context name with no metadata resolves to the declared scope path",
requestType: qbtypes.RequestTypeRaw,
keysMap: map[string][]*telemetrytypes.TelemetryFieldKey{},
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Filter: &qbtypes.Filter{},
SelectFields: []telemetrytypes.TelemetryFieldKey{
{Name: "name", FieldContext: telemetrytypes.FieldContextScope},
},
Limit: 10,
},
expected: qbtypes.Statement{
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.name::String <> '', scope.name::String, NULL) AS `__SELECT_KEY_3_name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
},
{
// A scope name that collides with a declared path: with both the declared
// scope.version and a scope attribute literally named `version` in metadata, a
// select on `{version, scope}` unions both (attribute first, declared fallback).
name: "scope select field unions a same-named scope attribute and the declared path",
requestType: qbtypes.RequestTypeRaw,
keysMap: map[string][]*telemetrytypes.TelemetryFieldKey{
"scope.version": {
{
Name: "scope.version",
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
},
"version": {
{
Name: "version",
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
},
},
query: qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Filter: &qbtypes.Filter{},
SelectFields: []telemetrytypes.TelemetryFieldKey{
{Name: "version", FieldContext: telemetrytypes.FieldContextScope},
},
Limit: 10,
},
expected: qbtypes.Statement{
Query: "SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, multiIf(scope.attributes.`version` IS NOT NULL, toString(scope.attributes.`version`::String), scope.version::String <> '', toString(scope.version::String), NULL) AS `__SELECT_KEY_3_version` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
},
}
for _, c := range cases {

View File

@@ -391,6 +391,83 @@ func TestConditionForResourceWithEvolution(t *testing.T) {
}
}
// TestConditionForScopeIntrinsicFields covers the scope.name/scope.version intrinsic
// fields against the "scope" JSON column. These are *declared* String paths on that
// column, so a row without a scope reads as ” and never NULL: presence must be an
// empty-string check, since "IS NOT NULL" would hold for every row. That also rules
// out treating them as nested attribute keys under scope.attributes, which are
// undeclared (Dynamic) paths and genuinely NULL when absent.
func TestConditionForScopeIntrinsicFields(t *testing.T) {
ctx := context.Background()
fm := NewFieldMapper(flaggertest.New(t))
conditionBuilder := NewConditionBuilder(fm, flaggertest.New(t))
testCases := []struct {
name string
key telemetrytypes.TelemetryFieldKey
operator qbtypes.FilterOperator
value any
expectedSQL string
}{
{
name: "Equal - scope.name",
key: telemetrytypes.TelemetryFieldKey{
Name: "scope.name",
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
operator: qbtypes.FilterOperatorEqual,
value: "io.signoz.payment",
expectedSQL: "(scope.name::String = ? AND scope.name::String <> '')",
},
{
name: "Equal - scope.version",
key: telemetrytypes.TelemetryFieldKey{
Name: "scope.version",
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
operator: qbtypes.FilterOperatorEqual,
value: "2.3.1",
expectedSQL: "(scope.version::String = ? AND scope.version::String <> '')",
},
{
name: "Exists - scope.name",
key: telemetrytypes.TelemetryFieldKey{
Name: "scope.name",
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
operator: qbtypes.FilterOperatorExists,
value: nil,
expectedSQL: "scope.name::String <> ''",
},
{
name: "NotExists - scope.version",
key: telemetrytypes.TelemetryFieldKey{
Name: "scope.version",
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
operator: qbtypes.FilterOperatorNotExists,
value: nil,
expectedSQL: "scope.version::String = ''",
},
}
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
conds, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, 0, 0, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
require.NoError(t, err)
sb.Where(conds...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
assert.Contains(t, sql, tc.expectedSQL)
assert.NotContains(t, sql, "scope.`scope.", "must not double-prefix the scope JSON path")
})
}
}
// TestConditionForSynthesizedKeys covers the KeyNotFound fallback: when a
// referenced attribute key has no metadata match, the builder synthesizes key(s) from
// user input and queries anyway, emitting a warning instead of failing.
@@ -585,101 +662,3 @@ func TestConditionForSynthesizedKeys(t *testing.T) {
assert.NotContains(t, sql, "mapContains")
})
}
// TestConditionForScope covers filters on the scope JSON column: declared paths, scope
// attributes, exists semantics, and the attribute-first union when a scope attribute
// shares a declared path's name.
func TestConditionForScope(t *testing.T) {
ctx := context.Background()
fm := NewFieldMapper(flaggertest.New(t))
cb := NewConditionBuilder(fm, flaggertest.New(t))
scopeName := IntrinsicFields["scope.name"]
declared := map[string][]*telemetrytypes.TelemetryFieldKey{"scope.name": {&scopeName}}
build := func(key telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, value any) (string, []any) {
t.Helper()
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, keys, qbtypes.ConditionBuilderOptions{}, op, value, sb)
require.NoError(t, err)
sb.Where(sb.Or(conds...))
return sb.BuildWithFlavor(sqlbuilder.ClickHouse)
}
t.Run("declared scope.name equality is exists-guarded", func(t *testing.T) {
key := telemetrytypes.TelemetryFieldKey{Name: "name", FieldContext: telemetrytypes.FieldContextScope}
sql, args := build(key, declared, qbtypes.FilterOperatorEqual, "otelcol")
assert.Contains(t, sql, "scope.name::String = ?")
assert.Contains(t, sql, "scope.name::String <> ''")
assert.Contains(t, args, "otelcol")
})
t.Run("declared scope.name exists", func(t *testing.T) {
key := telemetrytypes.TelemetryFieldKey{Name: "name", FieldContext: telemetrytypes.FieldContextScope}
sql, _ := build(key, declared, qbtypes.FilterOperatorExists, nil)
assert.Contains(t, sql, "scope.name::String <> ''")
})
t.Run("scope attribute equality guards the raw JSON path", func(t *testing.T) {
key := telemetrytypes.TelemetryFieldKey{Name: "telemetry.sdk.language", FieldContext: telemetrytypes.FieldContextScope}
keys := map[string][]*telemetrytypes.TelemetryFieldKey{
"telemetry.sdk.language": {{Name: "telemetry.sdk.language", Signal: telemetrytypes.SignalTraces, FieldContext: telemetrytypes.FieldContextScope, FieldDataType: telemetrytypes.FieldDataTypeString}},
}
sql, args := build(key, keys, qbtypes.FilterOperatorEqual, "python")
assert.Contains(t, sql, "scope.attributes.`telemetry.sdk.language`::String = ?")
assert.Contains(t, sql, "scope.attributes.`telemetry.sdk.language` IS NOT NULL")
assert.Contains(t, args, "python")
assert.NotContains(t, sql, "scope.`scope.")
})
t.Run("declared path wins over a same-named scope attribute", func(t *testing.T) {
keys := map[string][]*telemetrytypes.TelemetryFieldKey{
"scope.name": {&scopeName},
"name": {{Name: "name", Signal: telemetrytypes.SignalTraces, FieldContext: telemetrytypes.FieldContextScope, FieldDataType: telemetrytypes.FieldDataTypeString}},
}
key := telemetrytypes.TelemetryFieldKey{Name: "name", FieldContext: telemetrytypes.FieldContextScope}
sql, _ := build(key, keys, qbtypes.FilterOperatorEqual, "x")
assert.Contains(t, sql, "scope.name::String = ?")
assert.NotContains(t, sql, "scope.attributes.`name`")
explicit := telemetrytypes.TelemetryFieldKey{Name: "attributes.name", FieldContext: telemetrytypes.FieldContextScope}
sql, _ = build(explicit, keys, qbtypes.FilterOperatorEqual, "x")
assert.Contains(t, sql, "scope.attributes.`name`::String = ?")
})
t.Run("declared scope.version equality", func(t *testing.T) {
scopeVersion := IntrinsicFields["scope.version"]
key := telemetrytypes.TelemetryFieldKey{Name: "version", FieldContext: telemetrytypes.FieldContextScope}
keys := map[string][]*telemetrytypes.TelemetryFieldKey{"scope.version": {&scopeVersion}}
sql, args := build(key, keys, qbtypes.FilterOperatorEqual, "1.2.3")
assert.Contains(t, sql, "scope.version::String = ?")
assert.Contains(t, sql, "scope.version::String <> ''")
assert.Contains(t, args, "1.2.3")
})
t.Run("negative operator on declared path does not add existence guard", func(t *testing.T) {
key := telemetrytypes.TelemetryFieldKey{Name: "name", FieldContext: telemetrytypes.FieldContextScope}
sql, _ := build(key, declared, qbtypes.FilterOperatorNotEqual, "otelcol")
assert.Contains(t, sql, "scope.name::String <> ?")
assert.NotContains(t, sql, "= ''")
})
t.Run("IN on a scope attribute", func(t *testing.T) {
key := telemetrytypes.TelemetryFieldKey{Name: "telemetry.sdk.language", FieldContext: telemetrytypes.FieldContextScope}
keys := map[string][]*telemetrytypes.TelemetryFieldKey{
"telemetry.sdk.language": {{Name: "telemetry.sdk.language", Signal: telemetrytypes.SignalTraces, FieldContext: telemetrytypes.FieldContextScope, FieldDataType: telemetrytypes.FieldDataTypeString}},
}
sql, _ := build(key, keys, qbtypes.FilterOperatorIn, []any{"python", "go"})
assert.Contains(t, sql, "scope.attributes.`telemetry.sdk.language`::String = ?")
assert.Contains(t, sql, "scope.attributes.`telemetry.sdk.language` IS NOT NULL")
})
t.Run("numeric operand on a scope attribute coerces the string path to float", func(t *testing.T) {
key := telemetrytypes.TelemetryFieldKey{Name: "sampler.ratio", FieldContext: telemetrytypes.FieldContextScope}
keys := map[string][]*telemetrytypes.TelemetryFieldKey{
"sampler.ratio": {{Name: "sampler.ratio", Signal: telemetrytypes.SignalTraces, FieldContext: telemetrytypes.FieldContextScope, FieldDataType: telemetrytypes.FieldDataTypeString}},
}
sql, _ := build(key, keys, qbtypes.FilterOperatorGreaterThan, float64(0.5))
assert.Contains(t, sql, "toFloat64OrNull(scope.attributes.`sampler.ratio`::String) > ?")
})
}

View File

@@ -293,21 +293,20 @@ func (m *fieldMapper) resolveColumnExprs(
switch column.Type.GetType() {
case schema.ColumnTypeEnumJSON:
// The ::String cast is required because ClickHouse rejects Variant/Dynamic
// types in GROUP BY; revisit once the clickHouse dependency is updated.
// have to add ::string as clickHouse throws an error :- data types Variant/Dynamic are not allowed in GROUP BY
// once clickHouse dependency is updated, we need to check if we can remove it.
switch key.FieldContext {
case telemetrytypes.FieldContextResource:
exprs = append(exprs, fmt.Sprintf("%s.`%s`::String", columnName, key.Name))
existExprs = append(existExprs, fmt.Sprintf("%s.`%s` IS NOT NULL", columnName, key.Name))
case telemetrytypes.FieldContextScope:
if declared, ok := resolvedScopeDeclaredPath(key.Name); ok {
// declared typed String paths are non-Nullable: absent reads '' not NULL.
exprs = append(exprs, fmt.Sprintf("%s::String", declared))
existExprs = append(existExprs, fmt.Sprintf("%s::String <> ''", declared))
if isDeclaredScopePath(key.Name) {
// declared String paths on the scope column read '' for the missing case
exprs = append(exprs, fmt.Sprintf("%s::String", key.Name))
existExprs = append(existExprs, fmt.Sprintf("%s <> ''", key.Name))
} else {
attribute := scopeAttributeName(key.Name)
exprs = append(exprs, fmt.Sprintf("%s.attributes.`%s`::String", columnName, attribute))
existExprs = append(existExprs, fmt.Sprintf("%s.attributes.`%s` IS NOT NULL", columnName, attribute))
exprs = append(exprs, fmt.Sprintf("%s.attributes.`%s`::String", columnName, key.Name))
existExprs = append(existExprs, fmt.Sprintf("%s.attributes.`%s` IS NOT NULL", columnName, key.Name))
}
default:
return nil, nil, nil, errors.Newf(errors.TypeInvalidInput, errors.CodeInvalidInput, "only resource and scope context fields are supported for json columns, got %s", key.FieldContext.String)
@@ -353,9 +352,9 @@ func (m *fieldMapper) resolveColumnExprs(
return exprs, existExprs, columns, nil
}
// logicalForResolvedColumn upgrades a directly-resolvable key (the FieldFor
// probe succeeded) to its family when the metadata map proves membership;
// otherwise the key stays a single-member logical field.
// logicalForResolvedColumn returns the logical field for a directly-resolvable key: its
// semantic-convention family when the metadata map proves membership, otherwise the
// single-member field for the key as given.
func (m *fieldMapper) logicalForResolvedColumn(ctx context.Context, orgID valuer.UUID, field *telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) *telemetrytypes.LogicalField {
for _, logical := range querybuilder.MatchingLogicalFields(ctx, orgID, m.fl, field, keys) {
if logical.IsFamily() &&
@@ -429,23 +428,38 @@ func (m *fieldMapper) ColumnExpressionFor(
// Resolve the candidate logical field(s).
var candidates []*telemetrytypes.LogicalField
switch _, err := m.FieldFor(ctx, orgID, startNs, endNs, field); {
case err == nil:
// A directly-resolvable key upgrades to its family when the metadata
// map proves membership; otherwise it stays single-member.
candidates = []*telemetrytypes.LogicalField{m.logicalForResolvedColumn(ctx, orgID, field, keys)}
case errors.Is(err, qbtypes.ErrColumnNotFound):
// The legacy candidate flow, unchanged: column (when the bare name is
// one) plus metadata matches, else synthesized type-variant keys. The
// family step below only swaps candidates for their family; it never
// changes candidate order or non-family behavior.
raw := m.CandidateKeys(ctx, orgID, field, nil, keys)
if len(raw) == 0 {
return "", errors.Wrapf(err, errors.TypeInvalidInput, errors.CodeInvalidInput, "field `%s` not found", field.Name).WithSuggestions(errors.NewSuggestionsOnLevenshteinDistance(field.Name, errors.NounKeys, maps.Keys(keys))...)
switch field.FieldContext {
case telemetrytypes.FieldContextScope:
// FieldFor resolves any scope key to a single expression, so the probe below would skip
// the union. Resolve scope the way the filter path does: MatchingLogicalFields surfaces a
// same-named scope attribute (attribute-first) alongside the declared path, and
// CandidateKeys synthesizes when metadata knows neither.
matches := querybuilder.MatchingLogicalFields(ctx, orgID, m.fl, field, keys)
candidates, _ = querybuilder.ResolveLogicalFields(field, matches)
if len(candidates) == 0 {
candidates = querybuilder.WrapAsLogicalFields(field.Name, m.CandidateKeys(ctx, orgID, field, nil, keys))
}
if len(candidates) == 0 {
return "", errors.Wrapf(querybuilder.NewKeyNotFoundError(field.Name), errors.TypeInvalidInput, errors.CodeInvalidInput, "field `%s` not found", field.Name).WithSuggestions(errors.NewSuggestionsOnLevenshteinDistance(field.Name, errors.NounKeys, maps.Keys(keys))...)
}
candidates = m.upgradeToFamilies(ctx, orgID, field, querybuilder.WrapAsLogicalFields(field.Name, raw), keys)
default:
return "", err
switch _, err := m.FieldFor(ctx, orgID, startNs, endNs, field); {
case err == nil:
// A directly-resolvable key upgrades to its family when the metadata
// map proves membership; otherwise it stays single-member.
candidates = []*telemetrytypes.LogicalField{m.logicalForResolvedColumn(ctx, orgID, field, keys)}
case errors.Is(err, qbtypes.ErrColumnNotFound):
// The legacy candidate flow: column (when the bare name is one) plus metadata
// matches, else synthesized type-variant keys. The family step only swaps candidates
// for their family; it never changes candidate order or non-family behavior.
raw := m.CandidateKeys(ctx, orgID, field, nil, keys)
if len(raw) == 0 {
return "", errors.Wrapf(err, errors.TypeInvalidInput, errors.CodeInvalidInput, "field `%s` not found", field.Name).WithSuggestions(errors.NewSuggestionsOnLevenshteinDistance(field.Name, errors.NounKeys, maps.Keys(keys))...)
}
candidates = m.upgradeToFamilies(ctx, orgID, field, querybuilder.WrapAsLogicalFields(field.Name, raw), keys)
default:
return "", err
}
}
// Group-by/order (String) and aggregation (String/Float64): every candidate is
@@ -467,13 +481,10 @@ func (m *fieldMapper) ColumnExpressionFor(
return "", err
}
coerced := value
// a time column keeps its native type; coercing it would yield seconds, and a
// scope expression is already ::String so a String cast would be redundant
alreadyString := field.FieldContext == telemetrytypes.FieldContextScope &&
requiredDataType == telemetrytypes.FieldDataTypeString
// a time column keeps its native type; coercing it would yield seconds
if temporal, err := m.logicalIsTemporal(ctx, startNs, endNs, logical); err != nil {
return "", err
} else if !temporal && !alreadyString {
} else if !temporal {
coerced, _ = querybuilder.DataTypeCollisionHandledFieldName(logical.Single(), dummyValue, value, qbtypes.FilterOperatorUnknown)
}
stmts = append(stmts, guard, coerced)
@@ -499,9 +510,7 @@ func (m *fieldMapper) ColumnExpressionFor(
}
// Multiple candidates (collision / synth): multiIf picks the first that exists,
// stringified so branches share a type. Scope value expressions are already
// ::String, so they skip the redundant toString wrap.
scopeContext := field.FieldContext == telemetrytypes.FieldContextScope
// stringified so branches share a type.
args := make([]string, 0, len(candidates))
for _, logical := range candidates {
value, err := querybuilder.LogicalValueExpr(ctx, orgID, startNs, endNs, m, logical)
@@ -512,11 +521,7 @@ func (m *fieldMapper) ColumnExpressionFor(
if err != nil {
return "", err
}
if scopeContext {
args = append(args, fmt.Sprintf("%s, %s", guard, value))
} else {
args = append(args, fmt.Sprintf("%s, toString(%s)", guard, value))
}
args = append(args, fmt.Sprintf("%s, toString(%s)", guard, value))
}
return fmt.Sprintf("multiIf(%s, NULL)", strings.Join(args, ", ")), nil
}
@@ -598,13 +603,6 @@ func (m *fieldMapper) CandidateKeys(ctx context.Context, _ valuer.UUID, field *t
}
}
// Scope keys resolve only against the scope JSON column, so they never fall through to the
// name-only metadata match below: a same-named span or attribute entry is a different
// field (`scope.duration_nano` is not the duration_nano column).
if field.FieldContext == telemetrytypes.FieldContextScope {
return scopeCandidateKeys(field, keys)
}
// Metadata match by name, then the literal `{context}.{name}` spelling (a context can be
// a legitimate prefix in user data, e.g. `metric.max_count`). For a forgiving context
// this is the correction step (span.http.method -> attribute http.method).
@@ -627,93 +625,44 @@ func (m *fieldMapper) CandidateKeys(ctx context.Context, _ valuer.UUID, field *t
// strict context honored as-is: stripped interpretation first, literal spelling second
literal := telemetrytypes.NewTelemetryFieldKey(field.FieldContext.StringValue()+"."+field.Name, field.FieldContext, field.FieldDataType)
return append(querybuilder.SynthesizeKeys(field, value), querybuilder.SynthesizeKeys(literal, value)...)
case telemetrytypes.FieldContextScope:
// A short scope name that names a declared scope path (e.g. {name, scope} -> scope.name)
// resolves to that declared path, not an undeclared scope attribute.
if compound := field.FieldContext.StringValue() + "." + field.Name; isDeclaredScopePath(compound) {
return []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(compound, telemetrytypes.FieldContextScope, telemetrytypes.FieldDataTypeString)}
}
return []*telemetrytypes.TelemetryFieldKey{synthScopeAttributeKey(field)}
}
// contexts that don't exist on spans (log, body, …) have nothing to synthesize
return nil
}
// scopeCandidateKeys resolves a scope-context key against the scope JSON column. A name that
// addresses one home explicitly — a declared path, or the `attributes.` prefix — resolves to
// it alone, even without metadata. A short declared name is ambiguous: the declared path
// comes first, joined by a same-named scope attribute that metadata knows about, the way a
// bare name puts its column ahead of same-named attributes.
func scopeCandidateKeys(field *telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
scopeStringKey := func(name string) *telemetrytypes.TelemetryFieldKey {
return telemetrytypes.NewTelemetryFieldKey(name, telemetrytypes.FieldContextScope, telemetrytypes.FieldDataTypeString)
}
if declared, ok := resolvedScopeDeclaredPath(field.Name); ok {
return []*telemetrytypes.TelemetryFieldKey{scopeStringKey(declared)}
}
for _, name := range []string{field.Name, telemetrytypes.FieldContextScope.StringValue() + "." + field.Name} {
scoped := []*telemetrytypes.TelemetryFieldKey{}
for _, match := range keys[name] {
if match.FieldContext == telemetrytypes.FieldContextScope {
scoped = append(scoped, match)
}
}
if len(scoped) > 0 {
return scoped
}
}
return []*telemetrytypes.TelemetryFieldKey{synthScopeAttributeKey(field)}
}
// synthScopeAttributeKey guesses a scope attribute (scope.attributes.<name>) for a name
// absent from metadata — the scope analog of querybuilder.SynthesizeKeys.
// synthScopeAttributeKey guesses a scope attribute (scope.attributes.<name>) for a name absent
// from metadata — the scope analog of querybuilder.SynthesizeKeys.
func synthScopeAttributeKey(field *telemetrytypes.TelemetryFieldKey) *telemetrytypes.TelemetryFieldKey {
return telemetrytypes.NewTelemetryFieldKey(field.Name, telemetrytypes.FieldContextScope, telemetrytypes.FieldDataTypeString)
}
// scopeAttributeNamePrefix addresses the attributes home explicitly, so a name that also
// exists as a declared path stays reachable (`scope.attributes.name`).
const scopeAttributeNamePrefix = "attributes."
// scopeAttributeName is the key inside scope.attributes that name refers to.
func scopeAttributeName(name string) string {
return strings.TrimPrefix(name, scopeAttributeNamePrefix)
}
// resolvedScopeDeclaredPath returns the declared path a scope name refers to, resolving the
// short spelling users type (`scope.name` normalizes to `name`) to it. The declared path wins
// over a same-named scope attribute, which stays reachable as `scope.attributes.name` — the
// precedence a real column has over a same-named attribute elsewhere in the builder.
func resolvedScopeDeclaredPath(name string) (string, bool) {
if strings.HasPrefix(name, scopeAttributeNamePrefix) {
return "", false
}
if isDeclaredScopePath(name) {
return name, true
}
qualified := telemetrytypes.FieldContextScope.StringValue() + "." + name
return qualified, isDeclaredScopePath(qualified)
}
// isDeclaredScopePath reports whether name is a declared typed sub-path of the scope JSON
// column (scope.name / scope.version), as opposed to an entry in scope.attributes.
func isDeclaredScopePath(name string) bool {
f, ok := IntrinsicFields[name]
return ok && f.FieldContext == telemetrytypes.FieldContextScope
}
// scopeJSONExistsExpression renders the presence predicate for a scope JSON key, whose
// two homes differ: declared typed paths are non-Nullable (absent reads ”), while
// scope.attributes.* are Dynamic/Nullable. Returns ok=false for non-scope keys so the
// caller falls back to the generic exists expression.
// scopeJSONExistsExpression renders the existence predicate for the scope JSON column, the one
// signal-specific case the generic querybuilder.ExistsExpression must not carry.
func scopeJSONExistsExpression(key *telemetrytypes.TelemetryFieldKey, fieldExpression string, exists bool) (string, bool) {
if key.FieldContext != telemetrytypes.FieldContextScope {
return "", false
}
if _, ok := resolvedScopeDeclaredPath(key.Name); ok {
// Declared String paths are non-Nullable (absent reads '' not NULL).
if isDeclaredScopePath(key.Name) {
if exists {
return fieldExpression + " <> ''", true
}
return fieldExpression + " = ''", true
}
// The value expression casts the JSON path to String, folding a missing key's NULL to
// '', so presence must test the raw path — drop the ::String cast.
// Scope attribute: the value expression casts the JSON path to String, which folds a missing
// key's NULL to '', so presence must test the raw path — drop the ::String cast.
path := strings.TrimSuffix(fieldExpression, "::String")
if exists {
return path + " IS NOT NULL", true

View File

@@ -84,6 +84,33 @@ func TestGetFieldKeyName(t *testing.T) {
expectedResult: "multiIf(resource.`deployment.environment` IS NOT NULL, resource.`deployment.environment`::String, `resource_string_deployment$$environment_exists`, `resource_string_deployment$$environment`, NULL)",
expectedError: nil,
},
{
name: "Scope field - scope.name",
key: telemetrytypes.TelemetryFieldKey{
Name: "scope.name",
FieldContext: telemetrytypes.FieldContextScope,
},
expectedResult: "scope.name::String",
expectedError: nil,
},
{
name: "Scope field - scope.version",
key: telemetrytypes.TelemetryFieldKey{
Name: "scope.version",
FieldContext: telemetrytypes.FieldContextScope,
},
expectedResult: "scope.version::String",
expectedError: nil,
},
{
name: "Scope field - custom attribute",
key: telemetrytypes.TelemetryFieldKey{
Name: "custom.attr",
FieldContext: telemetrytypes.FieldContextScope,
},
expectedResult: "scope.attributes.`custom.attr`::String",
expectedError: nil,
},
{
// Query like `attribute.attribute_string:string` should resolve to `attributes_string['attribute_string']`.
name: "Attribute key whose name collides with contextual map column resolves as a map lookup",
@@ -305,170 +332,77 @@ func TestColumnExpressionForTimestampAttributeCollision(t *testing.T) {
})
}
// scopeKey builds a TelemetryFieldKey the way the API boundary would after Normalize.
func scopeKey(name string) telemetrytypes.TelemetryFieldKey {
return telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
}
}
// declaredScopeKeys injects the scope.name/scope.version intrinsics into the metadata map
// the way metadata.go does at query time; resolution of the declared paths depends on it.
func declaredScopeKeys() map[string][]*telemetrytypes.TelemetryFieldKey {
scopeName := IntrinsicFields["scope.name"]
scopeVersion := IntrinsicFields["scope.version"]
return map[string][]*telemetrytypes.TelemetryFieldKey{
"scope.name": {&scopeName},
"scope.version": {&scopeVersion},
}
}
func scopeAttribute(name string) *telemetrytypes.TelemetryFieldKey {
return &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
}
}
// TestColumnExpressionForScope covers the scope resolution matrix from PR #10920: declared
// paths, scope attributes, and the precedence between them when a scope attribute shares its
// name with a declared path.
func TestColumnExpressionForScope(t *testing.T) {
// TestColumnExpressionForScopeUnion covers select-side resolution of scope names that
// collide with a declared scope path. A short name under scope context (or the bare
// `scope.<x>` spelling that normalizes to it) binds to the declared path, and unions a
// same-named scope attribute when one is also in metadata. The full `scope.<x>` name under
// explicit scope context addresses the declared path alone.
func TestColumnExpressionForScopeUnion(t *testing.T) {
ctx := context.Background()
tsStart := uint64(time.Date(2024, 6, 1, 0, 0, 0, 0, time.UTC).UnixNano())
tsEnd := uint64(time.Date(2024, 6, 5, 0, 0, 0, 0, time.UTC).UnixNano())
fm := NewFieldMapper(flaggertest.New(t))
run := func(field telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) string {
t.Helper()
result, err := fm.ColumnExpressionFor(ctx, valuer.UUID{}, tsStart, tsEnd, &field, telemetrytypes.FieldDataTypeUnspecified, keys)
require.NoError(t, err)
return result
scopeKey := func(name string) *telemetrytypes.TelemetryFieldKey {
return &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
}
}
declaredOnly := map[string][]*telemetrytypes.TelemetryFieldKey{
"scope.name": {scopeKey("scope.name")},
"scope.version": {scopeKey("scope.version")},
}
withAttr := map[string][]*telemetrytypes.TelemetryFieldKey{
"scope.name": {scopeKey("scope.name")},
"scope.version": {scopeKey("scope.version")},
"name": {scopeKey("name")},
"version": {scopeKey("version")},
}
t.Run("short name binds to declared scope.name when no attribute exists", func(t *testing.T) {
assert.Equal(t,
"multiIf(scope.name::String <> '', scope.name::String, NULL)",
run(scopeKey("name"), declaredScopeKeys()))
})
t.Run("fully-qualified scope.name isolates the declared path", func(t *testing.T) {
assert.Equal(t,
"multiIf(scope.name::String <> '', scope.name::String, NULL)",
run(scopeKey("scope.name"), declaredScopeKeys()))
})
t.Run("short version binds to declared scope.version", func(t *testing.T) {
assert.Equal(t,
"multiIf(scope.version::String <> '', scope.version::String, NULL)",
run(scopeKey("version"), declaredScopeKeys()))
})
t.Run("plain scope attribute", func(t *testing.T) {
keys := declaredScopeKeys()
keys["testing.env"] = []*telemetrytypes.TelemetryFieldKey{scopeAttribute("testing.env")}
assert.Equal(t,
"multiIf(scope.attributes.`testing.env` IS NOT NULL, scope.attributes.`testing.env`::String, NULL)",
run(scopeKey("testing.env"), keys))
})
t.Run("scope attribute synthesized when absent from metadata", func(t *testing.T) {
assert.Equal(t,
"multiIf(scope.attributes.`testing.env` IS NOT NULL, scope.attributes.`testing.env`::String, NULL)",
run(scopeKey("testing.env"), declaredScopeKeys()))
})
t.Run("declared path wins over a same-named scope attribute", func(t *testing.T) {
keys := declaredScopeKeys()
keys["name"] = []*telemetrytypes.TelemetryFieldKey{scopeAttribute("name")}
assert.Equal(t,
"multiIf(scope.name::String <> '', scope.name::String, NULL)",
run(scopeKey("name"), keys))
})
t.Run("attributes prefix reaches the shadowed scope attribute", func(t *testing.T) {
keys := declaredScopeKeys()
keys["name"] = []*telemetrytypes.TelemetryFieldKey{scopeAttribute("name")}
assert.Equal(t,
"multiIf(scope.attributes.`name` IS NOT NULL, scope.attributes.`name`::String, NULL)",
run(scopeKey("attributes.name"), keys))
})
t.Run("fully-qualified scope.version isolates declared even with conflicting attribute", func(t *testing.T) {
keys := declaredScopeKeys()
keys["version"] = []*telemetrytypes.TelemetryFieldKey{scopeAttribute("version")}
assert.Equal(t,
"multiIf(scope.version::String <> '', scope.version::String, NULL)",
run(scopeKey("scope.version"), keys))
})
t.Run("group by keeps the scope expression unwrapped by toString", func(t *testing.T) {
keys := declaredScopeKeys()
keys["name"] = []*telemetrytypes.TelemetryFieldKey{scopeAttribute("name")}
result, err := fm.ColumnExpressionFor(ctx, valuer.UUID{}, tsStart, tsEnd, &[]telemetrytypes.TelemetryFieldKey{scopeKey("attributes.name")}[0], telemetrytypes.FieldDataTypeString, keys)
require.NoError(t, err)
assert.Equal(t,
"multiIf(scope.attributes.`name` IS NOT NULL, scope.attributes.`name`::String, NULL)",
result)
})
}
// TestFieldForScope covers the per-key SQL for a resolved scope key.
func TestFieldForScope(t *testing.T) {
ctx := context.Background()
tsStart := uint64(time.Date(2024, 6, 1, 0, 0, 0, 0, time.UTC).UnixNano())
tsEnd := uint64(time.Date(2024, 6, 5, 0, 0, 0, 0, time.UTC).UnixNano())
fm := NewFieldMapper(flaggertest.New(t))
cases := map[string]string{
"scope.name": "scope.name::String",
"scope.version": "scope.version::String",
"custom.attr": "scope.attributes.`custom.attr`::String",
}
for name, want := range cases {
t.Run(name, func(t *testing.T) {
key := scopeKey(name)
got, err := fm.FieldFor(ctx, valuer.UUID{}, tsStart, tsEnd, &key)
require.NoError(t, err)
assert.Equal(t, want, got)
// A scope path must never double-prefix the JSON column.
assert.NotContains(t, got, "scope.`scope.")
})
}
}
// TestExistsForScope covers the presence predicates: declared paths test <> ” (non-Nullable),
// scope attributes test the raw JSON path IS NOT NULL.
func TestExistsForScope(t *testing.T) {
ctx := context.Background()
tsStart := uint64(time.Date(2024, 6, 1, 0, 0, 0, 0, time.UTC).UnixNano())
tsEnd := uint64(time.Date(2024, 6, 5, 0, 0, 0, 0, time.UTC).UnixNano())
fm := NewFieldMapper(flaggertest.New(t))
cases := []struct {
name string
key string
exists bool
want string
testCases := []struct {
name string
key telemetrytypes.TelemetryFieldKey
keys map[string][]*telemetrytypes.TelemetryFieldKey
expectedResult string
}{
{"declared exists", "scope.name", true, "scope.name::String <> ''"},
{"declared not exists", "scope.name", false, "scope.name::String = ''"},
{"short declared exists", "name", true, "scope.name::String <> ''"},
{"attribute exists", "exception.type", true, "scope.attributes.`exception.type` IS NOT NULL"},
{"attribute not exists", "exception.type", false, "scope.attributes.`exception.type` IS NULL"},
{"shadowed attribute exists", "attributes.name", true, "scope.attributes.`name` IS NOT NULL"},
{"shadowed attribute not exists", "attributes.name", false, "scope.attributes.`name` IS NULL"},
{
name: "short name under scope context binds to the declared path",
key: telemetrytypes.TelemetryFieldKey{Name: "version", FieldContext: telemetrytypes.FieldContextScope},
keys: declaredOnly,
expectedResult: "multiIf(scope.version::String <> '', scope.version::String, NULL)",
},
{
name: "short name unions the declared path and a same-named scope attribute",
key: telemetrytypes.TelemetryFieldKey{Name: "version", FieldContext: telemetrytypes.FieldContextScope},
keys: withAttr,
expectedResult: "multiIf(scope.attributes.`version` IS NOT NULL, toString(scope.attributes.`version`::String), scope.version::String <> '', toString(scope.version::String), NULL)",
},
{
name: "full scope.version name under scope context addresses the declared path alone",
key: telemetrytypes.TelemetryFieldKey{Name: "scope.version", FieldContext: telemetrytypes.FieldContextScope},
keys: withAttr,
expectedResult: "multiIf(scope.version::String <> '', scope.version::String, NULL)",
},
{
name: "short scope name unions the declared scope.name and a same-named attribute",
key: telemetrytypes.TelemetryFieldKey{Name: "name", FieldContext: telemetrytypes.FieldContextScope},
keys: withAttr,
expectedResult: "multiIf(scope.attributes.`name` IS NOT NULL, toString(scope.attributes.`name`::String), scope.name::String <> '', toString(scope.name::String), NULL)",
},
{
name: "full scope.name name under scope context addresses the declared path alone",
key: telemetrytypes.TelemetryFieldKey{Name: "scope.name", FieldContext: telemetrytypes.FieldContextScope},
keys: withAttr,
expectedResult: "multiIf(scope.name::String <> '', scope.name::String, NULL)",
},
}
for _, tc := range cases {
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
key := scopeKey(tc.key)
got, err := fm.ExistsFor(ctx, valuer.UUID{}, tsStart, tsEnd, &key, tc.exists)
fm := NewFieldMapper(flaggertest.New(t))
result, err := fm.ColumnExpressionFor(ctx, valuer.UUID{}, 0, 0, &tc.key, telemetrytypes.FieldDataTypeUnspecified, tc.keys)
require.NoError(t, err)
assert.Equal(t, tc.want, got)
assert.Equal(t, tc.expectedResult, result)
})
}
}

View File

@@ -113,6 +113,20 @@ func BuildCompleteFieldKeyMap(releaseTime time.Time) map[string][]*telemetrytype
FieldDataType: telemetrytypes.FieldDataTypeBool,
},
},
"scope.name": {
{
Name: "scope.name",
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
},
"scope.version": {
{
Name: "scope.version",
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
},
// both spellings of an enabled semantic-convention family
"deployment.environment.name": {
{
@@ -128,21 +142,6 @@ func BuildCompleteFieldKeyMap(releaseTime time.Time) map[string][]*telemetrytype
FieldDataType: telemetrytypes.FieldDataTypeString,
},
},
// declared scope paths, mirroring the intrinsics metadata.go injects at query time
"scope.name": {
{
Name: "scope.name",
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
},
"scope.version": {
{
Name: "scope.version",
FieldContext: telemetrytypes.FieldContextScope,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
},
}
for _, keys := range keysMap {
for _, key := range keys {

View File

@@ -35,24 +35,6 @@ func TestGetFieldKeyFromKeyText(t *testing.T) {
FieldDataType: FieldDataTypeUnspecified,
},
},
{
// only the context prefix is stripped, so `attributes.` survives to address the
// scope attribute that a declared path of the same name would otherwise win
keyText: "scope.attributes.name",
expected: TelemetryFieldKey{
Name: "attributes.name",
FieldContext: FieldContextScope,
FieldDataType: FieldDataTypeUnspecified,
},
},
{
keyText: "scope.custom.attr:string",
expected: TelemetryFieldKey{
Name: "custom.attr",
FieldContext: FieldContextScope,
FieldDataType: FieldDataTypeString,
},
},
{
keyText: "attribute.http.method",
expected: TelemetryFieldKey{
@@ -312,6 +294,17 @@ func TestNormalize(t *testing.T) {
FieldDataType: FieldDataTypeString,
},
},
{
name: "Normalize keeps a prefix that does not match the set context",
input: TelemetryFieldKey{
Name: "scope.name",
FieldContext: FieldContextAttribute,
},
expected: TelemetryFieldKey{
Name: "scope.name",
FieldContext: FieldContextAttribute,
},
},
{
name: "Normalize body field",
input: TelemetryFieldKey{

View File

@@ -999,6 +999,8 @@ def generate_traces_with_corrupt_metadata() -> list[Traces]:
"cloud.provider": "integration",
"cloud.account.id": "000",
"trace_id": "corrupt_data",
"scope_name": "corrupt_data",
"scope.scope.name": "corrupt_data",
},
attributes={
"net.transport": "IP.TCP",
@@ -1007,7 +1009,10 @@ def generate_traces_with_corrupt_metadata() -> list[Traces]:
"http.request.method": "POST",
"http.response.status_code": "200",
"timestamp": "corrupt_data",
"version": "1.0.0",
"scope.scope.version": "1.0.0",
},
scope={"name": "io.signoz.http.server", "version": "2.0.0"},
),
Traces(
timestamp=now - timedelta(seconds=3.5),
@@ -1027,12 +1032,24 @@ def generate_traces_with_corrupt_metadata() -> list[Traces]:
"cloud.provider": "integration",
"cloud.account.id": "000",
"timestamp": "corrupt_data",
"scope.attributes.name": "corrupt_data",
},
attributes={
"db.name": "integration",
"db.operation": "SELECT",
"db.statement": "SELECT * FROM integration",
"trace_d": "corrupt_data",
"scope.attributes.version": "corrupt_data",
},
scope={
"name": "io.opentelemetry.contrib.http",
"version": "1.0.0",
"attributes": {
"telemetry.sdk.language": "cpp",
"name": "not-the-real-name",
"version": "not-the-real-version",
"attributes": "literally-a-key-named-attributes",
},
},
),
Traces(
@@ -1053,12 +1070,15 @@ def generate_traces_with_corrupt_metadata() -> list[Traces]:
"cloud.provider": "integration",
"cloud.account.id": "000",
"duration_nano": "corrupt_data",
"scope.scope.attributes.version": "corrupt_data",
},
attributes={
"http.request.method": "PATCH",
"http.status_code": "404",
"id": "1",
"scope.scope.version": "corrupt_data",
},
scope={"name": "io.signoz.http.client", "version": "2.0.0"},
),
Traces(
timestamp=now - timedelta(seconds=1),
@@ -1077,6 +1097,7 @@ def generate_traces_with_corrupt_metadata() -> list[Traces]:
"host.name": "linux-001",
"cloud.provider": "integration",
"cloud.account.id": "001",
"scope.scope.version": "corrupt_data",
},
attributes={
"message.type": "SENT",
@@ -1084,7 +1105,10 @@ def generate_traces_with_corrupt_metadata() -> list[Traces]:
"messaging.message.id": "001",
"duration_nano": "corrupt_data",
"id": 1,
"scope": "corrupt_data",
"scope.attributes.name": "corrupt_data",
},
scope={"name": "io.signoz.messaging", "version": "3.0.0"},
),
]

View File

@@ -302,6 +302,7 @@ class Traces(ABC):
db_operation: str
has_error: bool
is_remote: str
scope_json: dict[str, Any]
resource: list[TracesResource]
tag_attributes: list[TracesTagAttributes]
@@ -327,6 +328,7 @@ class Traces(ABC):
links: list[TracesLink] = [],
trace_state: str = "",
flags: np.uint32 = 0,
scope: dict[str, Any] = {},
resource_write_mode: Literal["legacy_only", "dual_write"] = "dual_write",
) -> None:
if timestamp is None:
@@ -408,6 +410,33 @@ class Traces(ABC):
# Calculate resource fingerprint
self.resource_fingerprint = LogsOrTracesFingerprint(self.resources_string).calculate()
# Process scope mirroring the InstrumentationScope on the OTLP span.
scope_name = scope.get("name", "")
scope_version = scope.get("version", "")
scope_string = {k: str(v) for k, v in scope.get("attributes", {}).items()}
self.scope_json = {
"name": scope_name,
"version": scope_version,
"attributes": scope_string,
}
scope_keys = {"scope.name": scope_name, "scope.version": scope_version}
scope_keys.update(scope_string)
for k, v in scope_keys.items():
if v == "":
continue
self.tag_attributes.append(
TracesTagAttributes(
timestamp=timestamp,
tag_key=k,
tag_type="scope",
tag_data_type="string",
string_value=v,
number_value=None,
)
)
self.attribute_keys.append(TracesResourceOrAttributeKeys(name=k, datatype="string", tag_type="scope"))
# Process attributes by type and populate custom fields
self.attribute_string = {}
self.attributes_number = {}
@@ -659,6 +688,7 @@ class Traces(ABC):
self.has_error,
self.is_remote,
self.resource_json,
self.scope_json,
],
dtype=object,
)
@@ -689,6 +719,7 @@ class Traces(ABC):
attributes=data.get("attributes", {}),
trace_state=data.get("trace_state", ""),
flags=data.get("flags", 0),
scope=data.get("scope", {}),
)
@classmethod
@@ -828,6 +859,7 @@ def insert_traces_to_clickhouse(conn, traces: list[Traces]) -> None:
"has_error",
"is_remote",
"resource",
"scope",
],
data=[trace.np_arr() for trace in traces],
)

View File

@@ -1240,6 +1240,13 @@ def test_traces_list_span_scope(
lambda x: {"duration_nano": int(x[1].duration_nano), "span_id": x[1].span_id, "timestamp": format_timestamp(x[1].timestamp), "trace_id": x[1].trace_id},
id="select_attribute_duration_order_intrinsic",
),
# Case 9: filter on the intrinsic scope.version. Only x[1] should match.
pytest.param(
BuilderQuery(signal="traces", name="A", select_fields=[TelemetryFieldKey("timestamp")], filter_expression="scope.version = '1.0.0'", limit=1),
HTTPStatus.OK,
lambda x: {"span_id": x[1].span_id, "timestamp": format_timestamp(x[1].timestamp), "trace_id": x[1].trace_id},
id="filter_scope_version",
),
],
)
def test_traces_list_with_corrupt_data(
@@ -1283,6 +1290,161 @@ def test_traces_list_with_corrupt_data(
assert get_rows(response)[0]["data"] == expected(traces)
@pytest.mark.parametrize(
"filter_expression,expected_indices",
[
# Intrinsic scope.name / scope.version resolve to the JSON sub-columns.
pytest.param("scope.name = 'io.signoz.payment'", [1], id="intrinsic_scope_name"),
pytest.param("scope.version = '2.3.1'", [0], id="intrinsic_scope_version"),
# A scope attribute resolves against the scope JSON column's attributes.
pytest.param("scope.telemetry.sdk.language = 'python'", [1], id="scope_attribute"),
# A scope attribute whose own name carries a `scope.` prefix. `scope.prefixed`
# normalizes to {prefixed, scope} and must still resolve to the attribute.
pytest.param("scope.prefixed = 'prefixed-val'", [0], id="scope_prefixed_attribute"),
# `env.tier` is a span attribute on span 0 and a scope attribute on
# span 1. Unprefixed -> no explicit context, so it is checked in every
# applicable context (attribute OR scope) and both spans match.
pytest.param("env.tier = 'gold'", [0, 1], id="bare_cross_context"),
# The explicit `scope.` prefix forces scope context only, so span 0's
# span attribute is ignored — only span 1 matches.
pytest.param("scope.env.tier = 'gold'", [1], id="scope_prefixed_cross_context"),
# `scope.name` matches BOTH the intrinsic scope.name field (span 0) and a
# scope attribute literally named `name` (span 1's scope attribute
# name='io.signoz.checkout').
pytest.param("scope.name = 'io.signoz.checkout'", [0, 1], id="scope_name_collision"),
# `scope.name` also matches a span attribute literally named `scope.name`
# (attribute context) — span 2 carries attribute scope.name='attr-scope-name'.
pytest.param("scope.name = 'attr-scope-name'", [2], id="scope_name_attribute_collision"),
# An unprefixed `name` resolves to the intrinsic span `name` column and a
# `name` scope attribute, but NOT the scope.name field. Span 2's span
# name and span 1's scope attribute `name` both equal 'io.signoz.checkout';
# span 0's scope.name field equals it too but is NOT matched.
pytest.param("name = 'io.signoz.checkout'", [1, 2], id="bare_name_excludes_scope_name_field"),
# A value that no resolvable key holds (scope.name/scope.version field,
# a `name`/`version` scope attribute, or a same-named attribute/resource)
# returns nothing.
pytest.param("scope.version = 'corrupt_data'", [], id="scope_version_no_match"),
pytest.param("scope.name = 'corrupt_data'", [], id="scope_name_no_match"),
],
)
def test_traces_list_with_scope_filter(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_traces: Callable[[list[Traces]], None],
filter_expression: str,
expected_indices: list[int],
) -> None:
"""
Setup three spans with different scope key resolution:
- x[0]: scope.name/version 'io.signoz.checkout'/'2.3.1'; span attribute
env.tier='gold'.
- x[1]: scope.name/version 'io.signoz.payment'/'4.5.6'; scope attributes
telemetry.sdk.language='python', env.tier='gold', and a `name` scope
attribute colliding with x[0]'s scope.name value.
- x[2]: span name 'io.signoz.checkout' (colliding with x[0]'s scope.name
value) and a span attribute literally named `scope.name`.
Tests:
- Filtering on scope.name / scope.version / a scope attribute.
- An unprefixed key is resolved across contexts (scope checked alongside
attribute / intrinsic), while a `scope.`-prefixed key is scope-only.
- `scope.name` hits the intrinsic field, a `name` scope attribute, and a
span attribute `scope.name` (cross-context), while a bare `name` hits
the span name column (and a `name` scope attribute) but never the
scope.name field.
"""
now = datetime.now(tz=UTC).replace(microsecond=0)
trace_id = TraceIdGenerator.trace_id()
span_ids = [TraceIdGenerator.span_id() for _ in range(3)]
traces = [
Traces(
timestamp=now - timedelta(seconds=4),
duration=timedelta(seconds=2),
trace_id=trace_id,
span_id=span_ids[0],
parent_span_id="",
name="GET /checkout",
kind=TracesKind.SPAN_KIND_SERVER,
status_code=TracesStatusCode.STATUS_CODE_OK,
resources={"service.name": "checkout"},
attributes={"http.request.method": "GET", "env.tier": "gold"},
scope={
"name": "io.signoz.checkout",
"version": "2.3.1",
# a scope attribute whose own name carries a `scope.` prefix
"attributes": {"telemetry.sdk.language": "go", "scope.prefixed": "prefixed-val"},
},
),
Traces(
timestamp=now - timedelta(seconds=2),
duration=timedelta(seconds=1),
trace_id=trace_id,
span_id=span_ids[1],
parent_span_id="",
name="POST /pay",
kind=TracesKind.SPAN_KIND_SERVER,
status_code=TracesStatusCode.STATUS_CODE_OK,
resources={"service.name": "payment"},
attributes={"http.request.method": "POST"},
# env.tier is a scope attribute here (cross-context with span 0);
# `name` is a scope attribute colliding with span 0's scope.name.
scope={
"name": "io.signoz.payment",
"version": "4.5.6",
"attributes": {
"telemetry.sdk.language": "python",
"env.tier": "gold",
"name": "io.signoz.checkout",
},
},
),
Traces(
timestamp=now - timedelta(seconds=1),
duration=timedelta(seconds=1),
trace_id=trace_id,
span_id=span_ids[2],
parent_span_id="",
# span name collides with span 0's scope.name value
name="io.signoz.checkout",
kind=TracesKind.SPAN_KIND_SERVER,
status_code=TracesStatusCode.STATUS_CODE_OK,
resources={"service.name": "probe"},
# a span attribute named `scope.name`
attributes={"scope.name": "attr-scope-name"},
scope={"name": "span-gamma", "version": "9.9.9"},
),
]
insert_traces(traces)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
start_ms = int((now - timedelta(minutes=1)).timestamp() * 1000)
end_ms = int((now + timedelta(seconds=1)).timestamp() * 1000)
response = make_query_request(
signoz,
token,
start_ms=start_ms,
end_ms=end_ms,
request_type=RequestType.RAW,
queries=[
BuilderQuery(
signal="traces",
name="A",
select_fields=[TelemetryFieldKey("timestamp")],
filter_expression=filter_expression,
limit=10,
).to_dict()
],
)
assert response.status_code == HTTPStatus.OK, response.text
got_span_ids = {row["data"]["span_id"] for row in get_rows(response)}
expected_span_ids = {traces[i].span_id for i in expected_indices}
assert got_span_ids == expected_span_ids
@pytest.mark.parametrize("surface", ["filter", "select", "order"])
def test_traces_list_unknown_span_context_synthesizes(
signoz: types.SigNoz,