Compare commits

...

1 Commits

Author SHA1 Message Date
srikanthccv
8b79b7aed7 feat(querybuilder): resolve semconv families across logs and metrics
Phase 2 of #6143, behind the resolve_semconv_families flag (default off),
rebuilt on the storage contract.

- Registry: each member carries the scope of its own rename edges. A
  fan-out keeps one membership per target, an unscoped rename stays
  unbounded, and an ambiguous name stays literal. Family fields are
  unexported, All() iterates, and make semconv-check backs CI.
- Gate: the family lookup applies to every signal. A string entry of a
  family signal under the resource or attribute context groups into the
  family. The traces storage keeps its behaviour.
- Logs: filters, select fields, group by, order by, and aggregations
  resolve the family through the shared read. The mid-migration state
  reads the one stored spelling.
- Metrics: label families expand into the stored spellings (dotted,
  normalized, resource_-prefixed). Every metric_name filter unions the
  storage names of a metric-name family. The querier resolves type and
  temporality through the same union. A family of labels keeps the
  metrics keyless contract: no guard, and no NULL group.
- Values suggestions union the family spellings. Related values resolve
  the family in the selected column, the search narrowing, and the
  existing-query filter.
- The legacy metric spellings live in a deletable shim. transition.go is
  removed; the v3 and v4 readers resolve the three CPU metric renames
  through the registry.
- db.system.name is disabled: its value domain also renamed, and no value
  mapping is read yet.
- Integration: the semconvfamilies suite covers the logs matrix, the
  metric label and metric-name families in every storage layout, and the
  fields/keys and fields/values pins.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014F37Rwd6xYrr5Abi6JJqJW
2026-09-15 20:28:00 +05:30
39 changed files with 2674 additions and 536 deletions

View File

@@ -67,7 +67,7 @@ jobs:
with:
go-version: "1.24"
- name: check-semconv-generated-files
run: go run ./scripts/semconv -check
run: make semconv-check
build:
if: |
github.event_name == 'merge_group' ||

View File

@@ -262,6 +262,10 @@ py-clean: ## Clear all pycache and pytest cache from tests directory recursively
semconv-generate: ## Regenerate semantic-convention families for Go and TypeScript
@go run ./scripts/semconv
.PHONY: semconv-check
semconv-check: ## Fail if the generated semantic-convention files are stale
@go run ./scripts/semconv -check
.PHONY: gen-mocks
gen-mocks:
@echo ">> Generating mocks"

View File

@@ -84,9 +84,9 @@ A storage answers four questions and nothing else:
| WhenAbsent | Absent row reads | Positive filter | Raw select | Multi-candidate column | Field keys |
|---|---|---|---|---|---|
| `AlwaysPresent` | a real value | no guard | no guard | no branch, ends the candidate list | table columns |
| `AbsentIsSentinel` | `''`, 0, false, and that is not a value | exists guard | exists guard | presence branch | map attributes, cast JSON paths, string families |
| `AbsentIsSentinel` | `''`, 0, false, and that is not a value | exists guard | exists guard | presence branch | map attributes, cast JSON paths, string families of such members |
| `AbsentIsNull` | NULL | no guard | no guard | presence branch | multi-era folds, body JSON paths, numeric families |
| `AbsentIsValue` | `''`, and that is the keyless contract | no guard | no guard | no presence branch | metrics labels, rule state history labels |
| `AbsentIsValue` | `''`, and that is the keyless contract | no guard | no guard | no presence branch | metrics labels, rule state history labels, and families of such members |
### The generic layer
@@ -109,7 +109,7 @@ The functions, from the outside in:
| `RejectsBodyFunction(traits, operator)` | Runs before resolution. A storage without body functions (`has`, `hasAny`, `hasAll`, `hasToken`, `search`) errors. The fingerprint side of a split skips the term, because the main query evaluates it. After resolution, `Condition` errors when `has`, `hasAny`, `hasAll`, or `hasToken` lands on a map-backed key (resource, attribute, scope), before the split can drop it. |
| `SharedCondition(...)` | The `Compile` of every storage without its own condition language: `LogicalRead`, the shared data-type collision cast, `OperatorCondition`, then the guard rule. |
| `OperatorCondition(...)` | The operator switch over an already cast read. A storage with its own cast policy composes with it. |
| `LogicalRead(...)` | The only place family expressions are built. A single-member field reads through its member. A family merges the member reads, current member first: `COALESCE(NULLIF(m1, ''), NULLIF(m2, ''), '')` for strings, `multiIf` with a NULL tail for numbers. It ORs the member presence tests. A row without any member reads what the tail of the merge reads. A member with a value map reads through `TransformRead`. `NOT EXISTS` is the read's `Absence`, the storage's own negated form. |
| `LogicalRead(...)` | The only place family expressions are built. A single-member field reads through its member. A family merges the member reads, current member first: `COALESCE(NULLIF(m1, ''), NULLIF(m2, ''), '')` for strings, `multiIf` with a NULL tail for numbers. It ORs the member presence tests. A row without any member reads what the tail of the merge reads. When every member reads its sentinel as a value, so does the family. A member with a value map reads through `TransformRead`. `NOT EXISTS` is the read's `Absence`, the storage's own negated form. |
### A resolved key

View File

@@ -1,32 +1,82 @@
// Code generated by scripts/semconv. DO NOT EDIT.
export type SemconvFamily = {
readonly current: string;
readonly old: readonly string[];
readonly kind: 'attribute' | 'metric';
// An empty contexts/signals/applyToMetrics array places no constraint on
// that axis.
export type SemconvMember = {
readonly name: string;
readonly contexts: readonly string[];
readonly signals: readonly string[];
readonly applyToMetrics: readonly string[];
};
export type SemconvFamily = {
readonly current: string;
readonly kind: 'attribute' | 'metric';
readonly members: readonly SemconvMember[];
readonly contexts: readonly string[];
readonly signals: readonly string[];
readonly valueMap: Readonly<Record<string, string>>;
};
export const SEMCONV_FAMILIES: readonly SemconvFamily[] = [
{
current: 'db.system.name',
old: ['db.system'],
kind: 'attribute',
current: 'container.cpu.usage',
kind: 'metric',
members: [
{
name: 'container.cpu.utilization',
contexts: [],
signals: [],
applyToMetrics: [],
},
],
contexts: [],
signals: [],
applyToMetrics: [],
valueMap: {},
},
{
current: 'deployment.environment.name',
old: ['deployment.environment'],
kind: 'attribute',
members: [
{
name: 'deployment.environment',
contexts: [],
signals: [],
applyToMetrics: [],
},
],
contexts: ['attribute', 'resource'],
signals: ['logs', 'metrics', 'traces'],
valueMap: {},
},
{
current: 'k8s.node.cpu.usage',
kind: 'metric',
members: [
{
name: 'k8s.node.cpu.utilization',
contexts: [],
signals: [],
applyToMetrics: [],
},
],
contexts: [],
signals: [],
valueMap: {},
},
{
current: 'k8s.pod.cpu.usage',
kind: 'metric',
members: [
{
name: 'k8s.pod.cpu.utilization',
contexts: [],
signals: [],
applyToMetrics: [],
},
],
contexts: [],
signals: [],
applyToMetrics: [],
valueMap: {},
},
] as const;

View File

@@ -93,7 +93,7 @@ func MustNewRegistry() featuretypes.Registry {
Name: FeatureResolveSemconvFamilies,
Kind: featuretypes.KindBoolean,
Stage: featuretypes.StageExperimental,
Description: "Controls whether trace queries resolve a semantic-convention name to all the spellings of its family",
Description: "Controls whether trace, log, and metric queries resolve a semantic-convention name to all the spellings of its family",
DefaultVariant: featuretypes.MustNewName("disabled"),
Variants: featuretypes.NewBooleanVariants(),
},

View File

@@ -435,6 +435,7 @@ func (m *module) buildFilterClause(ctx context.Context, orgID valuer.UUID, filte
whereClauseSelectors[idx].SelectorMatchType = telemetrytypes.FieldSelectorMatchTypeExact
}
whereClauseSelectors = querybuilder.ExpandKeySelectorsForFamilies(ctx, orgID, m.fl, whereClauseSelectors)
keys, _, err := m.telemetryMetadataStore.GetKeysMulti(ctx, orgID, whereClauseSelectors)
if err != nil {
return nil, err

View File

@@ -965,6 +965,7 @@ func (m *module) buildFilterClause(ctx context.Context, orgID valuer.UUID, filte
// whereClauseSelectors[idx].Source = query.Source
}
whereClauseSelectors = querybuilder.ExpandKeySelectorsForFamilies(ctx, orgID, m.fl, whereClauseSelectors)
keys, _, err := m.telemetryMetadataStore.GetKeysMulti(ctx, orgID, whereClauseSelectors)
if err != nil {
return nil, err

View File

@@ -422,6 +422,9 @@ func (q *querier) populateQBEvent(event *qbtypes.QBEvent, queries []qbtypes.Quer
// resolved: never-seen metrics and dormant metrics (seen but no data in
// the query window).
// - err: Internal when a metadata fetch fails.
//
// Metric metadata resolves through every storage name of a metric-name
// family, the same names the statement builder unions.
func (q *querier) resolveMetricMetadata(ctx context.Context, orgID valuer.UUID, queries []qbtypes.QueryEnvelope, start, end uint64, requestType qbtypes.RequestType) (missingMetricQueries []string, metricWarnings []string, err error) {
metricNames := make([]string, 0)
for idx := range queries {
@@ -434,7 +437,7 @@ func (q *querier) resolveMetricMetadata(ctx context.Context, orgID valuer.UUID,
}
for _, agg := range spec.Aggregations {
if agg.MetricName != "" {
metricNames = append(metricNames, agg.MetricName)
metricNames = append(metricNames, querybuilder.FamilyMetricNames(ctx, orgID, q.fl, agg.MetricName)...)
}
}
}
@@ -463,13 +466,19 @@ func (q *querier) resolveMetricMetadata(ctx context.Context, orgID valuer.UUID,
presentAggregations := make([]qbtypes.MetricAggregation, 0, len(spec.Aggregations))
for i := range spec.Aggregations {
if spec.Aggregations[i].MetricName != "" && spec.Aggregations[i].Temporality == metrictypes.Unknown {
if temp, ok := metricTemporality[spec.Aggregations[i].MetricName]; ok && temp != metrictypes.Unknown {
spec.Aggregations[i].Temporality = temp
for _, member := range querybuilder.FamilyMetricNames(ctx, orgID, q.fl, spec.Aggregations[i].MetricName) {
if temp, ok := metricTemporality[member]; ok && temp != metrictypes.Unknown {
spec.Aggregations[i].Temporality = temp
break
}
}
}
if spec.Aggregations[i].MetricName != "" && spec.Aggregations[i].Type == metrictypes.UnspecifiedType {
if foundMetricType, ok := metricTypes[spec.Aggregations[i].MetricName]; ok && foundMetricType != metrictypes.UnspecifiedType {
spec.Aggregations[i].Type = foundMetricType
for _, member := range querybuilder.FamilyMetricNames(ctx, orgID, q.fl, spec.Aggregations[i].MetricName) {
if foundMetricType, ok := metricTypes[member]; ok && foundMetricType != metrictypes.UnspecifiedType {
spec.Aggregations[i].Type = foundMetricType
break
}
}
}
if spec.Aggregations[i].Type == metrictypes.UnspecifiedType {

View File

@@ -51,10 +51,10 @@ import (
"github.com/SigNoz/signoz/pkg/query-service/constants"
chErrors "github.com/SigNoz/signoz/pkg/query-service/errors"
"github.com/SigNoz/signoz/pkg/query-service/metrics"
"github.com/SigNoz/signoz/pkg/query-service/model"
v3 "github.com/SigNoz/signoz/pkg/query-service/model/v3"
"github.com/SigNoz/signoz/pkg/query-service/utils"
"github.com/SigNoz/signoz/pkg/semconv"
)
const (
@@ -3199,7 +3199,14 @@ func (r *ClickHouseReader) GetMetricAttributeValues(ctx context.Context, orgID v
query = query + fmt.Sprintf(" LIMIT %d;", req.Limit)
}
names := []string{req.AggregateAttribute}
names = append(names, metrics.GetTransitionedMetric(req.AggregateAttribute))
current := semconv.Current(semconv.KindMetric, telemetrytypes.FieldKeySelector{
Name: req.AggregateAttribute,
Signal: telemetrytypes.SignalMetrics,
FieldContext: telemetrytypes.FieldContextMetric,
})
if current != req.AggregateAttribute {
names = append(names, current)
}
rows, err = r.db.Query(ctx, query, req.FilterAttributeKey, names, req.FilterAttributeKey, fmt.Sprintf("%%%s%%", req.SearchText), common.PastDayRoundOff())

View File

@@ -1,14 +0,0 @@
package metrics
var MetricsUnderTransition = map[string]string{
"k8s.pod.cpu.utilization": "k8s.pod.cpu.usage",
"k8s.node.cpu.utilization": "k8s.node.cpu.usage",
"container.cpu.utilization": "container.cpu.usage",
}
func GetTransitionedMetric(metric string) string {
if transitionedMetric, ok := MetricsUnderTransition[metric]; ok {
return transitionedMetric
}
return metric
}

View File

@@ -10,8 +10,9 @@ import (
"log/slog"
"github.com/SigNoz/signoz/pkg/query-service/constants"
"github.com/SigNoz/signoz/pkg/query-service/metrics"
v3 "github.com/SigNoz/signoz/pkg/query-service/model/v3"
"github.com/SigNoz/signoz/pkg/semconv"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// ValidateAndCastValue validates and casts the value of a key to the corresponding data type of the key
@@ -234,12 +235,12 @@ func ClickHouseFormattedValue(v interface{}) string {
func ClickHouseFormattedMetricNames(v interface{}) string {
if name, ok := v.(string); ok {
transitionedMetrics := metrics.GetTransitionedMetric(name)
if transitionedMetrics != name {
return ClickHouseFormattedValue([]interface{}{transitionedMetrics})
} else {
return ClickHouseFormattedValue([]interface{}{name})
}
current := semconv.Current(semconv.KindMetric, telemetrytypes.FieldKeySelector{
Name: name,
Signal: telemetrytypes.SignalMetrics,
FieldContext: telemetrytypes.FieldContextMetric,
})
return ClickHouseFormattedValue([]interface{}{current})
}
return ClickHouseFormattedValue(v)

View File

@@ -483,3 +483,22 @@ func TestGetEpochNanoSecs(t *testing.T) {
})
}
}
// Only the canonical dotted spelling of a metric-name family redirects on the
// legacy path. A normalized spelling keeps reading its own series.
func TestClickHouseFormattedMetricNames(t *testing.T) {
cases := []struct {
name string
expected string
}{
{name: "k8s.pod.cpu.utilization", expected: "['k8s.pod.cpu.usage']"},
{name: "k8s.pod.cpu.usage", expected: "['k8s.pod.cpu.usage']"},
{name: "k8s_pod_cpu_utilization", expected: "['k8s_pod_cpu_utilization']"},
{name: "http.server.duration", expected: "['http.server.duration']"},
}
for _, c := range cases {
if got := ClickHouseFormattedMetricNames(c.name); got != c.expected {
t.Errorf("ClickHouseFormattedMetricNames(%q) = %q, want %q", c.name, got, c.expected)
}
}
}

View File

@@ -4,57 +4,61 @@ import (
"context"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/semconv"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
// semconvFamiliesEnabled evaluates the resolve_semconv_families flag for the
// SemconvFamiliesEnabled evaluates the resolve_semconv_families flag for the
// org. A nil flagger means off, so a caller without family support stays
// literal by default.
func semconvFamiliesEnabled(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger) bool {
func SemconvFamiliesEnabled(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger) bool {
if fl == nil {
return false
}
return fl.BooleanOrEmpty(ctx, flagger.FeatureResolveSemconvFamilies, featuretypes.NewFlaggerEvaluationContext(orgID))
}
// ExpandKeySelectorsForFamilies adds selectors for the other members of each
// semantic-convention family that a selector names. The metadata fetched for
// a query then contains each spelling that MatchingLogicalFields can group.
// This function is the prefetch of the resolution layer: statement builders
// call it after they derive the selectors, and the metadata store stays
// family-blind (autocomplete responses keep the literal spelling that the
// user typed). It does nothing when the resolve_semconv_families flag is off
// for the org. Only trace selectors expand today, because that matches the
// family support. Fuzzy (search-style) selectors never expand.
// ExpandKeySelectorsForFamilies adds selectors for the other spellings of
// each semantic-convention family that a selector names. The metadata fetched
// for a query then contains each spelling that MatchingLogicalFields can
// group. This function is the prefetch of the resolution layer: statement
// builders call it after they derive the selectors, and the metadata store
// stays family-blind (autocomplete responses keep the literal spelling that
// the user typed). It does nothing when the resolve_semconv_families flag is
// off for the org. Fuzzy (search-style) selectors never expand.
//
// Every call site pairs this prefetch with Resolve at query time. A site
// without the prefetch degrades soft. The metadata lacks the sibling, and
// the name stays literal. It never merges wrong.
func ExpandKeySelectorsForFamilies(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger, selectors []*telemetrytypes.FieldKeySelector) []*telemetrytypes.FieldKeySelector {
if !semconvFamiliesEnabled(ctx, orgID, fl) {
if !SemconvFamiliesEnabled(ctx, orgID, fl) {
return selectors
}
// Two selectors can share a name under different contexts, signals, or
// data types, and each needs its own sibling selectors.
out := selectors
seen := make(map[string]bool, len(selectors))
for _, selector := range selectors {
seen[selector.Name] = true
seen[selector.Identity(selector.Name)] = true
}
for _, selector := range selectors {
if selector.Signal != telemetrytypes.SignalTraces ||
selector.SelectorMatchType == telemetrytypes.FieldSelectorMatchTypeFuzzy {
if selector.SelectorMatchType == telemetrytypes.FieldSelectorMatchTypeFuzzy {
continue
}
members := semconv.Members(semconv.KindAttribute, telemetrytypes.FieldKeySelector{
Name: selector.Name,
Signal: selector.Signal,
FieldContext: selector.FieldContext,
members := familySpellings(telemetrytypes.FieldKeySelector{
Name: selector.Name,
Signal: selector.Signal,
FieldContext: selector.FieldContext,
MetricContext: selector.MetricContext,
})
for _, member := range members {
if seen[member] {
if seen[selector.Identity(member)] {
continue
}
seen[member] = true
seen[selector.Identity(member)] = true
expanded := *selector
expanded.Name = member
out = append(out, &expanded)

View File

@@ -0,0 +1,179 @@
package querybuilder
import (
"context"
"strings"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/semconv"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
// Legacy metric storage spellings. The normalized era wrote label and metric
// names with underscores, and the span-metrics pipeline wrote resource
// attributes with a resource_ prefix. The vocabulary in pkg/semconv knows
// only the canonical dotted names.
const legacyResourcePrefix = "resource_"
// MetricLabelSpellings returns the storage spellings that may hold
// selector.Name in metric labels: every admitted family member expanded into
// its dotted, normalized, and resource_-prefixed layouts, ordered
// member-major with the requested shape's layout first. A name outside an
// enabled family is returned unchanged. So is a name the selector leaves
// ambiguous.
func MetricLabelSpellings(selector telemetrytypes.FieldKeySelector) []string {
lookupSelector := selector
lookupSelector.Name = strings.TrimPrefix(selector.Name, legacyResourcePrefix)
members, style, ok := metricVocabulary(semconv.KindAttribute, lookupSelector)
if !ok {
return []string{selector.Name}
}
result := make([]string, 0, len(members)*4)
resourceFirst := selector.FieldContext != telemetrytypes.FieldContextAttribute ||
strings.HasPrefix(selector.Name, legacyResourcePrefix)
for _, member := range members {
variants := []string{member, normalizedMetricSpelling(member)}
if style == legacySpellingNormalized {
variants[0], variants[1] = variants[1], variants[0]
}
if resourceFirst {
for _, variant := range variants {
result = appendUniqueSpelling(result, legacyResourcePrefix+variant)
}
}
for _, variant := range variants {
result = appendUniqueSpelling(result, variant)
}
if !resourceFirst {
for _, variant := range variants {
result = appendUniqueSpelling(result, legacyResourcePrefix+variant)
}
}
}
return result
}
// MetricNameSpellings returns the storage names of a metric-name family in
// the requested layout: both layouts are valid metric identities and must not
// be mixed in one query.
func MetricNameSpellings(name string) []string {
selector := telemetrytypes.FieldKeySelector{
Name: name,
Signal: telemetrytypes.SignalMetrics,
FieldContext: telemetrytypes.FieldContextMetric,
}
members, style, ok := metricVocabulary(semconv.KindMetric, selector)
if !ok {
return []string{name}
}
result := make([]string, 0, len(members))
for _, member := range members {
if style == legacySpellingNormalized {
member = normalizedMetricSpelling(member)
}
result = appendUniqueSpelling(result, member)
}
return result
}
type legacySpelling int
const (
legacySpellingDotted legacySpelling = iota
legacySpellingNormalized
)
// metricVocabulary resolves a metric spelling to its family members: first as
// the canonical dotted name, then by comparing the normalized layout of every
// vocabulary spelling. The ambiguity rule of the vocabulary applies to both
// layouts: a name that admits several families stays unresolved.
func metricVocabulary(kind semconv.Kind, selector telemetrytypes.FieldKeySelector) ([]string, legacySpelling, bool) {
if members := semconv.Members(kind, selector); len(members) > 1 {
return members, legacySpellingDotted, true
}
dotted, ok := denormalizedName(kind, selector)
if !ok {
return nil, legacySpellingDotted, false
}
dottedSelector := selector
dottedSelector.Name = dotted
members := semconv.Members(kind, dottedSelector)
if len(members) <= 1 {
return nil, legacySpellingDotted, false
}
return members, legacySpellingNormalized, true
}
// normalizedVocabulary indexes every vocabulary spelling by kind and
// normalized layout. Each family gives one candidate, the first of its
// spellings with that layout. Cross-family duplicates stay, so the
// ambiguity count below sees every family that carries the layout.
var normalizedVocabulary = buildNormalizedVocabulary()
func buildNormalizedVocabulary() map[semconv.Kind]map[string][]string {
index := make(map[semconv.Kind]map[string][]string)
for family := range semconv.All() {
if index[family.Kind()] == nil {
index[family.Kind()] = make(map[string][]string)
}
seen := make(map[string]bool)
for _, name := range append([]string{family.Current()}, family.Old()...) {
normalized := normalizedMetricSpelling(name)
if seen[normalized] {
continue
}
seen[normalized] = true
index[family.Kind()][normalized] = append(index[family.Kind()][normalized], name)
}
}
return index
}
// denormalizedName maps a normalized spelling back to its unique canonical
// vocabulary name. The reverse mapping is lossy in general (a dot and an
// underscore normalize identically), so only an unambiguous match resolves.
func denormalizedName(kind semconv.Kind, selector telemetrytypes.FieldKeySelector) (string, bool) {
found, foundName := 0, ""
for _, name := range normalizedVocabulary[kind][selector.Name] {
probe := selector
probe.Name = name
if len(semconv.Members(kind, probe)) > 1 {
found++
foundName = name
}
}
if found != 1 {
return "", false
}
return foundName, true
}
func normalizedMetricSpelling(name string) string {
return strings.ReplaceAll(name, ".", "_")
}
func appendUniqueSpelling(values []string, value string) []string {
for _, existing := range values {
if existing == value {
return values
}
}
return append(values, value)
}
// FamilyMetricNames returns the storage names a metric query must read: the
// requested name plus the other spellings of its metric-name family when the
// resolve_semconv_families flag is on for the org.
func FamilyMetricNames(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger, metricName string) []string {
if !SemconvFamiliesEnabled(ctx, orgID, fl) {
return []string{metricName}
}
return MetricNameSpellings(metricName)
}

View File

@@ -0,0 +1,54 @@
package querybuilder
import (
"testing"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/stretchr/testify/assert"
)
func TestMetricLabelSpellingsExpandsLegacyLayouts(t *testing.T) {
selector := telemetrytypes.FieldKeySelector{
Name: "deployment.environment",
Signal: telemetrytypes.SignalMetrics,
FieldContext: telemetrytypes.FieldContextResource,
}
assert.Equal(t, []string{
"resource_deployment.environment.name", "resource_deployment_environment_name",
"deployment.environment.name", "deployment_environment_name",
"resource_deployment.environment", "resource_deployment_environment",
"deployment.environment", "deployment_environment",
}, MetricLabelSpellings(selector), "members expand into dotted, normalized, and resource_-prefixed layouts")
}
func TestMetricLabelSpellingsPreservesRequestedLayout(t *testing.T) {
selector := telemetrytypes.FieldKeySelector{
Name: "resource_deployment_environment",
Signal: telemetrytypes.SignalMetrics,
FieldContext: telemetrytypes.FieldContextAttribute,
}
assert.Equal(t, []string{
"resource_deployment_environment_name", "resource_deployment.environment.name",
"deployment_environment_name", "deployment.environment.name",
"resource_deployment_environment", "resource_deployment.environment",
"deployment_environment", "deployment.environment",
}, MetricLabelSpellings(selector), "a normalized request lists normalized layouts first and keeps resource_ variants")
}
func TestMetricLabelSpellingsStaysLiteralOutsideTheVocabulary(t *testing.T) {
selector := telemetrytypes.FieldKeySelector{
Name: "http.route",
Signal: telemetrytypes.SignalMetrics,
}
assert.Equal(t, []string{"http.route"}, MetricLabelSpellings(selector))
}
func TestMetricNameSpellingsPreservesLayout(t *testing.T) {
assert.Equal(t, []string{"k8s.pod.cpu.usage", "k8s.pod.cpu.utilization"}, MetricNameSpellings("k8s.pod.cpu.utilization"))
assert.Equal(t, []string{"k8s_pod_cpu_usage", "k8s_pod_cpu_utilization"}, MetricNameSpellings("k8s_pod_cpu_utilization"))
assert.Equal(t, []string{"k8s.pod.cpu.usage", "k8s.pod.cpu.utilization"}, MetricNameSpellings("k8s.pod.cpu.usage"))
assert.Equal(t, []string{"http.server.duration"}, MetricNameSpellings("http.server.duration"))
}

View File

@@ -20,8 +20,9 @@ import (
// member reads, current member first. It is present when any member is
// present, and absent when no member is present. A row without any member
// reads what the tail of the merge reads: the sentinel for a string family,
// NULL for the others. A member with a value map reads in the current
// vocabulary.
// NULL for the others. When every member reads its sentinel as a value, so
// does the family, and the keyless contract of the signal survives the
// merge. A member with a value map reads in the current vocabulary.
func LogicalRead(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, logical *telemetrytypes.LogicalField) (qbtypes.Read, error) {
if !logical.IsFamily() {
return memberRead(ctx, q, storage, logical, 0)
@@ -35,7 +36,7 @@ func LogicalRead(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Stora
reads = append(reads, read)
}
merged := qbtypes.Read{WhenAbsent: familyAbsence(logical)}
merged := qbtypes.Read{WhenAbsent: familyAbsence(logical, reads)}
guards := make([]string, 0, len(reads))
for _, read := range reads {
guards = append(guards, read.Presence)
@@ -97,10 +98,16 @@ func clickHouseStringArray(values []string) string {
}
// familyAbsence is what the merged read yields for a row without any
// member: the sentinel tail of a string family, NULL for the others.
func familyAbsence(logical *telemetrytypes.LogicalField) qbtypes.Absent {
if logical.FieldDataType == telemetrytypes.FieldDataTypeString {
return qbtypes.AbsentIsSentinel
// member: the sentinel tail of a string family, NULL for the others. When
// every member's sentinel is a value, the tail is one too.
func familyAbsence(logical *telemetrytypes.LogicalField, reads []qbtypes.Read) qbtypes.Absent {
if logical.FieldDataType != telemetrytypes.FieldDataTypeString {
return qbtypes.AbsentIsNull
}
return qbtypes.AbsentIsNull
for _, read := range reads {
if read.WhenAbsent != qbtypes.AbsentIsValue {
return qbtypes.AbsentIsSentinel
}
}
return qbtypes.AbsentIsValue
}

View File

@@ -48,7 +48,7 @@ func TestFamiliesOffByDefault(t *testing.T) {
}},
}
fields := matchingLogicalFields(false, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
fields := matchingLogicalFields(false, telemetrytypes.SignalUnspecified, nil, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
require.Len(t, fields, 1)
assert.False(t, fields[0].IsFamily())
assert.Equal(t, []string{"deployment.environment.name"}, memberNames(fields[0]))
@@ -76,7 +76,7 @@ func TestMatchingLogicalFieldsGroupsFamilyMembers(t *testing.T) {
}
for _, requested := range []string{"deployment.environment.name", "deployment.environment"} {
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: requested}, fieldKeys)
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, nil, &telemetrytypes.TelemetryFieldKey{Name: requested}, fieldKeys)
require.Len(t, fields, 1, "a family is one logical field, requested via %s", requested)
logical := fields[0]
assert.Equal(t, requested, logical.Name, "response identity is the requested spelling")
@@ -106,7 +106,7 @@ func TestMatchingLogicalFieldsOrdersMembersByFamilyRank(t *testing.T) {
}},
}
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, nil, &telemetrytypes.TelemetryFieldKey{
Name: "deployment.environment.name",
FieldContext: telemetrytypes.FieldContextResource,
}, fieldKeys)
@@ -115,9 +115,8 @@ func TestMatchingLogicalFieldsOrdersMembersByFamilyRank(t *testing.T) {
assert.Equal(t, []string{"resource.deployment.environment.name", "deployment.environment"}, memberNames(fields[0]))
}
// Non-trace signals have no family support: the requested spelling stays
// literal, and a family member name never pulls in its siblings.
func TestMatchingLogicalFieldsKeepsLogsLiteral(t *testing.T) {
// Log entries group into families exactly like trace entries.
func TestMatchingLogicalFieldsGroupsLogEntries(t *testing.T) {
logsKey := func(name string) *telemetrytypes.TelemetryFieldKey {
return &telemetrytypes.TelemetryFieldKey{
Name: name,
@@ -131,10 +130,61 @@ func TestMatchingLogicalFieldsKeepsLogsLiteral(t *testing.T) {
"deployment.environment": {logsKey("deployment.environment")},
}
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
fields := matchingLogicalFields(true, telemetrytypes.SignalLogs, nil, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
require.Len(t, fields, 1)
assert.False(t, fields[0].IsFamily())
assert.Equal(t, []string{"deployment.environment.name"}, memberNames(fields[0]))
assert.True(t, fields[0].IsFamily())
assert.Equal(t, []string{"deployment.environment.name", "deployment.environment"}, memberNames(fields[0]))
}
// Metric entries group across the stored label spellings of the family, in
// member-major order: every spelling of the current name precedes the first
// spelling of the old one.
func TestMatchingLogicalFieldsGroupsMetricSpellings(t *testing.T) {
metricsKey := func(name string) *telemetrytypes.TelemetryFieldKey {
return &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalMetrics,
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeString,
}
}
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {metricsKey("deployment.environment.name")},
"deployment_environment_name": {metricsKey("deployment_environment_name")},
"deployment.environment": {metricsKey("deployment.environment")},
"deployment_environment": {metricsKey("deployment_environment")},
}
fields := matchingLogicalFields(true, telemetrytypes.SignalMetrics, nil, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment"}, fieldKeys)
require.Len(t, fields, 1)
assert.True(t, fields[0].IsFamily())
assert.Equal(t, []string{
"deployment.environment.name", "deployment_environment_name",
"deployment.environment", "deployment_environment",
}, memberNames(fields[0]))
}
// A non-string entry never joins a family: the merged read has no common
// ClickHouse type across the storages.
func TestMatchingLogicalFieldsKeepsNumberEntriesSingle(t *testing.T) {
numberKey := func(name string) *telemetrytypes.TelemetryFieldKey {
return &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalMetrics,
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeNumber,
}
}
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {numberKey("deployment.environment.name")},
"deployment.environment": {numberKey("deployment.environment")},
}
fields := matchingLogicalFields(true, telemetrytypes.SignalMetrics, nil, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment"}, fieldKeys)
require.Len(t, fields, 2)
for _, logical := range fields {
assert.False(t, logical.IsFamily())
}
}
// A family and a genuine same-name collision stack cleanly: the family stays
@@ -165,7 +215,7 @@ func TestResolveLogicalFieldsKeepsFamilyThroughAmbiguity(t *testing.T) {
}
requested := &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, requested, fieldKeys)
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, nil, requested, fieldKeys)
require.Len(t, fields, 2, "resource family + attribute collision")
resolved, warning := ResolveLogicalFields(requested, fields)
@@ -193,7 +243,7 @@ func TestMatchingLogicalFieldsNeverMergesAcrossDataTypes(t *testing.T) {
}},
}
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, nil, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
require.Len(t, fields, 2)
for _, logical := range fields {
assert.False(t, logical.IsFamily())
@@ -218,11 +268,15 @@ func TestExpandKeySelectorsForFamilies(t *testing.T) {
"service.name",
"deployment.environment.name",
"deployment.environment",
}, names, "one sibling selector for the trace family member; logs and non-family names untouched")
"deployment.environment",
}, names, "each selector identity gets its own sibling, and a non-family name stays untouched")
sibling := expanded[len(expanded)-1]
assert.Equal(t, telemetrytypes.SignalTraces, sibling.Signal)
assert.Equal(t, telemetrytypes.FieldSelectorMatchTypeExact, sibling.SelectorMatchType)
tracesSibling := expanded[len(expanded)-2]
assert.Equal(t, telemetrytypes.SignalTraces, tracesSibling.Signal)
assert.Equal(t, telemetrytypes.FieldSelectorMatchTypeExact, tracesSibling.SelectorMatchType)
logsSibling := expanded[len(expanded)-1]
assert.Equal(t, telemetrytypes.SignalLogs, logsSibling.Signal,
"a same-named selector under another signal must not take the sibling")
}
func TestExpandKeySelectorsForFamiliesDeduplicatesAndSkipsFuzzy(t *testing.T) {

View File

@@ -21,7 +21,7 @@ func NewQueryInfo(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger, si
EndNs: endNs,
Signal: signal,
Metric: metric,
FamiliesOn: semconvFamiliesEnabled(ctx, orgID, fl),
FamiliesOn: SemconvFamiliesEnabled(ctx, orgID, fl),
}
if fl != nil {
q.BodyJSONOn = fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
@@ -59,7 +59,7 @@ func Resolve(
traits := storage.Traits()
lookup := key
matches := matchingLogicalFields(q.FamiliesOn, q.Signal, key, fieldKeys)
matches := matchingLogicalFields(q.FamiliesOn, q.Signal, q.Metric, key, fieldKeys)
if len(matches) == 0 && slices.Contains(traits.OwnContexts, key.FieldContext) {
// a column the storage knows under the key's own context is the key
// as written, and only a miss corrects to the bare spelling
@@ -69,7 +69,7 @@ func Resolve(
}
}
lookup = telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextUnspecified, key.FieldDataType)
matches = matchingLogicalFields(q.FamiliesOn, q.Signal, lookup, fieldKeys)
matches = matchingLogicalFields(q.FamiliesOn, q.Signal, q.Metric, lookup, fieldKeys)
}
resolved := qbtypes.Resolved{Key: key, Ambiguous: len(matches) > 1}

View File

@@ -1013,30 +1013,39 @@ func assignIfEmpty(s *string, value string) {
}
// familyMemberNames returns the physical spellings to look up for the
// referenced key: the semantic-convention family members (current-first) when
// families are on and the query can resolve to traces, else just the requested
// name. Only the traces storage understands families today. Logs and
// metrics keep the requested spelling until theirs land.
func familyMemberNames(familiesOn bool, signal telemetrytypes.Signal, field *telemetrytypes.TelemetryFieldKey) []string {
// referenced key: the semantic-convention family spellings, current first,
// when families are on, else just the requested name. The key's own signal
// wins over the query's signal.
func familyMemberNames(familiesOn bool, signal telemetrytypes.Signal, metric *telemetrytypes.MetricContext, field *telemetrytypes.TelemetryFieldKey) []string {
if !familiesOn {
return []string{field.Name}
}
if signal != telemetrytypes.SignalUnspecified && signal != telemetrytypes.SignalTraces {
return []string{field.Name}
if field.Signal != telemetrytypes.SignalUnspecified {
signal = field.Signal
}
return semconv.Members(semconv.KindAttribute, telemetrytypes.FieldKeySelector{
Name: field.Name,
Signal: telemetrytypes.SignalTraces,
FieldContext: field.FieldContext,
return familySpellings(telemetrytypes.FieldKeySelector{
Name: field.Name,
Signal: signal,
FieldContext: field.FieldContext,
MetricContext: metric,
})
}
// familySpellings returns the storage spellings for the selector. The
// metrics signal expands each member into its stored label layouts.
func familySpellings(selector telemetrytypes.FieldKeySelector) []string {
if selector.Signal == telemetrytypes.SignalMetrics {
return MetricLabelSpellings(selector)
}
return semconv.Members(semconv.KindAttribute, selector)
}
// matchingLogicalFields resolves the referenced key against the metadata map
// into logical fields, honoring any context/data type the user specified.
//
// Physical keys that are members of one semantic-convention family (traces
// only today) group into one logical field per (signal, context, data type)
// identity, members ordered current-first. Every other matching key becomes
// Physical keys that are members of one semantic-convention family group
// into one logical field per (signal, context, data type) identity, members
// ordered current-first. Every other matching key becomes
// its own single-member logical field. Ambiguity is the length of the
// returned slice: one family is one element and is never ambiguous with
// itself, but the slice can hold several logical fields, including several
@@ -1046,9 +1055,9 @@ func familyMemberNames(familiesOn bool, signal telemetrytypes.Signal, field *tel
//
// Family grouping only happens when families are on for the query. Off,
// every match stays a single-member logical field.
func matchingLogicalFields(familiesOn bool, signal telemetrytypes.Signal, field *telemetrytypes.TelemetryFieldKey, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.LogicalField {
members := familyMemberNames(familiesOn, signal, field)
matches := collectMemberMatches(field, members, fieldKeys)
func matchingLogicalFields(familiesOn bool, signal telemetrytypes.Signal, metric *telemetrytypes.MetricContext, field *telemetrytypes.TelemetryFieldKey, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.LogicalField {
members := familyMemberNames(familiesOn, signal, metric, field)
matches := collectMemberMatches(field, members, metric, fieldKeys)
return groupIntoLogicalFields(field.Name, len(members) > 1, matches)
}
@@ -1074,32 +1083,29 @@ func matchesRequestedIdentity(field, item *telemetrytypes.TelemetryFieldKey, con
}
// inFamilyScope reports whether a match found under a sibling member name is
// legitimate: the entry must be trace metadata, and the member must be in the
// family of the requested name for the entry's context. A member lookup can
// otherwise find a same-named field in a scope where the family does not
// apply.
func inFamilyScope(field, item *telemetrytypes.TelemetryFieldKey, memberName string) bool {
if item.Signal != telemetrytypes.SignalTraces {
return false
}
return slices.Contains(semconv.Members(semconv.KindAttribute, telemetrytypes.FieldKeySelector{
Name: field.Name,
Signal: telemetrytypes.SignalTraces,
FieldContext: item.FieldContext,
// legitimate: the member must be a family spelling of the requested name for
// the entry's own signal and context. A member lookup can otherwise find a
// same-named field in a scope where the family does not apply.
func inFamilyScope(field, item *telemetrytypes.TelemetryFieldKey, memberName string, metric *telemetrytypes.MetricContext) bool {
return slices.Contains(familySpellings(telemetrytypes.FieldKeySelector{
Name: field.Name,
Signal: item.Signal,
FieldContext: item.FieldContext,
MetricContext: metric,
}), memberName)
}
// collectMemberMatches finds the metadata entries for every member spelling:
// first under the member names, then under their context-prefixed spellings
// (a context can be a legitimate part of a stored name, e.g. `attribute.key`).
func collectMemberMatches(field *telemetrytypes.TelemetryFieldKey, members []string, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) []memberMatch {
func collectMemberMatches(field *telemetrytypes.TelemetryFieldKey, members []string, metric *telemetrytypes.MetricContext, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) []memberMatch {
matches := make([]memberMatch, 0)
collect := func(lookupName string, rank int, memberName string, contextMatched bool) {
for _, item := range fieldKeys[lookupName] {
if !matchesRequestedIdentity(field, item, contextMatched) {
continue
}
if memberName != field.Name && !inFamilyScope(field, item, memberName) {
if memberName != field.Name && !inFamilyScope(field, item, memberName, metric) {
continue
}
matches = append(matches, memberMatch{key: item, rank: rank})
@@ -1117,18 +1123,21 @@ func collectMemberMatches(field *telemetrytypes.TelemetryFieldKey, members []str
return matches
}
// groupIntoLogicalFields turns matches into logical fields. Trace entries in
// family mode group by their (signal, context, data type) identity; every
// other entry becomes its own single-member field. Members sort by family
// rank at the end: precedence is a property of the family, not of the order
// in which the lookups found the members.
// groupIntoLogicalFields turns matches into logical fields. In family mode,
// a string entry of a family signal under the resource or attribute context
// groups by its (signal, context, data type) identity. Every other entry
// becomes its own single-member field. Members sort by family rank at the
// end: precedence is a property of the family, not of the order in which the
// lookups found the members.
func groupIntoLogicalFields(requestedName string, familyMode bool, matches []memberMatch) []*telemetrytypes.LogicalField {
fields := make([]*telemetrytypes.LogicalField, 0, len(matches))
groups := make(map[string]*telemetrytypes.LogicalField)
ranks := make(map[*telemetrytypes.TelemetryFieldKey]int)
for _, match := range matches {
if !familyMode || match.key.Signal != telemetrytypes.SignalTraces {
if !familyMode || !familySignal(match.key.Signal) ||
!familyFieldContext(match.key.FieldContext) ||
match.key.FieldDataType != telemetrytypes.FieldDataTypeString {
fields = append(fields, telemetrytypes.SingleLogicalField(requestedName, match.key))
continue
}
@@ -1168,3 +1177,19 @@ func groupHasMemberNamed(group *telemetrytypes.LogicalField, name string) bool {
}
return false
}
func familySignal(signal telemetrytypes.Signal) bool {
switch signal {
case telemetrytypes.SignalTraces, telemetrytypes.SignalLogs, telemetrytypes.SignalMetrics:
return true
}
return false
}
func familyFieldContext(fieldContext telemetrytypes.FieldContext) bool {
switch fieldContext {
case telemetrytypes.FieldContextResource, telemetrytypes.FieldContextAttribute:
return true
}
return false
}

View File

@@ -589,7 +589,7 @@ func TestVisitKey(t *testing.T) {
// and decides not-found handling. Replay that here against the generic
// builder behavior (error unless the key is ignored). The test maps carry
// no signal, so every logical field is single-member and flattens losslessly.
matching := matchingLogicalFields(false, telemetrytypes.SignalUnspecified, key, tt.fieldKeys)
matching := matchingLogicalFields(false, telemetrytypes.SignalUnspecified, nil, key, tt.fieldKeys)
resolved, warning := ResolveLogicalFields(key, matching)
keys := make([]*telemetrytypes.TelemetryFieldKey, 0, len(resolved))
for _, logical := range resolved {

View File

@@ -2,21 +2,37 @@
package semconv
import "github.com/SigNoz/signoz/pkg/types/telemetrytypes"
var families = []Family{
{
Current: "db.system.name",
Old: []string{"db.system"},
Kind: KindAttribute,
Contexts: nil,
Signals: nil,
ApplyToMetrics: nil,
current: "container.cpu.usage",
kind: KindMetric,
members: []Member{
{name: "container.cpu.utilization"},
},
},
{
Current: "deployment.environment.name",
Old: []string{"deployment.environment"},
Kind: KindAttribute,
Contexts: nil,
Signals: nil,
ApplyToMetrics: nil,
current: "deployment.environment.name",
kind: KindAttribute,
members: []Member{
{name: "deployment.environment"},
},
contexts: []telemetrytypes.FieldContext{telemetrytypes.FieldContextAttribute, telemetrytypes.FieldContextResource},
signals: []telemetrytypes.Signal{telemetrytypes.SignalLogs, telemetrytypes.SignalMetrics, telemetrytypes.SignalTraces},
},
{
current: "k8s.node.cpu.usage",
kind: KindMetric,
members: []Member{
{name: "k8s.node.cpu.utilization"},
},
},
{
current: "k8s.pod.cpu.usage",
kind: KindMetric,
members: []Member{
{name: "k8s.pod.cpu.utilization"},
},
},
}

View File

@@ -1,6 +1,7 @@
package semconv
import (
"iter"
"slices"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -14,16 +15,47 @@ type Kind struct {
valuer.String
}
// Family is one logical telemetry field. Old is ordered from the most recent
// predecessor to the oldest one and therefore also defines fallback order.
// Member is one historical spelling of a family, with the scope its rename
// edges declared. A nil axis places no constraint on that axis.
type Member struct {
name string
contexts []telemetrytypes.FieldContext
signals []telemetrytypes.Signal
applyToMetrics []string
}
func (m Member) Name() string {
return m.name
}
// Family is one logical telemetry field. Members are ordered from the most
// recent predecessor to the oldest one and therefore also define fallback
// order. The family-level contexts and signals come from the overlay. They
// gate where the family can resolve at all. The member scopes come from the
// schema edges. They gate which members apply for a given selector.
type Family struct {
Current string
Old []string
Kind Kind
Contexts []telemetrytypes.FieldContext
Signals []telemetrytypes.Signal
ApplyToMetrics []string
ValueMap map[string]string
current string
kind Kind
members []Member
contexts []telemetrytypes.FieldContext
signals []telemetrytypes.Signal
}
func (f Family) Current() string {
return f.current
}
func (f Family) Kind() Kind {
return f.kind
}
// Old returns the historical spellings in fallback order.
func (f Family) Old() []string {
names := make([]string, len(f.members))
for i, member := range f.members {
names[i] = member.name
}
return names
}
var (
@@ -38,90 +70,149 @@ func (Kind) Enum() []any {
return []any{KindAttribute, KindMetric}
}
// Lookup returns the enabled family containing selector.Name for kind. The
// returned family must not be modified.
func Lookup(kind Kind, selector telemetrytypes.FieldKeySelector) (Family, bool) {
idx, ok := lookupIndex(kind, selector)
if !ok {
return Family{}, false
}
return families[idx], true
}
// Members returns the current name first, followed by historical names in
// fallback order. A name outside an enabled family is returned unchanged. The
// returned slice must not be modified.
// Members returns the current name first, followed by the historical spellings
// admitted for the selector, in fallback order. A name outside an enabled
// family is returned unchanged. So is a name the selector leaves ambiguous.
// The returned slice must not be modified.
func Members(kind Kind, selector telemetrytypes.FieldKeySelector) []string {
idx, ok := lookupIndex(kind, selector)
if !ok {
return []string{selector.Name}
}
return familyMembers[idx]
return admittedMembers(idx, selector)
}
func admittedMembers(idx int, selector telemetrytypes.FieldKeySelector) []string {
admitted := 0
for _, member := range families[idx].members {
if memberAdmits(member, selector) {
admitted++
}
}
if admitted == len(families[idx].members) {
return familyMembers[idx]
}
names := make([]string, 0, admitted+1)
names = append(names, families[idx].current)
for _, member := range families[idx].members {
if memberAdmits(member, selector) {
names = append(names, member.name)
}
}
return names
}
// Current returns the current name for selector.Name, or the input name when
// it does not belong to an enabled family.
// it does not resolve to a family.
func Current(kind Kind, selector telemetrytypes.FieldKeySelector) string {
idx, ok := lookupIndex(kind, selector)
if !ok {
return selector.Name
}
return families[idx].Current
return families[idx].current
}
// All returns every enabled family. The returned slice and families must not be
// modified.
func All() []Family {
return families
func All() iter.Seq[Family] {
return func(yield func(Family) bool) {
for _, family := range families {
if !yield(family) {
return
}
}
}
}
func buildIndexes() (map[string][]int, [][]string) {
index := make(map[string][]int)
members := make([][]string, len(families))
add := func(name string, i int) {
if !slices.Contains(index[name], i) {
index[name] = append(index[name], i)
}
}
for i, family := range families {
members[i] = make([]string, 0, len(family.Old)+1)
members[i] = append(members[i], family.Current)
members[i] = append(members[i], family.Old...)
index[family.Current] = append(index[family.Current], i)
for _, old := range family.Old {
index[old] = append(index[old], i)
members[i] = make([]string, 0, len(family.members)+1)
members[i] = append(members[i], family.current)
add(family.current, i)
for _, member := range family.members {
members[i] = append(members[i], member.name)
add(member.name, i)
}
}
return index, members
}
// lookupIndex returns the family that resolves selector.Name for kind.
//
// The missing-information policy is the same on every axis (signal, field
// context, metric name): an axis the selector does not populate is a wildcard
// and constrains nothing. When the wildcards leave more than one family
// admitted, the name does not resolve. Resolution never picks an arbitrary
// winner. It asks for more information by staying literal.
func lookupIndex(kind Kind, selector telemetrytypes.FieldKeySelector) (int, bool) {
found, foundIdx := 0, 0
for _, idx := range memberToFamilies[selector.Name] {
if matchesSelector(families[idx], kind, selector) {
return idx, true
if familyAdmits(families[idx], kind, selector) {
found++
foundIdx = idx
}
}
return 0, false
if found != 1 {
return 0, false
}
return foundIdx, true
}
func matchesSelector(family Family, kind Kind, selector telemetrytypes.FieldKeySelector) bool {
if family.Kind != kind {
// familyAdmits reports whether the family resolves selector.Name: the
// family-level gate must admit the selector, and the name must be the current
// name or an admitted member.
func familyAdmits(family Family, kind Kind, selector telemetrytypes.FieldKeySelector) bool {
if family.kind != kind {
return false
}
if selector.Signal != telemetrytypes.SignalUnspecified && len(family.Signals) > 0 {
if !slices.Contains(family.Signals, selector.Signal) {
return false
if !axisAdmits(family.signals, selector.Signal, telemetrytypes.SignalUnspecified) {
return false
}
if !axisAdmits(family.contexts, selector.FieldContext, telemetrytypes.FieldContextUnspecified) {
return false
}
if selector.Name == family.current {
for _, member := range family.members {
if memberAdmits(member, selector) {
return true
}
}
return false
}
for _, member := range family.members {
if member.name == selector.Name && memberAdmits(member, selector) {
return true
}
}
return false
}
if selector.FieldContext != telemetrytypes.FieldContextUnspecified && len(family.Contexts) > 0 {
if !slices.Contains(family.Contexts, selector.FieldContext) {
return false
}
// memberAdmits reports whether the member applies for the selector under the
// wildcard policy: a selector axis without a value never constrains, and a
// member axis without a value admits every selector value.
func memberAdmits(member Member, selector telemetrytypes.FieldKeySelector) bool {
if !axisAdmits(member.signals, selector.Signal, telemetrytypes.SignalUnspecified) {
return false
}
if selector.Signal == telemetrytypes.SignalMetrics && len(family.ApplyToMetrics) > 0 {
if selector.MetricContext == nil {
return false
}
return slices.Contains(family.ApplyToMetrics, selector.MetricContext.MetricName)
if !axisAdmits(member.contexts, selector.FieldContext, telemetrytypes.FieldContextUnspecified) {
return false
}
if len(member.applyToMetrics) > 0 &&
selector.MetricContext != nil && selector.MetricContext.MetricName != "" &&
!slices.Contains(member.applyToMetrics, selector.MetricContext.MetricName) {
return false
}
return true
}
func axisAdmits[T comparable](scope []T, value T, unspecified T) bool {
if len(scope) == 0 || value == unspecified {
return true
}
return slices.Contains(scope, value)
}

View File

@@ -78,3 +78,116 @@ func TestMembersReturnsInputWhenKindDoesNotMatch(t *testing.T) {
"an attribute family must not match a metric-name lookup",
)
}
func TestFamilySignalsGateResolution(t *testing.T) {
swapFamilies(t, []Family{{
current: "gated.current",
kind: KindAttribute,
members: []Member{{name: "gated.old"}},
signals: []telemetrytypes.Signal{telemetrytypes.SignalLogs, telemetrytypes.SignalTraces},
}})
metrics := telemetrytypes.FieldKeySelector{Name: "gated.old", Signal: telemetrytypes.SignalMetrics}
logs := telemetrytypes.FieldKeySelector{Name: "gated.old", Signal: telemetrytypes.SignalLogs}
assert.Equal(t, []string{"gated.old"}, Members(KindAttribute, metrics),
"a family gated to traces and logs must stay literal for metrics")
assert.Equal(t, []string{"gated.current", "gated.old"}, Members(KindAttribute, logs),
"the gate admits the signals it lists")
}
func TestMetricNameFamilyResolves(t *testing.T) {
selector := telemetrytypes.FieldKeySelector{Name: "k8s.pod.cpu.utilization", Signal: telemetrytypes.SignalMetrics}
assert.Equal(t, []string{"k8s.pod.cpu.usage", "k8s.pod.cpu.utilization"}, Members(KindMetric, selector))
assert.Equal(t, "k8s.pod.cpu.usage", Current(KindMetric, selector))
assert.Equal(t, []string{"k8s.pod.cpu.utilization"}, Members(KindAttribute, selector),
"a metric-name family must not match an attribute lookup")
}
func TestMembersReturnsSharedSliceForUnscopedFamily(t *testing.T) {
selector := telemetrytypes.FieldKeySelector{Name: "deployment.environment", Signal: telemetrytypes.SignalTraces}
first := Members(KindAttribute, selector)
second := Members(KindAttribute, selector)
assert.Equal(t, &first[0], &second[0],
"a family whose members all admit must return the precomputed slice, not a copy")
}
func TestAllIteratesEnabledFamilies(t *testing.T) {
currents := []string{}
for family := range All() {
currents = append(currents, family.Current())
}
assert.Contains(t, currents, "deployment.environment.name")
assert.Contains(t, currents, "k8s.pod.cpu.usage")
}
// swapFamilies replaces the generated table for one test so scoped-member and
// fan-out behavior can be pinned without enabling such families for real.
func swapFamilies(t *testing.T, replacement []Family) {
t.Helper()
prevFamilies, prevIndex, prevMembers := families, memberToFamilies, familyMembers
families = replacement
memberToFamilies, familyMembers = buildIndexes()
t.Cleanup(func() {
families, memberToFamilies, familyMembers = prevFamilies, prevIndex, prevMembers
})
}
func TestFanOutResolvesOnlyWithEnoughInformation(t *testing.T) {
swapFamilies(t, []Family{
{
current: "cpu.mode",
kind: KindAttribute,
members: []Member{{name: "state", applyToMetrics: []string{"system.cpu.time"}}},
},
{
current: "db.client.connection.state",
kind: KindAttribute,
members: []Member{{name: "state", applyToMetrics: []string{"db.client.connections.usage"}}},
},
})
ambiguous := telemetrytypes.FieldKeySelector{Name: "state", Signal: telemetrytypes.SignalMetrics}
assert.Equal(t, []string{"state"}, Members(KindAttribute, ambiguous),
"without a metric name, a fanned-out member admits several families and must stay literal")
pinned := ambiguous
pinned.MetricContext = &telemetrytypes.MetricContext{MetricName: "system.cpu.time"}
assert.Equal(t, []string{"cpu.mode", "state"}, Members(KindAttribute, pinned),
"the metric name disambiguates the fan-out")
outside := ambiguous
outside.MetricContext = &telemetrytypes.MetricContext{MetricName: "http.server.duration"}
assert.Equal(t, []string{"state"}, Members(KindAttribute, outside),
"a metric outside every apply_to_metrics list resolves no family")
}
func TestMemberScopesFilterMembers(t *testing.T) {
swapFamilies(t, []Family{{
current: "user_agent.original",
kind: KindAttribute,
members: []Member{
{name: "http.user_agent", contexts: []telemetrytypes.FieldContext{telemetrytypes.FieldContextAttribute}, signals: []telemetrytypes.Signal{telemetrytypes.SignalTraces}},
{name: "browser.user_agent", contexts: []telemetrytypes.FieldContext{telemetrytypes.FieldContextResource}},
},
}})
resource := telemetrytypes.FieldKeySelector{
Name: "user_agent.original",
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextResource,
}
assert.Equal(t, []string{"user_agent.original", "browser.user_agent"}, Members(KindAttribute, resource),
"a strict resource lookup must not include the span-only member")
attribute := resource
attribute.FieldContext = telemetrytypes.FieldContextAttribute
assert.Equal(t, []string{"user_agent.original", "http.user_agent"}, Members(KindAttribute, attribute),
"a strict attribute lookup must not include the resource-only member")
strictResourceOldSpan := resource
strictResourceOldSpan.Name = "http.user_agent"
assert.Equal(t, []string{"http.user_agent"}, Members(KindAttribute, strictResourceOldSpan),
"an old spelling outside its own scope stays literal")
}

View File

@@ -0,0 +1,181 @@
package logsstatementbuilder
import (
"context"
"testing"
"time"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/require"
)
var logsFamilyReleaseTime = time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
func logsFamilyKey(name string) *telemetrytypes.TelemetryFieldKey {
return &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeString,
}
}
func logsFamilyBuilder(t *testing.T, familyOn bool, stored ...string) *logQueryStatementBuilder {
t.Helper()
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureResolveSemconvFamilies.String(): familyOn,
})
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys := logstelemetryschema.BuildCompleteFieldKeyMap(logsFamilyReleaseTime)
for _, name := range stored {
keys[name] = []*telemetrytypes.TelemetryFieldKey{logsFamilyKey(name)}
}
mockMetadataStore.KeysMap = keys
storage := logstelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
return NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore, storage, aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
}
func buildLogsFamilyQuery(t *testing.T, builder *logQueryStatementBuilder, query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]) string {
t.Helper()
releaseTimeNano := uint64(logsFamilyReleaseTime.UnixNano())
q, err := builder.Build(context.Background(), valuer.UUID{},
releaseTimeNano+uint64(24*time.Hour.Nanoseconds()),
releaseTimeNano+uint64(48*time.Hour.Nanoseconds()),
qbtypes.RequestTypeScalar, query, nil)
require.NoError(t, err)
return q.Query
}
// A filter on either spelling of an enabled family compiles to one merged
// condition over the log attribute maps. The flag default keeps it literal.
func TestStatementBuilderResolvesLogFamilies(t *testing.T) {
cases := []struct {
name string
familyOn bool
expected string
}{
{
name: "families on",
familyOn: true,
expected: "SELECT count() AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE (COALESCE(NULLIF(attributes_string['deployment.environment.name'], ''), NULLIF(attributes_string['deployment.environment'], ''), '') = ? AND (mapContains(attributes_string, 'deployment.environment.name') OR mapContains(attributes_string, 'deployment.environment'))) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? ORDER BY __result_0 DESC",
},
{
name: "families off",
familyOn: false,
expected: "SELECT count() AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE (attributes_string['deployment.environment'] = ? AND mapContains(attributes_string, 'deployment.environment')) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? ORDER BY __result_0 DESC",
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
builder := logsFamilyBuilder(t, c.familyOn, "deployment.environment.name", "deployment.environment")
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Aggregations: []qbtypes.LogAggregation{{Expression: "count()"}},
Filter: &qbtypes.Filter{Expression: "attribute.deployment.environment = 'production'"},
}
require.Equal(t, c.expected, buildLogsFamilyQuery(t, builder, query))
})
}
}
// The predicate of a filtered aggregation resolves the family exactly like
// the main WHERE clause.
func TestStatementBuilderResolvesLogFamilyFilteredAggregation(t *testing.T) {
cases := []struct {
name string
familyOn bool
expected string
}{
{name: "families on", familyOn: true, expected: "SELECT countIf((COALESCE(NULLIF(attributes_string['deployment.environment.name'], ''), NULLIF(attributes_string['deployment.environment'], ''), '') = ? AND (mapContains(attributes_string, 'deployment.environment.name') OR mapContains(attributes_string, 'deployment.environment')))) AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? ORDER BY __result_0 DESC"},
{name: "families off", familyOn: false, expected: "SELECT countIf((attributes_string['deployment.environment.name'] = ? AND mapContains(attributes_string, 'deployment.environment.name'))) AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? ORDER BY __result_0 DESC"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
builder := logsFamilyBuilder(t, c.familyOn, "deployment.environment.name", "deployment.environment")
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Aggregations: []qbtypes.LogAggregation{{Expression: "countIf(deployment.environment.name = 'production')"}},
}
require.Equal(t, c.expected, buildLogsFamilyQuery(t, builder, query))
})
}
}
// The mid-migration state: metadata holds one spelling of the family, and
// the query names the other. The filter and the group by both read the one
// stored spelling.
func TestStatementBuilderResolvesSingleSpellingAcrossNames(t *testing.T) {
cases := []struct {
name string
stored string
queried string
expected string
}{
{name: "old data queried by the current name", stored: "deployment.environment", queried: "deployment.environment.name", expected: "SELECT toString(multiIf(mapContains(attributes_string, 'deployment.environment'), attributes_string['deployment.environment'], NULL)) AS `__GROUP_BY_KEY_0_deployment.environment.name`, count() AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE (attributes_string['deployment.environment'] = ? AND mapContains(attributes_string, 'deployment.environment')) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_deployment.environment.name` ORDER BY __result_0 DESC"},
{name: "current data queried by the old name", stored: "deployment.environment.name", queried: "deployment.environment", expected: "SELECT toString(multiIf(mapContains(attributes_string, 'deployment.environment.name'), attributes_string['deployment.environment.name'], NULL)) AS `__GROUP_BY_KEY_0_deployment.environment`, count() AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE (attributes_string['deployment.environment.name'] = ? AND mapContains(attributes_string, 'deployment.environment.name')) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_deployment.environment` ORDER BY __result_0 DESC"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
builder := logsFamilyBuilder(t, true, c.stored)
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Aggregations: []qbtypes.LogAggregation{{Expression: "count()"}},
Filter: &qbtypes.Filter{Expression: c.queried + " = 'production'"},
GroupBy: []qbtypes.GroupByKey{
{TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: c.queried}},
},
}
require.Equal(t, c.expected, buildLogsFamilyQuery(t, builder, query))
})
}
}
// Group by resolves the family exactly like the filter. The merged column
// reads the spellings current-first with empty falling through, and a row
// with no member keeps the NULL group of a single key.
func TestStatementBuilderResolvesLogFamilyGroupBy(t *testing.T) {
cases := []struct {
name string
familyOn bool
expected string
}{
{name: "families on", familyOn: true, expected: "SELECT toString(multiIf((mapContains(attributes_string, 'deployment.environment.name') OR mapContains(attributes_string, 'deployment.environment')), COALESCE(NULLIF(attributes_string['deployment.environment.name'], ''), NULLIF(attributes_string['deployment.environment'], ''), ''), NULL)) AS `__GROUP_BY_KEY_0_deployment.environment.name`, count() AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_deployment.environment.name` ORDER BY __result_0 DESC"},
{name: "families off", familyOn: false, expected: "SELECT toString(multiIf(mapContains(attributes_string, 'deployment.environment.name'), attributes_string['deployment.environment.name'], NULL)) AS `__GROUP_BY_KEY_0_deployment.environment.name`, count() AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_deployment.environment.name` ORDER BY __result_0 DESC"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
builder := logsFamilyBuilder(t, c.familyOn, "deployment.environment.name", "deployment.environment")
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Aggregations: []qbtypes.LogAggregation{{Expression: "count()"}},
GroupBy: []qbtypes.GroupByKey{
{TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}},
},
}
require.Equal(t, c.expected, buildLogsFamilyQuery(t, builder, query))
})
}
}

View File

@@ -125,6 +125,7 @@ func (b *logQueryStatementBuilder) Build(
bodyJSONEnabled := b.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
keySelectors, warnings := getKeySelectors(query, bodyJSONEnabled)
keySelectors = querybuilder.ExpandKeySelectorsForFamilies(ctx, orgID, b.fl, keySelectors)
keys, _, err := b.metadataStore.GetKeysMulti(ctx, orgID, keySelectors)
if err != nil {
return nil, err

View File

@@ -0,0 +1,109 @@
package metricsstatementbuilder
import (
"context"
"testing"
"time"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/require"
)
func metricsFamilyKey(name string) *telemetrytypes.TelemetryFieldKey {
return &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalMetrics,
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeString,
}
}
func familyStatementBuilder(t *testing.T, familyOn bool, stored ...string) *StatementBuilder {
t.Helper()
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureResolveSemconvFamilies.String(): familyOn,
})
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys := make(map[string][]*telemetrytypes.TelemetryFieldKey, len(stored))
for _, name := range stored {
keys[name] = []*telemetrytypes.TelemetryFieldKey{metricsFamilyKey(name)}
}
mockMetadataStore.KeysMap = keys
return NewMetricQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
metricstelemetryschema.NewStorage(),
fl,
)
}
func familyQuery() qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation] {
return qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "k8s.pod.cpu.utilization",
Type: metrictypes.GaugeType,
Temporality: metrictypes.Unspecified,
TimeAggregation: metrictypes.TimeAggregationAvg,
SpaceAggregation: metrictypes.SpaceAggregationAvg,
},
},
Filter: &qbtypes.Filter{
Expression: "deployment.environment = 'production'",
},
GroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "deployment.environment.name",
},
},
},
}
}
// The flag merges the label spellings of the family and unions the storage
// names of the metric-name family, in the filter, the group by column, and
// every metric_name filter.
func TestStatementBuilderResolvesFamilies(t *testing.T) {
statementBuilder := familyStatementBuilder(t, true, "deployment.environment.name", "deployment.environment", "deployment_environment")
q, err := statementBuilder.Build(context.Background(), valuer.UUID{}, 1747947419000, 1747983448000, qbtypes.RequestTypeTimeSeries, familyQuery(), nil)
require.NoError(t, err)
require.Equal(t, "WITH __temporal_aggregation_cte AS (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_deployment.environment.name`, avg(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, COALESCE(NULLIF(JSONExtractString(labels, 'deployment.environment.name'), ''), NULLIF(JSONExtractString(labels, 'deployment.environment'), ''), NULLIF(JSONExtractString(labels, 'deployment_environment'), ''), '') AS `__GROUP_BY_KEY_0_deployment.environment.name` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?, ?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) AND COALESCE(NULLIF(JSONExtractString(labels, 'deployment.environment.name'), ''), NULLIF(JSONExtractString(labels, 'deployment.environment'), ''), NULLIF(JSONExtractString(labels, 'deployment_environment'), ''), '') = ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_deployment.environment.name`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?, ?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_deployment.environment.name` ORDER BY fingerprint, ts), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_deployment.environment.name`, avg(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_deployment.environment.name`) SELECT * FROM __spatial_aggregation_cte ORDER BY `__GROUP_BY_KEY_0_deployment.environment.name`, ts", q.Query)
require.Equal(t, []any{"k8s.pod.cpu.usage", "k8s.pod.cpu.utilization", uint64(1747936800000), uint64(1747983420000), "unspecified", "production", "k8s.pod.cpu.usage", "k8s.pod.cpu.utilization", uint64(1747947390000), uint64(1747983420000), 0}, q.Args)
}
// The mid-migration state: metadata holds only the old label spelling, and
// the query names the current one. The filter and the group by both read
// the one stored label.
func TestStatementBuilderResolvesSingleSpellingAcrossNames(t *testing.T) {
statementBuilder := familyStatementBuilder(t, true, "deployment.environment")
query := familyQuery()
query.Filter = &qbtypes.Filter{Expression: "deployment.environment.name = 'production'"}
q, err := statementBuilder.Build(context.Background(), valuer.UUID{}, 1747947419000, 1747983448000, qbtypes.RequestTypeTimeSeries, query, nil)
require.NoError(t, err)
require.Equal(t, "WITH __temporal_aggregation_cte AS (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_deployment.environment.name`, avg(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'deployment.environment') AS `__GROUP_BY_KEY_0_deployment.environment.name` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?, ?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) AND JSONExtractString(labels, 'deployment.environment') = ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_deployment.environment.name`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?, ?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_deployment.environment.name` ORDER BY fingerprint, ts), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_deployment.environment.name`, avg(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_deployment.environment.name`) SELECT * FROM __spatial_aggregation_cte ORDER BY `__GROUP_BY_KEY_0_deployment.environment.name`, ts", q.Query)
}
// With the flag at its default, both the labels and the metric name stay
// literal.
func TestStatementBuilderKeepsFamiliesLiteralByDefault(t *testing.T) {
statementBuilder := familyStatementBuilder(t, false, "deployment.environment.name", "deployment.environment", "deployment_environment")
q, err := statementBuilder.Build(context.Background(), valuer.UUID{}, 1747947419000, 1747983448000, qbtypes.RequestTypeTimeSeries, familyQuery(), nil)
require.NoError(t, err)
require.Equal(t, "WITH __temporal_aggregation_cte AS (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_deployment.environment.name`, avg(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'deployment.environment.name') AS `__GROUP_BY_KEY_0_deployment.environment.name` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) AND JSONExtractString(labels, 'deployment.environment') = ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_deployment.environment.name`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_deployment.environment.name` ORDER BY fingerprint, ts), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_deployment.environment.name`, avg(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_deployment.environment.name`) SELECT * FROM __spatial_aggregation_cte ORDER BY `__GROUP_BY_KEY_0_deployment.environment.name`, ts", q.Query)
require.Equal(t, []any{"k8s.pod.cpu.utilization", uint64(1747936800000), uint64(1747983420000), "unspecified", "production", "k8s.pod.cpu.utilization", uint64(1747947390000), uint64(1747983420000), 0}, q.Args)
}

View File

@@ -117,7 +117,9 @@ func (b *StatementBuilder) Build(
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
keySelectors := GetKeySelectors(query)
keySelectors := querybuilder.ExpandKeySelectorsForFamilies(ctx, orgID, b.flagger, GetKeySelectors(query))
metricNames := querybuilder.FamilyMetricNames(ctx, orgID, b.flagger, query.Aggregations[0].MetricName)
keySelectors = expandSelectorsForMetricNames(keySelectors, metricNames)
keys, _, err := b.metadataStore.GetKeysMulti(ctx, orgID, keySelectors)
if err != nil {
return nil, err
@@ -125,7 +127,32 @@ func (b *StatementBuilder) Build(
start, end = querybuilder.AdjustedMetricTimeRange(start, end, uint64(query.StepInterval.Seconds()), query)
return b.buildPipelineStatement(ctx, orgID, start, end, requestType, query, keys, variables)
return b.buildPipelineStatement(ctx, orgID, start, end, requestType, query, keys, metricNames, variables)
}
// expandSelectorsForMetricNames duplicates the selectors for each storage
// name of a metric-name family. Label-key metadata is filtered by the exact
// metric_name, so the series under the old name must contribute their keys
// too.
func expandSelectorsForMetricNames(selectors []*telemetrytypes.FieldKeySelector, metricNames []string) []*telemetrytypes.FieldKeySelector {
if len(metricNames) <= 1 {
return selectors
}
out := selectors
for _, selector := range selectors {
if selector.MetricContext == nil {
continue
}
for _, metricName := range metricNames {
if metricName == selector.MetricContext.MetricName {
continue
}
expanded := *selector
expanded.MetricContext = &telemetrytypes.MetricContext{MetricName: metricName}
out = append(out, &expanded)
}
}
return out
}
func (b *StatementBuilder) buildPipelineStatement(
@@ -135,6 +162,7 @@ func (b *StatementBuilder) buildPipelineStatement(
requestType qbtypes.RequestType,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
keys map[string][]*telemetrytypes.TelemetryFieldKey,
metricNames []string,
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
var (
@@ -166,13 +194,13 @@ func (b *StatementBuilder) buildPipelineStatement(
var filterWarnings []string
var err error
if timeSeriesCTE, timeSeriesCTEArgs, filterWarnings, err = b.buildTimeSeriesCTE(ctx, orgID, tsStart, tsEnd, cteQuery, keys, variables, tsTable); err != nil {
if timeSeriesCTE, timeSeriesCTEArgs, filterWarnings, err = b.buildTimeSeriesCTE(ctx, orgID, tsStart, tsEnd, cteQuery, keys, metricNames, variables, tsTable); err != nil {
return nil, err
}
if qbtypes.CanShortCircuitDelta(agg) {
// spatial_aggregation_cte directly for certain delta queries
if frag, args, err := b.buildTemporalAggDeltaFastPath(start, end, cteQuery, samplesTable, timeSeriesCTE, timeSeriesCTEArgs); err != nil {
if frag, args, err := b.buildTemporalAggDeltaFastPath(start, end, cteQuery, metricNames, samplesTable, timeSeriesCTE, timeSeriesCTEArgs); err != nil {
return nil, err
} else if frag != "" {
cteFragments = append(cteFragments, frag)
@@ -180,7 +208,7 @@ func (b *StatementBuilder) buildPipelineStatement(
}
} else {
// temporal_aggregation_cte
if frag, args, err := b.buildTemporalAggregationCTE(ctx, start, end, cteQuery, keys, samplesTable, timeSeriesCTE, timeSeriesCTEArgs); err != nil {
if frag, args, err := b.buildTemporalAggregationCTE(ctx, start, end, cteQuery, keys, metricNames, samplesTable, timeSeriesCTE, timeSeriesCTEArgs); err != nil {
return nil, err
} else if frag != "" {
cteFragments = append(cteFragments, frag)
@@ -201,16 +229,16 @@ func (b *StatementBuilder) buildPipelineStatement(
var tsArgs []any
// time series rows are written on hour boundaries
tsStart := start - (start % metricstelemetryschema.OneHourInMilliseconds)
if tsCTE, tsArgs, err = b.buildReducedTimeSeriesCTE(ctx, orgID, tsStart, end, cteQuery, keys, variables); err != nil {
if tsCTE, tsArgs, err = b.buildReducedTimeSeriesCTE(ctx, orgID, tsStart, end, cteQuery, keys, metricNames, variables); err != nil {
return nil, err
}
if qbtypes.CanShortCircuitReduced(agg) {
// spatial_aggregation_cte directly, no per-series level
if spatialFrag, spatialArgs, ok := b.buildReducedSpatialAggFastPath(start, end, cteQuery, tsCTE, tsArgs); ok {
if spatialFrag, spatialArgs, ok := b.buildReducedSpatialAggFastPath(start, end, cteQuery, metricNames, tsCTE, tsArgs); ok {
reducedFragments = []string{spatialFrag}
reducedArgs = [][]any{spatialArgs}
}
} else if temporalFrag, temporalArgs, ok := b.buildReducedTemporalAggregationCTE(start, end, cteQuery, tsCTE, tsArgs); ok {
} else if temporalFrag, temporalArgs, ok := b.buildReducedTemporalAggregationCTE(start, end, cteQuery, metricNames, tsCTE, tsArgs); ok {
spatialFrag, spatialArgs := b.buildReducedSpatialAggregationCTE(cteQuery)
reducedFragments = []string{temporalFrag, spatialFrag}
reducedArgs = [][]any{temporalArgs, spatialArgs}
@@ -268,6 +296,7 @@ func (b *StatementBuilder) buildReducedTimeSeriesCTE(
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
keys map[string][]*telemetrytypes.TelemetryFieldKey,
metricNames []string,
variables map[string]qbtypes.VariableItem,
) (string, []any, error) {
sb := sqlbuilder.NewSelectBuilder()
@@ -300,7 +329,7 @@ func (b *StatementBuilder) buildReducedTimeSeriesCTE(
sb.SelectMore(sqlbuilder.Escape(fmt.Sprintf("%s AS %s", col, GroupByColumnAlias(i, g.Name))))
}
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GTE("unix_milli", start),
sb.LTE("unix_milli", end),
)
@@ -325,6 +354,7 @@ func (b *StatementBuilder) buildReducedTimeSeriesCTE(
func (b *StatementBuilder) buildReducedSpatialAggFastPath(
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
metricNames []string,
timeSeriesCTE string,
timeSeriesCTEArgs []any,
) (string, []any, bool) {
@@ -345,7 +375,7 @@ func (b *StatementBuilder) buildReducedSpatialAggFastPath(
sb.From(fmt.Sprintf("%s.%s AS points FINAL", metricstelemetryschema.DBName, metricstelemetryschema.WhichReducedSamplesTableToUse(agg.Type)))
sb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.reduced_fingerprint = filtered_time_series.fingerprint")
sb.Where(
sb.In("metric_name", agg.MetricName),
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GTE("unix_milli", start),
sb.LT("unix_milli", end),
)
@@ -359,6 +389,7 @@ func (b *StatementBuilder) buildReducedSpatialAggFastPath(
func (b *StatementBuilder) buildReducedTemporalAggregationCTE(
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
metricNames []string,
timeSeriesCTE string,
timeSeriesCTEArgs []any,
) (string, []any, bool) {
@@ -387,7 +418,7 @@ func (b *StatementBuilder) buildReducedTemporalAggregationCTE(
sb.From(fmt.Sprintf("%s.%s AS points FINAL", metricstelemetryschema.DBName, metricstelemetryschema.WhichReducedSamplesTableToUse(agg.Type)))
sb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.reduced_fingerprint = filtered_time_series.fingerprint")
sb.Where(
sb.In("metric_name", agg.MetricName),
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GTE("unix_milli", start),
sb.LT("unix_milli", end),
)
@@ -428,6 +459,7 @@ func (b *StatementBuilder) buildReducedSpatialAggregationCTE(
func (b *StatementBuilder) buildTemporalAggDeltaFastPath(
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
metricNames []string,
samplesTable string,
timeSeriesCTE string,
timeSeriesCTEArgs []any,
@@ -469,7 +501,7 @@ func (b *StatementBuilder) buildTemporalAggDeltaFastPath(
sb.From(fmt.Sprintf("%s.%s AS points", metricstelemetryschema.DBName, samplesTable))
sb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.fingerprint = filtered_time_series.fingerprint")
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GTE("unix_milli", start),
sb.LT("unix_milli", end),
)
@@ -486,6 +518,7 @@ func (b *StatementBuilder) buildTimeSeriesCTE(
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
keys map[string][]*telemetrytypes.TelemetryFieldKey,
metricNames []string,
variables map[string]qbtypes.VariableItem,
tsTable string,
) (string, []any, []string, error) {
@@ -522,7 +555,7 @@ func (b *StatementBuilder) buildTimeSeriesCTE(
}
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GTE("unix_milli", start),
sb.LTE("unix_milli", end),
)
@@ -554,22 +587,24 @@ func (b *StatementBuilder) buildTemporalAggregationCTE(
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
_ map[string][]*telemetrytypes.TelemetryFieldKey,
metricNames []string,
samplesTable string,
timeSeriesCTE string,
timeSeriesCTEArgs []any,
) (string, []any, error) {
if query.Aggregations[0].Temporality == metrictypes.Delta {
return b.buildTemporalAggDelta(ctx, start, end, query, samplesTable, timeSeriesCTE, timeSeriesCTEArgs)
return b.buildTemporalAggDelta(ctx, start, end, query, metricNames, samplesTable, timeSeriesCTE, timeSeriesCTEArgs)
} else if query.Aggregations[0].Temporality != metrictypes.Multiple {
return b.buildTemporalAggCumulativeOrUnspecified(ctx, start, end, query, samplesTable, timeSeriesCTE, timeSeriesCTEArgs)
return b.buildTemporalAggCumulativeOrUnspecified(ctx, start, end, query, metricNames, samplesTable, timeSeriesCTE, timeSeriesCTEArgs)
}
return b.buildTemporalAggForMultipleTemporalities(ctx, start, end, query, samplesTable, timeSeriesCTE, timeSeriesCTEArgs)
return b.buildTemporalAggForMultipleTemporalities(ctx, start, end, query, metricNames, samplesTable, timeSeriesCTE, timeSeriesCTEArgs)
}
func (b *StatementBuilder) buildTemporalAggDelta(
_ context.Context,
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
metricNames []string,
samplesTable string,
timeSeriesCTE string,
timeSeriesCTEArgs []any,
@@ -601,7 +636,7 @@ func (b *StatementBuilder) buildTemporalAggDelta(
sb.From(fmt.Sprintf("%s.%s AS points", metricstelemetryschema.DBName, samplesTable))
sb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.fingerprint = filtered_time_series.fingerprint")
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GTE("unix_milli", start),
sb.LT("unix_milli", end),
)
@@ -617,6 +652,7 @@ func (b *StatementBuilder) buildTemporalAggCumulativeOrUnspecified(
_ context.Context,
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
metricNames []string,
samplesTable string,
timeSeriesCTE string,
timeSeriesCTEArgs []any,
@@ -642,7 +678,7 @@ func (b *StatementBuilder) buildTemporalAggCumulativeOrUnspecified(
baseSb.From(fmt.Sprintf("%s.%s AS points", metricstelemetryschema.DBName, samplesTable))
baseSb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.fingerprint = filtered_time_series.fingerprint")
baseSb.Where(
baseSb.In("metric_name", query.Aggregations[0].MetricName),
baseSb.In("metric_name", sqlbuilder.List(metricNames)),
baseSb.GTE("unix_milli", start),
baseSb.LT("unix_milli", end),
)
@@ -683,6 +719,7 @@ func (b *StatementBuilder) buildTemporalAggForMultipleTemporalities(
_ context.Context,
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
metricNames []string,
samplesTable string,
timeSeriesCTE string,
timeSeriesCTEArgs []any,
@@ -733,7 +770,7 @@ func (b *StatementBuilder) buildTemporalAggForMultipleTemporalities(
sb.From(fmt.Sprintf("%s.%s AS points", metricstelemetryschema.DBName, samplesTable))
sb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.fingerprint = filtered_time_series.fingerprint")
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GTE("unix_milli", start),
sb.LT("unix_milli", end),
)

View File

@@ -2062,3 +2062,51 @@ func TestStatementBuilderSemconvFamilies(t *testing.T) {
})
}
}
// The mid-migration state: metadata holds only the old spelling, and the
// query names the current one. The resource-filter condition and the group
// by column both read the stored spelling, so the filter and the groups
// agree.
func TestStatementBuilderSemconvSingleSpelling(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureResolveSemconvFamilies.String(): true,
})
storage := tracestelemetryschema.NewStorage()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment": {{
Name: "deployment.environment",
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextResource,
FieldDataType: telemetrytypes.FieldDataTypeString,
}},
}
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
aggExprRewriter,
nil,
fl,
false,
100000,
)
query := qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]{
Signal: telemetrytypes.SignalTraces,
StepInterval: qbtypes.Step{Duration: 30 * time.Second},
Aggregations: []qbtypes.TraceAggregation{{Expression: "count()"}},
Filter: &qbtypes.Filter{
Expression: "deployment.environment.name = 'production'",
},
GroupBy: []qbtypes.GroupByKey{
{TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}},
},
}
q, err := statementBuilder.Build(context.Background(), valuer.UUID{}, 1747947419000, 1747983448000, qbtypes.RequestTypeScalar, query, nil)
require.NoError(t, err)
require.Equal(t, "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'deployment.environment') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint) SELECT toString(multiIf(resource.`deployment.environment` IS NOT NULL, resource.`deployment.environment`::String, mapContains(resources_string, 'deployment.environment'), resources_string['deployment.environment'], NULL)) AS `__GROUP_BY_KEY_0_deployment.environment.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_deployment.environment.name` ORDER BY __result_0 DESC", q.Query)
}

View File

@@ -0,0 +1,82 @@
package telemetrymetadata
import (
"context"
"testing"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/assert"
)
// The flagger provider registration is process-global and keyed by provider
// name, so each flagger must be used before the next one is created.
func TestFamilyValueNames(t *testing.T) {
selector := &telemetrytypes.FieldValueSelector{
FieldKeySelector: &telemetrytypes.FieldKeySelector{Name: "deployment.environment"},
}
off := &telemetryMetaStore{fl: flaggertest.WithBooleanFlags(t, map[string]bool{})}
assert.Equal(t,
[]string{"deployment.environment"},
off.familyValueNames(context.Background(), valuer.UUID{}, telemetrytypes.SignalLogs, selector),
"the flag default keeps values literal")
on := &telemetryMetaStore{fl: flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureResolveSemconvFamilies.String(): true,
})}
assert.Equal(t,
[]string{"deployment.environment.name", "deployment.environment"},
on.familyValueNames(context.Background(), valuer.UUID{}, telemetrytypes.SignalLogs, selector),
"values for one spelling must cover the whole family")
assert.Equal(t,
[]string{
"resource_deployment.environment.name", "resource_deployment_environment_name",
"deployment.environment.name", "deployment_environment_name",
"resource_deployment.environment", "resource_deployment_environment",
"deployment.environment", "deployment_environment",
},
on.familyValueNames(context.Background(), valuer.UUID{}, telemetrytypes.SignalMetrics, selector),
"metric values must cover the stored label spellings")
}
// A family condition on the related values table follows the shared guard
// rule: the operator applies to the current-first merge, a positive operator
// takes the presence guard, and a negative operator keeps the keyless rows.
func TestConditionForFamilyMergedSemantics(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureResolveSemconvFamilies.String(): true,
})
storage := NewStorage()
resourceKey := func(name string) *telemetrytypes.TelemetryFieldKey {
return &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextResource,
FieldDataType: telemetrytypes.FieldDataTypeString,
}
}
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {resourceKey("deployment.environment.name")},
"deployment.environment": {resourceKey("deployment.environment")},
}
for operator, expected := range map[qbtypes.FilterOperator]string{
qbtypes.FilterOperatorEqual: "SELECT 1 WHERE (COALESCE(NULLIF(resource_attributes['deployment.environment.name'], ''), NULLIF(resource_attributes['deployment.environment'], ''), '') = ? AND (mapContains(resource_attributes, 'deployment.environment.name') OR mapContains(resource_attributes, 'deployment.environment')))",
qbtypes.FilterOperatorNotEqual: "SELECT 1 WHERE COALESCE(NULLIF(resource_attributes['deployment.environment.name'], ''), NULLIF(resource_attributes['deployment.environment'], ''), '') <> ?",
} {
sb := sqlbuilder.NewSelectBuilder()
q := querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0)
conds, _, err := querybuilder.Conditions(context.Background(), q, storage,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment"}, operator, "production", fieldKeys, false, sb)
assert.NoError(t, err)
sb.Select("1").Where(conds...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
assert.Equal(t, expected, sql, operator)
}
}

View File

@@ -15,6 +15,7 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/semconv"
"github.com/SigNoz/signoz/pkg/telemetryschema/audittelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema"
@@ -1292,25 +1293,41 @@ func (t *telemetryMetaStore) getRelatedValues(ctx context.Context, orgID valuer.
FieldDataType: fieldValueSelector.FieldDataType,
}
q := querybuilder.NewQueryInfo(ctx, orgID, nil, fieldValueSelector.Signal, nil, 0, 0)
selectRead, err := t.storage.Read(ctx, q, key)
selectColumn := selectRead.SQL
if err != nil {
// we don't have a explicit column to select from the related metadata table
// so we will select either from resource_attributes or attributes table
// in that order
resourceRead, _ := t.storage.Read(ctx, q, &telemetrytypes.TelemetryFieldKey{
Name: key.Name,
FieldContext: telemetrytypes.FieldContextResource,
FieldDataType: telemetrytypes.FieldDataTypeString,
})
attributeRead, _ := t.storage.Read(ctx, q, &telemetrytypes.TelemetryFieldKey{
Name: key.Name,
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeString,
})
selectColumn = fmt.Sprintf("if(notEmpty(%s), %s, %s)", resourceRead.SQL, resourceRead.SQL, attributeRead.SQL)
q := querybuilder.NewQueryInfo(ctx, orgID, t.fl, fieldValueSelector.Signal, nil, 0, 0)
// One column per family spelling, merged current-first, so the
// suggestions cover rows that carry only an old spelling.
names := t.familyValueNames(ctx, orgID, fieldValueSelector.Signal, fieldValueSelector)
memberColumns := make([]string, 0, len(names))
for _, name := range names {
memberKey := &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: fieldValueSelector.Signal,
FieldContext: fieldValueSelector.FieldContext,
FieldDataType: fieldValueSelector.FieldDataType,
}
memberRead, err := t.storage.Read(ctx, q, memberKey)
memberColumn := memberRead.SQL
if err != nil {
// we don't have a explicit column to select from the related metadata table
// so we will select either from resource_attributes or attributes table
// in that order
resourceRead, _ := t.storage.Read(ctx, q, &telemetrytypes.TelemetryFieldKey{
Name: name,
FieldContext: telemetrytypes.FieldContextResource,
FieldDataType: telemetrytypes.FieldDataTypeString,
})
attributeRead, _ := t.storage.Read(ctx, q, &telemetrytypes.TelemetryFieldKey{
Name: name,
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeString,
})
memberColumn = fmt.Sprintf("if(notEmpty(%s), %s, %s)", resourceRead.SQL, resourceRead.SQL, attributeRead.SQL)
}
memberColumns = append(memberColumns, memberColumn)
}
selectColumn := memberColumns[len(memberColumns)-1]
for i := len(memberColumns) - 2; i >= 0; i-- {
selectColumn = fmt.Sprintf("if(notEmpty(%s), %s, %s)", memberColumns[i], memberColumns[i], selectColumn)
}
sb := sqlbuilder.Select("DISTINCT " + selectColumn).From(t.relatedMetadataDBName + "." + t.relatedMetadataTblName)
@@ -1320,6 +1337,7 @@ func (t *telemetryMetaStore) getRelatedValues(ctx context.Context, orgID valuer.
for _, keySelector := range keySelectors {
keySelector.Signal = fieldValueSelector.Signal
}
keySelectors = querybuilder.ExpandKeySelectorsForFamilies(ctx, orgID, t.fl, keySelectors)
keys, _, err := t.GetKeysMulti(ctx, orgID, keySelectors)
if err != nil {
return nil, false, err
@@ -1361,20 +1379,20 @@ func (t *telemetryMetaStore) getRelatedValues(ctx context.Context, orgID valuer.
// search on attributes
key.FieldContext = telemetrytypes.FieldContextAttribute
attrConds, err := t.containsConditions(ctx, q, key, fieldValueSelector.Value, sb)
attrConds, err := t.containsConditions(ctx, q, key, names, fieldValueSelector.Value, sb)
if err == nil {
conds = append(conds, attrConds...)
}
// search on resource
key.FieldContext = telemetrytypes.FieldContextResource
resourceConds, err := t.containsConditions(ctx, q, key, fieldValueSelector.Value, sb)
resourceConds, err := t.containsConditions(ctx, q, key, names, fieldValueSelector.Value, sb)
if err == nil {
conds = append(conds, resourceConds...)
}
key.FieldContext = origContext
} else {
keyConds, err := t.containsConditions(ctx, q, key, fieldValueSelector.Value, sb)
keyConds, err := t.containsConditions(ctx, q, key, names, fieldValueSelector.Value, sb)
if err == nil {
conds = append(conds, keyConds...)
}
@@ -1432,7 +1450,7 @@ func (t *telemetryMetaStore) GetRelatedValues(ctx context.Context, orgID valuer.
return t.getRelatedValues(ctx, orgID, fieldValueSelector)
}
func (t *telemetryMetaStore) getSpanFieldValues(ctx context.Context, fieldValueSelector *telemetrytypes.FieldValueSelector) (*telemetrytypes.TelemetryFieldValues, bool, error) {
func (t *telemetryMetaStore) getSpanFieldValues(ctx context.Context, orgID valuer.UUID, fieldValueSelector *telemetrytypes.FieldValueSelector) (*telemetrytypes.TelemetryFieldValues, bool, error) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.TelemetrySignal: telemetrytypes.SignalTraces.StringValue(),
instrumentationtypes.CodeNamespace: "metadata",
@@ -1443,11 +1461,12 @@ func (t *telemetryMetaStore) getSpanFieldValues(ctx context.Context, fieldValueS
return values, true, nil
}
knownBool := isKnownBoolField(fieldValueSelector, tracestelemetryschema.IntrinsicFields, tracestelemetryschema.CalculatedFields)
names := t.familyValueNames(ctx, orgID, telemetrytypes.SignalTraces, fieldValueSelector)
// unix_milli is the hour of the span start
return t.getTagTableValues(ctx, t.tracesDBName+"."+t.tracesFieldsTblName, fieldValueSelector, knownBool)
return t.getTagTableValues(ctx, t.tracesDBName+"."+t.tracesFieldsTblName, fieldValueSelector, names, knownBool)
}
func (t *telemetryMetaStore) getLogFieldValues(ctx context.Context, fieldValueSelector *telemetrytypes.FieldValueSelector) (*telemetrytypes.TelemetryFieldValues, bool, error) {
func (t *telemetryMetaStore) getLogFieldValues(ctx context.Context, orgID valuer.UUID, fieldValueSelector *telemetrytypes.FieldValueSelector) (*telemetrytypes.TelemetryFieldValues, bool, error) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.TelemetrySignal: telemetrytypes.SignalLogs.StringValue(),
instrumentationtypes.CodeNamespace: "metadata",
@@ -1455,8 +1474,18 @@ func (t *telemetryMetaStore) getLogFieldValues(ctx context.Context, fieldValueSe
})
knownBool := isKnownBoolField(fieldValueSelector, logstelemetryschema.IntrinsicFields)
names := t.familyValueNames(ctx, orgID, telemetrytypes.SignalLogs, fieldValueSelector)
// unix_milli is the hour the log was ingested, not the log's own timestamp
return t.getTagTableValues(ctx, t.logsDBName+"."+t.logsFieldsTblName, fieldValueSelector, knownBool)
return t.getTagTableValues(ctx, t.logsDBName+"."+t.logsFieldsTblName, fieldValueSelector, names, knownBool)
}
// tagKeyCondition matches the requested key, or every spelling of its
// family when there is more than one.
func tagKeyCondition(sb *sqlbuilder.SelectBuilder, name string, names []string) string {
if len(names) > 1 {
return sb.In("tag_key", sqlbuilder.List(names))
}
return sb.E("tag_key", name)
}
// tagTableSinceDay restricts rows to the tag table's day partitions from the
@@ -1472,9 +1501,9 @@ func tagTableSinceDay(sb *sqlbuilder.SelectBuilder, startUnixMilli int64) {
// tagTableHasBoolRows reports whether the tag table holds a bool row for the
// key. Bool rows carry no value, so one row is enough to know the key takes
// the values true and false.
func (t *telemetryMetaStore) tagTableHasBoolRows(ctx context.Context, table string, selector *telemetrytypes.FieldValueSelector) (bool, error) {
func (t *telemetryMetaStore) tagTableHasBoolRows(ctx context.Context, table string, selector *telemetrytypes.FieldValueSelector, names []string) (bool, error) {
sb := sqlbuilder.Select("1").From(table)
sb.Where(sb.E("tag_key", selector.Name))
sb.Where(tagKeyCondition(sb, selector.Name, names))
sb.Where(sb.E("tag_data_type", telemetrytypes.FieldDataTypeBool.TagDataType()))
if selector.FieldContext != telemetrytypes.FieldContextUnspecified {
sb.Where(sb.E("tag_type", selector.FieldContext.TagType()))
@@ -1494,7 +1523,7 @@ func (t *telemetryMetaStore) tagTableHasBoolRows(ctx context.Context, table stri
// getTagTableValues returns the string and number values of the key from a
// tag table, and true and false when the key is a known bool field or the
// table holds bool rows for it. Bool rows do not count towards the limit.
func (t *telemetryMetaStore) getTagTableValues(ctx context.Context, table string, fieldValueSelector *telemetrytypes.FieldValueSelector, knownBool bool) (*telemetrytypes.TelemetryFieldValues, bool, error) {
func (t *telemetryMetaStore) getTagTableValues(ctx context.Context, table string, fieldValueSelector *telemetrytypes.FieldValueSelector, names []string, knownBool bool) (*telemetrytypes.TelemetryFieldValues, bool, error) {
limit := fieldValueSelector.Limit
if limit == 0 {
limit = 50
@@ -1507,7 +1536,7 @@ func (t *telemetryMetaStore) getTagTableValues(ctx context.Context, table string
return values, true, nil
}
} else if fieldValueSelector.FieldDataType == telemetrytypes.FieldDataTypeUnspecified {
hasBoolRows, err := t.tagTableHasBoolRows(ctx, table, fieldValueSelector)
hasBoolRows, err := t.tagTableHasBoolRows(ctx, table, fieldValueSelector, names)
if err != nil {
return nil, false, err
}
@@ -1519,7 +1548,7 @@ func (t *telemetryMetaStore) getTagTableValues(ctx context.Context, table string
sb := sqlbuilder.Select("DISTINCT string_value, number_value").From(table)
if fieldValueSelector.Name != "" {
sb.Where(sb.E("tag_key", fieldValueSelector.Name))
sb.Where(tagKeyCondition(sb, fieldValueSelector.Name, names))
}
sb.Where(sb.NE("tag_data_type", telemetrytypes.FieldDataTypeBool.TagDataType()))
@@ -1719,7 +1748,11 @@ func (t *telemetryMetaStore) getMetricFieldValues(ctx context.Context, orgID val
From(t.metricsDBName + "." + t.metricsFieldsTblName)
if fieldValueSelector.Name != "" {
sb.Where(sb.E("attr_name", fieldValueSelector.Name))
if names := t.familyValueNames(ctx, orgID, telemetrytypes.SignalMetrics, fieldValueSelector); len(names) > 1 {
sb.Where(sb.In("attr_name", sqlbuilder.List(names)))
} else {
sb.Where(sb.E("attr_name", fieldValueSelector.Name))
}
}
if fieldValueSelector.FieldContext != telemetrytypes.FieldContextUnspecified {
@@ -1731,7 +1764,11 @@ func (t *telemetryMetaStore) getMetricFieldValues(ctx context.Context, orgID val
}
if fieldValueSelector.MetricContext != nil && fieldValueSelector.MetricContext.MetricName != "" {
sb.Where(sb.E("metric_name", fieldValueSelector.MetricContext.MetricName))
if metricNames := querybuilder.FamilyMetricNames(ctx, orgID, t.fl, fieldValueSelector.MetricContext.MetricName); len(metricNames) > 1 {
sb.Where(sb.In("metric_name", sqlbuilder.List(metricNames)))
} else {
sb.Where(sb.E("metric_name", fieldValueSelector.MetricContext.MetricName))
}
}
if fieldValueSelector.MetricContext != nil && fieldValueSelector.MetricContext.MetricNamespace != "" {
sb.Where(sb.Like("metric_name", clickhousesql.LikePattern(fieldValueSelector.MetricContext.MetricNamespace)+"%"))
@@ -2053,12 +2090,12 @@ func (t *telemetryMetaStore) GetAllValues(ctx context.Context, orgID valuer.UUID
switch fieldValueSelector.Signal {
case telemetrytypes.SignalTraces:
values, complete, err = t.getSpanFieldValues(ctx, fieldValueSelector)
values, complete, err = t.getSpanFieldValues(ctx, orgID, fieldValueSelector)
case telemetrytypes.SignalLogs:
if fieldValueSelector.Source == telemetrytypes.SourceAudit {
values, complete, err = t.getAuditFieldValues(ctx, fieldValueSelector)
} else {
values, complete, err = t.getLogFieldValues(ctx, fieldValueSelector)
values, complete, err = t.getLogFieldValues(ctx, orgID, fieldValueSelector)
}
case telemetrytypes.SignalMetrics:
if fieldValueSelector.Source == telemetrytypes.SourceMeter {
@@ -2071,13 +2108,13 @@ func (t *telemetryMetaStore) GetAllValues(ctx context.Context, orgID valuer.UUID
mapOfRelatedValues := make(map[any]bool)
allUnspecifiedValues := &telemetrytypes.TelemetryFieldValues{}
tracesValues, tracesComplete, err := t.getSpanFieldValues(ctx, fieldValueSelector)
tracesValues, tracesComplete, err := t.getSpanFieldValues(ctx, orgID, fieldValueSelector)
if err == nil {
populateComplete := populateAllUnspecifiedValues(allUnspecifiedValues, mapOfValues, mapOfRelatedValues, tracesValues, limit)
complete = complete && tracesComplete && populateComplete
}
logsValues, logsComplete, err := t.getLogFieldValues(ctx, fieldValueSelector)
logsValues, logsComplete, err := t.getLogFieldValues(ctx, orgID, fieldValueSelector)
if err == nil {
populateComplete := populateAllUnspecifiedValues(allUnspecifiedValues, mapOfValues, mapOfRelatedValues, logsValues, limit)
complete = complete && logsComplete && populateComplete
@@ -2590,9 +2627,32 @@ func (t *telemetryMetaStore) fetchLastSeenInfoForTable(ctx context.Context, tabl
}
// containsConditions compiles a contains search on one key of the related
// values table. The key is its own metadata.
func (t *telemetryMetaStore) containsConditions(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, value string, sb *sqlbuilder.SelectBuilder) ([]string, error) {
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {key}}
// values table. The key and its family spellings are their own metadata,
// so the search narrows the suggestions across the whole family.
func (t *telemetryMetaStore) containsConditions(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, names []string, value string, sb *sqlbuilder.SelectBuilder) ([]string, error) {
fieldKeys := make(map[string][]*telemetrytypes.TelemetryFieldKey, len(names))
for _, name := range names {
fieldKeys[name] = []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(name, key.FieldContext, key.FieldDataType)}
}
conds, _, err := querybuilder.Conditions(ctx, q, t.storage, key, qbtypes.FilterOperatorContains, value, fieldKeys, false, sb)
return conds, err
}
// familyValueNames returns the spellings whose stored values merge into the
// suggestions for the requested name. With the flag off, the requested name
// alone.
func (t *telemetryMetaStore) familyValueNames(ctx context.Context, orgID valuer.UUID, signal telemetrytypes.Signal, fieldValueSelector *telemetrytypes.FieldValueSelector) []string {
if !querybuilder.SemconvFamiliesEnabled(ctx, orgID, t.fl) {
return []string{fieldValueSelector.Name}
}
selector := telemetrytypes.FieldKeySelector{
Name: fieldValueSelector.Name,
Signal: signal,
FieldContext: fieldValueSelector.FieldContext,
MetricContext: fieldValueSelector.MetricContext,
}
if signal == telemetrytypes.SignalMetrics {
return querybuilder.MetricLabelSpellings(selector)
}
return semconv.Members(semconv.KindAttribute, selector)
}

View File

@@ -247,6 +247,14 @@ type FieldKeySelector struct {
MetricContext *MetricContext `json:"metricContext,omitempty"`
}
// Identity is the selector's signal, context, data type, and the given
// name, joined into one key. Two selectors with the same identity fetch the
// same metadata. The metric context is not part of it: a caller that
// selects per metric name duplicates the selectors after this key is used.
func (s *FieldKeySelector) Identity(name string) string {
return s.Signal.StringValue() + ";" + s.FieldContext.StringValue() + ";" + s.FieldDataType.StringValue() + ";" + name
}
// MatchesKey reports whether a statically defined key satisfies the selector, so
// callers can suggest keys that were never ingested.
func (s *FieldKeySelector) MatchesKey(key *TelemetryFieldKey) bool {

View File

@@ -64,6 +64,9 @@ type overlayFile struct {
Families map[string]overlayFamily `yaml:"families"`
}
// Contexts and Signals set the family-level gate: the axes a family may
// resolve on at all. The Add* fields widen the schema-derived member scopes,
// for renames SigNoz applies beyond where the schema published them.
type overlayFamily struct {
Enabled *bool `yaml:"enabled"`
Kind string `yaml:"kind"`
@@ -79,27 +82,37 @@ type overlayFamily struct {
ValueMap map[string]string `yaml:"value_map"`
}
type edge struct {
old string
current string
kind string
// scope is the constraint a rename edge carries. A nil axis places no
// constraint on that axis.
type scope struct {
contexts []string
signals []string
allContexts bool
allSignals bool
applyToMetrics []string
}
type edge struct {
old string
current string
kind string
scope scope
}
type graphKey struct{ kind, name string }
type generatedFamily struct {
Current string
Old []string
Kind string
type generatedMember struct {
Name string
Contexts []string
Signals []string
ApplyToMetrics []string
ValueMap map[string]string
}
type generatedFamily struct {
Current string
Kind string
Members []generatedMember
Contexts []string
Signals []string
ValueMap map[string]string
}
func main() {
@@ -232,25 +245,31 @@ func collectEdges(schemas []schemaFile) ([]edge, error) {
{name: "metrics", section: version.Metrics},
}
for _, scoped := range sections {
contexts, signals, allContexts, allSignals, err := scopeForSection(scoped.name)
sectionScope, err := scopeForSection(scoped.name)
if err != nil {
return nil, err
}
for _, change := range scoped.section.Changes {
if change.RenameAttributes != nil {
if change.RenameAttributes.ApplyToMetrics != nil && len(change.RenameAttributes.ApplyToMetrics) == 0 {
return nil, fmt.Errorf(
"schema version %q has an explicitly empty apply_to_metrics; an empty list would be emitted as unconstrained",
versionName,
)
}
edgeScope := sectionScope
edgeScope.applyToMetrics = change.RenameAttributes.ApplyToMetrics
for _, old := range sortedMapKeys(change.RenameAttributes.AttributeMap) {
versionEdges = append(versionEdges, edge{
old: old, current: change.RenameAttributes.AttributeMap[old], kind: kindAttribute,
contexts: contexts, signals: signals,
allContexts: allContexts, allSignals: allSignals,
applyToMetrics: change.RenameAttributes.ApplyToMetrics,
scope: edgeScope,
})
}
}
for _, old := range sortedMapKeys(change.RenameMetrics) {
versionEdges = append(versionEdges, edge{
old: old, current: change.RenameMetrics[old], kind: kindMetric,
contexts: []string{"metric"}, signals: []string{"metrics"},
scope: scope{contexts: []string{"metric"}, signals: []string{"metrics"}},
})
}
}
@@ -311,99 +330,197 @@ func compareVersionParts(left, right [3]int) int {
return 0
}
func scopeForSection(section string) (contexts, signals []string, allContexts, allSignals bool, err error) {
func scopeForSection(section string) (scope, error) {
switch section {
case "all":
return nil, nil, true, true, nil
return scope{}, nil
case "resources":
return []string{"resource"}, nil, false, true, nil
return scope{contexts: []string{"resource"}}, nil
case "spans":
return []string{"attribute"}, []string{"traces"}, false, false, nil
return scope{contexts: []string{"attribute"}, signals: []string{"traces"}}, nil
case "logs":
return []string{"attribute"}, []string{"logs"}, false, false, nil
return scope{contexts: []string{"attribute"}, signals: []string{"logs"}}, nil
case "metrics":
return []string{"attribute"}, []string{"metrics"}, false, false, nil
return scope{contexts: []string{"attribute"}, signals: []string{"metrics"}}, nil
default:
return nil, nil, false, false, fmt.Errorf("unsupported schema section %q", section)
return scope{}, fmt.Errorf("unsupported schema section %q", section)
}
}
// unionScope widens per axis: no constraint on either side widens to no
// constraint. The metric-name axis only takes evidence from edges that can
// apply to metrics. A rename filed under a non-metrics section says nothing
// about metric scoping, and must not erase a scoped list into a wildcard.
func unionScope(left, right scope) scope {
return scope{
contexts: unionAxis(left.contexts, right.contexts),
signals: unionAxis(left.signals, right.signals),
applyToMetrics: applyToMetricsUnion(left, right),
}
}
func applyToMetricsUnion(left, right scope) []string {
leftApplies := coversMetrics(left.signals)
rightApplies := coversMetrics(right.signals)
switch {
case leftApplies && rightApplies:
return unionAxis(left.applyToMetrics, right.applyToMetrics)
case leftApplies:
return sortedCopy(left.applyToMetrics)
case rightApplies:
return sortedCopy(right.applyToMetrics)
}
return nil
}
func coversMetrics(signals []string) bool {
if signals == nil {
return true
}
for _, signal := range signals {
if signal == "metrics" {
return true
}
}
return false
}
func unionAxis(left, right []string) []string {
if left == nil || right == nil {
return nil
}
merged := appendUnique(append([]string(nil), left...), right...)
sort.Strings(merged)
return merged
}
// pathResult is one resolution path from a name to a family root: the root
// name and the hop count. Hops carry only reachability. A member's scope
// comes from its own rename edges, because the section a later rename is
// filed under says nothing about where the older spelling existed. The
// vendored schema files chained renames under different sections.
type pathResult struct {
root string
distance int
}
func rootsFor(next map[graphKey][]edge, kind, name string, distance int, seen map[string]bool) ([]pathResult, error) {
if seen[name] {
return nil, fmt.Errorf("rename cycle for %s %q", kind, name)
}
outgoing := next[graphKey{kind: kind, name: name}]
if len(outgoing) == 0 {
return []pathResult{{root: name, distance: distance}}, nil
}
seen[name] = true
defer delete(seen, name)
var results []pathResult
for _, hop := range outgoing {
hopResults, err := rootsFor(next, kind, hop.current, distance+1, seen)
if err != nil {
return nil, err
}
results = append(results, hopResults...)
}
return results, nil
}
func buildFamilies(schemas []schemaFile, overlay overlayFile) ([]generatedFamily, error) {
edges, err := collectEdges(schemas)
if err != nil {
return nil, err
}
next := make(map[graphKey]string)
// One old name can fan out into several families when its rename edges are
// scoped differently, so the graph keeps every successor.
next := make(map[graphKey][]edge)
for _, item := range edges {
key := graphKey{kind: item.kind, name: item.old}
if existing, ok := next[key]; ok && existing == item.current {
// Repeated entries are common in chained schema histories. Treat an
// identical edge as a no-op so it cannot sever a later edge in the
// same chain (A -> B, B -> C, then a repeated A -> B).
merged := false
for i, existing := range next[key] {
if existing.current == item.current {
// Repeated entries are common in chained schema histories. Merge
// the scopes instead of appending so a repeat cannot sever a
// later edge in the same chain (A -> B, B -> C, then a repeated
// A -> B).
next[key][i].scope = unionScope(existing.scope, item.scope)
merged = true
break
}
}
if merged {
continue
}
// Schema history occasionally repeats an old name with a newer direct
// destination or rolls a rename back. Edges are collected
// oldest-to-newest, so the latest published current name must be a root.
// oldest-to-newest, so the latest published current name must be a
// root. The delete removes every outgoing edge of the re-published
// name: a family only reachable through it becomes orphaned. The
// vendored history contains only true rollbacks, where the orphan is
// the correct result.
delete(next, graphKey{kind: item.kind, name: item.current})
next[key] = item.current
next[key] = append(next[key], item)
}
type memberState struct {
sc scope
distance int
}
type familyState struct {
family generatedFamily
distance map[string]int
allContexts bool
allSignals bool
family generatedFamily
members map[string]*memberState
}
states := map[graphKey]*familyState{}
for _, item := range edges {
root, distance, err := rootFor(next, item.kind, item.old)
if err != nil {
return nil, err
}
key := graphKey{kind: item.kind, name: root}
state := states[key]
if state == nil {
state = &familyState{
family: generatedFamily{Current: root, Kind: item.kind},
distance: map[string]int{},
for key, outgoing := range next {
for _, item := range outgoing {
results, err := rootsFor(next, key.kind, item.current, 1, map[string]bool{key.name: true})
if err != nil {
return nil, err
}
for _, result := range results {
rootKey := graphKey{kind: key.kind, name: result.root}
state := states[rootKey]
if state == nil {
state = &familyState{
family: generatedFamily{Current: result.root, Kind: key.kind},
members: map[string]*memberState{},
}
states[rootKey] = state
}
member := state.members[key.name]
if member == nil {
state.members[key.name] = &memberState{sc: item.scope, distance: result.distance}
continue
}
member.sc = unionScope(member.sc, item.scope)
if result.distance < member.distance {
member.distance = result.distance
}
}
states[key] = state
}
if prior, ok := state.distance[item.old]; !ok || distance < prior {
state.distance[item.old] = distance
}
state.allContexts = state.allContexts || item.allContexts
state.allSignals = state.allSignals || item.allSignals
state.family.Contexts = appendUnique(state.family.Contexts, item.contexts...)
state.family.Signals = appendUnique(state.family.Signals, item.signals...)
state.family.ApplyToMetrics = appendUnique(state.family.ApplyToMetrics, item.applyToMetrics...)
}
for _, state := range states {
for old := range state.distance {
if old != state.family.Current {
state.family.Old = append(state.family.Old, old)
}
names := make([]string, 0, len(state.members))
for name := range state.members {
names = append(names, name)
}
sort.Slice(state.family.Old, func(i, j int) bool {
left, right := state.family.Old[i], state.family.Old[j]
if state.distance[left] != state.distance[right] {
return state.distance[left] < state.distance[right]
sort.Slice(names, func(i, j int) bool {
left, right := state.members[names[i]], state.members[names[j]]
if left.distance != right.distance {
return left.distance < right.distance
}
return left < right
return names[i] < names[j]
})
if state.allContexts {
state.family.Contexts = nil
} else {
sort.Strings(state.family.Contexts)
for _, name := range names {
member := state.members[name]
state.family.Members = append(state.family.Members, generatedMember{
Name: name,
Contexts: sortedCopy(member.sc.contexts),
Signals: sortedCopy(member.sc.signals),
ApplyToMetrics: sortedCopy(member.sc.applyToMetrics),
})
}
if state.allSignals {
state.family.Signals = nil
} else {
sort.Strings(state.family.Signals)
}
sort.Strings(state.family.ApplyToMetrics)
}
for _, current := range sortedMapKeys(overlay.Families) {
@@ -414,6 +531,12 @@ func buildFamilies(schemas []schemaFile, overlay overlayFile) ([]generatedFamily
}
policy.Kind = kind
overlay.Families[current] = policy
if policy.ApplyToMetrics != nil && len(policy.ApplyToMetrics) == 0 {
return nil, fmt.Errorf(
"overlay family %q has an explicitly empty apply_to_metrics; an empty list would be emitted as unconstrained",
current,
)
}
key := graphKey{kind: kind, name: current}
state := states[key]
if state == nil {
@@ -424,10 +547,7 @@ func buildFamilies(schemas []schemaFile, overlay overlayFile) ([]generatedFamily
kind,
)
}
state = &familyState{
family: generatedFamily{Current: current, Kind: kind, Old: append([]string(nil), policy.Old...)},
distance: map[string]int{},
}
state = &familyState{family: generatedFamily{Current: current, Kind: kind}}
states[key] = state
}
applyOverlay(&state.family, policy)
@@ -446,7 +566,7 @@ func buildFamilies(schemas []schemaFile, overlay overlayFile) ([]generatedFamily
if !enabled {
continue
}
if len(state.family.Old) == 0 {
if len(state.family.Members) == 0 {
return nil, fmt.Errorf(
"enabled family %q with kind %q has no old members",
state.family.Current,
@@ -455,7 +575,6 @@ func buildFamilies(schemas []schemaFile, overlay overlayFile) ([]generatedFamily
}
sort.Strings(state.family.Contexts)
sort.Strings(state.family.Signals)
sort.Strings(state.family.ApplyToMetrics)
result = append(result, state.family)
}
@@ -468,21 +587,13 @@ func buildFamilies(schemas []schemaFile, overlay overlayFile) ([]generatedFamily
return result, nil
}
func rootFor(next map[graphKey]string, kind, name string) (string, int, error) {
seen := map[string]bool{}
distance := 0
for {
if seen[name] {
return "", 0, fmt.Errorf("rename cycle for %s %q", kind, name)
}
seen[name] = true
current, ok := next[graphKey{kind: kind, name: name}]
if !ok {
return name, distance, nil
}
name = current
distance++
func sortedCopy(values []string) []string {
if values == nil {
return nil
}
out := append([]string(nil), values...)
sort.Strings(out)
return out
}
func normalizedOverlayKind(current string, policy overlayFamily) (string, error) {
@@ -501,15 +612,29 @@ func applyOverlay(family *generatedFamily, policy overlayFamily) {
family.Kind = policy.Kind
}
if policy.Old != nil {
family.Old = append([]string(nil), policy.Old...)
family.Members = nil
for _, old := range policy.Old {
family.Members = append(family.Members, generatedMember{Name: old})
}
}
for _, old := range policy.AddOld {
if familyHasMember(family, old) {
continue
}
family.Members = append(family.Members, generatedMember{Name: old})
}
family.Old = appendUnique(family.Old, policy.AddOld...)
if len(policy.ExcludeOld) > 0 {
excluded := make(map[string]bool, len(policy.ExcludeOld))
for _, old := range policy.ExcludeOld {
excluded[old] = true
}
family.Old = deleteMatching(family.Old, excluded)
kept := family.Members[:0]
for _, member := range family.Members {
if !excluded[member.Name] {
kept = append(kept, member)
}
}
family.Members = kept
}
if policy.Contexts != nil {
family.Contexts = append([]string(nil), policy.Contexts...)
@@ -517,12 +642,28 @@ func applyOverlay(family *generatedFamily, policy overlayFamily) {
if policy.Signals != nil {
family.Signals = append([]string(nil), policy.Signals...)
}
family.Contexts = appendUnique(family.Contexts, policy.AddContexts...)
family.Signals = appendUnique(family.Signals, policy.AddSignals...)
if policy.ApplyToMetrics != nil {
family.ApplyToMetrics = append([]string(nil), policy.ApplyToMetrics...)
// Add* fields only widen: a nil gate already admits everything, so they
// extend a gate only when the overlay set one.
if family.Contexts != nil {
family.Contexts = appendUnique(family.Contexts, policy.AddContexts...)
}
if family.Signals != nil {
family.Signals = appendUnique(family.Signals, policy.AddSignals...)
}
for i := range family.Members {
if len(policy.AddContexts) > 0 {
family.Members[i].Contexts = unionAxis(family.Members[i].Contexts, sortedCopy(policy.AddContexts))
}
if len(policy.AddSignals) > 0 {
family.Members[i].Signals = unionAxis(family.Members[i].Signals, sortedCopy(policy.AddSignals))
}
if policy.ApplyToMetrics != nil {
family.Members[i].ApplyToMetrics = sortedCopy(policy.ApplyToMetrics)
}
if len(policy.AddApplyToMetrics) > 0 && family.Members[i].ApplyToMetrics != nil {
family.Members[i].ApplyToMetrics = unionAxis(family.Members[i].ApplyToMetrics, sortedCopy(policy.AddApplyToMetrics))
}
}
family.ApplyToMetrics = appendUnique(family.ApplyToMetrics, policy.AddApplyToMetrics...)
if policy.ValueMap != nil {
family.ValueMap = make(map[string]string, len(policy.ValueMap))
for old, current := range policy.ValueMap {
@@ -531,6 +672,15 @@ func applyOverlay(family *generatedFamily, policy overlayFamily) {
}
}
func familyHasMember(family *generatedFamily, name string) bool {
for _, member := range family.Members {
if member.Name == name {
return true
}
}
return false
}
func appendUnique(values []string, additions ...string) []string {
seen := make(map[string]bool, len(values)+len(additions))
for _, value := range values {
@@ -546,16 +696,6 @@ func appendUnique(values []string, additions ...string) []string {
return values
}
func deleteMatching(values []string, excluded map[string]bool) []string {
result := values[:0]
for _, value := range values {
if !excluded[value] {
result = append(result, value)
}
}
return result
}
func renderGo(families []generatedFamily) ([]byte, error) {
var out bytes.Buffer
out.WriteString("// Code generated by scripts/semconv. DO NOT EDIT.\n\n")
@@ -564,7 +704,11 @@ func renderGo(families []generatedFamily) ([]byte, error) {
for _, family := range families {
if len(family.Contexts) > 0 || len(family.Signals) > 0 {
needsTelemetryTypes = true
break
}
for _, member := range family.Members {
if len(member.Contexts) > 0 || len(member.Signals) > 0 {
needsTelemetryTypes = true
}
}
}
if needsTelemetryTypes {
@@ -572,32 +716,36 @@ func renderGo(families []generatedFamily) ([]byte, error) {
}
out.WriteString("var families = []Family{\n")
for _, family := range families {
contexts, err := goFieldContextSlice(family.Contexts)
if err != nil {
return nil, fmt.Errorf("render family %q: %w", family.Current, err)
}
signals, err := goSignalSlice(family.Signals)
if err != nil {
return nil, fmt.Errorf("render family %q: %w", family.Current, err)
}
out.WriteString("\t{\n")
fmt.Fprintf(&out, "\t\tCurrent: %s,\n", strconv.Quote(family.Current))
fmt.Fprintf(&out, "\t\tOld: %s,\n", goStringSlice(family.Old))
fmt.Fprintf(&out, "\t\tcurrent: %s,\n", strconv.Quote(family.Current))
if family.Kind == kindMetric {
out.WriteString("\t\tKind: KindMetric,\n")
out.WriteString("\t\tkind: KindMetric,\n")
} else {
out.WriteString("\t\tKind: KindAttribute,\n")
out.WriteString("\t\tkind: KindAttribute,\n")
}
fmt.Fprintf(&out, "\t\tContexts: %s,\n", contexts)
fmt.Fprintf(&out, "\t\tSignals: %s,\n", signals)
fmt.Fprintf(&out, "\t\tApplyToMetrics: %s,\n", goStringSlice(family.ApplyToMetrics))
if len(family.ValueMap) > 0 {
out.WriteString("\t\tValueMap: map[string]string{\n")
keys := sortedMapKeys(family.ValueMap)
for _, key := range keys {
fmt.Fprintf(&out, "\t\t\t%s: %s,\n", strconv.Quote(key), strconv.Quote(family.ValueMap[key]))
out.WriteString("\t\tmembers: []Member{\n")
for _, member := range family.Members {
if err := writeGoMember(&out, family.Current, member); err != nil {
return nil, err
}
out.WriteString("\t\t},\n")
}
out.WriteString("\t\t},\n")
if len(family.Contexts) > 0 {
contexts, err := goFieldContextSlice(family.Contexts)
if err != nil {
return nil, fmt.Errorf("render family %q: %w", family.Current, err)
}
fmt.Fprintf(&out, "\t\tcontexts: %s,\n", contexts)
}
if len(family.Signals) > 0 {
signals, err := goSignalSlice(family.Signals)
if err != nil {
return nil, fmt.Errorf("render family %q: %w", family.Current, err)
}
fmt.Fprintf(&out, "\t\tsignals: %s,\n", signals)
}
if len(family.ValueMap) > 0 {
return nil, fmt.Errorf("family %q carries a value map, and the Go registry has no value-map reader yet", family.Current)
}
out.WriteString("\t},\n")
}
@@ -605,6 +753,29 @@ func renderGo(families []generatedFamily) ([]byte, error) {
return format.Source(out.Bytes())
}
func writeGoMember(out *bytes.Buffer, current string, member generatedMember) error {
parts := []string{fmt.Sprintf("name: %s", strconv.Quote(member.Name))}
if len(member.Contexts) > 0 {
contexts, err := goFieldContextSlice(member.Contexts)
if err != nil {
return fmt.Errorf("render family %q member %q: %w", current, member.Name, err)
}
parts = append(parts, "contexts: "+contexts)
}
if len(member.Signals) > 0 {
signals, err := goSignalSlice(member.Signals)
if err != nil {
return fmt.Errorf("render family %q member %q: %w", current, member.Name, err)
}
parts = append(parts, "signals: "+signals)
}
if len(member.ApplyToMetrics) > 0 {
parts = append(parts, "applyToMetrics: "+goStringSlice(member.ApplyToMetrics))
}
fmt.Fprintf(out, "\t\t\t{%s},\n", strings.Join(parts, ", "))
return nil
}
func goStringSlice(values []string) string {
if len(values) == 0 {
return "nil"
@@ -659,21 +830,36 @@ func goSignalSlice(values []string) (string, error) {
func renderTypeScript(families []generatedFamily) []byte {
var out bytes.Buffer
out.WriteString("// Code generated by scripts/semconv. DO NOT EDIT.\n\n")
out.WriteString("// An empty contexts/signals/applyToMetrics array places no constraint on\n")
out.WriteString("// that axis.\n")
out.WriteString("export type SemconvMember = {\n")
out.WriteString("\treadonly name: string;\n")
out.WriteString("\treadonly contexts: readonly string[];\n")
out.WriteString("\treadonly signals: readonly string[];\n")
out.WriteString("\treadonly applyToMetrics: readonly string[];\n};\n\n")
out.WriteString("export type SemconvFamily = {\n")
out.WriteString("\treadonly current: string;\n\treadonly old: readonly string[];\n")
out.WriteString("\treadonly current: string;\n")
out.WriteString("\treadonly kind: 'attribute' | 'metric';\n")
out.WriteString("\treadonly members: readonly SemconvMember[];\n")
out.WriteString("\treadonly contexts: readonly string[];\n\treadonly signals: readonly string[];\n")
out.WriteString("\treadonly applyToMetrics: readonly string[];\n")
out.WriteString("\treadonly valueMap: Readonly<Record<string, string>>;\n};\n\n")
out.WriteString("export const SEMCONV_FAMILIES: readonly SemconvFamily[] = [\n")
for _, family := range families {
out.WriteString("\t{\n")
fmt.Fprintf(&out, "\t\tcurrent: %s,\n", tsString(family.Current))
fmt.Fprintf(&out, "\t\told: %s,\n", tsStringSlice(family.Old))
fmt.Fprintf(&out, "\t\tkind: %s,\n", tsString(family.Kind))
out.WriteString("\t\tmembers: [\n")
for _, member := range family.Members {
out.WriteString("\t\t\t{\n")
fmt.Fprintf(&out, "\t\t\t\tname: %s,\n", tsString(member.Name))
fmt.Fprintf(&out, "\t\t\t\tcontexts: %s,\n", tsStringSlice(member.Contexts))
fmt.Fprintf(&out, "\t\t\t\tsignals: %s,\n", tsStringSlice(member.Signals))
fmt.Fprintf(&out, "\t\t\t\tapplyToMetrics: %s,\n", tsStringSlice(member.ApplyToMetrics))
out.WriteString("\t\t\t},\n")
}
out.WriteString("\t\t],\n")
fmt.Fprintf(&out, "\t\tcontexts: %s,\n", tsStringSlice(family.Contexts))
fmt.Fprintf(&out, "\t\tsignals: %s,\n", tsStringSlice(family.Signals))
fmt.Fprintf(&out, "\t\tapplyToMetrics: %s,\n", tsStringSlice(family.ApplyToMetrics))
out.WriteString("\t\tvalueMap: {")
keys := sortedMapKeys(family.ValueMap)
for i, key := range keys {

View File

@@ -37,6 +37,11 @@ versions:
assert.ErrorContains(t, err, `schema version "latest"`, "malformed versions must not be silently reordered")
}
func TestParseSchemaVersionRejectsNonNumericComponent(t *testing.T) {
_, err := parseSchemaVersion("1.2.x")
assert.ErrorContains(t, err, `invalid numeric component "x"`, "non-numeric version components must fail generation")
}
func TestBuildFamiliesResolvesRenameChain(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
@@ -71,11 +76,13 @@ versions:
}})
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "c",
Old: []string{"b", "x", "a"},
Kind: kindAttribute,
Contexts: []string{"attribute"},
Signals: []string{"traces"},
Current: "c",
Kind: kindAttribute,
Members: []generatedMember{
{Name: "b", Contexts: []string{"attribute"}, Signals: []string{"traces"}},
{Name: "x", Contexts: []string{"attribute"}, Signals: []string{"traces"}},
{Name: "a", Contexts: []string{"attribute"}, Signals: []string{"traces"}},
},
}}, families, "predecessors should be ordered by distance and then name")
}
@@ -120,27 +127,239 @@ versions:
require.NoError(t, err)
assert.Equal(t, []generatedFamily{
{
Current: "all.current", Old: []string{"all.old"}, Kind: kindAttribute,
Contexts: nil, Signals: nil,
Current: "all.current", Kind: kindAttribute,
Members: []generatedMember{{Name: "all.old"}},
},
{
Current: "cpu.mode", Old: []string{"state"}, Kind: kindAttribute,
Contexts: []string{"attribute"}, Signals: []string{"metrics"},
Current: "cpu.mode", Kind: kindAttribute,
Members: []generatedMember{{
Name: "state", Contexts: []string{"attribute"}, Signals: []string{"metrics"},
ApplyToMetrics: []string{"system.cpu.time"},
}},
},
{
Current: "current.metric", Kind: kindMetric,
Members: []generatedMember{{Name: "old.metric", Contexts: []string{"metric"}, Signals: []string{"metrics"}}},
},
{
Current: "log.current", Kind: kindAttribute,
Members: []generatedMember{{Name: "log.old", Contexts: []string{"attribute"}, Signals: []string{"logs"}}},
},
{
Current: "resource.current", Kind: kindAttribute,
Members: []generatedMember{{Name: "resource.old", Contexts: []string{"resource"}}},
},
}, families, "schema sections should produce their documented per-member signal and context scopes")
}
func TestBuildFamiliesKeepsApplyToMetricsThroughNonMetricsRepeat(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
versions:
2.0.0:
logs:
changes:
- rename_attributes:
attribute_map:
old: current
1.0.0:
metrics:
changes:
- rename_attributes:
attribute_map:
old: current
apply_to_metrics: [system.cpu.time]
`), &schema), "test schema must decode")
families, err := buildFamilies([]schemaFile{schema}, overlayFile{DefaultEnabled: true})
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "current", Kind: kindAttribute,
Members: []generatedMember{{
Name: "old", Contexts: []string{"attribute"}, Signals: []string{"logs", "metrics"},
ApplyToMetrics: []string{"system.cpu.time"},
}},
}}, families, "a repeat under a non-metrics section widens the signals and must not erase the metric scope")
}
func TestBuildFamiliesRejectsExplicitlyEmptyOverlayApplyToMetrics(t *testing.T) {
enabled := true
_, err := buildFamilies(nil, overlayFile{Families: map[string]overlayFamily{
"current": {Enabled: &enabled, Old: []string{"old"}, ApplyToMetrics: []string{}},
}})
assert.ErrorContains(t, err, "explicitly empty apply_to_metrics", "the overlay must not silently widen an empty list to every metric")
}
func TestBuildFamiliesRejectsExplicitlyEmptyApplyToMetrics(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
versions:
1.0.0:
metrics:
changes:
- rename_attributes:
attribute_map:
old: current
apply_to_metrics: []
`), &schema), "test schema must decode")
_, err := buildFamilies([]schemaFile{schema}, overlayFile{})
assert.ErrorContains(t, err, "explicitly empty apply_to_metrics", "an empty list must not silently widen to every metric")
}
func TestBuildFamiliesKeepsFanOutSeparate(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
versions:
2.0.0:
metrics:
changes:
- rename_attributes:
attribute_map:
state: cpu.mode
apply_to_metrics: [system.cpu.time]
1.0.0:
metrics:
changes:
- rename_attributes:
attribute_map:
state: db.client.connection.state
apply_to_metrics: [db.client.connections.usage]
`), &schema), "test schema must decode")
families, err := buildFamilies([]schemaFile{schema}, overlayFile{DefaultEnabled: true})
require.NoError(t, err)
assert.Equal(t, []generatedFamily{
{
Current: "cpu.mode", Kind: kindAttribute,
Members: []generatedMember{{
Name: "state", Contexts: []string{"attribute"}, Signals: []string{"metrics"},
ApplyToMetrics: []string{"system.cpu.time"},
}},
},
{
Current: "current.metric", Old: []string{"old.metric"}, Kind: kindMetric,
Contexts: []string{"metric"}, Signals: []string{"metrics"},
Current: "db.client.connection.state", Kind: kindAttribute,
Members: []generatedMember{{
Name: "state", Contexts: []string{"attribute"}, Signals: []string{"metrics"},
ApplyToMetrics: []string{"db.client.connections.usage"},
}},
},
{
Current: "log.current", Old: []string{"log.old"}, Kind: kindAttribute,
Contexts: []string{"attribute"}, Signals: []string{"logs"},
}, families, "an old name with differently scoped rename targets must keep one membership per target")
}
func TestBuildFamiliesKeepsUnscopedRenameUnscoped(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
versions:
2.0.0:
metrics:
changes:
- rename_attributes:
attribute_map:
direction: network.io.direction
apply_to_metrics: [system.disk.io, system.disk.merged]
1.0.0:
metrics:
changes:
- rename_attributes:
attribute_map:
system.network.io.direction: network.io.direction
`), &schema), "test schema must decode")
families, err := buildFamilies([]schemaFile{schema}, overlayFile{DefaultEnabled: true})
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "network.io.direction", Kind: kindAttribute,
Members: []generatedMember{
{Name: "direction", Contexts: []string{"attribute"}, Signals: []string{"metrics"}, ApplyToMetrics: []string{"system.disk.io", "system.disk.merged"}},
{Name: "system.network.io.direction", Contexts: []string{"attribute"}, Signals: []string{"metrics"}},
},
{
Current: "resource.current", Old: []string{"resource.old"}, Kind: kindAttribute,
Contexts: []string{"resource"},
}}, families, "a rename without apply_to_metrics stays unbounded; a scoped sibling must not bound it")
}
func TestBuildFamiliesKeepsMemberScopesThroughChains(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
versions:
2.0.0:
metrics:
changes:
- rename_attributes:
attribute_map:
messaging.client_id: messaging.client.id
1.0.0:
spans:
changes:
- rename_attributes:
attribute_map:
messaging.kafka.client_id: messaging.client_id
`), &schema), "test schema must decode")
families, err := buildFamilies([]schemaFile{schema}, overlayFile{DefaultEnabled: true})
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "messaging.client.id", Kind: kindAttribute,
Members: []generatedMember{
{Name: "messaging.client_id", Contexts: []string{"attribute"}, Signals: []string{"metrics"}},
{Name: "messaging.kafka.client_id", Contexts: []string{"attribute"}, Signals: []string{"traces"}},
},
}, families, "schema sections should produce their documented signal and context scopes")
}}, families, "a member keeps the scope of its own rename edge; later hops only carry it to the root")
}
func TestBuildFamiliesRecordsCrossContextMembersSeparately(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
versions:
1.0.0:
spans:
changes:
- rename_attributes:
attribute_map:
http.user_agent: user_agent.original
resources:
changes:
- rename_attributes:
attribute_map:
browser.user_agent: user_agent.original
`), &schema), "test schema must decode")
families, err := buildFamilies([]schemaFile{schema}, overlayFile{DefaultEnabled: true})
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "user_agent.original", Kind: kindAttribute,
Members: []generatedMember{
{Name: "browser.user_agent", Contexts: []string{"resource"}},
{Name: "http.user_agent", Contexts: []string{"attribute"}, Signals: []string{"traces"}},
},
}}, families, "members renamed from different contexts must keep their own context scopes")
}
func TestBuildFamiliesMergesRepeatedEdgeScopes(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
versions:
2.0.0:
logs:
changes:
- rename_attributes:
attribute_map:
old: current
1.0.0:
spans:
changes:
- rename_attributes:
attribute_map:
old: current
`), &schema), "test schema must decode")
families, err := buildFamilies([]schemaFile{schema}, overlayFile{DefaultEnabled: true})
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "current", Kind: kindAttribute,
Members: []generatedMember{
{Name: "old", Contexts: []string{"attribute"}, Signals: []string{"logs", "traces"}},
},
}}, families, "the same rename filed under several sections widens the member scope")
}
func TestOverlayAddsFamilyWithoutSchemaHistory(t *testing.T) {
@@ -157,8 +376,8 @@ func TestOverlayAddsFamilyWithoutSchemaHistory(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "added.current",
Old: []string{"added.old"},
Kind: kindAttribute,
Members: []generatedMember{{Name: "added.old"}},
Contexts: []string{"resource"},
Signals: []string{"traces"},
}}, families, "an explicit overlay family should not require schema history")
@@ -179,26 +398,76 @@ versions:
enabled := true
families, err := buildFamilies([]schemaFile{schema}, overlayFile{Families: map[string]overlayFamily{
"current": {
Enabled: &enabled,
AddOld: []string{"older"},
ExcludeOld: []string{"old"},
AddContexts: []string{"resource"},
AddSignals: []string{"logs"},
ValueMap: map[string]string{"legacy": "current"},
Enabled: &enabled,
AddOld: []string{"older"},
ExcludeOld: []string{"old"},
AddSignals: []string{"logs"},
ValueMap: map[string]string{"legacy": "current"},
},
}})
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "current",
Old: []string{"older"},
Kind: kindAttribute,
Contexts: []string{"attribute", "resource"},
Signals: []string{"logs", "traces"},
Members: []generatedMember{{Name: "older"}},
ValueMap: map[string]string{"legacy": "current"},
}}, families, "overlay additions and exclusions should be applied to the generated family")
}
func TestOverlayAddSignalsWidensMemberScopes(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
versions:
1.0.0:
spans:
changes:
- rename_attributes:
attribute_map:
old: current
`), &schema), "test schema must decode")
enabled := true
families, err := buildFamilies([]schemaFile{schema}, overlayFile{Families: map[string]overlayFamily{
"current": {Enabled: &enabled, AddSignals: []string{"logs"}},
}})
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "current",
Kind: kindAttribute,
Members: []generatedMember{
{Name: "old", Contexts: []string{"attribute"}, Signals: []string{"logs", "traces"}},
},
}}, families, "add_signals widens the schema-derived member scopes and never narrows the family gate")
}
func TestOverlaySignalsSetTheFamilyGate(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
versions:
1.0.0:
all:
changes:
- rename_attributes:
attribute_map:
old: current
`), &schema), "test schema must decode")
enabled := true
families, err := buildFamilies([]schemaFile{schema}, overlayFile{Families: map[string]overlayFamily{
"current": {Enabled: &enabled, Signals: []string{"logs", "traces"}},
}})
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "current",
Kind: kindAttribute,
Members: []generatedMember{{Name: "old"}},
Signals: []string{"logs", "traces"},
}}, families, "the overlay signals list gates the family without touching member scopes")
}
func TestOverlayDisablesFamilyWhenDefaultIsEnabled(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
@@ -223,23 +492,21 @@ versions:
assert.Empty(t, families, "an explicitly disabled family must override default_enabled")
}
func TestRenderGoIsDeterministic(t *testing.T) {
func TestRenderGoRejectsValueMap(t *testing.T) {
families := []generatedFamily{{
Current: "current", Old: []string{"old"}, Kind: kindAttribute,
ValueMap: map[string]string{"b": "2", "a": "1"},
Current: "current", Kind: kindAttribute,
Members: []generatedMember{{Name: "old"}},
ValueMap: map[string]string{"a": "1"},
}}
first, err := renderGo(families)
require.NoError(t, err)
second, err := renderGo(families)
require.NoError(t, err)
assert.Equal(t, first, second, "Go generation must not depend on map iteration order")
_, err := renderGo(families)
assert.ErrorContains(t, err, "no value-map reader yet", "a value map must fail Go generation until a reader exists")
}
func TestRenderGoUsesCanonicalTelemetryTypes(t *testing.T) {
families := []generatedFamily{{
Current: "current", Old: []string{"old"}, Kind: kindAttribute,
Contexts: []string{"resource"}, Signals: []string{"traces"},
Current: "current", Kind: kindAttribute,
Members: []generatedMember{{Name: "old", Contexts: []string{"resource"}, Signals: []string{"traces"}}},
}}
output, err := renderGo(families)
@@ -248,15 +515,112 @@ func TestRenderGoUsesCanonicalTelemetryTypes(t *testing.T) {
assert.Contains(t, string(output), "telemetrytypes.SignalTraces", "generated signals should use telemetrytypes")
}
func TestRenderGoRejectsUnknownContext(t *testing.T) {
families := []generatedFamily{{
Current: "current", Kind: kindAttribute,
Members: []generatedMember{{Name: "old", Contexts: []string{"bogus"}}},
}}
_, err := renderGo(families)
assert.ErrorContains(t, err, `unsupported field context "bogus"`, "a bad overlay context must fail generation, not compile")
}
func TestRenderGoRejectsUnknownSignal(t *testing.T) {
families := []generatedFamily{{
Current: "current", Kind: kindAttribute,
Members: []generatedMember{{Name: "old"}},
Signals: []string{"bogus"},
}}
_, err := renderGo(families)
assert.ErrorContains(t, err, `unsupported signal "bogus"`, "a bad overlay signal must fail generation, not compile")
}
func TestRenderGoPinsOutput(t *testing.T) {
families := []generatedFamily{{
Current: "deployment.environment.name", Kind: kindAttribute,
Members: []generatedMember{{Name: "deployment.environment"}},
Signals: []string{"logs", "traces"},
}}
output, err := renderGo(families)
require.NoError(t, err)
assert.Equal(t, `// Code generated by scripts/semconv. DO NOT EDIT.
package semconv
import "github.com/SigNoz/signoz/pkg/types/telemetrytypes"
var families = []Family{
{
current: "deployment.environment.name",
kind: KindAttribute,
members: []Member{
{name: "deployment.environment"},
},
signals: []telemetrytypes.Signal{telemetrytypes.SignalLogs, telemetrytypes.SignalTraces},
},
}
`, string(output), "the emitted Go text is a contract; regeneration must be reviewable")
}
func TestRenderTypeScriptIsDeterministic(t *testing.T) {
families := []generatedFamily{{
Current: "current", Old: []string{"old"}, Kind: kindAttribute,
Current: "current", Kind: kindAttribute,
Members: []generatedMember{{Name: "old"}},
ValueMap: map[string]string{"b": "2", "a": "1"},
}}
assert.Equal(t, renderTypeScript(families), renderTypeScript(families), "TypeScript generation must not depend on map iteration order")
}
func TestRenderTypeScriptPinsOutput(t *testing.T) {
families := []generatedFamily{{
Current: "deployment.environment.name", Kind: kindAttribute,
Members: []generatedMember{{Name: "deployment.environment"}},
Signals: []string{"logs", "traces"},
}}
assert.Equal(t, `// Code generated by scripts/semconv. DO NOT EDIT.
// An empty contexts/signals/applyToMetrics array places no constraint on
// that axis.
export type SemconvMember = {
readonly name: string;
readonly contexts: readonly string[];
readonly signals: readonly string[];
readonly applyToMetrics: readonly string[];
};
export type SemconvFamily = {
readonly current: string;
readonly kind: 'attribute' | 'metric';
readonly members: readonly SemconvMember[];
readonly contexts: readonly string[];
readonly signals: readonly string[];
readonly valueMap: Readonly<Record<string, string>>;
};
export const SEMCONV_FAMILIES: readonly SemconvFamily[] = [
{
current: 'deployment.environment.name',
kind: 'attribute',
members: [
{
name: 'deployment.environment',
contexts: [],
signals: [],
applyToMetrics: [],
},
],
contexts: [],
signals: ['logs', 'traces'],
valueMap: {},
},
] as const;
`, string(renderTypeScript(families)), "the emitted TypeScript text is a contract; regeneration must be reviewable")
}
func TestBuildFamiliesHandlesRenameRollback(t *testing.T) {
var schema schemaFile
require.NoError(t, decodeKnownFields([]byte(`
@@ -280,11 +644,9 @@ versions:
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "original",
Old: []string{"temporary"},
Kind: kindMetric,
Contexts: []string{"metric"},
Signals: []string{"metrics"},
Current: "original",
Kind: kindMetric,
Members: []generatedMember{{Name: "temporary", Contexts: []string{"metric"}, Signals: []string{"metrics"}}},
}}, families, "the latest rollback destination should remain the family root")
}
@@ -356,11 +718,9 @@ versions:
require.NoError(t, err)
assert.Equal(t, []generatedFamily{{
Current: "shared.current",
Old: []string{"attribute.old"},
Kind: kindAttribute,
Contexts: []string{"attribute"},
Signals: []string{"traces"},
Current: "shared.current",
Kind: kindAttribute,
Members: []generatedMember{{Name: "attribute.old", Contexts: []string{"attribute"}, Signals: []string{"traces"}}},
}}, families, "a kind-less overlay policy should affect only the attribute family")
}

View File

@@ -2,10 +2,35 @@
#
# Families are keyed by their current OpenTelemetry name. Schema-derived
# families are disabled by default so rollout remains explicit and reversible.
# The signals list is the per-family rollout gate.
default_enabled: false
families:
deployment.environment.name:
enabled: true
signals: [traces, logs, metrics]
# The family names an attribute of the resource or the span/log, never a
# field inside a log body: without this gate a body-context key with the
# same path joins the family and skips the body-JSON machinery.
contexts: [resource, attribute]
db.system.name:
# The db.system value domain also renamed (mssql -> microsoft.sql_server
# and others), and no value mapping is read yet. A name-only merge matches
# half the history on every signal, so the family stays off until a
# value-mapping reader exists.
enabled: false
# These metric renames predate the vendored schema history, so the overlay
# declares the old names itself.
k8s.pod.cpu.usage:
enabled: true
kind: metric
old: [k8s.pod.cpu.utilization]
k8s.node.cpu.usage:
enabled: true
kind: metric
old: [k8s.node.cpu.utilization]
container.cpu.usage:
enabled: true
kind: metric
old: [container.cpu.utilization]

View File

@@ -1,22 +1,36 @@
from collections.abc import Callable, Generator
from datetime import UTC, datetime, timedelta
from http import HTTPStatus
from time import sleep
import pytest
from fixtures.logs import Logs
from fixtures.traces import TraceIdGenerator, Traces, TracesKind, TracesStatusCode
from fixtures import types
from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
from fixtures.logs import Logs, insert_logs_to_clickhouse
from fixtures.metrics import Metrics
from fixtures.querier import (
RequestType,
build_metrics_aggregation,
build_order_by,
build_raw_query,
build_scalar_query,
get_column_data_from_response,
get_scalar_table_data,
make_query_request,
)
from fixtures.traces import TraceIdGenerator, Traces, TracesKind, TracesStatusCode, insert_traces_to_clickhouse
PREFIX = "semconv-fam"
CURRENT_KEY = "deployment.environment.name"
OLD_KEY = "deployment.environment"
# Row identities. The span name, the log body, and service.name are the identity.
# Tests compare identity sets filtered by PREFIX, so reruns on a reused stack
# with leftover rows stay stable.
OLD = f"{PREFIX}-old" # only the old spelling, value "production"
NEW = f"{PREFIX}-new" # only the current spelling, value "production"
OLD = f"{PREFIX}-old"
NEW = f"{PREFIX}-new"
BOTH = f"{PREFIX}-both" # current "staging" and old "production" - the conflict row
NEITHER = f"{PREFIX}-neither" # no member at all
NEITHER = f"{PREFIX}-neither"
_ROWS = [
(OLD, {OLD_KEY: "production"}, timedelta(seconds=4)),
@@ -26,13 +40,17 @@ _ROWS = [
]
@pytest.fixture(name="family_fleet", scope="function")
def family_fleet(
insert_logs: Callable[[list[Logs]], None],
insert_traces: Callable[[list[Traces]], None],
) -> Generator[datetime]:
"""Yields the base timestamp of the inserted rows."""
now = datetime.now(tz=UTC).replace(microsecond=0) - timedelta(minutes=1)
@pytest.fixture(name="family_fleet", scope="package")
def family_fleet(clickhouse: types.TestContainerClickhouse) -> Generator[datetime]:
"""Seeds one time per package. The base aligns to the minute, so no row
offset crosses a 60s time-series bucket boundary."""
now = datetime.now(tz=UTC).replace(second=0, microsecond=0) - timedelta(minutes=1)
def insert_traces(traces: list[Traces]) -> None:
insert_traces_to_clickhouse(clickhouse.conn, traces)
def insert_logs(logs: list[Logs]) -> None:
insert_logs_to_clickhouse(clickhouse.conn, logs)
insert_traces(
[
@@ -62,3 +80,122 @@ def family_fleet(
]
)
yield now
def query_identities(
signoz: types.SigNoz,
token: str,
base: datetime,
expression: str,
signal: str,
identity_field: str,
identity_column: str,
) -> set[str]:
response = make_query_request(
signoz,
token,
start_ms=int((base - timedelta(minutes=2)).timestamp() * 1000),
end_ms=int((base + timedelta(minutes=1)).timestamp() * 1000),
request_type=RequestType.RAW,
queries=[
build_raw_query(
"A",
signal,
limit=100,
filter_expression=expression,
order=[build_order_by("timestamp", "asc")],
select_fields=[{"name": identity_field}],
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
return {name for name in get_column_data_from_response(response.json(), identity_column) if name.startswith(PREFIX)}
LABEL_METRIC = "semconv.fam.label.metric"
OLD_NAME_METRIC = "k8s.pod.cpu.utilization"
CURRENT_NAME_METRIC = "k8s.pod.cpu.usage"
NORMALIZED_OLD_NAME_METRIC = "k8s_pod_cpu_utilization"
NORMALIZED_CURRENT_NAME_METRIC = "k8s_pod_cpu_usage"
def scalar_query(
signoz: types.SigNoz,
token: str,
now: datetime,
metric_name: str,
filter_expression: str | None = None,
group_by: list[dict] | None = None,
):
return make_query_request(
signoz,
token,
start_ms=int((now - timedelta(minutes=30)).timestamp() * 1000),
end_ms=int(now.timestamp() * 1000),
request_type=RequestType.SCALAR,
queries=[
build_scalar_query(
name="A",
signal="metrics",
aggregations=[build_metrics_aggregation(metric_name, "latest", "sum", "unspecified", reduce_to="last")],
filter_expression=filter_expression,
group_by=group_by,
)
],
)
def label_metric_sum(signoz: types.SigNoz, token: str, now: datetime, filter_expression: str) -> float:
response = scalar_query(signoz, token, now, LABEL_METRIC, filter_expression=filter_expression)
assert response.status_code == HTTPStatus.OK, response.text
rows = get_scalar_table_data(response.json())
return rows[0][-1] if rows else 0.0
@pytest.fixture(name="metric_family_fleet", scope="function")
def metric_family_fleet(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> datetime:
"""Every series has a power-of-two value, so a missed member is a unique
wrong sum."""
now = datetime.now(tz=UTC)
# The querier clamps very recent metric samples (flux interval), so the
# fleet sits in the past.
seeded = now - timedelta(minutes=10)
gauge = {"temporality": "Unspecified", "type_": "Gauge", "is_monotonic": False}
insert_metrics(
[
Metrics(metric_name=LABEL_METRIC, labels={"deployment.environment.name": "staging"}, timestamp=seeded, value=1.0, **gauge),
Metrics(metric_name=LABEL_METRIC, labels={"deployment.environment": "production"}, timestamp=seeded, value=2.0, **gauge),
Metrics(metric_name=LABEL_METRIC, labels={"deployment_environment": "production"}, timestamp=seeded, value=4.0, **gauge),
Metrics(metric_name=LABEL_METRIC, labels={"region": "keyless"}, timestamp=seeded, value=8.0, **gauge),
Metrics(metric_name=OLD_NAME_METRIC, labels={"pod": "a"}, timestamp=seeded, value=16.0, **gauge),
Metrics(metric_name=CURRENT_NAME_METRIC, labels={"pod": "b"}, timestamp=seeded, value=32.0, **gauge),
Metrics(metric_name=LABEL_METRIC, labels={"deployment.environment.name": "staging", "deployment.environment": "production"}, timestamp=seeded, value=64.0, **gauge),
Metrics(metric_name=LABEL_METRIC, labels={"resource_deployment_environment": "production"}, timestamp=seeded, value=128.0, **gauge),
Metrics(metric_name=NORMALIZED_OLD_NAME_METRIC, labels={"pod": "c"}, timestamp=seeded, value=256.0, **gauge),
Metrics(metric_name=NORMALIZED_CURRENT_NAME_METRIC, labels={"pod": "d"}, timestamp=seeded, value=512.0, **gauge),
]
)
# Metric metadata lags the insert. Each probe reads a label value only
# its own series carries, so the family union of the flag-on instance
# cannot pass before every member is queryable.
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
probes = [
(LABEL_METRIC, "region = 'keyless'"),
(OLD_NAME_METRIC, "pod = 'a'"),
(CURRENT_NAME_METRIC, "pod = 'b'"),
(NORMALIZED_OLD_NAME_METRIC, "pod = 'c'"),
(NORMALIZED_CURRENT_NAME_METRIC, "pod = 'd'"),
]
deadline = datetime.now(tz=UTC) + timedelta(seconds=60)
while datetime.now(tz=UTC) < deadline:
seen = [get_scalar_table_data(scalar_query(signoz, token, datetime.now(tz=UTC), metric, filter_expression=expression).json()) for metric, expression in probes]
if all(rows for rows in seen):
return now
sleep(1)
raise AssertionError(f"seeded metrics never became queryable: {seen}")

View File

@@ -17,7 +17,7 @@ pkg/types/querybuildertypes/querybuildertypesv5/builder_elements.go):
Any implementation change that makes these assertions fail is a behavior
break, not a cleanup. Family-field behavior must mirror this matrix; see
queriertraces/13_semconv_evolution.py.
semconvfamilies/01_family_matrix.py.
Seed data lives in fixtures/queriercommon.py: GOLD and SILVER carry the
keys, NONE carries none. Every case asserts which identities a filter

View File

@@ -6,14 +6,19 @@ import pytest
from fixtures import types
from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
from fixtures.metadata import get_field_keys, get_field_values
from fixtures.querier import (
RequestType,
build_aggregation,
build_group_by_field,
build_order_by,
build_raw_query,
build_scalar_query,
build_traces_scalar_query,
get_all_warnings,
get_column_data_from_response,
get_scalar_columns,
get_scalar_table_data,
make_query_request,
)
from fixtures.semconvfamilies import (
@@ -24,6 +29,7 @@ from fixtures.semconvfamilies import (
OLD,
OLD_KEY,
PREFIX,
query_identities,
)
FILTER_MATRIX = [
@@ -33,6 +39,11 @@ FILTER_MATRIX = [
pytest.param("{key} IN ['production', 'staging']", {OLD, NEW, BOTH}, id="in_matches_merged_value"),
pytest.param("{key} NOT IN ['production']", {BOTH, NEITHER}, id="not_in_keeps_keyless"),
pytest.param("{key} LIKE '%prod%'", {OLD, NEW}, id="like_matches_merged_value"),
pytest.param("{key} NOT LIKE '%prod%'", {BOTH, NEITHER}, id="not_like_keeps_keyless"),
pytest.param("{key} ILIKE 'PROD%'", {OLD, NEW}, id="ilike_matches_merged_value"),
pytest.param("{key} CONTAINS 'oduct'", {OLD, NEW}, id="contains_matches_merged_value"),
pytest.param("{key} REGEXP '^prod.*'", {OLD, NEW}, id="regexp_matches_merged_value"),
pytest.param("{key} NOT CONTAINS 'prod'", {BOTH, NEITHER}, id="not_contains_keeps_keyless"),
pytest.param("{key} EXISTS", {OLD, NEW, BOTH}, id="exists_is_any_member"),
pytest.param("{key} NOT EXISTS", {NEITHER}, id="not_exists_is_no_member"),
pytest.param("{key} != 'production' AND {key} EXISTS", {BOTH}, id="neq_composed_with_exists"),
@@ -60,25 +71,26 @@ def test_family_filters(
"""The result set is a property of the family, not of the requested spelling."""
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
expression = expression_template.format(key=f"{context}.{requested_key}")
response = make_query_request(
signoz,
token,
start_ms=int((family_fleet - timedelta(minutes=2)).timestamp() * 1000),
end_ms=int((family_fleet + timedelta(minutes=1)).timestamp() * 1000),
request_type=RequestType.RAW,
queries=[
build_raw_query(
"A",
"traces",
limit=100,
filter_expression=expression,
order=[build_order_by("timestamp", "asc")],
select_fields=[{"name": "span.name"}],
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
matched = {name for name in get_column_data_from_response(response.json(), "name") if name.startswith(PREFIX)}
matched = query_identities(signoz, token, family_fleet, expression, "traces", "span.name", "name")
assert matched == expected, expression
@pytest.mark.parametrize("expression_template,expected", FILTER_MATRIX)
@pytest.mark.parametrize("requested_key", [CURRENT_KEY, OLD_KEY], ids=["current", "old"])
@pytest.mark.parametrize("context", ["resource", "attribute"])
def test_log_family_filters(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
family_fleet: datetime,
context: str,
requested_key: str,
expression_template: str,
expected: set[str],
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
expression = expression_template.format(key=f"{context}.{requested_key}")
matched = query_identities(signoz, token, family_fleet, expression, "logs", "body", "body")
assert matched == expected, expression
@@ -93,59 +105,7 @@ def test_flag_off_stays_literal(
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
expression = expression_template.format(key=f"resource.{CURRENT_KEY}")
response = make_query_request(
signoz_families_off,
token,
start_ms=int((family_fleet - timedelta(minutes=2)).timestamp() * 1000),
end_ms=int((family_fleet + timedelta(minutes=1)).timestamp() * 1000),
request_type=RequestType.RAW,
queries=[
build_raw_query(
"A",
"traces",
limit=100,
filter_expression=expression,
order=[build_order_by("timestamp", "asc")],
select_fields=[{"name": "span.name"}],
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
matched = {name for name in get_column_data_from_response(response.json(), "name") if name.startswith(PREFIX)}
assert matched == expected, expression
@pytest.mark.parametrize("expression_template,expected", LITERAL_MATRIX)
def test_logs_stay_literal_with_flag_on(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
family_fleet: datetime,
expression_template: str,
expected: set[str],
) -> None:
"""Only traces have family support today."""
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
expression = expression_template.format(key=f"resource.{CURRENT_KEY}")
response = make_query_request(
signoz,
token,
start_ms=int((family_fleet - timedelta(minutes=2)).timestamp() * 1000),
end_ms=int((family_fleet + timedelta(minutes=1)).timestamp() * 1000),
request_type=RequestType.RAW,
queries=[
build_raw_query(
"A",
"logs",
limit=100,
filter_expression=expression,
order=[build_order_by("timestamp", "asc")],
select_fields=[{"name": "body"}],
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
matched = {body for body in get_column_data_from_response(response.json(), "body") if body.startswith(PREFIX)}
matched = query_identities(signoz_families_off, token, family_fleet, expression, "traces", "span.name", "name")
assert matched == expected, expression
@@ -166,7 +126,7 @@ def test_group_by_merges_and_echoes_requested_spelling(
request_type=RequestType.SCALAR,
queries=[
build_traces_scalar_query(
[build_aggregation("count()")],
[build_aggregation("count_distinct(name)")],
filter_expression=f"service.name LIKE '{PREFIX}%'",
group_by=[build_group_by_field(requested_key, "string", "resource")],
)
@@ -174,14 +134,179 @@ def test_group_by_merges_and_echoes_requested_spelling(
)
assert response.status_code == HTTPStatus.OK, response.text
result = response.json()["data"]["data"]["results"][0]
group_column = result["columns"][0]
group_column = get_scalar_columns(response.json())[0]
assert group_column["name"] == requested_key, group_column
assert group_column["columnType"] == "group", group_column
groups = {row[0] for row in result["data"]}
assert {"production", "staging"}.issubset(groups), groups
assert None in groups, groups
# Distinct identities per group make the counts rerun-safe on a reused
# stack: OLD and NEW merge into production, BOTH is staging, NEITHER has
# no spelling at all.
groups = {tuple(row) for row in get_scalar_table_data(response.json())}
assert groups == {("production", 2), ("staging", 1), (None, 1)}, groups
@pytest.mark.parametrize("requested_key", [CURRENT_KEY, OLD_KEY], ids=["current", "old"])
def test_log_group_by_merges_and_echoes_requested_spelling(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
family_fleet: datetime,
requested_key: str,
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms=int((family_fleet - timedelta(minutes=2)).timestamp() * 1000),
end_ms=int((family_fleet + timedelta(minutes=1)).timestamp() * 1000),
request_type=RequestType.SCALAR,
queries=[
build_scalar_query(
name="A",
signal="logs",
aggregations=[build_aggregation("count_distinct(body)")],
filter_expression=f"service.name LIKE '{PREFIX}%'",
group_by=[build_group_by_field(requested_key, "string", "resource")],
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
group_column = get_scalar_columns(response.json())[0]
assert group_column["name"] == requested_key, group_column
groups = {tuple(row) for row in get_scalar_table_data(response.json())}
assert groups == {("production", 2), ("staging", 1), (None, 1)}, groups
def test_time_series_group_by_merges_family(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
family_fleet: datetime,
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms=int((family_fleet - timedelta(minutes=2)).timestamp() * 1000),
end_ms=int((family_fleet + timedelta(minutes=1)).timestamp() * 1000),
request_type=RequestType.TIME_SERIES,
queries=[
build_traces_scalar_query(
[build_aggregation("count_distinct(name)")],
filter_expression=f"service.name LIKE '{PREFIX}%'",
group_by=[build_group_by_field(CURRENT_KEY, "string", "resource")],
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
series = response.json()["data"]["data"]["results"][0]["aggregations"][0]["series"]
values_by_group = {}
for entry in series:
for label in entry.get("labels", []):
if label["key"]["name"] == CURRENT_KEY:
values_by_group[label["value"]] = max((point["value"] for point in entry["values"]), default=None)
assert values_by_group.get("production") == 2, values_by_group
assert values_by_group.get("staging") == 1, values_by_group
def test_raw_select_reads_merged_value(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
family_fleet: datetime,
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms=int((family_fleet - timedelta(minutes=2)).timestamp() * 1000),
end_ms=int((family_fleet + timedelta(minutes=1)).timestamp() * 1000),
request_type=RequestType.RAW,
queries=[
build_raw_query(
"A",
"traces",
limit=100,
filter_expression=f"service.name LIKE '{PREFIX}%'",
order=[build_order_by("timestamp", "asc")],
select_fields=[{"name": "span.name"}, {"name": f"resource.{CURRENT_KEY}"}],
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
rows = response.json()["data"]["data"]["results"][0]["rows"]
value_by_identity = {}
for row in rows:
data = row["data"]
if data.get("name", "").startswith(PREFIX):
value_by_identity[data["name"]] = data.get(CURRENT_KEY)
assert value_by_identity.get(OLD) == "production", value_by_identity
assert value_by_identity.get(NEW) == "production", value_by_identity
assert value_by_identity.get(BOTH) == "staging", value_by_identity
assert value_by_identity.get(NEITHER) in ("", None), value_by_identity
def test_order_by_family_key_sorts_merged_values(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
family_fleet: datetime,
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms=int((family_fleet - timedelta(minutes=2)).timestamp() * 1000),
end_ms=int((family_fleet + timedelta(minutes=1)).timestamp() * 1000),
request_type=RequestType.RAW,
queries=[
build_raw_query(
"A",
"traces",
limit=100,
filter_expression=f"{CURRENT_KEY} EXISTS AND service.name LIKE '{PREFIX}%'",
order=[build_order_by(f"resource.{CURRENT_KEY}", "asc")],
select_fields=[{"name": "span.name"}, {"name": f"resource.{CURRENT_KEY}"}],
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
rows = response.json()["data"]["data"]["results"][0]["rows"]
merged_values = [row["data"].get(CURRENT_KEY) for row in rows if row["data"].get("name", "").startswith(PREFIX)]
assert merged_values == sorted(merged_values), merged_values
assert set(merged_values) == {"production", "staging"}, merged_values
def test_qualified_family_key_emits_no_ambiguity_warning(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
family_fleet: datetime,
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms=int((family_fleet - timedelta(minutes=2)).timestamp() * 1000),
end_ms=int((family_fleet + timedelta(minutes=1)).timestamp() * 1000),
request_type=RequestType.RAW,
queries=[
build_raw_query(
"A",
"traces",
limit=10,
filter_expression=f"resource.{CURRENT_KEY} = 'production'",
order=[build_order_by("timestamp", "asc")],
select_fields=[{"name": "span.name"}],
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
assert get_all_warnings(response.json()) == [], response.json()["data"].get("warning")
def test_bare_name_prefers_resource_and_warns(
@@ -215,6 +340,38 @@ def test_bare_name_prefers_resource_and_warns(
matched = {name for name in get_column_data_from_response(response.json(), "name") if name.startswith(PREFIX)}
assert matched == {OLD, NEW}
warning = response.json()["data"].get("warning") or {}
messages = " ".join(entry.get("message", "") for entry in warning.get("warnings", []))
messages = " ".join(entry.get("message", "") for entry in get_all_warnings(response.json()))
assert "ambiguous" in messages.lower(), messages
def test_field_keys_stay_literal(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
family_fleet: datetime,
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = get_field_keys(signoz, token, {"signal": "traces", "searchText": "deployment.environment"})
assert response.status_code == HTTPStatus.OK, response.text
names = set(response.json()["data"]["keys"].keys())
assert CURRENT_KEY in names, names
assert OLD_KEY in names, names
def test_field_values_union_the_family(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
family_fleet: datetime,
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = get_field_values(
signoz,
token,
{"signal": "traces", "name": OLD_KEY, "fieldContext": "resource"},
)
assert response.status_code == HTTPStatus.OK, response.text
values = set(response.json()["data"]["values"].get("stringValues", []))
# staging exists only under the current spelling on the BOTH row, so only
# the family union makes it reachable from a query on the old spelling.
assert {"production", "staging"}.issubset(values), values

View File

@@ -0,0 +1,124 @@
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from http import HTTPStatus
from time import sleep
import pytest
from fixtures import types
from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
from fixtures.metrics import Metrics
from fixtures.querier import build_group_by_field, get_scalar_columns, get_scalar_table_data
from fixtures.semconvfamilies import (
CURRENT_NAME_METRIC,
LABEL_METRIC,
NORMALIZED_CURRENT_NAME_METRIC,
NORMALIZED_OLD_NAME_METRIC,
OLD_NAME_METRIC,
label_metric_sum,
scalar_query,
)
@pytest.mark.parametrize("requested_key", ["deployment.environment.name", "deployment.environment"], ids=["current", "old"])
def test_metric_label_filter_merges_stored_spellings(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
metric_family_fleet: datetime,
requested_key: str,
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
total = label_metric_sum(signoz, token, metric_family_fleet, f"{requested_key} = 'production'")
# The dotted old, the normalized, and the resource_ series. The conflict
# series (64) merges current-first to staging and stays out.
assert total == 134.0, total
def test_metric_label_group_by_merges_stored_spellings(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
metric_family_fleet: datetime,
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = scalar_query(
signoz,
token,
metric_family_fleet,
LABEL_METRIC,
group_by=[build_group_by_field("deployment.environment.name", "string", "attribute")],
)
assert response.status_code == HTTPStatus.OK, response.text
group_column = get_scalar_columns(response.json())[0]
assert group_column["name"] == "deployment.environment.name", group_column
groups = {row[0]: row[-1] for row in get_scalar_table_data(response.json())}
assert groups.get("production") == 134.0, groups
# The conflict series lands in the staging group: the current spelling
# wins the merge.
assert groups.get("staging") == 65.0, groups
assert groups.get("") == 8.0, groups
def test_metric_name_family_unions_storage_names(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
metric_family_fleet: datetime,
) -> None:
"""The requested layout decides the storage names. The dotted pair and the
normalized pair never mix."""
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
for requested in (OLD_NAME_METRIC, CURRENT_NAME_METRIC):
response = scalar_query(signoz, token, metric_family_fleet, requested)
assert response.status_code == HTTPStatus.OK, response.text
rows = get_scalar_table_data(response.json())
assert rows and rows[0][-1] == 48.0, (requested, rows)
for requested in (NORMALIZED_OLD_NAME_METRIC, NORMALIZED_CURRENT_NAME_METRIC):
response = scalar_query(signoz, token, metric_family_fleet, requested)
assert response.status_code == HTTPStatus.OK, response.text
rows = get_scalar_table_data(response.json())
assert rows and rows[0][-1] == 768.0, (requested, rows)
def test_metric_name_union_double_counts_dual_emission(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
now = datetime.now(tz=UTC)
seeded = now - timedelta(minutes=10)
gauge = {"temporality": "Unspecified", "type_": "Gauge", "is_monotonic": False}
insert_metrics(
[
Metrics(metric_name=OLD_NAME_METRIC, labels={"pod": "overlap"}, timestamp=seeded, value=16.0, **gauge),
Metrics(metric_name=CURRENT_NAME_METRIC, labels={"pod": "overlap"}, timestamp=seeded, value=16.0, **gauge),
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
deadline = datetime.now(tz=UTC) + timedelta(seconds=60)
rows = []
while datetime.now(tz=UTC) < deadline:
rows = get_scalar_table_data(scalar_query(signoz, token, datetime.now(tz=UTC), CURRENT_NAME_METRIC).json())
if rows:
break
sleep(1)
assert rows and rows[0][-1] == 32.0, rows
def test_metric_family_stays_literal_with_flag_off(
signoz_families_off: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
metric_family_fleet: datetime,
) -> None:
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
total = label_metric_sum(signoz_families_off, token, metric_family_fleet, "deployment.environment.name = 'production'")
assert total == 0.0, total
response = scalar_query(signoz_families_off, token, metric_family_fleet, OLD_NAME_METRIC)
assert response.status_code == HTTPStatus.OK, response.text
rows = get_scalar_table_data(response.json())
assert rows and rows[0][-1] == 16.0, rows

View File

@@ -51,5 +51,7 @@ def signoz_families_off(
request=request,
pytestconfig=pytestconfig,
cache_key="signoz-semconv-families-off",
env_overrides={},
env_overrides={
"SIGNOZ_FLAGGER_CONFIG_BOOLEAN_RESOLVE__SEMCONV__FAMILIES": False,
},
)