Compare commits

..

6 Commits

Author SHA1 Message Date
Tushar Vats
d435f31b13 feat(logs): implement search() across all log fields
Replace the search() stub with the real implementation. The logs condition
builder fans a case-insensitive match of the needle across every searchable
column - log columns, body/body_v2, and the attribute + resource maps - gated
by the allow_logs_search feature flag. Scoped forms such as
search('needle', body, resource) narrow the fan-out to the named field contexts.

The where-clause visitor emits FilterOperatorSearch and flags the statement as
scan-heavy; the logs statement builder attaches a CostGuard that the querier
enforces via EXPLAIN ESTIMATE against a configurable scan budget
(statementbuilder.Config.SearchMaxScanRows).
2026-07-29 19:39:16 +05:30
Tushar Vats
04637bd960 test(querier): assert rendered SQL in skip_resource_fingerprint tests (#12334)
Identical rows can't distinguish the optimized instance from the
fingerprint baseline, so the tests now also diff the rendered ClickHouse
statement via /query_range/preview: the baseline must join the fingerprint
CTE and the optimized instance must not.

The assertion is pinned to the "resource_fingerprint GLOBAL IN (SELECT
fingerprint FROM __resource_filter)" clause, a literal in the statement
builder. The resource condition the fallback path renders on the main
table is deliberately not asserted: its column form follows the resource
JSON evolution boundary, rendering as a multiIf over resources_string
while the query window straddles release_time and collapsing to
resource.`key`::String once the window sits past it.

Also fixes the fixture env prefix. skip_resource_fingerprint moved out of
the querier config namespace into its own, so
SIGNOZ_QUERIER_SKIP__RESOURCE__FINGERPRINT_* was being ignored and both
instances ran with the optimization disabled.
2026-07-29 12:51:03 +00:00
Swapnil Nakade
0703fbf961 feat: adding compute engine service in GCP integration (#12166)
* feat: adding gcp memorystore redis service

* refactor: updating dashboard title

* refactor: extending width of uptime gauge panel

* refactor: updating cpu utilization panel

* refactor: updating dashboard panel to use rate function instead of hack

* feat: adding compute engine service

* refactor: updating dashboard panels to use rate aggregation

* fix: correct typo and unit in compute engine dashboard

* refactor: migrating dashboard to v6
2026-07-29 11:28:58 +00:00
Tushar Vats
4e45620b72 refactor(statementbuilder): drop Builders bundle; fold factories into statement_builder.go (#12330)
Remove the statementbuilder.Builders aggregate. The querier chain (querier.New,
signozquerier.NewFactory, NewQuerierProviderFactories) now takes the six per-signal
statement builders individually, and signoz.go::newQueryStack returns them directly
via a plain multi-return. The statementbuilder package is now config-only.

Remove each <signal>statementbuilder/new.go and fold its NewFactory into that
package's statement_builder.go (traces' NewOperatorFactory into
trace_operator_statement_builder.go), giving the layout: struct -> NewFactory ->
New<X>QueryStatementBuilder.
2026-07-29 10:46:55 +00:00
Naman Verma
a9506f1354 fix: add meter source where needed during dashboard migration (#12322)
* fix: add meter source during migration

* chore: move source field handling to wrapinv5envelope

* fix: add migration to backfill source for migrated dashboards

* chore: shorten comments
2026-07-29 10:39:41 +00:00
Tushar Vats
e2e7caf1ca refactor(querier): 3-layer per-signal query architecture (#12304)
Split the five telemetry<signal> packages, which mixed three concerns, into
three per-signal layers with a cycle-free dependency direction:

- telemetryschema/<signal>telemetryschema — primitives (const + table
  selection, field_mapper, condition_builder, trace helpers); leaf layer.
- statementbuilder/<signal>statementbuilder — SQL generation. The parent
  statementbuilder package is contract-only (the Builders bundle + Config);
  each sub-package exposes a factory.ProviderFactory[..., statementbuilder.Config]
  whose New internalizes FieldMapper/ConditionBuilder/AggExprRewriter and reads
  SkipResourceFingerprint. Traces exposes two factories (query + operator).
  telemetryresourcefilter moves here as statementbuilder/resourcefilter.
- telemetrymetadata — key/value resolution; NewTelemetryMetaStore collapses
  from 24 args to (settings, telemetrystore, flagger), sourcing table names
  from the schema constants.

Centralize query-stack assembly in signoz.go via newQueryStack: build the
single metadata store, run each per-signal statement-builder factory, assemble
the statementbuilder.Builders bundle, and build the bucket cache — once. This
is the only place that imports the concrete sub-packages (so the edge runs
subs -> parent, cycle-free), and it removes the duplicate metadata store that
signozquerier used to build, leaving signozquerier a thin
querier.New(*statementbuilder.Builders) adapter.

Also:
- statementbuilder.Config owns SkipResourceFingerprint (moved off
  querier.Config). YAML key moves querier.skip_resource_fingerprint ->
  statementbuilder.skip_resource_fingerprint.
- Querier interface moves into querier.go (interfaces.go removed); BucketCache
  -> bucket_cache.go, Handler -> api.go.
- Add pkg/querier/queriertest.MockQuerier.
2026-07-29 09:24:08 +00:00
135 changed files with 4305 additions and 2331 deletions

View File

@@ -2,9 +2,9 @@ package main
import (
"github.com/SigNoz/signoz/pkg/metercollector"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
"github.com/SigNoz/signoz/pkg/types/zeustypes"
)
@@ -25,8 +25,8 @@ var meterConfigs = []metercollector.Config{
Name: zeustypes.MeterLogSize,
Unit: zeustypes.MeterUnitBytes,
Aggregation: zeustypes.MeterAggregationSum,
DBName: telemetrylogs.DBName,
TableName: telemetrylogs.LogsV2LocalTableName,
DBName: logstelemetryschema.DBName,
TableName: logstelemetryschema.LogsV2LocalTableName,
DefaultRetentionDays: retentiontypes.DefaultLogsRetentionDays,
},
},
@@ -36,8 +36,8 @@ var meterConfigs = []metercollector.Config{
Name: zeustypes.MeterSpanSize,
Unit: zeustypes.MeterUnitBytes,
Aggregation: zeustypes.MeterAggregationSum,
DBName: telemetrytraces.DBName,
TableName: telemetrytraces.SpanIndexV3LocalTableName,
DBName: tracestelemetryschema.DBName,
TableName: tracestelemetryschema.SpanIndexV3LocalTableName,
DefaultRetentionDays: retentiontypes.DefaultTracesRetentionDays,
},
},
@@ -47,8 +47,8 @@ var meterConfigs = []metercollector.Config{
Name: zeustypes.MeterDatapointCount,
Unit: zeustypes.MeterUnitCount,
Aggregation: zeustypes.MeterAggregationSum,
DBName: telemetrymetrics.DBName,
TableName: telemetrymetrics.SamplesV4LocalTableName,
DBName: metricstelemetryschema.DBName,
TableName: metricstelemetryschema.SamplesV4LocalTableName,
DefaultRetentionDays: retentiontypes.DefaultMetricsRetentionDays,
},
},

View File

@@ -1496,6 +1496,7 @@ components:
- redis
- cloudsql_postgres
- memorystore_redis
- computeengine
type: string
CloudintegrationtypesServiceMetadata:
properties:

View File

@@ -16,7 +16,7 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/metercollector"
"github.com/SigNoz/signoz/pkg/modules/retention"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/licensetypes"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
@@ -153,7 +153,7 @@ func (provider *Provider) Collect(
func buildOriginQuery(meterName string) (string, []any) {
sb := sqlbuilder.NewSelectBuilder()
sb.Select("toInt64(ifNull(min(unix_milli), 0))")
sb.From(telemetrymeter.DBName + "." + telemetrymeter.SamplesTableName)
sb.From(metertelemetryschema.DBName + "." + metertelemetryschema.SamplesTableName)
sb.Where(sb.Equal("metric_name", meterName))
return sb.BuildWithFlavor(sqlbuilder.ClickHouse)
}
@@ -171,7 +171,7 @@ func buildQuery(meterName string, segment *retentiontypes.RetentionPolicySegment
sb := sqlbuilder.NewSelectBuilder()
sb.Select(selects...)
sb.From(telemetrymeter.DBName + "." + telemetrymeter.SamplesTableName)
sb.From(metertelemetryschema.DBName + "." + metertelemetryschema.SamplesTableName)
sb.Where(
sb.Equal("metric_name", meterName),
sb.GTE("unix_milli", segment.StartMs),

View File

@@ -8,16 +8,16 @@ import (
sqlbuilder "github.com/huandu/go-sqlbuilder"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/metricreductionruletypes"
)
var (
reductionRulesTable = telemetrymetrics.DBName + "." + telemetrymetrics.ReductionRulesTableName
metadataTable = telemetrymetrics.DBName + "." + telemetrymetrics.AttributesMetadataTableName
bufferSeriesTable = telemetrymetrics.DBName + "." + telemetrymetrics.TimeseriesV4BufferTableName
reductionRulesTable = metricstelemetryschema.DBName + "." + metricstelemetryschema.ReductionRulesTableName
metadataTable = metricstelemetryschema.DBName + "." + metricstelemetryschema.AttributesMetadataTableName
bufferSeriesTable = metricstelemetryschema.DBName + "." + metricstelemetryschema.TimeseriesV4BufferTableName
)
const timeSeriesBucketMilli = int64(time.Hour / time.Millisecond)
@@ -192,7 +192,7 @@ func (c *clickhouse) ingestedSeriesCount(ctx context.Context, metricNames []stri
sb := sqlbuilder.NewSelectBuilder()
sb.Select("metric_name", "uniq(fingerprint)")
sb.From(telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4BufferTableName)
sb.From(metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4BufferTableName)
conds := []string{
sb.In("metric_name", names...),
sb.GE("unix_milli", startMs),
@@ -229,8 +229,8 @@ func (c *clickhouse) ingestedSeriesCount(ctx context.Context, metricNames []stri
// reduced sample tables.
func (c *clickhouse) reducedSeriesCount(ctx context.Context, metricNames []string, effectiveFrom map[string]int64, startMs, endMs int64) (map[string]uint64, error) {
out := make(map[string]uint64, len(metricNames))
for _, table := range []string{telemetrymetrics.SamplesV4ReducedLastTableName, telemetrymetrics.SamplesV4ReducedSumTableName} {
counts, err := c.reducedSeriesCountForTable(ctx, telemetrymetrics.DBName+"."+table, metricNames, effectiveFrom, startMs, endMs)
for _, table := range []string{metricstelemetryschema.SamplesV4ReducedLastTableName, metricstelemetryschema.SamplesV4ReducedSumTableName} {
counts, err := c.reducedSeriesCountForTable(ctx, metricstelemetryschema.DBName+"."+table, metricNames, effectiveFrom, startMs, endMs)
if err != nil {
return nil, err
}
@@ -300,9 +300,9 @@ func (c *clickhouse) RankByVolume(ctx context.Context, metricNames []string, eff
direction = "DESC"
}
ingestedTable := telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4BufferTableName
reducedLast := telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4ReducedLastTableName
reducedSum := telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4ReducedSumTableName
ingestedTable := metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4BufferTableName
reducedLast := metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4ReducedLastTableName
reducedSum := metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4ReducedSumTableName
sb := sqlbuilder.NewSelectBuilder()
sb.Select("base.metric_name AS metric_name", "ifNull(i.cnt, 0) AS ingested", "ifNull(d.cnt, 0) AS reduced")
@@ -352,16 +352,16 @@ func (c *clickhouse) SampleVolumeByMetric(ctx context.Context, metricNames []str
}
ctx = c.withThreads(ctx)
ingested, err := c.countSamplesByMetric(ctx, telemetrymetrics.DBName+"."+telemetrymetrics.SamplesV4BufferTableName, "count()", metricNames, effectiveFrom, startMs, endMs)
ingested, err := c.countSamplesByMetric(ctx, metricstelemetryschema.DBName+"."+metricstelemetryschema.SamplesV4BufferTableName, "count()", metricNames, effectiveFrom, startMs, endMs)
if err != nil {
return nil, err
}
last, err := c.countSamplesByMetric(ctx, telemetrymetrics.DBName+"."+telemetrymetrics.SamplesV4ReducedLastTableName, "uniq(reduced_fingerprint, unix_milli)", metricNames, effectiveFrom, startMs, endMs)
last, err := c.countSamplesByMetric(ctx, metricstelemetryschema.DBName+"."+metricstelemetryschema.SamplesV4ReducedLastTableName, "uniq(reduced_fingerprint, unix_milli)", metricNames, effectiveFrom, startMs, endMs)
if err != nil {
return nil, err
}
sum, err := c.countSamplesByMetric(ctx, telemetrymetrics.DBName+"."+telemetrymetrics.SamplesV4ReducedSumTableName, "uniq(reduced_fingerprint, unix_milli)", metricNames, effectiveFrom, startMs, endMs)
sum, err := c.countSamplesByMetric(ctx, metricstelemetryschema.DBName+"."+metricstelemetryschema.SamplesV4ReducedSumTableName, "uniq(reduced_fingerprint, unix_milli)", metricNames, effectiveFrom, startMs, endMs)
if err != nil {
return nil, err
}
@@ -419,7 +419,7 @@ func (c *clickhouse) TotalVolume(ctx context.Context, startMs, endMs int64) (uin
sb := sqlbuilder.NewSelectBuilder()
sb.Select("uniq(fingerprint)", "count()")
sb.From(telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4BufferTableName)
sb.From(metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4BufferTableName)
sb.Where(sb.GE("unix_milli", startMs), sb.LT("unix_milli", endMs))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -470,7 +470,7 @@ func (c *clickhouse) SampleTimeseries(ctx context.Context, ruledMetrics []string
func (c *clickhouse) totalSamplesByBucket(ctx context.Context, startMs, endMs int64) (map[int64]uint64, error) {
sb := sqlbuilder.NewSelectBuilder()
sb.Select(sampleBucketExpr, "count()")
sb.From(telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4BufferTableName)
sb.From(metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4BufferTableName)
sb.Where(sb.GE("unix_milli", startMs), sb.LT("unix_milli", endMs))
sb.GroupBy("bucket")
@@ -485,7 +485,7 @@ func (c *clickhouse) ruledIngestedSamplesByBucket(ctx context.Context, metricNam
sb := sqlbuilder.NewSelectBuilder()
sb.Select(sampleBucketExpr, "count()")
sb.From(telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4BufferTableName)
sb.From(metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4BufferTableName)
conds := []string{sb.In("metric_name", names...), sb.GE("unix_milli", startMs), sb.LT("unix_milli", endMs)}
if len(effectiveFrom) > 0 {
conds = append(conds, strictEffectiveFrom(sb, metricNames, effectiveFrom))
@@ -499,7 +499,7 @@ func (c *clickhouse) ruledIngestedSamplesByBucket(ctx context.Context, metricNam
// reduced 60s rows are versioned by computed_at, so count distinct buckets.
func (c *clickhouse) ruledRetainedSamplesByBucket(ctx context.Context, metricNames []string, effectiveFrom map[string]int64, startMs, endMs int64) (map[int64]uint64, error) {
out := make(map[int64]uint64)
for _, table := range []string{telemetrymetrics.SamplesV4ReducedLastTableName, telemetrymetrics.SamplesV4ReducedSumTableName} {
for _, table := range []string{metricstelemetryschema.SamplesV4ReducedLastTableName, metricstelemetryschema.SamplesV4ReducedSumTableName} {
names := make([]any, len(metricNames))
for i, name := range metricNames {
names[i] = name
@@ -507,7 +507,7 @@ func (c *clickhouse) ruledRetainedSamplesByBucket(ctx context.Context, metricNam
sb := sqlbuilder.NewSelectBuilder()
sb.Select(sampleBucketExpr, "uniq(reduced_fingerprint, unix_milli)")
sb.From(telemetrymetrics.DBName + "." + table)
sb.From(metricstelemetryschema.DBName + "." + table)
conds := []string{sb.In("metric_name", names...), sb.GE("unix_milli", startMs), sb.LT("unix_milli", endMs)}
if len(effectiveFrom) > 0 {
conds = append(conds, strictEffectiveFrom(sb, metricNames, effectiveFrom))

View File

@@ -2815,6 +2815,7 @@ export enum CloudintegrationtypesServiceIDDTO {
redis = 'redis',
cloudsql_postgres = 'cloudsql_postgres',
memorystore_redis = 'memorystore_redis',
computeengine = 'computeengine',
}
export type CloudintegrationtypesCloudIntegrationServiceDTOAnyOf = {
/**

View File

@@ -112,15 +112,12 @@ functionCall
;
/*
* Full-text search call: search('needle')
*
* Uses the shared functionParamList so future scoped forms like
* search(body, 'abc') / search(attribute, 'abc') need no grammar change. Today
* only a single needle is supported. Unlike bare/quoted free text (`fullText`),
* which only targets the body column, search() fans out across every field.
* Full-text search: search('needle') or scoped search('needle', body, ...).
* First param is the needle; the rest are field-context scopes (body/attribute/
* resource/log), quoted or bare. Handled in the visitor — no grammar change.
*/
searchCall
: SEARCH LPAREN functionParamList RPAREN
: SEARCH LPAREN valueList RPAREN
;
// Function parameters can be keys, single scalar values, or arrays

View File

@@ -29,13 +29,19 @@ func New(t *testing.T) flagger.Flagger {
// WithUseJSONBody returns a Flagger with use_json_body set to the given value.
func WithUseJSONBody(t *testing.T, enabled bool) flagger.Flagger {
return WithBooleanFlags(t, map[string]bool{
flagger.FeatureUseJSONBody.String(): enabled,
})
}
// WithBooleanFlags returns a Flagger with the given boolean feature flags set to
// the provided values (keyed by feature name, e.g. flagger.FeatureX.String()).
func WithBooleanFlags(t *testing.T, flags map[string]bool) flagger.Flagger {
t.Helper()
registry := flagger.MustNewRegistry()
cfg := flagger.Config{}
if enabled {
cfg.Config.Boolean = map[string]bool{
flagger.FeatureUseJSONBody.String(): true,
}
if len(flags) > 0 {
cfg.Config.Boolean = flags
}
fl, err := flagger.New(
context.Background(),

View File

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" viewBox="0 0 24 24"><defs><style>.cls-1{fill:#aecbfa;}.cls-2{fill:#669df6;}.cls-3{fill:#4285f4;}</style></defs><title>Icon_24px_ComputeEngine_Color</title><g data-name="Product Icons"><rect class="cls-1" x="9" y="9" width="6" height="6"/><rect class="cls-2" x="11" y="2" width="2" height="4"/><rect class="cls-2" x="7" y="2" width="2" height="4"/><rect class="cls-2" x="15" y="2" width="2" height="4"/><rect class="cls-3" x="11" y="18" width="2" height="4"/><rect class="cls-3" x="7" y="18" width="2" height="4"/><rect class="cls-3" x="15" y="18" width="2" height="4"/><rect class="cls-3" x="19" y="10" width="2" height="4" transform="translate(8 32) rotate(-90)"/><rect class="cls-3" x="19" y="14" width="2" height="4" transform="translate(4 36) rotate(-90)"/><rect class="cls-3" x="19" y="6" width="2" height="4" transform="translate(12 28) rotate(-90)"/><rect class="cls-2" x="3" y="10" width="2" height="4" transform="translate(-8 16) rotate(-90)"/><rect class="cls-2" x="3" y="14" width="2" height="4" transform="translate(-12 20) rotate(-90)"/><rect class="cls-2" x="3" y="6" width="2" height="4" transform="translate(-4 12) rotate(-90)"/><path class="cls-1" d="M5,5V19H19V5ZM17,17H7V7H17Z"/><polygon class="cls-2" points="9 15 15 15 12 12 9 15"/><polygon class="cls-3" points="12 12 15 15 15 9 12 12"/></g></svg>

After

Width:  |  Height:  |  Size: 1.4 KiB

View File

@@ -0,0 +1,124 @@
{
"id": "computeengine",
"title": "GCP Compute Engine",
"icon": "file://icon.svg",
"overview": "file://overview.md",
"supportedSignals": {
"metrics": true,
"logs": true
},
"dataCollected": {
"metrics": [
{
"name": "compute.googleapis.com/instance/uptime_total",
"unit": "Seconds",
"type": "Gauge",
"description": ""
},
{
"name": "compute.googleapis.com/instance/cpu/utilization",
"unit": "Percent",
"type": "Gauge",
"description": ""
},
{
"name": "compute.googleapis.com/guest/memory/bytes_used",
"unit": "Bytes",
"type": "Gauge",
"description": ""
},
{
"name": "compute.googleapis.com/guest/memory/percent_used",
"unit": "Percent",
"type": "Gauge",
"description": ""
},
{
"name": "agent.googleapis.com/memory/percent_used",
"unit": "Percent",
"type": "Gauge",
"description": ""
},
{
"name": "compute.googleapis.com/instance/disk/average_io_latency",
"unit": "Microseconds",
"type": "Gauge",
"description": ""
},
{
"name": "compute.googleapis.com/instance/disk/read_bytes_count",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "compute.googleapis.com/instance/disk/write_bytes_count",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "compute.googleapis.com/instance/disk/read_ops_count",
"unit": "Count",
"type": "Sum",
"description": ""
},
{
"name": "compute.googleapis.com/instance/disk/write_ops_count",
"unit": "Count",
"type": "Sum",
"description": ""
},
{
"name": "compute.googleapis.com/instance/disk/performance_status",
"unit": "None",
"type": "Gauge",
"description": ""
},
{
"name": "compute.googleapis.com/instance/network/received_bytes_count",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "compute.googleapis.com/instance/network/sent_bytes_count",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "compute.googleapis.com/instance/network/received_packets_count",
"unit": "Count",
"type": "Sum",
"description": ""
},
{
"name": "compute.googleapis.com/instance/network/sent_packets_count",
"unit": "Count",
"type": "Sum",
"description": ""
},
{
"name": "compute.googleapis.com/firewall/dropped_packets_count",
"unit": "Count",
"type": "Sum",
"description": ""
}
],
"logs": []
},
"telemetryCollectionStrategy": {
"gcp": {}
},
"assets": {
"dashboards": [
{
"id": "overview",
"title": "GCP Compute Engine Overview",
"description": "Overview of GCP Compute Engine metrics",
"definition": "file://assets/dashboards/overview.json"
}
]
}
}

View File

@@ -0,0 +1,3 @@
### Monitor GCP Compute Engine with SigNoz
Collect key GCP Compute Engine metrics and view them with an out of the box dashboard.

View File

@@ -5,14 +5,14 @@ import (
"fmt"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/valuer"
)
func (m *module) Collect(ctx context.Context, _ valuer.UUID) (map[string]any, error) {
stats := make(map[string]any)
metadataTable := fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName)
metadataTable := fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName)
var (
systemMetricCount uint64
k8sMetricCount uint64

View File

@@ -7,7 +7,7 @@ import (
"strings"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -281,7 +281,7 @@ func (m *module) getPerGroupContainerStatusCounts(
mergedFilterExpr := mergeFilterExpressions(userFilterExpr, buildPageGroupsFilterExpr(pageGroups))
samplesStartMs, flooredEndMs, tsAdjustedStart, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
// Built once; identical across the two fps CTEs (buildFilterClause hits the
// metadata store + parses the expression). AddWhereClause only reads it.
@@ -310,7 +310,7 @@ func (m *module) getPerGroupContainerStatusCounts(
)
}
stateFps.Select(stateFpsCols...)
stateFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
stateFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
stateFps.Where(
stateFps.E("metric_name", containerStatusStateMetricName),
stateFps.GE("unix_milli", tsAdjustedStart),
@@ -342,7 +342,7 @@ func (m *module) getPerGroupContainerStatusCounts(
containerState.Select(containerStateCols...)
containerState.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN state_fps AS fps ON samples.fingerprint = fps.fingerprint",
telemetrymetrics.DBName, distributedSamplesTable,
metricstelemetryschema.DBName, distributedSamplesTable,
))
containerState.Where(
containerState.E("samples.metric_name", containerStatusStateMetricName),
@@ -361,7 +361,7 @@ func (m *module) getPerGroupContainerStatusCounts(
fmt.Sprintf("JSONExtractString(labels, %s) AS container_name", reasonFps.Var(containerNameAttrKey)),
fmt.Sprintf("JSONExtractString(labels, %s) AS reason", reasonFps.Var(containerStatusReasonAttrKey)),
)
reasonFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
reasonFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
reasonFps.Where(
reasonFps.E("metric_name", containerStatusReasonMetricName),
reasonFps.GE("unix_milli", tsAdjustedStart),
@@ -391,7 +391,7 @@ func (m *module) getPerGroupContainerStatusCounts(
)
reasonInner.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN reason_fps AS fps ON samples.fingerprint = fps.fingerprint",
telemetrymetrics.DBName, distributedSamplesTable,
metricstelemetryschema.DBName, distributedSamplesTable,
))
reasonInner.Where(
reasonInner.E("samples.metric_name", containerStatusReasonMetricName),
@@ -546,7 +546,7 @@ func (m *module) getPerGroupContainerRestartCounts(
mergedFilterExpr := mergeFilterExpressions(userFilterExpr, buildPageGroupsFilterExpr(pageGroups))
samplesStartMs, flooredEndMs, tsAdjustedStart, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
var (
filterClause *sqlbuilder.WhereClause
@@ -572,7 +572,7 @@ func (m *module) getPerGroupContainerRestartCounts(
)
}
restartFps.Select(restartFpsCols...)
restartFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
restartFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
restartFps.Where(
restartFps.E("metric_name", containerRestartsMetricName),
restartFps.GE("unix_milli", tsAdjustedStart),
@@ -602,7 +602,7 @@ func (m *module) getPerGroupContainerRestartCounts(
containerRestarts.Select(containerRestartsCols...)
containerRestarts.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN restart_fps AS fps ON samples.fingerprint = fps.fingerprint",
telemetrymetrics.DBName, distributedSamplesTable,
metricstelemetryschema.DBName, distributedSamplesTable,
))
containerRestarts.Where(
containerRestarts.E("samples.metric_name", containerRestartsMetricName),
@@ -690,7 +690,7 @@ func (m *module) getPerGroupContainerReadyCounts(
mergedFilterExpr := mergeFilterExpressions(userFilterExpr, buildPageGroupsFilterExpr(pageGroups))
samplesStartMs, flooredEndMs, tsAdjustedStart, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
var (
filterClause *sqlbuilder.WhereClause
@@ -716,7 +716,7 @@ func (m *module) getPerGroupContainerReadyCounts(
)
}
readyFps.Select(readyFpsCols...)
readyFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
readyFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
readyFps.Where(
readyFps.E("metric_name", containerReadyMetricName),
readyFps.GE("unix_milli", tsAdjustedStart),
@@ -746,7 +746,7 @@ func (m *module) getPerGroupContainerReadyCounts(
containerReady.Select(containerReadyCols...)
containerReady.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN ready_fps AS fps ON samples.fingerprint = fps.fingerprint",
telemetrymetrics.DBName, distributedSamplesTable,
metricstelemetryschema.DBName, distributedSamplesTable,
))
containerReady.Where(
containerReady.E("samples.metric_name", containerReadyMetricName),

View File

@@ -9,7 +9,7 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
@@ -344,11 +344,11 @@ func alignedMetricWindow(startMs, endMs int64) (
flooredEndMs = flooredEndMs - (flooredEndMs % (adjustStep * 1000))
}
tsAdjustedStartMs, _, distributedTSTable, localTSTable := telemetrymetrics.WhichTSTableToUse(
tsAdjustedStartMs, _, distributedTSTable, localTSTable := metricstelemetryschema.WhichTSTableToUse(
samplesAdjustedStartMs, flooredEndMs, false, nil,
)
distributedSamplesTable, localSamplesTable := telemetrymetrics.WhichSamplesTableToUse(
distributedSamplesTable, localSamplesTable := metricstelemetryschema.WhichSamplesTableToUse(
samplesAdjustedStartMs, flooredEndMs,
metrictypes.UnspecifiedType, metrictypes.TimeAggregationUnspecified, false, nil,
)
@@ -362,7 +362,7 @@ func alignedMetricWindow(startMs, endMs int64) (
func (m *module) buildSamplesTblFingerprintSubQuery(metricNames []string, samplesTable string, flooredStart, flooredEnd uint64) *sqlbuilder.SelectBuilder {
fpSB := sqlbuilder.NewSelectBuilder()
fpSB.Select("DISTINCT fingerprint")
fpSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, samplesTable))
fpSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, samplesTable))
fpSB.Where(
fpSB.In("metric_name", sqlbuilder.List(metricNames)),
fpSB.GE("unix_milli", flooredStart),
@@ -376,7 +376,7 @@ func (m *module) buildSamplesTblFingerprintSubQuery(metricNames []string, sample
func (m *module) buildReducedSamplesTblFingerprintSubQuery(metricNames []string, flooredStart, flooredEnd uint64) *sqlbuilder.SelectBuilder {
lastSB := sqlbuilder.NewSelectBuilder()
lastSB.Select("reduced_fingerprint")
lastSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedLastTableName))
lastSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedLastTableName))
lastSB.Where(
lastSB.In("metric_name", sqlbuilder.List(metricNames)),
lastSB.GE("unix_milli", flooredStart),
@@ -385,7 +385,7 @@ func (m *module) buildReducedSamplesTblFingerprintSubQuery(metricNames []string,
sumSB := sqlbuilder.NewSelectBuilder()
sumSB.Select("reduced_fingerprint")
sumSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedSumTableName))
sumSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedSumTableName))
sumSB.Where(
sumSB.In("metric_name", sqlbuilder.List(metricNames)),
sumSB.GE("unix_milli", flooredStart),
@@ -478,7 +478,7 @@ func (m *module) getEarliestMetricTime(ctx context.Context, metricNames []string
sb := sqlbuilder.NewSelectBuilder()
sb.Select("min(first_reported_unix_milli) AS min_first_reported")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.Where(sb.In("metric_name", sqlbuilder.List(metricNames)))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -504,7 +504,7 @@ func (m *module) getMetricsExistence(ctx context.Context, metricNames []string)
sb := sqlbuilder.NewSelectBuilder()
sb.Select("metric_name", "count(*) AS cnt")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.Where(sb.In("metric_name", sqlbuilder.List(metricNames)))
sb.GroupBy("metric_name")
@@ -549,7 +549,7 @@ func (m *module) getAttributesExistence(ctx context.Context, metricNames, attrNa
}
sb := sqlbuilder.NewSelectBuilder()
sb.Select("attr_name", "count(*) AS cnt")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.Where(
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.In("attr_name", sqlbuilder.List(attrNames)),
@@ -656,7 +656,7 @@ func (m *module) getMetadata(
rawSrc := sqlbuilder.NewSelectBuilder()
rawSrc.Select("labels", "unix_milli")
rawSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTableName))
rawSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTableName))
rawSrc.Where(
rawSrc.In("metric_name", sqlbuilder.List(metricNames)),
rawSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -669,7 +669,7 @@ func (m *module) getMetadata(
reducedSrc := sqlbuilder.NewSelectBuilder()
reducedSrc.Select("labels", "unix_milli")
reducedSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedSrc.Where(
reducedSrc.In("metric_name", sqlbuilder.List(metricNames)),
reducedSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -683,7 +683,7 @@ func (m *module) getMetadata(
// Inner query reads over the union of raw + reduced series.
innerSB.From(innerSB.BuilderAs(sqlbuilder.UnionAll(rawSrc, reducedSrc), "series"))
} else {
innerSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTableName))
innerSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTableName))
innerSB.Where(
innerSB.In("metric_name", sqlbuilder.List(metricNames)),
innerSB.GE("unix_milli", tsAdjustedStartMs),
@@ -870,7 +870,7 @@ func (m *module) getPerGroupDistinctCounts(
rawSrc := sqlbuilder.NewSelectBuilder()
rawSrc.Select("labels")
rawSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTimeSeriesTbl))
rawSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTimeSeriesTbl))
rawSrc.Where(
rawSrc.In("metric_name", sqlbuilder.List(metricNames)),
rawSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -883,7 +883,7 @@ func (m *module) getPerGroupDistinctCounts(
reducedSrc := sqlbuilder.NewSelectBuilder()
reducedSrc.Select("labels")
reducedSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedSrc.Where(
reducedSrc.In("metric_name", sqlbuilder.List(metricNames)),
reducedSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -896,7 +896,7 @@ func (m *module) getPerGroupDistinctCounts(
sb.From(sb.BuilderAs(sqlbuilder.UnionAll(rawSrc, reducedSrc), "series"))
} else {
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTimeSeriesTbl))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTimeSeriesTbl))
sb.Where(
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GE("unix_milli", tsAdjustedStartMs),

View File

@@ -7,7 +7,7 @@ import (
"strings"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -72,7 +72,7 @@ func (m *module) getPerGroupHostStatusCounts(
rawSrc := sqlbuilder.NewSelectBuilder()
rawSrc.Select("labels")
rawSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTimeSeriesTableName))
rawSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTimeSeriesTableName))
rawSrc.Where(
rawSrc.In("metric_name", sqlbuilder.List(metricNames)),
rawSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -85,7 +85,7 @@ func (m *module) getPerGroupHostStatusCounts(
reducedSrc := sqlbuilder.NewSelectBuilder()
reducedSrc.Select("labels")
reducedSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedSrc.Where(
reducedSrc.In("metric_name", sqlbuilder.List(metricNames)),
reducedSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -101,7 +101,7 @@ func (m *module) getPerGroupHostStatusCounts(
fpSB := m.buildSamplesTblFingerprintSubQuery(metricNames, localSamplesTable, samplesStartMs, flooredEndMs)
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTimeSeriesTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTimeSeriesTableName))
sb.Where(
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GE("unix_milli", tsAdjustedStartMs),
@@ -357,7 +357,7 @@ func (m *module) getActiveHostsQuery(metricNames []string, hostNameAttr string,
sb := sqlbuilder.NewSelectBuilder()
sb.Distinct()
sb.Select("attr_string_value")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.Where(
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.E("attr_name", hostNameAttr),

View File

@@ -9,7 +9,7 @@ import (
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/modules/inframonitoring"
"github.com/SigNoz/signoz/pkg/querier"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
@@ -40,8 +40,8 @@ func NewModule(
providerSettings factory.ProviderSettings,
cfg inframonitoring.Config,
) inframonitoring.Module {
fieldMapper := telemetrymetrics.NewFieldMapper()
condBuilder := telemetrymetrics.NewConditionBuilder(fieldMapper)
fieldMapper := metricstelemetryschema.NewFieldMapper()
condBuilder := metricstelemetryschema.NewConditionBuilder(fieldMapper)
return &module{
telemetryStore: telemetryStore,
telemetryMetadataStore: telemetryMetadataStore,

View File

@@ -7,7 +7,7 @@ import (
"strings"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -186,7 +186,7 @@ func (m *module) getPerGroupNodeConditionCounts(
// Step-floor bounds + resolve tables in one shot to match QB v5 querier.
samplesStartMs, flooredEndMs, tsAdjustedStartMs, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
// ----- timeSeriesFPs -----
timeSeriesFPs := sqlbuilder.NewSelectBuilder()
@@ -200,7 +200,7 @@ func (m *module) getPerGroupNodeConditionCounts(
)
}
timeSeriesFPs.Select(timeSeriesFPsSelectCols...)
timeSeriesFPs.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
timeSeriesFPs.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
timeSeriesFPs.Where(
timeSeriesFPs.E("metric_name", nodeConditionMetricName),
timeSeriesFPs.GE("unix_milli", tsAdjustedStartMs),
@@ -237,7 +237,7 @@ func (m *module) getPerGroupNodeConditionCounts(
latestConditionPerNode.Select(latestConditionPerNodeSelectCols...)
latestConditionPerNode.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN time_series_fps AS tsfp ON samples.fingerprint = tsfp.fingerprint",
telemetrymetrics.DBName, distributedSamplesTable,
metricstelemetryschema.DBName, distributedSamplesTable,
))
latestConditionPerNode.Where(
latestConditionPerNode.E("samples.metric_name", nodeConditionMetricName),

View File

@@ -8,7 +8,7 @@ import (
"time"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -284,7 +284,7 @@ func (m *module) getPerGroupPodStatusCounts(
mergedFilterExpr := mergeFilterExpressions(userFilterExpr, buildPageGroupsFilterExpr(pageGroups))
samplesStartMs, flooredEndMs, tsAdjustedStart, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
// Build the merged filter clause once; it's identical across the three fps
// CTEs, and buildFilterClause hits the metadata store + parses the
@@ -313,7 +313,7 @@ func (m *module) getPerGroupPodStatusCounts(
)
}
phaseFps.Select(phaseFpsCols...)
phaseFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
phaseFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
phaseFps.Where(
phaseFps.E("metric_name", podPhaseMetricName),
phaseFps.GE("unix_milli", tsAdjustedStart),
@@ -340,7 +340,7 @@ func (m *module) getPerGroupPodStatusCounts(
phasePerPod.Select(phasePerPodCols...)
phasePerPod.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN phase_fps AS fps ON samples.fingerprint = fps.fingerprint",
telemetrymetrics.DBName, distributedSamplesTable,
metricstelemetryschema.DBName, distributedSamplesTable,
))
phasePerPod.Where(
phasePerPod.E("samples.metric_name", podPhaseMetricName),
@@ -357,7 +357,7 @@ func (m *module) getPerGroupPodStatusCounts(
"fingerprint",
fmt.Sprintf("JSONExtractString(labels, %s) AS pod_uid", podReasonFps.Var(podUIDAttrKey)),
)
podReasonFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
podReasonFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
podReasonFps.Where(
podReasonFps.E("metric_name", podStatusReasonMetricName),
podReasonFps.GE("unix_milli", tsAdjustedStart),
@@ -377,7 +377,7 @@ func (m *module) getPerGroupPodStatusCounts(
)
podReasonPerPod.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN pod_reason_fps AS fps ON samples.fingerprint = fps.fingerprint",
telemetrymetrics.DBName, distributedSamplesTable,
metricstelemetryschema.DBName, distributedSamplesTable,
))
podReasonPerPod.Where(
podReasonPerPod.E("samples.metric_name", podStatusReasonMetricName),
@@ -396,7 +396,7 @@ func (m *module) getPerGroupPodStatusCounts(
fmt.Sprintf("JSONExtractString(labels, %s) AS container_name", containerReasonFps.Var(containerNameAttrKey)),
fmt.Sprintf("JSONExtractString(labels, %s) AS reason", containerReasonFps.Var(containerStatusReasonAttrKey)),
)
containerReasonFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
containerReasonFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
containerReasonFps.Where(
containerReasonFps.E("metric_name", containerStatusReasonMetricName),
containerReasonFps.GE("unix_milli", tsAdjustedStart),
@@ -432,7 +432,7 @@ func (m *module) getPerGroupPodStatusCounts(
)
containerInner.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN container_reason_fps AS fps ON samples.fingerprint = fps.fingerprint",
telemetrymetrics.DBName, distributedSamplesTable,
metricstelemetryschema.DBName, distributedSamplesTable,
))
containerInner.Where(
containerInner.E("samples.metric_name", containerStatusReasonMetricName),
@@ -611,7 +611,7 @@ func (m *module) getPerGroupPodRestartCounts(
mergedFilterExpr := mergeFilterExpressions(userFilterExpr, buildPageGroupsFilterExpr(pageGroups))
samplesStartMs, flooredEndMs, tsAdjustedStart, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
var (
filterClause *sqlbuilder.WhereClause
@@ -637,7 +637,7 @@ func (m *module) getPerGroupPodRestartCounts(
)
}
restartFps.Select(restartFpsCols...)
restartFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
restartFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
restartFps.Where(
restartFps.E("metric_name", containerRestartsMetricName),
restartFps.GE("unix_milli", tsAdjustedStart),
@@ -667,7 +667,7 @@ func (m *module) getPerGroupPodRestartCounts(
containerRestarts.Select(containerRestartsCols...)
containerRestarts.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN restart_fps AS fps ON samples.fingerprint = fps.fingerprint",
telemetrymetrics.DBName, distributedSamplesTable,
metricstelemetryschema.DBName, distributedSamplesTable,
))
containerRestarts.Where(
containerRestarts.E("samples.metric_name", containerRestartsMetricName),

View File

@@ -19,8 +19,8 @@ import (
"github.com/SigNoz/signoz/pkg/modules/dashboard"
"github.com/SigNoz/signoz/pkg/modules/metricsexplorer"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/dashboardtypes"
@@ -49,8 +49,8 @@ type module struct {
// NewModule constructs the metrics module with the provided dependencies.
func NewModule(ts telemetrystore.TelemetryStore, telemetryMetadataStore telemetrytypes.MetadataStore, cache cache.Cache, ruleStore ruletypes.RuleStore, dashboardModule dashboard.Module, fl flagger.Flagger, providerSettings factory.ProviderSettings, cfg metricsexplorer.Config) metricsexplorer.Module {
fieldMapper := telemetrymetrics.NewFieldMapper()
condBuilder := telemetrymetrics.NewConditionBuilder(fieldMapper)
fieldMapper := metricstelemetryschema.NewFieldMapper()
condBuilder := metricstelemetryschema.NewConditionBuilder(fieldMapper)
return &module{
telemetryStore: ts,
fieldMapper: fieldMapper,
@@ -89,7 +89,7 @@ func (m *module) listMeterMetrics(ctx context.Context, params *metricsexplorerty
"argMax(temporality, unix_milli) AS temporality",
"argMax(is_monotonic, unix_milli) AS is_monotonic",
)
sb.From(fmt.Sprintf("%s.%s", telemetrymeter.DBName, telemetrymeter.SamplesTableName))
sb.From(fmt.Sprintf("%s.%s", metertelemetryschema.DBName, metertelemetryschema.SamplesTableName))
if params.Start != nil && params.End != nil {
sb.Where(sb.Between("unix_milli", *params.Start, *params.End))
@@ -160,11 +160,11 @@ func (m *module) listMetrics(ctx context.Context, orgID valuer.UUID, params *met
hasRange := params.Start != nil && params.End != nil
if hasRange {
var distributedTsTable string
rangeStart, rangeEnd, distributedTsTable, _ = telemetrymetrics.WhichTSTableToUse(uint64(*params.Start), uint64(*params.End), false, nil)
rawSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTsTable))
rangeStart, rangeEnd, distributedTsTable, _ = metricstelemetryschema.WhichTSTableToUse(uint64(*params.Start), uint64(*params.End), false, nil)
rawSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTsTable))
rawSB.Where(rawSB.Between("unix_milli", rangeStart, rangeEnd))
} else {
rawSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV41weekTableName))
rawSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV41weekTableName))
}
applyListFilters(rawSB)
@@ -174,7 +174,7 @@ func (m *module) listMetrics(ctx context.Context, orgID valuer.UUID, params *met
if reductionEnabled {
reducedSB := sqlbuilder.NewSelectBuilder()
reducedSB.Select("DISTINCT metric_name")
reducedSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
if hasRange {
reducedSB.Where(reducedSB.Between("unix_milli", rangeStart, rangeEnd))
}
@@ -512,7 +512,7 @@ func (m *module) CheckMetricExists(ctx context.Context, orgID valuer.UUID, metri
sb := sqlbuilder.NewSelectBuilder()
sb.Select("count(*) > 0 as metricExists")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.Where(sb.E("metric_name", metricName))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -534,7 +534,7 @@ func (m *module) HasNonSigNozMetrics(ctx context.Context) (bool, error) {
sb := sqlbuilder.NewSelectBuilder()
sb.Select("count(*) > 0")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV41weekTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV41weekTableName))
sb.Where("metric_name NOT LIKE 'signoz_%'")
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -570,10 +570,10 @@ func (m *module) InspectMetrics(
return nil, err
}
tsStart, _, tsTable, _ := telemetrymetrics.WhichTSTableToUse(start, end, false, nil)
tsStart, _, tsTable, _ := metricstelemetryschema.WhichTSTableToUse(start, end, false, nil)
tsSb := sqlbuilder.NewSelectBuilder()
tsSb.Select("fingerprint", "labels")
tsSb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, tsTable))
tsSb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, tsTable))
tsSb.Where(
tsSb.E("metric_name", req.MetricName),
tsSb.GE("unix_milli", tsStart),
@@ -625,7 +625,7 @@ func (m *module) InspectMetrics(
samplesSb := sqlbuilder.NewSelectBuilder()
samplesSb.Select("fingerprint", "unix_milli", "value")
samplesSb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4TableName))
samplesSb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4TableName))
samplesSb.Where(
samplesSb.In("fingerprint", fingerprints...),
samplesSb.E("metric_name", req.MetricName),
@@ -717,7 +717,7 @@ func (m *module) fetchUpdatedMetadata(ctx context.Context, orgID valuer.UUID, me
"argMax(temporality, created_at) AS temporality",
"argMax(is_monotonic, created_at) AS is_monotonic",
)
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.UpdatedMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.UpdatedMetadataTableName))
sb.Where(sb.In("metric_name", args...))
sb.GroupBy("metric_name")
@@ -777,7 +777,7 @@ func (m *module) fetchTimeseriesMetadata(ctx context.Context, orgID valuer.UUID,
"anyLast(temporality) AS temporality",
"argMax(is_monotonic, unix_milli) AS is_monotonic",
)
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4TableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4TableName))
sb.Where(sb.In("metric_name", args...))
sb.GroupBy("metric_name")
@@ -891,7 +891,7 @@ func (m *module) checkForLabelInMetric(ctx context.Context, metricName string, l
sb := sqlbuilder.NewSelectBuilder()
sb.Select("count(*) > 0 AS has_label")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.Where(sb.E("metric_name", metricName))
sb.Where(sb.E("attr_name", label))
sb.Limit(1)
@@ -914,7 +914,7 @@ func (m *module) insertMetricsMetadata(ctx context.Context, orgID valuer.UUID, r
createdAt := time.Now().UnixMilli()
ib := sqlbuilder.NewInsertBuilder()
ib.InsertInto(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.UpdatedMetadataTableName))
ib.InsertInto(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.UpdatedMetadataTableName))
ib.Cols("metric_name", "temporality", "is_monotonic", "type", "description", "unit", "created_at")
ib.Values(
req.MetricName,
@@ -1016,9 +1016,9 @@ func (m *module) fetchMetricsStatsWithSamples(
}
}
start, end, distributedTsTable, localTsTable := telemetrymetrics.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
distributedSamplesTable, _ := telemetrymetrics.WhichSamplesTableToUse(uint64(req.Start), uint64(req.End), metrictypes.UnspecifiedType, metrictypes.TimeAggregationUnspecified, false, nil)
countExp := telemetrymetrics.CountExpressionForSamplesTable(distributedSamplesTable)
start, end, distributedTsTable, localTsTable := metricstelemetryschema.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
distributedSamplesTable, _ := metricstelemetryschema.WhichSamplesTableToUse(uint64(req.Start), uint64(req.End), metrictypes.UnspecifiedType, metrictypes.TimeAggregationUnspecified, false, nil)
countExp := metricstelemetryschema.CountExpressionForSamplesTable(distributedSamplesTable)
// Timeseries counts per metric
tsSB := sqlbuilder.NewSelectBuilder()
@@ -1026,7 +1026,7 @@ func (m *module) fetchMetricsStatsWithSamples(
"metric_name",
"uniq(fingerprint) AS timeseries",
)
tsSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTsTable))
tsSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTsTable))
tsSB.Where(tsSB.Between("unix_milli", start, end))
tsSB.Where("NOT startsWith(metric_name, 'signoz')")
if filterWhereClause != nil {
@@ -1040,7 +1040,7 @@ func (m *module) fetchMetricsStatsWithSamples(
"metric_name",
fmt.Sprintf("%s AS samples", countExp),
)
samplesSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedSamplesTable))
samplesSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedSamplesTable))
samplesSB.Where(samplesSB.Between("unix_milli", req.Start, req.End))
samplesSB.Where("NOT startsWith(metric_name, 'signoz')")
@@ -1053,7 +1053,7 @@ func (m *module) fetchMetricsStatsWithSamples(
if filterWhereClause != nil {
fingerprintSB := sqlbuilder.NewSelectBuilder()
fingerprintSB.Select("fingerprint")
fingerprintSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTsTable))
fingerprintSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTsTable))
fingerprintSB.Where(fingerprintSB.Between("unix_milli", start, end))
fingerprintSB.Where("NOT startsWith(metric_name, 'signoz')")
fingerprintSB.AddWhereClause(sqlbuilder.CopyWhereClause(filterWhereClause))
@@ -1064,7 +1064,7 @@ func (m *module) fetchMetricsStatsWithSamples(
} else {
metricNamesSB := sqlbuilder.NewSelectBuilder()
metricNamesSB.Select("DISTINCT metric_name")
metricNamesSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTsTable))
metricNamesSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTsTable))
metricNamesSB.Where(metricNamesSB.Between("unix_milli", start, end))
metricNamesSB.Where("NOT startsWith(metric_name, 'signoz')")
@@ -1090,18 +1090,18 @@ func (m *module) fetchMetricsStatsWithSamples(
// the data lands either of the _reduced_last/_reduced_sum tables
reducedLastSB := sqlbuilder.NewSelectBuilder()
reducedLastSB.Select("metric_name", "uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedLastSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedLastTableName))
reducedLastSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedLastTableName))
reducedLastSB.Where(reducedLastSB.Between("unix_milli", req.Start, req.End))
reducedLastSB.Where("NOT startsWith(metric_name, 'signoz')")
reducedSumSB := sqlbuilder.NewSelectBuilder()
reducedSumSB.Select("metric_name", "uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedSumSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedSumTableName))
reducedSumSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedSumTableName))
reducedSumSB.Where(reducedSumSB.Between("unix_milli", req.Start, req.End))
reducedSumSB.Where("NOT startsWith(metric_name, 'signoz')")
// separate query for reduced series counts
reducedTsSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedTsSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedTsSB.Where(reducedTsSB.Between("unix_milli", start, end))
reducedTsSB.Where("NOT startsWith(metric_name, 'signoz')")
reducedTsSB.GroupBy("metric_name")
@@ -1112,7 +1112,7 @@ func (m *module) fetchMetricsStatsWithSamples(
// samples uses a separate cte with local table
reducedFpSB := sqlbuilder.NewSelectBuilder()
reducedFpSB.Select("fingerprint")
reducedFpSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedLocalTableName))
reducedFpSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedLocalTableName))
reducedFpSB.Where(reducedFpSB.Between("unix_milli", start, end))
reducedFpSB.Where("NOT startsWith(metric_name, 'signoz')")
reducedFpSB.AddWhereClause(sqlbuilder.CopyWhereClause(filterWhereClause))
@@ -1220,11 +1220,11 @@ func (m *module) computeTimeseriesTreemap(ctx context.Context, orgID valuer.UUID
}
}
start, end, distributedTsTable, _ := telemetrymetrics.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
start, end, distributedTsTable, _ := metricstelemetryschema.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
totalTSBuilder := sqlbuilder.NewSelectBuilder()
totalTSBuilder.Select("uniq(fingerprint) AS total_time_series")
totalTSBuilder.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTsTable))
totalTSBuilder.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTsTable))
totalTSBuilder.Where(totalTSBuilder.Between("unix_milli", start, end))
metricsSB := sqlbuilder.NewSelectBuilder()
@@ -1232,7 +1232,7 @@ func (m *module) computeTimeseriesTreemap(ctx context.Context, orgID valuer.UUID
"metric_name",
"uniq(fingerprint) AS total_value",
)
metricsSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTsTable))
metricsSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTsTable))
metricsSB.Where(metricsSB.Between("unix_milli", start, end))
metricsSB.Where("NOT startsWith(metric_name, 'signoz')")
if filterWhereClause != nil {
@@ -1244,7 +1244,7 @@ func (m *module) computeTimeseriesTreemap(ctx context.Context, orgID valuer.UUID
if reductionEnabled {
reducedTotalTSBuilder := sqlbuilder.NewSelectBuilder()
reducedTotalTSBuilder.Select("uniq(fingerprint) AS total_time_series")
reducedTotalTSBuilder.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedTotalTSBuilder.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedTotalTSBuilder.Where(reducedTotalTSBuilder.Between("unix_milli", start, end))
reducedMetricsSB := sqlbuilder.NewSelectBuilder()
@@ -1252,7 +1252,7 @@ func (m *module) computeTimeseriesTreemap(ctx context.Context, orgID valuer.UUID
"metric_name",
"uniq(fingerprint) AS total_value",
)
reducedMetricsSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedMetricsSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedMetricsSB.Where(reducedMetricsSB.Between("unix_milli", start, end))
reducedMetricsSB.Where("NOT startsWith(metric_name, 'signoz')")
if filterWhereClause != nil {
@@ -1338,15 +1338,15 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
}
}
start, end, distributedTsTable, localTsTable := telemetrymetrics.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
distributedSamplesTable, _ := telemetrymetrics.WhichSamplesTableToUse(uint64(req.Start), uint64(req.End), metrictypes.UnspecifiedType, metrictypes.TimeAggregationUnspecified, false, nil)
countExp := telemetrymetrics.CountExpressionForSamplesTable(distributedSamplesTable)
start, end, distributedTsTable, localTsTable := metricstelemetryschema.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
distributedSamplesTable, _ := metricstelemetryschema.WhichSamplesTableToUse(uint64(req.Start), uint64(req.End), metrictypes.UnspecifiedType, metrictypes.TimeAggregationUnspecified, false, nil)
countExp := metricstelemetryschema.CountExpressionForSamplesTable(distributedSamplesTable)
candidateLimit := req.Limit + 50
metricCandidatesSB := sqlbuilder.NewSelectBuilder()
metricCandidatesSB.Select("metric_name")
metricCandidatesSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTsTable))
metricCandidatesSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTsTable))
metricCandidatesSB.Where("NOT startsWith(metric_name, 'signoz')")
metricCandidatesSB.Where(metricCandidatesSB.Between("unix_milli", start, end))
if filterWhereClause != nil {
@@ -1364,7 +1364,7 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
if reductionEnabled {
reducedCandidatesSB = sqlbuilder.NewSelectBuilder()
reducedCandidatesSB.Select("metric_name")
reducedCandidatesSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedCandidatesSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedCandidatesSB.Where("NOT startsWith(metric_name, 'signoz')")
reducedCandidatesSB.Where(reducedCandidatesSB.Between("unix_milli", start, end))
if filterWhereClause != nil {
@@ -1379,7 +1379,7 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
totalSamplesSB := sqlbuilder.NewSelectBuilder()
totalSamplesSB.Select(fmt.Sprintf("%s AS total_samples", countExp))
totalSamplesSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedSamplesTable))
totalSamplesSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedSamplesTable))
totalSamplesSB.Where(totalSamplesSB.Between("unix_milli", req.Start, req.End))
sampleCountsSB := sqlbuilder.NewSelectBuilder()
@@ -1387,7 +1387,7 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
"metric_name",
fmt.Sprintf("%s AS samples", countExp),
)
sampleCountsSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedSamplesTable))
sampleCountsSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedSamplesTable))
sampleCountsSB.Where(sampleCountsSB.Between("unix_milli", req.Start, req.End))
sampleCountsSB.Where("metric_name GLOBAL IN (SELECT metric_name FROM __metric_candidates)")
@@ -1397,13 +1397,13 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
if reductionEnabled {
reducedLastSB = sqlbuilder.NewSelectBuilder()
reducedLastSB.Select("metric_name", "uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedLastSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedLastTableName))
reducedLastSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedLastTableName))
reducedLastSB.Where(reducedLastSB.Between("unix_milli", req.Start, req.End))
reducedLastSB.Where("metric_name GLOBAL IN (SELECT metric_name FROM __reduced_metric_candidates)")
reducedSumSB = sqlbuilder.NewSelectBuilder()
reducedSumSB.Select("metric_name", "uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedSumSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedSumTableName))
reducedSumSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedSumTableName))
reducedSumSB.Where(reducedSumSB.Between("unix_milli", req.Start, req.End))
reducedSumSB.Where("metric_name GLOBAL IN (SELECT metric_name FROM __reduced_metric_candidates)")
}
@@ -1411,7 +1411,7 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
if filterWhereClause != nil {
fingerprintSB := sqlbuilder.NewSelectBuilder()
fingerprintSB.Select("fingerprint")
fingerprintSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTsTable))
fingerprintSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTsTable))
fingerprintSB.Where(fingerprintSB.Between("unix_milli", start, end))
fingerprintSB.Where("NOT startsWith(metric_name, 'signoz')")
fingerprintSB.AddWhereClause(sqlbuilder.CopyWhereClause(filterWhereClause))
@@ -1423,7 +1423,7 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
if reductionEnabled {
reducedFingerprintSB := sqlbuilder.NewSelectBuilder()
reducedFingerprintSB.Select("fingerprint")
reducedFingerprintSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedLocalTableName))
reducedFingerprintSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedLocalTableName))
reducedFingerprintSB.Where(reducedFingerprintSB.Between("unix_milli", start, end))
reducedFingerprintSB.Where("NOT startsWith(metric_name, 'signoz')")
reducedFingerprintSB.AddWhereClause(sqlbuilder.CopyWhereClause(filterWhereClause))
@@ -1457,12 +1457,12 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
// Grand total includes reduced sample volume so percentages stay consistent.
reducedTotalLastSB := sqlbuilder.NewSelectBuilder()
reducedTotalLastSB.Select("uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedTotalLastSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedLastTableName))
reducedTotalLastSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedLastTableName))
reducedTotalLastSB.Where(reducedTotalLastSB.Between("unix_milli", req.Start, req.End))
reducedTotalSumSB := sqlbuilder.NewSelectBuilder()
reducedTotalSumSB.Select("uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedTotalSumSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedSumTableName))
reducedTotalSumSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedSumTableName))
reducedTotalSumSB.Where(reducedTotalSumSB.Between("unix_milli", req.Start, req.End))
reducedTotalSamplesSB := sqlbuilder.NewSelectBuilder()
@@ -1552,7 +1552,7 @@ func (m *module) getMetricDataPoints(ctx context.Context, metricName string) (ui
sb := sqlbuilder.NewSelectBuilder()
sb.Select("sum(count) AS data_points")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4Agg30mTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4Agg30mTableName))
sb.Where(sb.E("metric_name", metricName))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -1574,7 +1574,7 @@ func (m *module) getMetricLastReceived(ctx context.Context, metricName string) (
sb := sqlbuilder.NewSelectBuilder()
sb.Select("MAX(last_reported_unix_milli) AS last_received_time")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.Where(sb.E("metric_name", metricName))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -1599,7 +1599,7 @@ func (m *module) getTotalTimeSeriesForMetricName(ctx context.Context, metricName
sb := sqlbuilder.NewSelectBuilder()
sb.Select("uniq(fingerprint) AS time_series_count")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV41weekTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV41weekTableName))
sb.Where(sb.E("metric_name", metricName))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -1623,7 +1623,7 @@ func (m *module) getActiveTimeSeriesForMetricName(ctx context.Context, metricNam
sb := sqlbuilder.NewSelectBuilder()
sb.Select("uniq(fingerprint) AS active_time_series")
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4TableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4TableName))
sb.Where(sb.E("metric_name", metricName))
sb.Where(sb.GTE("unix_milli", milli))
@@ -1649,7 +1649,7 @@ func (m *module) fetchMetricAttributes(ctx context.Context, metricName string, s
"groupUniqArray(1000)(attr_string_value) AS values",
"uniq(attr_string_value) AS valueCount",
)
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.Where(sb.E("metric_name", metricName))
sb.Where("NOT startsWith(attr_name, '__')")

View File

@@ -9,7 +9,7 @@ import (
schemamigrator "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/modules/promote"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
@@ -52,7 +52,7 @@ func (m *module) ListPromotedAndIndexedPaths(ctx context.Context) ([]promotetype
response := []promotetypes.PromotePath{}
for _, path := range promotedPaths {
fullPath := telemetrylogs.BodyPromotedColumnPrefix + path
fullPath := logstelemetryschema.BodyPromotedColumnPrefix + path
path = telemetrytypes.BodyJSONStringSearchPrefix + path
item := promotetypes.PromotePath{
Path: path,
@@ -68,7 +68,7 @@ func (m *module) ListPromotedAndIndexedPaths(ctx context.Context) ([]promotetype
// add the paths that are not promoted but have indexes
for path, indexes := range aggr {
path := strings.TrimPrefix(path, telemetrylogs.BodyV2ColumnPrefix)
path := strings.TrimPrefix(path, logstelemetryschema.BodyV2ColumnPrefix)
path = telemetrytypes.BodyJSONStringSearchPrefix + path
response = append(response, promotetypes.PromotePath{
Path: path,
@@ -108,8 +108,8 @@ func (m *module) createIndexes(ctx context.Context, indexes []schemamigrator.Ind
for _, index := range indexes {
alterStmt := schemamigrator.AlterTableAddIndex{
Database: telemetrylogs.DBName,
Table: telemetrylogs.LogsV2LocalTableName,
Database: logstelemetryschema.DBName,
Table: logstelemetryschema.LogsV2LocalTableName,
Index: index,
}
op := alterStmt.OnCluster(m.telemetryStore.Cluster())
@@ -153,10 +153,10 @@ func (m *module) PromoteAndIndexPaths(
}
}
if len(it.Indexes) > 0 {
parentColumn := telemetrylogs.LogsV2BodyV2Column
parentColumn := logstelemetryschema.LogsV2BodyV2Column
// if the path is already promoted or is being promoted, add it to the promoted column
if _, promoted := existingPromotedPaths[it.Path]; promoted || it.Promote {
parentColumn = telemetrylogs.LogsV2BodyPromotedColumn
parentColumn = logstelemetryschema.LogsV2BodyPromotedColumn
}
for _, index := range it.Indexes {

View File

@@ -35,7 +35,7 @@ func (c *conditionBuilder) ConditionFor(
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken are logs-body-only; reject for rule state history.
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for rule state history.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}

View File

@@ -11,8 +11,8 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/modules/services"
"github.com/SigNoz/signoz/pkg/querier"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -39,7 +39,7 @@ func (m *module) FetchTopLevelOperations(ctx context.Context, start time.Time, s
ctx = m.withServicesContext(ctx, "FetchTopLevelOperations")
db := m.TelemetryStore.ClickhouseDB()
query := fmt.Sprintf("SELECT name, serviceName, max(time) as ts FROM %s.%s WHERE time >= @start", telemetrytraces.DBName, telemetrytraces.TopLevelOperationsTableName)
query := fmt.Sprintf("SELECT name, serviceName, max(time) as ts FROM %s.%s WHERE time >= @start", tracestelemetryschema.DBName, tracestelemetryschema.TopLevelOperationsTableName)
args := []any{clickhouse.Named("start", start)}
if len(services) > 0 {
query += " AND serviceName IN @services"

File diff suppressed because one or more lines are too long

View File

@@ -137,8 +137,8 @@ func filterqueryParserInit() {
0, 0, 191, 19, 1, 0, 0, 0, 192, 190, 1, 0, 0, 0, 193, 194, 7, 2, 0, 0,
194, 21, 1, 0, 0, 0, 195, 196, 7, 3, 0, 0, 196, 197, 5, 1, 0, 0, 197, 198,
3, 26, 13, 0, 198, 199, 5, 2, 0, 0, 199, 23, 1, 0, 0, 0, 200, 201, 5, 27,
0, 0, 201, 202, 5, 1, 0, 0, 202, 203, 3, 26, 13, 0, 203, 204, 5, 2, 0,
0, 204, 25, 1, 0, 0, 0, 205, 210, 3, 28, 14, 0, 206, 207, 5, 5, 0, 0, 207,
0, 0, 201, 202, 5, 1, 0, 0, 202, 203, 3, 18, 9, 0, 203, 204, 5, 2, 0, 0,
204, 25, 1, 0, 0, 0, 205, 210, 3, 28, 14, 0, 206, 207, 5, 5, 0, 0, 207,
209, 3, 28, 14, 0, 208, 206, 1, 0, 0, 0, 209, 212, 1, 0, 0, 0, 210, 208,
1, 0, 0, 0, 210, 211, 1, 0, 0, 0, 211, 27, 1, 0, 0, 0, 212, 210, 1, 0,
0, 0, 213, 217, 3, 34, 17, 0, 214, 217, 3, 32, 16, 0, 215, 217, 3, 30,
@@ -2945,7 +2945,7 @@ type ISearchCallContext interface {
// Getter signatures
SEARCH() antlr.TerminalNode
LPAREN() antlr.TerminalNode
FunctionParamList() IFunctionParamListContext
ValueList() IValueListContext
RPAREN() antlr.TerminalNode
// IsSearchCallContext differentiates from other interfaces.
@@ -2992,10 +2992,10 @@ func (s *SearchCallContext) LPAREN() antlr.TerminalNode {
return s.GetToken(FilterQueryParserLPAREN, 0)
}
func (s *SearchCallContext) FunctionParamList() IFunctionParamListContext {
func (s *SearchCallContext) ValueList() IValueListContext {
var t antlr.RuleContext
for _, ctx := range s.GetChildren() {
if _, ok := ctx.(IFunctionParamListContext); ok {
if _, ok := ctx.(IValueListContext); ok {
t = ctx.(antlr.RuleContext)
break
}
@@ -3005,7 +3005,7 @@ func (s *SearchCallContext) FunctionParamList() IFunctionParamListContext {
return nil
}
return t.(IFunctionParamListContext)
return t.(IValueListContext)
}
func (s *SearchCallContext) RPAREN() antlr.TerminalNode {
@@ -3064,7 +3064,7 @@ func (p *FilterQueryParser) SearchCall() (localctx ISearchCallContext) {
}
{
p.SetState(202)
p.FunctionParamList()
p.ValueList()
}
{
p.SetState(203)

View File

@@ -1,85 +0,0 @@
package clickhouseprometheusv2
import (
"context"
"sync"
"github.com/SigNoz/signoz/pkg/prometheus"
"github.com/prometheus/prometheus/model/labels"
"github.com/prometheus/prometheus/storage"
"github.com/prometheus/prometheus/util/annotations"
)
// statementRecorder collects the statements a PromQL evaluation would run.
// Safe for concurrent use: the engine may Select selectors concurrently.
type statementRecorder struct {
mu sync.Mutex
statements []prometheus.CapturedStatement
}
func (r *statementRecorder) record(query string, args []any) {
r.mu.Lock()
defer r.mu.Unlock()
r.statements = append(r.statements, prometheus.CapturedStatement{Query: query, Args: args})
}
func (r *statementRecorder) Statements() []prometheus.CapturedStatement {
r.mu.Lock()
defer r.mu.Unlock()
out := make([]prometheus.CapturedStatement, len(r.statements))
copy(out, r.statements)
return out
}
type captureQueryable struct {
client *client
recorder *statementRecorder
}
func (c *captureQueryable) Querier(mint, maxt int64) (storage.Querier, error) {
return &captureQuerier{
querier: querier{mint: mint, maxt: maxt, client: c.client},
recorder: c.recorder,
}, nil
}
// captureQuerier builds the same SQL as the live querier but records it and
// returns no data. The fingerprint filter always takes the subquery form:
// without executing the series lookup, the inline literal set is unknown.
type captureQuerier struct {
querier
recorder *statementRecorder
}
func (c *captureQuerier) Select(ctx context.Context, _ bool, hints *storage.SelectHints, matchers ...*labels.Matcher) storage.SeriesSet {
start, end := c.window(hints)
samplesQuery, args, err := buildSamplesQuery(start, end, metricNamesFromMatchers(matchers), matchers, c.lastSamplePerStepFor(ctx, hints))
if err != nil {
return storage.ErrSeriesSet(err)
}
c.recorder.record(samplesQuery, args)
return storage.EmptySeriesSet()
}
func (c *captureQuerier) LabelValues(context.Context, string, *storage.LabelHints, ...*labels.Matcher) ([]string, annotations.Annotations, error) {
return nil, nil, nil
}
func (c *captureQuerier) LabelNames(context.Context, *storage.LabelHints, ...*labels.Matcher) ([]string, annotations.Annotations, error) {
return nil, nil, nil
}
// metricNamesFromMatchers extracts the statically known metric name, if any.
// The live path derives names from the matched series; the capture path has
// no execution results, so only a __name__ equality contributes.
func metricNamesFromMatchers(matchers []*labels.Matcher) []string {
for _, m := range matchers {
if m.Name == metricNameLabel && m.Type == labels.MatchEqual && m.Value != "" {
return []string{m.Value}
}
}
return nil
}

View File

@@ -1,180 +0,0 @@
package clickhouseprometheusv2
import (
"context"
"encoding/json"
"log/slog"
"math"
"slices"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/prometheus"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/prometheus/prometheus/model/labels"
promValue "github.com/prometheus/prometheus/model/value"
)
// seriesLookup holds a series lookup's result: matched fingerprints with
// their labels, and the distinct metric names seen on them.
type seriesLookup struct {
fingerprints map[uint64]labels.Labels
metricNames []string
}
// client executes the series, samples and raw queries against ClickHouse.
type client struct {
settings factory.ScopedProviderSettings
telemetryStore telemetrystore.TelemetryStore
lookbackMs int64
}
func newClient(settings factory.ScopedProviderSettings, telemetryStore telemetrystore.TelemetryStore, cfg prometheus.Config) *client {
lookback := cfg.LookbackDelta
if lookback <= 0 {
// Mirror the engine: promql defaults an unset lookback to 5m.
lookback = defaultLookbackDelta
}
return &client{
settings: settings,
telemetryStore: telemetryStore,
lookbackMs: lookback.Milliseconds(),
}
}
func (c *client) withContext(ctx context.Context, functionName string) context.Context {
return ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.TelemetrySignal: telemetrytypes.SignalMetrics.StringValue(),
instrumentationtypes.CodeNamespace: "clickhouse-prometheus-v2",
instrumentationtypes.CodeFunctionName: functionName,
})
}
func (c *client) selectSeries(ctx context.Context, query string, args []any) (*seriesLookup, error) {
ctx = c.withContext(ctx, "selectSeries")
rows, err := c.telemetryStore.ClickhouseDB().Query(ctx, query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
lookup := &seriesLookup{fingerprints: make(map[uint64]labels.Labels)}
names := make(map[string]struct{})
var fingerprint uint64
var labelsJSON string
for rows.Next() {
if err := rows.Scan(&fingerprint, &labelsJSON); err != nil {
return nil, err
}
lset, err := unmarshalLabels(labelsJSON)
if err != nil {
return nil, err
}
lookup.fingerprints[fingerprint] = lset
if name := lset.Get(metricNameLabel); name != "" {
names[name] = struct{}{}
}
}
if err := rows.Err(); err != nil {
return nil, err
}
for name := range names {
lookup.metricNames = append(lookup.metricNames, name)
}
slices.Sort(lookup.metricNames)
return lookup, nil
}
// unmarshalLabels parses the labels JSON column, dropping empty-valued
// labels: empty means "absent" in Prometheus, but stored attribute JSON can
// carry them.
func unmarshalLabels(s string) (labels.Labels, error) {
m := make(map[string]string)
if err := json.Unmarshal([]byte(s), &m); err != nil {
return labels.EmptyLabels(), err
}
builder := labels.NewScratchBuilder(len(m))
for k, v := range m {
if v == "" {
continue
}
builder.Add(k, v)
}
builder.Sort()
return builder.Labels(), nil
}
// selectSamples assembles per-series sample slices from a samples query (raw
// or last-sample-per-step; same column shape), whose rows arrive ordered by
// (fingerprint, unix_milli). Fingerprints missing from the lookup are skipped:
// the samples query's semi-join re-runs the series predicates and can match
// series registered after the lookup ran — the lookup is the read snapshot.
// Stale flags map to the engine's StaleNaN. Duplicate
// timestamps pass through: uniqueness is ingest's job, and v1 feeds them to
// the engine as-is over the same dirty data.
func (c *client) selectSamples(ctx context.Context, query string, args []any, lookup *seriesLookup) ([]*series, error) {
ctx = c.withContext(ctx, "selectSamples")
rows, err := c.telemetryStore.ClickhouseDB().Query(ctx, query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
var (
result []*series
current *series
fingerprint uint64
prevFp uint64
timestampMs int64
val float64
flags uint32
first = true
haveCurrent bool
staleMarker = math.Float64frombits(promValue.StaleNaN)
unknownCount int
)
for rows.Next() {
if err := rows.Scan(&fingerprint, &timestampMs, &val, &flags); err != nil {
return nil, err
}
if first || fingerprint != prevFp {
first = false
prevFp = fingerprint
lset, ok := lookup.fingerprints[fingerprint]
if !ok {
unknownCount++
haveCurrent = false
continue
}
current = &series{lset: lset}
result = append(result, current)
haveCurrent = true
}
if !haveCurrent {
continue
}
if flags&1 == 1 {
val = staleMarker
}
current.ts = append(current.ts, timestampMs)
current.vs = append(current.vs, val)
}
if err := rows.Err(); err != nil {
return nil, err
}
if unknownCount > 0 {
c.settings.Logger().DebugContext(ctx, "skipped samples of fingerprints missing from series lookup",
slog.Int("unknown_fingerprints", unknownCount))
}
return result, nil
}

View File

@@ -1,49 +0,0 @@
// Package clickhouseprometheusv2 is the second-generation ClickHouse-backed
// Prometheus provider. It exists because the v1 provider fetches every raw
// sample of a query's union window through the remote-read protobuf layer
// and hands it to the engine — the cost is a function of ingested data, not
// of the question asked. Here the stock promql engine evaluates over a
// native storage.Querier: no translation layer, per-selector fetch windows,
// and fetch reductions that are provably invisible to the engine. Every
// reduction either preserves engine semantics exactly or is not performed.
//
// # Series lookup
//
// Matchers resolve to series once per selector (selectSeries) against the
// time-series tables, which hold one row per (fingerprint, bucket) at
// 1h/6h/1d/1w granularities; timeSeriesTableFor picks the table whose bucket
// fits the window and rounds the window start down to the bucket boundary.
// How matchers become SQL, and why regexes are anchored, is documented at
// applySeriesConditions. Empty-valued labels come off at this boundary: an
// empty value means "label absent" in Prometheus, but stored attribute JSON
// can carry them.
//
// # Sample fetch
//
// Samples are fetched per selector using the engine's per-selector hints,
// not the query-wide union window, so foo / foo offset 1d reads two narrow
// windows instead of the widest one twice. Instant selectors of
// subquery-free queries fetch only the last sample per step bucket — see
// lastSamplePerStep for the correctness argument — while range selectors
// always fetch raw: every sample feeds the range function. Row assembly maps
// stale flags to the engine's StaleNaN and merges series with identical
// label sets (sortAndMerge), because the engine assumes storages never emit
// duplicates.
//
// # Sharding
//
// samples_v4 and time_series_v4 (and all their rollups) shard on the same
// key — cityHash64(env, temporality, metric_name, fingerprint) — so a
// series' samples and catalog rows live on the same shard. The samples
// fetch exploits that: it restricts by a shard-local series subquery, not a
// GLOBAL broadcast of the matched set. Delta-temporality
// series stay invisible to PromQL exactly as they are in v1: the rollout
// gate is parity with v1, and a Delta stream fed to rate() as-if-cumulative
// would be wrong, not just new.
//
// # Observability
//
// Every statement carries a log_comment with
// code.namespace=clickhouse-prometheus-v2 and code.function.name naming the
// call site, so this provider's work is attributable in system.query_log.
package clickhouseprometheusv2

View File

@@ -1,72 +0,0 @@
package clickhouseprometheusv2
import (
"context"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/prometheus"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/prometheus/prometheus/storage"
)
// Provider ties the package together: its own engine and parser, and the
// ClickHouse client behind the native storage.Querier. See the package
// documentation for how the read path differs from v1. It is exported as a
// concrete type — callers hold it directly, and an interface with a single
// implementation would only hide that dependency.
type Provider struct {
settings factory.ScopedProviderSettings
engine *prometheus.Engine
parser prometheus.Parser
client *client
}
var (
_ prometheus.Prometheus = (*Provider)(nil)
_ prometheus.StatementCapturer = (*Provider)(nil)
)
func NewFactory(telemetryStore telemetrystore.TelemetryStore) factory.ProviderFactory[prometheus.Prometheus, prometheus.Config] {
return factory.NewProviderFactory(factory.MustNewName("clickhousev2"), func(ctx context.Context, providerSettings factory.ProviderSettings, config prometheus.Config) (prometheus.Prometheus, error) {
return New(ctx, providerSettings, config, telemetryStore)
})
}
func New(_ context.Context, providerSettings factory.ProviderSettings, config prometheus.Config, telemetryStore telemetrystore.TelemetryStore) (*Provider, error) {
settings := factory.NewScopedProviderSettings(providerSettings, "github.com/SigNoz/signoz/pkg/prometheus/clickhouseprometheusv2")
engine := prometheus.NewEngine(settings.Logger(), config)
parser := prometheus.NewParser()
client := newClient(settings, telemetryStore, config)
return &Provider{
settings: settings,
engine: engine,
parser: parser,
client: client,
}, nil
}
func (p *Provider) Engine() *prometheus.Engine {
return p.engine
}
func (p *Provider) Parser() prometheus.Parser {
return p.parser
}
func (p *Provider) Storage() storage.Queryable {
return p
}
func (p *Provider) Querier(mint, maxt int64) (storage.Querier, error) {
return &querier{mint: mint, maxt: maxt, client: p.client}, nil
}
// CapturingStorage implements prometheus.StatementCapturer: a storage that
// records each selector's SQL without executing it, for the preview path.
// A fresh recorder per call keeps concurrent dry-runs isolated.
func (p *Provider) CapturingStorage() (storage.Queryable, prometheus.StatementRecorder) {
recorder := &statementRecorder{}
return &captureQueryable{client: p.client, recorder: recorder}, recorder
}

View File

@@ -1,170 +0,0 @@
package clickhouseprometheusv2
import (
"context"
"fmt"
"slices"
"time"
"github.com/SigNoz/signoz/pkg/prometheus"
"github.com/huandu/go-sqlbuilder"
"github.com/prometheus/prometheus/model/labels"
"github.com/prometheus/prometheus/storage"
"github.com/prometheus/prometheus/util/annotations"
)
// defaultLookbackDelta mirrors promql's default when the config leaves the
// lookback unset; the engine and the storage must agree on it for
// last-sample-per-step bucket anchoring.
const defaultLookbackDelta = 5 * time.Minute
// querier is a native storage.Querier over ClickHouse: Select builds SQL
// directly from the matchers and hints, with no remote-read protobuf layer.
type querier struct {
mint, maxt int64
client *client
}
var _ storage.Querier = (*querier)(nil)
func (q *querier) Select(ctx context.Context, sortSeries bool, hints *storage.SelectHints, matchers ...*labels.Matcher) storage.SeriesSet {
start, end := q.window(hints)
seriesQuery, seriesArgs, err := buildSeriesQuery(start, end, matchers)
if err != nil {
return storage.ErrSeriesSet(err)
}
lookup, err := q.client.selectSeries(ctx, seriesQuery, seriesArgs)
if err != nil {
return storage.ErrSeriesSet(err)
}
if len(lookup.fingerprints) == 0 {
return storage.EmptySeriesSet()
}
list, err := q.fetchSamples(ctx, start, end, matchers, lookup, q.lastSamplePerStepFor(ctx, hints))
if err != nil {
return storage.ErrSeriesSet(err)
}
// The engine assumes storages never emit duplicate label sets.
list = sortAndMerge(list)
return newSeriesSet(list)
}
func (q *querier) LabelValues(ctx context.Context, name string, hints *storage.LabelHints, matchers ...*labels.Matcher) ([]string, annotations.Annotations, error) {
sb := sqlbuilder.NewSelectBuilder()
if name == metricNameLabel {
sb.Select("DISTINCT metric_name AS value")
} else {
sb.Select(fmt.Sprintf("DISTINCT JSONExtractString(labels, %s) AS value", sb.Var(name)))
}
adjustedStart, table := timeSeriesTableFor(q.mint, q.maxt)
sb.From(fmt.Sprintf("%s.%s", databaseName, table))
if err := applySeriesConditions(sb, adjustedStart, q.maxt, matchers); err != nil {
return nil, nil, err
}
sb.Where("value != ''")
if hints != nil && hints.Limit > 0 {
sb.Limit(hints.Limit)
}
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
values, err := q.selectStrings(ctx, "LabelValues", query, args)
if err != nil {
return nil, nil, err
}
slices.Sort(values)
return values, nil, nil
}
func (q *querier) LabelNames(ctx context.Context, hints *storage.LabelHints, matchers ...*labels.Matcher) ([]string, annotations.Annotations, error) {
sb := sqlbuilder.NewSelectBuilder()
sb.Select("DISTINCT arrayJoin(JSONExtractKeys(labels)) AS name")
adjustedStart, table := timeSeriesTableFor(q.mint, q.maxt)
sb.From(fmt.Sprintf("%s.%s", databaseName, table))
if err := applySeriesConditions(sb, adjustedStart, q.maxt, matchers); err != nil {
return nil, nil, err
}
if hints != nil && hints.Limit > 0 {
sb.Limit(hints.Limit)
}
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
names, err := q.selectStrings(ctx, "LabelNames", query, args)
if err != nil {
return nil, nil, err
}
slices.Sort(names)
return names, nil, nil
}
func (q *querier) Close() error {
return nil
}
// window returns the per-selector fetch window from the hints (already
// adjusted for offset, @, range and lookback); mint/maxt span the union of
// all selectors and are the fallback.
func (q *querier) window(hints *storage.SelectHints) (int64, int64) {
if hints != nil && hints.Start != 0 && hints.End != 0 && hints.Start <= hints.End {
return hints.Start, hints.End
}
return q.mint, q.maxt
}
// lastSamplePerStepFor decides whether the fetch can keep only the last
// sample per step bucket (see lastSamplePerStep). Only instant selectors
// (hints.Range == 0) of subquery-free queries qualify: range selectors need
// every raw sample, and subquery selectors evaluate at the subquery's own
// step while hints carry the top-level step — the subquery-free proof
// arrives as QueryTraits in the context. The first evaluation timestamp is
// recovered as hints.Start + lookback - 1ms, inverting how the engine
// derives hints.Start.
func (q *querier) lastSamplePerStepFor(ctx context.Context, hints *storage.SelectHints) *lastSamplePerStep {
if hints == nil || hints.Range != 0 || hints.Start <= 0 {
return nil
}
traits, ok := prometheus.QueryTraitsFromContext(ctx)
if !ok || !traits.SubqueryFree {
return nil
}
firstEval := hints.Start + q.client.lookbackMs - 1
if firstEval > hints.End {
// Defensive: never anchor a bucket past the window.
firstEval = hints.End
}
return &lastSamplePerStep{firstEvalMs: firstEval, stepMs: hints.Step}
}
// fetchSamples runs the samples query for the matched series (see
// buildSamplesQuery).
func (q *querier) fetchSamples(ctx context.Context, start, end int64, matchers []*labels.Matcher, lookup *seriesLookup, lastPerStep *lastSamplePerStep) ([]*series, error) {
query, args, err := buildSamplesQuery(start, end, lookup.metricNames, matchers, lastPerStep)
if err != nil {
return nil, err
}
return q.client.selectSamples(ctx, query, args, lookup)
}
func (q *querier) selectStrings(ctx context.Context, fn, query string, args []any) ([]string, error) {
ctx = q.client.withContext(ctx, fn)
rows, err := q.client.telemetryStore.ClickhouseDB().Query(ctx, query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
var out []string
var v string
for rows.Next() {
if err := rows.Scan(&v); err != nil {
return nil, err
}
out = append(out, v)
}
if err := rows.Err(); err != nil {
return nil, err
}
return out, nil
}

View File

@@ -1,184 +0,0 @@
package clickhouseprometheusv2
import (
"sort"
"github.com/prometheus/prometheus/model/histogram"
"github.com/prometheus/prometheus/model/labels"
"github.com/prometheus/prometheus/storage"
"github.com/prometheus/prometheus/tsdb/chunkenc"
"github.com/prometheus/prometheus/util/annotations"
)
// series is one time series with samples as parallel slices ordered by
// timestamp. Deliberately not storage.NewListSeries: that boxes every sample
// as an interface value, a per-sample allocation this fetch path exists to
// avoid.
type series struct {
lset labels.Labels
ts []int64
vs []float64
}
var _ storage.Series = (*series)(nil)
func (s *series) Labels() labels.Labels {
return s.lset
}
func (s *series) Iterator(it chunkenc.Iterator) chunkenc.Iterator {
if fit, ok := it.(*floatIterator); ok {
fit.reset(s)
return fit
}
fit := &floatIterator{}
fit.reset(s)
return fit
}
// floatIterator implements chunkenc.Iterator over a series' sample slices.
type floatIterator struct {
s *series
i int
}
var _ chunkenc.Iterator = (*floatIterator)(nil)
func (it *floatIterator) reset(s *series) {
it.s = s
it.i = -1
}
func (it *floatIterator) Next() chunkenc.ValueType {
it.i++
if it.i >= len(it.s.ts) {
return chunkenc.ValNone
}
return chunkenc.ValFloat
}
func (it *floatIterator) Seek(t int64) chunkenc.ValueType { //nolint:govet // stdmethods flags io.Seeker; this is chunkenc.Iterator's Seek
if it.i < 0 {
it.i = 0
}
if it.i >= len(it.s.ts) {
return chunkenc.ValNone
}
// The current position, once valid, must not move backwards.
if it.s.ts[it.i] >= t {
return chunkenc.ValFloat
}
it.i += sort.Search(len(it.s.ts)-it.i, func(j int) bool {
return it.s.ts[it.i+j] >= t
})
if it.i >= len(it.s.ts) {
return chunkenc.ValNone
}
return chunkenc.ValFloat
}
func (it *floatIterator) At() (int64, float64) {
return it.s.ts[it.i], it.s.vs[it.i]
}
func (it *floatIterator) AtHistogram(*histogram.Histogram) (int64, *histogram.Histogram) {
return 0, nil
}
func (it *floatIterator) AtFloatHistogram(*histogram.FloatHistogram) (int64, *histogram.FloatHistogram) {
return 0, nil
}
func (it *floatIterator) AtT() int64 {
return it.s.ts[it.i]
}
// AtST returns the current start timestamp; not tracked by this storage.
func (it *floatIterator) AtST() int64 {
return 0
}
func (it *floatIterator) Err() error {
return nil
}
// seriesSet iterates a fully materialized, label-sorted list of series.
type seriesSet struct {
series []*series
i int
}
var _ storage.SeriesSet = (*seriesSet)(nil)
func newSeriesSet(list []*series) *seriesSet {
return &seriesSet{series: list, i: -1}
}
func (s *seriesSet) Next() bool {
s.i++
return s.i < len(s.series)
}
func (s *seriesSet) At() storage.Series {
return s.series[s.i]
}
func (s *seriesSet) Err() error {
return nil
}
func (s *seriesSet) Warnings() annotations.Annotations {
return nil
}
// sortAndMerge orders series by label set and merges identical label sets
// by timestamp (first sample wins ties): distinct fingerprints can carry
// identical label sets, and the engine assumes storages never emit
// duplicates.
func sortAndMerge(list []*series) []*series {
if len(list) < 2 {
return list
}
sort.Slice(list, func(i, j int) bool {
return labels.Compare(list[i].lset, list[j].lset) < 0
})
out := list[:1]
for _, s := range list[1:] {
last := out[len(out)-1]
if labels.Compare(last.lset, s.lset) != 0 {
out = append(out, s)
continue
}
merged := mergeSamples(last, s)
out[len(out)-1] = merged
}
return out
}
func mergeSamples(a, b *series) *series {
ts := make([]int64, 0, len(a.ts)+len(b.ts))
vs := make([]float64, 0, len(a.ts)+len(b.ts))
i, j := 0, 0
for i < len(a.ts) && j < len(b.ts) {
switch {
case a.ts[i] < b.ts[j]:
ts = append(ts, a.ts[i])
vs = append(vs, a.vs[i])
i++
case a.ts[i] > b.ts[j]:
ts = append(ts, b.ts[j])
vs = append(vs, b.vs[j])
j++
default:
ts = append(ts, a.ts[i])
vs = append(vs, a.vs[i])
i++
j++
}
}
ts = append(ts, a.ts[i:]...)
vs = append(vs, a.vs[i:]...)
ts = append(ts, b.ts[j:]...)
vs = append(vs, b.vs[j:]...)
return &series{lset: a.lset, ts: ts, vs: vs}
}

View File

@@ -1,164 +0,0 @@
package clickhouseprometheusv2
import (
"fmt"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/huandu/go-sqlbuilder"
"github.com/prometheus/prometheus/model/labels"
)
// buildSeriesQuery renders the series lookup: one row per matched fingerprint
// with its labels.
func buildSeriesQuery(start, end int64, matchers []*labels.Matcher) (string, []any, error) {
adjustedStart, table := timeSeriesTableFor(start, end)
sb := sqlbuilder.NewSelectBuilder()
sb.Select("fingerprint", "any(labels)")
sb.From(fmt.Sprintf("%s.%s", databaseName, table))
if err := applySeriesConditions(sb, adjustedStart, end, matchers); err != nil {
return "", nil, err
}
sb.GroupBy("fingerprint")
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
return query, args, nil
}
// buildSamplesQuery renders the samples fetch for the matched series: a
// semi-join re-runs the series predicates against the shard-local series
// table (complete by fingerprint co-locality, see localTimeSeriesTable — a
// GLOBAL broadcast would ship the matched set to every shard, and ClickHouse
// materializes the subquery's set per shard before the scan, so it still
// engages the fingerprint primary-key column). metricNames (observed on the
// matched series when the selector had no __name__ equality) narrows the
// primary-key scan. A non-nil lastPerStep groups to the last sample per
// step bucket.
func buildSamplesQuery(start, end int64, metricNames []string, matchers []*labels.Matcher, lastPerStep *lastSamplePerStep) (string, []any, error) {
sb := sqlbuilder.NewSelectBuilder()
if lastPerStep != nil {
// Aliases must not shadow source columns: ClickHouse resolves aliases
// in WHERE too, and "max(unix_milli) AS unix_milli" would put an
// aggregate into the WHERE clause (error 184).
sb.Select("fingerprint", "max(unix_milli) AS ts", "argMax(value, unix_milli) AS val", "argMax(flags, unix_milli) AS fl")
} else {
sb.Select("fingerprint", "unix_milli", "value", "flags")
}
sb.From(fmt.Sprintf("%s.%s", databaseName, distributedSamplesV4))
switch len(metricNames) {
case 0:
// No name constraint derivable; the primary-key prefix goes unused.
case 1:
sb.Where(sb.EQ("metric_name", metricNames[0]))
default:
sb.Where(sb.In("metric_name", sqlbuilder.List(metricNames)))
}
// Semantically redundant (the fingerprints already come from these
// temporalities) but engages the leading primary-key column.
sb.Where("temporality IN ['Cumulative', 'Unspecified']")
sub := sqlbuilder.NewSelectBuilder()
sub.Select("fingerprint")
adjustedStart, table := timeSeriesTableFor(start, end)
sub.From(fmt.Sprintf("%s.%s", databaseName, localTimeSeriesTable(table)))
if err := applySeriesConditions(sub, adjustedStart, end, matchers); err != nil {
return "", nil, err
}
sb.Where(sb.In("fingerprint", sub))
sb.Where(sb.GTE("unix_milli", start), sb.LTE("unix_milli", end))
if lastPerStep != nil {
sb.GroupBy("fingerprint")
if expr := lastPerStep.bucketExpr(); expr != "" {
sb.GroupBy(expr)
}
sb.OrderBy("fingerprint", "ts")
} else {
sb.OrderBy("fingerprint", "unix_milli")
}
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
return query, args, nil
}
// applySeriesConditions adds the WHERE conditions of a series table scan:
// __name__ matchers (all four types) translate to the metric_name column,
// every other matcher to a JSONExtractString condition on the labels column.
// An equality matcher against "" matches series without the label, mirroring
// PromQL, because JSONExtractString returns "" for missing keys. Regexes are
// anchored: PromQL matchers match the whole value, ClickHouse match()
// searches for a substring.
func applySeriesConditions(sb *sqlbuilder.SelectBuilder, start, end int64, matchers []*labels.Matcher) error {
for _, m := range matchers {
if m.Name != metricNameLabel {
continue
}
switch m.Type {
case labels.MatchEqual:
sb.Where(sb.EQ("metric_name", m.Value))
case labels.MatchNotEqual:
sb.Where(sb.NE("metric_name", m.Value))
case labels.MatchRegexp:
sb.Where(fmt.Sprintf("match(metric_name, %s)", sb.Var(anchorRegex(m.Value))))
case labels.MatchNotRegexp:
sb.Where(fmt.Sprintf("NOT match(metric_name, %s)", sb.Var(anchorRegex(m.Value))))
default:
return errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported matcher type %q for __name__", m.Type)
}
}
sb.Where("temporality IN ['Cumulative', 'Unspecified']")
// Inclusive upper bound: registration rows are hour-floored (and 6h/1d/1w
// for the rollup tables) by the exporter, so a series first registered in
// the bucket starting exactly at `end` would otherwise be invisible while
// its samples (<= end) are in range.
sb.Where(sb.GTE("unix_milli", start), sb.LTE("unix_milli", end))
for _, m := range matchers {
if m.Name == metricNameLabel {
continue
}
switch m.Type {
case labels.MatchEqual:
sb.Where(fmt.Sprintf("JSONExtractString(labels, %s) = %s", sb.Var(m.Name), sb.Var(m.Value)))
case labels.MatchNotEqual:
sb.Where(fmt.Sprintf("JSONExtractString(labels, %s) != %s", sb.Var(m.Name), sb.Var(m.Value)))
case labels.MatchRegexp:
sb.Where(fmt.Sprintf("match(JSONExtractString(labels, %s), %s)", sb.Var(m.Name), sb.Var(anchorRegex(m.Value))))
case labels.MatchNotRegexp:
sb.Where(fmt.Sprintf("NOT match(JSONExtractString(labels, %s), %s)", sb.Var(m.Name), sb.Var(anchorRegex(m.Value))))
default:
return errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported matcher type %q", m.Type)
}
}
return nil
}
// anchorRegex turns a PromQL regex into its fully-anchored form (see
// applySeriesConditions).
func anchorRegex(v string) string {
return "^(?:" + v + ")$"
}
// lastSamplePerStep reduces an instant-selector fetch to the last sample of
// each step bucket: bucket 0 is (start, firstEval], bucket i is
// (firstEval+(i-1)·step, firstEval+i·step]. Anchoring buckets at the first
// evaluation timestamp makes every bucket boundary an evaluation timestamp,
// so a non-final sample of a bucket can never be the latest sample in any
// (t-lookback, t] the engine resolves — the reduction is lossless. Real
// timestamps are preserved, so the engine's own lookback and staleness
// handling remain exact.
type lastSamplePerStep struct {
firstEvalMs int64
stepMs int64
}
func (t *lastSamplePerStep) bucketExpr() string {
if t.stepMs <= 0 {
// Instant query: a single evaluation at firstEval; one bucket.
return ""
}
return fmt.Sprintf(
"if(unix_milli <= %d, 0, intDiv(unix_milli - %d - 1, %d) + 1)",
t.firstEvalMs, t.firstEvalMs, t.stepMs,
)
}

View File

@@ -1,139 +0,0 @@
package clickhouseprometheusv2
import (
"testing"
"time"
"github.com/prometheus/prometheus/model/labels"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func mustMatcher(t *testing.T, mt labels.MatchType, name, value string) *labels.Matcher {
t.Helper()
m, err := labels.NewMatcher(mt, name, value)
require.NoError(t, err)
return m
}
func TestTimeSeriesTableFor(t *testing.T) {
base := time.Date(2026, 7, 10, 3, 27, 0, 0, time.UTC).UnixMilli()
tests := []struct {
name string
span time.Duration
wantTable string
roundTo time.Duration
}{
{"under 6h uses hourly table", 2 * time.Hour, distributedTimeSeriesV4, time.Hour},
{"under 1d uses 6h table", 12 * time.Hour, distributedTimeSeriesV46hrs, 6 * time.Hour},
{"under 1w uses 1d table", 3 * 24 * time.Hour, distributedTimeSeriesV41day, 24 * time.Hour},
{"over 1w uses 1w table", 10 * 24 * time.Hour, distributedTimeSeriesV41week, 7 * 24 * time.Hour},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
start, table := timeSeriesTableFor(base, base+tt.span.Milliseconds())
assert.Equal(t, tt.wantTable, table)
assert.Zero(t, start%tt.roundTo.Milliseconds())
assert.LessOrEqual(t, start, base)
})
}
}
func TestBuildSeriesQuery(t *testing.T) {
start := int64(1_700_000_000_000)
end := start + time.Hour.Milliseconds()
// The series table window rounds down to the table's bucket boundary.
adjustedStart := start - (start % time.Hour.Milliseconds())
t.Run("equality name and label matchers", func(t *testing.T) {
query, args, err := buildSeriesQuery(start, end, []*labels.Matcher{
mustMatcher(t, labels.MatchEqual, "__name__", "http_requests_total"),
mustMatcher(t, labels.MatchEqual, "job", "api"),
})
require.NoError(t, err)
assert.Equal(t,
"SELECT fingerprint, any(labels) FROM signoz_metrics.distributed_time_series_v4 WHERE metric_name = ? AND temporality IN ['Cumulative', 'Unspecified'] AND unix_milli >= ? AND unix_milli <= ? AND JSONExtractString(labels, ?) = ? GROUP BY fingerprint",
query,
)
assert.Equal(t, []any{"http_requests_total", adjustedStart, end, "job", "api"}, args)
})
t.Run("regex matchers are anchored", func(t *testing.T) {
_, args, err := buildSeriesQuery(start, end, []*labels.Matcher{
mustMatcher(t, labels.MatchEqual, "__name__", "up"),
mustMatcher(t, labels.MatchRegexp, "instance", "prod.*"),
mustMatcher(t, labels.MatchNotRegexp, "env", "dev|test"),
})
require.NoError(t, err)
assert.Equal(t, []any{"up", adjustedStart, end, "instance", "^(?:prod.*)$", "env", "^(?:dev|test)$"}, args)
})
t.Run("regex name matcher uses metric_name column", func(t *testing.T) {
query, args, err := buildSeriesQuery(start, end, []*labels.Matcher{
mustMatcher(t, labels.MatchRegexp, "__name__", "node_cpu.*|node_memory.*"),
})
require.NoError(t, err)
assert.Contains(t, query, "match(metric_name, ?)")
assert.NotContains(t, query, "JSONExtractString")
assert.Equal(t, []any{"^(?:node_cpu.*|node_memory.*)$", adjustedStart, end}, args)
})
t.Run("no name matcher omits metric_name condition", func(t *testing.T) {
query, _, err := buildSeriesQuery(start, end, []*labels.Matcher{
mustMatcher(t, labels.MatchEqual, "job", "api"),
})
require.NoError(t, err)
assert.NotContains(t, query, "metric_name")
})
}
func TestBuildSamplesQuery(t *testing.T) {
start := int64(1_700_000_000_000)
end := start + time.Hour.Milliseconds()
adjustedStart := start - (start % time.Hour.Milliseconds())
matchers := []*labels.Matcher{
mustMatcher(t, labels.MatchEqual, "__name__", "up"),
mustMatcher(t, labels.MatchEqual, "job", "api"),
}
t.Run("raw fetch filters by a shard-local semi-join", func(t *testing.T) {
query, args, err := buildSamplesQuery(start, end, []string{"up"}, matchers, nil)
require.NoError(t, err)
assert.Contains(t, query, "fingerprint IN (SELECT fingerprint FROM signoz_metrics.time_series_v4 WHERE ")
assert.NotContains(t, query, "GLOBAL IN")
assert.Contains(t, query, "ORDER BY fingerprint, unix_milli")
// Args follow placeholder order: samples metric name, the semi-join's
// series predicates, then the samples window bounds.
assert.Equal(t, []any{"up", "up", adjustedStart, end, "job", "api", start, end}, args)
})
t.Run("last-sample-per-step groups by step bucket anchored at first eval", func(t *testing.T) {
lastPerStep := &lastSamplePerStep{firstEvalMs: start + 299_999, stepMs: 60_000}
query, _, err := buildSamplesQuery(start, end, []string{"up"}, matchers, lastPerStep)
require.NoError(t, err)
assert.Contains(t, query, "argMax(value, unix_milli) AS val")
assert.Contains(t, query, "argMax(flags, unix_milli) AS fl")
assert.Contains(t, query, "GROUP BY fingerprint, if(unix_milli <= 1700000299999, 0, intDiv(unix_milli - 1700000299999 - 1, 60000) + 1)")
assert.Contains(t, query, "ORDER BY fingerprint, ts")
// Aliases must not shadow the source columns referenced in WHERE.
assert.NotContains(t, query, "AS unix_milli")
assert.NotContains(t, query, "AS value")
assert.NotContains(t, query, "AS flags")
})
t.Run("instant query keeps one bucket", func(t *testing.T) {
lastPerStep := &lastSamplePerStep{firstEvalMs: end, stepMs: 0}
query, _, err := buildSamplesQuery(start, end, []string{"up"}, matchers, lastPerStep)
require.NoError(t, err)
assert.Contains(t, query, "GROUP BY fingerprint ORDER BY fingerprint, ts")
assert.NotContains(t, query, "intDiv")
})
t.Run("multiple metric names from regex selector", func(t *testing.T) {
query, args, err := buildSamplesQuery(start, end, []string{"node_cpu", "node_memory"}, matchers, nil)
require.NoError(t, err)
assert.Contains(t, query, "metric_name IN (?, ?)")
assert.Equal(t, []any{"node_cpu", "node_memory", "up", adjustedStart, end, "job", "api", start, end}, args)
})
}

View File

@@ -1,70 +0,0 @@
package clickhouseprometheusv2
import "time"
// TODO(srikanthccv): consolidate the metrics table names and the
// window-to-table selection across the query builders (telemetrymetrics)
// and the prometheus providers instead of each package carrying its own
// copy.
const (
// metricNameLabel is the reserved PromQL label holding the metric name.
metricNameLabel string = "__name__"
databaseName string = "signoz_metrics"
distributedTimeSeriesV4 string = "distributed_time_series_v4"
distributedTimeSeriesV46hrs string = "distributed_time_series_v4_6hrs"
distributedTimeSeriesV41day string = "distributed_time_series_v4_1day"
distributedTimeSeriesV41week string = "distributed_time_series_v4_1week"
distributedSamplesV4 string = "distributed_samples_v4"
localTimeSeriesV4 string = "time_series_v4"
localTimeSeriesV46hrs string = "time_series_v4_6hrs"
localTimeSeriesV41day string = "time_series_v4_1day"
localTimeSeriesV41week string = "time_series_v4_1week"
)
// localTimeSeriesTable maps a distributed time series table to its shard-local
// table. Samples and time series shard on the same key
// (cityHash64(env, temporality, metric_name, fingerprint)), so a query whose
// top-level FROM is the distributed samples table can join or semi-join the
// local time series table inside each shard: the shard rewrite runs the
// subquery against the shard's own series rows, which are exactly the series
// of the shard's samples. No broadcast, no initiator-side join.
func localTimeSeriesTable(distributed string) string {
switch distributed {
case distributedTimeSeriesV46hrs:
return localTimeSeriesV46hrs
case distributedTimeSeriesV41day:
return localTimeSeriesV41day
case distributedTimeSeriesV41week:
return localTimeSeriesV41week
default:
return localTimeSeriesV4
}
}
var (
oneHourInMilliseconds = time.Hour.Milliseconds()
sixHoursInMilliseconds = time.Hour.Milliseconds() * 6
oneDayInMilliseconds = time.Hour.Milliseconds() * 24
oneWeekInMilliseconds = time.Hour.Milliseconds() * 24 * 7
)
// timeSeriesTableFor returns the adjusted start and the time series table for
// the window. Time series tables hold one row per (fingerprint, bucket), with
// bucket granularities of 1h, 6h, 1d and 1w; the start is rounded down to the
// bucket boundary so a window beginning mid-bucket still matches the bucket's
// row.
func timeSeriesTableFor(start, end int64) (int64, string) {
switch {
case end-start < sixHoursInMilliseconds:
return start - (start % oneHourInMilliseconds), distributedTimeSeriesV4
case end-start < oneDayInMilliseconds:
return start - (start % sixHoursInMilliseconds), distributedTimeSeriesV46hrs
case end-start < oneWeekInMilliseconds:
return start - (start % oneDayInMilliseconds), distributedTimeSeriesV41day
default:
return start - (start % oneWeekInMilliseconds), distributedTimeSeriesV41week
}
}

View File

@@ -1,49 +0,0 @@
package prometheus
import (
"context"
"github.com/prometheus/prometheus/promql/parser"
)
type queryTraitsKey struct{}
// QueryTraits carries per-query facts a storage implementation cannot derive
// from SelectHints alone. Call sites that parse the PromQL expression attach
// traits to the context before handing it to the engine; storages treat a
// missing traits value as "unknown" and stay conservative.
type QueryTraits struct {
// SubqueryFree is true when the query contains no subquery expression.
// Subquery selectors are evaluated at the subquery's own step, but
// SelectHints.Step always carries the top-level step, so step-aligned
// storage optimizations (e.g. keeping only the last sample per step
// bucket) are safe only when this is true.
SubqueryFree bool
}
// DetectQueryTraits derives QueryTraits from a parsed PromQL expression.
func DetectQueryTraits(expr parser.Expr) QueryTraits {
subqueryFree := true
parser.Inspect(expr, func(node parser.Node, _ []parser.Node) error {
if _, ok := node.(*parser.SubqueryExpr); ok {
subqueryFree = false
}
return nil
})
return QueryTraits{SubqueryFree: subqueryFree}
}
// NewContextWithQueryTraits returns a context carrying the given traits.
func NewContextWithQueryTraits(ctx context.Context, traits QueryTraits) context.Context {
return context.WithValue(ctx, queryTraitsKey{}, traits)
}
// QueryTraitsFromContext returns the traits attached to ctx, if any.
//
// Context is used here, unlike for backend selection, because traits must
// cross the promql engine to reach storage.Querier.Select, and the engine's
// interfaces offer no other channel; the alternative is a Prometheus fork.
func QueryTraitsFromContext(ctx context.Context) (QueryTraits, bool) {
traits, ok := ctx.Value(queryTraitsKey{}).(QueryTraits)
return traits, ok
}

View File

@@ -22,6 +22,13 @@ import (
"github.com/SigNoz/signoz/pkg/variables"
)
type Handler interface {
QueryRange(rw http.ResponseWriter, req *http.Request)
QueryRangePreview(rw http.ResponseWriter, req *http.Request)
QueryRawStream(rw http.ResponseWriter, req *http.Request)
ReplaceVariables(rw http.ResponseWriter, req *http.Request)
}
type handler struct {
set factory.ProviderSettings
analytics analytics.Analytics

View File

@@ -15,6 +15,14 @@ import (
"github.com/SigNoz/signoz/pkg/valuer"
)
// BucketCache is the interface for bucket-based caching.
type BucketCache interface {
// cached portion + list of gaps to fetch
GetMissRanges(ctx context.Context, orgID valuer.UUID, q qbtypes.Query, step qbtypes.Step) (cached *qbtypes.Result, missing []*qbtypes.TimeRange)
// store fresh buckets for future hits
Put(ctx context.Context, orgID valuer.UUID, q qbtypes.Query, step qbtypes.Step, fresh *qbtypes.Result)
}
// bucketCache implements the BucketCache interface.
type bucketCache struct {
cache cache.Cache

View File

@@ -11,8 +11,8 @@ import (
"github.com/ClickHouse/clickhouse-go/v2"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -20,6 +20,8 @@ import (
"github.com/SigNoz/signoz/pkg/valuer"
)
const estimateTimeout = 5 * time.Second
const traceOutsideRangeWarn = "Query %s references a trace_id that exists between %s and %s (UTC) but lies outside the selected time range; adjust the time range to see results"
type builderQuery[T any] struct {
@@ -247,6 +249,10 @@ func (q *builderQuery[T]) Execute(ctx context.Context) (*qbtypes.Result, error)
return nil, err
}
if err := q.enforceEstimate(ctx, stmt); err != nil {
return nil, err
}
// Execute the query with proper context for partial value detection
result, err := q.executeWithContext(ctx, stmt.Query, stmt.Args)
if err != nil {
@@ -258,6 +264,70 @@ func (q *builderQuery[T]) Execute(ctx context.Context) (*qbtypes.Result, error)
return result, nil
}
// estimateRows runs EXPLAIN ESTIMATE for a cost-guarded statement and returns its
// estimated total scan rows. guarded=false means there is nothing to enforce (no
// CostGuard or a non-positive budget). A non-nil error means the query must be
// rejected: either the estimate itself failed (fail closed — we cannot honor the
// budget without it) or the parent context was cancelled (surfaced as-is). Callers
// enforce the budget so it can be applied per-statement or cumulatively.
func (q *builderQuery[T]) estimateRows(ctx context.Context, stmt *qbtypes.Statement) (int64, bool, error) {
if stmt.CostGuard == nil || stmt.CostGuard.MaxScanRows <= 0 {
return 0, false, nil
}
estCtx, cancel := context.WithTimeout(ctx, estimateTimeout)
defer cancel()
entries, err := q.telemetryStore.Estimate(estCtx, stmt.Query, stmt.Args...)
if err != nil {
// Parent cancellation isn't the budget's concern — surface it unchanged.
if ctx.Err() != nil {
return 0, true, ctx.Err()
}
// Fail closed: this is a scan-heavy statement and without an estimate we
// cannot bound its cost, so running it unbounded is exactly what the guard
// exists to prevent.
reason := fmt.Sprintf("This query is too broad to plan within %s; narrow the time range or add a more selective filter.", estimateTimeout)
if estCtx.Err() != context.DeadlineExceeded {
reason = "Could not estimate this query's scan cost; narrow the time range or add a more selective filter."
q.logger.WarnContext(ctx, "EXPLAIN ESTIMATE failed; rejecting scan-heavy query (fail-closed)", errors.Attr(err))
}
return 0, true, errors.NewInvalidInputf(errors.CodeInvalidInput, "%s", withAdvisory(stmt.CostGuard.Warning, reason))
}
var rows int64
for _, e := range entries {
rows += e.Rows
}
return rows, true, nil
}
// enforceEstimate rejects a single scan-heavy statement (Statement.CostGuard) whose
// EXPLAIN ESTIMATE rows exceed its budget, before executing. Budget 0 disables.
func (q *builderQuery[T]) enforceEstimate(ctx context.Context, stmt *qbtypes.Statement) error {
rows, guarded, err := q.estimateRows(ctx, stmt)
if err != nil {
return err
}
if !guarded {
return nil
}
if budget := stmt.CostGuard.MaxScanRows; rows > budget {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "%s",
withAdvisory(stmt.CostGuard.Warning, fmt.Sprintf("This query would scan about %d rows in this range, over the limit of %d; narrow the time range or add a more selective filter.", rows, budget)))
}
return nil
}
// withAdvisory prefixes reason with the requirement's advisory (e.g. the search()
// warning) when present, so the rejection leads with why the query is expensive.
func withAdvisory(advisory, reason string) string {
if advisory == "" {
return reason
}
return strings.TrimRight(advisory, ". ") + ". " + reason
}
// narrowWindowByTraceID inspects the filter for trace_id predicates and clamps
// [fromMS,toMS] to the time range stored in signoz_traces.distributed_trace_summary.
// Returns the (possibly narrowed) window, overlap=false when the trace lies
@@ -277,12 +347,12 @@ func (q *builderQuery[T]) narrowWindowByTraceID(ctx context.Context, fromMS, toM
return fromMS, toMS, true, ""
}
traceIDs, found := telemetrytraces.ExtractTraceIDsFromFilter(q.spec.Filter.Expression)
traceIDs, found := tracestelemetryschema.ExtractTraceIDsFromFilter(q.spec.Filter.Expression)
if !found || len(traceIDs) == 0 {
return fromMS, toMS, true, ""
}
finder := telemetrytraces.NewTraceTimeRangeFinder(q.telemetryStore)
finder := tracestelemetryschema.NewTraceTimeRangeFinder(q.telemetryStore)
traceStart, traceEnd, exists, err := finder.GetTraceTimeRangeMulti(ctx, traceIDs)
if err != nil {
return fromMS, toMS, true, ""
@@ -491,6 +561,11 @@ func (q *builderQuery[T]) executeWindowList(ctx context.Context) (*qbtypes.Resul
var warnings []string
var warningsDocURL string
// Cost guard is enforced against the cumulative estimate across the buckets we
// actually visit — a broad search() that fans every bucket must be bounded by
// the per-query budget, not let each bucket pass its slice independently.
var estimatedScan int64
for _, r := range buckets {
q.spec.Offset = 0
q.spec.Limit = need
@@ -501,6 +576,17 @@ func (q *builderQuery[T]) executeWindowList(ctx context.Context) (*qbtypes.Resul
}
warnings = stmt.Warnings
warningsDocURL = stmt.WarningsDocURL
rowsEst, guarded, err := q.estimateRows(ctx, stmt)
if err != nil {
return nil, err
}
if guarded {
estimatedScan += rowsEst
if budget := stmt.CostGuard.MaxScanRows; estimatedScan > budget {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "%s",
withAdvisory(stmt.CostGuard.Warning, fmt.Sprintf("This query would scan about %d rows across the time range, over the limit of %d; narrow the time range or add a more selective filter.", estimatedScan, budget)))
}
}
// Execute with proper context for partial value detection
res, err := q.executeWithContext(ctx, stmt.Query, stmt.Args)
if err != nil {

View File

@@ -6,18 +6,18 @@ import (
"time"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/valuer"
)
func (q *querier) Collect(ctx context.Context, _ valuer.UUID) (map[string]any, error) {
stats := make(map[string]any)
tracesTable := fmt.Sprintf("%s.%s", telemetrytraces.DBName, telemetrytraces.SpanIndexV3TableName)
logsTable := fmt.Sprintf("%s.%s", telemetrylogs.DBName, telemetrylogs.LogsV2TableName)
metricsTable := fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4TableName)
tracesTable := fmt.Sprintf("%s.%s", tracestelemetryschema.DBName, tracestelemetryschema.SpanIndexV3TableName)
logsTable := fmt.Sprintf("%s.%s", logstelemetryschema.DBName, logstelemetryschema.LogsV2TableName)
metricsTable := fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4TableName)
var (
traces uint64

View File

@@ -5,16 +5,11 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/statementbuilder"
)
const DefaultMaxConcurrentQueries = 8
type SkipResourceFingerprint struct {
Enabled bool `yaml:"enabled" mapstructure:"enabled"`
// If count of fingerprint is above threshold, skip the fingerprint subquery and filter on main table instead.
Threshold uint64 `yaml:"threshold" mapstructure:"threshold"`
}
// Config represents the configuration for the querier.
type Config struct {
// CacheTTL is the TTL for cached query results
@@ -23,10 +18,12 @@ type Config struct {
FluxInterval time.Duration `yaml:"flux_interval" mapstructure:"flux_interval"`
// MaxConcurrentQueries is the maximum number of concurrent queries for missing ranges
MaxConcurrentQueries int `yaml:"max_concurrent_queries" mapstructure:"max_concurrent_queries"`
// SkipResourceFingerprint configures when the resource fingerprint subquery is skipped in favor of main-table filtering.
SkipResourceFingerprint SkipResourceFingerprint `yaml:"skip_resource_fingerprint" mapstructure:"skip_resource_fingerprint"`
// LogTraceIDWindowPadding is the padding added to narrowed down timerange from trace summary to logs with trace_id filter.
LogTraceIDWindowPadding time.Duration `yaml:"log_trace_id_window_padding" mapstructure:"log_trace_id_window_padding"`
// StatementBuilder config is embedded (squashed) so its keys live directly under
// the querier namespace, e.g. querier.skip_resource_fingerprint / querier.search_max_scan_rows.
statementbuilder.Config `mapstructure:",squash" yaml:",squash"`
}
// NewConfigFactory creates a new config factory for querier.
@@ -37,14 +34,11 @@ func NewConfigFactory() factory.ConfigFactory {
func newConfig() factory.Config {
return Config{
// Default values
CacheTTL: 168 * time.Hour,
FluxInterval: 5 * time.Minute,
MaxConcurrentQueries: DefaultMaxConcurrentQueries,
SkipResourceFingerprint: SkipResourceFingerprint{
Enabled: false,
Threshold: 100000,
},
CacheTTL: 168 * time.Hour,
FluxInterval: 5 * time.Minute,
MaxConcurrentQueries: DefaultMaxConcurrentQueries,
LogTraceIDWindowPadding: 5 * time.Minute,
Config: statementbuilder.NewConfig(),
}
}
@@ -59,12 +53,13 @@ func (c Config) Validate() error {
if c.MaxConcurrentQueries <= 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "max_concurrent_queries must be positive, got %v", c.MaxConcurrentQueries)
}
if c.SkipResourceFingerprint.Enabled && c.SkipResourceFingerprint.Threshold == 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "skip_resource_fingerprint.threshold must be > 0 when enabled")
}
if c.LogTraceIDWindowPadding < 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "log_trace_id_window_padding must not be negative, got %v", c.LogTraceIDWindowPadding)
}
// The embedded statement-builder config's own Validate is shadowed by this method; call it explicitly.
if err := c.Config.Validate(); err != nil {
return err
}
return nil
}

View File

@@ -1,35 +0,0 @@
package querier
import (
"context"
"net/http"
"github.com/SigNoz/signoz/pkg/statsreporter"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
)
// Querier interface defines the contract for querying data.
type Querier interface {
QueryRange(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest) (*qbtypes.QueryRangeResponse, error)
QueryRawStream(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, client *qbtypes.RawStream)
statsreporter.StatsCollector
// QueryRangePreview validates and renders the queries without executing them.
QueryRangePreview(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, opts qbtypes.QueryRangePreviewOptions) (*qbtypes.QueryRangePreviewResponse, error)
}
// BucketCache is the interface for bucket-based caching.
type BucketCache interface {
// cached portion + list of gaps to fetch
GetMissRanges(ctx context.Context, orgID valuer.UUID, q qbtypes.Query, step qbtypes.Step) (cached *qbtypes.Result, missing []*qbtypes.TimeRange)
// store fresh buckets for future hits
Put(ctx context.Context, orgID valuer.UUID, q qbtypes.Query, step qbtypes.Step, fresh *qbtypes.Result)
}
type Handler interface {
QueryRange(rw http.ResponseWriter, req *http.Request)
// QueryRangePreview is the dry-run endpoint: validate and render without executing.
QueryRangePreview(rw http.ResponseWriter, req *http.Request)
QueryRawStream(rw http.ResponseWriter, req *http.Request)
ReplaceVariables(rw http.ResponseWriter, req *http.Request)
}

View File

@@ -21,6 +21,7 @@ import (
"github.com/SigNoz/signoz/pkg/prometheus"
"github.com/SigNoz/signoz/pkg/query-service/utils"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statsreporter"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
@@ -35,6 +36,15 @@ var (
intervalWarn = "Query %s is requesting aggregation interval %v seconds, which is smaller than the minimum allowed interval of %v seconds for selected time range. Using the minimum instead"
)
// Querier interface defines the contract for querying data.
type Querier interface {
QueryRange(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest) (*qbtypes.QueryRangeResponse, error)
QueryRawStream(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, client *qbtypes.RawStream)
statsreporter.StatsCollector
// QueryRangePreview validates and renders the queries without executing them.
QueryRangePreview(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, opts qbtypes.QueryRangePreviewOptions) (*qbtypes.QueryRangePreviewResponse, error)
}
type querier struct {
logger *slog.Logger
fl flagger.Flagger

View File

@@ -119,17 +119,17 @@ func TestQueryRange_MetricTypeFromStore(t *testing.T) {
providerSettings,
telemetryStore,
metadataStore,
nil, // prometheus
nil, // traceStmtBuilder
nil, // logStmtBuilder
nil, // auditStmtBuilder
&mockMetricStmtBuilder{}, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
flaggertest.New(t), // flagger
0, // logTraceIDWindowPadding
0, // maxConcurrentQueries
nil, // prometheus
nil, // traceStmtBuilder
nil, // logStmtBuilder
nil, // auditStmtBuilder
&mockMetricStmtBuilder{},
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
flaggertest.New(t), // flagger
0, // logTraceIDWindowPadding
0, // maxConcurrentQueries
)
req := &qbtypes.QueryRangeRequest{

View File

@@ -0,0 +1,52 @@
package queriertest
import (
"context"
"github.com/SigNoz/signoz/pkg/querier"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
)
// MockQuerier implements querier.Querier for testing. Each behavior can be
// overridden via a function field; unset fields return empty, non-error values.
type MockQuerier struct {
QueryRangeFunc func(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest) (*qbtypes.QueryRangeResponse, error)
QueryRawStreamFunc func(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, client *qbtypes.RawStream)
QueryRangePreviewFunc func(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, opts qbtypes.QueryRangePreviewOptions) (*qbtypes.QueryRangePreviewResponse, error)
CollectFunc func(ctx context.Context, orgID valuer.UUID) (map[string]any, error)
}
var _ querier.Querier = (*MockQuerier)(nil)
// NewMockQuerier creates a new MockQuerier.
func NewMockQuerier() *MockQuerier {
return &MockQuerier{}
}
func (m *MockQuerier) QueryRange(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest) (*qbtypes.QueryRangeResponse, error) {
if m.QueryRangeFunc != nil {
return m.QueryRangeFunc(ctx, orgID, req)
}
return &qbtypes.QueryRangeResponse{}, nil
}
func (m *MockQuerier) QueryRawStream(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, client *qbtypes.RawStream) {
if m.QueryRawStreamFunc != nil {
m.QueryRawStreamFunc(ctx, orgID, req, client)
}
}
func (m *MockQuerier) QueryRangePreview(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, opts qbtypes.QueryRangePreviewOptions) (*qbtypes.QueryRangePreviewResponse, error) {
if m.QueryRangePreviewFunc != nil {
return m.QueryRangePreviewFunc(ctx, orgID, req, opts)
}
return &qbtypes.QueryRangePreviewResponse{}, nil
}
func (m *MockQuerier) Collect(ctx context.Context, orgID valuer.UUID) (map[string]any, error) {
if m.CollectFunc != nil {
return m.CollectFunc(ctx, orgID)
}
return map[string]any{}, nil
}

View File

@@ -3,192 +3,54 @@ package signozquerier
import (
"context"
"github.com/SigNoz/signoz/pkg/cache"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/prometheus"
"github.com/SigNoz/signoz/pkg/querier"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetadata"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// NewFactory creates a new factory for the signoz querier provider.
// NewFactory creates a new factory for the signoz querier provider. The query
// stack (metadata store, statement builders, bucket cache) is assembled once in
// signoz.go and injected here.
func NewFactory(
telemetryStore telemetrystore.TelemetryStore,
prometheus prometheus.Prometheus,
cache cache.Cache,
metadataStore telemetrytypes.MetadataStore,
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation],
logStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation],
auditStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation],
metricStmtBuilder qbtypes.StatementBuilder[qbtypes.MetricAggregation],
meterStmtBuilder qbtypes.StatementBuilder[qbtypes.MetricAggregation],
traceOperatorStmtBuilder qbtypes.TraceOperatorStatementBuilder,
bucketCache querier.BucketCache,
flagger flagger.Flagger,
) factory.ProviderFactory[querier.Querier, querier.Config] {
return factory.NewProviderFactory(
factory.MustNewName("signoz"),
func(
ctx context.Context,
_ context.Context,
settings factory.ProviderSettings,
cfg querier.Config,
) (querier.Querier, error) {
return newProvider(ctx, settings, cfg, telemetryStore, prometheus, cache, flagger)
return querier.New(
settings,
telemetryStore,
metadataStore,
prometheus,
traceStmtBuilder,
logStmtBuilder,
auditStmtBuilder,
metricStmtBuilder,
meterStmtBuilder,
traceOperatorStmtBuilder,
bucketCache,
flagger,
cfg.LogTraceIDWindowPadding,
cfg.MaxConcurrentQueries,
), nil
},
)
}
func newProvider(
_ context.Context,
settings factory.ProviderSettings,
cfg querier.Config,
telemetryStore telemetrystore.TelemetryStore,
prometheus prometheus.Prometheus,
cache cache.Cache,
flagger flagger.Flagger,
) (querier.Querier, error) {
// Create telemetry metadata store
telemetryMetadataStore := telemetrymetadata.NewTelemetryMetaStore(
settings,
telemetryStore,
telemetrytraces.DBName,
telemetrytraces.TagAttributesV2TableName,
telemetrytraces.SpanAttributesKeysTblName,
telemetrytraces.SpanIndexV3TableName,
telemetrymetrics.DBName,
telemetrymetrics.AttributesMetadataTableName,
telemetrymeter.DBName,
telemetrymeter.SamplesAgg1dTableName,
telemetrylogs.DBName,
telemetrylogs.LogsV2TableName,
telemetrylogs.TagAttributesV2TableName,
telemetrylogs.LogAttributeKeysTblName,
telemetrylogs.LogResourceKeysTblName,
telemetryaudit.DBName,
telemetryaudit.AuditLogsTableName,
telemetryaudit.TagAttributesTableName,
telemetryaudit.LogAttributeKeysTblName,
telemetryaudit.LogResourceKeysTblName,
telemetrymetadata.DBName,
telemetrymetadata.AttributesMetadataTableName,
telemetrymetadata.ColumnEvolutionMetadataTableName,
flagger,
)
// Create trace statement builder
traceFieldMapper := telemetrytraces.NewFieldMapper()
traceConditionBuilder := telemetrytraces.NewConditionBuilder(traceFieldMapper)
traceAggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, traceFieldMapper, traceConditionBuilder, flagger)
traceStmtBuilder := telemetrytraces.NewTraceQueryStatementBuilder(
settings,
telemetryMetadataStore,
traceFieldMapper,
traceConditionBuilder,
traceAggExprRewriter,
telemetryStore,
flagger,
cfg.SkipResourceFingerprint.Enabled,
cfg.SkipResourceFingerprint.Threshold,
)
// Create trace operator statement builder
traceOperatorStmtBuilder := telemetrytraces.NewTraceOperatorStatementBuilder(
settings,
telemetryMetadataStore,
traceFieldMapper,
traceConditionBuilder,
traceStmtBuilder,
traceAggExprRewriter,
flagger,
)
// Create log statement builder
logFieldMapper := telemetrylogs.NewFieldMapper(flagger)
logConditionBuilder := telemetrylogs.NewConditionBuilder(logFieldMapper, flagger)
logAggExprRewriter := querybuilder.NewAggExprRewriter(
settings,
telemetrylogs.DefaultFullTextColumn,
logFieldMapper,
logConditionBuilder,
flagger,
)
logStmtBuilder := telemetrylogs.NewLogQueryStatementBuilder(
settings,
telemetryMetadataStore,
logFieldMapper,
logConditionBuilder,
logAggExprRewriter,
telemetrylogs.DefaultFullTextColumn,
flagger,
telemetryStore,
cfg.SkipResourceFingerprint.Enabled,
cfg.SkipResourceFingerprint.Threshold,
)
// Create audit statement builder
auditFieldMapper := telemetryaudit.NewFieldMapper()
auditConditionBuilder := telemetryaudit.NewConditionBuilder(auditFieldMapper)
auditAggExprRewriter := querybuilder.NewAggExprRewriter(
settings,
telemetryaudit.DefaultFullTextColumn,
auditFieldMapper,
auditConditionBuilder,
flagger,
)
auditStmtBuilder := telemetryaudit.NewAuditQueryStatementBuilder(
settings,
telemetryMetadataStore,
auditFieldMapper,
auditConditionBuilder,
auditAggExprRewriter,
telemetryaudit.DefaultFullTextColumn,
flagger,
)
// Create metric statement builder
metricFieldMapper := telemetrymetrics.NewFieldMapper()
metricConditionBuilder := telemetrymetrics.NewConditionBuilder(metricFieldMapper)
metricStmtBuilder := telemetrymetrics.NewMetricQueryStatementBuilder(
settings,
telemetryMetadataStore,
metricFieldMapper,
metricConditionBuilder,
flagger,
)
// Create meter statement builder
meterStmtBuilder := telemetrymeter.NewMeterQueryStatementBuilder(
settings,
telemetryMetadataStore,
metricFieldMapper,
metricConditionBuilder,
metricStmtBuilder,
)
// Create bucket cache
bucketCache := querier.NewBucketCache(
settings,
cache,
cfg.CacheTTL,
cfg.FluxInterval,
)
// Create and return the querier
return querier.New(
settings,
telemetryStore,
telemetryMetadataStore,
prometheus,
traceStmtBuilder,
logStmtBuilder,
auditStmtBuilder,
metricStmtBuilder,
meterStmtBuilder,
traceOperatorStmtBuilder,
bucketCache,
flagger,
cfg.LogTraceIDWindowPadding,
cfg.MaxConcurrentQueries,
), nil
}

View File

@@ -18,6 +18,13 @@ import (
"github.com/SigNoz/signoz/pkg/querier/signozquerier"
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/SigNoz/signoz/pkg/sqlstore/sqlstoretest"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/auditstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/logsstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/meterstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/metricsstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/tracesstatementbuilder"
"github.com/SigNoz/signoz/pkg/telemetrymetadata"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
"github.com/stretchr/testify/require"
@@ -105,7 +112,23 @@ func NewTestManager(t *testing.T, testOpts *TestManagerOptions) *Manager {
}
// Create querier with test values
providerFactory := signozquerier.NewFactory(telemetryStore, prometheus, cache, flagger)
metadataStore := telemetrymetadata.NewTelemetryMetaStore(providerSettings, telemetryStore, flagger)
cfg := statementbuilder.Config{}
ctx := context.Background()
traceStmtBuilder, err := tracesstatementbuilder.NewFactory(telemetryStore, metadataStore, flagger).New(ctx, providerSettings, cfg)
require.NoError(t, err)
traceOperatorStmtBuilder, err := tracesstatementbuilder.NewOperatorFactory(telemetryStore, metadataStore, flagger).New(ctx, providerSettings, cfg)
require.NoError(t, err)
logStmtBuilder, err := logsstatementbuilder.NewFactory(telemetryStore, metadataStore, flagger).New(ctx, providerSettings, cfg)
require.NoError(t, err)
auditStmtBuilder, err := auditstatementbuilder.NewFactory(metadataStore, flagger).New(ctx, providerSettings, cfg)
require.NoError(t, err)
metricStmtBuilder, err := metricsstatementbuilder.NewFactory(metadataStore, flagger).New(ctx, providerSettings, cfg)
require.NoError(t, err)
meterStmtBuilder, err := meterstatementbuilder.NewFactory(metadataStore, flagger).New(ctx, providerSettings, cfg)
require.NoError(t, err)
bucketCache := querier.NewBucketCache(providerSettings, cache, 0, 0)
providerFactory := signozquerier.NewFactory(telemetryStore, prometheus, metadataStore, traceStmtBuilder, logStmtBuilder, auditStmtBuilder, metricStmtBuilder, meterStmtBuilder, traceOperatorStmtBuilder, bucketCache, flagger)
mockQuerier, err := providerFactory.New(context.Background(), providerSettings, querier.Config{})
require.NoError(t, err)

View File

@@ -8,11 +8,11 @@ import (
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querier"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/logsstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/metricsstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/tracesstatementbuilder"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
"github.com/stretchr/testify/require"
@@ -24,7 +24,7 @@ func prepareQuerierForMetrics(t *testing.T, telemetryStore telemetrystore.Teleme
providerSettings := instrumentationtest.New().ToProviderSettings()
metadataStore := telemetrytypestest.NewMockMetadataStore()
flagger, err := flagger.New(
fl, err := flagger.New(
context.Background(),
instrumentationtest.New().ToProviderSettings(),
flagger.Config{},
@@ -32,15 +32,8 @@ func prepareQuerierForMetrics(t *testing.T, telemetryStore telemetrystore.Teleme
)
require.NoError(t, err)
metricFieldMapper := telemetrymetrics.NewFieldMapper()
metricConditionBuilder := telemetrymetrics.NewConditionBuilder(metricFieldMapper)
metricStmtBuilder := telemetrymetrics.NewMetricQueryStatementBuilder(
providerSettings,
metadataStore,
metricFieldMapper,
metricConditionBuilder,
flagger,
)
metricStmtBuilder, err := metricsstatementbuilder.NewFactory(metadataStore, fl).New(context.Background(), providerSettings, statementbuilder.Config{})
require.NoError(t, err)
return querier.New(
providerSettings,
@@ -54,7 +47,7 @@ func prepareQuerierForMetrics(t *testing.T, telemetryStore telemetrystore.Teleme
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
flagger,
fl,
0,
0, // maxConcurrentQueries (0 means default)
), metadataStore
@@ -73,40 +66,21 @@ func prepareQuerierForLogs(t *testing.T, telemetryStore telemetrystore.Telemetry
metadataStore.KeysMap = keysMap
fl := flaggertest.New(t)
logFieldMapper := telemetrylogs.NewFieldMapper(fl)
logConditionBuilder := telemetrylogs.NewConditionBuilder(logFieldMapper, fl)
logAggExprRewriter := querybuilder.NewAggExprRewriter(
providerSettings,
telemetrylogs.DefaultFullTextColumn,
logFieldMapper,
logConditionBuilder,
fl,
)
logStmtBuilder := telemetrylogs.NewLogQueryStatementBuilder(
providerSettings,
metadataStore,
logFieldMapper,
logConditionBuilder,
logAggExprRewriter,
telemetrylogs.DefaultFullTextColumn,
fl,
nil,
false,
100000,
)
logStmtBuilder, err := logsstatementbuilder.NewFactory(nil, metadataStore, fl).New(context.Background(), providerSettings, statementbuilder.Config{})
require.NoError(t, err)
return querier.New(
providerSettings,
telemetryStore,
metadataStore,
nil, // prometheus
nil, // traceStmtBuilder
logStmtBuilder, // logStmtBuilder
nil, // auditStmtBuilder
nil, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
nil, // prometheus
nil, // traceStmtBuilder
logStmtBuilder,
nil, // auditStmtBuilder
nil, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
fl,
5*time.Minute, // logTraceIDWindowPadding
0, // maxConcurrentQueries (0 means default)
@@ -126,36 +100,22 @@ func prepareQuerierForTraces(t *testing.T, telemetryStore telemetrystore.Telemet
}
metadataStore.KeysMap = keysMap
// Create trace statement builder
traceFieldMapper := telemetrytraces.NewFieldMapper()
traceConditionBuilder := telemetrytraces.NewConditionBuilder(traceFieldMapper)
fl := flaggertest.New(t)
traceAggExprRewriter := querybuilder.NewAggExprRewriter(providerSettings, nil, traceFieldMapper, traceConditionBuilder, fl)
traceStmtBuilder := telemetrytraces.NewTraceQueryStatementBuilder(
providerSettings,
metadataStore,
traceFieldMapper,
traceConditionBuilder,
traceAggExprRewriter,
telemetryStore,
fl,
false,
100000,
)
traceStmtBuilder, err := tracesstatementbuilder.NewFactory(telemetryStore, metadataStore, fl).New(context.Background(), providerSettings, statementbuilder.Config{})
require.NoError(t, err)
return querier.New(
providerSettings,
telemetryStore,
metadataStore,
nil, // prometheus
traceStmtBuilder, // traceStmtBuilder
nil, // logStmtBuilder
nil, // auditStmtBuilder
nil, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
nil, // prometheus
traceStmtBuilder,
nil, // logStmtBuilder
nil, // auditStmtBuilder
nil, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
fl,
0,
0, // maxConcurrentQueries (0 means default)

View File

@@ -8,6 +8,10 @@ const (
// BodyFullTextSearchDefaultWarning is emitted when a full-text search or "body" searches are hit
// with New JSON Body enhancements.
BodyFullTextSearchDefaultWarning = "Full text searches default to `body.message:string`. Use `body.<key>` to search a different field inside body"
// SearchWarning is emitted on every search() call. search() scans all fields,
// so it is slow and expensive; a specific field is cheaper.
SearchWarning = "search() runs across all fields and can be slow and expensive. Prefer a specific field, e.g. `<context>.<field_key>:<type>`"
)
var (

View File

@@ -161,14 +161,16 @@ func inferDataTypesFromList(values []any) []telemetrytypes.FieldDataType {
return out
}
// NewFunctionUnsupportedError returns the error for a has/hasAny/hasAll/hasToken operator
// on a builder that doesn't support it (logs body only), or nil for other operators.
// NewFunctionUnsupportedError returns the error for a has/hasAny/hasAll/hasToken/search
// operator on a builder that doesn't support it (logs only), or nil for other operators.
func NewFunctionUnsupportedError(operator qbtypes.FilterOperator) error {
switch operator {
case qbtypes.FilterOperatorHasToken:
return errors.NewInvalidInputf(errors.CodeInvalidInput, "function `hasToken` only supports body field as first parameter").WithUrl(hasTokenFunctionDocURL)
case qbtypes.FilterOperatorHas, qbtypes.FilterOperatorHasAny, qbtypes.FilterOperatorHasAll:
return errors.NewInvalidInputf(errors.CodeInvalidInput, "function `%s` supports only body JSON search", operator.FunctionName()).WithUrl(functionBodyJSONSearchDocURL)
case qbtypes.FilterOperatorSearch:
return errors.NewInvalidInputf(errors.CodeInvalidInput, "function `search` is only supported for logs")
default:
return nil
}

View File

@@ -43,6 +43,8 @@ type filterExpressionVisitor struct {
keysWithWarnings map[string]bool
startNs uint64
endNs uint64
requiresCostGuard bool
}
type FilterExprVisitorOpts struct {
@@ -81,9 +83,10 @@ func newFilterExpressionVisitor(opts FilterExprVisitorOpts) *filterExpressionVis
}
type PreparedWhereClause struct {
WhereClause *sqlbuilder.WhereClause
Warnings []string
WarningsDocURL string
WhereClause *sqlbuilder.WhereClause
Warnings []string
WarningsDocURL string
RequiresCostGuard bool
}
func (p PreparedWhereClause) IsEmpty() bool {
@@ -165,12 +168,12 @@ func PrepareWhereClause(query string, opts FilterExprVisitorOpts) (PreparedWhere
// Return empty where clause so callers can skip the WHERE clause
if cond == "" || cond == SkipConditionLiteral {
return PreparedWhereClause{WhereClause: nil, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL}, nil
return PreparedWhereClause{WhereClause: nil, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL, RequiresCostGuard: visitor.requiresCostGuard}, nil
}
whereClause := sqlbuilder.NewWhereClause().AddWhereExpr(visitor.builder.Args, cond)
return PreparedWhereClause{WhereClause: whereClause, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL}, nil
return PreparedWhereClause{WhereClause: whereClause, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL, RequiresCostGuard: visitor.requiresCostGuard}, nil
}
// Visit dispatches to the specific visit method based on node type.
@@ -776,11 +779,79 @@ func normalizeFunctionValue(operator qbtypes.FilterOperator, functionName string
return valueParams, nil
}
// VisitSearchCall handles search('needle'). The search() function is parsed but
// not yet implemented; reject it with a clear invalid-input error.
// VisitSearchCall handles search('needle') and its scoped forms, e.g.
// search('needle', body, resource). The first arg is the case-insensitive needle; the
// rest are field-context scopes (bare or quoted). Emits one FilterOperatorSearch per
// scope (none = keyless, covering every field) and ORs them.
func (v *filterExpressionVisitor) VisitSearchCall(ctx *grammar.SearchCallContext) any {
v.errors = append(v.errors, "function `search` is not yet supported")
return ErrorConditionLiteral
// Flag scan-heavy so the statement builder attaches the cost guard.
v.requiresCostGuard = true
valueList := ctx.ValueList()
if valueList == nil {
v.errors = append(v.errors, "function `search` expects a needle, e.g. search('error')")
return ErrorConditionLiteral
}
params := valueList.AllValue()
if len(params) == 0 {
v.errors = append(v.errors, "function `search` expects a needle, e.g. search('error')")
return ErrorConditionLiteral
}
searchText, ok := searchParamText(params[0])
if !ok {
v.errors = append(v.errors, "function `search` expects a needle as its first argument, e.g. search('error')")
return ErrorConditionLiteral
}
var fieldContexts []telemetrytypes.FieldContext
if len(params) == 1 {
fieldContexts = []telemetrytypes.FieldContext{telemetrytypes.FieldContextUnspecified}
} else {
for _, p := range params[1:] {
scopeText, sok := searchParamText(p)
if !sok {
v.errors = append(v.errors, "function `search` expects each scope to be a context, e.g. search('error', body, resource)")
return ErrorConditionLiteral
}
fc, fok := telemetrytypes.FieldContextFromText(scopeText)
if !fok {
v.errors = append(v.errors, fmt.Sprintf("invalid search scope %q; expected a field context: body, attribute, resource, or log", scopeText))
return ErrorConditionLiteral
}
fieldContexts = append(fieldContexts, fc)
}
}
var conds []string
for _, fieldContext := range fieldContexts {
key := telemetrytypes.NewTelemetryFieldKey("", fieldContext, telemetrytypes.FieldDataTypeUnspecified)
scoped, cok := v.buildConditions(key, nil, qbtypes.FilterOperatorSearch, searchText)
if !cok {
return ErrorConditionLiteral
}
conds = append(conds, scoped...)
}
if len(conds) == 0 {
return SkipConditionLiteral
}
if len(conds) == 1 {
return conds[0]
}
return v.builder.Or(conds...)
}
// searchParamText returns the raw token text of a search() argument (quoted or bare),
// not the visited value — so a bare word stays literal and search(1000000) doesn't
// become "1e+06".
func searchParamText(val grammar.IValueContext) (string, bool) {
if val == nil {
return "", false
}
if val.QUOTED_TEXT() != nil {
return trimQuotes(val.QUOTED_TEXT().GetText()), true
}
return val.GetText(), true
}
// VisitFunctionParamList handles the parameter list for function calls.

View File

@@ -94,7 +94,7 @@ type Config struct {
// Alertmanager config
Alertmanager alertmanager.Config `mapstructure:"alertmanager" yaml:"alertmanager"`
// Querier config
// Querier config (the statement-builder config is embedded under it)
Querier querier.Config `mapstructure:"querier"`
// Ruler config

View File

@@ -69,6 +69,8 @@ import (
"github.com/SigNoz/signoz/pkg/types/alertmanagertypes"
"github.com/SigNoz/signoz/pkg/types/dashboardtypes"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/version"
"github.com/SigNoz/signoz/pkg/web"
"github.com/SigNoz/signoz/pkg/web/noopweb"
@@ -227,6 +229,7 @@ func NewSQLMigrationProviderFactories(
sqlmigration.NewAddTelemetryTuplesFactory(sqlstore),
sqlmigration.NewAddTagRelationRankFactory(sqlstore, sqlschema),
sqlmigration.NewMigrateDashboardsV1ToV2Factory(sqlstore, sqlschema, dashboardStore, tagModule),
sqlmigration.NewFillDashboardMeterSourceFactory(sqlstore, dashboardStore),
)
}
@@ -285,9 +288,9 @@ func NewStatsReporterProviderFactories(aggregator statsreporter.Aggregator, orgG
)
}
func NewQuerierProviderFactories(telemetryStore telemetrystore.TelemetryStore, prometheus prometheus.Prometheus, cache cache.Cache, flagger flagger.Flagger) factory.NamedMap[factory.ProviderFactory[querier.Querier, querier.Config]] {
func NewQuerierProviderFactories(telemetryStore telemetrystore.TelemetryStore, prometheus prometheus.Prometheus, metadataStore telemetrytypes.MetadataStore, traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation], logStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation], auditStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation], metricStmtBuilder qbtypes.StatementBuilder[qbtypes.MetricAggregation], meterStmtBuilder qbtypes.StatementBuilder[qbtypes.MetricAggregation], traceOperatorStmtBuilder qbtypes.TraceOperatorStatementBuilder, bucketCache querier.BucketCache, flagger flagger.Flagger) factory.NamedMap[factory.ProviderFactory[querier.Querier, querier.Config]] {
return factory.MustNewNamedMap(
signozquerier.NewFactory(telemetryStore, prometheus, cache, flagger),
signozquerier.NewFactory(telemetryStore, prometheus, metadataStore, traceStmtBuilder, logStmtBuilder, auditStmtBuilder, metricStmtBuilder, meterStmtBuilder, traceOperatorStmtBuilder, bucketCache, flagger),
)
}

View File

@@ -48,16 +48,17 @@ import (
"github.com/SigNoz/signoz/pkg/sqlmigrator"
"github.com/SigNoz/signoz/pkg/sqlschema"
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/SigNoz/signoz/pkg/statementbuilder/auditstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/logsstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/meterstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/metricsstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/tracesstatementbuilder"
"github.com/SigNoz/signoz/pkg/statsreporter"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetadata"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
pkgtokenizer "github.com/SigNoz/signoz/pkg/tokenizer"
"github.com/SigNoz/signoz/pkg/types/authtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/version"
"github.com/SigNoz/signoz/pkg/zeus"
@@ -96,6 +97,62 @@ type SigNoz struct {
MeterReporter meterreporter.Reporter
}
// newQueryStack assembles the query stack once and returns, in order: the shared
// telemetry metadata store (reused elsewhere in signoz.New), the per-signal
// statement builders (trace, log, audit, metric, meter, trace-operator), and the
// bucket cache. It is the only place that imports the concrete statement-builder
// sub-packages.
func newQueryStack(
ctx context.Context,
settings factory.ProviderSettings,
config Config,
telemetryStore telemetrystore.TelemetryStore,
cache cache.Cache,
fl flagger.Flagger,
) (
telemetrytypes.MetadataStore,
qbtypes.StatementBuilder[qbtypes.TraceAggregation],
qbtypes.StatementBuilder[qbtypes.LogAggregation],
qbtypes.StatementBuilder[qbtypes.LogAggregation],
qbtypes.StatementBuilder[qbtypes.MetricAggregation],
qbtypes.StatementBuilder[qbtypes.MetricAggregation],
qbtypes.TraceOperatorStatementBuilder,
querier.BucketCache,
error,
) {
metadataStore := telemetrymetadata.NewTelemetryMetaStore(settings, telemetryStore, fl)
cfg := config.Querier.Config
traceStmtBuilder, err := tracesstatementbuilder.NewFactory(telemetryStore, metadataStore, fl).New(ctx, settings, cfg)
if err != nil {
return nil, nil, nil, nil, nil, nil, nil, nil, err
}
traceOperatorStmtBuilder, err := tracesstatementbuilder.NewOperatorFactory(telemetryStore, metadataStore, fl).New(ctx, settings, cfg)
if err != nil {
return nil, nil, nil, nil, nil, nil, nil, nil, err
}
logStmtBuilder, err := logsstatementbuilder.NewFactory(telemetryStore, metadataStore, fl).New(ctx, settings, cfg)
if err != nil {
return nil, nil, nil, nil, nil, nil, nil, nil, err
}
auditStmtBuilder, err := auditstatementbuilder.NewFactory(metadataStore, fl).New(ctx, settings, cfg)
if err != nil {
return nil, nil, nil, nil, nil, nil, nil, nil, err
}
metricStmtBuilder, err := metricsstatementbuilder.NewFactory(metadataStore, fl).New(ctx, settings, cfg)
if err != nil {
return nil, nil, nil, nil, nil, nil, nil, nil, err
}
meterStmtBuilder, err := meterstatementbuilder.NewFactory(metadataStore, fl).New(ctx, settings, cfg)
if err != nil {
return nil, nil, nil, nil, nil, nil, nil, nil, err
}
bucketCache := querier.NewBucketCache(settings, cache, config.Querier.CacheTTL, config.Querier.FluxInterval)
return metadataStore, traceStmtBuilder, logStmtBuilder, auditStmtBuilder, metricStmtBuilder, meterStmtBuilder, traceOperatorStmtBuilder, bucketCache, nil
}
func New(
ctx context.Context,
config Config,
@@ -254,12 +311,19 @@ func New(
return nil, err
}
// Assemble the query stack (metadata store, statement builders, bucket cache) once,
// and reuse the single metadata store everywhere downstream.
telemetryMetadataStore, traceStmtBuilder, logStmtBuilder, auditStmtBuilder, metricStmtBuilder, meterStmtBuilder, traceOperatorStmtBuilder, bucketCache, err := newQueryStack(ctx, providerSettings, config, telemetrystore, cache, flagger)
if err != nil {
return nil, err
}
// Initialize querier from the available querier provider factories
querier, err := factory.NewProviderFromNamedMap(
ctx,
providerSettings,
config.Querier,
NewQuerierProviderFactories(telemetrystore, prometheus, cache, flagger),
NewQuerierProviderFactories(telemetrystore, prometheus, telemetryMetadataStore, traceStmtBuilder, logStmtBuilder, auditStmtBuilder, metricStmtBuilder, meterStmtBuilder, traceOperatorStmtBuilder, bucketCache, flagger),
config.Querier.Provider(),
)
if err != nil {
@@ -427,35 +491,6 @@ func New(
return nil, err
}
// Initialize telemetry metadata store
// TODO: consolidate other telemetrymetadata.NewTelemetryMetaStore initializations to reuse this instance instead.
telemetryMetadataStore := telemetrymetadata.NewTelemetryMetaStore(
providerSettings,
telemetrystore,
telemetrytraces.DBName,
telemetrytraces.TagAttributesV2TableName,
telemetrytraces.SpanAttributesKeysTblName,
telemetrytraces.SpanIndexV3TableName,
telemetrymetrics.DBName,
telemetrymetrics.AttributesMetadataTableName,
telemetrymeter.DBName,
telemetrymeter.SamplesAgg1dTableName,
telemetrylogs.DBName,
telemetrylogs.LogsV2TableName,
telemetrylogs.TagAttributesV2TableName,
telemetrylogs.LogAttributeKeysTblName,
telemetrylogs.LogResourceKeysTblName,
telemetryaudit.DBName,
telemetryaudit.AuditLogsTableName,
telemetryaudit.TagAttributesTableName,
telemetryaudit.LogAttributeKeysTblName,
telemetryaudit.LogResourceKeysTblName,
telemetrymetadata.DBName,
telemetrymetadata.AttributesMetadataTableName,
telemetrymetadata.ColumnEvolutionMetadataTableName,
flagger,
)
global, err := factory.NewProviderFromNamedMap(
ctx,
providerSettings,

View File

@@ -0,0 +1,159 @@
package sqlmigration
import (
"context"
"log/slog"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/SigNoz/signoz/pkg/types"
"github.com/SigNoz/signoz/pkg/types/dashboardtypes"
qb "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/uptrace/bun"
"github.com/uptrace/bun/migrate"
)
// meterMetricNames are the SigNoz meter metrics; a metrics query aggregating one is a
// meter query.
var meterMetricNames = map[string]bool{
"signoz.meter.log.count": true,
"signoz.meter.log.size": true,
"signoz.meter.span.count": true,
"signoz.meter.span.size": true,
"signoz.meter.metric.datapoint.count": true,
"signoz.meter.platform.active": true,
}
type fillDashboardMeterSource struct {
sqlstore sqlstore.SQLStore
dashboardStore dashboardtypes.Store
settings factory.ProviderSettings
}
func NewFillDashboardMeterSourceFactory(sqlstore sqlstore.SQLStore, dashboardStore dashboardtypes.Store) factory.ProviderFactory[SQLMigration, Config] {
return factory.NewProviderFactory(
factory.MustNewName("fill_dashboard_meter_source"),
func(ctx context.Context, ps factory.ProviderSettings, c Config) (SQLMigration, error) {
return &fillDashboardMeterSource{sqlstore: sqlstore, dashboardStore: dashboardStore, settings: ps}, nil
},
)
}
func (migration *fillDashboardMeterSource) Register(migrations *migrate.Migrations) error {
return migrations.Register(migration.Up, migration.Down)
}
// Up resets the meter source that 103 dropped when converting v1 meter queries to v2:
// every metrics query aggregating a meter metric is set back to source "meter". One
// transaction; v1 (unparseable-as-v2) dashboards are skipped.
func (migration *fillDashboardMeterSource) Up(ctx context.Context, _ *bun.DB) error {
return migration.sqlstore.RunInTxCtx(ctx, nil, func(ctx context.Context) error {
var orgIDs []string
if err := migration.sqlstore.BunDBCtx(ctx).NewSelect().Model((*types.Organization)(nil)).Column("id").Scan(ctx, &orgIDs); err != nil {
return err
}
for _, id := range orgIDs {
orgID, err := valuer.NewUUID(id)
if err != nil {
return err
}
if err := migration.fillOrg(ctx, orgID); err != nil {
return err
}
}
return nil
})
}
// fillOrg fills the meter source on every v2 dashboard in the org that needs it. Runs
// inside the caller's transaction.
func (migration *fillDashboardMeterSource) fillOrg(ctx context.Context, orgID valuer.UUID) error {
// List, not ListV2: ListV2 paginates and excludes system dashboards; a migration needs every row.
storables, err := migration.dashboardStore.List(ctx, orgID)
if err != nil {
return err
}
logger := migration.settings.Logger
var stillInV1, skippedNoMeter, wouldMigrateButErrored, migrated int
for _, storable := range storables {
// ToDashboardV2 rejects non-v2 data, so v1 dashboards fall through untouched.
dashboard, err := storable.ToDashboardV2(nil)
if err != nil {
stillInV1++
continue
}
if !fillMeterSource(&dashboard.Spec) {
skippedNoMeter++
continue
}
restorable, err := dashboard.ToStorableDashboard()
if err != nil {
wouldMigrateButErrored++
logger.WarnContext(ctx, "failed to re-serialize dashboard after filling meter source", slog.String("org_id", orgID.String()), slog.String("dashboard_id", storable.ID.String()), slog.Any("error", err))
continue
}
if err := migration.dashboardStore.Update(ctx, orgID, restorable); err != nil {
return err
}
migrated++
}
logger.InfoContext(ctx, "filled meter source on v2 dashboards",
slog.String("org_id", orgID.String()),
slog.Int("total", len(storables)),
slog.Int("still_in_v1", stillInV1),
slog.Int("skipped_no_meter", skippedNoMeter),
slog.Int("would_migrate_but_errored", wouldMigrateButErrored),
slog.Int("migrated", migrated),
)
return nil
}
// fillMeterSource sets Source=meter on every metrics builder query in the spec whose
// aggregation targets a meter metric, reporting whether anything changed.
func fillMeterSource(spec *dashboardtypes.DashboardSpec) bool {
changed := false
for _, panel := range spec.Panels {
if panel == nil {
continue
}
for _, query := range panel.Spec.Queries {
switch plugin := query.Spec.Plugin.Spec.(type) {
case *qb.CompositeQuery:
for i := range plugin.Queries {
changed = setMeterSource(&plugin.Queries[i].Spec) || changed
}
case *dashboardtypes.BuilderQuerySpec:
changed = setMeterSource(&plugin.Spec) || changed
}
}
}
return changed
}
// setMeterSource sets source "meter" on a metrics query aggregating a meter metric. The
// type assertion doubles as the metrics-signal check; the already-meter guard keeps it
// idempotent. Written back through the pointer, as an interface value isn't addressable.
func setMeterSource(spec *any) bool {
query, ok := (*spec).(qb.QueryBuilderQuery[qb.MetricAggregation])
if !ok || query.Source == telemetrytypes.SourceMeter {
return false
}
for _, aggregation := range query.Aggregations {
if meterMetricNames[aggregation.MetricName] {
query.Source = telemetrytypes.SourceMeter
*spec = query
return true
}
}
return false
}
func (migration *fillDashboardMeterSource) Down(context.Context, *bun.DB) error {
return nil
}

View File

@@ -1,4 +1,4 @@
package telemetryaudit
package auditstatementbuilder
import (
"context"
@@ -10,7 +10,9 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryresourcefilter"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/resourcefilter"
"github.com/SigNoz/signoz/pkg/telemetryschema/audittelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -29,6 +31,25 @@ type auditQueryStatementBuilder struct {
var _ qbtypes.StatementBuilder[qbtypes.LogAggregation] = (*auditQueryStatementBuilder)(nil)
// NewFactory returns a provider factory for the audit statement builder. Its New
// internalizes the FieldMapper, ConditionBuilder, and AggExprRewriter.
func NewFactory(
metadataStore telemetrytypes.MetadataStore,
fl flagger.Flagger,
) factory.ProviderFactory[qbtypes.StatementBuilder[qbtypes.LogAggregation], statementbuilder.Config] {
return factory.NewProviderFactory(
factory.MustNewName("audit"),
func(_ context.Context, settings factory.ProviderSettings, _ statementbuilder.Config) (qbtypes.StatementBuilder[qbtypes.LogAggregation], error) {
fm := audittelemetryschema.NewFieldMapper()
cb := audittelemetryschema.NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, audittelemetryschema.DefaultFullTextColumn, fm, cb, fl)
return NewAuditQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, audittelemetryschema.DefaultFullTextColumn, fl,
), nil
},
)
}
func NewAuditQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
@@ -38,12 +59,12 @@ func NewAuditQueryStatementBuilder(
fullTextColumn *telemetrytypes.TelemetryFieldKey,
flagger flagger.Flagger,
) *auditQueryStatementBuilder {
auditSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryaudit")
auditSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/audittelemetryschema")
resourceFilterStmtBuilder := telemetryresourcefilter.New[qbtypes.LogAggregation](
resourceFilterStmtBuilder := resourcefilter.New[qbtypes.LogAggregation](
settings,
DBName,
LogsResourceTableName,
audittelemetryschema.DBName,
audittelemetryschema.LogsResourceTableName,
telemetrytypes.SignalLogs,
telemetrytypes.SourceAudit,
metadataStore,
@@ -191,8 +212,8 @@ func (b *auditQueryStatementBuilder) adjustKeys(ctx context.Context, keys map[st
}
func (b *auditQueryStatementBuilder) adjustKey(key *telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) []string {
if _, ok := IntrinsicFields[key.Name]; ok {
intrinsicField := IntrinsicFields[key.Name]
if _, ok := audittelemetryschema.IntrinsicFields[key.Name]; ok {
intrinsicField := audittelemetryschema.IntrinsicFields[key.Name]
return querybuilder.AdjustKey(key, keys, &intrinsicField)
}
return querybuilder.AdjustKey(key, keys, nil)
@@ -219,26 +240,26 @@ func (b *auditQueryStatementBuilder) buildListQuery(
cteArgs = append(cteArgs, args)
}
sb.Select(TimestampColumn)
sb.SelectMore(IDColumn)
sb.Select(audittelemetryschema.TimestampColumn)
sb.SelectMore(audittelemetryschema.IDColumn)
if len(query.SelectFields) == 0 {
sb.SelectMore(TraceIDColumn)
sb.SelectMore(SpanIDColumn)
sb.SelectMore(TraceFlagsColumn)
sb.SelectMore(SeverityTextColumn)
sb.SelectMore(SeverityNumberColumn)
sb.SelectMore(ScopeNameColumn)
sb.SelectMore(ScopeVersionColumn)
sb.SelectMore(BodyColumn)
sb.SelectMore(EventNameColumn)
sb.SelectMore(AttributesStringColumn)
sb.SelectMore(AttributesNumberColumn)
sb.SelectMore(AttributesBoolColumn)
sb.SelectMore(ResourceColumn)
sb.SelectMore(ScopeStringColumn)
sb.SelectMore(audittelemetryschema.TraceIDColumn)
sb.SelectMore(audittelemetryschema.SpanIDColumn)
sb.SelectMore(audittelemetryschema.TraceFlagsColumn)
sb.SelectMore(audittelemetryschema.SeverityTextColumn)
sb.SelectMore(audittelemetryschema.SeverityNumberColumn)
sb.SelectMore(audittelemetryschema.ScopeNameColumn)
sb.SelectMore(audittelemetryschema.ScopeVersionColumn)
sb.SelectMore(audittelemetryschema.BodyColumn)
sb.SelectMore(audittelemetryschema.EventNameColumn)
sb.SelectMore(audittelemetryschema.AttributesStringColumn)
sb.SelectMore(audittelemetryschema.AttributesNumberColumn)
sb.SelectMore(audittelemetryschema.AttributesBoolColumn)
sb.SelectMore(audittelemetryschema.ResourceColumn)
sb.SelectMore(audittelemetryschema.ScopeStringColumn)
} else {
for index := range query.SelectFields {
if query.SelectFields[index].Name == TimestampColumn || query.SelectFields[index].Name == IDColumn {
if query.SelectFields[index].Name == audittelemetryschema.TimestampColumn || query.SelectFields[index].Name == audittelemetryschema.IDColumn {
continue
}
@@ -250,7 +271,7 @@ func (b *auditQueryStatementBuilder) buildListQuery(
}
}
sb.From(fmt.Sprintf("%s.%s", DBName, AuditLogsTableName))
sb.From(fmt.Sprintf("%s.%s", audittelemetryschema.DBName, audittelemetryschema.AuditLogsTableName))
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables)
if err != nil {
@@ -340,7 +361,7 @@ func (b *auditQueryStatementBuilder) buildTimeSeriesQuery(
sb.SelectMore(fmt.Sprintf("%s AS __result_%d", rewritten, i))
}
sb.From(fmt.Sprintf("%s.%s", DBName, AuditLogsTableName))
sb.From(fmt.Sprintf("%s.%s", audittelemetryschema.DBName, audittelemetryschema.AuditLogsTableName))
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables)
if err != nil {
@@ -478,7 +499,7 @@ func (b *auditQueryStatementBuilder) buildScalarQuery(
}
}
sb.From(fmt.Sprintf("%s.%s", DBName, AuditLogsTableName))
sb.From(fmt.Sprintf("%s.%s", audittelemetryschema.DBName, audittelemetryschema.AuditLogsTableName))
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables)
if err != nil {

View File

@@ -1,4 +1,4 @@
package telemetryaudit
package auditstatementbuilder
import (
"context"
@@ -8,6 +8,7 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/audittelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
@@ -54,8 +55,8 @@ func newTestAuditStatementBuilder(t *testing.T) *auditQueryStatementBuilder {
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = auditFieldKeyMap()
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := audittelemetryschema.NewFieldMapper()
cb := audittelemetryschema.NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
return NewAuditQueryStatementBuilder(
@@ -64,7 +65,7 @@ func newTestAuditStatementBuilder(t *testing.T) *auditQueryStatementBuilder {
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
audittelemetryschema.DefaultFullTextColumn,
fl,
)
}

View File

@@ -0,0 +1,45 @@
package statementbuilder
import (
"github.com/SigNoz/signoz/pkg/errors"
)
// SkipResourceFingerprint configures when the resource fingerprint subquery is skipped in favor of main-table filtering.
type SkipResourceFingerprint struct {
Enabled bool `yaml:"enabled" mapstructure:"enabled"`
// If count of fingerprint is above threshold, skip the fingerprint subquery and filter on main table instead.
Threshold uint64 `yaml:"threshold" mapstructure:"threshold"`
}
// Config represents the configuration for the statement builders.
type Config struct {
// SkipResourceFingerprint configures when the resource fingerprint subquery is skipped in favor of main-table filtering.
SkipResourceFingerprint SkipResourceFingerprint `yaml:"skip_resource_fingerprint" mapstructure:"skip_resource_fingerprint"`
// SearchMaxScanRows caps the rows a search() query may scan; the log statement builder
// attaches it to the CostGuard and the querier enforces it via EXPLAIN ESTIMATE (0 disables).
SearchMaxScanRows int64 `yaml:"search_max_scan_rows" mapstructure:"search_max_scan_rows"`
}
// NewConfig returns the statement-builder config with its default values. It is
// embedded (squashed) into querier.Config rather than registered as a standalone
// config section, so its defaults are seeded from querier's newConfig.
func NewConfig() Config {
return Config{
SkipResourceFingerprint: SkipResourceFingerprint{
Enabled: false,
Threshold: 100000,
},
SearchMaxScanRows: 0,
}
}
// Validate validates the configuration.
func (c Config) Validate() error {
if c.SkipResourceFingerprint.Enabled && c.SkipResourceFingerprint.Threshold == 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "skip_resource_fingerprint.threshold must be > 0 when enabled")
}
if c.SearchMaxScanRows < 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "search_max_scan_rows must not be negative, got %v", c.SearchMaxScanRows)
}
return nil
}

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logsstatementbuilder
import (
"context"
@@ -8,6 +8,8 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
@@ -43,14 +45,15 @@ func TestStatementBuilderGroupByUnknownKey(t *testing.T) {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.useJSONBody)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore, fm, cb, aggExprRewriter,
DefaultFullTextColumn, fl, nil, false, 100000,
logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logsstatementbuilder
import (
"context"
@@ -11,6 +11,8 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
@@ -1225,7 +1227,7 @@ func TestJSONStmtBuilder_OrderBy(t *testing.T) {
func TestResourceAggrAndGroupBy_WithJSONEnabled(t *testing.T) {
statementBuilder, metadataStore := buildJSONTestStatementBuilder(t, false)
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keysMap := buildCompleteFieldKeyMap(releaseTime)
keysMap := logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
for _, keys := range keysMap {
for _, key := range keys {
metadataStore.SetKey(key)
@@ -1287,7 +1289,7 @@ func TestResourceAggrAndGroupBy_WithJSONEnabled(t *testing.T) {
func buildTestTelemetryMetadataStore(t *testing.T, addIndexes bool) *telemetrytypestest.MockMetadataStore {
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.SetStaticFields(IntrinsicFields)
mockMetadataStore.SetStaticFields(logstelemetryschema.IntrinsicFields)
types, _ := telemetrytypes.TestJSONTypeSet()
for path, fieldDataTypes := range types {
for _, fdt := range fieldDataTypes {
@@ -1307,15 +1309,15 @@ func buildTestTelemetryMetadataStore(t *testing.T, addIndexes bool) *telemetryty
Name: path,
FieldContext: telemetrytypes.FieldContextBody,
FieldDataType: fdt,
BaseColumn: LogsV2BodyV2Column,
IndexExpression: schemamigrator.JSONSubColumnIndexExpr(LogsV2BodyV2Column, path, jsonType.StringValue()),
BaseColumn: logstelemetryschema.LogsV2BodyV2Column,
IndexExpression: schemamigrator.JSONSubColumnIndexExpr(logstelemetryschema.LogsV2BodyV2Column, path, jsonType.StringValue()),
})
}
}
err := key.SetJSONAccessPlan(telemetrytypes.JSONColumnMetadata{
BaseColumn: LogsV2BodyV2Column,
PromotedColumn: LogsV2BodyPromotedColumn,
BaseColumn: logstelemetryschema.LogsV2BodyV2Column,
PromotedColumn: logstelemetryschema.LogsV2BodyPromotedColumn,
}, types)
require.NoError(t, err)
@@ -1331,8 +1333,8 @@ func buildJSONTestStatementBuilder(t *testing.T, addIndexes bool) (*logQueryStat
mockMetadataStore := buildTestTelemetryMetadataStore(t, addIndexes)
fl := flaggertest.WithUseJSONBody(t, true)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -1342,11 +1344,10 @@ func buildJSONTestStatementBuilder(t *testing.T, addIndexes bool) (*logQueryStat
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
false,
100000,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
return statementBuilder, mockMetadataStore

View File

@@ -0,0 +1,49 @@
package logsstatementbuilder
import (
"context"
"testing"
"time"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/require"
)
// TestSearchCostGuard covers the search() path end-to-end through Build: the
// statement carries a CostGuard with its scan budget and the advisory warning.
func TestSearchCostGuard(t *testing.T) {
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
ctx := context.Background()
start := uint64(releaseTime.Add(-5 * time.Minute).UnixMilli())
end := uint64(releaseTime.UnixMilli())
fl := flaggertest.WithBooleanFlags(t, map[string]bool{})
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
store := telemetrytypestest.NewMockMetadataStore()
store.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
sb := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
store, fm, cb, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{SearchMaxScanRows: 100000, SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
Filter: &qbtypes.Filter{Expression: "search('error')"},
Limit: 1,
}
stmt, err := sb.Build(ctx, valuer.UUID{}, start, end, qbtypes.RequestTypeRaw, query, nil)
require.NoError(t, err)
require.NotNil(t, stmt.CostGuard)
require.Contains(t, stmt.Warnings, querybuilder.SearchWarning)
}

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logsstatementbuilder
import (
"context"
@@ -10,7 +10,9 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryresourcefilter"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/resourcefilter"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -19,21 +21,54 @@ import (
"github.com/huandu/go-sqlbuilder"
)
const bodySearchDefaultWarning = "body searches default to `body.message:string`. Use `body.<key>` to search a different field inside body"
func bodyAliasExpression(bodyJSONEnabled bool) string {
if !bodyJSONEnabled {
return logstelemetryschema.LogsV2BodyColumn
}
return fmt.Sprintf("%s as body", logstelemetryschema.LogsV2BodyV2Column)
}
type logQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
resourceFilterResolver *telemetryresourcefilter.ResourceFingerprintResolver[qbtypes.LogAggregation]
resourceFilterResolver *resourcefilter.ResourceFingerprintResolver[qbtypes.LogAggregation]
aggExprRewriter qbtypes.AggExprRewriter
fl flagger.Flagger
skipResourceFingerprintEnabled bool
fullTextColumn *telemetrytypes.TelemetryFieldKey
fullTextColumn *telemetrytypes.TelemetryFieldKey
searchMaxScanRows int64
}
var _ qbtypes.StatementBuilder[qbtypes.LogAggregation] = (*logQueryStatementBuilder)(nil)
// NewFactory returns a provider factory for the logs statement builder. Its New
// internalizes the FieldMapper, ConditionBuilder, and AggExprRewriter, and reads
// SkipResourceFingerprint and SearchMaxScanRows from the config.
func NewFactory(
telemetryStore telemetrystore.TelemetryStore,
metadataStore telemetrytypes.MetadataStore,
fl flagger.Flagger,
) factory.ProviderFactory[qbtypes.StatementBuilder[qbtypes.LogAggregation], statementbuilder.Config] {
return factory.NewProviderFactory(
factory.MustNewName("logs"),
func(_ context.Context, settings factory.ProviderSettings, cfg statementbuilder.Config) (qbtypes.StatementBuilder[qbtypes.LogAggregation], error) {
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, logstelemetryschema.DefaultFullTextColumn, fm, cb, fl)
return NewLogQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, logstelemetryschema.DefaultFullTextColumn,
fl, telemetryStore, cfg,
), nil
},
)
}
func NewLogQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
@@ -43,25 +78,24 @@ func NewLogQueryStatementBuilder(
fullTextColumn *telemetrytypes.TelemetryFieldKey,
fl flagger.Flagger,
telemetryStore telemetrystore.TelemetryStore,
skipResourceFingerprintEnable bool,
skipResourceFingerprintThreshold uint64,
cfg statementbuilder.Config,
) *logQueryStatementBuilder {
logsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrylogs")
logsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema")
resourceFilterResolver := telemetryresourcefilter.NewResolver[qbtypes.LogAggregation](
resourceFilterResolver := resourcefilter.NewResolver[qbtypes.LogAggregation](
settings,
DBName,
LogsResourceV2TableName,
logstelemetryschema.DBName,
logstelemetryschema.LogsResourceV2TableName,
telemetrytypes.SignalLogs,
telemetrytypes.SourceUnspecified,
metadataStore,
fullTextColumn,
fl,
telemetryStore,
skipResourceFingerprintThreshold,
cfg.SkipResourceFingerprint.Threshold,
)
return &logQueryStatementBuilder{
b := &logQueryStatementBuilder{
logger: logsSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
@@ -69,9 +103,11 @@ func NewLogQueryStatementBuilder(
resourceFilterResolver: resourceFilterResolver,
aggExprRewriter: aggExprRewriter,
fl: fl,
skipResourceFingerprintEnabled: skipResourceFingerprintEnable,
skipResourceFingerprintEnabled: cfg.SkipResourceFingerprint.Enabled,
searchMaxScanRows: cfg.SearchMaxScanRows,
fullTextColumn: fullTextColumn,
}
return b
}
// Build builds a SQL query for logs based on the given parameters.
@@ -117,9 +153,23 @@ func (b *logQueryStatementBuilder) Build(
}
stmt.Warnings = append(stmt.Warnings, warnings...)
// The search flag is gated in the condition builder; here the advisory rides on
// the statement so the querier can enforce the scan budget from the CostGuard.
if stmt.CostGuard != nil && stmt.CostGuard.Warning != "" {
stmt.Warnings = append(stmt.Warnings, stmt.CostGuard.Warning)
}
return stmt, nil
}
// costGuardFor builds the cost guard for a scan-heavy (search()) statement,
// pairing the advisory with the configured scan budget. Returns nil otherwise.
func (b *logQueryStatementBuilder) costGuardFor(required bool) *qbtypes.CostGuard {
if !required {
return nil
}
return &qbtypes.CostGuard{Warning: querybuilder.SearchWarning, MaxScanRows: b.searchMaxScanRows}
}
func getKeySelectors(query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation], bodyJSONEnabled bool) ([]*telemetrytypes.FieldKeySelector, []string) {
var keySelectors []*telemetrytypes.FieldKeySelector
var warnings []string
@@ -174,7 +224,7 @@ func getKeySelectors(query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation], bo
// TODO(Piyush): Setup better for coming FTS support.
if bodyJSONEnabled {
for _, sel := range keySelectors {
if sel.Name == LogsV2BodyColumn {
if sel.Name == logstelemetryschema.LogsV2BodyColumn {
warnings = append(warnings, bodySearchDefaultWarning)
break
}
@@ -250,8 +300,8 @@ func (b *logQueryStatementBuilder) adjustKeys(ctx context.Context, keys map[stri
func (b *logQueryStatementBuilder) adjustKey(key *telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) []string {
// First check if it matches with any intrinsic fields
var intrinsicOrCalculatedField telemetrytypes.TelemetryFieldKey
if _, ok := IntrinsicFields[key.Name]; ok {
intrinsicOrCalculatedField = IntrinsicFields[key.Name]
if _, ok := logstelemetryschema.IntrinsicFields[key.Name]; ok {
intrinsicOrCalculatedField = logstelemetryschema.IntrinsicFields[key.Name]
return querybuilder.AdjustKey(key, keys, &intrinsicOrCalculatedField)
}
@@ -284,28 +334,28 @@ func (b *logQueryStatementBuilder) buildListQuery(
}
// Select timestamp and id by default
sb.Select(LogsV2TimestampColumn)
sb.SelectMore(LogsV2IDColumn)
sb.Select(logstelemetryschema.LogsV2TimestampColumn)
sb.SelectMore(logstelemetryschema.LogsV2IDColumn)
if len(query.SelectFields) == 0 {
// Select all default columns
sb.SelectMore(LogsV2TraceIDColumn)
sb.SelectMore(LogsV2SpanIDColumn)
sb.SelectMore(LogsV2TraceFlagsColumn)
sb.SelectMore(LogsV2SeverityTextColumn)
sb.SelectMore(LogsV2SeverityNumberColumn)
sb.SelectMore(LogsV2ScopeNameColumn)
sb.SelectMore(LogsV2ScopeVersionColumn)
sb.SelectMore(logstelemetryschema.LogsV2TraceIDColumn)
sb.SelectMore(logstelemetryschema.LogsV2SpanIDColumn)
sb.SelectMore(logstelemetryschema.LogsV2TraceFlagsColumn)
sb.SelectMore(logstelemetryschema.LogsV2SeverityTextColumn)
sb.SelectMore(logstelemetryschema.LogsV2SeverityNumberColumn)
sb.SelectMore(logstelemetryschema.LogsV2ScopeNameColumn)
sb.SelectMore(logstelemetryschema.LogsV2ScopeVersionColumn)
sb.SelectMore(bodyAliasExpression(b.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))))
sb.SelectMore(LogsV2AttributesStringColumn)
sb.SelectMore(LogsV2AttributesNumberColumn)
sb.SelectMore(LogsV2AttributesBoolColumn)
sb.SelectMore(LogsV2ResourcesStringColumn)
sb.SelectMore(LogsV2ScopeStringColumn)
sb.SelectMore(logstelemetryschema.LogsV2AttributesStringColumn)
sb.SelectMore(logstelemetryschema.LogsV2AttributesNumberColumn)
sb.SelectMore(logstelemetryschema.LogsV2AttributesBoolColumn)
sb.SelectMore(logstelemetryschema.LogsV2ResourcesStringColumn)
sb.SelectMore(logstelemetryschema.LogsV2ScopeStringColumn)
} else {
// Select specified columns
for index := range query.SelectFields {
if query.SelectFields[index].Name == LogsV2TimestampColumn || query.SelectFields[index].Name == LogsV2IDColumn {
if query.SelectFields[index].Name == logstelemetryschema.LogsV2TimestampColumn || query.SelectFields[index].Name == logstelemetryschema.LogsV2IDColumn {
continue
}
@@ -318,7 +368,7 @@ func (b *logQueryStatementBuilder) buildListQuery(
}
}
sb.From(fmt.Sprintf("%s.%s", DBName, LogsV2TableName))
sb.From(fmt.Sprintf("%s.%s", logstelemetryschema.DBName, logstelemetryschema.LogsV2TableName))
// Add filter conditions
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables, skipResourceFilter)
@@ -357,6 +407,7 @@ func (b *logQueryStatementBuilder) buildListQuery(
Args: finalArgs,
Warnings: preparedWhereClause.Warnings,
WarningsDocURL: preparedWhereClause.WarningsDocURL,
CostGuard: b.costGuardFor(preparedWhereClause.RequiresCostGuard),
}
return stmt, nil
@@ -424,7 +475,7 @@ func (b *logQueryStatementBuilder) buildTimeSeriesQuery(
}
// Add FROM clause
sb.From(fmt.Sprintf("%s.%s", DBName, LogsV2TableName))
sb.From(fmt.Sprintf("%s.%s", logstelemetryschema.DBName, logstelemetryschema.LogsV2TableName))
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables, skipResourceFilter)
@@ -523,6 +574,7 @@ func (b *logQueryStatementBuilder) buildTimeSeriesQuery(
Args: finalArgs,
Warnings: preparedWhereClause.Warnings,
WarningsDocURL: preparedWhereClause.WarningsDocURL,
CostGuard: b.costGuardFor(preparedWhereClause.RequiresCostGuard),
}
return stmt, nil
@@ -592,7 +644,7 @@ func (b *logQueryStatementBuilder) buildScalarQuery(
}
}
sb.From(fmt.Sprintf("%s.%s", DBName, LogsV2TableName))
sb.From(fmt.Sprintf("%s.%s", logstelemetryschema.DBName, logstelemetryschema.LogsV2TableName))
// Add filter conditions
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables, skipResourceFilter)
@@ -650,6 +702,7 @@ func (b *logQueryStatementBuilder) buildScalarQuery(
Args: finalArgs,
Warnings: preparedWhereClause.Warnings,
WarningsDocURL: preparedWhereClause.WarningsDocURL,
CostGuard: b.costGuardFor(preparedWhereClause.RequiresCostGuard),
}
return stmt, nil

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logsstatementbuilder
import (
"context"
@@ -11,6 +11,8 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -212,12 +214,12 @@ func TestStatementBuilderTimeSeries(t *testing.T) {
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keysMap := buildCompleteFieldKeyMap(releaseTime)
keysMap := logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = keysMap
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -227,11 +229,10 @@ func TestStatementBuilderTimeSeries(t *testing.T) {
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
false,
100000,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
for _, c := range cases {
@@ -355,12 +356,12 @@ func TestStatementBuilderListQuery(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fm := NewFieldMapper(fl)
fm := logstelemetryschema.NewFieldMapper(fl)
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
cb := NewConditionBuilder(fm, fl)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -370,11 +371,10 @@ func TestStatementBuilderListQuery(t *testing.T) {
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
false,
100000,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
for _, c := range cases {
@@ -507,11 +507,11 @@ func TestStatementBuilderListQueryResourceTests(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fm := NewFieldMapper(fl)
fm := logstelemetryschema.NewFieldMapper(fl)
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
cb := NewConditionBuilder(fm, fl)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -521,11 +521,10 @@ func TestStatementBuilderListQueryResourceTests(t *testing.T) {
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
false,
100000,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
for _, c := range cases {
@@ -585,11 +584,11 @@ func TestStatementBuilderTimeSeriesBodyGroupBy(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fm := NewFieldMapper(fl)
fm := logstelemetryschema.NewFieldMapper(fl)
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
cb := NewConditionBuilder(fm, fl)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -599,11 +598,10 @@ func TestStatementBuilderTimeSeriesBodyGroupBy(t *testing.T) {
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
false,
100000,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
for _, c := range cases {
@@ -684,9 +682,9 @@ func TestStatementBuilderListQueryServiceCollision(t *testing.T) {
ctx := context.Background()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMapCollision()
cb := NewConditionBuilder(fm, fl)
fm := logstelemetryschema.NewFieldMapper(fl)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMapCollision()
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -696,11 +694,10 @@ func TestStatementBuilderListQueryServiceCollision(t *testing.T) {
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
false,
100000,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
for _, c := range cases {
@@ -740,8 +737,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: IntrinsicFields["severity_text"],
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: logstelemetryschema.IntrinsicFields["severity_text"],
},
{
name: "intrinsic field with other key match - no override",
@@ -777,7 +774,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextBody,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "severity_number",
FieldContext: telemetrytypes.FieldContextBody,
@@ -791,8 +788,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["service.name"][0],
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["service.name"][0],
},
{
name: "single matching key with incorrect context specified - no override",
@@ -801,7 +798,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "service.name",
FieldContext: telemetrytypes.FieldContextAttribute,
@@ -815,8 +812,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["service.name"][0],
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["service.name"][0],
},
{
name: "multiple matching keys - all materialized",
@@ -825,7 +822,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "multi.mat.key",
FieldDataType: telemetrytypes.FieldDataTypeString,
@@ -839,7 +836,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "mixed.materialization.key",
FieldDataType: telemetrytypes.FieldDataTypeString,
@@ -853,8 +850,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["mixed.materialization.key"][0],
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["mixed.materialization.key"][0],
},
{
name: "no matching keys - unknown field",
@@ -863,7 +860,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "unknown.field",
FieldContext: telemetrytypes.FieldContextUnspecified,
@@ -878,7 +875,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "unknown.field",
FieldContext: telemetrytypes.FieldContextAttribute,
@@ -893,8 +890,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["mat.key"][0],
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["mat.key"][0],
},
{
name: "non-materialized field",
@@ -903,16 +900,16 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["user.id"][0],
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["user.id"][0],
},
}
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
fm := logstelemetryschema.NewFieldMapper(fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMapCollision()
cb := NewConditionBuilder(fm, fl)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMapCollision()
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -922,11 +919,10 @@ func TestAdjustKey(t *testing.T) {
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
false,
100000,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
for _, c := range cases {
@@ -1054,11 +1050,11 @@ func TestStmtBuilderBodyField(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.enableUseJSONBody)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
// build the key map
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
for _, field := range IntrinsicFields {
for _, field := range logstelemetryschema.IntrinsicFields {
f := field
mockMetadataStore.KeysMap[field.Name] = append(mockMetadataStore.KeysMap[field.Name], &f)
}
@@ -1069,11 +1065,10 @@ func TestStmtBuilderBodyField(t *testing.T) {
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
false,
100000,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
q, err := statementBuilder.Build(context.Background(), valuer.UUID{}, 1747947419000, 1747983448000, c.requestType, c.query, nil)
@@ -1155,11 +1150,11 @@ func TestStmtBuilderBodyFullTextSearch(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.enableUseJSONBody)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
// build the key map
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
for _, field := range IntrinsicFields {
for _, field := range logstelemetryschema.IntrinsicFields {
f := field
mockMetadataStore.KeysMap[field.Name] = append(mockMetadataStore.KeysMap[field.Name], &f)
}
@@ -1170,11 +1165,10 @@ func TestStmtBuilderBodyFullTextSearch(t *testing.T) {
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
false,
100000,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
q, err := statementBuilder.Build(context.Background(), valuer.UUID{}, 1747947419000, 1747983448000, c.requestType, c.query, nil)
@@ -1273,14 +1267,14 @@ func newSkipResourceFingerprintLogsBuilder(
t.Helper()
fl := flaggertest.New(t)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC))
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC))
aggExprRewriter := querybuilder.NewAggExprRewriter(
instrumentationtest.New().ToProviderSettings(),
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fm,
cb,
fl,
@@ -1292,10 +1286,9 @@ func newSkipResourceFingerprintLogsBuilder(
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
logstelemetryschema.DefaultFullTextColumn,
fl,
telemetryStore,
skipEnable,
threshold,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: skipEnable, Threshold: threshold}},
)
}

View File

@@ -1,4 +1,4 @@
package telemetrymeter
package meterstatementbuilder
import (
"context"
@@ -7,8 +7,12 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/metricsstatementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -21,19 +25,40 @@ type meterQueryStatementBuilder struct {
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
metricsStatementBuilder *telemetrymetrics.MetricQueryStatementBuilder
metricsStatementBuilder *metricsstatementbuilder.StatementBuilder
}
var _ qbtypes.StatementBuilder[qbtypes.MetricAggregation] = (*meterQueryStatementBuilder)(nil)
// NewFactory returns a provider factory for the meter statement builder. Its New
// reuses the metrics FieldMapper/ConditionBuilder and delegates the final SELECT
// to a metrics statement builder built via the metrics factory.
func NewFactory(
metadataStore telemetrytypes.MetadataStore,
fl flagger.Flagger,
) factory.ProviderFactory[qbtypes.StatementBuilder[qbtypes.MetricAggregation], statementbuilder.Config] {
return factory.NewProviderFactory(
factory.MustNewName("meter"),
func(ctx context.Context, settings factory.ProviderSettings, cfg statementbuilder.Config) (qbtypes.StatementBuilder[qbtypes.MetricAggregation], error) {
metricsStatementBuilder, err := metricsstatementbuilder.NewFactory(metadataStore, fl).New(ctx, settings, cfg)
if err != nil {
return nil, err
}
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
return NewMeterQueryStatementBuilder(settings, metadataStore, fm, cb, metricsStatementBuilder), nil
},
)
}
func NewMeterQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
metricsStatementBuilder *telemetrymetrics.MetricQueryStatementBuilder,
metricsStatementBuilder *metricsstatementbuilder.StatementBuilder,
) *meterQueryStatementBuilder {
metricsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrymeter")
metricsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema")
return &meterQueryStatementBuilder{
logger: metricsSettings.Logger(),
@@ -53,7 +78,7 @@ func (b *meterQueryStatementBuilder) Build(
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
keySelectors := telemetrymetrics.GetKeySelectors(query)
keySelectors := metricsstatementbuilder.GetKeySelectors(query)
keys, _, err := b.metadataStore.GetKeysMulti(ctx, orgID, keySelectors)
if err != nil {
return nil, err
@@ -133,8 +158,8 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDeltaFastPath(
sb.SelectMore(col)
}
tbl := WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
tbl := metertelemetryschema.WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := metertelemetryschema.AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
if err != nil {
return "", nil, err
}
@@ -143,7 +168,7 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDeltaFastPath(
}
sb.SelectMore(fmt.Sprintf("%s AS value", aggCol))
sb.From(fmt.Sprintf("%s.%s AS points", DBName, tbl))
sb.From(fmt.Sprintf("%s.%s AS points", metertelemetryschema.DBName, tbl))
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
sb.GTE("unix_milli", start),
@@ -222,8 +247,8 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDelta(
sb.SelectMore(col)
}
tbl := WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality,
tbl := metertelemetryschema.WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := metertelemetryschema.AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality,
query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
if err != nil {
return "", nil, err
@@ -234,7 +259,7 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDelta(
sb.SelectMore(fmt.Sprintf("%s AS per_series_value", aggCol))
sb.From(fmt.Sprintf("%s.%s AS points", DBName, tbl))
sb.From(fmt.Sprintf("%s.%s AS points", metertelemetryschema.DBName, tbl))
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
sb.GTE("unix_milli", start),
@@ -300,14 +325,14 @@ func (b *meterQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
baseSb.SelectMore(col)
}
tbl := WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
tbl := metertelemetryschema.WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := metertelemetryschema.AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
if err != nil {
return "", nil, err
}
baseSb.SelectMore(fmt.Sprintf("%s AS per_series_value", aggCol))
baseSb.From(fmt.Sprintf("%s.%s AS points", DBName, tbl))
baseSb.From(fmt.Sprintf("%s.%s AS points", metertelemetryschema.DBName, tbl))
baseSb.Where(
baseSb.In("metric_name", query.Aggregations[0].MetricName),
baseSb.GTE("unix_milli", start),
@@ -350,7 +375,7 @@ func (b *meterQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
for _, g := range query.GroupBy {
wrapped.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
wrapped.SelectMore(fmt.Sprintf("%s AS per_series_value", telemetrymetrics.RateTmpl))
wrapped.SelectMore(fmt.Sprintf("%s AS per_series_value", metricsstatementbuilder.RateTmpl))
wrapped.From(fmt.Sprintf("(%s) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)", innerQuery))
q, args := wrapped.BuildWithFlavor(sqlbuilder.ClickHouse, innerArgs...)
return fmt.Sprintf("__temporal_aggregation_cte AS (%s)", q), args, nil
@@ -361,7 +386,7 @@ func (b *meterQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
for _, g := range query.GroupBy {
wrapped.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
wrapped.SelectMore(fmt.Sprintf("%s AS per_series_value", telemetrymetrics.IncreaseTmpl))
wrapped.SelectMore(fmt.Sprintf("%s AS per_series_value", metricsstatementbuilder.IncreaseTmpl))
wrapped.From(fmt.Sprintf("(%s) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)", innerQuery))
q, args := wrapped.BuildWithFlavor(sqlbuilder.ClickHouse, innerArgs...)
return fmt.Sprintf("__temporal_aggregation_cte AS (%s)", q), args, nil

View File

@@ -1,4 +1,4 @@
package telemetrymeter
package meterstatementbuilder
import (
"context"
@@ -7,7 +7,8 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/statementbuilder/metricsstatementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -158,8 +159,8 @@ func TestStatementBuilder(t *testing.T) {
},
}
fm := telemetrymetrics.NewFieldMapper()
cb := telemetrymetrics.NewConditionBuilder(fm)
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
if err != nil {
@@ -169,7 +170,7 @@ func TestStatementBuilder(t *testing.T) {
flagger := flaggertest.New(t)
metricStmtBuilder := telemetrymetrics.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, fm, cb, flagger)
metricStmtBuilder := metricsstatementbuilder.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, fm, cb, flagger)
statementBuilder := NewMeterQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),

View File

@@ -1,4 +1,4 @@
package telemetrymetrics
package metricsstatementbuilder
import (
"context"
@@ -7,6 +7,7 @@ import (
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -118,8 +119,8 @@ func TestReducedStatementBuilder(t *testing.T) {
},
}
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
fl, err := flagger.New(context.Background(), instrumentationtest.New().ToProviderSettings(), flagger.Config{}, flagger.MustNewRegistry())
require.NoError(t, err)
sb := NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), telemetrytypestest.NewMockMetadataStore(), fm, cb, fl)

View File

@@ -1,4 +1,4 @@
package telemetrymetrics
package metricsstatementbuilder
import (
"context"
@@ -9,6 +9,8 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -29,7 +31,7 @@ const (
OthersMultiTemporality = `IF(LOWER(temporality) LIKE LOWER('delta'), %s, %s) AS per_series_value`
)
type MetricQueryStatementBuilder struct {
type StatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
@@ -37,7 +39,24 @@ type MetricQueryStatementBuilder struct {
flagger flagger.Flagger
}
var _ qbtypes.StatementBuilder[qbtypes.MetricAggregation] = (*MetricQueryStatementBuilder)(nil)
var _ qbtypes.StatementBuilder[qbtypes.MetricAggregation] = (*StatementBuilder)(nil)
// NewFactory returns a provider factory for the metrics statement builder. Its
// New internalizes the FieldMapper and ConditionBuilder and yields the concrete
// *StatementBuilder so the meter builder can reuse it.
func NewFactory(
metadataStore telemetrytypes.MetadataStore,
fl flagger.Flagger,
) factory.ProviderFactory[*StatementBuilder, statementbuilder.Config] {
return factory.NewProviderFactory(
factory.MustNewName("metrics"),
func(_ context.Context, settings factory.ProviderSettings, _ statementbuilder.Config) (*StatementBuilder, error) {
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
return NewMetricQueryStatementBuilder(settings, metadataStore, fm, cb, fl), nil
},
)
}
func NewMetricQueryStatementBuilder(
settings factory.ProviderSettings,
@@ -45,9 +64,9 @@ func NewMetricQueryStatementBuilder(
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
flagger flagger.Flagger,
) *MetricQueryStatementBuilder {
metricsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrymetrics")
return &MetricQueryStatementBuilder{
) *StatementBuilder {
metricsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema")
return &StatementBuilder{
logger: metricsSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
@@ -89,7 +108,7 @@ func GetKeySelectors(query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation])
return keySelectors
}
func (b *MetricQueryStatementBuilder) Build(
func (b *StatementBuilder) Build(
ctx context.Context,
orgID valuer.UUID,
start uint64,
@@ -126,7 +145,7 @@ func (b *MetricQueryStatementBuilder) Build(
return b.buildPipelineStatement(ctx, orgID, start, end, query, keys, variables)
}
func (b *MetricQueryStatementBuilder) buildPipelineStatement(
func (b *StatementBuilder) buildPipelineStatement(
ctx context.Context,
orgID valuer.UUID,
start, end uint64,
@@ -188,11 +207,11 @@ func (b *MetricQueryStatementBuilder) buildPipelineStatement(
// shaped exactly like samples_v4 / time_series_v4, so once the table names are
// chosen the rest of the pipeline is unchanged.
useBuffer := agg.Reduced &&
end-start < oneDayInMilliseconds &&
start >= uint64(time.Now().UnixMilli())-oneDayInMilliseconds
end-start < metricstelemetryschema.OneDayInMilliseconds &&
start >= uint64(time.Now().UnixMilli())-metricstelemetryschema.OneDayInMilliseconds
samplesTable, _ := WhichSamplesTableToUse(start, end, agg.Type, agg.TimeAggregation, useBuffer, agg.TableHints)
tsStart, tsEnd, _, tsTable := WhichTSTableToUse(start, end, useBuffer, agg.TableHints)
samplesTable, _ := metricstelemetryschema.WhichSamplesTableToUse(start, end, agg.Type, agg.TimeAggregation, useBuffer, agg.TableHints)
tsStart, tsEnd, _, tsTable := metricstelemetryschema.WhichTSTableToUse(start, end, useBuffer, agg.TableHints)
var timeSeriesCTE string
var timeSeriesCTEArgs []any
@@ -232,7 +251,7 @@ func (b *MetricQueryStatementBuilder) buildPipelineStatement(
var tsCTE string
var tsArgs []any
// time series rows are written on hour boundaries
tsStart := start - (start % oneHourInMilliseconds)
tsStart := start - (start % metricstelemetryschema.OneHourInMilliseconds)
if tsCTE, tsArgs, err = b.buildReducedTimeSeriesCTE(ctx, orgID, tsStart, end, query, keys, variables); err != nil {
return nil, err
}
@@ -282,7 +301,7 @@ func unionStatements(main, reduced *qbtypes.Statement, query qbtypes.QueryBuilde
return &qbtypes.Statement{Query: q, Args: args, Warnings: warnings}, nil
}
func (b *MetricQueryStatementBuilder) buildReducedTimeSeriesCTE(
func (b *StatementBuilder) buildReducedTimeSeriesCTE(
ctx context.Context,
orgID valuer.UUID,
start, end uint64,
@@ -312,7 +331,7 @@ func (b *MetricQueryStatementBuilder) buildReducedTimeSeriesCTE(
}
}
sb.From(fmt.Sprintf("%s.%s", DBName, TimeseriesV4ReducedLocalTableName))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedLocalTableName))
sb.Select("fingerprint")
for _, g := range query.GroupBy {
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
@@ -343,7 +362,7 @@ func (b *MetricQueryStatementBuilder) buildReducedTimeSeriesCTE(
// spatial_aggregation_cte in one level with no per-series grouping, so shards
// send one state per (step, group) instead of per (series, step, group).
// FINAL still dedups recomputed 60s buckets at scan time.
func (b *MetricQueryStatementBuilder) buildReducedSpatialAggFastPath(
func (b *StatementBuilder) buildReducedSpatialAggFastPath(
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
timeSeriesCTE string,
@@ -352,7 +371,7 @@ func (b *MetricQueryStatementBuilder) buildReducedSpatialAggFastPath(
agg := query.Aggregations[0]
stepSec := int64(query.StepInterval.Seconds())
value, _, ok := ReducedValueColumn(agg.Type, agg.SpaceAggregation)
value, _, ok := metricstelemetryschema.ReducedValueColumn(agg.Type, agg.SpaceAggregation)
if !ok {
return "", nil, false
}
@@ -362,8 +381,8 @@ func (b *MetricQueryStatementBuilder) buildReducedSpatialAggFastPath(
for _, g := range query.GroupBy {
sb.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
sb.SelectMore(fmt.Sprintf("%s AS value", ReducedTimeAggregationColumn(agg.TimeAggregation, stepSec, value)))
sb.From(fmt.Sprintf("%s.%s AS points FINAL", DBName, WhichReducedSamplesTableToUse(agg.Type)))
sb.SelectMore(fmt.Sprintf("%s AS value", metricstelemetryschema.ReducedTimeAggregationColumn(agg.TimeAggregation, stepSec, value)))
sb.From(fmt.Sprintf("%s.%s AS points FINAL", metricstelemetryschema.DBName, metricstelemetryschema.WhichReducedSamplesTableToUse(agg.Type)))
sb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.reduced_fingerprint = filtered_time_series.fingerprint")
sb.Where(
sb.In("metric_name", agg.MetricName),
@@ -377,7 +396,7 @@ func (b *MetricQueryStatementBuilder) buildReducedSpatialAggFastPath(
return fmt.Sprintf("__spatial_aggregation_cte AS (%s)", q), args, true
}
func (b *MetricQueryStatementBuilder) buildReducedTemporalAggregationCTE(
func (b *StatementBuilder) buildReducedTemporalAggregationCTE(
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
timeSeriesCTE string,
@@ -386,7 +405,7 @@ func (b *MetricQueryStatementBuilder) buildReducedTemporalAggregationCTE(
agg := query.Aggregations[0]
stepSec := int64(query.StepInterval.Seconds())
value, weight, ok := ReducedValueColumn(agg.Type, agg.SpaceAggregation)
value, weight, ok := metricstelemetryschema.ReducedValueColumn(agg.Type, agg.SpaceAggregation)
if !ok {
return "", nil, false
}
@@ -399,13 +418,13 @@ func (b *MetricQueryStatementBuilder) buildReducedTemporalAggregationCTE(
for _, g := range query.GroupBy {
sb.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
sb.SelectMore(fmt.Sprintf("%s AS per_series_value", ReducedTimeAggregationColumn(agg.TimeAggregation, stepSec, value)))
sb.SelectMore(fmt.Sprintf("%s AS per_series_value", metricstelemetryschema.ReducedTimeAggregationColumn(agg.TimeAggregation, stepSec, value)))
if weight != "" {
// count_series is a series count, not additive over time, so the avg
// denominator is reduced with avg
sb.SelectMore(fmt.Sprintf("avg(%s) AS per_series_weight", weight))
}
sb.From(fmt.Sprintf("%s.%s AS points FINAL", DBName, WhichReducedSamplesTableToUse(agg.Type)))
sb.From(fmt.Sprintf("%s.%s AS points FINAL", metricstelemetryschema.DBName, metricstelemetryschema.WhichReducedSamplesTableToUse(agg.Type)))
sb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.reduced_fingerprint = filtered_time_series.fingerprint")
sb.Where(
sb.In("metric_name", agg.MetricName),
@@ -419,7 +438,7 @@ func (b *MetricQueryStatementBuilder) buildReducedTemporalAggregationCTE(
return fmt.Sprintf("__temporal_aggregation_cte AS (%s)", q), args, true
}
func (b *MetricQueryStatementBuilder) buildReducedSpatialAggregationCTE(
func (b *StatementBuilder) buildReducedSpatialAggregationCTE(
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
) (string, []any) {
spatial := "sum(per_series_value)"
@@ -446,7 +465,7 @@ func (b *MetricQueryStatementBuilder) buildReducedSpatialAggregationCTE(
return fmt.Sprintf("__spatial_aggregation_cte AS (%s)", q), args
}
func (b *MetricQueryStatementBuilder) buildTemporalAggDeltaFastPath(
func (b *StatementBuilder) buildTemporalAggDeltaFastPath(
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
samplesTable string,
@@ -465,7 +484,7 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggDeltaFastPath(
sb.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
aggCol, err := AggregationColumnForSamplesTable(
aggCol, err := metricstelemetryschema.AggregationColumnForSamplesTable(
samplesTable, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation,
)
if err != nil {
@@ -483,7 +502,7 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggDeltaFastPath(
sb.SelectMore(fmt.Sprintf("%s AS value", aggCol))
sb.From(fmt.Sprintf("%s.%s AS points", DBName, samplesTable))
sb.From(fmt.Sprintf("%s.%s AS points", metricstelemetryschema.DBName, samplesTable))
sb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.fingerprint = filtered_time_series.fingerprint")
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
@@ -497,7 +516,7 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggDeltaFastPath(
return fmt.Sprintf("__spatial_aggregation_cte AS (%s)", q), args, nil
}
func (b *MetricQueryStatementBuilder) buildTimeSeriesCTE(
func (b *StatementBuilder) buildTimeSeriesCTE(
ctx context.Context,
orgID valuer.UUID,
start, end uint64,
@@ -529,7 +548,7 @@ func (b *MetricQueryStatementBuilder) buildTimeSeriesCTE(
}
}
sb.From(fmt.Sprintf("%s.%s", DBName, tsTable))
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, tsTable))
sb.Select("fingerprint")
for _, g := range query.GroupBy {
@@ -552,7 +571,7 @@ func (b *MetricQueryStatementBuilder) buildTimeSeriesCTE(
// the buffer holds both raw rows and the reduced catalog rows; the raw read
// only wants the original series
if tsTable == TimeseriesV4BufferLocalTableName {
if tsTable == metricstelemetryschema.TimeseriesV4BufferLocalTableName {
sb.Where(sb.EQ("is_reduced", false))
}
@@ -567,7 +586,7 @@ func (b *MetricQueryStatementBuilder) buildTimeSeriesCTE(
return fmt.Sprintf("(%s) AS filtered_time_series", q), args, nil
}
func (b *MetricQueryStatementBuilder) buildTemporalAggregationCTE(
func (b *StatementBuilder) buildTemporalAggregationCTE(
ctx context.Context,
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
@@ -584,7 +603,7 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggregationCTE(
return b.buildTemporalAggForMultipleTemporalities(ctx, start, end, query, samplesTable, timeSeriesCTE, timeSeriesCTEArgs)
}
func (b *MetricQueryStatementBuilder) buildTemporalAggDelta(
func (b *StatementBuilder) buildTemporalAggDelta(
_ context.Context,
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
@@ -605,7 +624,7 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggDelta(
sb.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
aggCol, err := AggregationColumnForSamplesTable(samplesTable, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation)
aggCol, err := metricstelemetryschema.AggregationColumnForSamplesTable(samplesTable, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation)
if err != nil {
return "", nil, err
}
@@ -616,7 +635,7 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggDelta(
sb.SelectMore(fmt.Sprintf("%s AS per_series_value", aggCol))
sb.From(fmt.Sprintf("%s.%s AS points", DBName, samplesTable))
sb.From(fmt.Sprintf("%s.%s AS points", metricstelemetryschema.DBName, samplesTable))
sb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.fingerprint = filtered_time_series.fingerprint")
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
@@ -631,7 +650,7 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggDelta(
return fmt.Sprintf("__temporal_aggregation_cte AS (%s)", q), args, nil
}
func (b *MetricQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
func (b *StatementBuilder) buildTemporalAggCumulativeOrUnspecified(
_ context.Context,
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
@@ -651,13 +670,13 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
baseSb.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
aggCol, err := AggregationColumnForSamplesTable(samplesTable, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation)
aggCol, err := metricstelemetryschema.AggregationColumnForSamplesTable(samplesTable, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation)
if err != nil {
return "", nil, err
}
baseSb.SelectMore(fmt.Sprintf("%s AS per_series_value", aggCol))
baseSb.From(fmt.Sprintf("%s.%s AS points", DBName, samplesTable))
baseSb.From(fmt.Sprintf("%s.%s AS points", metricstelemetryschema.DBName, samplesTable))
baseSb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.fingerprint = filtered_time_series.fingerprint")
baseSb.Where(
baseSb.In("metric_name", query.Aggregations[0].MetricName),
@@ -697,7 +716,7 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
}
}
func (b *MetricQueryStatementBuilder) buildTemporalAggForMultipleTemporalities(
func (b *StatementBuilder) buildTemporalAggForMultipleTemporalities(
_ context.Context,
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
@@ -716,11 +735,11 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggForMultipleTemporalities(
sb.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
aggForDeltaTemporality, err := AggregationColumnForSamplesTable(samplesTable, metrictypes.Delta, query.Aggregations[0].TimeAggregation)
aggForDeltaTemporality, err := metricstelemetryschema.AggregationColumnForSamplesTable(samplesTable, metrictypes.Delta, query.Aggregations[0].TimeAggregation)
if err != nil {
return "", nil, err
}
aggForCumulativeTemporality, err := AggregationColumnForSamplesTable(samplesTable, metrictypes.Cumulative, query.Aggregations[0].TimeAggregation)
aggForCumulativeTemporality, err := metricstelemetryschema.AggregationColumnForSamplesTable(samplesTable, metrictypes.Cumulative, query.Aggregations[0].TimeAggregation)
if err != nil {
return "", nil, err
}
@@ -748,7 +767,7 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggForMultipleTemporalities(
sb.SelectMore(expr)
}
sb.From(fmt.Sprintf("%s.%s AS points", DBName, samplesTable))
sb.From(fmt.Sprintf("%s.%s AS points", metricstelemetryschema.DBName, samplesTable))
sb.JoinWithOption(sqlbuilder.InnerJoin, timeSeriesCTE, "points.fingerprint = filtered_time_series.fingerprint")
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
@@ -762,7 +781,7 @@ func (b *MetricQueryStatementBuilder) buildTemporalAggForMultipleTemporalities(
return fmt.Sprintf("__temporal_aggregation_cte AS (%s)", queryWithWindowAndOrder), args, nil
}
func (b *MetricQueryStatementBuilder) buildSpatialAggregationCTE(
func (b *StatementBuilder) buildSpatialAggregationCTE(
_ context.Context,
_ uint64,
_ uint64,
@@ -788,7 +807,7 @@ func (b *MetricQueryStatementBuilder) buildSpatialAggregationCTE(
return fmt.Sprintf("__spatial_aggregation_cte AS (%s)", q), args
}
func (b *MetricQueryStatementBuilder) BuildFinalSelect(
func (b *StatementBuilder) BuildFinalSelect(
cteFragments []string,
cteArgs [][]any,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
@@ -833,7 +852,7 @@ func (b *MetricQueryStatementBuilder) BuildFinalSelect(
sb.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
aggQuery, err := AggregationQueryForHistogramCountWithParams(query.Aggregations[0].ComparisonSpaceAggregationParam)
aggQuery, err := metricstelemetryschema.AggregationQueryForHistogramCountWithParams(query.Aggregations[0].ComparisonSpaceAggregationParam)
if err != nil {
return nil, err
}

View File

@@ -1,4 +1,4 @@
package telemetrymetrics
package metricsstatementbuilder
import (
"context"
@@ -7,6 +7,7 @@ import (
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -252,8 +253,8 @@ func TestStatementBuilder(t *testing.T) {
},
}
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
if err != nil {

View File

@@ -1,4 +1,4 @@
package telemetryresourcefilter
package resourcefilter
import (
"context"

View File

@@ -1,4 +1,4 @@
package telemetryresourcefilter
package resourcefilter
import (
"context"

View File

@@ -1,4 +1,4 @@
package telemetryresourcefilter
package resourcefilter
import (
"context"

View File

@@ -1,4 +1,4 @@
package telemetryresourcefilter
package resourcefilter
import (
"context"

View File

@@ -1,4 +1,4 @@
package telemetryresourcefilter
package resourcefilter
import (
"context"
@@ -45,7 +45,7 @@ func New[T any](
fullTextColumn *telemetrytypes.TelemetryFieldKey,
fl flagger.Flagger,
) *resourceFilterStatementBuilder[T] {
set := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryresourcefilter")
set := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/statementbuilder/resourcefilter")
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
return &resourceFilterStatementBuilder[T]{

View File

@@ -1,4 +1,4 @@
package telemetryresourcefilter
package resourcefilter
import (
"context"

View File

@@ -1,4 +1,4 @@
package telemetrytraces
package tracesstatementbuilder
import (
"context"
@@ -10,7 +10,9 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryresourcefilter"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/resourcefilter"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -27,13 +29,35 @@ type traceQueryStatementBuilder struct {
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
resourceFilterResolver *telemetryresourcefilter.ResourceFingerprintResolver[qbtypes.TraceAggregation]
resourceFilterResolver *resourcefilter.ResourceFingerprintResolver[qbtypes.TraceAggregation]
aggExprRewriter qbtypes.AggExprRewriter
skipResourceFingerprintEnabled bool
}
var _ qbtypes.StatementBuilder[qbtypes.TraceAggregation] = (*traceQueryStatementBuilder)(nil)
// NewFactory returns a provider factory for the trace query statement builder. Its
// New internalizes the FieldMapper, ConditionBuilder, and AggExprRewriter, and reads
// SkipResourceFingerprint from the config.
func NewFactory(
telemetryStore telemetrystore.TelemetryStore,
metadataStore telemetrytypes.MetadataStore,
fl flagger.Flagger,
) factory.ProviderFactory[qbtypes.StatementBuilder[qbtypes.TraceAggregation], statementbuilder.Config] {
return factory.NewProviderFactory(
factory.MustNewName("traces"),
func(_ context.Context, settings factory.ProviderSettings, cfg statementbuilder.Config) (qbtypes.StatementBuilder[qbtypes.TraceAggregation], error) {
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, fm, cb, fl)
return NewTraceQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, telemetryStore, fl,
cfg.SkipResourceFingerprint.Enabled, cfg.SkipResourceFingerprint.Threshold,
), nil
},
)
}
func NewTraceQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
@@ -45,12 +69,12 @@ func NewTraceQueryStatementBuilder(
skipResourceFingerprintEnable bool,
skipResourceFingerprintThreshold uint64,
) *traceQueryStatementBuilder {
tracesSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrytraces")
tracesSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema")
resourceFilterResolver := telemetryresourcefilter.NewResolver[qbtypes.TraceAggregation](
resourceFilterResolver := resourcefilter.NewResolver[qbtypes.TraceAggregation](
settings,
DBName,
TracesResourceV3TableName,
tracestelemetryschema.DBName,
tracestelemetryschema.TracesResourceV3TableName,
telemetrytypes.SignalTraces,
telemetrytypes.SourceUnspecified,
metadataStore,
@@ -183,10 +207,10 @@ func getKeySelectors(query qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation])
// 3. We prepend (not append) so the intrinsic/calculated entry wins ordering
// in the multi_if SQL expression.
func mergeDeprecatedTraceKeys(keys map[string][]*telemetrytypes.TelemetryFieldKey) {
for fieldKeyName, fieldKey := range IntrinsicFieldsDeprecated {
for fieldKeyName, fieldKey := range tracestelemetryschema.IntrinsicFieldsDeprecated {
keys[fieldKeyName] = append([]*telemetrytypes.TelemetryFieldKey{&fieldKey}, keys[fieldKeyName]...)
}
for fieldKeyName, fieldKey := range CalculatedFieldsDeprecated {
for fieldKeyName, fieldKey := range tracestelemetryschema.CalculatedFieldsDeprecated {
keys[fieldKeyName] = append([]*telemetrytypes.TelemetryFieldKey{&fieldKey}, keys[fieldKeyName]...)
}
}
@@ -246,18 +270,18 @@ func adjustTraceKey(key *telemetrytypes.TelemetryFieldKey, keys map[string][]*te
*/
var isIntrinsicOrCalculatedField bool
var intrinsicOrCalculatedField telemetrytypes.TelemetryFieldKey
if _, ok := IntrinsicFields[key.Name]; ok {
if _, ok := tracestelemetryschema.IntrinsicFields[key.Name]; ok {
isIntrinsicOrCalculatedField = true
intrinsicOrCalculatedField = IntrinsicFields[key.Name]
} else if _, ok := CalculatedFields[key.Name]; ok {
intrinsicOrCalculatedField = tracestelemetryschema.IntrinsicFields[key.Name]
} else if _, ok := tracestelemetryschema.CalculatedFields[key.Name]; ok {
isIntrinsicOrCalculatedField = true
intrinsicOrCalculatedField = CalculatedFields[key.Name]
} else if _, ok := IntrinsicFieldsDeprecated[key.Name]; ok {
intrinsicOrCalculatedField = tracestelemetryschema.CalculatedFields[key.Name]
} else if _, ok := tracestelemetryschema.IntrinsicFieldsDeprecated[key.Name]; ok {
isIntrinsicOrCalculatedField = true
intrinsicOrCalculatedField = IntrinsicFieldsDeprecated[key.Name]
} else if _, ok := CalculatedFieldsDeprecated[key.Name]; ok {
intrinsicOrCalculatedField = tracestelemetryschema.IntrinsicFieldsDeprecated[key.Name]
} else if _, ok := tracestelemetryschema.CalculatedFieldsDeprecated[key.Name]; ok {
isIntrinsicOrCalculatedField = true
intrinsicOrCalculatedField = CalculatedFieldsDeprecated[key.Name]
intrinsicOrCalculatedField = tracestelemetryschema.CalculatedFieldsDeprecated[key.Name]
}
if isIntrinsicOrCalculatedField {
@@ -304,13 +328,13 @@ func (b *traceQueryStatementBuilder) buildListQuery(
}
if isSelectFieldsEmpty {
for _, col := range ContextualSpanColumns {
for _, col := range tracestelemetryschema.ContextualSpanColumns {
sb.SelectMore(col)
}
}
// From table
sb.From(fmt.Sprintf("%s.%s", DBName, SpanIndexV3TableName))
sb.From(fmt.Sprintf("%s.%s", tracestelemetryschema.DBName, tracestelemetryschema.SpanIndexV3TableName))
// Add filter conditions
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables, skipResourceFilter)
@@ -368,7 +392,7 @@ func (b *traceQueryStatementBuilder) buildTraceQuery(
distSB := sqlbuilder.NewSelectBuilder()
distSB.Select("trace_id")
distSB.From(fmt.Sprintf("%s.%s", DBName, SpanIndexV3TableName))
distSB.From(fmt.Sprintf("%s.%s", tracestelemetryschema.DBName, tracestelemetryschema.SpanIndexV3TableName))
var (
cteFragments []string
@@ -398,7 +422,7 @@ func (b *traceQueryStatementBuilder) buildTraceQuery(
// Build the inner subquery for root spans
innerSB := sqlbuilder.NewSelectBuilder()
innerSB.Select("trace_id", "duration_nano", sqlbuilder.Escape("resource_string_service$$name as `service.name`"), "name")
innerSB.From(fmt.Sprintf("%s.%s", DBName, SpanIndexV3TableName))
innerSB.From(fmt.Sprintf("%s.%s", tracestelemetryschema.DBName, tracestelemetryschema.SpanIndexV3TableName))
innerSB.Where("parent_span_id = ''")
// this only helps when there is a filter
@@ -528,7 +552,7 @@ func (b *traceQueryStatementBuilder) buildTimeSeriesQuery(
sb.SelectMore(fmt.Sprintf("%s AS __result_%d", rewritten, i))
}
sb.From(fmt.Sprintf("%s.%s", DBName, SpanIndexV3TableName))
sb.From(fmt.Sprintf("%s.%s", tracestelemetryschema.DBName, tracestelemetryschema.SpanIndexV3TableName))
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables, skipResourceFilter)
if err != nil {
return nil, err
@@ -690,7 +714,7 @@ func (b *traceQueryStatementBuilder) buildScalarQuery(
}
// From table
sb.From(fmt.Sprintf("%s.%s", DBName, SpanIndexV3TableName))
sb.From(fmt.Sprintf("%s.%s", tracestelemetryschema.DBName, tracestelemetryschema.SpanIndexV3TableName))
// Add filter conditions
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables, skipResourceFilter)
@@ -873,21 +897,21 @@ func (b *traceQueryStatementBuilder) maybeAttachResourceFilter(
// It must be called before adjustKeys so that normalization runs over the full set.
func (b *traceQueryStatementBuilder) expandRawSelectFields(query qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation]) qbtypes.QueryBuilderQuery[qbtypes.TraceAggregation] {
if len(query.SelectFields) == 0 {
selectFields := make([]telemetrytypes.TelemetryFieldKey, 0, len(IntrinsicSpanFields)+len(CalculatedSpanFields))
selectFields = append(selectFields, IntrinsicSpanFields...)
selectFields = append(selectFields, CalculatedSpanFields...)
selectFields := make([]telemetrytypes.TelemetryFieldKey, 0, len(tracestelemetryschema.IntrinsicSpanFields)+len(tracestelemetryschema.CalculatedSpanFields))
selectFields = append(selectFields, tracestelemetryschema.IntrinsicSpanFields...)
selectFields = append(selectFields, tracestelemetryschema.CalculatedSpanFields...)
query.SelectFields = selectFields
return query
}
selectFields := []telemetrytypes.TelemetryFieldKey{
{Name: SpanTimestampColumn, FieldContext: telemetrytypes.FieldContextSpan},
{Name: SpanTraceIDColumn, FieldContext: telemetrytypes.FieldContextSpan},
{Name: SpanSpanIDColumn, FieldContext: telemetrytypes.FieldContextSpan},
{Name: tracestelemetryschema.SpanTimestampColumn, FieldContext: telemetrytypes.FieldContextSpan},
{Name: tracestelemetryschema.SpanTraceIDColumn, FieldContext: telemetrytypes.FieldContextSpan},
{Name: tracestelemetryschema.SpanSpanIDColumn, FieldContext: telemetrytypes.FieldContextSpan},
}
for _, field := range query.SelectFields {
// TODO(tvats): If a user specifies attribute.timestamp in the select fields, this loop will basically ignore it, as we already added a field by default. This can be fixed once we close https://github.com/SigNoz/engineering-pod/issues/3693
if field.Name == SpanTimestampColumn || field.Name == SpanTraceIDColumn || field.Name == SpanSpanIDColumn {
if field.Name == tracestelemetryschema.SpanTimestampColumn || field.Name == tracestelemetryschema.SpanTraceIDColumn || field.Name == tracestelemetryschema.SpanSpanIDColumn {
continue
}
selectFields = append(selectFields, field)

View File

@@ -1,4 +1,4 @@
package telemetrytraces
package tracesstatementbuilder
import (
"context"
@@ -11,6 +11,7 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -375,10 +376,10 @@ func TestStatementBuilder(t *testing.T) {
}
fl := flaggertest.New(t)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
@@ -676,10 +677,10 @@ func TestStatementBuilderListQuery(t *testing.T) {
}
fl := flaggertest.New(t)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
@@ -740,7 +741,7 @@ func TestStatementBuilderListQueryWithCorruptData(t *testing.T) {
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextResource,
FieldDataType: telemetrytypes.FieldDataTypeString,
Evolutions: mockEvolutionData(time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)),
Evolutions: tracestelemetryschema.MockEvolutionData(time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)),
},
},
},
@@ -774,7 +775,7 @@ func TestStatementBuilderListQueryWithCorruptData(t *testing.T) {
Signal: telemetrytypes.SignalTraces,
FieldContext: telemetrytypes.FieldContextResource,
FieldDataType: telemetrytypes.FieldDataTypeString,
Evolutions: mockEvolutionData(time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)),
Evolutions: tracestelemetryschema.MockEvolutionData(time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)),
},
},
},
@@ -803,12 +804,12 @@ func TestStatementBuilderListQueryWithCorruptData(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.New(t)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = c.keysMap
if mockMetadataStore.KeysMap == nil {
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
}
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -878,10 +879,10 @@ func TestStatementBuilderGroupByResourceEvolution(t *testing.T) {
}
fl := flaggertest.New(t)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
@@ -1045,10 +1046,10 @@ func TestStatementBuilderTraceQuery(t *testing.T) {
}
fl := flaggertest.New(t)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
@@ -1096,8 +1097,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: IntrinsicFields["trace_id"],
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: tracestelemetryschema.IntrinsicFields["trace_id"],
},
{
name: "intrinsic field with metadata match with incorrect context - override",
@@ -1106,7 +1107,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextBody, // incorrect context
FieldDataType: telemetrytypes.FieldDataTypeInt64,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "duration_nano",
FieldContext: telemetrytypes.FieldContextSpan, // should be corrected
@@ -1120,7 +1121,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextSpan, // correct context
FieldDataType: telemetrytypes.FieldDataTypeInt64,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "duration_nano",
FieldContext: telemetrytypes.FieldContextSpan, // should be corrected
@@ -1134,8 +1135,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["service.name"][0],
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["service.name"][0],
},
{
name: "single matching key with context specified - override",
@@ -1144,8 +1145,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["cart.items_count"][0],
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["cart.items_count"][0],
},
{
name: "multiple matching keys - all materialized",
@@ -1182,7 +1183,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "mixed.materialization.key",
FieldDataType: telemetrytypes.FieldDataTypeString,
@@ -1196,7 +1197,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "mixed.materialization.key",
FieldContext: telemetrytypes.FieldContextAttribute,
@@ -1211,7 +1212,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "unknown.field",
Materialized: false,
@@ -1224,7 +1225,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "service.name",
FieldContext: telemetrytypes.FieldContextAttribute,
@@ -1239,7 +1240,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "cart.items_count",
FieldContext: telemetrytypes.FieldContextAttribute,
@@ -1254,7 +1255,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "user.id",
FieldContext: telemetrytypes.FieldContextAttribute,
@@ -1308,7 +1309,7 @@ func TestAdjustKeys(t *testing.T) {
},
},
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedSelectFields: []telemetrytypes.TelemetryFieldKey{
{
Name: "service.name",
@@ -1345,7 +1346,7 @@ func TestAdjustKeys(t *testing.T) {
},
},
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedGroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
@@ -1392,7 +1393,7 @@ func TestAdjustKeys(t *testing.T) {
},
},
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedOrder: []qbtypes.OrderBy{
{
Key: qbtypes.OrderByKey{
@@ -1451,7 +1452,7 @@ func TestAdjustKeys(t *testing.T) {
},
},
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedSelectFields: []telemetrytypes.TelemetryFieldKey{
{
Name: "trace_id",
@@ -1506,7 +1507,7 @@ func TestAdjustKeys(t *testing.T) {
},
},
},
keysMap: buildCompleteFieldKeyMap(releaseTime),
keysMap: tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
// After alias adjustment, name becomes "span.duration" with FieldContextUnspecified
// "span.duration" is not in keysMap, so context stays unspecified
expectedOrder: []qbtypes.OrderBy{
@@ -1574,14 +1575,14 @@ func TestAdjustKeys(t *testing.T) {
if c.expectDeprecatedFieldsAdd {
// Check that at least some deprecated fields were added
foundDeprecatedField := false
for fieldName := range IntrinsicFieldsDeprecated {
for fieldName := range tracestelemetryschema.IntrinsicFieldsDeprecated {
if _, ok := keysMapCopy[fieldName]; ok {
foundDeprecatedField = true
break
}
}
if !foundDeprecatedField {
for fieldName := range CalculatedFieldsDeprecated {
for fieldName := range tracestelemetryschema.CalculatedFieldsDeprecated {
if _, ok := keysMapCopy[fieldName]; ok {
foundDeprecatedField = true
break
@@ -1683,11 +1684,11 @@ func newSkipResourceFingerprintBuilder(
t.Helper()
fl := flaggertest.New(t)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(
instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl,
@@ -1713,10 +1714,10 @@ func TestStatementBuilderGroupByUnseenKey(t *testing.T) {
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
@@ -1755,10 +1756,10 @@ func TestStatementBuilderAggregationUnseenKey(t *testing.T) {
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(

View File

@@ -1,4 +1,4 @@
package telemetrytraces
package tracesstatementbuilder
import (
"context"
@@ -10,6 +10,7 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -136,7 +137,7 @@ func (b *traceOperatorCTEBuilder) buildAllSpansCTE(ctx context.Context) {
sb.Select("*")
sb.SelectMore(sqlbuilder.Escape("resource_string_service$$name") + " AS `service.name`")
sb.From(fmt.Sprintf("%s.%s", DBName, SpanIndexV3TableName))
sb.From(fmt.Sprintf("%s.%s", tracestelemetryschema.DBName, tracestelemetryschema.SpanIndexV3TableName))
startBucket := b.start/querybuilder.NsToSeconds - querybuilder.BucketAdjustment
endBucket := b.end / querybuilder.NsToSeconds
sb.Where(
@@ -245,7 +246,7 @@ func (b *traceOperatorCTEBuilder) buildQueryCTE(ctx context.Context, queryName s
sb := sqlbuilder.NewSelectBuilder()
sb.Select("*")
sb.From(fmt.Sprintf("%s.%s", DBName, SpanIndexV3TableName))
sb.From(fmt.Sprintf("%s.%s", tracestelemetryschema.DBName, tracestelemetryschema.SpanIndexV3TableName))
if resourceFilterCTEName != "" {
sb.Where(fmt.Sprintf("resource_fingerprint GLOBAL IN (SELECT fingerprint FROM %s)", resourceFilterCTEName))
}
@@ -436,9 +437,9 @@ func (b *traceOperatorCTEBuilder) buildFinalQuery(ctx context.Context, selectFro
if requestType == qbtypes.RequestTypeRaw {
isSelectFieldsEmpty = len(b.operator.SelectFields) == 0
if isSelectFieldsEmpty {
b.operator.SelectFields = make([]telemetrytypes.TelemetryFieldKey, 0, len(IntrinsicSpanFields)+len(CalculatedSpanFields))
b.operator.SelectFields = append(b.operator.SelectFields, IntrinsicSpanFields...)
b.operator.SelectFields = append(b.operator.SelectFields, CalculatedSpanFields...)
b.operator.SelectFields = make([]telemetrytypes.TelemetryFieldKey, 0, len(tracestelemetryschema.IntrinsicSpanFields)+len(tracestelemetryschema.CalculatedSpanFields))
b.operator.SelectFields = append(b.operator.SelectFields, tracestelemetryschema.IntrinsicSpanFields...)
b.operator.SelectFields = append(b.operator.SelectFields, tracestelemetryschema.CalculatedSpanFields...)
}
}
@@ -502,7 +503,7 @@ func (b *traceOperatorCTEBuilder) buildListQuery(ctx context.Context, selectFrom
}
if isSelectFieldsEmpty {
for _, col := range ContextualSpanColumns {
for _, col := range tracestelemetryschema.ContextualSpanColumns {
sb.SelectMore(col)
}
}

View File

@@ -1,4 +1,4 @@
package telemetrytraces
package tracesstatementbuilder
import (
"context"
@@ -8,6 +8,7 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
@@ -19,10 +20,10 @@ func newTestTraceOperatorStatementBuilder(t *testing.T) *traceOperatorStatementB
t.Helper()
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
traceStmtBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),

View File

@@ -1,4 +1,4 @@
package telemetrytraces
package tracesstatementbuilder
import (
"context"
@@ -8,7 +8,10 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryresourcefilter"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/resourcefilter"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -26,6 +29,32 @@ type traceOperatorStatementBuilder struct {
var _ qbtypes.TraceOperatorStatementBuilder = (*traceOperatorStatementBuilder)(nil)
// NewOperatorFactory returns a provider factory for the trace-operator statement
// builder. The operator delegates sub-query construction to a trace query statement
// builder, so it builds its own internally — mirroring how the meter factory builds
// its own metrics builder.
func NewOperatorFactory(
telemetryStore telemetrystore.TelemetryStore,
metadataStore telemetrytypes.MetadataStore,
fl flagger.Flagger,
) factory.ProviderFactory[qbtypes.TraceOperatorStatementBuilder, statementbuilder.Config] {
return factory.NewProviderFactory(
factory.MustNewName("traceoperator"),
func(_ context.Context, settings factory.ProviderSettings, cfg statementbuilder.Config) (qbtypes.TraceOperatorStatementBuilder, error) {
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, fm, cb, fl)
traceStmtBuilder := NewTraceQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, telemetryStore, fl,
cfg.SkipResourceFingerprint.Enabled, cfg.SkipResourceFingerprint.Threshold,
)
return NewTraceOperatorStatementBuilder(
settings, metadataStore, fm, cb, traceStmtBuilder, aggExprRewriter, fl,
), nil
},
)
}
func NewTraceOperatorStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
@@ -35,12 +64,12 @@ func NewTraceOperatorStatementBuilder(
aggExprRewriter qbtypes.AggExprRewriter,
flagger flagger.Flagger,
) *traceOperatorStatementBuilder {
tracesSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrytraces")
tracesSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema")
resourceFilterStmtBuilder := telemetryresourcefilter.New[qbtypes.TraceAggregation](
resourceFilterStmtBuilder := resourcefilter.New[qbtypes.TraceAggregation](
settings,
DBName,
TracesResourceV3TableName,
tracestelemetryschema.DBName,
tracestelemetryschema.TracesResourceV3TableName,
telemetrytypes.SignalTraces,
telemetrytypes.SourceUnspecified,
metadataStore,

View File

@@ -1,4 +1,4 @@
package telemetrytraces
package tracesstatementbuilder
import (
"context"
@@ -9,6 +9,7 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
@@ -20,11 +21,11 @@ import (
func TestTraceTimeRangeOptimization(t *testing.T) {
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap["trace_id"] = []*telemetrytypes.TelemetryFieldKey{{
Name: "trace_id",
FieldContext: telemetrytypes.FieldContextSpan,
@@ -129,7 +130,7 @@ func TestTraceTimeRangeOptimization(t *testing.T) {
assert.NotEmpty(t, stmt.Query)
if tt.query.Filter != nil && tt.query.Filter.Expression != "" {
traceIDs, found := ExtractTraceIDsFromFilter(tt.query.Filter.Expression)
traceIDs, found := tracestelemetryschema.ExtractTraceIDsFromFilter(tt.query.Filter.Expression)
assert.Equal(t, tt.expectTimeOptimization, found && len(traceIDs) > 0)
}
})

View File

@@ -1,12 +0,0 @@
package telemetryaudit
const (
DBName = "signoz_audit"
AuditLogsTableName = "distributed_logs"
AuditLogsLocalTableName = "logs"
TagAttributesTableName = "distributed_tag_attributes"
TagAttributesLocalTableName = "tag_attributes"
LogAttributeKeysTblName = "distributed_logs_attribute_keys"
LogResourceKeysTblName = "distributed_logs_resource_keys"
LogsResourceTableName = "distributed_logs_resource"
)

View File

@@ -1,15 +0,0 @@
package telemetrylogs
const (
DBName = "signoz_logs"
LogsV2TableName = "distributed_logs_v2"
LogsV2LocalTableName = "logs_v2"
TagAttributesV2TableName = "distributed_tag_attributes_v2"
TagAttributesV2LocalTableName = "tag_attributes_v2"
LogAttributeKeysTblName = "distributed_logs_attribute_keys"
LogResourceKeysTblName = "distributed_logs_resource_keys"
LogsResourceV2TableName = "distributed_logs_v2_resource"
PathTypesTableName = "distributed_json_path_types"
PromotedPathsTableName = "distributed_json_promoted_paths"
SkipIndexTableName = "system.data_skipping_indices"
)

View File

@@ -13,7 +13,7 @@ import (
"github.com/SigNoz/signoz-otel-collector/constants"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -139,8 +139,8 @@ func buildListLogsJSONIndexesQuery(cluster string, filters ...string) (string, [
"name", "type_full", "expr", "granularity",
).From(fmt.Sprintf("clusterAllReplicas('%s', %s)", cluster, SkipIndexTableName))
sb.Where(sb.Equal("database", telemetrylogs.DBName))
sb.Where(sb.Equal("table", telemetrylogs.LogsV2LocalTableName))
sb.Where(sb.Equal("database", logstelemetryschema.DBName))
sb.Where(sb.Equal("table", logstelemetryschema.LogsV2LocalTableName))
sb.Where(sb.Or(
sb.ILike("expr", fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyV2ColumnPrefix))),
sb.ILike("expr", fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyPromotedColumnPrefix))),
@@ -189,12 +189,12 @@ func (t *telemetryMetaStore) ListLogsJSONIndexes(ctx context.Context, filters ..
baseColumn := ""
fieldName := ""
switch {
case strings.HasPrefix(columnExpr, telemetrylogs.BodyV2ColumnPrefix):
baseColumn = telemetrylogs.BodyV2ColumnPrefix
fieldName = strings.TrimPrefix(columnExpr, telemetrylogs.BodyV2ColumnPrefix)
case strings.HasPrefix(columnExpr, telemetrylogs.BodyPromotedColumnPrefix):
baseColumn = telemetrylogs.BodyPromotedColumnPrefix
fieldName = strings.TrimPrefix(columnExpr, telemetrylogs.BodyPromotedColumnPrefix)
case strings.HasPrefix(columnExpr, logstelemetryschema.BodyV2ColumnPrefix):
baseColumn = logstelemetryschema.BodyV2ColumnPrefix
fieldName = strings.TrimPrefix(columnExpr, logstelemetryschema.BodyV2ColumnPrefix)
case strings.HasPrefix(columnExpr, logstelemetryschema.BodyPromotedColumnPrefix):
baseColumn = logstelemetryschema.BodyPromotedColumnPrefix
fieldName = strings.TrimPrefix(columnExpr, logstelemetryschema.BodyPromotedColumnPrefix)
}
fieldName = strings.ReplaceAll(fieldName, "`", "")
@@ -228,12 +228,12 @@ func (t *telemetryMetaStore) ListJSONValues(ctx context.Context, path string, li
}
if promoted {
path = telemetrylogs.BodyPromotedColumnPrefix + path
path = logstelemetryschema.BodyPromotedColumnPrefix + path
} else {
path = telemetrylogs.BodyV2ColumnPrefix + path
path = logstelemetryschema.BodyV2ColumnPrefix + path
}
from := fmt.Sprintf("%s.%s", telemetrylogs.DBName, telemetrylogs.LogsV2TableName)
from := fmt.Sprintf("%s.%s", logstelemetryschema.DBName, logstelemetryschema.LogsV2TableName)
colExpr := func(typ telemetrytypes.JSONDataType) string {
return fmt.Sprintf("dynamicElement(%s, '%s')", path, typ.StringValue())
}
@@ -382,7 +382,7 @@ func (t *telemetryMetaStore) IsPathPromoted(ctx context.Context, path string) (b
split := strings.Split(path, telemetrytypes.ArraySep)
pathSegment := split[0]
query := fmt.Sprintf("SELECT 1 FROM %s.%s WHERE signal = ? AND column_name = ? AND field_context = ? AND field_name = ? LIMIT 1", DBName, PromotedPathsTableName)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, query, telemetrytypes.SignalLogs, telemetrylogs.LogsV2BodyPromotedColumn, telemetrytypes.FieldContextBody, pathSegment)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, query, telemetrytypes.SignalLogs, logstelemetryschema.LogsV2BodyPromotedColumn, telemetrytypes.FieldContextBody, pathSegment)
if err != nil {
return false, errors.WrapInternalf(err, CodeFailCheckPathPromoted, "failed to check if path %s is promoted", path)
}
@@ -397,7 +397,7 @@ func (t *telemetryMetaStore) GetPromotedPaths(ctx context.Context, paths ...stri
sb := sqlbuilder.Select("field_name").From(fmt.Sprintf("%s.%s", DBName, PromotedPathsTableName))
conditions := []string{
sb.Equal("signal", telemetrytypes.SignalLogs),
sb.Equal("column_name", telemetrylogs.LogsV2BodyPromotedColumn),
sb.Equal("column_name", logstelemetryschema.LogsV2BodyPromotedColumn),
sb.Equal("field_context", telemetrytypes.FieldContextBody),
sb.NotEqual("field_name", "__all__"),
}
@@ -433,8 +433,8 @@ func (t *telemetryMetaStore) GetPromotedPaths(ctx context.Context, paths ...stri
// TODO(Piyush): Remove this function.
func CleanPathPrefixes(path string) string {
path = strings.TrimPrefix(path, telemetrytypes.BodyJSONStringSearchPrefix)
path = strings.TrimPrefix(path, telemetrylogs.BodyV2ColumnPrefix)
path = strings.TrimPrefix(path, telemetrylogs.BodyPromotedColumnPrefix)
path = strings.TrimPrefix(path, logstelemetryschema.BodyV2ColumnPrefix)
path = strings.TrimPrefix(path, logstelemetryschema.BodyPromotedColumnPrefix)
return path
}
@@ -454,7 +454,7 @@ func (t *telemetryMetaStore) PromotePaths(ctx context.Context, paths ...string)
if trimmed == "" {
continue
}
if err := batch.Append(telemetrytypes.SignalLogs, telemetrylogs.LogsV2BodyPromotedColumn, "JSON()", telemetrytypes.FieldContextBody, trimmed, 0, releaseTime); err != nil {
if err := batch.Append(telemetrytypes.SignalLogs, logstelemetryschema.LogsV2BodyPromotedColumn, "JSON()", telemetrytypes.FieldContextBody, trimmed, 0, releaseTime); err != nil {
_ = batch.Abort()
return errors.WrapInternalf(err, CodeFailedToAppendPath, "failed to append path")
}

View File

@@ -6,7 +6,7 @@ import (
"github.com/SigNoz/signoz-otel-collector/constants"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/stretchr/testify/require"
)
@@ -25,8 +25,8 @@ func TestBuildListLogsJSONIndexesQuery(t *testing.T) {
expectedSQL: "SELECT name, type_full, expr, granularity FROM clusterAllReplicas('test-cluster', system.data_skipping_indices) " +
"WHERE database = ? AND table = ? AND (LOWER(expr) LIKE LOWER(?) OR LOWER(expr) LIKE LOWER(?))",
expectedArgs: []any{
telemetrylogs.DBName,
telemetrylogs.LogsV2LocalTableName,
logstelemetryschema.DBName,
logstelemetryschema.LogsV2LocalTableName,
fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyV2ColumnPrefix)),
fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyPromotedColumnPrefix)),
},
@@ -38,8 +38,8 @@ func TestBuildListLogsJSONIndexesQuery(t *testing.T) {
expectedSQL: "SELECT name, type_full, expr, granularity FROM clusterAllReplicas('test-cluster', system.data_skipping_indices) " +
"WHERE database = ? AND table = ? AND (LOWER(expr) LIKE LOWER(?) OR LOWER(expr) LIKE LOWER(?)) AND (LOWER(replaceAll(expr, '`', '')) LIKE LOWER(?) OR LOWER(replaceAll(expr, '`', '')) LIKE LOWER(?))",
expectedArgs: []any{
telemetrylogs.DBName,
telemetrylogs.LogsV2LocalTableName,
logstelemetryschema.DBName,
logstelemetryschema.LogsV2LocalTableName,
fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyV2ColumnPrefix)),
fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyPromotedColumnPrefix)),
fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains("foo")),

View File

@@ -14,11 +14,12 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetryschema/audittelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
@@ -77,27 +78,6 @@ func escapeForLike(s string) string {
func NewTelemetryMetaStore(
settings factory.ProviderSettings,
telemetrystore telemetrystore.TelemetryStore,
tracesDBName string,
tracesFieldsTblName string,
spanAttributesKeysTblName string,
indexV3TblName string,
metricsDBName string,
metricsFieldsTblName string,
meterDBName string,
meterFieldsTblName string,
logsDBName string,
logsV2TblName string,
logsFieldsTblName string,
logAttributeKeysTblName string,
logResourceKeysTblName string,
auditDBName string,
auditLogsTblName string,
auditFieldsTblName string,
auditAttributeKeysTblName string,
auditResourceKeysTblName string,
relatedMetadataDBName string,
relatedMetadataTblName string,
columnEvolutionMetadataTblName string,
fl flagger.Flagger,
) telemetrytypes.MetadataStore {
metadataSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrymetadata")
@@ -108,32 +88,32 @@ func NewTelemetryMetaStore(
t := &telemetryMetaStore{
logger: metadataSettings.Logger(),
telemetrystore: telemetrystore,
tracesDBName: tracesDBName,
tracesFieldsTblName: tracesFieldsTblName,
spanAttributesKeysTblName: spanAttributesKeysTblName,
indexV3TblName: indexV3TblName,
metricsDBName: metricsDBName,
metricsFieldsTblName: metricsFieldsTblName,
meterDBName: meterDBName,
meterFieldsTblName: meterFieldsTblName,
logsDBName: logsDBName,
logsV2TblName: logsV2TblName,
logsFieldsTblName: logsFieldsTblName,
logAttributeKeysTblName: logAttributeKeysTblName,
logResourceKeysTblName: logResourceKeysTblName,
auditDBName: auditDBName,
auditLogsTblName: auditLogsTblName,
auditFieldsTblName: auditFieldsTblName,
auditAttributeKeysTblName: auditAttributeKeysTblName,
auditResourceKeysTblName: auditResourceKeysTblName,
relatedMetadataDBName: relatedMetadataDBName,
relatedMetadataTblName: relatedMetadataTblName,
columnEvolutionMetadataTblName: columnEvolutionMetadataTblName,
tracesDBName: tracestelemetryschema.DBName,
tracesFieldsTblName: tracestelemetryschema.TagAttributesV2TableName,
spanAttributesKeysTblName: tracestelemetryschema.SpanAttributesKeysTblName,
indexV3TblName: tracestelemetryschema.SpanIndexV3TableName,
metricsDBName: metricstelemetryschema.DBName,
metricsFieldsTblName: metricstelemetryschema.AttributesMetadataTableName,
meterDBName: metertelemetryschema.DBName,
meterFieldsTblName: metertelemetryschema.SamplesAgg1dTableName,
logsDBName: logstelemetryschema.DBName,
logsV2TblName: logstelemetryschema.LogsV2TableName,
logsFieldsTblName: logstelemetryschema.TagAttributesV2TableName,
logAttributeKeysTblName: logstelemetryschema.LogAttributeKeysTblName,
logResourceKeysTblName: logstelemetryschema.LogResourceKeysTblName,
auditDBName: audittelemetryschema.DBName,
auditLogsTblName: audittelemetryschema.AuditLogsTableName,
auditFieldsTblName: audittelemetryschema.TagAttributesTableName,
auditAttributeKeysTblName: audittelemetryschema.LogAttributeKeysTblName,
auditResourceKeysTblName: audittelemetryschema.LogResourceKeysTblName,
relatedMetadataDBName: DBName,
relatedMetadataTblName: AttributesMetadataTableName,
columnEvolutionMetadataTblName: ColumnEvolutionMetadataTableName,
jsonColumnMetadata: map[telemetrytypes.Signal]map[telemetrytypes.FieldContext]telemetrytypes.JSONColumnMetadata{
telemetrytypes.SignalLogs: {
telemetrytypes.FieldContextBody: telemetrytypes.JSONColumnMetadata{
BaseColumn: telemetrylogs.LogsV2BodyV2Column,
PromotedColumn: telemetrylogs.LogsV2BodyPromotedColumn,
BaseColumn: logstelemetryschema.LogsV2BodyV2Column,
PromotedColumn: logstelemetryschema.LogsV2BodyPromotedColumn,
},
},
},
@@ -309,8 +289,8 @@ func (t *telemetryMetaStore) getTracesKeys(ctx context.Context, fieldKeySelector
complete := rowCount <= limit
staticKeys := []string{"isRoot", "isEntryPoint"}
staticKeys = append(staticKeys, maps.Keys(telemetrytraces.IntrinsicFields)...)
staticKeys = append(staticKeys, maps.Keys(telemetrytraces.CalculatedFields)...)
staticKeys = append(staticKeys, maps.Keys(tracestelemetryschema.IntrinsicFields)...)
staticKeys = append(staticKeys, maps.Keys(tracestelemetryschema.CalculatedFields)...)
// Add matching intrinsic and matching calculated fields
// These don't count towards the limit
@@ -324,14 +304,14 @@ func (t *telemetryMetaStore) getTracesKeys(ctx context.Context, fieldKeySelector
}
if found {
if field, exists := telemetrytraces.IntrinsicFields[key]; exists {
if field, exists := tracestelemetryschema.IntrinsicFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
continue
}
if field, exists := telemetrytraces.CalculatedFields[key]; exists {
if field, exists := tracestelemetryschema.CalculatedFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
@@ -656,7 +636,7 @@ func (t *telemetryMetaStore) getLogsKeys(ctx context.Context, orgID valuer.UUID,
complete := rowCount <= limit
staticKeys := []string{}
staticKeys = append(staticKeys, maps.Keys(telemetrylogs.IntrinsicFields)...)
staticKeys = append(staticKeys, maps.Keys(logstelemetryschema.IntrinsicFields)...)
// Add matching intrinsic and matching calculated fields
// These don't count towards the limit
@@ -670,7 +650,7 @@ func (t *telemetryMetaStore) getLogsKeys(ctx context.Context, orgID valuer.UUID,
}
if found {
if field, exists := telemetrylogs.IntrinsicFields[key]; exists {
if field, exists := logstelemetryschema.IntrinsicFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
@@ -898,7 +878,7 @@ func (t *telemetryMetaStore) getAuditKeys(ctx context.Context, fieldKeySelectors
complete := rowCount <= limit
// Add intrinsic audit fields (same as logs intrinsics: body, severity_text, etc.)
staticKeys := maps.Keys(telemetryaudit.IntrinsicFields)
staticKeys := maps.Keys(audittelemetryschema.IntrinsicFields)
for _, key := range staticKeys {
found := false
for _, v := range searchTexts {
@@ -909,7 +889,7 @@ func (t *telemetryMetaStore) getAuditKeys(ctx context.Context, fieldKeySelectors
}
if found {
if field, exists := telemetryaudit.IntrinsicFields[key]; exists {
if field, exists := audittelemetryschema.IntrinsicFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
@@ -1176,7 +1156,7 @@ func enrichWithIntrinsicMetricKeys(keys map[string][]*telemetrytypes.TelemetryFi
continue
}
for name, key := range telemetrymetrics.IntrinsicMetricFieldDefinitions {
for name, key := range metricstelemetryschema.IntrinsicMetricFieldDefinitions {
if !selectorMatchesIntrinsicField(selector, key) {
continue
}
@@ -1892,7 +1872,7 @@ func (t *telemetryMetaStore) getIntrinsicMetricFieldValues(ctx context.Context,
instrumentationtypes.CodeNamespace: "metadata",
instrumentationtypes.CodeFunctionName: "getIntrinsicMetricFieldValues",
})
key, ok := telemetrymetrics.IntrinsicMetricFieldDefinitions[fieldValueSelector.Name]
key, ok := metricstelemetryschema.IntrinsicMetricFieldDefinitions[fieldValueSelector.Name]
if !ok {
return &telemetrytypes.TelemetryFieldValues{}, nil
}
@@ -1922,18 +1902,18 @@ func (t *telemetryMetaStore) getIntrinsicMetricFieldValues(ctx context.Context,
tableName string
extraConds []string
}{
{telemetrymetrics.TimeseriesV41weekTableName, nil},
{metricstelemetryschema.TimeseriesV41weekTableName, nil},
}
if reductionEnabled {
sources = append(sources,
struct {
tableName string
extraConds []string
}{telemetrymetrics.TimeseriesV4BufferTableName, []string{"is_reduced = false"}},
}{metricstelemetryschema.TimeseriesV4BufferTableName, []string{"is_reduced = false"}},
struct {
tableName string
extraConds []string
}{telemetrymetrics.TimeseriesV4ReducedTableName, nil},
}{metricstelemetryschema.TimeseriesV4ReducedTableName, nil},
)
}
@@ -2262,7 +2242,7 @@ func (t *telemetryMetaStore) fetchMetricsTemporalityAndType(ctx context.Context,
instrumentationtypes.CodeNamespace: "metadata",
instrumentationtypes.CodeFunctionName: "fetchMetricsTemporalityAndType",
})
adjustedStartTs, adjustedEndTs, tsTableName, _ := telemetrymetrics.WhichTSTableToUse(queryTimeRangeStartTs, queryTimeRangeEndTs, false, nil)
adjustedStartTs, adjustedEndTs, tsTableName, _ := metricstelemetryschema.WhichTSTableToUse(queryTimeRangeStartTs, queryTimeRangeEndTs, false, nil)
temporalities, types, err := t.fetchTemporalityTypeForTable(ctx, tsTableName, adjustedStartTs, adjustedEndTs, metricNames)
if err != nil {
@@ -2271,7 +2251,7 @@ func (t *telemetryMetaStore) fetchMetricsTemporalityAndType(ctx context.Context,
reduced := make(map[string]bool)
if reductionEnabled {
bufferTemporalities, bufferTypes, err := t.fetchTemporalityTypeForTable(ctx, telemetrymetrics.TimeseriesV4BufferTableName, adjustedStartTs, adjustedEndTs, metricNames, "is_reduced = false")
bufferTemporalities, bufferTypes, err := t.fetchTemporalityTypeForTable(ctx, metricstelemetryschema.TimeseriesV4BufferTableName, adjustedStartTs, adjustedEndTs, metricNames, "is_reduced = false")
if err != nil {
return nil, nil, nil, err
}
@@ -2284,7 +2264,7 @@ func (t *telemetryMetaStore) fetchMetricsTemporalityAndType(ctx context.Context,
}
}
reducedTemporalities, reducedTypes, err := t.fetchTemporalityTypeForTable(ctx, telemetrymetrics.TimeseriesV4ReducedTableName, adjustedStartTs, adjustedEndTs, metricNames)
reducedTemporalities, reducedTypes, err := t.fetchTemporalityTypeForTable(ctx, metricstelemetryschema.TimeseriesV4ReducedTableName, adjustedStartTs, adjustedEndTs, metricNames)
if err != nil {
return nil, nil, nil, err
}
@@ -2616,13 +2596,13 @@ func (t *telemetryMetaStore) GetFirstSeenFromMetricMetadata(ctx context.Context,
}
func (t *telemetryMetaStore) FetchLastSeenInfoMulti(ctx context.Context, orgID valuer.UUID, metricNames ...string) (map[string]int64, error) {
lastSeenInfo, err := t.fetchLastSeenInfoForTable(ctx, telemetrymetrics.TimeseriesV4TableName, metricNames)
lastSeenInfo, err := t.fetchLastSeenInfoForTable(ctx, metricstelemetryschema.TimeseriesV4TableName, metricNames)
if err != nil {
return nil, err
}
if t.fl.BooleanOrEmpty(ctx, flagger.FeatureEnableMetricsReduction, featuretypes.NewFlaggerEvaluationContext(orgID)) {
reducedLastSeen, err := t.fetchLastSeenInfoForTable(ctx, telemetrymetrics.TimeseriesV4ReducedTableName, metricNames)
reducedLastSeen, err := t.fetchLastSeenInfoForTable(ctx, metricstelemetryschema.TimeseriesV4ReducedTableName, metricNames)
if err != nil {
return nil, err
}

View File

@@ -9,13 +9,8 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -42,27 +37,6 @@ func TestGetFirstSeenFromMetricMetadata(t *testing.T) {
metadata := NewTelemetryMetaStore(
instrumentationtest.New().ToProviderSettings(),
mockTelemetryStore,
telemetrytraces.DBName,
telemetrytraces.TagAttributesV2TableName,
telemetrytraces.SpanAttributesKeysTblName,
telemetrytraces.SpanIndexV3TableName,
telemetrymetrics.DBName,
telemetrymetrics.AttributesMetadataTableName,
telemetrymeter.DBName,
telemetrymeter.SamplesAgg1dTableName,
telemetrylogs.DBName,
telemetrylogs.LogsV2TableName,
telemetrylogs.TagAttributesV2TableName,
telemetrylogs.LogAttributeKeysTblName,
telemetrylogs.LogResourceKeysTblName,
telemetryaudit.DBName,
telemetryaudit.AuditLogsTableName,
telemetryaudit.TagAttributesTableName,
telemetryaudit.LogAttributeKeysTblName,
telemetryaudit.LogResourceKeysTblName,
DBName,
AttributesMetadataTableName,
ColumnEvolutionMetadataTableName,
flaggertest.New(t),
)

View File

@@ -1,37 +0,0 @@
package telemetrymetrics
import "github.com/SigNoz/signoz/pkg/types/telemetrytypes"
var IntrinsicFields = []string{
"temporality",
"metric_name",
"type",
"is_monotonic",
}
var IntrinsicMetricFieldDefinitions = map[string]telemetrytypes.TelemetryFieldKey{
"metric_name": {
Name: "metric_name",
Signal: telemetrytypes.SignalMetrics,
FieldContext: telemetrytypes.FieldContextMetric,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
// "type": {
// Name: "type",
// Signal: telemetrytypes.SignalMetrics,
// FieldContext: telemetrytypes.FieldContextMetric,
// FieldDataType: telemetrytypes.FieldDataTypeString,
// },
// "temporality": {
// Name: "temporality",
// Signal: telemetrytypes.SignalMetrics,
// FieldContext: telemetrytypes.FieldContextMetric,
// FieldDataType: telemetrytypes.FieldDataTypeString,
// },
// "is_monotonic": {
// Name: "is_monotonic",
// Signal: telemetrytypes.SignalMetrics,
// FieldContext: telemetrytypes.FieldContextMetric,
// FieldDataType: telemetrytypes.FieldDataTypeBool,
// },
}

View File

@@ -1,4 +1,4 @@
package telemetryaudit
package audittelemetryschema
import (
"context"
@@ -134,7 +134,7 @@ func (c *conditionBuilder) ConditionFor(
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken are logs-body-only; reject for audit.
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for audit.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}

View File

@@ -1,4 +1,4 @@
package telemetryaudit
package audittelemetryschema
import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
@@ -127,3 +127,14 @@ var auditLogColumns = map[string]*schema.Column{
"scope_version": {Name: "scope_version", Type: schema.ColumnTypeString},
"scope_string": {Name: "scope_string", Type: schema.MapColumnType{KeyType: schema.LowCardinalityColumnType{ElementType: schema.ColumnTypeString}, ValueType: schema.ColumnTypeString}},
}
const (
DBName = "signoz_audit"
AuditLogsTableName = "distributed_logs"
AuditLogsLocalTableName = "logs"
TagAttributesTableName = "distributed_tag_attributes"
TagAttributesLocalTableName = "tag_attributes"
LogAttributeKeysTblName = "distributed_logs_attribute_keys"
LogResourceKeysTblName = "distributed_logs_resource_keys"
LogsResourceTableName = "distributed_logs_resource"
)

View File

@@ -1,4 +1,4 @@
package telemetryaudit
package audittelemetryschema
import (
"context"

View File

@@ -1,8 +1,9 @@
package telemetrylogs
package logstelemetryschema
import (
"context"
"fmt"
"regexp"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
@@ -27,6 +28,52 @@ func NewConditionBuilder(fm qbtypes.FieldMapper, fl flagger.Flagger) *conditionB
return &conditionBuilder{fm: fm, fl: fl}
}
// conditionForSearch ORs a case-insensitive match of the needle across the searchable
// columns of the key's field context (an unspecified context covers every column).
func (c *conditionBuilder) conditionForSearch(
ctx context.Context,
orgID valuer.UUID,
key *telemetrytypes.TelemetryFieldKey,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// Literal case-insensitive substring match: QuoteMeta the needle and LOWER both sides
// so the body path can use the LOWER(toString(body_v2)) skip index (a (?i) regex could not).
needle := regexp.QuoteMeta(fmt.Sprintf("%v", value))
useJSONBody := c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
var conditions []string
for _, col := range searchColumns(key.FieldContext, useJSONBody) {
switch col.Type.GetType() {
case schema.ColumnTypeEnumMap:
keysExpr := fmt.Sprintf("mapKeys(%s)", col.Name)
valsExpr := fmt.Sprintf("mapValues(%s)", col.Name)
// match() needs a String array; cast non-string map values first.
if mc, ok := col.Type.(schema.MapColumnType); ok && mc.ValueType.GetType() != schema.ColumnTypeEnumString {
valsExpr = fmt.Sprintf("arrayMap(x -> toString(x), mapValues(%s))", col.Name)
}
conditions = append(conditions, sb.Or(
fmt.Sprintf("arrayExists(x -> match(LOWER(x), LOWER(%s)), %s)", sb.Var(needle), keysExpr),
fmt.Sprintf("arrayExists(x -> match(LOWER(x), LOWER(%s)), %s)", sb.Var(needle), valsExpr),
))
case schema.ColumnTypeEnumJSON:
conditions = append(conditions, fmt.Sprintf("match(LOWER(toString(%s)), LOWER(%s))", col.Name, sb.Var(needle)))
case schema.ColumnTypeEnumString, schema.ColumnTypeEnumLowCardinality:
conditions = append(conditions, fmt.Sprintf("match(LOWER(%s), LOWER(%s))", col.Name, sb.Var(needle)))
default:
return nil, nil, errors.NewInternalf(errors.CodeInternal, "search does not support the column type of %q", col.Name)
}
}
if len(conditions) == 0 {
return nil, nil, nil
}
// The advisory rides on CostGuard (set by the visitor), not warnings.
return []string{sb.Or(conditions...)}, nil, nil
}
// isBodyJSONSearch reports whether a key addresses a path within the body JSON. Only
// an explicit Body context qualifies; a bare, context-less `body` (e.g. full-text
// `count_distinct(body)` or `body EXISTS`) is a full-text match, not a `$.body` path.
@@ -408,6 +455,11 @@ func (c *conditionBuilder) ConditionFor(
matches := querybuilder.MatchingFieldKeys(key, fieldKeys)
skipResourceFilter := options.SkipResourceFilter
// search() resolves its own (optional) scope; handle it before key resolution.
if operator == qbtypes.FilterOperatorSearch {
return c.conditionForSearch(ctx, orgID, key, value, sb)
}
keys, warning := querybuilder.ResolveKeys(key, matches)
var warnings []string
if warning != "" {

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logstelemetryschema
import (
"context"
@@ -548,7 +548,7 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("bare intrinsic column resolves to the column, not synthesized attributes", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "severity_text"}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{},qbtypes.FilterOperatorEqual, "ERROR", sb)
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "ERROR", sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -560,7 +560,7 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("attribute context", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextAttribute}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{},qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 2)
@@ -571,7 +571,7 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("resource context", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextResource}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{},qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 2)
@@ -582,7 +582,7 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("log context folds to attributes then body", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextLog}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{},qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 4)

View File

@@ -1,8 +1,6 @@
package telemetrylogs
package logstelemetryschema
import (
"fmt"
"github.com/SigNoz/signoz-otel-collector/constants"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -41,9 +39,8 @@ const (
// messageSubColumn is the ClickHouse sub-column that body searches map to
// when use_json_body feature flag is true.
messageSubField = "message"
messageSubColumn = "body_v2.message"
bodySearchDefaultWarning = "body searches default to `body.message:string`. Use `body.<key>` to search a different field inside body"
messageSubField = "message"
messageSubColumn = "body_v2.message"
// bodyMessageField is the field name addressing the message sub-field of the
// body when use_json_body is enabled (i.e. `body` + `.` + `message`). hasToken
@@ -137,10 +134,16 @@ var (
}
)
func bodyAliasExpression(bodyJSONEnabled bool) string {
if !bodyJSONEnabled {
return LogsV2BodyColumn
}
return fmt.Sprintf("%s as body", LogsV2BodyV2Column)
}
const (
DBName = "signoz_logs"
LogsV2TableName = "distributed_logs_v2"
LogsV2LocalTableName = "logs_v2"
TagAttributesV2TableName = "distributed_tag_attributes_v2"
TagAttributesV2LocalTableName = "tag_attributes_v2"
LogAttributeKeysTblName = "distributed_logs_attribute_keys"
LogResourceKeysTblName = "distributed_logs_resource_keys"
LogsResourceV2TableName = "distributed_logs_v2_resource"
PathTypesTableName = "distributed_json_path_types"
PromotedPathsTableName = "distributed_json_promoted_paths"
SkipIndexTableName = "system.data_skipping_indices"
)

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logstelemetryschema
import (
"context"
@@ -635,3 +635,37 @@ func (m *fieldMapper) existsExpressionFor(
}
return querybuilder.ExistsExpression(columns, key, tsStart, tsEnd, fieldExpression, exists)
}
// searchColumns is the single source of truth for the columns search() fans out
// across, by field context. Body is body_v2 JSON when useJSONBody, else body string.
func searchColumns(fieldContext telemetrytypes.FieldContext, useJSONBody bool) []*schema.Column {
switch fieldContext {
case telemetrytypes.FieldContextLog:
return []*schema.Column{
logsV2Columns[LogsV2SeverityTextColumn],
logsV2Columns[LogsV2TraceIDColumn],
logsV2Columns[LogsV2SpanIDColumn],
}
case telemetrytypes.FieldContextBody:
if useJSONBody {
return []*schema.Column{logsV2Columns[LogsV2BodyV2Column]}
}
return []*schema.Column{logsV2Columns[LogsV2BodyColumn]}
case telemetrytypes.FieldContextAttribute:
return []*schema.Column{
logsV2Columns[LogsV2AttributesStringColumn],
logsV2Columns[LogsV2AttributesNumberColumn],
logsV2Columns[LogsV2AttributesBoolColumn],
}
case telemetrytypes.FieldContextResource:
return []*schema.Column{
logsV2Columns[LogsV2ResourcesStringColumn],
}
default:
columns := searchColumns(telemetrytypes.FieldContextLog, useJSONBody)
columns = append(columns, searchColumns(telemetrytypes.FieldContextBody, useJSONBody)...)
columns = append(columns, searchColumns(telemetrytypes.FieldContextAttribute, useJSONBody)...)
columns = append(columns, searchColumns(telemetrytypes.FieldContextResource, useJSONBody)...)
return columns
}
}

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logstelemetryschema
import (
"context"

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logstelemetryschema
import (
"context"
@@ -19,7 +19,7 @@ func TestLikeAndILikeWithoutWildcards_Warns(t *testing.T) {
cb := NewConditionBuilder(fm, fl)
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := buildCompleteFieldKeyMap(releaseTime)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
@@ -57,7 +57,7 @@ func TestLikeAndILikeWithWildcards_NoWarn(t *testing.T) {
cb := NewConditionBuilder(fm, fl)
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := buildCompleteFieldKeyMap(releaseTime)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logstelemetryschema
import (
"context"
@@ -21,7 +21,7 @@ func TestFilterExprLogsBodyJSON(t *testing.T) {
cb := NewConditionBuilder(fm, fl)
// Define a comprehensive set of field keys to support all test cases
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := buildCompleteFieldKeyMap(releaseTime)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logstelemetryschema
import (
"context"
@@ -41,7 +41,7 @@ func TestFilterExprLogs(t *testing.T) {
cb := NewConditionBuilder(fm, fl)
// Define a comprehensive set of field keys to support all test cases
keys := buildCompleteFieldKeyMap(releaseTime)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
@@ -2476,7 +2476,7 @@ func TestFilterExprLogsConflictNegation(t *testing.T) {
// Define a comprehensive set of field keys to support all test cases
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := buildCompleteFieldKeyMap(releaseTime)
keys := BuildCompleteFieldKeyMap(releaseTime)
keys["body"] = []*telemetrytypes.TelemetryFieldKey{
{

View File

@@ -0,0 +1,288 @@
package logstelemetryschema
import (
"context"
"testing"
"time"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/require"
)
// searchFanOut returns the WHERE fragment search() fans out to. bodyExpr is the
// body match expression, which differs between the legacy string body and the
// body_v2 JSON column.
func searchFanOut(bodyExpr string) string {
return "(match(LOWER(severity_text), LOWER(?)) OR match(LOWER(trace_id), LOWER(?)) OR match(LOWER(span_id), LOWER(?)) OR " +
bodyExpr + " OR " +
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(attributes_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(attributes_string))) OR " +
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(attributes_number)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), arrayMap(x -> toString(x), mapValues(attributes_number)))) OR " +
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(attributes_bool)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), arrayMap(x -> toString(x), mapValues(attributes_bool)))) OR " +
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(resources_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(resources_string))))"
}
// searchArgs returns v repeated once per bound parameter search() emits — one per
// searchable column expression (currently 12).
func searchArgs(v any) []any {
const searchColumnParams = 12
args := make([]any, searchColumnParams)
for i := range args {
args[i] = v
}
return args
}
// TestFilterExprSearch covers the search('needle') function, which fans out
// across every searchable column via FilterOperatorSearch.
func TestFilterExprSearch(t *testing.T) {
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
inWindowStart := uint64(releaseTime.Add(-5 * time.Minute).UnixNano())
inWindowEnd := uint64(releaseTime.Add(5 * time.Minute).UnixNano())
legacyBody := "match(LOWER(body), LOWER(?))"
jsonBody := "match(LOWER(toString(body_v2)), LOWER(?))"
// Single-context scope fragments (the fan-out narrowed to one context).
logScope := "(match(LOWER(severity_text), LOWER(?)) OR match(LOWER(trace_id), LOWER(?)) OR match(LOWER(span_id), LOWER(?)))"
resourceScope := "(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(resources_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(resources_string)))"
serviceNameEq := "(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? " +
"AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)"
testCases := []struct {
name string
query string
jsonBodyEnabled bool
fullTextColumn *telemetrytypes.TelemetryFieldKey
startNs uint64
endNs uint64
shouldPass bool
expectedQuery string
expectedArgs []any
expectWarning bool
expectedErrorContains string
}{
{
name: "quoted, legacy body",
query: "search('error')",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("error"),
expectWarning: true,
},
{
name: "quoted, json body",
query: "search('error')",
jsonBodyEnabled: true,
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(jsonBody),
expectedArgs: searchArgs("error"),
expectWarning: true,
},
{
name: "bare word",
query: "search(timeout)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("timeout"),
expectWarning: true,
},
{
name: "negated",
query: "NOT search('error')",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE NOT (" + searchFanOut(legacyBody) + ")",
expectedArgs: searchArgs("error"),
expectWarning: true,
},
{
name: "combined with field filter",
query: "search('error') AND service.name=\"api\"",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE (" + searchFanOut(legacyBody) + " AND " + serviceNameEq + ")",
expectedArgs: append(searchArgs("error"), "api"),
expectWarning: true,
},
{
// A wide window is allowed at build time; scan cost is bounded by the
// querier's EXPLAIN ESTIMATE gate, not a window cap in the builder.
name: "wide window builds (estimate gate lives in querier)",
query: "search('error')",
fullTextColumn: DefaultFullTextColumn,
startNs: uint64(releaseTime.Add(-10 * time.Hour).UnixNano()),
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("error"),
expectWarning: true,
},
{
// search() is keyless and independent of fullTextColumn (which only
// governs bare/quoted free text). It must work even when unset.
name: "independent of full text column",
query: "search('error')",
fullTextColumn: nil,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("error"),
expectWarning: true,
},
{
// A context-prefixed bare word must be used as the literal needle, not
// normalized into a field key (Normalize would strip "resource.").
name: "bare word with context prefix is not normalized",
query: "search(resource.deployment)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("resource\\.deployment"),
expectWarning: true,
},
{
// A numeric needle must be the literal digits, not the %v rendering of a
// parsed float64 (which would make search(1000000) scan for "1e+06").
name: "numeric needle is not scientific notation",
query: "search(1000000)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("1000000"),
expectWarning: true,
},
{
name: "scoped to body, legacy",
query: "search('error', body)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE (" + legacyBody + ")",
expectedArgs: []any{"error"},
expectWarning: true,
},
{
name: "scoped to body, json",
query: "search('error', body)",
jsonBodyEnabled: true,
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE (" + jsonBody + ")",
expectedArgs: []any{"error"},
expectWarning: true,
},
{
name: "scoped to resource (quoted scope)",
query: "search('error', 'resource')",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE (" + resourceScope + ")",
expectedArgs: []any{"error", "error"},
expectWarning: true,
},
{
name: "scoped to log fields",
query: "search('error', log)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + logScope,
expectedArgs: []any{"error", "error", "error"},
expectWarning: true,
},
{
name: "scoped to multiple contexts",
query: "search('error', body, resource)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE ((" + legacyBody + ") OR (" + resourceScope + "))",
expectedArgs: []any{"error", "error", "error"},
expectWarning: true,
},
{
name: "invalid scope",
query: "search('error', 'timeout')",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: false,
expectedErrorContains: "invalid search scope",
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureUseJSONBody.String(): tc.jsonBodyEnabled,
})
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: tc.fullTextColumn,
StartNs: tc.startNs,
EndNs: tc.endNs,
}
clause, err := querybuilder.PrepareWhereClause(tc.query, opts)
if !tc.shouldPass {
require.Error(t, err)
require.True(t, detailContains(err, tc.expectedErrorContains),
"error %v should contain %q", err, tc.expectedErrorContains)
return
}
require.NoError(t, err)
require.False(t, clause.IsEmpty())
sql, args := clause.WhereClause.BuildWithFlavor(sqlbuilder.ClickHouse)
require.Equal(t, tc.expectedQuery, sql)
require.Equal(t, tc.expectedArgs, args)
if tc.expectWarning {
// The visitor only flags the cost guard; the statement builder
// materializes the advisory + budget from config downstream.
require.True(t, clause.RequiresCostGuard)
}
})
}
}

View File

@@ -1,4 +1,4 @@
package telemetrylogs
package logstelemetryschema
import (
"fmt"

Some files were not shown because too many files have changed in this diff Show More