chore: add initial version of query range design principles doc

* fix: add missing filtering for ip address for scalar data

In domain listing api for external api monitoring,
we have option to filter out the IP address but
it only handles timeseries and raw type data while
domain list handler returns scalar data.

* fix: switch to new derived attributes for ip filtering

---------

Co-authored-by: Nityananda Gohain <nityanandagohain@gmail.com>

.github/workflows/integrationci.yaml

@@ -54,7 +54,7 @@ jobs:
           - sqlite
         clickhouse-version:
           - 25.5.6
-          - 25.10.5
+          - 25.12.5
         schema-migrator-version:
           - v0.142.0
         postgres-version:

deploy/docker-swarm/otel-collector-config.yaml

@@ -82,6 +82,12 @@ exporters:
     timeout: 45s
     sending_queue:
       enabled: false
+  metadataexporter:
+    cache:
+      provider: in_memory
+    dsn: tcp://clickhouse:9000/signoz_metadata
+    enabled: true
+    timeout: 45s
 service:
   telemetry:
     logs:
@@ -93,19 +99,19 @@ service:
     traces:
       receivers: [otlp]
       processors: [signozspanmetrics/delta, batch]
-      exporters: [clickhousetraces, signozmeter]
+      exporters: [clickhousetraces, metadataexporter, signozmeter]
     metrics:
       receivers: [otlp]
       processors: [batch]
-      exporters: [signozclickhousemetrics, signozmeter]
+      exporters: [signozclickhousemetrics, metadataexporter, signozmeter]
     metrics/prometheus:
       receivers: [prometheus]
       processors: [batch]
-      exporters: [signozclickhousemetrics, signozmeter]
+      exporters: [signozclickhousemetrics, metadataexporter, signozmeter]
     logs:
       receivers: [otlp]
       processors: [batch]
-      exporters: [clickhouselogsexporter, signozmeter]
+      exporters: [clickhouselogsexporter, metadataexporter, signozmeter]
     metrics/meter:
       receivers: [signozmeter]
       processors: [batch/meter]

deploy/docker/otel-collector-config.yaml

@@ -82,6 +82,12 @@ exporters:
     timeout: 45s
     sending_queue:
       enabled: false
+  metadataexporter:
+    cache:
+      provider: in_memory
+    dsn: tcp://clickhouse:9000/signoz_metadata
+    enabled: true
+    timeout: 45s
 service:
   telemetry:
     logs:
@@ -93,19 +99,19 @@ service:
     traces:
       receivers: [otlp]
       processors: [signozspanmetrics/delta, batch]
-      exporters: [clickhousetraces, signozmeter]
+      exporters: [clickhousetraces, metadataexporter, signozmeter]
     metrics:
       receivers: [otlp]
       processors: [batch]
-      exporters: [signozclickhousemetrics, signozmeter]
+      exporters: [signozclickhousemetrics, metadataexporter, signozmeter]
     metrics/prometheus:
       receivers: [prometheus]
       processors: [batch]
-      exporters: [signozclickhousemetrics, signozmeter]
+      exporters: [signozclickhousemetrics, metadataexporter, signozmeter]
     logs:
       receivers: [otlp]
       processors: [batch]
-      exporters: [clickhouselogsexporter, signozmeter]
+      exporters: [clickhouselogsexporter, metadataexporter, signozmeter]
     metrics/meter:
       receivers: [signozmeter]
       processors: [batch/meter]

docs/contributing/query-range.md

@@ -0,0 +1,216 @@
+# Query Range v5 — Design Principles & Architectural Contracts
+
+## Purpose of This Document
+
+This document defines the design principles, invariants, and architectural contracts of the Query Range v5 system. It is intended for the authors working on the querier and querier related parts codebase. Any change to the system must align with the principles described here. If a change would violate a principle, it must be flagged and discussed.
+
+---
+
+## Core Architectural Principle
+
+**The user speaks OpenTelemetry. The storage speaks ClickHouse. The system translates between them. These two worlds must never leak into each other.**
+
+Every design choice in Query Range flows from this separation. The user-facing API surface deals exclusively in `TelemetryFieldKey`: a representation of fields as they exist in the OpenTelemetry data model. The storage layer deals in ClickHouse column expressions, table names, and SQL fragments. The translation between them is mediated by a small set of composable abstractions with strict boundaries.
+
+---
+
+## The Central Type: `TelemetryFieldKey`
+
+`TelemetryFieldKey` is the atomic unit of the entire query system. Every filter, aggregation, group-by, order-by, and select operation is expressed in terms of field keys. Understanding its design contracts is non-negotiable.
+
+### Identity
+
+A field key is identified by three dimensions:
+
+- **Name** — the field name as the user knows it (`service.name`, `http.method`, `trace_id`)
+- **FieldContext** — where the field lives in the OTel model (`resource`, `attribute`, `span`, `log`, `body`, `scope`, `event`, `metric`)
+- **FieldDataType** — the data type (`string`, `bool`, `number`/`float64`/`int64`, array variants)
+
+### Invariant: Same name does not mean same field
+
+`status` as an attribute, `status` as a body JSON key, and `status` as a span-level field are **three different fields**. The context disambiguates. Code that resolves or compares field keys must always consider all three dimensions, never just the name.
+
+### Invariant: Normalization happens once, at the boundary
+
+`TelemetryFieldKey.Normalize()` is called during JSON unmarshaling. After normalization, the text representation `resource.service.name:string` and the programmatic construction `{Name: "service.name", FieldContext: Resource, FieldDataType: String}` are identical.
+
+**Consequence:** Downstream code must never re-parse or re-normalize field keys. If you find yourself splitting on `.` or `:` deep in the pipeline, something is wrong — the normalization should have already happened.
+
+### Invariant: The text format is `context.name:datatype`
+
+Parsing rules (implemented in `GetFieldKeyFromKeyText` and `Normalize`):
+
+1. Data type is extracted from the right, after the last `:`.
+2. Field context is extracted from the left, before the first `.`, if it matches a known context prefix.
+3. Everything remaining is the name.
+
+Special case: `log.body.X` normalizes to `{FieldContext: body, Name: X}` — the `log.body.` prefix collapses because body fields under log are a nested context.
+
+### Invariant: Historical aliases must be preserved
+
+The `fieldContexts` map includes aliases (`tag` -> `attribute`, `spanfield` -> `span`, `logfield` -> `log`). These exist because older database entries use these names. Removing or changing these aliases will break existing saved queries and dashboard configurations.
+
+---
+
+## The Abstraction Stack
+
+The query pipeline is built from four interfaces that compose vertically. Each layer has a single responsibility. Each layer depends only on the layers below it. This layering is intentional and must be preserved.
+
+```
+StatementBuilder          <- Orchestrates everything into executable SQL
+  ├── AggExprRewriter     <- Rewrites aggregation expressions (maps field refs to columns)
+  ├── ConditionBuilder    <- Builds WHERE predicates (field + operator + value -> SQL)
+  └── FieldMapper         <- Maps TelemetryFieldKey -> ClickHouse column expression
+```
+
+### FieldMapper
+
+**Contract:** Given a `TelemetryFieldKey`, return a ClickHouse column expression that yields the value for that field when used in a SELECT.
+
+**Principle:** This is the *only* place where field-to-column translation happens. No other layer should contain knowledge of how fields map to storage. If you need a column expression, go through the FieldMapper.
+
+**Why:** The user says `http.request.method`. ClickHouse might store it as `attributes_string['http.request.method']`, or as a materialized column `` `attribute_string_http$$request$$method` ``, or via a JSON access path in a body column. This variation is entirely contained within the FieldMapper. Everything above it is storage-agnostic.
+
+### ConditionBuilder
+
+**Contract:** Given a field key, an operator, and a value, produce a valid SQL predicate for a WHERE clause.
+
+**Dependency:** Uses FieldMapper for the left-hand side of the condition.
+
+**Principle:** The ConditionBuilder owns all the complexity of operator semantics, i.e type casting, array operators (`hasAny`/`hasAll` vs `=`), existence checks, and negative operator behavior. This complexity must not leak upward into the StatementBuilder.
+
+### AggExprRewriter
+
+**Contract:** Given a user-facing aggregation expression like `sum(duration_nano)`, resolve field references within it and produce valid ClickHouse SQL.
+
+**Dependency:** Uses FieldMapper to resolve field names within expressions.
+
+**Principle:** Aggregation expressions are user-authored strings that contain field references. The rewriter parses them, identifies field references, resolves each through the FieldMapper, and reassembles the expression.
+
+### StatementBuilder
+
+**Contract:** Given a complete `QueryBuilderQuery`, a time range, and a request type, produces an executable SQL statement.
+
+**Dependency:** Uses all three abstractions above.
+
+**Principle:** This is the composition layer. It does not contain field mapping logic, condition building logic, or expression rewriting logic. It orchestrates the other abstractions. If you find storage-specific logic creeping into the StatementBuilder, push it down into the appropriate abstraction.
+
+### Invariant: No layer skipping
+
+The StatementBuilder must not call FieldMapper directly to build conditions, it goes through the ConditionBuilder. The AggExprRewriter must not hardcode column names, it goes through the FieldMapper. Skipping layers creates hidden coupling and makes the system fragile to storage changes.
+
+---
+
+## Design Decisions as Constraints
+
+### Constraint: Formula evaluation happens in Go, not in ClickHouse
+
+Formulas (`A + B`, `A / B`, `sqrt(A*A + B*B)`) are evaluated application-side by `FormulaEvaluator`, not via ClickHouse JOINs.
+
+**Why this is a constraint, not just an implementation choice:** The original JOIN-based approach was abandoned because ClickHouse evaluates joins right-to-left, serializing execution unnecessarily. Running queries independently allows parallelism and caching of intermediate results. Any future optimization must not reintroduce the JOIN pattern without solving the serialization problem.
+
+**Consequence:** Individual query results must be independently cacheable. Formula evaluation must handle label matching, timestamp alignment, and missing values without requiring the queries to coordinate at the SQL level.
+
+### Constraint: Zero-defaulting is aggregation-dependent
+
+Only additive/counting aggregations (`count`, `count_distinct`, `sum`, `rate`) default missing values to zero. Statistical aggregations (`avg`, `min`, `max`, percentiles) must show gaps.
+
+**Why:** Absence of data has different meanings. No error requests in a time bucket means error count = 0. No requests at all means average latency is *unknown*, not 0. Conflating these is a correctness bug, not a display preference.
+
+**Enforcement:** `GetQueriesSupportingZeroDefault` determines which queries can default to zero. The `FormulaEvaluator` consumes this via `canDefaultZero`. Changes to aggregation handling must preserve this distinction.
+
+### Constraint: Existence semantics differ for positive vs negative operators
+
+- **Positive operators** (`=`, `>`, `LIKE`, `IN`, etc.) implicitly assert field existence. `http.method = GET` means "the field exists AND equals GET".
+- **Negative operators** (`!=`, `NOT IN`, `NOT LIKE`, etc.) do **not** add an existence check. `http.method != GET` includes records where the field doesn't exist at all.
+
+**Why:** The user's intent with negative operators is ambiguous. Rather than guess, we take the broader interpretation. Users can add an explicit `EXISTS` filter if they want the narrower one. This is documented in `AddDefaultExistsFilter`.
+
+**Consequence:** Any new operator must declare its existence behavior in `AddDefaultExistsFilter`. Do not add operators without considering this.
+
+### Constraint: Post-processing functions operate on result sets, not in SQL
+
+Functions like `cutOffMin`, `ewma`, `median`, `timeShift`, `fillZero`, `runningDiff`, and `cumulativeSum` are applied in Go on the returned time series, not pushed into ClickHouse SQL.
+
+**Why:** These are sequential time-series transformations that require complete, ordered result sets. Pushing them into SQL would complicate query generation, prevent caching of raw results, and make the functions harder to test. They are applied via `ApplyFunctions` after query execution.
+
+**Consequence:** New time-series transformation functions should follow this pattern i.e implement them as Go functions on `*TimeSeries`, not as SQL modifications.
+
+### Constraint: The API surface rejects unknown fields with suggestions
+
+All request types use custom `UnmarshalJSON` that calls `DisallowUnknownFields`. Unknown fields trigger error messages with Levenshtein-based suggestions ("did you mean: 'groupBy'?").
+
+**Why:** Silent acceptance of unknown fields causes subtle bugs. A misspelled `groupBy` results in ungrouped data with no indication of what went wrong. Failing fast with suggestions turns errors into actionable feedback.
+
+**Consequence:** Any new request type or query spec struct must implement custom unmarshaling with `UnmarshalJSONWithContext`. Do not use default `json.Unmarshal` for user-facing types.
+
+### Constraint: Validation is context-sensitive to request type
+
+What's valid depends on the `RequestType`. For aggregation requests (`time_series`, `scalar`, `distribution`), fields like `groupBy`, `aggregations`, `having`, and aggregation-referenced `orderBy` are validated. For non-aggregation requests (`raw`, `raw_stream`, `trace`), these fields are ignored.
+
+**Why:** A raw log query doesn't have aggregations, so requiring `aggregations` would be wrong. But a time-series query without aggregations is meaningless. The validation rules are request-type-aware to avoid both false positives and false negatives.
+
+**Consequence:** When adding new fields to query specs, consider which request types they apply to and gate validation accordingly.
+
+---
+
+## The Composite Query Model
+
+### Structure
+
+A `QueryRangeRequest` contains a `CompositeQuery` which holds `[]QueryEnvelope`. Each envelope is a discriminated union: a `Type` field determines how `Spec` is decoded.
+
+### Invariant: Query names are unique within a composite query
+
+Builder queries must have unique names. Formulas reference queries by name (`A`, `B`, `A.0`, `A.my_alias`). Duplicate names would make formula evaluation ambiguous.
+
+### Invariant: Multi-aggregation uses indexed or aliased references
+
+A single builder query can have multiple aggregations. They are accessed in formulas via:
+- Index: `A.0`, `A.1` (zero-based)
+- Alias: `A.total`, `A.error_count`
+
+The default (just `A`) resolves to index 0. This is the formula evaluation contract and must be preserved.
+
+### Invariant: Type-specific decoding through signal detection
+
+Builder queries are decoded by first peeking at the `signal` field in the raw JSON, then unmarshaling into the appropriate generic type (`QueryBuilderQuery[TraceAggregation]`, `QueryBuilderQuery[LogAggregation]`, `QueryBuilderQuery[MetricAggregation]`). This two-pass decoding is intentional — it allows each signal to have its own aggregation schema while sharing the query structure.
+
+---
+
+## The Metadata Layer
+
+### MetadataStore
+
+The `MetadataStore` interface provides runtime field discovery and type resolution. It answers questions like "what fields exist for this signal?" and "what are the data types of field X?".
+
+### Principle: Fields can be ambiguous until resolved
+
+The same name can map to multiple `TelemetryFieldKey` variants (different contexts, different types). The metadata store returns *all* variants. Resolution to a single field happens during query building, using the query's signal and any explicit context/type hints from the user.
+
+**Consequence:** Code that calls `GetKey` or `GetKeys` must handle multiple results. Do not assume a name maps to a single field.
+
+### Principle: Materialized fields are a performance optimization, not a semantic distinction
+
+A materialized field and its non-materialized equivalent represent the same logical field. The `Materialized` flag tells the FieldMapper to generate a simpler column expression. The user should never need to know whether a field is materialized.
+
+### Principle: JSON body fields require access plans
+
+Fields inside JSON body columns (`body.response.errors[].code`) need pre-computed `JSONAccessPlan` trees that encode the traversal path, including branching at array boundaries between `Array(JSON)` and `Array(Dynamic)` representations. These plans are computed during metadata resolution, not during query execution.
+
+---
+
+## Summary of Inviolable Rules
+
+1. **User-facing types never contain ClickHouse column names or SQL fragments.**
+2. **Field-to-column translation only happens in FieldMapper.**
+3. **Normalization happens once at the API boundary, never deeper.**
+4. **Historical aliases in fieldContexts and fieldDataTypes must not be removed.**
+5. **Formula evaluation stays in Go — do not push it into ClickHouse JOINs.**
+6. **Zero-defaulting is aggregation-type-dependent — do not universally default to zero.**
+7. **Positive operators imply existence, negative operators do not.**
+8. **Post-processing functions operate on Go result sets, not in SQL.**
+9. **All user-facing types reject unknown JSON fields with suggestions.**
+10. **Validation rules are gated by request type.**
+11. **Query names must be unique within a composite query.**
+12. **The four-layer abstraction stack (FieldMapper -> ConditionBuilder -> AggExprRewriter -> StatementBuilder) must not be bypassed or flattened.**

frontend/src/AppRoutes/pageComponents.ts

@@ -308,3 +308,15 @@ export const PublicDashboardPage = Loadable(
 			/* webpackChunkName: "Public Dashboard Page" */ 'pages/PublicDashboard'
 		),
 );
+
+export const AlertTypeSelectionPage = Loadable(
+	() =>
+		import(
+			/* webpackChunkName: "Alert Type Selection Page" */ 'pages/AlertTypeSelection'
+		),
+);
+
+export const MeterExplorerPage = Loadable(
+	() =>
+		import(/* webpackChunkName: "Meter Explorer Page" */ 'pages/MeterExplorer'),
+);

frontend/src/AppRoutes/routes.ts

@@ -1,12 +1,10 @@
 import { RouteProps } from 'react-router-dom';
 import ROUTES from 'constants/routes';
-import AlertTypeSelectionPage from 'pages/AlertTypeSelection';
-import MessagingQueues from 'pages/MessagingQueues';
-import MeterExplorer from 'pages/MeterExplorer';
 
 import {
 	AlertHistory,
 	AlertOverview,
+	AlertTypeSelectionPage,
 	AllAlertChannels,
 	AllErrors,
 	ApiMonitoring,
@@ -29,6 +27,8 @@ import {
 	LogsExplorer,
 	LogsIndexToFields,
 	LogsSaveViews,
+	MessagingQueuesMainPage,
+	MeterExplorerPage,
 	MetricsExplorer,
 	OldLogsExplorer,
 	Onboarding,
@@ -399,28 +399,28 @@ const routes: AppRoutes[] = [
 	{
 		path: ROUTES.MESSAGING_QUEUES_KAFKA,
 		exact: true,
-		component: MessagingQueues,
+		component: MessagingQueuesMainPage,
 		key: 'MESSAGING_QUEUES_KAFKA',
 		isPrivate: true,
 	},
 	{
 		path: ROUTES.MESSAGING_QUEUES_CELERY_TASK,
 		exact: true,
-		component: MessagingQueues,
+		component: MessagingQueuesMainPage,
 		key: 'MESSAGING_QUEUES_CELERY_TASK',
 		isPrivate: true,
 	},
 	{
 		path: ROUTES.MESSAGING_QUEUES_OVERVIEW,
 		exact: true,
-		component: MessagingQueues,
+		component: MessagingQueuesMainPage,
 		key: 'MESSAGING_QUEUES_OVERVIEW',
 		isPrivate: true,
 	},
 	{
 		path: ROUTES.MESSAGING_QUEUES_KAFKA_DETAIL,
 		exact: true,
-		component: MessagingQueues,
+		component: MessagingQueuesMainPage,
 		key: 'MESSAGING_QUEUES_KAFKA_DETAIL',
 		isPrivate: true,
 	},
@@ -463,21 +463,21 @@ const routes: AppRoutes[] = [
 	{
 		path: ROUTES.METER,
 		exact: true,
-		component: MeterExplorer,
+		component: MeterExplorerPage,
 		key: 'METER',
 		isPrivate: true,
 	},
 	{
 		path: ROUTES.METER_EXPLORER,
 		exact: true,
-		component: MeterExplorer,
+		component: MeterExplorerPage,
 		key: 'METER_EXPLORER',
 		isPrivate: true,
 	},
 	{
 		path: ROUTES.METER_EXPLORER_VIEWS,
 		exact: true,
-		component: MeterExplorer,
+		component: MeterExplorerPage,
 		key: 'METER_EXPLORER_VIEWS',
 		isPrivate: true,
 	},

pkg/errors/http.go

@@ -18,17 +18,13 @@ type responseerroradditional struct {
 
 func AsJSON(cause error) *JSON {
 	// See if this is an instance of the base error or not
-	_, c, m, err, u, a := Unwrapb(cause)
+	_, c, m, _, u, a := Unwrapb(cause)
 
 	rea := make([]responseerroradditional, len(a))
 	for k, v := range a {
 		rea[k] = responseerroradditional{v}
 	}
 
-	if err != nil {
-		rea = append(rea, responseerroradditional{err.Error()})
-	}
-
 	return &JSON{
 		Code:    c.String(),
 		Message: m,

pkg/modules/thirdpartyapi/translator.go

@@ -120,6 +120,8 @@ func FilterResponse(results []*qbtypes.QueryRangeResponse) []*qbtypes.QueryRange
 					}
 				}
 				resultData.Rows = filteredRows
+			case *qbtypes.ScalarData:
+				resultData.Data = filterScalarDataIPs(resultData.Columns, resultData.Data)
 			}
 
 			filteredData = append(filteredData, result)
@@ -145,6 +147,39 @@ func shouldIncludeSeries(series *qbtypes.TimeSeries) bool {
 	return true
 }
 
+func filterScalarDataIPs(columns []*qbtypes.ColumnDescriptor, data [][]any) [][]any {
+	// Find column indices for server address fields
+	serverColIndices := make([]int, 0)
+	for i, col := range columns {
+		if col.Name == derivedKeyHTTPHost {
+			serverColIndices = append(serverColIndices, i)
+		}
+	}
+
+	if len(serverColIndices) == 0 {
+		return data
+	}
+
+	filtered := make([][]any, 0, len(data))
+	for _, row := range data {
+		includeRow := true
+		for _, colIdx := range serverColIndices {
+			if colIdx < len(row) {
+				if strVal, ok := row[colIdx].(string); ok {
+					if net.ParseIP(strVal) != nil {
+						includeRow = false
+						break
+					}
+				}
+			}
+		}
+		if includeRow {
+			filtered = append(filtered, row)
+		}
+	}
+	return filtered
+}
+
 func shouldIncludeRow(row *qbtypes.RawRow) bool {
 	if row.Data != nil {
 		if domainVal, ok := row.Data[derivedKeyHTTPHost]; ok {

pkg/modules/thirdpartyapi/translator_test.go

@@ -117,6 +117,59 @@ func TestFilterResponse(t *testing.T) {
 				},
 			},
 		},
+		{
+			name: "should filter out IP addresses from scalar data",
+			input: []*qbtypes.QueryRangeResponse{
+				{
+					Data: qbtypes.QueryData{
+						Results: []any{
+							&qbtypes.ScalarData{
+								QueryName: "endpoints",
+								Columns: []*qbtypes.ColumnDescriptor{
+									{
+										TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: derivedKeyHTTPHost},
+										Type:              qbtypes.ColumnTypeGroup,
+									},
+									{
+										TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: "endpoints"},
+										Type:              qbtypes.ColumnTypeAggregation,
+									},
+								},
+								Data: [][]any{
+									{"192.168.1.1", 10},
+									{"example.com", 20},
+									{"10.0.0.1", 5},
+								},
+							},
+						},
+					},
+				},
+			},
+			expected: []*qbtypes.QueryRangeResponse{
+				{
+					Data: qbtypes.QueryData{
+						Results: []any{
+							&qbtypes.ScalarData{
+								QueryName: "endpoints",
+								Columns: []*qbtypes.ColumnDescriptor{
+									{
+										TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: derivedKeyHTTPHost},
+										Type:              qbtypes.ColumnTypeGroup,
+									},
+									{
+										TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: "endpoints"},
+										Type:              qbtypes.ColumnTypeAggregation,
+									},
+								},
+								Data: [][]any{
+									{"example.com", 20},
+								},
+							},
+						},
+					},
+				},
+			},
+		},
 	}
 
 	for _, tt := range tests {

pkg/signoz/provider.go

@@ -169,6 +169,7 @@ func NewSQLMigrationProviderFactories(
 		sqlmigration.NewAddAnonymousPublicDashboardTransactionFactory(sqlstore),
 		sqlmigration.NewAddRootUserFactory(sqlstore, sqlschema),
 		sqlmigration.NewAddUserEmailOrgIDIndexFactory(sqlstore, sqlschema),
+		sqlmigration.NewMigrateRulesV4ToV5Factory(sqlstore, telemetryStore),
 	)
 }
 

pkg/sqlmigration/066_migrate_rules_v4_to_v5_post_deprecation.go

@@ -0,0 +1,209 @@
+package sqlmigration
+
+import (
+	"context"
+	"database/sql"
+	"encoding/json"
+	"log/slog"
+
+	"github.com/SigNoz/signoz/pkg/factory"
+	"github.com/SigNoz/signoz/pkg/sqlstore"
+	"github.com/SigNoz/signoz/pkg/telemetrystore"
+	"github.com/SigNoz/signoz/pkg/transition"
+	"github.com/uptrace/bun"
+	"github.com/uptrace/bun/migrate"
+)
+
+type migrateRulesV4ToV5 struct {
+	store          sqlstore.SQLStore
+	telemetryStore telemetrystore.TelemetryStore
+	logger         *slog.Logger
+}
+
+func NewMigrateRulesV4ToV5Factory(
+	store sqlstore.SQLStore,
+	telemetryStore telemetrystore.TelemetryStore,
+) factory.ProviderFactory[SQLMigration, Config] {
+	return factory.NewProviderFactory(
+		factory.MustNewName("migrate_rules_post_deprecation"),
+		func(ctx context.Context, ps factory.ProviderSettings, c Config) (SQLMigration, error) {
+			return &migrateRulesV4ToV5{
+				store:          store,
+				telemetryStore: telemetryStore,
+				logger:         ps.Logger,
+			}, nil
+		})
+}
+
+func (migration *migrateRulesV4ToV5) Register(migrations *migrate.Migrations) error {
+	if err := migrations.Register(migration.Up, migration.Down); err != nil {
+		return err
+	}
+	return nil
+}
+
+func (migration *migrateRulesV4ToV5) getLogDuplicateKeys(ctx context.Context) ([]string, error) {
+	query := `
+		SELECT name
+		FROM (
+			SELECT DISTINCT name FROM signoz_logs.distributed_logs_attribute_keys
+			INTERSECT
+			SELECT DISTINCT name FROM signoz_logs.distributed_logs_resource_keys
+		)
+		ORDER BY name
+	`
+
+	rows, err := migration.telemetryStore.ClickhouseDB().Query(ctx, query)
+	if err != nil {
+		migration.logger.WarnContext(ctx, "failed to query log duplicate keys", "error", err)
+		return nil, nil
+	}
+	defer rows.Close()
+
+	var keys []string
+	for rows.Next() {
+		var key string
+		if err := rows.Scan(&key); err != nil {
+			migration.logger.WarnContext(ctx, "failed to scan log duplicate key", "error", err)
+			continue
+		}
+		keys = append(keys, key)
+	}
+
+	return keys, nil
+}
+
+func (migration *migrateRulesV4ToV5) getTraceDuplicateKeys(ctx context.Context) ([]string, error) {
+	query := `
+		SELECT tagKey
+		FROM signoz_traces.distributed_span_attributes_keys
+		WHERE tagType IN ('tag', 'resource')
+		GROUP BY tagKey
+		HAVING COUNT(DISTINCT tagType) > 1
+		ORDER BY tagKey
+	`
+
+	rows, err := migration.telemetryStore.ClickhouseDB().Query(ctx, query)
+	if err != nil {
+		migration.logger.WarnContext(ctx, "failed to query trace duplicate keys", "error", err)
+		return nil, nil
+	}
+	defer rows.Close()
+
+	var keys []string
+	for rows.Next() {
+		var key string
+		if err := rows.Scan(&key); err != nil {
+			migration.logger.WarnContext(ctx, "failed to scan trace duplicate key", "error", err)
+			continue
+		}
+		keys = append(keys, key)
+	}
+
+	return keys, nil
+}
+
+func (migration *migrateRulesV4ToV5) Up(ctx context.Context, db *bun.DB) error {
+	logsKeys, err := migration.getLogDuplicateKeys(ctx)
+	if err != nil {
+		return err
+	}
+
+	tracesKeys, err := migration.getTraceDuplicateKeys(ctx)
+	if err != nil {
+		return err
+	}
+
+	tx, err := db.BeginTx(ctx, nil)
+	if err != nil {
+		return err
+	}
+
+	defer func() {
+		_ = tx.Rollback()
+	}()
+
+	var rules []struct {
+		ID   string         `bun:"id"`
+		Data map[string]any `bun:"data"`
+	}
+
+	err = tx.NewSelect().
+		Table("rule").
+		Column("id", "data").
+		Scan(ctx, &rules)
+	if err != nil {
+		if err == sql.ErrNoRows {
+			return nil
+		}
+		return err
+	}
+
+	alertsMigrator := transition.NewAlertMigrateV5(migration.logger, logsKeys, tracesKeys)
+
+	count := 0
+
+	for _, rule := range rules {
+		version, _ := rule.Data["version"].(string)
+
+		if version == "v5" {
+			continue
+		}
+
+		if version == "" {
+			migration.logger.WarnContext(ctx, "unexpected empty version for rule", "rule_id", rule.ID)
+		}
+
+		migration.logger.InfoContext(ctx, "migrating rule v4 to v5", "rule_id", rule.ID, "current_version", version)
+
+		// Check if the queries envelope already exists and is non-empty
+		hasQueriesEnvelope := false
+		if condition, ok := rule.Data["condition"].(map[string]any); ok {
+			if compositeQuery, ok := condition["compositeQuery"].(map[string]any); ok {
+				if queries, ok := compositeQuery["queries"].([]any); ok && len(queries) > 0 {
+					hasQueriesEnvelope = true
+				}
+			}
+		}
+
+		if hasQueriesEnvelope {
+			// already has queries envelope, just bump version
+			// this is because user made a mistake of choosing version
+			migration.logger.InfoContext(ctx, "rule already has queries envelope, bumping version", "rule_id", rule.ID)
+			rule.Data["version"] = "v5"
+		} else {
+			// old format, run full migration
+			migration.logger.InfoContext(ctx, "rule has old format, running full migration", "rule_id", rule.ID)
+			updated := alertsMigrator.Migrate(ctx, rule.Data)
+			if !updated {
+				migration.logger.WarnContext(ctx, "expected updated to be true but got false", "rule_id", rule.ID)
+				continue
+			}
+			rule.Data["version"] = "v5"
+		}
+
+		dataJSON, err := json.Marshal(rule.Data)
+		if err != nil {
+			return err
+		}
+
+		_, err = tx.NewUpdate().
+			Table("rule").
+			Set("data = ?", string(dataJSON)).
+			Where("id = ?", rule.ID).
+			Exec(ctx)
+		if err != nil {
+			return err
+		}
+		count++
+	}
+	if count != 0 {
+		migration.logger.InfoContext(ctx, "migrate v4 alerts", "count", count)
+	}
+
+	return tx.Commit()
+}
+
+func (migration *migrateRulesV4ToV5) Down(ctx context.Context, db *bun.DB) error {
+	return nil
+}

pkg/types/ruletypes/api_params.go

@@ -355,6 +355,10 @@ func (r *PostableRule) validate() error {
 		errs = append(errs, signozError.NewInvalidInputf(signozError.CodeInvalidInput, "composite query is required"))
 	}
 
+	if r.Version != "v5" {
+		errs = append(errs, signozError.NewInvalidInputf(signozError.CodeInvalidInput, "only version v5 is supported, got %q", r.Version))
+	}
+
 	if isAllQueriesDisabled(r.RuleCondition.CompositeQuery) {
 		errs = append(errs, signozError.NewInvalidInputf(signozError.CodeInvalidInput, "all queries are disabled in rule condition"))
 	}

pkg/types/ruletypes/api_params_test.go

@@ -108,6 +108,7 @@ func TestParseIntoRule(t *testing.T) {
 				"ruleType": "threshold_rule",
 				"evalWindow": "5m",
 				"frequency": "1m",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -150,6 +151,7 @@ func TestParseIntoRule(t *testing.T) {
 			content: []byte(`{
 				"alert": "DefaultsRule",
 				"ruleType": "threshold_rule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -187,6 +189,7 @@ func TestParseIntoRule(t *testing.T) {
 			initRule: PostableRule{},
 			content: []byte(`{
 				"alert": "PromQLRule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "promql",
@@ -256,6 +259,7 @@ func TestParseIntoRuleSchemaVersioning(t *testing.T) {
 			content: []byte(`{
 				"alert": "SeverityLabelTest",
 				"schemaVersion": "v1",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -344,6 +348,7 @@ func TestParseIntoRuleSchemaVersioning(t *testing.T) {
 			content: []byte(`{
 				"alert": "NoLabelsTest",
 				"schemaVersion": "v1",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -384,6 +389,7 @@ func TestParseIntoRuleSchemaVersioning(t *testing.T) {
 			content: []byte(`{
 				"alert": "OverwriteTest",
 				"schemaVersion": "v1",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -474,6 +480,7 @@ func TestParseIntoRuleSchemaVersioning(t *testing.T) {
 			content: []byte(`{
 				"alert": "V2Test",
 				"schemaVersion": "v2",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -517,6 +524,7 @@ func TestParseIntoRuleSchemaVersioning(t *testing.T) {
 			initRule: PostableRule{},
 			content: []byte(`{
 				"alert": "DefaultSchemaTest",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -569,6 +577,7 @@ func TestParseIntoRuleSchemaVersioning(t *testing.T) {
 func TestParseIntoRuleThresholdGeneration(t *testing.T) {
 	content := []byte(`{
 		"alert": "TestThresholds",
+		"version": "v5",
 		"condition": {
 			"compositeQuery": {
 				"queryType": "builder",
@@ -639,6 +648,7 @@ func TestParseIntoRuleMultipleThresholds(t *testing.T) {
 		"schemaVersion": "v2",
 		"alert": "MultiThresholdAlert",
 		"ruleType": "threshold_rule",
+		"version": "v5",
 		"condition": {
 			"compositeQuery": {
 				"queryType": "builder",
@@ -732,6 +742,7 @@ func TestAnomalyNegationEval(t *testing.T) {
 			ruleJSON: []byte(`{
 				"alert": "AnomalyBelowTest",
 				"ruleType": "anomaly_rule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -766,6 +777,7 @@ func TestAnomalyNegationEval(t *testing.T) {
 			ruleJSON: []byte(`{
 				"alert": "AnomalyBelowTest",
 				"ruleType": "anomaly_rule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -799,6 +811,7 @@ func TestAnomalyNegationEval(t *testing.T) {
 			ruleJSON: []byte(`{
 				"alert": "AnomalyAboveTest",
 				"ruleType": "anomaly_rule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -833,6 +846,7 @@ func TestAnomalyNegationEval(t *testing.T) {
 			ruleJSON: []byte(`{
 				"alert": "AnomalyAboveTest",
 				"ruleType": "anomaly_rule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -866,6 +880,7 @@ func TestAnomalyNegationEval(t *testing.T) {
 			ruleJSON: []byte(`{
 				"alert": "AnomalyBelowAllTest",
 				"ruleType": "anomaly_rule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -901,6 +916,7 @@ func TestAnomalyNegationEval(t *testing.T) {
 			ruleJSON: []byte(`{
 				"alert": "AnomalyBelowAllTest",
 				"ruleType": "anomaly_rule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -935,6 +951,7 @@ func TestAnomalyNegationEval(t *testing.T) {
 			ruleJSON: []byte(`{
 				"alert": "AnomalyOutOfBoundsTest",
 				"ruleType": "anomaly_rule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -969,6 +986,7 @@ func TestAnomalyNegationEval(t *testing.T) {
 			ruleJSON: []byte(`{
 				"alert": "ThresholdTest",
 				"ruleType": "threshold_rule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",
@@ -1003,6 +1021,7 @@ func TestAnomalyNegationEval(t *testing.T) {
 			ruleJSON: []byte(`{
 				"alert": "ThresholdTest",
 				"ruleType": "threshold_rule",
+				"version": "v5",
 				"condition": {
 					"compositeQuery": {
 						"queryType": "builder",

tests/integration/conftest.py

@@ -1,19 +1,5 @@
-import json
-import os
-import tempfile
-
 import pytest
 
-if not os.environ.get("DOCKER_CONFIG"):
-    os.environ["DOCKER_CONFIG"] = tempfile.mkdtemp(prefix="docker-config-")
-os.environ.setdefault("DOCKER_CREDENTIAL_STORE", "")
-os.environ.setdefault("DOCKER_CREDENTIAL_HELPER", "")
-
-docker_config_path = os.path.join(os.environ["DOCKER_CONFIG"], "config.json")
-if not os.path.exists(docker_config_path):
-    with open(docker_config_path, "w", encoding="utf-8") as config_file:
-        json.dump({"auths": {}, "credsStore": ""}, config_file)
-
 pytest_plugins = [
     "fixtures.auth",
     "fixtures.clickhouse",
@@ -35,7 +21,6 @@ pytest_plugins = [
     "fixtures.notification_channel",
     "fixtures.alerts",
     "fixtures.cloudintegrations",
-    "fixtures.jsontypeexporter",
 ]
 
 
@@ -73,7 +58,7 @@ def pytest_addoption(parser: pytest.Parser):
     parser.addoption(
         "--clickhouse-version",
         action="store",
-        default="25.8.6",
+        default="25.5.6",
         help="clickhouse version",
     )
     parser.addoption(
@@ -88,5 +73,3 @@ def pytest_addoption(parser: pytest.Parser):
         default="v0.129.7",
         help="schema migrator version",
     )
-
-

tests/integration/fixtures/jsontypeexporter.py

@@ -1,408 +0,0 @@
-"""
-Simpler version of jsontypeexporter for test fixtures.
-This exports JSON type metadata to the path_types table by parsing JSON bodies
-and extracting all paths with their types, similar to how the real jsontypeexporter works.
-"""
-import datetime
-import json
-from abc import ABC
-from typing import TYPE_CHECKING, Any, Callable, Dict, Generator, List, Optional, Set, Union
-
-import numpy as np
-import pytest
-
-from fixtures import types
-
-if TYPE_CHECKING:
-    from fixtures.logs import Logs
-
-
-class JSONPathType(ABC):
-    """Represents a JSON path with its type information"""
-    path: str
-    type: str
-    last_seen: np.uint64
-
-    def __init__(
-        self,
-        path: str,
-        type: str,  # pylint: disable=redefined-builtin
-        last_seen: Optional[datetime.datetime] = None,
-    ) -> None:
-        self.path = path
-        self.type = type
-        if last_seen is None:
-            last_seen = datetime.datetime.now()
-        self.last_seen = np.uint64(int(last_seen.timestamp() * 1e9))
-
-    def np_arr(self) -> np.array:
-        """Return path type data as numpy array for database insertion"""
-        return np.array([self.path, self.type, self.last_seen])
-
-
-# Constants matching jsontypeexporter
-ARRAY_SEPARATOR = "[]."  # Used in paths like "education[].name"
-ARRAY_SUFFIX = "[]"      # Used when traversing into array element objects
-
-
-def _infer_array_type(elements: List[Any]) -> Optional[str]:
-    """
-    Infer array type from array elements, matching jsontypeexporter's inferArrayMask logic.
-    Returns None if no valid type can be inferred.
-    """
-    if len(elements) == 0:
-        return None
-
-    # Collect element types (matching Go: types := make([]pcommon.ValueType, 0, s.Len()))
-    types = []
-    for elem in elements:
-        if elem is None:
-            continue
-        if isinstance(elem, dict):
-            types.append("JSON")
-        elif isinstance(elem, str):
-            types.append("String")
-        elif isinstance(elem, bool):
-            types.append("Bool")
-        elif isinstance(elem, float):
-            types.append("Float64")
-        elif isinstance(elem, int):
-            types.append("Int64")
-
-    if len(types) == 0:
-        return None
-
-    # Get unique types (matching Go: unique := make(map[pcommon.ValueType]struct{}, len(types)))
-    unique = set(types)
-
-    # Classify types (matching Go logic)
-    has_json = "JSON" in unique
-    has_primitive = any(t in unique for t in ["String", "Bool", "Float64", "Int64"])
-
-    if has_json:
-        # If only JSON → Array(JSON) (no primitive types)
-        if not has_primitive:
-            return "Array(JSON)"
-        # If there's JSON + any primitive → Dynamic
-        return "Array(Dynamic)"
-
-    # ---- Primitive Type Resolution ----
-    if "String" in unique:
-        # Strings cannot coerce with any other primitive
-        if len(unique) > 1:
-            return "Array(Dynamic)"
-        return "Array(Nullable(String))"
-
-    if "Float64" in unique:
-        return "Array(Nullable(Float64))"
-    if "Int64" in unique:
-        return "Array(Nullable(Int64))"
-    if "Bool" in unique:
-        return "Array(Nullable(Bool))"
-
-    return "Array(Dynamic)"
-
-
-def _python_type_to_clickhouse_type(value: Any) -> str:
-    """
-    Convert Python type to ClickHouse JSON type string.
-    Maps Python types to ClickHouse JSON data types.
-    """
-    if value is None:
-        return "String"  # Default for null values
-    
-    if isinstance(value, bool):
-        return "Bool"
-    elif isinstance(value, int):
-        return "Int64"
-    elif isinstance(value, float):
-        return "Float64"
-    elif isinstance(value, str):
-        return "String"
-    elif isinstance(value, list):
-        # Use the sophisticated array type inference
-        array_type = _infer_array_type(value)
-        return array_type if array_type else "Array(Dynamic)"
-    elif isinstance(value, dict):
-        return "JSON"
-    else:
-        return "String"  # Default fallback
-
-
-def _extract_json_paths(
-    obj: Any,
-    current_path: str = "",
-    path_types: Optional[Dict[str, Set[str]]] = None,
-    level: int = 0,
-) -> Dict[str, Set[str]]:
-    """
-    Recursively extract all paths and their types from a JSON object.
-    Matches jsontypeexporter's analyzePValue logic.
-    
-    Args:
-        obj: The JSON object to traverse
-        current_path: Current path being built (e.g., "user.name")
-        path_types: Dictionary mapping paths to sets of types found
-        level: Current nesting level (for depth limiting)
-    
-    Returns:
-        Dictionary mapping paths to sets of type strings
-    """
-    if path_types is None:
-        path_types = {}
-
-    if obj is None:
-        if current_path:
-            if current_path not in path_types:
-                path_types[current_path] = set()
-            path_types[current_path].add("String")  # Null defaults to String
-        return path_types
-    
-    if isinstance(obj, dict):
-        # For objects, add the object itself and recurse into keys
-        if current_path:
-            if current_path not in path_types:
-                path_types[current_path] = set()
-            path_types[current_path].add("JSON")
-        
-        for key, value in obj.items():
-            # Build the path for this key
-            if current_path:
-                new_path = f"{current_path}.{key}"
-            else:
-                new_path = key
-            
-            # Recurse into the value
-            _extract_json_paths(value, new_path, path_types, level + 1)
-    
-    elif isinstance(obj, list):
-        # Skip empty arrays
-        if len(obj) == 0:
-            return path_types
-        
-        # Collect types from array elements (matching Go: types := make([]pcommon.ValueType, 0, s.Len()))
-        types = []
-        
-        for item in obj:
-            if isinstance(item, dict):
-                # When traversing into array element objects, use ArraySuffix ([])
-                # This matches: prefix+ArraySuffix in the Go code
-                # Example: if current_path is "education", we use "education[]" to traverse into objects
-                array_prefix = current_path + ARRAY_SUFFIX if current_path else ""
-                for key, value in item.items():
-                    if array_prefix:
-                        # Use array separator: education[].name
-                        array_path = f"{array_prefix}.{key}"
-                    else:
-                        array_path = key
-                    # Recurse without increasing level (matching Go behavior)
-                    _extract_json_paths(value, array_path, path_types, level)
-                types.append("JSON")
-            elif isinstance(item, list):
-                # Arrays inside arrays are not supported - skip the whole path
-                # Matching Go: e.logger.Error("arrays inside arrays are not supported!", ...); return nil
-                return path_types
-            elif isinstance(item, str):
-                types.append("String")
-            elif isinstance(item, bool):
-                types.append("Bool")
-            elif isinstance(item, float):
-                types.append("Float64")
-            elif isinstance(item, int):
-                types.append("Int64")
-        
-        # Infer array type from collected types (matching Go: if mask := inferArrayMask(types); mask != 0)
-        if len(types) > 0:
-            array_type = _infer_array_type(obj)
-            if array_type and current_path:
-                if current_path not in path_types:
-                    path_types[current_path] = set()
-                path_types[current_path].add(array_type)
-    
-    else:
-        # Primitive value (string, number, bool)
-        if current_path:
-            if current_path not in path_types:
-                path_types[current_path] = set()
-            obj_type = _python_type_to_clickhouse_type(obj)
-            path_types[current_path].add(obj_type)
-    
-    return path_types
-
-
-def _parse_json_bodies_and_extract_paths(
-    json_bodies: List[str],
-    timestamp: Optional[datetime.datetime] = None,
-) -> List[JSONPathType]:
-    """
-    Parse JSON bodies and extract all paths with their types.
-    This mimics the behavior of jsontypeexporter.
-    
-    Args:
-        json_bodies: List of JSON body strings to parse
-        timestamp: Timestamp to use for last_seen (defaults to now)
-    
-    Returns:
-        List of JSONPathType objects with all discovered paths and types
-    """
-    if timestamp is None:
-        timestamp = datetime.datetime.now()
-    
-    # Aggregate all paths and their types across all JSON bodies
-    all_path_types: Dict[str, Set[str]] = {}
-    
-    for json_body in json_bodies:
-        try:
-            parsed = json.loads(json_body)
-            _extract_json_paths(parsed, "", all_path_types, level=0)
-        except (json.JSONDecodeError, TypeError):
-            # Skip invalid JSON
-            continue
-    
-    # Convert to list of JSONPathType objects
-    # Each path can have multiple types, so we create one JSONPathType per type
-    path_type_objects: List[JSONPathType] = []
-    for path, types_set in all_path_types.items():
-        for type_str in types_set:
-            path_type_objects.append(
-                JSONPathType(path=path, type=type_str, last_seen=timestamp)
-            )
-    
-    return path_type_objects
-
-
-@pytest.fixture(name="export_json_types", scope="function")
-def export_json_types(
-    clickhouse: types.TestContainerClickhouse,
-    request: pytest.FixtureRequest,  # To access migrator fixture
-) -> Generator[Callable[[Union[List[JSONPathType], List[str], List[Any]]], None], Any, None]:
-    """
-    Fixture for exporting JSON type metadata to the path_types table.
-    This is a simpler version of jsontypeexporter for test fixtures.
-    
-    The function can accept:
-    1. List of JSONPathType objects (manual specification)
-    2. List of JSON body strings (auto-extract paths)
-    3. List of Logs objects (extract from body_json field)
-    
-    Usage examples:
-        # Manual specification
-        export_json_types([
-            JSONPathType(path="user.name", type="String"),
-            JSONPathType(path="user.age", type="Int64"),
-        ])
-        
-        # Auto-extract from JSON strings
-        export_json_types([
-            '{"user": {"name": "alice", "age": 25}}',
-            '{"user": {"name": "bob", "age": 30}}',
-        ])
-        
-        # Auto-extract from Logs objects
-        export_json_types(logs_list)
-    """
-    # Ensure migrator has run to create the table
-    try:
-        request.getfixturevalue("migrator")
-    except Exception:
-        # If migrator fixture is not available, that's okay - table might already exist
-        pass
-    
-    def _export_json_types(
-        data: Union[List[JSONPathType], List[str], List[Any]],  # List[Logs] but avoiding circular import
-    ) -> None:
-        """
-        Export JSON type metadata to signoz_metadata.distributed_json_path_types table.
-        This table stores path and type information for body JSON fields.
-        """
-        path_types: List[JSONPathType] = []
-        
-        if len(data) == 0:
-            return
-        
-        # Determine input type and convert to JSONPathType list
-        first_item = data[0]
-        
-        if isinstance(first_item, JSONPathType):
-            # Already JSONPathType objects
-            path_types = data  # type: ignore
-        elif isinstance(first_item, str):
-            # List of JSON strings - parse and extract paths
-            path_types = _parse_json_bodies_and_extract_paths(data)  # type: ignore
-        else:
-            # Assume it's a list of Logs objects - extract body_json
-            json_bodies: List[str] = []
-            for log in data:  # type: ignore
-                # Try to get body_json attribute
-                if hasattr(log, "body_json") and log.body_json:
-                    json_bodies.append(log.body_json)
-                elif hasattr(log, "body") and log.body:
-                    # Fallback to body if body_json not available
-                    try:
-                        # Try to parse as JSON
-                        json.loads(log.body)
-                        json_bodies.append(log.body)
-                    except (json.JSONDecodeError, TypeError):
-                        pass
-            
-            if json_bodies:
-                path_types = _parse_json_bodies_and_extract_paths(json_bodies)
-        
-        if len(path_types) == 0:
-            return
-
-        clickhouse.conn.insert(
-            database="signoz_metadata",
-            table="distributed_json_path_types",
-            data=[path_type.np_arr() for path_type in path_types],
-            column_names=[
-                "path",
-                "type",
-                "last_seen",
-            ],
-        )
-
-    yield _export_json_types
-
-    # Cleanup - truncate the local table after tests (following pattern from logs fixture)
-    clickhouse.conn.query(
-        f"TRUNCATE TABLE signoz_metadata.json_path_types ON CLUSTER '{clickhouse.env['SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_CLUSTER']}' SYNC"
-    )
-
-
-@pytest.fixture(name="export_promoted_paths", scope="function")
-def export_promoted_paths(
-    clickhouse: types.TestContainerClickhouse,
-    request: pytest.FixtureRequest,  # To access migrator fixture
-) -> Generator[Callable[[List[str]], None], Any, None]:
-    """
-    Fixture for exporting promoted JSON paths to the promoted paths table.
-    """
-    # Ensure migrator has run to create the table
-    try:
-        request.getfixturevalue("migrator")
-    except Exception:
-        # If migrator fixture is not available, that's okay - table might already exist
-        pass
-
-    def _export_promoted_paths(paths: List[str]) -> None:
-        if len(paths) == 0:
-            return
-
-        now_ms = int(datetime.datetime.now().timestamp() * 1000)
-        rows = [(path, now_ms) for path in paths]
-        clickhouse.conn.insert(
-            database="signoz_metadata",
-            table="distributed_json_promoted_paths",
-            data=rows,
-            column_names=[
-                "path",
-                "created_at",
-            ],
-        )
-
-    yield _export_promoted_paths
-
-    clickhouse.conn.query(
-        f"TRUNCATE TABLE signoz_metadata.json_promoted_paths ON CLUSTER '{clickhouse.env['SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_CLUSTER']}' SYNC"
-    )

tests/integration/fixtures/logs.py

@@ -100,8 +100,6 @@ class Logs(ABC):
     severity_text: str
     severity_number: np.uint8
     body: str
-    body_json: str
-    body_json_promoted: str
     attributes_string: dict[str, str]
     attributes_number: dict[str, np.float64]
     attributes_bool: dict[str, bool]
@@ -121,8 +119,6 @@ class Logs(ABC):
         resources: dict[str, Any] = {},
         attributes: dict[str, Any] = {},
         body: str = "default body",
-        body_json: Optional[str] = None,
-        body_json_promoted: Optional[str] = None,
         severity_text: str = "INFO",
         trace_id: str = "",
         span_id: str = "",
@@ -166,32 +162,6 @@ class Logs(ABC):
 
         # Set body
         self.body = body
-        
-        # Set body_json - if body is JSON, parse and stringify it, otherwise use empty string
-        # ClickHouse accepts String input for JSON column
-        if body_json is not None:
-            self.body_json = body_json
-        else:
-            # Try to parse body as JSON, if successful use it, otherwise use empty string
-            try:
-                json.loads(body)
-                self.body_json = body
-            except (json.JSONDecodeError, TypeError):
-                self.body_json = ""
-        
-        # Set body_json_promoted - must be valid JSON
-        # Tests will explicitly pass promoted column's content, but we validate it
-        if body_json_promoted is not None:
-            # Validate that it's valid JSON
-            try:
-                json.loads(body_json_promoted)
-                self.body_json_promoted = body_json_promoted
-            except (json.JSONDecodeError, TypeError):
-                # If invalid, default to empty JSON object
-                self.body_json_promoted = "{}"
-        else:
-            # Default to empty JSON object (valid JSON)
-            self.body_json_promoted = "{}"
 
         # Process resources and attributes
         self.resources_string = {k: str(v) for k, v in resources.items()}
@@ -349,8 +319,6 @@ class Logs(ABC):
                 self.severity_text,
                 self.severity_number,
                 self.body,
-                self.body_json,
-                self.body_json_promoted,
                 self.attributes_string,
                 self.attributes_number,
                 self.attributes_bool,
@@ -495,8 +463,6 @@ def insert_logs(
                 "severity_text",
                 "severity_number",
                 "body",
-                "body_json",
-                "body_json_promoted",
                 "attributes_string",
                 "attributes_number",
                 "attributes_bool",

tests/integration/fixtures/migrator.py

@@ -20,14 +20,12 @@ def migrator(
     """
 
     def create() -> None:
-        # Hardcode version for new QB tests
-        version = "v0.129.13-rc.1"
+        version = request.config.getoption("--schema-migrator-version")
         client = docker.from_env()
 
         container = client.containers.run(
             image=f"signoz/signoz-schema-migrator:{version}",
             command=f"sync --replication=true --cluster-name=cluster --up= --dsn={clickhouse.env["SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN"]}",
-            environment={"ENABLE_LOGS_MIGRATIONS_V2": "1"},
             detach=True,
             auto_remove=False,
             network=network.id,
@@ -46,7 +44,6 @@ def migrator(
         container = client.containers.run(
             image=f"signoz/signoz-schema-migrator:{version}",
             command=f"async --replication=true --cluster-name=cluster --up= --dsn={clickhouse.env["SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN"]}",
-            environment={"ENABLE_LOGS_MIGRATIONS_V2": "1"},
             detach=True,
             auto_remove=False,
             network=network.id,

tests/integration/fixtures/signoz.py

@@ -73,7 +73,6 @@ def signoz(  # pylint: disable=too-many-arguments,too-many-positional-arguments
                 "SIGNOZ_ALERTMANAGER_SIGNOZ_POLL__INTERVAL": "5s",
                 "SIGNOZ_ALERTMANAGER_SIGNOZ_ROUTE_GROUP__WAIT": "1s",
                 "SIGNOZ_ALERTMANAGER_SIGNOZ_ROUTE_GROUP__INTERVAL": "5s",
-                "BODY_JSON_QUERY_ENABLED": "true",
             }
             | sqlstore.env
             | clickhouse.env