Compare commits

..

9 Commits

Author SHA1 Message Date
aks07
013aef1c85 refactor(quick-filters): use AuthZButton for the settings gear 2026-09-10 00:13:20 +05:30
aks07
f3c95c2593 fix(quick-filters): keep header action icons in the previous colour 2026-09-10 00:13:18 +05:30
aks07
dd866dc917 fix(quick-filters): show the settings gear whenever a signal is passed 2026-09-10 00:13:15 +05:30
aks07
f5adafcb11 chore(quick-filters): drop unused settings announcement localStorage key 2026-09-09 16:01:37 +05:30
aks07
72baa61711 feat(quick-filters): gate quick filters on quick-filter read and update
Filters content is behind read, settings gear behind read + update,
replacing the admin role check. Header actions moved to the ui Button so
AuthZTooltip can actually disable the gear. Dropped the old quick filters
settings announcement from the gear.
2026-09-09 16:01:37 +05:30
aks07
ec4512ae6a feat(authz): add quick-filter permission constants 2026-09-09 16:01:37 +05:30
nityanandagohain
6e7a2c1e1e chore: quickfilter authz 2026-09-09 16:01:37 +05:30
Gaurav Tewari
99dcd79979 chore(llm-observability): fork shared traces explorer files (#12795)
Some checks failed
build-staging / prepare (push) Has been cancelled
build-staging / js-build (push) Has been cancelled
build-staging / go-build (push) Has been cancelled
build-staging / staging (push) Has been cancelled
cacheci / tests (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
#### Description

- Copies, unmodified, the traces explorer files that the AI
observability explorer is about to diverge from. The point is the *next*
PR (#12796): with this baseline landed, that diff shows only the real
changes instead of ~1000 lines of brand-new files.
- No behaviour change. Nothing outside the fork imports the new copies
yet — the explorer's views are wired to them in #12796, alongside the
changes that need them.


Sources, all taken from this PR's merge base:

| Copied to `container/LLMObservability/` | Source |
| --- | --- |
| `ToolbarActions/LeftToolbarActions.tsx` |
`container/QueryBuilder/components/ToolbarActions/LeftToolbarActions.tsx`
|
| `ToolbarActions/ToolbarActions.styles.scss` |
`container/QueryBuilder/components/ToolbarActions/ToolbarActions.styles.scss`
|
| `Explorer/aiActions.ts` | `pages/TracesExplorer/aiActions.ts` |
| `Explorer/Controls/*` | `container/TracesExplorer/Controls/*` |
| `Explorer/TraceLoading/*` | `container/TracesExplorer/TraceLoading/*`
|
| `Explorer/TracesTable/*` | `container/TracesExplorer/TracesTable/*` |
| `Explorer/ListView/utils.tsx` |
`container/TracesExplorer/ListView/utils.tsx` |

#### Issues closed by this PR

#### Screenshots / Screen Recordings

#### Additional Information

---------

Co-authored-by: Gaurav Tewari <tewarig@users.noreply.github.com>
2026-09-09 09:40:13 +00:00
Srikanth Chekuri
d5af6f6d6b feat(telemetrymetadata): fields keys and values for the quick filters migration (#12794)
Some checks failed
build-staging / prepare (push) Has been cancelled
build-staging / js-build (push) Has been cancelled
build-staging / go-build (push) Has been cancelled
build-staging / staging (push) Has been cancelled
cacheci / tests (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
#### Description

- `fields/values` returns `boolValues` for bool fields: static span
fields such as `has_error`, stored bool attributes (decided by a
row-existence check, so they take no limit slots), and keys the caller
declares bool. `isRoot` and `isEntryPoint` return `true` only, matching
the condition builder.
- `fields/values` honors `startUnixMilli` for logs and traces against
the tag tables. The predicate is `toDate(unix_milli / 1000) >=
toDate(start)`, the partition expression, because the tag tables
deduplicate a value per day and keep whichever hour was inserted last.
For traces this means "span started on or after the start's day", for
logs "ingested on or after the start's day". `endUnixMilli` is ignored.
- `fields/keys` honors `fieldContext` and `fieldDataType` for the static
intrinsic and calculated keys, matches them case-insensitively, types
`isRoot`/`isEntryPoint` as bool, gives meter keys a data type, filters
the traces keys table by tag type for the scope context, and no longer
returns early for the log and scope contexts (logs and audit).
- The span context keeps returning attribute keys, since queries spell
span attributes as `span.<name>`.
- Default quick filters name the span and log fields the way the fields
API serves them (`has_error` as a span bool, `name`, `duration_nano`,
`response_status_code`, `http_host`, `http_url`, `trace_id` as span
fields, `severity_text` as a log field). Migration 126 rewrites the
static fields of every stored quick filter the same way, whatever
context the legacy seeds gave them. Deprecated names such as `hasError`
are not served.
- The API-backed checkbox hook merges `boolValues` into its options; the
`has_error` title is kept.
- The fields/values handler no longer drops `boolValues` from the
response.


🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-09-08 22:11:27 +00:00
119 changed files with 5506 additions and 3143 deletions

View File

@@ -102,6 +102,7 @@ func runGenerateAuthz(_ context.Context) error {
coretypes.NewResourceRef(coretypes.ResourceTelemetryResourceTraces).String(): true,
coretypes.NewResourceRef(coretypes.ResourceTelemetryResourceMetrics).String(): true,
coretypes.NewResourceRef(coretypes.ResourceTelemetryResourceMeterMetrics).String(): true,
coretypes.NewResourceRef(coretypes.ResourceMetaResourceQuickFilter).String(): true,
}
allowedTypes := map[string]bool{}

View File

@@ -54,93 +54,50 @@ The `fieldContexts` map includes aliases (`tag` -> `attribute`, `spanfield` -> `
## The Abstraction Stack
The query pipeline has three layers. The generic layer is written one time, in `pkg/querybuilder`. A storage is written one time per signal. The statement builders compose them. Each layer depends only on the layer below it. This layering is intentional and must be preserved.
The query pipeline is built from four interfaces that compose vertically. Each layer has a single responsibility. Each layer depends only on the layers below it. This layering is intentional and must be preserved.
```
StatementBuilder <- Composes one query into executable SQL
├── AggExprRewriter <- Rewrites aggregation expressions through the generic layer
├── filter visitor <- Parses the filter expression and compiles it one term at a time
└── querybuilder (generic) <- Resolution, the condition builder, the column expression builder
└── Storage <- What one signal's tables can answer about one field key
StatementBuilder <- Orchestrates everything into executable SQL
├── AggExprRewriter <- Rewrites aggregation expressions (maps field refs to columns)
├── ConditionBuilder <- Builds WHERE predicates (field + operator + value -> SQL)
└── FieldMapper <- Maps TelemetryFieldKey -> ClickHouse column expression
```
### Storage
### FieldMapper
**Contract:** `qbtypes.Storage` in `pkg/types/querybuildertypes/querybuildertypesv5/qb.go`. One implementation per signal: traces, logs, metrics, audit, rule state history, the resource fingerprint sub-query, and the related-values metadata.
**Contract:** Given a `TelemetryFieldKey`, return a ClickHouse column expression that yields the value for that field when used in a SELECT.
A storage answers four questions and nothing else:
**Principle:** This is the *only* place where field-to-column translation happens. No other layer should contain knowledge of how fields map to storage. If you need a column expression, go through the FieldMapper.
- `Read(key)`: one call for one field key. It returns the bare SQL read (no alias, no guard, no cast), the membership test (`Presence`, arg-free so it can sit inside guards) and its negation in the storage's own form (`Absence`), what a row without the key reads (`Absent`, below), whether the stage cast must leave the native type alone (`KeepType`: time columns, metrics labels), and whether the key filters but cannot be selected (`FilterOnly`: the legacy string body). It honors the materialization and the evolutions the key carries.
- `Fallback(key, operator, value)`: the field keys that could hold a key metadata does not report: column aliases, the type variants of a map read, body paths, and virtual keys that compile to structural predicates (a span search scope, a full-text search over a scope).
- `Traits()`: the storage's part in the resource fingerprint split, whether it supports body functions, what it does with an unknown key, and which contexts mean "this signal's own record".
- `Compile`: the one override, for a storage with its own condition language (the body JSON language in logs, the index hints of the resource fingerprint, the polarity form of the related values, the String-typed labels of metrics). Every other storage returns `querybuilder.SharedCondition`.
**Why:** The user says `http.request.method`. ClickHouse might store it as `attributes_string['http.request.method']`, or as a materialized column `` `attribute_string_http$$request$$method` ``, or via a JSON access path in a body column. This variation is entirely contained within the FieldMapper. Everything above it is storage-agnostic.
**Principle:** A storage describes its field keys. It never decides a guard, an ambiguity, a warning, or the shape of a fold. Those decisions are derived one time, in the generic layer, from those descriptions.
### ConditionBuilder
### Absent
**Contract:** Given a field key, an operator, and a value, produce a valid SQL predicate for a WHERE clause.
`Read` returns the field's `Absent`: what a row without the field reads. It is a property of the read, not of the column. `resource.x::String` reads the empty string for an absent row, the multi-era fold `multiIf(..., NULL)` reads NULL, and a table column always reads a real value. Every guard derives from it:
**Dependency:** Uses FieldMapper for the left-hand side of the condition.
| WhenAbsent | Absent row reads | Positive filter | Raw select | Multi-candidate column | Field keys |
|---|---|---|---|---|---|
| `AlwaysPresent` | a real value | no guard | no guard | no branch, ends the candidate list | table columns |
| `AbsentIsSentinel` | `''`, 0, false, and that is not a value | exists guard | exists guard | presence branch | map attributes, cast JSON paths, string families |
| `AbsentIsNull` | NULL | no guard | no guard | presence branch | multi-era folds, body JSON paths, numeric families |
| `AbsentIsValue` | `''`, and that is the keyless contract | no guard | no guard | no presence branch | metrics labels, rule state history labels |
**Principle:** The ConditionBuilder owns all the complexity of operator semantics, i.e type casting, array operators (`hasAny`/`hasAll` vs `=`), existence checks, and negative operator behavior. This complexity must not leak upward into the StatementBuilder.
### The generic layer
### AggExprRewriter
`pkg/querybuilder` needs two inputs, made one time per request:
**Contract:** Given a user-facing aggregation expression like `sum(duration_nano)`, resolve field references within it and produce valid ClickHouse SQL.
- The metadata keys: `keys := metadataStore.GetKeysMulti(...)`, the field keys the metadata store reports for the query's names, as `map[name][]*TelemetryFieldKey`.
- `q := querybuilder.NewQueryInfo(ctx, orgID, fl, signal, metric, startNs, endNs)`: the time range every read needs, the signal and the queried metric that family admission needs, and the query-path flags (`FamiliesOn`, `BodyJSONOn`), evaluated one time.
**Dependency:** Uses FieldMapper to resolve field names within expressions.
The functions, from the outside in:
**Principle:** Aggregation expressions are user-authored strings that contain field references. The rewriter parses them, identifies field references, resolves each through the FieldMapper, and reassembles the expression.
| Function | Does |
|---|---|
| `PrepareWhereClause(query, opts)` | The filter visitor. Parses the filter grammar and compiles each term through `RejectsBodyFunction`, `Resolve`, and `Condition`. Returns the WHERE clause, the warnings, and the cost-guard flag. |
| `NewAggExprRewriter(settings, fullTextColumn, storage, fl, signal)` | Parses an aggregation expression such as `sum(duration_nano)` and resolves each field reference through `ResolveColumn`. |
| `ResolveColumn(ctx, q, storage, key, target, metadata)` | `Resolve` with `FilterOperatorUnknown`, then `Column`. The column stages (raw select, order by, group by, aggregation arguments) call it. |
| `Resolve(ctx, q, storage, key, operator, value, metadata)` | One requested key to its meanings in this storage (see "A resolved key"). |
| `Condition(ctx, q, storage, resolved, dropResourceFields, operator, value, sb)` | A resolved key to the conditions of one filter term: the split narrows the fields, and each field compiles through `storage.Compile`. |
| `Conditions(...)` | `RejectsBodyFunction`, `Resolve`, and `Condition` in one call, for callers outside the visitor: the related-values metadata, the scoped traces predicate resolver, tests. |
| `Column(ctx, q, storage, resolved, target)` | A resolved key to one bare column expression. The caller aliases. |
| `RejectsBodyFunction(traits, operator)` | Before resolution: a storage without body functions (`has`, `hasAny`, `hasAll`, `hasToken`, `search`) errors; the fingerprint side of a split skips the term, because the main query evaluates it. After resolution, `Condition` errors when `has`, `hasAny`, `hasAll`, or `hasToken` lands on a map-backed key (resource, attribute, scope), before the split could drop it. |
| `SharedCondition(...)` | The `Compile` of every storage without its own condition language: `LogicalRead`, the shared data-type collision cast, `OperatorCondition`, then the guard rule. |
| `OperatorCondition(...)` | The operator switch over an already cast read. A storage with its own cast policy composes with it. |
| `LogicalRead(...)` | The only place family expressions are built. A single-member field reads through its member. A family merges the member reads current-first (`COALESCE(NULLIF(m1, ''), NULLIF(m2, ''), '')` for strings, `multiIf` with a NULL tail for numbers), ORs the member presence tests, and reads for a row without any member what the merge's tail reads. A member with a value map reads through `TransformRead`. `NOT EXISTS` is the read's `Absence`, the storage's own negated form. |
### StatementBuilder
### A resolved key
**Contract:** Given a complete `QueryBuilderQuery`, a time range, and a request type, produces an executable SQL statement.
`Resolve` turns one requested key into a `Resolved` value. It is the only thing the condition builder and the column expression builder receive. Compile it with the operator and value it was resolved with: the stage is the operand, and a nil value means a column or presence use.
**Dependency:** Uses all three abstractions above.
```go
type Resolved struct {
Key *TelemetryFieldKey // the spelling the request used
Fields []*LogicalField // its meanings in this storage, one per interpretation
FromFallback bool // the fields came from the storage's Fallback, not from metadata matches
Ambiguous bool // the matches held several interpretations
Skipped bool // the storage contributes nothing for this key
Warnings []string // the warnings to surface: ambiguity, not-found
}
```
A `LogicalField` is one meaning: one name, context, and data type, backed by one or more physical members. A family (one field with several spellings) is one logical field with several members, current spelling first. Ambiguity (one name, different fields) is several logical fields.
The resolution order is the same for every storage and every stage:
1. **Own context.** A key under one of the storage's own contexts (`span.x`, `log.x`) matches its own context first. Only when nothing matches does it look up as if it had no context, so `span.http.method` corrects to the attribute. Strict contexts (`resource.`, `attribute.`, `scope.`, `body.`) are honored as written.
2. **Matches.** The metadata keys under the key's spellings, grouped into families when the flag is on. Each combination of context and data type is one interpretation.
3. **Ambiguity.** A filter settles several interpretations by resource over attribute, with a warning. A column stage keeps every interpretation in metadata order and folds them, so a select or a group by sees the value wherever it is.
4. **Intrinsic column first**, bare keys only. A column every row has leads the list, whether metadata reports it or the storage's `Fallback` does. A `Fallback` column key carries the data type its column reads as (`querybuilder.ColumnDataType`), and a same-named metadata key of a contradicting type drops; a time column has no field data type and merges none. A metadata gap degrades to the correct column, never to a corrupt metadata key.
5. **Fallback.** With no match, the storage's fallback keys for the key. When the storage ignores unknown keys (a side query whose main query owns the error), the key is `Skipped`. Otherwise a key nothing can serve is an error with suggestions. The not-found warning fires only when every fallback key is a guess, that is, none of them is always present.
The condition builder then applies the fingerprint split (`MainOfSplit` drops the resource fields the sub-query serves and keeps fallback keys; `FingerprintOfSplit` keeps resource fields only), compiles each field, and the visitor joins the per-field conditions by the operator's polarity. The column expression builder reads each field through `LogicalRead`, casts for the coerced stages unless the read keeps its type, guards by `Absent`, and renders one candidate bare or several as `multiIf(..., NULL)`. A filter-only candidate drops; the error surfaces only when none remains.
**Principle:** This is the composition layer. It does not contain field mapping logic, condition building logic, or expression rewriting logic. It orchestrates the other abstractions. If you find storage-specific logic creeping into the StatementBuilder, push it down into the appropriate abstraction.
### Invariant: No layer skipping
A statement builder must not spell a column or a condition. It calls `ResolveColumn` and the filter visitor. A storage must not decide a guard or an ambiguity. It declares `Absent` and answers the four questions. Skipping layers recreates the per-signal copies the contract removed.
The StatementBuilder must not call FieldMapper directly to build conditions, it goes through the ConditionBuilder. The AggExprRewriter must not hardcode column names, it goes through the FieldMapper. Skipping layers creates hidden coupling and makes the system fragile to storage changes.
---
@@ -162,15 +119,14 @@ Only additive/counting aggregations (`count`, `count_distinct`, `sum`, `rate`) d
**Enforcement:** `GetQueriesSupportingZeroDefault` determines which queries can default to zero. The `FormulaEvaluator` consumes this via `canDefaultZero`. Changes to aggregation handling must preserve this distinction.
### Constraint: The exists guard derives from the operator and from the field
### Constraint: Existence semantics differ for positive vs negative operators
- **Positive operators** (`=`, `>`, `LIKE`, `IN`, etc.) implicitly assert field existence for a field that reads a sentinel when absent. `http.method = GET` on a map attribute means "the field exists AND equals GET".
- **Negative operators** (`!=`, `NOT IN`, `NOT LIKE`, etc.) never add an existence check. `http.method != GET` includes records where the field doesn't exist at all.
- A field that reads NULL when absent, and a table column, take no guard on any operator: the comparison already excludes the absent row, or there is no absent row.
- **Positive operators** (`=`, `>`, `LIKE`, `IN`, etc.) implicitly assert field existence. `http.method = GET` means "the field exists AND equals GET".
- **Negative operators** (`!=`, `NOT IN`, `NOT LIKE`, etc.) do **not** add an existence check. `http.method != GET` includes records where the field doesn't exist at all.
**Why:** The user's intent with negative operators is ambiguous. Rather than guess, we take the broader interpretation. Users can add an explicit `EXISTS` filter if they want the narrower one. The operator side is declared in `AddDefaultExistsFilter`; the field side is the `Absent` a storage returns from `Exists`.
**Why:** The user's intent with negative operators is ambiguous. Rather than guess, we take the broader interpretation. Users can add an explicit `EXISTS` filter if they want the narrower one. This is documented in `AddDefaultExistsFilter`.
**Consequence:** Any new operator must declare its existence behavior in `AddDefaultExistsFilter`. Any new read must declare what an absent row reads. Never add a guard by hand in a storage.
**Consequence:** Any new operator must declare its existence behavior in `AddDefaultExistsFilter`. Do not add operators without considering this.
### Constraint: Post-processing functions operate on result sets, not in SQL
@@ -232,11 +188,11 @@ The `MetadataStore` interface provides runtime field discovery and type resoluti
The same name can map to multiple `TelemetryFieldKey` variants (different contexts, different types). The metadata store returns *all* variants. Resolution to a single field happens during query building, using the query's signal and any explicit context/type hints from the user.
**Consequence:** Code that calls `GetKey` or `GetKeys` must handle multiple results. Do not assume a name maps to a single field. `querybuilder.Resolve` is where the variants settle: it returns every interpretation as a `LogicalField`, marks the result `Ambiguous`, and carries the warning.
**Consequence:** Code that calls `GetKey` or `GetKeys` must handle multiple results. Do not assume a name maps to a single field.
### Principle: Materialized fields are a performance optimization, not a semantic distinction
A materialized field and its non-materialized equivalent represent the same logical field. The `Materialized` flag tells the storage's `Read` to generate a simpler column expression. The user should never need to know whether a field is materialized.
A materialized field and its non-materialized equivalent represent the same logical field. The `Materialized` flag tells the FieldMapper to generate a simpler column expression. The user should never need to know whether a field is materialized.
### Principle: JSON body fields require access plans
@@ -247,14 +203,14 @@ Fields inside JSON body columns (`body.response.errors[].code`) need pre-compute
## Summary of Inviolable Rules
1. **User-facing types never contain ClickHouse column names or SQL fragments.**
2. **Field-to-column translation only happens in a Storage (`Read`, `Fallback`).**
2. **Field-to-column translation only happens in FieldMapper.**
3. **Normalization happens once at the API boundary, never deeper.**
4. **Historical aliases in fieldContexts and fieldDataTypes must not be removed.**
5. **Formula evaluation stays in Go — do not push it into ClickHouse JOINs.**
6. **Zero-defaulting is aggregation-type-dependent — do not universally default to zero.**
7. **The exists guard derives from `AddDefaultExistsFilter` and `Absent`; positive operators guard sentinel reads, negative operators never guard.**
7. **Positive operators imply existence, negative operators do not.**
8. **Post-processing functions operate on Go result sets, not in SQL.**
9. **All user-facing types reject unknown JSON fields with suggestions.**
10. **Validation rules are gated by request type.**
11. **Query names must be unique within a composite query.**
12. **The three-layer abstraction stack (Storage -> querybuilder generic layer -> StatementBuilder) must not be bypassed or flattened. A storage describes its field keys; the generic layer decides.**
12. **The four-layer abstraction stack (FieldMapper -> ConditionBuilder -> AggExprRewriter -> StatementBuilder) must not be bypassed or flattened.**

View File

@@ -20,7 +20,6 @@ export const Logout = async (): Promise<void> => {
deleteLocalStorageKey(LOCALSTORAGE.LOGGED_IN_USER_NAME);
deleteLocalStorageKey(LOCALSTORAGE.CHAT_SUPPORT);
deleteLocalStorageKey(LOCALSTORAGE.USER_ID);
deleteLocalStorageKey(LOCALSTORAGE.QUICK_FILTERS_SETTINGS_ANNOUNCEMENT);
window.dispatchEvent(new CustomEvent('LOGOUT'));
history.push(ROUTES.LOGIN);
};

View File

@@ -34,6 +34,7 @@ export function mockFieldsValuesAPI(response: {
relatedValues?: (string | null)[];
stringValues?: (string | null)[];
numberValues?: (number | null)[];
boolValues?: (boolean | null)[];
}): void {
server.use(
rest.get('http://localhost/api/v1/fields/values', (_, res, ctx) =>
@@ -46,6 +47,7 @@ export function mockFieldsValuesAPI(response: {
relatedValues: response.relatedValues ?? [],
stringValues: response.stringValues ?? [],
numberValues: response.numberValues ?? [],
boolValues: response.boolValues ?? [],
},
},
}),

View File

@@ -92,8 +92,12 @@ export function useFieldValues({
values.numberValues
?.filter((value): value is number => value !== null && value !== undefined)
.map((value) => value.toString()) || [];
const boolValues =
values.boolValues
?.filter((value): value is boolean => value !== null && value !== undefined)
.map((value) => value.toString()) || [];
return [...stringValues, ...numberValues];
return [...stringValues, ...numberValues, ...boolValues];
}, [data]);
return { relatedValues, allValues, isLoading, isFetching };

View File

@@ -95,6 +95,7 @@
gap: 10px;
width: 100%;
justify-content: flex-end;
--button-variant-link-color: var(--l1-foreground);
.divider-filter {
width: 1px;

View File

@@ -15,28 +15,30 @@ import {
ComboboxTrigger,
} from '@signozhq/ui/combobox';
import { Skeleton, Tooltip } from 'antd';
import { Button } from '@signozhq/ui/button';
import { Switch } from '@signozhq/ui/switch';
import { Typography } from '@signozhq/ui/typography';
import getLocalStorageKey from 'api/browser/localstorage/get';
import setLocalStorageKey from 'api/browser/localstorage/set';
import logEvent from 'api/common/logEvent';
import classNames from 'classnames';
import OverlayScrollbar from 'components/OverlayScrollbar/OverlayScrollbar';
import { LOCALSTORAGE } from 'constants/localStorage';
import { PANEL_TYPES } from 'constants/queryBuilder';
import { useApiMonitoringParams } from 'container/ApiMonitoring/queryParams';
import { useQueryBuilder } from 'hooks/queryBuilder/useQueryBuilder';
import { isFunction, isNull } from 'lodash-es';
import { useAppContext } from 'providers/App/App';
import { AuthZGuardContent } from 'lib/authz/components/AuthZGuard/AuthZGuardContent';
import AuthZButton from 'lib/authz/components/AuthZButton/AuthZButton';
import { useAuthZ } from 'lib/authz/hooks/useAuthZ/useAuthZ';
import {
QuickFilterManagePermissions,
QuickFilterReadPermission,
} from 'lib/authz/hooks/useAuthZ/permissions/quick-filter.permissions';
import { isFunction } from 'lodash-es';
import { Query } from 'types/api/queryBuilder/queryBuilderData';
import { USER_ROLES } from 'types/roles';
import Checkbox from './FilterRenderers/Checkbox/Checkbox';
import CheckboxV2 from './FilterRenderers/Checkbox/v2/CheckboxFilterV2';
import Duration from './FilterRenderers/Duration/Duration';
import Slider from './FilterRenderers/Slider/Slider';
import useFilterConfig from './hooks/useFilterConfig';
import AnnouncementTooltip from './QuickFiltersSettings/AnnouncementTooltip';
import QuickFiltersSettings from './QuickFiltersSettings/QuickFiltersSettings';
import { FiltersType, IQuickFiltersProps, QuickFiltersSource } from './types';
@@ -55,9 +57,7 @@ export default function QuickFilters(props: IQuickFiltersProps): JSX.Element {
showQueryName = true,
useFieldApis,
} = props;
const { user } = useAppContext();
const [isSettingsOpen, setIsSettingsOpen] = useState(false);
const isAdmin = user.role === USER_ROLES.ADMIN;
const [params, setParams] = useApiMonitoringParams();
const showIP = params.showIP ?? true;
@@ -68,6 +68,12 @@ export default function QuickFilters(props: IQuickFiltersProps): JSX.Element {
refetchCustomFilters,
isCustomFiltersLoading,
} = useFilterConfig({ signal, config });
const {
deniedPermissions: deniedSettingsPermissions,
isLoading: isSettingsChecking,
} = useAuthZ(QuickFilterManagePermissions, { enabled: isDynamicFilters });
const isSettingsDisabled =
isSettingsChecking || deniedSettingsPermissions.length > 0;
const {
currentQuery,
@@ -107,16 +113,6 @@ export default function QuickFilters(props: IQuickFiltersProps): JSX.Element {
const shouldShowDropdownInListView =
isListView && source === QuickFiltersSource.TRACES_EXPLORER;
const showAnnouncementTooltip = useMemo(() => {
const localStorageValue = getLocalStorageKey(
LOCALSTORAGE.QUICK_FILTERS_SETTINGS_ANNOUNCEMENT,
);
if (!isNull(localStorageValue)) {
return !(localStorageValue === 'false');
}
return true;
}, []);
const activeQueryIndex = useMemo(() => {
if (isListView) {
return source === QuickFiltersSource.TRACES_EXPLORER
@@ -232,49 +228,49 @@ export default function QuickFilters(props: IQuickFiltersProps): JSX.Element {
const renderRightActions = (): JSX.Element => (
<section className="right-actions">
<Tooltip title="Reset All">
<div className="right-action-icon-container">
<RefreshCw className="sync-icon" size="md" onClick={handleReset} />
</div>
<Button
variant="link"
color="secondary"
aria-label="Reset All"
className="right-action-icon-container"
onClick={handleReset}
prefix={<RefreshCw className="sync-icon" size="md" />}
/>
</Tooltip>
{showFilterCollapse && (
<Tooltip title="Collapse Filters">
<div className="right-action-icon-container">
<ArrowUpToLine
style={{ rotate: '270deg', cursor: 'pointer' }}
size="md"
onClick={handleFilterVisibilityChange}
/>
</div>
<Button
variant="link"
color="secondary"
aria-label="Collapse Filters"
className="right-action-icon-container"
onClick={handleFilterVisibilityChange}
prefix={<ArrowUpToLine style={{ rotate: '270deg' }} size="md" />}
/>
</Tooltip>
)}
{isDynamicFilters && isAdmin && (
<Tooltip title="Settings">
<div
className={classNames('right-action-icon-container', {
active: isSettingsOpen,
})}
>
<SettingsIcon
className="settings-icon"
data-testid="settings-icon"
width={14}
height={14}
onClick={(): void => setIsSettingsOpen(true)}
/>
<AnnouncementTooltip
show={showAnnouncementTooltip}
position={{ top: -5, left: 15 }}
title="Edit your quick filters"
message="You can now customize and re-arrange your quick filters panel. Select the quick filters youd need and hide away the rest for faster exploration."
onClose={(): void => {
setLocalStorageKey(
LOCALSTORAGE.QUICK_FILTERS_SETTINGS_ANNOUNCEMENT,
'false',
);
}}
/>
</div>
</Tooltip>
{isDynamicFilters && (
<AuthZButton
checks={QuickFilterManagePermissions}
variant="link"
color="secondary"
aria-label="Settings"
className={classNames('right-action-icon-container', {
active: isSettingsOpen,
})}
onClick={(): void => setIsSettingsOpen(true)}
testId="settings-icon-container"
prefix={
<Tooltip title="Settings" open={isSettingsDisabled ? false : undefined}>
<SettingsIcon
className="settings-icon"
data-testid="settings-icon"
width={14}
height={14}
/>
</Tooltip>
}
/>
)}
</section>
);
@@ -361,6 +357,21 @@ export default function QuickFilters(props: IQuickFiltersProps): JSX.Element {
</>
);
const filtersSkeleton = (
<div className="quick-filters-skeleton">
{Array.from({ length: 5 }).map((_, index) => (
// eslint-disable-next-line react/no-array-index-key
<Skeleton.Input active size="small" key={index} />
))}
</div>
);
const filtersContent = isCustomFiltersLoading ? (
filtersSkeleton
) : (
<OverlayScrollbar>{renderContent()}</OverlayScrollbar>
);
return (
<div className="quick-filters-container">
<div className="quick-filters">
@@ -371,15 +382,15 @@ export default function QuickFilters(props: IQuickFiltersProps): JSX.Element {
</section>
)}
{isCustomFiltersLoading ? (
<div className="quick-filters-skeleton">
{Array.from({ length: 5 }).map((_, index) => (
// eslint-disable-next-line react/no-array-index-key
<Skeleton.Input active size="small" key={index} />
))}
</div>
{signal ? (
<AuthZGuardContent
checks={[QuickFilterReadPermission]}
fallbackOnLoading={filtersSkeleton}
>
{filtersContent}
</AuthZGuardContent>
) : (
<OverlayScrollbar>{renderContent()}</OverlayScrollbar>
filtersContent
)}
</div>
<div className="quick-filters-settings-container">

View File

@@ -35,10 +35,7 @@ const useFilterConfig = ({
[data],
);
const isDynamicFilters = useMemo(
() => customFilters.length > 0,
[customFilters],
);
const isDynamicFilters = !!signal;
const filterConfig = useMemo(
() => getFilterConfig(signal, customFilters, config),

View File

@@ -0,0 +1,212 @@
import { ENVIRONMENT } from 'constants/env';
import {
ApiMonitoringParams,
useApiMonitoringParams,
} from 'container/ApiMonitoring/queryParams';
import { useQueryBuilder } from 'hooks/queryBuilder/useQueryBuilder';
import {
QuickFilterReadPermission,
QuickFilterUpdatePermission,
} from 'lib/authz/hooks/useAuthZ/permissions/quick-filter.permissions';
import {
AUTHZ_CHECK_URL,
setupAuthzAdmin,
setupAuthzDeny,
setupAuthzDenyAll,
} from 'lib/authz/utils/authz-test-utils';
import {
otherFiltersResponse,
quickFiltersAttributeValuesResponse,
quickFiltersListResponse,
} from 'mocks-server/__mockdata__/customQuickFilters';
import { server } from 'mocks-server/server';
import { rest } from 'msw';
import { render, screen, userEvent, waitFor } from 'tests/test-utils';
import QuickFilters from '../QuickFilters';
import { QuickFiltersSource, SignalType } from '../types';
import { QuickFiltersConfig } from './constants';
jest.mock('hooks/queryBuilder/useQueryBuilder', () => ({
useQueryBuilder: jest.fn(),
}));
jest.mock('container/ApiMonitoring/queryParams');
const mockUseApiMonitoringParams = jest.mocked(useApiMonitoringParams);
const BASE_URL = ENVIRONMENT.baseURL;
const SIGNAL = SignalType.LOGS;
const quickFiltersListURL = `${BASE_URL}/api/v2/quick_filters/${SIGNAL}`;
const fieldsKeysURL = `${BASE_URL}/api/v1/fields/keys`;
const attributeValuesURL = `${BASE_URL}/api/v3/autocomplete/attribute_values`;
const fieldsValuesURL = `${BASE_URL}/api/v1/fields/values`;
const NOT_AUTHORIZED_TEXT = /is not authorized to perform/i;
const FILTER_SERVICE_NAME = 'Service Name';
const SETTINGS_CONTAINER_TEST_ID = 'settings-icon-container';
beforeEach(() => {
(useQueryBuilder as jest.Mock).mockReturnValue({
currentQuery: {
builder: {
queryData: [
{
queryName: 'Test Query',
filters: { items: [] },
},
],
},
},
lastUsedQuery: 0,
redirectWithQueryBuilderData: jest.fn(),
});
mockUseApiMonitoringParams.mockReturnValue([
{ showIP: true } as ApiMonitoringParams,
jest.fn(),
]);
server.use(
rest.get(quickFiltersListURL, (_req, res, ctx) =>
res(ctx.status(200), ctx.json(quickFiltersListResponse)),
),
rest.get(fieldsKeysURL, (_req, res, ctx) =>
res(ctx.status(200), ctx.json(otherFiltersResponse)),
),
rest.get(attributeValuesURL, (_req, res, ctx) =>
res(ctx.status(200), ctx.json(quickFiltersAttributeValuesResponse)),
),
rest.get(fieldsValuesURL, (_req, res, ctx) =>
res(ctx.status(200), ctx.json(quickFiltersAttributeValuesResponse)),
),
);
});
afterEach(() => {
server.resetHandlers();
jest.clearAllMocks();
});
function renderWithSignal(): void {
render(
<QuickFilters
source={QuickFiltersSource.LOGS_EXPLORER}
signal={SIGNAL}
handleFilterVisibilityChange={jest.fn()}
/>,
);
}
function renderStaticConfig(): void {
render(
<QuickFilters
source={QuickFiltersSource.EXCEPTIONS}
config={QuickFiltersConfig}
handleFilterVisibilityChange={jest.fn()}
/>,
);
}
describe('QuickFilters - AuthZ', () => {
describe('read denied', () => {
it('shows the inline denial instead of the filters, header stays', async () => {
server.use(setupAuthzDeny(QuickFilterReadPermission));
renderWithSignal();
await expect(
screen.findByText(NOT_AUTHORIZED_TEXT),
).resolves.toBeInTheDocument();
expect(screen.queryByText(FILTER_SERVICE_NAME)).not.toBeInTheDocument();
expect(screen.getByText('Filters for')).toBeInTheDocument();
const settingsTrigger = await screen.findByTestId(
SETTINGS_CONTAINER_TEST_ID,
);
await waitFor(() =>
expect(settingsTrigger).toHaveAttribute(
'data-denied-permissions',
expect.stringContaining('read'),
),
);
});
});
describe('update denied', () => {
it('renders the filters but disables the settings trigger', async () => {
server.use(setupAuthzDeny(QuickFilterUpdatePermission));
renderWithSignal();
await expect(
screen.findByText(FILTER_SERVICE_NAME),
).resolves.toBeInTheDocument();
const settingsTrigger = await screen.findByTestId(
SETTINGS_CONTAINER_TEST_ID,
);
await waitFor(() =>
expect(settingsTrigger).toHaveAttribute(
'data-denied-permissions',
expect.stringContaining('update'),
),
);
await userEvent.click(settingsTrigger);
expect(screen.queryByText(/ADDED FILTERS/i)).not.toBeInTheDocument();
});
});
describe('all permissions granted', () => {
it('renders the filters and opens settings from the trigger', async () => {
server.use(setupAuthzAdmin());
renderWithSignal();
await expect(
screen.findByText(FILTER_SERVICE_NAME),
).resolves.toBeInTheDocument();
const settingsTrigger = await screen.findByTestId(
SETTINGS_CONTAINER_TEST_ID,
);
expect(settingsTrigger).not.toHaveAttribute('data-denied-permissions');
await userEvent.click(settingsTrigger);
await expect(
screen.findByText(/ADDED FILTERS/i),
).resolves.toBeInTheDocument();
});
});
describe('static config pages (no signal)', () => {
it('is not gated even when every permission is denied', async () => {
server.use(setupAuthzDenyAll());
renderStaticConfig();
await expect(
screen.findByText(FILTER_SERVICE_NAME),
).resolves.toBeInTheDocument();
expect(screen.queryByText(NOT_AUTHORIZED_TEXT)).not.toBeInTheDocument();
});
});
describe('permission check loading', () => {
it('shows the skeleton, not the filters or a denial', async () => {
server.use(
rest.post(AUTHZ_CHECK_URL, (_req, res, ctx) => res(ctx.delay('infinite'))),
);
renderWithSignal();
await waitFor(() =>
// eslint-disable-next-line testing-library/no-node-access
expect(
document.querySelector('.quick-filters-skeleton'),
).toBeInTheDocument(),
);
expect(screen.queryByText(NOT_AUTHORIZED_TEXT)).not.toBeInTheDocument();
expect(screen.queryByText(FILTER_SERVICE_NAME)).not.toBeInTheDocument();
});
});
});

View File

@@ -54,6 +54,7 @@ export interface IQuickFiltersProps {
source: QuickFiltersSource;
onFilterChange?: (query: Query) => void;
onQuickFilterChange?: (data: QuickFilterChangeEventData) => void;
/** Pass to fetch quick filters for this signal; omit to use `config` as-is */
signal?: SignalType;
className?: string;
showFilterCollapse?: boolean;

View File

@@ -8,6 +8,7 @@ import { FiltersType, IQuickFiltersConfig, SignalType } from './types';
const FILTER_TITLE_MAP: Record<string, string> = {
duration_nano: 'Duration',
hasError: 'Has Error (Status)',
has_error: 'Has Error (Status)',
};
const FILTER_TYPE_MAP: Record<string, FiltersType> = {

View File

@@ -31,7 +31,6 @@ export enum LOCALSTORAGE {
DONT_SHOW_SLOW_API_WARNING = 'DONT_SHOW_SLOW_API_WARNING',
METRICS_LIST_OPTIONS = 'METRICS_LIST_OPTIONS',
SHOW_EXCEPTIONS_QUICK_FILTERS = 'SHOW_EXCEPTIONS_QUICK_FILTERS',
QUICK_FILTERS_SETTINGS_ANNOUNCEMENT = 'QUICK_FILTERS_SETTINGS_ANNOUNCEMENT',
FUNNEL_STEPS = 'FUNNEL_STEPS',
SPAN_DETAILS_PINNED_ATTRIBUTES = 'SPAN_DETAILS_PINNED_ATTRIBUTES',
LAST_USED_CUSTOM_TIME_RANGES = 'LAST_USED_CUSTOM_TIME_RANGES',

View File

@@ -0,0 +1,14 @@
.container {
display: flex;
align-items: center;
justify-content: flex-end;
gap: 0.3rem;
margin: 8px 0;
}
.optionsTrigger {
display: flex;
align-items: center;
gap: 4px;
cursor: pointer;
}

View File

@@ -0,0 +1,82 @@
import { memo, useState } from 'react';
import { useTranslation } from 'react-i18next';
import { Settings } from '@signozhq/icons';
import FieldsSelector from 'components/FieldsSelector';
import Controls, { ControlsProps } from 'container/Controls';
import { OptionsMenuConfig } from 'container/OptionsMenu/types';
import useQueryPagination from 'hooks/queryPagination/useQueryPagination';
import { DataSource } from 'types/common/queryBuilder';
import styles from './Controls.module.scss';
function TraceExplorerControls({
isLoading,
totalCount,
perPageOptions,
config,
showSizeChanger = true,
}: TraceExplorerControlsProps): JSX.Element | null {
const { t } = useTranslation(['trace']);
const [isFieldsSelectorOpen, setIsFieldsSelectorOpen] = useState(false);
const {
pagination,
handleCountItemsPerPageChange,
handleNavigateNext,
handleNavigatePrevious,
} = useQueryPagination(totalCount, perPageOptions);
return (
<div className={styles.container}>
{config?.fieldsSelector && (
<>
<div
className={styles.optionsTrigger}
onClick={(): void => setIsFieldsSelectorOpen(true)}
>
{t('options_menu.options')}
<Settings size="md" />
</div>
<FieldsSelector
isOpen={isFieldsSelectorOpen}
title="Edit columns"
fields={config.fieldsSelector.value}
onFieldsChange={config.fieldsSelector.onFieldsChange}
onClose={(): void => setIsFieldsSelectorOpen(false)}
signal={DataSource.TRACES}
/>
</>
)}
<Controls
isLoading={isLoading}
totalCount={totalCount}
offset={pagination.offset}
countPerPage={pagination.limit}
perPageOptions={perPageOptions}
handleCountItemsPerPageChange={handleCountItemsPerPageChange}
handleNavigateNext={handleNavigateNext}
handleNavigatePrevious={handleNavigatePrevious}
showSizeChanger={showSizeChanger}
/>
</div>
);
}
TraceExplorerControls.defaultProps = {
config: null,
};
type TraceExplorerControlsProps = Pick<
ControlsProps,
'isLoading' | 'totalCount' | 'perPageOptions'
> & {
config?: OptionsMenuConfig | null;
showSizeChanger?: boolean;
};
TraceExplorerControls.defaultProps = {
showSizeChanger: true,
};
export default memo(TraceExplorerControls);

View File

@@ -0,0 +1,168 @@
import { Link } from 'react-router-dom';
import type { TableColumnsType as ColumnsType } from 'antd';
import { Badge } from '@signozhq/ui/badge';
import { Typography } from '@signozhq/ui/typography';
import { TelemetryFieldKey } from 'api/v5/v5';
import type { TracesTableRow } from '../TracesTable/getFieldColumn';
import { DATE_TIME_FORMATS } from 'constants/dateTimeFormats';
import ROUTES from 'constants/routes';
import { buildCompositeKey } from 'container/OptionsMenu/utils';
import { getMs } from 'container/Trace/Filters/Panel/PanelBody/Duration/util';
import { formUrlParams } from 'container/TraceDetail/utils';
import { TimestampInput } from 'hooks/useTimezoneFormatter/useTimezoneFormatter';
import { RowData } from 'lib/query/createTableColumnsFromQuery';
import LineClampedText from 'periscope/components/LineClampedText/LineClampedText';
import { ILog } from 'types/api/logs/log';
import { QueryDataV3 } from 'types/api/widgets/getQuery';
export function BlockLink({
children,
to,
openInNewTab,
}: {
children: React.ReactNode;
to: string;
openInNewTab: boolean;
}): any {
// Display block to make the whole cell clickable
return (
<Link
to={to}
style={{ display: 'block' }}
target={openInNewTab ? '_blank' : '_self'}
>
{children}
</Link>
);
}
export const transformDataWithDate = (
data: QueryDataV3[],
): Omit<ILog, 'timestamp'>[] =>
data[0]?.list?.map(({ data, timestamp }) => ({ ...data, date: timestamp })) ||
[];
export const getTraceLink = (record: Record<string, unknown>): string => {
function readId(value: unknown): string {
if (typeof value === 'string' || typeof value === 'number') {
return String(value);
}
return '';
}
const traceId = readId(record.traceID) || readId(record.trace_id);
const spanId = readId(record.spanID) || readId(record.span_id);
return `${ROUTES.TRACE}/${traceId}${formUrlParams({
spanId,
levelUp: 0,
levelDown: 0,
})}`;
};
export const getListColumns = (
selectedColumns: TelemetryFieldKey[],
formatTimezoneAdjustedTimestamp: (
input: TimestampInput,
format?: string,
) => string | number,
): ColumnsType<RowData> => {
const initialColumns: ColumnsType<RowData> = [
{
dataIndex: 'date',
key: 'date',
title: 'Timestamp',
width: 145,
render: (value, item): JSX.Element => {
const date =
typeof value === 'string'
? formatTimezoneAdjustedTimestamp(
value,
DATE_TIME_FORMATS.ISO_DATETIME_MS,
)
: formatTimezoneAdjustedTimestamp(
value / 1e6,
DATE_TIME_FORMATS.ISO_DATETIME_MS,
);
return (
<BlockLink to={getTraceLink(item)} openInNewTab={false}>
<Typography.Text>{date}</Typography.Text>
</BlockLink>
);
},
},
];
const columns: ColumnsType<RowData> =
selectedColumns.map((props) => {
const name = props?.name || (props as any)?.key;
const fieldContext = props?.fieldContext || (props as any)?.type;
return {
title: name,
dataIndex: name,
key: buildCompositeKey(name, fieldContext),
width: 145,
render: (value, item): JSX.Element => {
if (value === '') {
return (
<BlockLink to={getTraceLink(item)} openInNewTab={false}>
<Typography data-testid={name}>N/A</Typography>
</BlockLink>
);
}
if (
name === 'httpMethod' ||
name === 'responseStatusCode' ||
name === 'response_status_code' ||
name === 'http_method'
) {
return (
<BlockLink to={getTraceLink(item)} openInNewTab={false}>
<Badge data-testid={name} color="sakura" variant="outline">
{value}
</Badge>
</BlockLink>
);
}
if (name === 'durationNano' || name === 'duration_nano') {
return (
<BlockLink to={getTraceLink(item)} openInNewTab={false}>
<Typography data-testid={name}>{getMs(value)}ms</Typography>
</BlockLink>
);
}
return (
<BlockLink to={getTraceLink(item)} openInNewTab={false}>
<Typography data-testid={name}>
<LineClampedText text={value} lines={3} />
</Typography>
</BlockLink>
);
},
responsive: ['md'],
};
}) || [];
return [...initialColumns, ...columns];
};
// Reshapes the query-range list payload into table rows. `id` mirrors span_id so
// TanStack sees genuine row changes on orderBy toggles instead of falling back to
// positional ids; `timestamp` is lifted from the wrapping ListItem.
export const transformSpanRows = (data: QueryDataV3[]): TracesTableRow[] => {
const list = data[0]?.list;
if (!list) {
return [];
}
return list.map((item) => {
const row = item.data as Record<string, unknown>;
return {
...row,
timestamp: item.timestamp,
id: row.span_id,
};
}) as TracesTableRow[];
};

View File

@@ -0,0 +1,19 @@
.loading-traces {
padding: 24px 0;
height: 240px;
display: flex;
justify-content: center;
align-items: flex-start;
.loading-traces-content {
display: flex;
align-items: flex-start;
flex-direction: column;
.loading-gif {
height: 72px;
margin-left: -24px;
}
}
}

View File

@@ -0,0 +1,22 @@
import { useTranslation } from 'react-i18next';
import { Typography } from '@signozhq/ui/typography';
import { DataSource } from 'types/common/queryBuilder';
import loadingPlaneUrl from '@/assets/Icons/loading-plane.gif';
import './TraceLoading.styles.scss';
export function TracesLoading(): JSX.Element {
const { t } = useTranslation('common');
return (
<div className="loading-traces">
<div className="loading-traces-content">
<img className="loading-gif" src={loadingPlaneUrl} alt="wait-icon" />
<Typography>
{t('pending_data_placeholder', { dataSource: DataSource.TRACES })}
</Typography>
</div>
</div>
);
}

View File

@@ -0,0 +1,77 @@
import { generatePath, Link } from 'react-router-dom';
import { Badge } from '@signozhq/ui/badge';
import TanStackTable from 'components/TanStackTableView';
import { DATE_TIME_FORMATS } from 'constants/dateTimeFormats';
import ROUTES from 'constants/routes';
import { getMs } from 'container/Trace/Filters/Panel/PanelBody/Duration/util';
import { useTimezone } from 'providers/Timezone';
import {
DURATION_FIELD_NAMES,
STATUS_FIELD_NAMES,
TIMESTAMP_FIELD_NAMES,
TRACE_ID_FIELD_NAMES,
} from './constants';
import { stringifyCellValue } from './utils';
type FieldCellProps = {
name: string;
value: unknown;
};
function FieldCell({ name, value }: FieldCellProps): JSX.Element {
const { formatTimezoneAdjustedTimestamp } = useTimezone();
if (TIMESTAMP_FIELD_NAMES.has(name)) {
const ts = value as string | number;
const formatted =
typeof ts === 'string'
? formatTimezoneAdjustedTimestamp(ts, DATE_TIME_FORMATS.ISO_DATETIME_MS)
: formatTimezoneAdjustedTimestamp(
ts / 1e6,
DATE_TIME_FORMATS.ISO_DATETIME_MS,
);
const text = String(formatted);
return <TanStackTable.Text title={text}>{text}</TanStackTable.Text>;
}
if (value === '' || value == null) {
return <TanStackTable.Text data-testid={name}>-</TanStackTable.Text>;
}
const text = stringifyCellValue(value);
if (TRACE_ID_FIELD_NAMES.has(name)) {
return (
<Link
to={generatePath(ROUTES.TRACE_DETAIL, { id: text })}
data-testid="trace-id"
onClick={(e): void => e.stopPropagation()}
>
{text}
</Link>
);
}
if (STATUS_FIELD_NAMES.has(name)) {
return (
<Badge data-testid={name} color="sakura" variant="outline">
{text}
</Badge>
);
}
if (DURATION_FIELD_NAMES.has(name)) {
return (
<TanStackTable.Text data-testid={name}>{getMs(text)}ms</TanStackTable.Text>
);
}
return (
<TanStackTable.Text data-testid={name} title={text}>
{text}
</TanStackTable.Text>
);
}
export default FieldCell;

View File

@@ -0,0 +1,26 @@
.tableWrapper {
flex: 1;
min-height: 0;
display: flex;
flex-direction: column;
}
.tracesTable {
--tanstack-table-row-height: 54px;
--tanstack-table-header-height: 54px;
--tanstack-cell-padding-top-override: 5px;
--tanstack-cell-padding-bottom-override: 5px;
--tanstack-cell-padding-right-override: 15px;
--tanstack-cell-padding-left-override: 15px;
--tanstack-cell-header-padding-left-override: 5px;
--tanstack-cell-header-padding-left-first-column: 15px;
--tanstack-plain-body-line-clamp: 1;
--tanstack-table-cell-bg: var(--l2-background);
--tanstack-table-header-cell-bg: var(--l1-background-hover);
--tanstack-table-row-hover-bg: var(--l1-background-hover);
}

View File

@@ -0,0 +1,116 @@
import { useCallback } from 'react';
import { useHistory } from 'react-router-dom';
import ErrorInPlace from 'components/ErrorInPlace/ErrorInPlace';
import TanStackTable from 'components/TanStackTableView';
import type {
CellTypographySize,
TableColumnDef,
} from 'components/TanStackTableView/types';
import EmptyLogsSearch from 'container/EmptyLogsSearch/EmptyLogsSearch';
import NoLogs from 'container/NoLogs/NoLogs';
import { TracesLoading } from '../TraceLoading/TraceLoading';
import APIError from 'types/api/error';
import { DataSource, PanelTypeKeys } from 'types/common/queryBuilder';
import { getAbsoluteUrl } from 'utils/basePath';
import type { TracesTableRow } from './getFieldColumn';
import styles from './TracesTable.module.scss';
export type TracesTableProps = {
data: TracesTableRow[];
columns: TableColumnDef<TracesTableRow>[];
columnStorageKey?: string;
respectColumnOrder?: boolean;
panelType: PanelTypeKeys;
/** Builds the trace-detail href for a row; drives row click + cmd/ctrl-click. */
getRowHref: (row: TracesTableRow) => string;
isLoading: boolean;
isFetching: boolean;
isError: boolean;
error: APIError | Error | null;
isFilterApplied: boolean;
onColumnOrderChange?: (cols: TableColumnDef<TracesTableRow>[]) => void;
onColumnRemove?: (columnId: string) => void;
cellTypographySize?: CellTypographySize;
};
function TracesTable({
data,
columns,
columnStorageKey,
respectColumnOrder = false,
panelType,
getRowHref,
isLoading,
isFetching,
isError,
error,
isFilterApplied,
onColumnOrderChange,
onColumnRemove,
cellTypographySize = 'medium',
}: TracesTableProps): JSX.Element {
const history = useHistory();
const isDataAbsent =
!isLoading && !isFetching && !isError && data.length === 0;
const handleRowClick = useCallback(
(row: TracesTableRow): void => {
history.push(getRowHref(row));
},
[history, getRowHref],
);
const handleRowClickNewTab = useCallback(
(row: TracesTableRow): void => {
window.open(getAbsoluteUrl(getRowHref(row)), '_blank', 'noopener');
},
[getRowHref],
);
return (
<>
{isError && error && <ErrorInPlace error={error as APIError} />}
{(isLoading || (isFetching && data.length === 0)) && <TracesLoading />}
{isDataAbsent && !isFilterApplied && (
<NoLogs dataSource={DataSource.TRACES} />
)}
{isDataAbsent && isFilterApplied && (
<EmptyLogsSearch dataSource={DataSource.TRACES} panelType={panelType} />
)}
{!isError && data.length !== 0 && (
<div className={styles.tableWrapper}>
<TanStackTable<TracesTableRow>
data={data}
columns={columns}
className={styles.tracesTable}
columnStorageKey={columnStorageKey}
respectColumnOrder={respectColumnOrder}
isLoading={isFetching}
cellTypographySize={cellTypographySize}
onColumnOrderChange={onColumnOrderChange}
onColumnRemove={onColumnRemove}
onRowClick={handleRowClick}
onRowClickNewTab={handleRowClickNewTab}
getRowTestId={(row): string => `traces-table-row-${row.id}`}
/>
</div>
)}
</>
);
}
TracesTable.defaultProps = {
columnStorageKey: undefined,
respectColumnOrder: false,
onColumnOrderChange: undefined,
onColumnRemove: undefined,
cellTypographySize: 'medium',
};
export default TracesTable;

View File

@@ -0,0 +1,18 @@
// Field-name allowlists that drive signal-specific cell rendering. Both legacy
// camelCase and snake_case variants are listed because the API has shipped both.
export const TIMESTAMP_FIELD_NAMES = new Set(['timestamp']);
export const STATUS_FIELD_NAMES = new Set([
'httpMethod',
'http_method',
'http.method',
'http.request.method',
'responseStatusCode',
'response_status_code',
'http.status_code',
'http.response.status_code',
]);
export const DURATION_FIELD_NAMES = new Set(['durationNano', 'duration_nano']);
export const TRACE_ID_FIELD_NAMES = new Set(['traceID', 'trace_id']);

View File

@@ -0,0 +1,26 @@
import { TelemetryFieldKey } from 'api/v5/v5';
import type { TableColumnDef } from 'components/TanStackTableView/types';
import { buildCompositeKey } from 'container/OptionsMenu/utils';
import { TIMESTAMP_FIELD_NAMES } from './constants';
import FieldCell from './FieldCell';
export type TracesTableRow = { id: string } & Record<string, unknown>;
export function getFieldColumn(
field: TelemetryFieldKey,
): TableColumnDef<TracesTableRow> {
const { name, fieldContext, fieldDataType } = field;
const isTimestamp = TIMESTAMP_FIELD_NAMES.has(name);
return {
id: buildCompositeKey(name, fieldContext, fieldDataType),
header: name,
accessorFn: (row): unknown => row[name],
enableMove: !isTimestamp,
enableRemove: !isTimestamp,
canBeHidden: !isTimestamp,
width: { min: 192 },
cell: ({ value }): JSX.Element => <FieldCell name={name} value={value} />,
};
}

View File

@@ -0,0 +1,12 @@
export function stringifyCellValue(value: unknown): string {
if (value == null) {
return '';
}
if (typeof value === 'string') {
return value;
}
if (typeof value === 'number' || typeof value === 'boolean') {
return String(value);
}
return JSON.stringify(value);
}

View File

@@ -1,9 +1,6 @@
import { TelemetryFieldKey } from 'api/v5/v5';
import type { TableColumnDef } from 'components/TanStackTableView/types';
import {
getFieldColumn,
TracesTableRow,
} from 'container/TracesExplorer/TracesTable/getFieldColumn';
import { getFieldColumn, TracesTableRow } from '../TracesTable/getFieldColumn';
import { DEFAULT_PER_PAGE_OPTIONS } from 'hooks/queryPagination';
export const PER_PAGE_OPTIONS: number[] = [10, ...DEFAULT_PER_PAGE_OPTIONS];

View File

@@ -0,0 +1,235 @@
/**
* AI Assistant page-action factories for the Traces Explorer.
*
* Mirrors the logs equivalents — each factory closes over live page
* state/callbacks so `execute()` always operates on the current query, and
* the page component instantiates them via `useMemo` + `usePageActions`.
*
* See `pages/LogsExplorer/aiActions.ts` for the rationale behind writing
* BOTH `filters.items` and `filter.expression` and then re-using the same
* URL parser shape via `redirectWithQueryBuilderData`.
*/
import { convertFiltersToExpression } from 'components/QueryBuilderV2/utils';
import {
aiFilterToTagFilterItem,
FILTER_OP_ENUM,
FILTER_VALUE_DESCRIPTION,
FilterDeps,
replaceFirstQueryData,
} from 'container/AIAssistant/pageActions/builderQueryHelpers';
import {
ActionResult,
PageAction,
} from 'container/AIAssistant/pageActions/types';
import {
IBuilderQuery,
TagFilterItem,
} from 'types/api/queryBuilder/queryBuilderData';
interface AIFilter {
key: string;
op: string;
value: string;
}
interface RunQueryParams {
filters: AIFilter[];
}
interface AddFilterParams {
key: string;
op: string;
value: string;
}
type TracesView = 'list' | 'timeseries' | 'table' | 'trace';
interface ChangeViewParams {
view: TracesView;
}
interface SaveViewParams {
name: string;
}
/**
* Replace all active span filters and navigate to the updated query URL
* (which makes the WHERE clause reflect the new filters and triggers a re-run).
*/
export function tracesRunQueryAction(
deps: FilterDeps,
): PageAction<RunQueryParams> {
return {
id: 'traces.runQuery',
description: 'Replace the active trace filters and re-run the query',
parameters: {
type: 'object',
properties: {
filters: {
type: 'array',
description: 'Replacement filter list',
items: {
type: 'object',
properties: {
key: {
type: 'string',
description: 'Attribute key, e.g. service.name, http.status_code',
},
op: {
type: 'string',
enum: [...FILTER_OP_ENUM],
},
value: {
type: 'string',
description: FILTER_VALUE_DESCRIPTION,
},
},
required: ['key', 'op', 'value'],
},
},
},
required: ['filters'],
},
autoApply: true,
execute: async ({ filters }): Promise<ActionResult> => {
const baseQuery = deps.currentQuery.builder.queryData[0];
if (!baseQuery) {
throw new Error('No active query found in Traces Explorer.');
}
const tagItems = filters.map(aiFilterToTagFilterItem);
const newFilters = { items: tagItems, op: 'AND' };
const updatedBuilderQuery: IBuilderQuery = {
...baseQuery,
filters: newFilters,
filter: convertFiltersToExpression(newFilters),
};
deps.handleSetQueryData(0, updatedBuilderQuery);
deps.redirectWithQueryBuilderData(
replaceFirstQueryData(deps.currentQuery, updatedBuilderQuery),
);
return {
summary: `Query updated with ${filters.length} filter(s) and re-run.`,
};
},
getContext: (): Record<string, unknown> => ({
filters:
deps.currentQuery.builder.queryData[0]?.filters?.items?.map(
(f: TagFilterItem) => ({
key: f.key?.key,
op: f.op,
value: f.value,
}),
) ?? [],
}),
};
}
/**
* Append a single filter to the existing trace query and navigate to the
* updated URL.
*/
export function tracesAddFilterAction(
deps: FilterDeps,
): PageAction<AddFilterParams> {
return {
id: 'traces.addFilter',
description: 'Add a single filter to the current trace query and re-run',
parameters: {
type: 'object',
properties: {
key: {
type: 'string',
description: 'Attribute key, e.g. service.name, http.status_code',
},
op: {
type: 'string',
enum: [...FILTER_OP_ENUM],
},
value: {
type: 'string',
description: FILTER_VALUE_DESCRIPTION,
},
},
required: ['key', 'op', 'value'],
},
autoApply: true,
execute: async ({ key, op, value }): Promise<ActionResult> => {
const baseQuery = deps.currentQuery.builder.queryData[0];
if (!baseQuery) {
throw new Error('No active query found in Traces Explorer.');
}
const existing = baseQuery.filters?.items ?? [];
const newItem = aiFilterToTagFilterItem({ key, op, value });
const newFilters = { items: [...existing, newItem], op: 'AND' };
const updatedBuilderQuery: IBuilderQuery = {
...baseQuery,
filters: newFilters,
filter: convertFiltersToExpression(newFilters),
};
deps.handleSetQueryData(0, updatedBuilderQuery);
deps.redirectWithQueryBuilderData(
replaceFirstQueryData(deps.currentQuery, updatedBuilderQuery),
);
return { summary: `Filter added: ${key} ${op} "${value}". Query re-run.` };
},
};
}
/**
* Switch the traces explorer between list / timeseries / table / trace views.
*/
export function tracesChangeViewAction(deps: {
onChangeView: (view: TracesView) => void;
}): PageAction<ChangeViewParams> {
return {
id: 'traces.changeView',
description:
'Switch the Traces Explorer between list, timeseries, table, and trace views',
parameters: {
type: 'object',
properties: {
view: {
type: 'string',
enum: ['list', 'timeseries', 'table', 'trace'],
description: 'The panel view to switch to',
},
},
required: ['view'],
},
execute: async ({ view }): Promise<ActionResult> => {
deps.onChangeView(view);
return { summary: `Switched to the "${view}" view.` };
},
};
}
/**
* Save the current trace query as a named view (stub — wires to real API
* when available).
*/
export function tracesSaveViewAction(deps: {
onSaveView: (name: string) => Promise<void>;
}): PageAction<SaveViewParams> {
return {
id: 'traces.saveView',
description: 'Save the current trace query as a named view',
parameters: {
type: 'object',
properties: {
name: { type: 'string', description: 'Name for the saved view' },
},
required: ['name'],
},
execute: async ({ name }): Promise<ActionResult> => {
await deps.onSaveView(name);
return { summary: `View "${name}" saved.` };
},
};
}

View File

@@ -0,0 +1,132 @@
import {
ArrowUpToLine,
Atom,
Filter,
SquareMousePointer,
Terminal,
Binoculars,
} from '@signozhq/icons';
import { Button, Tooltip } from 'antd';
import cx from 'classnames';
import { ExplorerViews } from 'pages/LogsExplorer/utils';
import './ToolbarActions.styles.scss';
interface LeftToolbarActionsProps {
items: any;
selectedView: string;
onChangeSelectedView: (view: ExplorerViews) => void;
showFilter: boolean;
handleFilterVisibilityChange: () => void;
}
const activeTab = 'active-tab';
export default function LeftToolbarActions({
items,
selectedView,
onChangeSelectedView,
showFilter,
handleFilterVisibilityChange,
}: LeftToolbarActionsProps): JSX.Element {
const { clickhouse, list, timeseries, table, trace } = items;
return (
<div className="left-toolbar">
{!showFilter && (
<Tooltip title="Show Filters">
<Button onClick={handleFilterVisibilityChange} className="filter-btn">
<Filter size={12} />
<ArrowUpToLine size={12} style={{ transform: 'rotate(90deg)' }} />
</Button>
</Tooltip>
)}
<div className="left-toolbar-query-actions">
{list?.show && (
<Tooltip title="List View">
<Button
disabled={list.disabled}
className={cx(
'list-view-tab',
'explorer-view-option',
selectedView === list.key ? activeTab : '',
)}
onClick={(): void => onChangeSelectedView(list.key)}
>
<SquareMousePointer size={14} data-testid="search-view" />
List View
</Button>
</Tooltip>
)}
{trace?.show && (
<Tooltip title="Trace View">
<Button
disabled={trace.disabled}
className={cx(
'trace-view-tab',
'explorer-view-option',
selectedView === trace.key ? activeTab : '',
)}
onClick={(): void => onChangeSelectedView(trace.key)}
>
<SquareMousePointer size={14} data-testid="trace-view" />
Trace View
</Button>
</Tooltip>
)}
{timeseries?.show && (
<Tooltip title="Time Series">
<Button
disabled={timeseries.disabled}
className={cx(
'timeseries-view-tab',
'explorer-view-option',
selectedView === timeseries.key ? activeTab : '',
)}
onClick={(): void => onChangeSelectedView(timeseries.key)}
>
<Atom size={14} data-testid="query-builder-view" />
Time Series
</Button>
</Tooltip>
)}
{clickhouse?.show && (
<Tooltip title="Clickhouse">
<Button
disabled={clickhouse.disabled}
className={cx(
'clickhouse-view-tab',
'explorer-view-option',
selectedView === clickhouse.key ? activeTab : '',
)}
onClick={(): void => onChangeSelectedView(clickhouse.key)}
>
<Terminal size={14} data-testid="clickhouse-view" />
Clickhouse
</Button>
</Tooltip>
)}
{table?.show && (
<Tooltip title="Table">
<Button
disabled={table.disabled}
className={cx(
'table-view-tab',
'explorer-view-option',
selectedView === table.key ? activeTab : '',
)}
onClick={(): void => onChangeSelectedView(table.key)}
>
<Binoculars size={14} data-testid="query-builder-view-v2" />
Table
</Button>
</Tooltip>
)}
</div>
</div>
);
}

View File

@@ -0,0 +1,125 @@
.left-toolbar {
display: flex;
align-items: center;
.filter-btn {
display: flex;
align-items: center;
justify-content: center;
box-shadow: none;
height: 32px;
margin-right: 12px;
border: 1px solid var(--l1-border);
}
.left-toolbar-query-actions {
display: flex;
border-radius: 2px;
border: 1px solid var(--l1-border);
background: var(--l1-background);
flex-direction: row;
border-bottom: none;
margin-bottom: -1px;
.prom-ql-icon {
height: 14px;
width: 14px;
}
.explorer-view-option {
display: flex;
align-items: center;
justify-content: center;
flex-direction: row;
border: none;
padding: 9px;
box-shadow: none;
border-radius: 0px;
border-left: 1px solid var(--l1-border);
border-bottom: 1px solid var(--l1-border);
gap: 8px;
&.active-tab {
background-color: var(--primary-background);
border-bottom: 1px solid var(--primary-background);
color: var(--primary-foreground);
&:hover {
background-color: var(--primary-background) !important;
}
}
&:disabled {
background-color: var(--l3-background);
opacity: 0.6;
}
&:first-child {
border-left: 1px solid transparent;
}
&:hover {
background-color: transparent !important;
border-left: 1px solid transparent !important;
color: var(--l1-foreground);
}
}
}
.frequency-chart-view-controller {
display: flex;
align-items: center;
padding-left: 8px;
gap: 8px;
}
}
.right-toolbar {
display: flex;
align-items: center;
background-color: var(--bg-robin-600);
}
.right-actions {
display: flex;
align-items: center;
}
.loading-container {
display: flex;
gap: 8px;
align-items: center;
.loading-btn {
display: flex;
width: 32px;
height: 33px;
padding: 4px 10px;
justify-content: center;
align-items: center;
gap: 6px;
flex-shrink: 0;
border-radius: 2px;
background: var(--l3-background);
box-shadow: none;
border: none;
}
.cancel-run {
display: flex;
height: 33px;
padding: 4px 10px;
justify-content: center;
align-items: center;
gap: 6px;
flex: 1 0 0;
border-radius: 2px;
background: var(--danger-background);
border: none;
}
.cancel-run:hover {
background-color: var(--bg-cherry-400) !important;
color: var(--l1-foreground) !important;
}
}

View File

@@ -329,13 +329,14 @@ describe('transformTransactionGroupsToResourcePermissions', () => {
it('returns all resources from RESOURCE_ORDER even with empty transaction groups', () => {
const result = transformTransactionGroupsToResourcePermissions([]);
expect(result).toHaveLength(9);
expect(result).toHaveLength(10);
expect(result.map((r) => r.resourceKind)).toStrictEqual([
'factor-api-key',
'license',
'logs',
'meter-metrics',
'metrics',
'quick-filter',
'role',
'serviceaccount',
'subscription',
@@ -420,13 +421,14 @@ describe('createEmptyRolePermissions', () => {
it('creates permissions for all resources in RESOURCE_ORDER', () => {
const result = createEmptyRolePermissions();
expect(result).toHaveLength(9);
expect(result).toHaveLength(10);
expect(result.map((r) => r.resourceKind)).toStrictEqual([
'factor-api-key',
'license',
'logs',
'meter-metrics',
'metrics',
'quick-filter',
'role',
'serviceaccount',
'subscription',

View File

@@ -5,6 +5,7 @@ import {
FileKey,
Gauge,
Key,
ListFilter,
Logs,
Receipt,
Shield,
@@ -77,6 +78,14 @@ export const RESOURCE_PANELS: Record<AuthZResource, ResourcePanelConfig> = {
selectorPlaceholder: 'Type * to cover the workspace subscription',
docsAnchor: 'subscription',
},
'quick-filter': {
label: 'Quick Filters',
description: 'Quick filters shown in the logs, traces, and other explorers.',
icon: ListFilter,
selectorPlaceholder:
'Type quick filter ID, separate multiple with comma or space',
docsAnchor: 'quick-filter',
},
logs: {
label: 'Logs',
description: 'Log data collected across the workspace.',

View File

@@ -13,6 +13,11 @@ export default {
type: 'metaresource',
allowedVerbs: ['create', 'delete', 'list', 'read', 'update'],
},
{
kind: 'quick-filter',
type: 'metaresource',
allowedVerbs: ['list', 'read', 'update'],
},
{
kind: 'subscription',
type: 'metaresource',

View File

@@ -0,0 +1,16 @@
import { buildPermission } from '../utils';
export const QuickFilterReadPermission = buildPermission(
'read',
'quick-filter:*',
);
export const QuickFilterUpdatePermission = buildPermission(
'update',
'quick-filter:*',
);
// Editing quick filters needs read as well as update
export const QuickFilterManagePermissions = [
QuickFilterReadPermission,
QuickFilterUpdatePermission,
];

View File

@@ -82,6 +82,7 @@ func (handler *handler) GetFieldsValues(rw http.ResponseWriter, req *http.Reques
values := &telemetrytypes.TelemetryFieldValues{
StringValues: allValues.StringValues,
BoolValues: allValues.BoolValues,
NumberValues: allValues.NumberValues,
RelatedValues: relatedValues,
}

View File

@@ -446,12 +446,14 @@ func (m *module) buildFilterClause(ctx context.Context, orgID valuer.UUID, filte
}
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, orgID, m.fl, telemetrytypes.SignalMetrics, nil, querybuilder.ToNanoSecs(uint64(startMillis)), querybuilder.ToNanoSecs(uint64(endMillis))),
Storage: m.storage,
Logger: m.logger,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "metric_name", FieldContext: telemetrytypes.FieldContextMetric},
FieldKeys: keys,
Context: ctx,
Logger: m.logger,
FieldMapper: m.fieldMapper,
ConditionBuilder: m.condBuilder,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "metric_name", FieldContext: telemetrytypes.FieldContextMetric},
FieldKeys: keys,
StartNs: querybuilder.ToNanoSecs(uint64(startMillis)),
EndNs: querybuilder.ToNanoSecs(uint64(endMillis)),
}
whereClause, err := querybuilder.PrepareWhereClause(expression, opts)

View File

@@ -24,7 +24,8 @@ type module struct {
telemetryStore telemetrystore.TelemetryStore
telemetryMetadataStore telemetrytypes.MetadataStore
querier querier.Querier
storage qbtypes.Storage
fieldMapper qbtypes.FieldMapper
condBuilder qbtypes.ConditionBuilder
logger *slog.Logger
config inframonitoring.Config
fl flagger.Flagger
@@ -39,11 +40,14 @@ func NewModule(
providerSettings factory.ProviderSettings,
cfg inframonitoring.Config,
) inframonitoring.Module {
fieldMapper := metricstelemetryschema.NewFieldMapper()
condBuilder := metricstelemetryschema.NewConditionBuilder(fieldMapper)
return &module{
telemetryStore: telemetryStore,
telemetryMetadataStore: telemetryMetadataStore,
querier: querier,
storage: metricstelemetryschema.NewStorage(),
fieldMapper: fieldMapper,
condBuilder: condBuilder,
logger: providerSettings.Logger,
config: cfg,
fl: fl,

View File

@@ -37,7 +37,8 @@ import (
type module struct {
telemetryStore telemetrystore.TelemetryStore
telemetryMetadataStore telemetrytypes.MetadataStore
storage qbtypes.Storage
fieldMapper qbtypes.FieldMapper
condBuilder qbtypes.ConditionBuilder
logger *slog.Logger
cache cache.Cache
ruleStore ruletypes.RuleStore
@@ -48,9 +49,12 @@ type module struct {
// NewModule constructs the metrics module with the provided dependencies.
func NewModule(ts telemetrystore.TelemetryStore, telemetryMetadataStore telemetrytypes.MetadataStore, cache cache.Cache, ruleStore ruletypes.RuleStore, dashboardModule dashboard.Module, fl flagger.Flagger, providerSettings factory.ProviderSettings, cfg metricsexplorer.Config) metricsexplorer.Module {
fieldMapper := metricstelemetryschema.NewFieldMapper()
condBuilder := metricstelemetryschema.NewConditionBuilder(fieldMapper)
return &module{
telemetryStore: ts,
storage: metricstelemetryschema.NewStorage(),
fieldMapper: fieldMapper,
condBuilder: condBuilder,
logger: providerSettings.Logger,
telemetryMetadataStore: telemetryMetadataStore,
cache: cache,
@@ -971,12 +975,14 @@ func (m *module) buildFilterClause(ctx context.Context, orgID valuer.UUID, filte
}
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, orgID, m.fl, telemetrytypes.SignalMetrics, nil, querybuilder.ToNanoSecs(uint64(startMillis)), querybuilder.ToNanoSecs(uint64(endMillis))),
Storage: m.storage,
Logger: m.logger,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "metric_name", FieldContext: telemetrytypes.FieldContextMetric},
FieldKeys: keys,
Context: ctx,
Logger: m.logger,
FieldMapper: m.fieldMapper,
ConditionBuilder: m.condBuilder,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "metric_name", FieldContext: telemetrytypes.FieldContextMetric},
FieldKeys: keys,
StartNs: querybuilder.ToNanoSecs(uint64(startMillis)),
EndNs: querybuilder.ToNanoSecs(uint64(endMillis)),
}
whereClause, err := querybuilder.PrepareWhereClause(expression, opts)

View File

@@ -0,0 +1,150 @@
package implrulestatehistory
import (
"context"
"fmt"
"slices"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
type conditionBuilder struct {
fm qbtypes.FieldMapper
}
func newConditionBuilder(fm qbtypes.FieldMapper) qbtypes.ConditionBuilder {
return &conditionBuilder{fm: fm}
}
// Rule state history has no resource sub-query, so options are unused.
func (c *conditionBuilder) ConditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for rule state history.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}
// Rule state history fields have no family support, so every logical field
// is single-member and flattens losslessly to its physical key.
resolved, warning := querybuilder.ResolveLogicalFields(key, querybuilder.MatchingLogicalFields(ctx, orgID, nil, key, fieldKeys))
keys := querybuilder.SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
if len(keys) == 0 {
return nil, warnings, querybuilder.NewKeyNotFoundError(key.Name)
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
cond, err := c.conditionForKey(ctx, orgID, startNs, endNs, k, operator, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
}
return conds, warnings, nil
}
func (c *conditionBuilder) conditionForKey(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
if operator.IsStringSearchOperator() {
value = querybuilder.FormatValueForContains(value)
}
fieldName, err := c.fm.FieldFor(ctx, orgID, startNs, endNs, key)
if err != nil {
return "", err
}
switch operator {
case qbtypes.FilterOperatorEqual:
return sb.E(fieldName, value), nil
case qbtypes.FilterOperatorNotEqual:
return sb.NE(fieldName, value), nil
case qbtypes.FilterOperatorGreaterThan:
return sb.G(fieldName, value), nil
case qbtypes.FilterOperatorGreaterThanOrEq:
return sb.GE(fieldName, value), nil
case qbtypes.FilterOperatorLessThan:
return sb.LT(fieldName, value), nil
case qbtypes.FilterOperatorLessThanOrEq:
return sb.LE(fieldName, value), nil
case qbtypes.FilterOperatorLike:
return sb.Like(fieldName, value), nil
case qbtypes.FilterOperatorNotLike:
return sb.NotLike(fieldName, value), nil
case qbtypes.FilterOperatorILike:
return sb.ILike(fieldName, value), nil
case qbtypes.FilterOperatorNotILike:
return sb.NotILike(fieldName, value), nil
case qbtypes.FilterOperatorContains:
return sb.ILike(fieldName, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorNotContains:
return sb.NotILike(fieldName, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorRegexp:
return fmt.Sprintf(`match(%s, %s)`, sqlbuilder.Escape(fieldName), sb.Var(value)), nil
case qbtypes.FilterOperatorNotRegexp:
return fmt.Sprintf(`NOT match(%s, %s)`, sqlbuilder.Escape(fieldName), sb.Var(value)), nil
case qbtypes.FilterOperatorBetween:
values, ok := value.([]any)
if !ok || len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.Between(fieldName, values[0], values[1]), nil
case qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok || len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.NotBetween(fieldName, values[0], values[1]), nil
case qbtypes.FilterOperatorIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
return sb.In(fieldName, values), nil
case qbtypes.FilterOperatorNotIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
return sb.NotIn(fieldName, values), nil
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
intrinsic := []string{"rule_id", "rule_name", "overall_state", "overall_state_changed", "state", "state_changed", "unix_milli", "fingerprint", "value"}
if slices.Contains(intrinsic, key.Name) {
return "true", nil
}
if operator == qbtypes.FilterOperatorExists {
return fmt.Sprintf("JSONHas(labels, %s)", sb.Var(key.Name)), nil
}
return fmt.Sprintf("not JSONHas(labels, %s)", sb.Var(key.Name)), nil
}
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported operator: %v", operator)
}

View File

@@ -9,6 +9,7 @@ import (
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
@@ -25,15 +26,19 @@ var ruleStateHistoryColumns = map[string]*schema.Column{
"value": {Name: "value", Type: schema.ColumnTypeFloat64},
}
type storage struct{}
type fieldMapper struct{}
var _ qbtypes.Storage = (*storage)(nil)
func newStorage() *storage {
return &storage{}
func newFieldMapper() qbtypes.FieldMapper {
return &fieldMapper{}
}
func (m *storage) getColumn(_ context.Context, key *telemetrytypes.TelemetryFieldKey) (*schema.Column, error) { //nolint:unparam
// CandidateKeys returns nil: rule-state history has no attribute-map fallback, so a
// context-missing key stays unresolved and the caller errors.
func (m *fieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, _ *telemetrytypes.TelemetryFieldKey, _ any, _ map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
}
func (m *fieldMapper) getColumn(_ context.Context, key *telemetrytypes.TelemetryFieldKey) (*schema.Column, error) { //nolint:unparam
name := strings.TrimSpace(key.Name)
if col, ok := ruleStateHistoryColumns[name]; ok {
return col, nil
@@ -41,7 +46,7 @@ func (m *storage) getColumn(_ context.Context, key *telemetrytypes.TelemetryFiel
return ruleStateHistoryColumns["labels"], nil
}
func (m *storage) read(ctx context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
col, err := m.getColumn(ctx, key)
if err != nil {
return "", err
@@ -52,41 +57,36 @@ func (m *storage) read(ctx context.Context, _ qbtypes.QueryInfo, key *telemetryt
return col.Name, nil
}
// Read composes the bare read of one key with its membership test. A label
// inside the JSON is checked with JSONHas; absent, it reads the empty string,
// and that is the keyless contract here, so no query guards it. Every real
// column is always present.
func (m *storage) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (qbtypes.Read, error) {
func (m *fieldMapper) ColumnFor(ctx context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
col, err := m.getColumn(ctx, key)
if err != nil {
return qbtypes.Read{}, err
return nil, err
}
sql, err := m.read(ctx, q, key)
return []*schema.Column{col}, nil
}
// ExistsFor implements the per-key existence primitive of qbtypes.FieldMapper.
// A label inside the JSON gets a membership check, with the same condition
// that FieldFor uses for extraction; every real column always exists.
func (m *fieldMapper) ExistsFor(ctx context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey, exists bool) (string, error) {
col, err := m.getColumn(ctx, key)
if err != nil {
return qbtypes.Read{}, err
return "", err
}
if col.Name == "labels" && key.Name != "labels" {
presence := fmt.Sprintf("JSONHas(labels, %s)", querybuilder.ClickHouseStringLiteral(key.Name))
return qbtypes.Read{
SQL: sql,
Presence: presence,
Absence: "not " + presence,
WhenAbsent: qbtypes.AbsentIsValue,
}, nil
pred := fmt.Sprintf("JSONHas(labels, %s)", querybuilder.ClickHouseStringLiteral(key.Name))
if exists {
return pred, nil
}
return "not " + pred, nil
}
return qbtypes.Read{SQL: sql, Presence: "true", Absence: "false", WhenAbsent: qbtypes.AlwaysPresent}, nil
return "true", nil
}
// Fallback returns nil: rule-state history has no attribute-map fallback, so
// a key metadata does not hold stays unresolved and the caller errors.
func (m *storage) Fallback(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, qbtypes.FilterOperator, any) ([]*telemetrytypes.LogicalField, error) {
return nil, nil
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{}
}
func (m *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
return querybuilder.SharedCondition(ctx, q, m, logical, operator, value, sb)
func (m *fieldMapper) ColumnExpressionFor(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, field *telemetrytypes.TelemetryFieldKey, _ telemetrytypes.FieldDataType, _ map[string][]*telemetrytypes.TelemetryFieldKey) (string, error) {
colName, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, field)
if err != nil {
return "", err
}
return fmt.Sprintf("%s AS `%s`", sqlbuilder.Escape(colName), field.Name), nil
}

View File

@@ -25,15 +25,18 @@ const (
type store struct {
telemetryStore telemetrystore.TelemetryStore
telemetryMetadataStore telemetrytypes.MetadataStore
storage qbtypes.Storage
fieldMapper qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
logger *slog.Logger
}
func NewStore(telemetryStore telemetrystore.TelemetryStore, telemetryMetadataStore telemetrytypes.MetadataStore, logger *slog.Logger) rulestatehistorytypes.Store {
fm := newFieldMapper()
return &store{
telemetryStore: telemetryStore,
telemetryMetadataStore: telemetryMetadataStore,
storage: newStorage(),
fieldMapper: fm,
conditionBuilder: newConditionBuilder(fm),
logger: logger,
}
}
@@ -497,13 +500,15 @@ func (s *store) buildFilterClause(ctx context.Context, orgID valuer.UUID, filter
}
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, orgID, nil, telemetrytypes.SignalUnspecified, nil, querybuilder.ToNanoSecs(uint64(startMillis)), querybuilder.ToNanoSecs(uint64(endMillis))),
Storage: s.storage,
Logger: s.logger,
FieldKeys: fieldKeys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels", FieldContext: telemetrytypes.FieldContextAttribute},
Logger: s.logger,
FieldMapper: s.fieldMapper,
ConditionBuilder: s.conditionBuilder,
FieldKeys: fieldKeys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels", FieldContext: telemetrytypes.FieldContextAttribute},
}
opts.StartNs = querybuilder.ToNanoSecs(uint64(startMillis))
opts.EndNs = querybuilder.ToNanoSecs(uint64(endMillis))
prepared, err := querybuilder.PrepareWhereClause(expression, opts)
if err != nil {
return nil, err

View File

@@ -17,11 +17,11 @@ import (
)
type aggExprRewriter struct {
logger *slog.Logger
fullTextColumn *telemetrytypes.TelemetryFieldKey
storage qbtypes.Storage
flagger flagger.Flagger
signal telemetrytypes.Signal
logger *slog.Logger
fullTextColumn *telemetrytypes.TelemetryFieldKey
fieldMapper qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
flagger flagger.Flagger
}
var _ qbtypes.AggExprRewriter = (*aggExprRewriter)(nil)
@@ -29,18 +29,18 @@ var _ qbtypes.AggExprRewriter = (*aggExprRewriter)(nil)
func NewAggExprRewriter(
settings factory.ProviderSettings,
fullTextColumn *telemetrytypes.TelemetryFieldKey,
storage qbtypes.Storage,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
fl flagger.Flagger,
signal telemetrytypes.Signal,
) *aggExprRewriter {
set := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/querybuilder/agg_rewrite")
return &aggExprRewriter{
logger: set.Logger(),
fullTextColumn: fullTextColumn,
storage: storage,
flagger: fl,
signal: signal,
logger: set.Logger(),
fullTextColumn: fullTextColumn,
fieldMapper: fieldMapper,
conditionBuilder: conditionBuilder,
flagger: fl,
}
}
@@ -78,14 +78,18 @@ func (r *aggExprRewriter) Rewrite(
return "", nil, errors.NewInternalf(errors.CodeInternal, "no SELECT items for %q", expr)
}
visitor := &exprVisitor{
ctx: ctx,
query: NewQueryInfo(ctx, orgID, r.flagger, r.signal, nil, startNs, endNs),
logger: r.logger,
fieldKeys: keys,
fullTextColumn: r.fullTextColumn,
storage: r.storage,
}
visitor := newExprVisitor(
ctx,
orgID,
startNs,
endNs,
r.logger,
keys,
r.fullTextColumn,
r.fieldMapper,
r.conditionBuilder,
r.flagger,
)
// Rewrite the first select item (our expression)
if err := sel.SelectItems[0].Accept(visitor); err != nil {
return "", nil, err
@@ -126,19 +130,48 @@ func (r *aggExprRewriter) RewriteMulti(
return out, chArgsList, nil
}
// exprVisitor walks FunctionExpr nodes and resolves and renders their
// arguments.
// exprVisitor walks FunctionExpr nodes and applies the mappers.
type exprVisitor struct {
ctx context.Context
query qbtypes.QueryInfo
ctx context.Context
orgID valuer.UUID
startNs uint64
endNs uint64
chparser.DefaultASTVisitor
logger *slog.Logger
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey
fullTextColumn *telemetrytypes.TelemetryFieldKey
storage qbtypes.Storage
Modified bool
chArgs []any
isRate bool
logger *slog.Logger
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey
fullTextColumn *telemetrytypes.TelemetryFieldKey
fieldMapper qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
flagger flagger.Flagger
Modified bool
chArgs []any
isRate bool
}
func newExprVisitor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
logger *slog.Logger,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
fullTextColumn *telemetrytypes.TelemetryFieldKey,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
fl flagger.Flagger,
) *exprVisitor {
return &exprVisitor{
ctx: ctx,
orgID: orgID,
startNs: startNs,
endNs: endNs,
logger: logger,
fieldKeys: fieldKeys,
fullTextColumn: fullTextColumn,
fieldMapper: fieldMapper,
conditionBuilder: conditionBuilder,
flagger: fl,
}
}
// VisitFunctionExpr is invoked for each function call in the AST.
@@ -178,12 +211,15 @@ func (v *exprVisitor) VisitFunctionExpr(fn *chparser.FunctionExpr) error {
whereClause, err := PrepareWhereClause(
origPred,
FilterExprVisitorOpts{
Context: v.ctx,
Query: v.query,
Storage: v.storage,
Logger: v.logger,
FieldKeys: v.fieldKeys,
FullTextColumn: v.fullTextColumn,
Context: v.ctx,
OrgID: v.orgID,
Logger: v.logger,
FieldKeys: v.fieldKeys,
FieldMapper: v.fieldMapper,
ConditionBuilder: v.conditionBuilder,
FullTextColumn: v.fullTextColumn,
StartNs: v.startNs,
EndNs: v.endNs,
},
)
if err != nil {
@@ -208,7 +244,7 @@ func (v *exprVisitor) VisitFunctionExpr(fn *chparser.FunctionExpr) error {
for i := 0; i < len(args)-1; i++ {
origVal := chparser.Format(args[i])
fieldKey := telemetrytypes.GetFieldKeyFromKeyText(origVal)
expr, err := ResolveColumn(v.ctx, v.query, v.storage, &fieldKey, dataType, v.fieldKeys)
expr, err := v.fieldMapper.ColumnExpressionFor(v.ctx, v.orgID, v.startNs, v.endNs, &fieldKey, dataType, v.fieldKeys)
if err != nil {
return errors.WrapInvalidInputf(err, errors.CodeInvalidInput, "failed to get table field name for %q", origVal)
}
@@ -225,7 +261,7 @@ func (v *exprVisitor) VisitFunctionExpr(fn *chparser.FunctionExpr) error {
for i, arg := range args {
orig := chparser.Format(arg)
fieldKey := telemetrytypes.GetFieldKeyFromKeyText(orig)
expr, err := ResolveColumn(v.ctx, v.query, v.storage, &fieldKey, dataType, v.fieldKeys)
expr, err := v.fieldMapper.ColumnExpressionFor(v.ctx, v.orgID, v.startNs, v.endNs, &fieldKey, dataType, v.fieldKeys)
if err != nil {
return err
}

View File

@@ -1,90 +0,0 @@
package querybuilder
import (
"context"
"fmt"
"strings"
"github.com/SigNoz/signoz/pkg/errors"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// Column renders a resolved key as one bare column expression; the caller
// aliases. The coerced stages (group by, order, aggregation arguments) cast
// every candidate to the target type. The guard follows Absent: a sentinel
// read sits behind its presence test, so an absent row yields NULL; a column
// every row has reads bare and ends the candidate list; a NULL-reading field
// takes a presence branch only beside other candidates.
func Column(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
resolved qbtypes.Resolved,
target telemetrytypes.FieldDataType,
) (string, error) {
if len(resolved.Fields) == 0 {
return "", NewKeyNotFoundError(resolved.Key.Name, nil)
}
var targetValue any = ""
if target == telemetrytypes.FieldDataTypeFloat64 {
targetValue = 0.0
}
coerced := target != telemetrytypes.FieldDataTypeUnspecified
several := len(resolved.Fields) > 1
branches := make([]string, 0, len(resolved.Fields)*2)
filterOnly := false
for _, logical := range resolved.Fields {
read, err := LogicalRead(ctx, q, storage, logical)
if err != nil {
return "", err
}
if read.FilterOnly {
filterOnly = true
continue
}
// an array cannot sit inside Nullable or multiIf
if !several && bareRead(logical) {
return read.SQL, nil
}
expr := read.SQL
if coerced && !read.KeepType {
expr, _ = DataTypeCollisionHandledFieldName(logical.Single(), targetValue, expr, qbtypes.FilterOperatorUnknown)
}
// several native shapes share one multiIf, so every branch reads as text
branch := expr
if !coerced && several {
branch, _ = DataTypeCollisionHandledFieldName(logical.Single(), "", expr, qbtypes.FilterOperatorUnknown)
}
switch read.WhenAbsent {
case qbtypes.AlwaysPresent, qbtypes.AbsentIsValue:
if len(branches) == 0 {
return expr, nil
}
return fmt.Sprintf("multiIf(%s, %s)", strings.Join(branches, ", "), branch), nil
case qbtypes.AbsentIsNull:
if !several {
return expr, nil
}
branches = append(branches, read.Presence, branch)
default:
branches = append(branches, read.Presence, branch)
}
}
if len(branches) == 0 {
if filterOnly {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "`%s` can be filtered but not selected or grouped", resolved.Key.Name)
}
return "", NewKeyNotFoundError(resolved.Key.Name, nil)
}
return fmt.Sprintf("multiIf(%s, NULL)", strings.Join(branches, ", ")), nil
}
func bareRead(logical *telemetrytypes.LogicalField) bool {
key := logical.Single()
return strings.Contains(key.Name, telemetrytypes.ArraySep) ||
strings.Contains(key.Name, telemetrytypes.ArrayAnyIndex) ||
key.FieldDataType.IsArray()
}

View File

@@ -1,121 +0,0 @@
package querybuilder
import (
"context"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
)
// Condition compiles a resolved key into the conditions of one filter term:
// the storage's part in the fingerprint split narrows the fields, and every
// field compiles through the storage's Compile. It returns the per-field
// warnings; the resolution carries its own.
func Condition(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
resolved qbtypes.Resolved,
dropResourceFields bool,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
if resolved.Skipped {
return nil, nil, nil
}
if operator.IsFunctionOperator() && operator != qbtypes.FilterOperatorSearch {
for _, logical := range resolved.Fields {
switch logical.FieldContext {
case telemetrytypes.FieldContextResource, telemetrytypes.FieldContextAttribute, telemetrytypes.FieldContextScope:
// a body function on a map-backed key is a user error; the split must not hide it
return nil, nil, NewFunctionUnsupportedError(operator)
}
}
}
fields := resolved.Fields
switch storage.Traits().Split {
case qbtypes.MainOfSplit:
// the sub-query cannot know fallback keys, so those stay
if dropResourceFields && !resolved.FromFallback {
filtered := make([]*telemetrytypes.LogicalField, 0, len(fields))
for _, logical := range fields {
if logical.FieldContext != telemetrytypes.FieldContextResource {
filtered = append(filtered, logical)
}
}
if len(filtered) == 0 {
return nil, nil, nil
}
fields = filtered
}
case qbtypes.FingerprintOfSplit:
filtered := make([]*telemetrytypes.LogicalField, 0, len(fields))
for _, logical := range fields {
if logical.FieldContext == telemetrytypes.FieldContextResource {
filtered = append(filtered, logical)
}
}
fields = filtered
}
conds := make([]string, 0, len(fields))
var warnings []string
for _, logical := range fields {
compiled, err := storage.Compile(ctx, q, logical, operator, value, sb)
if err != nil {
return nil, nil, err
}
if compiled.Condition != "" {
conds = append(conds, compiled.Condition)
}
warnings = append(warnings, compiled.Warnings...)
}
return conds, warnings, nil
}
// RejectsBodyFunction reports the error a storage without body functions
// returns for one, before resolution; the fingerprint side of a split skips
// the term instead, because the main query evaluates it.
func RejectsBodyFunction(traits qbtypes.Traits, operator qbtypes.FilterOperator) (skip bool, err error) {
if !operator.IsFunctionOperator() && operator != qbtypes.FilterOperatorSearch {
return false, nil
}
if traits.SupportsBodyFunctions {
return false, nil
}
if traits.Split == qbtypes.FingerprintOfSplit {
return true, nil
}
return false, NewFunctionUnsupportedError(operator)
}
// Conditions resolves one key and compiles it: the condition builder for callers
// that do not run the filter visitor. The warnings carry the resolution's
// warnings first.
func Conditions(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
dropResourceFields bool,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
if _, err := RejectsBodyFunction(storage.Traits(), operator); err != nil {
return nil, nil, err
}
resolved, err := Resolve(ctx, q, storage, key, operator, value, fieldKeys)
if err != nil {
return nil, nil, err
}
conds, warnings, err := Condition(ctx, q, storage, resolved, dropResourceFields, operator, value, sb)
if err != nil {
return nil, nil, err
}
return conds, append(resolved.Warnings, warnings...), nil
}

View File

@@ -80,11 +80,6 @@ func ExistsExpression(columns []*schema.Column, key *telemetrytypes.TelemetryFie
return comparison("<>", "0"), nil
}
return comparison("=", "0"), nil
case schema.ColumnTypeEnumArray:
if exists {
return fmt.Sprintf("notEmpty(%s)", fieldExpression), nil
}
return fmt.Sprintf("empty(%s)", fieldExpression), nil
case schema.ColumnTypeEnumMap:
keyType := column.Type.(schema.MapColumnType).KeyType
if _, ok := keyType.(schema.LowCardinalityColumnType); !ok {

View File

@@ -1,165 +0,0 @@
package querybuilder
import (
"context"
"fmt"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
)
// SharedCondition is the Compile of every storage without its own condition
// language: the field's read, the shared data-type collision cast, the
// operator, then the guard rule. A sentinel-reading field takes the exists
// guard on the operators that would otherwise match the sentinel.
func SharedCondition(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
field *telemetrytypes.LogicalField,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (qbtypes.Compiled, error) {
condition, err := sharedOperator(ctx, q, storage, field, operator, value, sb)
if err != nil || condition == "" {
return qbtypes.Compiled{}, err
}
if !operator.AddDefaultExistsFilter() {
return qbtypes.Compiled{Condition: condition}, nil
}
read, err := LogicalRead(ctx, q, storage, field)
if err != nil {
return qbtypes.Compiled{}, err
}
if read.WhenAbsent != qbtypes.AbsentIsSentinel {
return qbtypes.Compiled{Condition: condition}, nil
}
return qbtypes.Compiled{Condition: sb.And(condition, sqlbuilder.Escape(read.Presence))}, nil
}
// sharedOperator expands a list per item, so each item takes its own cast,
// and casts the read against the operand for everything else.
func sharedOperator(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
field *telemetrytypes.LogicalField,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
switch operator {
case qbtypes.FilterOperatorIn, qbtypes.FilterOperatorNotIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
itemOperator := qbtypes.FilterOperatorEqual
if operator == qbtypes.FilterOperatorNotIn {
itemOperator = qbtypes.FilterOperatorNotEqual
}
conditions := make([]string, 0, len(values))
for _, item := range values {
condition, err := sharedOperator(ctx, q, storage, field, itemOperator, item, sb)
if err != nil {
return "", err
}
conditions = append(conditions, condition)
}
// `=`+OR and `!=`+AND instead of IN and NOT IN, to make use of the index
if operator == qbtypes.FilterOperatorIn {
return sb.Or(conditions...), nil
}
return sb.And(conditions...), nil
}
if operator.IsStringSearchOperator() {
value = FormatValueForContains(value)
}
logicalRead, err := LogicalRead(ctx, q, storage, field)
if err != nil {
return "", err
}
// Coercion switches only on the data type, which every member shares, so
// the first member stands in for the field.
read, value := DataTypeCollisionHandledFieldName(field.Single(), value, logicalRead.SQL, operator)
return OperatorCondition(ctx, q, storage, field, read, operator, value, sb)
}
// OperatorCondition renders one operator over an already cast read. It is
// the shared switch a storage with its own cast policy composes with. A list
// operator is the caller's to expand, so each item takes its own cast.
func OperatorCondition(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
field *telemetrytypes.LogicalField,
read string,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
switch operator {
case qbtypes.FilterOperatorEqual:
return sb.E(read, value), nil
case qbtypes.FilterOperatorNotEqual:
return sb.NE(read, value), nil
case qbtypes.FilterOperatorGreaterThan:
return sb.G(read, value), nil
case qbtypes.FilterOperatorGreaterThanOrEq:
return sb.GE(read, value), nil
case qbtypes.FilterOperatorLessThan:
return sb.LT(read, value), nil
case qbtypes.FilterOperatorLessThanOrEq:
return sb.LE(read, value), nil
case qbtypes.FilterOperatorLike:
return sb.Like(read, value), nil
case qbtypes.FilterOperatorNotLike:
return sb.NotLike(read, value), nil
case qbtypes.FilterOperatorILike:
return sb.ILike(read, value), nil
case qbtypes.FilterOperatorNotILike:
return sb.NotILike(read, value), nil
case qbtypes.FilterOperatorContains:
return sb.ILike(read, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorNotContains:
return sb.NotILike(read, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorRegexp:
// Note: Escape $$ to $$$$ to avoid sqlbuilder interpreting materialized $ signs
// Only needed because we are using sprintf instead of sb.Match (not implemented in sqlbuilder)
return fmt.Sprintf(`match(%s, %s)`, sqlbuilder.Escape(read), sb.Var(value)), nil
case qbtypes.FilterOperatorNotRegexp:
return fmt.Sprintf(`NOT match(%s, %s)`, sqlbuilder.Escape(read), sb.Var(value)), nil
case qbtypes.FilterOperatorBetween:
values, ok := value.([]any)
if !ok || len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.Between(read, values[0], values[1]), nil
case qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok || len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.NotBetween(read, values[0], values[1]), nil
// exists and not exists are key membership checks, so the storage's
// presence test answers them
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
logicalRead, err := LogicalRead(ctx, q, storage, field)
if err != nil {
return "", err
}
if operator == qbtypes.FilterOperatorNotExists {
return sqlbuilder.Escape(logicalRead.Absence), nil
}
return sqlbuilder.Escape(logicalRead.Presence), nil
}
return "", qbtypes.ErrUnsupportedOperator
}

View File

@@ -4,7 +4,6 @@ import (
"encoding/json"
"fmt"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -65,28 +64,6 @@ func ResolveLogicalFields(field *telemetrytypes.TelemetryFieldKey, logicalFields
return logicalFields, warning
}
// ColumnDataType is the field data type a table column reads as. A storage
// stamps it on the column key its Fallback returns, so the intrinsic-column
// step can drop a same-named metadata key of a contradicting type. A time
// column has no field data type and matches none.
func ColumnDataType(column *schema.Column) telemetrytypes.FieldDataType {
switch column.Type.GetType() {
case schema.ColumnTypeEnumBool:
return telemetrytypes.FieldDataTypeBool
case schema.ColumnTypeEnumInt8, schema.ColumnTypeEnumInt16, schema.ColumnTypeEnumInt32, schema.ColumnTypeEnumInt64,
schema.ColumnTypeEnumUInt8, schema.ColumnTypeEnumUInt16, schema.ColumnTypeEnumUInt32, schema.ColumnTypeEnumUInt64,
schema.ColumnTypeEnumFloat32, schema.ColumnTypeEnumFloat64:
return telemetrytypes.FieldDataTypeNumber
case schema.ColumnTypeEnumString, schema.ColumnTypeEnumFixedString:
return telemetrytypes.FieldDataTypeString
case schema.ColumnTypeEnumLowCardinality:
if lc, ok := column.Type.(schema.LowCardinalityColumnType); ok && lc.ElementType.GetType() == schema.ColumnTypeEnumString {
return telemetrytypes.FieldDataTypeString
}
}
return telemetrytypes.FieldDataTypeUnspecified
}
// WrapAsLogicalFields wraps physical keys (candidate or synthesized) as
// single-member logical fields addressed by the requested spelling.
func WrapAsLogicalFields(requestedName string, keys []*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.LogicalField {
@@ -97,14 +74,22 @@ func WrapAsLogicalFields(requestedName string, keys []*telemetrytypes.TelemetryF
return fields
}
// NewKeyNotFoundError builds the error for a key that neither metadata nor
// the storage can serve, with the closest known names as suggestions.
func NewKeyNotFoundError(name string, known []string) error {
err := errors.NewInvalidInputf(errors.CodeInvalidInput, "key `%s` not found", name).WithUrl(KeyNotFoundDocURL)
if len(known) == 0 {
return err
// SingleKeys flattens logical fields to their single members. It is the
// adapter for signals whose fields are single-member by construction (every
// signal without family support); their condition builders keep compiling per
// physical key.
func SingleKeys(fields []*telemetrytypes.LogicalField) []*telemetrytypes.TelemetryFieldKey {
keys := make([]*telemetrytypes.TelemetryFieldKey, 0, len(fields))
for _, field := range fields {
keys = append(keys, field.Single())
}
return err.WithSuggestions(errors.NewSuggestionsOnLevenshteinDistance(name, errors.NounKeys, known)...)
return keys
}
// NewKeyNotFoundError builds the error a condition builder returns when a filter term
// references a key it has no matching field key for.
func NewKeyNotFoundError(name string) error {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "key `%s` not found", name).WithUrl(KeyNotFoundDocURL)
}
// NewKeyNotFoundWarning is the warning surfaced when a referenced key is absent from

View File

@@ -7,99 +7,90 @@ import (
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
// LogicalRead is the only place family expressions are built. It composes
// exclusively from the storage's per-key Read, so every member honors its
// own storage: materialized columns, evolutions, and JSON plans ride the
// member keys, and a signal supports families the moment its reads are
// correct.
//
// A single-member field reads through its member. A family merges the member
// reads current-first, tests presence as any member present (and absence as
// none present), and reads for a row without any member what the merge's tail
// reads: the sentinel for a
// string family, NULL for the others. A member with a value map reads in the
// current vocabulary.
func LogicalRead(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, logical *telemetrytypes.LogicalField) (qbtypes.Read, error) {
// The two functions below are the only place family expressions are built.
// They compose exclusively from the mapper's per-key primitives (FieldFor,
// ExistsFor), so every member honors its own storage: materialized columns,
// evolutions, and JSON plans ride the member keys, and a signal supports
// families the moment its primitives are correct.
// LogicalValueExpr returns the value expression for a resolved logical field:
// the member's own expression for a single-member field, and a current-first
// merge across the members' expressions for a family.
func LogicalValueExpr(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
fm qbtypes.FieldMapper,
logical *telemetrytypes.LogicalField,
) (string, error) {
if !logical.IsFamily() {
return memberRead(ctx, q, storage, logical, 0)
}
reads := make([]qbtypes.Read, 0, len(logical.Members))
for i := range logical.Members {
read, err := memberRead(ctx, q, storage, logical, i)
if err != nil {
return qbtypes.Read{}, err
}
reads = append(reads, read)
return fm.FieldFor(ctx, orgID, tsStart, tsEnd, logical.Single())
}
merged := qbtypes.Read{WhenAbsent: familyAbsence(logical)}
guards := make([]string, 0, len(reads))
for _, read := range reads {
guards = append(guards, read.Presence)
merged.KeepType = merged.KeepType || read.KeepType
}
merged.Presence = "(" + strings.Join(guards, " OR ") + ")"
merged.Absence = "NOT " + merged.Presence
merged.FilterOnly = true
for _, read := range reads {
merged.FilterOnly = merged.FilterOnly && read.FilterOnly
memberExprs := make([]string, 0, len(logical.Members))
for _, member := range logical.Members {
expr, err := fm.FieldFor(ctx, orgID, tsStart, tsEnd, member)
if err != nil {
return "", err
}
memberExprs = append(memberExprs, expr)
}
if logical.FieldDataType == telemetrytypes.FieldDataTypeString {
// The trailing '' keeps single-key semantics for rows without any
// member: string maps read '' for an absent key, and negative
// operators must keep including such rows. A NULL tail would drop
// them: NULL != 'x' evaluates to NULL, and the row falls out of the
// result.
values := make([]string, 0, len(reads))
for _, read := range reads {
values = append(values, fmt.Sprintf("NULLIF(%s, '')", read.SQL))
// operators must keep including such rows (see AddDefaultExistsFilter).
// A NULL tail would drop them: NULL != 'x' evaluates to NULL, and the
// row falls out of the result.
values := make([]string, 0, len(memberExprs))
for _, expr := range memberExprs {
values = append(values, fmt.Sprintf("NULLIF(%s, '')", expr))
}
merged.SQL = "COALESCE(" + strings.Join(values, ", ") + ", '')"
return merged, nil
return "COALESCE(" + strings.Join(values, ", ") + ", '')", nil
}
// Numeric and boolean maps return zero for an absent key. If a family of
// either type is enabled, this tail must become zero too.
branches := make([]string, 0, len(reads)*2)
for _, read := range reads {
branches = append(branches, read.Presence, read.SQL)
branches := make([]string, 0, len(logical.Members)*2)
for i, member := range logical.Members {
guard, err := fm.ExistsFor(ctx, orgID, tsStart, tsEnd, member, true)
if err != nil {
return "", err
}
branches = append(branches, guard, memberExprs[i])
}
merged.SQL = "multiIf(" + strings.Join(branches, ", ") + ", NULL)"
return merged, nil
return "multiIf(" + strings.Join(branches, ", ") + ", NULL)", nil
}
func memberRead(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, logical *telemetrytypes.LogicalField, i int) (qbtypes.Read, error) {
read, err := storage.Read(ctx, q, logical.Members[i])
if err != nil {
return qbtypes.Read{}, err
// LogicalExistsExpr returns the existence predicate for a resolved logical
// field: the member's own predicate for a single-member field, presence of
// any member for a family.
func LogicalExistsExpr(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
fm qbtypes.FieldMapper,
logical *telemetrytypes.LogicalField,
exists bool,
) (string, error) {
if !logical.IsFamily() {
return fm.ExistsFor(ctx, orgID, tsStart, tsEnd, logical.Single(), exists)
}
if i < len(logical.ValueMaps) && logical.ValueMaps[i] != nil {
read.SQL = TransformRead(read.SQL, logical.ValueMaps[i])
}
return read, nil
}
// TransformRead brings a member's read into the current vocabulary: a stored
// value maps to its current value, any other value reads as it is.
func TransformRead(read string, valueMap *telemetrytypes.ValueMap) string {
return fmt.Sprintf("transform(%s, %s, %s, %s)", read, clickHouseStringArray(valueMap.Stored), clickHouseStringArray(valueMap.Current), read)
}
func clickHouseStringArray(values []string) string {
items := make([]string, 0, len(values))
for _, value := range values {
items = append(items, ClickHouseStringLiteral(value))
guards := make([]string, 0, len(logical.Members))
for _, member := range logical.Members {
guard, err := fm.ExistsFor(ctx, orgID, tsStart, tsEnd, member, true)
if err != nil {
return "", err
}
guards = append(guards, guard)
}
return "[" + strings.Join(items, ", ") + "]"
}
// familyAbsence is what the merged read yields for a row without any
// member: the sentinel tail of a string family, NULL for the others.
func familyAbsence(logical *telemetrytypes.LogicalField) qbtypes.Absent {
if logical.FieldDataType == telemetrytypes.FieldDataTypeString {
return qbtypes.AbsentIsSentinel
combined := "(" + strings.Join(guards, " OR ") + ")"
if exists {
return combined, nil
}
return qbtypes.AbsentIsNull
return "NOT " + combined, nil
}

View File

@@ -4,30 +4,39 @@ import (
"context"
"testing"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// stubStorage provides the one read the shared composition builds on.
type stubStorage struct{}
// stubFieldMapper provides just the two per-key primitives the shared
// composition builds on; the remaining FieldMapper methods are unused here.
type stubFieldMapper struct{}
func (stubStorage) Read(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (qbtypes.Read, error) {
return qbtypes.Read{SQL: "value(" + key.Name + ")", Presence: "has(" + key.Name + ")", Absence: "NOT has(" + key.Name + ")", WhenAbsent: qbtypes.AbsentIsSentinel}, nil
func (stubFieldMapper) FieldFor(_ context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return "value(" + key.Name + ")", nil
}
func (stubStorage) Fallback(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, qbtypes.FilterOperator, any) ([]*telemetrytypes.LogicalField, error) {
return nil, nil
func (stubFieldMapper) ExistsFor(_ context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey, exists bool) (string, error) {
if exists {
return "has(" + key.Name + ")", nil
}
return "NOT has(" + key.Name + ")", nil
}
func (stubStorage) Traits() qbtypes.Traits {
return qbtypes.Traits{}
func (stubFieldMapper) ColumnFor(context.Context, valuer.UUID, uint64, uint64, *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
return nil, qbtypes.ErrColumnNotFound
}
func (s stubStorage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
return SharedCondition(ctx, q, s, logical, operator, value, sb)
func (stubFieldMapper) ColumnExpressionFor(context.Context, valuer.UUID, uint64, uint64, *telemetrytypes.TelemetryFieldKey, telemetrytypes.FieldDataType, map[string][]*telemetrytypes.TelemetryFieldKey) (string, error) {
return "", qbtypes.ErrColumnNotFound
}
func (stubFieldMapper) CandidateKeys(context.Context, valuer.UUID, *telemetrytypes.TelemetryFieldKey, any, map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
}
func stringFamily(names ...string) *telemetrytypes.LogicalField {
@@ -38,21 +47,21 @@ func stringFamily(names ...string) *telemetrytypes.LogicalField {
return &telemetrytypes.LogicalField{Name: names[0], FieldDataType: telemetrytypes.FieldDataTypeString, Members: members}
}
func TestLogicalReadSingleMemberDelegatesToRead(t *testing.T) {
func TestLogicalValueExprSingleMemberDelegatesToFieldFor(t *testing.T) {
logical := telemetrytypes.SingleLogicalField("a", &telemetrytypes.TelemetryFieldKey{Name: "a"})
read, err := LogicalRead(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, logical)
expr, err := LogicalValueExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, logical)
require.NoError(t, err)
assert.Equal(t, "value(a)", read.SQL)
assert.Equal(t, "value(a)", expr)
}
func TestLogicalReadStringFamilyMergesCurrentFirst(t *testing.T) {
read, err := LogicalRead(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, stringFamily("current", "old"))
func TestLogicalValueExprStringFamilyMergesCurrentFirst(t *testing.T) {
expr, err := LogicalValueExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, stringFamily("current", "old"))
require.NoError(t, err)
// The trailing '' preserves keyless-row semantics for negative operators.
assert.Equal(t, "COALESCE(NULLIF(value(current), ''), NULLIF(value(old), ''), '')", read.SQL)
assert.Equal(t, "COALESCE(NULLIF(value(current), ''), NULLIF(value(old), ''), '')", expr)
}
func TestLogicalReadNumericFamilyGuardsEveryMember(t *testing.T) {
func TestLogicalValueExprNumericFamilyGuardsEveryMember(t *testing.T) {
logical := &telemetrytypes.LogicalField{
Name: "current",
FieldDataType: telemetrytypes.FieldDataTypeNumber,
@@ -61,24 +70,26 @@ func TestLogicalReadNumericFamilyGuardsEveryMember(t *testing.T) {
{Name: "old", FieldDataType: telemetrytypes.FieldDataTypeNumber},
},
}
read, err := LogicalRead(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, logical)
expr, err := LogicalValueExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, logical)
require.NoError(t, err)
assert.Equal(t, "multiIf(has(current), value(current), has(old), value(old), NULL)", read.SQL)
assert.Equal(t, "multiIf(has(current), value(current), has(old), value(old), NULL)", expr)
}
func TestLogicalReadSingleMemberDelegatesAbsence(t *testing.T) {
func TestLogicalExistsExprSingleMemberDelegatesToExistsFor(t *testing.T) {
logical := telemetrytypes.SingleLogicalField("a", &telemetrytypes.TelemetryFieldKey{Name: "a"})
read, err := LogicalRead(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, logical)
expr, err := LogicalExistsExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, logical, false)
require.NoError(t, err)
assert.Equal(t, "NOT has(a)", read.Absence)
assert.Equal(t, "NOT has(a)", expr)
}
func TestLogicalReadFamilyPresenceIsAnyMember(t *testing.T) {
func TestLogicalExistsExprFamilyIsAnyMemberPresence(t *testing.T) {
family := stringFamily("current", "old")
read, err := LogicalRead(context.Background(), qbtypes.QueryInfo{}, stubStorage{}, family)
expr, err := LogicalExistsExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, family, true)
require.NoError(t, err)
assert.Equal(t, "(has(current) OR has(old))", read.Presence)
assert.Equal(t, "(has(current) OR has(old))", expr)
assert.Equal(t, "NOT (has(current) OR has(old))", read.Absence)
expr, err = LogicalExistsExpr(context.Background(), valuer.UUID{}, 0, 0, stubFieldMapper{}, family, false)
require.NoError(t, err)
assert.Equal(t, "NOT (has(current) OR has(old))", expr)
}

View File

@@ -48,7 +48,7 @@ func TestFamiliesOffByDefault(t *testing.T) {
}},
}
fields := matchingLogicalFields(false, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, flaggertest.New(t), &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
require.Len(t, fields, 1)
assert.False(t, fields[0].IsFamily())
assert.Equal(t, []string{"deployment.environment.name"}, memberNames(fields[0]))
@@ -76,7 +76,7 @@ func TestMatchingLogicalFieldsGroupsFamilyMembers(t *testing.T) {
}
for _, requested := range []string{"deployment.environment.name", "deployment.environment"} {
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: requested}, fieldKeys)
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, familiesOn(t), &telemetrytypes.TelemetryFieldKey{Name: requested}, fieldKeys)
require.Len(t, fields, 1, "a family is one logical field, requested via %s", requested)
logical := fields[0]
assert.Equal(t, requested, logical.Name, "response identity is the requested spelling")
@@ -106,7 +106,7 @@ func TestMatchingLogicalFieldsOrdersMembersByFamilyRank(t *testing.T) {
}},
}
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, familiesOn(t), &telemetrytypes.TelemetryFieldKey{
Name: "deployment.environment.name",
FieldContext: telemetrytypes.FieldContextResource,
}, fieldKeys)
@@ -131,7 +131,7 @@ func TestMatchingLogicalFieldsKeepsLogsLiteral(t *testing.T) {
"deployment.environment": {logsKey("deployment.environment")},
}
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, familiesOn(t), &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
require.Len(t, fields, 1)
assert.False(t, fields[0].IsFamily())
assert.Equal(t, []string{"deployment.environment.name"}, memberNames(fields[0]))
@@ -165,7 +165,7 @@ func TestResolveLogicalFieldsKeepsFamilyThroughAmbiguity(t *testing.T) {
}
requested := &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, requested, fieldKeys)
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, familiesOn(t), requested, fieldKeys)
require.Len(t, fields, 2, "resource family + attribute collision")
resolved, warning := ResolveLogicalFields(requested, fields)
@@ -193,7 +193,7 @@ func TestMatchingLogicalFieldsNeverMergesAcrossDataTypes(t *testing.T) {
}},
}
fields := matchingLogicalFields(true, telemetrytypes.SignalUnspecified, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
fields := MatchingLogicalFields(context.Background(), valuer.UUID{}, familiesOn(t), &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, fieldKeys)
require.Len(t, fields, 2)
for _, logical := range fields {
assert.False(t, logical.IsFamily())

View File

@@ -1,201 +0,0 @@
package querybuilder
import (
"context"
"slices"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"golang.org/x/exp/maps"
)
// NewQueryInfo binds the context of one query and evaluates the query-path
// flags one time. A nil flagger keeps resolution literal and the log body in
// its legacy column.
func NewQueryInfo(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger, signal telemetrytypes.Signal, metric *telemetrytypes.MetricContext, startNs, endNs uint64) qbtypes.QueryInfo {
q := qbtypes.QueryInfo{
StartNs: startNs,
EndNs: endNs,
Signal: signal,
Metric: metric,
FamiliesOn: semconvFamiliesEnabled(ctx, orgID, fl),
}
if fl != nil {
q.BodyJSONOn = fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
}
return q
}
// Resolve turns one requested key into its meanings, one time per stage
// entry. The order is the same for every storage and every stage:
//
// 1. matches: the metadata keys under the key's spellings, grouped into
// families when the flag is on; a key under one of the storage's own
// contexts matches its own context first, and only then as if it had
// none;
// 2. ambiguity: in a filter, several interpretations settle by the
// resource-over-attribute policy, with a warning; a column stage keeps
// every interpretation in metadata order and folds them;
// 3. intrinsic column first: for a bare key, a column every row has leads, whether
// metadata reports it or the storage's own tables do, and sentinel
// fields of a contradicting type drop;
// 4. fallback: with no match, the storage's fallback keys; the not-found
// warning fires only when every one of them is a guess.
func Resolve(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
) (qbtypes.Resolved, error) {
traits := storage.Traits()
lookup := key
matches := matchingLogicalFields(q.FamiliesOn, q.Signal, key, fieldKeys)
if len(matches) == 0 && slices.Contains(traits.OwnContexts, key.FieldContext) {
bare := *key
bare.FieldContext = telemetrytypes.FieldContextUnspecified
lookup = &bare
matches = matchingLogicalFields(q.FamiliesOn, q.Signal, lookup, fieldKeys)
}
resolved := qbtypes.Resolved{Key: key, Ambiguous: len(matches) > 1}
fields := matches
if operator != qbtypes.FilterOperatorUnknown {
var warning string
fields, warning = ResolveLogicalFields(key, matches)
if warning != "" {
resolved.Warnings = append(resolved.Warnings, warning)
}
}
if lookup.FieldContext == telemetrytypes.FieldContextUnspecified && len(fields) > 0 {
fields = intrinsicColumnFirst(ctx, q, storage, key, operator, value, fields)
}
if len(fields) > 0 {
resolved.Fields = fields
return resolved, nil
}
fallback, err := storage.Fallback(ctx, q, key, operator, value)
if err != nil {
return qbtypes.Resolved{}, err
}
if len(fallback) == 0 {
if traits.UnknownKey == qbtypes.IgnoreUnknownKey {
resolved.Skipped = true
return resolved, nil
}
return qbtypes.Resolved{}, NewKeyNotFoundError(key.Name, maps.Keys(fieldKeys))
}
resolved.FromFallback = true
resolved.Fields = fallback
if fallbackIsGuess(ctx, q, storage, fallback) {
resolved.Warnings = append(resolved.Warnings, NewKeyNotFoundWarning(key.Name))
}
return resolved, nil
}
// ResolveColumn resolves one key for a column stage and renders it.
func ResolveColumn(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
key *telemetrytypes.TelemetryFieldKey,
target telemetrytypes.FieldDataType,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
resolved, err := Resolve(ctx, q, storage, key, qbtypes.FilterOperatorUnknown, nil, fieldKeys)
if err != nil {
return "", err
}
return Column(ctx, q, storage, resolved, target)
}
// intrinsicColumnFirst puts a column every row has first for a bare key. The column comes from the
// matches when metadata reports it, else from the storage's own fallback,
// so a metadata gap degrades to the correct column and never to a corrupt
// metadata key. A match of a contradicting data type drops.
func intrinsicColumnFirst(
ctx context.Context,
q qbtypes.QueryInfo,
storage qbtypes.Storage,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
fields []*telemetrytypes.LogicalField,
) []*telemetrytypes.LogicalField {
column := alwaysPresent(ctx, q, storage, fields)
if column == nil {
// a fallback that cannot answer this term has no column for it; its
// error belongs to the no-match path
if fallback, err := storage.Fallback(ctx, q, key, operator, value); err == nil {
column = alwaysPresent(ctx, q, storage, fallback)
}
}
if column == nil {
return fields
}
out := make([]*telemetrytypes.LogicalField, 0, len(fields)+1)
out = append(out, column)
for _, logical := range fields {
if logical == column || sameFact(logical, column) {
continue
}
if dataTypesConsistent(column.FieldDataType, logical.FieldDataType) {
out = append(out, logical)
}
}
return out
}
// alwaysPresent returns the first field every row has. A field the storage
// cannot test for presence is not that field.
func alwaysPresent(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, fields []*telemetrytypes.LogicalField) *telemetrytypes.LogicalField {
for _, logical := range fields {
if logical.IsFamily() {
continue
}
read, err := storage.Read(ctx, q, logical.Single())
if err == nil && read.WhenAbsent == qbtypes.AlwaysPresent {
return logical
}
}
return nil
}
func sameFact(a, b *telemetrytypes.LogicalField) bool {
return !a.IsFamily() && !b.IsFamily() &&
a.FieldContext == b.FieldContext && a.Single().Name == b.Single().Name
}
// dataTypesConsistent reports whether a metadata key's data type can
// describe the same stored value as the column's: an untyped metadata key
// matches anything, a column without a field data type (a time column)
// matches nothing, and the numeric kinds match each other.
func dataTypesConsistent(column, entry telemetrytypes.FieldDataType) bool {
if entry == telemetrytypes.FieldDataTypeUnspecified {
return true
}
if column == telemetrytypes.FieldDataTypeUnspecified {
return false
}
if column == entry {
return true
}
return isNumber(column) && isNumber(entry)
}
func isNumber(dt telemetrytypes.FieldDataType) bool {
return dt == telemetrytypes.FieldDataTypeInt64 || dt == telemetrytypes.FieldDataTypeFloat64 || dt == telemetrytypes.FieldDataTypeNumber
}
// fallbackIsGuess reports whether every fallback key is a guess. A column
// the storage knows is not one, and its presence means the key was found.
func fallbackIsGuess(ctx context.Context, q qbtypes.QueryInfo, storage qbtypes.Storage, fields []*telemetrytypes.LogicalField) bool {
return alwaysPresent(ctx, q, storage, fields) == nil
}

View File

@@ -9,10 +9,12 @@ import (
"strings"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
grammar "github.com/SigNoz/signoz/pkg/parser/filterquery/grammar"
"github.com/SigNoz/signoz/pkg/semconv"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/antlr4-go/antlr/v4"
sqlbuilder "github.com/huandu/go-sqlbuilder"
@@ -26,8 +28,10 @@ const stringMatchingOperatorDocURL = "https://signoz.io/docs/userguide/operators
// to convert the parsed filter expressions into ClickHouse WHERE clause.
type filterExpressionVisitor struct {
context context.Context
query qbtypes.QueryInfo
storage qbtypes.Storage
orgID valuer.UUID
fl flagger.Flagger
fieldMapper qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
warnings []string
mainWarnURL string
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey
@@ -40,31 +44,39 @@ type filterExpressionVisitor struct {
variables map[string]qbtypes.VariableItem
keysWithWarnings map[string]bool
startNs uint64
endNs uint64
requiresCostGuard bool
}
type FilterExprVisitorOpts struct {
Context context.Context
// Query is the request's context with the query-path flags evaluated
// one time; Storage answers the signal's part of every term.
Query qbtypes.QueryInfo
Storage qbtypes.Storage
OrgID valuer.UUID
// Flagger evaluates the resolve_semconv_families flag during resolution.
// A nil Flagger keeps resolution literal.
Flagger flagger.Flagger
Logger *slog.Logger
FieldMapper qbtypes.FieldMapper
ConditionBuilder qbtypes.ConditionBuilder
FieldKeys map[string][]*telemetrytypes.TelemetryFieldKey
Builder *sqlbuilder.SelectBuilder
FullTextColumn *telemetrytypes.TelemetryFieldKey
SkipResourceFilter bool
SkipFullTextFilter bool
Variables map[string]qbtypes.VariableItem
StartNs uint64
EndNs uint64
}
// newFilterExpressionVisitor creates a new filterExpressionVisitor.
func newFilterExpressionVisitor(opts FilterExprVisitorOpts) *filterExpressionVisitor {
return &filterExpressionVisitor{
context: opts.Context,
query: opts.Query,
storage: opts.Storage,
orgID: opts.OrgID,
fl: opts.Flagger,
fieldMapper: opts.FieldMapper,
conditionBuilder: opts.ConditionBuilder,
fieldKeys: opts.FieldKeys,
builder: opts.Builder,
fullTextColumn: opts.FullTextColumn,
@@ -72,6 +84,8 @@ func newFilterExpressionVisitor(opts FilterExprVisitorOpts) *filterExpressionVis
skipFullTextFilter: opts.SkipFullTextFilter,
variables: opts.Variables,
keysWithWarnings: make(map[string]bool),
startNs: opts.StartNs,
endNs: opts.EndNs,
}
}
@@ -353,7 +367,7 @@ func (v *filterExpressionVisitor) VisitPrimary(ctx *grammar.PrimaryContext) any
return ErrorConditionLiteral
}
}
conds, ok := v.compile(storageKey(v.fullTextColumn), qbtypes.FilterOperatorRegexp, FormatFullTextSearch(searchText))
conds, ok := v.buildConditions(v.fullTextColumn, []*telemetrytypes.LogicalField{telemetrytypes.SingleLogicalField(v.fullTextColumn.Name, v.fullTextColumn)}, qbtypes.FilterOperatorRegexp, FormatFullTextSearch(searchText))
if !ok {
return ErrorConditionLiteral
}
@@ -372,6 +386,7 @@ func (v *filterExpressionVisitor) VisitPrimary(ctx *grammar.PrimaryContext) any
// VisitComparison handles all comparison operators.
func (v *filterExpressionVisitor) VisitComparison(ctx *grammar.ComparisonContext) any {
key := v.Visit(ctx.Key()).(*telemetrytypes.TelemetryFieldKey)
matching := MatchingLogicalFields(v.context, v.orgID, v.fl, key, v.fieldKeys)
// Handle EXISTS specially
if ctx.EXISTS() != nil {
@@ -380,7 +395,7 @@ func (v *filterExpressionVisitor) VisitComparison(ctx *grammar.ComparisonContext
op = qbtypes.FilterOperatorNotExists
}
conds, ok := v.buildConditions(key, op, nil)
conds, ok := v.buildConditions(key, matching, op, nil)
if !ok {
return ErrorConditionLiteral
}
@@ -453,7 +468,7 @@ func (v *filterExpressionVisitor) VisitComparison(ctx *grammar.ComparisonContext
op = qbtypes.FilterOperatorNotIn
}
conds, ok := v.buildConditions(key, op, values)
conds, ok := v.buildConditions(key, matching, op, values)
if !ok {
return ErrorConditionLiteral
}
@@ -501,7 +516,7 @@ func (v *filterExpressionVisitor) VisitComparison(ctx *grammar.ComparisonContext
return ErrorConditionLiteral
}
conds, ok := v.buildConditions(key, op, []any{value1, value2})
conds, ok := v.buildConditions(key, matching, op, []any{value1, value2})
if !ok {
return ErrorConditionLiteral
}
@@ -585,7 +600,7 @@ func (v *filterExpressionVisitor) VisitComparison(ctx *grammar.ComparisonContext
}
}
conds, ok := v.buildConditions(key, op, value)
conds, ok := v.buildConditions(key, matching, op, value)
if !ok {
return ErrorConditionLiteral
}
@@ -667,7 +682,7 @@ func (v *filterExpressionVisitor) VisitFullText(ctx *grammar.FullTextContext) an
v.errors = append(v.errors, "full text search is not supported")
return ErrorConditionLiteral
}
conds, ok := v.compile(storageKey(v.fullTextColumn), qbtypes.FilterOperatorRegexp, FormatFullTextSearch(text))
conds, ok := v.buildConditions(v.fullTextColumn, []*telemetrytypes.LogicalField{telemetrytypes.SingleLogicalField(v.fullTextColumn.Name, v.fullTextColumn)}, qbtypes.FilterOperatorRegexp, FormatFullTextSearch(text))
if !ok {
return ErrorConditionLiteral
}
@@ -722,7 +737,7 @@ func (v *filterExpressionVisitor) VisitFunctionCall(ctx *grammar.FunctionCallCon
return ErrorConditionLiteral
}
conds, ok := v.buildConditions(key, operator, value)
conds, ok := v.buildConditions(key, MatchingLogicalFields(v.context, v.orgID, v.fl, key, v.fieldKeys), operator, value)
if !ok {
return ErrorConditionLiteral
}
@@ -778,12 +793,6 @@ func normalizeFunctionValue(operator qbtypes.FilterOperator, functionName string
// search term plus optional field-context scopes, ORing one FilterOperatorSearch per
// scope (no scope = keyless, covering every field).
func (v *filterExpressionVisitor) VisitSearchCall(ctx *grammar.SearchCallContext) any {
if skip, err := RejectsBodyFunction(v.storage.Traits(), qbtypes.FilterOperatorSearch); err != nil {
v.recordError(err)
return ErrorConditionLiteral
} else if skip {
return SkipConditionLiteral
}
// Flag scan-heavy so the statement builder attaches the cost guard.
v.requiresCostGuard = true
@@ -826,7 +835,7 @@ func (v *filterExpressionVisitor) VisitSearchCall(ctx *grammar.SearchCallContext
var conds []string
for _, fieldContext := range fieldContexts {
key := telemetrytypes.NewTelemetryFieldKey("", fieldContext, telemetrytypes.FieldDataTypeUnspecified)
scoped, cok := v.compile(storageKey(key), qbtypes.FilterOperatorSearch, searchText)
scoped, cok := v.buildConditions(key, nil, qbtypes.FilterOperatorSearch, searchText)
if !cok {
return ErrorConditionLiteral
}
@@ -918,55 +927,20 @@ func (v *filterExpressionVisitor) VisitKey(ctx *grammar.KeyContext) any {
return &fieldKey
}
// buildConditions resolves and compiles one filter term, folding its
// warnings and errors into the visitor state; ok is false when an error was
// recorded.
func (v *filterExpressionVisitor) buildConditions(key *telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, value any) ([]string, bool) {
if skip, err := RejectsBodyFunction(v.storage.Traits(), op); err != nil {
v.recordError(err)
return nil, false
} else if skip {
return nil, true
}
resolved, err := Resolve(v.context, v.query, v.storage, key, op, value, v.fieldKeys)
// buildConditions invokes the condition builder for a filter term, folding its
// warnings/errors into visitor state; returns false if an error was recorded.
func (v *filterExpressionVisitor) buildConditions(key *telemetrytypes.TelemetryFieldKey, matching []*telemetrytypes.LogicalField, op qbtypes.FilterOperator, value any) ([]string, bool) {
conds, warns, err := v.conditionBuilder.ConditionFor(v.context, v.orgID, v.startNs, v.endNs, key, v.fieldKeys, qbtypes.ConditionBuilderOptions{SkipResourceFilter: v.skipResourceFilter}, op, value, v.builder)
if err != nil {
v.recordError(err)
_, _, _, _, errURL, _ := errors.Unwrapb(err)
assignIfEmpty(&v.mainErrorURL, errURL)
v.errors = append(v.errors, err.Error())
return nil, false
}
v.addWarnings(resolved.Warnings, resolved.Ambiguous)
return v.compile(resolved, op, value)
}
// compile turns a resolved term into its conditions, folding the storage's
// warnings into the visitor state.
func (v *filterExpressionVisitor) compile(resolved qbtypes.Resolved, op qbtypes.FilterOperator, value any) ([]string, bool) {
conds, warns, err := Condition(v.context, v.query, v.storage, resolved, v.skipResourceFilter, op, value, v.builder)
if err != nil {
v.recordError(err)
return nil, false
}
v.addWarnings(warns, resolved.Ambiguous)
v.addWarnings(warns, len(matching) > 1)
return conds, true
}
func (v *filterExpressionVisitor) recordError(err error) {
_, _, _, _, errURL, _ := errors.Unwrapb(err)
assignIfEmpty(&v.mainErrorURL, errURL)
v.errors = append(v.errors, err.Error())
}
// storageKey is a key the storage knows without metadata, resolved as
// itself: the full-text column, or a search() scope that names a set of
// columns. It is a fallback key: the fingerprint sub-query cannot serve it,
// so the main query keeps it when the split runs.
func storageKey(key *telemetrytypes.TelemetryFieldKey) qbtypes.Resolved {
return qbtypes.Resolved{
Key: key,
Fields: []*telemetrytypes.LogicalField{telemetrytypes.SingleLogicalField(key.Name, key)},
FromFallback: true,
}
}
// addWarnings appends de-duplicated warnings to the visitor. ambiguous marks warnings
// from a multi-match key so the field-context doc URL is attached.
func (v *filterExpressionVisitor) addWarnings(warns []string, ambiguous bool) {
@@ -1014,14 +988,15 @@ func assignIfEmpty(s *string, value string) {
// familyMemberNames returns the physical spellings to look up for the
// referenced key: the semantic-convention family members (current-first) when
// families are on and the query can resolve to traces, else just the requested
// name. Only trace field mappers understand families today; logs and metrics
// keep the requested spelling until theirs land.
func familyMemberNames(familiesOn bool, signal telemetrytypes.Signal, field *telemetrytypes.TelemetryFieldKey) []string {
if !familiesOn {
// the resolve_semconv_families flag is on for the org and the key can resolve
// to traces, else just the requested name. Only trace field mappers understand
// families today; logs and metrics keep the requested spelling until theirs
// land.
func familyMemberNames(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger, field *telemetrytypes.TelemetryFieldKey) []string {
if !semconvFamiliesEnabled(ctx, orgID, fl) {
return []string{field.Name}
}
if signal != telemetrytypes.SignalUnspecified && signal != telemetrytypes.SignalTraces {
if field.Signal != telemetrytypes.SignalUnspecified && field.Signal != telemetrytypes.SignalTraces {
return []string{field.Name}
}
return semconv.Members(semconv.KindAttribute, telemetrytypes.FieldKeySelector{
@@ -1031,7 +1006,7 @@ func familyMemberNames(familiesOn bool, signal telemetrytypes.Signal, field *tel
})
}
// matchingLogicalFields resolves the referenced key against the metadata map
// MatchingLogicalFields resolves the referenced key against the metadata map
// into logical fields, honoring any context/data type the user specified.
//
// Physical keys that are members of one semantic-convention family (traces
@@ -1039,15 +1014,16 @@ func familyMemberNames(familiesOn bool, signal telemetrytypes.Signal, field *tel
// identity, members ordered current-first. Every other matching key becomes
// its own single-member logical field. Ambiguity is the length of the
// returned slice: one family is one element and is never ambiguous with
// itself, but the slice can hold several logical fields, including several
// itself, but the slice can hold several logical fields including several
// family fields, one per identity, when the family exists under more than
// one context or data type. Members alias the metadata map entries; nothing
// is copied or mutated.
//
// Family grouping only happens when families are on for the query. Off,
// every match stays a single-member logical field.
func matchingLogicalFields(familiesOn bool, signal telemetrytypes.Signal, field *telemetrytypes.TelemetryFieldKey, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.LogicalField {
members := familyMemberNames(familiesOn, signal, field)
// Family grouping only happens when the resolve_semconv_families flag is on
// for the org. A nil flagger means off: every match then stays a
// single-member logical field.
func MatchingLogicalFields(ctx context.Context, orgID valuer.UUID, fl flagger.Flagger, field *telemetrytypes.TelemetryFieldKey, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.LogicalField {
members := familyMemberNames(ctx, orgID, fl, field)
matches := collectMemberMatches(field, members, fieldKeys)
return groupIntoLogicalFields(field.Name, len(members) > 1, matches)
}

View File

@@ -10,6 +10,7 @@ import (
grammar "github.com/SigNoz/signoz/pkg/parser/filterquery/grammar"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/antlr4-go/antlr/v4"
sqlbuilder "github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/assert"
@@ -589,18 +590,15 @@ func TestVisitKey(t *testing.T) {
// and decides not-found handling. Replay that here against the generic
// builder behavior (error unless the key is ignored). The test maps carry
// no signal, so every logical field is single-member and flattens losslessly.
matching := matchingLogicalFields(false, telemetrytypes.SignalUnspecified, key, tt.fieldKeys)
matching := MatchingLogicalFields(context.Background(), valuer.UUID{}, nil, key, tt.fieldKeys)
resolved, warning := ResolveLogicalFields(key, matching)
keys := make([]*telemetrytypes.TelemetryFieldKey, 0, len(resolved))
for _, logical := range resolved {
keys = append(keys, logical.Single())
}
keys := SingleKeys(resolved)
var gotErrors []string
var gotMainErrURL, gotMainWrnURL string
var gotWarnings []string
if len(keys) == 0 && !tt.ignoreNotFoundKeys {
err := NewKeyNotFoundError(key.Name, nil)
err := NewKeyNotFoundError(key.Name)
gotErrors = append(gotErrors, err.Error())
_, _, _, _, gotMainErrURL, _ = errors.Unwrapb(err)
}
@@ -750,51 +748,99 @@ var visitTestKeys = map[string][]*telemetrytypes.TelemetryFieldKey{
"body": {{Name: "body", FieldContext: telemetrytypes.FieldContextLog, FieldDataType: telemetrytypes.FieldDataTypeString}},
}
// resourceStorage mirrors the fingerprint storage: only resource keys
// compile, and unknown keys and body functions are skipped.
type resourceStorage struct{}
type resourceConditionBuilder struct{}
func (resourceStorage) Read(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (qbtypes.Read, error) {
return qbtypes.Read{SQL: key.Name, Presence: "has(" + key.Name + ")", Absence: "NOT has(" + key.Name + ")", WhenAbsent: qbtypes.AbsentIsSentinel}, nil
}
func (b *resourceConditionBuilder) ConditionFor(
_ context.Context,
_ valuer.UUID,
_ uint64,
_ uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
_ any,
_ *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
func (resourceStorage) Fallback(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, qbtypes.FilterOperator, any) ([]*telemetrytypes.LogicalField, error) {
return nil, nil
}
func (resourceStorage) Traits() qbtypes.Traits {
return qbtypes.Traits{Split: qbtypes.FingerprintOfSplit, UnknownKey: qbtypes.IgnoreUnknownKey}
}
func (resourceStorage) Compile(_ context.Context, _ qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, _ qbtypes.FilterOperator, _ any, _ *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
return qbtypes.Compiled{Condition: fmt.Sprintf("%s_cond", logical.Single().Name)}, nil
}
// mainStorage mirrors a main-query storage: body functions apply to body keys
// only, an unknown key errors, and a body path without metadata matches is
// its own fallback.
type mainStorage struct{}
func (mainStorage) Read(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (qbtypes.Read, error) {
return qbtypes.Read{SQL: key.Name, Presence: "has(" + key.Name + ")", Absence: "NOT has(" + key.Name + ")", WhenAbsent: qbtypes.AbsentIsSentinel}, nil
}
func (mainStorage) Fallback(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, _ qbtypes.FilterOperator, _ any) ([]*telemetrytypes.LogicalField, error) {
if key.FieldContext != telemetrytypes.FieldContextBody {
return nil, nil
// mirror the real resource builder: function operators never apply to resources
if operator.IsFunctionOperator() {
return nil, nil, nil
}
return WrapAsLogicalFields(key.Name, []*telemetrytypes.TelemetryFieldKey{key}), nil
}
func (mainStorage) Traits() qbtypes.Traits {
return qbtypes.Traits{Split: qbtypes.MainOfSplit, SupportsBodyFunctions: true}
}
func (mainStorage) Compile(_ context.Context, _ qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, _ any, _ *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
if operator.IsFunctionOperator() && logical.FieldContext != telemetrytypes.FieldContextBody {
return qbtypes.Compiled{}, errors.NewInvalidInputf(errors.CodeInvalidInput, "function supports only body JSON search")
resolved, warning := ResolveLogicalFields(key, MatchingLogicalFields(context.Background(), valuer.UUID{}, nil, key, fieldKeys))
keys := SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
return qbtypes.Compiled{Condition: fmt.Sprintf("%s_cond", logical.Single().Name)}, nil
var conds []string
for _, k := range keys {
// only resource keys contribute; others (and unknown keys) are ignored
if k.FieldContext != telemetrytypes.FieldContextResource {
continue
}
conds = append(conds, fmt.Sprintf("%s_cond", k.Name))
}
return conds, warnings, nil
}
type conditionBuilder struct{}
func (b *conditionBuilder) ConditionFor(
_ context.Context,
_ valuer.UUID,
_ uint64,
_ uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
options qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
_ any,
_ *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken only support body fields; mirror the real
// condition builder which now owns this validation and errors for non-body keys.
switch operator {
case qbtypes.FilterOperatorHas, qbtypes.FilterOperatorHasAny, qbtypes.FilterOperatorHasAll, qbtypes.FilterOperatorHasToken:
if key.FieldContext != telemetrytypes.FieldContextBody {
return nil, nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "function supports only body JSON search")
}
return []string{fmt.Sprintf("%s_cond", key.Name)}, nil, nil
}
resolved, warning := ResolveLogicalFields(key, MatchingLogicalFields(context.Background(), valuer.UUID{}, nil, key, fieldKeys))
keys := SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
if len(keys) == 0 {
// errors on unknown keys (no IgnoreNotFoundKeys equivalent for this builder)
return nil, warnings, NewKeyNotFoundError(key.Name)
}
// A resource sub-query already covers the term; drop resource keys from the main query.
if options.SkipResourceFilter {
filtered := make([]*telemetrytypes.TelemetryFieldKey, 0, len(keys))
for _, k := range keys {
if k.FieldContext != telemetrytypes.FieldContextResource {
filtered = append(filtered, k)
}
}
if len(filtered) == 0 {
return nil, warnings, nil
}
keys = filtered
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
conds = append(conds, fmt.Sprintf("%s_cond", k.Name))
}
return conds, warnings, nil
}
// visitComparisonCase is a single test case for the TestVisitComparison_* family.
@@ -833,7 +879,7 @@ func visitComparisonOpts(t *testing.T) (rsbOpts, sbOpts FilterExprVisitorOpts) {
rsbOpts = FilterExprVisitorOpts{
Context: t.Context(),
FieldKeys: visitTestKeys,
Storage: &resourceStorage{},
ConditionBuilder: &resourceConditionBuilder{},
Variables: allVariable,
SkipResourceFilter: false,
SkipFullTextFilter: true,
@@ -841,7 +887,7 @@ func visitComparisonOpts(t *testing.T) (rsbOpts, sbOpts FilterExprVisitorOpts) {
sbOpts = FilterExprVisitorOpts{
Context: t.Context(),
FieldKeys: visitTestKeys,
Storage: &mainStorage{},
ConditionBuilder: &conditionBuilder{},
Variables: allVariable,
SkipResourceFilter: true,
SkipFullTextFilter: false,
@@ -1584,8 +1630,6 @@ func TestVisitComparison_FunctionCalls(t *testing.T) {
wantErrSB: true,
},
{
// SB: a body function on a resource key is a user error, even when the
// resource sub-query would otherwise cover x.
name: "has on resource key",
expr: "has(x, 'hello')",
wantRSB: "",

View File

@@ -253,6 +253,7 @@ func NewSQLMigrationProviderFactories(
sqlmigration.NewAddQuickFilterTuplesFactory(sqlstore),
sqlmigration.NewAddIngestionTuplesFactory(sqlstore),
sqlmigration.NewAddSubscriptionTuplesFactory(sqlstore),
sqlmigration.NewNormalizeQuickFilterFieldsFactory(sqlstore),
)
}

View File

@@ -0,0 +1,161 @@
package sqlmigration
import (
"context"
"encoding/json"
"log/slog"
"github.com/uptrace/bun"
"github.com/uptrace/bun/migrate"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/sqlstore"
)
type quickFilterSourceRow struct {
bun.BaseModel `bun:"table:quick_filter"`
ID string `bun:"id,pk"`
Source string `bun:"source"`
Filter string `bun:"filter"`
}
type quickFilterStaticField struct {
name string
fieldContext string
fieldDataType string
}
// quickFilterSpanFields are the span-level fields the fields API serves with
// the span context, keyed by every name a stored filter may carry for them.
var quickFilterSpanFields = func() map[string]quickFilterStaticField {
fields := map[string]quickFilterStaticField{}
for name, dataType := range map[string]string{
"trace_id": "string", "span_id": "string", "trace_state": "string", "parent_span_id": "string",
"flags": "number", "name": "string", "kind": "number", "kind_string": "string",
"duration_nano": "number", "status_code": "number", "status_message": "string", "status_code_string": "string",
"response_status_code": "string", "external_http_url": "string", "http_url": "string",
"external_http_method": "string", "http_method": "string", "http_host": "string",
"db_name": "string", "db_operation": "string", "has_error": "bool", "is_remote": "string",
} {
fields[name] = quickFilterStaticField{name: name, fieldContext: "span", fieldDataType: dataType}
}
for deprecated, current := range map[string]string{
"responseStatusCode": "response_status_code", "externalHttpUrl": "external_http_url", "httpUrl": "http_url",
"externalHttpMethod": "external_http_method", "httpMethod": "http_method", "httpHost": "http_host",
"dbName": "db_name", "dbOperation": "db_operation", "hasError": "has_error", "isRemote": "is_remote",
} {
fields[deprecated] = fields[current]
}
return fields
}()
// quickFilterLogFields are the log-level fields the fields API serves with
// the log context.
var quickFilterLogFields = map[string]quickFilterStaticField{
"body": {name: "body", fieldContext: "log", fieldDataType: "string"},
"severity_text": {name: "severity_text", fieldContext: "log", fieldDataType: "string"},
"severity_number": {name: "severity_number", fieldContext: "log", fieldDataType: "number"},
"trace_id": {name: "trace_id", fieldContext: "log", fieldDataType: "string"},
"span_id": {name: "span_id", fieldContext: "log", fieldDataType: "string"},
"trace_flags": {name: "trace_flags", fieldContext: "log", fieldDataType: "number"},
}
type normalizeQuickFilterFields struct {
settings factory.ProviderSettings
}
func NewNormalizeQuickFilterFieldsFactory(sqlstore sqlstore.SQLStore) factory.ProviderFactory[SQLMigration, Config] {
return factory.NewProviderFactory(factory.MustNewName("normalize_quick_filter_fields"), func(ctx context.Context, ps factory.ProviderSettings, c Config) (SQLMigration, error) {
return &normalizeQuickFilterFields{settings: ps}, nil
})
}
func (migration *normalizeQuickFilterFields) Register(migrations *migrate.Migrations) error {
return migrations.Register(migration.Up, migration.Down)
}
func (migration *normalizeQuickFilterFields) Up(ctx context.Context, db *bun.DB) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer func() { _ = tx.Rollback() }()
var rows []*quickFilterSourceRow
if err := tx.NewSelect().Model(&rows).Scan(ctx); err != nil {
return err
}
var migrated, skipped int
for _, row := range rows {
normalized, changed, ok := normalizeQuickFilterEntries(row.Source, row.Filter)
if !ok {
migration.settings.Logger.WarnContext(ctx, "quick filter could not be parsed, leaving it untouched", slog.String("quick_filter_id", row.ID), slog.String("raw_filter", row.Filter))
skipped++
continue
}
if !changed {
continue
}
migrated++
if _, err := tx.NewUpdate().Model((*quickFilterSourceRow)(nil)).Set("filter = ?", normalized).Where("id = ?", row.ID).Exec(ctx); err != nil {
return err
}
}
migration.settings.Logger.InfoContext(ctx, "normalized quick filter static fields", slog.Int("total", len(rows)), slog.Int("migrated", migrated), slog.Int("skipped", skipped))
return tx.Commit()
}
func (migration *normalizeQuickFilterFields) Down(context.Context, *bun.DB) error {
return nil
}
// normalizeQuickFilterEntries rewrites the static fields of a stored filter
// list to the name, context and data type the fields API serves them with:
// span fields for the trace-based sources, log fields for logs, whatever
// context the legacy seeds gave them. Other keys are left as they are;
// ok=false means unparseable.
func normalizeQuickFilterEntries(source string, filter string) (normalized string, changed bool, ok bool) {
var staticFields map[string]quickFilterStaticField
switch source {
case "traces", "api_monitoring", "exceptions", "ai_observability":
staticFields = quickFilterSpanFields
case "logs":
staticFields = quickFilterLogFields
default:
return "", false, true
}
var entries []telemetryFieldKeyOutput
if err := json.Unmarshal([]byte(filter), &entries); err != nil {
return "", false, false
}
for i, entry := range entries {
field, static := staticFields[entry.Name]
if !static {
continue
}
if entry.Name == field.name && entry.FieldContext == field.fieldContext && entry.FieldDataType == field.fieldDataType {
continue
}
entries[i].Name = field.name
entries[i].FieldContext = field.fieldContext
entries[i].FieldDataType = field.fieldDataType
changed = true
}
if !changed {
return "", false, true
}
normalizedJSON, err := marshalUnescaped(entries)
if err != nil {
return "", false, false
}
return string(normalizedJSON), true, true
}

View File

@@ -196,7 +196,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -267,7 +267,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
countIf(attribute_string_gen_ai$$request$$model_exists) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -340,7 +340,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -409,7 +409,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -479,7 +479,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -555,7 +555,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -632,7 +632,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS service.name,
any(multiIf(multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS service.name,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -832,7 +832,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `+"`service.name`"+`,
any(multiIf(multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `+"`service.name`"+`,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,
@@ -855,11 +855,12 @@ SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000
assert.Empty(t, attrStmt.Warnings)
// span. is the signal's own context, so it resolves to the same attribute
// (identical SQL) without a warning.
// span. corrects to the same attribute (identical SQL) but the span-context
// metadata lookup misses, surfacing a key-not-found warning.
spanStmt := build("span.output_tokens > 100")
assert.Equal(t, renderSQL(t, attrStmt), renderSQL(t, spanStmt))
assert.Empty(t, spanStmt.Warnings)
require.Len(t, spanStmt.Warnings, 1)
assert.Contains(t, spanStmt.Warnings[0], "key `output_tokens` not found in metadata")
// bare spelling is claimed by the aggregate alias
bareStmt := build("output_tokens > 100")
@@ -899,7 +900,7 @@ SELECT trace_id,
(max(toUnixTimestamp64Nano(timestamp) + duration_nano) - min(toUnixTimestamp64Nano(timestamp))) AS trace_duration_nano,
count() AS span_count,
anyIf(name, parent_span_id = '') AS root_span_name,
any(multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `+"`service.name`"+`,
any(multiIf(multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`+"`service.name`"+` IS NOT NULL, resource.`+"`service.name`"+`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `+"`service.name`"+`,
countIf(mapContains(attributes_string, 'gen_ai.request.model')) AS llm_call_count,
countIf(mapContains(attributes_string, 'gen_ai.tool.name')) AS tool_call_count,
uniqIf(multiIf(mapContains(attributes_string, 'gen_ai.tool.name'), attributes_string['gen_ai.tool.name'], NULL), mapContains(attributes_string, 'gen_ai.tool.name')) AS distinct_tool_count,

View File

@@ -154,7 +154,7 @@ func TestBuild_FullSQL_Scalar_GroupByIntrinsic(t *testing.T) {
assertSQLEqual(t, `
WITH __scoped_traces AS (
SELECT trace_id,
toString(name) AS __GROUP_BY_KEY_0_name,
toString(multiIf(name <> '', toString(name), NULL)) AS __GROUP_BY_KEY_0_name,
sum(multiIf(mapContains(attributes_number, 'gen_ai.usage.output_tokens'), toFloat64(attributes_number['gen_ai.usage.output_tokens']), NULL)) AS output_tokens
FROM signoz_traces.distributed_signoz_index_v3
WHERE timestamp >= '1747947419000000000'
@@ -718,7 +718,7 @@ SELECT sum(multiIf(mapContains(attributes_number, 'gen_ai.usage.output_tokens'),
FROM signoz_traces.distributed_signoz_index_v3
WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter)
AND trace_id GLOBAL IN (SELECT trace_id FROM __trace_scope)
AND (((mapContains(attributes_string, 'gen_ai.request.model') OR mapContains(attributes_string, 'gen_ai.tool.name') OR mapContains(attributes_string, 'gen_ai.agent.name'))) AND (multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = 'api'))
AND (((mapContains(attributes_string, 'gen_ai.request.model') OR mapContains(attributes_string, 'gen_ai.tool.name') OR mapContains(attributes_string, 'gen_ai.agent.name'))) AND ((multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = 'api' AND multiIf(resource.service.name IS NOT NULL, resource.service.name::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))
AND timestamp >= '1747947419000000000'
AND timestamp < '1747983448000000000'
AND ts_bucket_start >= 1747945619

View File

@@ -22,11 +22,11 @@ import (
type auditQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
storage qbtypes.Storage
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
resourceFilterStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation]
aggExprRewriter qbtypes.AggExprRewriter
fullTextColumn *telemetrytypes.TelemetryFieldKey
fl flagger.Flagger
}
var _ qbtypes.StatementBuilder[qbtypes.LogAggregation] = (*auditQueryStatementBuilder)(nil)
@@ -40,10 +40,11 @@ func NewFactory(
return factory.NewProviderFactory(
factory.MustNewName("audit"),
func(_ context.Context, settings factory.ProviderSettings, _ statementbuilder.Config) (qbtypes.StatementBuilder[qbtypes.LogAggregation], error) {
storage := audittelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, audittelemetryschema.DefaultFullTextColumn, storage, fl, telemetrytypes.SignalLogs)
fm := audittelemetryschema.NewFieldMapper()
cb := audittelemetryschema.NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, audittelemetryschema.DefaultFullTextColumn, fm, cb, fl)
return NewAuditQueryStatementBuilder(
settings, metadataStore, storage, aggExprRewriter, audittelemetryschema.DefaultFullTextColumn, fl,
settings, metadataStore, fm, cb, aggExprRewriter, audittelemetryschema.DefaultFullTextColumn, fl,
), nil
},
)
@@ -52,7 +53,8 @@ func NewFactory(
func NewAuditQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
storage qbtypes.Storage,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
aggExprRewriter qbtypes.AggExprRewriter,
fullTextColumn *telemetrytypes.TelemetryFieldKey,
flagger flagger.Flagger,
@@ -73,11 +75,11 @@ func NewAuditQueryStatementBuilder(
return &auditQueryStatementBuilder{
logger: auditSettings.Logger(),
metadataStore: metadataStore,
storage: storage,
fm: fieldMapper,
cb: conditionBuilder,
resourceFilterStmtBuilder: resourceFilterStmtBuilder,
aggExprRewriter: aggExprRewriter,
fullTextColumn: fullTextColumn,
fl: flagger,
}
}
@@ -226,7 +228,6 @@ func (b *auditQueryStatementBuilder) buildListQuery(
keys map[string][]*telemetrytypes.TelemetryFieldKey,
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
info := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalLogs, nil, start, end)
var (
cteFragments []string
cteArgs [][]any
@@ -262,11 +263,11 @@ func (b *auditQueryStatementBuilder) buildListQuery(
continue
}
colExpr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &query.SelectFields[index], telemetrytypes.FieldDataTypeUnspecified, keys)
colExpr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &query.SelectFields[index], telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
sb.SelectMore(fmt.Sprintf("%s AS `%s`", sqlbuilder.Escape(colExpr), query.SelectFields[index].Name))
sb.SelectMore(colExpr)
}
}
@@ -278,11 +279,11 @@ func (b *auditQueryStatementBuilder) buildListQuery(
}
for _, orderBy := range query.Order {
colExpr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
colExpr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
sb.OrderBy(fmt.Sprintf("%s %s", sqlbuilder.Escape(colExpr), orderBy.Direction.StringValue()))
sb.OrderBy(fmt.Sprintf("%s %s", colExpr, orderBy.Direction.StringValue()))
}
if query.Limit > 0 {
@@ -339,9 +340,8 @@ func (b *auditQueryStatementBuilder) buildTimeSeriesQuery(
var allGroupByArgs []any
fieldNames := make([]string, 0, len(query.GroupBy))
info := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalLogs, nil, start, end)
for _, gb := range query.GroupBy {
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -475,9 +475,8 @@ func (b *auditQueryStatementBuilder) buildScalarQuery(
var allGroupByArgs []any
info := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalLogs, nil, start, end)
for _, gb := range query.GroupBy {
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -567,13 +566,16 @@ func (b *auditQueryStatementBuilder) addFilterCondition(
if query.Filter != nil && query.Filter.Expression != "" {
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalLogs, nil, start, end),
Storage: b.storage,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
SkipResourceFilter: true,
FullTextColumn: b.fullTextColumn,
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {

View File

@@ -55,13 +55,15 @@ func newTestAuditStatementBuilder(t *testing.T) *auditQueryStatementBuilder {
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = auditFieldKeyMap()
storage := audittelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
fm := audittelemetryschema.NewFieldMapper()
cb := audittelemetryschema.NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
return NewAuditQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
audittelemetryschema.DefaultFullTextColumn,
fl,

View File

@@ -46,11 +46,12 @@ func TestStatementBuilderGroupByUnknownKey(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.useJSONBody)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
storage := logstelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore, storage, aggExprRewriter,
mockMetadataStore, fm, cb, aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)

View File

@@ -1125,7 +1125,7 @@ func TestJSONStmtBuilder_SelectField(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "SELECT timestamp, id, dynamicElement(body_v2.`user.name`, 'String') AS `__SELECT_KEY_0_user.name` FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? LIMIT ?",
Query: "SELECT timestamp, id, multiIf((dynamicElement(body_v2.`user.name`, 'String') IS NOT NULL), dynamicElement(body_v2.`user.name`, 'String'), NULL) AS `__SELECT_KEY_0_user.name` FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448), 10},
},
},
@@ -1202,7 +1202,7 @@ func TestJSONStmtBuilder_OrderBy(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, scope_name, scope_version, body_v2 as body, attributes_string, attributes_number, attributes_bool, resources_string, scope_string FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? ORDER BY dynamicElement(body_v2.`user.name`, 'String') asc LIMIT ?",
Query: "SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, scope_name, scope_version, body_v2 as body, attributes_string, attributes_number, attributes_bool, resources_string, scope_string FROM signoz_logs.distributed_logs_v2 WHERE timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? ORDER BY multiIf((dynamicElement(body_v2.`user.name`, 'String') IS NOT NULL), dynamicElement(body_v2.`user.name`, 'String'), NULL) asc LIMIT ?",
Args: []any{"1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448), 10},
},
},
@@ -1333,14 +1333,16 @@ func buildJSONTestStatementBuilder(t *testing.T, addIndexes bool) (*logQueryStat
mockMetadataStore := buildTestTelemetryMetadataStore(t, addIndexes)
fl := flaggertest.WithUseJSONBody(t, true)
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,

View File

@@ -26,13 +26,14 @@ func TestSearchCostGuard(t *testing.T) {
end := uint64(releaseTime.UnixMilli())
fl := flaggertest.WithBooleanFlags(t, map[string]bool{})
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
store := telemetrytypestest.NewMockMetadataStore()
store.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
sb := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
store, storage, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
store, fm, cb, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{SearchMaxScanRows: 100000, SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
@@ -56,13 +57,14 @@ func TestSearchCostGuardJSONBody(t *testing.T) {
end := uint64(releaseTime.UnixMilli())
fl := flaggertest.WithUseJSONBody(t, true)
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
store := telemetrytypestest.NewMockMetadataStore()
store.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
sb := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
store, storage, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
store, fm, cb, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{
SearchMaxScanRows: 100000,
SearchMaxScanRowsJSONBody: 10000,

View File

@@ -34,7 +34,8 @@ func bodyAliasExpression(bodyJSONEnabled bool) string {
type logQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
storage qbtypes.Storage
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
resourceFilterResolver *resourcefilter.ResourceFingerprintResolver[qbtypes.LogAggregation]
aggExprRewriter qbtypes.AggExprRewriter
fl flagger.Flagger
@@ -58,10 +59,11 @@ func NewFactory(
return factory.NewProviderFactory(
factory.MustNewName("logs"),
func(_ context.Context, settings factory.ProviderSettings, cfg statementbuilder.Config) (qbtypes.StatementBuilder[qbtypes.LogAggregation], error) {
storage := logstelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, logstelemetryschema.DefaultFullTextColumn, storage, fl, telemetrytypes.SignalLogs)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, logstelemetryschema.DefaultFullTextColumn, fm, cb, fl)
return NewLogQueryStatementBuilder(
settings, metadataStore, storage, aggExprRewriter, logstelemetryschema.DefaultFullTextColumn,
settings, metadataStore, fm, cb, aggExprRewriter, logstelemetryschema.DefaultFullTextColumn,
fl, telemetryStore, cfg,
), nil
},
@@ -71,7 +73,8 @@ func NewFactory(
func NewLogQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
storage qbtypes.Storage,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
aggExprRewriter qbtypes.AggExprRewriter,
fullTextColumn *telemetrytypes.TelemetryFieldKey,
fl flagger.Flagger,
@@ -96,7 +99,8 @@ func NewLogQueryStatementBuilder(
b := &logQueryStatementBuilder{
logger: logsSettings.Logger(),
metadataStore: metadataStore,
storage: storage,
fm: fieldMapper,
cb: conditionBuilder,
resourceFilterResolver: resourceFilterResolver,
aggExprRewriter: aggExprRewriter,
fl: fl,
@@ -318,7 +322,6 @@ func (b *logQueryStatementBuilder) buildListQuery(
keys map[string][]*telemetrytypes.TelemetryFieldKey,
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
info := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalLogs, nil, start, end)
var (
cteFragments []string
@@ -346,7 +349,7 @@ func (b *logQueryStatementBuilder) buildListQuery(
sb.SelectMore(logstelemetryschema.LogsV2SeverityNumberColumn)
sb.SelectMore(logstelemetryschema.LogsV2ScopeNameColumn)
sb.SelectMore(logstelemetryschema.LogsV2ScopeVersionColumn)
sb.SelectMore(bodyAliasExpression(info.BodyJSONOn))
sb.SelectMore(bodyAliasExpression(b.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))))
sb.SelectMore(logstelemetryschema.LogsV2AttributesStringColumn)
sb.SelectMore(logstelemetryschema.LogsV2AttributesNumberColumn)
sb.SelectMore(logstelemetryschema.LogsV2AttributesBoolColumn)
@@ -361,7 +364,7 @@ func (b *logQueryStatementBuilder) buildListQuery(
}
// get column expression for the field - use array index directly to avoid pointer to loop variable
colExpr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &query.SelectFields[index], telemetrytypes.FieldDataTypeUnspecified, keys)
colExpr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &query.SelectFields[index], telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
@@ -380,7 +383,7 @@ func (b *logQueryStatementBuilder) buildListQuery(
// Add order by
for _, orderBy := range query.Order {
colExpr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
colExpr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
@@ -444,13 +447,13 @@ func (b *logQueryStatementBuilder) buildTimeSeriesQuery(
))
// Keep original column expressions so we can build the tuple
info := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalLogs, nil, start, end)
bodyJSONEnabled := b.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
fieldNames := make([]string, 0, len(query.GroupBy))
for i, gb := range query.GroupBy {
if !info.BodyJSONOn && (strings.Contains(gb.Name, telemetrytypes.ArraySep) || strings.Contains(gb.Name, telemetrytypes.ArrayAnyIndex)) {
if !bodyJSONEnabled && (strings.Contains(gb.Name, telemetrytypes.ArraySep) || strings.Contains(gb.Name, telemetrytypes.ArrayAnyIndex)) {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "Group by/Aggregation isn't available for the Array Paths: %s", gb.Name)
}
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -609,13 +612,13 @@ func (b *logQueryStatementBuilder) buildScalarQuery(
allAggChArgs := []any{}
info := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalLogs, nil, start, end)
bodyJSONEnabled := b.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
fieldNames := make([]string, 0, len(query.GroupBy))
for i, gb := range query.GroupBy {
if !info.BodyJSONOn && (strings.Contains(gb.Name, telemetrytypes.ArraySep) || strings.Contains(gb.Name, telemetrytypes.ArrayAnyIndex)) {
if !bodyJSONEnabled && (strings.Contains(gb.Name, telemetrytypes.ArraySep) || strings.Contains(gb.Name, telemetrytypes.ArrayAnyIndex)) {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "Group by/Aggregation isn't available for the Array Paths: %s", gb.Name)
}
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -728,13 +731,16 @@ func (b *logQueryStatementBuilder) addFilterCondition(
// add filter expression
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalLogs, nil, start, end),
Storage: b.storage,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
SkipResourceFilter: skipResourceFilter,
FullTextColumn: b.fullTextColumn,
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {

View File

@@ -106,7 +106,7 @@ func TestStatementBuilderTimeSeries(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, countDistinct(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR (attributes_string['http.method'] = ? AND mapContains(attributes_string, 'http.method'))) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, countDistinct(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR (attributes_string['http.method'] = ? AND mapContains(attributes_string, 'http.method'))) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, countDistinct(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (attributes_string['http.method'] = ? AND mapContains(attributes_string, 'http.method'))) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, countDistinct(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS __result_0 FROM signoz_logs.distributed_logs_v2 WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (attributes_string['http.method'] = ? AND mapContains(attributes_string, 'http.method'))) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"redis-manual", "GET", "1705226400000000000", uint64(1705224600), "1705485600000000000", uint64(1705485600), 10, "redis-manual", "GET", "1705226400000000000", uint64(1705224600), "1705485600000000000", uint64(1705485600)},
},
expectedErr: nil,
@@ -218,14 +218,16 @@ func TestStatementBuilderTimeSeries(t *testing.T) {
mockMetadataStore.KeysMap = keysMap
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -354,18 +356,20 @@ func TestStatementBuilderListQuery(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -505,17 +509,19 @@ func TestStatementBuilderListQueryResourceTests(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -573,24 +579,26 @@ func TestStatementBuilderTimeSeriesBodyGroupBy(t *testing.T) {
},
},
},
expectedErrContains: "can be filtered but not selected or grouped",
expectedErrContains: "Operation isn't available for the body column",
},
}
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -605,8 +613,7 @@ func TestStatementBuilderTimeSeriesBodyGroupBy(t *testing.T) {
if c.expectedErrContains != "" {
require.Error(t, err)
_, _, message, _, _, _ := errors.Unwrapb(err)
require.Contains(t, message, c.expectedErrContains)
require.Contains(t, err.Error(), c.expectedErrContains)
} else {
require.NoError(t, err)
require.Equal(t, c.expected.Query, q.Query)
@@ -677,15 +684,17 @@ func TestStatementBuilderListQueryServiceCollision(t *testing.T) {
ctx := context.Background()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fl := flaggertest.New(t)
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMapCollision()
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -899,16 +908,18 @@ func TestAdjustKey(t *testing.T) {
}
fl := flaggertest.New(t)
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMapCollision()
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -1041,18 +1052,20 @@ func TestStmtBuilderBodyField(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.enableUseJSONBody)
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
// build the key map
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
for _, field := range logstelemetryschema.IntrinsicFields {
f := field
mockMetadataStore.KeysMap[field.Name] = append(mockMetadataStore.KeysMap[field.Name], &f)
}
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -1139,18 +1152,20 @@ func TestStmtBuilderBodyFullTextSearch(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.enableUseJSONBody)
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
// build the key map
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
for _, field := range logstelemetryschema.IntrinsicFields {
f := field
mockMetadataStore.KeysMap[field.Name] = append(mockMetadataStore.KeysMap[field.Name], &f)
}
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalLogs)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,
@@ -1254,16 +1269,24 @@ func newSkipResourceFingerprintLogsBuilder(
t.Helper()
fl := flaggertest.New(t)
storage := logstelemetryschema.NewStorage()
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC))
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), logstelemetryschema.DefaultFullTextColumn, storage, fl, telemetrytypes.SignalLogs)
aggExprRewriter := querybuilder.NewAggExprRewriter(
instrumentationtest.New().ToProviderSettings(),
logstelemetryschema.DefaultFullTextColumn,
fm,
cb,
fl,
)
return NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
fl,

View File

@@ -23,7 +23,8 @@ import (
type meterQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
storage qbtypes.Storage
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
metricsStatementBuilder *metricsstatementbuilder.StatementBuilder
}
@@ -43,7 +44,9 @@ func NewFactory(
if err != nil {
return nil, err
}
return NewMeterQueryStatementBuilder(settings, metadataStore, metricstelemetryschema.NewStorage(), metricsStatementBuilder), nil
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
return NewMeterQueryStatementBuilder(settings, metadataStore, fm, cb, metricsStatementBuilder), nil
},
)
}
@@ -51,7 +54,8 @@ func NewFactory(
func NewMeterQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
storage qbtypes.Storage,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
metricsStatementBuilder *metricsstatementbuilder.StatementBuilder,
) *meterQueryStatementBuilder {
metricsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema")
@@ -59,7 +63,8 @@ func NewMeterQueryStatementBuilder(
return &meterQueryStatementBuilder{
logger: metricsSettings.Logger(),
metadataStore: metadataStore,
storage: storage,
fm: fieldMapper,
cb: conditionBuilder,
metricsStatementBuilder: metricsStatementBuilder,
}
}
@@ -145,9 +150,8 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDeltaFastPath(
"toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(%d)) AS ts",
stepSec,
))
info := querybuilder.NewQueryInfo(ctx, orgID, nil, telemetrytypes.SignalMetrics, &telemetrytypes.MetricContext{MetricName: query.Aggregations[0].MetricName}, start, end)
for i, g := range query.GroupBy {
col, err := querybuilder.ResolveColumn(ctx, info, b.storage, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return "", nil, err
}
@@ -172,13 +176,16 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDeltaFastPath(
)
if query.Filter != nil && query.Filter.Expression != "" {
filterWhere, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: info,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
Context: ctx,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
return "", nil, err
@@ -232,9 +239,8 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDelta(
stepSec,
))
info := querybuilder.NewQueryInfo(ctx, orgID, nil, telemetrytypes.SignalMetrics, &telemetrytypes.MetricContext{MetricName: query.Aggregations[0].MetricName}, start, end)
for i, g := range query.GroupBy {
col, err := querybuilder.ResolveColumn(ctx, info, b.storage, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return "", nil, err
}
@@ -262,13 +268,16 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDelta(
if query.Filter != nil && query.Filter.Expression != "" {
filterWhere, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: info,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
Context: ctx,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
return "", nil, err
@@ -308,9 +317,8 @@ func (b *meterQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
"toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(%d)) AS ts",
stepSec,
))
info := querybuilder.NewQueryInfo(ctx, orgID, nil, telemetrytypes.SignalMetrics, &telemetrytypes.MetricContext{MetricName: query.Aggregations[0].MetricName}, start, end)
for i, g := range query.GroupBy {
col, err := querybuilder.ResolveColumn(ctx, info, b.storage, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return "", nil, err
}
@@ -332,13 +340,16 @@ func (b *meterQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
)
if query.Filter != nil && query.Filter.Expression != "" {
filterWhere, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: info,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
Context: ctx,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
return "", nil, err

View File

@@ -162,7 +162,8 @@ func TestStatementBuilder(t *testing.T) {
},
}
storage := metricstelemetryschema.NewStorage()
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
if err != nil {
@@ -172,12 +173,13 @@ func TestStatementBuilder(t *testing.T) {
flagger := flaggertest.New(t)
metricStmtBuilder := metricsstatementbuilder.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, storage, flagger)
metricStmtBuilder := metricsstatementbuilder.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, fm, cb, flagger)
statementBuilder := NewMeterQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
metricStmtBuilder,
)
@@ -200,7 +202,8 @@ func TestStatementBuilder(t *testing.T) {
}
func TestGroupByAliasAvoidsColumnCollision(t *testing.T) {
storage := metricstelemetryschema.NewStorage()
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
require.NoError(t, err)
@@ -211,8 +214,9 @@ func TestGroupByAliasAvoidsColumnCollision(t *testing.T) {
statementBuilder := NewMeterQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
metricsstatementbuilder.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, storage, flagger),
fm,
cb,
metricsstatementbuilder.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, fm, cb, flagger),
)
for _, groupBy := range []string{"ts", "value", "fingerprint", "service.name"} {

View File

@@ -144,10 +144,11 @@ func TestReducedStatementBuilder(t *testing.T) {
},
}
storage := metricstelemetryschema.NewStorage()
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
fl, err := flagger.New(context.Background(), instrumentationtest.New().ToProviderSettings(), flagger.Config{}, flagger.MustNewRegistry())
require.NoError(t, err)
sb := NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), telemetrytypestest.NewMockMetadataStore(), storage, fl)
sb := NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), telemetrytypestest.NewMockMetadataStore(), fm, cb, fl)
const start, end = uint64(1747000000000), uint64(1747172800000)

View File

@@ -34,7 +34,8 @@ const (
type StatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
storage qbtypes.Storage
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
flagger flagger.Flagger
}
@@ -50,7 +51,9 @@ func NewFactory(
return factory.NewProviderFactory(
factory.MustNewName("metrics"),
func(_ context.Context, settings factory.ProviderSettings, _ statementbuilder.Config) (*StatementBuilder, error) {
return NewMetricQueryStatementBuilder(settings, metadataStore, metricstelemetryschema.NewStorage(), fl), nil
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
return NewMetricQueryStatementBuilder(settings, metadataStore, fm, cb, fl), nil
},
)
}
@@ -58,14 +61,16 @@ func NewFactory(
func NewMetricQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
storage qbtypes.Storage,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
flagger flagger.Flagger,
) *StatementBuilder {
metricsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema")
return &StatementBuilder{
logger: metricsSettings.Logger(),
metadataStore: metadataStore,
storage: storage,
fm: fieldMapper,
cb: conditionBuilder,
flagger: flagger,
}
}
@@ -249,19 +254,21 @@ func (b *StatementBuilder) buildReducedTimeSeriesCTE(
variables map[string]qbtypes.VariableItem,
) (string, []any, error) {
sb := sqlbuilder.NewSelectBuilder()
info := querybuilder.NewQueryInfo(ctx, orgID, b.flagger, telemetrytypes.SignalMetrics, &telemetrytypes.MetricContext{MetricName: query.Aggregations[0].MetricName}, start, end)
var preparedWhereClause querybuilder.PreparedWhereClause
var err error
if query.Filter != nil && query.Filter.Expression != "" {
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: info,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
Context: ctx,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
return "", nil, err
@@ -271,7 +278,7 @@ func (b *StatementBuilder) buildReducedTimeSeriesCTE(
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedLocalTableName))
sb.Select("fingerprint")
for i, g := range query.GroupBy {
col, err := querybuilder.ResolveColumn(ctx, info, b.storage, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return "", nil, err
}
@@ -467,20 +474,22 @@ func (b *StatementBuilder) buildTimeSeriesCTE(
tsTable string,
) (string, []any, []string, error) {
sb := sqlbuilder.NewSelectBuilder()
info := querybuilder.NewQueryInfo(ctx, orgID, b.flagger, telemetrytypes.SignalMetrics, &telemetrytypes.MetricContext{MetricName: query.Aggregations[0].MetricName}, start, end)
var preparedWhereClause querybuilder.PreparedWhereClause
var err error
if query.Filter != nil && query.Filter.Expression != "" {
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: info,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
Context: ctx,
OrgID: orgID,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "labels"},
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
return "", nil, nil, err
@@ -491,7 +500,7 @@ func (b *StatementBuilder) buildTimeSeriesCTE(
sb.Select("fingerprint")
for i, g := range query.GroupBy {
col, err := querybuilder.ResolveColumn(ctx, info, b.storage, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
col, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &g.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return "", nil, nil, err
}

View File

@@ -376,7 +376,7 @@ func TestStatementBuilder(t *testing.T) {
expected: qbtypes.Statement{
Query: "WITH __temporal_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_k8s.statefulset.name`, multiIf(row_number() OVER rate_window = 1, nan, (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) < 0, per_series_value / (ts - lagInFrame(ts, 1) OVER rate_window), (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) / (ts - lagInFrame(ts, 1) OVER rate_window)) AS per_series_value FROM (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_k8s.statefulset.name`, max(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'k8s.statefulset.name') AS `__GROUP_BY_KEY_0_k8s.statefulset.name` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) AND JSONExtractString(labels, 'k8s.statefulset.name') = ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_k8s.statefulset.name`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_k8s.statefulset.name` ORDER BY fingerprint, ts) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_k8s.statefulset.name`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_k8s.statefulset.name`) SELECT * FROM __spatial_aggregation_cte ORDER BY `__GROUP_BY_KEY_0_k8s.statefulset.name`, ts",
Args: []any{"signoz_calls_total", uint64(1747936800000), uint64(1747983420000), "cumulative", "my-statefulset", "signoz_calls_total", uint64(1747947360000), uint64(1747983420000), 0},
Warnings: []string{"key `k8s.statefulset.name` not found in metadata; querying the underlying data directly. If this is unexpected, check the key name for typos."},
Warnings: []string{"label `k8s.statefulset.name` not found in metadata; check the label name for typos"},
},
expectedErr: nil,
},
@@ -414,7 +414,8 @@ func TestStatementBuilder(t *testing.T) {
},
}
storage := metricstelemetryschema.NewStorage()
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
if err != nil {
@@ -437,7 +438,8 @@ func TestStatementBuilder(t *testing.T) {
statementBuilder := NewMetricQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
flagger,
)
@@ -460,7 +462,8 @@ func TestStatementBuilder(t *testing.T) {
}
func TestGroupByAliasAvoidsColumnCollision(t *testing.T) {
storage := metricstelemetryschema.NewStorage()
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
require.NoError(t, err)
@@ -472,7 +475,8 @@ func TestGroupByAliasAvoidsColumnCollision(t *testing.T) {
statementBuilder := NewMetricQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
fl,
)

View File

@@ -5,12 +5,27 @@ import (
"fmt"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
type defaultConditionBuilder struct {
fm qbtypes.FieldMapper
// fl evaluates the resolve_semconv_families flag during resolution.
// A nil flagger keeps resolution literal.
fl flagger.Flagger
}
var _ qbtypes.ConditionBuilder = (*defaultConditionBuilder)(nil)
func NewConditionBuilder(fm qbtypes.FieldMapper, fl flagger.Flagger) *defaultConditionBuilder {
return &defaultConditionBuilder{fm: fm, fl: fl}
}
func valueForIndexFilter(op qbtypes.FilterOperator, key *telemetrytypes.TelemetryFieldKey, value any) any {
switch v := value.(type) {
case []any:
@@ -97,20 +112,55 @@ func memberPresenceCondition(sb *sqlbuilder.SelectBuilder, column string, member
return sb.And(conditions...)
}
// Compile weaves the index hints of the fingerprint table into each
// operator. It reads the members directly, so a member with a value map must
// translate its operand through StoredValues before the hint is built.
func (b *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, op qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
condition, err := b.conditionForLogicalField(ctx, q, logical, op, value, sb)
if err != nil {
return qbtypes.Compiled{}, err
// SkipResourceFilter is not applicable here: the fingerprint table only stores resource attributes.
func (b *defaultConditionBuilder) ConditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
op qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
matches := querybuilder.MatchingLogicalFields(ctx, orgID, b.fl, key, fieldKeys)
// has/hasAny/hasAll/hasToken are logs-body-only functions; they never apply to the
// resource fingerprint table, so skip them (the main query still evaluates them).
if op.IsFunctionOperator() {
return nil, nil, nil
}
return qbtypes.Compiled{Condition: condition}, nil
logicalFields, warning := querybuilder.ResolveLogicalFields(key, matches)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
conds := make([]string, 0, len(logicalFields))
for _, logical := range logicalFields {
// the resource fingerprint table only stores resource attributes; fields from
// any other context contribute no condition and are omitted. An empty result
// (including an unknown key) lets the caller skip this filter entirely.
if logical.FieldContext != telemetrytypes.FieldContextResource {
continue
}
cond, err := b.conditionForLogicalField(ctx, orgID, startNs, endNs, logical, op, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
}
return conds, warnings, nil
}
func (b *storage) conditionForLogicalField(
func (b *defaultConditionBuilder) conditionForLogicalField(
ctx context.Context,
q qbtypes.QueryInfo,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
logical *telemetrytypes.LogicalField,
op qbtypes.FilterOperator,
value any,
@@ -123,7 +173,7 @@ func (b *storage) conditionForLogicalField(
// Every resource-context key maps to the labels column, so any member
// resolves the column for the whole field.
columns, err := b.getColumn(ctx, q.StartNs, q.EndNs, logical.Single())
columns, err := b.fm.ColumnFor(ctx, orgID, startNs, endNs, logical.Single())
if err != nil {
return "", err
}
@@ -141,11 +191,10 @@ func (b *storage) conditionForLogicalField(
keyIdxFilter := keyIndexCondition(sb, column.Name, members)
singleValueIndexFilter := valueForIndexFilter(op, members[0], value)
logicalRead, err := querybuilder.LogicalRead(ctx, q, b, logical)
fieldName, err := querybuilder.LogicalValueExpr(ctx, orgID, startNs, endNs, b.fm, logical)
if err != nil {
return "", err
}
fieldName := logicalRead.SQL
switch op {
case qbtypes.FilterOperatorEqual:

View File

@@ -2,11 +2,12 @@ package resourcefilter
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"testing"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -200,12 +201,13 @@ func TestConditionBuilder(t *testing.T) {
},
}
storage := newStorage()
fm := NewFieldMapper()
conditionBuilder := NewConditionBuilder(fm, flaggertest.New(t))
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, _, err := querybuilder.Conditions(context.Background(), qbtypes.QueryInfo{}, storage, tc.key, tc.op, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {tc.key}}, false, sb)
cond, _, err := conditionBuilder.ConditionFor(context.Background(), valuer.UUID{}, 0, 0, tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.op, tc.value, sb)
sb.Where(cond...)
if tc.expectedErr != nil {

View File

@@ -2,7 +2,6 @@ package resourcefilter
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
"github.com/SigNoz/signoz/pkg/flagger"
@@ -19,8 +18,7 @@ import (
// the metadata map as trace resource attributes.
func TestFamilyEqualWidensIndexHintsToAnyMember(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true})
storage := newStorage()
cb := NewConditionBuilder(NewFieldMapper(), flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true}))
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {{
Name: "deployment.environment.name",
@@ -37,7 +35,9 @@ func TestFamilyEqualWidensIndexHintsToAnyMember(t *testing.T) {
}
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorEqual, "production", fieldKeys, false, sb)
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, 0, 0,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "production", sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -47,8 +47,7 @@ func TestFamilyEqualWidensIndexHintsToAnyMember(t *testing.T) {
}
func TestFamilyNotEqualDropsNegatedValueHint(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true})
storage := newStorage()
cb := NewConditionBuilder(NewFieldMapper(), flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true}))
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {{
Name: "deployment.environment.name",
@@ -65,7 +64,9 @@ func TestFamilyNotEqualDropsNegatedValueHint(t *testing.T) {
}
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorNotEqual, "production", fieldKeys, false, sb)
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, 0, 0,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorNotEqual, "production", sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -75,8 +76,7 @@ func TestFamilyNotEqualDropsNegatedValueHint(t *testing.T) {
}
func TestFamilyExistsIsAnyMemberPresence(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true})
storage := newStorage()
cb := NewConditionBuilder(NewFieldMapper(), flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true}))
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {{
Name: "deployment.environment.name",
@@ -93,7 +93,9 @@ func TestFamilyExistsIsAnyMemberPresence(t *testing.T) {
}
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorExists, nil, fieldKeys, false, sb)
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, 0, 0,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorExists, nil, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -102,7 +104,9 @@ func TestFamilyExistsIsAnyMemberPresence(t *testing.T) {
require.Equal(t, []any{true, true, "%deployment.environment.name%", "%deployment.environment%"}, args)
sb = sqlbuilder.NewSelectBuilder()
conds, _, err = querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorNotExists, nil, fieldKeys, false, sb)
conds, _, err = cb.ConditionFor(context.Background(), valuer.UUID{}, 0, 0,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorNotExists, nil, sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -114,8 +118,7 @@ func TestFamilyExistsIsAnyMemberPresence(t *testing.T) {
// With only one member in metadata the SQL keeps the exact pre-family shape,
// including the negated value hint on !=.
func TestSingleMemberShapesUnchanged(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true})
storage := newStorage()
cb := NewConditionBuilder(NewFieldMapper(), flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureResolveSemconvFamilies.String(): true}))
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{
"deployment.environment.name": {{
Name: "deployment.environment.name",
@@ -126,7 +129,9 @@ func TestSingleMemberShapesUnchanged(t *testing.T) {
}
sb := sqlbuilder.NewSelectBuilder()
conds, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalTraces, nil, 0, 0), storage, &telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"}, qbtypes.FilterOperatorNotEqual, "production", fieldKeys, false, sb)
conds, _, err := cb.ConditionFor(context.Background(), valuer.UUID{}, 0, 0,
&telemetrytypes.TelemetryFieldKey{Name: "deployment.environment.name"},
fieldKeys, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorNotEqual, "production", sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)

View File

@@ -7,6 +7,7 @@ import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
var (
@@ -17,15 +18,21 @@ var (
}
)
type storage struct{}
type defaultFieldMapper struct{}
var _ qbtypes.Storage = (*storage)(nil)
var _ qbtypes.FieldMapper = (*defaultFieldMapper)(nil)
func newStorage() *storage {
return &storage{}
// CandidateKeys returns nil: the resource filter has no attribute-map fallback, so a
// context-missing key stays unresolved and the caller errors.
func (m *defaultFieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, _ *telemetrytypes.TelemetryFieldKey, _ any, _ map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
}
func (m *storage) getColumn(
func NewFieldMapper() *defaultFieldMapper {
return &defaultFieldMapper{}
}
func (m *defaultFieldMapper) getColumn(
_ context.Context,
_, _ uint64,
key *telemetrytypes.TelemetryFieldKey,
@@ -39,8 +46,22 @@ func (m *storage) getColumn(
return nil, qbtypes.ErrColumnNotFound
}
func (m *storage) read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, q.StartNs, q.EndNs, key)
func (m *defaultFieldMapper) ColumnFor(
ctx context.Context,
_ valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
) ([]*schema.Column, error) {
return m.getColumn(ctx, tsStart, tsEnd, key)
}
func (m *defaultFieldMapper) FieldFor(
ctx context.Context,
_ valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
) (string, error) {
columns, err := m.getColumn(ctx, tsStart, tsEnd, key)
if err != nil {
return "", err
}
@@ -50,36 +71,44 @@ func (m *storage) read(ctx context.Context, q qbtypes.QueryInfo, key *telemetryt
return columns[0].Name, nil
}
// Read composes the bare read of one key with its membership test. Only a
// resource key has a presence notion in the labels JSON; anything else is a
// real column and always present.
func (m *storage) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (qbtypes.Read, error) {
columns, err := m.getColumn(ctx, q.StartNs, q.EndNs, key)
// ExistsFor reports key presence in the fingerprint labels JSON. Only resource
// context keys have a presence notion here; anything else is a real column and
// always present.
func (m *defaultFieldMapper) ExistsFor(
ctx context.Context,
_ valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
exists bool,
) (string, error) {
columns, err := m.getColumn(ctx, tsStart, tsEnd, key)
if err != nil {
return qbtypes.Read{}, err
}
sql, err := m.read(ctx, q, key)
if err != nil {
return qbtypes.Read{}, err
return "", err
}
if key.FieldContext != telemetrytypes.FieldContextResource {
return qbtypes.Read{SQL: sql, Presence: "true", Absence: "false", WhenAbsent: qbtypes.AlwaysPresent}, nil
if exists {
return "true", nil
}
return "false", nil
}
presence := fmt.Sprintf("simpleJSONHas(%s, '%s')", columns[0].Name, key.Name)
return qbtypes.Read{
SQL: sql,
Presence: presence,
Absence: "NOT " + presence,
WhenAbsent: qbtypes.AbsentIsSentinel,
}, nil
pred := fmt.Sprintf("simpleJSONHas(%s, '%s')", columns[0].Name, key.Name)
if exists {
return pred, nil
}
return "NOT " + pred, nil
}
// Fallback returns nil: the fingerprint table holds only what the metadata
// reports, and a term it cannot serve is the main query's to evaluate.
func (m *storage) Fallback(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, qbtypes.FilterOperator, any) ([]*telemetrytypes.LogicalField, error) {
return nil, nil
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{Split: qbtypes.FingerprintOfSplit, UnknownKey: qbtypes.IgnoreUnknownKey}
func (m *defaultFieldMapper) ColumnExpressionFor(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
_ telemetrytypes.FieldDataType,
_ map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
fieldExpression, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
return "", err
}
return fmt.Sprintf("%s AS `%s`", fieldExpression, key.Name), nil
}

View File

@@ -16,14 +16,15 @@ import (
// resourceFilterStatementBuilder builds resource fingerprint filter CTEs.
type resourceFilterStatementBuilder[T any] struct {
logger *slog.Logger
dbName string
tableName string
storage qbtypes.Storage
metadataStore telemetrytypes.MetadataStore
signal telemetrytypes.Signal
source telemetrytypes.Source
flagger flagger.Flagger
logger *slog.Logger
dbName string
tableName string
fieldMapper qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
metadataStore telemetrytypes.MetadataStore
signal telemetrytypes.Signal
source telemetrytypes.Source
flagger flagger.Flagger
fullTextColumn *telemetrytypes.TelemetryFieldKey
}
@@ -45,16 +46,19 @@ func New[T any](
fl flagger.Flagger,
) *resourceFilterStatementBuilder[T] {
set := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/statementbuilder/resourcefilter")
fm := NewFieldMapper()
cb := NewConditionBuilder(fm, fl)
return &resourceFilterStatementBuilder[T]{
logger: set.Logger(),
dbName: dbName,
tableName: tableName,
storage: newStorage(),
metadataStore: metadataStore,
signal: signal,
source: source,
flagger: fl,
fullTextColumn: fullTextColumn,
logger: set.Logger(),
dbName: dbName,
tableName: tableName,
fieldMapper: fm,
conditionBuilder: cb,
metadataStore: metadataStore,
signal: signal,
source: source,
flagger: fl,
fullTextColumn: fullTextColumn,
}
}
@@ -159,15 +163,19 @@ func (b *resourceFilterStatementBuilder[T]) addConditions(
// warnings would be encountered as part of the main condition already
filterWhereClause, err := querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, orgID, b.flagger, b.signal, nil, start, end),
Storage: b.storage,
OrgID: orgID,
Flagger: b.flagger,
Logger: b.logger,
FieldMapper: b.fieldMapper,
ConditionBuilder: b.conditionBuilder,
FieldKeys: keys,
FullTextColumn: b.fullTextColumn,
SkipFullTextFilter: true,
// the resource-filter storage ignores keys it can't resolve (and
// the resource-filter condition builder ignores keys it can't resolve (and
// skips function calls), so no "key not found" error arises here.
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {

View File

@@ -8,6 +8,7 @@ import (
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
// Aggregate renders one column's SQL through the resolvers and lists the attribute
@@ -15,7 +16,7 @@ import (
// constructors below; the zero value is not usable.
type Aggregate struct {
keys []*telemetrytypes.TelemetryFieldKey
render func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (expr string, err error)
render func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (expr string, err error)
}
// IntrinsicSpanKey references an intrinsic span-index field (timestamp, name, …).
@@ -46,15 +47,15 @@ const (
// CountAll renders count().
func CountAll() Aggregate {
return Aggregate{render: func(context.Context, qbtypes.QueryInfo, *columnResolver, *predicateResolver) (string, error) {
return Aggregate{render: func(context.Context, valuer.UUID, uint64, uint64, *columnResolver, *predicateResolver) (string, error) {
return "count()", nil
}}
}
// FieldReduce renders <fn>(<field>) over a field-mapper-resolved column.
func FieldReduce(fn AggFunc, key *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, _ *predicateResolver) (string, error) {
f, err := cols.Read(ctx, q, key)
return Aggregate{render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, _ *predicateResolver) (string, error) {
f, err := cols.FieldFor(ctx, orgID, startNs, endNs, key)
if err != nil {
return "", err
}
@@ -65,12 +66,12 @@ func FieldReduce(fn AggFunc, key *telemetrytypes.TelemetryFieldKey) Aggregate {
// TraceDuration renders the full-trace wall duration: last span end minus first
// span start.
func TraceDuration(tsKey, durationKey *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, _ *predicateResolver) (string, error) {
ts, err := cols.Read(ctx, q, tsKey)
return Aggregate{render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, _ *predicateResolver) (string, error) {
ts, err := cols.FieldFor(ctx, orgID, startNs, endNs, tsKey)
if err != nil {
return "", err
}
dur, err := cols.Read(ctx, q, durationKey)
dur, err := cols.FieldFor(ctx, orgID, startNs, endNs, durationKey)
if err != nil {
return "", err
}
@@ -82,52 +83,52 @@ func TraceDuration(tsKey, durationKey *telemetrytypes.TelemetryFieldKey) Aggrega
// FieldAnyWhere renders anyIf(<field>, <cond>) — the field value from any span
// matching the condition.
func FieldAnyWhere(valueKey, condKey *telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, condValue any) Aggregate {
return Aggregate{render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.Read(ctx, q, valueKey)
return Aggregate{render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.FieldFor(ctx, orgID, startNs, endNs, valueKey)
if err != nil {
return "", err
}
cond, err := preds.ConditionFor(ctx, q, condKey, op, condValue)
cond, err := preds.ConditionFor(ctx, orgID, startNs, endNs, condKey, op, condValue)
return fmt.Sprintf("anyIf(%s, %s)", v, cond), err
}}
}
// AnyValue renders any(<value>) over a metadata-resolved attribute value.
func AnyValue(key *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.FieldDataType) Aggregate {
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{key}, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, _ *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, q, key, dt)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{key}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, _ *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, orgID, startNs, endNs, key, dt)
return fmt.Sprintf("any(%s)", v), err
}}
}
// CountExists renders countIf(<key> EXISTS) — counts spans carrying key.
func CountExists(key *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{key}, render: func(ctx context.Context, q qbtypes.QueryInfo, _ *columnResolver, preds *predicateResolver) (string, error) {
cond, err := preds.ExistsFor(ctx, q, key)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{key}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, _ *columnResolver, preds *predicateResolver) (string, error) {
cond, err := preds.ExistsFor(ctx, orgID, startNs, endNs, key)
return fmt.Sprintf("countIf(%s)", cond), err
}}
}
// CondCount renders countIf(<cond>) over a condition-builder-resolved predicate.
func CondCount(key *telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, value any) Aggregate {
return Aggregate{render: func(ctx context.Context, q qbtypes.QueryInfo, _ *columnResolver, preds *predicateResolver) (string, error) {
cond, err := preds.ConditionFor(ctx, q, key, op, value)
return Aggregate{render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, _ *columnResolver, preds *predicateResolver) (string, error) {
cond, err := preds.ConditionFor(ctx, orgID, startNs, endNs, key, op, value)
return fmt.Sprintf("countIf(%s)", cond), err
}}
}
// Reduce renders <fn>(<value>) over a resolved numeric attribute value.
func Reduce(fn AggFunc, valueKey *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, _ *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, q, valueKey, telemetrytypes.FieldDataTypeFloat64)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, _ *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, orgID, startNs, endNs, valueKey, telemetrytypes.FieldDataTypeFloat64)
return fmt.Sprintf("%s(%s)", fn, v), err
}}
}
// ScopedReduce renders <fn>If(<field>, <gate mask>) over a field-mapper-resolved column.
func ScopedReduce(fn AggFunc, key *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (string, error) {
f, err := cols.Read(ctx, q, key)
return Aggregate{render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (string, error) {
f, err := cols.FieldFor(ctx, orgID, startNs, endNs, key)
if err != nil {
return "", err
}
@@ -138,12 +139,12 @@ func ScopedReduce(fn AggFunc, key *telemetrytypes.TelemetryFieldKey) Aggregate {
// ScopedToKeyColumn renders <fn>If(<field>, <scopeKey> EXISTS) — a span-index field
// aggregated over spans carrying scopeKey (e.g. max LLM latency).
func ScopedToKeyColumn(fn AggFunc, columnKey, scopeKey *telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{scopeKey}, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (string, error) {
col, err := cols.Read(ctx, q, columnKey)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{scopeKey}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (string, error) {
col, err := cols.FieldFor(ctx, orgID, startNs, endNs, columnKey)
if err != nil {
return "", err
}
cond, err := preds.ExistsFor(ctx, q, scopeKey)
cond, err := preds.ExistsFor(ctx, orgID, startNs, endNs, scopeKey)
return fmt.Sprintf("%sIf(%s, %s)", fn, col, cond), err
}}
}
@@ -155,28 +156,28 @@ func PickBy(valueKey *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.FieldD
if dir == PickEarliest {
fn = "argMinIf"
}
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, q, valueKey, dt)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, orgID, startNs, endNs, valueKey, dt)
if err != nil {
return "", err
}
order, err := cols.Read(ctx, q, orderKey)
order, err := cols.FieldFor(ctx, orgID, startNs, endNs, orderKey)
if err != nil {
return "", err
}
cond, err := preds.ExistsFor(ctx, q, valueKey)
cond, err := preds.ExistsFor(ctx, orgID, startNs, endNs, valueKey)
return fmt.Sprintf("%s(%s, %s, %s)", fn, v, order, cond), err
}}
}
// UniqCount renders uniqIf(<value>, <value> EXISTS) — distinct count of an attribute.
func UniqCount(valueKey *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.FieldDataType) Aggregate {
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, q, valueKey, dt)
return Aggregate{keys: []*telemetrytypes.TelemetryFieldKey{valueKey}, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, preds *predicateResolver) (string, error) {
v, err := cols.ValueFor(ctx, orgID, startNs, endNs, valueKey, dt)
if err != nil {
return "", err
}
cond, err := preds.ExistsFor(ctx, q, valueKey)
cond, err := preds.ExistsFor(ctx, orgID, startNs, endNs, valueKey)
return fmt.Sprintf("uniqIf(%s, %s)", v, cond), err
}}
}
@@ -184,10 +185,10 @@ func UniqCount(valueKey *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.Fie
// SumOfKeys renders coalesce(sum(<v1>), 0) + coalesce(sum(<v2>), 0) + …; coalesced
// because a key absent from every span sums to NULL and NULL + n = NULL.
func SumOfKeys(dt telemetrytypes.FieldDataType, valueKeys ...*telemetrytypes.TelemetryFieldKey) Aggregate {
return Aggregate{keys: valueKeys, render: func(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, _ *predicateResolver) (string, error) {
return Aggregate{keys: valueKeys, render: func(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, cols *columnResolver, _ *predicateResolver) (string, error) {
parts := make([]string, 0, len(valueKeys))
for _, k := range valueKeys {
v, err := cols.ValueFor(ctx, q, k, dt)
v, err := cols.ValueFor(ctx, orgID, startNs, endNs, k, dt)
if err != nil {
return "", err
}

View File

@@ -3,40 +3,35 @@ package scopedtracesstatementbuilder
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
// columnResolver resolves keys to bare column/value expressions through the
// shared storage. It binds no args, so its expressions embed in any builder;
// predicates (which do bind args) are the predicateResolver's job.
// columnResolver resolves keys to bare column/value expressions through the shared
// field mapper. It binds no args, so its expressions embed in any builder; predicates
// (which do bind args) are the predicateResolver's job.
type columnResolver struct {
storage qbtypes.Storage
keys map[string][]*telemetrytypes.TelemetryFieldKey
fm qbtypes.FieldMapper
keys map[string][]*telemetrytypes.TelemetryFieldKey
}
func newColumnResolver(storage qbtypes.Storage, keys map[string][]*telemetrytypes.TelemetryFieldKey) *columnResolver {
return &columnResolver{storage: storage, keys: keys}
func newColumnResolver(fm qbtypes.FieldMapper, keys map[string][]*telemetrytypes.TelemetryFieldKey) *columnResolver {
return &columnResolver{fm: fm, keys: keys}
}
// Read returns the bare read of one field key.
func (r *columnResolver) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
read, err := r.storage.Read(ctx, q, key)
if err != nil {
return "", err
}
return read.SQL, nil
func (r *columnResolver) FieldFor(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return r.fm.FieldFor(ctx, orgID, startNs, endNs, key)
}
// ValueFor returns the value expression for an attribute key.
func (r *columnResolver) ValueFor(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.FieldDataType) (string, error) {
func (r *columnResolver) ValueFor(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey, dt telemetrytypes.FieldDataType) (string, error) {
// TODO(nitya): Fix this as this is not correct way
if cands := r.keys[key.Name]; len(cands) > 0 {
key = cands[0]
}
expr, err := querybuilder.ResolveColumn(ctx, q, r.storage, key, dt, r.keys)
expr, err := r.fm.ColumnExpressionFor(ctx, orgID, startNs, endNs, key, dt, r.keys)
if err != nil {
return "", err
}

View File

@@ -3,31 +3,31 @@ package scopedtracesstatementbuilder
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
// predicateResolver resolves key + operator + value to boolean predicates through the
// shared storage. Args bind into sb as $n markers, so returned predicates can be
// embedded anywhere in sb; maskExpr is set by the builder after resolveMask
// shared condition builder. Args bind into sb as $n markers, so returned predicates
// can be embedded anywhere in sb; maskExpr is set by the builder after resolveMask
// (Scoped* aggregates embed it).
type predicateResolver struct {
storage qbtypes.Storage
cb qbtypes.ConditionBuilder
keys map[string][]*telemetrytypes.TelemetryFieldKey
sb *sqlbuilder.SelectBuilder
maskExpr string
}
func newPredicateResolver(storage qbtypes.Storage, keys map[string][]*telemetrytypes.TelemetryFieldKey, sb *sqlbuilder.SelectBuilder) *predicateResolver {
return &predicateResolver{storage: storage, keys: keys, sb: sb}
func newPredicateResolver(cb qbtypes.ConditionBuilder, keys map[string][]*telemetrytypes.TelemetryFieldKey, sb *sqlbuilder.SelectBuilder) *predicateResolver {
return &predicateResolver{cb: cb, keys: keys, sb: sb}
}
// ConditionFor returns a boolean predicate for key (materialized column when
// present, else map access), args bound into sb.
func (r *predicateResolver) ConditionFor(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, value any) (string, error) {
conds, _, err := querybuilder.Conditions(ctx, q, r.storage, key, op, value, r.keys, false, r.sb)
// ConditionFor returns a boolean predicate for key via the condition builder
// (materialized column when present, else map access), args bound into sb.
func (r *predicateResolver) ConditionFor(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey, op qbtypes.FilterOperator, value any) (string, error) {
conds, _, err := r.cb.ConditionFor(ctx, orgID, startNs, endNs, key, r.keys, qbtypes.ConditionBuilderOptions{}, op, value, r.sb)
if err != nil {
return "", err
}
@@ -42,6 +42,6 @@ func (r *predicateResolver) ConditionFor(ctx context.Context, q qbtypes.QueryInf
}
// ExistsFor returns the EXISTS predicate for key.
func (r *predicateResolver) ExistsFor(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return r.ConditionFor(ctx, q, key, qbtypes.FilterOperatorExists, nil)
func (r *predicateResolver) ExistsFor(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
return r.ConditionFor(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorExists, nil)
}

View File

@@ -31,7 +31,8 @@ var (
type scopedTraceStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
storage qbtypes.Storage
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
scope TraceScope
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation]
resourceFilterStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation]
@@ -57,7 +58,9 @@ func NewFactory(
if err != nil {
return nil, err
}
return NewScopedTraceStatementBuilder(settings, metadataStore, tracestelemetryschema.NewStorage(), scope, traceStmtBuilder, fl), nil
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
return NewScopedTraceStatementBuilder(settings, metadataStore, fm, cb, scope, traceStmtBuilder, fl), nil
},
)
}
@@ -67,7 +70,8 @@ func NewFactory(
func NewScopedTraceStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
storage qbtypes.Storage,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
scope TraceScope,
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation],
fl flagger.Flagger,
@@ -88,7 +92,8 @@ func NewScopedTraceStatementBuilder(
return &scopedTraceStatementBuilder{
logger: scopedSettings.Logger(),
metadataStore: metadataStore,
storage: storage,
fm: fieldMapper,
cb: conditionBuilder,
scope: scope,
traceStmtBuilder: traceStmtBuilder,
resourceFilterStmtBuilder: resourceFilterStmtBuilder,
@@ -224,13 +229,12 @@ func (b *scopedTraceStatementBuilder) buildTraceListQuery(
return nil, err
}
matchedSB := sqlbuilder.NewSelectBuilder()
q := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end)
maskExpr, resolved, err := b.resolveFor(ctx, q, keys, matchedSB)
maskExpr, resolved, err := b.resolveFor(ctx, orgID, start, end, keys, matchedSB)
if err != nil {
return nil, err
}
enrichSB := sqlbuilder.NewSelectBuilder()
_, enrichResolved, err := b.resolveFor(ctx, q, keys, enrichSB)
_, enrichResolved, err := b.resolveFor(ctx, orgID, start, end, keys, enrichSB)
if err != nil {
return nil, err
}
@@ -346,15 +350,15 @@ func (b *scopedTraceStatementBuilder) resolverFieldKeys() []*telemetrytypes.Tele
// resolveFor renders the gate mask and every scope column with condition args bound
// into sb.
func (b *scopedTraceStatementBuilder) resolveFor(ctx context.Context, q qbtypes.QueryInfo, keys map[string][]*telemetrytypes.TelemetryFieldKey, sb *sqlbuilder.SelectBuilder) (string, []resolvedColumn, error) {
cols := newColumnResolver(b.storage, keys)
preds := newPredicateResolver(b.storage, keys, sb)
maskExpr, err := b.resolveMask(ctx, q, preds)
func (b *scopedTraceStatementBuilder) resolveFor(ctx context.Context, orgID valuer.UUID, start, end uint64, keys map[string][]*telemetrytypes.TelemetryFieldKey, sb *sqlbuilder.SelectBuilder) (string, []resolvedColumn, error) {
cols := newColumnResolver(b.fm, keys)
preds := newPredicateResolver(b.cb, keys, sb)
maskExpr, err := b.resolveMask(ctx, orgID, start, end, preds)
if err != nil {
return "", nil, err
}
preds.maskExpr = maskExpr
resolved, err := b.resolveColumns(ctx, q, cols, preds)
resolved, err := b.resolveColumns(ctx, orgID, start, end, cols, preds)
if err != nil {
return "", nil, err
}
@@ -362,11 +366,11 @@ func (b *scopedTraceStatementBuilder) resolveFor(ctx context.Context, q qbtypes.
}
// resolveMask builds the per-span in-scope mask: OR of the gate keys' EXISTS predicates.
func (b *scopedTraceStatementBuilder) resolveMask(ctx context.Context, q qbtypes.QueryInfo, preds *predicateResolver) (string, error) {
func (b *scopedTraceStatementBuilder) resolveMask(ctx context.Context, orgID valuer.UUID, start, end uint64, preds *predicateResolver) (string, error) {
fieldKeys := b.scope.FieldKeys
parts := make([]string, 0, len(fieldKeys))
for _, key := range fieldKeys {
e, err := preds.ExistsFor(ctx, q, key)
e, err := preds.ExistsFor(ctx, orgID, start, end, key)
if err != nil {
return "", err
}
@@ -381,10 +385,10 @@ type resolvedColumn struct {
orderable bool
}
func (b *scopedTraceStatementBuilder) resolveColumns(ctx context.Context, q qbtypes.QueryInfo, cols *columnResolver, preds *predicateResolver) ([]resolvedColumn, error) {
func (b *scopedTraceStatementBuilder) resolveColumns(ctx context.Context, orgID valuer.UUID, start, end uint64, cols *columnResolver, preds *predicateResolver) ([]resolvedColumn, error) {
out := make([]resolvedColumn, 0, len(b.scope.Columns))
for _, c := range b.scope.Columns {
expr, err := c.Expr.render(ctx, q, cols, preds)
expr, err := c.Expr.render(ctx, orgID, start, end, cols, preds)
if err != nil {
return nil, err
}
@@ -453,7 +457,7 @@ func (b *scopedTraceStatementBuilder) splitFilter(ctx context.Context, orgID val
}
havingExpr = traceExpr
if strings.TrimSpace(spanExpr) != "" {
pred, warnings, url, err := b.resolveSpanPredicate(ctx, querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end), spanExpr, keys, variables, sb)
pred, warnings, url, err := b.resolveSpanPredicate(ctx, orgID, start, end, spanExpr, keys, variables, sb)
if err != nil {
return fp, err
}
@@ -482,17 +486,21 @@ func (b *scopedTraceStatementBuilder) splitFilter(ctx context.Context, orgID val
// resolveSpanPredicate resolves a span-level filter expression to a bare boolean
// predicate, args bound into sb; keys must cover the expression's selectors.
func (b *scopedTraceStatementBuilder) resolveSpanPredicate(ctx context.Context, q qbtypes.QueryInfo, expr string, keys map[string][]*telemetrytypes.TelemetryFieldKey, variables map[string]qbtypes.VariableItem, sb *sqlbuilder.SelectBuilder) (string, []string, string, error) {
func (b *scopedTraceStatementBuilder) resolveSpanPredicate(ctx context.Context, orgID valuer.UUID, start, end uint64, expr string, keys map[string][]*telemetrytypes.TelemetryFieldKey, variables map[string]qbtypes.VariableItem, sb *sqlbuilder.SelectBuilder) (string, []string, string, error) {
prepared, err := querybuilder.PrepareWhereClause(expr, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: q,
Storage: b.storage,
Logger: b.logger,
FieldKeys: keys,
Builder: sb,
Context: ctx,
OrgID: orgID,
Flagger: b.fl,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
Builder: sb,
// resource conditions are handled by __resource_filter
SkipResourceFilter: true,
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
return "", nil, "", err

View File

@@ -340,7 +340,7 @@ func (b *scopedTraceStatementBuilder) buildQualifiedStatement(
return nil, nil, err
}
sb := sqlbuilder.NewSelectBuilder()
maskExpr, resolved, err := b.resolveFor(ctx, querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end), keys, sb)
maskExpr, resolved, err := b.resolveFor(ctx, orgID, start, end, keys, sb)
if err != nil {
return nil, nil, err
}
@@ -475,15 +475,15 @@ func groupBySelectors(groupBy []qbtypes.GroupByKey) []*telemetrytypes.FieldKeySe
return selectors
}
// resolveGroupColumns resolves group-by keys through the storage for selection
// resolveGroupColumns resolves group-by keys through the field mapper for selection
// inside the per-trace scan; keys must cover the group-by selectors.
func (b *scopedTraceStatementBuilder) resolveGroupColumns(ctx context.Context, q qbtypes.QueryInfo, groupBy []qbtypes.GroupByKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) ([]groupColumn, error) {
func (b *scopedTraceStatementBuilder) resolveGroupColumns(ctx context.Context, orgID valuer.UUID, start, end uint64, groupBy []qbtypes.GroupByKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) ([]groupColumn, error) {
if len(groupBy) == 0 {
return nil, nil
}
out := make([]groupColumn, 0, len(groupBy))
for i := range groupBy {
expr, err := querybuilder.ResolveColumn(ctx, q, b.storage, &groupBy[i].TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &groupBy[i].TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -518,13 +518,12 @@ func (b *scopedTraceStatementBuilder) newScanContext(
) (*scanContext, error) {
sc := &scanContext{sb: sqlbuilder.NewSelectBuilder()}
var err error
q := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end)
sc.maskExpr, sc.resolved, err = b.resolveFor(ctx, q, keys, sc.sb)
sc.maskExpr, sc.resolved, err = b.resolveFor(ctx, orgID, start, end, keys, sc.sb)
if err != nil {
return nil, err
}
if strings.TrimSpace(spanExpr) != "" {
pred, warns, url, err := b.resolveSpanPredicate(ctx, q, spanExpr, keys, variables, sc.sb)
pred, warns, url, err := b.resolveSpanPredicate(ctx, orgID, start, end, spanExpr, keys, variables, sc.sb)
if err != nil {
return nil, err
}
@@ -597,7 +596,7 @@ func (b *scopedTraceStatementBuilder) buildTraceAggregationQuery(
}
}
groupCols, err := b.resolveGroupColumns(ctx, querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end), query.GroupBy, keys)
groupCols, err := b.resolveGroupColumns(ctx, orgID, start, end, query.GroupBy, keys)
if err != nil {
return nil, err
}

View File

@@ -8,6 +8,7 @@ import (
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
qbvariables "github.com/SigNoz/signoz/pkg/variables"
"github.com/huandu/go-sqlbuilder"
)
@@ -52,15 +53,14 @@ func (b *scopedTraceStatementBuilder) resolveTraceHaving(ctx context.Context, ex
fieldKeys[alias] = []*telemetrytypes.TelemetryFieldKey{key}
}
storage := &aliasStorage{allowed: allowed, used: make(map[string]struct{})}
cb := &aliasConditionBuilder{allowed: allowed, used: make(map[string]struct{})}
prepared, err := querybuilder.PrepareWhereClause(expr, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: qbtypes.QueryInfo{Signal: telemetrytypes.SignalTraces},
Storage: storage,
Logger: b.logger,
FieldKeys: fieldKeys,
Variables: variables,
Builder: sb,
Context: ctx,
Logger: b.logger,
ConditionBuilder: cb,
FieldKeys: fieldKeys,
Variables: variables,
Builder: sb,
})
if err != nil {
return nil, err
@@ -68,38 +68,37 @@ func (b *scopedTraceStatementBuilder) resolveTraceHaving(ctx context.Context, ex
if prepared.IsEmpty() {
return nil, nil //nolint:nilnil
}
return &traceHaving{pred: prepared.Expr, used: storage.used}, nil
return &traceHaving{pred: prepared.Expr, used: cb.used}, nil
}
// aliasStorage renders filter conditions directly against the per-trace
// aliases, recording the ones it touches. Every alias is a computed column,
// present in every row; a key resolving to no alias is an error.
type aliasStorage struct {
// aliasConditionBuilder renders filter conditions directly against the per-trace
// aliases, recording the ones it touches; a key resolving to no alias is an error.
type aliasConditionBuilder struct {
allowed map[string]struct{}
used map[string]struct{}
}
var _ qbtypes.Storage = (*aliasStorage)(nil)
var _ qbtypes.ConditionBuilder = (*aliasConditionBuilder)(nil)
func (s *aliasStorage) Read(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (qbtypes.Read, error) {
s.used[key.Name] = struct{}{}
return qbtypes.Read{SQL: quoteAlias(key.Name), Presence: "true", Absence: "false", WhenAbsent: qbtypes.AlwaysPresent}, nil
}
func (s *aliasStorage) Fallback(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, _ qbtypes.FilterOperator, _ any) ([]*telemetrytypes.LogicalField, error) {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput,
"aggregate %q cannot be used in a trace-level filter; filterable aggregates: %s",
key.Name, strings.Join(sortedAliases(s.allowed), ", "))
}
func (s *aliasStorage) Traits() qbtypes.Traits {
return qbtypes.Traits{}
}
// Compile supports the comparison operators only: an aggregate is a number.
func (s *aliasStorage) Compile(_ context.Context, _ qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, op qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
alias := logical.Single().Name
s.used[alias] = struct{}{}
func (c *aliasConditionBuilder) ConditionFor(
_ context.Context,
_ valuer.UUID,
_, _ uint64,
key *telemetrytypes.TelemetryFieldKey,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
op qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
matching := keys[key.Name]
if len(matching) == 0 {
return nil, nil, errors.NewInvalidInputf(errors.CodeInvalidInput,
"aggregate %q cannot be used in a trace-level filter; filterable aggregates: %s",
key.Name, strings.Join(sortedAliases(c.allowed), ", "))
}
alias := matching[0].Name
c.used[alias] = struct{}{}
col := quoteAlias(alias)
var cond string
@@ -129,7 +128,7 @@ func (s *aliasStorage) Compile(_ context.Context, _ qbtypes.QueryInfo, logical *
case qbtypes.FilterOperatorBetween, qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok || len(values) != 2 {
return qbtypes.Compiled{}, errors.NewInvalidInputf(errors.CodeInvalidInput,
return nil, nil, errors.NewInvalidInputf(errors.CodeInvalidInput,
"between on trace-level aggregate %q requires exactly two values", alias)
}
if op == qbtypes.FilterOperatorBetween {
@@ -138,8 +137,8 @@ func (s *aliasStorage) Compile(_ context.Context, _ qbtypes.QueryInfo, logical *
cond = sb.NotBetween(col, values[0], values[1])
}
default:
return qbtypes.Compiled{}, errors.NewInvalidInputf(errors.CodeInvalidInput,
return nil, nil, errors.NewInvalidInputf(errors.CodeInvalidInput,
"trace-level aggregate %q supports only comparison operators (=, !=, <, <=, >, >=, in, between)", alias)
}
return qbtypes.Compiled{Condition: cond}, nil
return []string{cond}, nil, nil
}

View File

@@ -27,7 +27,8 @@ var (
type traceQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
storage qbtypes.Storage
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
resourceFilterResolver *resourcefilter.ResourceFingerprintResolver[qbtypes.TraceAggregation]
aggExprRewriter qbtypes.AggExprRewriter
fl flagger.Flagger
@@ -53,10 +54,11 @@ func NewFactory(
return factory.NewProviderFactory(
factory.MustNewName("traces"),
func(_ context.Context, settings factory.ProviderSettings, cfg statementbuilder.Config) (qbtypes.StatementBuilder[qbtypes.TraceAggregation], error) {
storage := tracestelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, storage, fl, telemetrytypes.SignalTraces)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, fm, cb, fl)
return NewTraceQueryStatementBuilder(
settings, metadataStore, storage, aggExprRewriter, telemetryStore, fl,
settings, metadataStore, fm, cb, aggExprRewriter, telemetryStore, fl,
cfg.SkipResourceFingerprint.Enabled, cfg.SkipResourceFingerprint.Threshold,
), nil
},
@@ -66,7 +68,8 @@ func NewFactory(
func NewTraceQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
storage qbtypes.Storage,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
aggExprRewriter qbtypes.AggExprRewriter,
telemetryStore telemetrystore.TelemetryStore,
flagger flagger.Flagger,
@@ -91,7 +94,8 @@ func NewTraceQueryStatementBuilder(
return &traceQueryStatementBuilder{
logger: tracesSettings.Logger(),
metadataStore: metadataStore,
storage: storage,
fm: fieldMapper,
cb: conditionBuilder,
resourceFilterResolver: resourceFilterResolver,
aggExprRewriter: aggExprRewriter,
fl: flagger,
@@ -378,9 +382,8 @@ func (b *traceQueryStatementBuilder) buildListQuery(
cteArgs = append(cteArgs, scopeArgs...)
}
info := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end)
for i, field := range query.SelectFields {
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &field, telemetrytypes.FieldDataTypeUnspecified, keys)
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &field, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
@@ -404,7 +407,7 @@ func (b *traceQueryStatementBuilder) buildListQuery(
// Add order by
for _, orderBy := range query.Order {
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
@@ -592,9 +595,8 @@ func (b *traceQueryStatementBuilder) buildTimeSeriesQuery(
// Keep original column expressions so we can build the tuple
fieldNames := make([]string, 0, len(query.GroupBy))
info := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end)
for i, gb := range query.GroupBy {
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -756,9 +758,8 @@ func (b *traceQueryStatementBuilder) buildScalarQuery(
allAggChArgs := []any{}
fieldNames := make([]string, 0, len(query.GroupBy))
info := querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end)
for i, gb := range query.GroupBy {
expr, err := querybuilder.ResolveColumn(ctx, info, b.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := b.fm.ColumnExpressionFor(ctx, orgID, start, end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, err
}
@@ -869,12 +870,16 @@ func (b *traceQueryStatementBuilder) addFilterCondition(
// add filter expression
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, orgID, b.fl, telemetrytypes.SignalTraces, nil, start, end),
Storage: b.storage,
OrgID: orgID,
Flagger: b.fl,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
SkipResourceFilter: skipResourceFilter,
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {

View File

@@ -71,7 +71,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -100,7 +100,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR (attributes_string['http.request.method'] = ? AND mapContains(attributes_string, 'http.request.method'))) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR (attributes_string['http.request.method'] = ? AND mapContains(attributes_string, 'http.request.method'))) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (attributes_string['http.request.method'] = ? AND mapContains(attributes_string, 'http.request.method'))) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (attributes_string['http.request.method'] = ? AND mapContains(attributes_string, 'http.request.method'))) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"redis-manual", "GET", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "redis-manual", "GET", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -129,7 +129,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"redis-manual", "redis-manual", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "redis-manual", "redis-manual", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -160,7 +160,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(attribute_string_http$$route) AS `__GROUP_BY_KEY_0_httpRoute`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_httpRoute` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(attribute_string_http$$route) AS `__GROUP_BY_KEY_0_httpRoute`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_httpRoute`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_httpRoute` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_httpRoute`",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(attribute_string_http$$route <> '', attribute_string_http$$route, NULL)) AS `__GROUP_BY_KEY_0_httpRoute`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_httpRoute` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(attribute_string_http$$route <> '', attribute_string_http$$route, NULL)) AS `__GROUP_BY_KEY_0_httpRoute`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_httpRoute`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_httpRoute` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_httpRoute`",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -198,7 +198,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(attribute_string_http$$route) AS `__GROUP_BY_KEY_0_httpRoute`, toString(http_method) AS `__GROUP_BY_KEY_1_httpMethod`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (resource_string_service$$name = ? AND http_method <> '' AND kind_string = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(attribute_string_http$$route) AS `__GROUP_BY_KEY_0_httpRoute`, toString(http_method) AS `__GROUP_BY_KEY_1_httpMethod`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (resource_string_service$$name = ? AND http_method <> '' AND kind_string = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(attribute_string_http$$route <> '', attribute_string_http$$route, NULL)) AS `__GROUP_BY_KEY_0_httpRoute`, toString(multiIf(http_method <> '', http_method, NULL)) AS `__GROUP_BY_KEY_1_httpMethod`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((resource_string_service$$name = ? AND resource_string_service$$name <> '') AND http_method <> '' AND kind_string = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(attribute_string_http$$route <> '', attribute_string_http$$route, NULL)) AS `__GROUP_BY_KEY_0_httpRoute`, toString(multiIf(http_method <> '', http_method, NULL)) AS `__GROUP_BY_KEY_1_httpMethod`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE ((resource_string_service$$name = ? AND resource_string_service$$name <> '') AND http_method <> '' AND kind_string = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_httpRoute`, `__GROUP_BY_KEY_1_httpMethod`",
Args: []any{"redis-manual", "Server", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "redis-manual", "Server", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -239,7 +239,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(mapContains(attributes_number, 'metric.max_count'), toFloat64(attributes_number['metric.max_count']), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 desc LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(mapContains(attributes_number, 'metric.max_count'), toFloat64(attributes_number['metric.max_count']), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name` ORDER BY ts desc",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(mapContains(attributes_number, 'metric.max_count'), toFloat64(attributes_number['metric.max_count']), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 desc LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(mapContains(attributes_number, 'metric.max_count'), toFloat64(attributes_number['metric.max_count']), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name` ORDER BY ts desc",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -268,7 +268,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -307,7 +307,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY `__GROUP_BY_KEY_0_service.name` desc LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name` ORDER BY `__GROUP_BY_KEY_0_service.name` desc, ts desc",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY `__GROUP_BY_KEY_0_service.name` desc LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, sum(multiIf(`attribute_number_cart$$items_count_exists`, toFloat64(`attribute_number_cart$$items_count`), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name` ORDER BY `__GROUP_BY_KEY_0_service.name` desc, ts desc",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -338,7 +338,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(response_status_code) AS `__GROUP_BY_KEY_0_responseStatusCode`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_responseStatusCode` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(response_status_code) AS `__GROUP_BY_KEY_0_responseStatusCode`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_responseStatusCode`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_responseStatusCode` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_responseStatusCode`",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(response_status_code <> '', response_status_code, NULL)) AS `__GROUP_BY_KEY_0_responseStatusCode`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_responseStatusCode` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(response_status_code <> '', response_status_code, NULL)) AS `__GROUP_BY_KEY_0_responseStatusCode`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_responseStatusCode`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_responseStatusCode` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_responseStatusCode`",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -369,7 +369,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(response_status_code) AS `__GROUP_BY_KEY_0_responseStatusCode`, quantile(0.90)(toFloat64(duration_nano)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_responseStatusCode` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(response_status_code) AS `__GROUP_BY_KEY_0_responseStatusCode`, quantile(0.90)(toFloat64(duration_nano)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_responseStatusCode`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_responseStatusCode` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_responseStatusCode`",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), __limit_cte AS (SELECT toString(multiIf(response_status_code <> '', response_status_code, NULL)) AS `__GROUP_BY_KEY_0_responseStatusCode`, quantile(0.90)(multiIf(duration_nano <> 0, accurateCastOrNull(duration_nano, 'Float64'), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_responseStatusCode` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(response_status_code <> '', response_status_code, NULL)) AS `__GROUP_BY_KEY_0_responseStatusCode`, quantile(0.90)(multiIf(duration_nano <> 0, accurateCastOrNull(duration_nano, 'Float64'), NULL)) AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_responseStatusCode`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_responseStatusCode` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_responseStatusCode`",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -458,22 +458,24 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE (scope.version::String = ? AND scope.version::String <> '') AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"1.0.0", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1.0.0", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
},
}
fl := flaggertest.New(t)
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
nil,
fl,
@@ -669,7 +671,7 @@ func TestStatementBuilderListQuery(t *testing.T) {
Limit: 10,
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint) SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, name AS `__SELECT_KEY_3_name`, resource_string_service$$name AS `__SELECT_KEY_4_serviceName`, duration_nano AS `__SELECT_KEY_5_durationNano`, http_method AS `__SELECT_KEY_6_httpMethod`, multiIf(`attribute_string_mixed$$materialization$$key_exists`, `attribute_string_mixed$$materialization$$key`, multiIf(resource.`mixed.materialization.key` IS NOT NULL, resource.`mixed.materialization.key`::String, mapContains(resources_string, 'mixed.materialization.key'), resources_string['mixed.materialization.key'], NULL) IS NOT NULL, multiIf(resource.`mixed.materialization.key` IS NOT NULL, resource.`mixed.materialization.key`::String, mapContains(resources_string, 'mixed.materialization.key'), resources_string['mixed.materialization.key'], NULL), NULL) AS `__SELECT_KEY_7_mixed.materialization.key` FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint) SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, name AS `__SELECT_KEY_3_name`, resource_string_service$$name AS `__SELECT_KEY_4_serviceName`, duration_nano AS `__SELECT_KEY_5_durationNano`, http_method AS `__SELECT_KEY_6_httpMethod`, multiIf(`attribute_string_mixed$$materialization$$key_exists`, toString(`attribute_string_mixed$$materialization$$key`), multiIf(resource.`mixed.materialization.key` IS NOT NULL, resource.`mixed.materialization.key`::String, mapContains(resources_string, 'mixed.materialization.key'), resources_string['mixed.materialization.key'], NULL) IS NOT NULL, toString(multiIf(resource.`mixed.materialization.key` IS NOT NULL, resource.`mixed.materialization.key`::String, mapContains(resources_string, 'mixed.materialization.key'), resources_string['mixed.materialization.key'], NULL)), NULL) AS `__SELECT_KEY_7_mixed.materialization.key` FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
expectedErr: nil,
@@ -756,7 +758,7 @@ func TestStatementBuilderListQuery(t *testing.T) {
Limit: 10,
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint) SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, name AS `__SELECT_KEY_3_name`, resource_string_service$$name AS `__SELECT_KEY_4_serviceName`, duration_nano AS `__SELECT_KEY_5_durationNano`, http_method AS `__SELECT_KEY_6_httpMethod`, multiIf(mapContains(attributes_string, 'non-existent.key'), attributes_string['non-existent.key'], mapContains(attributes_number, 'non-existent.key'), toString(attributes_number['non-existent.key']), mapContains(attributes_bool, 'non-existent.key'), toString(attributes_bool['non-existent.key']), NULL) AS `__SELECT_KEY_7_non-existent.key` FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint) SELECT timestamp AS `__SELECT_KEY_0_timestamp`, trace_id AS `__SELECT_KEY_1_trace_id`, span_id AS `__SELECT_KEY_2_span_id`, name AS `__SELECT_KEY_3_name`, resource_string_service$$name AS `__SELECT_KEY_4_serviceName`, duration_nano AS `__SELECT_KEY_5_durationNano`, http_method AS `__SELECT_KEY_6_httpMethod`, multiIf(mapContains(attributes_string, 'non-existent.key'), toString(attributes_string['non-existent.key']), mapContains(attributes_number, 'non-existent.key'), toString(attributes_number['non-existent.key']), mapContains(attributes_bool, 'non-existent.key'), toString(attributes_bool['non-existent.key']), NULL) AS `__SELECT_KEY_7_non-existent.key` FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"redis-manual", "%service.name%", "%service.name\":\"redis-manual%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
expectedErr: nil,
@@ -764,15 +766,17 @@ func TestStatementBuilderListQuery(t *testing.T) {
}
fl := flaggertest.New(t)
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
nil,
fl,
@@ -1026,18 +1030,20 @@ func TestStatementBuilderListQueryWithCorruptData(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.New(t)
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = c.keysMap
if mockMetadataStore.KeysMap == nil {
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
}
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
nil,
fl,
@@ -1074,7 +1080,7 @@ func TestStatementBuilderGroupByResourceEvolution(t *testing.T) {
startMs: 1747947419000, // 2025-05-22 21:56:59 UTC, ~3m before release
endMs: 1747983448000, // 2025-05-23 07:57:28 UTC, ~10h after release
expected: qbtypes.Statement{
Query: "WITH __limit_cte AS (SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Query: "WITH __limit_cte AS (SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? GROUP BY `__GROUP_BY_KEY_0_service.name` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(timestamp, INTERVAL 30 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `__GROUP_BY_KEY_0_service.name`, count() AS __result_0 FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND (`__GROUP_BY_KEY_0_service.name`) GLOBAL IN (SELECT `__GROUP_BY_KEY_0_service.name` FROM __limit_cte) GROUP BY ts, `__GROUP_BY_KEY_0_service.name`",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10, "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
},
@@ -1099,15 +1105,17 @@ func TestStatementBuilderGroupByResourceEvolution(t *testing.T) {
}
fl := flaggertest.New(t)
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
nil,
fl,
@@ -1194,7 +1202,7 @@ func TestStatementBuilderTraceQuery(t *testing.T) {
Limit: 10,
},
expected: qbtypes.Statement{
Query: "WITH __toe AS (SELECT trace_id FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __toe_duration_sorted AS (SELECT trace_id, duration_nano, resource_string_service$$name as `service.name`, name FROM signoz_traces.distributed_signoz_index_v3 WHERE parent_span_id = '' AND trace_id GLOBAL IN __toe AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? ORDER BY duration_nano DESC LIMIT 1 BY trace_id) SELECT __toe_duration_sorted.`service.name` AS `service.name`, __toe_duration_sorted.name AS `name`, count() AS span_count, __toe_duration_sorted.duration_nano AS `duration_nano`, __toe_duration_sorted.trace_id AS `trace_id` FROM __toe INNER JOIN __toe_duration_sorted ON __toe.trace_id = __toe_duration_sorted.trace_id GROUP BY trace_id, duration_nano, name, `service.name` ORDER BY duration_nano DESC LIMIT 1 BY trace_id LIMIT ? SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Query: "WITH __toe AS (SELECT trace_id FROM signoz_traces.distributed_signoz_index_v3 WHERE ((match(`attribute_string_materialized$$key$$name`, ?) AND `attribute_string_materialized$$key$$name_exists`) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __toe_duration_sorted AS (SELECT trace_id, duration_nano, resource_string_service$$name as `service.name`, name FROM signoz_traces.distributed_signoz_index_v3 WHERE parent_span_id = '' AND trace_id GLOBAL IN __toe AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? ORDER BY duration_nano DESC LIMIT 1 BY trace_id) SELECT __toe_duration_sorted.`service.name` AS `service.name`, __toe_duration_sorted.name AS `name`, count() AS span_count, __toe_duration_sorted.duration_nano AS `duration_nano`, __toe_duration_sorted.trace_id AS `trace_id` FROM __toe INNER JOIN __toe_duration_sorted ON __toe.trace_id = __toe_duration_sorted.trace_id GROUP BY trace_id, duration_nano, name, `service.name` ORDER BY duration_nano DESC LIMIT 1 BY trace_id LIMIT ? SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Args: []any{"redis-manual", "redis-manual", "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), 10},
},
expectedErr: nil,
@@ -1264,15 +1272,17 @@ func TestStatementBuilderTraceQuery(t *testing.T) {
}
fl := flaggertest.New(t)
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
nil,
fl,
@@ -1900,17 +1910,21 @@ func newSkipResourceFingerprintBuilder(
t.Helper()
fl := flaggertest.New(t)
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(
instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl,
)
return NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
telemetryStore,
fl,
@@ -1926,15 +1940,17 @@ func TestStatementBuilderGroupByUnseenKey(t *testing.T) {
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
nil,
fl,
@@ -1966,15 +1982,17 @@ func TestStatementBuilderAggregationUnseenKey(t *testing.T) {
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
nil,
fl,
@@ -2039,15 +2057,17 @@ func TestStatementBuilderSemconvFamilies(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureResolveSemconvFamilies.String(): c.flag,
})
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
nil,
fl,

View File

@@ -264,11 +264,15 @@ func (b *traceOperatorCTEBuilder) buildQueryCTE(ctx context.Context, queryName s
query.Filter.Expression,
querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: querybuilder.NewQueryInfo(ctx, b.orgID, b.stmtBuilder.fl, telemetrytypes.SignalTraces, nil, b.start, b.end),
Storage: b.stmtBuilder.storage,
OrgID: b.orgID,
Flagger: b.stmtBuilder.fl,
Logger: b.stmtBuilder.logger,
FieldMapper: b.stmtBuilder.fm,
ConditionBuilder: b.stmtBuilder.cb,
FieldKeys: keys,
SkipResourceFilter: true,
StartNs: b.start,
EndNs: b.end,
},
)
if err != nil {
@@ -484,12 +488,11 @@ func (b *traceOperatorCTEBuilder) buildListQuery(ctx context.Context, selectFrom
}
// Add selectFields since we now have all base table columns
info := querybuilder.NewQueryInfo(ctx, b.orgID, b.stmtBuilder.fl, telemetrytypes.SignalTraces, nil, b.start, b.end)
for i, field := range b.operator.SelectFields {
if selectedFields[field.Name] {
continue
}
expr, err := querybuilder.ResolveColumn(ctx, info, b.stmtBuilder.storage, &field, telemetrytypes.FieldDataTypeUnspecified, keys)
expr, err := b.stmtBuilder.fm.ColumnExpressionFor(ctx, b.orgID, b.start, b.end, &field, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
b.stmtBuilder.logger.WarnContext(ctx, "failed to map select field",
slog.String("field", field.Name), errors.Attr(err))
@@ -510,7 +513,7 @@ func (b *traceOperatorCTEBuilder) buildListQuery(ctx context.Context, selectFrom
// Add order by support
orderApplied := false
for _, orderBy := range b.operator.Order {
expr, err := querybuilder.ResolveColumn(ctx, info, b.stmtBuilder.storage, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
expr, err := b.stmtBuilder.fm.ColumnExpressionFor(ctx, b.orgID, b.start, b.end, &orderBy.Key.TelemetryFieldKey, telemetrytypes.FieldDataTypeUnspecified, keys)
if err != nil {
return nil, err
}
@@ -628,9 +631,8 @@ func (b *traceOperatorCTEBuilder) buildTimeSeriesQuery(ctx context.Context, sele
int64(b.operator.StepInterval.Seconds()),
))
info := querybuilder.NewQueryInfo(ctx, b.orgID, b.stmtBuilder.fl, telemetrytypes.SignalTraces, nil, b.start, b.end)
for _, gb := range b.operator.GroupBy {
expr, err := querybuilder.ResolveColumn(ctx, info, b.stmtBuilder.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := b.stmtBuilder.fm.ColumnExpressionFor(ctx, b.orgID, b.start, b.end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, errors.NewInvalidInputf(
errors.CodeInvalidInput,
@@ -725,9 +727,8 @@ func (b *traceOperatorCTEBuilder) buildTraceQuery(ctx context.Context, selectFro
sb := sqlbuilder.NewSelectBuilder()
info := querybuilder.NewQueryInfo(ctx, b.orgID, b.stmtBuilder.fl, telemetrytypes.SignalTraces, nil, b.start, b.end)
for _, gb := range b.operator.GroupBy {
expr, err := querybuilder.ResolveColumn(ctx, info, b.stmtBuilder.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := b.stmtBuilder.fm.ColumnExpressionFor(ctx, b.orgID, b.start, b.end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, errors.NewInvalidInputf(
errors.CodeInvalidInput,
@@ -852,9 +853,8 @@ func (b *traceOperatorCTEBuilder) buildTraceQuery(ctx context.Context, selectFro
func (b *traceOperatorCTEBuilder) buildScalarQuery(ctx context.Context, selectFromCTE string, keys map[string][]*telemetrytypes.TelemetryFieldKey) (*qbtypes.Statement, error) {
sb := sqlbuilder.NewSelectBuilder()
info := querybuilder.NewQueryInfo(ctx, b.orgID, b.stmtBuilder.fl, telemetrytypes.SignalTraces, nil, b.start, b.end)
for _, gb := range b.operator.GroupBy {
expr, err := querybuilder.ResolveColumn(ctx, info, b.stmtBuilder.storage, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
expr, err := b.stmtBuilder.fm.ColumnExpressionFor(ctx, b.orgID, b.start, b.end, &gb.TelemetryFieldKey, telemetrytypes.FieldDataTypeString, keys)
if err != nil {
return nil, errors.NewInvalidInputf(
errors.CodeInvalidInput,

View File

@@ -20,17 +20,18 @@ func newTestTraceOperatorStatementBuilder(t *testing.T) *traceOperatorStatementB
t.Helper()
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
traceStmtBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore, storage, aggExprRewriter, nil, fl, false, 100000,
mockMetadataStore, fm, cb, aggExprRewriter, nil, fl, false, 100000,
)
return NewTraceOperatorStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore, storage, traceStmtBuilder, aggExprRewriter, fl,
mockMetadataStore, fm, cb, traceStmtBuilder, aggExprRewriter, fl,
)
}
@@ -284,7 +285,7 @@ func TestTraceOperatorStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH toDateTime64(1747947419000000000, 9) AS t_from, toDateTime64(1747983448000000000, 9) AS t_to, 1747945619 AS bucket_from, 1747983448 AS bucket_to, all_spans AS (SELECT *, resource_string_service$$name AS `service.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_A AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), A AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_A) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_B AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), B AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_B) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), A_DIR_DESC_B AS (SELECT p.* FROM A AS p INNER JOIN B AS c ON p.trace_id = c.trace_id AND p.span_id = c.parent_span_id) SELECT toStartOfInterval(timestamp, INTERVAL 60 SECOND) AS ts, toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `service.name`, count() AS __result_0 FROM A_DIR_DESC_B GROUP BY ts, `service.name` ORDER BY ts desc SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Query: "WITH toDateTime64(1747947419000000000, 9) AS t_from, toDateTime64(1747983448000000000, 9) AS t_to, 1747945619 AS bucket_from, 1747983448 AS bucket_to, all_spans AS (SELECT *, resource_string_service$$name AS `service.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_A AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), A AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_A) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_B AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), B AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_B) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), A_DIR_DESC_B AS (SELECT p.* FROM A AS p INNER JOIN B AS c ON p.trace_id = c.trace_id AND p.span_id = c.parent_span_id) SELECT toStartOfInterval(timestamp, INTERVAL 60 SECOND) AS ts, toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `service.name`, count() AS __result_0 FROM A_DIR_DESC_B GROUP BY ts, `service.name` ORDER BY ts desc SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), "frontend", "%service.name%", "%service.name\":\"frontend%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), "backend", "%service.name%", "%service.name\":\"backend%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448)},
},
expectedErr: nil,
@@ -343,7 +344,7 @@ func TestTraceOperatorStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH toDateTime64(1747947419000000000, 9) AS t_from, toDateTime64(1747983448000000000, 9) AS t_to, 1747945619 AS bucket_from, 1747983448 AS bucket_to, all_spans AS (SELECT *, resource_string_service$$name AS `service.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_A AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), A AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_A) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), B AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND toFloat64(response_status_code) < ?), A_AND_B AS (SELECT l.* FROM A AS l INNER JOIN B AS r ON l.trace_id = r.trace_id) SELECT toString(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL)) AS `service.name`, avg(toFloat64(duration_nano)) AS __result_0 FROM A_AND_B GROUP BY `service.name` ORDER BY __result_0 desc SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Query: "WITH toDateTime64(1747947419000000000, 9) AS t_from, toDateTime64(1747983448000000000, 9) AS t_to, 1747945619 AS bucket_from, 1747983448 AS bucket_to, all_spans AS (SELECT *, resource_string_service$$name AS `service.name` FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), __resource_filter_A AS (SELECT fingerprint FROM signoz_traces.distributed_traces_v3_resource WHERE (simpleJSONExtractString(labels, 'service.name') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ? GROUP BY fingerprint), A AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter_A) AND timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ?), B AS (SELECT * FROM signoz_traces.distributed_signoz_index_v3 WHERE timestamp >= ? AND timestamp < ? AND ts_bucket_start >= ? AND ts_bucket_start <= ? AND toFloat64(response_status_code) < ?), A_AND_B AS (SELECT l.* FROM A AS l INNER JOIN B AS r ON l.trace_id = r.trace_id) SELECT toString(multiIf(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL, multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL), NULL)) AS `service.name`, avg(multiIf(duration_nano <> 0, accurateCastOrNull(duration_nano, 'Float64'), mapContains(attributes_number, 'duration_nano'), toFloat64(attributes_number['duration_nano']), NULL)) AS __result_0 FROM A_AND_B GROUP BY `service.name` ORDER BY __result_0 desc SETTINGS distributed_product_mode='allow', max_memory_usage=10000000000",
Args: []any{"1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), "frontend", "%service.name%", "%service.name\":\"frontend%", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), "1747947419000000000", "1747983448000000000", uint64(1747945619), uint64(1747983448), float64(400)},
},
expectedErr: nil,

View File

@@ -20,7 +20,8 @@ import (
type traceOperatorStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
storage qbtypes.Storage
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation]
resourceFilterStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation]
aggExprRewriter qbtypes.AggExprRewriter
@@ -41,14 +42,15 @@ func NewOperatorFactory(
return factory.NewProviderFactory(
factory.MustNewName("traceoperator"),
func(_ context.Context, settings factory.ProviderSettings, cfg statementbuilder.Config) (qbtypes.TraceOperatorStatementBuilder, error) {
storage := tracestelemetryschema.NewStorage()
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, storage, fl, telemetrytypes.SignalTraces)
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, fm, cb, fl)
traceStmtBuilder := NewTraceQueryStatementBuilder(
settings, metadataStore, storage, aggExprRewriter, telemetryStore, fl,
settings, metadataStore, fm, cb, aggExprRewriter, telemetryStore, fl,
cfg.SkipResourceFingerprint.Enabled, cfg.SkipResourceFingerprint.Threshold,
)
return NewTraceOperatorStatementBuilder(
settings, metadataStore, storage, traceStmtBuilder, aggExprRewriter, fl,
settings, metadataStore, fm, cb, traceStmtBuilder, aggExprRewriter, fl,
), nil
},
)
@@ -57,7 +59,8 @@ func NewOperatorFactory(
func NewTraceOperatorStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
storage qbtypes.Storage,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation],
aggExprRewriter qbtypes.AggExprRewriter,
flagger flagger.Flagger,
@@ -78,7 +81,8 @@ func NewTraceOperatorStatementBuilder(
return &traceOperatorStatementBuilder{
logger: tracesSettings.Logger(),
metadataStore: metadataStore,
storage: storage,
fm: fieldMapper,
cb: conditionBuilder,
traceStmtBuilder: traceStmtBuilder,
resourceFilterStmtBuilder: resourceFilterStmtBuilder,
aggExprRewriter: aggExprRewriter,

View File

@@ -22,7 +22,8 @@ func TestTraceTimeRangeOptimization(t *testing.T) {
releaseTime := time.Date(2025, 5, 22, 22, 0, 0, 0, time.UTC)
fl := flaggertest.New(t)
storage := tracestelemetryschema.NewStorage()
fm := tracestelemetryschema.NewFieldMapper(fl)
cb := tracestelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = tracestelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
@@ -39,12 +40,13 @@ func TestTraceTimeRangeOptimization(t *testing.T) {
Signal: telemetrytypes.SignalTraces,
}}
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, storage, fl, telemetrytypes.SignalTraces)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewTraceQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
storage,
fm,
cb,
aggExprRewriter,
nil, // telemetryStore is nil - adaptive path is disabled
fl,

View File

@@ -0,0 +1,61 @@
package telemetrymetadata
import (
"strconv"
"strings"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// boolFieldValues is the suggestion set for a bool field, optionally narrowed
// by the search text.
func boolFieldValues(searchText string) *telemetrytypes.TelemetryFieldValues {
values := &telemetrytypes.TelemetryFieldValues{}
needle := strings.ToLower(searchText)
for _, v := range []bool{true, false} {
if needle == "" || strings.Contains(strconv.FormatBool(v), needle) {
values.BoolValues = append(values.BoolValues, v)
}
}
return values
}
// spanSearchScopeFieldValues is the suggestion set for a search-scope selector
// (isRoot, isEntryPoint), which only filters with true. ok is false for any
// other name.
func spanSearchScopeFieldValues(name, searchText string) (*telemetrytypes.TelemetryFieldValues, bool) {
for scopeName := range tracestelemetryschema.SpanSearchScopeFields {
if !strings.EqualFold(scopeName, name) {
continue
}
values := &telemetrytypes.TelemetryFieldValues{}
if needle := strings.ToLower(searchText); needle == "" || strings.Contains("true", needle) {
values.BoolValues = []bool{true}
}
return values, true
}
return nil, false
}
// isKnownBoolField is true when the caller asked for the bool data type, or
// when the name is one of the signal's static bool fields and the requested
// context does not rule that static field out.
func isKnownBoolField(selector *telemetrytypes.FieldValueSelector, staticFields ...map[string]telemetrytypes.TelemetryFieldKey) bool {
if selector.FieldDataType == telemetrytypes.FieldDataTypeBool {
return true
}
if selector.FieldDataType != telemetrytypes.FieldDataTypeUnspecified {
return false
}
for _, fields := range staticFields {
field, ok := fields[selector.Name]
if !ok || field.FieldDataType != telemetrytypes.FieldDataTypeBool {
continue
}
if selector.FieldContext == telemetrytypes.FieldContextUnspecified || selector.FieldContext == field.FieldContext {
return true
}
}
return false
}

View File

@@ -8,26 +8,61 @@ import (
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
// Compile keeps the related-values polarity form for a single key: the key
// must exist to apply the filter, and a row without it answers with the
// operator's polarity. A family compiles through the shared condition.
func (s *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
if logical.IsFamily() {
return querybuilder.SharedCondition(ctx, q, s, logical, operator, value, sb)
}
condition, err := s.conditionForKey(ctx, q, logical.Single(), operator, value, sb)
if err != nil {
return qbtypes.Compiled{}, err
}
return qbtypes.Compiled{Condition: condition}, nil
type conditionBuilder struct {
fm qbtypes.FieldMapper
}
func (s *storage) conditionForKey(
func NewConditionBuilder(fm qbtypes.FieldMapper) *conditionBuilder {
return &conditionBuilder{fm: fm}
}
// Metadata has no resource sub-query, so options are unused.
func (c *conditionBuilder) ConditionFor(
ctx context.Context,
q qbtypes.QueryInfo,
orgID valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
_ qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken are logs-body-only; reject to avoid malformed related-values SQL.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}
// an unknown key simply yields no condition rather than an error. Metadata
// fields have no family support, so every logical field is single-member
// and flattens losslessly to its physical key.
resolved, warning := querybuilder.ResolveLogicalFields(key, querybuilder.MatchingLogicalFields(ctx, orgID, nil, key, fieldKeys))
keys := querybuilder.SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
cond, err := c.conditionForKey(ctx, orgID, tsStart, tsEnd, k, operator, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
}
return conds, warnings, nil
}
func (c *conditionBuilder) conditionForKey(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
@@ -44,13 +79,13 @@ func (s *storage) conditionForKey(
value = querybuilder.FormatValueForContains(value)
}
columns, err := s.getColumn(ctx, q.StartNs, q.EndNs, key)
columns, err := c.fm.ColumnFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
// if we don't have a column, we can't build a condition for related values
return "", nil
}
fieldExpression, err := s.read(ctx, q, key)
fieldExpression, err := c.fm.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
// if we don't have a table field name, we can't build a condition for related values
return "", nil

View File

@@ -2,11 +2,11 @@ package telemetrymetadata
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -14,7 +14,7 @@ import (
func TestConditionFor(t *testing.T) {
ctx := context.Background()
storage := NewStorage()
conditionBuilder := NewConditionBuilder(NewFieldMapper())
testCases := []struct {
name string
@@ -198,7 +198,7 @@ func TestConditionFor(t *testing.T) {
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, _, err := querybuilder.Conditions(ctx, qbtypes.QueryInfo{}, storage, &tc.key, tc.operator, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, false, sb)
cond, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, 0, 0, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
sb.Where(cond...)
if tc.expectedError != nil {

View File

@@ -3,10 +3,15 @@ package telemetrymetadata
import (
"context"
"fmt"
"strings"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"golang.org/x/exp/maps"
)
var (
@@ -22,15 +27,20 @@ var (
}
)
type storage struct{}
var _ qbtypes.Storage = (*storage)(nil)
func NewStorage() qbtypes.Storage {
return &storage{}
type fieldMapper struct {
}
func (m *storage) getColumn(_ context.Context, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
// CandidateKeys returns nil: this mapper has no attribute-map fallback, so a context-missing
// key stays unresolved and the caller errors.
func (m *fieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, _ *telemetrytypes.TelemetryFieldKey, _ any, _ map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
}
func NewFieldMapper() qbtypes.FieldMapper {
return &fieldMapper{}
}
func (m *fieldMapper) getColumn(_ context.Context, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
switch key.FieldContext {
case telemetrytypes.FieldContextResource:
return []*schema.Column{attributeMetadataColumns["resource_attributes"]}, nil
@@ -40,8 +50,29 @@ func (m *storage) getColumn(_ context.Context, _, _ uint64, key *telemetrytypes.
return nil, qbtypes.ErrColumnNotFound
}
func (m *storage) read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, q.StartNs, q.EndNs, key)
func (m *fieldMapper) ColumnFor(ctx context.Context, _ valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
columns, err := m.getColumn(ctx, tsStart, tsEnd, key)
if err != nil {
return nil, err
}
return columns, nil
}
// ExistsFor implements the per-key existence primitive of qbtypes.FieldMapper.
func (m *fieldMapper) ExistsFor(ctx context.Context, _ valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey, exists bool) (string, error) {
columns, err := m.getColumn(ctx, tsStart, tsEnd, key)
if err != nil {
return "", err
}
pred := fmt.Sprintf("mapContains(%s, '%s')", columns[0].Name, key.Name)
if exists {
return pred, nil
}
return "NOT " + pred, nil
}
func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, startNs, endNs, key)
if err != nil {
return "", err
}
@@ -56,31 +87,48 @@ func (m *storage) read(ctx context.Context, q qbtypes.QueryInfo, key *telemetryt
return columns[0].Name, nil
}
// Read composes the bare read of one key with its map membership test.
func (m *storage) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (qbtypes.Read, error) {
columns, err := m.getColumn(ctx, q.StartNs, q.EndNs, key)
if err != nil {
return qbtypes.Read{}, err
}
sql, err := m.read(ctx, q, key)
if err != nil {
return qbtypes.Read{}, err
}
presence := fmt.Sprintf("mapContains(%s, '%s')", columns[0].Name, key.Name)
return qbtypes.Read{
SQL: sql,
Presence: presence,
Absence: "NOT " + presence,
WhenAbsent: qbtypes.AbsentIsSentinel,
}, nil
}
func (m *fieldMapper) ColumnExpressionFor(
ctx context.Context,
orgID valuer.UUID,
startNs, endNs uint64,
field *telemetrytypes.TelemetryFieldKey,
_ telemetrytypes.FieldDataType,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
// Fallback returns nil: the metadata tables have no attribute synthesis, so
// a key metadata does not hold yields no condition.
func (m *storage) Fallback(context.Context, qbtypes.QueryInfo, *telemetrytypes.TelemetryFieldKey, qbtypes.FilterOperator, any) ([]*telemetrytypes.LogicalField, error) {
return nil, nil
}
fieldExpression, err := m.FieldFor(ctx, orgID, startNs, endNs, field)
if errors.Is(err, qbtypes.ErrColumnNotFound) {
// the key didn't have the right context to be added to the query
// we try to use the context we know of
keysForField := keys[field.Name]
if len(keysForField) == 0 {
// is it a static field?
if _, ok := attributeMetadataColumns[field.Name]; ok {
// if it is, attach the column name directly
field.FieldContext = telemetrytypes.FieldContextSpan
fieldExpression, _ = m.FieldFor(ctx, orgID, startNs, endNs, field)
} else {
// - the context is not provided
// - there are not keys for the field
// - it is not a static field
// - the next best thing to do is see if there is a typo
// and suggest a correction
wrappedErr := errors.Wrapf(err, errors.TypeInvalidInput, errors.CodeInvalidInput, "field `%s` not found", field.Name).WithSuggestions(errors.NewSuggestionsOnLevenshteinDistance(field.Name, errors.NounKeys, maps.Keys(keys))...)
return "", wrappedErr
}
} else if len(keysForField) == 1 {
// we have a single key for the field, use it
fieldExpression, _ = m.FieldFor(ctx, orgID, startNs, endNs, keysForField[0])
} else {
// select any non-empty value from the keys
args := []string{}
for _, key := range keysForField {
fieldExpression, _ = m.FieldFor(ctx, orgID, startNs, endNs, key)
args = append(args, fmt.Sprintf("toString(%s) != '', toString(%s)", fieldExpression, fieldExpression))
}
fieldExpression = fmt.Sprintf("multiIf(%s, NULL)", strings.Join(args, ", "))
}
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{UnknownKey: qbtypes.IgnoreUnknownKey}
return fmt.Sprintf("%s AS `%s`", sqlbuilder.Escape(fieldExpression), field.Name), nil
}

View File

@@ -7,6 +7,7 @@ import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -124,11 +125,11 @@ func TestGetColumn(t *testing.T) {
},
}
storage := &storage{}
fm := NewFieldMapper()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
col, err := storage.getColumn(context.Background(), 0, 0, &tc.key)
col, err := fm.ColumnFor(context.Background(), valuer.UUID{}, 0, 0, &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
@@ -201,17 +202,17 @@ func TestGetFieldKeyName(t *testing.T) {
},
}
storage := NewStorage()
fm := NewFieldMapper()
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
read, err := storage.Read(ctx, qbtypes.QueryInfo{StartNs: tc.tsStart, EndNs: tc.tsEnd}, &tc.key)
result, err := fm.FieldFor(ctx, valuer.UUID{}, tc.tsStart, tc.tsEnd, &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
} else {
require.NoError(t, err)
assert.Equal(t, tc.expectedResult, read.SQL)
assert.Equal(t, tc.expectedResult, result)
}
})
}

View File

@@ -65,7 +65,8 @@ type telemetryMetaStore struct {
relatedMetadataTblName string
columnEvolutionMetadataTblName string
storage qbtypes.Storage
fm qbtypes.FieldMapper
conditionBuilder qbtypes.ConditionBuilder
fl flagger.Flagger
jsonColumnMetadata map[telemetrytypes.Signal]map[telemetrytypes.FieldContext]telemetrytypes.JSONColumnMetadata
}
@@ -81,6 +82,9 @@ func NewTelemetryMetaStore(
) telemetrytypes.MetadataStore {
metadataSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrymetadata")
fm := NewFieldMapper()
conditionBuilder := NewConditionBuilder(fm)
t := &telemetryMetaStore{
logger: metadataSettings.Logger(),
telemetrystore: telemetrystore,
@@ -113,8 +117,9 @@ func NewTelemetryMetaStore(
},
},
},
fl: fl,
storage: NewStorage(),
fl: fl,
fm: fm,
conditionBuilder: conditionBuilder,
}
return t
@@ -182,8 +187,6 @@ func (t *telemetryMetaStore) getTracesKeys(ctx context.Context, fieldKeySelector
).From(t.tracesDBName + "." + t.spanAttributesKeysTblName)
var limit int
searchTexts := []string{}
conds := []string{}
for _, fieldKeySelector := range fieldKeySelectors {
@@ -203,14 +206,12 @@ func (t *telemetryMetaStore) getTracesKeys(ctx context.Context, fieldKeySelector
fieldKeyConds = append(fieldKeyConds, sb.ILike("tagKey", "%"+escapeForLike(fieldKeySelector.Name)+"%"))
}
searchTexts = append(searchTexts, fieldKeySelector.Name)
// now look at the field context
// we don't write most of intrinsic fields to keys table
// for this reason we don't want to apply tagType if the field context
// is not attribute or resource attribute
if fieldKeySelector.FieldContext != telemetrytypes.FieldContextUnspecified &&
(fieldKeySelector.FieldContext == telemetrytypes.FieldContextAttribute ||
fieldKeySelector.FieldContext == telemetrytypes.FieldContextResource) {
// is not attribute, resource attribute or scope
switch fieldKeySelector.FieldContext {
case telemetrytypes.FieldContextAttribute, telemetrytypes.FieldContextResource, telemetrytypes.FieldContextScope:
fieldKeyConds = append(fieldKeyConds, sb.E("tagType", fieldKeySelector.FieldContext.TagType()))
}
@@ -283,41 +284,20 @@ func (t *telemetryMetaStore) getTracesKeys(ctx context.Context, fieldKeySelector
// hit the limit? (only counting DB results)
complete := rowCount <= limit
staticKeys := []string{"isRoot", "isEntryPoint"}
staticKeys = append(staticKeys, maps.Keys(tracestelemetryschema.IntrinsicFields)...)
staticKeys = append(staticKeys, maps.Keys(tracestelemetryschema.CalculatedFields)...)
// Add the matching static fields: the span scope selectors, the intrinsic
// columns and the calculated columns. These don't count towards the limit
staticFields := maps.Values(tracestelemetryschema.SpanSearchScopeFields)
staticFields = append(staticFields, maps.Values(tracestelemetryschema.IntrinsicFields)...)
staticFields = append(staticFields, maps.Values(tracestelemetryschema.CalculatedFields)...)
// Add matching intrinsic and matching calculated fields
// These don't count towards the limit
for _, key := range staticKeys {
found := false
for _, v := range searchTexts {
if v == "" || strings.Contains(key, v) {
found = true
break
}
for _, field := range staticFields {
if !staticFieldMatchesAny(field, fieldKeySelectors) {
continue
}
if found {
if field, exists := tracestelemetryschema.IntrinsicFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
continue
}
if field, exists := tracestelemetryschema.CalculatedFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
continue
}
keys = append(keys, &telemetrytypes.TelemetryFieldKey{
Name: key,
FieldContext: telemetrytypes.FieldContextSpan,
Signal: telemetrytypes.SignalTraces,
})
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; added {
continue
}
keys = append(keys, &field)
}
if err = t.updateColumnEvolutionMetadataForKeys(ctx, keys); err != nil {
@@ -537,12 +517,6 @@ func (t *telemetryMetaStore) getLogsKeys(ctx context.Context, orgID valuer.UUID,
allArgs = append(allArgs, args...)
}
if len(queries) == 0 {
// No matching contexts, return empty result
return []*telemetrytypes.TelemetryFieldKey{}, true, nil
}
// Combine queries with UNION ALL
var limit int
for _, fieldKeySelector := range fieldKeySelectors {
limit += fieldKeySelector.Limit
@@ -551,7 +525,15 @@ func (t *telemetryMetaStore) getLogsKeys(ctx context.Context, orgID valuer.UUID,
limit = 1000
}
mainQuery := fmt.Sprintf(`
keys := []*telemetrytypes.TelemetryFieldKey{}
parentTypes := make(map[string][]telemetrytypes.FieldDataType)
rowCount := 0
// the log and scope contexts have no keys table; they are served by the
// static fields appended below
if len(queries) > 0 {
// Combine queries with UNION ALL
mainQuery := fmt.Sprintf(`
SELECT tag_key, tag_type, tag_data_type, max(priority) as priority
FROM (
%s
@@ -561,103 +543,75 @@ func (t *telemetryMetaStore) getLogsKeys(ctx context.Context, orgID valuer.UUID,
LIMIT %d
`, strings.Join(queries, " UNION ALL "), limit+1)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, mainQuery, allArgs...)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetLogsKeys.Error())
}
defer rows.Close()
keys := []*telemetrytypes.TelemetryFieldKey{}
parentTypes := make(map[string][]telemetrytypes.FieldDataType)
rowCount := 0
searchTexts := []string{}
// Collect search texts for static field matching
for _, fieldKeySelector := range fieldKeySelectors {
searchTexts = append(searchTexts, fieldKeySelector.Name)
}
for rows.Next() {
rowCount++
// reached the limit, we know there are more results
if rowCount > limit {
break
}
var name string
var fieldContext telemetrytypes.FieldContext
var fieldDataType telemetrytypes.FieldDataType
var priority uint8
err = rows.Scan(&name, &fieldContext, &fieldDataType, &priority)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, mainQuery, allArgs...)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetLogsKeys.Error())
}
defer rows.Close()
// ArrayJSON/ArrayDynamic body rows for parent paths are needed by the JSON access plan
// builder (enrichJSONKeys). Always record them in parentTypes. Only skip adding to keys
// if the user did not also directly request this name — a field like "education" can be
// both a parent of "education[].name" and an explicitly queried field in its own right.
switch fieldDataType {
case telemetrytypes.FieldDataTypeArrayJSON, telemetrytypes.FieldDataTypeArrayDynamic:
if fieldContext == telemetrytypes.FieldContextBody && parentPaths[name] {
parentTypes[name] = append(parentTypes[name], fieldDataType)
if !mapOfRequestedSelectors[name] {
continue // skip; don't register the key.
for rows.Next() {
rowCount++
// reached the limit, we know there are more results
if rowCount > limit {
break
}
var name string
var fieldContext telemetrytypes.FieldContext
var fieldDataType telemetrytypes.FieldDataType
var priority uint8
err = rows.Scan(&name, &fieldContext, &fieldDataType, &priority)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetLogsKeys.Error())
}
// ArrayJSON/ArrayDynamic body rows for parent paths are needed by the JSON access plan
// builder (enrichJSONKeys). Always record them in parentTypes. Only skip adding to keys
// if the user did not also directly request this name — a field like "education" can be
// both a parent of "education[].name" and an explicitly queried field in its own right.
switch fieldDataType {
case telemetrytypes.FieldDataTypeArrayJSON, telemetrytypes.FieldDataTypeArrayDynamic:
if fieldContext == telemetrytypes.FieldContextBody && parentPaths[name] {
parentTypes[name] = append(parentTypes[name], fieldDataType)
if !mapOfRequestedSelectors[name] {
continue // skip; don't register the key.
}
}
}
}
key, ok := mapOfKeys[name+";"+fieldContext.StringValue()+";"+fieldDataType.StringValue()]
key, ok := mapOfKeys[name+";"+fieldContext.StringValue()+";"+fieldDataType.StringValue()]
// if there is no materialised column, create a key with the field context and data type
if !ok {
key = &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalLogs,
FieldContext: fieldContext,
FieldDataType: fieldDataType,
// if there is no materialised column, create a key with the field context and data type
if !ok {
key = &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalLogs,
FieldContext: fieldContext,
FieldDataType: fieldDataType,
}
}
keys = append(keys, key)
mapOfKeys[name+";"+fieldContext.StringValue()+";"+fieldDataType.StringValue()] = key
}
keys = append(keys, key)
mapOfKeys[name+";"+fieldContext.StringValue()+";"+fieldDataType.StringValue()] = key
}
if rows.Err() != nil {
return nil, false, errors.Wrap(rows.Err(), errors.TypeInternal, errors.CodeInternal, ErrFailedToGetLogsKeys.Error())
if rows.Err() != nil {
return nil, false, errors.Wrap(rows.Err(), errors.TypeInternal, errors.CodeInternal, ErrFailedToGetLogsKeys.Error())
}
}
// hit the limit? (only counting DB results)
complete := rowCount <= limit
staticKeys := []string{}
staticKeys = append(staticKeys, maps.Keys(logstelemetryschema.IntrinsicFields)...)
// Add matching intrinsic and matching calculated fields
// These don't count towards the limit
for _, key := range staticKeys {
found := false
for _, v := range searchTexts {
if v == "" || strings.Contains(key, v) {
found = true
break
}
// Add the matching intrinsic columns. These don't count towards the limit
for _, field := range maps.Values(logstelemetryschema.IntrinsicFields) {
if !staticFieldMatchesAny(field, fieldKeySelectors) {
continue
}
if found {
if field, exists := logstelemetryschema.IntrinsicFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
continue
}
keys = append(keys, &telemetrytypes.TelemetryFieldKey{
Name: key,
FieldContext: telemetrytypes.FieldContextLog,
Signal: telemetrytypes.SignalLogs,
})
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; added {
continue
}
keys = append(keys, &field)
}
// enrich body keys with promoted paths, indexes, and JSON access plans
@@ -801,10 +755,6 @@ func (t *telemetryMetaStore) getAuditKeys(ctx context.Context, fieldKeySelectors
allArgs = append(allArgs, args...)
}
if len(queries) == 0 {
return []*telemetrytypes.TelemetryFieldKey{}, true, nil
}
var limit int
for _, fieldKeySelector := range fieldKeySelectors {
limit += fieldKeySelector.Limit
@@ -813,7 +763,13 @@ func (t *telemetryMetaStore) getAuditKeys(ctx context.Context, fieldKeySelectors
limit = 1000
}
mainQuery := fmt.Sprintf(`
keys := []*telemetrytypes.TelemetryFieldKey{}
rowCount := 0
// the log and scope contexts have no keys table; they are served by the
// static fields appended below
if len(queries) > 0 {
mainQuery := fmt.Sprintf(`
SELECT tag_key, tag_type, tag_data_type, max(priority) as priority
FROM (
%s
@@ -823,73 +779,57 @@ func (t *telemetryMetaStore) getAuditKeys(ctx context.Context, fieldKeySelectors
LIMIT %d
`, strings.Join(queries, " UNION ALL "), limit+1)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, mainQuery, allArgs...)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
}
defer rows.Close()
keys := []*telemetrytypes.TelemetryFieldKey{}
rowCount := 0
searchTexts := []string{}
for _, fieldKeySelector := range fieldKeySelectors {
searchTexts = append(searchTexts, fieldKeySelector.Name)
}
for rows.Next() {
rowCount++
if rowCount > limit {
break
}
var name string
var fieldContext telemetrytypes.FieldContext
var fieldDataType telemetrytypes.FieldDataType
var priority uint8
err = rows.Scan(&name, &fieldContext, &fieldDataType, &priority)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, mainQuery, allArgs...)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
}
key, ok := mapOfKeys[name+";"+fieldContext.StringValue()+";"+fieldDataType.StringValue()]
defer rows.Close()
if !ok {
key = &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalLogs,
FieldContext: fieldContext,
FieldDataType: fieldDataType,
for rows.Next() {
rowCount++
if rowCount > limit {
break
}
var name string
var fieldContext telemetrytypes.FieldContext
var fieldDataType telemetrytypes.FieldDataType
var priority uint8
err = rows.Scan(&name, &fieldContext, &fieldDataType, &priority)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
}
key, ok := mapOfKeys[name+";"+fieldContext.StringValue()+";"+fieldDataType.StringValue()]
if !ok {
key = &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalLogs,
FieldContext: fieldContext,
FieldDataType: fieldDataType,
}
}
keys = append(keys, key)
mapOfKeys[name+";"+fieldContext.StringValue()+";"+fieldDataType.StringValue()] = key
}
keys = append(keys, key)
mapOfKeys[name+";"+fieldContext.StringValue()+";"+fieldDataType.StringValue()] = key
}
if rows.Err() != nil {
return nil, false, errors.Wrap(rows.Err(), errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
if rows.Err() != nil {
return nil, false, errors.Wrap(rows.Err(), errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
}
}
complete := rowCount <= limit
// Add intrinsic audit fields (same as logs intrinsics: body, severity_text, etc.)
staticKeys := maps.Keys(audittelemetryschema.IntrinsicFields)
for _, key := range staticKeys {
found := false
for _, v := range searchTexts {
if v == "" || strings.Contains(key, v) {
found = true
break
}
// Add the matching intrinsic audit fields (same as logs intrinsics: body, severity_text, etc.)
for _, field := range maps.Values(audittelemetryschema.IntrinsicFields) {
if !staticFieldMatchesAny(field, fieldKeySelectors) {
continue
}
if found {
if field, exists := audittelemetryschema.IntrinsicFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
}
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; added {
continue
}
keys = append(keys, &field)
}
return keys, complete, nil
@@ -1086,9 +1026,12 @@ func (t *telemetryMetaStore) getMeterSourceMetricKeys(ctx context.Context, field
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetMeterKeys.Error())
}
// meter labels are stored as strings in the labels JSON and have no
// attribute context, so only the data type is known
keys = append(keys, &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalMetrics,
Name: name,
Signal: telemetrytypes.SignalMetrics,
FieldDataType: telemetrytypes.FieldDataTypeString,
})
}
@@ -1357,25 +1300,23 @@ func (t *telemetryMetaStore) getRelatedValues(ctx context.Context, orgID valuer.
FieldDataType: fieldValueSelector.FieldDataType,
}
q := querybuilder.NewQueryInfo(ctx, orgID, nil, fieldValueSelector.Signal, nil, 0, 0)
selectRead, err := t.storage.Read(ctx, q, key)
selectColumn := selectRead.SQL
selectColumn, err := t.fm.FieldFor(ctx, orgID, 0, 0, key)
if err != nil {
// we don't have a explicit column to select from the related metadata table
// so we will select either from resource_attributes or attributes table
// in that order
resourceRead, _ := t.storage.Read(ctx, q, &telemetrytypes.TelemetryFieldKey{
resourceColumn, _ := t.fm.FieldFor(ctx, orgID, 0, 0, &telemetrytypes.TelemetryFieldKey{
Name: key.Name,
FieldContext: telemetrytypes.FieldContextResource,
FieldDataType: telemetrytypes.FieldDataTypeString,
})
attributeRead, _ := t.storage.Read(ctx, q, &telemetrytypes.TelemetryFieldKey{
attributeColumn, _ := t.fm.FieldFor(ctx, orgID, 0, 0, &telemetrytypes.TelemetryFieldKey{
Name: key.Name,
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeString,
})
selectColumn = fmt.Sprintf("if(notEmpty(%s), %s, %s)", resourceRead.SQL, resourceRead.SQL, attributeRead.SQL)
selectColumn = fmt.Sprintf("if(notEmpty(%s), %s, %s)", resourceColumn, resourceColumn, attributeColumn)
}
sb := sqlbuilder.Select("DISTINCT " + selectColumn).From(t.relatedMetadataDBName + "." + t.relatedMetadataTblName)
@@ -1391,11 +1332,11 @@ func (t *telemetryMetaStore) getRelatedValues(ctx context.Context, orgID valuer.
}
whereClause, err := querybuilder.PrepareWhereClause(fieldValueSelector.ExistingQuery, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Query: q,
Storage: t.storage,
Logger: t.logger,
FieldKeys: keys,
Context: ctx,
Logger: t.logger,
FieldMapper: t.fm,
ConditionBuilder: t.conditionBuilder,
FieldKeys: keys,
})
if err != nil {
t.logger.WarnContext(ctx, "error parsing existing query for related values", errors.Attr(err))
@@ -1426,20 +1367,20 @@ func (t *telemetryMetaStore) getRelatedValues(ctx context.Context, orgID valuer.
// search on attributes
key.FieldContext = telemetrytypes.FieldContextAttribute
attrConds, err := t.containsConditions(ctx, q, key, fieldValueSelector.Value, sb)
attrConds, _, err := t.conditionBuilder.ConditionFor(ctx, orgID, 0, 0, key, map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {key}}, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorContains, fieldValueSelector.Value, sb)
if err == nil {
conds = append(conds, attrConds...)
}
// search on resource
key.FieldContext = telemetrytypes.FieldContextResource
resourceConds, err := t.containsConditions(ctx, q, key, fieldValueSelector.Value, sb)
resourceConds, _, err := t.conditionBuilder.ConditionFor(ctx, orgID, 0, 0, key, map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {key}}, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorContains, fieldValueSelector.Value, sb)
if err == nil {
conds = append(conds, resourceConds...)
}
key.FieldContext = origContext
} else {
keyConds, err := t.containsConditions(ctx, q, key, fieldValueSelector.Value, sb)
keyConds, _, err := t.conditionBuilder.ConditionFor(ctx, orgID, 0, 0, key, map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {key}}, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorContains, fieldValueSelector.Value, sb)
if err == nil {
conds = append(conds, keyConds...)
}
@@ -1503,88 +1444,13 @@ func (t *telemetryMetaStore) getSpanFieldValues(ctx context.Context, fieldValueS
instrumentationtypes.CodeNamespace: "metadata",
instrumentationtypes.CodeFunctionName: "getSpanFieldValues",
})
// build the query to get the keys from the spans that match the field selection criteria
limit := fieldValueSelector.Limit
if limit == 0 {
limit = 50
if values, ok := spanSearchScopeFieldValues(fieldValueSelector.Name, fieldValueSelector.Value); ok {
return values, true, nil
}
sb := sqlbuilder.Select("DISTINCT string_value, number_value").From(t.tracesDBName + "." + t.tracesFieldsTblName)
if fieldValueSelector.Name != "" {
sb.Where(sb.E("tag_key", fieldValueSelector.Name))
}
// now look at the field context
if fieldValueSelector.FieldContext != telemetrytypes.FieldContextUnspecified {
sb.Where(sb.E("tag_type", fieldValueSelector.FieldContext.TagType()))
}
// now look at the field data type
if fieldValueSelector.FieldDataType != telemetrytypes.FieldDataTypeUnspecified {
sb.Where(sb.E("tag_data_type", fieldValueSelector.FieldDataType.TagDataType()))
}
if fieldValueSelector.Value != "" {
switch fieldValueSelector.FieldDataType {
case telemetrytypes.FieldDataTypeString:
sb.Where(sb.ILike("string_value", "%"+escapeForLike(fieldValueSelector.Value)+"%"))
case telemetrytypes.FieldDataTypeNumber:
sb.Where(sb.IsNotNull("number_value"))
sb.Where(sb.ILike("toString(number_value)", "%"+escapeForLike(fieldValueSelector.Value)+"%"))
case telemetrytypes.FieldDataTypeUnspecified:
// or b/w string and number
sb.Where(sb.Or(
sb.ILike("string_value", "%"+escapeForLike(fieldValueSelector.Value)+"%"),
sb.ILike("toString(number_value)", "%"+escapeForLike(fieldValueSelector.Value)+"%"),
))
}
}
// query one extra to check if we hit the limit
sb.Limit(limit + 1)
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, query, args...)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetLogsKeys.Error())
}
defer rows.Close()
values := &telemetrytypes.TelemetryFieldValues{}
seen := make(map[string]bool)
rowCount := 0
totalCount := 0 // Track total unique values
for rows.Next() {
rowCount++
var stringValue string
var numberValue float64
if err := rows.Scan(&stringValue, &numberValue); err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetLogsKeys.Error())
}
// Only add values if we haven't hit the limit yet
if totalCount < limit {
if _, ok := seen[stringValue]; !ok && stringValue != "" {
values.StringValues = append(values.StringValues, stringValue)
seen[stringValue] = true
totalCount++
}
if _, ok := seen[fmt.Sprintf("%f", numberValue)]; !ok && numberValue != 0 && totalCount < limit {
values.NumberValues = append(values.NumberValues, numberValue)
seen[fmt.Sprintf("%f", numberValue)] = true
totalCount++
}
}
}
// hit the limit?
complete := rowCount <= limit
return values, complete, nil
knownBool := isKnownBoolField(fieldValueSelector, tracestelemetryschema.IntrinsicFields, tracestelemetryschema.CalculatedFields)
// unix_milli is the hour of the span start
return t.getTagTableValues(ctx, t.tracesDBName+"."+t.tracesFieldsTblName, fieldValueSelector, knownBool)
}
func (t *telemetryMetaStore) getLogFieldValues(ctx context.Context, fieldValueSelector *telemetrytypes.FieldValueSelector) (*telemetrytypes.TelemetryFieldValues, bool, error) {
@@ -1593,17 +1459,77 @@ func (t *telemetryMetaStore) getLogFieldValues(ctx context.Context, fieldValueSe
instrumentationtypes.CodeNamespace: "metadata",
instrumentationtypes.CodeFunctionName: "getLogFieldValues",
})
// build the query to get the keys from the spans that match the field selection criteria
knownBool := isKnownBoolField(fieldValueSelector, logstelemetryschema.IntrinsicFields)
// unix_milli is the hour the log was ingested, not the log's own timestamp
return t.getTagTableValues(ctx, t.logsDBName+"."+t.logsFieldsTblName, fieldValueSelector, knownBool)
}
// tagTableSinceDay restricts rows to the tag table's day partitions from the
// start's day on. Partitions are toDate(unix_milli / 1000) in the server's
// timezone, and a value's surviving row within a day carries whichever hour
// was inserted last, so the day is the finest safe unit.
func tagTableSinceDay(sb *sqlbuilder.SelectBuilder, startUnixMilli int64) {
if startUnixMilli != 0 {
sb.Where(fmt.Sprintf("toDate(unix_milli / 1000) >= toDate(%d)", startUnixMilli/1000))
}
}
// tagTableHasBoolRows reports whether the tag table holds a bool row for the
// key. Bool rows carry no value, so one row is enough to know the key takes
// the values true and false.
func (t *telemetryMetaStore) tagTableHasBoolRows(ctx context.Context, table string, selector *telemetrytypes.FieldValueSelector) (bool, error) {
sb := sqlbuilder.Select("1").From(table)
sb.Where(sb.E("tag_key", selector.Name))
sb.Where(sb.E("tag_data_type", telemetrytypes.FieldDataTypeBool.TagDataType()))
if selector.FieldContext != telemetrytypes.FieldContextUnspecified {
sb.Where(sb.E("tag_type", selector.FieldContext.TagType()))
}
tagTableSinceDay(sb, selector.StartUnixMilli)
sb.Limit(1)
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, query, args...)
if err != nil {
return false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetLogsKeys.Error())
}
defer rows.Close()
return rows.Next(), rows.Err()
}
// getTagTableValues returns the string and number values of the key from a
// tag table, and true and false when the key is a known bool field or the
// table holds bool rows for it. Bool rows do not count towards the limit.
func (t *telemetryMetaStore) getTagTableValues(ctx context.Context, table string, fieldValueSelector *telemetrytypes.FieldValueSelector, knownBool bool) (*telemetrytypes.TelemetryFieldValues, bool, error) {
limit := fieldValueSelector.Limit
if limit == 0 {
limit = 50
}
sb := sqlbuilder.Select("DISTINCT string_value, number_value").From(t.logsDBName + "." + t.logsFieldsTblName)
values := &telemetrytypes.TelemetryFieldValues{}
if knownBool {
values.BoolValues = boolFieldValues(fieldValueSelector.Value).BoolValues
if fieldValueSelector.FieldDataType == telemetrytypes.FieldDataTypeBool {
return values, true, nil
}
} else if fieldValueSelector.FieldDataType == telemetrytypes.FieldDataTypeUnspecified {
hasBoolRows, err := t.tagTableHasBoolRows(ctx, table, fieldValueSelector)
if err != nil {
return nil, false, err
}
if hasBoolRows {
values.BoolValues = boolFieldValues(fieldValueSelector.Value).BoolValues
}
}
sb := sqlbuilder.Select("DISTINCT string_value, number_value").From(table)
if fieldValueSelector.Name != "" {
sb.Where(sb.E("tag_key", fieldValueSelector.Name))
}
sb.Where(sb.NE("tag_data_type", telemetrytypes.FieldDataTypeBool.TagDataType()))
tagTableSinceDay(sb, fieldValueSelector.StartUnixMilli)
if fieldValueSelector.FieldContext != telemetrytypes.FieldContextUnspecified {
sb.Where(sb.E("tag_type", fieldValueSelector.FieldContext.TagType()))
@@ -1640,7 +1566,6 @@ func (t *telemetryMetaStore) getLogFieldValues(ctx context.Context, fieldValueSe
}
defer rows.Close()
values := &telemetrytypes.TelemetryFieldValues{}
seen := make(map[string]bool)
rowCount := 0
totalCount := 0 // Track total unique values
@@ -2094,6 +2019,18 @@ func populateAllUnspecifiedValues(allUnspecifiedValues *telemetrytypes.Telemetry
}
}
for _, value := range values.BoolValues {
if totalCount >= limit {
complete = false
break
}
if _, ok := mapOfValues[value]; !ok {
mapOfValues[value] = true
allUnspecifiedValues.BoolValues = append(allUnspecifiedValues.BoolValues, value)
totalCount++
}
}
for _, value := range values.RelatedValues {
if totalCount >= limit {
complete = false
@@ -2464,6 +2401,10 @@ func (k *telemetryMetaStore) fetchEvolutionEntryFromClickHouse(ctx context.Conte
// updateColumnEvolutionMetadataForKeys updates the evolution field for keys.
func (k *telemetryMetaStore) updateColumnEvolutionMetadataForKeys(ctx context.Context, keysToUpdate []*telemetrytypes.TelemetryFieldKey) error {
// an empty selector list would run the evolution query without a filter
if len(keysToUpdate) == 0 {
return nil
}
var metadataKeySelectors []*telemetrytypes.EvolutionSelector
for _, keySelector := range keysToUpdate {
@@ -2651,11 +2592,3 @@ func (t *telemetryMetaStore) fetchLastSeenInfoForTable(ctx context.Context, tabl
}
return lastSeenInfo, nil
}
// containsConditions compiles a contains search on one key of the related
// values table; the key is its own metadata.
func (t *telemetryMetaStore) containsConditions(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, value string, sb *sqlbuilder.SelectBuilder) ([]string, error) {
fieldKeys := map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {key}}
conds, _, err := querybuilder.Conditions(ctx, q, t.storage, key, qbtypes.FilterOperatorContains, value, fieldKeys, false, sb)
return conds, err
}

View File

@@ -0,0 +1,53 @@
package telemetrymetadata
import (
"strings"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
func staticFieldMatchesAny(field telemetrytypes.TelemetryFieldKey, selectors []*telemetrytypes.FieldKeySelector) bool {
for _, selector := range selectors {
if staticFieldMatches(field, selector) {
return true
}
}
return false
}
// staticFieldMatches mirrors the keys-table lookup for a static field: the
// requested context and data type, when given, must agree with the field's,
// and the name matches case-insensitively, as a substring for fuzzy selectors
// and as the whole name for exact ones.
func staticFieldMatches(field telemetrytypes.TelemetryFieldKey, selector *telemetrytypes.FieldKeySelector) bool {
if selector.FieldContext != telemetrytypes.FieldContextUnspecified && selector.FieldContext != field.FieldContext {
return false
}
if selector.FieldDataType != telemetrytypes.FieldDataTypeUnspecified && !sameDataTypeFamily(selector.FieldDataType, field.FieldDataType) {
return false
}
if selector.Name == "" {
return true
}
if selector.SelectorMatchType == telemetrytypes.FieldSelectorMatchTypeExact {
return strings.EqualFold(field.Name, selector.Name)
}
return strings.Contains(strings.ToLower(field.Name), strings.ToLower(selector.Name))
}
// sameDataTypeFamily treats the numeric types as one family: static fields
// declare "number" while callers may ask for int64 or float64.
func sameDataTypeFamily(requested, actual telemetrytypes.FieldDataType) bool {
if requested == actual {
return true
}
return isNumericDataType(requested) && isNumericDataType(actual)
}
func isNumericDataType(dataType telemetrytypes.FieldDataType) bool {
switch dataType {
case telemetrytypes.FieldDataTypeNumber, telemetrytypes.FieldDataTypeInt64, telemetrytypes.FieldDataTypeFloat64:
return true
}
return false
}

View File

@@ -0,0 +1,215 @@
package audittelemetryschema
import (
"context"
"fmt"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
type conditionBuilder struct {
fm qbtypes.FieldMapper
}
func NewConditionBuilder(fm qbtypes.FieldMapper) *conditionBuilder {
return &conditionBuilder{fm: fm}
}
func (c *conditionBuilder) conditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs, endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
if operator.IsStringSearchOperator() {
value = querybuilder.FormatValueForContains(value)
}
fieldExpression, err := c.fm.FieldFor(ctx, orgID, startNs, endNs, key)
if err != nil {
return "", err
}
fieldExpression, value = querybuilder.DataTypeCollisionHandledFieldName(key, value, fieldExpression, operator)
switch operator {
case qbtypes.FilterOperatorEqual:
return sb.E(fieldExpression, value), nil
case qbtypes.FilterOperatorNotEqual:
return sb.NE(fieldExpression, value), nil
case qbtypes.FilterOperatorGreaterThan:
return sb.G(fieldExpression, value), nil
case qbtypes.FilterOperatorGreaterThanOrEq:
return sb.GE(fieldExpression, value), nil
case qbtypes.FilterOperatorLessThan:
return sb.LT(fieldExpression, value), nil
case qbtypes.FilterOperatorLessThanOrEq:
return sb.LE(fieldExpression, value), nil
case qbtypes.FilterOperatorLike:
return sb.Like(fieldExpression, value), nil
case qbtypes.FilterOperatorNotLike:
return sb.NotLike(fieldExpression, value), nil
case qbtypes.FilterOperatorILike:
return sb.ILike(fieldExpression, value), nil
case qbtypes.FilterOperatorNotILike:
return sb.NotILike(fieldExpression, value), nil
case qbtypes.FilterOperatorContains:
return sb.ILike(fieldExpression, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorNotContains:
return sb.NotILike(fieldExpression, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorRegexp:
return fmt.Sprintf(`match(%s, %s)`, sqlbuilder.Escape(fieldExpression), sb.Var(value)), nil
case qbtypes.FilterOperatorNotRegexp:
return fmt.Sprintf(`NOT match(%s, %s)`, sqlbuilder.Escape(fieldExpression), sb.Var(value)), nil
case qbtypes.FilterOperatorBetween:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrBetweenValues
}
if len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.Between(fieldExpression, values[0], values[1]), nil
case qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrBetweenValues
}
if len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.NotBetween(fieldExpression, values[0], values[1]), nil
case qbtypes.FilterOperatorIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
conditions := []string{}
for _, value := range values {
cond, err := c.conditionFor(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorEqual, value, sb)
if err != nil {
return "", err
}
conditions = append(conditions, cond)
}
return sb.Or(conditions...), nil
case qbtypes.FilterOperatorNotIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
conditions := []string{}
for _, value := range values {
cond, err := c.conditionFor(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorNotEqual, value, sb)
if err != nil {
return "", err
}
conditions = append(conditions, cond)
}
return sb.And(conditions...), nil
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
columns, err := c.fm.ColumnFor(ctx, orgID, startNs, endNs, key)
if err != nil {
return "", err
}
pred, err := querybuilder.ExistsExpression(columns, key, startNs, endNs, fieldExpression, operator == qbtypes.FilterOperatorExists)
if err != nil {
return "", err
}
return sqlbuilder.Escape(pred), nil
}
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported operator: %v", operator)
}
func (c *conditionBuilder) ConditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
options qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for audit.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}
// Audit fields have no family support, so every logical field is
// single-member and flattens losslessly to its physical key.
resolved, warning := querybuilder.ResolveLogicalFields(key, querybuilder.MatchingLogicalFields(ctx, orgID, nil, key, fieldKeys))
keys := querybuilder.SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
if len(keys) == 0 {
return nil, warnings, querybuilder.NewKeyNotFoundError(key.Name)
}
// Drop resource keys the sub-query already covers; if none remain, skip the term (not an error).
if options.SkipResourceFilter {
filtered := make([]*telemetrytypes.TelemetryFieldKey, 0, len(keys))
for _, k := range keys {
if k.FieldContext != telemetrytypes.FieldContextResource {
filtered = append(filtered, k)
}
}
if len(filtered) == 0 {
return nil, warnings, nil
}
keys = filtered
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
cond, err := c.conditionForKey(ctx, orgID, startNs, endNs, k, operator, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
}
return conds, warnings, nil
}
func (c *conditionBuilder) conditionForKey(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
condition, err := c.conditionFor(ctx, orgID, startNs, endNs, key, operator, value, sb)
if err != nil {
return "", err
}
if key.FieldContext == telemetrytypes.FieldContextLog || key.FieldContext == telemetrytypes.FieldContextScope {
return condition, nil
}
if operator.AddDefaultExistsFilter() {
existsCondition, err := c.conditionFor(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorExists, nil, sb)
if err != nil {
return "", err
}
return sb.And(condition, existsCondition), nil
}
return condition, nil
}

View File

@@ -9,18 +9,19 @@ import (
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
"golang.org/x/exp/maps"
)
type storage struct{}
type fieldMapper struct{}
var _ qbtypes.Storage = (*storage)(nil)
func NewStorage() qbtypes.Storage {
return &storage{}
func NewFieldMapper() qbtypes.FieldMapper {
return &fieldMapper{}
}
func (m *storage) getColumn(_ context.Context, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
func (m *fieldMapper) getColumn(_ context.Context, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
switch key.FieldContext {
case telemetrytypes.FieldContextResource:
return []*schema.Column{auditLogColumns["resource"]}, nil
@@ -52,7 +53,7 @@ func (m *storage) getColumn(_ context.Context, key *telemetrytypes.TelemetryFiel
return nil, qbtypes.ErrColumnNotFound
}
func (m *storage) read(ctx context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
func (m *fieldMapper) FieldFor(ctx context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, key)
if err != nil {
return "", err
@@ -92,58 +93,74 @@ func (m *storage) read(ctx context.Context, _ qbtypes.QueryInfo, key *telemetryt
return column.Name, nil
}
// Read composes the bare read of one key with its membership test and what
// an absent row reads.
func (m *storage) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (qbtypes.Read, error) {
func (m *fieldMapper) ColumnFor(ctx context.Context, _ valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
return m.getColumn(ctx, key)
}
// ExistsFor implements the per-key existence primitive of qbtypes.FieldMapper.
func (m *fieldMapper) ExistsFor(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey, exists bool) (string, error) {
fieldExpression, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
return "", err
}
columns, err := m.getColumn(ctx, key)
if err != nil {
return qbtypes.Read{}, err
return "", err
}
sql, err := m.read(ctx, q, key)
if err != nil {
return qbtypes.Read{}, err
}
presence, err := querybuilder.ExistsExpression(columns, key, q.StartNs, q.EndNs, sql, true)
if err != nil {
return qbtypes.Read{}, err
}
absence, err := querybuilder.ExistsExpression(columns, key, q.StartNs, q.EndNs, sql, false)
if err != nil {
return qbtypes.Read{}, err
}
return qbtypes.Read{SQL: sql, Presence: presence, Absence: absence, WhenAbsent: absentReads(columns[0])}, nil
return querybuilder.ExistsExpression(columns, key, tsStart, tsEnd, fieldExpression, exists)
}
// absentReads tells what a row without the key reads from the column: the
// empty value of a map or of a JSON path (its ::String cast folds NULL to
// the empty string), and a real value from every other column.
func absentReads(column *schema.Column) qbtypes.Absent {
switch column.Type.GetType() {
case schema.ColumnTypeEnumMap, schema.ColumnTypeEnumJSON:
return qbtypes.AbsentIsSentinel
func (m *fieldMapper) ColumnExpressionFor(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
field *telemetrytypes.TelemetryFieldKey,
requiredDataType telemetrytypes.FieldDataType,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
resolved := field
fieldExpression, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, field)
if errors.Is(err, qbtypes.ErrColumnNotFound) {
keysForField := keys[field.Name]
if len(keysForField) == 0 {
if _, ok := auditLogColumns[field.Name]; ok {
field.FieldContext = telemetrytypes.FieldContextLog
fieldExpression, _ = m.FieldFor(ctx, orgID, tsStart, tsEnd, field)
} else {
wrappedErr := errors.Wrapf(err, errors.TypeInvalidInput, errors.CodeInvalidInput, "field `%s` not found", field.Name).WithSuggestions(errors.NewSuggestionsOnLevenshteinDistance(field.Name, errors.NounKeys, maps.Keys(keys))...)
return "", wrappedErr
}
} else {
resolved = keysForField[0]
fieldExpression, _ = m.FieldFor(ctx, orgID, tsStart, tsEnd, keysForField[0])
}
}
return qbtypes.AlwaysPresent
// Group-by/order (String) and aggregation (String/Float64): exists-guarded and coerced
// to requiredDataType, returned bare (the caller adds any alias). Raw select
// (Unspecified) returns the aliased column expression.
if requiredDataType != telemetrytypes.FieldDataTypeUnspecified {
var dummyValue any = ""
if requiredDataType == telemetrytypes.FieldDataTypeFloat64 {
dummyValue = 0.0
}
columns, err := m.getColumn(ctx, resolved)
if err != nil {
return "", err
}
guard, err := querybuilder.ExistsExpression(columns, resolved, tsStart, tsEnd, fieldExpression, true)
if err != nil {
return "", err
}
coerced, _ := querybuilder.DataTypeCollisionHandledFieldName(resolved, dummyValue, fieldExpression, qbtypes.FilterOperatorUnknown)
return fmt.Sprintf("multiIf(%s, %s, NULL)", guard, coerced), nil
}
return fmt.Sprintf("%s AS `%s`", sqlbuilder.Escape(fieldExpression), field.Name), nil
}
// Fallback answers a column name with the column; audit has no attribute
// synthesis, so any other unknown key stays unresolved and the caller errors.
func (m *storage) Fallback(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, _ qbtypes.FilterOperator, _ any) ([]*telemetrytypes.LogicalField, error) {
schemaColumn, ok := auditLogColumns[key.Name]
if !ok {
return nil, nil
}
dataType := key.FieldDataType
if dataType == telemetrytypes.FieldDataTypeUnspecified {
dataType = querybuilder.ColumnDataType(schemaColumn)
}
column := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextLog, dataType)
return querybuilder.WrapAsLogicalFields(key.Name, []*telemetrytypes.TelemetryFieldKey{column}), nil
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{Split: qbtypes.MainOfSplit}
}
func (m *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
return querybuilder.SharedCondition(ctx, q, m, logical, operator, value, sb)
// CandidateKeys returns nil: audit has no synthesize-on-unknown-key fallback, so an
// unknown key stays unresolved and the caller errors.
func (m *fieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, _ *telemetrytypes.TelemetryFieldKey, _ any, _ map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
return nil
}

View File

@@ -7,26 +7,41 @@ import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
type conditionBuilder struct {
fm qbtypes.FieldMapper
fl flagger.Flagger
}
var _ qbtypes.ConditionBuilder = (*conditionBuilder)(nil)
func NewConditionBuilder(fm qbtypes.FieldMapper, fl flagger.Flagger) *conditionBuilder {
return &conditionBuilder{fm: fm, fl: fl}
}
// conditionForSearch ORs a case-insensitive match of the search term across the key
// context's searchable columns (unspecified context = every column).
func (s *storage) conditionForSearch(
q qbtypes.QueryInfo,
func (c *conditionBuilder) conditionForSearch(
ctx context.Context,
orgID valuer.UUID,
key *telemetrytypes.TelemetryFieldKey,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, error) {
) ([]string, []string, error) {
// QuoteMeta + LOWER on both sides, not (?i): a literal match that can still use the
// LOWER(toString(body_v2)) skip index.
term := regexp.QuoteMeta(fmt.Sprintf("%v", value))
useJSONBody := q.BodyJSONOn
useJSONBody := c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
var conditions []string
@@ -48,15 +63,15 @@ func (s *storage) conditionForSearch(
case schema.ColumnTypeEnumString, schema.ColumnTypeEnumLowCardinality:
conditions = append(conditions, fmt.Sprintf("match(LOWER(%s), LOWER(%s))", col.Name, sb.Var(term)))
default:
return nil, errors.NewInternalf(errors.CodeInternal, "search does not support the column type of %q", col.Name)
return nil, nil, errors.NewInternalf(errors.CodeInternal, "search does not support the column type of %q", col.Name)
}
}
if len(conditions) == 0 {
return nil, nil
return nil, nil, nil
}
// The advisory rides on CostGuard (set by the visitor), not warnings.
return []string{sb.Or(conditions...)}, nil
return []string{sb.Or(conditions...)}, nil, nil
}
// isBodyJSONSearch reports whether a key addresses a path within the body JSON. Only
@@ -76,8 +91,9 @@ func isBodyJSONSearch(key *telemetrytypes.TelemetryFieldKey, columns []*schema.C
// conditionForArrayFunction builds has/hasAny/hasAll over a body JSON path — via the JSON
// access plan (flag on) or legacy typed extraction (flag off).
func (s *storage) conditionForArrayFunction(
q qbtypes.QueryInfo,
func (c *conditionBuilder) conditionForArrayFunction(
ctx context.Context,
orgID valuer.UUID,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
@@ -94,7 +110,7 @@ func (s *storage) conditionForArrayFunction(
needle = args[0]
}
if q.BodyJSONOn {
if c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
// JSON access plan: data-type collision handling, nested array paths.
valueType, needle := InferDataType(needle, operator, key)
// A not-found (synthesized) body path carries no metadata plan; build an exhaustive
@@ -168,8 +184,9 @@ func firstTokenSeparator(s string) (string, bool) {
// conditionForHasToken builds a hasToken full-text search over the body column, resolving the
// column from the key name + use_json_body flag.
func (s *storage) conditionForHasToken(
q qbtypes.QueryInfo,
func (c *conditionBuilder) conditionForHasToken(
ctx context.Context,
orgID valuer.UUID,
key *telemetrytypes.TelemetryFieldKey,
value any,
sb *sqlbuilder.SelectBuilder,
@@ -194,7 +211,7 @@ func (s *storage) conditionForHasToken(
needleStr, sep, needleStr).WithUrl(hasTokenFunctionDocURL)
}
bodyJSONEnabled := q.BodyJSONOn
bodyJSONEnabled := c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
if !bodyJSONEnabled {
// legacy: token search over the plain body string column only.
@@ -229,9 +246,10 @@ func (s *storage) conditionForHasToken(
"function `hasToken` only supports the body field or a body JSON string field as first parameter").WithUrl(hasTokenFunctionDocURL)
}
func (s *storage) conditionForResolvedKey(
func (c *conditionBuilder) conditionForResolvedKey(
ctx context.Context,
q qbtypes.QueryInfo,
orgID valuer.UUID,
startNs, endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
@@ -239,13 +257,13 @@ func (s *storage) conditionForResolvedKey(
) (string, error) {
// hasToken resolves from the key name + flag alone (no column resolution), so handle it first.
if operator == qbtypes.FilterOperatorHasToken {
return s.conditionForHasToken(q, key, value, sb)
return c.conditionForHasToken(ctx, orgID, key, value, sb)
}
columns, err := s.getColumn(q, key)
columns, err := c.fm.ColumnFor(ctx, orgID, startNs, endNs, key)
if errors.Is(err, qbtypes.ErrColumnNotFound) && key.FieldContext == telemetrytypes.FieldContextUnspecified {
key = telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextBody, key.FieldDataType)
columns, err = s.getColumn(q, key)
columns, err = c.fm.ColumnFor(ctx, orgID, startNs, endNs, key)
}
if err != nil {
return "", err
@@ -253,12 +271,12 @@ func (s *storage) conditionForResolvedKey(
// has/hasAny/hasAll take the body-JSON path, not the normal operator paths.
if operator.IsArrayFunctionOperator() {
return s.conditionForArrayFunction(q, key, operator, value, columns, sb)
return c.conditionForArrayFunction(ctx, orgID, key, operator, value, columns, sb)
}
// TODO(Piyush): Update this to support multiple JSON columns based on evolutions
for _, column := range columns {
if column.Type.GetType() == schema.ColumnTypeEnumJSON && isBodyJSONSearch(key, columns) && q.BodyJSONOn && key.Name != messageSubField {
if column.Type.GetType() == schema.ColumnTypeEnumJSON && isBodyJSONSearch(key, columns) && c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) && key.Name != messageSubField {
valueType, value := InferDataType(value, operator, key)
if len(key.JSONPlan) == 0 {
keyCopy := telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, key.FieldDataType)
@@ -281,13 +299,13 @@ func (s *storage) conditionForResolvedKey(
value = querybuilder.FormatValueForContains(value)
}
fieldExpression, err := s.read(ctx, q, key)
fieldExpression, err := c.fm.FieldFor(ctx, orgID, startNs, endNs, key)
if err != nil {
return "", err
}
// Check if this is a body JSON search (legacy string-body path, JSON flag off).
if isBodyJSONSearch(key, columns) && !q.BodyJSONOn {
if isBodyJSONSearch(key, columns) && !c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
fieldExpression, value = GetBodyJSONKey(ctx, key, operator, value)
}
@@ -338,13 +356,13 @@ func (s *storage) conditionForResolvedKey(
return sb.NotILike(fieldExpression, value), nil
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
if isBodyJSONSearch(key, columns) && !q.BodyJSONOn {
if isBodyJSONSearch(key, columns) && !c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
if operator == qbtypes.FilterOperatorExists {
return GetBodyJSONKeyForExists(ctx, key, operator, value), nil
}
return "NOT " + GetBodyJSONKeyForExists(ctx, key, operator, value), nil
}
pred, err := querybuilder.ExistsExpression(columns, key, q.StartNs, q.EndNs, fieldExpression, operator == qbtypes.FilterOperatorExists)
pred, err := querybuilder.ExistsExpression(columns, key, startNs, endNs, fieldExpression, operator == qbtypes.FilterOperatorExists)
if err != nil {
return "", err
}
@@ -392,7 +410,7 @@ func (s *storage) conditionForResolvedKey(
// instead of using IN, we use `=` + `OR` to make use of index
conditions := []string{}
for _, value := range values {
cond, err := s.conditionForResolvedKey(ctx, q, key, qbtypes.FilterOperatorEqual, value, sb)
cond, err := c.conditionForResolvedKey(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorEqual, value, sb)
if err != nil {
return "", err
}
@@ -407,7 +425,7 @@ func (s *storage) conditionForResolvedKey(
// instead of using NOT IN, we use `!=` + `AND` to make use of index
conditions := []string{}
for _, value := range values {
cond, err := s.conditionForResolvedKey(ctx, q, key, qbtypes.FilterOperatorNotEqual, value, sb)
cond, err := c.conditionForResolvedKey(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorNotEqual, value, sb)
if err != nil {
return "", err
}
@@ -419,29 +437,132 @@ func (s *storage) conditionForResolvedKey(
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported operator: %v", operator)
}
// candidateLookupKeys returns the metadata map only for fold-contexts, where CandidateKeys
// would otherwise fold the prefix into the key name. Handing it the map lets a same-named
// key under another context resolve first (as ColumnExpressionFor does). Strict contexts
// (resource/attribute/scope) get nil so their explicit context is always honored.
func candidateLookupKeys(key *telemetrytypes.TelemetryFieldKey, fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey) map[string][]*telemetrytypes.TelemetryFieldKey {
if key.FieldContext == telemetrytypes.FieldContextLog {
return fieldKeys
}
return nil
}
func (c *conditionBuilder) ConditionFor(
ctx context.Context,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
fieldKeys map[string][]*telemetrytypes.TelemetryFieldKey,
options qbtypes.ConditionBuilderOptions,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
matches := querybuilder.MatchingLogicalFields(ctx, orgID, nil, key, fieldKeys)
skipResourceFilter := options.SkipResourceFilter
// search() resolves its own (optional) scope; handle it before key resolution.
if operator == qbtypes.FilterOperatorSearch {
return c.conditionForSearch(ctx, orgID, key, value, sb)
}
// Logs fields have no family support yet, so every logical field is
// single-member and flattens losslessly to its physical key.
resolved, warning := querybuilder.ResolveLogicalFields(key, matches)
keys := querybuilder.SingleKeys(resolved)
var warnings []string
if warning != "" {
warnings = append(warnings, warning)
}
synthesized := false
if len(keys) == 0 {
_, isIntrinsicColumn := logsV2Columns[key.Name]
switch {
case key.FieldContext == telemetrytypes.FieldContextBody && key.Name == "":
return nil, warnings, errors.NewInvalidInputf(errors.CodeInvalidInput, "missing key for body json search - expected key of the form `body.key` (ex: `body.status`)")
case key.FieldContext == telemetrytypes.FieldContextLog && isIntrinsicColumn:
keys = []*telemetrytypes.TelemetryFieldKey{key}
default:
// Fold-contexts get the metadata map so a same-named key under another context
// wins before the prefix folds into the key name (matching ColumnExpressionFor);
// strict contexts pass nil and stay honored as-is.
keys = c.fm.CandidateKeys(ctx, orgID, key, value, candidateLookupKeys(key, fieldKeys))
if operator.IsFunctionOperator() {
if key.FieldContext != telemetrytypes.FieldContextBody {
// has/hasAny/hasAll/hasToken are body-JSON only
return nil, warnings, querybuilder.NewFunctionUnsupportedError(operator)
}
bodyKeys := make([]*telemetrytypes.TelemetryFieldKey, 0, len(keys))
for _, k := range keys {
if k.FieldContext == telemetrytypes.FieldContextBody {
bodyKeys = append(bodyKeys, k)
}
}
keys = bodyKeys
}
if len(keys) == 0 {
return nil, warnings, querybuilder.NewKeyNotFoundError(key.Name)
}
synthesized = true
warnings = append(warnings, querybuilder.NewKeyNotFoundWarning(key.Name))
}
}
if skipResourceFilter && !synthesized {
filtered := make([]*telemetrytypes.TelemetryFieldKey, 0, len(keys))
for _, k := range keys {
if k.FieldContext != telemetrytypes.FieldContextResource {
filtered = append(filtered, k)
}
}
if len(filtered) == 0 {
return nil, warnings, nil
}
keys = filtered
}
conds := make([]string, 0, len(keys))
for _, k := range keys {
cond, err := c.conditionForKey(ctx, orgID, startNs, endNs, k, operator, value, sb)
if err != nil {
return nil, nil, err
}
conds = append(conds, cond)
if w := c.bodyFullTextDefaultWarning(ctx, orgID, startNs, endNs, k, operator); w != "" {
warnings = append(warnings, w)
}
}
return conds, warnings, nil
}
// bodyFullTextDefaultWarning returns the advisory shown when a regexp full-text
// search on `body` resolves to the body.message sub-field (JSON mode), else "". This
// keeps the JSON-vs-legacy decision in the builder rather than the filter visitor.
func (s *storage) bodyFullTextDefaultWarning(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, operator qbtypes.FilterOperator) string {
func (c *conditionBuilder) bodyFullTextDefaultWarning(ctx context.Context, orgID valuer.UUID, startNs, endNs uint64, key *telemetrytypes.TelemetryFieldKey, operator qbtypes.FilterOperator) string {
if operator != qbtypes.FilterOperatorRegexp || key.Name != LogsV2BodyColumn {
return ""
}
if field, err := s.read(ctx, q, key); err == nil && field == messageSubColumn {
if field, err := c.fm.FieldFor(ctx, orgID, startNs, endNs, key); err == nil && field == messageSubColumn {
return querybuilder.BodyFullTextSearchDefaultWarning
}
return ""
}
func (s *storage) conditionForKey(
func (c *conditionBuilder) conditionForKey(
ctx context.Context,
q qbtypes.QueryInfo,
orgID valuer.UUID,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
condition, err := s.conditionForResolvedKey(ctx, q, key, operator, value, sb)
condition, err := c.conditionForResolvedKey(ctx, orgID, startNs, endNs, key, operator, value, sb)
if err != nil {
return "", err
}
@@ -449,28 +570,22 @@ func (s *storage) conditionForKey(
// Skip adding exists filter for intrinsic fields i.e. Table level log context fields
buildExistCondition := operator.AddDefaultExistsFilter()
switch key.FieldContext {
case telemetrytypes.FieldContextLog:
case telemetrytypes.FieldContextLog, telemetrytypes.FieldContextScope:
// pass; No need to build exist condition for top level columns
// immediately return
return condition, nil
case telemetrytypes.FieldContextScope:
// scope_name and scope_version are columns; a scope attribute lives in
// a map and follows the keyless contract like an attribute
if !s.mapBacked(q, key) {
return condition, nil
}
case telemetrytypes.FieldContextResource, telemetrytypes.FieldContextAttribute:
// build exist condition for resource and attribute fields based on filter operator
case telemetrytypes.FieldContextBody:
// Querying JSON fields already account for Nullability of fields
// so additional exists checks are not needed
if q.BodyJSONOn {
if c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
return condition, nil
}
}
if buildExistCondition {
existsCondition, err := s.conditionForResolvedKey(ctx, q, key, qbtypes.FilterOperatorExists, nil, sb)
existsCondition, err := c.conditionForResolvedKey(ctx, orgID, startNs, endNs, key, qbtypes.FilterOperatorExists, nil, sb)
if err != nil {
return "", err
}
@@ -479,44 +594,3 @@ func (s *storage) conditionForKey(
return condition, nil
}
// mapBacked reports whether the key reads a map column, so a row can lack it.
func (s *storage) mapBacked(q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) bool {
columns, err := s.getColumn(q, key)
return err == nil && len(columns) == 1 && columns[0].Type.GetType() == schema.ColumnTypeEnumMap
}
// Compile keeps the body language: search over a scope, the body functions,
// the body JSON paths, and the body column forms that use its index. Every
// other field compiles through the shared condition.
func (s *storage) Compile(ctx context.Context, q qbtypes.QueryInfo, logical *telemetrytypes.LogicalField, operator qbtypes.FilterOperator, value any, sb *sqlbuilder.SelectBuilder) (qbtypes.Compiled, error) {
if operator == qbtypes.FilterOperatorSearch {
conditions, err := s.conditionForSearch(q, logical.Single(), value, sb)
if err != nil || len(conditions) == 0 {
return qbtypes.Compiled{}, err
}
return qbtypes.Compiled{Condition: conditions[0]}, nil
}
if logical.IsFamily() || !s.ownLanguage(logical.Single(), operator) {
return querybuilder.SharedCondition(ctx, q, s, logical, operator, value, sb)
}
key := logical.Single()
condition, err := s.conditionForKey(ctx, q, key, operator, value, sb)
if err != nil {
return qbtypes.Compiled{}, err
}
compiled := qbtypes.Compiled{Condition: condition}
if w := s.bodyFullTextDefaultWarning(ctx, q, key, operator); w != "" {
compiled.Warnings = append(compiled.Warnings, w)
}
return compiled, nil
}
// ownLanguage reports whether a term needs the body language: a body path,
// a body function, or the body column itself.
func (s *storage) ownLanguage(key *telemetrytypes.TelemetryFieldKey, operator qbtypes.FilterOperator) bool {
if operator.IsFunctionOperator() || key.FieldContext == telemetrytypes.FieldContextBody {
return true
}
return key.Name == LogsV2BodyColumn && (key.FieldContext == telemetrytypes.FieldContextLog || key.FieldContext == telemetrytypes.FieldContextUnspecified)
}

View File

@@ -2,7 +2,6 @@ package logstelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
"time"
@@ -126,13 +125,14 @@ func TestExistsConditionForWithEvolutions(t *testing.T) {
},
}
fl := flaggertest.New(t)
storage := NewStorage()
fm := NewFieldMapper(fl)
conditionBuilder := NewConditionBuilder(fm, fl)
ctx := context.Background()
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, _, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, tc.startTs, tc.endTs), storage, &tc.key, tc.operator, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, false, sb)
cond, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, tc.startTs, tc.endTs, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
sb.Where(cond...)
if tc.expectedError != nil {
@@ -517,12 +517,13 @@ func TestConditionFor(t *testing.T) {
},
}
fl := flaggertest.New(t)
storage := NewStorage()
fm := NewFieldMapper(fl)
conditionBuilder := NewConditionBuilder(fm, fl)
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
tc.key.Evolutions = tc.evolutions
cond, _, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &tc.key, tc.operator, tc.value, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, false, sb)
cond, _, err := conditionBuilder.ConditionFor(ctx, valuer.UUID{}, 0, 0, &tc.key, map[string][]*telemetrytypes.TelemetryFieldKey{tc.key.Name: {&tc.key}}, qbtypes.ConditionBuilderOptions{}, tc.operator, tc.value, sb)
sb.Where(cond...)
if tc.expectedError != nil {
@@ -542,12 +543,12 @@ func TestConditionFor(t *testing.T) {
func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
storage := NewStorage()
cb := NewConditionBuilder(NewFieldMapper(fl), fl)
t.Run("bare intrinsic column resolves to the column, not synthesized attributes", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "severity_text"}
conds, _, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &key, qbtypes.FilterOperatorEqual, "ERROR", nil, false, sb)
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "ERROR", sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -559,7 +560,7 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("attribute context", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextAttribute}
conds, warnings, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &key, qbtypes.FilterOperatorEqual, "v", nil, false, sb)
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 2)
@@ -570,22 +571,18 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("resource context", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextResource}
conds, warnings, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &key, qbtypes.FilterOperatorEqual, "v", nil, false, sb)
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 2)
// the resource read spans both eras and yields NULL for an absent key, so
// it takes no presence guard
sb.Where(conds...)
sql, _ := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
assert.Contains(t, sql, "resources_string['custom.key']")
assert.Contains(t, sql, "resources_string['resource.custom.key']")
assert.Contains(t, conds[0], "mapContains(resources_string, 'custom.key')")
assert.Contains(t, conds[1], "mapContains(resources_string, 'resource.custom.key')")
})
t.Run("log context folds to attributes then body", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextLog}
conds, warnings, err := querybuilder.Conditions(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &key, qbtypes.FilterOperatorEqual, "v", nil, false, sb)
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 4)
@@ -629,14 +626,15 @@ func TestConditionForMultipleKeys(t *testing.T) {
}
fl := flaggertest.New(t)
storage := &storage{}
fm := NewFieldMapper(fl)
conditionBuilder := NewConditionBuilder(fm, fl)
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
var err error
for _, key := range tc.keys {
cond, err := storage.conditionForResolvedKey(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), &key, tc.operator, tc.value, sb)
cond, err := conditionBuilder.conditionForResolvedKey(ctx, valuer.UUID{}, 0, 0, &key, tc.operator, tc.value, sb)
sb.Where(cond)
if err != nil {
t.Fatalf("Error getting condition for key %s: %v", key.Name, err)
@@ -888,12 +886,13 @@ func TestConditionForJSONBodySearch(t *testing.T) {
}
fl := flaggertest.New(t)
storage := &storage{}
fm := NewFieldMapper(fl)
conditionBuilder := NewConditionBuilder(fm, fl)
for _, tc := range testCases {
sb := sqlbuilder.NewSelectBuilder()
t.Run(tc.name, func(t *testing.T) {
cond, err := storage.conditionForResolvedKey(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), &tc.key, tc.operator, tc.value, sb)
cond, err := conditionBuilder.conditionForResolvedKey(ctx, valuer.UUID{}, 0, 0, &tc.key, tc.operator, tc.value, sb)
sb.Where(cond)
if tc.expectedError != nil {
@@ -931,7 +930,8 @@ func TestConditionForBodyIn(t *testing.T) {
}
fl := flaggertest.New(t)
storage := NewStorage()
fm := NewFieldMapper(fl)
conditionBuilder := NewConditionBuilder(fm, fl)
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
@@ -942,7 +942,9 @@ func TestConditionForBodyIn(t *testing.T) {
}
sb := sqlbuilder.NewSelectBuilder()
sb.Select("1").From("t")
cond, _, err := querybuilder.Conditions(context.Background(), querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), storage, &key, qbtypes.FilterOperatorIn, tc.values, map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {&key}}, false, sb)
cond, _, err := conditionBuilder.ConditionFor(context.Background(), valuer.UUID{}, 0, 0, &key,
map[string][]*telemetrytypes.TelemetryFieldKey{key.Name: {&key}}, qbtypes.ConditionBuilderOptions{},
qbtypes.FilterOperatorIn, tc.values, sb)
require.NoError(t, err)
sb.Where(cond...)

View File

@@ -8,9 +8,12 @@ import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz-otel-collector/utils"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/huandu/go-sqlbuilder"
)
@@ -60,15 +63,15 @@ var (
}
)
type storage struct{}
var _ qbtypes.Storage = (*storage)(nil)
func NewStorage() qbtypes.Storage {
return &storage{}
type fieldMapper struct {
fl flagger.Flagger
}
func (m *storage) getColumn(q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
func NewFieldMapper(fl flagger.Flagger) qbtypes.FieldMapper {
return &fieldMapper{fl: fl}
}
func (m *fieldMapper) getColumn(ctx context.Context, orgID valuer.UUID, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
switch key.FieldContext {
case telemetrytypes.FieldContextResource:
columns := []*schema.Column{logsV2Columns["resource"], logsV2Columns["resources_string"]}
@@ -92,7 +95,7 @@ func (m *storage) getColumn(q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFi
}
case telemetrytypes.FieldContextBody:
// Body context is for JSON body fields. Use body_v2 if feature flag is enabled.
if q.BodyJSONOn {
if m.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
if key.Name == messageSubField {
return []*schema.Column{logsV2Columns[messageSubColumn]}, nil
}
@@ -101,7 +104,7 @@ func (m *storage) getColumn(q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFi
// Fall back to legacy body column
return []*schema.Column{logsV2Columns["body"]}, nil
case telemetrytypes.FieldContextLog:
if key.Name == LogsV2BodyColumn && q.BodyJSONOn {
if key.Name == LogsV2BodyColumn && m.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
return []*schema.Column{logsV2Columns[messageSubColumn]}, nil
}
col, ok := logsV2Columns[key.Name]
@@ -114,13 +117,13 @@ func (m *storage) getColumn(q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFi
return nil, qbtypes.ErrColumnNotFound
}
func (m *storage) read(_ context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(q, key)
func (m *fieldMapper) FieldFor(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, orgID, key)
if err != nil {
return "", err
}
newColumns, evolutionsEntries, err := qbtypes.SelectEvolutionsForColumns(columns, key.Evolutions, q.StartNs, q.EndNs)
newColumns, evolutionsEntries, err := qbtypes.SelectEvolutionsForColumns(columns, key.Evolutions, tsStart, tsEnd)
if err != nil {
return "", err
}
@@ -210,8 +213,191 @@ func (m *storage) read(_ context.Context, q qbtypes.QueryInfo, key *telemetrytyp
return columns[0].Name, nil
}
func (m *fieldMapper) ColumnFor(ctx context.Context, orgID valuer.UUID, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
return m.getColumn(ctx, orgID, key)
}
func (m *fieldMapper) ColumnExpressionFor(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
field *telemetrytypes.TelemetryFieldKey,
requiredDataType telemetrytypes.FieldDataType,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
bodyJSONEnabled := m.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
var candidates []*telemetrytypes.TelemetryFieldKey
switch _, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, field); {
case err == nil:
if field.FieldContext == telemetrytypes.FieldContextBody && !bodyJSONEnabled {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "Operation isn't available for the body column")
}
candidates = []*telemetrytypes.TelemetryFieldKey{field}
case errors.Is(err, qbtypes.ErrColumnNotFound):
if _, ok := logsV2Columns[field.Name]; ok {
field.FieldContext = telemetrytypes.FieldContextLog
candidates = []*telemetrytypes.TelemetryFieldKey{field}
break
}
candidates = keys[field.Name]
if len(candidates) == 0 {
candidates = keys[fmt.Sprintf("%s.%s", field.FieldContext.StringValue(), field.Name)]
}
if len(candidates) == 0 {
// synthesized attribute candidates first, body path last; legacy body
// doesn't support group by/select, so bare keys keep attributes only
for _, key := range m.CandidateKeys(ctx, orgID, field, nil, nil) {
if !bodyJSONEnabled && field.FieldContext == telemetrytypes.FieldContextUnspecified &&
key.FieldContext == telemetrytypes.FieldContextBody {
continue
}
candidates = append(candidates, key)
}
}
if len(candidates) == 0 {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "field `%s` not found", field.Name)
}
default:
return "", err
}
// Group-by/order (String) and aggregation (String/Float64): every candidate is
// exists-guarded and coerced to requiredDataType, in a single multiIf. Raw select
// (Unspecified) keeps the lighter native shape below.
if requiredDataType != telemetrytypes.FieldDataTypeUnspecified {
// arrays cannot sit inside Nullable/multiIf, so a lone array candidate stays bare
if len(candidates) == 1 && (strings.Contains(candidates[0].Name, telemetrytypes.ArraySep) ||
strings.Contains(candidates[0].Name, telemetrytypes.ArrayAnyIndex) ||
candidates[0].FieldDataType.IsArray()) {
return m.FieldFor(ctx, orgID, tsStart, tsEnd, candidates[0])
}
var dummyValue any = ""
if requiredDataType == telemetrytypes.FieldDataTypeFloat64 {
dummyValue = 0.0
}
var stmts []string
for _, key := range candidates {
guard, err := m.ExistsFor(ctx, orgID, tsStart, tsEnd, key, true)
if err != nil {
return "", err
}
var fieldExpression string
if key.FieldContext == telemetrytypes.FieldContextBody && !bodyJSONEnabled {
fieldExpression, _ = GetBodyJSONKey(ctx, key, qbtypes.FilterOperatorUnknown, dummyValue)
} else {
fieldExpression, err = m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
return "", err
}
fieldExpression, _ = querybuilder.DataTypeCollisionHandledFieldName(key, dummyValue, fieldExpression, qbtypes.FilterOperatorUnknown)
}
stmts = append(stmts, guard, fieldExpression)
}
return fmt.Sprintf("multiIf(%s, NULL)", strings.Join(stmts, ", ")), nil
}
if len(candidates) == 1 {
// arrays cannot sit inside Nullable, so array candidates stay bare
if strings.Contains(candidates[0].Name, telemetrytypes.ArraySep) ||
strings.Contains(candidates[0].Name, telemetrytypes.ArrayAnyIndex) ||
candidates[0].FieldDataType.IsArray() {
return m.FieldFor(ctx, orgID, tsStart, tsEnd, candidates[0])
}
if !m.membershipGuarded(ctx, orgID, tsStart, tsEnd, candidates[0]) {
return m.FieldFor(ctx, orgID, tsStart, tsEnd, candidates[0])
}
guard, err := m.ExistsFor(ctx, orgID, tsStart, tsEnd, candidates[0], true)
if err != nil {
return "", err
}
var fieldExpression string
if candidates[0].FieldContext == telemetrytypes.FieldContextBody && !bodyJSONEnabled {
fieldExpression, _ = GetBodyJSONKey(ctx, candidates[0], qbtypes.FilterOperatorUnknown, "")
} else {
fieldExpression, err = m.FieldFor(ctx, orgID, tsStart, tsEnd, candidates[0])
if err != nil {
return "", err
}
}
return fmt.Sprintf("multiIf(%s, %s, NULL)", guard, fieldExpression), nil
}
var stmts []string
for _, key := range candidates {
guard, err := m.ExistsFor(ctx, orgID, tsStart, tsEnd, key, true)
if err != nil {
return "", err
}
stmts = append(stmts, guard)
var fieldExpression string
if key.FieldContext == telemetrytypes.FieldContextBody && !bodyJSONEnabled {
fieldExpression, _ = GetBodyJSONKey(ctx, key, qbtypes.FilterOperatorUnknown, "")
} else {
fieldExpression, err = m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
return "", err
}
fieldExpression, _ = querybuilder.DataTypeCollisionHandledFieldName(key, "", fieldExpression, qbtypes.FilterOperatorUnknown)
}
stmts = append(stmts, fieldExpression)
}
return fmt.Sprintf("multiIf(%s, NULL)", strings.Join(stmts, ", ")), nil
}
func (m *fieldMapper) CandidateKeys(_ context.Context, _ valuer.UUID, field *telemetrytypes.TelemetryFieldKey, value any, keys map[string][]*telemetrytypes.TelemetryFieldKey) []*telemetrytypes.TelemetryFieldKey {
if matches := keys[field.Name]; len(matches) > 0 {
return matches
}
if matches := keys[fmt.Sprintf("%s.%s", field.FieldContext.StringValue(), field.Name)]; len(matches) > 0 {
return matches
}
switch field.FieldContext {
case telemetrytypes.FieldContextBody:
if field.Name == "" {
return nil
}
return []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(field.Name, field.FieldContext, field.FieldDataType)}
case telemetrytypes.FieldContextUnspecified:
// a real column wins before synthesis (so adjustKeys is not needed to resolve these)
if _, ok := logsV2Columns[field.Name]; ok {
return []*telemetrytypes.TelemetryFieldKey{{Name: field.Name, FieldContext: telemetrytypes.FieldContextLog}}
}
bodyKey := telemetrytypes.NewTelemetryFieldKey(field.Name, telemetrytypes.FieldContextBody, field.FieldDataType)
if value == nil && bodyKey.FieldDataType == telemetrytypes.FieldDataTypeUnspecified {
bodyKey.FieldDataType = telemetrytypes.FieldDataTypeString
}
return append(querybuilder.SynthesizeKeys(field, value), bodyKey)
case telemetrytypes.FieldContextAttribute, telemetrytypes.FieldContextResource:
// stripped interpretation first, the literal `{context}.{name}` spelling second
literal := telemetrytypes.NewTelemetryFieldKey(field.FieldContext.StringValue()+"."+field.Name, field.FieldContext, field.FieldDataType)
return append(querybuilder.SynthesizeKeys(field, value), querybuilder.SynthesizeKeys(literal, value)...)
case telemetrytypes.FieldContextLog:
if _, ok := logsV2Columns[field.Name]; ok {
return []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(field.Name, field.FieldContext, field.FieldDataType)}
}
stripped := telemetrytypes.NewTelemetryFieldKey(field.Name, telemetrytypes.FieldContextUnspecified, field.FieldDataType)
literal := telemetrytypes.NewTelemetryFieldKey(field.FieldContext.StringValue()+"."+field.Name, telemetrytypes.FieldContextUnspecified, field.FieldDataType)
// attribute candidates first (stripped, then literal), body paths last
candidates := append(querybuilder.SynthesizeKeys(stripped, value), querybuilder.SynthesizeKeys(literal, value)...)
for _, key := range []*telemetrytypes.TelemetryFieldKey{stripped, literal} {
bodyKey := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextBody, key.FieldDataType)
if value == nil && bodyKey.FieldDataType == telemetrytypes.FieldDataTypeUnspecified {
bodyKey.FieldDataType = telemetrytypes.FieldDataTypeString
}
candidates = append(candidates, bodyKey)
}
return candidates
}
return nil
}
// buildFieldForJSON builds the field expression for body JSON fields using arrayConcat pattern.
func (m *storage) buildFieldForJSON(key *telemetrytypes.TelemetryFieldKey) (string, error) {
func (m *fieldMapper) buildFieldForJSON(key *telemetrytypes.TelemetryFieldKey) (string, error) {
plan := key.JSONPlan
if len(plan) == 0 {
if key.KeyNameContainsArray() {
@@ -282,7 +468,7 @@ func (m *storage) buildFieldForJSON(key *telemetrytypes.TelemetryFieldKey) (stri
}
// buildArrayConcat builds the arrayConcat pattern directly from the tree structure.
func (m *storage) buildArrayConcat(plan telemetrytypes.JSONAccessPlan) (string, error) {
func (m *fieldMapper) buildArrayConcat(plan telemetrytypes.JSONAccessPlan) (string, error) {
if len(plan) == 0 {
return "", errors.NewInternalf(CodeGroupByPlanEmpty, "group by plan is empty while building arrayConcat")
}
@@ -313,7 +499,7 @@ func (m *storage) buildArrayConcat(plan telemetrytypes.JSONAccessPlan) (string,
}
// buildArrayMap builds the arrayMap expression for a specific branch, handling all sub-branches.
func (m *storage) buildArrayMap(currentNode *telemetrytypes.JSONAccessNode, branchType telemetrytypes.JSONAccessBranchType) (string, error) {
func (m *fieldMapper) buildArrayMap(currentNode *telemetrytypes.JSONAccessNode, branchType telemetrytypes.JSONAccessBranchType) (string, error) {
if currentNode == nil {
return "", errors.NewInternalf(CodeCurrentNodeNil, "current node is nil while building arrayMap")
}
@@ -367,6 +553,90 @@ func (m *storage) buildArrayMap(currentNode *telemetrytypes.JSONAccessNode, bran
return fmt.Sprintf("arrayMap(%s->%s, %s)", currentNode.Alias(), nestedExpr, arrayExpr), nil
}
func (m *fieldMapper) membershipGuarded(ctx context.Context, orgID valuer.UUID, tsStart, tsEnd uint64, key *telemetrytypes.TelemetryFieldKey) bool {
if key.FieldContext == telemetrytypes.FieldContextBody {
return true
}
columns, err := m.getColumn(ctx, orgID, key)
if err != nil {
return false
}
newColumns, _, err := qbtypes.SelectEvolutionsForColumns(columns, key.Evolutions, tsStart, tsEnd)
if err != nil || len(newColumns) != 1 {
return false
}
columnType := newColumns[0].Type.GetType()
return columnType == schema.ColumnTypeEnumMap || columnType == schema.ColumnTypeEnumJSON
}
// ExistsFor implements the per-key existence primitive of qbtypes.FieldMapper.
func (m *fieldMapper) ExistsFor(
ctx context.Context,
orgID valuer.UUID,
tsStart, tsEnd uint64,
key *telemetrytypes.TelemetryFieldKey,
exists bool,
) (string, error) {
columns, err := m.getColumn(ctx, orgID, key)
if errors.Is(err, qbtypes.ErrColumnNotFound) && key.FieldContext == telemetrytypes.FieldContextUnspecified {
bodyKey := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextBody, key.FieldDataType)
columns, err = m.getColumn(ctx, orgID, bodyKey)
}
if err != nil {
return "", err
}
operator := qbtypes.FilterOperatorExists
if !exists {
operator = qbtypes.FilterOperatorNotExists
}
bodyJSONEnabled := m.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
for _, column := range columns {
if column.Type.GetType() == schema.ColumnTypeEnumJSON && isBodyJSONSearch(key, columns) && bodyJSONEnabled && key.Name != messageSubField {
valueType, value := InferDataType(nil, operator, key)
if len(key.JSONPlan) == 0 {
keyCopy := telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, key.FieldDataType)
if err := keyCopy.SetExhaustiveJSONAccessPlan(
telemetrytypes.JSONColumnMetadata{BaseColumn: LogsV2BodyV2Column}, valueType,
); err != nil {
return "", err
}
key = keyCopy
}
sb := sqlbuilder.NewSelectBuilder()
cond, err := NewJSONConditionBuilder(key, valueType).buildJSONCondition(operator, value, sb)
if err != nil {
return "", err
}
sb.Where(cond)
expr, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
expr = strings.TrimPrefix(expr, "WHERE ")
if len(args) > 0 {
expr, err = sqlbuilder.ClickHouse.Interpolate(expr, args)
if err != nil {
return "", err
}
}
return expr, nil
}
}
if isBodyJSONSearch(key, columns) && !bodyJSONEnabled {
if exists {
return GetBodyJSONKeyForExists(ctx, key, operator, nil), nil
}
return "NOT " + GetBodyJSONKeyForExists(ctx, key, operator, nil), nil
}
fieldExpression, err := m.FieldFor(ctx, orgID, tsStart, tsEnd, key)
if err != nil {
return "", err
}
return querybuilder.ExistsExpression(columns, key, tsStart, tsEnd, fieldExpression, exists)
}
// searchColumns is the single source of truth for the columns search() fans out across,
// by context; body is body_v2 when useJSONBody, else the legacy body string.
func searchColumns(fieldContext telemetrytypes.FieldContext, useJSONBody bool) []*schema.Column {
@@ -400,164 +670,3 @@ func searchColumns(fieldContext telemetrytypes.FieldContext, useJSONBody bool) [
return columns
}
}
// absentReads tells what a row without the key reads: NULL from a multi-era
// read, the empty value from a map or from a JSON path (its ::String cast
// folds NULL to the empty string), and a real value from every other column.
func absentReads(q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, columns []*schema.Column) qbtypes.Absent {
newColumns, _, err := qbtypes.SelectEvolutionsForColumns(columns, key.Evolutions, q.StartNs, q.EndNs)
if err != nil || len(newColumns) != 1 {
return qbtypes.AbsentIsNull
}
switch newColumns[0].Type.GetType() {
case schema.ColumnTypeEnumMap, schema.ColumnTypeEnumJSON:
return qbtypes.AbsentIsSentinel
}
return qbtypes.AlwaysPresent
}
// Read composes the bare read of one key with its membership test and what
// an absent row reads. A body JSON path tests presence through the JSON
// condition language; the legacy body path through its typed extraction; a
// column or map through the shared presence expression. Without the JSON
// body, a body path filters but cannot be selected.
func (m *storage) Read(ctx context.Context, q qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey) (qbtypes.Read, error) {
columns, err := m.getColumn(q, key)
if errors.Is(err, qbtypes.ErrColumnNotFound) && key.FieldContext == telemetrytypes.FieldContextUnspecified {
bodyKey := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextBody, key.FieldDataType)
columns, err = m.getColumn(q, bodyKey)
}
if err != nil {
return qbtypes.Read{}, err
}
sql, err := m.read(ctx, q, key)
if err != nil {
return qbtypes.Read{}, err
}
read := qbtypes.Read{SQL: sql, FilterOnly: key.FieldContext == telemetrytypes.FieldContextBody && !q.BodyJSONOn}
for _, column := range columns {
if column.Type.GetType() == schema.ColumnTypeEnumJSON && isBodyJSONSearch(key, columns) && q.BodyJSONOn && key.Name != messageSubField {
valueType, value := InferDataType(nil, qbtypes.FilterOperatorExists, key)
if len(key.JSONPlan) == 0 {
keyCopy := telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, key.FieldDataType)
if err := keyCopy.SetExhaustiveJSONAccessPlan(
telemetrytypes.JSONColumnMetadata{BaseColumn: LogsV2BodyV2Column}, valueType,
); err != nil {
return qbtypes.Read{}, err
}
key = keyCopy
}
jsonCondition := func(operator qbtypes.FilterOperator) (string, error) {
sb := sqlbuilder.NewSelectBuilder()
cond, err := NewJSONConditionBuilder(key, valueType).buildJSONCondition(operator, value, sb)
if err != nil {
return "", err
}
sb.Where(cond)
expr, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
expr = strings.TrimPrefix(expr, "WHERE ")
if len(args) > 0 {
return sqlbuilder.ClickHouse.Interpolate(expr, args)
}
return expr, nil
}
if read.Presence, err = jsonCondition(qbtypes.FilterOperatorExists); err != nil {
return qbtypes.Read{}, err
}
if read.Absence, err = jsonCondition(qbtypes.FilterOperatorNotExists); err != nil {
return qbtypes.Read{}, err
}
// a JSON path reads NULL when the row lacks it
read.WhenAbsent = qbtypes.AbsentIsNull
return read, nil
}
}
if isBodyJSONSearch(key, columns) && !q.BodyJSONOn {
read.Presence = GetBodyJSONKeyForExists(ctx, key, qbtypes.FilterOperatorExists, nil)
read.Absence = "NOT " + read.Presence
read.WhenAbsent = qbtypes.AbsentIsSentinel
return read, nil
}
if read.Presence, err = querybuilder.ExistsExpression(columns, key, q.StartNs, q.EndNs, sql, true); err != nil {
return qbtypes.Read{}, err
}
if read.Absence, err = querybuilder.ExistsExpression(columns, key, q.StartNs, q.EndNs, sql, false); err != nil {
return qbtypes.Read{}, err
}
read.WhenAbsent = absentReads(q, key, columns)
return read, nil
}
// Fallback answers a key metadata does not report: a column name is the
// column, a body path is itself, and any other name synthesizes its type
// variants under the stripped and the literal spelling, with the body path
// last. A body function narrows the answer to body paths.
func (m *storage) Fallback(_ context.Context, _ qbtypes.QueryInfo, key *telemetrytypes.TelemetryFieldKey, operator qbtypes.FilterOperator, value any) ([]*telemetrytypes.LogicalField, error) {
if key.FieldContext == telemetrytypes.FieldContextBody && key.Name == "" {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "missing key for body json search - expected key of the form `body.key` (ex: `body.status`)")
}
var keys []*telemetrytypes.TelemetryFieldKey
switch key.FieldContext {
case telemetrytypes.FieldContextBody:
keys = []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, key.FieldDataType)}
case telemetrytypes.FieldContextUnspecified:
if column, ok := logsV2Columns[key.Name]; ok {
keys = []*telemetrytypes.TelemetryFieldKey{{Name: key.Name, FieldContext: telemetrytypes.FieldContextLog, FieldDataType: querybuilder.ColumnDataType(column)}}
break
}
keys = append(querybuilder.SynthesizeKeys(key, value), bodyPath(key.Name, key.FieldDataType, value))
case telemetrytypes.FieldContextAttribute, telemetrytypes.FieldContextResource:
// a context can be a legitimate prefix in user data
literal := telemetrytypes.NewTelemetryFieldKey(key.FieldContext.StringValue()+"."+key.Name, key.FieldContext, key.FieldDataType)
keys = append(querybuilder.SynthesizeKeys(key, value), querybuilder.SynthesizeKeys(literal, value)...)
case telemetrytypes.FieldContextLog:
if column, ok := logsV2Columns[key.Name]; ok {
dataType := key.FieldDataType
if dataType == telemetrytypes.FieldDataTypeUnspecified {
dataType = querybuilder.ColumnDataType(column)
}
keys = []*telemetrytypes.TelemetryFieldKey{telemetrytypes.NewTelemetryFieldKey(key.Name, key.FieldContext, dataType)}
break
}
stripped := telemetrytypes.NewTelemetryFieldKey(key.Name, telemetrytypes.FieldContextUnspecified, key.FieldDataType)
literal := telemetrytypes.NewTelemetryFieldKey(key.FieldContext.StringValue()+"."+key.Name, telemetrytypes.FieldContextUnspecified, key.FieldDataType)
keys = append(querybuilder.SynthesizeKeys(stripped, value), querybuilder.SynthesizeKeys(literal, value)...)
for _, spelling := range []*telemetrytypes.TelemetryFieldKey{stripped, literal} {
keys = append(keys, bodyPath(spelling.Name, spelling.FieldDataType, value))
}
}
if operator.IsFunctionOperator() {
if key.FieldContext != telemetrytypes.FieldContextBody {
return nil, querybuilder.NewFunctionUnsupportedError(operator)
}
bodyKeys := make([]*telemetrytypes.TelemetryFieldKey, 0, len(keys))
for _, k := range keys {
if k.FieldContext == telemetrytypes.FieldContextBody {
bodyKeys = append(bodyKeys, k)
}
}
keys = bodyKeys
}
return querybuilder.WrapAsLogicalFields(key.Name, keys), nil
}
// bodyPath is the body JSON path for a name; without an operand to type it,
// a string path.
func bodyPath(name string, dataType telemetrytypes.FieldDataType, value any) *telemetrytypes.TelemetryFieldKey {
bodyKey := telemetrytypes.NewTelemetryFieldKey(name, telemetrytypes.FieldContextBody, dataType)
if value == nil && bodyKey.FieldDataType == telemetrytypes.FieldDataTypeUnspecified {
bodyKey.FieldDataType = telemetrytypes.FieldDataTypeString
}
return bodyKey
}
func (m *storage) Traits() qbtypes.Traits {
return qbtypes.Traits{
Split: qbtypes.MainOfSplit,
SupportsBodyFunctions: true,
OwnContexts: []telemetrytypes.FieldContext{telemetrytypes.FieldContextLog},
}
}

View File

@@ -2,7 +2,6 @@ package logstelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/querybuilder"
"testing"
"time"
@@ -16,6 +15,7 @@ import (
)
func TestGetColumn(t *testing.T) {
ctx := context.Background()
testCases := []struct {
name string
@@ -168,11 +168,11 @@ func TestGetColumn(t *testing.T) {
}
fl := flaggertest.New(t)
storage := &storage{}
fm := NewFieldMapper(fl)
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
col, err := storage.getColumn(querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), &tc.key)
col, err := fm.ColumnFor(ctx, valuer.UUID{}, 0, 0, &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
@@ -277,14 +277,14 @@ func TestGetFieldKeyName(t *testing.T) {
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fl := flaggertest.New(t)
storage := NewStorage()
read, err := storage.Read(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0), &tc.key)
fm := NewFieldMapper(fl)
result, err := fm.FieldFor(ctx, valuer.UUID{}, 0, 0, &tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
} else {
require.NoError(t, err)
assert.Equal(t, tc.expectedResult, read.SQL)
assert.Equal(t, tc.expectedResult, result)
}
})
}
@@ -519,19 +519,19 @@ func TestFieldForWithEvolutions(t *testing.T) {
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fl := flaggertest.New(t)
storage := NewStorage()
fm := NewFieldMapper(fl)
tsStart := uint64(tc.tsStartTime.UnixNano())
tsEnd := uint64(tc.tsEndTime.UnixNano())
tc.key.Evolutions = tc.evolutions
read, err := storage.Read(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, tsStart, tsEnd), tc.key)
result, err := fm.FieldFor(ctx, valuer.UUID{}, tsStart, tsEnd, tc.key)
if tc.expectedError != nil {
assert.Equal(t, tc.expectedError, err)
} else {
require.NoError(t, err)
assert.Equal(t, tc.expectedResult, read.SQL)
assert.Equal(t, tc.expectedResult, result)
}
})
}
@@ -585,15 +585,15 @@ func TestFieldForWithMaterialized(t *testing.T) {
}
fl := flaggertest.New(t)
storage := NewStorage()
fm := NewFieldMapper(fl)
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
start := uint64(tc.start.UnixNano())
end := uint64(tc.end.UnixNano())
read, err := storage.Read(ctx, querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, start, end), materializedKey)
result, err := fm.FieldFor(ctx, valuer.UUID{}, start, end, materializedKey)
require.NoError(t, err)
assert.Equal(t, tc.expectedResult, read.SQL)
assert.Equal(t, tc.expectedResult, result)
})
}
}

View File

@@ -2,8 +2,6 @@ package logstelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"testing"
"time"
@@ -17,17 +15,19 @@ import (
func TestLikeAndILikeWithoutWildcards_Warns(t *testing.T) {
fl := flaggertest.New(t)
ctx := context.Background()
storage := NewStorage()
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0),
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
Context: ctx,
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
}
tests := []string{
@@ -53,17 +53,19 @@ func TestLikeAndILikeWithoutWildcards_Warns(t *testing.T) {
// TestLikeAndILikeWithWildcards_NoWarn Tests that LIKE/ILIKE with wildcards do not add warnings.
func TestLikeAndILikeWithWildcards_NoWarn(t *testing.T) {
fl := flaggertest.New(t)
storage := NewStorage()
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0),
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn}
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn}
tests := []string{
"service.name LIKE 'demo-%'",

View File

@@ -3,7 +3,6 @@ package logstelemetryschema
import (
"context"
"fmt"
"github.com/SigNoz/signoz/pkg/valuer"
"testing"
"time"
@@ -18,17 +17,19 @@ import (
// TestFilterExprLogsBodyJSON tests a comprehensive set of query patterns for body JSON search.
func TestFilterExprLogsBodyJSON(t *testing.T) {
fl := flaggertest.New(t)
storage := NewStorage()
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
// Define a comprehensive set of field keys to support all test cases
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0),
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "body"},
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: &telemetrytypes.TelemetryFieldKey{Name: "body"},
}
testCases := []struct {

View File

@@ -3,7 +3,6 @@ package logstelemetryschema
import (
"context"
"fmt"
"github.com/SigNoz/signoz/pkg/valuer"
"strings"
"testing"
"time"
@@ -38,17 +37,21 @@ func TestFilterExprLogs(t *testing.T) {
fl := flaggertest.New(t)
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
ctx := context.Background()
storage := NewStorage()
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
// Define a comprehensive set of field keys to support all test cases
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, uint64(releaseTime.Add(-5*time.Minute).UnixNano()), uint64(releaseTime.Add(5*time.Minute).UnixNano())),
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
Context: ctx,
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
StartNs: uint64(releaseTime.Add(-5 * time.Minute).UnixNano()),
EndNs: uint64(releaseTime.Add(5 * time.Minute).UnixNano()),
}
testCases := []struct {
@@ -482,7 +485,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "FREETEXT with conditions",
query: "error service.name=authentication",
shouldPass: true,
expectedQuery: "WHERE (match(LOWER(body), LOWER(?)) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE (match(LOWER(body), LOWER(?)) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedArgs: []any{"error", "authentication"},
expectedErrorContains: "",
},
@@ -840,7 +843,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Basic equality",
query: "service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?",
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -1200,7 +1203,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "IN operator (parentheses)",
query: "service.name IN (\"api\", \"web\", \"auth\")",
shouldPass: true,
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedArgs: []any{"api", "web", "auth"},
expectedErrorContains: "",
},
@@ -1208,7 +1211,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "IN operator (parentheses)",
query: "environment IN (\"dev\", \"test\", \"staging\", \"prod\")",
shouldPass: true,
expectedQuery: "WHERE (multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ?)",
expectedQuery: "WHERE ((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ?) AND multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) IS NOT NULL)",
expectedArgs: []any{"dev", "test", "staging", "prod"},
expectedErrorContains: "",
},
@@ -1234,7 +1237,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "IN operator (brackets)",
query: "service.name IN [\"api\", \"web\", \"auth\"]",
shouldPass: true,
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedArgs: []any{"api", "web", "auth"},
expectedErrorContains: "",
},
@@ -1242,7 +1245,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "IN operator (brackets)",
query: "environment IN [\"dev\", \"test\", \"staging\", \"prod\"]",
shouldPass: true,
expectedQuery: "WHERE (multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ?)",
expectedQuery: "WHERE ((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? OR multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ?) AND multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) IS NOT NULL)",
expectedArgs: []any{"dev", "test", "staging", "prod"},
expectedErrorContains: "",
},
@@ -1606,7 +1609,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Explicit AND",
query: "status=200 AND service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedArgs: []any{float64(200), "api"},
expectedErrorContains: "",
},
@@ -1640,7 +1643,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Explicit OR",
query: "service.name=\"api\" OR service.name=\"web\"",
shouldPass: true,
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedArgs: []any{"api", "web"},
expectedErrorContains: "",
},
@@ -1666,7 +1669,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "NOT with expressions",
query: "NOT service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE NOT (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE NOT ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -1684,7 +1687,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "AND + OR combinations",
query: "status=200 AND (service.name=\"api\" OR service.name=\"web\")",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))))",
expectedArgs: []any{float64(200), "api", "web"},
expectedErrorContains: "",
},
@@ -1710,7 +1713,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "AND + NOT combinations",
query: "status=200 AND NOT service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND NOT (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND NOT ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))",
expectedArgs: []any{float64(200), "api"},
expectedErrorContains: "",
},
@@ -1728,7 +1731,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "OR + NOT combinations",
query: "NOT status=200 OR NOT service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) OR NOT (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))",
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) OR NOT ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))",
expectedArgs: []any{float64(200), "api"},
expectedErrorContains: "",
},
@@ -1746,7 +1749,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "AND + OR + NOT combinations",
query: "status=200 AND (service.name=\"api\" OR NOT duration>1000)",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR NOT ((toFloat64(attributes_number['duration']) > ? AND mapContains(attributes_number, 'duration'))))))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR NOT ((toFloat64(attributes_number['duration']) > ? AND mapContains(attributes_number, 'duration'))))))",
expectedArgs: []any{float64(200), "api", float64(1000)},
expectedErrorContains: "",
},
@@ -1762,7 +1765,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "AND + OR + NOT combinations",
query: "NOT (status=200 AND service.name=\"api\") OR count>0",
shouldPass: true,
expectedQuery: "WHERE (NOT ((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))) OR (toFloat64(attributes_number['count']) > ? AND mapContains(attributes_number, 'count')))",
expectedQuery: "WHERE (NOT ((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))) OR (toFloat64(attributes_number['count']) > ? AND mapContains(attributes_number, 'count')))",
expectedArgs: []any{float64(200), "api", float64(0)},
expectedErrorContains: "",
},
@@ -1772,7 +1775,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Implicit AND",
query: "status=200 service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedArgs: []any{float64(200), "api"},
expectedErrorContains: "",
},
@@ -1798,7 +1801,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Mixed implicit/explicit AND",
query: "status=200 AND service.name=\"api\" duration<1000",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND (toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) AND (toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')))",
expectedArgs: []any{float64(200), "api", float64(1000)},
expectedErrorContains: "",
},
@@ -1824,7 +1827,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Simple grouping",
query: "service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?",
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -1850,7 +1853,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Nested grouping",
query: "(((service.name=\"api\")))",
shouldPass: true,
expectedQuery: "WHERE (((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)))",
expectedQuery: "WHERE ((((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))))",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -1868,7 +1871,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Complex nested grouping",
query: "(status=200 AND (service.name=\"api\" OR service.name=\"web\"))",
shouldPass: true,
expectedQuery: "WHERE (((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))))",
expectedQuery: "WHERE (((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))))",
expectedArgs: []any{float64(200), "api", "web"},
expectedErrorContains: "",
},
@@ -1894,7 +1897,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Deep nesting",
query: "(((status=200 OR status=201) AND service.name=\"api\") OR ((status=202 OR status=203) AND service.name=\"web\"))",
shouldPass: true,
expectedQuery: "WHERE (((((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')))) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)) OR (((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')))) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))))",
expectedQuery: "WHERE (((((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')))) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))) OR (((((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')))) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))))",
expectedArgs: []any{float64(200), float64(201), "api", float64(202), float64(203), "web"},
expectedErrorContains: "",
},
@@ -1902,7 +1905,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Deep nesting",
query: "(count>0 AND ((duration<1000 AND service.name=\"api\") OR (duration<500 AND service.name=\"web\")))",
shouldPass: true,
expectedQuery: "WHERE (((toFloat64(attributes_number['count']) > ? AND mapContains(attributes_number, 'count')) AND (((((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)) OR (((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))))))",
expectedQuery: "WHERE (((toFloat64(attributes_number['count']) > ? AND mapContains(attributes_number, 'count')) AND (((((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))) OR (((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))))))",
expectedArgs: []any{float64(0), float64(1000), "api", float64(500), "web"},
expectedErrorContains: "",
},
@@ -1912,7 +1915,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "String quote styles",
query: "service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?",
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -1920,7 +1923,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "String quote styles",
query: "service.name='api'",
shouldPass: true,
expectedQuery: "WHERE multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?",
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)",
expectedArgs: []any{"api"},
expectedErrorContains: "",
},
@@ -2080,28 +2083,28 @@ func TestFilterExprLogs(t *testing.T) {
category: "Operator precedence",
query: "NOT status=200 AND service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedArgs: []any{float64(200), "api"}, // Should be (NOT status=200) AND service.name="api"
},
{
category: "Operator precedence",
query: "status=200 AND service.name=\"api\" OR service.name=\"web\"",
shouldPass: true,
expectedQuery: "WHERE (((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE (((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedArgs: []any{float64(200), "api", "web"}, // Should be (status=200 AND service.name="api") OR service.name="web"
},
{
category: "Operator precedence",
query: "NOT status=200 OR NOT service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) OR NOT (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?))",
expectedQuery: "WHERE (NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))) OR NOT ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)))",
expectedArgs: []any{float64(200), "api"}, // Should be (NOT status=200) OR (NOT service.name="api")
},
{
category: "Operator precedence",
query: "status=200 OR service.name=\"api\" AND level=\"ERROR\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND (attributes_string['level'] = ? AND mapContains(attributes_string, 'level'))))",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) OR ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) AND (attributes_string['level'] = ? AND mapContains(attributes_string, 'level'))))",
expectedArgs: []any{float64(200), "api", "ERROR"}, // Should be status=200 OR (service.name="api" AND level="ERROR")
},
@@ -2126,7 +2129,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Whitespace patterns",
query: "status=200 AND service.name=\"api\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedArgs: []any{float64(200), "api"}, // Multiple spaces
},
@@ -2296,7 +2299,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "More common filters",
query: "service.name=\"api\" AND (status>=500 OR duration>1000) AND NOT message CONTAINS \"expected\"",
shouldPass: true,
expectedQuery: "WHERE (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND (((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['duration']) > ? AND mapContains(attributes_number, 'duration')))) AND NOT ((LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message'))))",
expectedQuery: "WHERE ((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) AND (((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) OR (toFloat64(attributes_number['duration']) > ? AND mapContains(attributes_number, 'duration')))) AND NOT ((LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message'))))",
expectedArgs: []any{"api", float64(500), float64(1000), "%expected%"},
},
@@ -2362,7 +2365,7 @@ func TestFilterExprLogs(t *testing.T) {
category: "Unusual whitespace",
query: "status = 200 AND service.name = \"api\"",
shouldPass: true,
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?)",
expectedQuery: "WHERE ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status')) AND (multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL))",
expectedArgs: []any{float64(200), "api"},
},
{
@@ -2423,7 +2426,7 @@ func TestFilterExprLogs(t *testing.T) {
)
`,
shouldPass: true,
expectedQuery: "WHERE ((((((((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) AND (toFloat64(attributes_number['status']) < ? AND mapContains(attributes_number, 'status')))) OR (((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) AND (toFloat64(attributes_number['status']) < ? AND mapContains(attributes_number, 'status')) AND NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))))))) AND (((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) OR ((multiIf(resource.`service.type` IS NOT NULL, resource.`service.type`::String, mapContains(resources_string, 'service.type'), resources_string['service.type'], NULL) = ? AND NOT (multiIf(resource.`service.deprecated` IS NOT NULL, resource.`service.deprecated`::String, mapContains(resources_string, 'service.deprecated'), resources_string['service.deprecated'], NULL) = ?))))))) AND (((((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) OR ((toFloat64(attributes_number['duration']) BETWEEN ? AND ? AND mapContains(attributes_number, 'duration'))))) AND ((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) <> ? OR ((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? AND (attributes_bool['is_automated_test'] = ? AND mapContains(attributes_bool, 'is_automated_test')))))))) AND NOT ((((((LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message')) OR (LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message')))) AND (attributes_string['severity'] = ? AND mapContains(attributes_string, 'severity'))))))",
expectedQuery: "WHERE ((((((((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) AND (toFloat64(attributes_number['status']) < ? AND mapContains(attributes_number, 'status')))) OR (((toFloat64(attributes_number['status']) >= ? AND mapContains(attributes_number, 'status')) AND (toFloat64(attributes_number['status']) < ? AND mapContains(attributes_number, 'status')) AND NOT ((toFloat64(attributes_number['status']) = ? AND mapContains(attributes_number, 'status'))))))) AND ((((multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? OR multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?) AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL) OR (((multiIf(resource.`service.type` IS NOT NULL, resource.`service.type`::String, mapContains(resources_string, 'service.type'), resources_string['service.type'], NULL) = ? AND multiIf(resource.`service.type` IS NOT NULL, resource.`service.type`::String, mapContains(resources_string, 'service.type'), resources_string['service.type'], NULL) IS NOT NULL) AND NOT ((multiIf(resource.`service.deprecated` IS NOT NULL, resource.`service.deprecated`::String, mapContains(resources_string, 'service.deprecated'), resources_string['service.deprecated'], NULL) = ? AND multiIf(resource.`service.deprecated` IS NOT NULL, resource.`service.deprecated`::String, mapContains(resources_string, 'service.deprecated'), resources_string['service.deprecated'], NULL) IS NOT NULL)))))))) AND (((((toFloat64(attributes_number['duration']) < ? AND mapContains(attributes_number, 'duration')) OR ((toFloat64(attributes_number['duration']) BETWEEN ? AND ? AND mapContains(attributes_number, 'duration'))))) AND ((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) <> ? OR (((multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) = ? AND multiIf(resource.`environment` IS NOT NULL, resource.`environment`::String, mapContains(resources_string, 'environment'), resources_string['environment'], NULL) IS NOT NULL) AND (attributes_bool['is_automated_test'] = ? AND mapContains(attributes_bool, 'is_automated_test')))))))) AND NOT ((((((LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message')) OR (LOWER(attributes_string['message']) LIKE LOWER(?) AND mapContains(attributes_string, 'message')))) AND (attributes_string['severity'] = ? AND mapContains(attributes_string, 'severity'))))))",
expectedArgs: []any{
float64(200), float64(300), float64(400), float64(500), float64(404),
"api", "web", "auth",
@@ -2468,7 +2471,8 @@ func TestFilterExprLogs(t *testing.T) {
// TestFilterExprLogs tests a comprehensive set of query patterns for logs search.
func TestFilterExprLogsConflictNegation(t *testing.T) {
fl := flaggertest.New(t)
storage := NewStorage()
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
// Define a comprehensive set of field keys to support all test cases
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
@@ -2488,11 +2492,12 @@ func TestFilterExprLogsConflictNegation(t *testing.T) {
}
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, 0, 0),
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: DefaultFullTextColumn,
}
testCases := []struct {

View File

@@ -2,7 +2,6 @@ package logstelemetryschema
import (
"context"
"github.com/SigNoz/signoz/pkg/valuer"
"testing"
"time"
@@ -51,8 +50,8 @@ func TestFilterExprSearch(t *testing.T) {
logScope := "(match(LOWER(severity_text), LOWER(?)) OR match(LOWER(trace_id), LOWER(?)) OR match(LOWER(span_id), LOWER(?)))"
resourceScope := "(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(resources_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(resources_string)))"
// the read spans both eras and yields NULL for an absent key, so it takes no presence guard
serviceNameEq := "multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ?"
serviceNameEq := "(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? " +
"AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)"
testCases := []struct {
name string
@@ -244,15 +243,19 @@ func TestFilterExprSearch(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureUseJSONBody.String(): tc.jsonBodyEnabled,
})
storage := NewStorage()
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: storage, Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, tc.startNs, tc.endNs),
FieldKeys: keys,
FullTextColumn: tc.fullTextColumn,
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: tc.fullTextColumn,
StartNs: tc.startNs,
EndNs: tc.endNs,
}
clause, err := querybuilder.PrepareWhereClause(tc.query, opts)
@@ -279,54 +282,3 @@ func TestFilterExprSearch(t *testing.T) {
})
}
}
// A search scope is a field the resource fingerprint sub-query cannot serve,
// so the main query keeps it when the split runs.
func TestFilterExprSearchResourceScopeUnderSplit(t *testing.T) {
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
resourceScope := "(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(resources_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(resources_string)))"
legacyBody := "match(LOWER(body), LOWER(?))"
testCases := []struct {
name string
query string
expectedQuery string
expectedArgs []any
}{
{
name: "resource scope alone",
query: "search('checkout', resource)",
expectedQuery: "WHERE (" + resourceScope + ")",
expectedArgs: []any{"checkout", "checkout"},
},
{
name: "resource scope in a union",
query: "search('checkout', body, resource)",
expectedQuery: "WHERE ((" + legacyBody + ") OR (" + resourceScope + "))",
expectedArgs: []any{"checkout", "checkout", "checkout"},
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{flagger.FeatureUseJSONBody.String(): false})
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
Storage: NewStorage(),
Query: querybuilder.NewQueryInfo(context.Background(), valuer.UUID{}, fl, telemetrytypes.SignalLogs, nil, uint64(releaseTime.Add(-5*time.Minute).UnixNano()), uint64(releaseTime.Add(5*time.Minute).UnixNano())),
FieldKeys: BuildCompleteFieldKeyMap(releaseTime),
FullTextColumn: DefaultFullTextColumn,
SkipResourceFilter: true,
}
clause, err := querybuilder.PrepareWhereClause(tc.query, opts)
require.NoError(t, err)
require.False(t, clause.IsEmpty())
sql, args := clause.WhereClause.BuildWithFlavor(sqlbuilder.ClickHouse)
require.Equal(t, tc.expectedQuery, sql)
require.Equal(t, tc.expectedArgs, args)
})
}
}

Some files were not shown because too many files have changed in this diff Show More