Compare commits

..

15 Commits

Author SHA1 Message Date
grandwizard28
a6f630ed8e fix(querier): drop the block comment guard now the parser handles it
The parser no longer loops on an unterminated block comment, so the hand-rolled
scan that worked around it is redundant, and it was the riskier of the two: it
duplicated the lexer's handling of string literals and could have rejected a
legitimate query. Leave the parser as the single source of truth.

The panic case likewise no longer panics, so its test can no longer cover the
recover and is removed. The recover stays as insurance, since this reaches
user-authored SQL and the parser has regressed this way before.

Cover INTERSECT and EXCEPT, which the parser only started accepting in this
version and which reach a second query through the same rules.
2026-07-28 02:21:11 +05:30
grandwizard28
928a0676e5 Merge remote-tracking branch 'origin/main' into fix/restrict-clickhouse-sql-queries 2026-07-28 02:16:49 +05:30
Pandey
44828b4185 chore(deps): bump clickhouse-sql-parser to v0.5.2 (#12303)
Some checks are pending
build-staging / js-build (push) Blocked by required conditions
build-staging / prepare (push) Waiting to run
build-staging / go-build (push) Blocked by required conditions
build-staging / staging (push) Blocked by required conditions
Release Drafter / update_release_draft (push) Waiting to run
* chore(deps): bump clickhouse-sql-parser to v0.5.2

The parser hangs in an infinite loop on an unterminated block comment and panics
on a settings clause with no value, both reachable from user-authored SQL. v0.5.2
fixes those, along with Accept and Walk missing AST children, which anything
walking the tree to inspect a query depends on.

String() on Expr is gone in favour of FormatSQL, so the callers that rendered a
node back to SQL now go through the Format helper, which formats compactly the
way String() did.

* test(querierlogs): cover the rate aggregation family

rate and rate_sum divide by the query window rather than the step, and nothing
exercised that path, so a change to how the aggregation expression is rendered
would have gone unnoticed. Assert both against the inserted logs.

The window is now passed to the request helper instead of relying on its default,
since these are the only assertions in the file that depend on it.

* test(querierlogs): match the response rounding for rate expectations

Scalar responses round floats to three significant figures below one, and the
rates are the only values in this test that do not divide evenly, so compare
against the rounded form rather than the exact quotient.

* test(queriertraces): cover the rate aggregation family

Traces derives the scalar rate window separately from logs, so a divergence
between the two would go unnoticed with only the logs case. rate_sum here is
taken over the intrinsic duration column, which also puts it on the other side
of the response rounding boundary from the logs test.
2026-07-27 20:30:16 +00:00
grandwizard28
dc28991308 fix(querier): reject unterminated block comments before parsing
The parser loops forever on an unterminated block comment, so validating a query
containing one would spin a request goroutine at full CPU instead of rejecting
it. Detect it up front, skipping comment markers that sit inside string literals.

Cover the parser panicking on a settings clause with no value in its own test,
which asserts the input still panics the parser so the case cannot quietly stop
exercising the recover.
2026-07-28 00:03:40 +05:30
grandwizard28
9df54bb39b fix(querier): reject internal databases in user-authored clickhouse sql
Reading system or information_schema exposes grants, users and server metadata,
and ClickHouse read-only mode does not prevent it, so reject any table reference
into them.

Update the dashboard variables test for the new rejection message and cover both
a statement smuggled through a variable value and a system table read.
2026-07-27 23:53:21 +05:30
grandwizard28
e1cc7fd848 fix(querier): address review on clickhouse sql validation
Rename the validator to ValidateReadOnlySelect and collapse the multi-line error
constructions onto single lines.

Carry the parse failure in the message rather than as a wrapped cause. Neither
renderer surfaces the cause: render.Error reads only the message off the base
error, and RespondError reads Error(), which returns the cause alone and drops
the message. Both now show the same text with a 400.

Add unit tests covering statement kinds, table functions nested in joins, CTEs,
subqueries and unions, and readonly overrides.
2026-07-27 23:35:11 +05:30
Abhi kumar
33a0cd043c fix(dashboard-v2): stop the table drilldown crashing on a numeric group-by (#12298)
Clicking a numeric group-by column in a V2 table panel replaced the whole
dashboard with "Something went wrong :/".

getGroupContextMenuConfig indexed QUERY_BUILDER_OPERATORS_BY_TYPES with the
group-by's `dataType` and called `.filter` on the result. That map is keyed by
the V3 data types, but a V2 panel's group-by carries the V5 `fieldDataType`,
and the backend stores every `float64` as `number` — a key the map has never
had. The lookup returned undefined, `.filter` threw during render, and the
page-level error boundary swallowed the dashboard.

The same gap made isNumberDataType miss `number`, so the filter value was sent
as a quoted string instead of a number.

Resolve the operator list through a table that knows `number`, falling back to
the string set so an unmapped type can't white-screen a panel, and treat
`number` as numeric. No query payload changes.
2026-07-27 18:02:18 +00:00
grandwizard28
295ecc4f60 fix(querier): restrict user-authored clickhouse sql to read-only selects
Validate every user-authored ClickHouse statement before it runs: exactly one
statement, SELECT only, no table functions and no readonly override. Validation
runs on the rendered statement, since substituted variable values are user input
too.

Apply it to both entry points that reach the telemetry store with user SQL, the
clickhouse_sql query type in query_range and the dashboard variables query, and
replace the substring blacklist in the latter, which ran before substitution.

Also run these statements with readonly = 2 as a backstop. The connection is
shared with write paths, so it is opt-in per query and writers never set it.
2026-07-27 23:28:15 +05:30
Swapnil Nakade
1a6a693466 refactor(cloud-integrations): remove cloud_integration_id query param (#12300) 2026-07-27 17:57:09 +00:00
Abhi kumar
3e35d1ef64 fix(dashboards-v2): new panel reads clean on open, not dirty (#12297)
The panel-editor dirty check trips a new panel the instant it opens: an
untouched, query-less new panel (e.g. Time Series) showed Save enabled and a
discard-changes prompt with no user edit.

The isDirty third clause `(isNew && draft.spec.queries.length > 0)` gates a new
panel as savable once it carries a query — List auto-seeds one, other kinds open
query-less. It read the live `draft`, but the staged-query sync (added in
f514af469e) commits the builder-seeded query into `draft.spec.queries` on open
for every kind, so the clause tripped even for a query-less panel.

Read the immutable seed `panel.spec.queries` instead — `[]` for non-List (clean
on open), a real query for List / explorer-exports (savable on open). Genuine
edits still surface via isSpecDirty / isQueryDirty.

- PanelEditorContainer test: regression case where the staged sync populated the
  draft but the seed is query-less; the container mock previously returned
  draft === panel, so this class of bug slipped past coverage.
- Query-sync integration tests: a new panel is not query-dirty on mount across
  signals and List.
2026-07-27 17:40:01 +00:00
Aditya Singh
1483fbd2c5 fix(query-builder): coerce table sort values to string to avoid localeCompare crash (#12289)
The QueryTable column sorter called `.localeCompare` on values cast with
`as string`, which is compile-time only. Upstream `processTableRowValue`
normalizes numeric-looking strings to real numbers while null becomes the
string "N/A", so a grouped column can mix numbers and text. Comparing a
number cell against a text cell skipped the numeric branch and invoked
`(200).localeCompare(...)`, throwing "localeCompare is not a function" and
crashing the table on sort.

Extract the compare logic into `compareTableColumnValues` and coerce both
sides with `String(x ?? '')` so any value type sorts safely while
preserving empty-string semantics for null/undefined.

Affects all QueryTable surfaces: Traces/Logs table views, APM Top
Operations, and dashboard Table panels.

Fixes SIGNOZ-UI-5GC
2026-07-27 16:00:29 +00:00
Vikrant Gupta
1255637e25 fix(authz): changelog column type for postgres metastore (#12299) 2026-07-27 15:52:06 +00:00
Naman Verma
38639847be fix: clamp over-wide widgets, drop zero-width widgets, and properly handle invalid order by (#12295) 2026-07-27 15:48:18 +00:00
Aditya Singh
31cb4d7520 feat(logs): unwrap lone message field from json log body (#12206)
Some checks failed
build-staging / prepare (push) Has been cancelled
build-staging / js-build (push) Has been cancelled
build-staging / go-build (push) Has been cancelled
build-staging / staging (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
2026-07-27 13:31:56 +00:00
Ashwin Bhatkal
5ede27e8fb fix(ai-assistant): page context for the V2 panel editor (edit + create) and the v2 dashboards list in the picker (#12291)
* fix(ai-assistant): add page context for the V2 panel editor route

The V2 panel editor lives on `/dashboard/:dashboardId/panel/:panelId`,
which `getAutoContexts` never matched — the three existing dashboard
matchers are all `exact`, so a 4-segment URL fell through every branch
and the assistant opened with no context chip and a page type of
`other`.

A saved panel maps to `panel_edit` with its id. The unsaved `panel/new`
route has no id to attach yet and the schema requires a non-empty
`panel_edit.widgetId`, so it degrades to the dashboard context; the
in-progress query and time range still ride along in shared metadata.

* fix(ai-assistant): use the v2 dashboards list in the context picker

`GET /api/v1/dashboards` is now a stub that returns a V1-deprecated
error, and `useGetAllDashboard` routes errors through `useErrorModal` —
so opening the picker on the Dashboards tab both showed "Failed to load
dashboards" and popped a global error modal.

Switch to `useListDashboardsForUserV2`, the same source the V2 list page
and export picker already use. `resolveAutoContextName` reads the v2
query key from cache too, so auto-context chips resolve to real
dashboard titles again instead of falling back to a generic label.

The assistant was the last live caller of the v1 list: the only other
one, `container/ListOfDashboard`, is unreachable now that the routed
`pages/DashboardsListPage` renders the V2 page.

* fix(ai-assistant): report panel_create on the unsaved panel editor

`/dashboard/:id/panel/new` previously reported `dashboard_detail`, so the
assistant was told the user was looking at a dashboard rather than
creating a panel. It now reports `panel_create`, which renders a "New
panel" chip.

`PageTypeDTO` has no `panel_create` member — `alert_new` is the
precedent it's missing — so `resolvePageType` maps it to `panel_edit`
until the backend adds one. An unmapped key would fall through to
`other` and lose the empty-state chips. The context itself carries no
`widgetId`, which the schema only requires when `metadata.page` is
literally `panel_edit`.

Also drops the `DASHBOARD_WIDGET` matcher. That V1 route is unreachable
— nothing generates a link to it since V2 always builds panel-editor
paths — and it read the `:widgetId` path segment, which V1's own page
ignored in favour of `?widgetId=`, so its `widgetId` was never right.
2026-07-27 13:29:40 +00:00
47 changed files with 1438 additions and 429 deletions

View File

@@ -9934,14 +9934,9 @@ paths:
get:
deprecated: false
description: This endpoint lists the services metadata for the specified cloud
provider
provider, without any account context.
operationId: ListServicesMetadata
parameters:
- in: query
name: cloud_integration_id
required: false
schema:
type: string
- in: path
name: cloud_provider
required: true
@@ -9991,14 +9986,10 @@ paths:
/api/v1/cloud_integrations/{cloud_provider}/services/{service_id}:
get:
deprecated: false
description: This endpoint gets a service for the specified cloud provider
description: This endpoint gets a service definition for the specified cloud
provider, without any account context.
operationId: GetService
parameters:
- in: query
name: cloud_integration_id
required: false
schema:
type: string
- in: path
name: cloud_provider
required: true

View File

@@ -36,14 +36,12 @@ import type {
GetConnectionCredentials200,
GetConnectionCredentialsPathParameters,
GetService200,
GetServiceParams,
GetServicePathParameters,
ListAccountServicesMetadata200,
ListAccountServicesMetadataPathParameters,
ListAccounts200,
ListAccountsPathParameters,
ListServicesMetadata200,
ListServicesMetadataParams,
ListServicesMetadataPathParameters,
RenderErrorResponseDTO,
UpdateAccountPathParameters,
@@ -1162,30 +1160,24 @@ export const invalidateGetConnectionCredentials = async (
};
/**
* This endpoint lists the services metadata for the specified cloud provider
* This endpoint lists the services metadata for the specified cloud provider, without any account context.
* @summary List services metadata
*/
export const listServicesMetadata = (
{ cloudProvider }: ListServicesMetadataPathParameters,
params?: ListServicesMetadataParams,
signal?: AbortSignal,
) => {
return GeneratedAPIInstance<ListServicesMetadata200>({
url: `/api/v1/cloud_integrations/${cloudProvider}/services`,
method: 'GET',
params,
signal,
});
};
export const getListServicesMetadataQueryKey = (
{ cloudProvider }: ListServicesMetadataPathParameters,
params?: ListServicesMetadataParams,
) => {
return [
`/api/v1/cloud_integrations/${cloudProvider}/services`,
...(params ? [params] : []),
] as const;
export const getListServicesMetadataQueryKey = ({
cloudProvider,
}: ListServicesMetadataPathParameters) => {
return [`/api/v1/cloud_integrations/${cloudProvider}/services`] as const;
};
export const getListServicesMetadataQueryOptions = <
@@ -1193,7 +1185,6 @@ export const getListServicesMetadataQueryOptions = <
TError = ErrorType<RenderErrorResponseDTO>,
>(
{ cloudProvider }: ListServicesMetadataPathParameters,
params?: ListServicesMetadataParams,
options?: {
query?: UseQueryOptions<
Awaited<ReturnType<typeof listServicesMetadata>>,
@@ -1205,12 +1196,11 @@ export const getListServicesMetadataQueryOptions = <
const { query: queryOptions } = options ?? {};
const queryKey =
queryOptions?.queryKey ??
getListServicesMetadataQueryKey({ cloudProvider }, params);
queryOptions?.queryKey ?? getListServicesMetadataQueryKey({ cloudProvider });
const queryFn: QueryFunction<
Awaited<ReturnType<typeof listServicesMetadata>>
> = ({ signal }) => listServicesMetadata({ cloudProvider }, params, signal);
> = ({ signal }) => listServicesMetadata({ cloudProvider }, signal);
return {
queryKey,
@@ -1238,7 +1228,6 @@ export function useListServicesMetadata<
TError = ErrorType<RenderErrorResponseDTO>,
>(
{ cloudProvider }: ListServicesMetadataPathParameters,
params?: ListServicesMetadataParams,
options?: {
query?: UseQueryOptions<
Awaited<ReturnType<typeof listServicesMetadata>>,
@@ -1249,7 +1238,6 @@ export function useListServicesMetadata<
): UseQueryResult<TData, TError> & { queryKey: QueryKey } {
const queryOptions = getListServicesMetadataQueryOptions(
{ cloudProvider },
params,
options,
);
@@ -1266,11 +1254,10 @@ export function useListServicesMetadata<
export const invalidateListServicesMetadata = async (
queryClient: QueryClient,
{ cloudProvider }: ListServicesMetadataPathParameters,
params?: ListServicesMetadataParams,
options?: InvalidateOptions,
): Promise<QueryClient> => {
await queryClient.invalidateQueries(
{ queryKey: getListServicesMetadataQueryKey({ cloudProvider }, params) },
{ queryKey: getListServicesMetadataQueryKey({ cloudProvider }) },
options,
);
@@ -1278,29 +1265,26 @@ export const invalidateListServicesMetadata = async (
};
/**
* This endpoint gets a service for the specified cloud provider
* This endpoint gets a service definition for the specified cloud provider, without any account context.
* @summary Get service
*/
export const getService = (
{ cloudProvider, serviceId }: GetServicePathParameters,
params?: GetServiceParams,
signal?: AbortSignal,
) => {
return GeneratedAPIInstance<GetService200>({
url: `/api/v1/cloud_integrations/${cloudProvider}/services/${serviceId}`,
method: 'GET',
params,
signal,
});
};
export const getGetServiceQueryKey = (
{ cloudProvider, serviceId }: GetServicePathParameters,
params?: GetServiceParams,
) => {
export const getGetServiceQueryKey = ({
cloudProvider,
serviceId,
}: GetServicePathParameters) => {
return [
`/api/v1/cloud_integrations/${cloudProvider}/services/${serviceId}`,
...(params ? [params] : []),
] as const;
};
@@ -1309,7 +1293,6 @@ export const getGetServiceQueryOptions = <
TError = ErrorType<RenderErrorResponseDTO>,
>(
{ cloudProvider, serviceId }: GetServicePathParameters,
params?: GetServiceParams,
options?: {
query?: UseQueryOptions<
Awaited<ReturnType<typeof getService>>,
@@ -1321,12 +1304,11 @@ export const getGetServiceQueryOptions = <
const { query: queryOptions } = options ?? {};
const queryKey =
queryOptions?.queryKey ??
getGetServiceQueryKey({ cloudProvider, serviceId }, params);
queryOptions?.queryKey ?? getGetServiceQueryKey({ cloudProvider, serviceId });
const queryFn: QueryFunction<Awaited<ReturnType<typeof getService>>> = ({
signal,
}) => getService({ cloudProvider, serviceId }, params, signal);
}) => getService({ cloudProvider, serviceId }, signal);
return {
queryKey,
@@ -1352,7 +1334,6 @@ export function useGetService<
TError = ErrorType<RenderErrorResponseDTO>,
>(
{ cloudProvider, serviceId }: GetServicePathParameters,
params?: GetServiceParams,
options?: {
query?: UseQueryOptions<
Awaited<ReturnType<typeof getService>>,
@@ -1363,7 +1344,6 @@ export function useGetService<
): UseQueryResult<TData, TError> & { queryKey: QueryKey } {
const queryOptions = getGetServiceQueryOptions(
{ cloudProvider, serviceId },
params,
options,
);
@@ -1380,11 +1360,10 @@ export function useGetService<
export const invalidateGetService = async (
queryClient: QueryClient,
{ cloudProvider, serviceId }: GetServicePathParameters,
params?: GetServiceParams,
options?: InvalidateOptions,
): Promise<QueryClient> => {
await queryClient.invalidateQueries(
{ queryKey: getGetServiceQueryKey({ cloudProvider, serviceId }, params) },
{ queryKey: getGetServiceQueryKey({ cloudProvider, serviceId }) },
options,
);

View File

@@ -10204,14 +10204,6 @@ export type GetConnectionCredentials200 = {
export type ListServicesMetadataPathParameters = {
cloudProvider: string;
};
export type ListServicesMetadataParams = {
/**
* @type string
* @description undefined
*/
cloud_integration_id?: string;
};
export type ListServicesMetadata200 = {
data: CloudintegrationtypesGettableServicesMetadataDTO;
/**
@@ -10224,14 +10216,6 @@ export type GetServicePathParameters = {
cloudProvider: string;
serviceId: string;
};
export type GetServiceParams = {
/**
* @type string
* @description undefined
*/
cloud_integration_id?: string;
};
export type GetService200 = {
data: CloudintegrationtypesServiceDTO;
/**

View File

@@ -380,4 +380,88 @@ describe('convertV5ResponseToLegacy', () => {
},
});
});
describe('raw logs body: extract lone `message` field', () => {
function makeRawResult(
rows: Array<{ timestamp: string; data: Record<string, any> }>,
type: 'raw' | 'trace' = 'raw',
): ReturnType<typeof convertV5ResponseToLegacy> {
const v5Data = {
type,
data: { results: [{ queryName: 'A', rows }] },
meta: { rowsScanned: 0, bytesScanned: 0, durationMs: 0, stepIntervals: {} },
} as unknown as QueryRangeResponseV5;
const params = makeBaseParams(type as RequestType, [
{
type: 'builder_query',
spec: {
name: 'A',
signal: type === 'trace' ? 'traces' : 'logs',
stepInterval: 60,
disabled: false,
aggregations: [],
},
},
]);
const input: SuccessResponse<MetricRangePayloadV5, QueryRangeRequestV5> =
makeBaseSuccess({ data: v5Data }, params);
return convertV5ResponseToLegacy(input, { A: 'A' }, false);
}
it('unwraps body when it is an object with only a message field', () => {
const result = makeRawResult([
{ timestamp: '2026-07-21T00:00:00Z', data: { body: { message: 'hello' } } },
]);
expect(result.payload.data.result[0].list?.[0]?.data?.body).toBe('hello');
});
it('leaves body unchanged when the object has keys besides message', () => {
const body = { message: 'hello', level: 'INFO' };
const result = makeRawResult([
{ timestamp: '2026-07-21T00:00:00Z', data: { body } },
]);
expect(result.payload.data.result[0].list?.[0]?.data?.body).toStrictEqual(
body,
);
});
it('leaves a string body unchanged (use_json_body off)', () => {
const result = makeRawResult([
{
timestamp: '2026-07-21T00:00:00Z',
data: { body: '{"message":"hello"}' },
},
]);
expect(result.payload.data.result[0].list?.[0]?.data?.body).toBe(
'{"message":"hello"}',
);
});
it('stringifies the nested object when message is an object', () => {
const nested = { a: 1, b: 2 };
const result = makeRawResult([
{ timestamp: '2026-07-21T00:00:00Z', data: { body: { message: nested } } },
]);
expect(result.payload.data.result[0].list?.[0]?.data?.body).toBe(
JSON.stringify(nested),
);
});
it('does not add a body key to rows without a body (traces)', () => {
const result = makeRawResult(
[{ timestamp: '2026-07-21T00:00:00Z', data: { name: 'span-1' } }],
'trace',
);
const data = (result.payload.data.result[0].list?.[0] as any)?.data ?? {};
expect('body' in data).toBe(false);
});
});
});

View File

@@ -273,6 +273,19 @@ function convertScalarWithFormatForWeb(
});
}
function extractOnlyMessageBody(body: unknown): unknown {
const isJsonBody = body && typeof body === 'object' && !Array.isArray(body);
if (isJsonBody) {
const keys = Object.keys(body);
const hasOnlyMessageKey = keys.length === 1 && keys[0] === 'message';
if (hasOnlyMessageKey) {
const { message } = body as { message: unknown };
return typeof message === 'string' ? message : JSON.stringify(message);
}
}
return body;
}
/**
* Converts V5 RawData to legacy format
*/
@@ -285,14 +298,22 @@ function convertRawData(
queryName: rawData.queryName,
legend: legendMap[rawData.queryName] || rawData.queryName,
series: null,
list: rawData.rows?.map((row) => ({
timestamp: row.timestamp,
data: {
list: rawData.rows?.map((row) => {
const data = {
// Map raw data to ILog structure - spread row.data first to include all properties
...row.data,
date: row.timestamp,
} as any,
})),
} as any;
if ('body' in row.data) {
data.body = extractOnlyMessageBody(row.data.body);
}
return {
timestamp: row.timestamp,
data,
};
}),
nextCursor: rawData.nextCursor,
};
}

View File

@@ -167,6 +167,56 @@ describe('getAutoContexts', () => {
]);
});
it('returns panel edit context on the V2 panel editor', () => {
const dashboardId = 'dash-123';
const panelId = 'panel-abc';
const pathname = ROUTES.DASHBOARD_PANEL_EDITOR.replace(
':dashboardId',
dashboardId,
).replace(':panelId', panelId);
const contexts = getAutoContexts(pathname, '');
expect(contexts).toStrictEqual([
{
source: 'auto',
type: 'dashboard',
resourceId: dashboardId,
metadata: {
page: 'panel_edit',
widgetId: panelId,
},
},
]);
});
it('returns new panel context on the unsaved new-panel editor', () => {
const dashboardId = 'dash-123';
const pathname = ROUTES.DASHBOARD_PANEL_EDITOR.replace(
':dashboardId',
dashboardId,
).replace(':panelId', 'new');
const startTime = '1700000000000';
const endTime = '1700003600000';
const contexts = getAutoContexts(
pathname,
`?panelKind=TimeSeries&${QueryParams.startTime}=${startTime}&${QueryParams.endTime}=${endTime}`,
);
expect(contexts).toStrictEqual([
{
source: 'auto',
type: 'dashboard',
resourceId: dashboardId,
metadata: {
page: 'panel_create',
timeRange: { start: Number(startTime), end: Number(endTime) },
},
},
]);
});
it('returns empty array on alert overview without ruleId', () => {
const contexts = getAutoContexts(ROUTES.ALERT_OVERVIEW, '');

View File

@@ -17,6 +17,28 @@ describe('resolvePageType', () => {
expect(resolvePageType(pathname, '')).toBe(PageTypeDTO.dashboard_detail);
});
it('returns panel_edit on the V2 panel editor', () => {
const pathname = ROUTES.DASHBOARD_PANEL_EDITOR.replace(
':dashboardId',
'dash-123',
).replace(':panelId', 'panel-abc');
expect(resolvePageType(pathname, '')).toBe(PageTypeDTO.panel_edit);
});
// `panel_create` has no PageTypeDTO counterpart yet, so it reports
// `panel_edit` rather than degrading to `other`.
it('returns panel_edit on the unsaved new-panel editor', () => {
const pathname = ROUTES.DASHBOARD_PANEL_EDITOR.replace(
':dashboardId',
'dash-123',
).replace(':panelId', 'new');
expect(resolvePageType(pathname, '?panelKind=TimeSeries')).toBe(
PageTypeDTO.panel_edit,
);
});
it('returns alerts_triggered on alert history without ruleId', () => {
expect(resolvePageType(ROUTES.ALERT_HISTORY, '')).toBe(
PageTypeDTO.alerts_triggered,

View File

@@ -16,17 +16,22 @@ import { TooltipSimple } from '@signozhq/ui/tooltip';
import type { UploadFile } from 'antd';
import getSessionStorage from 'api/browser/sessionstorage/get';
import setSessionStorage from 'api/browser/sessionstorage/set';
import {
getListDashboardsForUserV2QueryKey,
useListDashboardsForUserV2,
} from 'api/generated/services/dashboard';
import {
getListRulesQueryKey,
useListRules,
} from 'api/generated/services/rules';
import type { ListRules200 } from 'api/generated/services/sigNoz.schemas';
import type {
DashboardtypesListedDashboardForUserV2DTO,
ListDashboardsForUserV2200,
ListDashboardsForUserV2Params,
ListRules200,
} from 'api/generated/services/sigNoz.schemas';
import logEvent from 'api/common/logEvent';
import { REACT_QUERY_KEY } from 'constants/reactQueryKeys';
import { useGetAllDashboard } from 'hooks/dashboard/useGetAllDashboard';
import { useQueryService } from 'hooks/useQueryService';
import type { SuccessResponseV2 } from 'types/api';
import type { Dashboard } from 'types/api/dashboard/getAll';
// eslint-disable-next-line
import { useSelector } from 'react-redux';
import { AppState } from 'store/reducers';
@@ -100,6 +105,8 @@ function autoContextLabel(ctx: MessageContext): string {
return 'Current dashboard';
case 'panel_edit':
return 'Editing panel';
case 'panel_create':
return 'New panel';
case 'panel_fullscreen':
return 'Panel (fullscreen)';
case 'dashboard_list':
@@ -164,6 +171,18 @@ const TEXTAREA_MAX_HEIGHT_PX = 200;
const HOME_SERVICES_INTERVAL = 30 * 60 * 1000;
/** sessionStorage key for the "voice input failed this tab" flag. */
const VOICE_UNAVAILABLE_KEY = 'ai-assistant-voice-unavailable';
/**
* The picker filters client-side, so it pulls one large page instead of
* paginating. Shared with `getQueryData` below — the params are part of the
* generated query key, so both sides must use the same object.
*/
const DASHBOARD_LIST_PARAMS: ListDashboardsForUserV2Params = { limit: 1000 };
function dashboardTitle(
dashboard: DashboardtypesListedDashboardForUserV2DTO,
): string {
return dashboard.spec.display?.name || dashboard.name || 'Untitled';
}
interface SelectedContextItem {
category: ContextCategory;
@@ -716,9 +735,11 @@ export default function ChatInput({
data: dashboardsResponse,
isLoading: isDashboardsLoading,
isError: isDashboardsError,
} = useGetAllDashboard({
enabled: isContextPickerOpen && activeContextCategory === 'Dashboards',
staleTime: Infinity,
} = useListDashboardsForUserV2(DASHBOARD_LIST_PARAMS, {
query: {
enabled: isContextPickerOpen && activeContextCategory === 'Dashboards',
staleTime: Infinity,
},
});
const {
@@ -765,12 +786,12 @@ export default function ChatInput({
return ctx.resourceId;
}
if (ctx.type === 'dashboard' && ctx.resourceId) {
const cached = queryClient.getQueryData<SuccessResponseV2<Dashboard[]>>(
REACT_QUERY_KEY.GET_ALL_DASHBOARDS,
const cached = queryClient.getQueryData<ListDashboardsForUserV2200>(
getListDashboardsForUserV2QueryKey(DASHBOARD_LIST_PARAMS),
);
const dash = cached?.data?.find((d) => d.id === ctx.resourceId);
if (dash?.data.title) {
return dash.data.title;
const dash = cached?.data?.dashboards?.find((d) => d.id === ctx.resourceId);
if (dash) {
return dashboardTitle(dash);
}
}
if (ctx.type === 'alert' && ctx.resourceId) {
@@ -800,9 +821,9 @@ export default function ChatInput({
const contextEntitiesByCategory: Record<ContextCategory, ContextEntityItem[]> =
{
Dashboards:
dashboardsResponse?.data?.map((dashboard) => ({
dashboardsResponse?.data?.dashboards?.map((dashboard) => ({
id: dashboard.id,
value: dashboard.data.title ?? 'Untitled',
value: dashboardTitle(dashboard),
})) ?? [],
Alerts:
alertsResponse?.data

View File

@@ -2,12 +2,13 @@ import { render, screen, userEvent, waitFor } from 'tests/test-utils';
// The prefill flow only depends on the context-picker data hooks resolving to
// empty lists (so the empty state renders) — mock them to skip real fetches.
jest.mock('hooks/dashboard/useGetAllDashboard', () => ({
useGetAllDashboard: (): unknown => ({
data: [],
jest.mock('api/generated/services/dashboard', () => ({
useListDashboardsForUserV2: (): unknown => ({
data: undefined,
isLoading: false,
isError: false,
}),
getListDashboardsForUserV2QueryKey: (): string[] => ['dashboards'],
}));
jest.mock('api/generated/services/rules', () => ({

View File

@@ -2,6 +2,7 @@ import type { MessageContext } from 'api/ai-assistant/chat';
import { QueryParams } from 'constants/query';
import ROUTES from 'constants/routes';
import { AlertListTabs } from 'pages/AlertList/types';
import { NEW_PANEL_ID } from 'pages/DashboardPageV2/DashboardContainer/PanelEditor/newPanelRoute';
import { matchPath } from 'react-router-dom';
/**
@@ -30,22 +31,23 @@ export function getAutoContexts(
// ── Dashboards ────────────────────────────────────────────────────────────
// Widget edit (panel_edit) — `/dashboard/:dashboardId/:widgetId`.
const widgetMatch = matchPath<{ dashboardId: string; widgetId: string }>(
// Panel editor (V2). `panel/new` has no widget id yet and the schema requires
// a non-empty `panel_edit.widgetId`, so it reports `panel_create` instead.
const panelEditorMatch = matchPath<{ dashboardId: string; panelId: string }>(
pathname,
{ path: ROUTES.DASHBOARD_WIDGET, exact: true },
{ path: ROUTES.DASHBOARD_PANEL_EDITOR, exact: true },
);
if (widgetMatch) {
if (panelEditorMatch) {
const { dashboardId, panelId } = panelEditorMatch.params;
const isNewPanel = panelId === NEW_PANEL_ID;
return [
{
source: 'auto',
type: 'dashboard',
resourceId: widgetMatch.params.dashboardId,
metadata: {
page: 'panel_edit',
widgetId: widgetMatch.params.widgetId,
...sharedMetadata,
},
resourceId: dashboardId,
metadata: isNewPanel
? { page: 'panel_create', ...sharedMetadata }
: { page: 'panel_edit', widgetId: panelId, ...sharedMetadata },
},
];
}

View File

@@ -9,6 +9,8 @@ const PAGE_METADATA_TO_DTO: Record<string, PageTypeDTO> = {
dashboard_detail: PageTypeDTO.dashboard_detail,
dashboard_list: PageTypeDTO.dashboard_list,
panel_edit: PageTypeDTO.panel_edit,
// There is no panel_create, so sending panel_edit temporarily
panel_create: PageTypeDTO.panel_edit,
panel_fullscreen: PageTypeDTO.panel_fullscreen,
logs_explorer: PageTypeDTO.logs_explorer,
trace_detail: PageTypeDTO.trace_detail,

View File

@@ -147,7 +147,6 @@ function ServiceDetails({
cloudProvider: type,
serviceId: serviceId || '',
},
undefined,
{
query: {
enabled: !!serviceId && !cloudAccountId,

View File

@@ -39,9 +39,12 @@ function ServicesList({
const {
data: providerServicesMetadata,
isLoading: isProviderServicesLoading,
} = useListServicesMetadata({ cloudProvider: type }, undefined, {
query: { enabled: !isAccountsLoading && !hasConnectedAccounts },
});
} = useListServicesMetadata(
{ cloudProvider: type },
{
query: { enabled: !isAccountsLoading && !hasConnectedAccounts },
},
);
const servicesMetadata = hasConnectedAccounts
? accountServicesMetadata

View File

@@ -0,0 +1,82 @@
import { render, screen } from '@testing-library/react';
import { PANEL_TYPES } from 'constants/queryBuilder';
import { ClickedData } from 'periscope/components/ContextMenu';
import { DataTypes } from 'types/api/queryBuilder/queryAutocompleteResponse';
import { Query } from 'types/api/queryBuilder/queryBuilderData';
import { getGroupContextMenuConfig } from '../contextConfig';
const GROUP_KEY = 'http.status_code';
const makeQuery = (dataType: string): Query =>
({
builder: {
queryData: [
{
queryName: 'A',
groupBy: [{ key: GROUP_KEY, dataType, type: 'attribute' }],
},
],
queryFormulas: [],
queryTraceOperator: [],
},
promql: [],
clickhouse_sql: [],
}) as unknown as Query;
const clickedData: ClickedData = {
column: { dataIndex: GROUP_KEY },
record: { key: GROUP_KEY, timestamp: 0 },
};
const renderGroupMenu = (query: Query): void => {
const { items } = getGroupContextMenuConfig({
query,
clickedData,
panelType: PANEL_TYPES.TABLE,
onColumnClick: jest.fn(),
});
render(<div>{items}</div>);
};
describe('getGroupContextMenuConfig', () => {
// A `number` group-by used to throw: it isn't a key of QUERY_BUILDER_OPERATORS_BY_TYPES.
it('renders comparison operators for a `number` group-by column', () => {
renderGroupMenu(makeQuery('number'));
expect(screen.getByText(`Filter by ${GROUP_KEY}`)).toBeInTheDocument();
expect(screen.getByText('Is this')).toBeInTheDocument();
expect(screen.getByText('Is greater than or equal to')).toBeInTheDocument();
expect(screen.getByText('Is less than')).toBeInTheDocument();
});
it('renders only equality operators for a string group-by column', () => {
renderGroupMenu(makeQuery(DataTypes.String));
expect(screen.getByText('Is this')).toBeInTheDocument();
expect(screen.getByText('Is not this')).toBeInTheDocument();
expect(
screen.queryByText('Is greater than or equal to'),
).not.toBeInTheDocument();
});
it('falls back to equality operators when the column has no known data type', () => {
renderGroupMenu(makeQuery(''));
expect(screen.getByText('Is this')).toBeInTheDocument();
expect(
screen.queryByText('Is greater than or equal to'),
).not.toBeInTheDocument();
});
it('returns no items for a non-table panel', () => {
const config = getGroupContextMenuConfig({
query: makeQuery('number'),
clickedData,
panelType: PANEL_TYPES.TIME_SERIES,
onColumnClick: jest.fn(),
});
expect(config.items).toBeUndefined();
});
});

View File

@@ -1,8 +1,11 @@
import { DataTypes } from 'types/api/queryBuilder/queryAutocompleteResponse';
import { Query } from 'types/api/queryBuilder/queryBuilderData';
import {
getOperatorsByDataType,
getQueryData,
getViewQuery,
isNumberDataType,
isValidQueryName,
} from '../drilldownUtils';
import { METRIC_TO_LOGS_TRACES_MAPPINGS } from '../metricsCorrelationUtils';
@@ -687,4 +690,41 @@ describe('drilldownUtils', () => {
expect(expr).toContain(`name = 'GET /api'`);
});
});
describe('getOperatorsByDataType', () => {
it('gives numeric operators for the V5 `number` data type', () => {
const operators = getOperatorsByDataType('number');
expect(operators).toContain('>=');
expect(operators).toContain('<');
expect(operators).not.toContain('LIKE');
});
it('gives numeric operators for the V3 int64 / float64 data types', () => {
expect(getOperatorsByDataType(DataTypes.Int64)).toContain('>=');
expect(getOperatorsByDataType(DataTypes.Float64)).toContain('>=');
});
it('falls back to the string operators for unmapped, empty and missing types', () => {
const stringOperators = getOperatorsByDataType(DataTypes.String);
expect(getOperatorsByDataType('[]string')).toStrictEqual(stringOperators);
expect(getOperatorsByDataType('')).toStrictEqual(stringOperators);
expect(getOperatorsByDataType(undefined)).toStrictEqual(stringOperators);
});
});
describe('isNumberDataType', () => {
it.each([DataTypes.Int64, DataTypes.Float64, 'number' as DataTypes])(
'treats %s as numeric',
(dataType) => {
expect(isNumberDataType(dataType)).toBe(true);
},
);
it.each([DataTypes.String, DataTypes.bool, DataTypes.EMPTY, undefined])(
'does not treat %s as numeric',
(dataType) => {
expect(isNumberDataType(dataType)).toBe(false);
},
);
});
});

View File

@@ -0,0 +1,111 @@
import { render, screen } from '@testing-library/react';
import {
DashboardtypesQueryDTO,
Querybuildertypesv5BuilderQuerySpecDTO,
Querybuildertypesv5CompositeQueryDTO,
} from 'api/generated/services/sigNoz.schemas';
import { PANEL_TYPES } from 'constants/queryBuilder';
import {
fromPerses,
toPerses,
} from 'pages/DashboardPageV2/DashboardContainer/queryV5/persesQueryAdapters';
import { ClickedData } from 'periscope/components/ContextMenu';
import { getGroupContextMenuConfig } from '../contextConfig';
import {
addFilterToQuery,
getBaseMeta,
isNumberDataType,
} from '../drilldownUtils';
const GROUP_KEY = 'panja.pinger.status_code';
/**
* A saved V2 table panel grouped by a numeric attribute. The backend rewrites `float64` to
* `number` when it stores the panel, so `number` is what a reload actually carries.
*/
const savedPanelQueries = [
{
kind: 'signoz/scalar',
spec: {
plugin: {
kind: 'signoz/CompositeQuery',
spec: {
queries: [
{
type: 'builder_query',
spec: {
name: 'A',
signal: 'metrics',
aggregations: [{ metricName: 'probe_checks', spaceAggregation: 'sum' }],
groupBy: [
{
name: GROUP_KEY,
fieldDataType: 'number',
fieldContext: 'attribute',
},
],
filter: { expression: '' },
disabled: false,
},
},
],
},
},
},
},
] as unknown as DashboardtypesQueryDTO[];
describe('drilldown on a numeric group-by column (V2 table panel)', () => {
const v1Query = fromPerses(savedPanelQueries, PANEL_TYPES.TABLE);
const groupByDataType = getBaseMeta(v1Query, GROUP_KEY)?.dataType;
it('sees the `number` type the panel was stored with', () => {
expect(groupByDataType).toBe('number');
});
it('offers the numeric operators instead of throwing', () => {
const clickedData: ClickedData = {
column: { dataIndex: GROUP_KEY },
record: { key: GROUP_KEY, timestamp: 0 },
};
const { items } = getGroupContextMenuConfig({
query: v1Query,
clickedData,
panelType: PANEL_TYPES.TABLE,
onColumnClick: jest.fn(),
});
render(<div>{items}</div>);
expect(screen.getByText(`Filter by ${GROUP_KEY}`)).toBeInTheDocument();
expect(screen.getByText('Is this')).toBeInTheDocument();
expect(screen.getByText('Is greater than or equal to')).toBeInTheDocument();
});
it('filters on an unquoted number', () => {
// What the filter hooks branch on before coercing the clicked value.
expect(isNumberDataType(groupByDataType)).toBe(true);
const refined = addFilterToQuery(v1Query, [
{ filterKey: GROUP_KEY, filterValue: 200, operator: '=' },
]);
expect(refined.builder.queryData[0].filter?.expression).toBe(
`${GROUP_KEY} = 200`,
);
});
it('leaves the stored query shape untouched on the way back out', () => {
const [envelope] = toPerses(v1Query, PANEL_TYPES.TABLE);
const composite = envelope.spec.plugin
.spec as Querybuildertypesv5CompositeQueryDTO;
const [query] = composite.queries ?? [];
// The generated envelope union doesn't discriminate `spec` by `type`.
const spec = query?.spec as Querybuildertypesv5BuilderQuerySpecDTO;
expect(spec.groupBy?.[0]).toMatchObject({
name: GROUP_KEY,
fieldDataType: 'number',
fieldContext: 'attribute',
});
});
});

View File

@@ -1,12 +1,9 @@
import { ReactNode } from 'react';
import {
PANEL_TYPES,
QUERY_BUILDER_OPERATORS_BY_TYPES,
} from 'constants/queryBuilder';
import { PANEL_TYPES } from 'constants/queryBuilder';
import ContextMenu, { ClickedData } from 'periscope/components/ContextMenu';
import { IBuilderQuery, Query } from 'types/api/queryBuilder/queryBuilderData';
import { getBaseMeta } from './drilldownUtils';
import { getBaseMeta, getOperatorsByDataType } from './drilldownUtils';
import { SUPPORTED_OPERATORS } from './menuOptions';
import { BreakoutAttributeType } from './types';
@@ -49,15 +46,9 @@ export function getGroupContextMenuConfig({
}: Omit<ContextMenuConfigParams, 'configType'>): GroupContextMenuConfig {
const filterKey = clickedData?.column?.dataIndex;
const filterDataType =
getBaseMeta(query, filterKey as string)?.dataType || 'string';
const filterDataType = getBaseMeta(query, filterKey as string)?.dataType;
const operators =
QUERY_BUILDER_OPERATORS_BY_TYPES[
filterDataType as keyof typeof QUERY_BUILDER_OPERATORS_BY_TYPES
];
const filterOperators = operators.filter(
const filterOperators = getOperatorsByDataType(filterDataType).filter(
(operator) => SUPPORTED_OPERATORS[operator],
);

View File

@@ -3,6 +3,7 @@ import { convertFiltersToExpressionWithExistingQuery } from 'components/QueryBui
import {
initialQueryBuilderFormValuesMap,
OPERATORS,
QUERY_BUILDER_OPERATORS_BY_TYPES,
} from 'constants/queryBuilder';
import ROUTES from 'constants/routes';
import { isApmMetric } from 'container/PanelWrapper/utils';
@@ -54,13 +55,44 @@ export const getRoute = (key: string): string => {
}
};
/**
* A group-by's `dataType` comes from the field metadata, which reports a numeric as `number`
* (V5, what a saved V2 panel carries) or as `int64` / `float64` (V3) — all three are numeric.
*/
const NUMERIC_DATA_TYPES: string[] = [
DataTypes.Int64,
DataTypes.Float64,
'number',
];
export const isNumberDataType = (dataType: DataTypes | undefined): boolean => {
if (!dataType) {
return false;
}
return dataType === DataTypes.Int64 || dataType === DataTypes.Float64;
return NUMERIC_DATA_TYPES.includes(dataType);
};
/**
* `QUERY_BUILDER_OPERATORS_BY_TYPES` is keyed by the V3 data types, so the `number` a saved V2
* panel carries misses it — and a raw lookup returns `undefined`, which throws on the caller's
* `.filter`. Resolve through this table and fall back to the string set.
*/
const OPERATOR_DATA_TYPES: Record<
string,
keyof typeof QUERY_BUILDER_OPERATORS_BY_TYPES
> = {
[DataTypes.String]: 'string',
[DataTypes.Int64]: 'int64',
[DataTypes.Float64]: 'float64',
[DataTypes.bool]: 'bool',
number: 'float64',
};
export const getOperatorsByDataType = (dataType?: string): string[] =>
QUERY_BUILDER_OPERATORS_BY_TYPES[
OPERATOR_DATA_TYPES[dataType ?? ''] ?? 'string'
];
export interface FilterData {
filterKey: string;
filterValue: string | number;

View File

@@ -0,0 +1,94 @@
import {
compareTableColumnValues,
RowData,
} from '../createTableColumnsFromQuery';
// Builds a minimal RowData row. Values are intentionally loosely typed because
// real query responses can put objects/arrays into cells despite RowData's
// declared `string | number` index signature (that mismatch is the bug under test).
const row = (value: unknown, dataIndex = 'col'): RowData =>
({
timestamp: 0,
key: 'k',
[dataIndex]: value,
}) as unknown as RowData;
describe('compareTableColumnValues', () => {
it('sorts numerically when both cells are numbers', () => {
expect(compareTableColumnValues(row(1), row(2), 'col')).toBeLessThan(0);
expect(compareTableColumnValues(row(2), row(1), 'col')).toBeGreaterThan(0);
expect(compareTableColumnValues(row(5), row(5), 'col')).toBe(0);
});
it('sorts numeric-looking strings numerically, not lexically', () => {
// "10" vs "9": numeric branch => 10 - 9 > 0. A string compare would be < 0.
expect(compareTableColumnValues(row('10'), row('9'), 'col')).toBeGreaterThan(
0,
);
});
it('prefers the `<dataIndex>_without_unit` value for numeric comparison', () => {
const a = row('2 ms');
const b = row('10 ms');
a.col_without_unit = 2;
b.col_without_unit = 10;
expect(compareTableColumnValues(a, b, 'col')).toBeLessThan(0);
});
it('falls back to locale string compare for non-numeric strings', () => {
expect(compareTableColumnValues(row('abc'), row('abd'), 'col')).toBe(
'abc'.localeCompare('abd'),
);
});
it('treats null/undefined cells as empty string (no "null"/"undefined")', () => {
// Empty string sorts before a real word, and two empties are equal.
expect(compareTableColumnValues(row(null), row('a'), 'col')).toBeLessThan(0);
expect(compareTableColumnValues(row(undefined), row(undefined), 'col')).toBe(
0,
);
// If null coerced to the literal "null", this would sort after "a".
expect(
compareTableColumnValues(row(null), row('a'), 'col'),
).not.toBeGreaterThan(0);
});
it('does not throw when a cell value is an object', () => {
const a = row({ foo: 'bar' });
const b = row({ foo: 'baz' });
expect(() => compareTableColumnValues(a, b, 'col')).not.toThrow();
expect(typeof compareTableColumnValues(a, b, 'col')).toBe('number');
});
it('does not throw when a cell value is an array', () => {
const a = row([1, 2]);
const b = row([3, 4]);
expect(() => compareTableColumnValues(a, b, 'col')).not.toThrow();
// String([1,2]) === "1,2" < String([3,4]) === "3,4"
expect(compareTableColumnValues(a, b, 'col')).toBeLessThan(0);
});
it('does not throw when one cell is numeric and the other is an object', () => {
const a = row(42);
const b = row({ foo: 'bar' });
expect(() => compareTableColumnValues(a, b, 'col')).not.toThrow();
expect(typeof compareTableColumnValues(a, b, 'col')).toBe('number');
});
it('does not throw for a number cell compared against an "N/A" cell', () => {
// http.status_code column: [null, "200", ...] -> ["N/A", 200, ...]
const numberCell = row(200); // numeric string "200" becomes the number 200
const naCell = row('N/A'); // null becomes the string "N/A"
// Both orderings — antd's sorter compares pairs in both directions.
expect(() =>
compareTableColumnValues(numberCell, naCell, 'col'),
).not.toThrow();
expect(() =>
compareTableColumnValues(naCell, numberCell, 'col'),
).not.toThrow();
expect(typeof compareTableColumnValues(numberCell, naCell, 'col')).toBe(
'number',
);
});
});

View File

@@ -637,6 +637,21 @@ const generateData = (
return data;
};
export const compareTableColumnValues = (
a: RowData,
b: RowData,
dataIndex: string,
): number => {
const valueA = Number(a[`${dataIndex}_without_unit`] ?? a[dataIndex]);
const valueB = Number(b[`${dataIndex}_without_unit`] ?? b[dataIndex]);
if (!isNaN(valueA) && !isNaN(valueB)) {
return valueA - valueB;
}
return String(a[dataIndex] ?? '').localeCompare(String(b[dataIndex] ?? ''));
};
const generateTableColumns = (
dynamicColumns: DynamicColumns,
renderColumnCell?: QueryTableProps['renderColumnCell'],
@@ -650,18 +665,8 @@ const generateTableColumns = (
title: item.title,
width: QUERY_TABLE_CONFIG.width,
render: renderColumnCell && renderColumnCell[dataIndex],
sorter: (a: RowData, b: RowData): number => {
const valueA = Number(a[`${dataIndex}_without_unit`] ?? a[dataIndex]);
const valueB = Number(b[`${dataIndex}_without_unit`] ?? b[dataIndex]);
if (!isNaN(valueA) && !isNaN(valueB)) {
return valueA - valueB;
}
return ((a[dataIndex] as string) || '').localeCompare(
(b[dataIndex] as string) || '',
);
},
sorter: (a: RowData, b: RowData): number =>
compareTableColumnValues(a, b, dataIndex),
};
return [...acc, column];

View File

@@ -174,11 +174,13 @@ const baseProps = {
function setup(
panel: DashboardtypesPanelDTO,
overrides?: Partial<React.ComponentProps<typeof PanelEditorContainer>>,
draftOverrides?: { isSpecDirty?: boolean },
draftOverrides?: { isSpecDirty?: boolean; draft?: DashboardtypesPanelDTO },
): void {
// The live draft can diverge from the seed `panel`; default to the seed.
const draftPanel = draftOverrides?.draft ?? panel;
mockUseDraft.mockReturnValue({
draft: panel,
spec: panel.spec,
draft: draftPanel,
spec: draftPanel.spec,
setSpec: mockSetSpec,
isSpecDirty: draftOverrides?.isSpecDirty ?? false,
});
@@ -271,6 +273,23 @@ describe('PanelEditorContainer composition', () => {
);
});
it('keeps a query-less new panel clean after the staged-query sync seeds its draft (regression)', () => {
// Staged-query sync populated the draft with the seed; dirty must read the seed
// `panel` (query-less), not the draft, else an untouched new panel reads dirty.
const committedDraft = makePanel('signoz/TimeSeriesPanel', [
{ spec: { plugin: { kind: 'signoz/CompositeQuery', spec: {} } } },
]);
setup(
makePanel('signoz/TimeSeriesPanel'),
{ isNew: true },
{ draft: committedDraft },
);
expect(mockHeaderProps).toHaveBeenCalledWith(
expect.objectContaining({ isDirty: false }),
);
});
it('marks a new panel that already has a query saveable (e.g. list auto-runs one)', () => {
const seededQuery = {
spec: { plugin: { kind: 'signoz/BuilderQuery', spec: { signal: 'logs' } } },

View File

@@ -2,6 +2,7 @@ import { renderHook, waitFor } from '@testing-library/react';
import type {
DashboardtypesPanelDTO,
DashboardtypesQueryDTO,
TelemetrytypesSignalDTO,
} from 'api/generated/services/sigNoz.schemas';
import { QueryParams } from 'constants/query';
import { initialQueriesMap, PANEL_TYPES } from 'constants/queryBuilder';
@@ -89,6 +90,33 @@ describe('usePanelEditorQuerySync (real query builder)', () => {
expect(result.current.isQueryDirty).toBe(false);
});
it.each([
[DataSource.METRICS, PANEL_TYPES.TIME_SERIES],
[DataSource.LOGS, PANEL_TYPES.TIME_SERIES],
[DataSource.TRACES, PANEL_TYPES.TIME_SERIES],
[DataSource.LOGS, PANEL_TYPES.LIST],
])(
'a NEW %s panel (%s, no savedQueries) is NOT query-dirty on mount',
async (ds, pt) => {
const { result } = renderHook(
() =>
usePanelEditorQuerySync({
draft: makePanel([]),
panelType: pt,
setSpec: jest.fn(),
refetch: jest.fn(),
alwaysSerializeQuery: true,
signal: ds as unknown as TelemetrytypesSignalDTO,
// savedQueries omitted — new panel.
}),
{ wrapper: AllTheProviders },
);
await waitFor(() => expect(result.current.isQueryDirty).toBe(false));
expect(result.current.isQueryDirty).toBe(false);
},
);
it('an untouched panel with a minimal/older stored query is NOT dirty (drift fix)', async () => {
// An older saved query carries only a few fields; the builder re-emits many more
// (source, stepInterval, filter, spaceAggregation, …). Comparing raw would read

View File

@@ -159,11 +159,12 @@ function PanelEditorContainer({
return section?.controls;
}, [panelDefinition]);
// A new panel is savable once it has a query to run — List auto-seeds one; other
// kinds open query-less, so there's nothing to save until the user builds one.
// New panels are savable once seeded with a query (List auto-seeds one). Read the
// seed `panel`, not the live `draft` — the staged-query sync commits the seed into
// the draft on open, which would falsely dirty an untouched query-less new panel.
const isDirty = useMemo(
() => isSpecDirty || isQueryDirty || (isNew && draft.spec.queries.length > 0),
[isSpecDirty, isQueryDirty, isNew, draft.spec.queries.length],
() => isSpecDirty || isQueryDirty || (isNew && panel.spec.queries.length > 0),
[isSpecDirty, isQueryDirty, isNew, panel.spec.queries.length],
);
const isListPanel = panelKind === 'signoz/ListPanel';

2
go.mod
View File

@@ -4,7 +4,7 @@ go 1.25.7
require (
dario.cat/mergo v1.0.2
github.com/AfterShip/clickhouse-sql-parser v0.4.16
github.com/AfterShip/clickhouse-sql-parser v0.5.2
github.com/ClickHouse/clickhouse-go/v2 v2.44.0
github.com/DATA-DOG/go-sqlmock v1.5.2
github.com/SigNoz/clickhouse-go-mock v0.14.0

4
go.sum
View File

@@ -66,8 +66,8 @@ dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/AfterShip/clickhouse-sql-parser v0.4.16 h1:gpl+wXclYUKT0p4+gBq22XeRYWwEoZ9f35vogqMvkLQ=
github.com/AfterShip/clickhouse-sql-parser v0.4.16/go.mod h1:W0Z82wJWkJxz2RVun/RMwxue3g7ut47Xxl+SFqdJGus=
github.com/AfterShip/clickhouse-sql-parser v0.5.2 h1:KCxdmvuVawVF4yl0ZS33q7olgmLZwvZG5BjWHp6o414=
github.com/AfterShip/clickhouse-sql-parser v0.5.2/go.mod h1:Qi3qvPTfZb/aFwI5V4WFOahgjsLJa4MzVijIAfwOhDw=
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible h1:fcYLmCpyNYRnvJbPerq7U0hS+6+I79yEDJBqVNcqUzU=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0 h1:fou+2+WFTib47nS+nz/ozhEBnvU96bKHy6LjRsY4E28=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0/go.mod h1:t76Ruy8AHvUAC8GfMWJMa0ElSbuIcO03NLpynfbgsPA=

View File

@@ -136,9 +136,8 @@ func (provider *provider) addCloudIntegrationRoutes(router *mux.Router) error {
ID: "ListServicesMetadata",
Tags: []string{"cloudintegration"},
Summary: "List services metadata",
Description: "This endpoint lists the services metadata for the specified cloud provider",
Description: "This endpoint lists the services metadata for the specified cloud provider, without any account context.",
Request: nil,
RequestQuery: new(citypes.ListServicesMetadataParams),
RequestContentType: "",
Response: new(citypes.GettableServicesMetadata),
ResponseContentType: "application/json",
@@ -177,9 +176,8 @@ func (provider *provider) addCloudIntegrationRoutes(router *mux.Router) error {
ID: "GetService",
Tags: []string{"cloudintegration"},
Summary: "Get service",
Description: "This endpoint gets a service for the specified cloud provider",
Description: "This endpoint gets a service definition for the specified cloud provider, without any account context.",
Request: nil,
RequestQuery: new(citypes.GetServiceParams),
RequestContentType: "",
Response: new(citypes.Service),
ResponseContentType: "application/json",

View File

@@ -251,23 +251,7 @@ func (handler *handler) ListServicesMetadata(rw http.ResponseWriter, r *http.Req
return
}
queryParams := new(cloudintegrationtypes.ListServicesMetadataParams)
if err := binding.Query.BindQuery(r.URL.Query(), queryParams); err != nil {
render.Error(rw, err)
return
}
orgID := valuer.MustNewUUID(claims.OrgID)
// check if integration account exists and is not removed.
if !queryParams.CloudIntegrationID.IsZero() {
_, err := handler.module.GetConnectedAccount(ctx, orgID, queryParams.CloudIntegrationID, provider)
if err != nil {
render.Error(rw, err)
return
}
}
services, err := handler.module.ListServicesMetadata(ctx, orgID, provider, queryParams.CloudIntegrationID)
services, err := handler.module.ListServicesMetadata(ctx, valuer.MustNewUUID(claims.OrgID), provider, valuer.UUID{})
if err != nil {
render.Error(rw, err)
return
@@ -336,22 +320,7 @@ func (handler *handler) GetService(rw http.ResponseWriter, r *http.Request) {
return
}
queryParams := new(cloudintegrationtypes.GetServiceParams)
if err := binding.Query.BindQuery(r.URL.Query(), queryParams); err != nil {
render.Error(rw, err)
return
}
// check if integration account exists and is not removed.
if !queryParams.CloudIntegrationID.IsZero() {
_, err := handler.module.GetConnectedAccount(ctx, valuer.MustNewUUID(claims.OrgID), queryParams.CloudIntegrationID, provider)
if err != nil {
render.Error(rw, err)
return
}
}
svc, err := handler.module.GetService(ctx, valuer.MustNewUUID(claims.OrgID), serviceID, provider, queryParams.CloudIntegrationID)
svc, err := handler.module.GetService(ctx, valuer.MustNewUUID(claims.OrgID), serviceID, provider, valuer.UUID{})
if err != nil {
render.Error(rw, err)
return

View File

@@ -100,9 +100,22 @@ func (q *chSQLQuery) renderVars(query string, vars map[string]qbtypes.VariableIt
return newQuery.String(), nil
}
func (q *chSQLQuery) render() (string, error) {
rendered, err := q.renderVars(q.query.Query, q.vars, q.fromMS, q.toMS)
if err != nil {
return "", err
}
if err := querybuilder.ValidateReadOnlySelect(rendered); err != nil {
return "", err
}
return rendered, nil
}
// Statement renders the SQL without executing it, for the preview path.
func (q *chSQLQuery) Statement(_ context.Context) (*qbtypes.Statement, error) {
rendered, err := q.renderVars(q.query.Query, q.vars, q.fromMS, q.toMS)
rendered, err := q.render()
if err != nil {
return nil, err
}
@@ -124,11 +137,13 @@ func (q *chSQLQuery) Execute(ctx context.Context) (*qbtypes.Result, error) {
elapsed += p.Elapsed
}))
query, err := q.renderVars(q.query.Query, q.vars, q.fromMS, q.toMS)
query, err := q.render()
if err != nil {
return nil, err
}
ctx = ctxtypes.SetClickhouseReadOnly(ctx)
rows, err := q.telemetryStore.ClickhouseDB().Query(ctx, query, q.args...)
if err != nil {
return nil, err

View File

@@ -11,6 +11,7 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/modules/thirdpartyapi"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/queryparser"
"log/slog"
@@ -1021,21 +1022,6 @@ func prepareQuery(r *http.Request) (string, error) {
return "", fmt.Errorf("query is required")
}
notAllowedOps := []string{
"alter table",
"drop table",
"truncate table",
"drop database",
"drop view",
"drop function",
}
for _, op := range notAllowedOps {
if strings.Contains(strings.ToLower(query), op) {
return "", fmt.Errorf("operation %s is not allowed", op)
}
}
vars := make(map[string]string)
for k, v := range postData.Variables {
vars[k] = metrics.FormattedValue(v)
@@ -1051,30 +1037,34 @@ func prepareQuery(r *http.Request) (string, error) {
return "", tmplErr
}
if !constants.IsDotMetricsEnabled {
return queryBuf.String(), nil
}
query = queryBuf.String()
// Now handle $var replacements (simple string replace)
keys := make([]string, 0, len(vars))
for k := range vars {
keys = append(keys, k)
if constants.IsDotMetricsEnabled {
// Now handle $var replacements (simple string replace)
keys := make([]string, 0, len(vars))
for k := range vars {
keys = append(keys, k)
}
sort.Slice(keys, func(i, j int) bool {
return len(keys[i]) > len(keys[j])
})
newQuery := query
for _, k := range keys {
placeholder := "$" + k
v := vars[k]
newQuery = strings.ReplaceAll(newQuery, placeholder, v)
}
query = newQuery
}
sort.Slice(keys, func(i, j int) bool {
return len(keys[i]) > len(keys[j])
})
newQuery := query
for _, k := range keys {
placeholder := "$" + k
v := vars[k]
newQuery = strings.ReplaceAll(newQuery, placeholder, v)
if err := querybuilder.ValidateReadOnlySelect(query); err != nil {
return "", err
}
return newQuery, nil
return query, nil
}
func (aH *APIHandler) Get(rw http.ResponseWriter, r *http.Request) {
@@ -1092,7 +1082,7 @@ func (aH *APIHandler) queryDashboardVarsV2(w http.ResponseWriter, r *http.Reques
return
}
dashboardVars, err := aH.reader.QueryDashboardVars(r.Context(), query)
dashboardVars, err := aH.reader.QueryDashboardVars(ctxtypes.SetClickhouseReadOnly(r.Context()), query)
if err != nil {
RespondError(w, &model.ApiError{Typ: model.ErrorBadData, Err: err}, nil)
return

View File

@@ -67,7 +67,26 @@ func TestPrepareQuery(t *testing.T) {
Query: "ALTER TABLE signoz_table DELETE where true",
},
expectedErr: true,
errMsg: "operation alter table is not allowed",
errMsg: "only SELECT statements are allowed in ClickHouse SQL queries",
},
{
name: "query smuggles a statement through a variable",
postData: &model.DashboardVars{
Query: "select * from table where id = {{.id}}",
Variables: map[string]interface{}{
"id": "1'; DROP TABLE signoz_table; --",
},
},
expectedErr: true,
errMsg: "ClickHouse SQL must contain exactly one statement",
},
{
name: "query reads a system table",
postData: &model.DashboardVars{
Query: "SELECT name FROM system.users",
},
expectedErr: true,
errMsg: "the ClickHouse system database is not allowed in SQL queries",
},
{
name: "query text produces template exec error",

View File

@@ -96,9 +96,9 @@ func (r *aggExprRewriter) Rewrite(
}
if visitor.isRate {
return fmt.Sprintf("%s/%d", sel.SelectItems[0].String(), rateInterval), visitor.chArgs, nil
return fmt.Sprintf("%s/%d", chparser.Format(sel.SelectItems[0]), rateInterval), visitor.chArgs, nil
}
return sel.SelectItems[0].String(), visitor.chArgs, nil
return chparser.Format(sel.SelectItems[0]), visitor.chArgs, nil
}
// RewriteMulti rewrites a slice of expressions.
@@ -207,7 +207,7 @@ func (v *exprVisitor) VisitFunctionExpr(fn *chparser.FunctionExpr) error {
// Handle *If functions with predicate + values
if aggFunc.FuncCombinator {
// Map the predicate (last argument)
origPred := args[len(args)-1].String()
origPred := chparser.Format(args[len(args)-1])
whereClause, err := PrepareWhereClause(
origPred,
FilterExprVisitorOpts{
@@ -242,7 +242,7 @@ func (v *exprVisitor) VisitFunctionExpr(fn *chparser.FunctionExpr) error {
// Map each value column argument
for i := 0; i < len(args)-1; i++ {
origVal := args[i].String()
origVal := chparser.Format(args[i])
fieldKey := telemetrytypes.GetFieldKeyFromKeyText(origVal)
expr, err := v.fieldMapper.ColumnExpressionFor(v.ctx, v.orgID, v.startNs, v.endNs, &fieldKey, dataType, v.fieldKeys)
if err != nil {
@@ -259,7 +259,7 @@ func (v *exprVisitor) VisitFunctionExpr(fn *chparser.FunctionExpr) error {
} else {
// Non-If functions: map every argument as a column/value
for i, arg := range args {
orig := arg.String()
orig := chparser.Format(arg)
fieldKey := telemetrytypes.GetFieldKeyFromKeyText(orig)
expr, err := v.fieldMapper.ColumnExpressionFor(v.ctx, v.orgID, v.startNs, v.endNs, &fieldKey, dataType, v.fieldKeys)
if err != nil {

View File

@@ -0,0 +1,68 @@
package querybuilder
import (
"strings"
chparser "github.com/AfterShip/clickhouse-sql-parser/parser"
"github.com/SigNoz/signoz/pkg/errors"
)
// internalDatabases hold server metadata, credentials and grants rather than telemetry.
var internalDatabases = map[string]struct{}{
"system": {},
"information_schema": {},
}
// ValidateReadOnlySelect rejects a user-authored ClickHouse statement unless it is a
// single SELECT that reads telemetry: no table function, no internal database and no
// lowering of the readonly setting. It must run on the rendered statement, since the
// substituted variable values are user input too.
func ValidateReadOnlySelect(query string) (err error) {
// The parser has a history of panicking on malformed input rather than returning
// an error. No input is known to still do so, but this reaches user-authored SQL,
// so a regression must not take the process down.
defer func() {
if recovered := recover(); recovered != nil {
err = errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid ClickHouse SQL: %v", recovered)
}
}()
stmts, parseErr := chparser.NewParser(query).ParseStmts()
if parseErr != nil {
// The cause is carried in the message because the renderers drop the wrapped error.
return errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid ClickHouse SQL: %s", parseErr.Error())
}
if len(stmts) != 1 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "ClickHouse SQL must contain exactly one statement")
}
selectQuery, ok := stmts[0].(*chparser.SelectQuery)
if !ok {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "only SELECT statements are allowed in ClickHouse SQL queries")
}
visitor := &chparser.DefaultASTVisitor{Visit: func(node chparser.Expr) error {
switch expr := node.(type) {
case *chparser.TableFunctionExpr:
// Source table functions remain usable in ClickHouse read-only mode.
return errors.NewInvalidInputf(errors.CodeInvalidInput, "ClickHouse table functions are not allowed in SQL queries: %s", chparser.Format(expr.Name))
case *chparser.TableIdentifier:
// Reading these is unaffected by ClickHouse read-only mode.
if expr.Database == nil {
return nil
}
if _, ok := internalDatabases[strings.ToLower(expr.Database.Name)]; ok {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "the ClickHouse %s database is not allowed in SQL queries", expr.Database.Name)
}
case *chparser.SettingExpr:
// A query-level setting takes precedence over the context setting.
if strings.EqualFold(expr.Name.Name, "readonly") {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "the ClickHouse readonly setting cannot be overridden")
}
}
return nil
}}
return selectQuery.Accept(visitor)
}

View File

@@ -0,0 +1,326 @@
package querybuilder
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestValidateReadOnlySelect(t *testing.T) {
testCases := []struct {
name string
query string
pass bool
}{
{
name: "Select_Valid",
query: "SELECT region AS r, zone FROM metrics WHERE metric_name = 'cpu' GROUP BY region, zone",
pass: true,
},
{
name: "SelectWithTrailingSemicolon_Valid",
query: "SELECT count() FROM signoz_logs.distributed_logs_v2;",
pass: true,
},
{
name: "CommonTableExpression_Valid",
query: "WITH t AS (SELECT fingerprint FROM signoz_metrics.time_series_v4 GROUP BY fingerprint) SELECT * FROM t",
pass: true,
},
{
name: "Join_Valid",
query: "SELECT * FROM t1 LEFT JOIN t2 ON t1.a = t2.b",
pass: true,
},
{
name: "GlobalIn_Valid",
query: "SELECT a FROM t WHERE a GLOBAL IN (SELECT b FROM t2)",
pass: true,
},
{
name: "Union_Valid",
query: "SELECT * FROM t UNION ALL SELECT * FROM t2",
pass: true,
},
{
name: "WindowFunction_Valid",
query: "SELECT sum(v) OVER (PARTITION BY a ORDER BY t) FROM t",
pass: true,
},
{
name: "UnrelatedSetting_Valid",
query: "SELECT * FROM t SETTINGS max_threads = 4",
pass: true,
},
{
name: "Empty_Invalid",
query: "",
pass: false,
},
{
name: "Unparseable_Invalid",
query: "SELECT FROM WHERE",
pass: false,
},
{
name: "TerminatedBlockComment_Valid",
query: "SELECT /* keep me */ count() FROM t",
pass: true,
},
{
name: "BlockCommentMarkerInsideStringLiteral_Valid",
query: "SELECT count() FROM t WHERE body = '/* not a comment'",
pass: true,
},
{
// The parser used to loop forever on this. It now reads the comment to the
// end of the input, so this doubles as a canary for that regression.
name: "TrailingUnterminatedBlockComment_Valid",
query: "SELECT count() FROM t /* unterminated",
pass: true,
},
{
name: "UnterminatedBlockCommentOnly_Invalid",
query: "/* x",
pass: false,
},
{
name: "Intersect_Valid",
query: "SELECT * FROM t INTERSECT SELECT * FROM t2",
pass: true,
},
{
name: "IntersectReadingInternalDatabase_Invalid",
query: "SELECT * FROM t INTERSECT SELECT * FROM system.users",
pass: false,
},
{
name: "ExceptReadingTableFunction_Invalid",
query: "SELECT * FROM t EXCEPT SELECT * FROM url('http://x', CSV, 'a String')",
pass: false,
},
{
name: "MultipleStatements_Invalid",
query: "SELECT 1; DROP TABLE signoz_logs.logs_v2",
pass: false,
},
{
name: "Drop_Invalid",
query: "DROP TABLE signoz_logs.logs_v2",
pass: false,
},
{
name: "Insert_Invalid",
query: "INSERT INTO signoz_logs.logs_v2 SELECT * FROM signoz_logs.logs_v2",
pass: false,
},
{
name: "AlterDelete_Invalid",
query: "ALTER TABLE signoz_logs.logs_v2 DELETE WHERE 1 = 1",
pass: false,
},
{
name: "Truncate_Invalid",
query: "TRUNCATE TABLE signoz_logs.logs_v2",
pass: false,
},
{
name: "CreateTable_Invalid",
query: "CREATE TABLE evil (a Int) ENGINE = Memory",
pass: false,
},
{
name: "AttachTable_Invalid",
query: "ATTACH TABLE x",
pass: false,
},
{
name: "Optimize_Invalid",
query: "OPTIMIZE TABLE signoz_logs.logs_v2 FINAL",
pass: false,
},
{
name: "Grant_Invalid",
query: "GRANT ALL ON *.* TO admin",
pass: false,
},
{
name: "Describe_Invalid",
query: "DESCRIBE TABLE signoz_logs.logs_v2",
pass: false,
},
{
name: "Set_Invalid",
query: "SET readonly = 0",
pass: false,
},
{
name: "ShowGrants_Invalid",
query: "SHOW GRANTS",
pass: false,
},
{
name: "System_Invalid",
query: "SYSTEM SHUTDOWN",
pass: false,
},
{
name: "Kill_Invalid",
query: "KILL QUERY WHERE 1",
pass: false,
},
{
name: "IntoOutfile_Invalid",
query: "SELECT * FROM t INTO OUTFILE '/tmp/x.csv'",
pass: false,
},
{
name: "UrlTableFunction_Invalid",
query: "SELECT * FROM url('http://attacker.example/x', CSV, 'a String')",
pass: false,
},
{
name: "FileTableFunction_Invalid",
query: "SELECT * FROM file('/etc/passwd', CSV, 'a String')",
pass: false,
},
{
name: "RemoteTableFunction_Invalid",
query: "SELECT * FROM remote('attacker:9000', system.users)",
pass: false,
},
{
name: "S3TableFunction_Invalid",
query: "SELECT * FROM s3('https://x/y.csv', 'CSV')",
pass: false,
},
{
name: "MysqlTableFunction_Invalid",
query: "SELECT * FROM mysql('host:3306', 'db', 'tbl', 'u', 'p')",
pass: false,
},
{
name: "ExecutableTableFunction_Invalid",
query: "SELECT * FROM executable('script.sh', CSV, 'a String')",
pass: false,
},
{
name: "ClusterTableFunction_Invalid",
query: "SELECT * FROM cluster('c', system, users)",
pass: false,
},
{
name: "TableFunctionInJoin_Invalid",
query: "SELECT * FROM t1 JOIN url('http://x', CSV, 'a String') u ON 1 = 1",
pass: false,
},
{
name: "TableFunctionInScalarSubquery_Invalid",
query: "SELECT (SELECT * FROM url('http://x', CSV, 'a String'))",
pass: false,
},
{
name: "TableFunctionInCommonTableExpression_Invalid",
query: "WITH c AS (SELECT * FROM url('http://x', CSV, 'a String')) SELECT * FROM c",
pass: false,
},
{
name: "TableFunctionInNestedSubquery_Invalid",
query: "SELECT * FROM (SELECT * FROM (SELECT * FROM file('/etc/passwd', CSV, 'a String')))",
pass: false,
},
{
name: "TableFunctionInWhereSubquery_Invalid",
query: "SELECT * FROM t WHERE a IN (SELECT * FROM url('http://x', CSV, 'a String'))",
pass: false,
},
{
name: "TableFunctionInUnion_Invalid",
query: "SELECT * FROM t UNION ALL SELECT * FROM url('http://x', CSV, 'a String')",
pass: false,
},
{
name: "SystemUsers_Invalid",
query: "SELECT * FROM system.users",
pass: false,
},
{
name: "SystemGrants_Invalid",
query: "SELECT * FROM system.grants",
pass: false,
},
{
name: "SystemQuoted_Invalid",
query: "SELECT count() FROM `system`.`tables`",
pass: false,
},
{
name: "SystemUppercase_Invalid",
query: "SELECT * FROM SYSTEM.USERS",
pass: false,
},
{
name: "SystemInSubquery_Invalid",
query: "SELECT * FROM (SELECT name FROM system.parts)",
pass: false,
},
{
name: "SystemInCommonTableExpression_Invalid",
query: "WITH c AS (SELECT * FROM system.query_log) SELECT * FROM c",
pass: false,
},
{
name: "SystemInJoin_Invalid",
query: "SELECT * FROM signoz_logs.distributed_logs_v2 AS l JOIN system.users AS u ON 1 = 1",
pass: false,
},
{
name: "SystemInUnion_Invalid",
query: "SELECT name FROM t UNION ALL SELECT name FROM system.tables",
pass: false,
},
{
name: "InformationSchema_Invalid",
query: "SELECT * FROM information_schema.tables",
pass: false,
},
{
name: "InformationSchemaUppercase_Invalid",
query: "SELECT * FROM INFORMATION_SCHEMA.COLUMNS",
pass: false,
},
{
name: "TableNamedSystemInTelemetryDatabase_Valid",
query: "SELECT * FROM signoz_logs.system",
pass: true,
},
{
name: "ReadonlySettingOverride_Invalid",
query: "SELECT * FROM t SETTINGS readonly = 0",
pass: false,
},
{
name: "ReadonlySettingOverrideUppercase_Invalid",
query: "SELECT * FROM t SETTINGS READONLY = 0",
pass: false,
},
{
name: "ReadonlySettingOverrideAlongsideOthers_Invalid",
query: "SELECT * FROM t SETTINGS max_threads = 4, readonly = 0",
pass: false,
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
err := ValidateReadOnlySelect(testCase.query)
if testCase.pass {
assert.NoError(t, err)
return
}
assert.Error(t, err)
})
}
}

View File

@@ -316,17 +316,17 @@ func (e *ClickHouseFilterExtractor) extractColumnStrByExpr(expr clickhouse.Expr)
// FunctionExpr is a function call like "toDate(timestamp)"
case *clickhouse.FunctionExpr:
// For function expressions, return the complete function call string
return ex.String()
return clickhouse.Format(ex)
// ColumnExpr is a column expression like "m.region", "toDate(timestamp)"
case *clickhouse.ColumnExpr:
// ColumnExpr wraps another expression - extract the underlying expression
if ex.Expr != nil {
return e.extractColumnStrByExpr(ex.Expr)
}
return ex.String()
return clickhouse.Format(ex)
default:
// For other expression types, return the string representation
return expr.String()
return clickhouse.Format(expr)
}
}
@@ -524,7 +524,7 @@ func (e *ClickHouseFilterExtractor) extractFullExpression(expr clickhouse.Expr)
if expr == nil {
return ""
}
return expr.String()
return clickhouse.Format(expr)
}
// isSimpleColumnReference checks if an expression is just a simple column reference
@@ -657,7 +657,7 @@ func (e *ClickHouseFilterExtractor) extractCTEName(cte *clickhouse.CTEStmt) stri
case *clickhouse.Ident:
return name.Name
default:
return cte.Expr.String()
return clickhouse.Format(cte.Expr)
}
}

View File

@@ -103,7 +103,7 @@ func (migration *addTelemetryTuples) Up(ctx context.Context, db *bun.DB) error {
INSERT INTO changelog (store, object_type, object_id, relation, _user, operation, ulid, inserted_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (store, ulid, object_type) DO NOTHING`,
storeID, tuple.objectType, objectID, tuple.relation, user, "TUPLE_OPERATION_WRITE", tupleID, now,
storeID, tuple.objectType, objectID, tuple.relation, user, 0, tupleID, now,
)
if err != nil {
return err

View File

@@ -27,7 +27,7 @@ func (v *TelemetryFieldVisitor) VisitColumnDef(expr *parser.ColumnDef) error {
}
// Parse column name to extract context and data type
columnName := expr.Name.String()
columnName := parser.Format(expr.Name)
// Remove backticks if present
columnName = strings.TrimPrefix(columnName, "`")
@@ -75,7 +75,7 @@ func (v *TelemetryFieldVisitor) VisitColumnDef(expr *parser.ColumnDef) error {
// Extract field name from the DEFAULT expression
// The DEFAULT expression should be something like: resources_string['k8s.cluster.name']
// We need to extract the key inside the square brackets
defaultExprStr := expr.DefaultExpr.String()
defaultExprStr := parser.Format(expr.DefaultExpr)
// Look for the pattern: map['key']
startIdx := strings.Index(defaultExprStr, "['")

View File

@@ -72,6 +72,12 @@ func (h *provider) BeforeQuery(ctx context.Context, _ *telemetrystore.QueryEvent
settings["result_overflow_mode"] = ctx.Value("result_overflow_mode")
}
// readonly = 2 allows reads and per-query setting changes, but no writes, DDL or
// SYSTEM statements, and ClickHouse refuses to lower it from within the statement.
if readOnly, ok := ctx.Value(ctxtypes.ClickhouseContextReadOnlyKey).(bool); ok && readOnly {
settings["readonly"] = 2
}
// TODO(srikanthccv): enable it when the "Cannot read all data" issue is fixed
// https://github.com/ClickHouse/ClickHouse/issues/82283
settings["secondary_indices_enable_bulk_filtering"] = false

View File

@@ -44,10 +44,6 @@ type GettableServicesMetadata struct {
Services []*ServiceMetadata `json:"services" required:"true" nullable:"false"`
}
type ListServicesMetadataParams struct {
CloudIntegrationID valuer.UUID `query:"cloud_integration_id" required:"false"`
}
// Service represents a cloud integration service with its definition,
// cloud integration service is non nil only when the service entry exists in DB with ANY config (enabled or disabled).
type Service struct {
@@ -63,10 +59,6 @@ type ServiceAssets struct {
Dashboards []*ServiceDashboard `json:"dashboards" required:"true" nullable:"false"`
}
type GetServiceParams struct {
CloudIntegrationID valuer.UUID `query:"cloud_integration_id" required:"false"`
}
type UpdatableService struct {
Config *ServiceConfig `json:"config" required:"true" nullable:"false"`
}

View File

@@ -6,9 +6,17 @@ type ctxKey string
const (
ClickhouseContextMaxThreadsKey ctxKey = "clickhouse_max_threads"
ClickhouseContextReadOnlyKey ctxKey = "clickhouse_readonly"
)
// SetClickhouseMaxThreads stores the max threads value in context.
func SetClickhouseMaxThreads(ctx context.Context, maxThreads int) context.Context {
return context.WithValue(ctx, ClickhouseContextMaxThreadsKey, maxThreads)
}
// SetClickhouseReadOnly marks the context so the statement runs under a read-only
// ClickHouse session. The telemetry store connection is shared with write paths, so
// those must never set this.
func SetClickhouseReadOnly(ctx context.Context) context.Context {
return context.WithValue(ctx, ClickhouseContextReadOnlyKey, true)
}

View File

@@ -180,6 +180,11 @@ func placedWidgetIDs(data StorableDashboardData) map[string]bool {
for _, e := range layout {
if m, ok := e.(map[string]any); ok {
if i, ok := m["i"].(string); ok && i != "" {
// A zero-width placement doesn't render in the v1 UI; treat it as
// unplaced so the widget is dropped rather than migrated invisibly.
if w, ok := m["w"].(float64); ok && w <= 0 {
continue
}
ids[i] = true
}
}
@@ -320,6 +325,9 @@ func compactGridItemsVertically(items []dashboard.GridItem) {
if l.X < 0 {
l.X = 0
}
if l.X+l.Width > gridColumnCount { // still overflows (wider than the grid) → clamp width so x+width = cols
l.Width = gridColumnCount - l.X
}
if l.Y < 0 {
l.Y = 0
}

View File

@@ -3,6 +3,7 @@ package dashboardtypes
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"regexp"
"strings"
@@ -108,26 +109,12 @@ func normalizeFunctionArgs(query map[string]any) {
}
}
// malformedOrderByValueKeys are v4 order-by columnNames meaning "order by the aggregation value"
// that the v5 aggregation validator rejects (validateOrderByForAggregation). All resolve
// to the same aggregation key. Add more as they surface. The frontend passes these
// through (the query-service resolves them), but the v2 dashboard validator only accepts
// a real aggregation key.
var malformedOrderByValueKeys = map[string]bool{
"#SIGNOZ_VALUE": true,
"A": true,
"A.count()": true,
"__result": true,
"value": true,
"A.p99(duration_nano)": true,
"aws_Kafka_MessagesInPerSec_max": true,
"byte_in_count": true,
"(http_server_request_duration_ms.bucket)": true,
}
// normalizeOrderByKeys rewrites any orderBy columnName in orderByValueKeys to the
// v5-valid aggregation key. Left untouched if the key can't resolve (no aggregation to
// name).
// normalizeOrderByKeys rewrites any orderBy columnName the v5 aggregation validator
// would reject (validateOrderByForAggregation) to the canonical aggregation value key.
// v1 tolerated free-form "order by the value" aliases (#SIGNOZ_VALUE, the query name,
// the raw metric/expression) that the query-service resolved at query time; v2 accepts
// only a real order key. Anything already valid (a group-by key, an aggregation
// expression/alias/index) is left alone. No-op if no aggregation key can be named.
func normalizeOrderByKeys(query map[string]any) {
orders, ok := query["orderBy"].([]any)
if !ok {
@@ -137,17 +124,88 @@ func normalizeOrderByKeys(query map[string]any) {
if !ok {
return
}
valid := validAggregationOrderKeys(query)
for _, o := range orders {
order, ok := o.(map[string]any)
if !ok {
continue
}
if cn, _ := order["columnName"].(string); malformedOrderByValueKeys[cn] {
if cn, _ := order["columnName"].(string); cn != "" && !valid[cn] {
order["columnName"] = key
}
}
}
// validAggregationOrderKeys is a line-for-line mirror of validateOrderByForAggregation
// (querybuildertypesv5/validation.go) over the untyped query map instead of a typed
// QueryBuilderQuery. Keep the two in lockstep — every insertion here must match one
// there — so a side-by-side read makes any drift obvious. The only adaptations: fields
// are read out of maps, the type switch on the aggregation becomes a switch on the
// query signal (metrics vs logs/traces), and a not-yet-upgraded group-by still carries
// the v4 "key" instead of the v5 "name".
func validAggregationOrderKeys(query map[string]any) map[string]bool {
validOrderKeys := make(map[string]bool)
for _, gb := range asObjects(query["groupBy"]) {
name, _ := gb["name"].(string)
if name == "" {
name, _ = gb["key"].(string)
}
validOrderKeys[name] = true
}
signal := signalFromDataSource(query["dataSource"])
for i, agg := range asObjects(query["aggregations"]) {
validOrderKeys[fmt.Sprintf("%d", i)] = true
switch signal {
// TraceAggregation / LogAggregation (identical bodies in the validator).
case telemetrytypes.SignalTraces, telemetrytypes.SignalLogs:
if alias, _ := agg["alias"].(string); alias != "" {
validOrderKeys[alias] = true
}
expression, _ := agg["expression"].(string)
validOrderKeys[expression] = true
// MetricAggregation.
case telemetrytypes.SignalMetrics:
// Also allow the generic __result pattern
validOrderKeys["__result"] = true
metricName, _ := agg["metricName"].(string)
spaceRaw, _ := agg["spaceAggregation"].(string)
timeRaw, _ := agg["timeAggregation"].(string)
spaceAggregation := metrictypes.SpaceAggregation{String: valuer.NewString(spaceRaw)}
timeAggregation := metrictypes.TimeAggregation{String: valuer.NewString(timeRaw)}
validOrderKeys[fmt.Sprintf("%s(%s)", spaceAggregation.StringValue(), metricName)] = true
if timeAggregation != metrictypes.TimeAggregationUnspecified {
validOrderKeys[fmt.Sprintf("%s(%s)", timeAggregation.StringValue(), metricName)] = true
}
if timeAggregation != metrictypes.TimeAggregationUnspecified && spaceAggregation != metrictypes.SpaceAggregationUnspecified {
validOrderKeys[fmt.Sprintf("%s(%s(%s))", spaceAggregation.StringValue(), timeAggregation.StringValue(), metricName)] = true
}
}
}
return validOrderKeys
}
// asObjects returns the map elements of a []any, skipping non-object entries.
func asObjects(raw any) []map[string]any {
items, ok := raw.([]any)
if !ok {
return nil
}
out := make([]map[string]any, 0, len(items))
for _, it := range items {
if m, ok := it.(map[string]any); ok {
out = append(out, m)
}
}
return out
}
// aggregationOrderKey names the first aggregation the way validateOrderByForAggregation
// expects: "space(metricName)" for metrics, the expression for logs/traces.
func aggregationOrderKey(query map[string]any) (string, bool) {

View File

@@ -1738,6 +1738,47 @@ func TestConvertV1WidgetQueryRewritesValueOrderKeyAfterDefaultAggregation(t *tes
assert.Equal(t, "count()", spec.Order[0].Key.Name, "#SIGNOZ_VALUE resolves to the injected default aggregation")
}
// A metric query ordering by a value alias the v5 validator rejects (here the raw
// metric name, never enumerated anywhere) is rewritten to the canonical aggregation
// key, while a genuine group-by order key is left alone. Guards the allowlist
// approach: validity is derived from the query, not a hardcoded set of bad keys.
func TestConvertV1WidgetQueryRewritesUnknownMetricValueOrderKey(t *testing.T) {
widget := map[string]any{
"id": "m-1",
"panelTypes": "graph",
"query": map[string]any{
"queryType": "builder",
"builder": map[string]any{
"queryData": []any{
map[string]any{
"queryName": "A",
"expression": "A",
"dataSource": "metrics",
"aggregations": []any{map[string]any{"metricName": "http_requests_total", "spaceAggregation": "sum"}},
"groupBy": []any{map[string]any{"key": "service.name", "dataType": "string", "type": "resource"}},
"orderBy": []any{
map[string]any{"columnName": "http_requests_total", "order": "desc"},
map[string]any{"columnName": "service.name", "order": "asc"},
},
},
},
},
},
}
queries := (&v1Decoder{}).convertV1WidgetQuery(widget, PanelKindTimeSeries)
require.Len(t, queries, 1)
wrapper, ok := queries[0].Spec.Plugin.Spec.(*BuilderQuerySpec)
require.True(t, ok)
spec, ok := wrapper.Spec.(qb.QueryBuilderQuery[qb.MetricAggregation])
require.True(t, ok, "metrics query should dispatch to MetricAggregation, got %T", wrapper.Spec)
require.Len(t, spec.Order, 2)
assert.Equal(t, "sum(http_requests_total)", spec.Order[0].Key.Name, "unknown value alias rewritten to the aggregation key")
assert.Equal(t, "service.name", spec.Order[1].Key.Name, "valid group-by order key left alone")
}
func TestConvertV1WidgetQueryInjectsCountForNoopOnAggregationPanel(t *testing.T) {
// A logs query with the list-style "noop" operator placed on an aggregation
// panel (graph). createAggregationsShapeSafe drops noop, leaving no aggregation;
@@ -2084,6 +2125,26 @@ func TestConvertV1LayoutsClampsXBounds(t *testing.T) {
assert.Equal(t, 6, grid.Items[1].X) // x+w=16>12 shifted left to 12-6
}
func TestConvertV1LayoutsClampsOverwideWidth(t *testing.T) {
// A widget wider than the 12-col grid (e.g. carried over from a 24-col v1 grid)
// can't be shifted to fit, so its width is clamped so x+width = grid width —
// otherwise it overflows and v2 validation rejects it.
data := StorableDashboardData{
"widgets": []any{map[string]any{"id": "wide", "panelTypes": "graph", "query": singleLogsBuilderQuery()}},
"layout": []any{map[string]any{"i": "wide", "x": float64(0), "y": float64(0), "w": float64(24), "h": float64(6)}},
}
d := &v1Decoder{}
layouts := d.convertV1Layouts(data, d.convertV1Panels(data["widgets"]))
require.NoError(t, d.errIfHasMalformedFields())
require.Len(t, layouts, 1)
grid, ok := layouts[0].Spec.(*dashboard.GridLayoutSpec)
require.True(t, ok)
require.Len(t, grid.Items, 1)
assert.Equal(t, 0, grid.Items[0].X)
assert.Equal(t, 12, grid.Items[0].Width) // w=24 clamped to 12 so x+width = 12
}
// TestConvertV1LayoutsToleratesNonObjectPanelMap covers templates that store
// panelMap as {rowID: []widgetID} instead of the canonical {rowID: {widgets,
// collapsed}}. The frontend reads such an entry as "not collapsed" (it accesses
@@ -2152,6 +2213,33 @@ func TestConvertV1LayoutsDropsEntryForUnrenderableWidget(t *testing.T) {
assert.Equal(t, "#/spec/panels/p-1", spec.Items[0].Content.Ref)
}
func TestRetainPlacedWidgetsDropsZeroWidth(t *testing.T) {
// z-1 is placed with zero width, which the v1 UI doesn't render. It must be
// dropped entirely: no panel, and no dangling layout entry.
data := StorableDashboardData{
"widgets": []any{
map[string]any{"id": "p-1", "panelTypes": "graph", "query": singleLogsBuilderQuery()},
map[string]any{"id": "z-1", "panelTypes": "graph", "query": singleLogsBuilderQuery()},
},
"layout": []any{
map[string]any{"i": "p-1", "x": float64(0), "y": float64(0), "w": float64(6), "h": float64(6)},
map[string]any{"i": "z-1", "x": float64(6), "y": float64(0), "w": float64(0), "h": float64(6)},
},
}
d := &v1Decoder{}
panels := d.convertV1Panels(retainPlacedWidgets(data))
require.Contains(t, panels, "p-1")
require.NotContains(t, panels, "z-1", "a zero-width widget is not retained → no panel")
layouts := d.convertV1Layouts(data, panels)
require.Len(t, layouts, 1)
spec, ok := layouts[0].Spec.(*dashboard.GridLayoutSpec)
require.True(t, ok)
require.Len(t, spec.Items, 1, "the zero-width widget's layout entry is dropped too")
assert.Equal(t, "#/spec/panels/p-1", spec.Items[0].Content.Ref)
}
func TestConvertV1LayoutsDropsCollapsedChildWithNoPanel(t *testing.T) {
// A collapsed section lists a child ("ghost") that has no widget at all — a
// deleted widget still referenced in panelMap. It produces no panel and no

View File

@@ -140,49 +140,6 @@ type seriesLookup struct {
// maps a variable to its series keys, letting evaluation iterate a single
// variable's series directly.
variableToSeriesKeys map[string][]string
// seriesKey -> label set encoded as id pairs, used as the "subset" side of isSubset
// during dedup and evaluation so label comparison is an integer compare.
encodedLabels map[string][]encodedLabel
}
// encodedLabel is a label's key name and value replaced with stable integer ids by an
// encodedLabelsBuilder, so comparing two encodedLabels is an integer compare rather than
// a per-byte string compare (runtime.efaceeq/strequal/memeqbody).
type encodedLabel struct {
keyID uint32
valueID uint32
}
// encodedLabelsBuilder stores a stable uint32 id for each distinct label key name and label
// value (dictionary encoding). Populated single-threaded in buildSeriesLookup and
// read-only thereafter, so the parallel evaluation phase never writes to it.
type encodedLabelsBuilder struct {
encodedKeyIDs map[string]uint32
encodedValueIDs map[any]uint32
}
func newEncodedLabelsBuilder() *encodedLabelsBuilder {
return &encodedLabelsBuilder{encodedKeyIDs: map[string]uint32{}, encodedValueIDs: map[any]uint32{}}
}
func (b *encodedLabelsBuilder) encode(labels []*Label) []encodedLabel {
out := make([]encodedLabel, len(labels))
for i, label := range labels {
encodedKey, ok := b.encodedKeyIDs[label.Key.Name]
if !ok {
encodedKey = uint32(len(b.encodedKeyIDs)) // acts as a sequential counter.
b.encodedKeyIDs[label.Key.Name] = encodedKey
}
encodedValue, ok := b.encodedValueIDs[label.Value]
if !ok {
encodedValue = uint32(len(b.encodedValueIDs)) // acts as a sequential counter.
b.encodedValueIDs[label.Value] = encodedValue
}
out[i] = encodedLabel{keyID: encodedKey, valueID: encodedValue}
}
return out
}
// FormulaEvaluator handles formula evaluation b/w time series from different aggregations
@@ -342,7 +299,7 @@ func (fe *FormulaEvaluator) EvaluateFormula(timeSeriesData map[string]*TimeSerie
// Work per label-set is cheap enough that spawning a goroutine per item
// costs more in scheduler signaling than it saves in parallelism.
const numWorkers = 4
workCh := make(chan labelSetCandidate, len(uniqueLabelSets))
workCh := make(chan []*Label, len(uniqueLabelSets))
resultChan := make(chan *TimeSeries, len(uniqueLabelSets))
var wg sync.WaitGroup
@@ -391,12 +348,8 @@ func (fe *FormulaEvaluator) buildSeriesLookup(timeSeriesData map[string]*TimeSer
seriesMetadata: make(map[string]*TimeSeries),
variableToSeriesKeys: make(map[string][]string),
encodedLabels: make(map[string][]encodedLabel),
}
encodedLabelsBuilder := newEncodedLabelsBuilder()
for variable, aggRef := range fe.aggRefs {
// We are only interested in the time series data for the queries that are
// involved in the formula expression.
@@ -446,7 +399,6 @@ func (fe *FormulaEvaluator) buildSeriesLookup(timeSeriesData map[string]*TimeSer
if _, exists := lookup.data[seriesKey]; !exists {
lookup.data[seriesKey] = make(map[int64]float64, len(series.Values))
lookup.seriesMetadata[seriesKey] = series
lookup.encodedLabels[seriesKey] = encodedLabelsBuilder.encode(series.Labels)
lookup.variableToSeriesKeys[variable] = append(lookup.variableToSeriesKeys[variable], seriesKey)
}
@@ -509,74 +461,58 @@ func (fe *FormulaEvaluator) buildSeriesKey(variable string, seriesIndex int, lab
// The result of any expression that uses the series with `{"service": "frontend", "operation": "GET /api"}`
// and `{"service": "frontend"}` would be the series with `{"service": "frontend", "operation": "GET /api"}`
// So, we create a set of labels sets that can be termed as candidates for the final result.
func (fe *FormulaEvaluator) findUniqueLabelSets(lookup *seriesLookup) []labelSetCandidate {
// original labels (for the result series) paired with their encoded form (for the
// subset comparison).
type labelSet struct {
labels []*Label
encoded []encodedLabel
}
allLabelSets := make([]labelSet, 0, len(lookup.seriesMetadata))
for key, series := range lookup.seriesMetadata {
allLabelSets = append(allLabelSets, labelSet{labels: series.Labels, encoded: lookup.encodedLabels[key]})
func (fe *FormulaEvaluator) findUniqueLabelSets(lookup *seriesLookup) [][]*Label {
var allLabelSets [][]*Label
// Collect all label sets from series metadata
for _, series := range lookup.seriesMetadata {
allLabelSets = append(allLabelSets, series.Labels)
}
// sort the label sets by the number of labels in descending order.
slices.SortFunc(allLabelSets, func(i, j labelSet) int {
if len(i.labels) > len(j.labels) {
// sort the label sets by the number of labels in descending order
slices.SortFunc(allLabelSets, func(i, j []*Label) int {
if len(i) > len(j) {
return -1
}
if len(i.labels) < len(j.labels) {
if len(i) < len(j) {
return 1
}
return 0
})
// Find unique label sets using integer-id label comparison.
var uniqueSets []labelSetCandidate
// Find unique label sets using proper label comparison
var uniqueSets [][]*Label
var uniqueMaps []map[string]any
for _, labelSet := range allLabelSets {
isUnique := true
for _, uniqueSet := range uniqueSets {
if isSubset(uniqueSet.encodedKeyToEncodedValueMap, labelSet.encoded) {
for _, uniqueMap := range uniqueMaps {
if isSubset(uniqueMap, labelSet) {
isUnique = false
break
}
}
if isUnique {
uniqueSets = append(uniqueSets, labelSetCandidate{
labels: labelSet.labels,
encodedKeyToEncodedValueMap: encodedLabelsToMap(labelSet.encoded),
})
uniqueSets = append(uniqueSets, labelSet)
uniqueMaps = append(uniqueMaps, labelsToMap(labelSet))
}
}
return uniqueSets
}
// labelSetCandidate is a maximal label set kept by findUniqueLabelSets: the original
// labels (preserved for the result series) plus a keyID->valueID map used as the
// "superset" when matching series during evaluation.
type labelSetCandidate struct {
labels []*Label
encodedKeyToEncodedValueMap map[uint32]uint32
}
func encodedLabelsToMap(labels []encodedLabel) map[uint32]uint32 {
m := make(map[uint32]uint32, len(labels))
func labelsToMap(labels []*Label) map[string]any {
m := make(map[string]any, len(labels))
for _, label := range labels {
m[label.keyID] = label.valueID
m[label.Key.Name] = label.Value
}
return m
}
// isSubset reports whether every label in subset is present with the same value in
// supersetMap (i.e. subset ⊆ superset).
func isSubset(supersetMap map[uint32]uint32, subset []encodedLabel) bool {
func isSubset(supersetMap map[string]any, subset []*Label) bool {
for _, label := range subset {
// each key and value of string/any type in the overall label set in the whole result
// has a corresponding unique uint32 assigned to it by buildSeriesLookup, which helps us
// do a simple integer comparison in place of a much more expensive str/any comparison.
if valID, ok := supersetMap[label.keyID]; !ok || valID != label.valueID {
if val, ok := supersetMap[label.Key.Name]; !ok || val != label.Value {
return false
}
}
@@ -584,7 +520,7 @@ func isSubset(supersetMap map[uint32]uint32, subset []encodedLabel) bool {
}
// evaluateForLabelSet performs formula evaluation for a specific label set.
func (fe *FormulaEvaluator) evaluateForLabelSet(targetLabels labelSetCandidate, lookup *seriesLookup) *TimeSeries {
func (fe *FormulaEvaluator) evaluateForLabelSet(targetLabels []*Label, lookup *seriesLookup) *TimeSeries {
// Find matching series for each variable
variableData := make(map[string]map[int64]float64)
// not every series would have a value for every timestamp
@@ -592,14 +528,14 @@ func (fe *FormulaEvaluator) evaluateForLabelSet(targetLabels labelSetCandidate,
// for the variable
var allTimestamps = make(map[int64]struct{})
// target.encodedKeyToEncodedValueMap is the superset map, precomputed once in
// findUniqueLabelSets and reused across every series comparison below.
targetMap := targetLabels.encodedKeyToEncodedValueMap
// targetLabels is fixed for this call, so build its lookup once and reuse it
// across every series comparison below.
targetMap := labelsToMap(targetLabels)
for variable := range fe.aggRefs {
// only this variable's series.
for _, seriesKey := range lookup.variableToSeriesKeys[variable] {
if isSubset(targetMap, lookup.encodedLabels[seriesKey]) {
if isSubset(targetMap, lookup.seriesMetadata[seriesKey].Labels) {
if timestampData, exists := lookup.data[seriesKey]; exists {
variableData[variable] = timestampData
// Collect all timestamps
@@ -687,8 +623,8 @@ func (fe *FormulaEvaluator) evaluateForLabelSet(targetLabels labelSetCandidate,
}
// Preserve original label structure and metadata
resultLabels := make([]*Label, len(targetLabels.labels))
copy(resultLabels, targetLabels.labels)
resultLabels := make([]*Label, len(targetLabels))
copy(resultLabels, targetLabels)
return &TimeSeries{
Labels: resultLabels,

View File

@@ -69,7 +69,7 @@ func (qp *QueryProcessor) ProcessQuery(query string, transformer FilterTransform
// Reconstruct the query
var resultBuilder strings.Builder
for _, stmt := range stmts {
resultBuilder.WriteString(stmt.String())
resultBuilder.WriteString(parser.Format(stmt))
resultBuilder.WriteString(";")
}

View File

@@ -31,7 +31,7 @@ def test_list_services_without_account(
create_user_admin: types.Operation, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
) -> None:
"""List available services without specifying a cloud_integration_id."""
"""List the cloud provider's supported services"""
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = requests.get(
@@ -54,38 +54,6 @@ def test_list_services_without_account(
assert "enabled" in service, "Service should have 'enabled' field"
def test_list_services_with_account(
signoz: types.SigNoz,
create_user_admin: types.Operation, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
create_cloud_integration_account: Callable,
) -> None:
"""List services filtered to a specific account — all disabled by default."""
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
account = create_cloud_integration_account(admin_token, CLOUD_PROVIDER)
account_id = account["id"]
checkin = simulate_agent_checkin(signoz, admin_token, CLOUD_PROVIDER, account_id, str(uuid.uuid4()))
assert checkin.status_code == HTTPStatus.OK, f"Check-in failed: {checkin.text}"
response = requests.get(
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/services?cloud_integration_id={account_id}"),
headers={"Authorization": f"Bearer {admin_token}"},
timeout=10,
)
assert response.status_code == HTTPStatus.OK, f"Expected 200, got {response.status_code}"
data = response.json()["data"]
assert "services" in data, "Response should contain 'services' field"
assert len(data["services"]) > 0, "services list should be non-empty"
for svc in data["services"]:
assert "enabled" in svc, "Each service should have 'enabled' field"
assert svc["enabled"] is False, f"Service {svc['id']} should be disabled before any config is set"
EC2_SERVICE_ID = "ec2"
@@ -154,34 +122,6 @@ def test_get_service_details_without_account(
assert data["cloudIntegrationService"] is None, "cloudIntegrationService should be null without account context"
def test_get_service_details_with_account(
signoz: types.SigNoz,
create_user_admin: types.Operation, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
create_cloud_integration_account: Callable,
) -> None:
"""Get service details with account context — cloudIntegrationService is null before first UpdateService."""
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
account = create_cloud_integration_account(admin_token, CLOUD_PROVIDER)
account_id = account["id"]
checkin = simulate_agent_checkin(signoz, admin_token, CLOUD_PROVIDER, account_id, str(uuid.uuid4()))
assert checkin.status_code == HTTPStatus.OK, f"Check-in failed: {checkin.text}"
response = requests.get(
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/services/{SERVICE_ID}?cloud_integration_id={account_id}"),
headers={"Authorization": f"Bearer {admin_token}"},
timeout=10,
)
assert response.status_code == HTTPStatus.OK, f"Expected 200, got {response.status_code}"
data = response.json()["data"]
assert data["id"] == SERVICE_ID
assert data["cloudIntegrationService"] is None, "cloudIntegrationService should be null before any service config is set"
def test_get_account_service(
signoz: types.SigNoz,
create_user_admin: types.Operation, # pylint: disable=unused-argument
@@ -252,7 +192,7 @@ def test_update_service_config(
assert put_response.status_code == HTTPStatus.NO_CONTENT, f"Expected 204, got {put_response.status_code}: {put_response.text}"
get_response = requests.get(
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/services/{SERVICE_ID}?cloud_integration_id={account_id}"),
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/accounts/{account_id}/services/{SERVICE_ID}"),
headers={"Authorization": f"Bearer {admin_token}"},
timeout=10,
)
@@ -302,7 +242,7 @@ def test_update_service_config_disable(
assert r.status_code == HTTPStatus.NO_CONTENT, f"Disable failed: {r.status_code}: {r.text}"
get_response = requests.get(
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/services/{SERVICE_ID}?cloud_integration_id={account_id}"),
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/accounts/{account_id}/services/{SERVICE_ID}"),
headers={"Authorization": f"Bearer {admin_token}"},
timeout=10,
)
@@ -355,7 +295,7 @@ def test_list_services_account_removed(
get_token: Callable[[str, str], str],
create_cloud_integration_account: Callable,
) -> None:
"""List services with a cloud_integration_id for a deleted account returns 404."""
"""List services for a deleted account returns 404."""
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
account = create_cloud_integration_account(admin_token, CLOUD_PROVIDER)
@@ -372,7 +312,7 @@ def test_list_services_account_removed(
assert delete_response.status_code == HTTPStatus.NO_CONTENT, f"Expected 204 on delete, got {delete_response.status_code}"
response = requests.get(
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/services?cloud_integration_id={account_id}"),
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/accounts/{account_id}/services"),
headers={"Authorization": f"Bearer {admin_token}"},
timeout=10,
)
@@ -386,7 +326,7 @@ def test_get_service_details_account_removed(
get_token: Callable[[str, str], str],
create_cloud_integration_account: Callable,
) -> None:
"""Get service details with a cloud_integration_id for a deleted account returns 404."""
"""Get service details for a deleted account returns 404."""
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
account = create_cloud_integration_account(admin_token, CLOUD_PROVIDER)
@@ -403,7 +343,7 @@ def test_get_service_details_account_removed(
assert delete_response.status_code == HTTPStatus.NO_CONTENT, f"Expected 204 on delete, got {delete_response.status_code}"
response = requests.get(
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/services/{SERVICE_ID}?cloud_integration_id={account_id}"),
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/accounts/{account_id}/services/{SERVICE_ID}"),
headers={"Authorization": f"Bearer {admin_token}"},
timeout=10,
)
@@ -468,7 +408,7 @@ def test_enable_metrics_provisions_dashboards(
# Assertion 1: GetService returns provisioned dashboard UUIDs
get_svc_response = requests.get(
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/services/{SERVICE_ID}?cloud_integration_id={account_id}"),
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/accounts/{account_id}/services/{SERVICE_ID}"),
headers={"Authorization": f"Bearer {admin_token}"},
timeout=10,
)
@@ -533,7 +473,7 @@ def test_disable_metrics_deprovisions_dashboards(
# Capture the provisioned dashboard IDs before disabling
get_svc_response = requests.get(
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/services/{SERVICE_ID}?cloud_integration_id={account_id}"),
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/accounts/{account_id}/services/{SERVICE_ID}"),
headers={"Authorization": f"Bearer {admin_token}"},
timeout=10,
)
@@ -552,7 +492,7 @@ def test_disable_metrics_deprovisions_dashboards(
# Assertion 1: GetService no longer returns UUID dashboard IDs
get_svc_after = requests.get(
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/services/{SERVICE_ID}?cloud_integration_id={account_id}"),
signoz.self.host_configs["8080"].get(f"/api/v1/cloud_integrations/{CLOUD_PROVIDER}/accounts/{account_id}/services/{SERVICE_ID}"),
headers={"Authorization": f"Bearer {admin_token}"},
timeout=10,
)

View File

@@ -20,12 +20,15 @@ from fixtures.querier import (
# Numeric aggregation-function coverage for logs — the logs counterpart of
# queriertraces/02_aggregation.py::test_traces_aggregate_functions. Logs querier
# tests elsewhere only ever use count()/count_distinct(); here a grouped scalar
# query computes sum / avg / min / max / p50 / p90 / p95 / p99 / countIf /
# count_distinct over a numeric log attribute and asserts every value.
# query computes sum / avg / min / max / p50 / p90 / p95 / p99 / countIf / rate /
# rate_sum / count_distinct over a numeric log attribute and asserts every value.
#
# Log number attributes are stored as Float64, so aggregates come back as floats;
# percentiles are matched with pytest.approx (ClickHouse quantile() and
# numpy.percentile both linear-interpolate, but avoid ULP-level exact equality).
#
# The rate family divides by a window the caller does not otherwise see, so the
# lookback is passed explicitly instead of taken from the request helper default.
def test_logs_aggregate_functions(
@@ -41,11 +44,14 @@ def test_logs_aggregate_functions(
Tests:
A grouped scalar query computes count / sum / avg / min / max / p50 / p90 /
p95 / p99 over latency_ms, countIf over a numeric threshold, and
count_distinct over a string attribute — all matching values derived from the
inserted logs, ordered by count() desc.
p95 / p99 over latency_ms, countIf over a numeric threshold, rate and
rate_sum over the query window, and count_distinct over a string attribute —
all matching values derived from the inserted logs, ordered by count() desc.
"""
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
# A scalar rate divides by the whole query window, so both sides agree on it.
lookback_minutes = 5
rate_interval_seconds = lookback_minutes * 60
# (service, latency_ms, endpoint)
specs = [
("svc-a", 10, "/x"),
@@ -80,12 +86,14 @@ def test_logs_aggregate_functions(
build_aggregation("p95(latency_ms)", "p95_l"),
build_aggregation("p99(latency_ms)", "p99_l"),
build_aggregation("countIf(latency_ms >= 25)", "slow"),
build_aggregation("rate()", "rate_all"),
build_aggregation("rate_sum(latency_ms)", "rate_sum_l"),
build_aggregation("count_distinct(endpoint)", "endpoints"),
],
group_by=[build_group_by_field("service.name", "string", "resource")],
order=[build_order_by("count()", "desc")],
)
response = make_scalar_query_request(signoz, token, now, [query])
response = make_scalar_query_request(signoz, token, now, [query], lookback_minutes=lookback_minutes)
assert response.status_code == HTTPStatus.OK, response.text
data = get_scalar_table_data(response.json())
@@ -109,6 +117,12 @@ def test_logs_aggregate_functions(
float(np.percentile(latencies, 95)), # p95(latency_ms)
float(np.percentile(latencies, 99)), # p99(latency_ms)
sum(1 for latency in latencies if latency >= 25), # countIf(latency_ms >= 25)
# Response floats are rounded to 3 decimals at or above 1 and to 3
# significant figures below it (roundToNonZeroDecimals). Every other
# value here is exact; the rates are the only ones that repeat, and
# they stay below 1 for this fixture, so mirror the latter form.
float(f"{len(latencies) / rate_interval_seconds:.3g}"), # rate()
float(f"{sum(latencies) / rate_interval_seconds:.3g}"), # rate_sum(latency_ms)
len(set(endpoints)), # count_distinct(endpoint)
]
assert by_service[service] == pytest.approx(expected), f"{service}: {by_service[service]} != {expected}"

View File

@@ -150,12 +150,17 @@ def test_traces_aggregate_functions(
Tests:
A grouped scalar query computes count / sum / avg / min / max / p50 / p90 over
duration_nano, countIf over the intrinsic status_code and the calculated
response_status_code, and avg over a numeric attribute — all matching values
derived from the inserted spans. Under the corrupt variant the same-named
colliding attributes must not change any of these.
response_status_code, rate and rate_sum over the query window, and avg over a
numeric attribute — all matching values derived from the inserted spans. Under
the corrupt variant the same-named colliding attributes must not change any of
these.
"""
extra_attrs, extra_resources = trace_noise(noise)
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
# A scalar rate divides by the whole query window, so both sides agree on it.
# Traces derives this separately from logs (telemetrytraces/statement_builder.go).
lookback_minutes = 5
rate_interval_seconds = lookback_minutes * 60
def mk(service: str, dur_s: float, status: TracesStatusCode, rsc: str, latency: float) -> Traces:
return Traces(
@@ -189,6 +194,8 @@ def test_traces_aggregate_functions(
build_aggregation("p50(duration_nano)", "p50_d"),
build_aggregation("p90(duration_nano)", "p90_d"),
build_aggregation("countIf(status_code = 2)", "errs"),
build_aggregation("rate()", "rate_all"),
build_aggregation("rate_sum(duration_nano)", "rate_sum_d"),
build_aggregation("avg(latency_ms)", "avg_lat"),
]
query = build_traces_scalar_query(
@@ -196,7 +203,7 @@ def test_traces_aggregate_functions(
group_by=[build_group_by_field("service.name", "string", "resource")],
order=[build_order_by("count()", "desc")],
)
response = make_scalar_query_request(signoz, token, now, [query])
response = make_scalar_query_request(signoz, token, now, [query], lookback_minutes=lookback_minutes)
assert response.status_code == HTTPStatus.OK, response.text
data = get_scalar_table_data(response.json())
@@ -214,6 +221,11 @@ def test_traces_aggregate_functions(
float(np.percentile(durations, 50)), # p50(duration_nano)
float(np.percentile(durations, 90)), # p90(duration_nano)
sum(1 for s in group if int(s.status_code) == 2), # countIf(status_code = 2)
# Response floats are rounded to 3 significant figures below 1 and to 3
# decimals at or above it (roundToNonZeroDecimals). The two rates land on
# either side of that boundary, so each mirrors its own form.
float(f"{len(group) / rate_interval_seconds:.3g}"), # rate()
round(sum(durations) / rate_interval_seconds, 3), # rate_sum(duration_nano)
sum(latencies) / len(latencies), # avg(latency_ms)
)