Compare commits

..

29 Commits

Author SHA1 Message Date
Pandey
875c0c336d Merge branch 'main' into platform-pod/issues/1934 2026-04-06 22:56:01 +05:30
Srikanth Chekuri
20c338b643 Merge branch 'main' into platform-pod/issues/1934 2026-04-05 16:55:46 +05:30
grandwizard28
32ec41e080 style: formatting fix in tables file 2026-04-04 19:54:34 +05:30
grandwizard28
db535b8df7 style(audit): remove module docstring from integration test 2026-04-04 19:16:02 +05:30
grandwizard28
0c633ae829 fix(audit): remove silent empty-string guards in metadata store
Remove guards that silently returned nil/empty when audit DB params
were empty. All call sites now pass real constants, so misconfiguration
should fail loudly rather than produce silent empty results.
2026-04-04 19:14:16 +05:30
grandwizard28
47d6d234af refactor(audit): inline all data setup, use distinct scenarios per test
Remove _insert_standard_audit_events helper. Each test now owns its
data: list_all uses alert-rule/saved-view/user resource types,
scalar_count uses multiple failures from different principals (count=2),
leak test uses a single organization event. Parametrized filter tests
keep the original 5-event dataset.
2026-04-04 19:02:52 +05:30
grandwizard28
80e03eb053 refactor(audit): inline DEFAULT_ORDER using build_order_by
Use build_order_by from querier fixtures instead of OrderBy/
TelemetryFieldKey dataclasses. Allow BuilderQuery.order to accept
plain dicts alongside OrderBy objects.
2026-04-04 18:59:43 +05:30
grandwizard28
b24334ecce style(audit): use snake_case for pytest parametrize IDs 2026-04-04 18:51:34 +05:30
grandwizard28
dfbba7cada refactor(audit): inline _time_window into test functions 2026-04-04 18:49:18 +05:30
grandwizard28
04a80d52d7 refactor(audit): remove wrapper helpers, inline make_query_request calls
Remove _query_audit_raw and _query_audit_scalar helpers. Use
make_query_request, BuilderQuery, and build_scalar_query directly.
Compute time window at test execution time via _time_window() to
avoid stale module-level timestamps.
2026-04-04 18:47:16 +05:30
grandwizard28
2972bb2cda refactor(audit): reuse querier fixtures in integration tests
Add source param to BuilderQuery and build_scalar_query in the
querier fixture. Replace custom _build_audit_query and
_build_audit_ts_query helpers with BuilderQuery and
build_scalar_query from the shared fixtures.
2026-04-04 18:43:25 +05:30
grandwizard28
837fdf2873 fix(audit): add source to resource filter for correct metadata routing
Add source param to telemetryresourcefilter.New so the resource
filter's key selectors include Source when calling GetKeysMulti.
Without this, audit resource keys route to signoz_logs metadata
tables instead of signoz_audit. Fix scalar test to use table
response format (columns+data, not rows).
2026-04-04 17:45:41 +05:30
grandwizard28
34adaab599 refactor(audit): use pytest.mark.parametrize for filter integration tests
Consolidate filter test functions into a single parametrized test.
6/8 tests passing; resource kind+ID filter and scalar count need
further investigation (resource filter JSON key extraction with
dotted keys, scalar response format).
2026-04-04 17:28:29 +05:30
grandwizard28
fbf40d8f70 fix(audit): rename resource.name to resource.kind, move to resource attributes
Align with schema change from SigNoz/signoz#10826:
- signoz.audit.resource.name renamed to signoz.audit.resource.kind
- resource.kind and resource.id moved from event attributes to OTel
  Resource attributes (resource JSON column)
- Materialized columns reduced from 7 to 5 (resource.kind and
  resource.id no longer materialized)
2026-04-04 17:18:00 +05:30
grandwizard28
ee07cd4fa7 feat(audit): internalize resource filter in audit statement builder
Build the resource filter internally pointing at
signoz_audit.distributed_logs_resource. Add LogsResourceTableName
constant. Remove resourceFilterStmtBuilder from constructor params.
Update test expectations to use the audit resource table.
2026-04-04 16:26:47 +05:30
grandwizard28
6b42a249cc fix(audit): remove debug assertion from integration test 2026-04-04 16:26:47 +05:30
grandwizard28
a2d09e2cf3 fix(audit): revert sb.As in getAuditKeys, fix fixture column_names
Revert getAuditKeys to use raw SQL strings instead of sb.As() which
incorrectly treated string literals as column references. Add explicit
column_names to all ClickHouse insert calls in the audit fixture.
2026-04-04 16:26:47 +05:30
grandwizard28
c7e1b37f4b test(audit): add integration tests for audit log querying
Cover the documented query patterns: list all events, filter by
principal ID, filter by outcome, filter by resource name+ID,
filter by principal type, scalar count for alerting, and
isolation test ensuring audit data doesn't leak into regular logs.
2026-04-04 16:26:46 +05:30
grandwizard28
5bdbc23180 feat(audit): wire audit statement builder into querier
Add auditStmtBuilder to querier struct and route LogAggregation
queries with source=audit to it in all three dispatch locations
(main query, live tail, shiftedQuery). Create and wire the full
audit query stack in signozquerier provider.
2026-04-04 16:26:46 +05:30
grandwizard28
db9ad4b0a6 fix(audit): rename tag_attributes_v2 to tag_attributes
Migration uses tag_attributes/distributed_tag_attributes (no _v2
suffix). Rename constants and update all references including the
integration test fixture.
2026-04-04 16:26:46 +05:30
grandwizard28
a3eb7013b3 feat(audit): add integration test fixture for audit logs
AuditLog fixture inserts into all 5 signoz_audit tables matching
the schema migration DDL: distributed_logs (no resources_string,
has event_name), distributed_logs_resource, distributed_tag_attributes_v2,
distributed_logs_attribute_keys, distributed_logs_resource_keys.
2026-04-04 16:26:46 +05:30
grandwizard28
409576c12a fix(audit): align table names with schema migration
Migration uses logs/distributed_logs (not logs_v2/distributed_logs_v2).
Rename LogsV2TableName to LogsTableName and LogsV2LocalTableName to
LogsLocalTableName to match the actual signoz_audit DDL.
2026-04-04 16:26:46 +05:30
grandwizard28
3d9a8bae04 style(audit): move column map to const.go, use sqlbuilder.As in metadata
Move logsV2Columns from field_mapper.go to const.go to colocate all
column definitions. Switch getAuditKeys() to use sb.As() instead of
raw string formatting. Fix FieldContext alignment.
2026-04-04 16:26:46 +05:30
grandwizard28
1bc0ad5a88 refactor(audit): inline field key map into test file
Remove test_data.go and inline the audit field key map directly
into statement_builder_test.go with a compact helper function.
2026-04-04 16:26:34 +05:30
grandwizard28
74ae9baf81 test(audit): add statement builder tests
Cover all three request types (list, time series, scalar) with
audit-specific query patterns: materialized column filters, AND/OR
conditions, limit CTEs, and group-by expressions.
2026-04-04 16:26:33 +05:30
grandwizard28
2c493a5bcc feat(audit): add audit field value autocomplete support
Wire distributed_tag_attributes_v2 for signoz_audit into the
metadata store. Add getAuditFieldValues() and route SignalLogs +
SourceAudit to it in GetFieldValues().
2026-04-04 16:26:33 +05:30
grandwizard28
6514f8f852 fix(audit): align field mapper with actual audit DDL schema
Remove resources_string (not in audit table DDL).
Add event_name as intrinsic column.
Resource context resolves only through the resource JSON column.
2026-04-04 16:26:33 +05:30
grandwizard28
8467b6c268 chore: address review comments
Comment out SourceAudit from Enum() until frontend is ready.
Use actual audit table constants in metadata test helpers.
2026-04-04 16:26:33 +05:30
grandwizard28
7c686ecc90 feat(audit): add telemetry audit query infrastructure
Add pkg/telemetryaudit/ with tables, field mapper, condition builder,
and statement builder for querying audit logs from signoz_audit database.
Add SourceAudit to source enum and integrate audit key resolution
into the metadata store.
2026-04-04 16:26:33 +05:30
46 changed files with 3298 additions and 731 deletions

View File

@@ -1,4 +1,4 @@
import { ReactChild, useCallback, useMemo } from 'react';
import { ReactChild, useCallback, useEffect, useMemo, useState } from 'react';
import { matchPath, Redirect, useLocation } from 'react-router-dom';
import getLocalStorageApi from 'api/browser/localstorage/get';
import setLocalStorageApi from 'api/browser/localstorage/set';
@@ -8,10 +8,12 @@ import { LOCALSTORAGE } from 'constants/localStorage';
import { ORG_PREFERENCES } from 'constants/orgPreferences';
import ROUTES from 'constants/routes';
import { useGetTenantLicense } from 'hooks/useGetTenantLicense';
import history from 'lib/history';
import { isEmpty } from 'lodash-es';
import { useAppContext } from 'providers/App/App';
import { LicensePlatform, LicenseState } from 'types/api/licensesV3/getActive';
import { OrgPreference } from 'types/api/preferences/preference';
import { Organization } from 'types/api/user/getOrganization';
import { USER_ROLES } from 'types/roles';
import { routePermission } from 'utils/permission';
@@ -23,7 +25,6 @@ import routes, {
SUPPORT_ROUTE,
} from './routes';
// eslint-disable-next-line sonarjs/cognitive-complexity
function PrivateRoute({ children }: PrivateRouteProps): JSX.Element {
const location = useLocation();
const { pathname } = location;
@@ -56,12 +57,7 @@ function PrivateRoute({ children }: PrivateRouteProps): JSX.Element {
const currentRoute = mapRoutes.get('current');
const { isCloudUser: isCloudUserVal } = useGetTenantLicense();
const orgData = useMemo(() => {
if (org && org.length > 0 && org[0].id !== undefined) {
return org[0];
}
return undefined;
}, [org]);
const [orgData, setOrgData] = useState<Organization | undefined>(undefined);
const { data: usersData, isFetching: isFetchingUsers } = useListUsers({
query: {
@@ -79,7 +75,214 @@ function PrivateRoute({ children }: PrivateRouteProps): JSX.Element {
return remainingUsers.length === 1;
}, [usersData?.data]);
// Handle old routes - redirect to new routes
useEffect(() => {
if (
isCloudUserVal &&
!isFetchingOrgPreferences &&
orgPreferences &&
!isFetchingUsers &&
usersData &&
usersData.data
) {
const isOnboardingComplete = orgPreferences?.find(
(preference: OrgPreference) =>
preference.name === ORG_PREFERENCES.ORG_ONBOARDING,
)?.value;
// Don't redirect to onboarding if workspace has issues (blocked, suspended, or restricted)
// User needs access to settings/billing to fix payment issues
const isWorkspaceBlocked = trialInfo?.workSpaceBlock;
const isWorkspaceSuspended = activeLicense?.state === LicenseState.DEFAULTED;
const isWorkspaceAccessRestricted =
activeLicense?.state === LicenseState.TERMINATED ||
activeLicense?.state === LicenseState.EXPIRED ||
activeLicense?.state === LicenseState.CANCELLED;
const hasWorkspaceIssue =
isWorkspaceBlocked || isWorkspaceSuspended || isWorkspaceAccessRestricted;
if (hasWorkspaceIssue) {
return;
}
const isFirstUser = checkFirstTimeUser();
if (
isFirstUser &&
!isOnboardingComplete &&
// if the current route is allowed to be overriden by org onboarding then only do the same
!ROUTES_NOT_TO_BE_OVERRIDEN.includes(pathname)
) {
history.push(ROUTES.ONBOARDING);
}
}
}, [
checkFirstTimeUser,
isCloudUserVal,
isFetchingOrgPreferences,
isFetchingUsers,
orgPreferences,
usersData,
pathname,
trialInfo?.workSpaceBlock,
activeLicense?.state,
]);
const navigateToWorkSpaceBlocked = useCallback((): void => {
const isRouteEnabledForWorkspaceBlockedState =
isAdmin &&
(pathname === ROUTES.SETTINGS ||
pathname === ROUTES.ORG_SETTINGS ||
pathname === ROUTES.MEMBERS_SETTINGS ||
pathname === ROUTES.BILLING ||
pathname === ROUTES.MY_SETTINGS);
if (
pathname &&
pathname !== ROUTES.WORKSPACE_LOCKED &&
!isRouteEnabledForWorkspaceBlockedState
) {
history.push(ROUTES.WORKSPACE_LOCKED);
}
}, [isAdmin, pathname]);
const navigateToWorkSpaceAccessRestricted = useCallback((): void => {
if (pathname && pathname !== ROUTES.WORKSPACE_ACCESS_RESTRICTED) {
history.push(ROUTES.WORKSPACE_ACCESS_RESTRICTED);
}
}, [pathname]);
useEffect(() => {
if (!isFetchingActiveLicense && activeLicense) {
const isTerminated = activeLicense.state === LicenseState.TERMINATED;
const isExpired = activeLicense.state === LicenseState.EXPIRED;
const isCancelled = activeLicense.state === LicenseState.CANCELLED;
const isWorkspaceAccessRestricted = isTerminated || isExpired || isCancelled;
const { platform } = activeLicense;
if (isWorkspaceAccessRestricted && platform === LicensePlatform.CLOUD) {
navigateToWorkSpaceAccessRestricted();
}
}
}, [
isFetchingActiveLicense,
activeLicense,
navigateToWorkSpaceAccessRestricted,
]);
useEffect(() => {
if (!isFetchingActiveLicense) {
const shouldBlockWorkspace = trialInfo?.workSpaceBlock;
if (
shouldBlockWorkspace &&
activeLicense?.platform === LicensePlatform.CLOUD
) {
navigateToWorkSpaceBlocked();
}
}
}, [
isFetchingActiveLicense,
trialInfo?.workSpaceBlock,
activeLicense?.platform,
navigateToWorkSpaceBlocked,
]);
const navigateToWorkSpaceSuspended = useCallback((): void => {
if (pathname && pathname !== ROUTES.WORKSPACE_SUSPENDED) {
history.push(ROUTES.WORKSPACE_SUSPENDED);
}
}, [pathname]);
useEffect(() => {
if (!isFetchingActiveLicense && activeLicense) {
const shouldSuspendWorkspace =
activeLicense.state === LicenseState.DEFAULTED;
if (
shouldSuspendWorkspace &&
activeLicense.platform === LicensePlatform.CLOUD
) {
navigateToWorkSpaceSuspended();
}
}
}, [isFetchingActiveLicense, activeLicense, navigateToWorkSpaceSuspended]);
useEffect(() => {
if (org && org.length > 0 && org[0].id !== undefined) {
setOrgData(org[0]);
}
}, [org]);
// if the feature flag is enabled and the current route is /get-started then redirect to /get-started-with-signoz-cloud
useEffect(() => {
if (
currentRoute?.path === ROUTES.GET_STARTED &&
featureFlags?.find((e) => e.name === FeatureKeys.ONBOARDING_V3)?.active
) {
history.push(ROUTES.GET_STARTED_WITH_CLOUD);
}
}, [currentRoute, featureFlags]);
// eslint-disable-next-line sonarjs/cognitive-complexity
useEffect(() => {
// if it is an old route navigate to the new route
if (isOldRoute) {
// this will be handled by the redirect component below
return;
}
// if the current route is public dashboard then don't redirect to login
const isPublicDashboard = currentRoute?.path === ROUTES.PUBLIC_DASHBOARD;
if (isPublicDashboard) {
return;
}
// if the current route
if (currentRoute) {
const { isPrivate, key } = currentRoute;
if (isPrivate) {
if (isLoggedInState) {
const route = routePermission[key];
if (route && route.find((e) => e === user.role) === undefined) {
history.push(ROUTES.UN_AUTHORIZED);
}
} else {
setLocalStorageApi(LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT, pathname);
history.push(ROUTES.LOGIN);
}
} else if (isLoggedInState) {
const fromPathname = getLocalStorageApi(
LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT,
);
if (fromPathname) {
history.push(fromPathname);
setLocalStorageApi(LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT, '');
} else if (pathname !== ROUTES.SOMETHING_WENT_WRONG) {
history.push(ROUTES.HOME);
}
} else {
// do nothing as the unauthenticated routes are LOGIN and SIGNUP and the LOGIN container takes care of routing to signup if
// setup is not completed
}
} else if (isLoggedInState) {
const fromPathname = getLocalStorageApi(
LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT,
);
if (fromPathname) {
history.push(fromPathname);
setLocalStorageApi(LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT, '');
} else {
history.push(ROUTES.HOME);
}
} else {
setLocalStorageApi(LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT, pathname);
history.push(ROUTES.LOGIN);
}
}, [isLoggedInState, pathname, user, isOldRoute, currentRoute, location]);
if (isOldRoute) {
const redirectUrl = oldNewRoutesMapping[pathname];
return (
@@ -93,143 +296,7 @@ function PrivateRoute({ children }: PrivateRouteProps): JSX.Element {
);
}
// Public dashboard - no redirect needed
const isPublicDashboard = currentRoute?.path === ROUTES.PUBLIC_DASHBOARD;
if (isPublicDashboard) {
return <>{children}</>;
}
// Check for workspace access restriction (cloud only)
const isCloudPlatform = activeLicense?.platform === LicensePlatform.CLOUD;
if (!isFetchingActiveLicense && activeLicense && isCloudPlatform) {
const isTerminated = activeLicense.state === LicenseState.TERMINATED;
const isExpired = activeLicense.state === LicenseState.EXPIRED;
const isCancelled = activeLicense.state === LicenseState.CANCELLED;
const isWorkspaceAccessRestricted = isTerminated || isExpired || isCancelled;
if (
isWorkspaceAccessRestricted &&
pathname !== ROUTES.WORKSPACE_ACCESS_RESTRICTED
) {
return <Redirect to={ROUTES.WORKSPACE_ACCESS_RESTRICTED} />;
}
// Check for workspace suspended (DEFAULTED)
const shouldSuspendWorkspace = activeLicense.state === LicenseState.DEFAULTED;
if (shouldSuspendWorkspace && pathname !== ROUTES.WORKSPACE_SUSPENDED) {
return <Redirect to={ROUTES.WORKSPACE_SUSPENDED} />;
}
}
// Check for workspace blocked (trial expired)
if (!isFetchingActiveLicense && isCloudPlatform && trialInfo?.workSpaceBlock) {
const isRouteEnabledForWorkspaceBlockedState =
isAdmin &&
(pathname === ROUTES.SETTINGS ||
pathname === ROUTES.ORG_SETTINGS ||
pathname === ROUTES.MEMBERS_SETTINGS ||
pathname === ROUTES.BILLING ||
pathname === ROUTES.MY_SETTINGS);
if (
pathname !== ROUTES.WORKSPACE_LOCKED &&
!isRouteEnabledForWorkspaceBlockedState
) {
return <Redirect to={ROUTES.WORKSPACE_LOCKED} />;
}
}
// Check for onboarding redirect (cloud users, first user, onboarding not complete)
if (
isCloudUserVal &&
!isFetchingOrgPreferences &&
orgPreferences &&
!isFetchingUsers &&
usersData &&
usersData.data
) {
const isOnboardingComplete = orgPreferences?.find(
(preference: OrgPreference) =>
preference.name === ORG_PREFERENCES.ORG_ONBOARDING,
)?.value;
// Don't redirect to onboarding if workspace has issues
const isWorkspaceBlocked = trialInfo?.workSpaceBlock;
const isWorkspaceSuspended = activeLicense?.state === LicenseState.DEFAULTED;
const isWorkspaceAccessRestricted =
activeLicense?.state === LicenseState.TERMINATED ||
activeLicense?.state === LicenseState.EXPIRED ||
activeLicense?.state === LicenseState.CANCELLED;
const hasWorkspaceIssue =
isWorkspaceBlocked || isWorkspaceSuspended || isWorkspaceAccessRestricted;
if (!hasWorkspaceIssue) {
const isFirstUser = checkFirstTimeUser();
if (
isFirstUser &&
!isOnboardingComplete &&
!ROUTES_NOT_TO_BE_OVERRIDEN.includes(pathname) &&
pathname !== ROUTES.ONBOARDING
) {
return <Redirect to={ROUTES.ONBOARDING} />;
}
}
}
// Check for GET_STARTED → GET_STARTED_WITH_CLOUD redirect (feature flag)
if (
currentRoute?.path === ROUTES.GET_STARTED &&
featureFlags?.find((e) => e.name === FeatureKeys.ONBOARDING_V3)?.active
) {
return <Redirect to={ROUTES.GET_STARTED_WITH_CLOUD} />;
}
// Main routing logic
if (currentRoute) {
const { isPrivate, key } = currentRoute;
if (isPrivate) {
if (isLoggedInState) {
const route = routePermission[key];
if (route && route.find((e) => e === user.role) === undefined) {
return <Redirect to={ROUTES.UN_AUTHORIZED} />;
}
} else {
// Save current path and redirect to login
setLocalStorageApi(LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT, pathname);
return <Redirect to={ROUTES.LOGIN} />;
}
} else if (isLoggedInState) {
// Non-private route, but user is logged in
const fromPathname = getLocalStorageApi(
LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT,
);
if (fromPathname) {
setLocalStorageApi(LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT, '');
return <Redirect to={fromPathname} />;
}
if (pathname !== ROUTES.SOMETHING_WENT_WRONG) {
return <Redirect to={ROUTES.HOME} />;
}
}
// Non-private route, user not logged in - let login/signup pages handle it
} else if (isLoggedInState) {
// Unknown route, logged in
const fromPathname = getLocalStorageApi(
LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT,
);
if (fromPathname) {
setLocalStorageApi(LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT, '');
return <Redirect to={fromPathname} />;
}
return <Redirect to={ROUTES.HOME} />;
} else {
// Unknown route, not logged in
setLocalStorageApi(LOCALSTORAGE.UNAUTHENTICATED_ROUTE_HIT, pathname);
return <Redirect to={ROUTES.LOGIN} />;
}
// NOTE: disabling this rule as there is no need to have div
return <>{children}</>;
}

View File

@@ -6,6 +6,7 @@ import { FeatureKeys } from 'constants/features';
import { LOCALSTORAGE } from 'constants/localStorage';
import { ORG_PREFERENCES } from 'constants/orgPreferences';
import ROUTES from 'constants/routes';
import history from 'lib/history';
import { AppContext } from 'providers/App/App';
import { IAppContext, IUser } from 'providers/App/types';
import {
@@ -21,6 +22,19 @@ import { ROLES, USER_ROLES } from 'types/roles';
import PrivateRoute from '../Private';
// Mock history module
jest.mock('lib/history', () => ({
__esModule: true,
default: {
push: jest.fn(),
location: { pathname: '/', search: '', hash: '' },
listen: jest.fn(),
createHref: jest.fn(),
},
}));
const mockHistoryPush = history.push as jest.Mock;
// Mock localStorage APIs
const mockLocalStorage: Record<string, string> = {};
jest.mock('api/browser/localstorage/get', () => ({
@@ -225,18 +239,20 @@ function renderPrivateRoute(options: RenderPrivateRouteOptions = {}): void {
}
// Generic assertion helpers for navigation behavior
// Using location-based assertions since Private.tsx now uses Redirect component
// Using these allows easier refactoring when switching from history.push to Redirect component
async function assertRedirectsTo(targetRoute: string): Promise<void> {
await waitFor(() => {
expect(screen.getByTestId('location-display')).toHaveTextContent(targetRoute);
expect(mockHistoryPush).toHaveBeenCalledWith(targetRoute);
});
}
function assertStaysOnRoute(expectedRoute: string): void {
expect(screen.getByTestId('location-display')).toHaveTextContent(
expectedRoute,
);
function assertNoRedirect(): void {
expect(mockHistoryPush).not.toHaveBeenCalled();
}
function assertDoesNotRedirectTo(targetRoute: string): void {
expect(mockHistoryPush).not.toHaveBeenCalledWith(targetRoute);
}
function assertRendersChildren(): void {
@@ -334,7 +350,7 @@ describe('PrivateRoute', () => {
});
assertRendersChildren();
assertStaysOnRoute('/public/dashboard/abc123');
assertNoRedirect();
});
it('should render children for public dashboard route when logged in without redirecting', () => {
@@ -346,7 +362,7 @@ describe('PrivateRoute', () => {
assertRendersChildren();
// Critical: without the isPublicDashboard early return, logged-in users
// would be redirected to HOME due to the non-private route handling
assertStaysOnRoute('/public/dashboard/abc123');
assertNoRedirect();
});
});
@@ -404,7 +420,7 @@ describe('PrivateRoute', () => {
});
assertRendersChildren();
assertStaysOnRoute(ROUTES.HOME);
assertNoRedirect();
});
it('should redirect to unauthorized when VIEWER tries to access admin-only route /alerts/new', async () => {
@@ -513,7 +529,7 @@ describe('PrivateRoute', () => {
appContext: { isLoggedIn: true },
});
assertStaysOnRoute(ROUTES.SOMETHING_WENT_WRONG);
assertDoesNotRedirectTo(ROUTES.HOME);
});
});
@@ -525,7 +541,7 @@ describe('PrivateRoute', () => {
});
// Should not redirect - login page handles its own routing
assertStaysOnRoute(ROUTES.LOGIN);
assertNoRedirect();
});
it('should not redirect when not logged in user visits signup page', () => {
@@ -534,7 +550,7 @@ describe('PrivateRoute', () => {
appContext: { isLoggedIn: false },
});
assertStaysOnRoute(ROUTES.SIGN_UP);
assertNoRedirect();
});
it('should not redirect when not logged in user visits password reset page', () => {
@@ -543,7 +559,7 @@ describe('PrivateRoute', () => {
appContext: { isLoggedIn: false },
});
assertStaysOnRoute(ROUTES.PASSWORD_RESET);
assertNoRedirect();
});
it('should not redirect when not logged in user visits forgot password page', () => {
@@ -552,7 +568,7 @@ describe('PrivateRoute', () => {
appContext: { isLoggedIn: false },
});
assertStaysOnRoute(ROUTES.FORGOT_PASSWORD);
assertNoRedirect();
});
});
@@ -641,7 +657,7 @@ describe('PrivateRoute', () => {
});
// Admin should be able to access settings even when workspace is blocked
assertStaysOnRoute(ROUTES.SETTINGS);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_LOCKED);
});
it('should allow ADMIN to access /settings/billing when workspace is blocked', () => {
@@ -657,7 +673,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.BILLING);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_LOCKED);
});
it('should allow ADMIN to access /settings/org-settings when workspace is blocked', () => {
@@ -673,7 +689,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.ORG_SETTINGS);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_LOCKED);
});
it('should allow ADMIN to access /settings/members when workspace is blocked', () => {
@@ -689,7 +705,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.MEMBERS_SETTINGS);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_LOCKED);
});
it('should allow ADMIN to access /settings/my-settings when workspace is blocked', () => {
@@ -705,7 +721,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.MY_SETTINGS);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_LOCKED);
});
it('should redirect VIEWER to workspace locked even when trying to access settings', async () => {
@@ -816,7 +832,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.WORKSPACE_LOCKED);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_LOCKED);
});
it('should not redirect self-hosted users to workspace locked even when workSpaceBlock is true', () => {
@@ -833,7 +849,7 @@ describe('PrivateRoute', () => {
isCloudUser: false,
});
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_LOCKED);
});
});
@@ -903,7 +919,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.WORKSPACE_ACCESS_RESTRICTED);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_ACCESS_RESTRICTED);
});
it('should not redirect self-hosted users to workspace access restricted when license is terminated', () => {
@@ -920,7 +936,7 @@ describe('PrivateRoute', () => {
isCloudUser: false,
});
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_ACCESS_RESTRICTED);
});
it('should not redirect when license is ACTIVE', () => {
@@ -937,7 +953,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_ACCESS_RESTRICTED);
});
it('should not redirect when license is EVALUATING', () => {
@@ -954,7 +970,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_ACCESS_RESTRICTED);
});
});
@@ -990,7 +1006,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.WORKSPACE_SUSPENDED);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_SUSPENDED);
});
it('should not redirect self-hosted users to workspace suspended when license is defaulted', () => {
@@ -1007,7 +1023,7 @@ describe('PrivateRoute', () => {
isCloudUser: false,
});
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_SUSPENDED);
});
});
@@ -1027,11 +1043,6 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
// Wait for the users query to complete and trigger re-render
await act(async () => {
await Promise.resolve();
});
await assertRedirectsTo(ROUTES.ONBOARDING);
});
@@ -1047,7 +1058,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.ONBOARDING);
});
it('should not redirect to onboarding when onboarding is already complete', async () => {
@@ -1073,7 +1084,7 @@ describe('PrivateRoute', () => {
// Critical: if isOnboardingComplete check is broken (always false),
// this test would fail because all other conditions for redirect ARE met
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.ONBOARDING);
});
it('should not redirect to onboarding for non-cloud users', () => {
@@ -1088,7 +1099,7 @@ describe('PrivateRoute', () => {
isCloudUser: false,
});
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.ONBOARDING);
});
it('should not redirect to onboarding when on /workspace-locked route', () => {
@@ -1103,7 +1114,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.WORKSPACE_LOCKED);
assertDoesNotRedirectTo(ROUTES.ONBOARDING);
});
it('should not redirect to onboarding when on /workspace-suspended route', () => {
@@ -1118,7 +1129,7 @@ describe('PrivateRoute', () => {
isCloudUser: true,
});
assertStaysOnRoute(ROUTES.WORKSPACE_SUSPENDED);
assertDoesNotRedirectTo(ROUTES.ONBOARDING);
});
it('should not redirect to onboarding when workspace is blocked and accessing billing', async () => {
@@ -1145,7 +1156,7 @@ describe('PrivateRoute', () => {
});
// Should NOT redirect to onboarding - user needs to access billing to fix payment
assertStaysOnRoute(ROUTES.BILLING);
assertDoesNotRedirectTo(ROUTES.ONBOARDING);
});
it('should not redirect to onboarding when workspace is blocked and accessing settings', async () => {
@@ -1169,7 +1180,7 @@ describe('PrivateRoute', () => {
await Promise.resolve();
});
assertStaysOnRoute(ROUTES.SETTINGS);
assertDoesNotRedirectTo(ROUTES.ONBOARDING);
});
it('should not redirect to onboarding when workspace is suspended (DEFAULTED)', async () => {
@@ -1196,7 +1207,7 @@ describe('PrivateRoute', () => {
});
// Should redirect to WORKSPACE_SUSPENDED, not ONBOARDING
await assertRedirectsTo(ROUTES.WORKSPACE_SUSPENDED);
assertDoesNotRedirectTo(ROUTES.ONBOARDING);
});
it('should not redirect to onboarding when workspace is access restricted (TERMINATED)', async () => {
@@ -1223,7 +1234,7 @@ describe('PrivateRoute', () => {
});
// Should redirect to WORKSPACE_ACCESS_RESTRICTED, not ONBOARDING
await assertRedirectsTo(ROUTES.WORKSPACE_ACCESS_RESTRICTED);
assertDoesNotRedirectTo(ROUTES.ONBOARDING);
});
it('should not redirect to onboarding when workspace is access restricted (EXPIRED)', async () => {
@@ -1249,7 +1260,7 @@ describe('PrivateRoute', () => {
await Promise.resolve();
});
await assertRedirectsTo(ROUTES.WORKSPACE_ACCESS_RESTRICTED);
assertDoesNotRedirectTo(ROUTES.ONBOARDING);
});
});
@@ -1291,7 +1302,7 @@ describe('PrivateRoute', () => {
},
});
assertStaysOnRoute(ROUTES.GET_STARTED);
assertDoesNotRedirectTo(ROUTES.GET_STARTED_WITH_CLOUD);
});
it('should not redirect when on GET_STARTED and ONBOARDING_V3 feature flag is not present', () => {
@@ -1303,7 +1314,7 @@ describe('PrivateRoute', () => {
},
});
assertStaysOnRoute(ROUTES.GET_STARTED);
assertDoesNotRedirectTo(ROUTES.GET_STARTED_WITH_CLOUD);
});
it('should not redirect when on different route even if ONBOARDING_V3 is active', () => {
@@ -1323,7 +1334,7 @@ describe('PrivateRoute', () => {
},
});
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.GET_STARTED_WITH_CLOUD);
});
});
@@ -1339,7 +1350,7 @@ describe('PrivateRoute', () => {
},
});
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_LOCKED);
});
it('should not fetch users when org data is not available', () => {
@@ -1382,7 +1393,9 @@ describe('PrivateRoute', () => {
},
});
assertStaysOnRoute(ROUTES.HOME);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_LOCKED);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_SUSPENDED);
assertDoesNotRedirectTo(ROUTES.WORKSPACE_ACCESS_RESTRICTED);
});
});
@@ -1423,40 +1436,22 @@ describe('PrivateRoute', () => {
await assertRedirectsTo(ROUTES.UN_AUTHORIZED);
});
it('should allow ADMIN to access /services route', () => {
renderPrivateRoute({
initialRoute: ROUTES.APPLICATION,
appContext: {
isLoggedIn: true,
user: createMockUser({ role: USER_ROLES.ADMIN as ROLES }),
},
it('should allow all roles to access /services route', () => {
const roles = [USER_ROLES.ADMIN, USER_ROLES.EDITOR, USER_ROLES.VIEWER];
roles.forEach((role) => {
jest.clearAllMocks();
renderPrivateRoute({
initialRoute: ROUTES.APPLICATION,
appContext: {
isLoggedIn: true,
user: createMockUser({ role: role as ROLES }),
},
});
assertDoesNotRedirectTo(ROUTES.UN_AUTHORIZED);
});
assertStaysOnRoute(ROUTES.APPLICATION);
});
it('should allow EDITOR to access /services route', () => {
renderPrivateRoute({
initialRoute: ROUTES.APPLICATION,
appContext: {
isLoggedIn: true,
user: createMockUser({ role: USER_ROLES.EDITOR as ROLES }),
},
});
assertStaysOnRoute(ROUTES.APPLICATION);
});
it('should allow VIEWER to access /services route', () => {
renderPrivateRoute({
initialRoute: ROUTES.APPLICATION,
appContext: {
isLoggedIn: true,
user: createMockUser({ role: USER_ROLES.VIEWER as ROLES }),
},
});
assertStaysOnRoute(ROUTES.APPLICATION);
});
it('should redirect VIEWER from /onboarding route (admin only)', async () => {
@@ -1486,7 +1481,7 @@ describe('PrivateRoute', () => {
});
assertRendersChildren();
assertStaysOnRoute(ROUTES.CHANNELS_NEW);
assertDoesNotRedirectTo(ROUTES.UN_AUTHORIZED);
});
it('should allow EDITOR to access /get-started route', () => {
@@ -1498,7 +1493,7 @@ describe('PrivateRoute', () => {
},
});
assertStaysOnRoute(ROUTES.GET_STARTED);
assertDoesNotRedirectTo(ROUTES.UN_AUTHORIZED);
});
});

View File

@@ -13,9 +13,7 @@ export interface HostListPayload {
orderBy?: {
columnName: string;
order: 'asc' | 'desc';
} | null;
start?: number;
end?: number;
};
}
export interface TimeSeriesValue {

View File

@@ -10,12 +10,7 @@ import APIError from 'types/api/error';
import './ErrorContent.styles.scss';
interface ErrorContentProps {
error:
| APIError
| {
code: number;
message: string;
};
error: APIError;
icon?: ReactNode;
}
@@ -25,15 +20,7 @@ function ErrorContent({ error, icon }: ErrorContentProps): JSX.Element {
errors: errorMessages,
code: errorCode,
message: errorMessage,
} =
error && 'error' in error
? error?.error?.error || {}
: {
url: undefined,
errors: [],
code: error.code || 500,
message: error.message || 'Something went wrong',
};
} = error?.error?.error || {};
return (
<section className="error-content">
{/* Summary Header */}

View File

@@ -13,7 +13,7 @@ import uPlot from 'uplot';
import { ChartProps } from '../types';
const TOOLTIP_WIDTH_PADDING = 120;
const TOOLTIP_WIDTH_PADDING = 60;
const TOOLTIP_MIN_WIDTH = 200;
export default function ChartWrapper({

View File

@@ -1,52 +1,41 @@
import { useCallback, useEffect, useMemo, useState } from 'react';
import { useQuery } from 'react-query';
// eslint-disable-next-line no-restricted-imports
import { useSelector } from 'react-redux';
import { VerticalAlignTopOutlined } from '@ant-design/icons';
import { Button, Tooltip, Typography } from 'antd';
import logEvent from 'api/common/logEvent';
import {
getHostLists,
HostListPayload,
HostListResponse,
} from 'api/infraMonitoring/getHostLists';
import { HostListPayload } from 'api/infraMonitoring/getHostLists';
import HostMetricDetail from 'components/HostMetricsDetail';
import QuickFilters from 'components/QuickFilters/QuickFilters';
import { QuickFiltersSource } from 'components/QuickFilters/types';
import { InfraMonitoringEvents } from 'constants/events';
import { FeatureKeys } from 'constants/features';
import { REACT_QUERY_KEY } from 'constants/reactQueryKeys';
import {
useInfraMonitoringCurrentPage,
useInfraMonitoringFiltersHosts,
useInfraMonitoringOrderByHosts,
} from 'container/InfraMonitoringK8s/hooks';
import { usePageSize } from 'container/InfraMonitoringK8s/utils';
import { useGetHostList } from 'hooks/infraMonitoring/useGetHostList';
import { useQueryBuilder } from 'hooks/queryBuilder/useQueryBuilder';
import { useQueryOperations } from 'hooks/queryBuilder/useQueryBuilderOperations';
import { Filter } from 'lucide-react';
import { parseAsString, useQueryState } from 'nuqs';
import { useAppContext } from 'providers/App/App';
import { useGlobalTimeStore } from 'store/globalTime';
import {
getAutoRefreshQueryKey,
NANO_SECOND_MULTIPLIER,
} from 'store/globalTime/utils';
import { ErrorResponse, SuccessResponse } from 'types/api';
import {
IBuilderQuery,
Query,
TagFilter,
} from 'types/api/queryBuilder/queryBuilderData';
import { AppState } from 'store/reducers';
import { IBuilderQuery, Query } from 'types/api/queryBuilder/queryBuilderData';
import { GlobalReducer } from 'types/reducer/globalTime';
import { FeatureKeys } from '../../constants/features';
import { useAppContext } from '../../providers/App/App';
import HostsListControls from './HostsListControls';
import HostsListTable from './HostsListTable';
import { getHostListsQuery, GetHostsQuickFiltersConfig } from './utils';
import './InfraMonitoring.styles.scss';
const defaultFilters: TagFilter = { items: [], op: 'and' };
const baseQuery = getHostListsQuery();
function HostsList(): JSX.Element {
const { maxTime, minTime } = useSelector<AppState, GlobalReducer>(
(state) => state.globalTime,
);
const [currentPage, setCurrentPage] = useInfraMonitoringCurrentPage();
const [filters, setFilters] = useInfraMonitoringFiltersHosts();
const [orderBy, setOrderBy] = useInfraMonitoringOrderByHosts();
@@ -73,48 +62,56 @@ function HostsList(): JSX.Element {
const { pageSize, setPageSize } = usePageSize('hosts');
const selectedTime = useGlobalTimeStore((s) => s.selectedTime);
const isRefreshEnabled = useGlobalTimeStore((s) => s.isRefreshEnabled);
const refreshInterval = useGlobalTimeStore((s) => s.refreshInterval);
const getMinMaxTime = useGlobalTimeStore((s) => s.getMinMaxTime);
const query = useMemo(() => {
const baseQuery = getHostListsQuery();
return {
...baseQuery,
limit: pageSize,
offset: (currentPage - 1) * pageSize,
filters,
start: Math.floor(minTime / 1000000),
end: Math.floor(maxTime / 1000000),
orderBy,
};
}, [pageSize, currentPage, filters, minTime, maxTime, orderBy]);
const queryKey = useMemo(
() =>
getAutoRefreshQueryKey(
selectedTime,
REACT_QUERY_KEY.GET_HOST_LIST,
const queryKey = useMemo(() => {
if (selectedHostName) {
return [
'hostList',
String(pageSize),
String(currentPage),
JSON.stringify(filters),
JSON.stringify(orderBy),
),
[pageSize, currentPage, filters, orderBy, selectedTime],
);
];
}
return [
'hostList',
String(pageSize),
String(currentPage),
JSON.stringify(filters),
JSON.stringify(orderBy),
String(minTime),
String(maxTime),
];
}, [
pageSize,
currentPage,
filters,
orderBy,
selectedHostName,
minTime,
maxTime,
]);
const { data, isFetching, isLoading, isError } = useQuery<
SuccessResponse<HostListResponse> | ErrorResponse,
Error
>({
queryKey,
queryFn: ({ signal }) => {
const { minTime, maxTime } = getMinMaxTime();
const payload: HostListPayload = {
...baseQuery,
limit: pageSize,
offset: (currentPage - 1) * pageSize,
filters: filters ?? defaultFilters,
orderBy,
start: Math.floor(minTime / NANO_SECOND_MULTIPLIER),
end: Math.floor(maxTime / NANO_SECOND_MULTIPLIER),
};
return getHostLists(payload, signal);
const { data, isFetching, isLoading, isError } = useGetHostList(
query as HostListPayload,
{
queryKey,
enabled: !!query,
keepPreviousData: true,
},
enabled: true,
keepPreviousData: true,
refetchInterval: isRefreshEnabled ? refreshInterval : false,
});
);
const hostMetricsData = useMemo(() => data?.payload?.data?.records || [], [
data,
@@ -230,7 +227,7 @@ function HostsList(): JSX.Element {
isError={isError}
tableData={data}
hostMetricsData={hostMetricsData}
filters={filters ?? defaultFilters}
filters={filters || { items: [], op: 'AND' }}
currentPage={currentPage}
setCurrentPage={setCurrentPage}
onHostClick={handleHostClick}

View File

@@ -10,7 +10,6 @@ import {
} from 'antd';
import type { SorterResult } from 'antd/es/table/interface';
import logEvent from 'api/common/logEvent';
import ErrorContent from 'components/ErrorModal/components/ErrorContent';
import { InfraMonitoringEvents } from 'constants/events';
import { isModifierKeyPressed } from 'utils/app';
import { openInNewTab } from 'utils/navigation';
@@ -27,40 +26,9 @@ import {
function EmptyOrLoadingView(
viewState: EmptyOrLoadingViewProps,
): React.ReactNode {
if (viewState.showTableLoadingState) {
return (
<div className="hosts-list-loading-state">
<Skeleton.Input
className="hosts-list-loading-state-item"
size="large"
block
active
/>
<Skeleton.Input
className="hosts-list-loading-state-item"
size="large"
block
active
/>
<Skeleton.Input
className="hosts-list-loading-state-item"
size="large"
block
active
/>
</div>
);
}
const { isError, data } = viewState;
if (isError || data?.error || (data?.statusCode || 0) >= 300) {
return (
<ErrorContent
error={{
code: data?.statusCode || 500,
message: data?.error || 'Something went wrong',
}}
/>
);
const { isError, errorMessage } = viewState;
if (isError) {
return <Typography>{errorMessage || 'Something went wrong'}</Typography>;
}
if (viewState.showHostsEmptyState) {
return (
@@ -108,6 +76,30 @@ function EmptyOrLoadingView(
</div>
);
}
if (viewState.showTableLoadingState) {
return (
<div className="hosts-list-loading-state">
<Skeleton.Input
className="hosts-list-loading-state-item"
size="large"
block
active
/>
<Skeleton.Input
className="hosts-list-loading-state-item"
size="large"
block
active
/>
<Skeleton.Input
className="hosts-list-loading-state-item"
size="large"
block
active
/>
</div>
);
}
return null;
}
@@ -198,8 +190,7 @@ export default function HostsListTable({
!isLoading &&
formattedHostMetricsData.length === 0 &&
(!sentAnyHostMetricsData || isSendingIncorrectK8SAgentMetrics) &&
!filters.items.length &&
!endTimeBeforeRetention;
!filters.items.length;
const showEndTimeBeforeRetentionMessage =
!isFetching &&
@@ -220,7 +211,7 @@ export default function HostsListTable({
const emptyOrLoadingView = EmptyOrLoadingView({
isError,
data,
errorMessage: data?.error ?? '',
showHostsEmptyState,
sentAnyHostMetricsData,
isSendingIncorrectK8SAgentMetrics,

View File

@@ -2,8 +2,8 @@ import { QueryClient, QueryClientProvider } from 'react-query';
// eslint-disable-next-line no-restricted-imports
import { Provider } from 'react-redux';
import { MemoryRouter } from 'react-router-dom';
import { render, waitFor } from '@testing-library/react';
import * as getHostListsApi from 'api/infraMonitoring/getHostLists';
import { render } from '@testing-library/react';
import * as useGetHostListHooks from 'hooks/infraMonitoring/useGetHostList';
import { withNuqsTestingAdapter } from 'nuqs/adapters/testing';
import * as appContextHooks from 'providers/App/App';
import * as timezoneHooks from 'providers/Timezone';
@@ -19,10 +19,6 @@ jest.mock('lib/getMinMax', () => ({
maxTime: 1713738000000,
isValidShortHandDateTimeFormat: jest.fn().mockReturnValue(true),
})),
getMinMaxForSelectedTime: jest.fn().mockReturnValue({
minTime: 1713734400000000000,
maxTime: 1713738000000000000,
}),
}));
jest.mock('container/TopNav/DateTimeSelectionV2', () => ({
__esModule: true,
@@ -45,13 +41,7 @@ jest.mock('components/CustomTimePicker/CustomTimePicker', () => ({
),
}));
const queryClient = new QueryClient({
defaultOptions: {
queries: {
retry: false,
},
},
});
const queryClient = new QueryClient();
jest.mock('react-redux', () => ({
...jest.requireActual('react-redux'),
@@ -90,40 +80,27 @@ jest.spyOn(timezoneHooks, 'useTimezone').mockReturnValue({
offset: 0,
},
} as any);
jest.spyOn(getHostListsApi, 'getHostLists').mockResolvedValue({
statusCode: 200,
error: null,
message: 'Success',
payload: {
status: 'success',
data: {
type: 'list',
records: [
{
hostName: 'test-host',
active: true,
os: 'linux',
cpu: 0.75,
cpuTimeSeries: { labels: {}, labelsArray: [], values: [] },
memory: 0.65,
memoryTimeSeries: { labels: {}, labelsArray: [], values: [] },
wait: 0.03,
waitTimeSeries: { labels: {}, labelsArray: [], values: [] },
load15: 0.5,
load15TimeSeries: { labels: {}, labelsArray: [], values: [] },
},
],
groups: null,
total: 1,
sentAnyHostMetricsData: true,
isSendingK8SAgentMetrics: false,
endTimeBeforeRetention: false,
jest.spyOn(useGetHostListHooks, 'useGetHostList').mockReturnValue({
data: {
payload: {
data: {
records: [
{
hostName: 'test-host',
active: true,
cpu: 0.75,
memory: 0.65,
wait: 0.03,
},
],
isSendingK8SAgentMetrics: false,
sentAnyHostMetricsData: true,
},
},
},
params: {} as any,
});
isLoading: false,
isError: false,
} as any);
jest.spyOn(appContextHooks, 'useAppContext').mockReturnValue({
user: {
role: 'admin',
@@ -151,11 +128,7 @@ jest.spyOn(appContextHooks, 'useAppContext').mockReturnValue({
const Wrapper = withNuqsTestingAdapter({ searchParams: {} });
describe('HostsList', () => {
beforeEach(() => {
queryClient.clear();
});
it('renders hosts list table', async () => {
it('renders hosts list table', () => {
const { container } = render(
<Wrapper>
<QueryClientProvider client={queryClient}>
@@ -167,12 +140,10 @@ describe('HostsList', () => {
</QueryClientProvider>
</Wrapper>,
);
await waitFor(() => {
expect(container.querySelector('.hosts-list-table')).toBeInTheDocument();
});
expect(container.querySelector('.hosts-list-table')).toBeInTheDocument();
});
it('renders filters', async () => {
it('renders filters', () => {
const { container } = render(
<Wrapper>
<QueryClientProvider client={queryClient}>
@@ -184,8 +155,6 @@ describe('HostsList', () => {
</QueryClientProvider>
</Wrapper>,
);
await waitFor(() => {
expect(container.querySelector('.filters')).toBeInTheDocument();
});
expect(container.querySelector('.filters')).toBeInTheDocument();
});
});

View File

@@ -1,13 +1,8 @@
import { Dispatch, SetStateAction } from 'react';
import { InfoCircleOutlined } from '@ant-design/icons';
import { Color } from '@signozhq/design-tokens';
import {
Progress,
TableColumnType as ColumnType,
Tag,
Tooltip,
Typography,
} from 'antd';
import { Progress, TabsProps, Tag, Tooltip, Typography } from 'antd';
import { TableColumnType as ColumnType } from 'antd';
import { SortOrder } from 'antd/lib/table/interface';
import {
HostData,
@@ -18,6 +13,8 @@ import {
FiltersType,
IQuickFiltersConfig,
} from 'components/QuickFilters/types';
import TabLabel from 'components/TabLabel';
import { PANEL_TYPES } from 'constants/queryBuilder';
import { TriangleAlert } from 'lucide-react';
import { ErrorResponse, SuccessResponse } from 'types/api';
import { DataTypes } from 'types/api/queryBuilder/queryAutocompleteResponse';
@@ -25,6 +22,9 @@ import { TagFilter } from 'types/api/queryBuilder/queryBuilderData';
import { DataSource } from 'types/common/queryBuilder';
import { OrderBySchemaType } from '../InfraMonitoringK8s/schemas';
import HostsList from './HostsList';
import './InfraMonitoring.styles.scss';
export interface HostRowData {
key?: string;
@@ -112,10 +112,7 @@ export interface HostsListTableProps {
export interface EmptyOrLoadingViewProps {
isError: boolean;
data:
| ErrorResponse<string>
| SuccessResponse<HostListResponse, unknown>
| undefined;
errorMessage: string;
showHostsEmptyState: boolean;
sentAnyHostMetricsData: boolean;
isSendingIncorrectK8SAgentMetrics: boolean;
@@ -144,6 +141,14 @@ function mapOrderByToSortOrder(
: undefined;
}
export const getTabsItems = (): TabsProps['items'] => [
{
label: <TabLabel label="List View" isDisabled={false} tooltipText="" />,
key: PANEL_TYPES.LIST,
children: <HostsList />,
},
];
export const getHostsListColumns = (
orderBy: OrderBySchemaType,
): ColumnType<HostRowData>[] => [

View File

@@ -0,0 +1,42 @@
import { useMemo } from 'react';
import { useQuery, UseQueryOptions, UseQueryResult } from 'react-query';
import {
getHostLists,
HostListPayload,
HostListResponse,
} from 'api/infraMonitoring/getHostLists';
import { REACT_QUERY_KEY } from 'constants/reactQueryKeys';
import { ErrorResponse, SuccessResponse } from 'types/api';
type UseGetHostList = (
requestData: HostListPayload,
options?: UseQueryOptions<
SuccessResponse<HostListResponse> | ErrorResponse,
Error
>,
headers?: Record<string, string>,
) => UseQueryResult<SuccessResponse<HostListResponse> | ErrorResponse, Error>;
export const useGetHostList: UseGetHostList = (
requestData,
options,
headers,
) => {
const queryKey = useMemo(() => {
if (options?.queryKey && Array.isArray(options.queryKey)) {
return [...options.queryKey];
}
if (options?.queryKey && typeof options.queryKey === 'string') {
return options.queryKey;
}
return [REACT_QUERY_KEY.GET_HOST_LIST, requestData];
}, [options?.queryKey, requestData]);
return useQuery<SuccessResponse<HostListResponse> | ErrorResponse, Error>({
queryFn: ({ signal }) => getHostLists(requestData, signal, headers),
...options,
queryKey,
});
};

View File

@@ -1,72 +0,0 @@
.uplot-tooltip-container {
font-family: 'Inter';
font-size: 12px;
background: var(--bg-ink-300);
-webkit-font-smoothing: antialiased;
color: var(--bg-vanilla-100);
border-radius: 6px;
border: 1px solid var(--bg-ink-100);
display: flex;
flex-direction: column;
gap: 8px;
&.lightMode {
background: var(--bg-vanilla-100);
color: var(--bg-ink-500);
border: 1px solid var(--bg-vanilla-300);
.uplot-tooltip-list {
&::-webkit-scrollbar-thumb {
background: var(--bg-vanilla-400);
}
}
.uplot-tooltip-divider {
background-color: var(--bg-vanilla-300);
}
}
.uplot-tooltip-header-container {
padding: 1rem 1rem 0 1rem;
display: flex;
flex-direction: column;
gap: 8px;
&:last-child {
padding-bottom: 1rem;
}
.uplot-tooltip-header {
font-size: 13px;
font-weight: 500;
}
}
.uplot-tooltip-divider {
width: 100%;
height: 1px;
background-color: var(--bg-ink-100);
}
.uplot-tooltip-list {
// Virtuoso absolutely positions its item rows; left: 0 prevents accidental
// horizontal offset when the scroller has padding or transform applied.
div[data-viewport-type='element'] {
left: 0;
padding: 4px 8px 4px 16px;
}
&::-webkit-scrollbar {
width: 0.3rem;
}
&::-webkit-scrollbar-track {
background: transparent;
}
&::-webkit-scrollbar-thumb {
background: var(--bg-slate-100);
border-radius: 0.5rem;
}
}
}

View File

@@ -0,0 +1,70 @@
.uplot-tooltip-container {
font-family: 'Inter';
font-size: 12px;
background: var(--bg-ink-300);
-webkit-font-smoothing: antialiased;
color: var(--bg-vanilla-100);
border-radius: 6px;
padding: 1rem 0.5rem 0.5rem 1rem;
border: 1px solid var(--bg-ink-100);
display: flex;
flex-direction: column;
gap: 8px;
&.lightMode {
background: var(--bg-vanilla-100);
color: var(--bg-ink-500);
border: 1px solid var(--bg-vanilla-300);
.uplot-tooltip-list {
&::-webkit-scrollbar-thumb {
background: var(--bg-vanilla-400);
}
}
}
.uplot-tooltip-header {
font-size: 13px;
font-weight: 500;
}
.uplot-tooltip-list-container {
overflow-y: auto;
max-height: 330px;
.uplot-tooltip-list {
&::-webkit-scrollbar {
width: 0.3rem;
}
&::-webkit-scrollbar-track {
background: transparent;
}
&::-webkit-scrollbar-thumb {
background: var(--bg-slate-100);
border-radius: 0.5rem;
}
}
}
.uplot-tooltip-item {
display: flex;
align-items: center;
gap: 8px;
margin-bottom: 4px;
.uplot-tooltip-item-marker {
border-radius: 50%;
border-width: 2px;
width: 12px;
height: 12px;
flex-shrink: 0;
}
.uplot-tooltip-item-content {
white-space: wrap;
word-break: break-all;
}
}
}

View File

@@ -7,14 +7,12 @@ import { useIsDarkMode } from 'hooks/useDarkMode';
import { useTimezone } from 'providers/Timezone';
import { TooltipProps } from '../types';
import TooltipItem from './components/TooltipItem/TooltipItem';
import Styles from './Tooltip.module.scss';
import './Tooltip.styles.scss';
// Fallback per-item height used for the initial size estimate before
// Virtuoso reports the real total height via totalListHeightChanged.
const TOOLTIP_LIST_MAX_HEIGHT = 330;
const TOOLTIP_ITEM_HEIGHT = 38;
const LIST_MAX_HEIGHT = 300;
const TOOLTIP_LIST_PADDING = 10;
export default function Tooltip({
uPlotInstance,
@@ -23,26 +21,27 @@ export default function Tooltip({
showTooltipHeader = true,
}: TooltipProps): JSX.Element {
const isDarkMode = useIsDarkMode();
const [listHeight, setListHeight] = useState(0);
const tooltipContent = content ?? [];
const { timezone: userTimezone } = useTimezone();
const [totalListHeight, setTotalListHeight] = useState(0);
const tooltipContent = useMemo(() => content ?? [], [content]);
const resolvedTimezone = timezone?.value ?? userTimezone.value;
const resolvedTimezone = useMemo(() => {
if (!timezone) {
return userTimezone.value;
}
return timezone.value;
}, [timezone, userTimezone]);
const headerTitle = useMemo(() => {
if (!showTooltipHeader) {
return null;
}
const data = uPlotInstance.data;
const cursorIdx = uPlotInstance.cursor.idx;
if (cursorIdx == null) {
return null;
}
const timestamp = uPlotInstance.data[0]?.[cursorIdx];
if (timestamp == null) {
return null;
}
return dayjs(timestamp * 1000)
return dayjs(data[0][cursorIdx] * 1000)
.tz(resolvedTimezone)
.format(DATE_TIME_FORMATS.MONTH_DATETIME_SECONDS);
}, [
@@ -52,68 +51,60 @@ export default function Tooltip({
showTooltipHeader,
]);
const activeItem = useMemo(
() => tooltipContent.find((item) => item.isActive) ?? null,
[tooltipContent],
);
// Use the measured height from Virtuoso when available; fall back to a
// per-item estimate on the first render. Math.ceil prevents a 1 px
// subpixel rounding gap from triggering a spurious scrollbar.
const virtuosoHeight = useMemo(() => {
return totalListHeight > 0
? Math.ceil(Math.min(totalListHeight, LIST_MAX_HEIGHT))
: Math.min(tooltipContent.length * TOOLTIP_ITEM_HEIGHT, LIST_MAX_HEIGHT);
}, [totalListHeight, tooltipContent.length]);
const showHeader = showTooltipHeader || activeItem != null;
// With a single series the active item is fully represented in the header —
// hide the divider and list to avoid showing a duplicate row.
const showList = tooltipContent.length > 1;
const showDivider = showList && showHeader;
const virtuosoStyle = useMemo(() => {
return {
height:
listHeight > 0
? Math.min(listHeight + TOOLTIP_LIST_PADDING, TOOLTIP_LIST_MAX_HEIGHT)
: Math.min(
tooltipContent.length * TOOLTIP_ITEM_HEIGHT,
TOOLTIP_LIST_MAX_HEIGHT,
),
width: '100%',
};
}, [listHeight, tooltipContent.length]);
return (
<div
className={cx(Styles.uplotTooltipContainer, !isDarkMode && Styles.lightMode)}
className={cx(
'uplot-tooltip-container',
isDarkMode ? 'darkMode' : 'lightMode',
)}
data-testid="uplot-tooltip-container"
>
{showHeader && (
<div className={Styles.uplotTooltipHeaderContainer}>
{showTooltipHeader && headerTitle && (
<div
className={Styles.uplotTooltipHeader}
data-testid="uplot-tooltip-header"
>
<span>{headerTitle}</span>
</div>
)}
{activeItem && (
<TooltipItem
item={activeItem}
isItemActive={true}
containerTestId="uplot-tooltip-pinned"
markerTestId="uplot-tooltip-pinned-marker"
contentTestId="uplot-tooltip-pinned-content"
/>
)}
{showTooltipHeader && (
<div className="uplot-tooltip-header" data-testid="uplot-tooltip-header">
<span>{headerTitle}</span>
</div>
)}
{showDivider && <span className={Styles.uplotTooltipDivider} />}
{showList && (
<Virtuoso
className={Styles.uplotTooltipList}
data-testid="uplot-tooltip-list"
data={tooltipContent}
style={{ height: virtuosoHeight, width: '100%' }}
totalListHeightChanged={setTotalListHeight}
itemContent={(_, item): JSX.Element => (
<TooltipItem item={item} isItemActive={false} />
)}
/>
)}
<div className="uplot-tooltip-list-container">
{tooltipContent.length > 0 ? (
<Virtuoso
className="uplot-tooltip-list"
data-testid="uplot-tooltip-list"
data={tooltipContent}
style={virtuosoStyle}
totalListHeightChanged={setListHeight}
itemContent={(_, item): JSX.Element => (
<div className="uplot-tooltip-item" data-testid="uplot-tooltip-item">
<div
className="uplot-tooltip-item-marker"
style={{ borderColor: item.color }}
data-is-legend-marker={true}
data-testid="uplot-tooltip-item-marker"
/>
<div
className="uplot-tooltip-item-content"
style={{ color: item.color, fontWeight: item.isActive ? 700 : 400 }}
data-testid="uplot-tooltip-item-content"
>
{item.label}: {item.tooltipValue}
</div>
</div>
)}
/>
) : null}
</div>
</div>
);
}

View File

@@ -133,30 +133,46 @@ describe('Tooltip', () => {
expect(screen.queryByText(unexpectedTitle)).not.toBeInTheDocument();
});
it('renders single active item in header only, without a list', () => {
it('renders lightMode class when dark mode is disabled', () => {
const uPlotInstance = createUPlotInstance(null);
const content = [createTooltipContent({ isActive: true })];
mockUseIsDarkMode.mockReturnValue(false);
renderTooltip({ uPlotInstance, content });
renderTooltip({ uPlotInstance });
// Active item is shown in the header, not duplicated in a list
expect(screen.queryByTestId('uplot-tooltip-list')).toBeNull();
expect(screen.getByTestId('uplot-tooltip-pinned')).toBeInTheDocument();
const pinnedContent = screen.getByTestId('uplot-tooltip-pinned-content');
expect(pinnedContent).toHaveTextContent('Series A');
expect(pinnedContent).toHaveTextContent('10');
const container = screen.getByTestId('uplot-tooltip-container');
expect(container).toHaveClass('lightMode');
expect(container).not.toHaveClass('darkMode');
});
it('renders list when multiple series are present', () => {
it('renders darkMode class when dark mode is enabled', () => {
const uPlotInstance = createUPlotInstance(null);
const content = [
createTooltipContent({ isActive: true }),
createTooltipContent({ label: 'Series B', isActive: false }),
];
mockUseIsDarkMode.mockReturnValue(true);
renderTooltip({ uPlotInstance });
const container = screen.getByTestId('uplot-tooltip-container');
expect(container).toHaveClass('darkMode');
expect(container).not.toHaveClass('lightMode');
});
it('renders tooltip items when content is provided', () => {
const uPlotInstance = createUPlotInstance(null);
const content = [createTooltipContent()];
renderTooltip({ uPlotInstance, content });
expect(screen.getByTestId('uplot-tooltip-list')).toBeInTheDocument();
const list = screen.queryByTestId('uplot-tooltip-list');
expect(list).not.toBeNull();
const marker = screen.getByTestId('uplot-tooltip-item-marker');
const itemContent = screen.getByTestId('uplot-tooltip-item-content');
expect(marker).toHaveStyle({ borderColor: '#ff0000' });
expect(itemContent).toHaveStyle({ color: '#ff0000', fontWeight: '700' });
expect(itemContent).toHaveTextContent('Series A: 10');
});
it('does not render tooltip list when content is empty', () => {
@@ -176,7 +192,7 @@ describe('Tooltip', () => {
renderTooltip({ uPlotInstance, content });
const list = screen.getByTestId('uplot-tooltip-list');
expect(list).toHaveStyle({ height: '200px' });
expect(list).toHaveStyle({ height: '210px' });
});
it('sets tooltip list height based on content length when Virtuoso reports 0 height', () => {

View File

@@ -189,7 +189,7 @@ describe('Tooltip utils', () => {
];
}
it('builds tooltip content in series-index order with isActive flag set correctly', () => {
it('builds tooltip content with active series first', () => {
const data: AlignedData = [[0], [10], [20], [30]];
const series = createSeriesConfig();
const dataIndexes = [null, 0, 0, 0];
@@ -206,21 +206,21 @@ describe('Tooltip utils', () => {
});
expect(result).toHaveLength(2);
// Series are returned in series-index order (A=index 1 before B=index 2)
// Active (series index 2) should come first
expect(result[0]).toMatchObject<Partial<TooltipContentItem>>({
label: 'A',
value: 10,
tooltipValue: 'formatted-10',
color: '#ff0000',
isActive: false,
});
expect(result[1]).toMatchObject<Partial<TooltipContentItem>>({
label: 'B',
value: 20,
tooltipValue: 'formatted-20',
color: 'color-2',
isActive: true,
});
expect(result[1]).toMatchObject<Partial<TooltipContentItem>>({
label: 'A',
value: 10,
tooltipValue: 'formatted-10',
color: '#ff0000',
isActive: false,
});
});
it('skips series with null data index or non-finite values', () => {
@@ -273,31 +273,5 @@ describe('Tooltip utils', () => {
expect(result[0].value).toBe(30);
expect(result[1].value).toBe(30);
});
it('returns items in series-index order', () => {
// Series values in non-sorted order: 3, 1, 4, 2
const data: AlignedData = [[0], [3], [1], [4], [2]];
const series: Series[] = [
{ label: 'x', show: true } as Series,
{ label: 'C', show: true, stroke: '#aaaaaa' } as Series,
{ label: 'A', show: true, stroke: '#bbbbbb' } as Series,
{ label: 'D', show: true, stroke: '#cccccc' } as Series,
{ label: 'B', show: true, stroke: '#dddddd' } as Series,
];
const dataIndexes = [null, 0, 0, 0, 0];
const u = createUPlotInstance();
const result = buildTooltipContent({
data,
series,
dataIndexes,
activeSeriesIndex: null,
uPlotInstance: u,
yAxisUnit,
decimalPrecision,
});
expect(result.map((item) => item.value)).toEqual([3, 1, 4, 2]);
});
});
});

View File

@@ -1,36 +0,0 @@
.uplot-tooltip-item {
display: flex;
align-items: center;
gap: 8px;
padding: 4px 0;
.uplot-tooltip-item-marker {
border-radius: 50%;
border-style: solid;
border-width: 2px;
width: 12px;
height: 12px;
flex-shrink: 0;
}
.uplot-tooltip-item-content {
width: 100%;
display: flex;
align-items: center;
gap: 8px;
justify-content: space-between;
.uplot-tooltip-item-label {
white-space: normal;
overflow-wrap: anywhere;
}
&-separator {
flex: 1;
border-width: 0.5px;
border-style: dashed;
min-width: 24px;
opacity: 0.5;
}
}
}

View File

@@ -1,49 +0,0 @@
import { TooltipContentItem } from '../../../types';
import Styles from './TooltipItem.module.scss';
interface TooltipItemProps {
item: TooltipContentItem;
isItemActive: boolean;
containerTestId?: string;
markerTestId?: string;
contentTestId?: string;
}
export default function TooltipItem({
item,
isItemActive,
containerTestId = 'uplot-tooltip-item',
markerTestId = 'uplot-tooltip-item-marker',
contentTestId = 'uplot-tooltip-item-content',
}: TooltipItemProps): JSX.Element {
return (
<div
className={Styles.uplotTooltipItem}
style={{
opacity: isItemActive ? 1 : 0.7,
fontWeight: isItemActive ? 700 : 400,
}}
data-testid={containerTestId}
>
<div
className={Styles.uplotTooltipItemMarker}
style={{ borderColor: item.color }}
data-is-legend-marker={true}
data-testid={markerTestId}
/>
<div
className={Styles.uplotTooltipItemContent}
style={{ color: item.color }}
data-testid={contentTestId}
>
<span className={Styles.uplotTooltipItemLabel}>{item.label}</span>
<span
className={Styles.uplotTooltipItemContentSeparator}
style={{ borderColor: item.color }}
/>
<span>{item.tooltipValue}</span>
</div>
</div>
);
}

View File

@@ -38,16 +38,16 @@ export function getTooltipBaseValue({
// When series are hidden, we must use the next *visible* series, not index+1,
// since hidden series keep raw values and would produce negative/wrong results.
if (isStackedBarChart && baseValue !== null && series) {
let nextVisibleSeriesIdx = -1;
for (let seriesIdx = index + 1; seriesIdx < series.length; seriesIdx++) {
if (series[seriesIdx]?.show) {
nextVisibleSeriesIdx = seriesIdx;
let nextVisibleIdx = -1;
for (let j = index + 1; j < series.length; j++) {
if (series[j]?.show) {
nextVisibleIdx = j;
break;
}
}
if (nextVisibleSeriesIdx >= 1) {
const nextStackedValue = data[nextVisibleSeriesIdx][dataIndex] ?? 0;
baseValue = baseValue - nextStackedValue;
if (nextVisibleIdx >= 1) {
const nextValue = data[nextVisibleIdx][dataIndex] ?? 0;
baseValue = baseValue - nextValue;
}
}
return baseValue;
@@ -72,15 +72,16 @@ export function buildTooltipContent({
decimalPrecision?: PrecisionOption;
isStackedBarChart?: boolean;
}): TooltipContentItem[] {
const items: TooltipContentItem[] = [];
const active: TooltipContentItem[] = [];
const rest: TooltipContentItem[] = [];
for (let seriesIndex = 1; seriesIndex < series.length; seriesIndex += 1) {
const seriesItem = series[seriesIndex];
if (!seriesItem?.show) {
for (let index = 1; index < series.length; index += 1) {
const s = series[index];
if (!s?.show) {
continue;
}
const dataIndex = dataIndexes[seriesIndex];
const dataIndex = dataIndexes[index];
// Skip series with no data at the current cursor position
if (dataIndex === null) {
continue;
@@ -88,22 +89,30 @@ export function buildTooltipContent({
const baseValue = getTooltipBaseValue({
data,
index: seriesIndex,
index,
dataIndex,
isStackedBarChart,
series,
});
const isActive = index === activeSeriesIndex;
if (Number.isFinite(baseValue) && baseValue !== null) {
items.push({
label: String(seriesItem.label ?? ''),
const item: TooltipContentItem = {
label: String(s.label ?? ''),
value: baseValue,
tooltipValue: getToolTipValue(baseValue, yAxisUnit, decimalPrecision),
color: resolveSeriesColor(seriesItem.stroke, uPlotInstance, seriesIndex),
isActive: seriesIndex === activeSeriesIndex,
});
color: resolveSeriesColor(s.stroke, uPlotInstance, index),
isActive,
};
if (isActive) {
active.push(item);
} else {
rest.push(item);
}
}
}
return items;
return [...active, ...rest];
}

View File

@@ -36,8 +36,8 @@ const HOVER_DISMISS_DELAY_MS = 100;
export default function TooltipPlugin({
config,
render,
maxWidth = 450,
maxHeight = 600,
maxWidth = 300,
maxHeight = 400,
syncMode = DashboardCursorSync.None,
syncKey = '_tooltip_sync_global_',
pinnedTooltipElement,

View File

@@ -97,9 +97,6 @@ export default defineConfig(
javascriptEnabled: true,
},
},
modules: {
localsConvention: 'camelCaseOnly',
},
},
define: {
// TODO: Remove this in favor of import.meta.env

View File

@@ -32,7 +32,7 @@ func newConfig() factory.Config {
Domain: "signozserviceaccount.com",
},
Analytics: AnalyticsConfig{
Enabled: false,
Enabled: true,
},
}
}

View File

@@ -40,6 +40,7 @@ type querier struct {
promEngine prometheus.Prometheus
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation]
logStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation]
auditStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation]
metricStmtBuilder qbtypes.StatementBuilder[qbtypes.MetricAggregation]
meterStmtBuilder qbtypes.StatementBuilder[qbtypes.MetricAggregation]
traceOperatorStmtBuilder qbtypes.TraceOperatorStatementBuilder
@@ -56,6 +57,7 @@ func New(
promEngine prometheus.Prometheus,
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation],
logStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation],
auditStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation],
metricStmtBuilder qbtypes.StatementBuilder[qbtypes.MetricAggregation],
meterStmtBuilder qbtypes.StatementBuilder[qbtypes.MetricAggregation],
traceOperatorStmtBuilder qbtypes.TraceOperatorStatementBuilder,
@@ -69,6 +71,7 @@ func New(
promEngine: promEngine,
traceStmtBuilder: traceStmtBuilder,
logStmtBuilder: logStmtBuilder,
auditStmtBuilder: auditStmtBuilder,
metricStmtBuilder: metricStmtBuilder,
meterStmtBuilder: meterStmtBuilder,
traceOperatorStmtBuilder: traceOperatorStmtBuilder,
@@ -361,7 +364,11 @@ func (q *querier) QueryRange(ctx context.Context, orgID valuer.UUID, req *qbtype
case qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]:
spec.ShiftBy = extractShiftFromBuilderQuery(spec)
timeRange := adjustTimeRangeForShift(spec, qbtypes.TimeRange{From: req.Start, To: req.End}, req.RequestType)
bq := newBuilderQuery(q.logger, q.telemetryStore, q.logStmtBuilder, spec, timeRange, req.RequestType, tmplVars)
stmtBuilder := q.logStmtBuilder
if spec.Source == telemetrytypes.SourceAudit {
stmtBuilder = q.auditStmtBuilder
}
bq := newBuilderQuery(q.logger, q.telemetryStore, stmtBuilder, spec, timeRange, req.RequestType, tmplVars)
queries[spec.Name] = bq
steps[spec.Name] = spec.StepInterval
case qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]:
@@ -550,7 +557,11 @@ func (q *querier) QueryRawStream(ctx context.Context, orgID valuer.UUID, req *qb
case <-tick:
// timestamp end is not specified here
timeRange := adjustTimeRangeForShift(spec, qbtypes.TimeRange{From: tsStart}, req.RequestType)
bq := newBuilderQuery(q.logger, q.telemetryStore, q.logStmtBuilder, spec, timeRange, req.RequestType, map[string]qbtypes.VariableItem{
liveTailStmtBuilder := q.logStmtBuilder
if spec.Source == telemetrytypes.SourceAudit {
liveTailStmtBuilder = q.auditStmtBuilder
}
bq := newBuilderQuery(q.logger, q.telemetryStore, liveTailStmtBuilder, spec, timeRange, req.RequestType, map[string]qbtypes.VariableItem{
"id": {
Value: updatedLogID,
},
@@ -850,7 +861,11 @@ func (q *querier) createRangedQuery(originalQuery qbtypes.Query, timeRange qbtyp
specCopy := qt.spec.Copy()
specCopy.ShiftBy = extractShiftFromBuilderQuery(specCopy)
adjustedTimeRange := adjustTimeRangeForShift(specCopy, timeRange, qt.kind)
return newBuilderQuery(q.logger, q.telemetryStore, q.logStmtBuilder, specCopy, adjustedTimeRange, qt.kind, qt.variables)
shiftStmtBuilder := q.logStmtBuilder
if qt.spec.Source == telemetrytypes.SourceAudit {
shiftStmtBuilder = q.auditStmtBuilder
}
return newBuilderQuery(q.logger, q.telemetryStore, shiftStmtBuilder, specCopy, adjustedTimeRange, qt.kind, qt.variables)
case *builderQuery[qbtypes.MetricAggregation]:
specCopy := qt.spec.Copy()

View File

@@ -9,6 +9,7 @@ import (
"github.com/SigNoz/signoz/pkg/prometheus"
"github.com/SigNoz/signoz/pkg/querier"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetadata"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
@@ -63,6 +64,11 @@ func newProvider(
telemetrylogs.TagAttributesV2TableName,
telemetrylogs.LogAttributeKeysTblName,
telemetrylogs.LogResourceKeysTblName,
telemetryaudit.DBName,
telemetryaudit.LogsTableName,
telemetryaudit.TagAttributesTableName,
telemetryaudit.LogAttributeKeysTblName,
telemetryaudit.LogResourceKeysTblName,
telemetrymetadata.DBName,
telemetrymetadata.AttributesMetadataLocalTableName,
telemetrymetadata.ColumnEvolutionMetadataTableName,
@@ -82,13 +88,13 @@ func newProvider(
telemetryStore,
)
// ADD: Create trace operator statement builder
// Create trace operator statement builder
traceOperatorStmtBuilder := telemetrytraces.NewTraceOperatorStatementBuilder(
settings,
telemetryMetadataStore,
traceFieldMapper,
traceConditionBuilder,
traceStmtBuilder, // Pass the regular trace statement builder
traceStmtBuilder,
traceAggExprRewriter,
)
@@ -112,6 +118,26 @@ func newProvider(
telemetrylogs.GetBodyJSONKey,
)
// Create audit statement builder
auditFieldMapper := telemetryaudit.NewFieldMapper()
auditConditionBuilder := telemetryaudit.NewConditionBuilder(auditFieldMapper)
auditAggExprRewriter := querybuilder.NewAggExprRewriter(
settings,
telemetryaudit.DefaultFullTextColumn,
auditFieldMapper,
auditConditionBuilder,
nil,
)
auditStmtBuilder := telemetryaudit.NewAuditQueryStatementBuilder(
settings,
telemetryMetadataStore,
auditFieldMapper,
auditConditionBuilder,
auditAggExprRewriter,
telemetryaudit.DefaultFullTextColumn,
nil,
)
// Create metric statement builder
metricFieldMapper := telemetrymetrics.NewFieldMapper()
metricConditionBuilder := telemetrymetrics.NewConditionBuilder(metricFieldMapper)
@@ -148,6 +174,7 @@ func newProvider(
prometheus,
traceStmtBuilder,
logStmtBuilder,
auditStmtBuilder,
metricStmtBuilder,
meterStmtBuilder,
traceOperatorStmtBuilder,

View File

@@ -46,6 +46,7 @@ func prepareQuerierForMetrics(t *testing.T, telemetryStore telemetrystore.Teleme
nil, // prometheus
nil, // traceStmtBuilder
nil, // logStmtBuilder
nil, // auditStmtBuilder
metricStmtBuilder,
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
@@ -91,6 +92,7 @@ func prepareQuerierForLogs(telemetryStore telemetrystore.TelemetryStore, keysMap
nil, // prometheus
nil, // traceStmtBuilder
logStmtBuilder, // logStmtBuilder
nil, // auditStmtBuilder
nil, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
@@ -131,6 +133,7 @@ func prepareQuerierForTraces(telemetryStore telemetrystore.TelemetryStore, keysM
nil, // prometheus
traceStmtBuilder, // traceStmtBuilder
nil, // logStmtBuilder
nil, // auditStmtBuilder
nil, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder

View File

@@ -33,6 +33,7 @@ import (
"github.com/SigNoz/signoz/pkg/sqlschema"
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/SigNoz/signoz/pkg/statsreporter"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetadata"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
@@ -395,6 +396,11 @@ func New(
telemetrylogs.TagAttributesV2TableName,
telemetrylogs.LogAttributeKeysTblName,
telemetrylogs.LogResourceKeysTblName,
telemetryaudit.DBName,
telemetryaudit.LogsTableName,
telemetryaudit.TagAttributesTableName,
telemetryaudit.LogAttributeKeysTblName,
telemetryaudit.LogResourceKeysTblName,
telemetrymetadata.DBName,
telemetrymetadata.AttributesMetadataLocalTableName,
telemetrymetadata.ColumnEvolutionMetadataTableName,

View File

@@ -0,0 +1,204 @@
package telemetryaudit
import (
"context"
"fmt"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
)
type conditionBuilder struct {
fm qbtypes.FieldMapper
}
func NewConditionBuilder(fm qbtypes.FieldMapper) *conditionBuilder {
return &conditionBuilder{fm: fm}
}
func (c *conditionBuilder) conditionFor(
ctx context.Context,
startNs, endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
columns, err := c.fm.ColumnFor(ctx, startNs, endNs, key)
if err != nil {
return "", err
}
if operator.IsStringSearchOperator() {
value = querybuilder.FormatValueForContains(value)
}
fieldExpression, err := c.fm.FieldFor(ctx, startNs, endNs, key)
if err != nil {
return "", err
}
fieldExpression, value = querybuilder.DataTypeCollisionHandledFieldName(key, value, fieldExpression, operator)
switch operator {
case qbtypes.FilterOperatorEqual:
return sb.E(fieldExpression, value), nil
case qbtypes.FilterOperatorNotEqual:
return sb.NE(fieldExpression, value), nil
case qbtypes.FilterOperatorGreaterThan:
return sb.G(fieldExpression, value), nil
case qbtypes.FilterOperatorGreaterThanOrEq:
return sb.GE(fieldExpression, value), nil
case qbtypes.FilterOperatorLessThan:
return sb.LT(fieldExpression, value), nil
case qbtypes.FilterOperatorLessThanOrEq:
return sb.LE(fieldExpression, value), nil
case qbtypes.FilterOperatorLike:
return sb.Like(fieldExpression, value), nil
case qbtypes.FilterOperatorNotLike:
return sb.NotLike(fieldExpression, value), nil
case qbtypes.FilterOperatorILike:
return sb.ILike(fieldExpression, value), nil
case qbtypes.FilterOperatorNotILike:
return sb.NotILike(fieldExpression, value), nil
case qbtypes.FilterOperatorContains:
return sb.ILike(fieldExpression, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorNotContains:
return sb.NotILike(fieldExpression, fmt.Sprintf("%%%s%%", value)), nil
case qbtypes.FilterOperatorRegexp:
return fmt.Sprintf(`match(%s, %s)`, sqlbuilder.Escape(fieldExpression), sb.Var(value)), nil
case qbtypes.FilterOperatorNotRegexp:
return fmt.Sprintf(`NOT match(%s, %s)`, sqlbuilder.Escape(fieldExpression), sb.Var(value)), nil
case qbtypes.FilterOperatorBetween:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrBetweenValues
}
if len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.Between(fieldExpression, values[0], values[1]), nil
case qbtypes.FilterOperatorNotBetween:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrBetweenValues
}
if len(values) != 2 {
return "", qbtypes.ErrBetweenValues
}
return sb.NotBetween(fieldExpression, values[0], values[1]), nil
case qbtypes.FilterOperatorIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
conditions := []string{}
for _, value := range values {
conditions = append(conditions, sb.E(fieldExpression, value))
}
return sb.Or(conditions...), nil
case qbtypes.FilterOperatorNotIn:
values, ok := value.([]any)
if !ok {
return "", qbtypes.ErrInValues
}
conditions := []string{}
for _, value := range values {
conditions = append(conditions, sb.NE(fieldExpression, value))
}
return sb.And(conditions...), nil
case qbtypes.FilterOperatorExists, qbtypes.FilterOperatorNotExists:
var value any
column := columns[0]
switch column.Type.GetType() {
case schema.ColumnTypeEnumJSON:
if operator == qbtypes.FilterOperatorExists {
return sb.IsNotNull(fieldExpression), nil
}
return sb.IsNull(fieldExpression), nil
case schema.ColumnTypeEnumLowCardinality:
switch elementType := column.Type.(schema.LowCardinalityColumnType).ElementType; elementType.GetType() {
case schema.ColumnTypeEnumString:
value = ""
if operator == qbtypes.FilterOperatorExists {
return sb.NE(fieldExpression, value), nil
}
return sb.E(fieldExpression, value), nil
default:
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "exists operator is not supported for low cardinality column type %s", elementType)
}
case schema.ColumnTypeEnumString:
value = ""
if operator == qbtypes.FilterOperatorExists {
return sb.NE(fieldExpression, value), nil
}
return sb.E(fieldExpression, value), nil
case schema.ColumnTypeEnumUInt64, schema.ColumnTypeEnumUInt32, schema.ColumnTypeEnumUInt8:
value = 0
if operator == qbtypes.FilterOperatorExists {
return sb.NE(fieldExpression, value), nil
}
return sb.E(fieldExpression, value), nil
case schema.ColumnTypeEnumMap:
keyType := column.Type.(schema.MapColumnType).KeyType
if _, ok := keyType.(schema.LowCardinalityColumnType); !ok {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "key type %s is not supported for map column type %s", keyType, column.Type)
}
switch valueType := column.Type.(schema.MapColumnType).ValueType; valueType.GetType() {
case schema.ColumnTypeEnumString, schema.ColumnTypeEnumBool, schema.ColumnTypeEnumFloat64:
leftOperand := fmt.Sprintf("mapContains(%s, '%s')", column.Name, key.Name)
if key.Materialized {
leftOperand = telemetrytypes.FieldKeyToMaterializedColumnNameForExists(key)
}
if operator == qbtypes.FilterOperatorExists {
return sb.E(leftOperand, true), nil
}
return sb.NE(leftOperand, true), nil
default:
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "exists operator is not supported for map column type %s", valueType)
}
default:
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "exists operator is not supported for column type %s", column.Type)
}
}
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported operator: %v", operator)
}
func (c *conditionBuilder) ConditionFor(
ctx context.Context,
startNs uint64,
endNs uint64,
key *telemetrytypes.TelemetryFieldKey,
operator qbtypes.FilterOperator,
value any,
sb *sqlbuilder.SelectBuilder,
) (string, error) {
condition, err := c.conditionFor(ctx, startNs, endNs, key, operator, value, sb)
if err != nil {
return "", err
}
buildExistCondition := operator.AddDefaultExistsFilter()
switch key.FieldContext {
case telemetrytypes.FieldContextLog, telemetrytypes.FieldContextScope:
return condition, nil
case telemetrytypes.FieldContextResource, telemetrytypes.FieldContextAttribute:
// build exist condition for resource and attribute fields based on filter operator
}
if buildExistCondition {
existsCondition, err := c.conditionFor(ctx, startNs, endNs, key, qbtypes.FilterOperatorExists, nil, sb)
if err != nil {
return "", err
}
return sb.And(condition, existsCondition), nil
}
return condition, nil
}

128
pkg/telemetryaudit/const.go Normal file
View File

@@ -0,0 +1,128 @@
package telemetryaudit
import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
const (
// Internal Columns.
IDColumn = "id"
TimestampBucketStartColumn = "ts_bucket_start"
ResourceFingerPrintColumn = "resource_fingerprint"
// Intrinsic Columns.
TimestampColumn = "timestamp"
ObservedTimestampColumn = "observed_timestamp"
BodyColumn = "body"
EventNameColumn = "event_name"
TraceIDColumn = "trace_id"
SpanIDColumn = "span_id"
TraceFlagsColumn = "trace_flags"
SeverityTextColumn = "severity_text"
SeverityNumberColumn = "severity_number"
ScopeNameColumn = "scope_name"
ScopeVersionColumn = "scope_version"
// Contextual Columns.
AttributesStringColumn = "attributes_string"
AttributesNumberColumn = "attributes_number"
AttributesBoolColumn = "attributes_bool"
ScopeStringColumn = "scope_string"
)
var (
DefaultFullTextColumn = &telemetrytypes.TelemetryFieldKey{
Name: "body",
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextLog,
FieldDataType: telemetrytypes.FieldDataTypeString,
}
IntrinsicFields = map[string]telemetrytypes.TelemetryFieldKey{
"body": {
Name: "body",
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextLog,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
"trace_id": {
Name: "trace_id",
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextLog,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
"span_id": {
Name: "span_id",
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextLog,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
"trace_flags": {
Name: "trace_flags",
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextLog,
FieldDataType: telemetrytypes.FieldDataTypeNumber,
},
"severity_text": {
Name: "severity_text",
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextLog,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
"severity_number": {
Name: "severity_number",
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextLog,
FieldDataType: telemetrytypes.FieldDataTypeNumber,
},
"event_name": {
Name: "event_name",
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextLog,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
}
DefaultSortingOrder = []qbtypes.OrderBy{
{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: TimestampColumn,
},
},
Direction: qbtypes.OrderDirectionDesc,
},
{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: IDColumn,
},
},
Direction: qbtypes.OrderDirectionDesc,
},
}
)
var logsV2Columns = map[string]*schema.Column{
"ts_bucket_start": {Name: "ts_bucket_start", Type: schema.ColumnTypeUInt64},
"resource_fingerprint": {Name: "resource_fingerprint", Type: schema.ColumnTypeString},
"timestamp": {Name: "timestamp", Type: schema.ColumnTypeUInt64},
"observed_timestamp": {Name: "observed_timestamp", Type: schema.ColumnTypeUInt64},
"id": {Name: "id", Type: schema.ColumnTypeString},
"trace_id": {Name: "trace_id", Type: schema.ColumnTypeString},
"span_id": {Name: "span_id", Type: schema.ColumnTypeString},
"trace_flags": {Name: "trace_flags", Type: schema.ColumnTypeUInt32},
"severity_text": {Name: "severity_text", Type: schema.LowCardinalityColumnType{ElementType: schema.ColumnTypeString}},
"severity_number": {Name: "severity_number", Type: schema.ColumnTypeUInt8},
"body": {Name: "body", Type: schema.ColumnTypeString},
"attributes_string": {Name: "attributes_string", Type: schema.MapColumnType{KeyType: schema.LowCardinalityColumnType{ElementType: schema.ColumnTypeString}, ValueType: schema.ColumnTypeString}},
"attributes_number": {Name: "attributes_number", Type: schema.MapColumnType{KeyType: schema.LowCardinalityColumnType{ElementType: schema.ColumnTypeString}, ValueType: schema.ColumnTypeFloat64}},
"attributes_bool": {Name: "attributes_bool", Type: schema.MapColumnType{KeyType: schema.LowCardinalityColumnType{ElementType: schema.ColumnTypeString}, ValueType: schema.ColumnTypeBool}},
"resource": {Name: "resource", Type: schema.JSONColumnType{}},
"event_name": {Name: "event_name", Type: schema.ColumnTypeString},
"scope_name": {Name: "scope_name", Type: schema.ColumnTypeString},
"scope_version": {Name: "scope_version", Type: schema.ColumnTypeString},
"scope_string": {Name: "scope_string", Type: schema.MapColumnType{KeyType: schema.LowCardinalityColumnType{ElementType: schema.ColumnTypeString}, ValueType: schema.ColumnTypeString}},
}

View File

@@ -0,0 +1,155 @@
package telemetryaudit
import (
"context"
"fmt"
"strings"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
"golang.org/x/exp/maps"
)
type fieldMapper struct{}
func NewFieldMapper() qbtypes.FieldMapper {
return &fieldMapper{}
}
func (m *fieldMapper) getColumn(_ context.Context, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
switch key.FieldContext {
case telemetrytypes.FieldContextResource:
return []*schema.Column{logsV2Columns["resource"]}, nil
case telemetrytypes.FieldContextScope:
switch key.Name {
case "name", "scope.name", "scope_name":
return []*schema.Column{logsV2Columns["scope_name"]}, nil
case "version", "scope.version", "scope_version":
return []*schema.Column{logsV2Columns["scope_version"]}, nil
}
return []*schema.Column{logsV2Columns["scope_string"]}, nil
case telemetrytypes.FieldContextAttribute:
switch key.FieldDataType {
case telemetrytypes.FieldDataTypeString:
return []*schema.Column{logsV2Columns["attributes_string"]}, nil
case telemetrytypes.FieldDataTypeInt64, telemetrytypes.FieldDataTypeFloat64, telemetrytypes.FieldDataTypeNumber:
return []*schema.Column{logsV2Columns["attributes_number"]}, nil
case telemetrytypes.FieldDataTypeBool:
return []*schema.Column{logsV2Columns["attributes_bool"]}, nil
}
case telemetrytypes.FieldContextLog, telemetrytypes.FieldContextUnspecified:
col, ok := logsV2Columns[key.Name]
if !ok {
return nil, qbtypes.ErrColumnNotFound
}
return []*schema.Column{col}, nil
}
return nil, qbtypes.ErrColumnNotFound
}
func (m *fieldMapper) FieldFor(ctx context.Context, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) (string, error) {
columns, err := m.getColumn(ctx, key)
if err != nil {
return "", err
}
exprs := []string{}
existExpr := []string{}
for _, column := range columns {
switch column.Type.GetType() {
case schema.ColumnTypeEnumJSON:
if key.FieldContext == telemetrytypes.FieldContextResource {
exprs = append(exprs, fmt.Sprintf("%s.`%s`::String", column.Name, key.Name))
existExpr = append(existExpr, fmt.Sprintf("%s.`%s` IS NOT NULL", column.Name, key.Name))
} else {
return "", errors.Newf(errors.TypeInvalidInput, errors.CodeInvalidInput, "only resource context fields are supported for json columns in audit, got %s", key.FieldContext.String)
}
case schema.ColumnTypeEnumLowCardinality:
switch elementType := column.Type.(schema.LowCardinalityColumnType).ElementType; elementType.GetType() {
case schema.ColumnTypeEnumString:
exprs = append(exprs, column.Name)
default:
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported low cardinality element type %s", elementType)
}
case schema.ColumnTypeEnumString, schema.ColumnTypeEnumUInt64, schema.ColumnTypeEnumUInt32, schema.ColumnTypeEnumUInt8:
exprs = append(exprs, column.Name)
case schema.ColumnTypeEnumMap:
keyType := column.Type.(schema.MapColumnType).KeyType
if _, ok := keyType.(schema.LowCardinalityColumnType); !ok {
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "key type %s is not supported for map column type %s", keyType, column.Type)
}
switch valueType := column.Type.(schema.MapColumnType).ValueType; valueType.GetType() {
case schema.ColumnTypeEnumString, schema.ColumnTypeEnumBool, schema.ColumnTypeEnumFloat64:
if key.Materialized {
exprs = append(exprs, telemetrytypes.FieldKeyToMaterializedColumnName(key))
existExpr = append(existExpr, fmt.Sprintf("%s==true", telemetrytypes.FieldKeyToMaterializedColumnNameForExists(key)))
} else {
exprs = append(exprs, fmt.Sprintf("%s['%s']", column.Name, key.Name))
existExpr = append(existExpr, fmt.Sprintf("mapContains(%s, '%s')", column.Name, key.Name))
}
default:
return "", errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported map value type %s", valueType)
}
}
}
if len(exprs) == 1 {
return exprs[0], nil
} else if len(exprs) > 1 {
if len(existExpr) != len(exprs) {
return "", errors.New(errors.TypeInternal, errors.CodeInternal, "length of exist exprs doesn't match to that of exprs")
}
finalExprs := []string{}
for i, expr := range exprs {
finalExprs = append(finalExprs, fmt.Sprintf("%s, %s", existExpr[i], expr))
}
return "multiIf(" + strings.Join(finalExprs, ", ") + ", NULL)", nil
}
return columns[0].Name, nil
}
func (m *fieldMapper) ColumnFor(ctx context.Context, _, _ uint64, key *telemetrytypes.TelemetryFieldKey) ([]*schema.Column, error) {
return m.getColumn(ctx, key)
}
func (m *fieldMapper) ColumnExpressionFor(
ctx context.Context,
tsStart, tsEnd uint64,
field *telemetrytypes.TelemetryFieldKey,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
) (string, error) {
fieldExpression, err := m.FieldFor(ctx, tsStart, tsEnd, field)
if errors.Is(err, qbtypes.ErrColumnNotFound) {
keysForField := keys[field.Name]
if len(keysForField) == 0 {
if _, ok := logsV2Columns[field.Name]; ok {
field.FieldContext = telemetrytypes.FieldContextLog
fieldExpression, _ = m.FieldFor(ctx, tsStart, tsEnd, field)
} else {
correction, found := telemetrytypes.SuggestCorrection(field.Name, maps.Keys(keys))
if found {
return "", errors.Wrap(err, errors.TypeInvalidInput, errors.CodeInvalidInput, correction)
}
return "", errors.Wrapf(err, errors.TypeInvalidInput, errors.CodeInvalidInput, "field `%s` not found", field.Name)
}
} else if len(keysForField) == 1 {
fieldExpression, _ = m.FieldFor(ctx, tsStart, tsEnd, keysForField[0])
} else {
args := []string{}
for _, key := range keysForField {
fieldExpression, _ = m.FieldFor(ctx, tsStart, tsEnd, key)
args = append(args, fmt.Sprintf("toString(%s) != '', toString(%s)", fieldExpression, fieldExpression))
}
fieldExpression = fmt.Sprintf("multiIf(%s, NULL)", strings.Join(args, ", "))
}
}
return fmt.Sprintf("%s AS `%s`", sqlbuilder.Escape(fieldExpression), field.Name), nil
}

View File

@@ -0,0 +1,611 @@
package telemetryaudit
import (
"context"
"fmt"
"log/slog"
"strings"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryresourcefilter"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
)
type auditQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
resourceFilterStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation]
aggExprRewriter qbtypes.AggExprRewriter
fullTextColumn *telemetrytypes.TelemetryFieldKey
jsonKeyToKey qbtypes.JsonKeyToFieldFunc
}
var _ qbtypes.StatementBuilder[qbtypes.LogAggregation] = (*auditQueryStatementBuilder)(nil)
func NewAuditQueryStatementBuilder(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
aggExprRewriter qbtypes.AggExprRewriter,
fullTextColumn *telemetrytypes.TelemetryFieldKey,
jsonKeyToKey qbtypes.JsonKeyToFieldFunc,
) *auditQueryStatementBuilder {
auditSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryaudit")
resourceFilterStmtBuilder := telemetryresourcefilter.New[qbtypes.LogAggregation](
settings,
DBName,
LogsResourceTableName,
telemetrytypes.SignalLogs,
telemetrytypes.SourceAudit,
metadataStore,
fullTextColumn,
jsonKeyToKey,
)
return &auditQueryStatementBuilder{
logger: auditSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
cb: conditionBuilder,
resourceFilterStmtBuilder: resourceFilterStmtBuilder,
aggExprRewriter: aggExprRewriter,
fullTextColumn: fullTextColumn,
jsonKeyToKey: jsonKeyToKey,
}
}
func (b *auditQueryStatementBuilder) Build(
ctx context.Context,
start uint64,
end uint64,
requestType qbtypes.RequestType,
query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation],
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
start = querybuilder.ToNanoSecs(start)
end = querybuilder.ToNanoSecs(end)
keySelectors := getKeySelectors(query)
keys, _, err := b.metadataStore.GetKeysMulti(ctx, keySelectors)
if err != nil {
return nil, err
}
query = b.adjustKeys(ctx, keys, query, requestType)
q := sqlbuilder.NewSelectBuilder()
var stmt *qbtypes.Statement
switch requestType {
case qbtypes.RequestTypeRaw, qbtypes.RequestTypeRawStream:
stmt, err = b.buildListQuery(ctx, q, query, start, end, keys, variables)
case qbtypes.RequestTypeTimeSeries:
stmt, err = b.buildTimeSeriesQuery(ctx, q, query, start, end, keys, variables)
case qbtypes.RequestTypeScalar:
stmt, err = b.buildScalarQuery(ctx, q, query, start, end, keys, false, variables)
default:
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported request type: %s", requestType)
}
if err != nil {
return nil, err
}
return stmt, nil
}
func getKeySelectors(query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]) []*telemetrytypes.FieldKeySelector {
var keySelectors []*telemetrytypes.FieldKeySelector
for idx := range query.Aggregations {
aggExpr := query.Aggregations[idx]
selectors := querybuilder.QueryStringToKeysSelectors(aggExpr.Expression)
keySelectors = append(keySelectors, selectors...)
}
if query.Filter != nil && query.Filter.Expression != "" {
whereClauseSelectors := querybuilder.QueryStringToKeysSelectors(query.Filter.Expression)
keySelectors = append(keySelectors, whereClauseSelectors...)
}
for idx := range query.GroupBy {
groupBy := query.GroupBy[idx]
keySelectors = append(keySelectors, &telemetrytypes.FieldKeySelector{
Name: groupBy.Name,
Signal: telemetrytypes.SignalLogs,
FieldContext: groupBy.FieldContext,
FieldDataType: groupBy.FieldDataType,
})
}
for idx := range query.SelectFields {
selectField := query.SelectFields[idx]
keySelectors = append(keySelectors, &telemetrytypes.FieldKeySelector{
Name: selectField.Name,
Signal: telemetrytypes.SignalLogs,
FieldContext: selectField.FieldContext,
FieldDataType: selectField.FieldDataType,
})
}
for idx := range query.Order {
keySelectors = append(keySelectors, &telemetrytypes.FieldKeySelector{
Name: query.Order[idx].Key.Name,
Signal: telemetrytypes.SignalLogs,
FieldContext: query.Order[idx].Key.FieldContext,
FieldDataType: query.Order[idx].Key.FieldDataType,
})
}
for idx := range keySelectors {
keySelectors[idx].Signal = telemetrytypes.SignalLogs
keySelectors[idx].Source = telemetrytypes.SourceAudit
keySelectors[idx].SelectorMatchType = telemetrytypes.FieldSelectorMatchTypeExact
}
return keySelectors
}
func (b *auditQueryStatementBuilder) adjustKeys(ctx context.Context, keys map[string][]*telemetrytypes.TelemetryFieldKey, query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation], requestType qbtypes.RequestType) qbtypes.QueryBuilderQuery[qbtypes.LogAggregation] {
keys["id"] = append([]*telemetrytypes.TelemetryFieldKey{{
Name: "id",
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextLog,
FieldDataType: telemetrytypes.FieldDataTypeString,
}}, keys["id"]...)
keys["timestamp"] = append([]*telemetrytypes.TelemetryFieldKey{{
Name: "timestamp",
Signal: telemetrytypes.SignalLogs,
FieldContext: telemetrytypes.FieldContextLog,
FieldDataType: telemetrytypes.FieldDataTypeNumber,
}}, keys["timestamp"]...)
actions := querybuilder.AdjustKeysForAliasExpressions(&query, requestType)
actions = append(actions, querybuilder.AdjustDuplicateKeys(&query)...)
for idx := range query.SelectFields {
actions = append(actions, b.adjustKey(&query.SelectFields[idx], keys)...)
}
for idx := range query.GroupBy {
actions = append(actions, b.adjustKey(&query.GroupBy[idx].TelemetryFieldKey, keys)...)
}
for idx := range query.Order {
actions = append(actions, b.adjustKey(&query.Order[idx].Key.TelemetryFieldKey, keys)...)
}
for _, action := range actions {
b.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
}
return query
}
func (b *auditQueryStatementBuilder) adjustKey(key *telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) []string {
if _, ok := IntrinsicFields[key.Name]; ok {
intrinsicField := IntrinsicFields[key.Name]
return querybuilder.AdjustKey(key, keys, &intrinsicField)
}
return querybuilder.AdjustKey(key, keys, nil)
}
func (b *auditQueryStatementBuilder) buildListQuery(
ctx context.Context,
sb *sqlbuilder.SelectBuilder,
query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation],
start, end uint64,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
var (
cteFragments []string
cteArgs [][]any
)
if frag, args, err := b.maybeAttachResourceFilter(ctx, sb, query, start, end, variables); err != nil {
return nil, err
} else if frag != "" {
cteFragments = append(cteFragments, frag)
cteArgs = append(cteArgs, args)
}
sb.Select(TimestampColumn)
sb.SelectMore(IDColumn)
if len(query.SelectFields) == 0 {
sb.SelectMore(TraceIDColumn)
sb.SelectMore(SpanIDColumn)
sb.SelectMore(TraceFlagsColumn)
sb.SelectMore(SeverityTextColumn)
sb.SelectMore(SeverityNumberColumn)
sb.SelectMore(ScopeNameColumn)
sb.SelectMore(ScopeVersionColumn)
sb.SelectMore(BodyColumn)
sb.SelectMore(EventNameColumn)
sb.SelectMore(AttributesStringColumn)
sb.SelectMore(AttributesNumberColumn)
sb.SelectMore(AttributesBoolColumn)
sb.SelectMore(ScopeStringColumn)
} else {
for index := range query.SelectFields {
if query.SelectFields[index].Name == TimestampColumn || query.SelectFields[index].Name == IDColumn {
continue
}
colExpr, err := b.fm.ColumnExpressionFor(ctx, start, end, &query.SelectFields[index], keys)
if err != nil {
return nil, err
}
sb.SelectMore(colExpr)
}
}
sb.From(fmt.Sprintf("%s.%s", DBName, LogsTableName))
preparedWhereClause, err := b.addFilterCondition(ctx, sb, start, end, query, keys, variables)
if err != nil {
return nil, err
}
for _, orderBy := range query.Order {
colExpr, err := b.fm.ColumnExpressionFor(ctx, start, end, &orderBy.Key.TelemetryFieldKey, keys)
if err != nil {
return nil, err
}
sb.OrderBy(fmt.Sprintf("%s %s", colExpr, orderBy.Direction.StringValue()))
}
if query.Limit > 0 {
sb.Limit(query.Limit)
} else {
sb.Limit(100)
}
if query.Offset > 0 {
sb.Offset(query.Offset)
}
mainSQL, mainArgs := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
finalSQL := querybuilder.CombineCTEs(cteFragments) + mainSQL
finalArgs := querybuilder.PrependArgs(cteArgs, mainArgs)
stmt := &qbtypes.Statement{
Query: finalSQL,
Args: finalArgs,
}
if preparedWhereClause != nil {
stmt.Warnings = preparedWhereClause.Warnings
stmt.WarningsDocURL = preparedWhereClause.WarningsDocURL
}
return stmt, nil
}
func (b *auditQueryStatementBuilder) buildTimeSeriesQuery(
ctx context.Context,
sb *sqlbuilder.SelectBuilder,
query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation],
start, end uint64,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
var (
cteFragments []string
cteArgs [][]any
)
if frag, args, err := b.maybeAttachResourceFilter(ctx, sb, query, start, end, variables); err != nil {
return nil, err
} else if frag != "" {
cteFragments = append(cteFragments, frag)
cteArgs = append(cteArgs, args)
}
sb.SelectMore(fmt.Sprintf(
"toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL %d SECOND) AS ts",
int64(query.StepInterval.Seconds()),
))
var allGroupByArgs []any
fieldNames := make([]string, 0, len(query.GroupBy))
for _, gb := range query.GroupBy {
expr, args, err := querybuilder.CollisionHandledFinalExpr(ctx, start, end, &gb.TelemetryFieldKey, b.fm, b.cb, keys, telemetrytypes.FieldDataTypeString, b.jsonKeyToKey)
if err != nil {
return nil, err
}
colExpr := fmt.Sprintf("toString(%s) AS `%s`", expr, gb.Name)
allGroupByArgs = append(allGroupByArgs, args...)
sb.SelectMore(colExpr)
fieldNames = append(fieldNames, fmt.Sprintf("`%s`", gb.Name))
}
allAggChArgs := make([]any, 0)
for i, agg := range query.Aggregations {
rewritten, chArgs, err := b.aggExprRewriter.Rewrite(ctx, start, end, agg.Expression, uint64(query.StepInterval.Seconds()), keys)
if err != nil {
return nil, err
}
allAggChArgs = append(allAggChArgs, chArgs...)
sb.SelectMore(fmt.Sprintf("%s AS __result_%d", rewritten, i))
}
sb.From(fmt.Sprintf("%s.%s", DBName, LogsTableName))
preparedWhereClause, err := b.addFilterCondition(ctx, sb, start, end, query, keys, variables)
if err != nil {
return nil, err
}
var finalSQL string
var finalArgs []any
if query.Limit > 0 && len(query.GroupBy) > 0 {
cteSB := sqlbuilder.NewSelectBuilder()
cteStmt, err := b.buildScalarQuery(ctx, cteSB, query, start, end, keys, true, variables)
if err != nil {
return nil, err
}
cteFragments = append(cteFragments, fmt.Sprintf("__limit_cte AS (%s)", cteStmt.Query))
cteArgs = append(cteArgs, cteStmt.Args)
tuple := fmt.Sprintf("(%s)", strings.Join(fieldNames, ", "))
sb.Where(fmt.Sprintf("%s GLOBAL IN (SELECT %s FROM __limit_cte)", tuple, strings.Join(fieldNames, ", ")))
sb.GroupBy("ts")
sb.GroupBy(querybuilder.GroupByKeys(query.GroupBy)...)
if query.Having != nil && query.Having.Expression != "" {
rewriter := querybuilder.NewHavingExpressionRewriter()
rewrittenExpr, err := rewriter.RewriteForLogs(query.Having.Expression, query.Aggregations)
if err != nil {
return nil, err
}
sb.Having(rewrittenExpr)
}
if len(query.Order) != 0 {
for _, orderBy := range query.Order {
_, ok := aggOrderBy(orderBy, query)
if !ok {
sb.OrderBy(fmt.Sprintf("`%s` %s", orderBy.Key.Name, orderBy.Direction.StringValue()))
}
}
sb.OrderBy("ts desc")
}
combinedArgs := append(allGroupByArgs, allAggChArgs...)
mainSQL, mainArgs := sb.BuildWithFlavor(sqlbuilder.ClickHouse, combinedArgs...)
finalSQL = querybuilder.CombineCTEs(cteFragments) + mainSQL
finalArgs = querybuilder.PrependArgs(cteArgs, mainArgs)
} else {
sb.GroupBy("ts")
sb.GroupBy(querybuilder.GroupByKeys(query.GroupBy)...)
if query.Having != nil && query.Having.Expression != "" {
rewriter := querybuilder.NewHavingExpressionRewriter()
rewrittenExpr, err := rewriter.RewriteForLogs(query.Having.Expression, query.Aggregations)
if err != nil {
return nil, err
}
sb.Having(rewrittenExpr)
}
if len(query.Order) != 0 {
for _, orderBy := range query.Order {
_, ok := aggOrderBy(orderBy, query)
if !ok {
sb.OrderBy(fmt.Sprintf("`%s` %s", orderBy.Key.Name, orderBy.Direction.StringValue()))
}
}
sb.OrderBy("ts desc")
}
combinedArgs := append(allGroupByArgs, allAggChArgs...)
mainSQL, mainArgs := sb.BuildWithFlavor(sqlbuilder.ClickHouse, combinedArgs...)
finalSQL = querybuilder.CombineCTEs(cteFragments) + mainSQL
finalArgs = querybuilder.PrependArgs(cteArgs, mainArgs)
}
stmt := &qbtypes.Statement{
Query: finalSQL,
Args: finalArgs,
}
if preparedWhereClause != nil {
stmt.Warnings = preparedWhereClause.Warnings
stmt.WarningsDocURL = preparedWhereClause.WarningsDocURL
}
return stmt, nil
}
func (b *auditQueryStatementBuilder) buildScalarQuery(
ctx context.Context,
sb *sqlbuilder.SelectBuilder,
query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation],
start, end uint64,
keys map[string][]*telemetrytypes.TelemetryFieldKey,
skipResourceCTE bool,
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
var (
cteFragments []string
cteArgs [][]any
)
if frag, args, err := b.maybeAttachResourceFilter(ctx, sb, query, start, end, variables); err != nil {
return nil, err
} else if frag != "" && !skipResourceCTE {
cteFragments = append(cteFragments, frag)
cteArgs = append(cteArgs, args)
}
allAggChArgs := []any{}
var allGroupByArgs []any
for _, gb := range query.GroupBy {
expr, args, err := querybuilder.CollisionHandledFinalExpr(ctx, start, end, &gb.TelemetryFieldKey, b.fm, b.cb, keys, telemetrytypes.FieldDataTypeString, b.jsonKeyToKey)
if err != nil {
return nil, err
}
colExpr := fmt.Sprintf("toString(%s) AS `%s`", expr, gb.Name)
allGroupByArgs = append(allGroupByArgs, args...)
sb.SelectMore(colExpr)
}
rateInterval := (end - start) / querybuilder.NsToSeconds
if len(query.Aggregations) > 0 {
for idx := range query.Aggregations {
aggExpr := query.Aggregations[idx]
rewritten, chArgs, err := b.aggExprRewriter.Rewrite(ctx, start, end, aggExpr.Expression, rateInterval, keys)
if err != nil {
return nil, err
}
allAggChArgs = append(allAggChArgs, chArgs...)
sb.SelectMore(fmt.Sprintf("%s AS __result_%d", rewritten, idx))
}
}
sb.From(fmt.Sprintf("%s.%s", DBName, LogsTableName))
preparedWhereClause, err := b.addFilterCondition(ctx, sb, start, end, query, keys, variables)
if err != nil {
return nil, err
}
sb.GroupBy(querybuilder.GroupByKeys(query.GroupBy)...)
if query.Having != nil && query.Having.Expression != "" {
rewriter := querybuilder.NewHavingExpressionRewriter()
rewrittenExpr, err := rewriter.RewriteForLogs(query.Having.Expression, query.Aggregations)
if err != nil {
return nil, err
}
sb.Having(rewrittenExpr)
}
for _, orderBy := range query.Order {
idx, ok := aggOrderBy(orderBy, query)
if ok {
sb.OrderBy(fmt.Sprintf("__result_%d %s", idx, orderBy.Direction.StringValue()))
} else {
sb.OrderBy(fmt.Sprintf("`%s` %s", orderBy.Key.Name, orderBy.Direction.StringValue()))
}
}
if len(query.Order) == 0 {
sb.OrderBy("__result_0 DESC")
}
if query.Limit > 0 {
sb.Limit(query.Limit)
}
combinedArgs := append(allGroupByArgs, allAggChArgs...)
mainSQL, mainArgs := sb.BuildWithFlavor(sqlbuilder.ClickHouse, combinedArgs...)
finalSQL := querybuilder.CombineCTEs(cteFragments) + mainSQL
finalArgs := querybuilder.PrependArgs(cteArgs, mainArgs)
stmt := &qbtypes.Statement{
Query: finalSQL,
Args: finalArgs,
}
if preparedWhereClause != nil {
stmt.Warnings = preparedWhereClause.Warnings
stmt.WarningsDocURL = preparedWhereClause.WarningsDocURL
}
return stmt, nil
}
func (b *auditQueryStatementBuilder) addFilterCondition(
ctx context.Context,
sb *sqlbuilder.SelectBuilder,
start, end uint64,
query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation],
keys map[string][]*telemetrytypes.TelemetryFieldKey,
variables map[string]qbtypes.VariableItem,
) (*querybuilder.PreparedWhereClause, error) {
var preparedWhereClause *querybuilder.PreparedWhereClause
var err error
if query.Filter != nil && query.Filter.Expression != "" {
preparedWhereClause, err = querybuilder.PrepareWhereClause(query.Filter.Expression, querybuilder.FilterExprVisitorOpts{
Context: ctx,
Logger: b.logger,
FieldMapper: b.fm,
ConditionBuilder: b.cb,
FieldKeys: keys,
SkipResourceFilter: true,
FullTextColumn: b.fullTextColumn,
JsonKeyToKey: b.jsonKeyToKey,
Variables: variables,
StartNs: start,
EndNs: end,
})
if err != nil {
return nil, err
}
}
if preparedWhereClause != nil {
sb.AddWhereClause(preparedWhereClause.WhereClause)
}
startBucket := start/querybuilder.NsToSeconds - querybuilder.BucketAdjustment
var endBucket uint64
if end != 0 {
endBucket = end / querybuilder.NsToSeconds
}
if start != 0 {
sb.Where(sb.GE("timestamp", fmt.Sprintf("%d", start)), sb.GE("ts_bucket_start", startBucket))
}
if end != 0 {
sb.Where(sb.L("timestamp", fmt.Sprintf("%d", end)), sb.LE("ts_bucket_start", endBucket))
}
return preparedWhereClause, nil
}
func aggOrderBy(k qbtypes.OrderBy, q qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]) (int, bool) {
for i, agg := range q.Aggregations {
if k.Key.Name == agg.Alias || k.Key.Name == agg.Expression || k.Key.Name == fmt.Sprintf("%d", i) {
return i, true
}
}
return 0, false
}
func (b *auditQueryStatementBuilder) maybeAttachResourceFilter(
ctx context.Context,
sb *sqlbuilder.SelectBuilder,
query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation],
start, end uint64,
variables map[string]qbtypes.VariableItem,
) (cteSQL string, cteArgs []any, err error) {
stmt, err := b.resourceFilterStmtBuilder.Build(ctx, start, end, qbtypes.RequestTypeRaw, query, variables)
if err != nil {
return "", nil, err
}
sb.Where("resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter)")
return fmt.Sprintf("__resource_filter AS (%s)", stmt.Query), stmt.Args, nil
}

View File

@@ -0,0 +1,223 @@
package telemetryaudit
import (
"context"
"testing"
"time"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
"github.com/stretchr/testify/require"
)
func auditFieldKeyMap() map[string][]*telemetrytypes.TelemetryFieldKey {
key := func(name string, ctx telemetrytypes.FieldContext, dt telemetrytypes.FieldDataType, materialized bool) *telemetrytypes.TelemetryFieldKey {
return &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalLogs,
FieldContext: ctx,
FieldDataType: dt,
Materialized: materialized,
}
}
attr := telemetrytypes.FieldContextAttribute
res := telemetrytypes.FieldContextResource
str := telemetrytypes.FieldDataTypeString
i64 := telemetrytypes.FieldDataTypeInt64
return map[string][]*telemetrytypes.TelemetryFieldKey{
"service.name": {key("service.name", res, str, false)},
"signoz.audit.action": {key("signoz.audit.action", attr, str, true)},
"signoz.audit.outcome": {key("signoz.audit.outcome", attr, str, true)},
"signoz.audit.principal.email": {key("signoz.audit.principal.email", attr, str, true)},
"signoz.audit.principal.id": {key("signoz.audit.principal.id", attr, str, true)},
"signoz.audit.principal.type": {key("signoz.audit.principal.type", attr, str, true)},
"signoz.audit.resource.kind": {key("signoz.audit.resource.kind", res, str, false)},
"signoz.audit.resource.id": {key("signoz.audit.resource.id", res, str, false)},
"signoz.audit.action_category": {key("signoz.audit.action_category", attr, str, false)},
"signoz.audit.error.type": {key("signoz.audit.error.type", attr, str, false)},
"signoz.audit.error.code": {key("signoz.audit.error.code", attr, str, false)},
"http.request.method": {key("http.request.method", attr, str, false)},
"http.response.status_code": {key("http.response.status_code", attr, i64, false)},
}
}
func newTestAuditStatementBuilder() *auditQueryStatementBuilder {
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = auditFieldKeyMap()
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, nil)
return NewAuditQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore,
fm,
cb,
aggExprRewriter,
DefaultFullTextColumn,
nil,
)
}
func TestStatementBuilder(t *testing.T) {
statementBuilder := newTestAuditStatementBuilder()
ctx := context.Background()
testCases := []struct {
name string
requestType qbtypes.RequestType
query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]
expected qbtypes.Statement
expectedErr error
}{
// List: all actions by a specific user (materialized principal.id filter)
{
name: "ListByPrincipalID",
requestType: qbtypes.RequestTypeRaw,
query: qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
Source: telemetrytypes.SourceAudit,
Filter: &qbtypes.Filter{
Expression: "signoz.audit.principal.id = '019a-1234-abcd-5678'",
},
Limit: 100,
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_audit.distributed_logs_resource WHERE true AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ?) SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, scope_name, scope_version, body, event_name, attributes_string, attributes_number, attributes_bool, scope_string FROM signoz_audit.distributed_logs WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND (`attribute_string_signoz$$audit$$principal$$id` = ? AND `attribute_string_signoz$$audit$$principal$$id_exists` = ?) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{uint64(1747945619), uint64(1747983448), "019a-1234-abcd-5678", true, "1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448), 100},
},
},
// List: all failed actions (materialized outcome filter)
{
name: "ListByOutcomeFailure",
requestType: qbtypes.RequestTypeRaw,
query: qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
Source: telemetrytypes.SourceAudit,
Filter: &qbtypes.Filter{
Expression: "signoz.audit.outcome = 'failure'",
},
Limit: 100,
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_audit.distributed_logs_resource WHERE true AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ?) SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, scope_name, scope_version, body, event_name, attributes_string, attributes_number, attributes_bool, scope_string FROM signoz_audit.distributed_logs WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND (`attribute_string_signoz$$audit$$outcome` = ? AND `attribute_string_signoz$$audit$$outcome_exists` = ?) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{uint64(1747945619), uint64(1747983448), "failure", true, "1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448), 100},
},
},
// List: change history of a specific dashboard (two materialized column AND)
{
name: "ListByResourceKindAndID",
requestType: qbtypes.RequestTypeRaw,
query: qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
Source: telemetrytypes.SourceAudit,
Filter: &qbtypes.Filter{
Expression: "signoz.audit.resource.kind = 'dashboard' AND signoz.audit.resource.id = '019b-5678-efgh-9012'",
},
Limit: 100,
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_audit.distributed_logs_resource WHERE ((simpleJSONExtractString(labels, 'signoz.audit.resource.kind') = ? AND labels LIKE ? AND labels LIKE ?) AND (simpleJSONExtractString(labels, 'signoz.audit.resource.id') = ? AND labels LIKE ? AND labels LIKE ?)) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ?) SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, scope_name, scope_version, body, event_name, attributes_string, attributes_number, attributes_bool, scope_string FROM signoz_audit.distributed_logs WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND true AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"dashboard", "%signoz.audit.resource.kind%", "%signoz.audit.resource.kind\":\"dashboard%", "019b-5678-efgh-9012", "%signoz.audit.resource.id%", "%signoz.audit.resource.id\":\"019b-5678-efgh-9012%", uint64(1747945619), uint64(1747983448), "1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448), 100},
},
},
// List: all dashboard deletions (compliance — resource.kind + action AND)
{
name: "ListByResourceKindAndAction",
requestType: qbtypes.RequestTypeRaw,
query: qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
Source: telemetrytypes.SourceAudit,
Filter: &qbtypes.Filter{
Expression: "signoz.audit.resource.kind = 'dashboard' AND signoz.audit.action = 'delete'",
},
Limit: 100,
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_audit.distributed_logs_resource WHERE (simpleJSONExtractString(labels, 'signoz.audit.resource.kind') = ? AND labels LIKE ? AND labels LIKE ?) AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ?) SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, scope_name, scope_version, body, event_name, attributes_string, attributes_number, attributes_bool, scope_string FROM signoz_audit.distributed_logs WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND (`attribute_string_signoz$$audit$$action` = ? AND `attribute_string_signoz$$audit$$action_exists` = ?) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{"dashboard", "%signoz.audit.resource.kind%", "%signoz.audit.resource.kind\":\"dashboard%", uint64(1747945619), uint64(1747983448), "delete", true, "1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448), 100},
},
},
// List: all actions by service accounts (materialized principal.type)
{
name: "ListByPrincipalType",
requestType: qbtypes.RequestTypeRaw,
query: qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
Source: telemetrytypes.SourceAudit,
Filter: &qbtypes.Filter{
Expression: "signoz.audit.principal.type = 'service_account'",
},
Limit: 100,
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_audit.distributed_logs_resource WHERE true AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ?) SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, scope_name, scope_version, body, event_name, attributes_string, attributes_number, attributes_bool, scope_string FROM signoz_audit.distributed_logs WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND (`attribute_string_signoz$$audit$$principal$$type` = ? AND `attribute_string_signoz$$audit$$principal$$type_exists` = ?) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? LIMIT ?",
Args: []any{uint64(1747945619), uint64(1747983448), "service_account", true, "1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448), 100},
},
},
// Scalar: alert — count forbidden errors (outcome + action AND)
{
name: "ScalarCountByOutcomeAndAction",
requestType: qbtypes.RequestTypeScalar,
query: qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
Source: telemetrytypes.SourceAudit,
StepInterval: qbtypes.Step{Duration: 60 * time.Second},
Filter: &qbtypes.Filter{
Expression: "signoz.audit.outcome = 'failure' AND signoz.audit.action = 'update'",
},
Aggregations: []qbtypes.LogAggregation{
{Expression: "count()"},
},
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_audit.distributed_logs_resource WHERE true AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ?) SELECT count() AS __result_0 FROM signoz_audit.distributed_logs WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND ((`attribute_string_signoz$$audit$$outcome` = ? AND `attribute_string_signoz$$audit$$outcome_exists` = ?) AND (`attribute_string_signoz$$audit$$action` = ? AND `attribute_string_signoz$$audit$$action_exists` = ?)) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? ORDER BY __result_0 DESC",
Args: []any{uint64(1747945619), uint64(1747983448), "failure", true, "update", true, "1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448)},
},
},
// TimeSeries: failures grouped by principal email with top-N limit
{
name: "TimeSeriesFailuresGroupedByPrincipal",
requestType: qbtypes.RequestTypeTimeSeries,
query: qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
Source: telemetrytypes.SourceAudit,
StepInterval: qbtypes.Step{Duration: 60 * time.Second},
Aggregations: []qbtypes.LogAggregation{
{Expression: "count()"},
},
Filter: &qbtypes.Filter{
Expression: "signoz.audit.outcome = 'failure'",
},
GroupBy: []qbtypes.GroupByKey{
{TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: "signoz.audit.principal.email"}},
},
Limit: 5,
},
expected: qbtypes.Statement{
Query: "WITH __resource_filter AS (SELECT fingerprint FROM signoz_audit.distributed_logs_resource WHERE true AND seen_at_ts_bucket_start >= ? AND seen_at_ts_bucket_start <= ?), __limit_cte AS (SELECT toString(multiIf(`attribute_string_signoz$$audit$$principal$$email_exists` = ?, `attribute_string_signoz$$audit$$principal$$email`, NULL)) AS `signoz.audit.principal.email`, count() AS __result_0 FROM signoz_audit.distributed_logs WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND (`attribute_string_signoz$$audit$$outcome` = ? AND `attribute_string_signoz$$audit$$outcome_exists` = ?) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? GROUP BY `signoz.audit.principal.email` ORDER BY __result_0 DESC LIMIT ?) SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 60 SECOND) AS ts, toString(multiIf(`attribute_string_signoz$$audit$$principal$$email_exists` = ?, `attribute_string_signoz$$audit$$principal$$email`, NULL)) AS `signoz.audit.principal.email`, count() AS __result_0 FROM signoz_audit.distributed_logs WHERE resource_fingerprint GLOBAL IN (SELECT fingerprint FROM __resource_filter) AND (`attribute_string_signoz$$audit$$outcome` = ? AND `attribute_string_signoz$$audit$$outcome_exists` = ?) AND timestamp >= ? AND ts_bucket_start >= ? AND timestamp < ? AND ts_bucket_start <= ? AND (`signoz.audit.principal.email`) GLOBAL IN (SELECT `signoz.audit.principal.email` FROM __limit_cte) GROUP BY ts, `signoz.audit.principal.email`",
Args: []any{uint64(1747945619), uint64(1747983448), true, "failure", true, "1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448), 5, true, "failure", true, "1747947419000000000", uint64(1747945619), "1747983448000000000", uint64(1747983448)},
},
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
q, err := statementBuilder.Build(ctx, 1747947419000, 1747983448000, testCase.requestType, testCase.query, nil)
if testCase.expectedErr != nil {
require.Error(t, err)
require.Contains(t, err.Error(), testCase.expectedErr.Error())
} else {
require.NoError(t, err)
require.Equal(t, testCase.expected.Query, q.Query)
require.Equal(t, testCase.expected.Args, q.Args)
}
})
}
}

View File

@@ -0,0 +1,12 @@
package telemetryaudit
const (
DBName = "signoz_audit"
LogsTableName = "distributed_logs"
LogsLocalTableName = "logs"
TagAttributesTableName = "distributed_tag_attributes"
TagAttributesLocalTableName = "tag_attributes"
LogAttributeKeysTblName = "distributed_logs_attribute_keys"
LogResourceKeysTblName = "distributed_logs_resource_keys"
LogsResourceTableName = "distributed_logs_resource"
)

View File

@@ -45,6 +45,7 @@ func NewLogQueryStatementBuilder(
DBName,
LogsResourceV2TableName,
telemetrytypes.SignalLogs,
telemetrytypes.SourceUnspecified,
metadataStore,
fullTextColumn,
jsonKeyToKey,

View File

@@ -13,6 +13,7 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
@@ -27,6 +28,7 @@ import (
var (
ErrFailedToGetTracesKeys = errors.Newf(errors.TypeInternal, errors.CodeInternal, "failed to get traces keys")
ErrFailedToGetLogsKeys = errors.Newf(errors.TypeInternal, errors.CodeInternal, "failed to get logs keys")
ErrFailedToGetAuditKeys = errors.Newf(errors.TypeInternal, errors.CodeInternal, "failed to get audit keys")
ErrFailedToGetTblStatement = errors.Newf(errors.TypeInternal, errors.CodeInternal, "failed to get tbl statement")
ErrFailedToGetMetricsKeys = errors.Newf(errors.TypeInternal, errors.CodeInternal, "failed to get metrics keys")
ErrFailedToGetMeterKeys = errors.Newf(errors.TypeInternal, errors.CodeInternal, "failed to get meter keys")
@@ -50,6 +52,11 @@ type telemetryMetaStore struct {
logAttributeKeysTblName string
logResourceKeysTblName string
logsV2TblName string
auditDBName string
auditV2TblName string
auditFieldsTblName string
auditAttributeKeysTblName string
auditResourceKeysTblName string
relatedMetadataDBName string
relatedMetadataTblName string
columnEvolutionMetadataTblName string
@@ -79,6 +86,11 @@ func NewTelemetryMetaStore(
logsFieldsTblName string,
logAttributeKeysTblName string,
logResourceKeysTblName string,
auditDBName string,
auditV2TblName string,
auditFieldsTblName string,
auditAttributeKeysTblName string,
auditResourceKeysTblName string,
relatedMetadataDBName string,
relatedMetadataTblName string,
columnEvolutionMetadataTblName string,
@@ -101,6 +113,11 @@ func NewTelemetryMetaStore(
logsFieldsTblName: logsFieldsTblName,
logAttributeKeysTblName: logAttributeKeysTblName,
logResourceKeysTblName: logResourceKeysTblName,
auditDBName: auditDBName,
auditV2TblName: auditV2TblName,
auditFieldsTblName: auditFieldsTblName,
auditAttributeKeysTblName: auditAttributeKeysTblName,
auditResourceKeysTblName: auditResourceKeysTblName,
relatedMetadataDBName: relatedMetadataDBName,
relatedMetadataTblName: relatedMetadataTblName,
columnEvolutionMetadataTblName: columnEvolutionMetadataTblName,
@@ -592,6 +609,232 @@ func (t *telemetryMetaStore) getLogsKeys(ctx context.Context, fieldKeySelectors
return keys, complete, nil
}
func (t *telemetryMetaStore) auditTblStatementToFieldKeys(ctx context.Context) ([]*telemetrytypes.TelemetryFieldKey, error) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.TelemetrySignal: telemetrytypes.SignalLogs.StringValue(),
instrumentationtypes.CodeNamespace: "metadata",
instrumentationtypes.CodeFunctionName: "auditTblStatementToFieldKeys",
})
query := fmt.Sprintf("SHOW CREATE TABLE %s.%s", t.auditDBName, t.auditV2TblName)
statements := []telemetrytypes.ShowCreateTableStatement{}
err := t.telemetrystore.ClickhouseDB().Select(ctx, &statements, query)
if err != nil {
return nil, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetTblStatement.Error())
}
materialisedKeys, err := ExtractFieldKeysFromTblStatement(statements[0].Statement)
if err != nil {
return nil, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
}
for idx := range materialisedKeys {
materialisedKeys[idx].Signal = telemetrytypes.SignalLogs
}
return materialisedKeys, nil
}
func (t *telemetryMetaStore) getAuditKeys(ctx context.Context, fieldKeySelectors []*telemetrytypes.FieldKeySelector) ([]*telemetrytypes.TelemetryFieldKey, bool, error) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.TelemetrySignal: telemetrytypes.SignalLogs.StringValue(),
instrumentationtypes.CodeNamespace: "metadata",
instrumentationtypes.CodeFunctionName: "getAuditKeys",
})
if len(fieldKeySelectors) == 0 {
return nil, true, nil
}
matKeys, err := t.auditTblStatementToFieldKeys(ctx)
if err != nil {
return nil, false, err
}
mapOfKeys := make(map[string]*telemetrytypes.TelemetryFieldKey)
for _, key := range matKeys {
mapOfKeys[key.Name+";"+key.FieldContext.StringValue()+";"+key.FieldDataType.StringValue()] = key
}
var queries []string
var allArgs []any
queryAttributeTable := false
queryResourceTable := false
for _, selector := range fieldKeySelectors {
if selector.FieldContext == telemetrytypes.FieldContextUnspecified {
queryAttributeTable = true
queryResourceTable = true
break
} else if selector.FieldContext == telemetrytypes.FieldContextAttribute {
queryAttributeTable = true
} else if selector.FieldContext == telemetrytypes.FieldContextResource {
queryResourceTable = true
}
}
tablesToQuery := []struct {
fieldContext telemetrytypes.FieldContext
shouldQuery bool
}{
{telemetrytypes.FieldContextAttribute, queryAttributeTable},
{telemetrytypes.FieldContextResource, queryResourceTable},
}
for _, table := range tablesToQuery {
if !table.shouldQuery {
continue
}
fieldContext := table.fieldContext
var tblName string
if fieldContext == telemetrytypes.FieldContextAttribute {
tblName = t.auditDBName + "." + t.auditAttributeKeysTblName
} else {
tblName = t.auditDBName + "." + t.auditResourceKeysTblName
}
sb := sqlbuilder.Select(
"name AS tag_key",
fmt.Sprintf("'%s' AS tag_type", fieldContext.TagType()),
"lower(datatype) AS tag_data_type",
fmt.Sprintf("%d AS priority", getPriorityForContext(fieldContext)),
).From(tblName)
var limit int
conds := []string{}
for _, fieldKeySelector := range fieldKeySelectors {
if fieldKeySelector.FieldContext != telemetrytypes.FieldContextUnspecified && fieldKeySelector.FieldContext != fieldContext {
continue
}
fieldKeyConds := []string{}
if fieldKeySelector.SelectorMatchType == telemetrytypes.FieldSelectorMatchTypeExact {
fieldKeyConds = append(fieldKeyConds, sb.E("name", fieldKeySelector.Name))
} else {
fieldKeyConds = append(fieldKeyConds, sb.ILike("name", "%"+escapeForLike(fieldKeySelector.Name)+"%"))
}
if fieldKeySelector.FieldDataType != telemetrytypes.FieldDataTypeUnspecified {
fieldKeyConds = append(fieldKeyConds, sb.E("datatype", fieldKeySelector.FieldDataType.TagDataType()))
}
if len(fieldKeyConds) > 0 {
conds = append(conds, sb.And(fieldKeyConds...))
}
limit += fieldKeySelector.Limit
}
if len(conds) > 0 {
sb.Where(sb.Or(conds...))
}
sb.GroupBy("name", "datatype")
if limit == 0 {
limit = 1000
}
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
queries = append(queries, query)
allArgs = append(allArgs, args...)
}
if len(queries) == 0 {
return []*telemetrytypes.TelemetryFieldKey{}, true, nil
}
var limit int
for _, fieldKeySelector := range fieldKeySelectors {
limit += fieldKeySelector.Limit
}
if limit == 0 {
limit = 1000
}
mainQuery := fmt.Sprintf(`
SELECT tag_key, tag_type, tag_data_type, max(priority) as priority
FROM (
%s
) AS combined_results
GROUP BY tag_key, tag_type, tag_data_type
ORDER BY priority
LIMIT %d
`, strings.Join(queries, " UNION ALL "), limit+1)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, mainQuery, allArgs...)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
}
defer rows.Close()
keys := []*telemetrytypes.TelemetryFieldKey{}
rowCount := 0
searchTexts := []string{}
for _, fieldKeySelector := range fieldKeySelectors {
searchTexts = append(searchTexts, fieldKeySelector.Name)
}
for rows.Next() {
rowCount++
if rowCount > limit {
break
}
var name string
var fieldContext telemetrytypes.FieldContext
var fieldDataType telemetrytypes.FieldDataType
var priority uint8
err = rows.Scan(&name, &fieldContext, &fieldDataType, &priority)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
}
key, ok := mapOfKeys[name+";"+fieldContext.StringValue()+";"+fieldDataType.StringValue()]
if !ok {
key = &telemetrytypes.TelemetryFieldKey{
Name: name,
Signal: telemetrytypes.SignalLogs,
FieldContext: fieldContext,
FieldDataType: fieldDataType,
}
}
keys = append(keys, key)
mapOfKeys[name+";"+fieldContext.StringValue()+";"+fieldDataType.StringValue()] = key
}
if rows.Err() != nil {
return nil, false, errors.Wrap(rows.Err(), errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
}
complete := rowCount <= limit
// Add intrinsic audit fields (same as logs intrinsics: body, severity_text, etc.)
staticKeys := maps.Keys(telemetryaudit.IntrinsicFields)
for _, key := range staticKeys {
found := false
for _, v := range searchTexts {
if v == "" || strings.Contains(key, v) {
found = true
break
}
}
if found {
if field, exists := telemetryaudit.IntrinsicFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
}
}
}
return keys, complete, nil
}
func getPriorityForContext(ctx telemetrytypes.FieldContext) int {
switch ctx {
case telemetrytypes.FieldContextLog:
@@ -889,7 +1132,11 @@ func (t *telemetryMetaStore) GetKeys(ctx context.Context, fieldKeySelector *tele
case telemetrytypes.SignalTraces:
keys, complete, err = t.getTracesKeys(ctx, selectors)
case telemetrytypes.SignalLogs:
keys, complete, err = t.getLogsKeys(ctx, selectors)
if fieldKeySelector.Source == telemetrytypes.SourceAudit {
keys, complete, err = t.getAuditKeys(ctx, selectors)
} else {
keys, complete, err = t.getLogsKeys(ctx, selectors)
}
case telemetrytypes.SignalMetrics:
if fieldKeySelector.Source == telemetrytypes.SourceMeter {
keys, complete, err = t.getMeterSourceMetricKeys(ctx, selectors)
@@ -938,6 +1185,7 @@ func (t *telemetryMetaStore) GetKeys(ctx context.Context, fieldKeySelector *tele
func (t *telemetryMetaStore) GetKeysMulti(ctx context.Context, fieldKeySelectors []*telemetrytypes.FieldKeySelector) (map[string][]*telemetrytypes.TelemetryFieldKey, bool, error) {
logsSelectors := []*telemetrytypes.FieldKeySelector{}
auditSelectors := []*telemetrytypes.FieldKeySelector{}
tracesSelectors := []*telemetrytypes.FieldKeySelector{}
metricsSelectors := []*telemetrytypes.FieldKeySelector{}
meterSourceMetricsSelectors := []*telemetrytypes.FieldKeySelector{}
@@ -945,7 +1193,11 @@ func (t *telemetryMetaStore) GetKeysMulti(ctx context.Context, fieldKeySelectors
for _, fieldKeySelector := range fieldKeySelectors {
switch fieldKeySelector.Signal {
case telemetrytypes.SignalLogs:
logsSelectors = append(logsSelectors, fieldKeySelector)
if fieldKeySelector.Source == telemetrytypes.SourceAudit {
auditSelectors = append(auditSelectors, fieldKeySelector)
} else {
logsSelectors = append(logsSelectors, fieldKeySelector)
}
case telemetrytypes.SignalTraces:
tracesSelectors = append(tracesSelectors, fieldKeySelector)
case telemetrytypes.SignalMetrics:
@@ -965,6 +1217,10 @@ func (t *telemetryMetaStore) GetKeysMulti(ctx context.Context, fieldKeySelectors
if err != nil {
return nil, false, err
}
auditKeys, auditComplete, err := t.getAuditKeys(ctx, auditSelectors)
if err != nil {
return nil, false, err
}
tracesKeys, tracesComplete, err := t.getTracesKeys(ctx, tracesSelectors)
if err != nil {
return nil, false, err
@@ -979,12 +1235,15 @@ func (t *telemetryMetaStore) GetKeysMulti(ctx context.Context, fieldKeySelectors
return nil, false, err
}
// Complete only if all queries are complete
complete := logsComplete && tracesComplete && metricsComplete
complete := logsComplete && auditComplete && tracesComplete && metricsComplete
mapOfKeys := make(map[string][]*telemetrytypes.TelemetryFieldKey)
for _, key := range logsKeys {
mapOfKeys[key.Name] = append(mapOfKeys[key.Name], key)
}
for _, key := range auditKeys {
mapOfKeys[key.Name] = append(mapOfKeys[key.Name], key)
}
for _, key := range tracesKeys {
mapOfKeys[key.Name] = append(mapOfKeys[key.Name], key)
}
@@ -1338,6 +1597,96 @@ func (t *telemetryMetaStore) getLogFieldValues(ctx context.Context, fieldValueSe
return values, complete, nil
}
func (t *telemetryMetaStore) getAuditFieldValues(ctx context.Context, fieldValueSelector *telemetrytypes.FieldValueSelector) (*telemetrytypes.TelemetryFieldValues, bool, error) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.TelemetrySignal: telemetrytypes.SignalLogs.StringValue(),
instrumentationtypes.CodeNamespace: "metadata",
instrumentationtypes.CodeFunctionName: "getAuditFieldValues",
})
limit := fieldValueSelector.Limit
if limit == 0 {
limit = 50
}
sb := sqlbuilder.Select("DISTINCT string_value, number_value").From(t.auditDBName + "." + t.auditFieldsTblName)
if fieldValueSelector.Name != "" {
sb.Where(sb.E("tag_key", fieldValueSelector.Name))
}
if fieldValueSelector.FieldContext != telemetrytypes.FieldContextUnspecified {
sb.Where(sb.E("tag_type", fieldValueSelector.FieldContext.TagType()))
}
if fieldValueSelector.FieldDataType != telemetrytypes.FieldDataTypeUnspecified {
sb.Where(sb.E("tag_data_type", fieldValueSelector.FieldDataType.TagDataType()))
}
if fieldValueSelector.Value != "" {
switch fieldValueSelector.FieldDataType {
case telemetrytypes.FieldDataTypeString:
sb.Where(sb.ILike("string_value", "%"+escapeForLike(fieldValueSelector.Value)+"%"))
case telemetrytypes.FieldDataTypeNumber:
sb.Where(sb.IsNotNull("number_value"))
sb.Where(sb.ILike("toString(number_value)", "%"+escapeForLike(fieldValueSelector.Value)+"%"))
case telemetrytypes.FieldDataTypeUnspecified:
sb.Where(sb.Or(
sb.ILike("string_value", "%"+escapeForLike(fieldValueSelector.Value)+"%"),
sb.ILike("toString(number_value)", "%"+escapeForLike(fieldValueSelector.Value)+"%"),
))
}
}
sb.Limit(limit + 1)
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, query, args...)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
}
defer rows.Close()
values := &telemetrytypes.TelemetryFieldValues{}
seen := make(map[string]bool)
rowCount := 0
totalCount := 0
for rows.Next() {
rowCount++
var stringValue string
var numberValue float64
err = rows.Scan(&stringValue, &numberValue)
if err != nil {
return nil, false, errors.Wrap(err, errors.TypeInternal, errors.CodeInternal, ErrFailedToGetAuditKeys.Error())
}
if stringValue != "" && !seen[stringValue] {
if totalCount >= limit {
break
}
values.StringValues = append(values.StringValues, stringValue)
seen[stringValue] = true
totalCount++
}
if numberValue != 0 {
if totalCount >= limit {
break
}
if !seen[fmt.Sprintf("%f", numberValue)] {
values.NumberValues = append(values.NumberValues, numberValue)
seen[fmt.Sprintf("%f", numberValue)] = true
totalCount++
}
}
}
complete := rowCount <= limit
return values, complete, nil
}
// getMetricFieldValues returns field values and whether the result is complete.
func (t *telemetryMetaStore) getMetricFieldValues(ctx context.Context, fieldValueSelector *telemetrytypes.FieldValueSelector) (*telemetrytypes.TelemetryFieldValues, bool, error) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
@@ -1628,7 +1977,11 @@ func (t *telemetryMetaStore) GetAllValues(ctx context.Context, fieldValueSelecto
case telemetrytypes.SignalTraces:
values, complete, err = t.getSpanFieldValues(ctx, fieldValueSelector)
case telemetrytypes.SignalLogs:
values, complete, err = t.getLogFieldValues(ctx, fieldValueSelector)
if fieldValueSelector.Source == telemetrytypes.SourceAudit {
values, complete, err = t.getAuditFieldValues(ctx, fieldValueSelector)
} else {
values, complete, err = t.getLogFieldValues(ctx, fieldValueSelector)
}
case telemetrytypes.SignalMetrics:
if fieldValueSelector.Source == telemetrytypes.SourceMeter {
values, complete, err = t.getMeterSourceMetricFieldValues(ctx, fieldValueSelector)

View File

@@ -5,6 +5,7 @@ import (
"testing"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
@@ -37,6 +38,11 @@ func TestGetFirstSeenFromMetricMetadata(t *testing.T) {
telemetrylogs.TagAttributesV2TableName,
telemetrylogs.LogAttributeKeysTblName,
telemetrylogs.LogResourceKeysTblName,
telemetryaudit.DBName,
telemetryaudit.LogsTableName,
telemetryaudit.TagAttributesTableName,
telemetryaudit.LogAttributeKeysTblName,
telemetryaudit.LogResourceKeysTblName,
DBName,
AttributesMetadataLocalTableName,
ColumnEvolutionMetadataTableName,

View File

@@ -7,6 +7,7 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
@@ -36,6 +37,11 @@ func newTestTelemetryMetaStoreTestHelper(store telemetrystore.TelemetryStore) te
telemetrylogs.TagAttributesV2TableName,
telemetrylogs.LogAttributeKeysTblName,
telemetrylogs.LogResourceKeysTblName,
telemetryaudit.DBName,
telemetryaudit.LogsTableName,
telemetryaudit.TagAttributesTableName,
telemetryaudit.LogAttributeKeysTblName,
telemetryaudit.LogResourceKeysTblName,
DBName,
AttributesMetadataLocalTableName,
ColumnEvolutionMetadataTableName,

View File

@@ -21,6 +21,7 @@ type resourceFilterStatementBuilder[T any] struct {
conditionBuilder qbtypes.ConditionBuilder
metadataStore telemetrytypes.MetadataStore
signal telemetrytypes.Signal
source telemetrytypes.Source
fullTextColumn *telemetrytypes.TelemetryFieldKey
jsonKeyToKey qbtypes.JsonKeyToFieldFunc
@@ -37,6 +38,7 @@ func New[T any](
dbName string,
tableName string,
signal telemetrytypes.Signal,
source telemetrytypes.Source,
metadataStore telemetrytypes.MetadataStore,
fullTextColumn *telemetrytypes.TelemetryFieldKey,
jsonKeyToKey qbtypes.JsonKeyToFieldFunc,
@@ -52,6 +54,7 @@ func New[T any](
conditionBuilder: cb,
metadataStore: metadataStore,
signal: signal,
source: source,
fullTextColumn: fullTextColumn,
jsonKeyToKey: jsonKeyToKey,
}
@@ -72,6 +75,7 @@ func (b *resourceFilterStatementBuilder[T]) getKeySelectors(query qbtypes.QueryB
continue
}
keySelectors[idx].Signal = b.signal
keySelectors[idx].Source = b.source
keySelectors[idx].SelectorMatchType = telemetrytypes.FieldSelectorMatchTypeExact
filteredKeySelectors = append(filteredKeySelectors, keySelectors[idx])
}

View File

@@ -375,6 +375,7 @@ func TestResourceFilterStatementBuilder_Traces(t *testing.T) {
"signoz_traces",
"distributed_traces_v3_resource",
telemetrytypes.SignalTraces,
telemetrytypes.SourceUnspecified,
mockMetadataStore,
nil,
nil,
@@ -592,6 +593,7 @@ func TestResourceFilterStatementBuilder_Logs(t *testing.T) {
"signoz_logs",
"distributed_logs_v2_resource",
telemetrytypes.SignalLogs,
telemetrytypes.SourceUnspecified,
mockMetadataStore,
nil,
nil,
@@ -653,6 +655,7 @@ func TestResourceFilterStatementBuilder_Variables(t *testing.T) {
"signoz_traces",
"distributed_traces_v3_resource",
telemetrytypes.SignalTraces,
telemetrytypes.SourceUnspecified,
mockMetadataStore,
nil,
nil,

View File

@@ -49,6 +49,7 @@ func NewTraceQueryStatementBuilder(
DBName,
TracesResourceV3TableName,
telemetrytypes.SignalTraces,
telemetrytypes.SourceUnspecified,
metadataStore,
nil,
nil,

View File

@@ -39,6 +39,7 @@ func NewTraceOperatorStatementBuilder(
DBName,
TracesResourceV3TableName,
telemetrytypes.SignalTraces,
telemetrytypes.SourceUnspecified,
metadataStore,
nil,
nil,

View File

@@ -20,7 +20,6 @@ var (
ErrCodeAPIKeyAlreadyExists = errors.MustNewCode("api_key_already_exists")
ErrCodeAPIKeytNotFound = errors.MustNewCode("api_key_not_found")
ErrCodeAPIKeyExpired = errors.MustNewCode("api_key_expired")
errInvalidAPIKeyName = errors.New(errors.TypeInvalidInput, ErrCodeAPIKeyInvalidInput, "name must be 180 characters long and contain only lowercase letters (a-z) and hyphens (-)")
)
type FactorAPIKey struct {
@@ -113,7 +112,7 @@ func (key *PostableFactorAPIKey) UnmarshalJSON(data []byte) error {
}
if match := factorAPIKeyNameRegex.MatchString(temp.Name); !match {
return errInvalidAPIKeyName
return errors.Newf(errors.TypeInvalidInput, ErrCodeAPIKeyInvalidInput, "name must conform to the regex: %s", factorAPIKeyNameRegex.String())
}
if temp.ExpiresAt != 0 && time.Now().After(time.Unix(int64(temp.ExpiresAt), 0)) {
@@ -133,7 +132,7 @@ func (key *UpdatableFactorAPIKey) UnmarshalJSON(data []byte) error {
}
if match := factorAPIKeyNameRegex.MatchString(temp.Name); !match {
return errInvalidAPIKeyName
return errors.Newf(errors.TypeInvalidInput, ErrCodeAPIKeyInvalidInput, "name must conform to the regex: %s", factorAPIKeyNameRegex.String())
}
if temp.ExpiresAt != 0 && time.Now().After(time.Unix(int64(temp.ExpiresAt), 0)) {

View File

@@ -23,7 +23,6 @@ var (
ErrCodeServiceAccountNotFound = errors.MustNewCode("service_account_not_found")
ErrCodeServiceAccountRoleAlreadyExists = errors.MustNewCode("service_account_role_already_exists")
ErrCodeServiceAccountOperationUnsupported = errors.MustNewCode("service_account_operation_unsupported")
errInvalidServiceAccountName = errors.New(errors.TypeInvalidInput, ErrCodeServiceAccountInvalidInput, "name must be 150 characters long and contain only lowercase letters (a-z) and hyphens (-)")
)
var (
@@ -215,7 +214,7 @@ func (serviceAccount *PostableServiceAccount) UnmarshalJSON(data []byte) error {
}
if match := serviceAccountNameRegex.MatchString(temp.Name); !match {
return errInvalidServiceAccountName
return errors.Newf(errors.TypeInvalidInput, ErrCodeServiceAccountInvalidInput, "name must conform to the regex: %s", serviceAccountNameRegex.String())
}
*serviceAccount = PostableServiceAccount(temp)

View File

@@ -7,11 +7,13 @@ type Source struct {
}
var (
SourceAudit = Source{valuer.NewString("audit")}
SourceMeter = Source{valuer.NewString("meter")}
SourceUnspecified = Source{valuer.NewString("")}
)
// Enum returns the acceptable values for Source.
// TODO: Add SourceAudit once the frontend is ready for consumption.
func (Source) Enum() []any {
return []any{
SourceMeter,

View File

@@ -12,6 +12,7 @@ pytest_plugins = [
"fixtures.sqlite",
"fixtures.zookeeper",
"fixtures.signoz",
"fixtures.audit",
"fixtures.logs",
"fixtures.traces",
"fixtures.metrics",

View File

@@ -0,0 +1,404 @@
import datetime
import json
from abc import ABC
from typing import Any, Callable, Generator, List, Optional
import numpy as np
import pytest
from ksuid import KsuidMs
from fixtures import types
from fixtures.fingerprint import LogsOrTracesFingerprint
class AuditResource(ABC):
labels: str
fingerprint: str
seen_at_ts_bucket_start: np.int64
def __init__(
self,
labels: dict[str, str],
fingerprint: str,
seen_at_ts_bucket_start: np.int64,
) -> None:
self.labels = json.dumps(labels, separators=(",", ":"))
self.fingerprint = fingerprint
self.seen_at_ts_bucket_start = seen_at_ts_bucket_start
def np_arr(self) -> np.array:
return np.array(
[
self.labels,
self.fingerprint,
self.seen_at_ts_bucket_start,
]
)
class AuditResourceOrAttributeKeys(ABC):
name: str
datatype: str
def __init__(self, name: str, datatype: str) -> None:
self.name = name
self.datatype = datatype
def np_arr(self) -> np.array:
return np.array([self.name, self.datatype])
class AuditTagAttributes(ABC):
unix_milli: np.int64
tag_key: str
tag_type: str
tag_data_type: str
string_value: str
int64_value: Optional[np.int64]
float64_value: Optional[np.float64]
def __init__(
self,
timestamp: datetime.datetime,
tag_key: str,
tag_type: str,
tag_data_type: str,
string_value: Optional[str],
int64_value: Optional[np.int64],
float64_value: Optional[np.float64],
) -> None:
self.unix_milli = np.int64(int(timestamp.timestamp() * 1e3))
self.tag_key = tag_key
self.tag_type = tag_type
self.tag_data_type = tag_data_type
self.string_value = string_value or ""
self.int64_value = int64_value
self.float64_value = float64_value
def np_arr(self) -> np.array:
return np.array(
[
self.unix_milli,
self.tag_key,
self.tag_type,
self.tag_data_type,
self.string_value,
self.int64_value,
self.float64_value,
]
)
class AuditLog(ABC):
"""Represents a single audit log event in signoz_audit.
Matches the ClickHouse DDL from the schema migration (ticket #1936):
- Database: signoz_audit
- Local table: logs
- Distributed table: distributed_logs
- No resources_string column (resource JSON only)
- Has event_name column
- 7 materialized columns auto-populated from attributes_string at INSERT time
"""
ts_bucket_start: np.uint64
resource_fingerprint: str
timestamp: np.uint64
observed_timestamp: np.uint64
id: str
trace_id: str
span_id: str
trace_flags: np.uint32
severity_text: str
severity_number: np.uint8
body: str
scope_name: str
scope_version: str
scope_string: dict[str, str]
attributes_string: dict[str, str]
attributes_number: dict[str, np.float64]
attributes_bool: dict[str, bool]
resource_json: dict[str, str]
event_name: str
resource: List[AuditResource]
tag_attributes: List[AuditTagAttributes]
resource_keys: List[AuditResourceOrAttributeKeys]
attribute_keys: List[AuditResourceOrAttributeKeys]
def __init__(
self,
timestamp: Optional[datetime.datetime] = None,
resources: dict[str, Any] = {},
attributes: dict[str, Any] = {},
body: str = "",
event_name: str = "",
severity_text: str = "INFO",
trace_id: str = "",
span_id: str = "",
trace_flags: np.uint32 = 0,
scope_name: str = "signoz.audit",
scope_version: str = "",
) -> None:
if timestamp is None:
timestamp = datetime.datetime.now()
self.tag_attributes = []
self.attribute_keys = []
self.resource_keys = []
self.timestamp = np.uint64(int(timestamp.timestamp() * 1e9))
self.observed_timestamp = self.timestamp
minute = timestamp.minute
bucket_minute = 0 if minute < 30 else 30
bucket_start = timestamp.replace(minute=bucket_minute, second=0, microsecond=0)
self.ts_bucket_start = np.uint64(int(bucket_start.timestamp()))
self.id = str(KsuidMs(datetime=timestamp))
self.trace_id = trace_id
self.span_id = span_id
self.trace_flags = trace_flags
self.severity_text = severity_text
self.severity_number = np.uint8(9 if severity_text == "INFO" else 17)
self.body = body
self.event_name = event_name
# Resources — JSON column only (no resources_string in audit DDL)
self.resource_json = {k: str(v) for k, v in resources.items()}
for k, v in self.resource_json.items():
self.tag_attributes.append(
AuditTagAttributes(
timestamp=timestamp,
tag_key=k,
tag_type="resource",
tag_data_type="string",
string_value=str(v),
int64_value=None,
float64_value=None,
)
)
self.resource_keys.append(
AuditResourceOrAttributeKeys(name=k, datatype="string")
)
self.resource_fingerprint = LogsOrTracesFingerprint(
self.resource_json
).calculate()
# Process attributes by type
self.attributes_string = {}
self.attributes_number = {}
self.attributes_bool = {}
for k, v in attributes.items():
if isinstance(v, bool):
self.attributes_bool[k] = v
self.tag_attributes.append(
AuditTagAttributes(
timestamp=timestamp,
tag_key=k,
tag_type="tag",
tag_data_type="bool",
string_value=None,
int64_value=None,
float64_value=None,
)
)
self.attribute_keys.append(
AuditResourceOrAttributeKeys(name=k, datatype="bool")
)
elif isinstance(v, int):
self.attributes_number[k] = np.float64(v)
self.tag_attributes.append(
AuditTagAttributes(
timestamp=timestamp,
tag_key=k,
tag_type="tag",
tag_data_type="int64",
string_value=None,
int64_value=np.int64(v),
float64_value=None,
)
)
self.attribute_keys.append(
AuditResourceOrAttributeKeys(name=k, datatype="int64")
)
elif isinstance(v, float):
self.attributes_number[k] = np.float64(v)
self.tag_attributes.append(
AuditTagAttributes(
timestamp=timestamp,
tag_key=k,
tag_type="tag",
tag_data_type="float64",
string_value=None,
int64_value=None,
float64_value=np.float64(v),
)
)
self.attribute_keys.append(
AuditResourceOrAttributeKeys(name=k, datatype="float64")
)
else:
self.attributes_string[k] = str(v)
self.tag_attributes.append(
AuditTagAttributes(
timestamp=timestamp,
tag_key=k,
tag_type="tag",
tag_data_type="string",
string_value=str(v),
int64_value=None,
float64_value=None,
)
)
self.attribute_keys.append(
AuditResourceOrAttributeKeys(name=k, datatype="string")
)
self.scope_name = scope_name
self.scope_version = scope_version
self.scope_string = {}
self.resource = [
AuditResource(
labels=self.resource_json,
fingerprint=self.resource_fingerprint,
seen_at_ts_bucket_start=self.ts_bucket_start,
)
]
def np_arr(self) -> np.array:
return np.array(
[
self.ts_bucket_start,
self.resource_fingerprint,
self.timestamp,
self.observed_timestamp,
self.id,
self.trace_id,
self.span_id,
self.trace_flags,
self.severity_text,
self.severity_number,
self.body,
self.scope_name,
self.scope_version,
self.scope_string,
self.attributes_string,
self.attributes_number,
self.attributes_bool,
self.resource_json,
self.event_name,
]
)
@pytest.fixture(name="insert_audit_logs", scope="function")
def insert_audit_logs(
clickhouse: types.TestContainerClickhouse,
) -> Generator[Callable[[List[AuditLog]], None], Any, None]:
def _insert_audit_logs(logs: List[AuditLog]) -> None:
resources: List[AuditResource] = []
for log in logs:
resources.extend(log.resource)
if len(resources) > 0:
clickhouse.conn.insert(
database="signoz_audit",
table="distributed_logs_resource",
data=[resource.np_arr() for resource in resources],
column_names=[
"labels",
"fingerprint",
"seen_at_ts_bucket_start",
],
)
tag_attributes: List[AuditTagAttributes] = []
for log in logs:
tag_attributes.extend(log.tag_attributes)
if len(tag_attributes) > 0:
clickhouse.conn.insert(
database="signoz_audit",
table="distributed_tag_attributes",
data=[ta.np_arr() for ta in tag_attributes],
column_names=[
"unix_milli",
"tag_key",
"tag_type",
"tag_data_type",
"string_value",
"int64_value",
"float64_value",
],
)
attribute_keys: List[AuditResourceOrAttributeKeys] = []
for log in logs:
attribute_keys.extend(log.attribute_keys)
if len(attribute_keys) > 0:
clickhouse.conn.insert(
database="signoz_audit",
table="distributed_logs_attribute_keys",
data=[ak.np_arr() for ak in attribute_keys],
column_names=["name", "datatype"],
)
resource_keys: List[AuditResourceOrAttributeKeys] = []
for log in logs:
resource_keys.extend(log.resource_keys)
if len(resource_keys) > 0:
clickhouse.conn.insert(
database="signoz_audit",
table="distributed_logs_resource_keys",
data=[rk.np_arr() for rk in resource_keys],
column_names=["name", "datatype"],
)
clickhouse.conn.insert(
database="signoz_audit",
table="distributed_logs",
data=[log.np_arr() for log in logs],
column_names=[
"ts_bucket_start",
"resource_fingerprint",
"timestamp",
"observed_timestamp",
"id",
"trace_id",
"span_id",
"trace_flags",
"severity_text",
"severity_number",
"body",
"scope_name",
"scope_version",
"scope_string",
"attributes_string",
"attributes_number",
"attributes_bool",
"resource",
"event_name",
],
)
yield _insert_audit_logs
cluster = clickhouse.env["SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_CLUSTER"]
for table in [
"logs",
"logs_resource",
"tag_attributes",
"logs_attribute_keys",
"logs_resource_keys",
]:
clickhouse.conn.query(
f"TRUNCATE TABLE signoz_audit.{table} ON CLUSTER '{cluster}' SYNC"
)

View File

@@ -38,6 +38,7 @@ class OrderBy:
class BuilderQuery:
signal: str
name: str = "A"
source: Optional[str] = None
limit: Optional[int] = None
filter_expression: Optional[str] = None
select_fields: Optional[List[TelemetryFieldKey]] = None
@@ -48,6 +49,8 @@ class BuilderQuery:
"signal": self.signal,
"name": self.name,
}
if self.source:
spec["source"] = self.source
if self.limit is not None:
spec["limit"] = self.limit
if self.filter_expression:
@@ -55,7 +58,9 @@ class BuilderQuery:
if self.select_fields:
spec["selectFields"] = [f.to_dict() for f in self.select_fields]
if self.order:
spec["order"] = [o.to_dict() for o in self.order]
spec["order"] = [
o.to_dict() if hasattr(o, "to_dict") else o for o in self.order
]
return {"type": "builder_query", "spec": spec}
@@ -76,7 +81,9 @@ class TraceOperatorQuery:
if self.limit is not None:
spec["limit"] = self.limit
if self.order:
spec["order"] = [o.to_dict() for o in self.order]
spec["order"] = [
o.to_dict() if hasattr(o, "to_dict") else o for o in self.order
]
return {"type": "builder_trace_operator", "spec": spec}
@@ -442,6 +449,7 @@ def build_scalar_query(
signal: str,
aggregations: List[Dict],
*,
source: Optional[str] = None,
group_by: Optional[List[Dict]] = None,
order: Optional[List[Dict]] = None,
limit: Optional[int] = None,
@@ -458,6 +466,9 @@ def build_scalar_query(
"aggregations": aggregations,
}
if source:
spec["source"] = source
if group_by:
spec["groupBy"] = group_by

View File

@@ -0,0 +1,441 @@
from datetime import datetime, timedelta, timezone
from http import HTTPStatus
from typing import Callable, List
import pytest
from fixtures import types
from fixtures.audit import AuditLog
from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
from fixtures.querier import (
BuilderQuery,
build_logs_aggregation,
build_order_by,
build_scalar_query,
make_query_request,
)
def test_audit_list_all(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_audit_logs: Callable[[List[AuditLog]], None],
) -> None:
"""List audit events across multiple resource types — verify count, ordering, and fields."""
now = datetime.now(tz=timezone.utc)
insert_audit_logs(
[
AuditLog(
timestamp=now - timedelta(seconds=3),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "alert-rule",
"signoz.audit.resource.id": "alert-001",
},
attributes={
"signoz.audit.principal.id": "user-010",
"signoz.audit.principal.email": "ops@acme.com",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "create",
"signoz.audit.outcome": "success",
},
body="ops@acme.com (user-010) created alert-rule (alert-001)",
event_name="alert-rule.created",
severity_text="INFO",
),
AuditLog(
timestamp=now - timedelta(seconds=2),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "saved-view",
"signoz.audit.resource.id": "view-001",
},
attributes={
"signoz.audit.principal.id": "user-010",
"signoz.audit.principal.email": "ops@acme.com",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "update",
"signoz.audit.outcome": "success",
},
body="ops@acme.com (user-010) updated saved-view (view-001)",
event_name="saved-view.updated",
severity_text="INFO",
),
AuditLog(
timestamp=now - timedelta(seconds=1),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "user",
"signoz.audit.resource.id": "user-020",
},
attributes={
"signoz.audit.principal.id": "user-010",
"signoz.audit.principal.email": "ops@acme.com",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "update",
"signoz.audit.action_category": "access_control",
"signoz.audit.outcome": "success",
},
body="ops@acme.com (user-010) updated user (user-020)",
event_name="user.role.changed",
severity_text="INFO",
),
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
now = datetime.now(tz=timezone.utc)
response = make_query_request(
signoz,
token,
start_ms=int((now - timedelta(seconds=30)).timestamp() * 1000),
end_ms=int(now.timestamp() * 1000),
queries=[
BuilderQuery(
signal="logs",
source="audit",
limit=100,
order=[build_order_by("timestamp"), build_order_by("id")],
).to_dict()
],
request_type="raw",
)
assert response.status_code == HTTPStatus.OK
assert response.json()["status"] == "success"
rows = response.json()["data"]["data"]["results"][0]["rows"]
assert len(rows) == 3
# Most recent first
assert rows[0]["data"]["event_name"] == "user.role.changed"
assert rows[1]["data"]["event_name"] == "saved-view.updated"
assert rows[2]["data"]["event_name"] == "alert-rule.created"
# Verify event_name and body are present
assert rows[0]["data"]["body"] == "ops@acme.com (user-010) updated user (user-020)"
assert rows[0]["data"]["severity_text"] == "INFO"
@pytest.mark.parametrize(
"filter_expression,expected_count,expected_event_names",
[
pytest.param(
"signoz.audit.principal.id = 'user-001'",
3,
{"session.login", "dashboard.updated", "dashboard.created"},
id="filter_by_principal_id",
),
pytest.param(
"signoz.audit.outcome = 'failure'",
1,
{"dashboard.deleted"},
id="filter_by_outcome_failure",
),
pytest.param(
"signoz.audit.resource.kind = 'dashboard'"
" AND signoz.audit.resource.id = 'dash-001'",
3,
{"dashboard.deleted", "dashboard.updated", "dashboard.created"},
id="filter_by_resource_kind_and_id",
),
pytest.param(
"signoz.audit.principal.type = 'service_account'",
1,
{"serviceaccount.apikey.created"},
id="filter_by_principal_type",
),
pytest.param(
"signoz.audit.resource.kind = 'dashboard'"
" AND signoz.audit.action = 'delete'",
1,
{"dashboard.deleted"},
id="filter_by_resource_kind_and_action",
),
],
)
def test_audit_filter(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_audit_logs: Callable[[List[AuditLog]], None],
filter_expression: str,
expected_count: int,
expected_event_names: set,
) -> None:
"""Parametrized audit filter tests covering the documented query patterns."""
now = datetime.now(tz=timezone.utc)
insert_audit_logs(
[
AuditLog(
timestamp=now - timedelta(seconds=5),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "dashboard",
"signoz.audit.resource.id": "dash-001",
},
attributes={
"signoz.audit.principal.id": "user-001",
"signoz.audit.principal.email": "alice@acme.com",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "create",
"signoz.audit.action_category": "configuration_change",
"signoz.audit.outcome": "success",
},
body="alice@acme.com created dashboard",
event_name="dashboard.created",
),
AuditLog(
timestamp=now - timedelta(seconds=4),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "dashboard",
"signoz.audit.resource.id": "dash-001",
},
attributes={
"signoz.audit.principal.id": "user-001",
"signoz.audit.principal.email": "alice@acme.com",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "update",
"signoz.audit.action_category": "configuration_change",
"signoz.audit.outcome": "success",
},
body="alice@acme.com updated dashboard",
event_name="dashboard.updated",
),
AuditLog(
timestamp=now - timedelta(seconds=3),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "dashboard",
"signoz.audit.resource.id": "dash-001",
},
attributes={
"signoz.audit.principal.id": "user-002",
"signoz.audit.principal.email": "viewer@acme.com",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "delete",
"signoz.audit.action_category": "configuration_change",
"signoz.audit.outcome": "failure",
"signoz.audit.error.type": "forbidden",
"signoz.audit.error.code": "authz_forbidden",
},
body="viewer@acme.com failed to delete dashboard",
event_name="dashboard.deleted",
severity_text="ERROR",
),
AuditLog(
timestamp=now - timedelta(seconds=2),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "serviceaccount",
"signoz.audit.resource.id": "sa-001",
},
attributes={
"signoz.audit.principal.id": "sa-001",
"signoz.audit.principal.email": "",
"signoz.audit.principal.type": "service_account",
"signoz.audit.action": "create",
"signoz.audit.action_category": "access_control",
"signoz.audit.outcome": "success",
},
body="sa-001 created serviceaccount",
event_name="serviceaccount.apikey.created",
),
AuditLog(
timestamp=now - timedelta(seconds=1),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "session",
"signoz.audit.resource.id": "*",
},
attributes={
"signoz.audit.principal.id": "user-001",
"signoz.audit.principal.email": "alice@acme.com",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "login",
"signoz.audit.action_category": "access_control",
"signoz.audit.outcome": "success",
},
body="alice@acme.com login session",
event_name="session.login",
),
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
now = datetime.now(tz=timezone.utc)
response = make_query_request(
signoz,
token,
start_ms=int((now - timedelta(seconds=30)).timestamp() * 1000),
end_ms=int(now.timestamp() * 1000),
queries=[
BuilderQuery(
signal="logs",
source="audit",
limit=100,
filter_expression=filter_expression,
order=[build_order_by("timestamp"), build_order_by("id")],
).to_dict()
],
request_type="raw",
)
assert response.status_code == HTTPStatus.OK
rows = response.json()["data"]["data"]["results"][0]["rows"]
assert len(rows) == expected_count
actual_event_names = {row["data"]["event_name"] for row in rows}
assert actual_event_names == expected_event_names
def test_audit_scalar_count_failures(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_audit_logs: Callable[[List[AuditLog]], None],
) -> None:
"""Alert query — count multiple failures from different principals."""
now = datetime.now(tz=timezone.utc)
insert_audit_logs(
[
AuditLog(
timestamp=now - timedelta(seconds=3),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "dashboard",
"signoz.audit.resource.id": "dash-100",
},
attributes={
"signoz.audit.principal.id": "user-050",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "delete",
"signoz.audit.outcome": "failure",
},
body="user-050 failed to delete dashboard",
event_name="dashboard.deleted",
severity_text="ERROR",
),
AuditLog(
timestamp=now - timedelta(seconds=2),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "alert-rule",
"signoz.audit.resource.id": "alert-200",
},
attributes={
"signoz.audit.principal.id": "user-060",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "update",
"signoz.audit.outcome": "failure",
},
body="user-060 failed to update alert-rule",
event_name="alert-rule.updated",
severity_text="ERROR",
),
AuditLog(
timestamp=now - timedelta(seconds=1),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "dashboard",
"signoz.audit.resource.id": "dash-100",
},
attributes={
"signoz.audit.principal.id": "user-050",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "update",
"signoz.audit.outcome": "success",
},
body="user-050 updated dashboard",
event_name="dashboard.updated",
),
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
now = datetime.now(tz=timezone.utc)
response = make_query_request(
signoz,
token,
start_ms=int((now - timedelta(seconds=30)).timestamp() * 1000),
end_ms=int(now.timestamp() * 1000),
queries=[
build_scalar_query(
name="A",
signal="logs",
source="audit",
aggregations=[build_logs_aggregation("count()")],
filter_expression="signoz.audit.outcome = 'failure'",
)
],
request_type="scalar",
)
assert response.status_code == HTTPStatus.OK
assert response.json()["status"] == "success"
scalar_data = response.json()["data"]["data"]["results"][0].get("data", [])
assert len(scalar_data) == 1
assert scalar_data[0][0] == 2
def test_audit_does_not_leak_into_logs(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_audit_logs: Callable[[List[AuditLog]], None],
) -> None:
"""A single audit event in signoz_audit must not appear in regular log queries."""
now = datetime.now(tz=timezone.utc)
insert_audit_logs(
[
AuditLog(
timestamp=now - timedelta(seconds=1),
resources={
"service.name": "signoz",
"signoz.audit.resource.kind": "organization",
"signoz.audit.resource.id": "org-999",
},
attributes={
"signoz.audit.principal.id": "user-admin",
"signoz.audit.principal.type": "user",
"signoz.audit.action": "update",
"signoz.audit.outcome": "success",
},
body="user-admin updated organization (org-999)",
event_name="organization.updated",
),
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
now = datetime.now(tz=timezone.utc)
response = make_query_request(
signoz,
token,
start_ms=int((now - timedelta(seconds=30)).timestamp() * 1000),
end_ms=int(now.timestamp() * 1000),
queries=[
BuilderQuery(
signal="logs",
limit=100,
order=[build_order_by("timestamp"), build_order_by("id")],
).to_dict()
],
request_type="raw",
)
assert response.status_code == HTTPStatus.OK
rows = response.json()["data"]["data"]["results"][0].get("rows") or []
audit_bodies = [
row["data"]["body"]
for row in rows
if "signoz.audit"
in row["data"].get("attributes_string", {}).get("signoz.audit.action", "")
]
assert len(audit_bodies) == 0