Compare commits

...

8 Commits

Author SHA1 Message Date
Tushar Vats
ae6e170211 refactor(logparsingpipeline): unmarshal collector config once 2026-10-13 18:08:55 +05:30
Tushar Vats
8bedbfe650 refactor(logparsingpipeline): remove json_body_dual_ingestion flag 2026-10-13 17:45:06 +05:30
Tushar Vats
2a3c693986 refactor(logparsingpipeline): rename normalizer position helpers 2026-10-13 16:45:33 +05:30
Tushar Vats
5549f016f3 test(logparsingpipeline): cover processor ordering around signozlogsnormalizer 2026-10-13 10:58:17 +05:30
Tushar Vats
b099142f76 feat(logparsingpipeline): order user pipelines around signozlogsnormalizer
User pipelines anchor on the first normalizer per its opamp.position; the prepended normalize operator is dropped when the processor already runs first.
2026-10-13 02:07:51 +05:30
Tushar Vats
ba01bc0418 fix(tests): use promoted_paths API in create_json_index fixture (#13105)
#### Description

- `create_json_index` still hit `/api/v1/logs/promote_paths`, removed in
#12839 → `test_indexed_paths` failed with 404.
- Fixture now posts to `/api/v1/promoted_paths` with `signal: logs` /
`context: body`; paths are bare (no `body.` prefix) per #13021.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-10-12 12:41:14 +00:00
Aditya Singh
566089d48e fix(explorer): preserve order by in list view for explorers (#13085)
#### Description

- The list view kept its Order by value in component state, so the sort
never reached the URL. It was lost on back navigation from trace
details, ignored on a deep link, and never saved with a view. The sort
now lives in `compositeQuery`.
- The order-by sanitizer dropped every valid list sort, because a list
panel orders by row fields rather than a group-by key or an aggregation.
It is now gated on panel type and unchanged for Time Series and Table.
- A raw panel (list or trace) cannot order by an aggregation. So,
switching to a list or trace panel now clears the order, so each view
applies its own default.
- Run Query set a flag that suppressed the sanitizer. The flag was never
cleared, so every later init on the page skipped sanitizing. It is now
cleared after the init it applies to.
- Same change applied to the LLM Observability list and trace views.
Those two tabs offer different fields, so before the clear each one
could send the other's.

#### Issues closed by this PR

Closes https://github.com/SigNoz/engineering-pod/issues/6043

Closes https://github.com/SigNoz/signoz/issues/12355

#### Additional Information

- An order no longer survives a round trip through a list or trace
panel. Set a sort on Time Series, open List, and the aggregated order is
gone. This is the trade for closing the leak between the two raw tabs.
- A pre-existing link or saved view that carries an aggregation order on
a list panel falls back to that view's default. The app no longer
produces such a link.

#### Screenshot

Traces back navigation keeps the order by



https://github.com/user-attachments/assets/c98215ee-479c-4d6c-bce8-c8dca3185d99


Switch to table or time series preserves the order by if valid else
resets. (This was existing behaviour onload but did not work on tab
switch due to a run query flag issue we fixed..mentioned in desc above)



https://github.com/user-attachments/assets/26a68377-cf04-442e-8d9e-683de6e773a1




Switch to list or trace view resets the order to respective
default(existing behaviour for traces and llm. new for logs)



https://github.com/user-attachments/assets/680447e7-7dd1-44ac-bcd4-716aa3e796e7
2026-10-12 12:08:08 +00:00
Jugal Kishore
785c2e2195 feat(onboarding): add Bun/Elysia, SNMP, Vercel Eve, Microsoft Agent Framework, Android datasources (#13100)
Some checks failed
build-staging / prepare (push) Has been cancelled
build-staging / js-build (push) Has been cancelled
build-staging / go-build (push) Has been cancelled
build-staging / staging (push) Has been cancelled
cacheci / tests (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
#### Description

- Adds Bun / Elysia, SNMP, Vercel Eve and Microsoft Agent Framework to
the onboarding picker.
- Replaces the Android Java and Android Kotlin cards with one Android
card, since the new doc supersedes both.
- New logos: `bun`, `microsoft`. Android reuses the existing mark.

#### Issues closed by this PR

Closes SigNoz/signoz.io#4270
Closes SigNoz/signoz.io#4240
Closes SigNoz/signoz.io#4236
Closes SigNoz/signoz.io#4231
Closes SigNoz/signoz.io#4219
2026-10-12 07:22:30 +00:00
27 changed files with 983 additions and 172 deletions

View File

@@ -1 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" xml:space="preserve" viewBox="0 0 60 60"><linearGradient id="a" x1="15.959" x2="44.307" y1="-13.014" y2="15.333" gradientTransform="matrix(1 0 0 -1 0 61)" gradientUnits="userSpaceOnUse"><stop offset=".097" style="stop-color:#0095d5"/><stop offset=".301" style="stop-color:#238ad9"/><stop offset=".621" style="stop-color:#557bde"/><stop offset=".864" style="stop-color:#7472e2"/><stop offset="1" style="stop-color:#806ee3"/></linearGradient><path d="m0 60 30.1-30.1L60 60z" style="fill:url(#a)"/><linearGradient id="b" x1="4.209" x2="20.673" y1="48.941" y2="65.405" gradientTransform="matrix(1 0 0 -1 0 61)" gradientUnits="userSpaceOnUse"><stop offset=".118" style="stop-color:#0095d5"/><stop offset=".418" style="stop-color:#3c83dc"/><stop offset=".696" style="stop-color:#6d74e1"/><stop offset=".833" style="stop-color:#806ee3"/></linearGradient><path d="M0 0h30.1L0 32.5z" style="fill:url(#b)"/><linearGradient id="c" x1="-10.102" x2="45.731" y1="5.836" y2="61.669" gradientTransform="matrix(1 0 0 -1 0 61)" gradientUnits="userSpaceOnUse"><stop offset=".107" style="stop-color:#c757bc"/><stop offset=".214" style="stop-color:#d0609a"/><stop offset=".425" style="stop-color:#e1725c"/><stop offset=".605" style="stop-color:#ee7e2f"/><stop offset=".743" style="stop-color:#f58613"/><stop offset=".823" style="stop-color:#f88909"/></linearGradient><path d="M30.1 0 0 31.7V60l30.1-30.1L60 0z" style="fill:url(#c)"/></svg>

Before

Width:  |  Height:  |  Size: 1.4 KiB

View File

Before

Width:  |  Height:  |  Size: 1.5 KiB

After

Width:  |  Height:  |  Size: 1.5 KiB

View File

@@ -0,0 +1 @@
<svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path fill="#9CA3AF" d="M12 22.596c6.628 0 12-4.338 12-9.688 0-3.318-2.057-6.248-5.219-7.986-1.286-.715-2.297-1.357-3.139-1.89C14.058 2.025 13.08 1.404 12 1.404c-1.097 0-2.334.785-3.966 1.821a49.92 49.92 0 0 1-2.816 1.697C2.057 6.66 0 9.59 0 12.908c0 5.35 5.372 9.687 12 9.687v.001ZM10.599 4.715c.334-.759.503-1.58.498-2.409 0-.145.202-.187.23-.029.658 2.783-.902 4.162-2.057 4.624-.124.048-.199-.121-.103-.209a5.763 5.763 0 0 0 1.432-1.977Zm2.058-.102a5.82 5.82 0 0 0-.782-2.306v-.016c-.069-.123.086-.263.185-.172 1.962 2.111 1.307 4.067.556 5.051-.082.103-.23-.003-.189-.126a5.85 5.85 0 0 0 .23-2.431Zm1.776-.561a5.727 5.727 0 0 0-1.612-1.806v-.014c-.112-.085-.024-.274.114-.218 2.595 1.087 2.774 3.18 2.459 4.407a.116.116 0 0 1-.049.071.11.11 0 0 1-.153-.026.122.122 0 0 1-.022-.083 5.891 5.891 0 0 0-.737-2.331Zm-5.087.561c-.617.546-1.282.76-2.063 1-.117 0-.195-.078-.156-.181 1.752-.909 2.376-1.649 2.999-2.778 0 0 .155-.118.188.085 0 .304-.349 1.329-.968 1.874Zm4.945 11.237a2.957 2.957 0 0 1-.937 1.553c-.346.346-.8.565-1.286.62a2.178 2.178 0 0 1-1.327-.62 2.955 2.955 0 0 1-.925-1.553.244.244 0 0 1 .064-.198.234.234 0 0 1 .193-.069h3.965a.226.226 0 0 1 .19.07c.05.053.073.125.063.197Zm-5.458-2.176a1.862 1.862 0 0 1-2.384-.245 1.98 1.98 0 0 1-.233-2.447c.207-.319.503-.566.848-.713a1.84 1.84 0 0 1 1.092-.11c.366.075.703.261.967.531a1.98 1.98 0 0 1 .408 2.114 1.931 1.931 0 0 1-.698.869v.001Zm8.495.005a1.86 1.86 0 0 1-2.381-.253 1.964 1.964 0 0 1-.547-1.366c0-.384.11-.76.32-1.079.207-.319.503-.567.849-.713a1.844 1.844 0 0 1 1.093-.108c.367.076.704.262.968.534a1.98 1.98 0 0 1 .4 2.117 1.932 1.932 0 0 1-.702.868Z"/></svg>

After

Width:  |  Height:  |  Size: 1.7 KiB

View File

@@ -0,0 +1 @@
<svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path fill="#F25022" d="M1 1h10.5v10.5H1z"/><path fill="#7FBA00" d="M12.5 1H23v10.5H12.5z"/><path fill="#00A4EF" d="M1 12.5h10.5V23H1z"/><path fill="#FFB900" d="M12.5 12.5H23V23H12.5z"/></svg>

After

Width:  |  Height:  |  Size: 264 B

View File

@@ -6,7 +6,6 @@ import {
useCallback,
useEffect,
useMemo,
useState,
} from 'react';
import { QueryKey } from 'react-query';
// eslint-disable-next-line no-restricted-imports
@@ -34,6 +33,8 @@ import { Warning } from 'types/api';
import { DataSource } from 'types/common/queryBuilder';
import { GlobalReducer } from 'types/reducer/globalTime';
import { getListOrderBy, setListOrderBy } from 'utils/explorerUtils';
import TraceExplorerControls from '../Controls';
import { getListViewQuery } from '../explorerUtils';
import {
@@ -58,12 +59,16 @@ function ListView({
setIsLoadingQueries,
queryKeyRef,
}: ListViewProps): JSX.Element {
const { stagedQuery, panelType: panelTypeFromQueryBuilder } =
useQueryBuilder();
const {
stagedQuery,
currentQuery,
redirectWithQueryBuilderData,
panelType: panelTypeFromQueryBuilder,
} = useQueryBuilder();
const panelType = panelTypeFromQueryBuilder || PANEL_TYPES.LIST;
const [orderBy, setOrderBy] = useState<string>('timestamp:desc');
const orderBy = useMemo(() => getListOrderBy(stagedQuery), [stagedQuery]);
const {
selectedTime: globalSelectedTime,
@@ -92,7 +97,6 @@ function ListView({
stagedQuery,
panelType,
paginationConfig,
orderBy,
],
[
stagedQuery,
@@ -101,7 +105,6 @@ function ListView({
paginationConfig,
maxTime,
minTime,
orderBy,
],
);
@@ -167,9 +170,12 @@ function ListView({
[queryTableData],
);
const handleOrderChange = useCallback((value: string) => {
setOrderBy(value);
}, []);
const handleOrderChange = useCallback(
(value: string) => {
redirectWithQueryBuilderData(setListOrderBy(currentQuery, value));
},
[currentQuery, redirectWithQueryBuilderData],
);
useEffect(() => {
if (!isLoading && !isFetching && !isError && rows.length !== 0) {

View File

@@ -6,7 +6,6 @@ import {
useCallback,
useEffect,
useMemo,
useState,
} from 'react';
import { QueryKey } from 'react-query';
// eslint-disable-next-line no-restricted-imports
@@ -23,6 +22,7 @@ import { TracesTableRow } from '../TracesTable/getFieldColumn';
import TracesTable from '../TracesTable/TracesTable';
import { useGetQueryRange } from 'hooks/queryBuilder/useGetQueryRange';
import { useQueryBuilder } from 'hooks/queryBuilder/useQueryBuilder';
import { getListOrderBy, setListOrderBy } from 'utils/explorerUtils';
import { Pagination } from 'hooks/queryPagination';
import useUrlQueryData from 'hooks/useUrlQueryData';
import { ArrowUp10, Minus } from '@signozhq/icons';
@@ -57,9 +57,13 @@ function TracesView({
setIsLoadingQueries,
queryKeyRef,
}: TracesViewProps): JSX.Element {
const { stagedQuery, panelType } = useQueryBuilder();
const { stagedQuery, currentQuery, redirectWithQueryBuilderData, panelType } =
useQueryBuilder();
const [orderBy, setOrderBy] = useState<string>(TRACE_VIEW_DEFAULT_ORDER_BY);
const orderBy = useMemo(
() => getListOrderBy(stagedQuery, TRACE_VIEW_DEFAULT_ORDER_BY),
[stagedQuery],
);
const {
columns,
@@ -94,7 +98,6 @@ function TracesView({
stagedQuery,
panelType,
paginationQueryData,
orderBy,
],
[
globalSelectedTime,
@@ -103,7 +106,6 @@ function TracesView({
stagedQuery,
panelType,
paginationQueryData,
orderBy,
],
);
@@ -165,9 +167,12 @@ function TracesView({
}
}, [isLoading, isFetching, isError, rows.length]);
const handleOrderChange = useCallback((value: string): void => {
setOrderBy(value);
}, []);
const handleOrderChange = useCallback(
(value: string): void => {
redirectWithQueryBuilderData(setListOrderBy(currentQuery, value));
},
[currentQuery, redirectWithQueryBuilderData],
);
// Without the full column set there is no pool to pick from, so the control is dropped.
const fieldsSelectorConfig = useMemo(

View File

@@ -1,6 +1,7 @@
import { initialQueriesMap } from 'constants/queryBuilder';
import { cloneDeep } from 'lodash-es';
import { OrderByPayload, Query } from 'types/api/queryBuilder/queryBuilderData';
import { parseListOrderBy } from 'utils/explorerUtils';
export const getListViewQuery = (
stagedQuery: Query,
@@ -11,12 +12,7 @@ export const getListViewQuery = (
: cloneDeep(initialQueriesMap.traces);
const orderByPayload: OrderByPayload[] = orderBy
? [
{
columnName: orderBy.split(':')[0],
order: orderBy.split(':')[1] as 'asc' | 'desc',
},
]
? [parseListOrderBy(orderBy)]
: [];
for (let i = 0; i < query.builder.queryData.length; i++) {

View File

@@ -16,14 +16,14 @@ function LogsActionsContainer({
showFrequencyChart,
handleToggleFrequencyChart,
orderBy,
setOrderBy,
onOrderChange,
explorerActions,
}: {
listQuery: any;
showFrequencyChart: boolean;
handleToggleFrequencyChart: () => void;
orderBy: string;
setOrderBy: (value: string) => void;
onOrderChange: (value: string) => void;
explorerActions: ReactNode;
}): JSX.Element {
const { options, config } = useOptionsMenu({
@@ -78,7 +78,7 @@ function LogsActionsContainer({
<ListViewOrderBy
value={orderBy}
onChange={(value): void => setOrderBy(value)}
onChange={onOrderChange}
dataSource={DataSource.LOGS}
/>
</div>

View File

@@ -59,6 +59,7 @@ import { QueryDataV3 } from 'types/api/widgets/getQuery';
import { DataSource } from 'types/common/queryBuilder';
import { GlobalReducer } from 'types/reducer/globalTime';
import { v4 } from 'uuid';
import { getListOrderBy, setListOrderBy } from 'utils/explorerUtils';
import LogsActionsContainer from './LogsActionsContainer';
@@ -101,7 +102,8 @@ function LogsExplorerViewsContainer({
const currentMinTimeRef = useRef<number>(minTime);
// Context
const { stagedQuery, panelType } = useQueryBuilder();
const { stagedQuery, currentQuery, redirectWithQueryBuilderData, panelType } =
useQueryBuilder();
const selectedPanelType = panelType || PANEL_TYPES.LIST;
@@ -111,7 +113,7 @@ function LogsExplorerViewsContainer({
const [requestData, setRequestData] = useState<Query | null>(null);
const [listChartQuery, setListChartQuery] = useState<Query | null>(null);
const [orderBy, setOrderBy] = useState<string>('timestamp:desc');
const orderBy = useMemo(() => getListOrderBy(stagedQuery), [stagedQuery]);
const { yAxisUnit, onUnitChange } = useUrlYAxisUnit('');
@@ -192,6 +194,13 @@ function LogsExplorerViewsContainer({
'custom',
);
const handleOrderChange = useCallback(
(value: string): void => {
redirectWithQueryBuilderData(setListOrderBy(currentQuery, value));
},
[currentQuery, redirectWithQueryBuilderData],
);
const getRequestData = useCallback(
(
query: Query | null,
@@ -323,18 +332,10 @@ function LogsExplorerViewsContainer({
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [data]);
// Store previous orderDirection to detect changes
const prevOrderByRef = useRef(orderBy);
useEffect(() => {
const orderByChanged =
prevOrderByRef.current !== orderBy && selectedPanelType === PANEL_TYPES.LIST;
prevOrderByRef.current = orderBy;
if (
requestData?.id !== stagedQuery?.id ||
currentMinTimeRef.current !== minTime ||
orderByChanged
currentMinTimeRef.current !== minTime
) {
const newRequestData = getRequestData(stagedQuery, {
filters: listQuery?.filters || initialFilters,
@@ -358,7 +359,6 @@ function LogsExplorerViewsContainer({
activeLogId,
selectedPanelType,
maxTime,
orderBy,
]);
const chartData = useMemo(() => {
@@ -430,7 +430,7 @@ function LogsExplorerViewsContainer({
showFrequencyChart={showFrequencyChart}
handleToggleFrequencyChart={handleToggleFrequencyChart}
orderBy={orderBy}
setOrderBy={setOrderBy}
onOrderChange={handleOrderChange}
explorerActions={explorerActions}
/>
)}

View File

@@ -1,8 +1,7 @@
import agnoUrl from '@/assets/Logos/agno.svg';
import amazonBedrockUrl from '@/assets/Logos/amazon-bedrock.svg';
import amazonMskUrl from '@/assets/Logos/amazon-msk.svg';
import androidJavaMonitoringUrl from '@/assets/Logos/android-java-monitoring.svg';
import androidKotlinMonitoringUrl from '@/assets/Logos/android-kotlin-monitoring.svg';
import androidUrl from '@/assets/Logos/android.svg';
import anthropicApiMonitoringUrl from '@/assets/Logos/anthropic-api-monitoring.svg';
import antigravityUrl from '@/assets/Logos/antigravity.svg';
import apacheDruidUrl from '@/assets/Logos/apache-druid.svg';
@@ -25,6 +24,7 @@ import azureSqlDatabaseMetricsUrl from '@/assets/Logos/azure-sql-database-metric
import azureVmUrl from '@/assets/Logos/azure-vm.svg';
import azureUrl from '@/assets/Logos/azure.svg';
import basetenUrl from '@/assets/Logos/baseten.svg';
import bunUrl from '@/assets/Logos/bun.svg';
import cassandraUrl from '@/assets/Logos/cassandra.svg';
import celeryUrl from '@/assets/Logos/celery.svg';
import certManagerUrl from '@/assets/Logos/cert-manager.svg';
@@ -126,6 +126,7 @@ import logstashUrl from '@/assets/Logos/logstash.svg';
import mastraUrl from '@/assets/Logos/mastra.svg';
import metaUrl from '@/assets/Logos/meta.svg';
import metricsUrl from '@/assets/Logos/metrics.svg';
import microsoftUrl from '@/assets/Logos/microsoft.svg';
import microsoftSqlServerUrl from '@/assets/Logos/microsoft-sql-server.svg';
import mistralUrl from '@/assets/Logos/mistral.svg';
import modalUrl from '@/assets/Logos/modal.svg';
@@ -4712,70 +4713,40 @@ const onboardingConfigWithLinks = [
link: '/docs/messaging-queues/celery-setup/',
},
{
dataSource: 'android-java',
label: 'Android Java',
imgUrl: androidJavaMonitoringUrl,
dataSource: 'android',
label: 'Android',
imgUrl: androidUrl,
tags: ['mobile app monitoring'],
module: 'apm',
relatedSearchKeywords: [
'android',
'android app tracing',
'android java',
'android java instrumentation',
'android java monitoring',
'android java to signoz',
'android kotlin',
'android logs',
'android metrics',
'android observability',
'android performance monitoring',
'android-java',
'android to signoz',
'apm',
'app',
'application performance monitoring',
'java',
'kotlin',
'logs',
'metrics',
'mobile',
'mobile app monitoring',
'monitor android apps',
'monitoring',
'observability',
'opentelemetry android java',
'otel android sdk',
'opentelemetry android',
'otel android agent',
'traces',
'tracing',
],
link: '/docs/instrumentation/mobile-instrumentation/opentelemetry-java/',
},
{
dataSource: 'android-kotlin',
label: 'Android Kotlin',
imgUrl: androidKotlinMonitoringUrl,
tags: ['mobile app monitoring'],
module: 'apm',
relatedSearchKeywords: [
'android',
'android app performance kotlin',
'android kotlin',
'android kotlin monitoring',
'android kotlin to signoz',
'android observability kotlin',
'android telemetry sdk',
'android-kotlin',
'apm',
'app',
'application performance monitoring',
'kotlin',
'kotlin app logs',
'kotlin mobile tracing',
'mobile',
'mobile app monitoring',
'monitor kotlin apps',
'monitoring',
'observability',
'opentelemetry android kotlin',
'otel kotlin sdk',
'traces',
'tracing',
],
link: '/docs/instrumentation/mobile-instrumentation/opentelemetry-kotlin/',
id: 'android',
link: '/docs/instrumentation/mobile-instrumentation/opentelemetry-android/',
},
{
dataSource: 'flutter',
@@ -7987,5 +7958,112 @@ const onboardingConfigWithLinks = [
id: 'aws-lambda-microvms',
link: '/docs/aws-monitoring/lambda-microvms/',
},
{
dataSource: 'bun-elysia',
label: 'Bun / Elysia',
imgUrl: bunUrl,
tags: ['apm/traces', 'metrics'],
module: 'apm',
relatedSearchKeywords: [
'apm',
'application performance monitoring',
'bun',
'bun elysia',
'bun metrics',
'bun monitoring',
'bun observability',
'bun traces',
'bunjs',
'elysia',
'elysiajs',
'javascript',
'metrics',
'observability',
'opentelemetry bun',
'traces',
'tracing',
'typescript',
],
id: 'bun-elysia',
link: '/docs/instrumentation/opentelemetry-bun/',
},
{
dataSource: 'snmp',
label: 'SNMP',
imgUrl: metricsUrl,
tags: ['metrics'],
module: 'metrics',
relatedSearchKeywords: [
'firewall',
'metrics',
'monitoring',
'network',
'network devices',
'network monitoring',
'observability',
'opentelemetry snmp',
'router',
'snmp',
'snmp metrics',
'snmp receiver',
'switch',
],
id: 'snmp',
link: '/docs/metrics-management/opentelemetry-snmp-metrics/',
},
{
dataSource: 'vercel-eve',
label: 'Vercel Eve',
imgUrl: vercelUrl,
tags: ['LLM Monitoring'],
module: 'apm',
relatedSearchKeywords: [
'agent',
'agents',
'ai agent',
'eve',
'eve dev',
'eve observability',
'llm',
'llm monitoring',
'monitoring',
'observability',
'token usage',
'traces',
'tracing',
'vercel',
'vercel agents',
'vercel eve',
],
id: 'vercel-eve',
link: '/docs/eve-observability/',
},
{
dataSource: 'microsoft-agent-framework',
label: 'Microsoft Agent Framework',
imgUrl: microsoftUrl,
tags: ['LLM Monitoring'],
module: 'apm',
relatedSearchKeywords: [
'agent',
'agent framework',
'agents',
'ai agent',
'llm',
'llm monitoring',
'maf',
'microsoft',
'microsoft agent framework',
'microsoft agent framework observability',
'monitoring',
'observability',
'token usage',
'traces',
'tracing',
'workflows',
],
id: 'microsoft-agent-framework',
link: '/docs/microsoft-agent-framework-observability/',
},
];
export default onboardingConfigWithLinks;

View File

@@ -7,7 +7,6 @@ import {
useCallback,
useEffect,
useMemo,
useState,
} from 'react';
// eslint-disable-next-line no-restricted-imports
import { useSelector } from 'react-redux';
@@ -34,6 +33,7 @@ import { useQueryBuilder } from 'hooks/queryBuilder/useQueryBuilder';
import { Pagination } from 'hooks/queryPagination';
import { getDefaultPaginationConfig } from 'hooks/queryPagination/utils';
import useUrlQueryData from 'hooks/useUrlQueryData';
import { getListOrderBy, setListOrderBy } from 'utils/explorerUtils';
import { ArrowUp10, Minus } from '@signozhq/icons';
import { AppState } from 'store/reducers';
import { Warning } from 'types/api';
@@ -66,12 +66,16 @@ function ListView({
queryKeyRef,
headerActions,
}: ListViewProps): JSX.Element {
const { stagedQuery, panelType: panelTypeFromQueryBuilder } =
useQueryBuilder();
const {
stagedQuery,
currentQuery,
redirectWithQueryBuilderData,
panelType: panelTypeFromQueryBuilder,
} = useQueryBuilder();
const panelType = panelTypeFromQueryBuilder || PANEL_TYPES.LIST;
const [orderBy, setOrderBy] = useState<string>('timestamp:desc');
const orderBy = useMemo(() => getListOrderBy(stagedQuery), [stagedQuery]);
const {
selectedTime: globalSelectedTime,
@@ -123,7 +127,6 @@ function ListView({
panelType,
paginationConfig,
selectColumnsSignature,
orderBy,
],
[
stagedQuery,
@@ -133,7 +136,6 @@ function ListView({
selectColumnsSignature,
maxTime,
minTime,
orderBy,
],
);
@@ -215,9 +217,12 @@ function ListView({
[config],
);
const handleOrderChange = useCallback((value: string) => {
setOrderBy(value);
}, []);
const handleOrderChange = useCallback(
(value: string) => {
redirectWithQueryBuilderData(setListOrderBy(currentQuery, value));
},
[currentQuery, redirectWithQueryBuilderData],
);
useEffect(() => {
if (!isLoading && !isFetching && !isError && rows.length !== 0) {

View File

@@ -2,6 +2,7 @@ import { initialQueriesMap, PANEL_TYPES } from 'constants/queryBuilder';
import { OptionsQuery } from 'container/OptionsMenu/types';
import { cloneDeep, set } from 'lodash-es';
import { OrderByPayload, Query } from 'types/api/queryBuilder/queryBuilderData';
import { parseListOrderBy } from 'utils/explorerUtils';
export const getListViewQuery = (
stagedQuery: Query,
@@ -12,12 +13,7 @@ export const getListViewQuery = (
: cloneDeep(initialQueriesMap.traces);
const orderByPayload: OrderByPayload[] = orderBy
? [
{
columnName: orderBy.split(':')[0],
order: orderBy.split(':')[1] as 'asc' | 'desc',
},
]
? [parseListOrderBy(orderBy)]
: [];
for (let i = 0; i < query.builder.queryData.length; i++) {

View File

@@ -0,0 +1,108 @@
import { renderHook } from '@testing-library/react';
import { initialQueriesMap, PANEL_TYPES } from 'constants/queryBuilder';
import { useQueryBuilder } from 'hooks/queryBuilder/useQueryBuilder';
import { useHandleExplorerTabChange } from 'hooks/useHandleExplorerTabChange';
import { IBuilderQuery, Query } from 'types/api/queryBuilder/queryBuilderData';
jest.mock('hooks/queryBuilder/useQueryBuilder');
jest.mock('hooks/saveViews/useGetSavedViewParams', () => ({
useGetSavedViewParams: (): { viewName: string; viewKey: string } => ({
viewName: '',
viewKey: '',
}),
}));
const mockedUseQueryBuilder = jest.mocked(useQueryBuilder);
const countOrderBy = [{ columnName: 'count()', order: 'desc' }];
const queryWithOrder = (orderBy: typeof countOrderBy): Query => ({
...initialQueriesMap.traces,
builder: {
...initialQueriesMap.traces.builder,
queryData: initialQueriesMap.traces.builder.queryData.map((item) => ({
...item,
orderBy,
})),
},
});
const setUpQueryBuilder = (
from: PANEL_TYPES,
currentQuery: Query,
): jest.Mock => {
const redirectWithQueryBuilderData = jest.fn();
mockedUseQueryBuilder.mockReturnValue({
currentQuery,
panelType: from,
redirectWithQueryBuilderData,
updateAllQueriesOperators: (query: Query): Query => query,
updateQueriesData: (
query: Query,
type: 'queryData',
updateCallback: (item: IBuilderQuery, index: number) => IBuilderQuery,
): Query => ({
...query,
builder: {
...query.builder,
[type]: query.builder[type].map(updateCallback),
},
}),
} as unknown as ReturnType<typeof useQueryBuilder>);
return redirectWithQueryBuilderData;
};
const switchTab = (from: PANEL_TYPES, to: PANEL_TYPES, query: Query): Query => {
const redirect = setUpQueryBuilder(from, query);
const { result } = renderHook(() => useHandleExplorerTabChange());
result.current.handleExplorerTabChange(to);
return redirect.mock.calls[0][0] as Query;
};
describe('explorer tab change clears the order for raw panels', () => {
it.each([
[PANEL_TYPES.TIME_SERIES, PANEL_TYPES.LIST],
[PANEL_TYPES.TABLE, PANEL_TYPES.LIST],
[PANEL_TYPES.TRACE, PANEL_TYPES.LIST],
[PANEL_TYPES.LIST, PANEL_TYPES.TRACE],
])('clears it on %s -> %s', (from, to) => {
const query = switchTab(from, to, queryWithOrder(countOrderBy));
query.builder.queryData.forEach((item) => {
expect(item.orderBy).toStrictEqual([]);
});
});
it.each([
[PANEL_TYPES.LIST, PANEL_TYPES.TIME_SERIES],
[PANEL_TYPES.LIST, PANEL_TYPES.TABLE],
])('keeps it on %s -> %s, where the sanitizer validates it', (from, to) => {
const query = switchTab(from, to, queryWithOrder(countOrderBy));
query.builder.queryData.forEach((item) => {
expect(item.orderBy).toStrictEqual(countOrderBy);
});
});
it('leaves the order alone when the caller supplies its own query', () => {
const saved = queryWithOrder([{ columnName: 'timestamp', order: 'asc' }]);
const redirect = setUpQueryBuilder(
PANEL_TYPES.TIME_SERIES,
queryWithOrder(countOrderBy),
);
const { result } = renderHook(() => useHandleExplorerTabChange());
result.current.handleExplorerTabChange(PANEL_TYPES.LIST, { query: saved });
const sent = redirect.mock.calls[0][0] as Query;
sent.builder.queryData.forEach((item) => {
expect(item.orderBy).toStrictEqual([
{ columnName: 'timestamp', order: 'asc' },
]);
});
});
});

View File

@@ -2,7 +2,6 @@ import { useCallback } from 'react';
import { QueryParams } from 'constants/query';
import { initialAutocompleteData, PANEL_TYPES } from 'constants/queryBuilder';
import ROUTES from 'constants/routes';
import { SIGNOZ_VALUE } from 'container/QueryBuilder/filters/OrderByFilter/constants';
import { Query } from 'types/api/queryBuilder/queryBuilderData';
import { DataSource } from 'types/common/queryBuilder';
@@ -47,7 +46,7 @@ export const useHandleExplorerTabChange = (): {
) {
query = updateQueriesData(query, 'queryData', (item) => ({
...item,
orderBy: item.orderBy.filter((item) => item.columnName !== SIGNOZ_VALUE),
orderBy: [],
aggregateAttribute: initialAutocompleteData,
}));
}

View File

@@ -66,6 +66,7 @@ import {
import { sanitizeOrderByForExplorer } from 'utils/sanitizeOrderBy';
import { v4 as uuid } from 'uuid';
import { getUnstableCurrentSearchParams } from 'utils/getUnstableCurrentSearchParams';
import { readPanelTypeFromUrl } from 'utils/readPanelTypeFromUrl';
export const QueryBuilderContext = createContext<QueryBuilderContextType>({
currentQuery: initialQueriesMap.metrics,
@@ -182,6 +183,11 @@ export function QueryBuilderProvider({
})) ?? [],
};
const urlPanelType = readPanelTypeFromUrl(PANEL_TYPES.LIST);
const isRawPanel =
urlPanelType === PANEL_TYPES.LIST || urlPanelType === PANEL_TYPES.TRACE;
const setupedQueryData = builder.queryData.map((item) => {
const currentElement: IBuilderQuery = {
...item,
@@ -200,8 +206,10 @@ export function QueryBuilderProvider({
// Explorer pages: sanitize stale orderBy before first query
const isExplorer =
location.pathname === ROUTES.LOGS_EXPLORER ||
location.pathname === ROUTES.TRACES_EXPLORER;
(location.pathname === ROUTES.LOGS_EXPLORER ||
location.pathname === ROUTES.TRACES_EXPLORER ||
location.pathname === ROUTES.AI_OBSERVABILITY_EXPLORER) &&
!isRawPanel;
if (isExplorer) {
const sanitizedOrderBy = sanitizeOrderByForExplorer(currentElement);
return calledFromHandleRunQuery
@@ -262,6 +270,8 @@ export function QueryBuilderProvider({
setStagedQuery(nextQuery);
setCurrentQuery(newQueryState);
setQueryType(type);
setCalledFromHandleRunQuery(false);
},
[prepareQueryBuilderData],
);
@@ -1038,7 +1048,8 @@ export function QueryBuilderProvider({
const handleRunQuery = useCallback(() => {
const isExplorer =
location.pathname === ROUTES.LOGS_EXPLORER ||
location.pathname === ROUTES.TRACES_EXPLORER;
location.pathname === ROUTES.TRACES_EXPLORER ||
location.pathname === ROUTES.AI_OBSERVABILITY_EXPLORER;
if (isExplorer) {
setCalledFromHandleRunQuery(true);
}

View File

@@ -0,0 +1,176 @@
import { initialQueriesMap, PANEL_TYPES } from 'constants/queryBuilder';
import ROUTES from 'constants/routes';
import { useQueryBuilder } from 'hooks/queryBuilder/useQueryBuilder';
import {
act,
AllTheProviders,
renderHook,
RenderHookResult,
} from 'tests/test-utils';
import { Query } from 'types/api/queryBuilder/queryBuilderData';
import { QueryBuilderContextType } from 'types/common/queryBuilder';
import {
__resetSearchParamsGetter,
__setSearchParamsGetterForTest,
} from 'utils/getUnstableCurrentSearchParams';
const timestampOrderBy = [{ columnName: 'timestamp', order: 'desc' }];
const queryWithTimestampOrder = (base: Query): Query => ({
...base,
builder: {
...base.builder,
queryData: base.builder.queryData.map((item) => ({
...item,
orderBy: timestampOrderBy,
})),
},
});
const renderOn = (
route: string,
panelType: PANEL_TYPES | null,
): RenderHookResult<QueryBuilderContextType, unknown> => {
const search = new URLSearchParams();
if (panelType) {
search.set('panelTypes', JSON.stringify(panelType));
}
__setSearchParamsGetterForTest(() => search);
return renderHook(() => useQueryBuilder(), {
wrapper: ({ children }): JSX.Element => (
<AllTheProviders initialRoute={route}>{children}</AllTheProviders>
),
});
};
const initOn = (
route: string,
panelType: PANEL_TYPES | null,
query: Query,
): Query | null => {
const { result } = renderOn(route, panelType);
act(() => {
result.current.initQueryBuilderData(query);
});
return result.current.stagedQuery;
};
describe('explorer orderBy sanitizing by panel type', () => {
afterEach(() => {
__resetSearchParamsGetter();
});
it('keeps the order on the logs list panel', () => {
const staged = initOn(
ROUTES.LOGS_EXPLORER,
PANEL_TYPES.LIST,
queryWithTimestampOrder(initialQueriesMap.logs),
);
expect(staged?.builder.queryData[0].orderBy).toStrictEqual(timestampOrderBy);
});
it('keeps the order when panelTypes is absent, since the explorers default to list', () => {
const staged = initOn(
ROUTES.LOGS_EXPLORER,
null,
queryWithTimestampOrder(initialQueriesMap.logs),
);
expect(staged?.builder.queryData[0].orderBy).toStrictEqual(timestampOrderBy);
});
it('keeps the order on the traces trace panel', () => {
const staged = initOn(
ROUTES.TRACES_EXPLORER,
PANEL_TYPES.TRACE,
queryWithTimestampOrder(initialQueriesMap.traces),
);
expect(staged?.builder.queryData[0].orderBy).toStrictEqual(timestampOrderBy);
});
// engineering-pod#2813: a timestamp order carried from the list view into the
// time series tab is not a group-by key or an aggregation, and errors the query.
it('drops the order on the time series panel', () => {
const staged = initOn(
ROUTES.LOGS_EXPLORER,
PANEL_TYPES.TIME_SERIES,
queryWithTimestampOrder(initialQueriesMap.logs),
);
expect(staged?.builder.queryData[0].orderBy).toStrictEqual([]);
});
it('keeps the order on the LLM observability list panel', () => {
const staged = initOn(
ROUTES.AI_OBSERVABILITY_EXPLORER,
PANEL_TYPES.LIST,
queryWithTimestampOrder(initialQueriesMap.traces),
);
expect(staged?.builder.queryData[0].orderBy).toStrictEqual(timestampOrderBy);
});
it('drops the order on the LLM observability time series panel', () => {
const staged = initOn(
ROUTES.AI_OBSERVABILITY_EXPLORER,
PANEL_TYPES.TIME_SERIES,
queryWithTimestampOrder(initialQueriesMap.traces),
);
expect(staged?.builder.queryData[0].orderBy).toStrictEqual([]);
});
it('drops the order on the table panel', () => {
const staged = initOn(
ROUTES.TRACES_EXPLORER,
PANEL_TYPES.TABLE,
queryWithTimestampOrder(initialQueriesMap.traces),
);
expect(staged?.builder.queryData[0].orderBy).toStrictEqual([]);
});
});
// handleRunQuery means "respect the query I just built" for the init it causes.
// It used to stay set for the rest of the visit, so every later init skipped
// sanitizing as well.
describe('the run bypass is one-shot', () => {
afterEach(() => {
__resetSearchParamsGetter();
});
it('skips sanitizing for the init after a run, and sanitizes the next one', () => {
const { result } = renderOn(ROUTES.LOGS_EXPLORER, PANEL_TYPES.TIME_SERIES);
act(() => {
result.current.handleRunQuery();
});
const afterRun = {
...queryWithTimestampOrder(initialQueriesMap.logs),
id: 'run-1',
};
act(() => {
result.current.initQueryBuilderData(afterRun);
});
expect(
result.current.stagedQuery?.builder.queryData[0].orderBy,
).toStrictEqual(timestampOrderBy);
const next = {
...queryWithTimestampOrder(initialQueriesMap.logs),
id: 'run-2',
};
act(() => {
result.current.initQueryBuilderData(next);
});
expect(
result.current.stagedQuery?.builder.queryData[0].orderBy,
).toStrictEqual([]);
});
});

View File

@@ -0,0 +1,137 @@
import { initialQueriesMap } from 'constants/queryBuilder';
import { Query } from 'types/api/queryBuilder/queryBuilderData';
import {
DEFAULT_LIST_ORDER_BY,
getListOrderBy,
getRawPanelOrderBy,
parseListOrderBy,
setListOrderBy,
} from 'utils/explorerUtils';
const withOrderBy = (
orderBy: { columnName: string; order: string }[],
): Query => ({
...initialQueriesMap.logs,
builder: {
...initialQueriesMap.logs.builder,
queryData: initialQueriesMap.logs.builder.queryData.map((item) => ({
...item,
orderBy,
})),
},
});
describe('getRawPanelOrderBy', () => {
it('keeps row fields', () => {
expect(
getRawPanelOrderBy(
withOrderBy([{ columnName: 'duration_nano', order: 'desc' }]),
),
).toStrictEqual([{ columnName: 'duration_nano', order: 'desc' }]);
});
it('keeps a group-by key, which is a row field too', () => {
expect(
getRawPanelOrderBy(
withOrderBy([{ columnName: 'service.name', order: 'asc' }]),
),
).toStrictEqual([{ columnName: 'service.name', order: 'asc' }]);
});
it('drops an aggregation carried in from an aggregated panel', () => {
expect(
getRawPanelOrderBy(withOrderBy([{ columnName: 'count()', order: 'desc' }])),
).toStrictEqual([]);
});
it('drops the aggregated value sentinel', () => {
expect(
getRawPanelOrderBy(
withOrderBy([{ columnName: '#SIGNOZ_VALUE', order: 'desc' }]),
),
).toStrictEqual([]);
});
it('keeps the usable entry and drops the rest', () => {
expect(
getRawPanelOrderBy(
withOrderBy([
{ columnName: 'count()', order: 'desc' },
{ columnName: 'timestamp', order: 'desc' },
]),
),
).toStrictEqual([{ columnName: 'timestamp', order: 'desc' }]);
});
});
describe('getListOrderBy', () => {
it('falls back to the default when the query holds only an aggregation', () => {
expect(
getListOrderBy(withOrderBy([{ columnName: 'count()', order: 'desc' }])),
).toBe(DEFAULT_LIST_ORDER_BY);
});
it('reads the first order entry as a select value', () => {
expect(
getListOrderBy(withOrderBy([{ columnName: 'timestamp', order: 'asc' }])),
).toBe('timestamp:asc');
});
it('lower-cases a hand-written direction from a deep link', () => {
expect(
getListOrderBy(withOrderBy([{ columnName: 'timestamp', order: 'ASC' }])),
).toBe('timestamp:asc');
});
it('shows the primary entry when the query also carries the id tiebreaker', () => {
const query = withOrderBy([
{ columnName: 'timestamp', order: 'desc' },
{ columnName: 'id', order: 'desc' },
]);
expect(getListOrderBy(query)).toBe('timestamp:desc');
});
it('falls back to the default when the query carries no order', () => {
expect(getListOrderBy(withOrderBy([]))).toBe(DEFAULT_LIST_ORDER_BY);
expect(getListOrderBy(null)).toBe(DEFAULT_LIST_ORDER_BY);
});
it('falls back to the given default', () => {
expect(getListOrderBy(withOrderBy([]), 'last_activity_time:desc')).toBe(
'last_activity_time:desc',
);
});
});
describe('parseListOrderBy', () => {
it('splits a select value into an order payload', () => {
expect(parseListOrderBy('durationNano:asc')).toStrictEqual({
columnName: 'durationNano',
order: 'asc',
});
});
});
describe('setListOrderBy', () => {
it('replaces the order on every queryData entry', () => {
const base = withOrderBy([{ columnName: 'timestamp', order: 'desc' }]);
const next = setListOrderBy(base, 'severity_text:asc');
next.builder.queryData.forEach((item) => {
expect(item.orderBy).toStrictEqual([
{ columnName: 'severity_text', order: 'asc' },
]);
});
});
it('does not mutate the query it is given', () => {
const base = withOrderBy([{ columnName: 'timestamp', order: 'desc' }]);
setListOrderBy(base, 'timestamp:asc');
expect(base.builder.queryData[0].orderBy).toStrictEqual([
{ columnName: 'timestamp', order: 'desc' },
]);
});
});

View File

@@ -0,0 +1,45 @@
import { PANEL_TYPES } from 'constants/queryBuilder';
import {
__resetSearchParamsGetter,
__setSearchParamsGetterForTest,
} from 'utils/getUnstableCurrentSearchParams';
import { readPanelTypeFromUrl } from 'utils/readPanelTypeFromUrl';
const withParam = (value?: string): void => {
const search = new URLSearchParams();
if (value !== undefined) {
search.set('panelTypes', value);
}
__setSearchParamsGetterForTest(() => search);
};
describe('readPanelTypeFromUrl', () => {
afterEach(() => {
__resetSearchParamsGetter();
});
it('decodes the JSON-encoded value the app writes', () => {
withParam(JSON.stringify(PANEL_TYPES.TIME_SERIES));
expect(readPanelTypeFromUrl()).toBe(PANEL_TYPES.TIME_SERIES);
});
it('accepts an unencoded value from a hand-written link', () => {
withParam('list');
expect(readPanelTypeFromUrl()).toBe(PANEL_TYPES.LIST);
});
it('returns the raw value instead of throwing on a malformed one', () => {
withParam('{list');
expect(readPanelTypeFromUrl()).toBe('{list');
});
it('returns the fallback when the param is absent', () => {
withParam();
expect(readPanelTypeFromUrl(PANEL_TYPES.LIST)).toBe(PANEL_TYPES.LIST);
expect(readPanelTypeFromUrl()).toBeNull();
});
});

View File

@@ -1,8 +1,69 @@
import { QueryParams } from 'constants/query';
import { PANEL_TYPES } from 'constants/queryBuilder';
import ROUTES from 'constants/routes';
import { SIGNOZ_VALUE } from 'container/QueryBuilder/filters/OrderByFilter/constants';
import { ExplorerViews } from 'pages/LogsExplorer/utils';
import { Query } from 'types/api/queryBuilder/queryBuilderData';
import { OrderByPayload, Query } from 'types/api/queryBuilder/queryBuilderData';
import { getParsedAggregationOptionsForOrderBy } from './aggregationConverter';
export const DEFAULT_LIST_ORDER_BY = 'timestamp:desc';
export const getRawPanelOrderBy = (query: Query | null): OrderByPayload[] => {
const queryData = query?.builder?.queryData?.[0];
if (!queryData) {
return [];
}
const disallowed = new Set<string>([SIGNOZ_VALUE]);
getParsedAggregationOptionsForOrderBy(queryData).forEach((aggregation) => {
if (aggregation?.key) {
disallowed.add(aggregation.key);
}
});
return (queryData.orderBy ?? []).filter(
(item) => !disallowed.has(item.columnName),
);
};
export const getListOrderBy = (
query: Query | null,
fallback: string = DEFAULT_LIST_ORDER_BY,
): string => {
const [orderBy] = getRawPanelOrderBy(query);
if (!orderBy?.columnName || !orderBy?.order) {
return fallback;
}
return `${orderBy.columnName}:${orderBy.order.toLowerCase()}`;
};
export const parseListOrderBy = (orderBy: string): OrderByPayload => {
const [columnName, order] = orderBy.split(':');
return {
columnName: columnName || 'timestamp',
order: order || 'desc',
};
};
export const setListOrderBy = (query: Query, orderBy: string): Query => {
const nextOrderBy = [parseListOrderBy(orderBy)];
return {
...query,
builder: {
...query.builder,
queryData: query.builder.queryData.map((item) => ({
...item,
orderBy: nextOrderBy,
})),
},
};
};
// Mapping between panel types and explorer views
export const panelTypeToExplorerView: Record<PANEL_TYPES, ExplorerViews> = {

View File

@@ -0,0 +1,21 @@
import { QueryParams } from 'constants/query';
import { PANEL_TYPES } from 'constants/queryBuilder';
import { getUnstableCurrentSearchParams } from './getUnstableCurrentSearchParams';
// The param is JSON-encoded, but a hand-written link can carry it unencoded.
export const readPanelTypeFromUrl = (
fallback: PANEL_TYPES | null = null,
): PANEL_TYPES | null => {
const raw = getUnstableCurrentSearchParams().get(QueryParams.panelTypes);
if (!raw) {
return fallback;
}
try {
return JSON.parse(raw) as PANEL_TYPES;
} catch {
return raw as PANEL_TYPES;
}
};

View File

@@ -9,7 +9,6 @@ var (
FeaturePutMetersInZeus = featuretypes.MustNewName("put_meters_in_zeus")
FeatureUseMeterReporter = featuretypes.MustNewName("use_meter_reporter")
FeatureUseJSONBody = featuretypes.MustNewName("use_json_body")
FeatureJSONBodyDualIngestion = featuretypes.MustNewName("json_body_dual_ingestion")
FeatureEnableMetricsReduction = featuretypes.MustNewName("enable_metrics_reduction")
FeatureResolveSemconvFamilies = featuretypes.MustNewName("resolve_semconv_families")
FeatureUseTraceAttributesJSON = featuretypes.MustNewName("use_trace_attributes_json")
@@ -65,14 +64,6 @@ func MustNewRegistry() featuretypes.Registry {
DefaultVariant: featuretypes.MustNewName("disabled"),
Variants: featuretypes.NewBooleanVariants(),
},
&featuretypes.Feature{
Name: FeatureJSONBodyDualIngestion,
Kind: featuretypes.KindBoolean,
Stage: featuretypes.StageExperimental,
Description: "Controls whether the collector's normalize operator keeps the original log body so it is ingested into both the legacy body and the JSON body columns",
DefaultVariant: featuretypes.MustNewName("disabled"),
Variants: featuretypes.NewBooleanVariants(),
},
&featuretypes.Feature{
Name: FeatureEnableMetricsReduction,
Kind: featuretypes.KindBoolean,

View File

@@ -2,6 +2,7 @@ package logparsingpipeline
import (
"encoding/json"
"slices"
"strings"
"sync"
@@ -24,10 +25,13 @@ var (
)
const (
memoryLimiterProcessor = "memory_limiter"
memoryLimiterProcessorPrefix = "memory_limiter/"
batchProcessor = "batch"
batchProcessorPrefix = "batch/"
memoryLimiterProcessor = "memory_limiter"
memoryLimiterProcessorPrefix = "memory_limiter/"
batchProcessor = "batch"
batchProcessorPrefix = "batch/"
normalizerProcessor = "signozlogsnormalizer"
normalizerProcessorPrefix = "signozlogsnormalizer/"
userPipelinesBeforeNormalizer = "before"
)
// check if the processors already exist
@@ -90,9 +94,14 @@ func getOtelPipelineFromConfig(config map[string]interface{}) (*otelPipeline, er
// 2. signoz user-pipeline processors (in the order given by signozPipelineProcessorNames)
// 3. custom processors (non-signoz, non-memory_limiter, non-batch processors from the current config)
// 4. batch processors (any processor named "batch" or "batch/<id>") and anything after them
//
// If a signozlogsnormalizer processor is present, the order above is skipped: user-pipeline
// processors go right after the first one, or right before it when its config sets
// `opamp.position: before`, and the other processors keep their places.
func buildCollectorPipelineProcessorsList(
currentCollectorProcessors []string,
signozPipelineProcessorNames []string,
processorConfigs map[string]interface{},
) ([]string, error) {
lockLogsPipelineSpec.Lock()
defer lockLogsPipelineSpec.Unlock()
@@ -103,6 +112,23 @@ func buildCollectorPipelineProcessorsList(
userPipelineSet[p] = struct{}{}
}
if normalizerIdx := slices.IndexFunc(currentCollectorProcessors, isNormalizerProcessor); normalizerIdx >= 0 {
before := opampPosition(processorConfigs, currentCollectorProcessors[normalizerIdx]) == userPipelinesBeforeNormalizer
result := make([]string, 0, len(currentCollectorProcessors)+len(signozPipelineProcessorNames))
for idx, p := range currentCollectorProcessors {
if idx == normalizerIdx && before {
result = append(result, signozPipelineProcessorNames...)
}
if _, inUserPipelines := userPipelineSet[p]; !inUserPipelines && !hasSignozPipelineProcessorPrefix(p) {
result = append(result, p)
}
if idx == normalizerIdx && !before {
result = append(result, signozPipelineProcessorNames...)
}
}
return result, nil
}
var memoryLimiters []string
var customProcessors []string
batchIdx := -1
@@ -135,13 +161,51 @@ func buildCollectorPipelineProcessorsList(
return result, nil
}
func GenerateCollectorConfigWithPipelines(config []byte, pipelines []pipelinetypes.GettablePipeline) ([]byte, error) {
func isNormalizerProcessor(name string) bool {
return name == normalizerProcessor || strings.HasPrefix(name, normalizerProcessorPrefix)
}
// collectorNormalizesFirst reports whether the logs pipeline of the collector config runs
// signozlogsnormalizer ahead of user pipelines, judged by the first one as in buildCollectorPipelineProcessorsList.
// Malformed configs report false; generateCollectorConfigWithPipelines surfaces their errors.
func collectorNormalizesFirst(collectorConf map[string]interface{}) bool {
p, err := getOtelPipelineFromConfig(collectorConf)
if err != nil || p.Pipelines.Logs == nil {
return false
}
normalizerIdx := slices.IndexFunc(p.Pipelines.Logs.Processors, isNormalizerProcessor)
if normalizerIdx < 0 {
return false
}
processorConfigs, _ := collectorConf["processors"].(map[string]interface{})
return opampPosition(processorConfigs, p.Pipelines.Logs.Processors[normalizerIdx]) != userPipelinesBeforeNormalizer
}
func opampPosition(processorConfigs map[string]interface{}, name string) string {
config, _ := processorConfigs[name].(map[string]interface{})
opamp, _ := config["opamp"].(map[string]interface{})
position, _ := opamp["position"].(string)
return position
}
func unmarshalCollectorConfig(config []byte) (map[string]interface{}, error) {
var collectorConf map[string]interface{}
err := yaml.Unmarshal([]byte(config), &collectorConf)
if err != nil {
if err := yaml.Unmarshal(config, &collectorConf); err != nil {
return nil, errors.WrapInvalidInputf(err, CodeCollectorConfigUnmarshalFailed, "could not unmarshal collector config")
}
return collectorConf, nil
}
func GenerateCollectorConfigWithPipelines(config []byte, pipelines []pipelinetypes.GettablePipeline) ([]byte, error) {
collectorConf, err := unmarshalCollectorConfig(config)
if err != nil {
return nil, err
}
return generateCollectorConfigWithPipelines(collectorConf, pipelines)
}
// generateCollectorConfigWithPipelines modifies collectorConf in place.
func generateCollectorConfigWithPipelines(collectorConf map[string]any, pipelines []pipelinetypes.GettablePipeline) ([]byte, error) {
signozPipelineProcessors, signozPipelineProcNames, err := PreparePipelineProcessor(pipelines)
if err != nil {
return nil, err
@@ -180,7 +244,9 @@ func GenerateCollectorConfigWithPipelines(config []byte, pipelines []pipelinetyp
return nil, errors.NewInternalf(CodeCollectorConfigLogsPipelineNotFound, "logs pipeline doesn't exist")
}
updatedProcessorList, _ := buildCollectorPipelineProcessorsList(p.Pipelines.Logs.Processors, signozPipelineProcNames)
updatedProcessorList, _ := buildCollectorPipelineProcessorsList(
p.Pipelines.Logs.Processors, signozPipelineProcNames, collectorConf["processors"].(map[string]interface{}),
)
p.Pipelines.Logs.Processors = updatedProcessorList
// add the new processor to the data ( no checks required as the keys will exists)

View File

@@ -5,11 +5,14 @@ import (
"fmt"
"testing"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/query-service/constants"
"github.com/SigNoz/signoz/pkg/query-service/model"
v3 "github.com/SigNoz/signoz/pkg/query-service/model/v3"
"github.com/SigNoz/signoz/pkg/types/pipelinetypes"
"github.com/SigNoz/signoz/pkg/valuer"
. "github.com/smartystreets/goconvey/convey"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
)
@@ -206,13 +209,74 @@ var BuildLogsPipelineTestData = []struct {
func TestBuildLogsPipeline(t *testing.T) {
for _, test := range BuildLogsPipelineTestData {
Convey(test.Name, t, func() {
v, err := buildCollectorPipelineProcessorsList(test.fromCollector, test.userPipelines)
v, err := buildCollectorPipelineProcessorsList(test.fromCollector, test.userPipelines, nil)
So(err, ShouldBeNil)
So(v, ShouldResemble, test.finalOutput)
})
}
}
func TestBuildLogsPipelineWithNormalizer(t *testing.T) {
a := constants.LogsPPLPfx + "a"
b := constants.LogsPPLPfx + "b"
before := map[string]any{"opamp": map[string]any{"position": "before"}}
after := map[string]any{"opamp": map[string]any{"position": "after"}}
testCases := []struct {
name string
fromCollector []string
userPipelines []string
configs map[string]any
expected []string
}{
{
name: "ExistingAndStaleUserProcessorsReplaced",
fromCollector: []string{"memory_limiter", a, constants.LogsPPLPfx + "stale", constants.OldLogsPPLPfx + "old", "signozlogsnormalizer", "custom", "batch"},
userPipelines: []string{a, b},
expected: []string{"memory_limiter", "signozlogsnormalizer", a, b, "custom", "batch"},
},
{
name: "NamedNormalizerBefore_CustomProcessorsKeepPlace",
fromCollector: []string{"memory_limiter", "custom_a", "signozlogsnormalizer/x", a, "custom_b", "batch"},
userPipelines: []string{a},
configs: map[string]any{"signozlogsnormalizer/x": before},
expected: []string{"memory_limiter", "custom_a", a, "signozlogsnormalizer/x", "custom_b", "batch"},
},
{
name: "FirstNormalizerBefore_SecondIgnored",
fromCollector: []string{"memory_limiter", "signozlogsnormalizer/first", "signozlogsnormalizer/second", "batch"},
userPipelines: []string{a},
configs: map[string]any{"signozlogsnormalizer/first": before, "signozlogsnormalizer/second": after},
expected: []string{"memory_limiter", a, "signozlogsnormalizer/first", "signozlogsnormalizer/second", "batch"},
},
{
name: "FirstNormalizerAfter_SecondIgnored",
fromCollector: []string{"memory_limiter", "signozlogsnormalizer/first", "signozlogsnormalizer/second", "batch"},
userPipelines: []string{a},
configs: map[string]any{"signozlogsnormalizer/first": after, "signozlogsnormalizer/second": before},
expected: []string{"memory_limiter", "signozlogsnormalizer/first", a, "signozlogsnormalizer/second", "batch"},
},
{
name: "NormalizerAfterBatch",
fromCollector: []string{"memory_limiter", "batch", "signozlogsnormalizer"},
userPipelines: []string{a},
expected: []string{"memory_limiter", "batch", "signozlogsnormalizer", a},
},
{
name: "NoUserPipelines_StaleRemoved",
fromCollector: []string{"memory_limiter", a, "signozlogsnormalizer", "batch"},
expected: []string{"memory_limiter", "signozlogsnormalizer", "batch"},
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
processors, err := buildCollectorPipelineProcessorsList(testCase.fromCollector, testCase.userPipelines, testCase.configs)
require.NoError(t, err)
assert.Equal(t, testCase.expected, processors)
})
}
}
func TestPipelineAliasCollisionsDontResultInDuplicateCollectorProcessors(t *testing.T) {
require := require.New(t)
@@ -437,3 +501,61 @@ func TestPipeCharInAliasDoesntBreakCollectorConfig(t *testing.T) {
}, result[0].Attributes_string,
)
}
func TestRecommendAgentConfigWithNormalizerProcessorAndJSONBody(t *testing.T) {
userPipeline := makeTestAddAttributePipeline()
controller := &LogParsingPipelineController{
fl: flaggertest.WithUseJSONBody(t, true),
GetIntegrationPipelines: func(context.Context, string) ([]pipelinetypes.GettablePipeline, error) {
return []pipelinetypes.GettablePipeline{userPipeline}, nil
},
}
normalizeOperator := CollectorConfProcessorName(getNormalizePipeline())
user := CollectorConfProcessorName(userPipeline)
testCases := []struct {
name string
processors string
logs string
expected []any
}{
{
name: "NoNormalizer_OperatorPrepended",
logs: "[memory_limiter, batch]",
expected: []any{"memory_limiter", normalizeOperator, user, "batch"},
},
{
name: "NormalizerDefault_OperatorDropped",
logs: "[memory_limiter, signozlogsnormalizer, batch]",
expected: []any{"memory_limiter", "signozlogsnormalizer", user, "batch"},
},
{
name: "NormalizerAfter_OperatorDropped",
processors: "signozlogsnormalizer:\n opamp:\n position: after\n",
logs: "[memory_limiter, signozlogsnormalizer, batch]",
expected: []any{"memory_limiter", "signozlogsnormalizer", user, "batch"},
},
{
name: "NormalizerBefore_OperatorKept",
processors: "signozlogsnormalizer:\n opamp:\n position: before\n",
logs: "[memory_limiter, signozlogsnormalizer, batch]",
expected: []any{"memory_limiter", normalizeOperator, user, "signozlogsnormalizer", "batch"},
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
conf := fmt.Sprintf(
"processors:\n memory_limiter: {}\n batch: {}\n %s\nservice:\n pipelines:\n logs:\n receivers: [otlp]\n processors: %s\n exporters: [clickhouselogsexporter]\n",
testCase.processors, testCase.logs,
)
recommended, _, err := controller.RecommendAgentConfig(valuer.GenerateUUID(), []byte(conf), nil)
require.NoError(t, err)
var recommendedConf map[string]any
require.NoError(t, yaml.Unmarshal(recommended, &recommendedConf))
logs := recommendedConf["service"].(map[string]any)["pipelines"].(map[string]any)["logs"].(map[string]any)
assert.Equal(t, testCase.expected, logs["processors"])
})
}
}

View File

@@ -220,26 +220,17 @@ func (ic *LogParsingPipelineController) ValidatePipelines(ctx context.Context,
return err
}
// withNormalizePipeline places normalize where the read path dictates. Ahead of user pipelines
// when queries run on body_v2 (use_json_body), so operators see the body the explorer shows.
// After them when dual ingestion alone writes body_v2, so operators keep seeing the raw body
// users still query. Absent when neither flag is on.
func (ic *LogParsingPipelineController) withNormalizePipeline(ctx context.Context, orgID valuer.UUID, pipelines []pipelinetypes.GettablePipeline) []pipelinetypes.GettablePipeline {
evalCtx := featuretypes.NewFlaggerEvaluationContext(orgID)
dualIngestion := ic.fl.BooleanOrEmpty(ctx, flagger.FeatureJSONBodyDualIngestion, evalCtx)
switch {
case ic.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, evalCtx):
return append([]pipelinetypes.GettablePipeline{getNormalizePipeline(dualIngestion)}, pipelines...)
case dualIngestion:
return append(slices.Clone(pipelines), getNormalizePipeline(true))
default:
// withNormalizePipeline places normalize ahead of user pipelines when queries run on body_v2
// (use_json_body), so operators see the body the explorer shows. Absent when the flag is off
// or collectorNormalizesFirst.
func (ic *LogParsingPipelineController) withNormalizePipeline(ctx context.Context, orgID valuer.UUID, pipelines []pipelinetypes.GettablePipeline, collectorNormalizesFirst bool) []pipelinetypes.GettablePipeline {
if collectorNormalizesFirst || !ic.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
return pipelines
}
return append([]pipelinetypes.GettablePipeline{getNormalizePipeline()}, pipelines...)
}
// stashOriginalBody makes normalize carry the pre-normalization body in an internal attribute
// for the exporter to restore into the legacy body column.
func getNormalizePipeline(stashOriginalBody bool) pipelinetypes.GettablePipeline {
func getNormalizePipeline() pipelinetypes.GettablePipeline {
return pipelinetypes.GettablePipeline{
StoreablePipeline: pipelinetypes.StoreablePipeline{
Name: "Default Pipeline - PreProcessing Body",
@@ -258,11 +249,10 @@ func getNormalizePipeline(stashOriginalBody bool) pipelinetypes.GettablePipeline
},
Config: []pipelinetypes.PipelineOperator{
{
ID: uuid.NewString(),
Type: "normalize",
Enabled: true,
If: "body != nil",
JSONBodyDualIngestion: stashOriginalBody,
ID: uuid.NewString(),
Type: "normalize",
Enabled: true,
If: "body != nil",
},
},
}
@@ -371,11 +361,8 @@ func (ic *LogParsingPipelineController) PreviewLogsPipelines(
}
// The collector gets the same pipeline prepended over opamp; see RecommendAgentConfig.
// Under dual ingestion alone it runs after user operators and only feeds body_v2, which
// the explorer does not show yet, so the preview leaves it out. The original-body stash
// is left off: the preview has no exporter to restore and strip it.
if ic.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID)) {
pipelines = append([]pipelinetypes.GettablePipeline{getNormalizePipeline(false)}, pipelines...)
pipelines = append([]pipelinetypes.GettablePipeline{getNormalizePipeline()}, pipelines...)
}
result, collectorLogs, err := SimulatePipelinesProcessing(ctx, pipelines, request.Logs)
@@ -396,14 +383,14 @@ func (pc *LogParsingPipelineController) AgentFeatureType() agentConf.AgentFeatur
// Implements agentConf.AgentFeature interface.
// RecommendAgentConfig generates the collector config to be sent to agents.
// The normalize pipeline (when use_json_body or json_body_dual_ingestion is on) is placed
// The normalize pipeline (when use_json_body is on) is placed
// here, after rawPipelineData is serialized. So it is only present in the config sent to
// the collector and never persisted to the database as part of the user's pipeline list.
//
// NOTE: The configId sent to agents is derived from the pipeline version number
// (e.g. "LogPipelines:5"), not the YAML content. If server-side logic changes
// the generated YAML without bumping the version (e.g. toggling the use_json_body or
// json_body_dual_ingestion flags or updating operator IfExpressions), agents that already
// the generated YAML without bumping the version (e.g. toggling the use_json_body
// flag or updating operator IfExpressions), agents that already
// applied that version will not re-apply the new config. In such cases, users must save a
// new pipeline version via the API to force agents to pick up the change.
func (pc *LogParsingPipelineController) RecommendAgentConfig(
@@ -431,10 +418,15 @@ func (pc *LogParsingPipelineController) RecommendAgentConfig(
return nil, "", err
}
// normalize is only for sending to the collector, never persisted
enrichedPipelines = pc.withNormalizePipeline(ctx, orgId, enrichedPipelines)
collectorConf, err := unmarshalCollectorConfig(currentConfYaml)
if err != nil {
return nil, "", err
}
updatedConf, err := GenerateCollectorConfigWithPipelines(currentConfYaml, enrichedPipelines)
// normalize is only for sending to the collector, never persisted
enrichedPipelines = pc.withNormalizePipeline(ctx, orgId, enrichedPipelines, collectorNormalizesFirst(collectorConf))
updatedConf, err := generateCollectorConfigWithPipelines(collectorConf, enrichedPipelines)
if err != nil {
return nil, "", err
}

View File

@@ -122,9 +122,6 @@ type PipelineOperator struct {
EnablePaths bool `json:"enable_paths,omitempty" yaml:"enable_paths,omitempty"`
PathPrefix string `json:"path_prefix,omitempty" yaml:"path_prefix,omitempty"`
// normalize fields, set by the server only
JSONBodyDualIngestion bool `json:"-" yaml:"json_body_dual_ingestion,omitempty"`
// Used in Severity Parsing and JSON Flattening mapping
Mapping map[string][]string `json:"mapping,omitempty" yaml:"mapping,omitempty"`
// severity parser fields

View File

@@ -311,14 +311,15 @@ def create_json_index(
) -> Generator[Callable[[str, list[dict[str, Any]]], None]]:
"""
Create ClickHouse data-skipping indexes on body_v2 JSON sub-columns via
POST /api/v1/logs/promote_paths.
POST /api/v1/promoted_paths.
**Must be called BEFORE insert_logs** so that newly inserted data parts are
covered by the index and the QB uses the indexed condition path.
Each entry in `paths` follows the PromotePath API shape:
Each entry in `paths` follows the PromotePath API shape (signal/context are
filled in as logs/body):
{
"path": "body.user.name", # must start with "body."
"path": "user.name", # bare path, no "body." prefix
"indexes": [
{
"fieldDataType": "string", # string | int64 | float64
@@ -337,9 +338,9 @@ def create_json_index(
token = get_token(...)
export_json_types(logs_list)
create_json_body_index(token, [
{"path": "body.user.name",
{"path": "user.name",
"indexes": [{"fieldDataType": "string", "type": "ngrambf_v1(3, 256, 2, 0)", "granularity": 1}]},
{"path": "body.user.age",
{"path": "user.age",
"indexes": [{"fieldDataType": "int64", "type": "minmax", "granularity": 1}]},
])
insert_logs(logs_list) # data inserted after index exists — index is built automatically
@@ -348,18 +349,15 @@ def create_json_index(
def _create_json_body_index(token: str, paths: list[dict[str, Any]]) -> None:
response = requests.post(
signoz.self.host_configs["8080"].get("/api/v1/logs/promote_paths"),
signoz.self.host_configs["8080"].get("/api/v1/promoted_paths"),
headers={"authorization": f"Bearer {token}"},
json=paths,
json=[{"signal": "logs", "context": "body", **path} for path in paths],
timeout=30,
)
assert response.status_code == HTTPStatus.CREATED, f"Failed to create JSON body indexes: {response.status_code} {response.text}"
for path in paths:
# The API strips the "body." prefix before storing — mirror that here
# so our cleanup query uses the bare path (e.g. "user.name").
raw = path["path"].removeprefix("body.")
if raw not in created_paths:
created_paths.append(raw)
if path["path"] not in created_paths:
created_paths.append(path["path"])
yield _create_json_body_index

View File

@@ -404,7 +404,7 @@ def test_indexed_paths(
token,
[
{
"path": "body.user.raw-data.name",
"path": "user.raw-data.name",
"indexes": [
{
"fieldDataType": "string",
@@ -414,7 +414,7 @@ def test_indexed_paths(
],
},
{
"path": "body.user.raw-data.age",
"path": "user.raw-data.age",
"indexes": [
{
"fieldDataType": "int64",