test(tokenizer): verify last_observed_at is flushed to the sql store

Shortens the GC interval for the passwordauthn package so the flush
runs within a test.

.claude/skills/storybook-visual-diff/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: storybook-visual-diff
-description: Screenshot a set of SigNoz Storybook stories, then pixel-diff two runs to see what a CSS or component change did, with the changes tinted over the new shot. Also surveys where a component is used across the UI, ringing each instance in red and collecting every one into a single contact sheet. Use when asked to take story screenshots, capture a visual baseline, compare before/after of a style change, find which pages a change affects, or show every place a component appears.
+description: Screenshot a set of SigNoz Storybook stories, then pixel-diff two runs to see what a CSS or component change did, with the changes tinted over the new shot. Use when asked to take story screenshots, capture a visual baseline, compare before/after of a style change, or find which pages a change affects.
 ---
 
 # Storybook visual diff
@@ -22,7 +22,7 @@ says, take it and do not ask again; ask only for what is genuinely missing, in
 
 | To settle | Ask | Options |
 | --- | --- | --- |
-| Job | "What should this run produce?" | shoot only · survey where a component is used (§4) · baseline for a change you are about to make · compare against a change already in the working tree · compare this branch against another (`main` by default, or one the user names) · compare two configurations of the same story (`--args`, clock, width) · noise floor (same tree twice) |
+| Job | "What should this run produce?" | shoot only · baseline for a change you are about to make · compare against a change already in the working tree · compare this branch against another (`main` by default, or one the user names) · compare two configurations of the same story (`--args`, clock, width) · noise floor (same tree twice) |
 | Scope | "Which stories?" | offer 2-3 concrete selections read off `index.json` (a page, a `--title` prefix, everything), never open-ended |
 | Themes | "Which themes?" | dark · dark + light |
 | Read-out | "How should the diff read?" | `green` (changed pixels over the after shot) · `green-parallel` (before \| after \| diff, side by side) · `red` · `red-parallel` · `none` (keep both runs, do not diff) |
@@ -39,7 +39,6 @@ The job decides which loop below to run:
 | Job | Loop |
 | --- | --- |
 | **shoot only** | §1, §2, stop. Report the paths. No diff, no second run. |
-| **usage survey** | §1, §4, stop. One run, no diff: the question is where a component appears, not what moved. |
 | **baseline first** | the full loop, stopping after step 2 to hand the change back. The user makes it, then continue at step 4. |
 | **change already in the tree** | the tree *is* the after state. `git stash` (or check out the base commit) to shoot the before, restore, shoot the after. Confirm the working tree is clean enough to stash before touching it, and restore it even if a capture fails. |
 | **branch vs branch** | shoot the current branch, then `git switch <base>` in place (stash first if the tree is dirty), restart the dev server, shoot again, switch back and unstash. Restart matters: HMR does not survive a whole-branch swap cleanly. Get the tree back to where it started even if a capture fails. |
@@ -123,8 +122,6 @@ node scripts/story-shots.mjs .story-shots/baseline \
 | `--clock <iso\|live>` | wall clock the page reads, passed to the preview as `?storyClock`; `live` unfreezes it |
 | `--motion` | keep animations and transitions running (sets the `motion` global to `live`) |
 | `--ignore <selector>` | hide matching elements, on top of `[data-shot-ignore]` and `[data-chromatic="ignore"]` |
-| `--highlight <selector>` | also write `<id>--highlight.png`, every match ringed in red with 6px of padding |
-| `--crop <selector>` | also write one `crops/<id>--<n>.png` per match, and montage the theme's crops into `crops.png` |
 | `--flat` | write `<out>/<id>.png`, no theme directory |
 | `--no-caption` | leave the caption band off the shots |
 | `--list` | print the matched stories and exit |
@@ -132,9 +129,8 @@ node scripts/story-shots.mjs .story-shots/baseline \
 Files land at `<out>/<theme>/<story-id>.png`, next to a `shots.json` recording
 what each shot is (id, title, name, theme, `ok`/`busy`, the caption's height in
 rows) and how the run was configured (args, clock, width, height, grow, motion,
-settle, ignore, highlight, crop). Keep the flags identical between the two runs
-or the diff pairs nothing. `--highlight` and `--crop` write extra files beside
-the shots; §4 is what they are for.
+settle, ignore). Keep the flags identical between the two runs or the diff pairs
+nothing.
 
 Every shot carries the caption band described below, so a single screenshot says
 what it is on its own. `--no-caption` leaves it off, and so does a machine
@@ -212,75 +208,10 @@ pixelmatch's `includeAA: false`. So the script implements that comparison:
 A pair whose shots are different sizes is compared over the overlap, and every
 row and column that exists in only one of them counts as changed.
 
-Pairing is by `<theme>/<story-id>.png`. A file that exists on one side only (a
-story added on the feature branch, renamed, retitled, or one whose capture
-failed) has nothing to compare against. It is not skipped: the side that has the
-shot is written out, captioned `missing previous` or `missing current`, and every
-one of its pixels counts as changed, so it sorts to the top of the report and is
-printed with that note. In the parallel modes the run that does not have it gets
-a placeholder tile saying so, in the theme's own colours, so the montage keeps
-its three tiles. Without this a whole component going missing reads as a clean
-run.
-
-## 4. Surveying where a component is used
-
-A different question from a diff: not *what moved*, but *where does this
-component appear and what does each instance look like*. One run answers it.
-
-```bash
-node scripts/story-shots.mjs .story-shots/button-group --port 6007 --theme dark,light \
-  --highlight '.ant-btn-group, div[role="group"][class*="button-group"]' \
-  --crop '.ant-btn-group, div[role="group"][class*="button-group"]' \
-  --stories pages-home--default,pages-alerts-history--default,...
-```
-
-Four things come out, per theme:
-
-- `<theme>/<id>.png` — the page as it is.
-- `<theme>/<id>--highlight.png` — the same page with every instance ringed in
-  red. This is what says *where on the page*, which a crop cannot.
-- `<theme>/crops/<id>--<n>.png` — each instance on its own.
-- `<theme>/crops.png` — every crop of that theme in one labelled contact sheet.
-
-The sheet is the useful artifact. Twelve instances across nine pages is one
-image to read, not twelve files to open in turn, and the label under each says
-which story it came from.
-
-### Finding the selector and the stories
-
-1. **Grep the source for the import, not the tag.** `Button.Group` and
-   `ButtonGroup` are two different components in this repo: antd's, and
-   `@signozhq/ui/button`'s. A survey that greps one misses the other.
-2. **Read the rendered markup, not the JSX.** `--crop` takes a CSS selector
-   against the DOM. antd's group is `.ant-btn-group`; the design-system one is a
-   `div[role="group"]` whose class is a hashed CSS module, hence
-   `[class*="button-group"]`. Open the built component under
-   `node_modules/@signozhq/ui/dist/` when the class is not obvious.
-3. **Map each source file to the story that renders it.** Follow the consumers:
-   a container renders inside a page, and the page's story is the one to shoot.
-   A component behind a drawer or a tab needs the story whose `args` open it
-   (`--args drawer:endpoint-stats`), not the page default.
-4. **Let the run itself confirm the mapping.** Each story logs `N cropped`. A
-   `0 cropped` line means that story never reaches the state, so swap the story
-   rather than the selector.
-
-### What a zero means
-
-- **`0 cropped` on a story** — the component is not on that page in that state.
-  Wrong story, or the state is behind an interaction the story has no `play`
-  for. A modal nobody opens cannot be surveyed; say so instead of shooting the
-  page it sits behind.
-- **An instance in the source with no crop** — a container whose children are
-  all conditional renders as a 0x0 box. Both `--crop` and `--highlight` skip
-  anything under 1px, since there is nothing on screen to ring. That is a
-  finding about the component, not a failure of the run: it is in the tree and
-  invisible.
-
-A story logged `viewport-sized content, stopped chasing Npx` is shot back at
-`--height` with its own scrollbar, and what is below the fold there is laid out
-but never painted. The crops of such a story are taken at the chased height
-instead, so they are not the black rectangles the page shot would give; the
-`--highlight` shot still shows only what fits the viewport.
+Pairing is by `<theme>/<story-id>.png`, so a story that exists on only one side
+(new on the feature branch, renamed, retitled) has nothing to pair with and is
+skipped silently. On a branch-vs-branch run, compare the two runs' file lists
+before reading the numbers.
 
 ## What makes a shot reproducible
 
@@ -361,6 +292,3 @@ for `--ignore` when a region cannot be settled.
 - **Stories behind a hover, drawer or modal** only render what their `play`
   reaches. If a state is missing from the shot, the story needs the `play`, not
   the script.
-- **The caption's temporary file is written beside the shot**, not in the system
-  temp directory. `/tmp` is often a different filesystem, and the rename back
-  over the shot then fails with `EXDEV: cross-device link not permitted`.

.github/CODEOWNERS

@@ -280,3 +280,9 @@ go.mod @therealpandey
 /frontend/src/components/MessagingQueues/ @SigNoz/events-frontend
 /frontend/src/components/MessagingQueueHealthCheck/ @SigNoz/events-frontend
 /frontend/src/hooks/messagingQueue/ @SigNoz/events-frontend
+
+## Storybook
+/frontend/.storybook/ @H4ad
+/frontend/src/storybook/ @H4ad
+/.claude/skills/signoz-page-story/ @H4ad
+/.claude/skills/storybook-visual-diff/ @H4ad

ee/query-service/app/api/featureFlags.go

@@ -80,15 +80,6 @@ func (ah *APIHandler) getFeatureFlags(w http.ResponseWriter, r *http.Request) {
 		Route:      "",
 	})
 
-	aiObservability := ah.Signoz.Flagger.BooleanOrEmpty(ctx, flagger.FeatureEnableAIObservability, evalCtx)
-	featureSet = append(featureSet, &licensetypes.Feature{
-		Name:       valuer.NewString(flagger.FeatureEnableAIObservability.String()),
-		Active:     aiObservability,
-		Usage:      0,
-		UsageLimit: -1,
-		Route:      "",
-	})
-
 	metricsReduction := ah.Signoz.Flagger.BooleanOrEmpty(ctx, flagger.FeatureEnableMetricsReduction, evalCtx)
 	featureSet = append(featureSet, &licensetypes.Feature{
 		Name:       valuer.NewString(flagger.FeatureEnableMetricsReduction.String()),

frontend/scripts/story-shots-caption.mjs

@@ -15,8 +15,6 @@ export const CONFIG_KEYS = [
 	'motion',
 	'settle',
 	'ignore',
-	'highlight',
-	'crop',
 ];
 
 let tools;

frontend/scripts/story-shots-diff.mjs

@@ -1,7 +1,8 @@
 #!/usr/bin/env node
-import { mkdir, readdir, readFile, rm, writeFile } from 'node:fs/promises';
+import { mkdir, readdir, readFile, writeFile } from 'node:fs/promises';
 import { parseArgs } from 'node:util';
 import path from 'node:path';
+import os from 'node:os';
 
 import {
 	bodyFont,
@@ -17,8 +18,7 @@ import {
 
 /**
  * Pairs the PNGs of two story-shots.mjs runs by relative path and reports what
- * moved, per pair, largest first. A shot only one run has is reported too,
- * labelled with the side it is missing from and counted as changed in full.
+ * moved, per pair, largest first.
  *
  * The comparison is Chromatic's: a pixel counts as changed when its YIQ
  * distance from the baseline pixel is over `threshold` of the largest distance
@@ -55,8 +55,7 @@ if (opts.help || !baseDir || !afterDir || !MODES.has(opts.mode)) {
   --tint <#rrggbb>       override the mode's highlight colour
   --no-caption           do not stamp the story and the run settings on top
 
-Prints "<changed pixels>  <relative path>", largest first; a shot only one run
-has is printed as "(missing previous)" or "(missing current)". Needs ImageMagick.`);
+Prints "<changed pixels>  <relative path>", largest first. Needs ImageMagick.`);
 	process.exit(opts.help ? 0 : 1);
 }
 
@@ -326,11 +325,6 @@ const shotOf = (run, rel) =>
 
 const captionOf = (run, rel) => shotOf(run, rel)?.caption ?? 0;
 
-/** What the shot was shot in, falling back to the directory it sits in. */
-const themeOf = (rel) =>
-	(shotOf(afterRun, rel) ?? shotOf(baseRun, rel))?.theme ??
-	rel.split(path.sep)[0];
-
 /** ImageMagick's inline crop, so a tile shows the shot without its caption. */
 const withoutCaption = (file, { width, height }, top) =>
 	top > 0 ? `${file}[${width}x${height}+0+${top}]` : file;
@@ -375,156 +369,20 @@ const pngs = async (dir, prefix = '') => {
 const results = [];
 await mkdir(outDir, { recursive: true });
 
-/** The half-built tiles, under the output directory so nothing is left elsewhere. */
-const scratch = Object.fromEntries(
-	['body', 'diff', 'shot', 'missing'].map((name) => [
-		name,
-		path.join(outDir, `.story-shots-${process.pid}-${name}.png`),
-	]),
-);
-
-/** One labelled tile of a parallel montage. */
-const tile = (label, file, background) => [
-	'(',
-	`label:${literal(label)}`,
-	file,
-	'-gravity',
-	'center',
-	'-append',
-	'-bordercolor',
-	background,
-	'-border',
-	'12',
-	')',
-];
-
-/** The tiles side by side under one caption. */
-const montage = ({
-	tiles,
-	width,
-	background,
-	foreground,
-	caption,
-	target,
-	theme,
-}) => {
-	magick([
-		'-background',
-		background,
-		'-fill',
-		foreground,
-		...bodyFont(),
-		'-pointsize',
-		// The tiles end up side by side, so they are read at the montage's width.
-		String(Math.round(pointsize(width * 3) * 0.62)),
-		...tiles.flat(),
-		'-gravity',
-		'north',
-		'+append',
-		caption.length ? scratch.body : target,
-	]);
-	if (caption.length) {
-		stamp({ lines: caption, from: scratch.body, to: target, theme });
-	}
-};
-
-/**
- * A tile standing in for a shot the run does not have, sized like the one it
- * does. The gutter's colours are the theme's own inverted, so they go back the
- * other way here and the tile reads as a shot rather than as a hole.
- */
-const placeholder = (
-	file,
-	{ width, height },
-	text,
-	{ background, foreground },
-) =>
-	magick([
-		'-size',
-		`${width}x${height}`,
-		'-background',
-		foreground,
-		'-fill',
-		background,
-		'-gravity',
-		'center',
-		...bodyFont(),
-		'-pointsize',
-		String(pointsize(width * 3)),
-		`label:${literal(text)}`,
-		file,
-	]);
-
-const [baseFiles, afterFiles] = await Promise.all([
-	pngs(baseDir),
-	pngs(afterDir),
-]);
-const inBase = new Set(baseFiles);
-const inAfter = new Set(afterFiles);
-
-for (const rel of [...new Set([...baseFiles, ...afterFiles])].sort((a, b) =>
-	a.localeCompare(b),
-)) {
+for (const rel of (await pngs(baseDir)).sort()) {
 	const afterFile = path.join(afterDir, rel);
-	const target = path.join(outDir, rel);
-	await mkdir(path.join(outDir, path.dirname(rel)), { recursive: true });
-
-	// A story added, removed or renamed since the baseline has nothing to
-	// compare against, so the side that does have it is written out under the
-	// label of the side that does not, and every one of its pixels counts.
-	if (!inBase.has(rel) || !inAfter.has(rel)) {
-		const gone = inAfter.has(rel) ? 'previous' : 'current';
-		const held = gone === 'previous' ? 'current' : 'previous';
-		const run = gone === 'previous' ? afterRun : baseRun;
-		const image = readRgba(
-			gone === 'previous' ? afterFile : path.join(baseDir, rel),
-			captionOf(run, rel),
-		);
-		const theme = themeOf(rel);
-		const colors = palette(theme);
-		const caption = captionLines(rel, [`missing ${gone}`]);
-
-		if (opts.mode.endsWith('-parallel')) {
-			// The montage keeps its three tiles: the run that has the shot shows it,
-			// and the run that does not, like the diff, says so in its place. There
-			// is nothing to compare, so nothing is tinted.
-			await writeRgba(image, scratch.shot);
-			placeholder(scratch.missing, image, `missing ${gone}`, colors);
-
-			const sides = {
-				[held]: tile(sideLabel(held, run), scratch.shot, colors.background),
-				[gone]: tile(
-					sideLabel(gone, gone === 'previous' ? baseRun : afterRun),
-					scratch.missing,
-					colors.background,
-				),
-			};
-			montage({
-				tiles: [
-					sides.previous,
-					sides.current,
-					tile('diff', scratch.missing, colors.background),
-				],
-				width: image.width,
-				...colors,
-				caption,
-				target,
-				theme,
-			});
-		} else {
-			await writeRgba(image, caption.length ? scratch.body : target);
-			if (caption.length) {
-				stamp({ lines: caption, from: scratch.body, to: target, theme });
-			}
-		}
-
-		results.push([image.width * image.height, rel, `missing ${gone}`]);
+	const base = readRgba(path.join(baseDir, rel), captionOf(baseRun, rel));
+	let after;
+	try {
+		after = readRgba(afterFile, captionOf(afterRun, rel));
+	} catch {
+		console.error(`missing in after: ${rel}`);
 		continue;
 	}
 
-	const base = readRgba(path.join(baseDir, rel), captionOf(baseRun, rel));
-	const after = readRgba(afterFile, captionOf(afterRun, rel));
+	await mkdir(path.join(outDir, path.dirname(rel)), { recursive: true });
 	const diff = diffPair(base, after, opts.mode);
+	const target = path.join(outDir, rel);
 	const parallel = opts.mode.endsWith('-parallel');
 	// With no tiles to label, a run's own settings go in the caption instead.
 	const caption = captionLines(
@@ -533,54 +391,70 @@ for (const rel of [...new Set([...baseFiles, ...afterFiles])].sort((a, b) =>
 			? []
 			: [sideLabel('previous', baseRun), sideLabel('current', afterRun)],
 	);
+	const diffFile = path.join(os.tmpdir(), `story-shots-${process.pid}.png`);
+	const body = path.join(os.tmpdir(), `story-shots-${process.pid}-body.png`);
 
 	// The gutter is the opposite of the theme's own background, so the tiles and
 	// the caption keep an edge instead of bleeding into it.
-	const theme = themeOf(rel);
+	const shot = shotOf(afterRun, rel) ?? shotOf(baseRun, rel);
+	const theme = shot?.theme ?? rel.split(path.sep)[0];
 	const { background, foreground } = palette(theme);
 
 	if (parallel) {
-		await writeRgba(diff, scratch.diff);
-		montage({
-			tiles: [
-				tile(
-					sideLabel('previous', baseRun),
-					withoutCaption(path.join(baseDir, rel), base, captionOf(baseRun, rel)),
-					background,
-				),
-				tile(
-					sideLabel('current', afterRun),
-					withoutCaption(afterFile, after, captionOf(afterRun, rel)),
-					background,
-				),
-				tile('diff', scratch.diff, background),
-			],
-			width: after.width,
+		await writeRgba(diff, diffFile);
+		const tile = (label, file) => [
+			'(',
+			`label:${literal(label)}`,
+			file,
+			'-gravity',
+			'center',
+			'-append',
+			'-bordercolor',
 			background,
+			'-border',
+			'12',
+			')',
+		];
+		magick([
+			'-background',
+			background,
+			'-fill',
 			foreground,
-			caption,
-			target,
-			theme,
-		});
-	} else {
-		await writeRgba(diff, caption.length ? scratch.body : target);
+			...bodyFont(),
+			'-pointsize',
+			// The tiles end up side by side, so they are read at the montage's width.
+			String(Math.round(pointsize(after.width * 3) * 0.62)),
+			...tile(
+				sideLabel('previous', baseRun),
+				withoutCaption(path.join(baseDir, rel), base, captionOf(baseRun, rel)),
+			),
+			...tile(
+				sideLabel('current', afterRun),
+				withoutCaption(afterFile, after, captionOf(afterRun, rel)),
+			),
+			...tile('diff', diffFile),
+			'-gravity',
+			'north',
+			'+append',
+			caption.length ? body : target,
+		]);
 		if (caption.length) {
-			stamp({ lines: caption, from: scratch.body, to: target, theme });
+			stamp({ lines: caption, from: body, to: target, theme });
+		}
+	} else {
+		await writeRgba(diff, caption.length ? body : target);
+		if (caption.length) {
+			stamp({ lines: caption, from: body, to: target, theme });
 		}
 	}
 
 	results.push([diff.changed, rel]);
 }
 
-await Promise.all(
-	Object.values(scratch).map((file) => rm(file, { force: true })),
-);
-
 results
 	.sort((a, b) => b[0] - a[0])
-	.forEach(([changed, rel, note]) => {
-		const suffix = note ? `  (${note})` : '';
-		console.log(`${String(changed).padStart(10)}  ${rel}${suffix}`);
-	});
+	.forEach(([changed, rel]) =>
+		console.log(`${String(changed).padStart(10)}  ${rel}`),
+	);
 
 console.error(`diffs in ${outDir}`);

frontend/scripts/story-shots.mjs

@@ -1,19 +1,15 @@
 #!/usr/bin/env node
 import { spawnSync } from 'node:child_process';
-import { renameSync, rmSync } from 'node:fs';
-import { mkdir, writeFile } from 'node:fs/promises';
+import { mkdir, rename, writeFile } from 'node:fs/promises';
 import { createRequire } from 'node:module';
 import { parseArgs } from 'node:util';
 import { pathToFileURL } from 'node:url';
 import path from 'node:path';
+import os from 'node:os';
 
 import {
-	bodyFont,
 	CONFIG_KEYS,
 	hasMagick,
-	literal,
-	magick,
-	palette,
 	settingsLine,
 	stamp,
 } from './story-shots-caption.mjs';
@@ -43,8 +39,6 @@ const { values: opts, positionals } = parseArgs({
 		clock: { type: 'string', default: FROZEN_CLOCK },
 		motion: { type: 'boolean', default: false },
 		ignore: { type: 'string', multiple: true, default: [] },
-		highlight: { type: 'string', multiple: true, default: [] },
-		crop: { type: 'string', multiple: true, default: [] },
 		flat: { type: 'boolean', default: false },
 		'no-caption': { type: 'boolean', default: false },
 		list: { type: 'boolean', default: false },
@@ -78,10 +72,6 @@ if (opts.help || (!outDir && !opts.list)) {
   --clock <iso|live>  wall clock the page reads (default ${FROZEN_CLOCK})
   --motion            keep animations and transitions running
   --ignore <selector> hide matching elements, on top of [data-shot-ignore]
-  --highlight <selector>
-                      also shoot <id>--highlight.png, every match ringed in red
-  --crop <selector>   also write one <id>--<n>.png per match under <theme>/crops,
-                      and montage them into <theme>/crops.png
   --flat              write <out>/<id>.png instead of <out>/<theme>/<id>.png
   --no-caption        do not stamp the story and the run settings on the shot
   --list              print the matched stories and exit
@@ -145,17 +135,10 @@ if (!stories.length) {
 	process.exit(1);
 }
 
-/** A repeatable, comma-separated flag read as one CSS selector list. */
-const selectorList = (values) =>
-	values
-		.flatMap((value) => value.split(','))
-		.map((value) => value.trim())
-		.filter(Boolean)
-		.join(', ');
-
-const ignoreSelectors = selectorList(opts.ignore);
-const highlightSelector = selectorList(opts.highlight);
-const cropSelector = selectorList(opts.crop);
+const ignoreSelectors = opts.ignore
+	.flatMap((value) => value.split(','))
+	.map((value) => value.trim())
+	.filter(Boolean);
 
 if (opts.clock !== 'live' && Number.isNaN(Date.parse(opts.clock))) {
 	console.error(`--clock: not a date: ${opts.clock}`);
@@ -169,12 +152,13 @@ if (opts.clock !== 'live' && Number.isNaN(Date.parse(opts.clock))) {
  * their bottom - is done by the preview itself, so a Chromatic build and a shot
  * from here see the same page.
  */
-const ignoreCss = (ignore) => {
-	const extra = ignore ? `, ${ignore}` : '';
-	return `[data-shot-ignore], [data-chromatic='ignore']${extra} {
+const ignoreCss = (
+	ignore,
+) => `[data-shot-ignore], [data-chromatic='ignore']${ignore
+	.map((selector) => `, ${selector}`)
+	.join('')} {
 	visibility: hidden !important;
 }`;
-};
 
 /**
  * Playwright is not a frontend dependency: it lives in `tests/e2e`, or globally,
@@ -264,9 +248,7 @@ const runConfig = {
 	grow: opts.grow,
 	motion: opts.motion ? 'live' : 'still',
 	settle: opts.settle,
-	ignore: ignoreSelectors,
-	highlight: highlightSelector,
-	crop: cropSelector,
+	ignore: ignoreSelectors.join(', '),
 };
 
 const captioning = !opts['no-caption'] && hasMagick();
@@ -279,8 +261,6 @@ const configLine = settingsLine(runConfig, CONFIG_KEYS);
 
 for (const theme of themes.length ? themes : [null]) {
 	const dir = opts.flat ? outDir : path.join(outDir, theme ?? 'default');
-	const cropDir = path.join(dir, 'crops');
-	const crops = [];
 	await mkdir(dir, { recursive: true });
 	if (theme) {
 		console.log(`\n[${theme}]`);
@@ -315,9 +295,6 @@ for (const theme of themes.length ? themes : [null]) {
 		// the viewport, kept only to flag the story in the log.
 		let chasing = 0;
 
-		// What the story's line in the log says beyond ok/busy.
-		const notes = [];
-
 		const url = new URL(`${base}/iframe.html`);
 		url.searchParams.set('viewMode', 'story');
 		url.searchParams.set('id', story.id);
@@ -470,160 +447,47 @@ for (const theme of themes.length ? themes : [null]) {
 				shot = next;
 			}
 
-			/**
-			 * The band goes on the shot itself so a single screenshot says what it
-			 * is, and its height is returned so a diff can take it back off. The
-			 * temporary is written beside the shot rather than in the system temp
-			 * directory: those are often separate filesystems, and a rename across
-			 * one fails with EXDEV.
-			 */
-			const caption = (target, lines) => {
-				if (!captioning) {
-					return 0;
-				}
+			const file = path.join(dir, `${story.id}.png`);
+			await writeFile(file, shot);
 
+			// The band goes on the shot itself so a single screenshot says what it
+			// is, and its height is recorded so a diff can take it back off.
+			let caption = 0;
+			if (captioning) {
 				const temporary = path.join(
-					path.dirname(target),
-					`.caption-${process.pid}.png`,
+					os.tmpdir(),
+					`story-shots-caption-${process.pid}.png`,
 				);
-				const rows = stamp({
+				caption = stamp({
 					lines: [
 						`${story.title}/${story.name}`,
 						[story.id, theme ?? 'default', stable ? '' : '(busy)']
 							.filter(Boolean)
 							.join('  '),
-						...lines,
+						configLine,
 					].filter(Boolean),
-					from: target,
+					from: file,
 					to: temporary,
 					theme: theme ?? 'dark',
 				});
-				renameSync(temporary, target);
-				return rows;
-			};
-
-			const file = path.join(dir, `${story.id}.png`);
-			await writeFile(file, shot);
-
-			const record = (relative, caption) =>
-				shots.push({
-					file: path.posix.join(opts.flat ? '' : (theme ?? 'default'), relative),
-					id: story.id,
-					title: story.title,
-					name: story.name,
-					theme: theme ?? 'default',
-					status: stable ? 'ok' : 'busy',
-					caption,
-				});
-
-			record(`${story.id}.png`, caption(file, [configLine]));
-
-			// The crops are taken before anything is drawn over the page, so a
-			// component's own shot carries no ring and no label: the montage at the
-			// end of the theme is what names them.
-			if (cropSelector) {
-				await mkdir(cropDir, { recursive: true });
-
-				// A page that sizes itself in `vh` was shot back at `--height` with
-				// its own scrollbar, and what is below the fold there is laid out but
-				// never painted: cropping it gives a black rectangle. The crops alone
-				// are taken at the height the rounds had reached, which is where the
-				// page does paint.
-				if (chasing) {
-					await page.setViewportSize({
-						width: Number(opts.width),
-						height: chasing,
-					});
-					await page.waitForTimeout(Number(opts.settle));
-				}
-
-				const matches = page.locator(cropSelector);
-				let kept = 0;
-				for (let index = 0; index < (await matches.count()); index += 1) {
-					const element = matches.nth(index);
-					// A group whose children are all conditional renders as a 0x0 box.
-					// It has no counterpart on screen, so there is nothing to crop.
-					const box = await element.boundingBox();
-					if (!box || box.width < 1 || box.height < 1) {
-						continue;
-					}
-
-					kept += 1;
-					const relative = `${story.id}--${kept}.png`;
-					// The scroll that brings an element into view needs a frame before
-					// the crop, or the region comes back unpainted.
-					await element.scrollIntoViewIfNeeded({ timeout: 15_000 });
-					await page.waitForTimeout(250);
-					await element.screenshot({
-						path: path.join(cropDir, relative),
-						timeout: 15_000,
-					});
-					crops.push({
-						file: path.join(cropDir, relative),
-						label: `${story.title}/${story.name}  #${kept}`,
-					});
-					record(path.posix.join('crops', relative), 0);
-				}
-				notes.push(`${kept} cropped`);
-
-				if (chasing) {
-					await page.setViewportSize({
-						width: Number(opts.width),
-						height: Number(opts.height),
-					});
-					await page.waitForTimeout(Number(opts.settle));
-				}
+				await rename(temporary, file);
 			}
 
-			if (highlightSelector) {
-				const ringed = await page.evaluate(
-					([selector, padding]) => {
-						const layer = document.createElement('div');
-						// The shot is viewport-sized, so the rings are placed in viewport
-						// coordinates and survive a page that stayed scrollable.
-						layer.style.cssText =
-							'position:fixed;inset:0;pointer-events:none;z-index:2147483647';
-						let drawn = 0;
-						for (const element of document.querySelectorAll(selector)) {
-							const box = element.getBoundingClientRect();
-							if (box.width < 1 || box.height < 1) {
-								continue;
-							}
-
-							drawn += 1;
-							const ring = document.createElement('div');
-							ring.style.cssText = `position:fixed;box-sizing:border-box;border:3px solid #ff003a;border-radius:4px;left:${
-								box.left - padding
-							}px;top:${box.top - padding}px;width:${
-								box.width + padding * 2
-							}px;height:${box.height + padding * 2}px`;
-							layer.append(ring);
-						}
-						document.documentElement.append(layer);
-						window.__storyShotsHighlight = layer;
-						return drawn;
-					},
-					[highlightSelector, 6],
-				);
-
-				const highlighted = path.join(dir, `${story.id}--highlight.png`);
-				await writeFile(highlighted, await page.screenshot());
-				record(
-					`${story.id}--highlight.png`,
-					caption(highlighted, [`${ringed} highlighted`, configLine]),
-				);
-				notes.push(`${ringed} highlighted`);
-				await page.evaluate(() => {
-					window.__storyShotsHighlight?.remove();
-					delete window.__storyShotsHighlight;
-				});
-			}
-			if (chasing) {
-				notes.push(`viewport-sized content, stopped chasing ${chasing}px`);
-			}
+			shots.push({
+				file: path.posix.join(
+					opts.flat ? '' : (theme ?? 'default'),
+					`${story.id}.png`,
+				),
+				id: story.id,
+				title: story.title,
+				name: story.name,
+				theme: theme ?? 'default',
+				status: stable ? 'ok' : 'busy',
+				caption,
+			});
 			console.log(
 				`  ${stable ? 'ok  ' : 'busy'} ${story.id}${
-					notes.length ? ` (${notes.join(', ')})` : ''
+					chasing ? ` (viewport-sized content, stopped chasing ${chasing}px)` : ''
 				}`,
 			);
 		} catch (error) {
@@ -633,52 +497,6 @@ for (const theme of themes.length ? themes : [null]) {
 			await context.close();
 		}
 	}
-
-	// One image of every crop the theme produced, labelled with the story it came
-	// from. A component that appears on eight pages is a survey rather than eight
-	// screenshots to open one after another.
-	if (crops.length && captioning) {
-		const sheet = path.join(dir, 'crops.png');
-		const { background, foreground } = palette(theme ?? 'dark');
-
-		// `montage -label` sizes every tile to the widest *image*, so a label
-		// longer than its crop runs under the next one. Each crop is composed with
-		// its own label first, which sizes the tile to whichever of the two is
-		// wider, and the sheet is then a montage of finished tiles.
-		const tiles = crops.map(({ file, label }, index) => {
-			const tile = path.join(cropDir, `.tile-${index}.png`);
-			magick([
-				'-background',
-				background,
-				'-fill',
-				foreground,
-				...bodyFont(),
-				'-pointsize',
-				'16',
-				file,
-				`label:${literal(label)}`,
-				'-gravity',
-				'center',
-				'-append',
-				tile,
-			]);
-			return tile;
-		});
-
-		magick([
-			'montage',
-			'-background',
-			background,
-			'-tile',
-			'2x',
-			'-geometry',
-			'+16+16',
-			...tiles,
-			sheet,
-		]);
-		tiles.forEach((tile) => rmSync(tile, { force: true }));
-		console.log(`  ${crops.length} crops -> ${sheet}`);
-	}
 }
 
 await browser.close();

frontend/src/AppRoutes/Private.tsx

@@ -8,7 +8,6 @@ import { ORG_PREFERENCES } from 'constants/orgPreferences';
 import ROUTES from 'constants/routes';
 import { useGetTenantLicense } from 'hooks/useGetTenantLicense';
 import { useIsAIAssistantEnabled } from 'hooks/useIsAIAssistantEnabled';
-import { useIsAIObservabilityEnabled } from 'hooks/useIsAIObservabilityEnabled';
 import { isEmpty } from 'lodash-es';
 import { useAppContext } from 'providers/App/App';
 import { LicensePlatform, LicenseState } from 'types/api/licensesV3/getActive';
@@ -44,7 +43,6 @@ function PrivateRoute({ children }: PrivateRouteProps): JSX.Element {
 
 	const isAdmin = user.role === USER_ROLES.ADMIN;
 	const isAIAssistantEnabled = useIsAIAssistantEnabled();
-	const isAIObservabilityEnabled = useIsAIObservabilityEnabled();
 	const mapRoutes = useMemo(
 		() =>
 			new Map(
@@ -135,14 +133,6 @@ function PrivateRoute({ children }: PrivateRouteProps): JSX.Element {
 		return <Redirect to={ROUTES.HOME} />;
 	}
 
-	if (
-		(pathname.startsWith(`${ROUTES.AI_OBSERVABILITY_BASE}/`) ||
-			pathname === ROUTES.AI_OBSERVABILITY_BASE) &&
-		!isAIObservabilityEnabled
-	) {
-		return <Redirect to={ROUTES.HOME} />;
-	}
-
 	// Check for workspace access restriction (cloud only)
 	const isCloudPlatform = activeLicense?.platform === LicensePlatform.CLOUD;
 

frontend/src/components/NewSelect/__test__/CustomSelect.test.tsx

@@ -1,4 +1,5 @@
 import { fireEvent, render, screen, waitFor } from '@testing-library/react';
+import userEvent from '@testing-library/user-event';
 
 import CustomSelect from '../CustomSelect';
 
@@ -203,4 +204,21 @@ describe('CustomSelect Component', () => {
 		// Check onChange was called
 		expect(handleChange).toHaveBeenCalled();
 	});
+	it('tells the consumer its search was cleared when the dropdown closes', async () => {
+		// The component clears its own search text on close. A consumer running a
+		// server-side search needs to hear that, or its results outlive the dropdown.
+		const onSearch = jest.fn();
+		const user = userEvent.setup();
+		render(<CustomSelect options={mockOptions} onSearch={onSearch} />);
+
+		const selectElement = screen.getByRole('combobox');
+		await user.click(selectElement);
+		await user.type(selectElement, 'opt');
+
+		expect(onSearch).toHaveBeenLastCalledWith('opt');
+
+		await user.keyboard('{Escape}');
+
+		expect(onSearch).toHaveBeenLastCalledWith('');
+	});
 });

frontend/src/components/NewSelect/styles.scss

@@ -258,6 +258,10 @@ $custom-border-color: #2c3044;
 		overflow: hidden;
 
 		.group-label {
+			display: flex;
+			align-items: center;
+			gap: 4px;
+
 			font-weight: 500;
 			padding: 4px 12px;
 			font-size: 13px;
@@ -442,7 +446,7 @@ $custom-border-color: #2c3044;
 		.group-label {
 			display: flex;
 			align-items: center;
-			justify-content: space-between;
+			gap: 4px;
 
 			font-weight: 500;
 			padding: 4px 12px;

frontend/src/constants/features.ts

@@ -8,6 +8,5 @@ export enum FeatureKeys {
 	PREMIUM_SUPPORT = 'premium_support',
 	ANOMALY_DETECTION = 'anomaly_detection',
 	USE_JSON_BODY = 'use_json_body',
-	ENABLE_AI_OBSERVABILITY = 'enable_ai_observability',
 	ENABLE_METRICS_REDUCTION = 'enable_metrics_reduction',
 }

frontend/src/container/LLMObservability/AttributeMapping/LLMObservabilityAttributeMapping.tsx

@@ -1,5 +1,6 @@
-import { useCallback } from 'react';
+import { useCallback, useEffect } from 'react';
 import { Tabs } from 'antd';
+import logEvent from 'api/common/logEvent';
 import { useConfirmableAction } from 'hooks/useConfirmableAction';
 
 import AttributeMappingActions from './components/AttributeMappingActions/AttributeMappingActions';
@@ -20,6 +21,10 @@ function LLMObservabilityAttributeMapping(): JSX.Element {
 	const groupDrawer = useGroupFormDrawer();
 	const spanTest = useTestSpanMapper(editor.snapshot, editor.groups);
 
+	useEffect(() => {
+		void logEvent('AI Observability Attribute Mapping: Page visited', {});
+	}, []);
+
 	const { discard } = editor;
 	// Discarding wipes the whole working copy, so gate it behind a confirm
 	// prompt rather than firing straight from the button.

frontend/src/container/LLMObservability/AttributeMapping/TestTab/useTestSpanMapper.ts

@@ -1,4 +1,5 @@
 import { useCallback, useEffect, useMemo, useState } from 'react';
+import logEvent from 'api/common/logEvent';
 import {
 	RenderErrorResponseDTO,
 	SpantypesSpanMapperTestSpanDTO,
@@ -126,6 +127,7 @@ export function useTestSpanMapper(
 			{ data: body },
 			{
 				onSuccess: (response) => {
+					void logEvent('AI Observability Attribute Mapping: Test run', {});
 					setTestedAttributes(submittedAttributes);
 					setTestedResource(submittedResource);
 					setResult(response.data?.spans ?? []);

frontend/src/container/LLMObservability/AttributeMapping/hooks/useAttributeMappingEditor.ts

@@ -2,6 +2,7 @@ import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
 import { cloneDeep, isEqual } from 'lodash-es';
 import { toast } from '@signozhq/ui/sonner';
 import { useQueryClient } from 'react-query';
+import logEvent from 'api/common/logEvent';
 import {
 	useCreateSpanMapper,
 	useCreateSpanMapperGroup,
@@ -262,6 +263,7 @@ export function useAttributeMappingEditor(): AttributeMappingEditor {
 		setSaveError(null);
 		try {
 			await persistDraft(snapshot, draft, mutations);
+			void logEvent('AI Observability Attribute Mapping: Changes saved', {});
 			// Refresh the groups list in place — it stays mounted, so this just
 			// swaps in fresh data without a loading flash. Using the query's own
 			// refetch keeps it scoped to the groups list; the per-group mapper

frontend/src/container/LLMObservability/Explorer/Explorer.tsx

@@ -173,7 +173,7 @@ function Explorer(): JSX.Element {
 
 	useEffect(() => {
 		if (!logEventCalledRef.current) {
-			logEvent('Traces Explorer: Page visited', {});
+			logEvent('AI Observability Explorer: Page visited', {});
 			logEventCalledRef.current = true;
 		}
 	}, []);

frontend/src/container/LLMObservability/Explorer/ListView/ListView.tsx

@@ -173,7 +173,7 @@ function ListView({
 
 	useEffect(() => {
 		if (!isLoading && !isFetching && !isError && rows.length !== 0) {
-			void logEvent('Traces Explorer: Data present', {
+			void logEvent('AI Observability Explorer: Data present', {
 				panelType,
 			});
 		}

frontend/src/container/LLMObservability/Explorer/TracesView/TracesView.tsx

@@ -159,7 +159,7 @@ function TracesView({
 
 	useEffect(() => {
 		if (!isLoading && !isFetching && !isError && rows.length !== 0) {
-			void logEvent('Traces Explorer: Data present', {
+			void logEvent('AI Observability Explorer: Data present', {
 				panelType: 'TRACE',
 			});
 		}

frontend/src/container/LLMObservability/Overview/Overview.tsx

@@ -1,4 +1,6 @@
+import { useEffect } from 'react';
 import { Typography } from '@signozhq/ui/typography';
+import logEvent from 'api/common/logEvent';
 import Spinner from 'components/Spinner';
 import DashboardContainer from 'pages/DashboardPage/DashboardContainer';
 
@@ -8,6 +10,10 @@ import styles from './Overview.module.scss';
 function Overview(): JSX.Element {
 	const { dashboard, isLoading, isError, error, refetch } = useSystemDashboard();
 
+	useEffect(() => {
+		void logEvent('AI Observability Overview: Page visited', {});
+	}, []);
+
 	const renderContent = (): JSX.Element => {
 		if (isLoading) {
 			return <Spinner tip="Loading dashboard..." />;

frontend/src/container/LLMObservability/Settings/ModelPricing/LLMObservabilityModelPricing.tsx

@@ -1,5 +1,7 @@
+import { useEffect } from 'react';
 import { Badge } from '@signozhq/ui/badge';
 import { Tabs } from 'antd';
+import logEvent from 'api/common/logEvent';
 import { useListUnmappedLLMModels } from 'api/generated/services/llmpricingrules';
 import { parseAsStringEnum, useQueryState } from 'nuqs';
 
@@ -20,6 +22,10 @@ function LLMObservabilityModelPricing(): JSX.Element {
 	const { data } = useListUnmappedLLMModels();
 	const unpricedCount = data?.data?.items?.length ?? 0;
 
+	useEffect(() => {
+		void logEvent('AI Observability Model Pricing: Page visited', {});
+	}, []);
+
 	return (
 		<div
 			className={styles.llmObservabilityModelPricing}

frontend/src/container/LLMObservability/Settings/ModelPricing/ModelCostTabPanel/components/ModelCostDrawer/hooks/useModelCostDrawer.ts

@@ -1,6 +1,7 @@
 import { useCallback, useState } from 'react';
 import { toast } from '@signozhq/ui/sonner';
 import { useQueryClient } from 'react-query';
+import logEvent from 'api/common/logEvent';
 import {
 	getListLLMPricingRulesQueryKey,
 	getListUnmappedLLMModelsQueryKey,
@@ -94,6 +95,10 @@ export function useModelCostDrawer(): UseModelCostDrawerResult {
 				await createOrUpdate({
 					data: { rules: [buildRulePayload(draft)] },
 				});
+				void logEvent('AI Observability Model Pricing: Model cost saved', {
+					mode,
+					modelName: draft.modelName,
+				});
 				await invalidateList();
 				setIsOpen(false);
 				setSelectedRuleId(null);

frontend/src/container/LLMObservability/Settings/ModelPricing/ModelCostTabPanel/hooks/useModelCostDelete.ts

@@ -1,6 +1,7 @@
 import { useCallback, useState } from 'react';
 import { toast } from '@signozhq/ui/sonner';
 import { useQueryClient } from 'react-query';
+import logEvent from 'api/common/logEvent';
 import {
 	getListLLMPricingRulesQueryKey,
 	useDeleteLLMPricingRule,
@@ -46,6 +47,9 @@ export function useModelCostDelete(): UseModelCostDeleteResult {
 		}
 		try {
 			await deleteRuleApi({ pathParams: { id: pendingDelete.id } });
+			void logEvent('AI Observability Model Pricing: Model cost deleted', {
+				modelName: pendingDelete.modelName,
+			});
 			await queryClient.invalidateQueries({
 				queryKey: getListLLMPricingRulesQueryKey(),
 			});

frontend/src/container/LLMObservability/Settings/ModelPricing/UnpricedModelsTab/hooks/useUnpricedModelMapping.ts

@@ -1,6 +1,7 @@
 import { useCallback, useState } from 'react';
 import { toast } from '@signozhq/ui/sonner';
 import { useQueryClient } from 'react-query';
+import logEvent from 'api/common/logEvent';
 import {
 	getListLLMPricingRulesQueryKey,
 	getListUnmappedLLMModelsQueryKey,
@@ -45,6 +46,10 @@ export function useUnpricedModelMapping(): UseUnpricedModelMappingResult {
 			setIsSaving(true);
 			try {
 				await createOrUpdate({ data: { rules: [payload] } });
+				void logEvent('AI Observability Model Pricing: Unpriced model mapped', {
+					modelName: model.modelName,
+					billingModelName: rule.modelName,
+				});
 				await Promise.all([
 					queryClient.invalidateQueries({
 						queryKey: getListUnmappedLLMModelsQueryKey(),

frontend/src/container/SideNav/SideNav.tsx

@@ -47,7 +47,6 @@ import { useKeyboardHotkeys } from 'hooks/hotkeys/useKeyboardHotkeys';
 import useComponentPermission from 'hooks/useComponentPermission';
 import { useGetTenantLicense } from 'hooks/useGetTenantLicense';
 import { useIsAIAssistantEnabled } from 'hooks/useIsAIAssistantEnabled';
-import { useIsAIObservabilityEnabled } from 'hooks/useIsAIObservabilityEnabled';
 import { useNotifications } from 'hooks/useNotifications';
 import history from 'lib/history';
 import { isArray } from 'lodash-es';
@@ -255,7 +254,6 @@ function SideNav({ isPinned }: { isPinned: boolean }): JSX.Element {
 	const isAdmin = user.role === USER_ROLES.ADMIN;
 	const isEditor = user.role === USER_ROLES.EDITOR;
 	const isAIAssistantEnabled = useIsAIAssistantEnabled();
-	const isAIObservabilityEnabled = useIsAIObservabilityEnabled();
 	const aiAssistantActiveConversationId = useAIAssistantStore(
 		(s) => s.activeConversationId,
 	);
@@ -295,9 +293,6 @@ function SideNav({ isPinned }: { isPinned: boolean }): JSX.Element {
 			if (item.key === ROUTES.INTEGRATIONS) {
 				return shouldShowIntegrationsValue;
 			}
-			if (item.key === ROUTES.AI_OBSERVABILITY_OVERVIEW) {
-				return isAIObservabilityEnabled;
-			}
 			return item.isEnabled;
 		};
 
@@ -314,7 +309,6 @@ function SideNav({ isPinned }: { isPinned: boolean }): JSX.Element {
 		isEnterpriseSelfHostedUser,
 		isAdmin,
 		isEditor,
-		isAIObservabilityEnabled,
 	]);
 
 	// Track if we've done the initial sync (to avoid overwriting user actions during session)

frontend/src/container/SideNav/menuItems.tsx

@@ -293,9 +293,7 @@ export const defaultMoreMenuItems: SidebarItem[] = [
 		label: 'AI Observability',
 		icon: <Brain size={16} />,
 		isBeta: true,
-		// Gated behind the `enable_ai_observability` feature flag in
-		// SideNav's `computedSecondaryMenuItems`; disabled by default.
-		isEnabled: false,
+		isEnabled: true,
 		itemKey: 'ai-observability',
 	},
 	{

frontend/src/hooks/useIsAIObservabilityEnabled.ts

@@ -1,11 +0,0 @@
-import { FeatureKeys } from 'constants/features';
-import { useAppContext } from 'providers/App/App';
-
-export function useIsAIObservabilityEnabled(): boolean {
-	const { featureFlags } = useAppContext();
-	return (
-		featureFlags?.find(
-			(flag) => flag.name === FeatureKeys.ENABLE_AI_OBSERVABILITY,
-		)?.active || false
-	);
-}

frontend/src/lib/uPlotV2/components/Legend/constants.ts

@@ -10,6 +10,11 @@ export const MIN_LEGEND_ITEM_WIDTH = 110;
 /** Marker + row padding, on top of the estimated label width. */
 export const LEGEND_ITEM_EXTRA_WIDTH = 16;
 
+/** Must match `.gridList`'s column gap and `.scroller`'s padding-right, or the
+ * reserved row count disagrees with the grid that gets laid out. */
+export const LEGEND_COLUMN_GAP = 8;
+export const LEGEND_SCROLLER_PADDING_RIGHT = 4;
+
 /** Must match `.row`'s height and the grid's row gap, or the reserved
  * rectangle clips a row. */
 export const LEGEND_ROW_HEIGHT = 28;

frontend/src/lib/visualization/charts/__tests__/utils.test.ts

@@ -113,7 +113,7 @@ describe('calculateChartDimensions', () => {
 	});
 
 	it('BOTTOM: items one past a row still reserve two rows', () => {
-		// 1000px wide fits 5 of these per row, so 6 items need a second row.
+		// 1000px wide fits 4 of these per row, so 6 items need a second row.
 		const dims = calculateChartDimensions({
 			containerWidth: 1000,
 			containerHeight: 500,
@@ -123,6 +123,19 @@ describe('calculateChartDimensions', () => {
 		expect(dims.legendHeight).toBe(70);
 	});
 
+	it('BOTTOM: reserves the rows the grid actually lays out, not the rows a bare width estimate allows', () => {
+		// The item width alone suggests three fit on one row; the grid's per-item
+		// padding and column gap leave room for two.
+		const dims = calculateChartDimensions({
+			containerWidth: 412,
+			containerHeight: 310,
+			legendConfig: { position: LegendPosition.BOTTOM },
+			seriesLabels: ['P99', 'P95', 'P50'],
+		});
+		expect(dims.legendHeight).toBe(70);
+		expect(dims.height).toBe(240);
+	});
+
 	it('BOTTOM: drops to a single row rather than take half a short panel', () => {
 		const dims = calculateChartDimensions({
 			containerWidth: 1000,

frontend/src/lib/visualization/charts/utils.ts

@@ -1,8 +1,11 @@
 import {
 	LEGEND_MAX_BOTTOM_ROWS,
 	MIN_LEGEND_ITEM_WIDTH,
+	LEGEND_COLUMN_GAP,
+	LEGEND_ITEM_EXTRA_WIDTH,
 	LEGEND_ROW_GAP,
 	LEGEND_ROW_HEIGHT,
+	LEGEND_SCROLLER_PADDING_RIGHT,
 	MAX_LEGEND_WIDTH,
 } from 'lib/uPlotV2/components/Legend/constants';
 import { LegendConfig, LegendPosition } from 'lib/uPlotV2/components/types';
@@ -143,9 +146,16 @@ export function calculateChartDimensions({
 	const legendItemWidth = Math.ceil(
 		Math.min(approxLegendItemWidth, MAX_LEGEND_WIDTH),
 	);
+	// Must resolve to the same track count as `.gridList`'s `auto-fill`; a more
+	// generous one under-reserves rows and the grid's last row is clipped away.
+	const gridWidth =
+		containerWidth - LEGEND_PADDING * 2 - LEGEND_SCROLLER_PADDING_RIGHT;
 	const legendItemsPerRow = Math.max(
 		1,
-		Math.floor((containerWidth - LEGEND_PADDING * 2) / legendItemWidth),
+		Math.floor(
+			(gridWidth + LEGEND_COLUMN_GAP) /
+				(legendItemWidth + LEGEND_ITEM_EXTRA_WIDTH + LEGEND_COLUMN_GAP),
+		),
 	);
 
 	// The wrapper's bottom padding is inside this height (border-box).
@@ -163,8 +173,8 @@ export function calculateChartDimensions({
 	);
 
 	// Without this, short grid panels hand most of their area to the legend and
-	// the chart — the pie donut especially — collapses to a sliver. Dropping a
-	// whole row beats clipping one.
+	// the chart — the pie donut especially — collapses to a sliver. The dropped
+	// row's items are clipped rather than removed, so they are scroll-only here.
 	const legendRowCount =
 		neededRowCount > 1 &&
 		heightForRows(neededRowCount) > containerHeight * MAX_SHORT_PANEL_LEGEND_RATIO

frontend/src/pages/DashboardPage/DashboardContainer/VariablesBar/__tests__/ValueSelector.test.tsx

@@ -35,6 +35,12 @@ function renderSelector(
 	);
 }
 
+async function openDropdown(): Promise<void> {
+	const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime });
+	const control = screen.getByTestId('variable-select-env');
+	await user.click(control.querySelector('input') as HTMLInputElement);
+}
+
 /** Hovers an element and lets the tooltip's open delay elapse. */
 async function hover(element: HTMLElement): Promise<void> {
 	const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime });
@@ -112,17 +118,95 @@ describe('ValueSelector', () => {
 		});
 	});
 
+	describe('a dynamic variable', () => {
+		function renderDynamic(
+			complete: boolean,
+			relatedValues: string[],
+		): jest.Mock {
+			const onSearch = jest.fn();
+			render(
+				<TooltipProvider>
+					<ValueSelector
+						options={OPTIONS}
+						variableType="dynamic"
+						multiSelect
+						showAllOption
+						selection={{ value: [], allSelected: false }}
+						onChange={jest.fn()}
+						emptyFallback={{ value: [], allSelected: false }}
+						testId="variable-select-env"
+						dynamic={{
+							values: OPTIONS,
+							relatedValues,
+							complete,
+							onSearch,
+							onSearchReset: jest.fn(),
+						}}
+					/>
+				</TooltipProvider>,
+			);
+			return onSearch;
+		}
+
+		it('splits related values out of the full list', async () => {
+			renderDynamic(true, ['checkout-service-prod']);
+
+			await openDropdown();
+
+			expect(
+				screen.getByRole('heading', { level: 2, name: /Related Values/ }),
+			).toBeInTheDocument();
+			expect(
+				screen.getByRole('heading', { level: 2, name: /All Values/ }),
+			).toBeInTheDocument();
+		});
+
+		it('still opens its dropdown in single-select', async () => {
+			// The shared single select spreads unknown props over its own handlers, so
+			// passing it an `onDropdownVisibleChange` silently kills its open state.
+			render(
+				<TooltipProvider>
+					<ValueSelector
+						options={OPTIONS}
+						variableType="dynamic"
+						multiSelect={false}
+						showAllOption={false}
+						selection={{ value: '', allSelected: false }}
+						onChange={jest.fn()}
+						emptyFallback={{ value: '', allSelected: false }}
+						testId="variable-select-env"
+						dynamic={{
+							values: OPTIONS,
+							relatedValues: [],
+							complete: false,
+							onSearch: jest.fn(),
+							onSearchReset: jest.fn(),
+						}}
+					/>
+				</TooltipProvider>,
+			);
+
+			await openDropdown();
+
+			expect(screen.getByText('cart-service-prod')).toBeInTheDocument();
+		});
+
+		it('routes typing to the API search when the list is truncated', async () => {
+			const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime });
+			const onSearch = renderDynamic(false, []);
+
+			await openDropdown();
+			await user.keyboard('pay');
+
+			expect(onSearch).toHaveBeenLastCalledWith('pay');
+		});
+	});
+
 	describe('clearing', () => {
 		function clearIcon(): Element | null {
 			return document.querySelector('.ant-select-clear');
 		}
 
-		async function openDropdown(): Promise<void> {
-			const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime });
-			const control = screen.getByTestId('variable-select-env');
-			await user.click(control.querySelector('input') as HTMLInputElement);
-		}
-
 		it('offers no clear icon while the list is closed', () => {
 			renderSelector({ value: VALUES, allSelected: false }, OPTIONS);
 

frontend/src/pages/DashboardPage/DashboardContainer/VariablesBar/__tests__/useFetchedVariableOptions.test.tsx

@@ -114,4 +114,149 @@ describe('useFetchedVariableOptions', () => {
 
 		await waitFor(() => expect(result.current.options).toStrictEqual(['prod']));
 	});
+
+	it('keeps related values as their own section and as selectable options', async () => {
+		mockGetFieldValues.mockResolvedValue({
+			data: {
+				normalizedValues: ['cart', 'payments'],
+				relatedValues: ['checkout'],
+				complete: true,
+			},
+		});
+
+		useDashboardStore.setState({
+			variableFetchStates: { env: VariableFetchState.Loading },
+			variableCycleIds: { env: 1 },
+		});
+
+		const variable = dynamicVariable('env');
+		const { result } = renderHook(
+			() => useFetchedVariableOptions(variable, [variable], {}),
+			{ wrapper },
+		);
+
+		await waitFor(() =>
+			expect(result.current.dynamic?.relatedValues).toStrictEqual(['checkout']),
+		);
+		expect(result.current.dynamic?.values).toStrictEqual(['cart', 'payments']);
+		// A related value the unscoped list never returned is still selectable.
+		expect(result.current.options).toStrictEqual([
+			'cart',
+			'payments',
+			'checkout',
+		]);
+	});
+	it('sends the search to the API when the list is incomplete', async () => {
+		mockGetFieldValues.mockImplementation((_signal, _name, searchText) =>
+			Promise.resolve({
+				data: searchText
+					? { normalizedValues: ['payments'], relatedValues: [], complete: false }
+					: { normalizedValues: ['cart'], relatedValues: [], complete: false },
+			}),
+		);
+
+		useDashboardStore.setState({
+			variableFetchStates: { env: VariableFetchState.Loading },
+			variableCycleIds: { env: 1 },
+		});
+
+		const variable = dynamicVariable('env');
+		const { result } = renderHook(
+			() => useFetchedVariableOptions(variable, [variable], {}),
+			{ wrapper },
+		);
+
+		await waitFor(() =>
+			expect(result.current.dynamic?.values).toStrictEqual(['cart']),
+		);
+
+		act(() => {
+			result.current.dynamic?.onSearch('pay');
+		});
+
+		await waitFor(() =>
+			expect(result.current.dynamic?.values).toStrictEqual(['payments']),
+		);
+		expect(mockGetFieldValues).toHaveBeenCalledWith(
+			undefined,
+			'service.name',
+			'pay',
+			1_000,
+			2_000,
+			undefined,
+			expect.anything(),
+		);
+		// The search narrows the dropdown only — the selectable set is the full list,
+		// so a pick made before searching is never reconciled away.
+		expect(result.current.options).toStrictEqual(['cart']);
+
+		// Clearing falls straight back to the base fetch's options — synchronously, so
+		// closing the dropdown cannot leave the last search's results on screen for a
+		// debounce interval. They come from the cache of a separate query the search
+		// never touched, so nothing is refetched.
+		act(() => {
+			result.current.dynamic?.onSearchReset();
+		});
+
+		expect(result.current.dynamic?.values).toStrictEqual(['cart']);
+		expect(mockGetFieldValues).toHaveBeenCalledTimes(2);
+	});
+	it('marks a client error as not retryable', async () => {
+		mockGetFieldValues.mockRejectedValue(
+			Object.assign(new Error('bad request'), { response: { status: 400 } }),
+		);
+
+		useDashboardStore.setState({
+			variableFetchStates: { env: VariableFetchState.Loading },
+			variableCycleIds: { env: 1 },
+		});
+
+		const variable = dynamicVariable('env');
+		const { result } = renderHook(
+			() => useFetchedVariableOptions(variable, [variable], {}),
+			{ wrapper },
+		);
+
+		await waitFor(() => expect(result.current.isRetryable).toBe(false));
+	});
+	it('scopes the fetch by a sibling dynamic selection, skipping ALL', async () => {
+		mockGetFieldValues.mockResolvedValue(fieldValues(['cart']));
+
+		useDashboardStore.setState({
+			variableFetchStates: { env: VariableFetchState.Loading },
+			variableCycleIds: { env: 1 },
+		});
+
+		const env = dynamicVariable('env');
+		const namespace: VariableFormModel = {
+			...dynamicVariable('namespace'),
+			dynamicAttribute: 'k8s.namespace.name',
+		};
+		const region: VariableFormModel = {
+			...dynamicVariable('region'),
+			dynamicAttribute: 'cloud.region',
+		};
+
+		renderHook(
+			() =>
+				useFetchedVariableOptions(env, [env, namespace, region], {
+					namespace: { value: ['prod'], allSelected: false },
+					// ALL means "no filter", so it contributes nothing to existingQuery —
+					// which is why the backend returns no related values for it.
+					region: { value: null, allSelected: true },
+				}),
+			{ wrapper },
+		);
+
+		await waitFor(() =>
+			expect(mockGetFieldValues).toHaveBeenCalledWith(
+				undefined,
+				'service.name',
+				undefined,
+				1_000,
+				2_000,
+				"k8s.namespace.name = 'prod'",
+			),
+		);
+	});
 });

frontend/src/pages/DashboardPage/DashboardContainer/VariablesBar/components/selectors/ValueSelector.tsx

@@ -4,7 +4,9 @@ import { CustomMultiSelect, CustomSelect } from 'components/NewSelect';
 import type { OptionData } from 'components/NewSelect/types';
 import { DashboardDetailEvents } from 'pages/DashboardPage/constants/events';
 
+import type { DynamicVariableOptions } from '../../hooks/useFetchedVariableOptions';
 import type { VariableSelection } from '../../selectionTypes';
+import { dynamicVariableOptions } from '../../utils/dynamicVariableOptions';
 import { areSelectionsEqual } from '../../utils/resolveVariableSelection';
 import { selectionFromCommittedValues } from '../../utils/selectionUtils';
 import OverflowValuesTooltip from './OverflowValuesTooltip';
@@ -24,6 +26,10 @@ interface ValueSelectorProps {
 	/** Option-fetch error surfaced in the dropdown, with a retry action. */
 	errorMessage?: string | null;
 	onRetry?: () => void;
+	/** Hides the retry action for an error that retrying cannot fix. */
+	isRetryable?: boolean;
+	/** DYNAMIC only: sectioned rendering and server-side search. */
+	dynamic?: DynamicVariableOptions;
 }
 
 function ValueSelector({
@@ -38,10 +44,15 @@ function ValueSelector({
 	testId,
 	errorMessage,
 	onRetry,
+	isRetryable = true,
+	dynamic,
 }: ValueSelectorProps): JSX.Element {
 	const optionData = useMemo<OptionData[]>(
-		() => options.map((option) => ({ label: option, value: option })),
-		[options],
+		() =>
+			dynamic
+				? dynamicVariableOptions(dynamic.values, dynamic.relatedValues)
+				: options.map((option) => ({ label: option, value: option })),
+		[options, dynamic],
 	);
 
 	// All-selected → the full option set so CustomMultiSelect engages its "all"
@@ -119,6 +130,7 @@ function ValueSelector({
 				loading={loading}
 				errorMessage={errorMessage}
 				onRetry={onRetry}
+				showRetryButton={isRetryable}
 				showSearch
 				// Clearing belongs to the open list: on the closed control the icon would
 				// appear on hover, in a row of variable pills, for an action whose result is
@@ -136,6 +148,11 @@ function ValueSelector({
 				)}
 				// Offer ALL only once options load, else a concrete value reads as "all".
 				enableAllSelection={showAllOption && options.length > 0}
+				isDynamicVariable={!!dynamic}
+				onSearch={dynamic?.onSearch}
+				showIncompleteDataMessage={
+					!!dynamic && !dynamic.complete && dynamic.values.length > 0
+				}
 				onDropdownVisibleChange={(open): void => {
 					if (open) {
 						setDraft(committedValues);
@@ -144,6 +161,7 @@ function ValueSelector({
 					}
 
 					setIsOpen(false);
+					dynamic?.onSearchReset();
 					commit(draft);
 				}}
 				onChange={(next): void => {
@@ -180,8 +198,14 @@ function ValueSelector({
 			loading={loading}
 			errorMessage={errorMessage}
 			onRetry={onRetry}
+			showRetryButton={isRetryable}
 			showSearch
 			placeholder="Select value"
+			isDynamicVariable={!!dynamic}
+			onSearch={dynamic?.onSearch}
+			showIncompleteDataMessage={
+				!!dynamic && !dynamic.complete && dynamic.values.length > 0
+			}
 			onChange={(next): void => {
 				void logEvent(
 					DashboardDetailEvents.VariableValueSelected,

frontend/src/pages/DashboardPage/DashboardContainer/VariablesBar/components/selectors/VariableValueControl.tsx

@@ -42,11 +42,8 @@ function VariableValueControl({
 	onChange,
 	onAutoSelect,
 }: VariableValueControlProps): JSX.Element {
-	const { options, loading, errorMessage, onRetry } = useVariableOptions(
-		variable,
-		variables,
-		selections,
-	);
+	const { options, loading, errorMessage, onRetry, isRetryable, dynamic } =
+		useVariableOptions(variable, variables, selections);
 
 	useAutoSelect(variable, options, selection, onAutoSelect);
 
@@ -65,6 +62,8 @@ function VariableValueControl({
 			loading={loading}
 			errorMessage={errorMessage}
 			onRetry={onRetry}
+			isRetryable={isRetryable}
+			dynamic={dynamic}
 			selection={selection}
 			onChange={onChange}
 			emptyFallback={emptyFallback}

frontend/src/pages/DashboardPage/DashboardContainer/VariablesBar/hooks/useDynamicVariableSearch.ts

@@ -0,0 +1,89 @@
+import { useCallback, useState } from 'react';
+import { useQuery } from 'react-query';
+import { getFieldValues } from 'api/dynamicVariables/getFieldValues';
+import { DEBOUNCE_DELAY } from 'constants/queryBuilderFilterConfig';
+import useDebounce from 'hooks/useDebounce';
+
+interface UseDynamicVariableSearchProps {
+	signal?: 'traces' | 'logs' | 'metrics';
+	attribute?: string;
+	startUnixMilli: number;
+	endUnixMilli: number;
+	existingQuery?: string;
+	/** Only a truncated list needs the API — a complete one is filtered in the dropdown. */
+	enabled: boolean;
+}
+
+export interface DynamicVariableSearch {
+	/** Results while a server search is in effect, else null — render the base options. */
+	results: { values: string[]; relatedValues: string[] } | null;
+	isSearching: boolean;
+	onSearch: (text: string) => void;
+	reset: () => void;
+}
+
+/**
+ * Server-side value search for a DYNAMIC variable, deliberately kept off the fetch
+ * engine's own query: a keystroke must not settle the variable's fetch cycle and
+ * re-cascade its dependent variables and panels.
+ */
+export function useDynamicVariableSearch({
+	signal,
+	attribute,
+	startUnixMilli,
+	endUnixMilli,
+	existingQuery,
+	enabled,
+}: UseDynamicVariableSearchProps): DynamicVariableSearch {
+	const [searchText, setSearchText] = useState('');
+	const debouncedSearchText = useDebounce(searchText, DEBOUNCE_DELAY);
+	const isActive =
+		enabled && !!attribute && !!searchText && !!debouncedSearchText;
+
+	const { data, isFetching } = useQuery(
+		[
+			'dashboard-variable-dynamic-search',
+			signal,
+			attribute,
+			debouncedSearchText,
+			existingQuery,
+			startUnixMilli,
+			endUnixMilli,
+		],
+		({ signal: abortSignal }) =>
+			getFieldValues(
+				signal,
+				attribute,
+				debouncedSearchText,
+				startUnixMilli,
+				endUnixMilli,
+				existingQuery,
+				abortSignal,
+			),
+		{ enabled: isActive, refetchOnWindowFocus: false, keepPreviousData: true },
+	);
+
+	const reset = useCallback((): void => setSearchText(''), []);
+
+	// No results yet falls back to the base options rather than an empty dropdown:
+	// the select filters them locally, so the list narrows while the API answers.
+	const results = isActive ? data?.data : undefined;
+	if (!results) {
+		return {
+			results: null,
+			isSearching: isActive && isFetching,
+			onSearch: setSearchText,
+			reset,
+		};
+	}
+
+	return {
+		results: {
+			values: results.normalizedValues ?? [],
+			relatedValues: results.relatedValues ?? [],
+		},
+		isSearching: isFetching,
+		onSearch: setSearchText,
+		reset,
+	};
+}

frontend/src/pages/DashboardPage/DashboardContainer/VariablesBar/hooks/useFetchedVariableOptions.ts

@@ -9,6 +9,7 @@ import {
 	DASHBOARD_CACHE_TIME_ON_REFRESH_ENABLED,
 } from 'constants/queryCacheTime';
 import type { AppState } from 'store/reducers';
+import { isRetryableError } from 'utils/errorUtils';
 import type { GlobalReducer } from 'types/reducer/globalTime';
 
 import {
@@ -20,13 +21,29 @@ import { useDashboardStore } from '../../store/useDashboardStore';
 import { buildExistingDynamicVariableQuery } from '../utils/dynamicFilter';
 import type { VariableSelectionMap } from '../selectionTypes';
 import { selectionToPayload } from '../utils/selectionUtils';
+import { useDynamicVariableSearch } from './useDynamicVariableSearch';
 import { useVariableFetchState } from './useVariableFetchState';
 
+export interface DynamicVariableOptions {
+	/** ALL VALUES section — narrowed to the API's matches while a search is active. */
+	values: string[];
+	/** RELATED VALUES section — scoped by the sibling dynamic variables' selections. */
+	relatedValues: string[];
+	/** false when the backend truncated the list, so searching has to hit the API. */
+	complete: boolean;
+	onSearch: (text: string) => void;
+	onSearchReset: () => void;
+}
+
 export interface VariableOptions {
 	options: string[];
 	loading: boolean;
 	errorMessage: string | null;
 	onRetry?: () => void;
+	/** false for a client error, where retrying the same request cannot help. */
+	isRetryable?: boolean;
+	/** DYNAMIC only: what the dropdown renders, sectioned and search-aware. */
+	dynamic?: DynamicVariableOptions;
 }
 
 /**
@@ -150,10 +167,68 @@ export function useFetchedVariableOptions(
 		return sortValuesByOrder(values, variable.sort).map(String);
 	}, [dynamicResult.data, variable.sort]);
 
+	const dynamicRelatedOptions = useMemo(
+		() =>
+			sortValuesByOrder(
+				dynamicResult.data?.data?.relatedValues ?? [],
+				variable.sort,
+			).map(String),
+		[dynamicResult.data, variable.sort],
+	);
+
+	// Related values are scoped by the sibling selections, so they can name values the
+	// unscoped list never returned — the selectable set is the union of both sections.
+	const dynamicSelectableOptions = useMemo(
+		() => [...new Set([...dynamicOptions, ...dynamicRelatedOptions])],
+		[dynamicOptions, dynamicRelatedOptions],
+	);
+
+	const isDynamicListComplete = dynamicResult.data?.data?.complete ?? true;
+
+	const search = useDynamicVariableSearch({
+		signal: signalForApi(variable.dynamicSignal),
+		attribute: variable.dynamicAttribute,
+		startUnixMilli: minTime,
+		endUnixMilli: maxTime,
+		existingQuery: existingQuery || undefined,
+		enabled: variable.type === 'DYNAMIC' && !isDynamicListComplete,
+	});
+
+	// One stable object: the select rebuilds its whole option list whenever this
+	// identity changes, so it must not be a literal rebuilt on every render.
+	const dynamicDisplay = useMemo<DynamicVariableOptions>(() => {
+		const display = search.results
+			? {
+					values: sortValuesByOrder(search.results.values, variable.sort).map(
+						String,
+					),
+					relatedValues: sortValuesByOrder(
+						search.results.relatedValues,
+						variable.sort,
+					).map(String),
+				}
+			: { values: dynamicOptions, relatedValues: dynamicRelatedOptions };
+
+		return {
+			...display,
+			complete: isDynamicListComplete,
+			onSearch: search.onSearch,
+			onSearchReset: search.reset,
+		};
+	}, [
+		search.results,
+		search.onSearch,
+		search.reset,
+		isDynamicListComplete,
+		dynamicOptions,
+		dynamicRelatedOptions,
+		variable.sort,
+	]);
+
 	// Flag a variable that settled with zero options so dependent panels fall through
 	// to "no data" instead of waiting forever. hasFetchedOnce excludes the pre-fetch state.
 	const effectiveOptions =
-		variable.type === 'DYNAMIC' ? dynamicOptions : queryOptions;
+		variable.type === 'DYNAMIC' ? dynamicSelectableOptions : queryOptions;
 	useEffect(() => {
 		if (variable.type !== 'QUERY' && variable.type !== 'DYNAMIC') {
 			return;
@@ -175,14 +250,16 @@ export function useFetchedVariableOptions(
 
 	if (variable.type === 'DYNAMIC') {
 		return {
-			options: dynamicOptions,
-			loading: dynamicResult.isFetching || isVariableWaiting,
+			options: dynamicSelectableOptions,
+			loading: dynamicResult.isFetching || isVariableWaiting || search.isSearching,
 			errorMessage: dynamicResult.error
 				? (dynamicResult.error as Error).message || null
 				: null,
 			onRetry: (): void => {
 				void dynamicResult.refetch();
 			},
+			isRetryable: !dynamicResult.error || isRetryableError(dynamicResult.error),
+			dynamic: dynamicDisplay,
 		};
 	}
 	return {
@@ -194,5 +271,6 @@ export function useFetchedVariableOptions(
 		onRetry: (): void => {
 			void queryResult.refetch();
 		},
+		isRetryable: !queryResult.error || isRetryableError(queryResult.error),
 	};
 }

frontend/src/pages/DashboardPage/DashboardContainer/VariablesBar/utils/dynamicVariableOptions.ts

@@ -0,0 +1,27 @@
+import type { OptionData } from 'components/NewSelect/types';
+
+const toOptions = (values: string[]): OptionData[] =>
+	values.map((value) => ({ label: value, value }));
+
+/**
+ * Dropdown options for a DYNAMIC variable: values scoped by the other dynamic
+ * variables' selections get their own section above the unscoped list. Without
+ * related values there is nothing to contrast, so the list stays flat.
+ */
+export function dynamicVariableOptions(
+	values: string[],
+	relatedValues: string[],
+): OptionData[] {
+	if (relatedValues.length === 0) {
+		return toOptions(values);
+	}
+
+	return [
+		{
+			label: 'Related Values',
+			value: 'relatedValues',
+			options: toOptions(relatedValues),
+		},
+		{ label: 'All Values', value: 'allValues', options: toOptions(values) },
+	];
+}

frontend/src/pages/DashboardPage/DashboardContainer/hooks/__tests__/useGetQueryRangeV5.test.tsx

@@ -0,0 +1,113 @@
+import type { PropsWithChildren } from 'react';
+import { QueryClient, QueryClientProvider, UseQueryResult } from 'react-query';
+import { renderHook, RenderHookResult, waitFor } from '@testing-library/react';
+import { AxiosError, AxiosHeaders } from 'axios';
+import { queryRangeV5 } from 'api/generated/services/querier';
+import type {
+	Querybuildertypesv5QueryRangeRequestDTO,
+	QueryRangeV5200,
+} from 'api/generated/services/sigNoz.schemas';
+
+import { useGetQueryRangeV5 } from '../useGetQueryRangeV5';
+
+jest.mock('api/generated/services/querier', () => ({
+	queryRangeV5: jest.fn(),
+}));
+
+const mockQueryRangeV5 = queryRangeV5 as jest.Mock;
+
+const REQUEST = {} as Querybuildertypesv5QueryRangeRequestDTO;
+const KEY_A = ['query-range', 'panel-1', 'window-a'];
+const KEY_B = ['query-range', 'panel-1', 'window-b'];
+
+function clientError(): AxiosError {
+	return new AxiosError('bad query', 'ERR_BAD_REQUEST', undefined, undefined, {
+		status: 400,
+		statusText: 'Bad Request',
+		data: {},
+		headers: {},
+		config: { headers: new AxiosHeaders() },
+	});
+}
+
+interface Props {
+	enabled: boolean;
+	queryKey?: unknown[];
+}
+
+function renderQuery(
+	initial: Props,
+	client = new QueryClient(),
+): RenderHookResult<UseQueryResult<QueryRangeV5200, Error>, Props> {
+	const wrapper = ({ children }: PropsWithChildren): JSX.Element => (
+		<QueryClientProvider client={client}>{children}</QueryClientProvider>
+	);
+	return renderHook(
+		({ enabled, queryKey = KEY_A }: Props) =>
+			useGetQueryRangeV5({ requestPayload: REQUEST, queryKey, enabled }),
+		{ wrapper, initialProps: initial },
+	);
+}
+
+describe('useGetQueryRangeV5 enabled gating', () => {
+	beforeEach(() => {
+		mockQueryRangeV5.mockReset();
+	});
+
+	it('does not fetch while disabled and fetches once when enabled', async () => {
+		mockQueryRangeV5.mockResolvedValue({ status: 'success', data: {} });
+		const { rerender } = renderQuery({ enabled: false });
+		expect(mockQueryRangeV5).not.toHaveBeenCalled();
+
+		rerender({ enabled: true });
+		await waitFor(() => expect(mockQueryRangeV5).toHaveBeenCalledTimes(1));
+	});
+
+	it('serves a successful key from cache when re-enabled', async () => {
+		mockQueryRangeV5.mockResolvedValue({ status: 'success', data: {} });
+		const { result, rerender } = renderQuery({ enabled: true });
+		await waitFor(() => expect(result.current.isSuccess).toBe(true));
+
+		rerender({ enabled: false });
+		rerender({ enabled: true });
+		await waitFor(() => expect(result.current.isFetching).toBe(false));
+		expect(mockQueryRangeV5).toHaveBeenCalledTimes(1);
+	});
+
+	it('does not re-run an errored key when re-enabled', async () => {
+		mockQueryRangeV5.mockRejectedValue(clientError());
+		const { result, rerender } = renderQuery({ enabled: true });
+		await waitFor(() => expect(result.current.isError).toBe(true));
+
+		rerender({ enabled: false });
+		rerender({ enabled: true });
+		await waitFor(() => expect(result.current.isFetching).toBe(false));
+		expect(mockQueryRangeV5).toHaveBeenCalledTimes(1);
+		expect(result.current.isError).toBe(true);
+	});
+
+	it('still gates a new key while disabled after a prior key errored', async () => {
+		mockQueryRangeV5.mockRejectedValue(clientError());
+		const { result, rerender } = renderQuery({ enabled: true });
+		await waitFor(() => expect(result.current.isError).toBe(true));
+
+		rerender({ enabled: false });
+		rerender({ enabled: false, queryKey: KEY_B });
+		await waitFor(() => expect(result.current.isFetching).toBe(false));
+		expect(mockQueryRangeV5).toHaveBeenCalledTimes(1);
+
+		rerender({ enabled: true, queryKey: KEY_B });
+		await waitFor(() => expect(mockQueryRangeV5).toHaveBeenCalledTimes(2));
+	});
+
+	it('re-runs an errored key on manual refetch', async () => {
+		mockQueryRangeV5.mockRejectedValue(clientError());
+		const { result } = renderQuery({ enabled: true });
+		await waitFor(() => expect(result.current.isError).toBe(true));
+
+		mockQueryRangeV5.mockResolvedValue({ status: 'success', data: {} });
+		await result.current.refetch();
+		await waitFor(() => expect(result.current.isSuccess).toBe(true));
+		expect(mockQueryRangeV5).toHaveBeenCalledTimes(2);
+	});
+});

frontend/src/pages/DashboardPage/DashboardContainer/hooks/useGetQueryRangeV5.ts

@@ -1,4 +1,4 @@
-import { useQuery, UseQueryResult } from 'react-query';
+import { useQuery, useQueryClient, UseQueryResult } from 'react-query';
 import { isAxiosError } from 'axios';
 import { queryRangeV5 } from 'api/generated/services/querier';
 import type {
@@ -50,10 +50,15 @@ export function useGetQueryRangeV5({
 	keepPreviousData,
 	cacheTime,
 }: UseGetQueryRangeV5Args): UseQueryResult<QueryRangeV5200, Error> {
+	const queryClient = useQueryClient();
+	// An errored key has no data, so it is always stale; keep it enabled so
+	// re-enabling doesn't re-run it.
+	const hasErrored = queryClient.getQueryState(queryKey)?.status === 'error';
+
 	return useQuery<QueryRangeV5200, Error>({
 		queryKey,
 		queryFn: ({ signal }) => queryRangeV5(requestPayload, signal),
-		enabled,
+		enabled: enabled || hasErrored,
 		retry: retryUnlessClientError,
 		keepPreviousData,
 		cacheTime,

pkg/analytics/segmentanalytics/logger.go

@@ -2,6 +2,7 @@ package segmentanalytics
 
 import (
 	"context"
+	"fmt"
 
 	"github.com/SigNoz/signoz/pkg/factory"
 	segment "github.com/segmentio/analytics-go/v3"
@@ -18,11 +19,11 @@ func newSegmentLogger(settings factory.ScopedProviderSettings) segment.Logger {
 }
 
 func (logger *logger) Logf(format string, args ...interface{}) {
-	// the no lint directive is needed because the segmentlogger is not a slog.Logger
-	logger.settings.Logger().InfoContext(context.TODO(), format, args...) //nolint:sloglint
+	// the no lint directive is needed because the segment logger is not a slog.Logger
+	logger.settings.Logger().InfoContext(context.TODO(), fmt.Sprintf(format, args...)) //nolint:sloglint
 }
 
 func (logger *logger) Errorf(format string, args ...interface{}) {
 	// the no lint directive is needed because the segment logger is not a slog.Logger
-	logger.settings.Logger().ErrorContext(context.TODO(), format, args...) //nolint:sloglint
+	logger.settings.Logger().ErrorContext(context.TODO(), fmt.Sprintf(format, args...)) //nolint:sloglint
 }

pkg/flagger/registry.go

@@ -9,7 +9,6 @@ var (
 	FeaturePutMetersInZeus        = featuretypes.MustNewName("put_meters_in_zeus")
 	FeatureUseMeterReporter       = featuretypes.MustNewName("use_meter_reporter")
 	FeatureUseJSONBody            = featuretypes.MustNewName("use_json_body")
-	FeatureEnableAIObservability  = featuretypes.MustNewName("enable_ai_observability")
 	FeatureEnableMetricsReduction = featuretypes.MustNewName("enable_metrics_reduction")
 	FeatureResolveSemconvFamilies = featuretypes.MustNewName("resolve_semconv_families")
 )
@@ -64,14 +63,6 @@ func MustNewRegistry() featuretypes.Registry {
 			DefaultVariant: featuretypes.MustNewName("disabled"),
 			Variants:       featuretypes.NewBooleanVariants(),
 		},
-		&featuretypes.Feature{
-			Name:           FeatureEnableAIObservability,
-			Kind:           featuretypes.KindBoolean,
-			Stage:          featuretypes.StageExperimental,
-			Description:    "Controls whether ai observability is enabled",
-			DefaultVariant: featuretypes.MustNewName("disabled"),
-			Variants:       featuretypes.NewBooleanVariants(),
-		},
 		&featuretypes.Feature{
 			Name:           FeatureEnableMetricsReduction,
 			Kind:           featuretypes.KindBoolean,

pkg/query-service/app/http_handler.go

@@ -1478,15 +1478,6 @@ func (aH *APIHandler) getFeatureFlags(w http.ResponseWriter, r *http.Request) {
 		Route:      "",
 	})
 
-	aiObservability := aH.Signoz.Flagger.BooleanOrEmpty(r.Context(), flagger.FeatureEnableAIObservability, evalCtx)
-	featureSet = append(featureSet, &licensetypes.Feature{
-		Name:       valuer.NewString(flagger.FeatureEnableAIObservability.String()),
-		Active:     aiObservability,
-		Usage:      0,
-		UsageLimit: -1,
-		Route:      "",
-	})
-
 	aH.Respond(w, featureSet)
 }
 

pkg/tokenizer/opaquetokenizer/provider.go

@@ -364,12 +364,26 @@ func (provider *provider) gc(ctx context.Context, org *types.Organization) error
 }
 
 func (provider *provider) flushLastObservedAt(ctx context.Context, org *types.Organization) error {
-	accessTokenToLastObservedAt, err := provider.listLastObservedAtDesc(ctx, org.ID)
+	tokens, err := provider.tokenStore.ListByOrgID(ctx, org.ID)
 	if err != nil {
 		return err
 	}
 
-	if err := provider.tokenStore.UpdateLastObservedAtByAccessToken(ctx, accessTokenToLastObservedAt); err != nil {
+	observedTokens := make([]*authtypes.StorableToken, 0, len(tokens))
+	for _, token := range tokens {
+		cachedLastObservedAt, ok := provider.lastObservedAtCache.Get(lastObservedAtCacheKey(token.AccessToken, token.UserID))
+		if !ok {
+			continue
+		}
+
+		if err := token.UpdateLastObservedAt(cachedLastObservedAt); err != nil {
+			continue
+		}
+
+		observedTokens = append(observedTokens, token)
+	}
+
+	if err := provider.tokenStore.UpdateLastObservedAt(ctx, observedTokens); err != nil {
 		return err
 	}
 

pkg/tokenizer/tokenizerstore/sqltokenizerstore/store.go

@@ -232,15 +232,16 @@ func (store *store) ListByUserID(ctx context.Context, userID valuer.UUID) ([]*au
 	return tokens, nil
 }
 
-func (store *store) UpdateLastObservedAtByAccessToken(ctx context.Context, accessTokenToLastObservedAt []map[string]any) error {
-	if len(accessTokenToLastObservedAt) == 0 {
+func (store *store) UpdateLastObservedAt(ctx context.Context, tokens []*authtypes.StorableToken) error {
+	if len(tokens) == 0 {
 		return nil
 	}
 
 	values := store.
 		sqlstore.
 		BunDBCtx(ctx).
-		NewValues(&accessTokenToLastObservedAt)
+		NewValues(&tokens).
+		Column("id", "last_observed_at", "updated_at")
 
 	_, err := store.
 		sqlstore.
@@ -250,8 +251,8 @@ func (store *store) UpdateLastObservedAtByAccessToken(ctx context.Context, acces
 		Model((*authtypes.StorableToken)(nil)).
 		TableExpr("update_cte").
 		Set("last_observed_at = update_cte.last_observed_at").
-		Where("auth_token.access_token = update_cte.access_token").
-		Where("auth_token.user_id = update_cte.user_id").
+		Set("updated_at = update_cte.updated_at").
+		Where("auth_token.id = update_cte.id").
 		Exec(ctx)
 	if err != nil {
 		return err

pkg/tokenizer/tokenizerstore/sqltokenizerstore/store_test.go

@@ -0,0 +1,74 @@
+package sqltokenizerstore
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"github.com/DATA-DOG/go-sqlmock"
+	"github.com/SigNoz/signoz/pkg/sqlstore"
+	"github.com/SigNoz/signoz/pkg/sqlstore/sqlstoretest"
+	"github.com/SigNoz/signoz/pkg/types/authtypes"
+	"github.com/SigNoz/signoz/pkg/valuer"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestUpdateLastObservedAt(t *testing.T) {
+	testCases := []struct {
+		name          string
+		provider      string
+		tokens        []*authtypes.StorableToken
+		expectedQuery string
+	}{
+		{
+			name:          "Sqlite_Empty",
+			provider:      "sqlite",
+			tokens:        nil,
+			expectedQuery: "",
+		},
+		{
+			name:          "Postgres_Empty",
+			provider:      "postgres",
+			tokens:        []*authtypes.StorableToken{},
+			expectedQuery: "",
+		},
+		{
+			name:     "Sqlite_OneToken",
+			provider: "sqlite",
+			tokens: []*authtypes.StorableToken{
+				{ID: valuer.MustNewUUID("019984d1-0000-7000-8000-000000000001"), AccessToken: "access-one", RefreshToken: "refresh-one", LastObservedAt: time.Date(2026, 9, 22, 10, 0, 0, 0, time.UTC), UpdatedAt: time.Date(2026, 9, 22, 10, 0, 1, 0, time.UTC)},
+			},
+			expectedQuery: `WITH "update_cte" ("id", "last_observed_at", "updated_at") AS (VALUES ('019984d1-0000-7000-8000-000000000001', '2026-09-22 10:00:00+00:00', '2026-09-22 10:00:01+00:00')) UPDATE "auth_token" AS "auth_token" SET last_observed_at = update_cte.last_observed_at, updated_at = update_cte.updated_at FROM update_cte WHERE (auth_token.id = update_cte.id)`,
+		},
+		{
+			name:     "Postgres_TwoTokens",
+			provider: "postgres",
+			tokens: []*authtypes.StorableToken{
+				{ID: valuer.MustNewUUID("019984d1-0000-7000-8000-000000000002"), AccessToken: "access-two", RefreshToken: "refresh-two", LastObservedAt: time.Date(2026, 9, 22, 11, 0, 0, 0, time.UTC), UpdatedAt: time.Date(2026, 9, 22, 11, 0, 1, 0, time.UTC)},
+				{ID: valuer.MustNewUUID("019984d1-0000-7000-8000-000000000003"), AccessToken: "access-three", RefreshToken: "refresh-three", LastObservedAt: time.Date(2026, 9, 22, 12, 0, 0, 0, time.UTC), UpdatedAt: time.Date(2026, 9, 22, 12, 0, 1, 0, time.UTC)},
+			},
+			expectedQuery: `WITH "update_cte" ("id", "last_observed_at", "updated_at") AS (VALUES ('019984d1-0000-7000-8000-000000000002'::text, '2026-09-22 11:00:00+00:00'::TIMESTAMPTZ, '2026-09-22 11:00:01+00:00'::TIMESTAMPTZ), ('019984d1-0000-7000-8000-000000000003'::text, '2026-09-22 12:00:00+00:00'::TIMESTAMPTZ, '2026-09-22 12:00:01+00:00'::TIMESTAMPTZ)) UPDATE "auth_token" AS "auth_token" SET last_observed_at = update_cte.last_observed_at, updated_at = update_cte.updated_at FROM update_cte WHERE (auth_token.id = update_cte.id)`,
+		},
+	}
+
+	for _, testCase := range testCases {
+		t.Run(testCase.name, func(t *testing.T) {
+			var executedQuery string
+			matcher := sqlmock.QueryMatcherFunc(func(_, actual string) error {
+				executedQuery = actual
+				return nil
+			})
+
+			sqlStore := sqlstoretest.New(sqlstore.Config{Provider: testCase.provider}, matcher)
+			if testCase.expectedQuery != "" {
+				sqlStore.Mock().ExpectExec("").WillReturnResult(sqlmock.NewResult(0, int64(len(testCase.tokens))))
+			}
+
+			err := NewStore(sqlStore).UpdateLastObservedAt(context.Background(), testCase.tokens)
+			require.NoError(t, err)
+			require.NoError(t, sqlStore.Mock().ExpectationsWereMet())
+			assert.Equal(t, testCase.expectedQuery, executedQuery)
+		})
+	}
+}

pkg/types/authtypes/token.go

@@ -258,6 +258,6 @@ type TokenStore interface {
 	// Delete a token by userID.
 	DeleteByUserID(context.Context, valuer.UUID) error
 
-	// Update last observed at by access token.
-	UpdateLastObservedAtByAccessToken(context.Context, []map[string]any) error
+	// Update last observed at of the given tokens.
+	UpdateLastObservedAt(context.Context, []*StorableToken) error
 }

tests/e2e/tests/llm-o11y/attribute-mapping.spec.ts

@@ -9,6 +9,7 @@ import { authToken } from '../../helpers/common';
 test.describe.configure({ mode: 'serial' });
 
 const GROUP_NAME = 'e2e-attr-mapping-happy';
+const GROUP_CONDITION = 'my_company.llm.';
 
 const TARGET_ATTR = 'gen_ai.content.prompt';
 const SOURCE_ATTR = 'my_company.llm.input';
@@ -34,6 +35,8 @@ test.describe('LLM Observability — Attribute Mapping', () => {
 		const groupDrawer = page.getByTestId('group-form-drawer');
 		await expect(groupDrawer).toBeVisible();
 		await page.getByTestId('group-form-name').fill(GROUP_NAME);
+		await page.getByTestId('group-form-attribute-add').click();
+		await page.getByTestId('group-form-attribute-0').fill(GROUP_CONDITION);
 		await page.getByTestId('group-form-save').click();
 		await expect(groupDrawer).toBeHidden();
 

tests/integration/tests/passwordauthn/09_last_observed_at.py

@@ -0,0 +1,36 @@
+import time
+from collections.abc import Callable
+from http import HTTPStatus
+
+import requests
+from sqlalchemy import sql
+
+from fixtures import types
+from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
+
+
+def test_last_observed_at_is_flushed(signoz: types.SigNoz, get_token: Callable[[str, str], str]) -> None:
+    """Verify the tokenizer GC persists the cached last observed at of a used token to the sql store."""
+    token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
+
+    response = requests.get(
+        signoz.self.host_configs["8080"].get("/api/v2/users/me"),
+        headers={"Authorization": f"Bearer {token}"},
+        timeout=5,
+    )
+    assert response.status_code == HTTPStatus.OK
+
+    deadline = time.time() + 30
+    while time.time() < deadline:
+        with signoz.sqlstore.conn.connect() as conn:
+            row = conn.execute(
+                sql.text("SELECT last_observed_at FROM auth_token WHERE access_token = :access_token"),
+                {"access_token": token},
+            ).fetchone()
+
+        if row is not None and row[0] is not None:
+            return
+
+        time.sleep(1)
+
+    raise AssertionError("last_observed_at was not flushed to the sql store within 30s")

tests/integration/tests/passwordauthn/conftest.py

@@ -1,4 +1,3 @@
-# TODO: remove this file once enable_ai_observability flag is removed or defaulted to True
 import pytest
 from testcontainers.core.container import Network
 
@@ -7,7 +6,7 @@ from fixtures.signoz import create_signoz
 
 
 @pytest.fixture(name="signoz", scope="package")
-def signoz_e2e(  # pylint: disable=too-many-arguments,too-many-positional-arguments
+def signoz_passwordauthn(
     network: Network,
     zeus: types.TestContainerDocker,
     gateway: types.TestContainerDocker,
@@ -17,12 +16,7 @@ def signoz_e2e(  # pylint: disable=too-many-arguments,too-many-positional-argume
     pytestconfig: pytest.Config,
 ) -> types.SigNoz:
     """
-    E2E-scoped SigNoz override. Enables the experimental AI/LLM Observability
-    module (disabled by default in pkg/flagger/registry.go) so its routes render
-    instead of redirecting to /home — required by the llm-o11y e2e specs. Scoped
-    to the e2e package via this conftest so normal integration tests keep the
-    stock feature set. Follows the same pattern as
-    tests/integration/tests/metricreduction/conftest.py.
+    Package-scoped fixture for SigNoz with a short tokenizer GC interval so the last observed at flush runs within a test.
     """
     return create_signoz(
         network=network,
@@ -32,8 +26,8 @@ def signoz_e2e(  # pylint: disable=too-many-arguments,too-many-positional-argume
         clickhouse=clickhouse,
         request=request,
         pytestconfig=pytestconfig,
-        cache_key="signoz_e2e",
+        cache_key="signoz-passwordauthn",
         env_overrides={
-            "SIGNOZ_FLAGGER_CONFIG_BOOLEAN_ENABLE__AI__OBSERVABILITY": True,
+            "SIGNOZ_TOKENIZER_OPAQUE_GC_INTERVAL": "5s",
         },
     )