Compare commits

...

132 Commits

Author SHA1 Message Date
Naman Verma
215f6f8e0f fix: distinguish between no enabled and >1 enabled queries 2026-09-09 22:07:31 +05:30
Naman Verma
802f7e4a3d fix: more concise err message in promql no le case 2026-09-09 22:03:39 +05:30
Naman Verma
81f1105fbc fix: throw error if no le bucket if found in promql returned data 2026-09-09 21:56:43 +05:30
Naman Verma
575aa57426 Merge branch 'main' into nv/heatmap 2026-09-09 21:19:41 +05:30
Naman Verma
f75d3d8724 test: move rejection tests 2026-09-09 21:18:58 +05:30
Naman Verma
43538b92db fix: no value to be sent if values array is present 2026-09-09 19:57:51 +05:30
Naman Verma
ef65c2c08c chore: remove unneeded nil check 2026-09-09 19:46:47 +05:30
Naman Verma
8e41d3d30d Merge branch 'main' into nv/heatmap 2026-09-09 19:34:30 +05:30
Pandey
84a802edba docs(security): use GitHub private vulnerability reporting (#12820)
#### Description

- Switches the vulnerability reporting channel in `SECURITY.md` to
GitHub's private vulnerability reporting (Security → Report a
vulnerability), which is already enabled on this repo.
- Keeps `security@signoz.io` as a fallback for reporters who can't use
GitHub.
- Adds a short how-to and sets expectations for what happens after a
report (private advisory, coordinated fix, credit + CVE).
- Adds a `CODEOWNERS` entry so `SECURITY.md` is owned by @therealpandey.

#### Additional Information

- Docs / repo config only; no code changes.
2026-09-09 13:52:53 +00:00
Pandey
f78bd492d8 fix(tracefunnel): require view access on trace funnel analytics endpoints (#12817)
#### Description

- The twelve `/api/v1/trace-funnels/analytics/*` route registrations
were missing an authorization wrapper. The six payload routes
(`/analytics/*`) were reachable without authentication; the six
`/{funnel_id}/analytics/*` routes ran without the role check the
middleware applies.
- Wrap all twelve in `am.ViewAccess`, matching the sibling trace-funnel
CRUD routes, so they require an authenticated viewer.

#### Additional Information

- Applies to both community and enterprise editions
(`RegisterTraceFunnelsRoutes` is shared by both servers).
- No change for authenticated viewers; anonymous and role-less callers
now get 401/403.
- Security advisory:
https://github.com/SigNoz/signoz/security/advisories/GHSA-v549-7j2x-qjm5.
2026-09-09 12:39:46 +00:00
Gaurav Tewari
99dcd79979 chore(llm-observability): fork shared traces explorer files (#12795)
Some checks failed
build-staging / prepare (push) Has been cancelled
build-staging / js-build (push) Has been cancelled
build-staging / go-build (push) Has been cancelled
build-staging / staging (push) Has been cancelled
cacheci / tests (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
#### Description

- Copies, unmodified, the traces explorer files that the AI
observability explorer is about to diverge from. The point is the *next*
PR (#12796): with this baseline landed, that diff shows only the real
changes instead of ~1000 lines of brand-new files.
- No behaviour change. Nothing outside the fork imports the new copies
yet — the explorer's views are wired to them in #12796, alongside the
changes that need them.


Sources, all taken from this PR's merge base:

| Copied to `container/LLMObservability/` | Source |
| --- | --- |
| `ToolbarActions/LeftToolbarActions.tsx` |
`container/QueryBuilder/components/ToolbarActions/LeftToolbarActions.tsx`
|
| `ToolbarActions/ToolbarActions.styles.scss` |
`container/QueryBuilder/components/ToolbarActions/ToolbarActions.styles.scss`
|
| `Explorer/aiActions.ts` | `pages/TracesExplorer/aiActions.ts` |
| `Explorer/Controls/*` | `container/TracesExplorer/Controls/*` |
| `Explorer/TraceLoading/*` | `container/TracesExplorer/TraceLoading/*`
|
| `Explorer/TracesTable/*` | `container/TracesExplorer/TracesTable/*` |
| `Explorer/ListView/utils.tsx` |
`container/TracesExplorer/ListView/utils.tsx` |

#### Issues closed by this PR

#### Screenshots / Screen Recordings

#### Additional Information

---------

Co-authored-by: Gaurav Tewari <tewarig@users.noreply.github.com>
2026-09-09 09:40:13 +00:00
Naman Verma
b249ab8275 fix: dont build heatmap bucket options for missing metrics 2026-09-09 12:25:40 +05:30
Naman Verma
61bca035a3 chore: use increase as time aggregation for regular percentiles as well 2026-09-09 11:55:40 +05:30
Naman Verma
9983b3344b Merge branch 'nv/heatmap' of https://github.com/SigNoz/signoz into nv/heatmap 2026-09-09 10:13:05 +05:30
Naman Verma
1313a289b6 Merge branch 'main' into nv/heatmap 2026-09-07 23:01:08 +05:30
Naman Verma
82a789c19e test: fix pylint in integration test 2026-09-07 21:01:03 +05:30
Naman Verma
ec09873573 chore: push temporary poc for testing 2026-09-07 21:01:03 +05:30
Naman Verma
00cc773792 test: add integration tests 2026-09-07 21:01:03 +05:30
Naman Verma
cf5dd79216 fix: return unsupported for logs and traces 2026-09-07 21:01:03 +05:30
Naman Verma
6546a0abc6 test: delete heatmap UTs (to be covered in integration tests) 2026-09-07 21:01:03 +05:30
Naman Verma
3874f3a6ec test: delete heatmap validation UTs (to be covered in integration tests) 2026-09-07 21:01:03 +05:30
Naman Verma
86db0fd26b chore: var renames 2026-09-07 21:01:03 +05:30
Naman Verma
f382a958db chore: code refactoring of bucketFormulaOutputAsHeatmap 2026-09-07 21:01:03 +05:30
Naman Verma
7c3481b230 chore: comments, code structure changes and renamings in resolveHeatmapBucketAxis 2026-09-07 21:01:03 +05:30
Naman Verma
24f3116527 chore: map var renames 2026-09-07 21:01:03 +05:30
Naman Verma
17f0f49398 chore: some code movement and clarity around ReindexValuesToNewUpperBounds 2026-09-07 21:01:03 +05:30
Naman Verma
f681bacdcb chore: method rename 2026-09-07 21:01:03 +05:30
Naman Verma
ded4ed3d76 chore: add comment for logs and traces supporting only 1 aggregation 2026-09-07 21:01:03 +05:30
Naman Verma
1f84f9c166 chore: remove comment 2026-09-07 21:01:03 +05:30
Naman Verma
87127b580c chore: upperbound instead of boundary 2026-09-07 21:01:03 +05:30
Naman Verma
fc4bd13fce test: remove unit tests that can be covered in integration tests 2026-09-07 21:01:03 +05:30
Naman Verma
075a4b164d chore: revert UT change 2026-09-07 21:01:03 +05:30
Naman Verma
2170189d14 chore: shorten comments 2026-09-07 21:01:03 +05:30
Naman Verma
b7720b655e chore: shorten comments 2026-09-07 21:01:03 +05:30
Naman Verma
f34973b897 chore: use the term upper bound instead of boundary 2026-09-07 21:01:03 +05:30
Naman Verma
d9d041a8fa fix: use separate typedef for cumulative column 2026-09-07 21:01:03 +05:30
Naman Verma
a0e66e3592 fix: return empty instead of err if promql resp does not have le 2026-09-07 21:01:03 +05:30
Naman Verma
a0dff26fbc test: remove unit tests that can be covered in integration tests 2026-09-07 21:01:03 +05:30
Naman Verma
db0b1b2b2a chore: remove capacity vars for an easier read 2026-09-07 21:01:03 +05:30
Naman Verma
fd8ed18592 chore: better var name 2026-09-07 21:01:03 +05:30
Naman Verma
392b5826fd chore: better comment 2026-09-07 21:01:03 +05:30
Naman Verma
4b4023d706 chore: move HeatmapBucketColumn 2026-09-07 21:01:03 +05:30
Naman Verma
f8d803d787 fix: allow other panel types in heatmap 2026-09-07 21:01:03 +05:30
Naman Verma
a33320c665 test: add unit tests for cumulative metrics 2026-09-07 21:01:03 +05:30
Naman Verma
695a481989 fix: use heatmapBucketing.LogScale correctly 2026-09-07 21:01:03 +05:30
Naman Verma
3fc763b95b chore: break down method for easier reading 2026-09-07 21:01:03 +05:30
Naman Verma
027b65574d chore: shorten comments 2026-09-07 21:01:03 +05:30
Naman Verma
43ec98b309 chore: rename method name 2026-09-07 21:01:03 +05:30
Naman Verma
a196a436be chore: code movement 2026-09-07 21:01:03 +05:30
Naman Verma
3f456b6602 chore: add separate file for heatmap accumulator 2026-09-07 21:01:03 +05:30
Naman Verma
f777e068c3 chore: reduce comment size 2026-09-07 21:01:03 +05:30
Naman Verma
5a690695e6 chore: split toResult method into two 2026-09-07 21:01:03 +05:30
Naman Verma
bf362d15db chore: shorten comment 2026-09-07 21:01:03 +05:30
Naman Verma
7e0f70ffa7 chore: remove unneeded comment 2026-09-07 21:01:03 +05:30
Naman Verma
590046adf3 fix: add heatmap to hasData check 2026-09-07 21:01:03 +05:30
Naman Verma
106cb9f6c2 chore: shorten comment 2026-09-07 21:01:03 +05:30
Naman Verma
f95699b1f7 chore: method name change 2026-09-07 21:01:03 +05:30
Naman Verma
1e75b66379 chore: method name change 2026-09-07 21:01:03 +05:30
Naman Verma
b80013d075 fix: return 501 for exp histogram instead of 400 2026-09-07 21:01:03 +05:30
Naman Verma
4a6e25708d chore: remove unneeded comments 2026-09-07 21:01:03 +05:30
Naman Verma
3b8f9a7830 chore: move MaxNumBuckets const to where it is actually used 2026-09-07 21:01:03 +05:30
Naman Verma
68bff5b053 chore: remove unneeded comments 2026-09-07 21:01:03 +05:30
Naman Verma
732c8ae18a chore: remove unneeded comments 2026-09-07 21:01:03 +05:30
Naman Verma
6911630c2a fix: dont allow bucket options in non heatmap requests 2026-09-07 21:01:03 +05:30
Naman Verma
2d78d8add9 chore: move valid bucket kinds to additional part of err 2026-09-07 21:01:03 +05:30
Naman Verma
8e47c7d337 chore: move consts to where they are actually used 2026-09-07 21:01:03 +05:30
Naman Verma
991e734c33 chore: minor code movement 2026-09-07 21:01:03 +05:30
Naman Verma
40c881b8f5 chore: remove unneeded comment 2026-09-07 21:01:03 +05:30
Naman Verma
a2c8dd5275 chore: remove unneeded comment 2026-09-07 21:01:03 +05:30
Naman Verma
32ae7a26d5 chore: shorten comment 2026-09-07 21:01:03 +05:30
Naman Verma
4067b04fa2 fix: update dashboard schema to latest spec 2026-09-07 21:01:03 +05:30
Naman Verma
70409f5ee3 fix: make ResolveHeatmapBucketing a method on MetricAggregation 2026-09-07 21:01:03 +05:30
Naman Verma
5d9953bb7e feat: add heatmap support in query and dashboards 2026-09-07 21:01:03 +05:30
Naman Verma
edfff0ee72 test: fix pylint in integration test 2026-09-07 16:42:23 +05:30
Naman Verma
44556060a3 chore: push temporary poc for testing 2026-09-07 15:54:09 +05:30
Naman Verma
545e0342d2 test: add integration tests 2026-09-07 14:57:47 +05:30
Naman Verma
2f559f34d3 fix: return unsupported for logs and traces 2026-09-06 12:14:39 +05:30
Naman Verma
e4d90b91b7 test: delete heatmap UTs (to be covered in integration tests) 2026-09-06 05:07:18 +05:30
Naman Verma
f33616a598 test: delete heatmap validation UTs (to be covered in integration tests) 2026-09-06 04:51:05 +05:30
Naman Verma
2ac94fc2fc chore: var renames 2026-09-06 04:46:39 +05:30
Naman Verma
7292a6946b chore: code refactoring of bucketFormulaOutputAsHeatmap 2026-09-06 04:32:54 +05:30
Naman Verma
dff49a4135 chore: comments, code structure changes and renamings in resolveHeatmapBucketAxis 2026-09-06 04:09:09 +05:30
Naman Verma
690c207519 chore: map var renames 2026-09-06 03:19:59 +05:30
Naman Verma
d9b6d1f425 chore: some code movement and clarity around ReindexValuesToNewUpperBounds 2026-09-06 03:18:01 +05:30
Naman Verma
1eb974d91a chore: method rename 2026-09-06 02:55:54 +05:30
Naman Verma
1141f6bf14 chore: add comment for logs and traces supporting only 1 aggregation 2026-09-06 02:50:12 +05:30
Naman Verma
a3fb926726 chore: remove comment 2026-09-06 02:25:27 +05:30
Naman Verma
fe1e86f385 chore: upperbound instead of boundary 2026-09-06 02:12:39 +05:30
Naman Verma
e30d9a1a14 test: remove unit tests that can be covered in integration tests 2026-09-06 01:51:03 +05:30
Naman Verma
c9a345713c chore: revert UT change 2026-09-06 01:49:52 +05:30
Naman Verma
22f8eba44a chore: shorten comments 2026-09-06 01:41:53 +05:30
Naman Verma
e8ad9044d5 chore: shorten comments 2026-09-06 01:40:08 +05:30
Naman Verma
576cd5052a chore: use the term upper bound instead of boundary 2026-09-06 01:37:33 +05:30
Naman Verma
4214ccf93d fix: use separate typedef for cumulative column 2026-09-06 01:34:44 +05:30
Naman Verma
63f5c17238 fix: return empty instead of err if promql resp does not have le 2026-09-06 01:25:53 +05:30
Naman Verma
fd6abd67dd test: remove unit tests that can be covered in integration tests 2026-09-05 16:25:23 +05:30
Naman Verma
a78c35d61c chore: remove capacity vars for an easier read 2026-09-05 16:19:46 +05:30
Naman Verma
b2c4916a86 chore: better var name 2026-09-05 16:08:02 +05:30
Naman Verma
feff1a0179 chore: better comment 2026-09-05 15:53:47 +05:30
Naman Verma
eafac1be69 chore: move HeatmapBucketColumn 2026-09-05 15:35:14 +05:30
Naman Verma
510d6985d0 fix: allow other panel types in heatmap 2026-09-05 15:23:15 +05:30
Naman Verma
758aacb065 test: add unit tests for cumulative metrics 2026-09-05 15:13:11 +05:30
Naman Verma
5599c352d0 fix: use heatmapBucketing.LogScale correctly 2026-09-05 14:29:47 +05:30
Naman Verma
96d0a2340c chore: break down method for easier reading 2026-09-05 14:19:11 +05:30
Naman Verma
c6b415ad04 chore: shorten comments 2026-09-05 14:14:39 +05:30
Naman Verma
9b922f297f chore: rename method name 2026-09-05 13:53:08 +05:30
Naman Verma
e81a4c4046 chore: code movement 2026-09-05 13:44:11 +05:30
Naman Verma
9ba13848ea chore: add separate file for heatmap accumulator 2026-09-05 13:16:18 +05:30
Naman Verma
2578f25dd4 chore: reduce comment size 2026-09-05 12:46:35 +05:30
Naman Verma
82352973bb chore: split toResult method into two 2026-09-05 12:36:47 +05:30
Naman Verma
3eb55692f1 chore: shorten comment 2026-09-05 04:04:45 +05:30
Naman Verma
61ceb6d300 chore: remove unneeded comment 2026-09-05 04:03:43 +05:30
Naman Verma
e33fc20579 fix: add heatmap to hasData check 2026-09-05 03:50:44 +05:30
Naman Verma
b130a61049 chore: shorten comment 2026-09-05 03:25:14 +05:30
Naman Verma
132e9bb740 chore: method name change 2026-09-05 03:15:13 +05:30
Naman Verma
9d577731e1 chore: method name change 2026-09-05 03:07:14 +05:30
Naman Verma
4d54568fe1 fix: return 501 for exp histogram instead of 400 2026-09-05 03:04:41 +05:30
Naman Verma
6f82624523 chore: remove unneeded comments 2026-09-05 02:53:15 +05:30
Naman Verma
da0bc4f438 chore: move MaxNumBuckets const to where it is actually used 2026-09-05 02:51:29 +05:30
Naman Verma
2dcb764632 chore: remove unneeded comments 2026-09-05 02:50:27 +05:30
Naman Verma
1bc25e16b8 chore: remove unneeded comments 2026-09-05 02:50:03 +05:30
Naman Verma
eebc6b400d fix: dont allow bucket options in non heatmap requests 2026-09-05 02:49:41 +05:30
Naman Verma
30c4508113 chore: move valid bucket kinds to additional part of err 2026-09-05 02:38:00 +05:30
Naman Verma
c4fe619920 chore: move consts to where they are actually used 2026-09-05 02:33:59 +05:30
Naman Verma
d597adddda chore: minor code movement 2026-09-05 02:31:10 +05:30
Naman Verma
9a05f33e7a chore: remove unneeded comment 2026-09-05 02:25:48 +05:30
Naman Verma
3812937c6f chore: remove unneeded comment 2026-09-05 02:25:29 +05:30
Naman Verma
34aa719cba chore: shorten comment 2026-09-05 02:24:15 +05:30
Naman Verma
cdfce4c4ab fix: update dashboard schema to latest spec 2026-09-05 02:12:34 +05:30
Naman Verma
d8ef672f83 Merge branch 'main' into nv/heatmap 2026-09-05 02:00:57 +05:30
Naman Verma
00efffc127 fix: make ResolveHeatmapBucketing a method on MetricAggregation 2026-09-03 13:50:09 +05:30
Naman Verma
dec922a83f feat: add heatmap support in query and dashboards 2026-09-03 12:07:27 +05:30
56 changed files with 6276 additions and 164 deletions

4
.github/CODEOWNERS vendored
View File

@@ -15,6 +15,10 @@
.github @therealpandey
go.mod @therealpandey
# Security
/SECURITY.md @therealpandey
# Scaffold Owners
/pkg/config/ @therealpandey

View File

@@ -1,17 +1,26 @@
# Security Policy
SigNoz is looking forward to working with security researchers across the world to keep SigNoz and our users safe. If you have found an issue in our systems/applications, please reach out to us.
SigNoz is looking forward to working with security researchers across the world to keep SigNoz and our users safe. If you have found an issue in our systems/applications, please report it to us privately.
## Supported Versions
We always recommend using the latest version of SigNoz to ensure you get all security updates
We always recommend using the latest version of SigNoz to ensure you get all security updates.
## Reporting a Vulnerability
If you believe you have found a security vulnerability within SigNoz, please let us know right away. We'll try and fix the problem as soon as possible.
**Do not report vulnerabilities using public GitHub issues**. Instead, email <security@signoz.io> with a detailed account of the issue. Please submit one issue per email, this helps us triage vulnerabilities.
**Do not report vulnerabilities using public GitHub issues, discussions, or pull requests.**
Once we've received your email we'll keep you updated as we fix the vulnerability.
Instead, report it privately through GitHub's private vulnerability reporting:
1. Go to the [**Security** tab](https://github.com/SigNoz/signoz/security) of this repository.
2. Click **Report a vulnerability**, or use [this link](https://github.com/SigNoz/signoz/security/advisories/new).
3. Describe the issue with as much detail as you can — affected version, impact, and steps to reproduce help us triage faster. Please submit one report per vulnerability.
This opens a private advisory visible only to you and the SigNoz maintainers. We'll respond there, keep you updated as we work on a fix, and coordinate disclosure. If the report is valid we'll credit you on the published advisory and request a CVE.
If you're unable to use GitHub's private reporting, you can email <security@signoz.io> instead.
## Thanks

View File

@@ -3474,6 +3474,79 @@ components:
- tags
- spec
type: object
DashboardtypesHeatmapAxes:
properties:
yScale:
$ref: '#/components/schemas/DashboardtypesHeatmapYScale'
type: object
DashboardtypesHeatmapChartAppearance:
properties:
colors:
$ref: '#/components/schemas/DashboardtypesHeatmapColors'
type: object
DashboardtypesHeatmapColorMode:
enum:
- palette
- opacity
type: string
DashboardtypesHeatmapColorScale:
enum:
- log
- sqrt
- linear
type: string
DashboardtypesHeatmapColors:
properties:
fill:
type: string
maxCount:
nullable: true
type: number
minCount:
nullable: true
type: number
mode:
$ref: '#/components/schemas/DashboardtypesHeatmapColorMode'
palette:
$ref: '#/components/schemas/DashboardtypesHeatmapPalette'
scale:
$ref: '#/components/schemas/DashboardtypesHeatmapColorScale'
steps:
type: integer
type: object
DashboardtypesHeatmapPalette:
enum:
- ice
- moss
- rust
- graphite
- ember
- lagoon
- orchid
- verdant
- lava
- beacon
type: string
DashboardtypesHeatmapPanelSpec:
properties:
axes:
$ref: '#/components/schemas/DashboardtypesHeatmapAxes'
chartAppearance:
$ref: '#/components/schemas/DashboardtypesHeatmapChartAppearance'
formatting:
$ref: '#/components/schemas/DashboardtypesPanelFormatting'
legend:
$ref: '#/components/schemas/DashboardtypesLegend'
visualization:
$ref: '#/components/schemas/DashboardtypesBasicVisualization'
type: object
DashboardtypesHeatmapYScale:
enum:
- auto
- linear
- log
- symlog
type: string
DashboardtypesHistogramBuckets:
properties:
bucketCount:
@@ -3826,6 +3899,7 @@ components:
discriminator:
mapping:
signoz/BarChartPanel: '#/components/schemas/DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesBarChartPanelSpec'
signoz/HeatmapPanel: '#/components/schemas/DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesHeatmapPanelSpec'
signoz/HistogramPanel: '#/components/schemas/DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesHistogramPanelSpec'
signoz/ListPanel: '#/components/schemas/DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesListPanelSpec'
signoz/NumberPanel: '#/components/schemas/DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesNumberPanelSpec'
@@ -3841,6 +3915,7 @@ components:
- $ref: '#/components/schemas/DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesTablePanelSpec'
- $ref: '#/components/schemas/DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesHistogramPanelSpec'
- $ref: '#/components/schemas/DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesListPanelSpec'
- $ref: '#/components/schemas/DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesHeatmapPanelSpec'
type: object
DashboardtypesPanelPluginKind:
enum:
@@ -3851,6 +3926,7 @@ components:
- signoz/TablePanel
- signoz/HistogramPanel
- signoz/ListPanel
- signoz/HeatmapPanel
type: string
DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesBarChartPanelSpec:
properties:
@@ -3864,6 +3940,18 @@ components:
- kind
- spec
type: object
DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesHeatmapPanelSpec:
properties:
kind:
enum:
- signoz/HeatmapPanel
type: string
spec:
$ref: '#/components/schemas/DashboardtypesHeatmapPanelSpec'
required:
- kind
- spec
type: object
DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesHistogramPanelSpec:
properties:
kind:
@@ -7395,10 +7483,7 @@ components:
$ref: '#/components/schemas/Querybuildertypesv5TimeSeries'
type: array
meta:
properties:
unit:
type: string
type: object
$ref: '#/components/schemas/Querybuildertypesv5AggregationMeta'
predictedSeries:
items:
$ref: '#/components/schemas/Querybuildertypesv5TimeSeries'
@@ -7413,12 +7498,51 @@ components:
$ref: '#/components/schemas/Querybuildertypesv5TimeSeries'
type: array
type: object
Querybuildertypesv5Bucket:
Querybuildertypesv5AggregationMeta:
properties:
step:
format: double
type: number
buckets:
items:
format: double
type: number
type: array
unit:
type: string
type: object
Querybuildertypesv5BucketOptions:
discriminator:
mapping:
linear: '#/components/schemas/Querybuildertypesv5BucketOptionsLinear'
log: '#/components/schemas/Querybuildertypesv5BucketOptionsLog'
propertyName: kind
oneOf:
- $ref: '#/components/schemas/Querybuildertypesv5BucketOptionsLinear'
- $ref: '#/components/schemas/Querybuildertypesv5BucketOptionsLog'
type: object
Querybuildertypesv5BucketOptionsLinear:
properties:
kind:
$ref: '#/components/schemas/Querybuildertypesv5BucketsKind'
spec:
$ref: '#/components/schemas/Querybuildertypesv5LinearBucketsSpec'
required:
- kind
- spec
type: object
Querybuildertypesv5BucketOptionsLog:
properties:
kind:
$ref: '#/components/schemas/Querybuildertypesv5BucketsKind'
spec:
$ref: '#/components/schemas/Querybuildertypesv5LogBucketsSpec'
required:
- kind
- spec
type: object
Querybuildertypesv5BucketsKind:
enum:
- linear
- log
type: string
Querybuildertypesv5BuilderQuerySpec:
discriminator:
mapping:
@@ -7599,6 +7723,16 @@ components:
value:
type: string
type: object
Querybuildertypesv5LinearBucketsSpec:
properties:
maxValue:
format: double
type: number
numBuckets:
type: integer
required:
- maxValue
type: object
Querybuildertypesv5LogAggregation:
properties:
alias:
@@ -7606,6 +7740,12 @@ components:
expression:
type: string
type: object
Querybuildertypesv5LogBucketsSpec:
properties:
scale:
nullable: true
type: integer
type: object
Querybuildertypesv5MetricAggregation:
properties:
comparisonSpaceAggregationParam:
@@ -8064,6 +8204,8 @@ components:
queries (traces, logs, metrics), formulas, joins, trace operators, PromQL,
and ClickHouse SQL queries.
properties:
bucketOptions:
$ref: '#/components/schemas/Querybuildertypesv5BucketOptions'
compositeQuery:
$ref: '#/components/schemas/Querybuildertypesv5CompositeQuery'
end:
@@ -8163,6 +8305,7 @@ components:
- raw
- raw_stream
- trace
- heatmap
type: string
Querybuildertypesv5ScalarData:
properties:
@@ -8237,8 +8380,6 @@ components:
type: object
Querybuildertypesv5TimeSeriesValue:
properties:
bucket:
$ref: '#/components/schemas/Querybuildertypesv5Bucket'
partial:
type: boolean
timestamp:

View File

@@ -0,0 +1,14 @@
.container {
display: flex;
align-items: center;
justify-content: flex-end;
gap: 0.3rem;
margin: 8px 0;
}
.optionsTrigger {
display: flex;
align-items: center;
gap: 4px;
cursor: pointer;
}

View File

@@ -0,0 +1,82 @@
import { memo, useState } from 'react';
import { useTranslation } from 'react-i18next';
import { Settings } from '@signozhq/icons';
import FieldsSelector from 'components/FieldsSelector';
import Controls, { ControlsProps } from 'container/Controls';
import { OptionsMenuConfig } from 'container/OptionsMenu/types';
import useQueryPagination from 'hooks/queryPagination/useQueryPagination';
import { DataSource } from 'types/common/queryBuilder';
import styles from './Controls.module.scss';
function TraceExplorerControls({
isLoading,
totalCount,
perPageOptions,
config,
showSizeChanger = true,
}: TraceExplorerControlsProps): JSX.Element | null {
const { t } = useTranslation(['trace']);
const [isFieldsSelectorOpen, setIsFieldsSelectorOpen] = useState(false);
const {
pagination,
handleCountItemsPerPageChange,
handleNavigateNext,
handleNavigatePrevious,
} = useQueryPagination(totalCount, perPageOptions);
return (
<div className={styles.container}>
{config?.fieldsSelector && (
<>
<div
className={styles.optionsTrigger}
onClick={(): void => setIsFieldsSelectorOpen(true)}
>
{t('options_menu.options')}
<Settings size="md" />
</div>
<FieldsSelector
isOpen={isFieldsSelectorOpen}
title="Edit columns"
fields={config.fieldsSelector.value}
onFieldsChange={config.fieldsSelector.onFieldsChange}
onClose={(): void => setIsFieldsSelectorOpen(false)}
signal={DataSource.TRACES}
/>
</>
)}
<Controls
isLoading={isLoading}
totalCount={totalCount}
offset={pagination.offset}
countPerPage={pagination.limit}
perPageOptions={perPageOptions}
handleCountItemsPerPageChange={handleCountItemsPerPageChange}
handleNavigateNext={handleNavigateNext}
handleNavigatePrevious={handleNavigatePrevious}
showSizeChanger={showSizeChanger}
/>
</div>
);
}
TraceExplorerControls.defaultProps = {
config: null,
};
type TraceExplorerControlsProps = Pick<
ControlsProps,
'isLoading' | 'totalCount' | 'perPageOptions'
> & {
config?: OptionsMenuConfig | null;
showSizeChanger?: boolean;
};
TraceExplorerControls.defaultProps = {
showSizeChanger: true,
};
export default memo(TraceExplorerControls);

View File

@@ -0,0 +1,168 @@
import { Link } from 'react-router-dom';
import type { TableColumnsType as ColumnsType } from 'antd';
import { Badge } from '@signozhq/ui/badge';
import { Typography } from '@signozhq/ui/typography';
import { TelemetryFieldKey } from 'api/v5/v5';
import type { TracesTableRow } from '../TracesTable/getFieldColumn';
import { DATE_TIME_FORMATS } from 'constants/dateTimeFormats';
import ROUTES from 'constants/routes';
import { buildCompositeKey } from 'container/OptionsMenu/utils';
import { getMs } from 'container/Trace/Filters/Panel/PanelBody/Duration/util';
import { formUrlParams } from 'container/TraceDetail/utils';
import { TimestampInput } from 'hooks/useTimezoneFormatter/useTimezoneFormatter';
import { RowData } from 'lib/query/createTableColumnsFromQuery';
import LineClampedText from 'periscope/components/LineClampedText/LineClampedText';
import { ILog } from 'types/api/logs/log';
import { QueryDataV3 } from 'types/api/widgets/getQuery';
export function BlockLink({
children,
to,
openInNewTab,
}: {
children: React.ReactNode;
to: string;
openInNewTab: boolean;
}): any {
// Display block to make the whole cell clickable
return (
<Link
to={to}
style={{ display: 'block' }}
target={openInNewTab ? '_blank' : '_self'}
>
{children}
</Link>
);
}
export const transformDataWithDate = (
data: QueryDataV3[],
): Omit<ILog, 'timestamp'>[] =>
data[0]?.list?.map(({ data, timestamp }) => ({ ...data, date: timestamp })) ||
[];
export const getTraceLink = (record: Record<string, unknown>): string => {
function readId(value: unknown): string {
if (typeof value === 'string' || typeof value === 'number') {
return String(value);
}
return '';
}
const traceId = readId(record.traceID) || readId(record.trace_id);
const spanId = readId(record.spanID) || readId(record.span_id);
return `${ROUTES.TRACE}/${traceId}${formUrlParams({
spanId,
levelUp: 0,
levelDown: 0,
})}`;
};
export const getListColumns = (
selectedColumns: TelemetryFieldKey[],
formatTimezoneAdjustedTimestamp: (
input: TimestampInput,
format?: string,
) => string | number,
): ColumnsType<RowData> => {
const initialColumns: ColumnsType<RowData> = [
{
dataIndex: 'date',
key: 'date',
title: 'Timestamp',
width: 145,
render: (value, item): JSX.Element => {
const date =
typeof value === 'string'
? formatTimezoneAdjustedTimestamp(
value,
DATE_TIME_FORMATS.ISO_DATETIME_MS,
)
: formatTimezoneAdjustedTimestamp(
value / 1e6,
DATE_TIME_FORMATS.ISO_DATETIME_MS,
);
return (
<BlockLink to={getTraceLink(item)} openInNewTab={false}>
<Typography.Text>{date}</Typography.Text>
</BlockLink>
);
},
},
];
const columns: ColumnsType<RowData> =
selectedColumns.map((props) => {
const name = props?.name || (props as any)?.key;
const fieldContext = props?.fieldContext || (props as any)?.type;
return {
title: name,
dataIndex: name,
key: buildCompositeKey(name, fieldContext),
width: 145,
render: (value, item): JSX.Element => {
if (value === '') {
return (
<BlockLink to={getTraceLink(item)} openInNewTab={false}>
<Typography data-testid={name}>N/A</Typography>
</BlockLink>
);
}
if (
name === 'httpMethod' ||
name === 'responseStatusCode' ||
name === 'response_status_code' ||
name === 'http_method'
) {
return (
<BlockLink to={getTraceLink(item)} openInNewTab={false}>
<Badge data-testid={name} color="sakura" variant="outline">
{value}
</Badge>
</BlockLink>
);
}
if (name === 'durationNano' || name === 'duration_nano') {
return (
<BlockLink to={getTraceLink(item)} openInNewTab={false}>
<Typography data-testid={name}>{getMs(value)}ms</Typography>
</BlockLink>
);
}
return (
<BlockLink to={getTraceLink(item)} openInNewTab={false}>
<Typography data-testid={name}>
<LineClampedText text={value} lines={3} />
</Typography>
</BlockLink>
);
},
responsive: ['md'],
};
}) || [];
return [...initialColumns, ...columns];
};
// Reshapes the query-range list payload into table rows. `id` mirrors span_id so
// TanStack sees genuine row changes on orderBy toggles instead of falling back to
// positional ids; `timestamp` is lifted from the wrapping ListItem.
export const transformSpanRows = (data: QueryDataV3[]): TracesTableRow[] => {
const list = data[0]?.list;
if (!list) {
return [];
}
return list.map((item) => {
const row = item.data as Record<string, unknown>;
return {
...row,
timestamp: item.timestamp,
id: row.span_id,
};
}) as TracesTableRow[];
};

View File

@@ -0,0 +1,19 @@
.loading-traces {
padding: 24px 0;
height: 240px;
display: flex;
justify-content: center;
align-items: flex-start;
.loading-traces-content {
display: flex;
align-items: flex-start;
flex-direction: column;
.loading-gif {
height: 72px;
margin-left: -24px;
}
}
}

View File

@@ -0,0 +1,22 @@
import { useTranslation } from 'react-i18next';
import { Typography } from '@signozhq/ui/typography';
import { DataSource } from 'types/common/queryBuilder';
import loadingPlaneUrl from '@/assets/Icons/loading-plane.gif';
import './TraceLoading.styles.scss';
export function TracesLoading(): JSX.Element {
const { t } = useTranslation('common');
return (
<div className="loading-traces">
<div className="loading-traces-content">
<img className="loading-gif" src={loadingPlaneUrl} alt="wait-icon" />
<Typography>
{t('pending_data_placeholder', { dataSource: DataSource.TRACES })}
</Typography>
</div>
</div>
);
}

View File

@@ -0,0 +1,77 @@
import { generatePath, Link } from 'react-router-dom';
import { Badge } from '@signozhq/ui/badge';
import TanStackTable from 'components/TanStackTableView';
import { DATE_TIME_FORMATS } from 'constants/dateTimeFormats';
import ROUTES from 'constants/routes';
import { getMs } from 'container/Trace/Filters/Panel/PanelBody/Duration/util';
import { useTimezone } from 'providers/Timezone';
import {
DURATION_FIELD_NAMES,
STATUS_FIELD_NAMES,
TIMESTAMP_FIELD_NAMES,
TRACE_ID_FIELD_NAMES,
} from './constants';
import { stringifyCellValue } from './utils';
type FieldCellProps = {
name: string;
value: unknown;
};
function FieldCell({ name, value }: FieldCellProps): JSX.Element {
const { formatTimezoneAdjustedTimestamp } = useTimezone();
if (TIMESTAMP_FIELD_NAMES.has(name)) {
const ts = value as string | number;
const formatted =
typeof ts === 'string'
? formatTimezoneAdjustedTimestamp(ts, DATE_TIME_FORMATS.ISO_DATETIME_MS)
: formatTimezoneAdjustedTimestamp(
ts / 1e6,
DATE_TIME_FORMATS.ISO_DATETIME_MS,
);
const text = String(formatted);
return <TanStackTable.Text title={text}>{text}</TanStackTable.Text>;
}
if (value === '' || value == null) {
return <TanStackTable.Text data-testid={name}>-</TanStackTable.Text>;
}
const text = stringifyCellValue(value);
if (TRACE_ID_FIELD_NAMES.has(name)) {
return (
<Link
to={generatePath(ROUTES.TRACE_DETAIL, { id: text })}
data-testid="trace-id"
onClick={(e): void => e.stopPropagation()}
>
{text}
</Link>
);
}
if (STATUS_FIELD_NAMES.has(name)) {
return (
<Badge data-testid={name} color="sakura" variant="outline">
{text}
</Badge>
);
}
if (DURATION_FIELD_NAMES.has(name)) {
return (
<TanStackTable.Text data-testid={name}>{getMs(text)}ms</TanStackTable.Text>
);
}
return (
<TanStackTable.Text data-testid={name} title={text}>
{text}
</TanStackTable.Text>
);
}
export default FieldCell;

View File

@@ -0,0 +1,26 @@
.tableWrapper {
flex: 1;
min-height: 0;
display: flex;
flex-direction: column;
}
.tracesTable {
--tanstack-table-row-height: 54px;
--tanstack-table-header-height: 54px;
--tanstack-cell-padding-top-override: 5px;
--tanstack-cell-padding-bottom-override: 5px;
--tanstack-cell-padding-right-override: 15px;
--tanstack-cell-padding-left-override: 15px;
--tanstack-cell-header-padding-left-override: 5px;
--tanstack-cell-header-padding-left-first-column: 15px;
--tanstack-plain-body-line-clamp: 1;
--tanstack-table-cell-bg: var(--l2-background);
--tanstack-table-header-cell-bg: var(--l1-background-hover);
--tanstack-table-row-hover-bg: var(--l1-background-hover);
}

View File

@@ -0,0 +1,116 @@
import { useCallback } from 'react';
import { useHistory } from 'react-router-dom';
import ErrorInPlace from 'components/ErrorInPlace/ErrorInPlace';
import TanStackTable from 'components/TanStackTableView';
import type {
CellTypographySize,
TableColumnDef,
} from 'components/TanStackTableView/types';
import EmptyLogsSearch from 'container/EmptyLogsSearch/EmptyLogsSearch';
import NoLogs from 'container/NoLogs/NoLogs';
import { TracesLoading } from '../TraceLoading/TraceLoading';
import APIError from 'types/api/error';
import { DataSource, PanelTypeKeys } from 'types/common/queryBuilder';
import { getAbsoluteUrl } from 'utils/basePath';
import type { TracesTableRow } from './getFieldColumn';
import styles from './TracesTable.module.scss';
export type TracesTableProps = {
data: TracesTableRow[];
columns: TableColumnDef<TracesTableRow>[];
columnStorageKey?: string;
respectColumnOrder?: boolean;
panelType: PanelTypeKeys;
/** Builds the trace-detail href for a row; drives row click + cmd/ctrl-click. */
getRowHref: (row: TracesTableRow) => string;
isLoading: boolean;
isFetching: boolean;
isError: boolean;
error: APIError | Error | null;
isFilterApplied: boolean;
onColumnOrderChange?: (cols: TableColumnDef<TracesTableRow>[]) => void;
onColumnRemove?: (columnId: string) => void;
cellTypographySize?: CellTypographySize;
};
function TracesTable({
data,
columns,
columnStorageKey,
respectColumnOrder = false,
panelType,
getRowHref,
isLoading,
isFetching,
isError,
error,
isFilterApplied,
onColumnOrderChange,
onColumnRemove,
cellTypographySize = 'medium',
}: TracesTableProps): JSX.Element {
const history = useHistory();
const isDataAbsent =
!isLoading && !isFetching && !isError && data.length === 0;
const handleRowClick = useCallback(
(row: TracesTableRow): void => {
history.push(getRowHref(row));
},
[history, getRowHref],
);
const handleRowClickNewTab = useCallback(
(row: TracesTableRow): void => {
window.open(getAbsoluteUrl(getRowHref(row)), '_blank', 'noopener');
},
[getRowHref],
);
return (
<>
{isError && error && <ErrorInPlace error={error as APIError} />}
{(isLoading || (isFetching && data.length === 0)) && <TracesLoading />}
{isDataAbsent && !isFilterApplied && (
<NoLogs dataSource={DataSource.TRACES} />
)}
{isDataAbsent && isFilterApplied && (
<EmptyLogsSearch dataSource={DataSource.TRACES} panelType={panelType} />
)}
{!isError && data.length !== 0 && (
<div className={styles.tableWrapper}>
<TanStackTable<TracesTableRow>
data={data}
columns={columns}
className={styles.tracesTable}
columnStorageKey={columnStorageKey}
respectColumnOrder={respectColumnOrder}
isLoading={isFetching}
cellTypographySize={cellTypographySize}
onColumnOrderChange={onColumnOrderChange}
onColumnRemove={onColumnRemove}
onRowClick={handleRowClick}
onRowClickNewTab={handleRowClickNewTab}
getRowTestId={(row): string => `traces-table-row-${row.id}`}
/>
</div>
)}
</>
);
}
TracesTable.defaultProps = {
columnStorageKey: undefined,
respectColumnOrder: false,
onColumnOrderChange: undefined,
onColumnRemove: undefined,
cellTypographySize: 'medium',
};
export default TracesTable;

View File

@@ -0,0 +1,18 @@
// Field-name allowlists that drive signal-specific cell rendering. Both legacy
// camelCase and snake_case variants are listed because the API has shipped both.
export const TIMESTAMP_FIELD_NAMES = new Set(['timestamp']);
export const STATUS_FIELD_NAMES = new Set([
'httpMethod',
'http_method',
'http.method',
'http.request.method',
'responseStatusCode',
'response_status_code',
'http.status_code',
'http.response.status_code',
]);
export const DURATION_FIELD_NAMES = new Set(['durationNano', 'duration_nano']);
export const TRACE_ID_FIELD_NAMES = new Set(['traceID', 'trace_id']);

View File

@@ -0,0 +1,26 @@
import { TelemetryFieldKey } from 'api/v5/v5';
import type { TableColumnDef } from 'components/TanStackTableView/types';
import { buildCompositeKey } from 'container/OptionsMenu/utils';
import { TIMESTAMP_FIELD_NAMES } from './constants';
import FieldCell from './FieldCell';
export type TracesTableRow = { id: string } & Record<string, unknown>;
export function getFieldColumn(
field: TelemetryFieldKey,
): TableColumnDef<TracesTableRow> {
const { name, fieldContext, fieldDataType } = field;
const isTimestamp = TIMESTAMP_FIELD_NAMES.has(name);
return {
id: buildCompositeKey(name, fieldContext, fieldDataType),
header: name,
accessorFn: (row): unknown => row[name],
enableMove: !isTimestamp,
enableRemove: !isTimestamp,
canBeHidden: !isTimestamp,
width: { min: 192 },
cell: ({ value }): JSX.Element => <FieldCell name={name} value={value} />,
};
}

View File

@@ -0,0 +1,12 @@
export function stringifyCellValue(value: unknown): string {
if (value == null) {
return '';
}
if (typeof value === 'string') {
return value;
}
if (typeof value === 'number' || typeof value === 'boolean') {
return String(value);
}
return JSON.stringify(value);
}

View File

@@ -1,9 +1,6 @@
import { TelemetryFieldKey } from 'api/v5/v5';
import type { TableColumnDef } from 'components/TanStackTableView/types';
import {
getFieldColumn,
TracesTableRow,
} from 'container/TracesExplorer/TracesTable/getFieldColumn';
import { getFieldColumn, TracesTableRow } from '../TracesTable/getFieldColumn';
import { DEFAULT_PER_PAGE_OPTIONS } from 'hooks/queryPagination';
export const PER_PAGE_OPTIONS: number[] = [10, ...DEFAULT_PER_PAGE_OPTIONS];

View File

@@ -0,0 +1,235 @@
/**
* AI Assistant page-action factories for the Traces Explorer.
*
* Mirrors the logs equivalents — each factory closes over live page
* state/callbacks so `execute()` always operates on the current query, and
* the page component instantiates them via `useMemo` + `usePageActions`.
*
* See `pages/LogsExplorer/aiActions.ts` for the rationale behind writing
* BOTH `filters.items` and `filter.expression` and then re-using the same
* URL parser shape via `redirectWithQueryBuilderData`.
*/
import { convertFiltersToExpression } from 'components/QueryBuilderV2/utils';
import {
aiFilterToTagFilterItem,
FILTER_OP_ENUM,
FILTER_VALUE_DESCRIPTION,
FilterDeps,
replaceFirstQueryData,
} from 'container/AIAssistant/pageActions/builderQueryHelpers';
import {
ActionResult,
PageAction,
} from 'container/AIAssistant/pageActions/types';
import {
IBuilderQuery,
TagFilterItem,
} from 'types/api/queryBuilder/queryBuilderData';
interface AIFilter {
key: string;
op: string;
value: string;
}
interface RunQueryParams {
filters: AIFilter[];
}
interface AddFilterParams {
key: string;
op: string;
value: string;
}
type TracesView = 'list' | 'timeseries' | 'table' | 'trace';
interface ChangeViewParams {
view: TracesView;
}
interface SaveViewParams {
name: string;
}
/**
* Replace all active span filters and navigate to the updated query URL
* (which makes the WHERE clause reflect the new filters and triggers a re-run).
*/
export function tracesRunQueryAction(
deps: FilterDeps,
): PageAction<RunQueryParams> {
return {
id: 'traces.runQuery',
description: 'Replace the active trace filters and re-run the query',
parameters: {
type: 'object',
properties: {
filters: {
type: 'array',
description: 'Replacement filter list',
items: {
type: 'object',
properties: {
key: {
type: 'string',
description: 'Attribute key, e.g. service.name, http.status_code',
},
op: {
type: 'string',
enum: [...FILTER_OP_ENUM],
},
value: {
type: 'string',
description: FILTER_VALUE_DESCRIPTION,
},
},
required: ['key', 'op', 'value'],
},
},
},
required: ['filters'],
},
autoApply: true,
execute: async ({ filters }): Promise<ActionResult> => {
const baseQuery = deps.currentQuery.builder.queryData[0];
if (!baseQuery) {
throw new Error('No active query found in Traces Explorer.');
}
const tagItems = filters.map(aiFilterToTagFilterItem);
const newFilters = { items: tagItems, op: 'AND' };
const updatedBuilderQuery: IBuilderQuery = {
...baseQuery,
filters: newFilters,
filter: convertFiltersToExpression(newFilters),
};
deps.handleSetQueryData(0, updatedBuilderQuery);
deps.redirectWithQueryBuilderData(
replaceFirstQueryData(deps.currentQuery, updatedBuilderQuery),
);
return {
summary: `Query updated with ${filters.length} filter(s) and re-run.`,
};
},
getContext: (): Record<string, unknown> => ({
filters:
deps.currentQuery.builder.queryData[0]?.filters?.items?.map(
(f: TagFilterItem) => ({
key: f.key?.key,
op: f.op,
value: f.value,
}),
) ?? [],
}),
};
}
/**
* Append a single filter to the existing trace query and navigate to the
* updated URL.
*/
export function tracesAddFilterAction(
deps: FilterDeps,
): PageAction<AddFilterParams> {
return {
id: 'traces.addFilter',
description: 'Add a single filter to the current trace query and re-run',
parameters: {
type: 'object',
properties: {
key: {
type: 'string',
description: 'Attribute key, e.g. service.name, http.status_code',
},
op: {
type: 'string',
enum: [...FILTER_OP_ENUM],
},
value: {
type: 'string',
description: FILTER_VALUE_DESCRIPTION,
},
},
required: ['key', 'op', 'value'],
},
autoApply: true,
execute: async ({ key, op, value }): Promise<ActionResult> => {
const baseQuery = deps.currentQuery.builder.queryData[0];
if (!baseQuery) {
throw new Error('No active query found in Traces Explorer.');
}
const existing = baseQuery.filters?.items ?? [];
const newItem = aiFilterToTagFilterItem({ key, op, value });
const newFilters = { items: [...existing, newItem], op: 'AND' };
const updatedBuilderQuery: IBuilderQuery = {
...baseQuery,
filters: newFilters,
filter: convertFiltersToExpression(newFilters),
};
deps.handleSetQueryData(0, updatedBuilderQuery);
deps.redirectWithQueryBuilderData(
replaceFirstQueryData(deps.currentQuery, updatedBuilderQuery),
);
return { summary: `Filter added: ${key} ${op} "${value}". Query re-run.` };
},
};
}
/**
* Switch the traces explorer between list / timeseries / table / trace views.
*/
export function tracesChangeViewAction(deps: {
onChangeView: (view: TracesView) => void;
}): PageAction<ChangeViewParams> {
return {
id: 'traces.changeView',
description:
'Switch the Traces Explorer between list, timeseries, table, and trace views',
parameters: {
type: 'object',
properties: {
view: {
type: 'string',
enum: ['list', 'timeseries', 'table', 'trace'],
description: 'The panel view to switch to',
},
},
required: ['view'],
},
execute: async ({ view }): Promise<ActionResult> => {
deps.onChangeView(view);
return { summary: `Switched to the "${view}" view.` };
},
};
}
/**
* Save the current trace query as a named view (stub — wires to real API
* when available).
*/
export function tracesSaveViewAction(deps: {
onSaveView: (name: string) => Promise<void>;
}): PageAction<SaveViewParams> {
return {
id: 'traces.saveView',
description: 'Save the current trace query as a named view',
parameters: {
type: 'object',
properties: {
name: { type: 'string', description: 'Name for the saved view' },
},
required: ['name'],
},
execute: async ({ name }): Promise<ActionResult> => {
await deps.onSaveView(name);
return { summary: `View "${name}" saved.` };
},
};
}

View File

@@ -0,0 +1,132 @@
import {
ArrowUpToLine,
Atom,
Filter,
SquareMousePointer,
Terminal,
Binoculars,
} from '@signozhq/icons';
import { Button, Tooltip } from 'antd';
import cx from 'classnames';
import { ExplorerViews } from 'pages/LogsExplorer/utils';
import './ToolbarActions.styles.scss';
interface LeftToolbarActionsProps {
items: any;
selectedView: string;
onChangeSelectedView: (view: ExplorerViews) => void;
showFilter: boolean;
handleFilterVisibilityChange: () => void;
}
const activeTab = 'active-tab';
export default function LeftToolbarActions({
items,
selectedView,
onChangeSelectedView,
showFilter,
handleFilterVisibilityChange,
}: LeftToolbarActionsProps): JSX.Element {
const { clickhouse, list, timeseries, table, trace } = items;
return (
<div className="left-toolbar">
{!showFilter && (
<Tooltip title="Show Filters">
<Button onClick={handleFilterVisibilityChange} className="filter-btn">
<Filter size={12} />
<ArrowUpToLine size={12} style={{ transform: 'rotate(90deg)' }} />
</Button>
</Tooltip>
)}
<div className="left-toolbar-query-actions">
{list?.show && (
<Tooltip title="List View">
<Button
disabled={list.disabled}
className={cx(
'list-view-tab',
'explorer-view-option',
selectedView === list.key ? activeTab : '',
)}
onClick={(): void => onChangeSelectedView(list.key)}
>
<SquareMousePointer size={14} data-testid="search-view" />
List View
</Button>
</Tooltip>
)}
{trace?.show && (
<Tooltip title="Trace View">
<Button
disabled={trace.disabled}
className={cx(
'trace-view-tab',
'explorer-view-option',
selectedView === trace.key ? activeTab : '',
)}
onClick={(): void => onChangeSelectedView(trace.key)}
>
<SquareMousePointer size={14} data-testid="trace-view" />
Trace View
</Button>
</Tooltip>
)}
{timeseries?.show && (
<Tooltip title="Time Series">
<Button
disabled={timeseries.disabled}
className={cx(
'timeseries-view-tab',
'explorer-view-option',
selectedView === timeseries.key ? activeTab : '',
)}
onClick={(): void => onChangeSelectedView(timeseries.key)}
>
<Atom size={14} data-testid="query-builder-view" />
Time Series
</Button>
</Tooltip>
)}
{clickhouse?.show && (
<Tooltip title="Clickhouse">
<Button
disabled={clickhouse.disabled}
className={cx(
'clickhouse-view-tab',
'explorer-view-option',
selectedView === clickhouse.key ? activeTab : '',
)}
onClick={(): void => onChangeSelectedView(clickhouse.key)}
>
<Terminal size={14} data-testid="clickhouse-view" />
Clickhouse
</Button>
</Tooltip>
)}
{table?.show && (
<Tooltip title="Table">
<Button
disabled={table.disabled}
className={cx(
'table-view-tab',
'explorer-view-option',
selectedView === table.key ? activeTab : '',
)}
onClick={(): void => onChangeSelectedView(table.key)}
>
<Binoculars size={14} data-testid="query-builder-view-v2" />
Table
</Button>
</Tooltip>
)}
</div>
</div>
);
}

View File

@@ -0,0 +1,125 @@
.left-toolbar {
display: flex;
align-items: center;
.filter-btn {
display: flex;
align-items: center;
justify-content: center;
box-shadow: none;
height: 32px;
margin-right: 12px;
border: 1px solid var(--l1-border);
}
.left-toolbar-query-actions {
display: flex;
border-radius: 2px;
border: 1px solid var(--l1-border);
background: var(--l1-background);
flex-direction: row;
border-bottom: none;
margin-bottom: -1px;
.prom-ql-icon {
height: 14px;
width: 14px;
}
.explorer-view-option {
display: flex;
align-items: center;
justify-content: center;
flex-direction: row;
border: none;
padding: 9px;
box-shadow: none;
border-radius: 0px;
border-left: 1px solid var(--l1-border);
border-bottom: 1px solid var(--l1-border);
gap: 8px;
&.active-tab {
background-color: var(--primary-background);
border-bottom: 1px solid var(--primary-background);
color: var(--primary-foreground);
&:hover {
background-color: var(--primary-background) !important;
}
}
&:disabled {
background-color: var(--l3-background);
opacity: 0.6;
}
&:first-child {
border-left: 1px solid transparent;
}
&:hover {
background-color: transparent !important;
border-left: 1px solid transparent !important;
color: var(--l1-foreground);
}
}
}
.frequency-chart-view-controller {
display: flex;
align-items: center;
padding-left: 8px;
gap: 8px;
}
}
.right-toolbar {
display: flex;
align-items: center;
background-color: var(--bg-robin-600);
}
.right-actions {
display: flex;
align-items: center;
}
.loading-container {
display: flex;
gap: 8px;
align-items: center;
.loading-btn {
display: flex;
width: 32px;
height: 33px;
padding: 4px 10px;
justify-content: center;
align-items: center;
gap: 6px;
flex-shrink: 0;
border-radius: 2px;
background: var(--l3-background);
box-shadow: none;
border: none;
}
.cancel-run {
display: flex;
height: 33px;
padding: 4px 10px;
justify-content: center;
align-items: center;
gap: 6px;
flex: 1 0 0;
border-radius: 2px;
background: var(--danger-background);
border: none;
}
.cancel-run:hover {
background-color: var(--bg-cherry-400) !important;
color: var(--l1-foreground) !important;
}
}

74
heatmap-poc/main.go Normal file
View File

@@ -0,0 +1,74 @@
package main
import (
"embed"
"io/fs"
"log/slog"
"net/http"
"net/http/httputil"
"net/url"
"os"
)
//go:embed static
var embedded embed.FS
// diskAssets is preferred over the embedded copy when it exists, so editing the
// UI and reloading the page needs no rebuild. It resolves against the launch
// config's cwd, the repo root.
const diskAssets = "heatmap-poc/static"
func main() {
addr := envOr("HEATMAP_POC_ADDR", "localhost:8099")
upstream, err := url.Parse(envOr("SIGNOZ_URL", "http://localhost:8080"))
if err != nil {
slog.Error("SIGNOZ_URL is not a URL", "error", err)
os.Exit(1)
}
apiKey := os.Getenv("SIGNOZ_API_KEY")
if apiKey == "" {
slog.Warn("SIGNOZ_API_KEY is unset, every upstream call will be rejected as unauthenticated")
}
mux := http.NewServeMux()
mux.Handle("/api/", &httputil.ReverseProxy{
Rewrite: func(r *httputil.ProxyRequest) {
r.SetURL(upstream)
r.Out.Host = upstream.Host
r.Out.Header.Set("SIGNOZ-API-KEY", apiKey)
},
ErrorHandler: func(rw http.ResponseWriter, _ *http.Request, err error) {
slog.Error("upstream call failed", "error", err)
http.Error(rw, err.Error(), http.StatusBadGateway)
},
})
mux.Handle("/", http.FileServerFS(assets()))
slog.Info("listening", "url", "http://"+addr, "upstream", upstream.String())
if err := http.ListenAndServe(addr, mux); err != nil {
slog.Error("server stopped", "error", err)
os.Exit(1)
}
}
func assets() fs.FS {
if stat, err := os.Stat(diskAssets); err == nil && stat.IsDir() {
slog.Info("serving the UI from disk", "dir", diskAssets)
return os.DirFS(diskAssets)
}
slog.Info("serving the embedded UI")
sub, err := fs.Sub(embedded, "static")
if err != nil {
slog.Error("embedded assets are unreadable", "error", err)
os.Exit(1)
}
return sub
}
func envOr(key, fallback string) string {
if value := os.Getenv(key); value != "" {
return value
}
return fallback
}

788
heatmap-poc/static/app.js Normal file
View File

@@ -0,0 +1,788 @@
const TIME_AGGREGATIONS = ["latest", "sum", "avg", "min", "max", "count", "count_distinct", "rate", "increase"];
const SPACE_AGGREGATIONS = ["sum", "avg", "min", "max", "count"];
// the types whose samples reach the reader as plain values, so a bucket axis has
// to be chosen for them
const VALUE_TYPES = new Set(["gauge", "sum", "summary"]);
// single hue, dark to light: a count's magnitude is the only thing it encodes
const RAMP = ["#1c3557", "#22406c", "#284c82", "#2e5998", "#3668ae", "#4a80c4", "#689dd6", "#8dbbe6", "#b9d8f5"];
const ZERO_FILL = "#0e1016";
const PAD = { left: 78, right: 8, top: 8, bottom: 22 };
const MAX_CHART_HEIGHT = 560;
const MAX_JSON_CHARS = 300_000;
const state = {
mode: "metric",
rows: [],
groupBy: [],
bucketKind: "default",
colorScale: "linear",
grid: null,
emptyReason: "Run a query to draw the heatmap.",
hidden: new Set(),
requestText: "",
responseText: "",
};
const catalogue = new Map();
const attributesByMetric = new Map();
let nextRowId = 0;
const $ = (selector) => document.querySelector(selector);
const esc = (value) => String(value).replace(/[&<>"]/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;" })[c]);
function timeWindow() {
const end = Date.now();
return { start: end - Number($("#range").value) * 60_000, end };
}
function rowLetter(index) {
return String.fromCharCode(65 + index);
}
function typeSupport(type) {
if (VALUE_TYPES.has(type)) {
return { aggregations: true, buckets: true };
}
if (type === "histogram") {
return { aggregations: false, buckets: false, note: "Read with increase/sum over its own le labels. Bucket options are rejected." };
}
if (type === "exponentialhistogram") {
return { aggregations: false, buckets: false, bad: true, note: "Exponential histograms are not supported yet — this comes back 501." };
}
return { aggregations: false, buckets: false, bad: true, note: "No type is recorded for this metric, so no bucket axis can be chosen — this comes back 400." };
}
function scaleHint(scale) {
const perTwice = 2 ** scale;
return perTwice >= 1 ? `${perTwice} bucket${perTwice === 1 ? "" : "s"} per 2x` : `1 bucket per ${2 ** -scale}x`;
}
function formatNumber(value) {
if (!Number.isFinite(value)) {
return value > 0 ? "∞" : "-∞";
}
if (value === 0) {
return "0";
}
const magnitude = Math.abs(value);
if (magnitude >= 1e6 || magnitude < 1e-3) {
return value.toExponential(1).replace("e+", "e");
}
if (Number.isInteger(value)) {
return String(value);
}
const text = value.toPrecision(magnitude >= 1 ? 4 : 3);
return text.includes(".") ? text.replace(/0+$/, "").replace(/\.$/, "") : text;
}
function formatTime(ms, spanMs) {
const at = new Date(ms);
const clock = at.toTimeString().slice(0, 5);
return spanMs > 24 * 3600_000 ? `${String(at.getMonth() + 1).padStart(2, "0")}-${String(at.getDate()).padStart(2, "0")} ${clock}` : clock;
}
/* ---------- upstream ---------- */
async function getJSON(path, params) {
const response = await fetch(`${path}?${new URLSearchParams(params)}`);
if (!response.ok) {
throw new Error(`${path} came back ${response.status}: ${(await response.text()).slice(0, 400)}`);
}
return response.json();
}
async function searchMetrics(searchText) {
const { start, end } = timeWindow();
const body = await getJSON("/api/v2/metrics", { start, end, limit: 60, searchText });
const metrics = body?.data?.metrics ?? [];
for (const metric of metrics) {
catalogue.set(metric.metricName, metric);
}
return metrics;
}
async function metricAttributes(metricName) {
if (attributesByMetric.has(metricName)) {
return attributesByMetric.get(metricName);
}
const { start, end } = timeWindow();
const body = await getJSON("/api/v2/metrics/attributes", { metricName, start, end });
const keys = (body?.data?.attributes ?? []).map((attribute) => attribute.key);
attributesByMetric.set(metricName, keys);
return keys;
}
/* ---------- metric rows ---------- */
function addRow() {
state.rows.push({ id: nextRowId++, metric: "", type: "", timeAggregation: "max", spaceAggregation: "max", filter: "" });
renderRows();
}
function renderRows() {
const host = $("#rows");
host.textContent = "";
state.rows.forEach((row, index) => {
const node = $("#row-template").content.firstElementChild.cloneNode(true);
const input = node.querySelector(".metric-input");
const list = node.querySelector("datalist");
const badge = node.querySelector(".badge");
const aggregations = node.querySelector(".agg-fields");
const note = node.querySelector(".row-note");
const listId = `metrics-${row.id}`;
node.querySelector(".row-name").textContent = rowLetter(index);
list.id = listId;
input.setAttribute("list", listId);
input.value = row.metric;
node.querySelector(".filter-input").value = row.filter;
node.querySelector(".remove-row").hidden = state.mode !== "formula" || state.rows.length < 2;
const support = typeSupport(row.type);
if (row.metric) {
badge.hidden = false;
badge.textContent = row.type || "no type";
badge.classList.toggle("bad", Boolean(support.bad));
aggregations.hidden = !support.aggregations;
note.hidden = !support.note;
note.textContent = support.note ?? "";
note.classList.toggle("warn", Boolean(support.bad));
}
for (const [select, options, chosen] of [
[node.querySelector(".time-agg"), TIME_AGGREGATIONS, row.timeAggregation],
[node.querySelector(".space-agg"), SPACE_AGGREGATIONS, row.spaceAggregation],
]) {
select.innerHTML = options.map((option) => `<option value="${option}"${option === chosen ? " selected" : ""}>${option}</option>`).join("");
}
const fillOptions = async () => {
try {
const metrics = await searchMetrics(input.value.trim());
list.innerHTML = metrics.map((metric) => `<option value="${esc(metric.metricName)}" label="${esc(metric.type || "no type")}"></option>`).join("");
} catch (error) {
showBanner(error.message);
}
};
let searchTimer = 0;
input.addEventListener("input", () => {
clearTimeout(searchTimer);
searchTimer = setTimeout(fillOptions, 220);
});
input.addEventListener("focus", () => {
if (!list.children.length) {
fillOptions();
}
});
input.addEventListener("change", async () => {
const name = input.value.trim();
row.metric = name;
row.type = catalogue.get(name)?.type ?? "";
if (name && !catalogue.has(name)) {
try {
await searchMetrics(name);
row.type = catalogue.get(name)?.type ?? "";
} catch (error) {
showBanner(error.message);
}
}
renderRows();
refreshPanes();
refreshGroupOptions();
});
node.querySelector(".time-agg").addEventListener("change", (event) => {
row.timeAggregation = event.target.value;
});
node.querySelector(".space-agg").addEventListener("change", (event) => {
row.spaceAggregation = event.target.value;
});
node.querySelector(".filter-input").addEventListener("change", (event) => {
row.filter = event.target.value.trim();
});
node.querySelector(".remove-row").addEventListener("click", () => {
state.rows = state.rows.filter((candidate) => candidate.id !== row.id);
renderRows();
refreshPanes();
refreshGroupOptions();
});
host.append(node);
});
}
/* ---------- group by ---------- */
function renderGroupChips() {
$("#group-chips").innerHTML = state.groupBy
.map((key) => `<span class="chip">${esc(key)}<button type="button" data-key="${esc(key)}" title="Remove">&times;</button></span>`)
.join("");
}
async function refreshGroupOptions() {
const metrics = state.rows.map((row) => row.metric).filter(Boolean);
const hint = $("#group-hint");
if (!metrics.length) {
$("#group-options").innerHTML = "";
hint.hidden = false;
hint.textContent = "Pick a metric to load its attributes.";
return;
}
try {
const keys = new Set();
for (const row of state.rows.filter((candidate) => candidate.metric)) {
for (const key of await metricAttributes(row.metric)) {
// the builder strips `le` from a histogram's group by and reads the
// bucket axis off it instead, so offering it here would do nothing
if (key !== "le" || row.type !== "histogram") {
keys.add(key);
}
}
}
const available = [...keys].filter((key) => !state.groupBy.includes(key)).sort();
$("#group-options").innerHTML = available.map((key) => `<option value="${esc(key)}"></option>`).join("");
hint.hidden = available.length > 0;
hint.textContent = available.length ? "" : "No further attributes on the selected metrics in this window.";
} catch (error) {
showBanner(error.message);
}
}
/* ---------- panes ---------- */
// A formula is bucketed from its own output, so its inputs may be any type. In
// metric mode the one metric decides, and an unpicked one keeps the pane up.
function bucketsAllowed() {
if (state.mode === "promql") {
return false;
}
if (state.mode === "formula") {
return true;
}
const row = state.rows[0];
return !row?.metric || typeSupport(row.type).buckets;
}
function refreshPanes() {
const isPromql = state.mode === "promql";
const isFormula = state.mode === "formula";
$("#builder-pane").hidden = isPromql;
$("#promql-pane").hidden = !isPromql;
$("#add-row").hidden = !isFormula;
$("#formula-field").hidden = !isFormula;
const allowed = bucketsAllowed();
$("#bucket-pane").hidden = !allowed;
if (!allowed) {
setBucketKind("default");
}
}
function setBucketKind(kind) {
state.bucketKind = kind;
for (const button of $("#bucket-kinds").children) {
button.classList.toggle("on", button.dataset.kind === kind);
}
$("#bucket-default-hint").hidden = kind !== "default";
$("#scale-field").hidden = kind !== "log";
$("#linear-fields").hidden = kind !== "linear";
$("#scale-hint").textContent = scaleHint(Number($("#scale").value));
}
function setMode(mode) {
state.mode = mode;
for (const button of $("#modes").children) {
button.classList.toggle("on", button.dataset.mode === mode);
}
if (mode === "metric") {
state.rows = state.rows.slice(0, 1);
}
if (!state.rows.length) {
addRow();
}
if (mode === "formula" && state.rows.length < 2) {
addRow();
}
renderRows();
refreshPanes();
refreshGroupOptions();
}
/* ---------- request ---------- */
function buildRequest() {
const { start, end } = timeWindow();
const step = Number($("#step").value) || 60;
const request = {
schemaVersion: "v1",
start,
end,
requestType: "heatmap",
compositeQuery: { queries: [] },
formatOptions: { formatTableResultForUI: false, fillGaps: false },
noCache: $("#no-cache").checked,
};
if (state.mode === "promql") {
const query = $("#promql").value.trim();
if (!query) {
throw new Error("Enter a PromQL query.");
}
request.compositeQuery.queries.push({ type: "promql", spec: { name: "A", query, step, disabled: false } });
return request;
}
const inFormula = state.mode === "formula";
state.rows.forEach((row, index) => {
if (!row.metric) {
throw new Error(`Query ${rowLetter(index)} has no metric selected.`);
}
const histogram = row.type === "histogram";
const spec = {
name: rowLetter(index),
signal: "metrics",
aggregations: [
{
metricName: row.metric,
timeAggregation: histogram ? "increase" : row.timeAggregation,
spaceAggregation: histogram ? "sum" : row.spaceAggregation,
},
],
stepInterval: step,
// only the enabled query renders the heatmap, so a formula's inputs ride
// along disabled
disabled: inFormula,
};
if (state.groupBy.length) {
spec.groupBy = state.groupBy.map((name) => ({ name }));
}
if (row.filter) {
spec.filter = { expression: row.filter };
}
request.compositeQuery.queries.push({ type: "builder_query", spec });
});
if (inFormula) {
const expression = $("#formula").value.trim();
if (!expression) {
throw new Error("A formula heatmap needs an expression.");
}
request.compositeQuery.queries.push({ type: "builder_formula", spec: { name: "F1", expression, disabled: false } });
}
if (state.bucketKind === "log") {
request.bucketOptions = { kind: "log", spec: { scale: Number($("#scale").value) } };
} else if (state.bucketKind === "linear") {
request.bucketOptions = { kind: "linear", spec: { maxValue: Number($("#max-value").value), numBuckets: Number($("#num-buckets").value) } };
}
return request;
}
async function run() {
let request;
try {
request = buildRequest();
} catch (error) {
showBanner(error.message);
return;
}
state.window = { start: request.start, end: request.end };
state.requestText = JSON.stringify(request, null, "\t");
renderJSON("#request", state.requestText);
$("#run").disabled = true;
$("#run").textContent = "Running…";
showBanner("");
try {
const response = await fetch("/api/v5/query_range", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(request),
});
const text = await response.text();
let parsed = null;
try {
parsed = JSON.parse(text);
state.responseText = JSON.stringify(parsed, null, "\t");
} catch {
state.responseText = text;
}
renderJSON("#response", state.responseText);
if (!response.ok) {
const problem = parsed?.error;
const detail = (problem?.errors ?? []).map((entry) => entry.message ?? JSON.stringify(entry)).join("\n");
showBanner(`${response.status} ${problem?.code ?? ""}\n${problem?.message ?? text.slice(0, 600)}${detail ? `\n${detail}` : ""}`.trim());
setGrid(null, "The request was rejected — see the message above.");
return;
}
const warning = parsed?.data?.warning;
if (warning?.message) {
showBanner([warning.message, ...(warning.warnings ?? []).map((entry) => entry.message)].join("\n"));
}
const grid = buildGrid(parsed);
setGrid(grid, "The response carried no series, so there is nothing to draw.");
} catch (error) {
showBanner(error.message);
setGrid(null, "The request could not be sent — see the message above.");
} finally {
$("#run").disabled = false;
$("#run").textContent = "Run query";
}
}
/* ---------- response ---------- */
function buildGrid(body) {
const results = body?.data?.data?.results ?? [];
const result = results.find((entry) => Array.isArray(entry?.aggregations) && entry.aggregations.length);
if (!result) {
return null;
}
const aggregation = result.aggregations[0];
const buckets = aggregation.meta?.buckets ?? [];
const rows = buckets.length + 1;
const partial = new Set();
const series = (aggregation.series ?? []).map((entry) => {
const labels = (entry.labels ?? []).map((label) => `${label.key?.name ?? "?"}=${label.value}`);
const byTs = new Map();
let total = 0;
for (const point of entry.values ?? []) {
const counts = point.values ?? [];
byTs.set(point.timestamp, counts);
total += counts.reduce((sum, count) => sum + count, 0);
if (point.partial) {
partial.add(point.timestamp);
}
}
return { key: labels.join(", ") || "(no labels)", byTs, total };
});
const timestamps = [...new Set(series.flatMap((entry) => [...entry.byTs.keys()]))].sort((a, b) => a - b);
series.sort((a, b) => b.total - a.total || a.key.localeCompare(b.key));
return { queryName: result.queryName, buckets, rows, timestamps, partial, series };
}
function setGrid(grid, emptyReason) {
state.grid = grid;
state.emptyReason = emptyReason;
state.hidden = new Set();
renderGroups();
renderChart();
}
function visibleSeries() {
return state.grid.series.filter((entry) => !state.hidden.has(entry.key));
}
function renderGroups() {
const grid = state.grid;
const card = $("#groups-card");
if (!grid || grid.series.length < 2) {
card.hidden = true;
return;
}
card.hidden = false;
$("#group-count").textContent = `${grid.series.length - state.hidden.size} of ${grid.series.length} shown`;
$("#groups").innerHTML = grid.series
.map(
(entry) => `<label><input type="checkbox" data-key="${esc(entry.key)}"${state.hidden.has(entry.key) ? "" : " checked"}>
<span class="name" title="${esc(entry.key)}">${esc(entry.key)}</span>
<span class="total">${formatNumber(entry.total)}</span></label>`,
)
.join("");
}
/* ---------- chart ---------- */
function bucketRange(grid, row) {
if (row === grid.rows - 1) {
return grid.buckets.length ? `> ${formatNumber(grid.buckets[grid.buckets.length - 1])}` : "overflow, the response carried no bucket bounds";
}
const upper = formatNumber(grid.buckets[row]);
return row === 0 ? `<= ${upper}` : `(${formatNumber(grid.buckets[row - 1])}, ${upper}]`;
}
function colorFor(count, max) {
if (count <= 0) {
return ZERO_FILL;
}
const fraction = max <= 0 ? 1 : state.colorScale === "log" ? Math.log1p(count) / Math.log1p(max) : count / max;
return RAMP[Math.min(RAMP.length - 1, Math.max(0, Math.round(fraction * (RAMP.length - 1))))];
}
function renderChart() {
const host = $("#chart");
const legend = $("#legend");
const grid = state.grid;
legend.textContent = "";
if (!grid) {
host.innerHTML = `<p class="empty">${esc(state.emptyReason)}</p>`;
return;
}
if (!grid.timestamps.length) {
host.innerHTML = '<p class="empty">The query returned no columns.</p>';
return;
}
const columns = grid.timestamps.length;
const shown = visibleSeries();
const matrix = grid.timestamps.map((ts) => {
const column = new Array(grid.rows).fill(0);
for (const entry of shown) {
const counts = entry.byTs.get(ts);
if (!counts) {
continue;
}
for (let row = 0; row < grid.rows; row++) {
column[row] += counts[row] ?? 0;
}
}
return column;
});
const max = Math.max(0, ...matrix.flat());
// clientWidth carries the 12px padding on either side of #chart
const available = host.clientWidth - 24 - PAD.left - PAD.right;
const cellWidth = Math.max(3, available / columns);
const cellHeight = Math.min(22, Math.max(4, MAX_CHART_HEIGHT / grid.rows));
const plotWidth = cellWidth * columns;
const plotHeight = cellHeight * grid.rows;
const gap = cellWidth >= 7 && cellHeight >= 7 ? 1 : 0;
const span = grid.timestamps[columns - 1] - grid.timestamps[0];
const cells = [];
for (let column = 0; column < columns; column++) {
for (let row = 0; row < grid.rows; row++) {
const x = PAD.left + column * cellWidth;
const y = PAD.top + (grid.rows - 1 - row) * cellHeight;
cells.push(`<rect x="${x.toFixed(2)}" y="${y.toFixed(2)}" width="${(cellWidth - gap).toFixed(2)}" height="${(cellHeight - gap).toFixed(2)}" fill="${colorFor(matrix[column][row], max)}"/>`);
}
}
const rowStride = Math.max(1, Math.ceil(13 / cellHeight));
const rowLabels = [];
for (let row = 0; row < grid.rows; row++) {
if (row % rowStride !== 0 && row !== grid.rows - 1) {
continue;
}
const y = PAD.top + (grid.rows - 1 - row) * cellHeight + cellHeight / 2;
const text = row === grid.rows - 1 ? "∞" : formatNumber(grid.buckets[row]);
rowLabels.push(`<text class="axis-label" x="${PAD.left - 6}" y="${(y + 3.2).toFixed(2)}" text-anchor="end">${esc(text)}</text>`);
}
const columnStride = Math.max(1, Math.ceil(58 / cellWidth));
const columnLabels = [];
for (let column = 0; column < columns; column += columnStride) {
const ts = grid.timestamps[column];
const x = PAD.left + column * cellWidth;
columnLabels.push(`<text class="axis-label" x="${x.toFixed(2)}" y="${(PAD.top + plotHeight + 14).toFixed(2)}">${esc(formatTime(ts, span))}${grid.partial.has(ts) ? "*" : ""}</text>`);
}
host.innerHTML = `<svg width="${PAD.left + plotWidth + PAD.right}" height="${PAD.top + plotHeight + PAD.bottom}">
${cells.join("")}
<line class="axis-line" x1="${PAD.left}" y1="${PAD.top + plotHeight + 0.5}" x2="${PAD.left + plotWidth}" y2="${PAD.top + plotHeight + 0.5}"/>
${rowLabels.join("")}${columnLabels.join("")}
<rect class="cursor-cell" hidden/>
</svg>`;
// a heatmap cannot fill gaps, so a chart much shorter than the window asked
// for means those columns hold no data at all rather than being hidden
const asked = state.window ? state.window.end - state.window.start : span;
const covered = `${formatTime(grid.timestamps[0], asked)}${formatTime(grid.timestamps[columns - 1], asked)}`;
const coverage =
state.window && span < 0.9 * asked
? `${covered}, the only columns with data in the ${formatTime(state.window.start, asked)}${formatTime(state.window.end, asked)} requested`
: covered;
legend.innerHTML = `<span>0</span>
<div class="swatches"><div class="swatch" style="background:${ZERO_FILL};border:1px solid var(--line)"></div>${RAMP.map((color) => `<div class="swatch" style="background:${color}"></div>`).join("")}</div>
<span>${formatNumber(max)} per cell</span>
<span>· ${grid.rows} buckets × ${columns} columns · ${shown.length} of ${grid.series.length} series · ${esc(coverage)}${grid.partial.size ? " · * partial column" : ""}</span>`;
attachHover(host.querySelector("svg"), { grid, matrix, shown, columns, cellWidth, cellHeight, plotHeight, span, max });
}
function attachHover(svg, view) {
const tooltip = $("#tooltip");
const cursor = svg.querySelector(".cursor-cell");
svg.addEventListener("mouseleave", () => {
tooltip.hidden = true;
cursor.setAttribute("hidden", "");
});
svg.addEventListener("mousemove", (event) => {
const box = svg.getBoundingClientRect();
const column = Math.floor((event.clientX - box.left - PAD.left) / view.cellWidth);
const row = view.grid.rows - 1 - Math.floor((event.clientY - box.top - PAD.top) / view.cellHeight);
if (column < 0 || column >= view.columns || row < 0 || row >= view.grid.rows) {
tooltip.hidden = true;
cursor.setAttribute("hidden", "");
return;
}
cursor.removeAttribute("hidden");
cursor.setAttribute("x", PAD.left + column * view.cellWidth);
cursor.setAttribute("y", PAD.top + (view.grid.rows - 1 - row) * view.cellHeight);
cursor.setAttribute("width", view.cellWidth);
cursor.setAttribute("height", view.cellHeight);
const ts = view.grid.timestamps[column];
const breakdown = view.shown
.map((entry) => ({ key: entry.key, count: entry.byTs.get(ts)?.[row] ?? 0 }))
.filter((entry) => entry.count > 0)
.sort((a, b) => b.count - a.count);
tooltip.innerHTML = [
`<b>${esc(formatNumber(view.matrix[column][row]))}</b> in ${esc(bucketRange(view.grid, row))}`,
`${esc(new Date(ts).toTimeString().slice(0, 8))}${view.grid.partial.has(ts) ? " (partial)" : ""}`,
...breakdown.slice(0, 6).map((entry) => ` ${esc(entry.key)} ${esc(formatNumber(entry.count))}`),
breakdown.length > 6 ? `${breakdown.length - 6} more` : "",
]
.filter(Boolean)
.join("\n");
tooltip.hidden = false;
const width = tooltip.offsetWidth;
tooltip.style.left = `${Math.min(event.clientX + 14, window.innerWidth - width - 8)}px`;
tooltip.style.top = `${Math.min(event.clientY + 14, window.innerHeight - tooltip.offsetHeight - 8)}px`;
});
}
/* ---------- chrome ---------- */
function showBanner(message) {
const banner = $("#banner");
banner.textContent = message;
banner.hidden = !message;
}
function renderJSON(selector, text) {
$(selector).textContent = text.length > MAX_JSON_CHARS ? `${text.slice(0, MAX_JSON_CHARS)}\n… truncated for display, Copy takes the whole thing` : text;
}
function wire() {
$("#modes").addEventListener("click", (event) => {
if (event.target.dataset.mode) {
setMode(event.target.dataset.mode);
}
});
$("#bucket-kinds").addEventListener("click", (event) => {
if (event.target.dataset.kind) {
setBucketKind(event.target.dataset.kind);
}
});
$("#scale").addEventListener("input", () => {
$("#scale-hint").textContent = scaleHint(Number($("#scale").value));
});
$("#add-row").addEventListener("click", () => {
addRow();
refreshPanes();
});
$("#range").addEventListener("change", () => {
attributesByMetric.clear();
refreshGroupOptions();
});
$("#group-input").addEventListener("change", (event) => {
const key = event.target.value.trim();
if (key && !state.groupBy.includes(key)) {
state.groupBy.push(key);
renderGroupChips();
refreshGroupOptions();
}
event.target.value = "";
});
$("#group-chips").addEventListener("click", (event) => {
const key = event.target.dataset.key;
if (key) {
state.groupBy = state.groupBy.filter((candidate) => candidate !== key);
renderGroupChips();
refreshGroupOptions();
}
});
$("#form").addEventListener("submit", (event) => {
event.preventDefault();
run();
});
$("#color-scale").addEventListener("change", (event) => {
state.colorScale = event.target.value;
renderChart();
});
$("#groups").addEventListener("change", (event) => {
const key = event.target.dataset.key;
if (!key) {
return;
}
if (event.target.checked) {
state.hidden.delete(key);
} else {
state.hidden.add(key);
}
$("#group-count").textContent = `${state.grid.series.length - state.hidden.size} of ${state.grid.series.length} shown`;
renderChart();
});
$("#select-all").addEventListener("click", () => {
state.hidden.clear();
renderGroups();
renderChart();
});
$("#select-none").addEventListener("click", () => {
state.hidden = new Set(state.grid.series.map((entry) => entry.key));
renderGroups();
renderChart();
});
for (const button of document.querySelectorAll("[data-copy]")) {
button.addEventListener("click", async () => {
const text = button.dataset.copy === "request" ? state.requestText : state.responseText;
try {
await navigator.clipboard.writeText(text);
button.textContent = "Copied";
} catch {
button.textContent = "Copy failed";
}
setTimeout(() => {
button.textContent = "Copy";
}, 1200);
});
}
let resizeTimer = 0;
window.addEventListener("resize", () => {
clearTimeout(resizeTimer);
resizeTimer = setTimeout(renderChart, 120);
});
}
wire();
setMode("metric");
renderGroupChips();
setBucketKind("default");

View File

@@ -0,0 +1,168 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Heatmap POC</title>
<link rel="stylesheet" href="style.css">
</head>
<body>
<header>
<h1>Heatmap POC</h1>
<p class="sub"><code>POST /api/v5/query_range</code> with <code>requestType: "heatmap"</code></p>
</header>
<div class="layout">
<form id="form" class="card form">
<div class="segmented" id="modes">
<button type="button" data-mode="metric" class="on">Metric</button>
<button type="button" data-mode="formula">Formula</button>
<button type="button" data-mode="promql">PromQL</button>
</div>
<div class="grid-2">
<label>Time range
<select id="range">
<option value="15">Last 15 minutes</option>
<option value="60" selected>Last 1 hour</option>
<option value="180">Last 3 hours</option>
<option value="360">Last 6 hours</option>
<option value="1440">Last 24 hours</option>
<option value="10080">Last 7 days</option>
</select>
</label>
<label>Step (seconds)
<input id="step" type="number" min="1" step="1" value="60">
</label>
</div>
<section id="builder-pane">
<div id="rows"></div>
<button type="button" id="add-row" class="ghost" hidden>+ Add metric</button>
<label id="formula-field" hidden>Formula
<input id="formula" placeholder="A / B" autocomplete="off">
</label>
<div class="chips-field">
<span class="chips-label">Group by</span>
<div id="group-chips" class="chips"></div>
<input id="group-input" list="group-options" placeholder="Add an attribute…" autocomplete="off">
<datalist id="group-options"></datalist>
<p class="hint" id="group-hint" hidden></p>
</div>
</section>
<section id="promql-pane" hidden>
<label>PromQL
<textarea id="promql" rows="4" spellcheck="false" placeholder='sum by (le) (increase({__name__="http.server.duration.bucket"}[5m]))'></textarea>
</label>
<p class="hint">The bucket axis comes from the <code>le</code> labels the query returns, so bucket options are rejected here. A dotted metric name needs the <code>{__name__="…"}</code> form.</p>
</section>
<fieldset id="bucket-pane">
<legend>Bucket options</legend>
<div class="segmented small" id="bucket-kinds">
<button type="button" data-kind="default" class="on">Default</button>
<button type="button" data-kind="log">Log</button>
<button type="button" data-kind="linear">Linear</button>
</div>
<p class="hint" id="bucket-default-hint">Omitted from the request. The backend falls back to a log axis at scale 4.</p>
<label id="scale-field" hidden>Scale
<input id="scale" type="number" min="-4" max="4" step="1" value="4">
<span class="hint" id="scale-hint"></span>
</label>
<div class="grid-2" id="linear-fields" hidden>
<label>Max value
<input id="max-value" type="number" min="0" step="any" value="1000">
</label>
<label>Number of buckets
<input id="num-buckets" type="number" min="1" max="512" step="1" value="60">
</label>
</div>
</fieldset>
<div class="run-row">
<button type="submit" id="run">Run query</button>
<label class="inline"><input type="checkbox" id="no-cache" checked> Bypass cache</label>
</div>
</form>
<div class="results">
<p id="banner" hidden></p>
<section class="card">
<div class="card-head">
<h2>Heatmap</h2>
<div class="head-tools">
<label class="inline">Colour scale
<select id="color-scale">
<option value="linear" selected>Linear</option>
<option value="log">Log</option>
</select>
</label>
</div>
</div>
<div id="chart" class="chart"><p class="empty">Run a query to draw the heatmap.</p></div>
<div id="legend" class="legend"></div>
</section>
<section class="card" id="groups-card" hidden>
<div class="card-head">
<h2>Groups <span id="group-count" class="count"></span></h2>
<div class="head-tools">
<button type="button" class="ghost" id="select-all">All</button>
<button type="button" class="ghost" id="select-none">None</button>
</div>
</div>
<div id="groups" class="groups"></div>
</section>
<div class="grid-2 json-grid">
<section class="card">
<div class="card-head">
<h2>Request</h2>
<button type="button" class="ghost" data-copy="request">Copy</button>
</div>
<pre id="request" class="json"></pre>
</section>
<section class="card">
<div class="card-head">
<h2>Response</h2>
<button type="button" class="ghost" data-copy="response">Copy</button>
</div>
<pre id="response" class="json"></pre>
</section>
</div>
</div>
</div>
<div id="tooltip" class="tooltip" hidden></div>
<template id="row-template">
<div class="metric-row">
<span class="row-name"></span>
<div class="row-body">
<div class="metric-field">
<input class="metric-input" placeholder="Search metrics…" autocomplete="off" spellcheck="false">
<datalist></datalist>
<span class="badge" hidden></span>
</div>
<div class="agg-fields" hidden>
<label>Time
<select class="time-agg"></select>
</label>
<label>Space
<select class="space-agg"></select>
</label>
</div>
<input class="filter-input" placeholder='Filter, e.g. service = "api"' autocomplete="off" spellcheck="false">
<p class="row-note hint" hidden></p>
</div>
<button type="button" class="remove-row" title="Remove this metric">&times;</button>
</div>
</template>
<script src="app.js" type="module"></script>
</body>
</html>

View File

@@ -0,0 +1,552 @@
:root {
color-scheme: dark;
--surface: #0b0d12;
--card: #12151d;
--card-head: #171b25;
--line: #242a38;
--line-soft: #1b2030;
--ink: #e6e9f0;
--ink-soft: #a2abbd;
--ink-faint: #6b7488;
--accent: #4a80c4;
--accent-ink: #b9d8f5;
--danger: #e0736b;
--danger-bg: #2a1618;
--cell-zero: #0e1016;
}
* {
box-sizing: border-box;
}
/* the display rules below would otherwise beat the UA's [hidden] rule, and SVG
elements never honoured the attribute on their own */
[hidden] {
display: none !important;
}
body {
margin: 0;
padding: 20px;
background: var(--surface);
color: var(--ink);
font: 13px/1.5 ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif;
}
header {
margin-bottom: 16px;
}
h1 {
margin: 0;
font-size: 17px;
font-weight: 600;
letter-spacing: -0.01em;
}
h2 {
margin: 0;
font-size: 12px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.06em;
color: var(--ink-soft);
}
.sub {
margin: 3px 0 0;
color: var(--ink-faint);
}
code {
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
font-size: 0.92em;
color: var(--ink-soft);
}
.layout {
display: grid;
grid-template-columns: minmax(320px, 380px) minmax(0, 1fr);
gap: 16px;
align-items: start;
}
@media (max-width: 900px) {
.layout {
grid-template-columns: minmax(0, 1fr);
}
}
.card {
background: var(--card);
border: 1px solid var(--line);
border-radius: 8px;
overflow: hidden;
}
.card-head {
display: flex;
align-items: center;
justify-content: space-between;
gap: 10px;
padding: 8px 12px;
background: var(--card-head);
border-bottom: 1px solid var(--line);
}
.head-tools {
display: flex;
align-items: center;
gap: 8px;
}
.count {
color: var(--ink-faint);
font-weight: 400;
text-transform: none;
letter-spacing: 0;
}
/* ---------- form ---------- */
.form {
padding: 14px;
display: grid;
gap: 14px;
position: sticky;
top: 20px;
}
label {
display: grid;
gap: 5px;
font-size: 12px;
color: var(--ink-soft);
}
label.inline {
display: inline-flex;
align-items: center;
gap: 6px;
}
.grid-2 {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 10px;
}
input,
select,
textarea {
width: 100%;
padding: 6px 8px;
background: var(--surface);
color: var(--ink);
border: 1px solid var(--line);
border-radius: 5px;
font: inherit;
}
input[type="checkbox"] {
width: auto;
accent-color: var(--accent);
}
textarea {
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
resize: vertical;
}
input:focus-visible,
select:focus-visible,
textarea:focus-visible,
button:focus-visible {
outline: 2px solid var(--accent);
outline-offset: 1px;
}
fieldset {
margin: 0;
padding: 10px 12px 12px;
border: 1px solid var(--line);
border-radius: 6px;
display: grid;
gap: 10px;
}
legend {
padding: 0 5px;
font-size: 11px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.06em;
color: var(--ink-soft);
}
button {
padding: 6px 12px;
background: var(--accent);
color: #fff;
border: 1px solid transparent;
border-radius: 5px;
font: inherit;
font-weight: 500;
cursor: pointer;
}
button:hover {
filter: brightness(1.12);
}
button.ghost {
background: transparent;
color: var(--ink-soft);
border-color: var(--line);
font-weight: 400;
}
button.ghost:hover {
color: var(--ink);
border-color: var(--ink-faint);
filter: none;
}
button[disabled] {
opacity: 0.5;
cursor: default;
filter: none;
}
.segmented {
display: flex;
gap: 2px;
padding: 2px;
background: var(--surface);
border: 1px solid var(--line);
border-radius: 6px;
}
.segmented button {
flex: 1;
background: transparent;
color: var(--ink-soft);
font-weight: 400;
}
.segmented button.on {
background: var(--line);
color: var(--ink);
}
.segmented button:hover {
filter: none;
color: var(--ink);
}
.segmented.small button {
padding: 4px 8px;
font-size: 12px;
}
.hint {
margin: 0;
font-size: 11px;
color: var(--ink-faint);
}
.hint.warn {
color: var(--danger);
}
.run-row {
display: flex;
align-items: center;
gap: 12px;
}
.run-row button {
flex: 1;
}
/* ---------- metric rows ---------- */
#rows {
display: grid;
gap: 8px;
}
.metric-row {
display: grid;
grid-template-columns: auto minmax(0, 1fr) auto;
gap: 8px;
align-items: start;
padding: 8px;
background: var(--surface);
border: 1px solid var(--line-soft);
border-radius: 6px;
}
.row-name {
width: 20px;
padding-top: 6px;
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
font-weight: 600;
color: var(--accent-ink);
text-align: center;
}
.row-body {
display: grid;
gap: 8px;
min-width: 0;
}
.metric-field {
display: flex;
align-items: center;
gap: 6px;
min-width: 0;
}
.badge {
flex: none;
padding: 2px 6px;
background: var(--line);
border-radius: 4px;
font-size: 10px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.04em;
color: var(--accent-ink);
white-space: nowrap;
}
.badge.bad {
background: var(--danger-bg);
color: var(--danger);
}
.agg-fields {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 8px;
}
.remove-row {
padding: 2px 7px;
background: transparent;
color: var(--ink-faint);
border-color: transparent;
font-size: 15px;
line-height: 1.2;
}
.remove-row:hover {
color: var(--danger);
filter: none;
}
/* ---------- group by chips ---------- */
.chips-field {
display: grid;
gap: 6px;
}
.chips-label {
font-size: 12px;
color: var(--ink-soft);
}
.chips {
display: flex;
flex-wrap: wrap;
gap: 5px;
}
.chips:empty {
display: none;
}
.chip {
display: inline-flex;
align-items: center;
gap: 5px;
padding: 2px 4px 2px 8px;
background: var(--line);
border-radius: 11px;
font-size: 12px;
}
.chip button {
padding: 0 3px;
background: transparent;
color: var(--ink-faint);
border: 0;
font-size: 13px;
line-height: 1;
}
.chip button:hover {
color: var(--danger);
filter: none;
}
/* ---------- results ---------- */
.results {
display: grid;
gap: 16px;
min-width: 0;
}
#banner {
margin: 0;
padding: 10px 12px;
background: var(--danger-bg);
border: 1px solid #4a2427;
border-radius: 8px;
color: var(--danger);
white-space: pre-wrap;
}
.chart {
padding: 12px;
overflow: auto;
max-height: 620px;
}
.chart svg {
display: block;
}
.empty {
margin: 0;
padding: 28px 0;
color: var(--ink-faint);
text-align: center;
}
.axis-label {
fill: var(--ink-faint);
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
font-size: 10px;
}
.axis-line {
stroke: var(--line);
stroke-width: 1;
}
.cursor-cell {
fill: none;
stroke: var(--accent-ink);
stroke-width: 1.5;
pointer-events: none;
}
.legend {
display: flex;
align-items: center;
gap: 10px;
padding: 0 12px 12px;
color: var(--ink-faint);
font-size: 11px;
}
.legend:empty {
display: none;
}
.legend .swatches {
display: flex;
gap: 2px;
}
.legend .swatch {
width: 20px;
height: 10px;
border-radius: 2px;
}
.groups {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
gap: 2px 12px;
padding: 10px 12px;
max-height: 220px;
overflow: auto;
}
.groups label {
display: flex;
align-items: center;
gap: 7px;
min-width: 0;
padding: 2px 0;
color: var(--ink);
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
font-size: 11px;
cursor: pointer;
}
.groups .name {
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.groups .total {
margin-left: auto;
flex: none;
color: var(--ink-faint);
}
.json-grid {
align-items: start;
}
@media (max-width: 1200px) {
.json-grid {
grid-template-columns: minmax(0, 1fr);
}
}
.json {
margin: 0;
padding: 12px;
max-height: 420px;
overflow: auto;
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
font-size: 11px;
line-height: 1.55;
color: var(--ink-soft);
white-space: pre;
tab-size: 2;
}
.tooltip {
position: fixed;
z-index: 10;
max-width: 320px;
padding: 7px 9px;
background: #1c212e;
border: 1px solid var(--line);
border-radius: 6px;
box-shadow: 0 6px 18px rgb(0 0 0 / 45%);
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
font-size: 11px;
line-height: 1.6;
pointer-events: none;
white-space: pre;
}
.tooltip b {
color: var(--accent-ink);
font-weight: 600;
}

View File

@@ -451,7 +451,7 @@ func (bc *bucketCache) mergeBuckets(ctx context.Context, buckets []*qbtypes.Cach
// Merge values based on type
var mergedValue any
switch resultType {
case qbtypes.RequestTypeTimeSeries:
case qbtypes.RequestTypeTimeSeries, qbtypes.RequestTypeHeatmap:
mergedValue = bc.mergeTimeSeriesValues(ctx, buckets)
// Raw and Scalar types are not cached, so no merge needed
}
@@ -476,14 +476,36 @@ func (bc *bucketCache) mergeTimeSeriesValues(ctx context.Context, buckets []*qbt
}
seriesMap := make(map[seriesKey]*qbtypes.TimeSeries, estimatedSeries)
decodedTimeSeriesData := make([]*qbtypes.TimeSeriesData, 0, len(buckets))
// Alias and Meta are taken from whichever cached bucket covers the latest
// range, and the buckets do not arrive in StartMs order, so keep the winner
// per AggregationBucket.Index alongside the StartMs that won it.
aggregationIndexToLatest := map[int]*qbtypes.AggregationBucket{}
aggregationIndexToLatestStartMs := map[int]uint64{}
for _, bucket := range buckets {
var tsData *qbtypes.TimeSeriesData
if err := json.Unmarshal(bucket.Value, &tsData); err != nil {
bc.logger.ErrorContext(ctx, "failed to unmarshal time series data", errors.Attr(err))
continue
}
decodedTimeSeriesData = append(decodedTimeSeriesData, tsData)
for _, aggBucket := range tsData.Aggregations {
if _, seen := aggregationIndexToLatest[aggBucket.Index]; !seen || bucket.StartMs >= aggregationIndexToLatestStartMs[aggBucket.Index] {
aggregationIndexToLatest[aggBucket.Index] = aggBucket
aggregationIndexToLatestStartMs[aggBucket.Index] = bucket.StartMs
}
}
}
mergedUpperBounds := qbtypes.MergeBucketUpperBounds(decodedTimeSeriesData...)
for _, tsData := range decodedTimeSeriesData {
for _, aggBucket := range tsData.Aggregations {
aggBucket.ReindexValuesToNewUpperBounds(mergedUpperBounds[aggBucket.Index])
for _, series := range aggBucket.Series {
// Create series key from labels
key := seriesKey{
@@ -556,10 +578,15 @@ func (bc *bucketCache) mergeTimeSeriesValues(ctx context.Context, buckets []*qbt
}
}
result.Aggregations = append(result.Aggregations, &qbtypes.AggregationBucket{
aggBucket := &qbtypes.AggregationBucket{
Index: index,
Series: seriesList,
})
}
if latest, ok := aggregationIndexToLatest[index]; ok {
aggBucket.Alias = latest.Alias
aggBucket.Meta = latest.Meta
}
result.Aggregations = append(result.Aggregations, aggBucket)
}
return result
@@ -572,7 +599,7 @@ func (bc *bucketCache) isEmptyResult(result *qbtypes.Result) (isEmpty bool, isFi
}
switch result.Type {
case qbtypes.RequestTypeTimeSeries:
case qbtypes.RequestTypeTimeSeries, qbtypes.RequestTypeHeatmap:
if tsData, ok := result.Value.(*qbtypes.TimeSeriesData); ok {
// No aggregations at all means truly empty
if len(tsData.Aggregations) == 0 {
@@ -699,14 +726,19 @@ func (bc *bucketCache) trimResultToFluxBoundary(result *qbtypes.Result, fluxBoun
}
switch result.Type {
case qbtypes.RequestTypeTimeSeries:
case qbtypes.RequestTypeTimeSeries, qbtypes.RequestTypeHeatmap:
// Trim time series data
if tsData, ok := result.Value.(*qbtypes.TimeSeriesData); ok && tsData != nil {
trimmedData := &qbtypes.TimeSeriesData{}
for _, aggBucket := range tsData.Aggregations {
// Meta has to survive the trim: a heatmap's counts are
// positional against Meta.Buckets, so a cached bucket that
// lost its axis cannot be read back against anything.
trimmedBucket := &qbtypes.AggregationBucket{
Index: aggBucket.Index,
Alias: aggBucket.Alias,
Meta: aggBucket.Meta,
}
for _, series := range aggBucket.Series {
@@ -766,7 +798,7 @@ func (bc *bucketCache) filterResultToTimeRange(result *qbtypes.Result, startMs,
}
switch result.Type {
case qbtypes.RequestTypeTimeSeries:
case qbtypes.RequestTypeTimeSeries, qbtypes.RequestTypeHeatmap:
if tsData, ok := result.Value.(*qbtypes.TimeSeriesData); ok {
filteredData := &qbtypes.TimeSeriesData{
Aggregations: make([]*qbtypes.AggregationBucket, 0, len(tsData.Aggregations)),

View File

@@ -92,6 +92,10 @@ func (q *builderQuery[T]) Fingerprint() string {
// This needs to include all fields that affect the query results
parts := []string{q.queryType.StringValue()}
// A heatmap and a time series query can share every spec field and still
// return different rows, so the request type has to separate their entries
parts = append(parts, fmt.Sprintf("requestType=%s", q.kind.StringValue()))
// Add signal type
parts = append(parts, fmt.Sprintf("signal=%s", q.spec.Signal.StringValue()))
@@ -130,6 +134,9 @@ func (q *builderQuery[T]) Fingerprint() string {
}
part += ":" + route
}
if a.HeatmapBucketing != nil {
part += ":" + fingerprintHeatmapBucketing(*a.HeatmapBucketing)
}
aggParts = append(aggParts, part)
}
}
@@ -185,6 +192,16 @@ func (q *builderQuery[T]) Fingerprint() string {
return strings.Join(parts, "&")
}
// fingerprintHeatmapBucketing captures only what changes the rows ClickHouse
// returns, which is why LogBucketsSpec.Scale is absent: coarsening it happens in
// postprocessing, so every scale reads one cache entry.
func fingerprintHeatmapBucketing(b qbtypes.HeatmapBucketing) string {
if b.Kind == qbtypes.BucketsKindLinear {
return fmt.Sprintf("%s:%v:%d", b.Kind.StringValue(), b.MaxValue, b.NumBuckets)
}
return b.Kind.StringValue()
}
func fingerprintGroupByKey(gb qbtypes.GroupByKey) string {
return fingerprintFieldKey(gb.TelemetryFieldKey)
}
@@ -412,7 +429,7 @@ func (q *builderQuery[T]) narrowWindowByTraceID(ctx context.Context, fromMS, toM
func emptyResultFor(kind qbtypes.RequestType, queryName string) *qbtypes.Result {
var value any
switch kind {
case qbtypes.RequestTypeTimeSeries:
case qbtypes.RequestTypeTimeSeries, qbtypes.RequestTypeHeatmap:
value = &qbtypes.TimeSeriesData{QueryName: queryName}
case qbtypes.RequestTypeScalar:
value = &qbtypes.ScalarData{QueryName: queryName}
@@ -465,8 +482,9 @@ func (q *builderQuery[T]) executeWithContext(ctx context.Context, query string,
queryWindow := &qbtypes.TimeRange{From: q.fromMS, To: q.toMS}
kind := q.kind
// all metric queries are time series then reduced if required
if q.spec.Signal == telemetrytypes.SignalMetrics {
// all metric queries are time series then reduced if required, except
// heatmaps, whose statement returns a row per bucket rather than per point
if q.spec.Signal == telemetrytypes.SignalMetrics && kind != qbtypes.RequestTypeHeatmap {
kind = qbtypes.RequestTypeTimeSeries
}

View File

@@ -6,6 +6,7 @@ import (
"time"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/stretchr/testify/assert"
@@ -120,6 +121,170 @@ func TestBuilderQueryFingerprintQueryType(t *testing.T) {
assert.Empty(t, ai.Fingerprint())
}
func TestBuilderQueryFingerprintHeatmapBucketing(t *testing.T) {
coarseLogScale := 1
testCases := []struct {
description string
left *builderQuery[qbtypes.MetricAggregation]
right *builderQuery[qbtypes.MetricAggregation]
expectedEqual bool
}{
{
// fingerprintHeatmapBucketing leaves LogScale out, so the two are
// indistinguishable here by design
description: "a coarser logScale reads the same cache entry",
left: &builderQuery[qbtypes.MetricAggregation]{
queryType: qbtypes.QueryTypeBuilder,
kind: qbtypes.RequestTypeHeatmap,
spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{{
MetricName: "system.memory.usage",
Type: metrictypes.GaugeType,
HeatmapBucketing: &qbtypes.HeatmapBucketing{Kind: qbtypes.BucketsKindLog, LogScale: qbtypes.MaxLogScale, NumBuckets: qbtypes.DefaultNumBuckets},
}},
},
},
right: &builderQuery[qbtypes.MetricAggregation]{
queryType: qbtypes.QueryTypeBuilder,
kind: qbtypes.RequestTypeHeatmap,
spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{{
MetricName: "system.memory.usage",
Type: metrictypes.GaugeType,
HeatmapBucketing: &qbtypes.HeatmapBucketing{Kind: qbtypes.BucketsKindLog, LogScale: qbtypes.MaxLogScale, NumBuckets: qbtypes.DefaultNumBuckets},
}},
},
},
expectedEqual: true,
},
{
description: "linear separates on maxValue",
left: &builderQuery[qbtypes.MetricAggregation]{
queryType: qbtypes.QueryTypeBuilder,
kind: qbtypes.RequestTypeHeatmap,
spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{{
MetricName: "system.memory.usage",
Type: metrictypes.GaugeType,
HeatmapBucketing: &qbtypes.HeatmapBucketing{Kind: qbtypes.BucketsKindLinear, MaxValue: 500, NumBuckets: 25},
}},
},
},
right: &builderQuery[qbtypes.MetricAggregation]{
queryType: qbtypes.QueryTypeBuilder,
kind: qbtypes.RequestTypeHeatmap,
spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{{
MetricName: "system.memory.usage",
Type: metrictypes.GaugeType,
HeatmapBucketing: &qbtypes.HeatmapBucketing{Kind: qbtypes.BucketsKindLinear, MaxValue: 800, NumBuckets: 25},
}},
},
},
expectedEqual: false,
},
{
description: "linear separates on numBuckets",
left: &builderQuery[qbtypes.MetricAggregation]{
queryType: qbtypes.QueryTypeBuilder,
kind: qbtypes.RequestTypeHeatmap,
spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{{
MetricName: "system.memory.usage",
Type: metrictypes.GaugeType,
HeatmapBucketing: &qbtypes.HeatmapBucketing{Kind: qbtypes.BucketsKindLinear, MaxValue: 500, NumBuckets: 25},
}},
},
},
right: &builderQuery[qbtypes.MetricAggregation]{
queryType: qbtypes.QueryTypeBuilder,
kind: qbtypes.RequestTypeHeatmap,
spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{{
MetricName: "system.memory.usage",
Type: metrictypes.GaugeType,
HeatmapBucketing: &qbtypes.HeatmapBucketing{Kind: qbtypes.BucketsKindLinear, MaxValue: 500, NumBuckets: 40},
}},
},
},
expectedEqual: false,
},
{
description: "linear and log are separate entries",
left: &builderQuery[qbtypes.MetricAggregation]{
queryType: qbtypes.QueryTypeBuilder,
kind: qbtypes.RequestTypeHeatmap,
spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{{
MetricName: "system.memory.usage",
Type: metrictypes.GaugeType,
HeatmapBucketing: &qbtypes.HeatmapBucketing{Kind: qbtypes.BucketsKindLinear, MaxValue: 500, NumBuckets: 25},
}},
},
},
right: &builderQuery[qbtypes.MetricAggregation]{
queryType: qbtypes.QueryTypeBuilder,
kind: qbtypes.RequestTypeHeatmap,
spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{{
MetricName: "system.memory.usage",
Type: metrictypes.GaugeType,
HeatmapBucketing: &qbtypes.HeatmapBucketing{Kind: qbtypes.BucketsKindLog, LogScale: qbtypes.MaxLogScale, NumBuckets: qbtypes.DefaultNumBuckets},
}},
},
},
expectedEqual: false,
},
}
for _, testCase := range testCases {
t.Run(testCase.description, func(t *testing.T) {
if testCase.expectedEqual {
assert.Equal(t, testCase.left.Fingerprint(), testCase.right.Fingerprint())
return
}
assert.NotEqual(t, testCase.left.Fingerprint(), testCase.right.Fingerprint())
})
}
t.Run("a coarser scale is carried but reads the same cache entry", func(t *testing.T) {
finest := (&qbtypes.BucketOptions{Kind: qbtypes.BucketsKindLog, Spec: qbtypes.LogBucketsSpec{}}).ToHeatmapBucketing()
coarse := (&qbtypes.BucketOptions{Kind: qbtypes.BucketsKindLog, Spec: qbtypes.LogBucketsSpec{Scale: &coarseLogScale}}).ToHeatmapBucketing()
assert.NotEqual(t, finest.LogScale, coarse.LogScale)
assert.Equal(t, fingerprintHeatmapBucketing(finest), fingerprintHeatmapBucketing(coarse))
})
t.Run("a histogram folds in no bucket options at all", func(t *testing.T) {
// resolveHeatmapBucketing leaves histograms nil, so bucketOptions sent
// alongside one must not fragment its cache
histogram := &builderQuery[qbtypes.MetricAggregation]{
queryType: qbtypes.QueryTypeBuilder,
kind: qbtypes.RequestTypeHeatmap,
spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{{
MetricName: "signoz_latency",
Type: metrictypes.HistogramType,
}},
},
}
fingerprint := histogram.Fingerprint()
assert.NotContains(t, fingerprint, qbtypes.BucketsKindLog.StringValue())
assert.NotContains(t, fingerprint, qbtypes.BucketsKindLinear.StringValue())
})
}
func TestMakeBucketsOrder(t *testing.T) {
// Test that makeBuckets returns buckets in reverse chronological order by default
// Using milliseconds as input - need > 1 hour range to get multiple buckets

View File

@@ -31,6 +31,11 @@ var (
// written clickhouse query. The column alias indcate which value is
// to be considered as final result (or target).
legacyReservedColumnTargetAliases = []string{"__result", "__value", "result", "res", "value"}
// userHeatmapBucketColumn is the alias a user written clickhouse query can
// give its bucket upper bound column, alongside the HeatmapBucketColumn the
// statement builder emits.
userHeatmapBucketColumn = "bucket"
)
// stripKeyAlias removes the __SELECT_KEY_<n>_ / __GROUP_BY_KEY_<n>_ prefix from a result
@@ -83,6 +88,8 @@ func consume(rows driver.Rows, kind qbtypes.RequestType, queryWindow *qbtypes.Ti
payload, err = readAsTimeSeries(rows, queryWindow, step, queryName)
case qbtypes.RequestTypeScalar:
payload, err = readAsScalar(rows, queryName)
case qbtypes.RequestTypeHeatmap:
payload, err = readAsHeatmap(rows, queryWindow, step, queryName)
case qbtypes.RequestTypeRaw, qbtypes.RequestTypeTrace, qbtypes.RequestTypeRawStream:
payload, err = readAsRaw(rows, queryName)
// TODO: add support for other request types
@@ -112,35 +119,6 @@ func readAsTimeSeries(rows driver.Rows, queryWindow *qbtypes.TimeRange, step qbt
stepMs := uint64(step.Milliseconds())
// Helper function to check if a timestamp represents a partial value
isPartialValue := func(timestamp int64) bool {
if stepMs == 0 || queryWindow == nil {
return false
}
timestampMs := uint64(timestamp)
// For the first interval, check if query start is misaligned
// The first complete interval starts at the first timestamp >= queryWindow.From that is aligned to step
firstCompleteInterval := queryWindow.From
if queryWindow.From%stepMs != 0 {
// Round up to next step boundary
firstCompleteInterval = ((queryWindow.From / stepMs) + 1) * stepMs
}
// If timestamp is before the first complete interval, it's partial
if timestampMs < firstCompleteInterval {
return true
}
// For the last interval, check if it would extend beyond query end
if timestampMs+stepMs > queryWindow.To {
return queryWindow.To%stepMs != 0
}
return false
}
// Pre-allocate for labels based on column count
lblValsCapacity := len(colNames) - 1 // -1 for timestamp
if lblValsCapacity < 0 {
@@ -271,7 +249,7 @@ func readAsTimeSeries(rows driver.Rows, queryWindow *qbtypes.TimeRange, step qbt
series.Values = append(series.Values, &qbtypes.TimeSeriesValue{
Timestamp: ts,
Value: val,
Partial: isPartialValue(ts),
Partial: isPartialValue(ts, queryWindow, stepMs),
})
}
}
@@ -315,6 +293,120 @@ func readAsTimeSeries(rows driver.Rows, queryWindow *qbtypes.TimeRange, step qbt
}, nil
}
func isHeatmapBucketColumn(colName string) bool {
name := stripKeyAlias(colName)
return name == qbtypes.HeatmapBucketColumn || name == userHeatmapBucketColumn
}
// readAsHeatmap folds one row per cell — (timestamp, group labels, bucket upper
// bound, count) — into one series per group.
func readAsHeatmap(rows driver.Rows, queryWindow *qbtypes.TimeRange, step qbtypes.Step, queryName string) (*qbtypes.TimeSeriesData, error) {
colTypes := rows.ColumnTypes()
colNames := rows.Columns()
if !slices.ContainsFunc(colNames, isHeatmapBucketColumn) {
// there is no heatmap bucket column so empty response is returned.
return &qbtypes.TimeSeriesData{QueryName: queryName}, nil
}
slots := make([]any, len(colTypes))
for i, ct := range colTypes {
slots[i] = reflect.New(ct.ScanType()).Interface()
}
stepMs := uint64(step.Milliseconds())
accumulator := newHeatmapAccumulator()
for rows.Next() {
if err := rows.Scan(slots...); err != nil {
return nil, err
}
var (
ts int64
upperBound float64
count float64
lblVals []string
lblObjs []*qbtypes.Label
)
for idx, ptr := range slots {
name := stripKeyAlias(colNames[idx])
value := derefValue(ptr)
if t, ok := value.(time.Time); ok {
ts = t.UnixMilli()
continue
}
switch name {
case qbtypes.HeatmapBucketColumn, userHeatmapBucketColumn:
upperBound = numericAsFloat(value)
default:
if aggRe.MatchString(name) || slices.Contains(legacyReservedColumnTargetAliases, name) {
count = numericAsFloat(value)
continue
}
// a nullable label column comes back as a nil any, which would
// otherwise key the series on the literal "<nil>"
if value == nil {
value = ""
}
lblVals = append(lblVals, fmt.Sprint(value))
lblObjs = append(lblObjs, &qbtypes.Label{
Key: telemetrytypes.TelemetryFieldKey{Name: name},
Value: value,
})
}
}
if ts == 0 || !isValidBucketUpperBound(upperBound) || math.IsNaN(count) || math.IsInf(count, 0) {
continue
}
sort.Strings(lblVals)
labelsKey := strings.Join(lblVals, ",")
accumulator.addCell(labelsKey, lblObjs, ts, upperBound, count)
}
if err := rows.Err(); err != nil {
return nil, err
}
return accumulator.foldSeries(queryWindow, stepMs, queryName), nil
}
// isPartialValue reports whether the step interval starting at timestamp is only
// partly covered by the query window, which happens when the window boundaries
// are not step-aligned.
func isPartialValue(timestamp int64, queryWindow *qbtypes.TimeRange, stepMs uint64) bool {
if stepMs == 0 || queryWindow == nil {
return false
}
timestampMs := uint64(timestamp)
// For the first interval, check if query start is misaligned
// The first complete interval starts at the first timestamp >= queryWindow.From that is aligned to step
firstCompleteInterval := queryWindow.From
if queryWindow.From%stepMs != 0 {
// Round up to next step boundary
firstCompleteInterval = ((queryWindow.From / stepMs) + 1) * stepMs
}
// If timestamp is before the first complete interval, it's partial
if timestampMs < firstCompleteInterval {
return true
}
// For the last interval, check if it would extend beyond query end
if timestampMs+stepMs > queryWindow.To {
return queryWindow.To%stepMs != 0
}
return false
}
func isNumericKind(t reflect.Type) bool {
if t == nil {
return false

View File

@@ -0,0 +1,116 @@
package querier
import (
"math"
"slices"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
)
// heatmapColumn maps a bucket's upper bound to the count in it, holding one
// timestamp's cells. Keyed rather than indexed by band because the axis is only
// known once every cell has been seen.
type heatmapColumn map[float64]float64
func isValidBucketUpperBound(upperBound float64) bool {
return !math.IsNaN(upperBound) && !math.IsInf(upperBound, -1)
}
// heatmapSeries accumulates one group's columns while the rows are read.
type heatmapSeries struct {
labels []*qbtypes.Label
columnsByTimestamp map[int64]heatmapColumn
}
// heatmapAccumulator collects cells from either reader and folds them into one
// series per group.
type heatmapAccumulator struct {
seriesByKey map[string]*heatmapSeries
seriesOrder []string
upperBounds map[float64]struct{}
}
func newHeatmapAccumulator() *heatmapAccumulator {
return &heatmapAccumulator{
seriesByKey: map[string]*heatmapSeries{},
upperBounds: map[float64]struct{}{},
}
}
// addCell files one cell under the group labelsKey identifies, keeping the
// labels from the first cell seen for it.
func (a *heatmapAccumulator) addCell(labelsKey string, lbls []*qbtypes.Label, ts int64, upperBound, count float64) {
series, ok := a.seriesByKey[labelsKey]
if !ok {
series = &heatmapSeries{labels: lbls, columnsByTimestamp: map[int64]heatmapColumn{}}
a.seriesByKey[labelsKey] = series
a.seriesOrder = append(a.seriesOrder, labelsKey)
}
if series.columnsByTimestamp[ts] == nil {
series.columnsByTimestamp[ts] = heatmapColumn{}
}
series.columnsByTimestamp[ts][upperBound] += count
if !math.IsInf(upperBound, 1) {
a.upperBounds[upperBound] = struct{}{}
}
}
// foldSeries turns the collected cells into one series per group, in the order
// the groups first appeared.
func (a *heatmapAccumulator) foldSeries(queryWindow *qbtypes.TimeRange, stepMs uint64, queryName string) *qbtypes.TimeSeriesData {
if len(a.seriesOrder) == 0 {
return &qbtypes.TimeSeriesData{QueryName: queryName}
}
upperBounds := make([]float64, 0, len(a.upperBounds))
for upperBound := range a.upperBounds {
upperBounds = append(upperBounds, upperBound)
}
slices.Sort(upperBounds)
// the index past the last upper bound is where the +Inf overflow lands
upperBoundToIndex := make(map[float64]int, len(upperBounds)+1)
for index, upperBound := range upperBounds {
upperBoundToIndex[upperBound] = index
}
upperBoundToIndex[math.Inf(1)] = len(upperBounds)
bucket := &qbtypes.AggregationBucket{
Index: 0,
Alias: "__result_0",
Meta: qbtypes.AggregationMeta{Buckets: upperBounds},
Series: make([]*qbtypes.TimeSeries, 0, len(a.seriesOrder)),
}
for _, labelsKey := range a.seriesOrder {
accumulated := a.seriesByKey[labelsKey]
timestamps := make([]int64, 0, len(accumulated.columnsByTimestamp))
for ts := range accumulated.columnsByTimestamp {
timestamps = append(timestamps, ts)
}
slices.Sort(timestamps)
series := &qbtypes.TimeSeries{
Labels: accumulated.labels,
Values: make([]*qbtypes.TimeSeriesValue, 0, len(timestamps)),
}
for _, ts := range timestamps {
values := make([]float64, len(upperBounds)+1)
for upperBound, count := range accumulated.columnsByTimestamp[ts] {
values[upperBoundToIndex[upperBound]] = count
}
series.Values = append(series.Values, &qbtypes.TimeSeriesValue{
Timestamp: ts,
Values: values,
Partial: isPartialValue(ts, queryWindow, stepMs),
})
}
bucket.Series = append(bucket.Series, series)
}
return &qbtypes.TimeSeriesData{
QueryName: queryName,
Aggregations: []*qbtypes.AggregationBucket{bucket},
}
}

101
pkg/querier/heatmap_test.go Normal file
View File

@@ -0,0 +1,101 @@
package querier
import (
"testing"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestMergeTimeSeriesResultsUnionsHeatmapAxes(t *testing.T) {
// a log axis holds whichever bands the data reached, so a wide cached range
// and a narrow fresh one routinely disagree on which bands exist
cached := &qbtypes.TimeSeriesData{
QueryName: "A",
Aggregations: []*qbtypes.AggregationBucket{{
Index: 0,
Meta: qbtypes.AggregationMeta{Buckets: []float64{1, 4, 16}},
Series: []*qbtypes.TimeSeries{{
Labels: []*qbtypes.Label{{Key: telemetrytypes.TelemetryFieldKey{Name: "host.name"}, Value: "node-1"}},
Values: []*qbtypes.TimeSeriesValue{{Timestamp: 1710000000000, Values: []float64{1, 2, 3, 4}}},
}},
}},
}
fresh := []*qbtypes.Result{{
Value: &qbtypes.TimeSeriesData{
QueryName: "A",
Aggregations: []*qbtypes.AggregationBucket{{
Index: 0,
Meta: qbtypes.AggregationMeta{Buckets: []float64{2, 4}},
Series: []*qbtypes.TimeSeries{{
Labels: []*qbtypes.Label{{Key: telemetrytypes.TelemetryFieldKey{Name: "host.name"}, Value: "node-1"}},
Values: []*qbtypes.TimeSeriesValue{{Timestamp: 1710000060000, Values: []float64{5, 6, 7}}},
}},
}},
},
}}
merged := (&querier{}).mergeTimeSeriesResults(cached, fresh)
require.Len(t, merged.Aggregations, 1)
aggBucket := merged.Aggregations[0]
assert.Equal(t, []float64{1, 2, 4, 16}, aggBucket.Meta.Buckets)
require.Len(t, aggBucket.Series, 1)
require.Len(t, aggBucket.Series[0].Values, 2)
// the cached 16 band survives even though the fresh range never reached it
assert.Equal(t, []float64{1, 0, 2, 3, 4}, aggBucket.Series[0].Values[0].Values)
// and the fresh 2 band survives even though the cached range never had it
assert.Equal(t, []float64{0, 5, 6, 0, 7}, aggBucket.Series[0].Values[1].Values)
}
func TestTrimResultToFluxBoundaryKeepsTheHeatmapAxis(t *testing.T) {
cache := &bucketCache{logger: instrumentationtest.New().Logger()}
result := &qbtypes.Result{
Type: qbtypes.RequestTypeHeatmap,
Value: &qbtypes.TimeSeriesData{
Aggregations: []*qbtypes.AggregationBucket{{
Index: 0,
Alias: "__result_0",
Meta: qbtypes.AggregationMeta{Unit: "By", Buckets: []float64{1, 2, 4}},
Series: []*qbtypes.TimeSeries{{
Values: []*qbtypes.TimeSeriesValue{
{Timestamp: 1710000000000, Values: []float64{1, 2, 3, 4}},
},
}},
}},
},
}
trimmed := cache.trimResultToFluxBoundary(result, 1710000060000)
tsData, ok := trimmed.Value.(*qbtypes.TimeSeriesData)
require.True(t, ok)
require.Len(t, tsData.Aggregations, 1)
// the counts are positional against the axis, so a cached bucket that lost
// Meta.Buckets would be realigned from an empty axis and collapse into the
// overflow slot on the way back out
aggBucket := tsData.Aggregations[0]
assert.Equal(t, []float64{1, 2, 4}, aggBucket.Meta.Buckets)
assert.Equal(t, "By", aggBucket.Meta.Unit)
assert.Equal(t, "__result_0", aggBucket.Alias)
}
func TestRealignFromAnEmptyAxisCollapsesIntoTheOverflow(t *testing.T) {
// pins the behaviour the trim bug exposed: with no axis to read the counts
// against, everything lands in the overflow slot
aggBucket := &qbtypes.AggregationBucket{
Series: []*qbtypes.TimeSeries{{
Values: []*qbtypes.TimeSeriesValue{{Timestamp: 1710000000000, Values: []float64{7, 8, 9, 10}}},
}},
}
aggBucket.ReindexValuesToNewUpperBounds([]float64{1, 2, 4})
assert.Equal(t, []float64{0, 0, 0, 7}, aggBucket.Series[0].Values[0].Values)
}

View File

@@ -195,6 +195,16 @@ func postProcessBuilderQuery[T any](
return result
}
// resolveHeatmapBucketAxis brings the bucket axis to the resolution the caller
// asked for. Downscaling runs first so AddHeatmapBucketsWithNoCounts adds them
// at that resolution rather than the finer one ClickHouse bucketed at.
func resolveHeatmapBucketAxis(tsData *qbtypes.TimeSeriesData, bucketing qbtypes.HeatmapBucketing) {
if bucketing.Kind == qbtypes.BucketsKindLog {
qbtypes.DownscaleHeatmapResolution(tsData, bucketing.LogScale)
}
qbtypes.AddHeatmapBucketsWithNoCounts(tsData, bucketing)
}
// postProcessMetricQuery applies postprocessing to a metric query result.
func postProcessMetricQuery(
q *querier,
@@ -216,6 +226,12 @@ func postProcessMetricQuery(
}
}
if req.RequestType == qbtypes.RequestTypeHeatmap && config.HeatmapBucketing != nil {
if tsData, ok := result.Value.(*qbtypes.TimeSeriesData); ok {
resolveHeatmapBucketAxis(tsData, *config.HeatmapBucketing)
}
}
result = q.applySeriesLimit(result, query.Limit, query.Order)
if len(query.Functions) > 0 {
@@ -342,6 +358,19 @@ func (q *querier) applyFormulas(ctx context.Context, results map[string]*qbtypes
result = q.applySeriesLimit(result, formula.Limit, formula.Order)
results[name] = result
}
case qbtypes.RequestTypeHeatmap:
// The queries a formula reads were run as time series, so what
// arrives here is one value per group per timestamp.
result := q.processTimeSeriesFormula(ctx, results, formula, req)
if result != nil {
if tsData, ok := result.Value.(*qbtypes.TimeSeriesData); ok {
bucketing := req.BucketOptions.ToHeatmapBucketing()
bucketFormulaOutputAsHeatmap(tsData, bucketing)
resolveHeatmapBucketAxis(tsData, bucketing)
}
result = q.applySeriesLimit(result, formula.Limit, formula.Order)
results[name] = result
}
case qbtypes.RequestTypeScalar:
result := q.processScalarFormula(ctx, results, formula, req)
// For scalar results, apply limit by processScalarFormula itself since it needs to be applied before converting back to scalar format
@@ -410,6 +439,89 @@ func (q *querier) processTimeSeriesFormula(
return result
}
func bucketFormulaOutputAsHeatmap(tsData *qbtypes.TimeSeriesData, bucketing qbtypes.HeatmapBucketing) {
// A formula is one expression, so processTimeSeriesFormula gives it one
// aggregation.
if tsData == nil || len(tsData.Aggregations) == 0 || tsData.Aggregations[0] == nil {
return
}
aggBucket := tsData.Aggregations[0]
calculateUpperBound := calculateLogValueUpperBound
if bucketing.Kind == qbtypes.BucketsKindLinear {
calculateUpperBound = func(value float64) float64 {
return calculateLinearValueUpperBound(bucketing, value)
}
}
// +Inf is the open-above overflow rather than an upper bound of its own, and
// a NaN value has no bucket at all, so neither goes on the axis.
upperBounds := []float64{}
for _, series := range aggBucket.Series {
for _, point := range series.Values {
upperBound := calculateUpperBound(point.Value)
if !math.IsNaN(upperBound) && !math.IsInf(upperBound, 0) {
upperBounds = append(upperBounds, upperBound)
}
}
}
slices.Sort(upperBounds)
upperBounds = slices.Compact(upperBounds)
upperBoundToIndex := make(map[float64]int, len(upperBounds))
for index, upperBound := range upperBounds {
upperBoundToIndex[upperBound] = index
}
overflowIndex := len(upperBounds)
for _, series := range aggBucket.Series {
for _, point := range series.Values {
upperBound := calculateUpperBound(point.Value)
point.Values = make([]float64, overflowIndex+1)
point.Value = 0
switch {
case math.IsNaN(upperBound):
case math.IsInf(upperBound, 1):
point.Values[overflowIndex] = 1
default:
point.Values[upperBoundToIndex[upperBound]] = 1
}
}
}
aggBucket.Meta.Buckets = upperBounds
}
// calculateLinearValueUpperBound and calculateLogValueUpperBound are the Go side
// of what renderLinearUpperBoundExpr and renderLogUpperBoundExpr emit, and have
// to stay identical to them: a formula heatmap and a metric heatmap that
// disagreed here would put their counts in different buckets.
func calculateLinearValueUpperBound(bucketing qbtypes.HeatmapBucketing, value float64) float64 {
if value > bucketing.MaxValue {
return math.Inf(1)
}
numBuckets := float64(bucketing.NumBuckets)
index := math.Min(math.Max(math.Ceil(value*numBuckets/bucketing.MaxValue), 1), numBuckets)
return index * bucketing.MaxValue / numBuckets
}
// Like renderLogUpperBoundExpr, this reads MaxLogScale rather than the requested
// scale: ClickHouse buckets at the finest resolution and resolveHeatmapBucketAxis
// folds the axis down afterwards.
func calculateLogValueUpperBound(value float64) float64 {
if value <= 0 {
return 0
}
if value <= qbtypes.MinLogUpperBound {
return qbtypes.MinLogUpperBound
}
if value > qbtypes.MaxLogUpperBound {
return math.Inf(1)
}
bucketsPerDoubling := math.Exp2(qbtypes.MaxLogScale)
return math.Exp2(math.Ceil(math.Log2(value)*bucketsPerDoubling) / bucketsPerDoubling)
}
func (q *querier) processScalarFormula(
ctx context.Context,
results map[string]*qbtypes.Result,
@@ -494,7 +606,7 @@ func (q *querier) processScalarFormula(
bucket := &qbtypes.AggregationBucket{
Index: aggIdx,
Alias: scalarData.Columns[colIdx].Name,
Meta: scalarData.Columns[colIdx].Meta,
Meta: qbtypes.AggregationMeta{Unit: scalarData.Columns[colIdx].Meta.Unit},
Series: make([]*qbtypes.TimeSeries, 0),
}
@@ -667,13 +779,14 @@ func convertTimeSeriesDataToScalar(tsData *qbtypes.TimeSeriesData, queryName str
if name == "" {
name = fmt.Sprintf("__result_%d", agg.Index)
}
columns = append(columns, &qbtypes.ColumnDescriptor{
column := &qbtypes.ColumnDescriptor{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: name},
QueryName: queryName,
AggregationIndex: int64(agg.Index),
Meta: agg.Meta,
Type: qbtypes.ColumnTypeAggregation,
})
}
column.Meta.Unit = agg.Meta.Unit
columns = append(columns, column)
}
// Build rows.

View File

@@ -50,7 +50,7 @@ func (q *querier) QueryRangePreview(
env := []qbtypes.QueryEnvelope{req.CompositeQuery.Queries[idx]}
ps.Warnings = append(ps.Warnings, q.adjustStepInterval(env, req.Start, req.End)...)
missingMetricQueries, metricWarnings, mErr := q.resolveMetricMetadata(ctx, orgID, env, req.Start, req.End)
missingMetricQueries, metricWarnings, mErr := q.resolveMetricMetadata(ctx, orgID, env, req.Start, req.End, req.RequestType, req.BucketOptions)
if mErr != nil {
// Report this query's error but keep previewing the rest.
ps.Error = mErr

View File

@@ -0,0 +1,140 @@
package querier
import (
"fmt"
"math"
"slices"
"sort"
"strconv"
"strings"
"github.com/prometheus/prometheus/model/labels"
"github.com/prometheus/prometheus/promql"
"github.com/SigNoz/signoz/pkg/errors"
qbv5 "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// promHistogramBucketLabel is the label a classic histogram carries its
// cumulative upper bound on.
const promHistogramBucketLabel = "le"
// cumulativeColumn maps a bucket's upper bound to the cumulative count at it.
// Differencing turns it into the per-band counts a heatmapColumn holds.
type cumulativeColumn map[float64]float64
// promHeatmapGroup assembles one group across the several matrix series its `le`
// values arrive as, since differencing needs all of them.
type promHeatmapGroup struct {
labels []*qbv5.Label
labelsKey string
cumulative map[int64]cumulativeColumn
}
// foldMatrixAsHeatmap folds a matrix of one cumulative series per (group, `le`)
// into one series per group whose points hold a count per band.
func foldMatrixAsHeatmap(matrix promql.Matrix, queryWindow *qbv5.TimeRange, stepMs uint64, queryName string) (*qbv5.TimeSeriesData, error) {
groups, groupOrder := collectCumulativeGroups(matrix)
// An empty matrix is only ever the window having no data, but series that
// all lack `le` say the expression itself cannot draw a heatmap.
if len(matrix) > 0 && len(groups) == 0 {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput,
"the query returned no `le` labels to build a bucket axis from").
WithAdditional("Keep `le` through the aggregation, e.g. sum by (le) (rate(metric_bucket[5m]))")
}
accumulator := newHeatmapAccumulator()
for _, labelsKey := range groupOrder {
groups[labelsKey].addDifferencedCells(accumulator)
}
return accumulator.foldSeries(queryWindow, stepMs, queryName), nil
}
// collectCumulativeGroups reads the matrix into one group per label set. A series
// without `le` has no band to sit in, so an expression that dropped the label
// draws nothing.
func collectCumulativeGroups(matrix promql.Matrix) (groups map[string]*promHeatmapGroup, groupOrder []string) {
groups = map[string]*promHeatmapGroup{}
for _, promSeries := range matrix {
upperBound, ok := extractBucketUpperBound(promSeries.Metric)
if !ok {
continue
}
lbls, labelsKey := extractHeatmapGroup(promSeries.Metric)
group, ok := groups[labelsKey]
if !ok {
group = &promHeatmapGroup{labels: lbls, labelsKey: labelsKey, cumulative: map[int64]cumulativeColumn{}}
groups[labelsKey] = group
groupOrder = append(groupOrder, labelsKey)
}
for _, point := range promSeries.Floats {
// skipping widens the band above onto the next upper bound that has
// a count, which is what lagInFrame does with an absent row
if math.IsNaN(point.F) || math.IsInf(point.F, 0) {
continue
}
if group.cumulative[point.T] == nil {
group.cumulative[point.T] = cumulativeColumn{}
}
group.cumulative[point.T][upperBound] = point.F
}
}
return groups, groupOrder
}
func extractBucketUpperBound(metric labels.Labels) (float64, bool) {
raw := metric.Get(promHistogramBucketLabel)
if raw == "" {
return 0, false
}
upperBound, err := strconv.ParseFloat(raw, 64)
if err != nil || !isValidBucketUpperBound(upperBound) {
return 0, false
}
return upperBound, true
}
// extractHeatmapGroup returns a series' group labels — everything but `le`.
func extractHeatmapGroup(metric labels.Labels) ([]*qbv5.Label, string) {
lbls := make([]*qbv5.Label, 0, metric.Len())
pairs := make([]string, 0, metric.Len())
metric.Range(func(l labels.Label) {
if l.Name == promHistogramBucketLabel || excludePromLabel(l.Name) {
return
}
lbls = append(lbls, &qbv5.Label{
Key: telemetrytypes.TelemetryFieldKey{Name: l.Name},
Value: l.Value,
})
pairs = append(pairs, fmt.Sprintf("%s=%s", l.Name, l.Value))
})
sort.Strings(pairs)
return lbls, strings.Join(pairs, ",")
}
// each cell is its upper bound's cumulative count minus the one below it.
func (g *promHeatmapGroup) addDifferencedCells(accumulator *heatmapAccumulator) {
for ts, cumulative := range g.cumulative {
upperBounds := make([]float64, 0, len(cumulative))
for upperBound := range cumulative {
upperBounds = append(upperBounds, upperBound)
}
slices.Sort(upperBounds)
previous := float64(0)
for _, upperBound := range upperBounds {
accumulator.addCell(g.labelsKey, g.labels, ts, upperBound, math.Max(cumulative[upperBound]-previous, 0))
previous = cumulative[upperBound]
}
}
}

View File

@@ -0,0 +1,88 @@
package querier
import (
"log/slog"
"math"
"testing"
"time"
"github.com/prometheus/prometheus/model/labels"
"github.com/prometheus/prometheus/promql"
qbv5 "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// The cache key is the fingerprint alone, so two request types over one
// expression must not produce the same one — a time series payload served to a
// heatmap request has no axis and reads back as a single collapsed band.
func TestFingerprintSeparatesHeatmapFromTimeSeries(t *testing.T) {
fingerprintFor := func(requestType qbv5.RequestType) string {
q := &promqlQuery{
logger: slog.New(slog.DiscardHandler),
query: qbv5.PromQuery{Name: "A", Query: "sum by (le) (increase(signoz_latency_bucket[5m]))", Step: qbv5.Step{Duration: time.Minute}},
tr: qbv5.TimeRange{From: 1710000000000, To: 1710003600000},
requestType: requestType,
}
return q.Fingerprint()
}
heatmap := fingerprintFor(qbv5.RequestTypeHeatmap)
timeSeries := fingerprintFor(qbv5.RequestTypeTimeSeries)
assert.NotEmpty(t, heatmap, "a heatmap decomposes into time buckets like a time series")
assert.NotEqual(t, timeSeries, heatmap)
assert.Empty(t, fingerprintFor(qbv5.RequestTypeScalar), "a scalar result is its window's last point")
}
func TestFoldMatrixAsHeatmapClampsADecreasingCumulativeCount(t *testing.T) {
at := int64(1710000000000)
matrix := promql.Matrix{
{
Metric: labels.FromStrings("le", "5"),
Floats: []promql.FPoint{{T: at, F: 10}},
},
{
Metric: labels.FromStrings("le", "10"),
Floats: []promql.FPoint{{T: at, F: 4}},
},
}
data, err := foldMatrixAsHeatmap(matrix, &qbv5.TimeRange{From: 1710000000000, To: 1710000060000}, uint64(time.Minute.Milliseconds()), "A")
require.NoError(t, err)
require.Len(t, data.Aggregations, 1)
// a cumulative count that went backwards would difference to -6
assert.Equal(t, []float64{10, 0, 0}, data.Aggregations[0].Series[0].Values[0].Values)
}
func TestFoldMatrixAsHeatmapWidensTheBandOverAMissingUpperBound(t *testing.T) {
at := int64(1710000000000)
matrix := promql.Matrix{
{
Metric: labels.FromStrings("le", "5"),
Floats: []promql.FPoint{{T: at, F: 3}},
},
{
Metric: labels.FromStrings("le", "10"),
Floats: []promql.FPoint{{T: at, F: math.NaN()}},
},
{
Metric: labels.FromStrings("le", "20"),
Floats: []promql.FPoint{{T: at, F: 30}},
},
}
data, err := foldMatrixAsHeatmap(matrix, &qbv5.TimeRange{From: 1710000000000, To: 1710000060000}, uint64(time.Minute.Milliseconds()), "A")
require.NoError(t, err)
require.Len(t, data.Aggregations, 1)
aggregation := data.Aggregations[0]
// 10 carried nothing to difference against, so it is not on the axis at all
// and 20 differences against 5, holding what (5,10] and (10,20] would split
assert.Equal(t, []float64{5, 20}, aggregation.Meta.Buckets)
assert.Equal(t, []float64{3, 27, 0}, aggregation.Series[0].Values[0].Values)
}

View File

@@ -155,7 +155,10 @@ func (q *promqlQuery) Fingerprint() string {
if q.opts.serve != nil {
return ""
}
if q.requestType != qbv5.RequestTypeTimeSeries {
switch q.requestType {
case qbv5.RequestTypeTimeSeries, qbv5.RequestTypeHeatmap:
default:
return ""
}
@@ -166,6 +169,8 @@ func (q *promqlQuery) Fingerprint() string {
}
parts := []string{
"promql",
// one expression returns a different shape per request type
fmt.Sprintf("requestType=%s", q.requestType.StringValue()),
query,
q.query.Step.String(),
}
@@ -369,7 +374,7 @@ func (q *promqlQuery) Execute(ctx context.Context) (*qbv5.Result, error) {
}
return nil, err
}
return q.toResult(matrix, nil, began, &statsMu, &rowsScanned, &bytesScanned), nil
return q.toResult(matrix, nil, began, &statsMu, &rowsScanned, &bytesScanned)
}
// When the serving provider has the RangeExecutor capability
@@ -385,7 +390,7 @@ func (q *promqlQuery) Execute(ctx context.Context) (*qbv5.Result, error) {
return nil, err
}
if served {
return q.toResult(matrix, nil, began, &statsMu, &rowsScanned, &bytesScanned), nil
return q.toResult(matrix, nil, began, &statsMu, &rowsScanned, &bytesScanned)
}
}
@@ -446,28 +451,60 @@ func (q *promqlQuery) Execute(ctx context.Context) (*qbv5.Result, error) {
}
warnings, _ := res.Warnings.AsStrings(query, 10, 0)
return q.toResult(matrix, warnings, began, &statsMu, &rowsScanned, &bytesScanned), nil
return q.toResult(matrix, warnings, began, &statsMu, &rowsScanned, &bytesScanned)
}
// toResult converts an evaluated matrix into the v5 result shape, attaching
// the ClickHouse scan stats accumulated during evaluation.
func (q *promqlQuery) toResult(matrix promql.Matrix, warnings []string, began time.Time, statsMu *sync.Mutex, rowsScanned, bytesScanned *uint64) *qbv5.Result {
// Hide only known SigNoz storage keys: label names are user data and may
// legitimately start with "__" (e.g. __address__), so a blanket dunder
// strip mangles user labelsets. The __scope./__resource. prefixes cover
// every exporter version's keys.
excludeLabel := func(labelName string) bool {
return labelName == "__temporality__" ||
strings.HasPrefix(labelName, "__scope.") ||
strings.HasPrefix(labelName, "__resource.")
// excludePromLabel hides only known SigNoz storage keys: label names are user
// data and may legitimately start with "__" (e.g. __address__), so a blanket
// dunder strip mangles user labelsets. The __scope./__resource. prefixes cover
// every exporter version's keys.
func excludePromLabel(labelName string) bool {
return labelName == "__temporality__" ||
strings.HasPrefix(labelName, "__scope.") ||
strings.HasPrefix(labelName, "__resource.")
}
// collectExecStats snapshots the scan counters a query accumulated. Callers take
// it at the point they are done with the matrix, so the duration covers the
// shaping they did.
func collectExecStats(began time.Time, statsMu *sync.Mutex, rowsScanned, bytesScanned *uint64) qbv5.ExecStats {
statsMu.Lock()
defer statsMu.Unlock()
return qbv5.ExecStats{
RowsScanned: *rowsScanned,
BytesScanned: *bytesScanned,
DurationMS: uint64(time.Since(began).Milliseconds()),
}
}
func (q *promqlQuery) toResult(matrix promql.Matrix, warnings []string, began time.Time, statsMu *sync.Mutex, rowsScanned, bytesScanned *uint64) (*qbv5.Result, error) {
if q.requestType == qbv5.RequestTypeHeatmap {
return q.toResultForHeatmap(matrix, warnings, began, statsMu, rowsScanned, bytesScanned)
}
return q.toResultForTimeSeriesAndScalar(matrix, warnings, began, statsMu, rowsScanned, bytesScanned), nil
}
func (q *promqlQuery) toResultForHeatmap(matrix promql.Matrix, warnings []string, began time.Time, statsMu *sync.Mutex, rowsScanned, bytesScanned *uint64) (*qbv5.Result, error) {
tsData, err := foldMatrixAsHeatmap(matrix, &q.tr, uint64(q.query.Step.Milliseconds()), q.query.Name)
if err != nil {
return nil, err
}
return &qbv5.Result{
Type: q.requestType,
Value: tsData,
Warnings: warnings,
Stats: collectExecStats(began, statsMu, rowsScanned, bytesScanned),
}, nil
}
func (q *promqlQuery) toResultForTimeSeriesAndScalar(matrix promql.Matrix, warnings []string, began time.Time, statsMu *sync.Mutex, rowsScanned, bytesScanned *uint64) *qbv5.Result {
var series []*qbv5.TimeSeries
for _, v := range matrix {
var s qbv5.TimeSeries
lbls := make([]*qbv5.Label, 0, v.Metric.Len())
v.Metric.Range(func(l labels.Label) {
if excludeLabel(l.Name) {
if excludePromLabel(l.Name) {
return
}
lbls = append(lbls, &qbv5.Label{
@@ -495,13 +532,7 @@ func (q *promqlQuery) toResult(matrix promql.Matrix, warnings []string, began ti
series = append(series, &s)
}
statsMu.Lock()
stats := qbv5.ExecStats{
RowsScanned: *rowsScanned,
BytesScanned: *bytesScanned,
DurationMS: uint64(time.Since(began).Milliseconds()),
}
statsMu.Unlock()
stats := collectExecStats(began, statsMu, rowsScanned, bytesScanned)
tsData := &qbv5.TimeSeriesData{QueryName: q.query.Name}
// No bucket at all when nothing survived: a bucket holding no series reads

View File

@@ -495,7 +495,8 @@ func TestToResultDropsNonFiniteValues(t *testing.T) {
var mu sync.Mutex
var rows, bytes uint64
result := q.toResult(matrix, nil, time.Now(), &mu, &rows, &bytes)
result, err := q.toResult(matrix, nil, time.Now(), &mu, &rows, &bytes)
require.NoError(t, err)
tsData, ok := result.Value.(*qbv5.TimeSeriesData)
require.True(t, ok)
@@ -526,7 +527,9 @@ func TestToResultDropsSeriesAndBucketLeftEmpty(t *testing.T) {
var mu sync.Mutex
var rows, bytes uint64
tsData, ok := q.toResult(matrix, nil, time.Now(), &mu, &rows, &bytes).Value.(*qbv5.TimeSeriesData)
result, err := q.toResult(matrix, nil, time.Now(), &mu, &rows, &bytes)
require.NoError(t, err)
tsData, ok := result.Value.(*qbv5.TimeSeriesData)
require.True(t, ok)
require.Len(t, tsData.Aggregations, 1)
require.Len(t, tsData.Aggregations[0].Series, 1, "the all-NaN series is gone")
@@ -535,7 +538,9 @@ func TestToResultDropsSeriesAndBucketLeftEmpty(t *testing.T) {
allNaN := promql.Matrix{
{Metric: labels.FromStrings("job_name", "idleJob"), Floats: []promql.FPoint{{T: 1000, F: math.NaN()}}},
}
tsData, ok = q.toResult(allNaN, nil, time.Now(), &mu, &rows, &bytes).Value.(*qbv5.TimeSeriesData)
result, err = q.toResult(allNaN, nil, time.Now(), &mu, &rows, &bytes)
require.NoError(t, err)
tsData, ok = result.Value.(*qbv5.TimeSeriesData)
require.True(t, ok)
assert.Empty(t, tsData.Aggregations)
}

View File

@@ -156,7 +156,7 @@ func (q *querier) QueryRange(ctx context.Context, orgID valuer.UUID, req *qbtype
// We need to set if it is unspecified or adjust it if value is not within recommended range
intervalWarnings := q.adjustStepInterval(req.CompositeQuery.Queries, req.Start, req.End)
missingMetricQueries, metricWarnings, err := q.resolveMetricMetadata(ctx, orgID, req.CompositeQuery.Queries, req.Start, req.End)
missingMetricQueries, metricWarnings, err := q.resolveMetricMetadata(ctx, orgID, req.CompositeQuery.Queries, req.Start, req.End, req.RequestType, req.BucketOptions)
if err != nil {
return nil, err
}
@@ -177,7 +177,7 @@ func (q *querier) QueryRange(ctx context.Context, orgID valuer.UUID, req *qbtype
preseededResults := make(map[string]any)
for _, name := range missingMetricQueries {
switch req.RequestType {
case qbtypes.RequestTypeTimeSeries:
case qbtypes.RequestTypeTimeSeries, qbtypes.RequestTypeHeatmap:
preseededResults[name] = &qbtypes.TimeSeriesData{QueryName: name}
case qbtypes.RequestTypeScalar:
preseededResults[name] = &qbtypes.ScalarData{QueryName: name}
@@ -334,15 +334,22 @@ func (q *querier) buildQueries(
if missingMetricQuerySet[spec.Name] {
continue
}
requestType := req.RequestType
if requestType == qbtypes.RequestTypeHeatmap && spec.Disabled {
// A disabled query in a heatmap request feeds a formula, and the
// formula converts time series into heatmap data, so its inputs
// run as time series queries.
requestType = qbtypes.RequestTypeTimeSeries
}
spec.ShiftBy = extractShiftFromBuilderQuery(spec)
timeRange := adjustTimeRangeForShift(spec, qbtypes.TimeRange{From: req.Start, To: req.End}, req.RequestType)
timeRange := adjustTimeRangeForShift(spec, qbtypes.TimeRange{From: req.Start, To: req.End}, requestType)
var bq *builderQuery[qbtypes.MetricAggregation]
if spec.Source == telemetrytypes.SourceMeter {
event.Source = telemetrytypes.SourceMeter.StringValue()
bq = newBuilderQuery(q.logger, q.telemetryStore, orgID, q.meterStmtBuilder, query.Type, spec, timeRange, req.RequestType, tmplVars, builderConfig{})
bq = newBuilderQuery(q.logger, q.telemetryStore, orgID, q.meterStmtBuilder, query.Type, spec, timeRange, requestType, tmplVars, builderConfig{})
} else {
bq = newBuilderQuery(q.logger, q.telemetryStore, orgID, q.metricStmtBuilder, query.Type, spec, timeRange, req.RequestType, tmplVars, builderConfig{})
bq = newBuilderQuery(q.logger, q.telemetryStore, orgID, q.metricStmtBuilder, query.Type, spec, timeRange, requestType, tmplVars, builderConfig{})
}
queries[spec.Name] = bq
@@ -415,7 +422,7 @@ func (q *querier) populateQBEvent(event *qbtypes.QBEvent, queries []qbtypes.Quer
// resolved: never-seen metrics and dormant metrics (seen but no data in
// the query window).
// - err: Internal when a metadata fetch fails.
func (q *querier) resolveMetricMetadata(ctx context.Context, orgID valuer.UUID, queries []qbtypes.QueryEnvelope, start, end uint64) (missingMetricQueries []string, metricWarnings []string, err error) {
func (q *querier) resolveMetricMetadata(ctx context.Context, orgID valuer.UUID, queries []qbtypes.QueryEnvelope, start, end uint64, requestType qbtypes.RequestType, bucketOptions *qbtypes.BucketOptions) (missingMetricQueries []string, metricWarnings []string, err error) {
metricNames := make([]string, 0)
for idx := range queries {
if queries[idx].Type != qbtypes.QueryTypeBuilder {
@@ -473,6 +480,13 @@ func (q *querier) resolveMetricMetadata(ctx context.Context, orgID valuer.UUID,
if err := spec.Aggregations[i].ValidateForTypeAndTemporality(); err != nil {
return nil, nil, err
}
// Only the enabled query is used to render the heatmap, so bucket
// options are only applied to the enabled query.
if requestType == qbtypes.RequestTypeHeatmap && !spec.Disabled {
if err := spec.Aggregations[i].VerifyAndApplyBucketOptions(bucketOptions); err != nil {
return nil, nil, err
}
}
if reducedMetricsSet[spec.Aggregations[i].MetricName] {
spec.Aggregations[i].Reduced = true
}
@@ -679,7 +693,7 @@ func (q *querier) run(
if val, ok := result.Value.(*qbtypes.RawData); ok && val != nil {
return len(val.Rows) != 0
}
case qbtypes.RequestTypeTimeSeries:
case qbtypes.RequestTypeTimeSeries, qbtypes.RequestTypeHeatmap:
if val, ok := result.Value.(*qbtypes.TimeSeriesData); ok && val != nil {
if len(val.Aggregations) != 0 {
anyNonEmpty := false
@@ -1000,7 +1014,7 @@ func (q *querier) mergeResults(cached *qbtypes.Result, fresh []*qbtypes.Result)
// Merge all fresh results including the first one
switch merged.Type {
case qbtypes.RequestTypeTimeSeries:
case qbtypes.RequestTypeTimeSeries, qbtypes.RequestTypeHeatmap:
// Pass nil as cached value to ensure proper merging of all fresh results
merged.Value = q.mergeTimeSeriesResults(nil, fresh)
}
@@ -1023,7 +1037,7 @@ func (q *querier) mergeResults(cached *qbtypes.Result, fresh []*qbtypes.Result)
}
switch merged.Type {
case qbtypes.RequestTypeTimeSeries:
case qbtypes.RequestTypeTimeSeries, qbtypes.RequestTypeHeatmap:
merged.Value = q.mergeTimeSeriesResults(cached.Value.(*qbtypes.TimeSeriesData), fresh)
}
@@ -1044,6 +1058,16 @@ func (q *querier) mergeResults(cached *qbtypes.Result, fresh []*qbtypes.Result)
return merged
}
func mergeBucketUpperBounds(cachedValue *qbtypes.TimeSeriesData, freshResults []*qbtypes.Result) map[int][]float64 {
upperBoundSources := make([]*qbtypes.TimeSeriesData, 0, len(freshResults)+1)
upperBoundSources = append(upperBoundSources, cachedValue)
for _, result := range freshResults {
freshTS, _ := result.Value.(*qbtypes.TimeSeriesData)
upperBoundSources = append(upperBoundSources, freshTS)
}
return qbtypes.MergeBucketUpperBounds(upperBoundSources...)
}
// mergeTimeSeriesResults merges time series data.
func (q *querier) mergeTimeSeriesResults(cachedValue *qbtypes.TimeSeriesData, freshResults []*qbtypes.Result) *qbtypes.TimeSeriesData {
@@ -1052,12 +1076,15 @@ func (q *querier) mergeTimeSeriesResults(cachedValue *qbtypes.TimeSeriesData, fr
// Map to store aggregation bucket metadata
bucketMetadata := make(map[int]*qbtypes.AggregationBucket)
mergedUpperBounds := mergeBucketUpperBounds(cachedValue, freshResults)
// Process cached data if available
if cachedValue != nil && cachedValue.Aggregations != nil {
for _, aggBucket := range cachedValue.Aggregations {
if seriesMap[aggBucket.Index] == nil {
seriesMap[aggBucket.Index] = make(map[string]*qbtypes.TimeSeries)
}
aggBucket.ReindexValuesToNewUpperBounds(mergedUpperBounds[aggBucket.Index])
if bucketMetadata[aggBucket.Index] == nil {
bucketMetadata[aggBucket.Index] = aggBucket
}
@@ -1109,6 +1136,7 @@ func (q *querier) mergeTimeSeriesResults(cachedValue *qbtypes.TimeSeriesData, fr
}
for _, aggBucket := range freshTS.Aggregations {
aggBucket.ReindexValuesToNewUpperBounds(mergedUpperBounds[aggBucket.Index])
for _, series := range aggBucket.Series {
key := qbtypes.GetUniqueSeriesKey(series.Labels)

View File

@@ -4070,20 +4070,20 @@ func (aH *APIHandler) RegisterTraceFunnelsRoutes(router *mux.Router, am *middlew
Methods(http.MethodPut)
// Analytics endpoints
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/validate", aH.handleValidateTraces).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/overview", aH.handleFunnelAnalytics).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/steps", aH.handleStepAnalytics).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/steps/overview", aH.handleFunnelStepAnalytics).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/slow-traces", aH.handleFunnelSlowTraces).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/error-traces", aH.handleFunnelErrorTraces).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/validate", am.ViewAccess(aH.handleValidateTraces)).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/overview", am.ViewAccess(aH.handleFunnelAnalytics)).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/steps", am.ViewAccess(aH.handleStepAnalytics)).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/steps/overview", am.ViewAccess(aH.handleFunnelStepAnalytics)).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/slow-traces", am.ViewAccess(aH.handleFunnelSlowTraces)).Methods("POST")
traceFunnelsRouter.HandleFunc("/{funnel_id}/analytics/error-traces", am.ViewAccess(aH.handleFunnelErrorTraces)).Methods("POST")
// Analytics endpoints
traceFunnelsRouter.HandleFunc("/analytics/validate", aH.handleValidateTracesWithPayload).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/overview", aH.handleFunnelAnalyticsWithPayload).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/steps", aH.handleStepAnalyticsWithPayload).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/steps/overview", aH.handleFunnelStepAnalyticsWithPayload).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/slow-traces", aH.handleFunnelSlowTracesWithPayload).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/error-traces", aH.handleFunnelErrorTracesWithPayload).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/validate", am.ViewAccess(aH.handleValidateTracesWithPayload)).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/overview", am.ViewAccess(aH.handleFunnelAnalyticsWithPayload)).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/steps", am.ViewAccess(aH.handleStepAnalyticsWithPayload)).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/steps/overview", am.ViewAccess(aH.handleFunnelStepAnalyticsWithPayload)).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/slow-traces", am.ViewAccess(aH.handleFunnelSlowTracesWithPayload)).Methods("POST")
traceFunnelsRouter.HandleFunc("/analytics/error-traces", am.ViewAccess(aH.handleFunnelErrorTracesWithPayload)).Methods("POST")
}
func (aH *APIHandler) handleValidateTraces(w http.ResponseWriter, r *http.Request) {

View File

@@ -129,7 +129,7 @@ func (b *meterQueryStatementBuilder) buildPipelineStatement(
}
// final SELECT
return b.metricsStatementBuilder.BuildFinalSelect(cteFragments, cteArgs, query)
return b.metricsStatementBuilder.BuildFinalSelect(cteFragments, cteArgs, qbtypes.RequestTypeTimeSeries, query)
}
func (b *meterQueryStatementBuilder) buildTemporalAggDeltaFastPath(

View File

@@ -122,7 +122,7 @@ func TestReducedStatementBuilder(t *testing.T) {
name: "histogram_p99",
query: reducedQuery("test.metric.bucket", metrictypes.HistogramType, metrictypes.Cumulative, metrictypes.TimeAggregationUnspecified, metrictypes.SpaceAggregationPercentile99),
expected: qbtypes.Statement{
Query: "SELECT * FROM (WITH __temporal_aggregation_cte AS (SELECT ts, `le`, multiIf(row_number() OVER rate_window = 1, nan, (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) < 0, per_series_value / (ts - lagInFrame(ts, 1) OVER rate_window), (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) / (ts - lagInFrame(ts, 1) OVER rate_window)) AS per_series_value FROM (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(300)) AS ts, `le`, max(max) AS per_series_value FROM signoz_metrics.distributed_samples_v4_agg_5m AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_1day WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `le` ORDER BY fingerprint, ts) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)), __spatial_aggregation_cte AS (SELECT ts, `le`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `le`) SELECT ts, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.990) AS value FROM __spatial_aggregation_cte GROUP BY ts ORDER BY ts) UNION ALL SELECT * FROM (WITH __spatial_aggregation_cte AS (SELECT toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(300)) AS ts, `le`, sum(`sum`) / 300 AS value FROM signoz_metrics.distributed_samples_v4_reduced_sum_60s AS points FINAL INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_reduced WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? GROUP BY fingerprint, `le`) AS filtered_time_series ON points.reduced_fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY ts, `le`) SELECT ts, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.990) AS value FROM __spatial_aggregation_cte GROUP BY ts ORDER BY ts) ORDER BY ts SETTINGS do_not_merge_across_partitions_select_final = 1, optimize_move_to_prewhere_if_final = 1",
Query: "SELECT * FROM (WITH __temporal_aggregation_cte AS (SELECT ts, `le`, multiIf(row_number() OVER rate_window = 1, nan, (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) < 0, per_series_value, per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) AS per_series_value FROM (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(300)) AS ts, `le`, max(max) AS per_series_value FROM signoz_metrics.distributed_samples_v4_agg_5m AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_1day WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `le` ORDER BY fingerprint, ts) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)), __spatial_aggregation_cte AS (SELECT ts, `le`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `le`) SELECT ts, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.990) AS value FROM __spatial_aggregation_cte GROUP BY ts ORDER BY ts) UNION ALL SELECT * FROM (WITH __spatial_aggregation_cte AS (SELECT toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(300)) AS ts, `le`, sum(`sum`) AS value FROM signoz_metrics.distributed_samples_v4_reduced_sum_60s AS points FINAL INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_reduced WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? GROUP BY fingerprint, `le`) AS filtered_time_series ON points.reduced_fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY ts, `le`) SELECT ts, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.990) AS value FROM __spatial_aggregation_cte GROUP BY ts ORDER BY ts) ORDER BY ts SETTINGS do_not_merge_across_partitions_select_final = 1, optimize_move_to_prewhere_if_final = 1",
Args: []any{"test.metric.bucket", uint64(1746921600000), uint64(1747172760000), "cumulative", "test.metric.bucket", uint64(1746999900000), uint64(1747172760000), 0, "test.metric.bucket", uint64(1746997200000), uint64(1747172760000), "test.metric.bucket", uint64(1746999900000), uint64(1747172760000)},
},
},
@@ -130,7 +130,7 @@ func TestReducedStatementBuilder(t *testing.T) {
name: "histogram_p99_group_by",
query: withGroupBy(reducedQuery("test.metric.bucket", metrictypes.HistogramType, metrictypes.Cumulative, metrictypes.TimeAggregationUnspecified, metrictypes.SpaceAggregationPercentile99), "service.name"),
expected: qbtypes.Statement{
Query: "SELECT * FROM (WITH __temporal_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, `le`, multiIf(row_number() OVER rate_window = 1, nan, (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) < 0, per_series_value / (ts - lagInFrame(ts, 1) OVER rate_window), (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) / (ts - lagInFrame(ts, 1) OVER rate_window)) AS per_series_value FROM (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(300)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, max(max) AS per_series_value FROM signoz_metrics.distributed_samples_v4_agg_5m AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_1day WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_service.name`, `le` ORDER BY fingerprint, ts) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.990) AS value FROM __spatial_aggregation_cte GROUP BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY `__GROUP_BY_KEY_0_service.name`, ts) UNION ALL SELECT * FROM (WITH __spatial_aggregation_cte AS (SELECT toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(300)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(`sum`) / 300 AS value FROM signoz_metrics.distributed_samples_v4_reduced_sum_60s AS points FINAL INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_reduced WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.reduced_fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.990) AS value FROM __spatial_aggregation_cte GROUP BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY `__GROUP_BY_KEY_0_service.name`, ts) ORDER BY `__GROUP_BY_KEY_0_service.name`, ts SETTINGS do_not_merge_across_partitions_select_final = 1, optimize_move_to_prewhere_if_final = 1",
Query: "SELECT * FROM (WITH __temporal_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, `le`, multiIf(row_number() OVER rate_window = 1, nan, (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) < 0, per_series_value, per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) AS per_series_value FROM (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(300)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, max(max) AS per_series_value FROM signoz_metrics.distributed_samples_v4_agg_5m AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_1day WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_service.name`, `le` ORDER BY fingerprint, ts) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.990) AS value FROM __spatial_aggregation_cte GROUP BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY `__GROUP_BY_KEY_0_service.name`, ts) UNION ALL SELECT * FROM (WITH __spatial_aggregation_cte AS (SELECT toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(300)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(`sum`) AS value FROM signoz_metrics.distributed_samples_v4_reduced_sum_60s AS points FINAL INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_reduced WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.reduced_fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.990) AS value FROM __spatial_aggregation_cte GROUP BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY `__GROUP_BY_KEY_0_service.name`, ts) ORDER BY `__GROUP_BY_KEY_0_service.name`, ts SETTINGS do_not_merge_across_partitions_select_final = 1, optimize_move_to_prewhere_if_final = 1",
Args: []any{"test.metric.bucket", uint64(1746921600000), uint64(1747172760000), "cumulative", "test.metric.bucket", uint64(1746999900000), uint64(1747172760000), 0, "test.metric.bucket", uint64(1746997200000), uint64(1747172760000), "test.metric.bucket", uint64(1746999900000), uint64(1747172760000)},
},
},

View File

@@ -4,9 +4,13 @@ import (
"context"
"fmt"
"log/slog"
"math"
"slices"
"strconv"
"strings"
"time"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
@@ -113,7 +117,7 @@ func (b *StatementBuilder) Build(
orgID valuer.UUID,
start uint64,
end uint64,
_ qbtypes.RequestType,
requestType qbtypes.RequestType,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
@@ -125,13 +129,14 @@ func (b *StatementBuilder) Build(
start, end = querybuilder.AdjustedMetricTimeRange(start, end, uint64(query.StepInterval.Seconds()), query)
return b.buildPipelineStatement(ctx, orgID, start, end, query, keys, variables)
return b.buildPipelineStatement(ctx, orgID, start, end, requestType, query, keys, variables)
}
func (b *StatementBuilder) buildPipelineStatement(
ctx context.Context,
orgID valuer.UUID,
start, end uint64,
requestType qbtypes.RequestType,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
keys map[string][]*telemetrytypes.TelemetryFieldKey,
variables map[string]qbtypes.VariableItem,
@@ -144,7 +149,7 @@ func (b *StatementBuilder) buildPipelineStatement(
cteQuery := query
if query.Aggregations[0].Type == metrictypes.HistogramType {
query.GroupBy = slices.DeleteFunc(slices.Clone(query.GroupBy), isHistogramBucket)
cteQuery = histogramCTEQuery(query)
cteQuery = rewriteQueryForHistogramCTE(query)
}
agg := cteQuery.Aggregations[0]
@@ -216,7 +221,7 @@ func (b *StatementBuilder) buildPipelineStatement(
}
}
mainStmt, err := b.BuildFinalSelect(cteFragments, cteArgs, query)
mainStmt, err := b.BuildFinalSelect(cteFragments, cteArgs, requestType, query)
if err != nil {
return nil, err
}
@@ -224,13 +229,27 @@ func (b *StatementBuilder) buildPipelineStatement(
if reducedFragments == nil {
return mainStmt, nil
}
reducedStmt, err := b.BuildFinalSelect(reducedFragments, reducedArgs, query)
reducedStmt, err := b.BuildFinalSelect(reducedFragments, reducedArgs, requestType, query)
if err != nil {
return nil, err
}
return unionStatements(mainStmt, reducedStmt, query)
}
func rewriteQueryForHistogramCTE(query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]) qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation] {
query.GroupBy = append(slices.Clone(query.GroupBy), qbtypes.GroupByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: histogramBucketKey},
})
// The CTE yields one observation count per `le` per step. The space
// aggregation the caller asked for is applied downstream, over the `le` array.
query.Aggregations = slices.Clone(query.Aggregations)
query.Aggregations[0].TimeAggregation = metrictypes.TimeAggregationIncrease
query.Aggregations[0].SpaceAggregation = metrictypes.SpaceAggregationSum
return query
}
func unionStatements(main, reduced *qbtypes.Statement, query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]) (*qbtypes.Statement, error) {
orderBy := "ts"
for i, g := range query.GroupBy {
@@ -758,11 +777,9 @@ func (b *StatementBuilder) buildSpatialAggregationCTE(
func (b *StatementBuilder) BuildFinalSelect(
cteFragments []string,
cteArgs [][]any,
requestType qbtypes.RequestType,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
) (*qbtypes.Statement, error) {
metricType := query.Aggregations[0].Type
spaceAgg := query.Aggregations[0].SpaceAggregation
combined := querybuilder.CombineCTEs(cteFragments)
var args []any
@@ -770,6 +787,22 @@ func (b *StatementBuilder) BuildFinalSelect(
args = append(args, a...)
}
if requestType == qbtypes.RequestTypeHeatmap {
return buildHeatmapFinalSelect(combined, args, query)
}
return buildAggregationFinalSelect(combined, args, query)
}
// buildAggregationFinalSelect reads __spatial_aggregation_cte as one value per
// (group, timestamp), which is what every request type but heatmap wants.
func buildAggregationFinalSelect(
combined string,
args []any,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
) (*qbtypes.Statement, error) {
metricType := query.Aggregations[0].Type
spaceAgg := query.Aggregations[0].SpaceAggregation
sb := sqlbuilder.NewSelectBuilder()
if metricType == metrictypes.HistogramType && spaceAgg.IsPercentile() {
@@ -842,24 +875,129 @@ func (b *StatementBuilder) BuildFinalSelect(
return &qbtypes.Statement{Query: combined + q, Args: append(args, a...)}, nil
}
const histogramBucketKey = "le"
const (
histogramBucketKey = "le"
heatmapValueAlias = "__result_0"
heatmapWindow = "__heatmap_window"
)
func isHistogramBucket(k qbtypes.GroupByKey) bool { return k.Name == histogramBucketKey }
func histogramCTEQuery(query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]) qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation] {
query.GroupBy = append(slices.Clone(query.GroupBy), qbtypes.GroupByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{Name: histogramBucketKey},
})
query.Aggregations = slices.Clone(query.Aggregations)
if query.Aggregations[0].SpaceAggregation.IsPercentile() {
query.Aggregations[0].TimeAggregation = metrictypes.TimeAggregationRate
} else {
query.Aggregations[0].TimeAggregation = metrictypes.TimeAggregationIncrease
// buildHeatmapFinalSelect turns __spatial_aggregation_cte into one row per
// heatmap cell: (ts, group labels..., bucket upper bound, count).
func buildHeatmapFinalSelect(
combined string,
args []any,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
) (*qbtypes.Statement, error) {
if query.Aggregations[0].Type == metrictypes.HistogramType {
return buildHistogramHeatmapFinalSelect(combined, args, query)
}
query.Aggregations[0].SpaceAggregation = metrictypes.SpaceAggregationSum
return buildValueHeatmapFinalSelect(combined, args, query)
}
return query
// buildHistogramHeatmapFinalSelect differences the cumulative per-`le` counts in
// __spatial_aggregation_cte into a count per band. The upper bound reported is the
// `le` itself, so the `le=+Inf` row reaches the reader as an infinite upper bound
// for it to fold into the overflow band.
func buildHistogramHeatmapFinalSelect(
combined string,
args []any,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
) (*qbtypes.Statement, error) {
groupAliases := GroupByAliases(query.GroupBy)
partitionBy := append(append([]string{}, groupAliases...), "ts")
sb := sqlbuilder.NewSelectBuilder()
sb.Select("ts")
sb.SelectMore(groupAliases...)
sb.SelectMore(fmt.Sprintf("toFloat64(%s) AS %s", histogramBucketKey, qbtypes.HeatmapBucketColumn))
// a partial scrape can break monotonicity across `le`, and a negative cell
// count has no meaning
sb.SelectMore(fmt.Sprintf(
"greatest(value - lagInFrame(value, 1, 0) OVER %s, 0) AS %s",
heatmapWindow, heatmapValueAlias,
))
// sqlbuilder has no WINDOW clause; appending it to FROM lands it between FROM
// and ORDER BY, since these statements carry no WHERE or GROUP BY
sb.From(fmt.Sprintf(
"__spatial_aggregation_cte WINDOW %s AS (PARTITION BY %s ORDER BY toFloat64(%s))",
heatmapWindow, strings.Join(partitionBy, ", "), histogramBucketKey,
))
sb.OrderBy(groupAliases...)
sb.OrderBy("ts", fmt.Sprintf("toFloat64(%s)", histogramBucketKey))
q, a := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
return &qbtypes.Statement{Query: combined + q, Args: append(args, a...)}, nil
}
// buildValueHeatmapFinalSelect places each spatially aggregated value in a band
// of the requested axis. __spatial_aggregation_cte holds one row per (group,
// timestamp), so every cell counts exactly one.
func buildValueHeatmapFinalSelect(
combined string,
args []any,
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
) (*qbtypes.Statement, error) {
upperBound, err := renderHeatmapUpperBoundExpr(*query.Aggregations[0].HeatmapBucketing)
if err != nil {
return nil, err
}
groupAliases := GroupByAliases(query.GroupBy)
sb := sqlbuilder.NewSelectBuilder()
sb.Select("ts")
sb.SelectMore(groupAliases...)
sb.SelectMore(fmt.Sprintf("%s AS %s", upperBound, qbtypes.HeatmapBucketColumn))
sb.SelectMore(fmt.Sprintf("toFloat64(1) AS %s", heatmapValueAlias))
sb.From("__spatial_aggregation_cte")
sb.OrderBy(groupAliases...)
sb.OrderBy("ts", qbtypes.HeatmapBucketColumn)
q, a := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
return &qbtypes.Statement{Query: combined + q, Args: append(args, a...)}, nil
}
// renderHeatmapUpperBoundExpr renders the upper bound of the band `value` falls in.
func renderHeatmapUpperBoundExpr(bucketing qbtypes.HeatmapBucketing) (string, error) {
switch bucketing.Kind {
case qbtypes.BucketsKindLinear:
return renderLinearUpperBoundExpr(bucketing), nil
case qbtypes.BucketsKindLog:
return renderLogUpperBoundExpr(), nil
default:
return "", errors.NewInvalidInputf(errors.CodeInvalidInput,
"unsupported bucketsScaling %q for heatmap requests", bucketing.Kind.StringValue())
}
}
func renderLinearUpperBoundExpr(bucketing qbtypes.HeatmapBucketing) string {
maxValue := formatFloat(bucketing.MaxValue)
numBuckets := strconv.Itoa(bucketing.NumBuckets)
return fmt.Sprintf(
"multiIf(value > %s, toFloat64('+Inf'), least(greatest(ceil(value * %s / %s), 1), %s) * %s / %s)",
maxValue, numBuckets, maxValue, numBuckets, maxValue, numBuckets,
)
}
// ClickHouse buckets at MaxLogScale whatever HeatmapBucketing.LogScale asks for;
// postprocessing folds the axis down afterwards.
func renderLogUpperBoundExpr() string {
bandsPerDoubling := formatFloat(math.Exp2(qbtypes.MaxLogScale))
lowest := formatFloat(qbtypes.MinLogUpperBound)
highest := formatFloat(qbtypes.MaxLogUpperBound)
return fmt.Sprintf(
"multiIf(value <= 0, toFloat64(0), value <= %s, %s, value > %s, toFloat64('+Inf'), pow(2, ceil(log2(value) * %s) / %s))",
lowest, lowest, highest, bandsPerDoubling, bandsPerDoubling,
)
}
// formatFloat renders a float64 as the shortest literal that reads back as the
// same value, so an upper bound computed from it is identical on every row.
func formatFloat(v float64) string {
return strconv.FormatFloat(v, 'g', -1, 64)
}
func GroupByColumnAlias(i int, name string) string {

View File

@@ -212,7 +212,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __spatial_aggregation_cte AS (SELECT toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(value)/30 AS value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) AND JSONExtractString(labels, 'service.name') = ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.950) AS value FROM __spatial_aggregation_cte GROUP BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY `__GROUP_BY_KEY_0_service.name`, ts",
Query: "WITH __spatial_aggregation_cte AS (SELECT toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(value) AS value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) AND JSONExtractString(labels, 'service.name') = ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.950) AS value FROM __spatial_aggregation_cte GROUP BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY `__GROUP_BY_KEY_0_service.name`, ts",
Args: []any{"signoz_latency", uint64(1747936800000), uint64(1747983420000), "delta", "cartservice", "signoz_latency", uint64(1747947390000), uint64(1747983420000)},
},
expectedErr: nil,
@@ -250,7 +250,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __spatial_aggregation_cte AS (SELECT toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(value)/30 AS value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) AND JSONExtractString(labels, 'service.name') = ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.950) AS value FROM __spatial_aggregation_cte GROUP BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY `__GROUP_BY_KEY_0_service.name`, ts",
Query: "WITH __spatial_aggregation_cte AS (SELECT toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(value) AS value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) AND JSONExtractString(labels, 'service.name') = ? GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.950) AS value FROM __spatial_aggregation_cte GROUP BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY `__GROUP_BY_KEY_0_service.name`, ts",
Args: []any{"signoz_latency", uint64(1747936800000), uint64(1747983420000), "delta", "cartservice", "signoz_latency", uint64(1747947390000), uint64(1747983420000)},
},
expectedErr: nil,
@@ -284,6 +284,199 @@ func TestStatementBuilder(t *testing.T) {
},
expectedErr: nil,
},
{
name: "test_histogram_heatmap_sum",
requestType: qbtypes.RequestTypeHeatmap,
query: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
StepInterval: qbtypes.Step{Duration: 60 * time.Second},
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "signoz_latency",
Type: metrictypes.HistogramType,
Temporality: metrictypes.Delta,
TimeAggregation: metrictypes.TimeAggregationIncrease,
SpaceAggregation: metrictypes.SpaceAggregationSum,
},
},
GroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "service.name",
},
},
},
},
expected: qbtypes.Statement{
Query: "WITH __spatial_aggregation_cte AS (SELECT toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(60)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(value) AS value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, toFloat64(le) AS __bucket, greatest(value - lagInFrame(value, 1, 0) OVER __heatmap_window, 0) AS __result_0 FROM __spatial_aggregation_cte WINDOW __heatmap_window AS (PARTITION BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY toFloat64(le)) ORDER BY `__GROUP_BY_KEY_0_service.name`, ts, toFloat64(le)",
Args: []any{"signoz_latency", uint64(1747936800000), uint64(1747983420000), "delta", "signoz_latency", uint64(1747947360000), uint64(1747983420000)},
},
expectedErr: nil,
},
{
name: "test_histogram_heatmap_percentile",
requestType: qbtypes.RequestTypeHeatmap,
query: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
StepInterval: qbtypes.Step{Duration: 60 * time.Second},
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "signoz_latency",
Type: metrictypes.HistogramType,
Temporality: metrictypes.Delta,
TimeAggregation: metrictypes.TimeAggregationRate,
SpaceAggregation: metrictypes.SpaceAggregationPercentile95,
},
},
GroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "service.name",
},
},
},
},
expected: qbtypes.Statement{
Query: "WITH __spatial_aggregation_cte AS (SELECT toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(60)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(value) AS value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, toFloat64(le) AS __bucket, greatest(value - lagInFrame(value, 1, 0) OVER __heatmap_window, 0) AS __result_0 FROM __spatial_aggregation_cte WINDOW __heatmap_window AS (PARTITION BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY toFloat64(le)) ORDER BY `__GROUP_BY_KEY_0_service.name`, ts, toFloat64(le)",
Args: []any{"signoz_latency", uint64(1747936800000), uint64(1747983420000), "delta", "signoz_latency", uint64(1747947360000), uint64(1747983420000)},
},
expectedErr: nil,
},
{
name: "test_gauge_heatmap_log",
requestType: qbtypes.RequestTypeHeatmap,
query: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
StepInterval: qbtypes.Step{Duration: 60 * time.Second},
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "system.memory.usage",
Type: metrictypes.GaugeType,
Temporality: metrictypes.Unspecified,
TimeAggregation: metrictypes.TimeAggregationAvg,
SpaceAggregation: metrictypes.SpaceAggregationSum,
HeatmapBucketing: &qbtypes.HeatmapBucketing{
Kind: qbtypes.BucketsKindLog,
LogScale: qbtypes.MaxLogScale,
NumBuckets: qbtypes.DefaultNumBuckets,
},
},
},
GroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "host.name",
},
},
},
},
expected: qbtypes.Statement{
Query: "WITH __temporal_aggregation_cte AS (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(60)) AS ts, `__GROUP_BY_KEY_0_host.name`, avg(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'host.name') AS `__GROUP_BY_KEY_0_host.name` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `__GROUP_BY_KEY_0_host.name`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_host.name` ORDER BY fingerprint, ts), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_host.name`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_host.name`) SELECT ts, `__GROUP_BY_KEY_0_host.name`, multiIf(value <= 0, toFloat64(0), value <= 2.3283064365386963e-10, 2.3283064365386963e-10, value > 1.8446744073709552e+19, toFloat64('+Inf'), pow(2, ceil(log2(value) * 16) / 16)) AS __bucket, toFloat64(1) AS __result_0 FROM __spatial_aggregation_cte ORDER BY `__GROUP_BY_KEY_0_host.name`, ts, __bucket",
Args: []any{"system.memory.usage", uint64(1747936800000), uint64(1747983420000), "unspecified", "system.memory.usage", uint64(1747947360000), uint64(1747983420000), 0},
},
expectedErr: nil,
},
{
name: "test_gauge_heatmap_linear",
requestType: qbtypes.RequestTypeHeatmap,
query: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
StepInterval: qbtypes.Step{Duration: 60 * time.Second},
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "system.memory.usage",
Type: metrictypes.GaugeType,
Temporality: metrictypes.Unspecified,
TimeAggregation: metrictypes.TimeAggregationAvg,
SpaceAggregation: metrictypes.SpaceAggregationSum,
HeatmapBucketing: &qbtypes.HeatmapBucketing{
Kind: qbtypes.BucketsKindLinear,
LogScale: qbtypes.MaxLogScale,
MaxValue: 500,
NumBuckets: 25,
},
},
},
GroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "host.name",
},
},
},
},
expected: qbtypes.Statement{
Query: "WITH __temporal_aggregation_cte AS (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(60)) AS ts, `__GROUP_BY_KEY_0_host.name`, avg(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'host.name') AS `__GROUP_BY_KEY_0_host.name` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `__GROUP_BY_KEY_0_host.name`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_host.name` ORDER BY fingerprint, ts), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_host.name`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_host.name`) SELECT ts, `__GROUP_BY_KEY_0_host.name`, multiIf(value > 500, toFloat64('+Inf'), least(greatest(ceil(value * 25 / 500), 1), 25) * 500 / 25) AS __bucket, toFloat64(1) AS __result_0 FROM __spatial_aggregation_cte ORDER BY `__GROUP_BY_KEY_0_host.name`, ts, __bucket",
Args: []any{"system.memory.usage", uint64(1747936800000), uint64(1747983420000), "unspecified", "system.memory.usage", uint64(1747947360000), uint64(1747983420000), 0},
},
expectedErr: nil,
},
{
// cumulative keeps CanShortCircuitDelta false, so the counts reach the
// bucket differencing through the temporal CTE rather than the delta
// fast path
name: "test_histogram_heatmap_cumulative",
requestType: qbtypes.RequestTypeHeatmap,
query: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
StepInterval: qbtypes.Step{Duration: 60 * time.Second},
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "http_server_duration_bucket",
Type: metrictypes.HistogramType,
Temporality: metrictypes.Cumulative,
TimeAggregation: metrictypes.TimeAggregationRate,
SpaceAggregation: metrictypes.SpaceAggregationPercentile95,
},
},
GroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "service.name",
},
},
},
},
expected: qbtypes.Statement{
Query: "WITH __temporal_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, `le`, multiIf(row_number() OVER rate_window = 1, nan, (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) < 0, per_series_value, per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) AS per_series_value FROM (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(60)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, max(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_service.name`, `le` ORDER BY fingerprint, ts) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, toFloat64(le) AS __bucket, greatest(value - lagInFrame(value, 1, 0) OVER __heatmap_window, 0) AS __result_0 FROM __spatial_aggregation_cte WINDOW __heatmap_window AS (PARTITION BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY toFloat64(le)) ORDER BY `__GROUP_BY_KEY_0_service.name`, ts, toFloat64(le)",
Args: []any{"http_server_duration_bucket", uint64(1747936800000), uint64(1747983420000), "cumulative", "http_server_duration_bucket", uint64(1747947300000), uint64(1747983420000), 0},
},
expectedErr: nil,
},
{
name: "test_sum_heatmap_cumulative",
requestType: qbtypes.RequestTypeHeatmap,
query: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Signal: telemetrytypes.SignalMetrics,
StepInterval: qbtypes.Step{Duration: 60 * time.Second},
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "signoz_calls_total",
Type: metrictypes.SumType,
Temporality: metrictypes.Cumulative,
TimeAggregation: metrictypes.TimeAggregationIncrease,
SpaceAggregation: metrictypes.SpaceAggregationSum,
HeatmapBucketing: &qbtypes.HeatmapBucketing{
Kind: qbtypes.BucketsKindLog,
LogScale: qbtypes.MaxLogScale,
NumBuckets: qbtypes.DefaultNumBuckets,
},
},
},
GroupBy: []qbtypes.GroupByKey{
{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "service.name",
},
},
},
},
expected: qbtypes.Statement{
Query: "WITH __temporal_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, multiIf(row_number() OVER rate_window = 1, nan, (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) < 0, per_series_value, per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) AS per_series_value FROM (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(60)) AS ts, `__GROUP_BY_KEY_0_service.name`, max(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_service.name` ORDER BY fingerprint, ts) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, multiIf(value <= 0, toFloat64(0), value <= 2.3283064365386963e-10, 2.3283064365386963e-10, value > 1.8446744073709552e+19, toFloat64('+Inf'), pow(2, ceil(log2(value) * 16) / 16)) AS __bucket, toFloat64(1) AS __result_0 FROM __spatial_aggregation_cte ORDER BY `__GROUP_BY_KEY_0_service.name`, ts, __bucket",
Args: []any{"signoz_calls_total", uint64(1747936800000), uint64(1747983420000), "cumulative", "signoz_calls_total", uint64(1747947300000), uint64(1747983420000), 0},
},
expectedErr: nil,
},
{
name: "test_gauge_avg_sum",
requestType: qbtypes.RequestTypeTimeSeries,
@@ -342,7 +535,7 @@ func TestStatementBuilder(t *testing.T) {
},
},
expected: qbtypes.Statement{
Query: "WITH __temporal_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, `le`, multiIf(row_number() OVER rate_window = 1, nan, (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) < 0, per_series_value / (ts - lagInFrame(ts, 1) OVER rate_window), (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) / (ts - lagInFrame(ts, 1) OVER rate_window)) AS per_series_value FROM (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, max(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_service.name`, `le` ORDER BY fingerprint, ts) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.950) AS value FROM __spatial_aggregation_cte GROUP BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY `__GROUP_BY_KEY_0_service.name`, ts",
Query: "WITH __temporal_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, `le`, multiIf(row_number() OVER rate_window = 1, nan, (per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) < 0, per_series_value, per_series_value - lagInFrame(per_series_value, 1) OVER rate_window) AS per_series_value FROM (SELECT fingerprint, toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), toIntervalSecond(30)) AS ts, `__GROUP_BY_KEY_0_service.name`, `le`, max(value) AS per_series_value FROM signoz_metrics.distributed_samples_v4 AS points INNER JOIN (SELECT fingerprint, JSONExtractString(labels, 'service.name') AS `__GROUP_BY_KEY_0_service.name`, JSONExtractString(labels, 'le') AS `le` FROM signoz_metrics.time_series_v4_6hrs WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli <= ? AND LOWER(temporality) LIKE LOWER(?) GROUP BY fingerprint, `__GROUP_BY_KEY_0_service.name`, `le`) AS filtered_time_series ON points.fingerprint = filtered_time_series.fingerprint WHERE metric_name IN (?) AND unix_milli >= ? AND unix_milli < ? GROUP BY fingerprint, ts, `__GROUP_BY_KEY_0_service.name`, `le` ORDER BY fingerprint, ts) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)), __spatial_aggregation_cte AS (SELECT ts, `__GROUP_BY_KEY_0_service.name`, `le`, sum(per_series_value) AS value FROM __temporal_aggregation_cte WHERE isNaN(per_series_value) = ? GROUP BY ts, `__GROUP_BY_KEY_0_service.name`, `le`) SELECT ts, `__GROUP_BY_KEY_0_service.name`, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.950) AS value FROM __spatial_aggregation_cte GROUP BY `__GROUP_BY_KEY_0_service.name`, ts ORDER BY `__GROUP_BY_KEY_0_service.name`, ts",
Args: []any{"http_server_duration_bucket", uint64(1747936800000), uint64(1747983420000), "cumulative", "http_server_duration_bucket", uint64(1747947360000), uint64(1747983420000), 0},
},
expectedErr: nil,

View File

@@ -524,6 +524,96 @@ func TestInvalidateUnknownPluginKind(t *testing.T) {
}
}
// TestHeatmapPanelQueryKinds pins the panel allowlist to what validateHeatmap
// accepts in querybuildertypesv5: everything but a trace operator.
func TestHeatmapPanelQueryKinds(t *testing.T) {
testCases := []struct {
description string
queryPluginKind string
queryPluginSpec string
expectedAllowed bool
}{
{
description: "a metrics builder query is allowed",
queryPluginKind: "signoz/BuilderQuery",
queryPluginSpec: `{"name": "A", "signal": "metrics", "aggregations": [
{"metricName": "http.server.request.duration", "timeAggregation": "increase", "spaceAggregation": "sum"}
]}`,
expectedAllowed: true,
},
{
description: "a promql query is allowed",
queryPluginKind: "signoz/PromQLQuery",
queryPluginSpec: `{"name": "A", "query": "sum by (le) (increase(signoz_latency_bucket[5m]))"}`,
expectedAllowed: true,
},
{
description: "a clickhouse query is allowed",
queryPluginKind: "signoz/ClickHouseSQL",
queryPluginSpec: `{"name": "A", "query": "SELECT ts, bucket, value FROM cells"}`,
expectedAllowed: true,
},
{
description: "a formula is allowed",
queryPluginKind: "signoz/Formula",
queryPluginSpec: `{"name": "F1", "expression": "A / B"}`,
expectedAllowed: true,
},
{
description: "a composite query is allowed, since a formula needs its disabled inputs alongside it",
queryPluginKind: "signoz/CompositeQuery",
queryPluginSpec: `{"queries": [
{"type": "builder_query", "spec": {"name": "A", "signal": "metrics", "disabled": true, "aggregations": [
{"metricName": "http.server.request.duration", "timeAggregation": "increase", "spaceAggregation": "sum"}
]}},
{"type": "builder_formula", "spec": {"name": "F1", "expression": "A * 2"}}
]}`,
expectedAllowed: true,
},
{
description: "a trace operator is refused",
queryPluginKind: "signoz/TraceOperator",
queryPluginSpec: `{"name": "T1", "expression": "A => B"}`,
expectedAllowed: false,
},
}
for _, testCase := range testCases {
t.Run(testCase.description, func(t *testing.T) {
data := fmt.Sprintf(`{
"variables": [],
"panels": {
"p1": {
"kind": "Panel",
"spec": {
"links": [],
"plugin": {"kind": "signoz/HeatmapPanel", "spec": {}},
"queries": [{
"kind": "heatmap",
"spec": {
"plugin": {"kind": %q, "spec": %s}
}
}]
}
}
},
"links": [],
"layouts": []
}`, testCase.queryPluginKind, testCase.queryPluginSpec)
_, err := unmarshalDashboard([]byte(data))
if testCase.expectedAllowed {
require.NoError(t, err)
return
}
require.Error(t, err)
assert.Contains(t, err.Error(), "is not supported by panel kind")
})
}
}
func TestInvalidateOneInvalidPanel(t *testing.T) {
data := []byte(`{
"variables": [],

View File

@@ -35,6 +35,7 @@ func (PanelPlugin) PrepareJSONSchema(s *jsonschema.Schema) error {
string(PanelKindTable): schemaRef("DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesTablePanelSpec"),
string(PanelKindHistogram): schemaRef("DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesHistogramPanelSpec"),
string(PanelKindList): schemaRef("DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesListPanelSpec"),
string(PanelKindHeatmap): schemaRef("DashboardtypesPanelPluginVariantGithubComSigNozSignozPkgTypesDashboardtypesHeatmapPanelSpec"),
})
}
@@ -65,6 +66,7 @@ func (PanelPlugin) JSONSchemaOneOf() []any {
PanelPluginVariant[TablePanelSpec]{Kind: string(PanelKindTable)},
PanelPluginVariant[HistogramPanelSpec]{Kind: string(PanelKindHistogram)},
PanelPluginVariant[ListPanelSpec]{Kind: string(PanelKindList)},
PanelPluginVariant[HeatmapPanelSpec]{Kind: string(PanelKindHeatmap)},
}
}
@@ -228,6 +230,7 @@ var (
PanelKindTable: func() any { return new(TablePanelSpec) },
PanelKindHistogram: func() any { return new(HistogramPanelSpec) },
PanelKindList: func() any { return new(ListPanelSpec) },
PanelKindHeatmap: func() any { return new(HeatmapPanelSpec) },
}
queryPluginSpecs = map[QueryPluginKind]func() any{
QueryKindBuilder: func() any { return new(BuilderQuerySpec) },
@@ -250,6 +253,7 @@ var (
PanelKindPieChart: {QueryKindBuilder, QueryKindComposite, QueryKindFormula, QueryKindTraceOperator, QueryKindClickHouseSQL},
PanelKindTable: {QueryKindBuilder, QueryKindComposite, QueryKindFormula, QueryKindTraceOperator, QueryKindClickHouseSQL},
PanelKindList: {QueryKindBuilder},
PanelKindHeatmap: {QueryKindBuilder, QueryKindComposite, QueryKindFormula, QueryKindPromQL, QueryKindClickHouseSQL},
}
)

View File

@@ -173,10 +173,11 @@ const (
PanelKindTable PanelPluginKind = "signoz/TablePanel"
PanelKindHistogram PanelPluginKind = "signoz/HistogramPanel"
PanelKindList PanelPluginKind = "signoz/ListPanel"
PanelKindHeatmap PanelPluginKind = "signoz/HeatmapPanel"
)
func (PanelPluginKind) Enum() []any {
return []any{PanelKindTimeSeries, PanelKindBarChart, PanelKindNumber, PanelKindPieChart, PanelKindTable, PanelKindHistogram, PanelKindList}
return []any{PanelKindTimeSeries, PanelKindBarChart, PanelKindNumber, PanelKindPieChart, PanelKindTable, PanelKindHistogram, PanelKindList, PanelKindHeatmap}
}
type TimeSeriesPanelSpec struct {
@@ -237,6 +238,56 @@ type ListPanelSpec struct {
SelectFields []telemetrytypes.TelemetryFieldKey `json:"selectFields,omitzero" validate:"dive"`
}
type HeatmapPanelSpec struct {
Visualization BasicVisualization `json:"visualization"`
Formatting PanelFormatting `json:"formatting"`
Axes HeatmapAxes `json:"axes"`
Legend Legend `json:"legend"`
ChartAppearance HeatmapChartAppearance `json:"chartAppearance"`
}
// HeatmapAxes carries only the Y scale. The shared Axes type models a value
// axis with soft bounds, where a heatmap's Y axis is the bucket boundaries the
// response already fixed.
type HeatmapAxes struct {
YScale HeatmapYScale `json:"yScale"`
}
type HeatmapChartAppearance struct {
Colors HeatmapColors `json:"colors"`
}
type HeatmapColors struct {
Mode HeatmapColorMode `json:"mode"`
Palette HeatmapPalette `json:"palette"`
Scale HeatmapColorScale `json:"scale"`
Steps int `json:"steps" validate:"omitempty,min=2,max=128"`
// MinCount and MaxCount clamp the colour scale; nil derives them from the
// grid, 0 and the highest count in it.
MinCount *float64 `json:"minCount"`
MaxCount *float64 `json:"maxCount"`
// Fill applies in opacity mode; empty means the selected group's legend colour.
Fill string `json:"fill"`
}
func (c *HeatmapColors) UnmarshalJSON(data []byte) error {
type alias HeatmapColors
var tmp alias
if err := json.Unmarshal(data, &tmp); err != nil {
return errors.WrapInvalidInputf(err, ErrCodeDashboardInvalidInput, "invalid heatmap colors")
}
*c = HeatmapColors(tmp)
return c.validate()
}
func (c HeatmapColors) validate() error {
if c.MinCount != nil && c.MaxCount != nil && *c.MinCount > *c.MaxCount {
return errors.NewInvalidInputf(ErrCodeDashboardInvalidInput,
"heatmap colors.minCount %v is greater than colors.maxCount %v", *c.MinCount, *c.MaxCount)
}
return nil
}
// ══════════════════════════════════════════════
// Panel common types
// ══════════════════════════════════════════════
@@ -709,3 +760,168 @@ func (p *PrecisionOption) UnmarshalJSON(data []byte) error {
return errors.NewInvalidInputf(ErrCodeDashboardInvalidInput, "invalid precision option %q: must be `0`, `1`, `2`, `3`, `4`, or `full`", v)
}
}
type HeatmapColorMode struct{ valuer.String }
var (
HeatmapColorModePalette = HeatmapColorMode{valuer.NewString("palette")} // default
HeatmapColorModeOpacity = HeatmapColorMode{valuer.NewString("opacity")}
)
func (HeatmapColorMode) Enum() []any {
return []any{HeatmapColorModePalette, HeatmapColorModeOpacity}
}
func (m HeatmapColorMode) ValueOrDefault() string {
if m.IsZero() {
return HeatmapColorModePalette.StringValue()
}
return m.StringValue()
}
func (m HeatmapColorMode) MarshalJSON() ([]byte, error) {
return json.Marshal(m.ValueOrDefault())
}
func (m *HeatmapColorMode) UnmarshalJSON(data []byte) error {
var v string
if err := json.Unmarshal(data, &v); err != nil {
return errors.WrapInvalidInputf(err, ErrCodeDashboardInvalidInput, "invalid heatmap color mode: must be a string, one of `palette` or `opacity`")
}
mode := HeatmapColorMode{valuer.NewString(v)}
switch mode {
case HeatmapColorModePalette, HeatmapColorModeOpacity:
*m = mode
return nil
default:
return errors.NewInvalidInputf(ErrCodeDashboardInvalidInput, "invalid heatmap color mode %q: must be `palette` or `opacity`", v)
}
}
type HeatmapPalette struct{ valuer.String }
var (
HeatmapPaletteIce = HeatmapPalette{valuer.NewString("ice")}
HeatmapPaletteMoss = HeatmapPalette{valuer.NewString("moss")}
HeatmapPaletteRust = HeatmapPalette{valuer.NewString("rust")}
HeatmapPaletteGraphite = HeatmapPalette{valuer.NewString("graphite")}
HeatmapPaletteEmber = HeatmapPalette{valuer.NewString("ember")}
HeatmapPaletteLagoon = HeatmapPalette{valuer.NewString("lagoon")}
HeatmapPaletteOrchid = HeatmapPalette{valuer.NewString("orchid")}
HeatmapPaletteVerdant = HeatmapPalette{valuer.NewString("verdant")}
HeatmapPaletteLava = HeatmapPalette{valuer.NewString("lava")} // default
HeatmapPaletteBeacon = HeatmapPalette{valuer.NewString("beacon")}
)
func (HeatmapPalette) Enum() []any {
return []any{
HeatmapPaletteIce, HeatmapPaletteMoss, HeatmapPaletteRust, HeatmapPaletteGraphite,
HeatmapPaletteEmber, HeatmapPaletteLagoon, HeatmapPaletteOrchid, HeatmapPaletteVerdant,
HeatmapPaletteLava, HeatmapPaletteBeacon,
}
}
func (p HeatmapPalette) ValueOrDefault() string {
if p.IsZero() {
return HeatmapPaletteLava.StringValue()
}
return p.StringValue()
}
func (p HeatmapPalette) MarshalJSON() ([]byte, error) {
return json.Marshal(p.ValueOrDefault())
}
func (p *HeatmapPalette) UnmarshalJSON(data []byte) error {
var v string
if err := json.Unmarshal(data, &v); err != nil {
return errors.WrapInvalidInputf(err, ErrCodeDashboardInvalidInput, "invalid heatmap palette: must be a string, one of `ice`, `moss`, `rust`, `graphite`, `ember`, `lagoon`, `orchid`, `verdant`, `lava`, or `beacon`")
}
palette := HeatmapPalette{valuer.NewString(v)}
switch palette {
case HeatmapPaletteIce, HeatmapPaletteMoss, HeatmapPaletteRust, HeatmapPaletteGraphite,
HeatmapPaletteEmber, HeatmapPaletteLagoon, HeatmapPaletteOrchid, HeatmapPaletteVerdant,
HeatmapPaletteLava, HeatmapPaletteBeacon:
*p = palette
return nil
default:
return errors.NewInvalidInputf(ErrCodeDashboardInvalidInput, "invalid heatmap palette %q: must be `ice`, `moss`, `rust`, `graphite`, `ember`, `lagoon`, `orchid`, `verdant`, `lava`, or `beacon`", v)
}
}
type HeatmapYScale struct{ valuer.String }
var (
HeatmapYScaleAuto = HeatmapYScale{valuer.NewString("auto")} // default
HeatmapYScaleLinear = HeatmapYScale{valuer.NewString("linear")}
HeatmapYScaleLog = HeatmapYScale{valuer.NewString("log")}
HeatmapYScaleSymlog = HeatmapYScale{valuer.NewString("symlog")}
)
func (HeatmapYScale) Enum() []any {
return []any{HeatmapYScaleAuto, HeatmapYScaleLinear, HeatmapYScaleLog, HeatmapYScaleSymlog}
}
func (s HeatmapYScale) ValueOrDefault() string {
if s.IsZero() {
return HeatmapYScaleAuto.StringValue()
}
return s.StringValue()
}
func (s HeatmapYScale) MarshalJSON() ([]byte, error) {
return json.Marshal(s.ValueOrDefault())
}
func (s *HeatmapYScale) UnmarshalJSON(data []byte) error {
var v string
if err := json.Unmarshal(data, &v); err != nil {
return errors.WrapInvalidInputf(err, ErrCodeDashboardInvalidInput, "invalid heatmap y scale: must be a string, one of `auto`, `linear`, `log`, or `symlog`")
}
scale := HeatmapYScale{valuer.NewString(v)}
switch scale {
case HeatmapYScaleAuto, HeatmapYScaleLinear, HeatmapYScaleLog, HeatmapYScaleSymlog:
*s = scale
return nil
default:
return errors.NewInvalidInputf(ErrCodeDashboardInvalidInput, "invalid heatmap y scale %q: must be `auto`, `linear`, `log`, or `symlog`", v)
}
}
type HeatmapColorScale struct{ valuer.String }
var (
HeatmapColorScaleLog = HeatmapColorScale{valuer.NewString("log")} // default
HeatmapColorScaleSqrt = HeatmapColorScale{valuer.NewString("sqrt")}
HeatmapColorScaleLinear = HeatmapColorScale{valuer.NewString("linear")}
)
func (HeatmapColorScale) Enum() []any {
return []any{HeatmapColorScaleLog, HeatmapColorScaleSqrt, HeatmapColorScaleLinear}
}
func (s HeatmapColorScale) ValueOrDefault() string {
if s.IsZero() {
return HeatmapColorScaleLog.StringValue()
}
return s.StringValue()
}
func (s HeatmapColorScale) MarshalJSON() ([]byte, error) {
return json.Marshal(s.ValueOrDefault())
}
func (s *HeatmapColorScale) UnmarshalJSON(data []byte) error {
var v string
if err := json.Unmarshal(data, &v); err != nil {
return errors.WrapInvalidInputf(err, ErrCodeDashboardInvalidInput, "invalid heatmap color scale: must be a string, one of `log`, `sqrt`, or `linear`")
}
scale := HeatmapColorScale{valuer.NewString(v)}
switch scale {
case HeatmapColorScaleLog, HeatmapColorScaleSqrt, HeatmapColorScaleLinear:
*s = scale
return nil
default:
return errors.NewInvalidInputf(ErrCodeDashboardInvalidInput, "invalid heatmap color scale %q: must be `log`, `sqrt`, or `linear`", v)
}
}

View File

@@ -540,6 +540,8 @@ type MetricAggregation struct {
// reduce to operator for metric scalar requests
ReduceTo ReduceTo `json:"reduceTo,omitzero"`
HeatmapBucketing *HeatmapBucketing `json:"-"`
Reduced bool `json:"-"`
}
@@ -554,6 +556,10 @@ func (m MetricAggregation) Copy() MetricAggregation {
valueFilterCopy := *m.ValueFilter
c.ValueFilter = &valueFilterCopy
}
if m.HeatmapBucketing != nil {
bucketingCopy := *m.HeatmapBucketing
c.HeatmapBucketing = &bucketingCopy
}
return c
}

View File

@@ -0,0 +1,296 @@
package querybuildertypesv5
import (
"math"
"slices"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
)
const (
// HeatmapBucketColumn is the alias a heatmap statement gives the column holding
// a row's bucket upper bound. Every other aggregation returns a single numeric
// column the reader treats as the value; this name tells the two apart.
HeatmapBucketColumn = "__bucket"
DefaultNumBuckets = 60
// MaxLogScale is the resolution ClickHouse buckets every log heatmap at:
// 2^MaxLogScale bands per doubling. It is both the default and the finest
// available, since a coarser LogBucketsSpec.Scale folds down from it.
MaxLogScale = 4
// MinLogScale is one band per 16x, the coarsest axis worth rendering.
MinLogScale = -4
// A positive value approaching zero runs its band index off to -inf, so
// without a clamp one near-zero sample would stretch the axis by thousands
// of bands once AddHeatmapBucketsWithNoCounts spans it.
MinLogBandIndex = -512 // 2^-32, about 2.3e-10
MaxLogBandIndex = 1024 // 2^64, about 1.8e19
)
// MinLogUpperBound and MaxLogUpperBound are the ends the log axis is clamped
// to. They do not vary with the requested scale.
var (
MinLogUpperBound = math.Exp2(float64(MinLogBandIndex) / math.Exp2(MaxLogScale))
MaxLogUpperBound = math.Exp2(float64(MaxLogBandIndex) / math.Exp2(MaxLogScale))
)
// HeatmapBucketing is the bucket axis a heatmap statement builds in ClickHouse,
// resolved from BucketOptions once the metric type is known. It stays nil for
// histograms, whose upper bounds come from their own `le` labels.
type HeatmapBucketing struct {
Kind BucketsKind
// LogScale is the resolution the caller asked for. ClickHouse always buckets
// at MaxLogScale, and postprocessing folds the axis down to this.
LogScale int
// MaxValue and NumBuckets are linear only.
MaxValue float64
NumBuckets int
}
// ToHeatmapBucketing fills in what the caller left unset. An absent
// BucketOptions resolves to the finest log axis, the one kind that needs nothing
// from the caller.
func (b *BucketOptions) ToHeatmapBucketing() HeatmapBucketing {
resolved := HeatmapBucketing{
Kind: BucketsKindLog,
LogScale: MaxLogScale,
NumBuckets: DefaultNumBuckets,
}
if b == nil {
return resolved
}
switch spec := b.Spec.(type) {
case LinearBucketsSpec:
resolved.Kind = BucketsKindLinear
resolved.MaxValue = spec.MaxValue
if spec.NumBuckets > 0 {
resolved.NumBuckets = spec.NumBuckets
}
case LogBucketsSpec:
if spec.Scale != nil {
resolved.LogScale = *spec.Scale
}
}
return resolved
}
// This cannot be called in validateHeatmap cuz type is resolved in querier.go.
func (a *MetricAggregation) VerifyAndApplyBucketOptions(bucketOptions *BucketOptions) error {
switch a.Type {
case metrictypes.HistogramType:
if bucketOptions != nil {
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"bucketOptions are not supported for histogram metrics: %q takes its bucket axis from its own `le` labels, so nothing in the spec would be applied", a.MetricName)
}
a.HeatmapBucketing = nil
return nil
// A summary carries no upper bounds of its own either, and its samples reach
// the final select the same way a gauge's do, so it buckets identically.
case metrictypes.GaugeType, metrictypes.SumType, metrictypes.SummaryType:
bucketing := bucketOptions.ToHeatmapBucketing()
a.HeatmapBucketing = &bucketing
return nil
case metrictypes.UnspecifiedType:
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"heatmaps need a metric whose type is known: no type is recorded for %q, so its bucket axis cannot be chosen", a.MetricName)
case metrictypes.ExpHistogramType:
return errors.Newf(errors.TypeUnsupported, errors.CodeUnsupported,
"heatmaps are not supported for exponential histograms yet: %q keeps its bucket counts in a sketch column, which needs its own reader", a.MetricName)
default:
return errors.Newf(errors.TypeUnsupported, errors.CodeUnsupported,
"heatmaps are not supported for %s metrics", a.Type.StringValue())
}
}
func MergeBucketUpperBounds(tsData ...*TimeSeriesData) map[int][]float64 {
upperBoundsByAggregation := map[int][]float64{}
for _, data := range tsData {
if data == nil {
continue
}
for _, aggBucket := range data.Aggregations {
if len(aggBucket.Meta.Buckets) == 0 {
continue
}
upperBoundsByAggregation[aggBucket.Index] = append(upperBoundsByAggregation[aggBucket.Index], aggBucket.Meta.Buckets...)
}
}
for index, upperBounds := range upperBoundsByAggregation {
slices.Sort(upperBounds)
upperBoundsByAggregation[index] = slices.Compact(upperBounds)
}
return upperBoundsByAggregation
}
// DownscaleHeatmapResolution folds the MaxLogScale axis ClickHouse buckets at
// down to toScale, merging every 2^(MaxLogScale-toScale) adjacent bands into
// one. The coarser upper bounds are a subset of the finer ones, so the fold is
// exact.
func DownscaleHeatmapResolution(tsData *TimeSeriesData, toScale int) {
if tsData == nil || toScale >= MaxLogScale {
return
}
for _, aggBucket := range tsData.Aggregations {
downscaleHeatmapResolutionForAggregation(aggBucket, toScale)
}
}
func downscaleHeatmapResolutionForAggregation(aggBucket *AggregationBucket, toScale int) {
if aggBucket == nil || len(aggBucket.Meta.Buckets) == 0 {
return
}
factor := int(math.Exp2(float64(MaxLogScale - toScale)))
// Merging is by index in the exponential mapping, not by position in
// Meta.Buckets, which lists only the upper bounds some series reached.
coarseUpperBounds := make([]float64, 0, len(aggBucket.Meta.Buckets))
upperBoundToCoarseIndex := make(map[float64]int, len(aggBucket.Meta.Buckets))
mergedInto := make([]int, len(aggBucket.Meta.Buckets))
for index, upperBound := range aggBucket.Meta.Buckets {
coarsened := coarsenUpperBound(upperBound, toScale, factor)
coarseIndex, ok := upperBoundToCoarseIndex[coarsened]
if !ok {
coarseIndex = len(coarseUpperBounds)
coarseUpperBounds = append(coarseUpperBounds, coarsened)
upperBoundToCoarseIndex[coarsened] = coarseIndex
}
mergedInto[index] = coarseIndex
}
overflowIndex := len(coarseUpperBounds)
for _, series := range aggBucket.Series {
for _, point := range series.Values {
if len(point.Values) == 0 {
continue
}
coarseCounts := make([]float64, overflowIndex+1)
for index, count := range point.Values {
if index >= len(mergedInto) {
coarseCounts[overflowIndex] += count
continue
}
coarseCounts[mergedInto[index]] += count
}
point.Values = coarseCounts
}
}
aggBucket.Meta.Buckets = coarseUpperBounds
}
// coarsenUpperBound moves an upper bound from the MaxLogScale exponential axis
// onto the toScale one. The zero band has no exponent to rescale and stays put.
func coarsenUpperBound(upperBound float64, toScale, factor int) float64 {
if upperBound <= 0 || math.IsInf(upperBound, 0) || math.IsNaN(upperBound) {
return upperBound
}
index := int(math.Round(math.Log2(upperBound) * math.Exp2(MaxLogScale)))
merged := int(math.Ceil(float64(index) / float64(factor)))
return math.Exp2(float64(merged) / math.Exp2(float64(toScale)))
}
// AddHeatmapBucketsWithNoCounts spans the range from the lowest upper bound some
// series reached to the highest. Meta.Buckets leaves the ones in between out
// entirely, so without this a gap renders with its two sides touching.
//
// Only a value-derived axis can be spanned: its upper bounds come from an index
// that is a pure function of the value, so the ones in between are known without
// having seen them. Nothing says what sits between two `le` labels.
func AddHeatmapBucketsWithNoCounts(tsData *TimeSeriesData, bucketing HeatmapBucketing) {
if tsData == nil {
return
}
for _, aggBucket := range tsData.Aggregations {
addHeatmapBucketsWithNoCountsForAggregation(aggBucket, bucketing)
}
}
func addHeatmapBucketsWithNoCountsForAggregation(aggBucket *AggregationBucket, bucketing HeatmapBucketing) {
if aggBucket == nil || len(aggBucket.Meta.Buckets) == 0 {
return
}
// The zero bucket holds everything at or below zero. It has no index on either
// axis and sits below every other upper bound, so it keeps index 0 and the
// fill runs over the rest.
offset := 0
if aggBucket.Meta.Buckets[0] <= 0 {
offset = 1
}
positive := aggBucket.Meta.Buckets[offset:]
if len(positive) == 0 {
return
}
// Only finite upper bounds have an index, and the fill sizes a slice from
// one. Nothing should put +Inf or NaN on the axis, but bail if it happens.
indexes := make([]int, len(positive))
for i, upperBound := range positive {
if math.IsInf(upperBound, 0) || math.IsNaN(upperBound) {
return
}
indexes[i] = bucketing.calculateIndexOfUpperBound(upperBound)
}
lowest, highest := slices.Min(indexes), slices.Max(indexes)
denseUpperBounds := append([]float64{}, aggBucket.Meta.Buckets[:offset]...)
for index := lowest; index <= highest; index++ {
denseUpperBounds = append(denseUpperBounds, bucketing.calculateUpperBoundAtIndex(index))
}
if len(denseUpperBounds) == len(aggBucket.Meta.Buckets) {
return
}
// Counts map through their index rather than by matching upper bounds, so a
// regenerated upper bound differing from ClickHouse's in its last bit still
// lands where it came from.
shiftedTo := make([]int, len(aggBucket.Meta.Buckets))
for i, index := range indexes {
shiftedTo[i+offset] = index - lowest + offset
}
overflowIndex := len(denseUpperBounds)
for _, series := range aggBucket.Series {
for _, point := range series.Values {
if len(point.Values) == 0 {
continue
}
denseCounts := make([]float64, overflowIndex+1)
for index, count := range point.Values {
if index >= len(shiftedTo) {
denseCounts[overflowIndex] += count
continue
}
denseCounts[shiftedTo[index]] += count
}
point.Values = denseCounts
}
}
aggBucket.Meta.Buckets = denseUpperBounds
}
// calculateIndexOfUpperBound and calculateUpperBoundAtIndex are inverses over
// the axis being returned, so they read h.LogScale rather than the MaxLogScale
// ClickHouse bucketed at: k * maxValue / numBuckets on a linear axis,
// 2^(k / 2^scale) on a log one.
func (h HeatmapBucketing) calculateIndexOfUpperBound(upperBound float64) int {
if h.Kind == BucketsKindLinear {
return int(math.Round(upperBound * float64(h.NumBuckets) / h.MaxValue))
}
return int(math.Round(math.Log2(upperBound) * math.Exp2(float64(h.LogScale))))
}
func (h HeatmapBucketing) calculateUpperBoundAtIndex(index int) float64 {
if h.Kind == BucketsKindLinear {
return float64(index) * h.MaxValue / float64(h.NumBuckets)
}
return math.Exp2(float64(index) / math.Exp2(float64(h.LogScale)))
}

View File

@@ -397,6 +397,8 @@ type QueryRangeRequest struct {
PromQLProvider string `json:"-"`
FormatOptions *FormatOptions `json:"formatOptions,omitempty"`
BucketOptions *BucketOptions `json:"bucketOptions,omitempty"`
}
// PrepareJSONSchema adds description to the QueryRangeRequest schema.
@@ -734,3 +736,130 @@ func (r *QueryRangeRequest) GetQueriesSupportingZeroDefault() map[string]bool {
return canDefaultZero
}
type BucketOptions struct {
Kind BucketsKind `json:"kind"`
Spec any `json:"spec"`
}
type BucketsKind struct {
valuer.String
}
var (
BucketsKindLinear = BucketsKind{valuer.NewString("linear")}
BucketsKindLog = BucketsKind{valuer.NewString("log")}
)
// Enum implements jsonschema.Enum.
func (BucketsKind) Enum() []any {
return []any{
BucketsKindLinear,
BucketsKindLog,
}
}
// LinearBucketsSpec divides (0, MaxValue] into NumBuckets equal bands.
type LinearBucketsSpec struct {
// Everything above MaxValue is counted in the trailing overflow band. Evenly
// spaced upper bounds have no top to divide without it, so it is required.
MaxValue float64 `json:"maxValue" required:"true"`
NumBuckets int `json:"numBuckets,omitempty"`
}
// LogBucketsSpec spaces upper bounds at 2^Scale bands per doubling, the mapping
// an exponential histogram uses.
type LogBucketsSpec struct {
// ClickHouse always buckets at MaxLogScale and the surplus is folded away
// afterwards, so every Scale reads the same cache entry. MaxLogScale applies
// when unset.
Scale *int `json:"scale,omitempty"`
}
func (b *BucketOptions) UnmarshalJSON(data []byte) error {
var shadow struct {
Kind BucketsKind `json:"kind"`
Spec json.RawMessage `json:"spec"`
}
if err := binding.JSON.BindBody(bytes.NewReader(data), &shadow, binding.WithDisallowUnknownFields(true)); err != nil {
return err
}
b.Kind = shadow.Kind
// An absent spec is a malformed pair rather than a request for defaults;
// `"spec": {}` asks for those.
if len(shadow.Spec) == 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"bucketOptions spec is required, use an empty object for the kind's defaults")
}
switch shadow.Kind {
case BucketsKindLinear:
var spec LinearBucketsSpec
if err := binding.JSON.BindBody(bytes.NewReader(shadow.Spec), &spec, binding.WithDisallowUnknownFields(true), binding.WithUnknownFieldContext("linear buckets spec")); err != nil {
return err
}
b.Spec = spec
case BucketsKindLog:
var spec LogBucketsSpec
if err := binding.JSON.BindBody(bytes.NewReader(shadow.Spec), &spec, binding.WithDisallowUnknownFields(true), binding.WithUnknownFieldContext("log buckets spec")); err != nil {
return err
}
b.Spec = spec
default:
return errors.NewInvalidInputf(
errors.CodeInvalidInput,
"invalid bucketOptions kind: %s",
shadow.Kind.StringValue(),
).WithAdditional(
"Valid bucket kinds are: linear, log",
)
}
return nil
}
// bucketOptionsLinear and bucketOptionsLog are the OpenAPI schemas for the two
// BucketOptions variants. They have to be named types: the reflector turns an
// anonymous one into an inline subschema, leaving the discriminator mapping in
// PrepareJSONSchema pointing at components that were never emitted. `kind` is
// required:"true" on both so oapi-codegen renders the discriminator non-pointer.
type bucketOptionsLinear struct {
Kind BucketsKind `json:"kind" required:"true" description:"How the upper bounds are spaced."`
Spec LinearBucketsSpec `json:"spec" required:"true" description:"The evenly spaced bucket specification."`
}
type bucketOptionsLog struct {
Kind BucketsKind `json:"kind" required:"true" description:"How the upper bounds are spaced."`
Spec LogBucketsSpec `json:"spec" required:"true" description:"The logarithmic bucket specification."`
}
var _ jsonschema.OneOfExposer = BucketOptions{}
func (BucketOptions) JSONSchemaOneOf() []any {
return []any{
bucketOptionsLinear{},
bucketOptionsLog{},
}
}
var _ jsonschema.Preparer = BucketOptions{}
// PrepareJSONSchema marks the options as a `kind`-discriminated union;
// signoz.attachDiscriminators promotes it and strips the base properties.
func (BucketOptions) PrepareJSONSchema(s *jsonschema.Schema) error {
if s.ExtraProperties == nil {
s.ExtraProperties = map[string]any{}
}
s.ExtraProperties["x-signoz-discriminator"] = map[string]any{
"propertyName": "kind",
"mapping": map[string]string{
BucketsKindLinear.StringValue(): "#/components/schemas/Querybuildertypesv5BucketOptionsLinear",
BucketsKindLog.StringValue(): "#/components/schemas/Querybuildertypesv5BucketOptionsLog",
},
}
return nil
}

View File

@@ -19,11 +19,11 @@ func (r *RequestType) UnmarshalJSON(data []byte) error {
}
v := RequestType{valuer.NewString(s)}
switch v {
case RequestTypeScalar, RequestTypeTimeSeries, RequestTypeRaw, RequestTypeRawStream, RequestTypeTrace, RequestTypeDistribution:
case RequestTypeScalar, RequestTypeTimeSeries, RequestTypeRaw, RequestTypeRawStream, RequestTypeTrace, RequestTypeDistribution, RequestTypeHeatmap:
*r = v
return nil
default:
return errors.NewInvalidInputf(errors.CodeInvalidInput, "unknown request type %q; allowed values: %s", s, "`scalar`, `time_series`, `raw`, `raw_stream`, `trace`, `distribution`")
return errors.NewInvalidInputf(errors.CodeInvalidInput, "unknown request type %q; allowed values: %s", s, "`scalar`, `time_series`, `raw`, `raw_stream`, `trace`, `distribution`, `heatmap`")
}
}
@@ -41,6 +41,9 @@ var (
RequestTypeTrace = RequestType{valuer.NewString("trace")}
// []Bucket (struct{Lower,Upper,Count float64}), example: histogram.
RequestTypeDistribution = RequestType{valuer.NewString("distribution")}
// TimeSeriesData carrying one count per histogram bucket at each timestamp,
// with the shared bucket upper bounds on the aggregation's meta.
RequestTypeHeatmap = RequestType{valuer.NewString("heatmap")}
)
// IsAggregation returns true for request types that produce aggregated results
@@ -49,7 +52,7 @@ var (
// For non-aggregation types (raw, raw_stream, trace), those fields are ignored
// and don't need to be validated.
func (r RequestType) IsAggregation() bool {
return r == RequestTypeTimeSeries || r == RequestTypeScalar || r == RequestTypeDistribution
return r == RequestTypeTimeSeries || r == RequestTypeScalar || r == RequestTypeDistribution || r == RequestTypeHeatmap
}
// Enum implements jsonschema.Enum; returns the acceptable values for RequestType.
@@ -60,6 +63,7 @@ func (RequestType) Enum() []any {
RequestTypeRaw,
RequestTypeRawStream,
RequestTypeTrace,
RequestTypeHeatmap,
// RequestTypeDistribution,
}
}

View File

@@ -138,12 +138,10 @@ type TimeSeriesData struct {
}
type AggregationBucket struct {
Index int `json:"index"` // or string Alias
Alias string `json:"alias"`
Meta struct {
Unit string `json:"unit,omitempty"`
} `json:"meta,omitempty"`
Series []*TimeSeries `json:"series"` // no extra nesting
Index int `json:"index"` // or string Alias
Alias string `json:"alias"`
Meta AggregationMeta `json:"meta,omitempty"`
Series []*TimeSeries `json:"series"` // no extra nesting
PredictedSeries []*TimeSeries `json:"predictedSeries,omitempty"`
UpperBoundSeries []*TimeSeries `json:"upperBoundSeries,omitempty"`
@@ -151,6 +149,54 @@ type AggregationBucket struct {
AnomalyScores []*TimeSeries `json:"anomalyScores,omitempty"`
}
// ReindexValuesToNewUpperBounds moves each count to the index its upper bound
// holds in onto, a superset of Meta.Buckets. No count changes, only its position
// in Values.
func (a *AggregationBucket) ReindexValuesToNewUpperBounds(onto []float64) {
if a == nil {
return
}
from := a.Meta.Buckets
if len(onto) == 0 || slices.Equal(from, onto) {
return
}
upperBoundToIndex := make(map[float64]int, len(onto))
for index, upperBound := range onto {
upperBoundToIndex[upperBound] = index
}
for _, series := range a.Series {
for _, point := range series.Values {
if len(point.Values) == 0 {
continue
}
reindexed := make([]float64, len(onto)+1)
for index, count := range point.Values {
if index >= len(from) {
reindexed[len(onto)] = count
break
}
if newIndex, ok := upperBoundToIndex[from[index]]; ok {
reindexed[newIndex] = count
}
}
point.Values = reindexed
}
}
a.Meta.Buckets = onto
}
type AggregationMeta struct {
Unit string `json:"unit,omitempty"`
// Buckets holds ascending upper bounds shared by every series in the
// AggregationBucket, set only for heatmap results. Each point's Values holds
// len(Buckets)+1 counts: one per bound, then the open-above overflow.
Buckets []float64 `json:"buckets,omitempty"`
}
type TimeSeries struct {
Labels []*Label `json:"labels,omitempty"`
Values []*TimeSeriesValue `json:"values"`
@@ -254,13 +300,9 @@ type TimeSeriesValue struct {
// on the client side, these partial values are rendered differently.
Partial bool `json:"partial,omitempty"`
// for the heatmap type chart
// Values holds one count per histogram bucket for heatmap results, in the
// order of the aggregation's Meta.Buckets. Value is omitted in that case.
Values []float64 `json:"values,omitempty"`
Bucket *Bucket `json:"bucket,omitempty"`
}
type Bucket struct {
Step float64 `json:"step"`
}
type ColumnType struct {
@@ -481,13 +523,20 @@ func (t TimeSeriesValue) MarshalJSON() ([]byte, error) {
}
}
// a heatmap point's counts are spread across Values, so there is no one
// number Value could carry
var sanitizedValue any
if t.Values == nil {
sanitizedValue = sanitizeValue(t.Value)
}
return json.Marshal(&struct {
*Alias
Value any `json:"value"`
Value any `json:"value,omitempty"`
Values any `json:"values,omitempty"`
}{
Alias: (*Alias)(&t),
Value: sanitizeValue(t.Value),
Value: sanitizedValue,
Values: sanitizedValues,
})
}

View File

@@ -50,13 +50,21 @@ func TestTimeSeriesValue_MarshalJSON(t *testing.T) {
expected: `{"timestamp":1234567890,"value":"-Inf"}`,
},
{
name: "values array with NaN",
name: "zero value",
value: TimeSeriesValue{
Timestamp: 1234567890,
Value: 0,
},
expected: `{"timestamp":1234567890,"value":0}`,
},
{
name: "values array with NaN, and no value beside it",
value: TimeSeriesValue{
Timestamp: 1234567890,
Value: 1.0,
Values: []float64{1.0, math.NaN(), 3.0, math.Inf(1)},
},
expected: `{"timestamp":1234567890,"value":1,"values":[1,"NaN",3,"Inf"]}`,
expected: `{"timestamp":1234567890,"values":[1,"NaN",3,"Inf"]}`,
},
}

View File

@@ -127,7 +127,7 @@ func calculateSeriesValue(series *TimeSeries) float64 {
// For single-point series, return that value directly
if len(series.Values) == 1 {
value := series.Values[0].Value
value := calculatePointValue(series.Values[0])
if math.IsNaN(value) || math.IsInf(value, 0) {
return 0.0
}
@@ -139,10 +139,11 @@ func calculateSeriesValue(series *TimeSeries) float64 {
var count float64
for _, point := range series.Values {
if math.IsNaN(point.Value) || math.IsInf(point.Value, 0) {
value := calculatePointValue(point)
if math.IsNaN(value) || math.IsInf(value, 0) {
continue
}
sum += point.Value
sum += value
count++
}
@@ -154,6 +155,25 @@ func calculateSeriesValue(series *TimeSeries) float64 {
return sum / count
}
// calculatePointValue returns what a point contributes to its series' rank.
// Heatmap points carry one count per bucket in Values and leave Value at zero,
// so they rank on the total across buckets.
func calculatePointValue(point *TimeSeriesValue) float64 {
if len(point.Values) == 0 {
return point.Value
}
var total float64
for _, value := range point.Values {
if math.IsNaN(value) || math.IsInf(value, 0) {
continue
}
total += value
}
return total
}
// convertValueToString converts various types to string for comparison.
func convertValueToString(value any) string {
switch v := value.(type) {

View File

@@ -1,10 +1,12 @@
package querybuildertypesv5
import (
"math"
"testing"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestApplySeriesLimit(t *testing.T) {
@@ -232,3 +234,81 @@ func TestApplySeriesLimit(t *testing.T) {
assert.Equal(t, 40.0, result[2].Values[0].Value)
})
}
func TestApplySeriesLimitRanksHeatmapSeriesByBucketTotals(t *testing.T) {
// A reshaped heatmap point leaves Value at zero and holds one count per
// bucket in Values, so ranking has to sum the buckets to see any difference.
series := []*TimeSeries{
{
Labels: []*Label{{
Key: telemetrytypes.TelemetryFieldKey{Name: "service.name"},
Value: "quiet",
}},
Values: []*TimeSeriesValue{
{Timestamp: 1000, Values: []float64{1, 2, 0}},
{Timestamp: 1060, Values: []float64{0, 1, 0}},
},
},
{
Labels: []*Label{{
Key: telemetrytypes.TelemetryFieldKey{Name: "service.name"},
Value: "busy",
}},
Values: []*TimeSeriesValue{
{Timestamp: 1000, Values: []float64{40, 60, 5}},
{Timestamp: 1060, Values: []float64{30, 70, 5}},
},
},
{
Labels: []*Label{{
Key: telemetrytypes.TelemetryFieldKey{Name: "service.name"},
Value: "middling",
}},
Values: []*TimeSeriesValue{
{Timestamp: 1000, Values: []float64{5, 5, 0}},
{Timestamp: 1060, Values: []float64{4, 6, 0}},
},
},
}
result := ApplySeriesLimit(series, nil, 2)
require.Len(t, result, 2)
assert.Equal(t, "busy", result[0].Labels[0].Value)
assert.Equal(t, "middling", result[1].Labels[0].Value)
}
func TestCalculatePointValue(t *testing.T) {
testCases := []struct {
description string
point *TimeSeriesValue
expectedValue float64
}{
{
description: "a plain time series point ranks on its single value",
point: &TimeSeriesValue{Timestamp: 1000, Value: 7},
expectedValue: 7,
},
{
description: "a heatmap point ranks on the total across its buckets",
point: &TimeSeriesValue{Timestamp: 1000, Values: []float64{1, 12, 14, 3}},
expectedValue: 30,
},
{
description: "non-finite bucket counts are skipped",
point: &TimeSeriesValue{Timestamp: 1000, Values: []float64{2, math.NaN(), math.Inf(1), 3}},
expectedValue: 5,
},
{
description: "an empty bucket list falls back to the single value",
point: &TimeSeriesValue{Timestamp: 1000, Value: 4, Values: []float64{}},
expectedValue: 4,
},
}
for _, testCase := range testCases {
t.Run(testCase.description, func(t *testing.T) {
assert.Equal(t, testCase.expectedValue, calculatePointValue(testCase.point))
})
}
}

View File

@@ -2,6 +2,7 @@ package querybuildertypesv5
import (
"fmt"
"math"
"slices"
"strings"
@@ -65,6 +66,8 @@ func wrapValidationError(cause error, contextIdentifier string, errorFormat stri
const (
// Maximum limit for query results.
MaxQueryLimit = 10000
MaxNumBuckets = 512
)
// ValidationOption is a functional option for configuring validation behaviour.
@@ -581,7 +584,7 @@ func (r *QueryRangeRequest) Validate(opts ...ValidationOption) error {
// Validate request type
switch r.RequestType {
case RequestTypeRaw, RequestTypeRawStream, RequestTypeTrace, RequestTypeTimeSeries, RequestTypeScalar:
case RequestTypeRaw, RequestTypeRawStream, RequestTypeTrace, RequestTypeTimeSeries, RequestTypeScalar, RequestTypeHeatmap:
opts = append(opts, GetValidationOptions(r.RequestType)...)
default:
return errors.NewInvalidInputf(
@@ -589,10 +592,14 @@ func (r *QueryRangeRequest) Validate(opts ...ValidationOption) error {
"invalid request type: %s",
r.RequestType,
).WithAdditional(
"Valid request types are: raw, timeseries, scalar",
"Valid request types are: raw, timeseries, scalar, heatmap",
)
}
if err := r.validateHeatmap(); err != nil {
return err
}
// raw/trace request types don't support metric queries;
// metrics are always aggregated and there is no raw form.
if r.RequestType == RequestTypeRaw || r.RequestType == RequestTypeRawStream || r.RequestType == RequestTypeTrace {
@@ -630,11 +637,15 @@ func (r *QueryRangeRequest) ValidateRequestScope() ([]ValidationOption, error) {
var opts []ValidationOption
switch r.RequestType {
case RequestTypeRaw, RequestTypeRawStream, RequestTypeTrace, RequestTypeTimeSeries, RequestTypeScalar:
case RequestTypeRaw, RequestTypeRawStream, RequestTypeTrace, RequestTypeTimeSeries, RequestTypeScalar, RequestTypeHeatmap:
opts = GetValidationOptions(r.RequestType)
default:
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid request type: %s", r.RequestType).
WithAdditional("Valid request types are: raw, timeseries, scalar")
WithAdditional("Valid request types are: raw, timeseries, scalar, heatmap")
}
if err := r.validateHeatmap(); err != nil {
return nil, err
}
if r.RequestType == RequestTypeRaw || r.RequestType == RequestTypeRawStream || r.RequestType == RequestTypeTrace {
@@ -838,9 +849,149 @@ func validateQueryEnvelope(envelope QueryEnvelope, opts ...ValidationOption) err
}
}
func (r *QueryRangeRequest) validateHeatmap() error {
if r.RequestType != RequestTypeHeatmap {
if r.BucketOptions != nil {
return errors.NewInvalidInputf(
errors.CodeInvalidInput,
"bucketOptions are only supported for heatmap requests, got %s",
r.RequestType,
)
}
return nil
}
if r.FormatOptions != nil && r.FormatOptions.FillGaps {
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"fillGaps is not supported for heatmap requests: an absent column means collection stopped, which a zero-filled column would hide")
}
if err := r.BucketOptions.validateBucketOptions(); err != nil {
return err
}
enabled := 0
for _, envelope := range r.CompositeQuery.Queries {
switch spec := envelope.Spec.(type) {
case QueryBuilderQuery[MetricAggregation]:
if err := validateHeatmapQuery(spec.Functions, spec.Having); err != nil {
return err
}
if spec.Disabled {
continue
}
enabled++
case QueryBuilderFormula:
if err := validateHeatmapQuery(spec.Functions, spec.Having); err != nil {
return err
}
if spec.Disabled {
continue
}
enabled++
case ClickHouseQuery:
if spec.Disabled {
continue
}
enabled++
case PromQuery:
if r.BucketOptions != nil {
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"bucketOptions are not supported for promql heatmap requests: the bucket axis comes from the `le` labels the query returns, so nothing in the spec would be applied")
}
if spec.Disabled {
continue
}
enabled++
// An AI query decodes to the traces spec, so it lands here too. Admitting
// either signal means capping Aggregations at one: each carries its own
// Meta.Buckets, and a heatmap renders against a single bucket axis.
case QueryBuilderQuery[LogAggregation], QueryBuilderQuery[TraceAggregation]:
return errors.New(errors.TypeUnsupported, errors.CodeUnsupported,
"heatmaps are not supported for the logs and traces signals yet")
default:
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"heatmap requests support one metrics builder query, one formula over them, one clickhouse query, or one promql query, got %q", envelope.Type.StringValue())
}
}
// A disabled query is a formula input rather than something to draw, so a
// request can carry queries and still have none to render.
switch {
case len(r.CompositeQuery.Queries) == 0:
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"at least one query is required")
case enabled == 0:
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"a heatmap needs one enabled query, but every query is disabled").
WithAdditional("Enable the query whose distribution you want to plot")
case enabled > 1:
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"a heatmap renders one distribution, but %d queries are enabled", enabled).
WithAdditional(
"Disable the queries you don't want to plot, keeping only the one whose distribution you want to show",
"A formula can stay enabled with the queries it reads disabled",
)
}
return nil
}
func (b *BucketOptions) validateBucketOptions() error {
if b == nil {
return nil
}
switch spec := b.Spec.(type) {
case LinearBucketsSpec:
if math.IsNaN(spec.MaxValue) || math.IsInf(spec.MaxValue, 0) || spec.MaxValue <= 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"linear buckets need a finite maxValue greater than 0, got %v", spec.MaxValue)
}
if spec.NumBuckets < 0 || spec.NumBuckets > MaxNumBuckets {
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"numBuckets must be between 1 and %d, got %d", MaxNumBuckets, spec.NumBuckets)
}
case LogBucketsSpec:
if spec.Scale != nil && (*spec.Scale < MinLogScale || *spec.Scale > MaxLogScale) {
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"scale must be between %d and %d, got %d", MinLogScale, MaxLogScale, *spec.Scale)
}
default:
return errors.NewInvalidInputf(
errors.CodeInvalidInput,
"invalid bucketOptions kind: %s",
b.Kind.StringValue(),
).WithAdditional(
"Valid bucket kinds are: linear, log",
)
}
return nil
}
// validateHeatmapQuery refuses the per-query settings that cannot mean anything
// on a heatmap. It runs on disabled queries too: a disabled query is a formula
// input, so whatever it does still reaches the cells.
func validateHeatmapQuery(functions []Function, having *Having) error {
if len(functions) > 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"functions are not supported for heatmap requests: a heatmap point is a count per bucket, not a single value")
}
if having != nil && having.Expression != "" {
return errors.NewInvalidInputf(errors.CodeInvalidInput,
"having is not supported for heatmap requests: it filters individual cells, which breaks the cumulative differencing")
}
return nil
}
func GetValidationOptions(requestType RequestType) []ValidationOption {
switch requestType {
case RequestTypeTimeSeries:
case RequestTypeTimeSeries, RequestTypeHeatmap:
return []ValidationOption{WithSkipSelectFieldValidation(), WithTimestampGroupByValidation()}
case RequestTypeScalar:
return []ValidationOption{WithSkipSelectFieldValidation(), WithReduceToValidation()}

View File

@@ -33,6 +33,7 @@ class RequestType:
TIME_SERIES = "time_series"
SCALAR = "scalar"
TABLE = "table"
HEATMAP = "heatmap"
@dataclass
@@ -173,6 +174,7 @@ def make_query_request(
request_type: str = RequestType.TIME_SERIES,
format_options: dict | None = None,
variables: dict | None = None,
bucket_options: dict | None = None,
no_cache: bool = True,
timeout: int = QUERY_TIMEOUT,
headers: dict | None = None,
@@ -191,6 +193,8 @@ def make_query_request(
}
if variables:
payload["variables"] = variables
if bucket_options is not None:
payload["bucketOptions"] = bucket_options
return requests.post(
signoz.self.host_configs["8080"].get("/api/v5/query_range"),
@@ -359,6 +363,20 @@ def build_formula_query(
return {"type": "builder_formula", "spec": spec}
def build_log_bucket_options(scale: int | None = None) -> dict:
spec: dict[str, Any] = {}
if scale is not None:
spec["scale"] = scale
return {"kind": "log", "spec": spec}
def build_linear_bucket_options(max_value: float, num_buckets: int | None = None) -> dict:
spec: dict[str, Any] = {"maxValue": max_value}
if num_buckets is not None:
spec["numBuckets"] = num_buckets
return {"kind": "linear", "spec": spec}
def build_function(name: str, *args: Any) -> dict:
func: dict[str, Any] = {"name": name}
if args:
@@ -393,6 +411,24 @@ def get_all_series(response_json: dict, query_name: str) -> list[dict]:
return aggregations[0].get("series", [])
def get_heatmap_buckets(response_json: dict, query_name: str) -> list[float]:
"""The ascending bucket upper bounds a heatmap result's counts are positional against.
Each point holds one more count than there are bounds: the trailing one is the open-above overflow."""
results = response_json.get("data", {}).get("data", {}).get("results", [])
result = find_named_result(results, query_name)
if not result:
return []
aggregations = result.get("aggregations", [])
if not aggregations:
return []
return aggregations[0].get("meta", {}).get("buckets", [])
def get_heatmap_columns(response_json: dict, query_name: str) -> list[dict]:
"""A heatmap result's points for its single series, oldest first."""
return sorted(get_series_values(response_json, query_name), key=lambda point: point["timestamp"])
def get_scalar_value(response_json: dict, query_name: str) -> float | None:
values = get_series_values(response_json, query_name)
if values:

View File

@@ -0,0 +1,24 @@
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "1"}, "timestamp": "2025-01-10T10:00:00+00:00", "value": 10, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "2"}, "timestamp": "2025-01-10T10:00:00+00:00", "value": 20, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "4"}, "timestamp": "2025-01-10T10:00:00+00:00", "value": 30, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "+Inf"}, "timestamp": "2025-01-10T10:00:00+00:00", "value": 40, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "1"}, "timestamp": "2025-01-10T10:00:00+00:00", "value": 50, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "2"}, "timestamp": "2025-01-10T10:00:00+00:00", "value": 60, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "4"}, "timestamp": "2025-01-10T10:00:00+00:00", "value": 70, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "+Inf"}, "timestamp": "2025-01-10T10:00:00+00:00", "value": 80, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "1"}, "timestamp": "2025-01-10T10:01:00+00:00", "value": 11, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "2"}, "timestamp": "2025-01-10T10:01:00+00:00", "value": 23, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "4"}, "timestamp": "2025-01-10T10:01:00+00:00", "value": 33, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "+Inf"}, "timestamp": "2025-01-10T10:01:00+00:00", "value": 44, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "1"}, "timestamp": "2025-01-10T10:01:00+00:00", "value": 50, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "2"}, "timestamp": "2025-01-10T10:01:00+00:00", "value": 61, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "4"}, "timestamp": "2025-01-10T10:01:00+00:00", "value": 72, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "+Inf"}, "timestamp": "2025-01-10T10:01:00+00:00", "value": 82, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "1"}, "timestamp": "2025-01-10T10:02:00+00:00", "value": 13, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "2"}, "timestamp": "2025-01-10T10:02:00+00:00", "value": 25, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "4"}, "timestamp": "2025-01-10T10:02:00+00:00", "value": 39, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "api", "le": "+Inf"}, "timestamp": "2025-01-10T10:02:00+00:00", "value": 50, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "1"}, "timestamp": "2025-01-10T10:02:00+00:00", "value": 51, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "2"}, "timestamp": "2025-01-10T10:02:00+00:00", "value": 62, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "4"}, "timestamp": "2025-01-10T10:02:00+00:00", "value": 73, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}
{"metric_name": "heatmap_request_duration_bucket", "labels": {"__temporality__": "Cumulative", "service": "web", "le": "+Inf"}, "timestamp": "2025-01-10T10:02:00+00:00", "value": 85, "temporality": "Cumulative", "type_": "Histogram", "is_monotonic": true, "flags": 0, "description": "", "unit": "", "env": "default", "resource_attrs": {}, "scope_attrs": {}}

View File

@@ -0,0 +1,642 @@
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from http import HTTPStatus
import pytest
from fixtures import types
from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
from fixtures.fs import get_testdata_file_path
from fixtures.metrics import Metrics
from fixtures.querier import (
RequestType,
assert_identical_query_response,
build_builder_query,
build_formula_query,
build_linear_bucket_options,
build_log_bucket_options,
get_all_series,
get_all_warnings,
get_heatmap_buckets,
get_heatmap_columns,
index_series_by_label,
make_query_request,
)
HISTOGRAM_FILE = get_testdata_file_path("histogram_data_1h.jsonl")
HISTOGRAM_COUNTERS_FILE = get_testdata_file_path("heatmap_histogram_3m.jsonl")
MINUTE_MS = 60_000
def test_gauge_heatmap(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
minutes = 3
start_ms = int((now - timedelta(minutes=minutes + 1)).timestamp() * 1000)
end_ms = int(now.timestamp() * 1000)
metric_name = "test_heatmap_gauge"
value_by_host = {f"host-{host:02d}": (200, 400, 800)[host // 8] + host for host in range(24)}
insert_metrics(
[
Metrics(
metric_name=metric_name,
labels={"host": host},
timestamp=now - timedelta(minutes=minutes - minute),
value=value,
type_="Gauge",
is_monotonic=False,
)
for host, value in value_by_host.items()
for minute in range(minutes)
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[build_builder_query("A", metric_name, "max", "max", group_by=["host"])],
request_type=RequestType.HEATMAP,
bucket_options=build_log_bucket_options(0),
)
assert response.status_code == HTTPStatus.OK, response.text
data = response.json()
# a group by gives one series per host, and all of them are counted against
# this one axis
assert get_heatmap_buckets(data, "A") == pytest.approx([256.0, 512.0, 1024.0])
columns_by_host = {host: sorted(series["values"], key=lambda column: column["timestamp"]) for host, series in index_series_by_label(get_all_series(data, "A"), "host").items()}
assert len(columns_by_host) == len(value_by_host)
# a column carries its per-bucket counts and no `value`, since no single
# number stands for a spread
assert all("value" not in column for columns in columns_by_host.values() for column in columns)
# a column holds one count per bucket plus a trailing one for the overflow
for host, columns in columns_by_host.items():
occupied = int(host.removeprefix("host-")) // 8
assert [column["values"] for column in columns] == [[1 if slot == occupied else 0 for slot in range(4)]] * minutes
# summed across the hosts, a column is the spread of the 24 of them
for minute in range(minutes):
assert [sum(columns[minute]["values"][slot] for columns in columns_by_host.values()) for slot in range(4)] == [8, 8, 8, 0]
def test_sum_heatmap(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
minutes = 3
start_ms = int((now - timedelta(minutes=minutes + 1)).timestamp() * 1000)
end_ms = int(now.timestamp() * 1000)
metric_name = "test_heatmap_sum"
value_by_endpoint = {f"/endpoint-{endpoint:02d}": (100, 800)[endpoint // 8] + endpoint for endpoint in range(16)}
insert_metrics(
[
Metrics(
metric_name=metric_name,
labels={"endpoint": endpoint},
timestamp=now - timedelta(minutes=minutes - minute),
value=value,
temporality="Cumulative",
type_="Sum",
)
for endpoint, value in value_by_endpoint.items()
for minute in range(minutes)
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[build_builder_query("A", metric_name, "max", "max", temporality="cumulative", group_by=["endpoint"])],
request_type=RequestType.HEATMAP,
bucket_options=build_log_bucket_options(0),
)
assert response.status_code == HTTPStatus.OK, response.text
data = response.json()
assert get_heatmap_buckets(data, "A") == pytest.approx([128.0, 256.0, 512.0, 1024.0])
columns_by_endpoint = {endpoint: sorted(series["values"], key=lambda column: column["timestamp"]) for endpoint, series in index_series_by_label(get_all_series(data, "A"), "endpoint").items()}
assert len(columns_by_endpoint) == len(value_by_endpoint)
for endpoint, columns in columns_by_endpoint.items():
occupied = 0 if int(endpoint.removeprefix("/endpoint-")) < 8 else 3
assert [column["values"] for column in columns] == [[1 if slot == occupied else 0 for slot in range(5)]] * minutes
for minute in range(minutes):
assert [sum(columns[minute]["values"][slot] for columns in columns_by_endpoint.values()) for slot in range(5)] == [8, 0, 0, 8, 0]
def test_histogram_heatmap(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
start_ms = int((now - timedelta(minutes=65)).timestamp() * 1000)
end_ms = int(now.timestamp() * 1000)
metric_name = "test_heatmap_histogram"
insert_metrics(
Metrics.load_from_file(
HISTOGRAM_FILE,
base_time=now - timedelta(minutes=60),
metric_name_override=metric_name,
)
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[build_builder_query("A", metric_name, "increase", "p50", group_by=["le"], filter_expression='endpoint = "/health"')],
request_type=RequestType.HEATMAP,
)
assert response.status_code == HTTPStatus.OK, response.text
data = response.json()
# a histogram's axis is its recorded `le` bounds exactly, since nothing is
# known about what sits between two of them; `le=+Inf` has no finite bound
# and is counted in the trailing overflow slot
assert get_heatmap_buckets(data, "A") == [1000, 1500, 2000, 4000, 5000, 6000, 8000]
columns = get_heatmap_columns(data, "A")
assert columns
for column in columns:
assert len(column["values"]) == 8
assert all(count >= 0 for count in column["values"])
assert any(sum(column["values"]) > 0 for column in columns)
def test_linear_buckets(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
start_ms = int((now - timedelta(minutes=30)).timestamp() * 1000)
end_ms = int(now.timestamp() * 1000)
metric_name = "test_heatmap_linear"
# 100 wide buckets: 100 lands on the first, 250 on the third, and 1500 is
# past maxValue so it counts in the overflow
values = [100, 250, 1500]
insert_metrics(
[
Metrics(
metric_name=metric_name,
labels={"service": "api"},
timestamp=now - timedelta(minutes=len(values) - minute),
value=value,
type_="Gauge",
is_monotonic=False,
)
for minute, value in enumerate(values)
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[build_builder_query("A", metric_name, "max", "max")],
request_type=RequestType.HEATMAP,
bucket_options=build_linear_bucket_options(1000, 10),
)
assert response.status_code == HTTPStatus.OK, response.text
data = response.json()
# 200 is on the axis although nothing reached it, so the gap between 100 and
# 300 renders as a gap
assert get_heatmap_buckets(data, "A") == pytest.approx([100.0, 200.0, 300.0])
columns = get_heatmap_columns(data, "A")
assert [column["values"] for column in columns] == [
[1, 0, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1],
]
def test_zero_bucket(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
start_ms = int((now - timedelta(minutes=30)).timestamp() * 1000)
end_ms = int(now.timestamp() * 1000)
metric_name = "test_heatmap_zero_bucket"
values = [0, 256, 1024]
insert_metrics(
[
Metrics(
metric_name=metric_name,
labels={"service": "api"},
timestamp=now - timedelta(minutes=len(values) - minute),
value=value,
type_="Gauge",
is_monotonic=False,
)
for minute, value in enumerate(values)
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[build_builder_query("A", metric_name, "max", "max")],
request_type=RequestType.HEATMAP,
bucket_options=build_log_bucket_options(0),
)
assert response.status_code == HTTPStatus.OK, response.text
data = response.json()
# a log axis cannot place a value at or below zero, so those share a bucket
# of their own beneath the rest, and 512 is spanned above it
assert get_heatmap_buckets(data, "A") == pytest.approx([0.0, 256.0, 512.0, 1024.0])
columns = get_heatmap_columns(data, "A")
assert [column["values"] for column in columns] == [
[1, 0, 0, 0, 0],
[0, 1, 0, 0, 0],
[0, 0, 0, 1, 0],
]
def test_single_bucket(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
start_ms = int((now - timedelta(minutes=30)).timestamp() * 1000)
end_ms = int(now.timestamp() * 1000)
metric_name = "test_heatmap_single_bucket"
values = [256, 256, 256]
insert_metrics(
[
Metrics(
metric_name=metric_name,
labels={"service": "api"},
timestamp=now - timedelta(minutes=len(values) - minute),
value=value,
type_="Gauge",
is_monotonic=False,
)
for minute, value in enumerate(values)
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[build_builder_query("A", metric_name, "max", "max")],
request_type=RequestType.HEATMAP,
)
assert response.status_code == HTTPStatus.OK, response.text
# a single bucket leaves no gap to span
data = response.json()
assert get_heatmap_buckets(data, "A") == [256.0]
assert [column["values"] for column in get_heatmap_columns(data, "A")] == [[1, 0], [1, 0], [1, 0]]
# the values seeded below are 256 and 512, so each axis here runs from the bucket
# holding 256 to the one holding 512 at that option's resolution: 16 log buckets
# to the 2x, one bucket per 16x, or a linear bucket every maxValue/numBuckets
@pytest.mark.parametrize(
"bucket_options, expected_buckets",
[
(None, [256 * 2 ** (step / 16) for step in range(17)]),
({"kind": "log", "spec": {}}, [256 * 2 ** (step / 16) for step in range(17)]),
(build_log_bucket_options(4), [256 * 2 ** (step / 16) for step in range(17)]),
(build_log_bucket_options(-4), [2**16]),
(build_linear_bucket_options(1024), [step * 1024 / 60 for step in range(15, 31)]),
(build_linear_bucket_options(1024, 512), [step * 1024 / 512 for step in range(128, 257)]),
],
ids=["absent", "log_defaults", "the_finest_scale", "the_coarsest_scale", "linear_without_num_buckets", "the_most_buckets"],
)
def test_bucket_option_limits(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
bucket_options: dict | None,
expected_buckets: list[float],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
start_ms = int((now - timedelta(minutes=30)).timestamp() * 1000)
end_ms = int(now.timestamp() * 1000)
metric_name = "test_heatmap_bucket_option_limits"
values = [256, 512]
insert_metrics(
[
Metrics(
metric_name=metric_name,
labels={"service": "api"},
timestamp=now - timedelta(minutes=len(values) - minute),
value=value,
type_="Gauge",
is_monotonic=False,
)
for minute, value in enumerate(values)
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[build_builder_query("A", metric_name, "max", "max")],
request_type=RequestType.HEATMAP,
bucket_options=bucket_options,
)
assert response.status_code == HTTPStatus.OK, response.text
data = response.json()
assert get_heatmap_buckets(data, "A") == pytest.approx(expected_buckets)
columns = get_heatmap_columns(data, "A")
assert len(columns) == len(values)
for column in columns:
assert len(column["values"]) == len(expected_buckets) + 1
assert sum(column["values"]) == 1
def test_formula_heatmap(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
start_ms = int((now - timedelta(minutes=30)).timestamp() * 1000)
end_ms = int(now.timestamp() * 1000)
metric_name = "test_heatmap_formula"
values = [100, 260, 295, 512, 1500]
insert_metrics(
[
Metrics(
metric_name=metric_name,
labels={"service": "api"},
timestamp=now - timedelta(minutes=len(values) - minute),
value=value,
type_="Gauge",
is_monotonic=False,
)
for minute, value in enumerate(values)
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
from_metric = make_query_request(
signoz,
token,
start_ms,
end_ms,
[build_builder_query("A", metric_name, "max", "max")],
request_type=RequestType.HEATMAP,
)
assert from_metric.status_code == HTTPStatus.OK, from_metric.text
# a formula over the same query has to land its counts on the same axis
from_formula = make_query_request(
signoz,
token,
start_ms,
end_ms,
[
build_builder_query("A", metric_name, "max", "max", disabled=True),
build_formula_query("F1", "A"),
],
request_type=RequestType.HEATMAP,
)
assert from_formula.status_code == HTTPStatus.OK, from_formula.text
assert get_heatmap_buckets(from_formula.json(), "F1") == pytest.approx(get_heatmap_buckets(from_metric.json(), "A"))
assert [column["values"] for column in get_heatmap_columns(from_formula.json(), "F1")] == [column["values"] for column in get_heatmap_columns(from_metric.json(), "A")]
def test_promql_heatmap(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
end_ms = (int((datetime.now(tz=UTC) - timedelta(minutes=5)).timestamp() * 1000) // MINUTE_MS) * MINUTE_MS
start_ms = end_ms - MINUTE_MS
# the file's three columns are one minute apart, and the first sits a minute
# before the query window so the earliest step has something to increase over.
# Every counter in it stays above its own rise across a window, below which
# increase clips its back-extrapolation at the counter's zero point.
insert_metrics(Metrics.load_from_file(HISTOGRAM_COUNTERS_FILE, base_time=datetime.fromtimestamp((start_ms - MINUTE_MS) / 1000, tz=UTC)))
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[{"type": "promql", "spec": {"name": "A", "query": "sum by (le) (increase(heatmap_request_duration_bucket[2m]))", "step": 60}}],
request_type=RequestType.HEATMAP,
)
assert response.status_code == HTTPStatus.OK, response.text
data = response.json()
assert get_heatmap_buckets(data, "A") == [1, 2, 4]
series = get_all_series(data, "A")
assert len(series) == 1
# `le` is what the bucket axis is read off, so it is never a group label
assert series[0].get("labels") in (None, [])
# what the query returns per `le` is still cumulative across `le`, so each
# count here is its own minus the one below it, and `le=+Inf` has no finite
# bound to sit on and lands in the trailing slot. increase over a 2m window of
# minutely samples extrapolates one minute's rise to two, hence the scaling.
assert [column["values"] for column in get_heatmap_columns(data, "A")] == [[2, 6, 2, 2], [6, 0, 8, 4]]
def test_promql_heatmap_with_no_data(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
) -> None:
end_ms = (int((datetime.now(tz=UTC) - timedelta(minutes=5)).timestamp() * 1000) // MINUTE_MS) * MINUTE_MS
start_ms = end_ms - MINUTE_MS
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[{"type": "promql", "spec": {"name": "A", "query": "sum by (le) (increase(promql_heatmap_bucket_never_written[2m]))", "step": 60}}],
request_type=RequestType.HEATMAP,
)
# a window holding nothing is not a query that can never draw a heatmap, so
# it comes back empty where the latter is rejected
assert response.status_code == HTTPStatus.OK, response.text
data = response.json()
assert get_heatmap_buckets(data, "A") == []
assert get_heatmap_columns(data, "A") == []
def test_clickhouse_heatmap(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
start = now - timedelta(minutes=2)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
# a clickhouse statement names its bucket upper bound column `bucket`
response = make_query_request(
signoz,
token,
int(start.timestamp() * 1000),
int(now.timestamp() * 1000),
[
{
"type": "clickhouse_sql",
"spec": {
"name": "A",
"query": (f"SELECT toDateTime({int(start.timestamp())}) AS ts, toFloat64(10) AS bucket, toFloat64(3) AS `__result_0` UNION ALL SELECT toDateTime({int(start.timestamp())}) AS ts, toFloat64(20) AS bucket, toFloat64(7) AS `__result_0`"),
"disabled": False,
},
}
],
request_type=RequestType.HEATMAP,
)
assert response.status_code == HTTPStatus.OK, response.text
data = response.json()
assert get_heatmap_buckets(data, "A") == [10, 20]
# a clickhouse heatmap takes no bucketOptions, so the counts land exactly
# where the statement put them, plus the overflow slot
assert [column["values"] for column in get_heatmap_columns(data, "A")] == [[3, 7, 0]]
def test_cached_heatmap_matches_uncached(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
metric_name = "test_heatmap_cache"
# the first half sits 16x below the second, so the cached range and the fresh
# one reach disjoint parts of the axis and neither may lose its counts
insert_metrics(
[
Metrics(
metric_name=metric_name,
labels={"service": "api"},
timestamp=now - timedelta(minutes=60 - minute),
value=256 if minute < 15 else 4096,
type_="Gauge",
is_monotonic=False,
)
for minute in range(30)
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
query = [build_builder_query("A", metric_name, "max", "max")]
wide_start_ms = int((now - timedelta(minutes=60)).timestamp() * 1000)
wide_end_ms = int((now - timedelta(minutes=30)).timestamp() * 1000)
warmup = make_query_request(
signoz,
token,
wide_start_ms,
int((now - timedelta(minutes=45)).timestamp() * 1000),
query,
request_type=RequestType.HEATMAP,
no_cache=False,
)
assert warmup.status_code == HTTPStatus.OK, warmup.text
from_cache = make_query_request(signoz, token, wide_start_ms, wide_end_ms, query, request_type=RequestType.HEATMAP, no_cache=False)
assert from_cache.status_code == HTTPStatus.OK, from_cache.text
uncached = make_query_request(signoz, token, wide_start_ms, wide_end_ms, query, request_type=RequestType.HEATMAP, no_cache=True)
assert uncached.status_code == HTTPStatus.OK, uncached.text
assert_identical_query_response(from_cache, uncached)
# 256 and 4096 are 16x apart, which the axis covers at 16 buckets per 2x, and
# every column holds the one value its minute recorded
assert len(get_heatmap_buckets(uncached.json(), "A")) == 65
assert [sum(column["values"]) for column in get_heatmap_columns(uncached.json(), "A")] == [1] * 30
def test_metric_with_no_data(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
missing_metric = "test_heatmap_metric_that_is_never_written"
response = make_query_request(
signoz,
token,
int((now - timedelta(minutes=30)).timestamp() * 1000),
int(now.timestamp() * 1000),
[build_builder_query("A", missing_metric, "max", "max")],
request_type=RequestType.HEATMAP,
bucket_options=build_log_bucket_options(0),
)
# a metric with nothing in the window carries no type to choose an axis
# from, and that is an empty heatmap rather than a rejected request
assert response.status_code == HTTPStatus.OK, response.text
data = response.json()
assert get_heatmap_buckets(data, "A") == []
assert get_heatmap_columns(data, "A") == []
assert any(missing_metric in warning["message"] for warning in get_all_warnings(data))

View File

@@ -0,0 +1,292 @@
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from http import HTTPStatus
from uuid import uuid4
import pytest
from fixtures import types
from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
from fixtures.fs import get_testdata_file_path
from fixtures.metrics import Metrics
from fixtures.querier import (
RequestType,
build_builder_query,
build_formula_query,
build_function,
build_log_bucket_options,
get_error_message,
make_query_request,
)
HISTOGRAM_FILE = get_testdata_file_path("histogram_data_1h.jsonl")
MINUTE_MS = 60_000
METRIC_NAME = "test_heatmap_rejections"
QUERY = [build_builder_query("A", METRIC_NAME, "max", "max")]
@pytest.mark.parametrize(
"queries, request_options, expected_message",
[
pytest.param(
[{"type": "promql", "spec": {"name": "A", "query": METRIC_NAME}}],
{"bucket_options": build_log_bucket_options(2)},
"bucketOptions are not supported for promql heatmap requests",
id="bucket_options_on_a_promql_query",
),
pytest.param(
QUERY,
{"format_options": {"formatTableResultForUI": False, "fillGaps": True}},
"fillGaps is not supported for heatmap requests",
id="fill_gaps",
),
pytest.param(
[
build_builder_query("A", METRIC_NAME, "max", "max"),
build_builder_query("B", METRIC_NAME, "min", "min"),
],
{},
"a heatmap renders one distribution, but 2 queries are enabled",
id="two_enabled_queries",
),
pytest.param(
[
build_builder_query("A", METRIC_NAME, "max", "max"),
build_builder_query("B", METRIC_NAME, "min", "min", disabled=True),
build_formula_query("F1", "B"),
],
{},
"a heatmap renders one distribution, but 2 queries are enabled",
id="a_formula_beside_an_enabled_query",
),
pytest.param(
[build_builder_query("A", METRIC_NAME, "max", "max", disabled=True)],
{},
"a heatmap needs one enabled query, but every query is disabled",
id="only_a_disabled_query",
),
pytest.param(
[],
{},
"at least one query is required",
id="no_queries",
),
pytest.param(
[build_builder_query("A", METRIC_NAME, "max", "max", functions=[build_function("absolute")])],
{},
"functions are not supported for heatmap requests",
id="functions_on_the_query",
),
pytest.param(
[
build_builder_query("A", METRIC_NAME, "max", "max", disabled=True, functions=[build_function("absolute")]),
build_formula_query("F1", "A"),
],
{},
"functions are not supported for heatmap requests",
id="functions_on_a_disabled_formula_input",
),
pytest.param(
[
build_builder_query("A", METRIC_NAME, "max", "max", disabled=True),
build_formula_query("F1", "A", functions=[build_function("absolute")]),
],
{},
"functions are not supported for heatmap requests",
id="functions_on_the_formula",
),
pytest.param(
[
{
"type": "builder_query",
"spec": {
"name": "A",
"signal": "metrics",
"aggregations": [{"metricName": METRIC_NAME, "timeAggregation": "max", "spaceAggregation": "max"}],
"stepInterval": 60,
"having": {"expression": "value > 1"},
},
}
],
{},
"having is not supported for heatmap requests",
id="having_on_the_query",
),
pytest.param(
QUERY,
{"bucket_options": {"kind": "quadratic", "spec": {}}},
"invalid bucketOptions kind",
id="unknown_bucket_kind",
),
pytest.param(
QUERY,
{"bucket_options": {"kind": "log"}},
"bucketOptions spec is required",
id="log_without_a_spec",
),
pytest.param(
QUERY,
{"bucket_options": {"kind": "linear"}},
"bucketOptions spec is required",
id="linear_without_a_spec",
),
pytest.param(
QUERY,
{"bucket_options": {"kind": "linear", "spec": {"maxValue": 1000, "scale": 2}}},
'unknown field "scale" in linear buckets spec',
id="scale_under_the_linear_kind",
),
pytest.param(
QUERY,
{"bucket_options": {"kind": "log", "spec": {"scale": 5}}},
"scale must be between -4 and 4",
id="scale_above_the_maximum",
),
pytest.param(
QUERY,
{"bucket_options": {"kind": "log", "spec": {"scale": -5}}},
"scale must be between -4 and 4",
id="scale_below_the_minimum",
),
pytest.param(
QUERY,
{"bucket_options": {"kind": "linear", "spec": {"maxValue": 0}}},
"linear buckets need a finite maxValue greater than 0",
id="zero_max_value",
),
pytest.param(
QUERY,
{"bucket_options": {"kind": "linear", "spec": {"maxValue": -10}}},
"linear buckets need a finite maxValue greater than 0",
id="negative_max_value",
),
pytest.param(
QUERY,
{"bucket_options": {"kind": "linear", "spec": {"maxValue": 1000, "numBuckets": 513}}},
"numBuckets must be between 1 and 512",
id="too_many_buckets",
),
],
)
def test_heatmap_request_is_rejected(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
queries: list[dict],
request_options: dict,
expected_message: str,
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
int((now - timedelta(minutes=30)).timestamp() * 1000),
int(now.timestamp() * 1000),
queries,
request_type=RequestType.HEATMAP,
**request_options,
)
assert response.status_code == HTTPStatus.BAD_REQUEST, response.text
assert expected_message in get_error_message(response.json())
@pytest.mark.parametrize(
"request_type",
[RequestType.TIME_SERIES, RequestType.SCALAR, RequestType.RAW],
ids=["time_series", "scalar", "raw"],
)
def test_bucket_options_outside_a_heatmap(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
request_type: str,
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
int((now - timedelta(minutes=30)).timestamp() * 1000),
int(now.timestamp() * 1000),
QUERY,
request_type=request_type,
bucket_options=build_log_bucket_options(2),
)
assert response.status_code == HTTPStatus.BAD_REQUEST, response.text
assert "bucketOptions are only supported for heatmap requests" in get_error_message(response.json())
def test_promql_returning_no_le_is_rejected(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
metric = f"promql_heatmap_gauge_{uuid4().hex[:8]}"
end_ms = (int((datetime.now(tz=UTC) - timedelta(minutes=5)).timestamp() * 1000) // MINUTE_MS) * MINUTE_MS
start_ms = end_ms - MINUTE_MS
# the metric has to return something, since an empty result is the window
# having no data rather than a query that can never draw a heatmap
insert_metrics(
[
Metrics(
metric_name=metric,
labels={"service": "api"},
timestamp=datetime.fromtimestamp(ts_ms / 1000, tz=UTC),
value=42.0,
type_="Gauge",
is_monotonic=False,
)
for ts_ms in range(start_ms, end_ms + 1, MINUTE_MS)
]
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[{"type": "promql", "spec": {"name": "A", "query": metric, "step": 60}}],
request_type=RequestType.HEATMAP,
)
assert response.status_code == HTTPStatus.BAD_REQUEST, response.text
assert "no `le` labels to build a bucket axis from" in get_error_message(response.json())
def test_histogram_rejects_bucket_options(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_metrics: Callable[[list[Metrics]], None],
) -> None:
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
start_ms = int((now - timedelta(minutes=65)).timestamp() * 1000)
end_ms = int(now.timestamp() * 1000)
metric_name = "test_heatmap_histogram_with_bucket_options"
insert_metrics(
Metrics.load_from_file(
HISTOGRAM_FILE,
base_time=now - timedelta(minutes=60),
metric_name_override=metric_name,
)
)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms,
end_ms,
[build_builder_query("A", metric_name, "increase", "p50")],
request_type=RequestType.HEATMAP,
bucket_options=build_log_bucket_options(2),
)
assert response.status_code == HTTPStatus.BAD_REQUEST, response.text
assert "bucketOptions are not supported for histogram metrics" in get_error_message(response.json())