Compare commits

..

29 Commits

Author SHA1 Message Date
swapnil-signoz
e769b208a5 refactor: updating dashboard and integration config 2026-07-30 00:51:31 +05:30
swapnil-signoz
3a6442bd70 Merge branch 'issue_5737' into issue-5739 2026-07-30 00:02:28 +05:30
swapnil-signoz
9685fb6591 Merge branch 'main' of https://github.com/SigNoz/signoz into issue-5739 2026-07-29 23:58:55 +05:30
swapnil-signoz
22359a24b6 Merge branch 'main' of https://github.com/SigNoz/signoz into issue_5737 2026-07-29 23:45:32 +05:30
swapnil-signoz
2c032a1874 feat: adding gcp cloud storage service 2026-07-29 23:42:24 +05:30
swapnil-signoz
3acb5e5d68 Merge branch 'issue_5737' into issue-5739 2026-07-29 23:40:49 +05:30
swapnil-signoz
8366a31edd refactor: updating icon in dashboard JSON 2026-07-29 23:15:09 +05:30
Vinicius Lourenço
b88ee12cd5 chore(docs): add authz guide (#12317)
* chore(docs): add authz guide

* chore(assets): clarify what withAuthZContent means
2026-07-29 17:18:15 +00:00
Pandey
6f3dd0b7ad fix(querybuilder): give each clickhouse sql refusal its own error code (#12339)
Some checks failed
build-staging / prepare (push) Has been cancelled
build-staging / js-build (push) Has been cancelled
build-staging / go-build (push) Has been cancelled
build-staging / staging (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
* fix(querybuilder): give each clickhouse sql refusal its own error code

Every refusal used CodeInvalidInput, so the only way to tell why a statement was
refused was to match on the message. Counting the failures on a live deployment
meant regexing the parser's text out of the log, and grouping by cause meant
knowing which of five different messages belonged to the same underlying gap.

One code per reason instead, which arrives as exception.code on the warning that
LogIfStatementIsNotValid writes, so the failures can be grouped and alerted on
directly.

Pin the expected code on each failing case, so the mapping is checked rather
than assumed.

* chore(querybuilder): drop the comment above the error codes

The names say it.

* fix(querybuilder): separate the parser panic code from the parse failure

A panic is a parser defect worth alerting on; a parse failure is a grammar gap
that shows up in ordinary traffic. Sharing one code lumps the two together.
2026-07-29 15:36:10 +00:00
Pandey
d32d93cd39 chore(deps): bump clickhouse-sql-parser to v0.5.3 (#12331)
* chore(deps): bump clickhouse-sql-parser to v0.5.3

v0.5.3 stops lexing a signed number after a closing bracket as one literal, so
`(now()-1)` and `arr[1]-1` parse where they used to fail. That was the third of
the three gaps the validator trips over on real dashboard SQL, and the only one
where the statement had to be rewritten to be accepted.

Move the two queries it covers out of the failing set, which is what that test
exists to prompt. On the production corpus the rejection rate goes from 6.1% to
5.6% of v5 query shapes; the remaining two gaps, `interval` used as a column name
and the standard trim(BOTH x FROM y) syntax, are still open upstream.

* chore(querybuilder): link the upstream issue on the signed literal cases
2026-07-29 15:33:00 +00:00
swapnil-signoz
28f10b3567 chore: generating openapi spec 2026-07-29 20:28:55 +05:30
swapnil-signoz
5d6ee1b97a Merge branch 'main' of https://github.com/SigNoz/signoz into issue_5737 2026-07-29 20:16:24 +05:30
swapnil-signoz
a61cb1fc15 feat: adding gcp gke service 2026-07-29 20:15:24 +05:30
Tushar Vats
e9a931788c chore(statementbuilder): log key adjustment actions at debug level (#12337)
These fired at info on every query build across the audit, logs and traces
builders. The behavior is settled, so drop them to debug and remove the
TODOs that asked for exactly this.
2026-07-29 14:37:09 +00:00
Ashwin Bhatkal
e51c464417 fix(frontend): alias lodash to lodash-es to survive a page-level AMD loader (#12332)
@grafana/data's ESM build imports bare CJS `lodash` in ~28 modules.
lodash's UMD footer checks for an AMD loader before assigning
module.exports, so when a third-party script has already defined
window.define.amd, lodash takes that branch and never exports. rolldown
can't statically detect lodash's dynamic named exports, so those imports
compile to runtime namespace reads against an empty object and every
method is undefined.

lodash-es is the same version, real ESM, and tree-shakes.
2026-07-29 14:03:32 +00:00
swapnil-signoz
642c49db37 refactor: migrating dashboard to v6 2026-07-29 01:32:45 +05:30
swapnil-signoz
f98d2291fd Merge branch 'main' into issue_5721 2026-07-28 23:01:05 +05:30
swapnil-signoz
22833ea8fe Merge branch 'issue_5721' of https://github.com/SigNoz/signoz into issue_5721 2026-07-22 17:30:14 +05:30
swapnil-signoz
8eb0a24492 fix: correct typo and unit in compute engine dashboard 2026-07-22 17:29:22 +05:30
swapnil-signoz
7e0b19800c Merge branch 'main' into issue_5721 2026-07-22 17:13:37 +05:30
Swapnil Nakade
3bc476c2ed Merge branch 'main' into issue_5721 2026-07-22 16:32:23 +05:30
swapnil-signoz
6a834ee944 Merge branch 'main' into issue_5721 2026-07-22 12:57:39 +05:30
swapnil-signoz
ba6a78aea7 refactor: updating dashboard panels to use rate aggregation 2026-07-22 10:38:46 +05:30
swapnil-signoz
ca98231b18 feat: adding compute engine service 2026-07-22 10:10:59 +05:30
swapnil-signoz
1644ecd6fc refactor: updating dashboard panel to use rate function instead of hack 2026-07-22 01:32:13 +05:30
swapnil-signoz
f1e2e9f4f7 refactor: updating cpu utilization panel 2026-07-19 22:25:47 +05:30
swapnil-signoz
a59e372f07 refactor: extending width of uptime gauge panel 2026-07-14 22:59:52 +05:30
swapnil-signoz
ca368a5b38 refactor: updating dashboard title 2026-07-14 22:51:25 +05:30
swapnil-signoz
dab5ceee0d feat: adding gcp memorystore redis service 2026-07-14 22:40:31 +05:30
58 changed files with 3300 additions and 1160 deletions

View File

@@ -1497,6 +1497,8 @@ components:
- cloudsql_postgres
- memorystore_redis
- computeengine
- gke
- cloudstorage
type: string
CloudintegrationtypesServiceMetadata:
properties:

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 139 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 49 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 86 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 64 KiB

View File

@@ -0,0 +1,136 @@
# AuthZ Guide
How to structure a page so it works with the permission system.
We are migrating from the `ADMIN | EDITOR | VIEWER` roles to per-action checks. Instead of granting `VIEWER` and
exposing everything, a user can now be granted access to a single resource, eg: `Logs` only.
- [Prerequisites](#prerequisites)
- [Core rules](#core-rules)
- [Page patterns](#page-patterns)
- [What "blocked" means](#what-blocked-means)
- [Migration checklist](#migration-checklist)
- [Testing](#testing)
## Prerequisites
Check whether the resource your page represents is supported: see
[permissions.type.ts](../src/lib/authz/hooks/useAuthZ/permissions.type.ts) for the current resources and their allowed
verbs.
**If the resource is not listed there, skip authz for now**. The backend does not enforce it yet, so any frontend
check would be decorative. Revisit once the resource is generated into that file (it is auto-generated from the
backend).
## Core rules
These hold for every page. The per-pattern sections below only add to them.
1. **A page is always reachable, regardless of permission.** The intended action must be known before permission can
be checked, so the route renders first and individual pieces are gated.
2. **`update` requires both `read` and `update`.** A user who can write but cannot read the current value must not get
an edit affordance.
3. **`delete` is independent of `read`.** Delete controls stay visible for a user who holds `delete` but not `read`.
4. **Never gate per row.** If a user can `list`, render every row. Check `read` only when the row is opened (drawer or
detail route).
5. **Gate the narrowest thing that works**, a button over a section, a section over a page.
6. **A resource may be gated while a sub-resource is not.** A user without `read` on Service Accounts can still hold
`create` on API Keys, so blocking the outer container would hide work they are allowed to do.
7. **Verbs not covered here** (`attach`, `detach`, `assignee`) behave like `delete`: gate the control that triggers
them, not the surrounding content.
## Page patterns
### List page
![Visual rules for structuring a list page](./assets/list-example.svg)
Without `list`, but with any of `read` / `create` / `update`, only the table is blocked:
- Title, description, search filters and action buttons stay visible.
- Filters and any control that drives the table are non-interactive.
- The create button stays enabled if the user holds `create`.
### Edit page
![Visual rules for structuring an edit page](./assets/edit-example.svg)
Without `read`:
- Block the content with `withAuthZContent`, not the whole route.
- Keep delete visible (rule 3).
- Keep update blocked (rule 2).
### Drawer
![Visual rules for structuring a drawer](./assets/drawer-example.svg)
Without `read`:
- Block the drawer body, not the drawer itself.
- Prefer routing that carries the resource ID in the URL, so delete stays reachable without `read`.
- Keep update blocked (rule 2).
- Watch for sub-resources the user may still be allowed to act on (rule 6).
### Create
Without `create`:
| Entry point | Gate with |
| --------------------- | ------------------------------------------------------------- |
| Button | `AuthZButton` |
| Dedicated create page | `withAuthZPage` |
| Create drawer opened directly (deep link) | `withAuthZContent` on the body + `AuthZButton` on footer actions |
### Delete
Gate the control with `AuthZButton`, or `AuthZTooltip` for a non-button trigger such as an icon or menu item.
### Quick filters
![Visual rules for structuring a page with quick filters](./assets/quick-filters-example.svg)
## What "blocked" means
Blocking is always a visible denial, never a silent removal. Use the components in
[`lib/authz/components`](../src/lib/authz/components/README.md) rather than hand-rolling a check, they carry the
denial message and the loading state.
| Scope | Component | Denied state |
| --------------- | ----------------------------------------- | ------------------------------------------- |
| Button | `AuthZButton` | Disabled + tooltip |
| Any element | `AuthZTooltip` | Disabled child + tooltip |
| Section | `withAuthZContent` / `AuthZGuardContent` | Inline `PermissionDeniedCallout` |
| Page / route | `withAuthZPage` / `AuthZGuardPage` | `PermissionDeniedFullPage` |
| Custom fallback | `withAuthZ` / `AuthZGuard` | Whatever you pass as `fallback` |
Prefer the HOC (`withAuthZ*`); reach for the JSX guard (`AuthZGuard*`) when the gate depends on conditional rendering
and an HOC cannot express it. The components README has the full decision tree and how to build the `checks` array.
## Migration checklist
Use the existing components. Create a new one only if none fit.
- [ ] Can I apply a `withAuthZ*` variant directly, or must I extract the content into a component first?
- If extraction is needed, declare the new content component in the same file to keep the diff small, then move it
to its own file in a follow-up commit or PR.
- [ ] Is the layout structured to respect the [core rules](#core-rules)?
- If not, raise it in the frontend Slack channel before reshaping the page.
- [ ] Am I gating a shared component rather than a page or section?
- If so, it must stay functional with no permission. Example: the Query Builder works without field suggestions when
the user cannot read them.
## Testing
### Devtool
Press `Cmd + K` (`Ctrl + K` on Windows/Linux) to open the shortcuts, search for `AuthZ`, and pick the first
result. The devtool simulates granted and denied permissions across the UI.
Available only in local development, it is stripped from production builds.
### Unit tests
[`lib/authz/utils/README.md`](../src/lib/authz/utils/README.md) covers the `*.authz.test.tsx` naming convention, the
MSW handlers (`setupAuthzAdmin`, `setupAuthzDenyAll`, `setupAuthzDeny`, `setupAuthzAllow`,
`setupAuthzGrantByPrefix`), and how to test the loading state.

View File

@@ -2816,6 +2816,7 @@ export enum CloudintegrationtypesServiceIDDTO {
cloudsql_postgres = 'cloudsql_postgres',
memorystore_redis = 'memorystore_redis',
computeengine = 'computeengine',
gke = 'gke',
}
export type CloudintegrationtypesCloudIntegrationServiceDTOAnyOf = {
/**

View File

@@ -1 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="1200" height="800" viewBox="-19.2 -28.483 166.401 170.898"><g transform="translate(0 -7.034)"><linearGradient id="a" x1="64" x2="64" y1="7.034" y2="120.789" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="#4387fd"/><stop offset="1" stop-color="#4683ea"/></linearGradient><path fill="url(#a)" d="M27.79 115.217 1.54 69.749a11.5 11.5 0 0 1 0-11.499l26.25-45.467a11.5 11.5 0 0 1 9.96-5.75h52.5a11.5 11.5 0 0 1 9.959 5.75l26.25 45.467a11.5 11.5 0 0 1 0 11.5l-26.25 45.466a11.5 11.5 0 0 1-9.959 5.75h-52.5a11.5 11.5 0 0 1-9.96-5.75z"/></g><g transform="translate(0 -7.034)"><defs><path id="b" d="M27.791 115.217 1.541 69.749a11.5 11.5 0 0 1 0-11.499l26.25-45.467a11.5 11.5 0 0 1 9.959-5.75h52.5a11.5 11.5 0 0 1 9.96 5.75l26.25 45.467a11.5 11.5 0 0 1 0 11.5l-26.25 45.466a11.5 11.5 0 0 1-9.96 5.75h-52.5a11.5 11.5 0 0 1-9.959-5.75z"/></defs><clipPath id="c"><use xlink:href="#b" width="100%" height="100%" overflow="visible"/></clipPath><path d="m49.313 53.875-7.01 6.99 5.957 5.958-5.898 10.476 44.635 44.636 10.816.002L118.936 84 85.489 50.55z" clip-path="url(#c)" opacity=".07"/></g><path fill="#fff" d="M84.7 43.236H43.264c-.667 0-1.212.546-1.212 1.214v8.566c0 .666.546 1.212 1.212 1.212H84.7c.667 0 1.213-.546 1.213-1.212v-8.568c0-.666-.545-1.213-1.212-1.213m-6.416 7.976a2.484 2.484 0 0 1-2.477-2.48 2.475 2.475 0 0 1 2.477-2.477c1.37 0 2.48 1.103 2.48 2.477a2.48 2.48 0 0 1-2.48 2.48m6.415 8.491-41.436.002c-.667 0-1.212.546-1.212 1.214v8.565c0 .666.546 1.213 1.212 1.213H84.7c.667 0 1.213-.547 1.213-1.213v-8.567c0-.666-.545-1.214-1.212-1.214m-6.416 7.976a2.483 2.483 0 0 1-2.477-2.48 2.475 2.475 0 0 1 2.477-2.477 2.48 2.48 0 1 1 0 4.956"/></svg>
<svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" viewBox="0 0 24 24"><defs><style>.cls-1{fill:#aecbfa;}.cls-2{fill:#669df6;}.cls-3{fill:#4285f4;}</style></defs><title>Icon_24px_CloudStorage_Color</title><g data-name="Product Icons"><path class="cls-1" d="M2,4.5H22a0,0,0,0,1,0,0v5a0,0,0,0,1,0,0H2a0,0,0,0,1,0,0v-5A0,0,0,0,1,2,4.5Z"/><path class="cls-2" d="M2,14.5H22a0,0,0,0,1,0,0v5a0,0,0,0,1,0,0H2a0,0,0,0,1,0,0v-5A0,0,0,0,1,2,14.5Z"/><rect class="cls-3" x="4.5" y="6.33" width="3.33" height="1.33"/><rect class="cls-3" x="4.5" y="16.33" width="3.33" height="1.33"/><circle class="cls-3" cx="18.5" cy="7" r="1"/><circle class="cls-3" cx="18.5" cy="17" r="1"/></g></svg>

Before

Width:  |  Height:  |  Size: 1.7 KiB

After

Width:  |  Height:  |  Size: 689 B

View File

@@ -1 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24"><path d="M19.73 6.56a1.73 1.73 0 0 0-1.68 1.68 1.83 1.83 0 0 0 .89 1.48v6.9l-5.16 3.06.8 1.28 5.55-3.25a.84.84 0 0 0 .4-.69v-7.3a1.64 1.64 0 0 0 .89-1.48 1.61 1.61 0 0 0-1.69-1.68" style="fill:#aecbfa"/><path d="m18 5.48-5.61-3.16a1.18 1.18 0 0 0-.79 0L5.25 6a1.72 1.72 0 0 0-2.68 1.35A1.73 1.73 0 0 0 4.26 9a1.73 1.73 0 0 0 1.68-1.65L12 3.9l5.15 3ZM11.2 18.5a1.57 1.57 0 0 0-.89.29l-5.16-3V9.92H3.56v6.31a.84.84 0 0 0 .4.69L9.52 20v.09a1.69 1.69 0 0 0 3.37 0 1.65 1.65 0 0 0-1.69-1.59" data-name="Path" style="fill:#aecbfa"/><path d="M16.96 8.63 12.1 5.78 7.13 8.63l4.97 2.77z" data-name="Path" style="fill:#669df6"/><path d="M12.1 12.38 6.84 9.32v2.47l5.26 2.96zM12.1 15.73l-5.26-3.05v2.07l5.26 3.06z" style="fill:#669df6"/><path d="M12.09 12.38v2.37l5.26-3.06V9.33Zm4.32-.94a.38.38 0 1 1 0-.76.38.38 0 0 1 0 .76M12.09 15.73v2.07l5.26-3.05v-2.07Zm4.32-1.07a.38.38 0 1 1 0-.76.38.38 0 0 1 0 .76" style="fill:#4285f4"/></svg>
<svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" viewBox="0 0 24 24"><defs><style>.cls-1{fill:#4285f4;}.cls-1,.cls-2,.cls-4{fill-rule:evenodd;}.cls-2{fill:#669df6;}.cls-3,.cls-4{fill:#aecbfa;}</style></defs><title>Icon_24px_K8Engine_Color</title><g data-name="Product Icons"><g ><polygon class="cls-1" points="14.68 13.06 19.23 15.69 19.23 16.68 14.29 13.83 14.68 13.06"/><polygon class="cls-2" points="9.98 13.65 4.77 16.66 4.45 15.86 9.53 12.92 9.98 13.65"/><rect class="cls-3" x="11.55" y="3.29" width="0.86" height="5.78"/><path class="cls-4" d="M3.25,7V17L12,22l8.74-5V7L12,2Zm15.63,8.89L12,19.78,5.12,15.89V8.11L12,4.22l6.87,3.89v7.78Z"/><polygon class="cls-4" points="11.98 11.5 15.96 9.21 11.98 6.91 8.01 9.21 11.98 11.5"/><polygon class="cls-2" points="11.52 12.3 7.66 10.01 7.66 14.6 11.52 16.89 11.52 12.3"/><polygon class="cls-1" points="12.48 12.3 12.48 16.89 16.34 14.6 16.34 10.01 12.48 12.3"/></g></g></svg>

Before

Width:  |  Height:  |  Size: 988 B

After

Width:  |  Height:  |  Size: 941 B

View File

@@ -143,6 +143,12 @@ export default defineConfig(({ mode }): UserConfig => {
plugins,
resolve: {
alias: {
// @grafana/data imports bare CJS `lodash`, whose UMD footer checks for an
// AMD loader before assigning module.exports. Any third-party script that
// defines window.define.amd first leaves the bundled namespace empty, so
// every lodash method reached through it is undefined at runtime. lodash-es
// is the same version, real ESM, and tree-shakes.
lodash: 'lodash-es',
'@': resolve(__dirname, './src'),
utils: resolve(__dirname, './src/utils'),
types: resolve(__dirname, './src/types'),

2
go.mod
View File

@@ -4,7 +4,7 @@ go 1.25.7
require (
dario.cat/mergo v1.0.2
github.com/AfterShip/clickhouse-sql-parser v0.5.2
github.com/AfterShip/clickhouse-sql-parser v0.5.3
github.com/ClickHouse/clickhouse-go/v2 v2.44.0
github.com/DATA-DOG/go-sqlmock v1.5.2
github.com/SigNoz/clickhouse-go-mock v0.14.0

4
go.sum
View File

@@ -66,8 +66,8 @@ dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/AfterShip/clickhouse-sql-parser v0.5.2 h1:KCxdmvuVawVF4yl0ZS33q7olgmLZwvZG5BjWHp6o414=
github.com/AfterShip/clickhouse-sql-parser v0.5.2/go.mod h1:Qi3qvPTfZb/aFwI5V4WFOahgjsLJa4MzVijIAfwOhDw=
github.com/AfterShip/clickhouse-sql-parser v0.5.3 h1:6iap8XGjuSjD3w7r1UNrg66ljBugcv2P39s4eo/ZLRw=
github.com/AfterShip/clickhouse-sql-parser v0.5.3/go.mod h1:Qi3qvPTfZb/aFwI5V4WFOahgjsLJa4MzVijIAfwOhDw=
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible h1:fcYLmCpyNYRnvJbPerq7U0hS+6+I79yEDJBqVNcqUzU=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0 h1:fou+2+WFTib47nS+nz/ozhEBnvU96bKHy6LjRsY4E28=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0/go.mod h1:t76Ruy8AHvUAC8GfMWJMa0ElSbuIcO03NLpynfbgsPA=

View File

@@ -112,12 +112,15 @@ functionCall
;
/*
* Full-text search: search('needle') or scoped search('needle', body, ...).
* First param is the needle; the rest are field-context scopes (body/attribute/
* resource/log), quoted or bare. Handled in the visitor — no grammar change.
* Full-text search call: search('needle')
*
* Uses the shared functionParamList so future scoped forms like
* search(body, 'abc') / search(attribute, 'abc') need no grammar change. Today
* only a single needle is supported. Unlike bare/quoted free text (`fullText`),
* which only targets the body column, search() fans out across every field.
*/
searchCall
: SEARCH LPAREN valueList RPAREN
: SEARCH LPAREN functionParamList RPAREN
;
// Function parameters can be keys, single scalar values, or arrays

View File

@@ -29,19 +29,13 @@ func New(t *testing.T) flagger.Flagger {
// WithUseJSONBody returns a Flagger with use_json_body set to the given value.
func WithUseJSONBody(t *testing.T, enabled bool) flagger.Flagger {
return WithBooleanFlags(t, map[string]bool{
flagger.FeatureUseJSONBody.String(): enabled,
})
}
// WithBooleanFlags returns a Flagger with the given boolean feature flags set to
// the provided values (keyed by feature name, e.g. flagger.FeatureX.String()).
func WithBooleanFlags(t *testing.T, flags map[string]bool) flagger.Flagger {
t.Helper()
registry := flagger.MustNewRegistry()
cfg := flagger.Config{}
if len(flags) > 0 {
cfg.Config.Boolean = flags
if enabled {
cfg.Config.Boolean = map[string]bool{
flagger.FeatureUseJSONBody.String(): true,
}
}
fl, err := flagger.New(
context.Background(),

View File

@@ -0,0 +1,956 @@
{
"schemaVersion": "v6",
"image": "/assets/Logos/gcp-cloud-storage",
"name": "",
"generateName": true,
"tags": [
{
"key": "tag",
"value": "observability"
}
],
"spec": {
"display": {
"name": "GCP Cloud Storage Overview",
"description": "Dashboard for GCP Cloud Storage overview"
},
"variables": [
{
"kind": "ListVariable",
"spec": {
"display": {
"name": "project_id",
"description": "GCP Project ID"
},
"allowAllValue": false,
"allowMultiple": false,
"customAllValue": "",
"capturingRegexp": "",
"sort": "none",
"plugin": {
"kind": "signoz/DynamicVariable",
"spec": {
"name": "project_id",
"signal": "metrics"
}
},
"name": "project_id"
}
},
{
"kind": "ListVariable",
"spec": {
"display": {
"name": "bucket_name",
"description": "GCS bucket name"
},
"allowAllValue": true,
"allowMultiple": true,
"customAllValue": "",
"capturingRegexp": "",
"sort": "none",
"plugin": {
"kind": "signoz/DynamicVariable",
"spec": {
"name": "bucket_name",
"signal": "metrics"
}
},
"name": "bucket_name"
}
}
],
"panels": {
"1f0a4d5b-6c2e-4a71-8b3d-9e5c07a41d20": {
"kind": "Panel",
"spec": {
"display": {
"name": "Bucket size",
"description": "Total size of all objects in the bucket, grouped by storage class."
},
"plugin": {
"kind": "signoz/TablePanel",
"spec": {
"visualization": {
"timePreference": "global_time"
},
"formatting": {
"columnUnits": {
"A": "By"
},
"decimalPrecision": "2"
},
"thresholds": null
}
},
"queries": [
{
"kind": "scalar",
"spec": {
"name": "A",
"plugin": {
"kind": "signoz/BuilderQuery",
"spec": {
"name": "A",
"signal": "metrics",
"source": "",
"aggregations": [
{
"metricName": "storage.googleapis.com/storage/v2/total_bytes",
"temporality": "",
"timeAggregation": "latest",
"spaceAggregation": "sum",
"reduceTo": "last"
}
],
"disabled": false,
"filter": {
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
},
"groupBy": [
{
"name": "bucket_name",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "storage_class",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
}
],
"order": [],
"having": {
"expression": ""
},
"functions": [],
"legend": "Size"
}
}
}
}
],
"links": []
}
},
"29ed9824-21db-44c3-9537-12be458f5c20": {
"kind": "Panel",
"spec": {
"display": {
"name": "Total object bytes",
"description": "Total size of all objects in the bucket, grouped by storage class and type."
},
"plugin": {
"kind": "signoz/TimeSeriesPanel",
"spec": {
"visualization": {
"timePreference": "global_time",
"fillSpans": false
},
"formatting": {
"unit": "By",
"decimalPrecision": "2"
},
"chartAppearance": {
"lineInterpolation": "spline",
"showPoints": false,
"lineStyle": "solid",
"fillMode": "none",
"spanGaps": {
"fillOnlyBelow": false,
"fillLessThan": ""
}
},
"axes": {
"softMin": 0,
"softMax": 0,
"isLogScale": false
},
"legend": {
"position": "right",
"mode": "list",
"customColors": null
},
"thresholds": null
}
},
"queries": [
{
"kind": "time_series",
"spec": {
"name": "A",
"plugin": {
"kind": "signoz/BuilderQuery",
"spec": {
"name": "A",
"signal": "metrics",
"source": "",
"aggregations": [
{
"metricName": "storage.googleapis.com/storage/v2/total_bytes",
"temporality": "",
"timeAggregation": "latest",
"spaceAggregation": "sum",
"reduceTo": "avg"
}
],
"disabled": false,
"filter": {
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
},
"groupBy": [
{
"name": "bucket_name",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "storage_class",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "type",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
}
],
"order": [],
"having": {
"expression": ""
},
"functions": [],
"legend": "{{bucket_name}}/{{storage_class}}/{{type}}"
}
}
}
}
],
"links": []
}
},
"cbc53174-5527-4bce-b32d-317403849e48": {
"kind": "Panel",
"spec": {
"display": {
"name": "Total object count",
"description": "Total number of objects and multipart-uploads per bucket, grouped by storage class and type, where type can be live-object, noncurrent-object, soft-deleted-object or multipart-upload."
},
"plugin": {
"kind": "signoz/TimeSeriesPanel",
"spec": {
"visualization": {
"timePreference": "global_time",
"fillSpans": false
},
"formatting": {
"unit": "{count}",
"decimalPrecision": "2"
},
"chartAppearance": {
"lineInterpolation": "spline",
"showPoints": false,
"lineStyle": "solid",
"fillMode": "none",
"spanGaps": {
"fillOnlyBelow": false,
"fillLessThan": ""
}
},
"axes": {
"softMin": 0,
"softMax": 0,
"isLogScale": false
},
"legend": {
"position": "right",
"mode": "list",
"customColors": null
},
"thresholds": null
}
},
"queries": [
{
"kind": "time_series",
"spec": {
"name": "A",
"plugin": {
"kind": "signoz/BuilderQuery",
"spec": {
"name": "A",
"signal": "metrics",
"source": "",
"aggregations": [
{
"metricName": "storage.googleapis.com/storage/v2/total_count",
"temporality": "",
"timeAggregation": "latest",
"spaceAggregation": "sum",
"reduceTo": "avg"
}
],
"disabled": false,
"filter": {
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
},
"groupBy": [
{
"name": "bucket_name",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "storage_class",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "type",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
}
],
"order": [],
"having": {
"expression": ""
},
"functions": [],
"legend": "{{bucket_name}}/{{storage_class}}/{{type}}"
}
}
}
}
],
"links": []
}
},
"76e70ce5-47e4-4b5b-9715-49fa21c58b0c": {
"kind": "Panel",
"spec": {
"display": {
"name": "Deleted bytes",
"description": "Rate of deleted bytes per bucket, grouped by storage class."
},
"plugin": {
"kind": "signoz/TimeSeriesPanel",
"spec": {
"visualization": {
"timePreference": "global_time",
"fillSpans": false
},
"formatting": {
"unit": "By/s",
"decimalPrecision": "2"
},
"chartAppearance": {
"lineInterpolation": "spline",
"showPoints": false,
"lineStyle": "solid",
"fillMode": "none",
"spanGaps": {
"fillOnlyBelow": false,
"fillLessThan": ""
}
},
"axes": {
"softMin": 0,
"softMax": 0,
"isLogScale": false
},
"legend": {
"position": "right",
"mode": "list",
"customColors": null
},
"thresholds": null
}
},
"queries": [
{
"kind": "time_series",
"spec": {
"name": "A",
"plugin": {
"kind": "signoz/BuilderQuery",
"spec": {
"name": "A",
"signal": "metrics",
"source": "",
"aggregations": [
{
"metricName": "storage.googleapis.com/storage/v2/deleted_bytes",
"temporality": "",
"timeAggregation": "rate",
"spaceAggregation": "sum",
"reduceTo": "sum"
}
],
"disabled": false,
"filter": {
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
},
"groupBy": [
{
"name": "bucket_name",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "storage_class",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
}
],
"order": [],
"having": {
"expression": ""
},
"functions": [],
"legend": "{{bucket_name}}/{{storage_class}}"
}
}
}
}
],
"links": []
}
},
"c5af2813-34ec-41a9-8d53-c9acd67f569f": {
"kind": "Panel",
"spec": {
"display": {
"name": "Request count",
"description": "Rate of API calls, grouped by the API method name and response code."
},
"plugin": {
"kind": "signoz/TimeSeriesPanel",
"spec": {
"visualization": {
"timePreference": "global_time",
"fillSpans": false
},
"formatting": {
"unit": "{count}/s",
"decimalPrecision": "2"
},
"chartAppearance": {
"lineInterpolation": "spline",
"showPoints": false,
"lineStyle": "solid",
"fillMode": "none",
"spanGaps": {
"fillOnlyBelow": false,
"fillLessThan": ""
}
},
"axes": {
"softMin": 0,
"softMax": 0,
"isLogScale": false
},
"legend": {
"position": "right",
"mode": "list",
"customColors": null
},
"thresholds": null
}
},
"queries": [
{
"kind": "time_series",
"spec": {
"name": "A",
"plugin": {
"kind": "signoz/BuilderQuery",
"spec": {
"name": "A",
"signal": "metrics",
"source": "",
"aggregations": [
{
"metricName": "storage.googleapis.com/api/request_count",
"temporality": "",
"timeAggregation": "rate",
"spaceAggregation": "sum",
"reduceTo": "sum"
}
],
"disabled": false,
"filter": {
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
},
"groupBy": [
{
"name": "bucket_name",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "method",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "response_code",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
}
],
"order": [],
"having": {
"expression": ""
},
"functions": [],
"legend": "{{bucket_name}}/{{method}} - {{response_code}}"
}
}
}
}
],
"links": []
}
},
"70333210-68c3-4387-a6b7-afa2459d27dd": {
"kind": "Panel",
"spec": {
"display": {
"name": "Authentication count",
"description": "Rate of HMAC/RSA signed requests, grouped by authentication method, API method name and response code."
},
"plugin": {
"kind": "signoz/TimeSeriesPanel",
"spec": {
"visualization": {
"timePreference": "global_time",
"fillSpans": false
},
"formatting": {
"unit": "{count}/s",
"decimalPrecision": "2"
},
"chartAppearance": {
"lineInterpolation": "spline",
"showPoints": false,
"lineStyle": "solid",
"fillMode": "none",
"spanGaps": {
"fillOnlyBelow": false,
"fillLessThan": ""
}
},
"axes": {
"softMin": 0,
"softMax": 0,
"isLogScale": false
},
"legend": {
"position": "right",
"mode": "list",
"customColors": null
},
"thresholds": null
}
},
"queries": [
{
"kind": "time_series",
"spec": {
"name": "A",
"plugin": {
"kind": "signoz/BuilderQuery",
"spec": {
"name": "A",
"signal": "metrics",
"source": "",
"aggregations": [
{
"metricName": "storage.googleapis.com/authn/authentication_count",
"temporality": "",
"timeAggregation": "rate",
"spaceAggregation": "sum",
"reduceTo": "sum"
}
],
"disabled": false,
"filter": {
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
},
"groupBy": [
{
"name": "bucket_name",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "authentication_method",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "method",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "response_code",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
}
],
"order": [],
"having": {
"expression": ""
},
"functions": [],
"legend": "{{bucket_name}}/{{authentication_method}}/{{method}} - {{response_code}}"
}
}
}
}
],
"links": []
}
},
"7a98084c-2cd3-4146-a831-f5382162ffe5": {
"kind": "Panel",
"spec": {
"display": {
"name": "Received bytes",
"description": "Rate of bytes received over the network, grouped by the API method name and response code."
},
"plugin": {
"kind": "signoz/TimeSeriesPanel",
"spec": {
"visualization": {
"timePreference": "global_time",
"fillSpans": false
},
"formatting": {
"unit": "By/s",
"decimalPrecision": "2"
},
"chartAppearance": {
"lineInterpolation": "spline",
"showPoints": false,
"lineStyle": "solid",
"fillMode": "none",
"spanGaps": {
"fillOnlyBelow": false,
"fillLessThan": ""
}
},
"axes": {
"softMin": 0,
"softMax": 0,
"isLogScale": false
},
"legend": {
"position": "right",
"mode": "list",
"customColors": null
},
"thresholds": null
}
},
"queries": [
{
"kind": "time_series",
"spec": {
"name": "A",
"plugin": {
"kind": "signoz/BuilderQuery",
"spec": {
"name": "A",
"signal": "metrics",
"source": "",
"aggregations": [
{
"metricName": "storage.googleapis.com/network/received_bytes_count",
"temporality": "",
"timeAggregation": "rate",
"spaceAggregation": "sum",
"reduceTo": "sum"
}
],
"disabled": false,
"filter": {
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
},
"groupBy": [
{
"name": "bucket_name",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "method",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "response_code",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
}
],
"order": [],
"having": {
"expression": ""
},
"functions": [],
"legend": "{{bucket_name}}/{{method}} - {{response_code}}"
}
}
}
}
],
"links": []
}
},
"7138bf98-b41a-46d8-90ef-8e7470819cdf": {
"kind": "Panel",
"spec": {
"display": {
"name": "Sent bytes",
"description": "Rate of bytes sent over the network, grouped by the API method name and response code."
},
"plugin": {
"kind": "signoz/TimeSeriesPanel",
"spec": {
"visualization": {
"timePreference": "global_time",
"fillSpans": false
},
"formatting": {
"unit": "By/s",
"decimalPrecision": "2"
},
"chartAppearance": {
"lineInterpolation": "spline",
"showPoints": false,
"lineStyle": "solid",
"fillMode": "none",
"spanGaps": {
"fillOnlyBelow": false,
"fillLessThan": ""
}
},
"axes": {
"softMin": 0,
"softMax": 0,
"isLogScale": false
},
"legend": {
"position": "right",
"mode": "list",
"customColors": null
},
"thresholds": null
}
},
"queries": [
{
"kind": "time_series",
"spec": {
"name": "A",
"plugin": {
"kind": "signoz/BuilderQuery",
"spec": {
"name": "A",
"signal": "metrics",
"source": "",
"aggregations": [
{
"metricName": "storage.googleapis.com/network/sent_bytes_count",
"temporality": "",
"timeAggregation": "rate",
"spaceAggregation": "sum",
"reduceTo": "sum"
}
],
"disabled": false,
"filter": {
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
},
"groupBy": [
{
"name": "bucket_name",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "method",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
},
{
"name": "response_code",
"signal": "",
"fieldContext": "attribute",
"fieldDataType": "string"
}
],
"order": [],
"having": {
"expression": ""
},
"functions": [],
"legend": "{{bucket_name}}/{{method}} - {{response_code}}"
}
}
}
}
],
"links": []
}
}
},
"layouts": [
{
"kind": "Grid",
"spec": {
"items": [
{
"x": 0,
"y": 0,
"width": 6,
"height": 6,
"content": {
"$ref": "#/spec/panels/1f0a4d5b-6c2e-4a71-8b3d-9e5c07a41d20"
}
}
]
}
},
{
"kind": "Grid",
"spec": {
"display": {
"title": "Storage",
"collapse": {
"open": true
}
},
"items": [
{
"x": 0,
"y": 0,
"width": 6,
"height": 6,
"content": {
"$ref": "#/spec/panels/29ed9824-21db-44c3-9537-12be458f5c20"
}
},
{
"x": 6,
"y": 0,
"width": 6,
"height": 6,
"content": {
"$ref": "#/spec/panels/cbc53174-5527-4bce-b32d-317403849e48"
}
},
{
"x": 0,
"y": 6,
"width": 6,
"height": 6,
"content": {
"$ref": "#/spec/panels/76e70ce5-47e4-4b5b-9715-49fa21c58b0c"
}
}
]
}
},
{
"kind": "Grid",
"spec": {
"display": {
"title": "API & Authentication",
"collapse": {
"open": true
}
},
"items": [
{
"x": 0,
"y": 0,
"width": 6,
"height": 6,
"content": {
"$ref": "#/spec/panels/c5af2813-34ec-41a9-8d53-c9acd67f569f"
}
},
{
"x": 6,
"y": 0,
"width": 6,
"height": 6,
"content": {
"$ref": "#/spec/panels/70333210-68c3-4387-a6b7-afa2459d27dd"
}
}
]
}
},
{
"kind": "Grid",
"spec": {
"display": {
"title": "Network",
"collapse": {
"open": true
}
},
"items": [
{
"x": 0,
"y": 0,
"width": 6,
"height": 6,
"content": {
"$ref": "#/spec/panels/7a98084c-2cd3-4146-a831-f5382162ffe5"
}
},
{
"x": 6,
"y": 0,
"width": 6,
"height": 6,
"content": {
"$ref": "#/spec/panels/7138bf98-b41a-46d8-90ef-8e7470819cdf"
}
}
]
}
}
],
"duration": "",
"refreshInterval": "",
"links": []
}
}

View File

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" viewBox="0 0 24 24"><defs><style>.cls-1{fill:#aecbfa;}.cls-2{fill:#669df6;}.cls-3{fill:#4285f4;}</style></defs><title>Icon_24px_CloudStorage_Color</title><g data-name="Product Icons"><path class="cls-1" d="M2,4.5H22a0,0,0,0,1,0,0v5a0,0,0,0,1,0,0H2a0,0,0,0,1,0,0v-5A0,0,0,0,1,2,4.5Z"/><path class="cls-2" d="M2,14.5H22a0,0,0,0,1,0,0v5a0,0,0,0,1,0,0H2a0,0,0,0,1,0,0v-5A0,0,0,0,1,2,14.5Z"/><rect class="cls-3" x="4.5" y="6.33" width="3.33" height="1.33"/><rect class="cls-3" x="4.5" y="16.33" width="3.33" height="1.33"/><circle class="cls-3" cx="18.5" cy="7" r="1"/><circle class="cls-3" cx="18.5" cy="17" r="1"/></g></svg>

After

Width:  |  Height:  |  Size: 689 B

View File

@@ -0,0 +1,70 @@
{
"id": "cloudstorage",
"title": "GCP Cloud Storage",
"icon": "file://icon.svg",
"overview": "file://overview.md",
"supportedSignals": {
"metrics": true,
"logs": true
},
"dataCollected": {
"metrics": [
{
"name": "storage.googleapis.com/storage/v2/total_bytes",
"unit": "Bytes",
"type": "Gauge",
"description": ""
},
{
"name": "storage.googleapis.com/storage/v2/total_count",
"unit": "Count",
"type": "Gauge",
"description": ""
},
{
"name": "storage.googleapis.com/storage/v2/deleted_bytes",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "storage.googleapis.com/api/request_count",
"unit": "Count",
"type": "Sum",
"description": ""
},
{
"name": "storage.googleapis.com/network/received_bytes_count",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "storage.googleapis.com/network/sent_bytes_count",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "storage.googleapis.com/authn/authentication_count",
"unit": "Count",
"type": "Sum",
"description": ""
}
],
"logs": []
},
"telemetryCollectionStrategy": {
"gcp": {}
},
"assets": {
"dashboards": [
{
"id": "overview",
"title": "GCP Cloud Storage Overview",
"description": "Overview of GCP Cloud Storage metrics",
"definition": "file://assets/dashboards/overview.json"
}
]
}
}

View File

@@ -0,0 +1,3 @@
### Monitor GCP Cloud Storage with SigNoz
Collect key GCP Cloud Storage metrics and view them with an out of the box dashboard.

View File

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" viewBox="0 0 24 24"><defs><style>.cls-1{fill:#4285f4;}.cls-1,.cls-2,.cls-4{fill-rule:evenodd;}.cls-2{fill:#669df6;}.cls-3,.cls-4{fill:#aecbfa;}</style></defs><title>Icon_24px_K8Engine_Color</title><g data-name="Product Icons"><g ><polygon class="cls-1" points="14.68 13.06 19.23 15.69 19.23 16.68 14.29 13.83 14.68 13.06"/><polygon class="cls-2" points="9.98 13.65 4.77 16.66 4.45 15.86 9.53 12.92 9.98 13.65"/><rect class="cls-3" x="11.55" y="3.29" width="0.86" height="5.78"/><path class="cls-4" d="M3.25,7V17L12,22l8.74-5V7L12,2Zm15.63,8.89L12,19.78,5.12,15.89V8.11L12,4.22l6.87,3.89v7.78Z"/><polygon class="cls-4" points="11.98 11.5 15.96 9.21 11.98 6.91 8.01 9.21 11.98 11.5"/><polygon class="cls-2" points="11.52 12.3 7.66 10.01 7.66 14.6 11.52 16.89 11.52 12.3"/><polygon class="cls-1" points="12.48 12.3 12.48 16.89 16.34 14.6 16.34 10.01 12.48 12.3"/></g></g></svg>

After

Width:  |  Height:  |  Size: 941 B

View File

@@ -0,0 +1,124 @@
{
"id": "gke",
"title": "GCP Kubernetes Engine",
"icon": "file://icon.svg",
"overview": "file://overview.md",
"supportedSignals": {
"metrics": true,
"logs": true
},
"dataCollected": {
"metrics": [
{
"name": "kubernetes.io/container/cpu/limit_utilization",
"unit": "None",
"type": "Gauge",
"description": ""
},
{
"name": "kubernetes.io/container/cpu/request_utilization",
"unit": "None",
"type": "Gauge",
"description": ""
},
{
"name": "kubernetes.io/container/memory/limit_utilization",
"unit": "None",
"type": "Gauge",
"description": ""
},
{
"name": "kubernetes.io/container/memory/request_utilization",
"unit": "None",
"type": "Gauge",
"description": ""
},
{
"name": "kubernetes.io/container/restart_count",
"unit": "Count",
"type": "Sum",
"description": ""
},
{
"name": "kubernetes.io/pod/latencies/pod_first_ready",
"unit": "Seconds",
"type": "Gauge",
"description": ""
},
{
"name": "kubernetes.io/pod/network/received_bytes_count",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "kubernetes.io/pod/network/sent_bytes_count",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "kubernetes.io/pod/volume/utilization",
"unit": "None",
"type": "Gauge",
"description": ""
},
{
"name": "kubernetes.io/node/cpu/allocatable_utilization",
"unit": "None",
"type": "Gauge",
"description": ""
},
{
"name": "kubernetes.io/node/memory/allocatable_utilization",
"unit": "None",
"type": "Gauge",
"description": ""
},
{
"name": "kubernetes.io/node/ephemeral_storage/used_bytes",
"unit": "Bytes",
"type": "Gauge",
"description": ""
},
{
"name": "kubernetes.io/node/ephemeral_storage/allocatable_bytes",
"unit": "Bytes",
"type": "Gauge",
"description": ""
},
{
"name": "kubernetes.io/node/network/received_bytes_count",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "kubernetes.io/node/network/sent_bytes_count",
"unit": "Bytes",
"type": "Sum",
"description": ""
},
{
"name": "kubernetes.io/node/status_condition",
"unit": "None",
"type": "Gauge",
"description": ""
}
],
"logs": []
},
"telemetryCollectionStrategy": {
"gcp": {}
},
"assets": {
"dashboards": [
{
"id": "overview",
"title": "GCP Kubernetes Engine Overview",
"description": "Overview of GCP Kubernetes Engine metrics",
"definition": "file://assets/dashboards/overview.json"
}
]
}
}

View File

@@ -0,0 +1,3 @@
### Monitor GCP Kubernetes Engine with SigNoz
Collect key GCP Kubernetes Engine metrics and view them with an out of the box dashboard.

View File

@@ -35,7 +35,7 @@ func (c *conditionBuilder) ConditionFor(
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for rule state history.
// has/hasAny/hasAll/hasToken are logs-body-only; reject for rule state history.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}

File diff suppressed because one or more lines are too long

View File

@@ -137,8 +137,8 @@ func filterqueryParserInit() {
0, 0, 191, 19, 1, 0, 0, 0, 192, 190, 1, 0, 0, 0, 193, 194, 7, 2, 0, 0,
194, 21, 1, 0, 0, 0, 195, 196, 7, 3, 0, 0, 196, 197, 5, 1, 0, 0, 197, 198,
3, 26, 13, 0, 198, 199, 5, 2, 0, 0, 199, 23, 1, 0, 0, 0, 200, 201, 5, 27,
0, 0, 201, 202, 5, 1, 0, 0, 202, 203, 3, 18, 9, 0, 203, 204, 5, 2, 0, 0,
204, 25, 1, 0, 0, 0, 205, 210, 3, 28, 14, 0, 206, 207, 5, 5, 0, 0, 207,
0, 0, 201, 202, 5, 1, 0, 0, 202, 203, 3, 26, 13, 0, 203, 204, 5, 2, 0,
0, 204, 25, 1, 0, 0, 0, 205, 210, 3, 28, 14, 0, 206, 207, 5, 5, 0, 0, 207,
209, 3, 28, 14, 0, 208, 206, 1, 0, 0, 0, 209, 212, 1, 0, 0, 0, 210, 208,
1, 0, 0, 0, 210, 211, 1, 0, 0, 0, 211, 27, 1, 0, 0, 0, 212, 210, 1, 0,
0, 0, 213, 217, 3, 34, 17, 0, 214, 217, 3, 32, 16, 0, 215, 217, 3, 30,
@@ -2945,7 +2945,7 @@ type ISearchCallContext interface {
// Getter signatures
SEARCH() antlr.TerminalNode
LPAREN() antlr.TerminalNode
ValueList() IValueListContext
FunctionParamList() IFunctionParamListContext
RPAREN() antlr.TerminalNode
// IsSearchCallContext differentiates from other interfaces.
@@ -2992,10 +2992,10 @@ func (s *SearchCallContext) LPAREN() antlr.TerminalNode {
return s.GetToken(FilterQueryParserLPAREN, 0)
}
func (s *SearchCallContext) ValueList() IValueListContext {
func (s *SearchCallContext) FunctionParamList() IFunctionParamListContext {
var t antlr.RuleContext
for _, ctx := range s.GetChildren() {
if _, ok := ctx.(IValueListContext); ok {
if _, ok := ctx.(IFunctionParamListContext); ok {
t = ctx.(antlr.RuleContext)
break
}
@@ -3005,7 +3005,7 @@ func (s *SearchCallContext) ValueList() IValueListContext {
return nil
}
return t.(IValueListContext)
return t.(IFunctionParamListContext)
}
func (s *SearchCallContext) RPAREN() antlr.TerminalNode {
@@ -3064,7 +3064,7 @@ func (p *FilterQueryParser) SearchCall() (localctx ISearchCallContext) {
}
{
p.SetState(202)
p.ValueList()
p.FunctionParamList()
}
{
p.SetState(203)

View File

@@ -20,8 +20,6 @@ import (
"github.com/SigNoz/signoz/pkg/valuer"
)
const estimateTimeout = 5 * time.Second
const traceOutsideRangeWarn = "Query %s references a trace_id that exists between %s and %s (UTC) but lies outside the selected time range; adjust the time range to see results"
type builderQuery[T any] struct {
@@ -249,10 +247,6 @@ func (q *builderQuery[T]) Execute(ctx context.Context) (*qbtypes.Result, error)
return nil, err
}
if err := q.enforceEstimate(ctx, stmt); err != nil {
return nil, err
}
// Execute the query with proper context for partial value detection
result, err := q.executeWithContext(ctx, stmt.Query, stmt.Args)
if err != nil {
@@ -264,70 +258,6 @@ func (q *builderQuery[T]) Execute(ctx context.Context) (*qbtypes.Result, error)
return result, nil
}
// estimateRows runs EXPLAIN ESTIMATE for a cost-guarded statement and returns its
// estimated total scan rows. guarded=false means there is nothing to enforce (no
// CostGuard or a non-positive budget). A non-nil error means the query must be
// rejected: either the estimate itself failed (fail closed — we cannot honor the
// budget without it) or the parent context was cancelled (surfaced as-is). Callers
// enforce the budget so it can be applied per-statement or cumulatively.
func (q *builderQuery[T]) estimateRows(ctx context.Context, stmt *qbtypes.Statement) (int64, bool, error) {
if stmt.CostGuard == nil || stmt.CostGuard.MaxScanRows <= 0 {
return 0, false, nil
}
estCtx, cancel := context.WithTimeout(ctx, estimateTimeout)
defer cancel()
entries, err := q.telemetryStore.Estimate(estCtx, stmt.Query, stmt.Args...)
if err != nil {
// Parent cancellation isn't the budget's concern — surface it unchanged.
if ctx.Err() != nil {
return 0, true, ctx.Err()
}
// Fail closed: this is a scan-heavy statement and without an estimate we
// cannot bound its cost, so running it unbounded is exactly what the guard
// exists to prevent.
reason := fmt.Sprintf("This query is too broad to plan within %s; narrow the time range or add a more selective filter.", estimateTimeout)
if estCtx.Err() != context.DeadlineExceeded {
reason = "Could not estimate this query's scan cost; narrow the time range or add a more selective filter."
q.logger.WarnContext(ctx, "EXPLAIN ESTIMATE failed; rejecting scan-heavy query (fail-closed)", errors.Attr(err))
}
return 0, true, errors.NewInvalidInputf(errors.CodeInvalidInput, "%s", withAdvisory(stmt.CostGuard.Warning, reason))
}
var rows int64
for _, e := range entries {
rows += e.Rows
}
return rows, true, nil
}
// enforceEstimate rejects a single scan-heavy statement (Statement.CostGuard) whose
// EXPLAIN ESTIMATE rows exceed its budget, before executing. Budget 0 disables.
func (q *builderQuery[T]) enforceEstimate(ctx context.Context, stmt *qbtypes.Statement) error {
rows, guarded, err := q.estimateRows(ctx, stmt)
if err != nil {
return err
}
if !guarded {
return nil
}
if budget := stmt.CostGuard.MaxScanRows; rows > budget {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "%s",
withAdvisory(stmt.CostGuard.Warning, fmt.Sprintf("This query would scan about %d rows in this range, over the limit of %d; narrow the time range or add a more selective filter.", rows, budget)))
}
return nil
}
// withAdvisory prefixes reason with the requirement's advisory (e.g. the search()
// warning) when present, so the rejection leads with why the query is expensive.
func withAdvisory(advisory, reason string) string {
if advisory == "" {
return reason
}
return strings.TrimRight(advisory, ". ") + ". " + reason
}
// narrowWindowByTraceID inspects the filter for trace_id predicates and clamps
// [fromMS,toMS] to the time range stored in signoz_traces.distributed_trace_summary.
// Returns the (possibly narrowed) window, overlap=false when the trace lies
@@ -561,11 +491,6 @@ func (q *builderQuery[T]) executeWindowList(ctx context.Context) (*qbtypes.Resul
var warnings []string
var warningsDocURL string
// Cost guard is enforced against the cumulative estimate across the buckets we
// actually visit — a broad search() that fans every bucket must be bounded by
// the per-query budget, not let each bucket pass its slice independently.
var estimatedScan int64
for _, r := range buckets {
q.spec.Offset = 0
q.spec.Limit = need
@@ -576,17 +501,6 @@ func (q *builderQuery[T]) executeWindowList(ctx context.Context) (*qbtypes.Resul
}
warnings = stmt.Warnings
warningsDocURL = stmt.WarningsDocURL
rowsEst, guarded, err := q.estimateRows(ctx, stmt)
if err != nil {
return nil, err
}
if guarded {
estimatedScan += rowsEst
if budget := stmt.CostGuard.MaxScanRows; estimatedScan > budget {
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "%s",
withAdvisory(stmt.CostGuard.Warning, fmt.Sprintf("This query would scan about %d rows across the time range, over the limit of %d; narrow the time range or add a more selective filter.", estimatedScan, budget)))
}
}
// Execute with proper context for partial value detection
res, err := q.executeWithContext(ctx, stmt.Query, stmt.Args)
if err != nil {

View File

@@ -5,7 +5,6 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/statementbuilder"
)
const DefaultMaxConcurrentQueries = 8
@@ -20,10 +19,6 @@ type Config struct {
MaxConcurrentQueries int `yaml:"max_concurrent_queries" mapstructure:"max_concurrent_queries"`
// LogTraceIDWindowPadding is the padding added to narrowed down timerange from trace summary to logs with trace_id filter.
LogTraceIDWindowPadding time.Duration `yaml:"log_trace_id_window_padding" mapstructure:"log_trace_id_window_padding"`
// StatementBuilder config is embedded (squashed) so its keys live directly under
// the querier namespace, e.g. querier.skip_resource_fingerprint / querier.search_max_scan_rows.
statementbuilder.Config `mapstructure:",squash" yaml:",squash"`
}
// NewConfigFactory creates a new config factory for querier.
@@ -34,11 +29,10 @@ func NewConfigFactory() factory.ConfigFactory {
func newConfig() factory.Config {
return Config{
// Default values
CacheTTL: 168 * time.Hour,
FluxInterval: 5 * time.Minute,
CacheTTL: 168 * time.Hour,
FluxInterval: 5 * time.Minute,
MaxConcurrentQueries: DefaultMaxConcurrentQueries,
LogTraceIDWindowPadding: 5 * time.Minute,
Config: statementbuilder.NewConfig(),
}
}
@@ -56,10 +50,6 @@ func (c Config) Validate() error {
if c.LogTraceIDWindowPadding < 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "log_trace_id_window_padding must not be negative, got %v", c.LogTraceIDWindowPadding)
}
// The embedded statement-builder config's own Validate is shadowed by this method; call it explicitly.
if err := c.Config.Validate(); err != nil {
return err
}
return nil
}

View File

@@ -9,6 +9,16 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
)
var (
CodeClickHouseSQLParserPanic = errors.MustNewCode("clickhouse_sql_parser_panic")
CodeClickHouseSQLUnparseable = errors.MustNewCode("clickhouse_sql_unparseable")
CodeClickHouseSQLNotSingleStatement = errors.MustNewCode("clickhouse_sql_not_single_statement")
CodeClickHouseSQLNotSelect = errors.MustNewCode("clickhouse_sql_not_select")
CodeClickHouseSQLTableFunction = errors.MustNewCode("clickhouse_sql_table_function")
CodeClickHouseSQLInternalDatabase = errors.MustNewCode("clickhouse_sql_internal_database")
CodeClickHouseSQLReadonlyOverride = errors.MustNewCode("clickhouse_sql_readonly_override")
)
// internalDatabases hold server metadata, credentials and grants rather than telemetry.
var internalDatabases = map[string]struct{}{
"system": {},
@@ -20,30 +30,30 @@ func ErrIfStatementIsNotValid(query string) (err error) {
defer func() {
// The parser has a history of panicking on malformed input rather than returning an error.
if recovered := recover(); recovered != nil {
err = errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid ClickHouse SQL (recovered): %v", recovered)
err = errors.NewInvalidInputf(CodeClickHouseSQLParserPanic, "invalid ClickHouse SQL (recovered): %v", recovered)
}
}()
stmts, parseErr := chparser.NewParser(query).ParseStmts()
if parseErr != nil {
// Wrapped rather than formatted in, so that callers can recover the parser's *ParseError and read the position off it.
return errors.WrapInvalidInputf(parseErr, errors.CodeInvalidInput, "invalid ClickHouse SQL: %s", parseErr.Error())
return errors.WrapInvalidInputf(parseErr, CodeClickHouseSQLUnparseable, "invalid ClickHouse SQL: %s", parseErr.Error())
}
if len(stmts) != 1 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "ClickHouse SQL must contain exactly one statement, found %d statements", len(stmts))
return errors.NewInvalidInputf(CodeClickHouseSQLNotSingleStatement, "ClickHouse SQL must contain exactly one statement, found %d statements", len(stmts))
}
selectQuery, ok := stmts[0].(*chparser.SelectQuery)
if !ok {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "only SELECT statements are allowed in ClickHouse SQL queries")
return errors.NewInvalidInputf(CodeClickHouseSQLNotSelect, "only SELECT statements are allowed in ClickHouse SQL queries")
}
visitor := &chparser.DefaultASTVisitor{Visit: func(node chparser.Expr) error {
switch expr := node.(type) {
case *chparser.TableFunctionExpr:
// Source table functions remain usable in ClickHouse read-only mode.
return errors.NewInvalidInputf(errors.CodeInvalidInput, "ClickHouse table functions are not allowed in SQL queries: %s", chparser.Format(expr.Name))
return errors.NewInvalidInputf(CodeClickHouseSQLTableFunction, "ClickHouse table functions are not allowed in SQL queries: %s", chparser.Format(expr.Name))
case *chparser.TableIdentifier:
// Reading these is unaffected by ClickHouse read-only mode.
@@ -52,13 +62,13 @@ func ErrIfStatementIsNotValid(query string) (err error) {
}
if _, ok := internalDatabases[strings.ToLower(expr.Database.Name)]; ok {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "the ClickHouse %s database is not allowed in SQL queries", expr.Database.Name)
return errors.NewInvalidInputf(CodeClickHouseSQLInternalDatabase, "the ClickHouse %s database is not allowed in SQL queries", expr.Database.Name)
}
case *chparser.SettingExpr:
// A query-level setting takes precedence over the context setting.
if strings.EqualFold(expr.Name.Name, "readonly") {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "the ClickHouse readonly setting cannot be overridden")
return errors.NewInvalidInputf(CodeClickHouseSQLReadonlyOverride, "the ClickHouse readonly setting cannot be overridden")
}
}

View File

@@ -3,6 +3,8 @@ package querybuilder
import (
"testing"
"github.com/SigNoz/signoz/pkg/errors"
chparser "github.com/AfterShip/clickhouse-sql-parser/parser"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -30,10 +32,15 @@ func TestErrIfStatementIsNotValid_Pass(t *testing.T) {
{"TrailingUnterminatedBlockComment", "SELECT count() FROM t /* unterminated"},
// The rule keys on the database, not on the table name.
{"TableNamedSystemInTelemetryDatabase", "SELECT * FROM signoz_logs.system"},
{"SignedLiteralAfterClosingParen", "SELECT (toUnixTimestamp(now()) - 3600)*1000000000"},
{"SignedLiteralAfterClosingParenSpaced", "SELECT (toUnixTimestamp(now()) - 3600)*1000000000"},
// order by interval
{"OrderByInterval", "SELECT toStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS interval ORDER BY interval"},
{"OrderByIntervalAndDirection", "SELECT toStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS `interval` ORDER BY `interval` ASC"},
// Unspaced, so rejected until the parser stopped lexing a signed literal after a
// closing bracket. The spaced form above no longer needs to be spaced.
// https://github.com/AfterShip/clickhouse-sql-parser/issues/286
{"SignedLiteralAfterClosingParenUnspaced", "SELECT now() AS ts, toFloat64(count()) AS value FROM ( SELECT attributes_string['TableName'] AS T, attributes_string['MissingId'] AS M, max(fromUnixTimestamp64Nano(timestamp)) AS last_seen, dateDiff('minute', min(fromUnixTimestamp64Nano(timestamp)), max(fromUnixTimestamp64Nano(timestamp))) AS age_min FROM signoz_logs.distributed_logs_v2 WHERE body='missing_map_record' AND timestamp >= (toUnixTimestamp(now())-3600)*1000000000 GROUP BY T, M ) WHERE age_min >= 20 AND last_seen >= now() - toIntervalMinute(8)"},
{"SignedLiteralAfterClosingParenMinimal", "SELECT (1)-1"},
{"TrimFunction", "SELECT trimBoth('/api/endpoint/', '/');"},
}
@@ -47,49 +54,52 @@ func TestErrIfStatementIsNotValid_Pass(t *testing.T) {
func TestErrIfStatementIsNotValid_Fail(t *testing.T) {
testCases := []struct {
name string
query string
name string
query string
expectedCode errors.Code
}{
// Not a single statement, or not a statement at all.
{"Empty", ""},
{"UnterminatedBlockCommentOnly", "/* x"},
{"Unparseable", "SELECT FROM WHERE"},
{"MultipleStatements", "SELECT 1; DROP TABLE signoz_logs.logs_v2"},
{"Empty", "", CodeClickHouseSQLNotSingleStatement},
{"UnterminatedBlockCommentOnly", "/* x", CodeClickHouseSQLUnparseable},
{"Unparseable", "SELECT FROM WHERE", CodeClickHouseSQLUnparseable},
{"MultipleStatements", "SELECT 1; DROP TABLE signoz_logs.logs_v2", CodeClickHouseSQLNotSingleStatement},
// Parses, but is not a SELECT.
{"Drop", "DROP TABLE signoz_logs.logs_v2"},
{"Insert", "INSERT INTO signoz_logs.logs_v2 SELECT * FROM signoz_logs.logs_v2"},
{"AlterDelete", "ALTER TABLE signoz_logs.logs_v2 DELETE WHERE 1 = 1"},
{"CreateTable", "CREATE TABLE evil (a Int) ENGINE = Memory"},
{"Grant", "GRANT ALL ON *.* TO admin"},
{"Set", "SET readonly = 0"},
{"Drop", "DROP TABLE signoz_logs.logs_v2", CodeClickHouseSQLNotSelect},
{"Insert", "INSERT INTO signoz_logs.logs_v2 SELECT * FROM signoz_logs.logs_v2", CodeClickHouseSQLNotSelect},
{"AlterDelete", "ALTER TABLE signoz_logs.logs_v2 DELETE WHERE 1 = 1", CodeClickHouseSQLNotSelect},
{"CreateTable", "CREATE TABLE evil (a Int) ENGINE = Memory", CodeClickHouseSQLNotSelect},
{"Grant", "GRANT ALL ON *.* TO admin", CodeClickHouseSQLNotSelect},
{"Set", "SET readonly = 0", CodeClickHouseSQLNotSelect},
// These the parser rejects outright rather than classifying.
{"ShowGrants", "SHOW GRANTS"},
{"IntoOutfile", "SELECT * FROM t INTO OUTFILE '/tmp/x.csv'"},
{"ShowGrants", "SHOW GRANTS", CodeClickHouseSQLUnparseable},
{"IntoOutfile", "SELECT * FROM t INTO OUTFILE '/tmp/x.csv'", CodeClickHouseSQLUnparseable},
// Table functions, which read through something other than a telemetry table.
{"UrlTableFunction", "SELECT * FROM url('http://attacker.example/x', CSV, 'a String')"},
{"FileTableFunction", "SELECT * FROM file('/etc/passwd', CSV, 'a String')"},
{"ExecutableTableFunction", "SELECT * FROM executable('script.sh', CSV, 'a String')"},
{"TableFunctionInJoin", "SELECT * FROM t1 JOIN url('http://x', CSV, 'a String') u ON 1 = 1"},
{"TableFunctionInCommonTableExpression", "WITH c AS (SELECT * FROM url('http://x', CSV, 'a String')) SELECT * FROM c"},
{"TableFunctionInWhereSubquery", "SELECT * FROM t WHERE a IN (SELECT * FROM file('/etc/passwd', CSV, 'a String'))"},
{"TableFunctionInUnion", "SELECT * FROM t UNION ALL SELECT * FROM url('http://x', CSV, 'a String')"},
{"UrlTableFunction", "SELECT * FROM url('http://attacker.example/x', CSV, 'a String')", CodeClickHouseSQLTableFunction},
{"FileTableFunction", "SELECT * FROM file('/etc/passwd', CSV, 'a String')", CodeClickHouseSQLTableFunction},
{"ExecutableTableFunction", "SELECT * FROM executable('script.sh', CSV, 'a String')", CodeClickHouseSQLTableFunction},
{"TableFunctionInJoin", "SELECT * FROM t1 JOIN url('http://x', CSV, 'a String') u ON 1 = 1", CodeClickHouseSQLTableFunction},
{"TableFunctionInCommonTableExpression", "WITH c AS (SELECT * FROM url('http://x', CSV, 'a String')) SELECT * FROM c", CodeClickHouseSQLTableFunction},
{"TableFunctionInWhereSubquery", "SELECT * FROM t WHERE a IN (SELECT * FROM file('/etc/passwd', CSV, 'a String'))", CodeClickHouseSQLTableFunction},
{"TableFunctionInUnion", "SELECT * FROM t UNION ALL SELECT * FROM url('http://x', CSV, 'a String')", CodeClickHouseSQLTableFunction},
// Internal databases, which hold grants and server metadata rather than telemetry.
{"SystemUsers", "SELECT * FROM system.users"},
{"SystemUppercase", "SELECT * FROM SYSTEM.USERS"},
{"SystemQuoted", "SELECT count() FROM `system`.`tables`"},
{"SystemInSubquery", "SELECT * FROM (SELECT name FROM system.parts)"},
{"SystemInJoin", "SELECT * FROM signoz_logs.distributed_logs_v2 AS l JOIN system.users AS u ON 1 = 1"},
{"SystemInIntersect", "SELECT * FROM t INTERSECT SELECT * FROM system.users"},
{"InformationSchema", "SELECT * FROM information_schema.tables"},
{"SystemUsers", "SELECT * FROM system.users", CodeClickHouseSQLInternalDatabase},
{"SystemUppercase", "SELECT * FROM SYSTEM.USERS", CodeClickHouseSQLInternalDatabase},
{"SystemQuoted", "SELECT count() FROM `system`.`tables`", CodeClickHouseSQLInternalDatabase},
{"SystemInSubquery", "SELECT * FROM (SELECT name FROM system.parts)", CodeClickHouseSQLInternalDatabase},
{"SystemInJoin", "SELECT * FROM signoz_logs.distributed_logs_v2 AS l JOIN system.users AS u ON 1 = 1", CodeClickHouseSQLInternalDatabase},
{"SystemInIntersect", "SELECT * FROM t INTERSECT SELECT * FROM system.users", CodeClickHouseSQLInternalDatabase},
{"InformationSchema", "SELECT * FROM information_schema.tables", CodeClickHouseSQLInternalDatabase},
// A query-level setting takes precedence over the one the caller applies.
{"ReadonlySettingOverride", "SELECT * FROM t SETTINGS readonly = 0"},
{"ReadonlySettingOverrideAmongOthers", "SELECT * FROM t SETTINGS max_threads = 4, readonly = 0"},
{"ReadonlySettingOverride", "SELECT * FROM t SETTINGS readonly = 0", CodeClickHouseSQLReadonlyOverride},
{"ReadonlySettingOverrideAmongOthers", "SELECT * FROM t SETTINGS max_threads = 4, readonly = 0", CodeClickHouseSQLReadonlyOverride},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
err := ErrIfStatementIsNotValid(testCase.query)
assert.Error(t, err)
assert.True(t, errors.Asc(err, testCase.expectedCode), "expected code %s, got %v", testCase.expectedCode, err)
})
}
}
@@ -122,18 +132,6 @@ func TestErrIfStatementIsNotValid_ShouldPassButFails(t *testing.T) {
expectedStopsAfter: "trim(BOTH '",
fix: "SELECT trimBoth(replaceOne( JSONExtractString(body, 'Action'), 'health_status: ', '' ), ' ')",
},
{
name: "SignedLiteralAfterClosingParen",
query: "SELECT now() AS ts, toFloat64(count()) AS value FROM ( SELECT attributes_string['TableName'] AS T, attributes_string['MissingId'] AS M, max(fromUnixTimestamp64Nano(timestamp)) AS last_seen, dateDiff('minute', min(fromUnixTimestamp64Nano(timestamp)), max(fromUnixTimestamp64Nano(timestamp))) AS age_min FROM signoz_logs.distributed_logs_v2 WHERE body='missing_map_record' AND timestamp >= (toUnixTimestamp(now())-3600)*1000000000 GROUP BY T, M ) WHERE age_min >= 20 AND last_seen >= now() - toIntervalMinute(8)",
expectedStopsAfter: "(toUnixTimestamp(now())",
fix: "SELECT (toUnixTimestamp(now()) - 3600) * 1000000000",
},
{
name: "SignedLiteralAfterClosingParenMinimal",
query: "SELECT (1)-1",
expectedStopsAfter: "(1)",
fix: "SELECT (1) - 1",
},
}
for _, testCase := range testCases {

View File

@@ -8,10 +8,6 @@ const (
// BodyFullTextSearchDefaultWarning is emitted when a full-text search or "body" searches are hit
// with New JSON Body enhancements.
BodyFullTextSearchDefaultWarning = "Full text searches default to `body.message:string`. Use `body.<key>` to search a different field inside body"
// SearchWarning is emitted on every search() call. search() scans all fields,
// so it is slow and expensive; a specific field is cheaper.
SearchWarning = "search() runs across all fields and can be slow and expensive. Prefer a specific field, e.g. `<context>.<field_key>:<type>`"
)
var (

View File

@@ -161,16 +161,14 @@ func inferDataTypesFromList(values []any) []telemetrytypes.FieldDataType {
return out
}
// NewFunctionUnsupportedError returns the error for a has/hasAny/hasAll/hasToken/search
// operator on a builder that doesn't support it (logs only), or nil for other operators.
// NewFunctionUnsupportedError returns the error for a has/hasAny/hasAll/hasToken operator
// on a builder that doesn't support it (logs body only), or nil for other operators.
func NewFunctionUnsupportedError(operator qbtypes.FilterOperator) error {
switch operator {
case qbtypes.FilterOperatorHasToken:
return errors.NewInvalidInputf(errors.CodeInvalidInput, "function `hasToken` only supports body field as first parameter").WithUrl(hasTokenFunctionDocURL)
case qbtypes.FilterOperatorHas, qbtypes.FilterOperatorHasAny, qbtypes.FilterOperatorHasAll:
return errors.NewInvalidInputf(errors.CodeInvalidInput, "function `%s` supports only body JSON search", operator.FunctionName()).WithUrl(functionBodyJSONSearchDocURL)
case qbtypes.FilterOperatorSearch:
return errors.NewInvalidInputf(errors.CodeInvalidInput, "function `search` is only supported for logs")
default:
return nil
}

View File

@@ -43,8 +43,6 @@ type filterExpressionVisitor struct {
keysWithWarnings map[string]bool
startNs uint64
endNs uint64
requiresCostGuard bool
}
type FilterExprVisitorOpts struct {
@@ -83,10 +81,9 @@ func newFilterExpressionVisitor(opts FilterExprVisitorOpts) *filterExpressionVis
}
type PreparedWhereClause struct {
WhereClause *sqlbuilder.WhereClause
Warnings []string
WarningsDocURL string
RequiresCostGuard bool
WhereClause *sqlbuilder.WhereClause
Warnings []string
WarningsDocURL string
}
func (p PreparedWhereClause) IsEmpty() bool {
@@ -168,12 +165,12 @@ func PrepareWhereClause(query string, opts FilterExprVisitorOpts) (PreparedWhere
// Return empty where clause so callers can skip the WHERE clause
if cond == "" || cond == SkipConditionLiteral {
return PreparedWhereClause{WhereClause: nil, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL, RequiresCostGuard: visitor.requiresCostGuard}, nil
return PreparedWhereClause{WhereClause: nil, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL}, nil
}
whereClause := sqlbuilder.NewWhereClause().AddWhereExpr(visitor.builder.Args, cond)
return PreparedWhereClause{WhereClause: whereClause, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL, RequiresCostGuard: visitor.requiresCostGuard}, nil
return PreparedWhereClause{WhereClause: whereClause, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL}, nil
}
// Visit dispatches to the specific visit method based on node type.
@@ -779,79 +776,11 @@ func normalizeFunctionValue(operator qbtypes.FilterOperator, functionName string
return valueParams, nil
}
// VisitSearchCall handles search('needle') and its scoped forms, e.g.
// search('needle', body, resource). The first arg is the case-insensitive needle; the
// rest are field-context scopes (bare or quoted). Emits one FilterOperatorSearch per
// scope (none = keyless, covering every field) and ORs them.
// VisitSearchCall handles search('needle'). The search() function is parsed but
// not yet implemented; reject it with a clear invalid-input error.
func (v *filterExpressionVisitor) VisitSearchCall(ctx *grammar.SearchCallContext) any {
// Flag scan-heavy so the statement builder attaches the cost guard.
v.requiresCostGuard = true
valueList := ctx.ValueList()
if valueList == nil {
v.errors = append(v.errors, "function `search` expects a needle, e.g. search('error')")
return ErrorConditionLiteral
}
params := valueList.AllValue()
if len(params) == 0 {
v.errors = append(v.errors, "function `search` expects a needle, e.g. search('error')")
return ErrorConditionLiteral
}
searchText, ok := searchParamText(params[0])
if !ok {
v.errors = append(v.errors, "function `search` expects a needle as its first argument, e.g. search('error')")
return ErrorConditionLiteral
}
var fieldContexts []telemetrytypes.FieldContext
if len(params) == 1 {
fieldContexts = []telemetrytypes.FieldContext{telemetrytypes.FieldContextUnspecified}
} else {
for _, p := range params[1:] {
scopeText, sok := searchParamText(p)
if !sok {
v.errors = append(v.errors, "function `search` expects each scope to be a context, e.g. search('error', body, resource)")
return ErrorConditionLiteral
}
fc, fok := telemetrytypes.FieldContextFromText(scopeText)
if !fok {
v.errors = append(v.errors, fmt.Sprintf("invalid search scope %q; expected a field context: body, attribute, resource, or log", scopeText))
return ErrorConditionLiteral
}
fieldContexts = append(fieldContexts, fc)
}
}
var conds []string
for _, fieldContext := range fieldContexts {
key := telemetrytypes.NewTelemetryFieldKey("", fieldContext, telemetrytypes.FieldDataTypeUnspecified)
scoped, cok := v.buildConditions(key, nil, qbtypes.FilterOperatorSearch, searchText)
if !cok {
return ErrorConditionLiteral
}
conds = append(conds, scoped...)
}
if len(conds) == 0 {
return SkipConditionLiteral
}
if len(conds) == 1 {
return conds[0]
}
return v.builder.Or(conds...)
}
// searchParamText returns the raw token text of a search() argument (quoted or bare),
// not the visited value — so a bare word stays literal and search(1000000) doesn't
// become "1e+06".
func searchParamText(val grammar.IValueContext) (string, bool) {
if val == nil {
return "", false
}
if val.QUOTED_TEXT() != nil {
return trimQuotes(val.QUOTED_TEXT().GetText()), true
}
return val.GetText(), true
v.errors = append(v.errors, "function `search` is not yet supported")
return ErrorConditionLiteral
}
// VisitFunctionParamList handles the parameter list for function calls.

View File

@@ -39,6 +39,7 @@ import (
"github.com/SigNoz/signoz/pkg/sqlmigrator"
"github.com/SigNoz/signoz/pkg/sqlschema"
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statsreporter"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/tokenizer"
@@ -94,9 +95,12 @@ type Config struct {
// Alertmanager config
Alertmanager alertmanager.Config `mapstructure:"alertmanager" yaml:"alertmanager"`
// Querier config (the statement-builder config is embedded under it)
// Querier config
Querier querier.Config `mapstructure:"querier"`
// StatementBuilder config
StatementBuilder statementbuilder.Config `mapstructure:"statementbuilder"`
// Ruler config
Ruler ruler.Config `mapstructure:"ruler"`
@@ -166,6 +170,7 @@ func NewConfig(ctx context.Context, logger *slog.Logger, resolverConfig config.R
prometheus.NewConfigFactory(),
alertmanager.NewConfigFactory(),
querier.NewConfigFactory(),
statementbuilder.NewConfigFactory(),
ruler.NewConfigFactory(),
emailing.NewConfigFactory(),
sharder.NewConfigFactory(),

View File

@@ -122,7 +122,7 @@ func newQueryStack(
) {
metadataStore := telemetrymetadata.NewTelemetryMetaStore(settings, telemetryStore, fl)
cfg := config.Querier.Config
cfg := config.StatementBuilder
traceStmtBuilder, err := tracesstatementbuilder.NewFactory(telemetryStore, metadataStore, fl).New(ctx, settings, cfg)
if err != nil {
return nil, nil, nil, nil, nil, nil, nil, nil, err

View File

@@ -205,7 +205,7 @@ func (b *auditQueryStatementBuilder) adjustKeys(ctx context.Context, keys map[st
}
for _, action := range actions {
b.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
b.logger.DebugContext(ctx, "key adjustment action", slog.String("action", action))
}
return query

View File

@@ -2,6 +2,7 @@ package statementbuilder
import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
)
// SkipResourceFingerprint configures when the resource fingerprint subquery is skipped in favor of main-table filtering.
@@ -15,21 +16,19 @@ type SkipResourceFingerprint struct {
type Config struct {
// SkipResourceFingerprint configures when the resource fingerprint subquery is skipped in favor of main-table filtering.
SkipResourceFingerprint SkipResourceFingerprint `yaml:"skip_resource_fingerprint" mapstructure:"skip_resource_fingerprint"`
// SearchMaxScanRows caps the rows a search() query may scan; the log statement builder
// attaches it to the CostGuard and the querier enforces it via EXPLAIN ESTIMATE (0 disables).
SearchMaxScanRows int64 `yaml:"search_max_scan_rows" mapstructure:"search_max_scan_rows"`
}
// NewConfig returns the statement-builder config with its default values. It is
// embedded (squashed) into querier.Config rather than registered as a standalone
// config section, so its defaults are seeded from querier's newConfig.
func NewConfig() Config {
// NewConfigFactory creates a new config factory for the statement builders.
func NewConfigFactory() factory.ConfigFactory {
return factory.NewConfigFactory(factory.MustNewName("statementbuilder"), newConfig)
}
func newConfig() factory.Config {
return Config{
SkipResourceFingerprint: SkipResourceFingerprint{
Enabled: false,
Threshold: 100000,
},
SearchMaxScanRows: 0,
}
}
@@ -38,8 +37,5 @@ func (c Config) Validate() error {
if c.SkipResourceFingerprint.Enabled && c.SkipResourceFingerprint.Threshold == 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "skip_resource_fingerprint.threshold must be > 0 when enabled")
}
if c.SearchMaxScanRows < 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "search_max_scan_rows must not be negative, got %v", c.SearchMaxScanRows)
}
return nil
}

View File

@@ -8,7 +8,6 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -52,8 +51,7 @@ func TestStatementBuilderGroupByUnknownKey(t *testing.T) {
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore, fm, cb, aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
logstelemetryschema.DefaultFullTextColumn, fl, nil, false, 100000,
)
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{

View File

@@ -11,7 +11,6 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -1347,7 +1346,8 @@ func buildJSONTestStatementBuilder(t *testing.T, addIndexes bool) (*logQueryStat
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
false,
100000,
)
return statementBuilder, mockMetadataStore

View File

@@ -1,49 +0,0 @@
package logsstatementbuilder
import (
"context"
"testing"
"time"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/require"
)
// TestSearchCostGuard covers the search() path end-to-end through Build: the
// statement carries a CostGuard with its scan budget and the advisory warning.
func TestSearchCostGuard(t *testing.T) {
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
ctx := context.Background()
start := uint64(releaseTime.Add(-5 * time.Minute).UnixMilli())
end := uint64(releaseTime.UnixMilli())
fl := flaggertest.WithBooleanFlags(t, map[string]bool{})
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
store := telemetrytypestest.NewMockMetadataStore()
store.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
sb := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
store, fm, cb, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
statementbuilder.Config{SearchMaxScanRows: 100000, SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
)
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
Signal: telemetrytypes.SignalLogs,
Filter: &qbtypes.Filter{Expression: "search('error')"},
Limit: 1,
}
stmt, err := sb.Build(ctx, valuer.UUID{}, start, end, qbtypes.RequestTypeRaw, query, nil)
require.NoError(t, err)
require.NotNil(t, stmt.CostGuard)
require.Contains(t, stmt.Warnings, querybuilder.SearchWarning)
}

View File

@@ -41,15 +41,14 @@ type logQueryStatementBuilder struct {
fl flagger.Flagger
skipResourceFingerprintEnabled bool
fullTextColumn *telemetrytypes.TelemetryFieldKey
searchMaxScanRows int64
fullTextColumn *telemetrytypes.TelemetryFieldKey
}
var _ qbtypes.StatementBuilder[qbtypes.LogAggregation] = (*logQueryStatementBuilder)(nil)
// NewFactory returns a provider factory for the logs statement builder. Its New
// internalizes the FieldMapper, ConditionBuilder, and AggExprRewriter, and reads
// SkipResourceFingerprint and SearchMaxScanRows from the config.
// SkipResourceFingerprint from the config.
func NewFactory(
telemetryStore telemetrystore.TelemetryStore,
metadataStore telemetrytypes.MetadataStore,
@@ -63,7 +62,7 @@ func NewFactory(
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, logstelemetryschema.DefaultFullTextColumn, fm, cb, fl)
return NewLogQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, logstelemetryschema.DefaultFullTextColumn,
fl, telemetryStore, cfg,
fl, telemetryStore, cfg.SkipResourceFingerprint.Enabled, cfg.SkipResourceFingerprint.Threshold,
), nil
},
)
@@ -78,7 +77,8 @@ func NewLogQueryStatementBuilder(
fullTextColumn *telemetrytypes.TelemetryFieldKey,
fl flagger.Flagger,
telemetryStore telemetrystore.TelemetryStore,
cfg statementbuilder.Config,
skipResourceFingerprintEnable bool,
skipResourceFingerprintThreshold uint64,
) *logQueryStatementBuilder {
logsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema")
@@ -92,10 +92,10 @@ func NewLogQueryStatementBuilder(
fullTextColumn,
fl,
telemetryStore,
cfg.SkipResourceFingerprint.Threshold,
skipResourceFingerprintThreshold,
)
b := &logQueryStatementBuilder{
return &logQueryStatementBuilder{
logger: logsSettings.Logger(),
metadataStore: metadataStore,
fm: fieldMapper,
@@ -103,11 +103,9 @@ func NewLogQueryStatementBuilder(
resourceFilterResolver: resourceFilterResolver,
aggExprRewriter: aggExprRewriter,
fl: fl,
skipResourceFingerprintEnabled: cfg.SkipResourceFingerprint.Enabled,
searchMaxScanRows: cfg.SearchMaxScanRows,
skipResourceFingerprintEnabled: skipResourceFingerprintEnable,
fullTextColumn: fullTextColumn,
}
return b
}
// Build builds a SQL query for logs based on the given parameters.
@@ -153,23 +151,9 @@ func (b *logQueryStatementBuilder) Build(
}
stmt.Warnings = append(stmt.Warnings, warnings...)
// The search flag is gated in the condition builder; here the advisory rides on
// the statement so the querier can enforce the scan budget from the CostGuard.
if stmt.CostGuard != nil && stmt.CostGuard.Warning != "" {
stmt.Warnings = append(stmt.Warnings, stmt.CostGuard.Warning)
}
return stmt, nil
}
// costGuardFor builds the cost guard for a scan-heavy (search()) statement,
// pairing the advisory with the configured scan budget. Returns nil otherwise.
func (b *logQueryStatementBuilder) costGuardFor(required bool) *qbtypes.CostGuard {
if !required {
return nil
}
return &qbtypes.CostGuard{Warning: querybuilder.SearchWarning, MaxScanRows: b.searchMaxScanRows}
}
func getKeySelectors(query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation], bodyJSONEnabled bool) ([]*telemetrytypes.FieldKeySelector, []string) {
var keySelectors []*telemetrytypes.FieldKeySelector
var warnings []string
@@ -290,8 +274,7 @@ func (b *logQueryStatementBuilder) adjustKeys(ctx context.Context, keys map[stri
}
for _, action := range actions {
// TODO: change to debug level once we are confident about the behavior
b.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
b.logger.DebugContext(ctx, "key adjustment action", slog.String("action", action))
}
return query
@@ -407,7 +390,6 @@ func (b *logQueryStatementBuilder) buildListQuery(
Args: finalArgs,
Warnings: preparedWhereClause.Warnings,
WarningsDocURL: preparedWhereClause.WarningsDocURL,
CostGuard: b.costGuardFor(preparedWhereClause.RequiresCostGuard),
}
return stmt, nil
@@ -574,7 +556,6 @@ func (b *logQueryStatementBuilder) buildTimeSeriesQuery(
Args: finalArgs,
Warnings: preparedWhereClause.Warnings,
WarningsDocURL: preparedWhereClause.WarningsDocURL,
CostGuard: b.costGuardFor(preparedWhereClause.RequiresCostGuard),
}
return stmt, nil
@@ -702,7 +683,6 @@ func (b *logQueryStatementBuilder) buildScalarQuery(
Args: finalArgs,
Warnings: preparedWhereClause.Warnings,
WarningsDocURL: preparedWhereClause.WarningsDocURL,
CostGuard: b.costGuardFor(preparedWhereClause.RequiresCostGuard),
}
return stmt, nil

View File

@@ -11,7 +11,6 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
@@ -232,7 +231,8 @@ func TestStatementBuilderTimeSeries(t *testing.T) {
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
false,
100000,
)
for _, c := range cases {
@@ -374,7 +374,8 @@ func TestStatementBuilderListQuery(t *testing.T) {
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
false,
100000,
)
for _, c := range cases {
@@ -524,7 +525,8 @@ func TestStatementBuilderListQueryResourceTests(t *testing.T) {
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
false,
100000,
)
for _, c := range cases {
@@ -601,7 +603,8 @@ func TestStatementBuilderTimeSeriesBodyGroupBy(t *testing.T) {
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
false,
100000,
)
for _, c := range cases {
@@ -697,7 +700,8 @@ func TestStatementBuilderListQueryServiceCollision(t *testing.T) {
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
false,
100000,
)
for _, c := range cases {
@@ -922,7 +926,8 @@ func TestAdjustKey(t *testing.T) {
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
false,
100000,
)
for _, c := range cases {
@@ -1068,7 +1073,8 @@ func TestStmtBuilderBodyField(t *testing.T) {
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
false,
100000,
)
q, err := statementBuilder.Build(context.Background(), valuer.UUID{}, 1747947419000, 1747983448000, c.requestType, c.query, nil)
@@ -1168,7 +1174,8 @@ func TestStmtBuilderBodyFullTextSearch(t *testing.T) {
logstelemetryschema.DefaultFullTextColumn,
fl,
nil,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
false,
100000,
)
q, err := statementBuilder.Build(context.Background(), valuer.UUID{}, 1747947419000, 1747983448000, c.requestType, c.query, nil)
@@ -1289,6 +1296,7 @@ func newSkipResourceFingerprintLogsBuilder(
logstelemetryschema.DefaultFullTextColumn,
fl,
telemetryStore,
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: skipEnable, Threshold: threshold}},
skipEnable,
threshold,
)
}

View File

@@ -127,8 +127,7 @@ func (b *traceQueryStatementBuilder) Build(
}
for _, action := range adjustTraceKeys(keys, &query, requestType) {
// TODO: change to debug level once we are confident about the behavior
b.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
b.logger.DebugContext(ctx, "key adjustment action", slog.String("action", action))
}
// Create SQL builder
q := sqlbuilder.NewSelectBuilder()

View File

@@ -223,8 +223,7 @@ func (b *traceOperatorCTEBuilder) buildQueryCTE(ctx context.Context, queryName s
// The CTE only selects spans matching the filter. Aggregations, group by
// and order by run later in buildFinalQuery, so RequestTypeRaw is fine here.
for _, action := range adjustTraceKeys(keys, query, qbtypes.RequestTypeRaw) {
// TODO: change to debug level once we are confident about the behavior
b.stmtBuilder.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
b.stmtBuilder.logger.DebugContext(ctx, "key adjustment action", slog.String("action", action))
}
// Build resource filter CTE for this specific query
@@ -576,7 +575,7 @@ func (b *traceOperatorCTEBuilder) adjustOperatorKeys(ctx context.Context, keys m
b.operator.Order = tmp.Order
for _, action := range actions {
b.stmtBuilder.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
b.stmtBuilder.logger.DebugContext(ctx, "key adjustment action", slog.String("action", action))
}
}

View File

@@ -134,7 +134,7 @@ func (c *conditionBuilder) ConditionFor(
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for audit.
// has/hasAny/hasAll/hasToken are logs-body-only; reject for audit.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}

View File

@@ -3,7 +3,6 @@ package logstelemetryschema
import (
"context"
"fmt"
"regexp"
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
@@ -28,52 +27,6 @@ func NewConditionBuilder(fm qbtypes.FieldMapper, fl flagger.Flagger) *conditionB
return &conditionBuilder{fm: fm, fl: fl}
}
// conditionForSearch ORs a case-insensitive match of the needle across the searchable
// columns of the key's field context (an unspecified context covers every column).
func (c *conditionBuilder) conditionForSearch(
ctx context.Context,
orgID valuer.UUID,
key *telemetrytypes.TelemetryFieldKey,
value any,
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// Literal case-insensitive substring match: QuoteMeta the needle and LOWER both sides
// so the body path can use the LOWER(toString(body_v2)) skip index (a (?i) regex could not).
needle := regexp.QuoteMeta(fmt.Sprintf("%v", value))
useJSONBody := c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
var conditions []string
for _, col := range searchColumns(key.FieldContext, useJSONBody) {
switch col.Type.GetType() {
case schema.ColumnTypeEnumMap:
keysExpr := fmt.Sprintf("mapKeys(%s)", col.Name)
valsExpr := fmt.Sprintf("mapValues(%s)", col.Name)
// match() needs a String array; cast non-string map values first.
if mc, ok := col.Type.(schema.MapColumnType); ok && mc.ValueType.GetType() != schema.ColumnTypeEnumString {
valsExpr = fmt.Sprintf("arrayMap(x -> toString(x), mapValues(%s))", col.Name)
}
conditions = append(conditions, sb.Or(
fmt.Sprintf("arrayExists(x -> match(LOWER(x), LOWER(%s)), %s)", sb.Var(needle), keysExpr),
fmt.Sprintf("arrayExists(x -> match(LOWER(x), LOWER(%s)), %s)", sb.Var(needle), valsExpr),
))
case schema.ColumnTypeEnumJSON:
conditions = append(conditions, fmt.Sprintf("match(LOWER(toString(%s)), LOWER(%s))", col.Name, sb.Var(needle)))
case schema.ColumnTypeEnumString, schema.ColumnTypeEnumLowCardinality:
conditions = append(conditions, fmt.Sprintf("match(LOWER(%s), LOWER(%s))", col.Name, sb.Var(needle)))
default:
return nil, nil, errors.NewInternalf(errors.CodeInternal, "search does not support the column type of %q", col.Name)
}
}
if len(conditions) == 0 {
return nil, nil, nil
}
// The advisory rides on CostGuard (set by the visitor), not warnings.
return []string{sb.Or(conditions...)}, nil, nil
}
// isBodyJSONSearch reports whether a key addresses a path within the body JSON. Only
// an explicit Body context qualifies; a bare, context-less `body` (e.g. full-text
// `count_distinct(body)` or `body EXISTS`) is a full-text match, not a `$.body` path.
@@ -455,11 +408,6 @@ func (c *conditionBuilder) ConditionFor(
matches := querybuilder.MatchingFieldKeys(key, fieldKeys)
skipResourceFilter := options.SkipResourceFilter
// search() resolves its own (optional) scope; handle it before key resolution.
if operator == qbtypes.FilterOperatorSearch {
return c.conditionForSearch(ctx, orgID, key, value, sb)
}
keys, warning := querybuilder.ResolveKeys(key, matches)
var warnings []string
if warning != "" {

View File

@@ -635,37 +635,3 @@ func (m *fieldMapper) existsExpressionFor(
}
return querybuilder.ExistsExpression(columns, key, tsStart, tsEnd, fieldExpression, exists)
}
// searchColumns is the single source of truth for the columns search() fans out
// across, by field context. Body is body_v2 JSON when useJSONBody, else body string.
func searchColumns(fieldContext telemetrytypes.FieldContext, useJSONBody bool) []*schema.Column {
switch fieldContext {
case telemetrytypes.FieldContextLog:
return []*schema.Column{
logsV2Columns[LogsV2SeverityTextColumn],
logsV2Columns[LogsV2TraceIDColumn],
logsV2Columns[LogsV2SpanIDColumn],
}
case telemetrytypes.FieldContextBody:
if useJSONBody {
return []*schema.Column{logsV2Columns[LogsV2BodyV2Column]}
}
return []*schema.Column{logsV2Columns[LogsV2BodyColumn]}
case telemetrytypes.FieldContextAttribute:
return []*schema.Column{
logsV2Columns[LogsV2AttributesStringColumn],
logsV2Columns[LogsV2AttributesNumberColumn],
logsV2Columns[LogsV2AttributesBoolColumn],
}
case telemetrytypes.FieldContextResource:
return []*schema.Column{
logsV2Columns[LogsV2ResourcesStringColumn],
}
default:
columns := searchColumns(telemetrytypes.FieldContextLog, useJSONBody)
columns = append(columns, searchColumns(telemetrytypes.FieldContextBody, useJSONBody)...)
columns = append(columns, searchColumns(telemetrytypes.FieldContextAttribute, useJSONBody)...)
columns = append(columns, searchColumns(telemetrytypes.FieldContextResource, useJSONBody)...)
return columns
}
}

View File

@@ -1,288 +0,0 @@
package logstelemetryschema
import (
"context"
"testing"
"time"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/huandu/go-sqlbuilder"
"github.com/stretchr/testify/require"
)
// searchFanOut returns the WHERE fragment search() fans out to. bodyExpr is the
// body match expression, which differs between the legacy string body and the
// body_v2 JSON column.
func searchFanOut(bodyExpr string) string {
return "(match(LOWER(severity_text), LOWER(?)) OR match(LOWER(trace_id), LOWER(?)) OR match(LOWER(span_id), LOWER(?)) OR " +
bodyExpr + " OR " +
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(attributes_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(attributes_string))) OR " +
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(attributes_number)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), arrayMap(x -> toString(x), mapValues(attributes_number)))) OR " +
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(attributes_bool)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), arrayMap(x -> toString(x), mapValues(attributes_bool)))) OR " +
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(resources_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(resources_string))))"
}
// searchArgs returns v repeated once per bound parameter search() emits — one per
// searchable column expression (currently 12).
func searchArgs(v any) []any {
const searchColumnParams = 12
args := make([]any, searchColumnParams)
for i := range args {
args[i] = v
}
return args
}
// TestFilterExprSearch covers the search('needle') function, which fans out
// across every searchable column via FilterOperatorSearch.
func TestFilterExprSearch(t *testing.T) {
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
inWindowStart := uint64(releaseTime.Add(-5 * time.Minute).UnixNano())
inWindowEnd := uint64(releaseTime.Add(5 * time.Minute).UnixNano())
legacyBody := "match(LOWER(body), LOWER(?))"
jsonBody := "match(LOWER(toString(body_v2)), LOWER(?))"
// Single-context scope fragments (the fan-out narrowed to one context).
logScope := "(match(LOWER(severity_text), LOWER(?)) OR match(LOWER(trace_id), LOWER(?)) OR match(LOWER(span_id), LOWER(?)))"
resourceScope := "(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(resources_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(resources_string)))"
serviceNameEq := "(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? " +
"AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)"
testCases := []struct {
name string
query string
jsonBodyEnabled bool
fullTextColumn *telemetrytypes.TelemetryFieldKey
startNs uint64
endNs uint64
shouldPass bool
expectedQuery string
expectedArgs []any
expectWarning bool
expectedErrorContains string
}{
{
name: "quoted, legacy body",
query: "search('error')",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("error"),
expectWarning: true,
},
{
name: "quoted, json body",
query: "search('error')",
jsonBodyEnabled: true,
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(jsonBody),
expectedArgs: searchArgs("error"),
expectWarning: true,
},
{
name: "bare word",
query: "search(timeout)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("timeout"),
expectWarning: true,
},
{
name: "negated",
query: "NOT search('error')",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE NOT (" + searchFanOut(legacyBody) + ")",
expectedArgs: searchArgs("error"),
expectWarning: true,
},
{
name: "combined with field filter",
query: "search('error') AND service.name=\"api\"",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE (" + searchFanOut(legacyBody) + " AND " + serviceNameEq + ")",
expectedArgs: append(searchArgs("error"), "api"),
expectWarning: true,
},
{
// A wide window is allowed at build time; scan cost is bounded by the
// querier's EXPLAIN ESTIMATE gate, not a window cap in the builder.
name: "wide window builds (estimate gate lives in querier)",
query: "search('error')",
fullTextColumn: DefaultFullTextColumn,
startNs: uint64(releaseTime.Add(-10 * time.Hour).UnixNano()),
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("error"),
expectWarning: true,
},
{
// search() is keyless and independent of fullTextColumn (which only
// governs bare/quoted free text). It must work even when unset.
name: "independent of full text column",
query: "search('error')",
fullTextColumn: nil,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("error"),
expectWarning: true,
},
{
// A context-prefixed bare word must be used as the literal needle, not
// normalized into a field key (Normalize would strip "resource.").
name: "bare word with context prefix is not normalized",
query: "search(resource.deployment)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("resource\\.deployment"),
expectWarning: true,
},
{
// A numeric needle must be the literal digits, not the %v rendering of a
// parsed float64 (which would make search(1000000) scan for "1e+06").
name: "numeric needle is not scientific notation",
query: "search(1000000)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + searchFanOut(legacyBody),
expectedArgs: searchArgs("1000000"),
expectWarning: true,
},
{
name: "scoped to body, legacy",
query: "search('error', body)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE (" + legacyBody + ")",
expectedArgs: []any{"error"},
expectWarning: true,
},
{
name: "scoped to body, json",
query: "search('error', body)",
jsonBodyEnabled: true,
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE (" + jsonBody + ")",
expectedArgs: []any{"error"},
expectWarning: true,
},
{
name: "scoped to resource (quoted scope)",
query: "search('error', 'resource')",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE (" + resourceScope + ")",
expectedArgs: []any{"error", "error"},
expectWarning: true,
},
{
name: "scoped to log fields",
query: "search('error', log)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE " + logScope,
expectedArgs: []any{"error", "error", "error"},
expectWarning: true,
},
{
name: "scoped to multiple contexts",
query: "search('error', body, resource)",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: true,
expectedQuery: "WHERE ((" + legacyBody + ") OR (" + resourceScope + "))",
expectedArgs: []any{"error", "error", "error"},
expectWarning: true,
},
{
name: "invalid scope",
query: "search('error', 'timeout')",
fullTextColumn: DefaultFullTextColumn,
startNs: inWindowStart,
endNs: inWindowEnd,
shouldPass: false,
expectedErrorContains: "invalid search scope",
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
flagger.FeatureUseJSONBody.String(): tc.jsonBodyEnabled,
})
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
keys := BuildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),
Logger: instrumentationtest.New().Logger(),
FieldMapper: fm,
ConditionBuilder: cb,
FieldKeys: keys,
FullTextColumn: tc.fullTextColumn,
StartNs: tc.startNs,
EndNs: tc.endNs,
}
clause, err := querybuilder.PrepareWhereClause(tc.query, opts)
if !tc.shouldPass {
require.Error(t, err)
require.True(t, detailContains(err, tc.expectedErrorContains),
"error %v should contain %q", err, tc.expectedErrorContains)
return
}
require.NoError(t, err)
require.False(t, clause.IsEmpty())
sql, args := clause.WhereClause.BuildWithFlavor(sqlbuilder.ClickHouse)
require.Equal(t, tc.expectedQuery, sql)
require.Equal(t, tc.expectedArgs, args)
if tc.expectWarning {
// The visitor only flags the cost guard; the statement builder
// materializes the advisory + budget from config downstream.
require.True(t, clause.RequiresCostGuard)
}
})
}
}

View File

@@ -157,7 +157,7 @@ func (c *conditionBuilder) ConditionFor(
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for metrics.
// has/hasAny/hasAll/hasToken are logs-body-only; reject for metrics.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}

View File

@@ -205,7 +205,7 @@ func (c *conditionBuilder) ConditionFor(
sb *sqlbuilder.SelectBuilder,
) ([]string, []string, error) {
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for traces.
// has/hasAny/hasAll/hasToken are logs-body-only; reject for traces.
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
return nil, nil, err
}

View File

@@ -44,6 +44,8 @@ var (
GCPServiceCloudSQLPostgres = ServiceID{valuer.NewString("cloudsql_postgres")}
GCPServiceMemorystoreRedis = ServiceID{valuer.NewString("memorystore_redis")}
GCPServiceComputeEngine = ServiceID{valuer.NewString("computeengine")}
GCPServiceGKE = ServiceID{valuer.NewString("gke")}
GCPServiceCloudStorage = ServiceID{valuer.NewString("cloudstorage")}
)
func (ServiceID) Enum() []any {
@@ -78,6 +80,8 @@ func (ServiceID) Enum() []any {
GCPServiceCloudSQLPostgres,
GCPServiceMemorystoreRedis,
GCPServiceComputeEngine,
GCPServiceGKE,
GCPServiceCloudStorage,
}
}
@@ -118,6 +122,8 @@ var SupportedServices = map[CloudProviderType][]ServiceID{
GCPServiceCloudSQLPostgres,
GCPServiceMemorystoreRedis,
GCPServiceComputeEngine,
GCPServiceGKE,
GCPServiceCloudStorage,
},
}

View File

@@ -118,10 +118,6 @@ const (
FilterOperatorHasToken
FilterOperatorHasAny
FilterOperatorHasAll
// FilterOperatorSearch backs search('needle'): a keyless search the condition
// builder fans out across every searchable column.
FilterOperatorSearch
)
var operatorInverseMapping = map[FilterOperator]FilterOperator{
@@ -190,8 +186,7 @@ func (f FilterOperator) IsNegativeOperator() bool {
FilterOperatorIn,
FilterOperatorExists,
FilterOperatorRegexp,
FilterOperatorContains,
FilterOperatorSearch:
FilterOperatorContains:
return false
}
return true
@@ -248,11 +243,10 @@ func (f FilterOperator) IsArrayFunctionOperator() bool {
}
// IsFunctionOperator reports whether the operator is a query function
// (has/hasAny/hasAll/hasToken/search). These are logs-only; other signals reject
// them and the resource-fingerprint builder skips them.
// (has/hasAny/hasAll/hasToken); these apply to the logs body column only.
func (f FilterOperator) IsFunctionOperator() bool {
switch f {
case FilterOperatorHas, FilterOperatorHasAny, FilterOperatorHasAll, FilterOperatorHasToken, FilterOperatorSearch:
case FilterOperatorHas, FilterOperatorHasAny, FilterOperatorHasAll, FilterOperatorHasToken:
return true
default:
return false
@@ -271,8 +265,6 @@ func (f FilterOperator) FunctionName() string {
return "hasAll"
case FilterOperatorHasToken:
return "hasToken"
case FilterOperatorSearch:
return "search"
default:
return ""
}

View File

@@ -57,12 +57,6 @@ type Statement struct {
Args []any
Warnings []string
WarningsDocURL string
CostGuard *CostGuard
}
type CostGuard struct {
Warning string
MaxScanRows int64
}
// StatementBuilder builds the query.

View File

@@ -79,14 +79,6 @@ var (
}
)
// FieldContextFromText resolves a context word (e.g. "body", "resource") to its
// FieldContext, applying the same aliases as key parsing (e.g. "tag" -> attribute). ok
// is false for an unknown word, so callers can reject it instead of getting unspecified.
func FieldContextFromText(text string) (FieldContext, bool) {
fc, ok := fieldContexts[strings.ToLower(strings.TrimSpace(text))]
return fc, ok
}
// UnmarshalJSON implements the json.Unmarshaler interface.
func (f *FieldContext) UnmarshalJSON(data []byte) error {
var str string

View File

@@ -412,16 +412,10 @@ func (v *variableReplacementVisitor) VisitFunctionCall(ctx *grammar.FunctionCall
}
func (v *variableReplacementVisitor) VisitSearchCall(ctx *grammar.SearchCallContext) any {
if ctx.ValueList() == nil {
if ctx.FunctionParamList() == nil {
return "search()"
}
// VisitValueList already wraps the args in parens and returns the skip
// marker if any arg resolves to __all__.
result := v.Visit(ctx.ValueList()).(string)
if result == specialSkipMarker {
return specialSkipMarker
}
return "search" + result
return "search(" + v.Visit(ctx.FunctionParamList()).(string) + ")"
}
func (v *variableReplacementVisitor) VisitFunctionParamList(ctx *grammar.FunctionParamListContext) any {

View File

@@ -1,124 +0,0 @@
import json
from collections import namedtuple
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from http import HTTPStatus
import pytest
from fixtures import types
from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
from fixtures.logs import Logs
from fixtures.querier import build_order_by, build_raw_query, get_rows, make_query_request
# search() with use_json_body on (see conftest.py): body matches run against body_v2 via
# LOWER(toString(body_v2)); the map/log fan-out is unchanged from querierlogs/15_search.
# The response `body` comes back as parsed JSON, so a plain-string body is {"message": <body>}.
Bodies = namedtuple("Bodies", ["a", "b", "c", "d"])
@pytest.mark.parametrize(
"expression,expected",
[
# body matches now run through body_v2 (toString of {"message": <body>})
pytest.param("search('login')", lambda b: {b.a}, id="keyless_body"),
pytest.param("search('checkout')", lambda b: {b.a, b.d}, id="keyless_body_and_resource"),
pytest.param("search('CHECKOUT')", lambda b: {b.a, b.d}, id="keyless_case_insensitive"),
pytest.param("search('login', body)", lambda b: {b.a}, id="scope_body"),
pytest.param("search('checkout', body)", lambda b: {b.a}, id="scope_body_excludes_resource"),
# the map / log fan-out is flag-independent — sanity that it still works here
pytest.param("search('checkout', resource)", lambda b: {b.a, b.d}, id="scope_resource"),
pytest.param("search('acme', attribute)", lambda b: {b.a, b.c}, id="scope_attribute"),
pytest.param("search('error', log)", lambda b: {b.b, b.d}, id="scope_log_severity"),
pytest.param("search('checkout', body, resource)", lambda b: {b.a, b.d}, id="scopes_union"),
pytest.param("NOT search('login')", lambda b: {b.b, b.c, b.d}, id="negated"),
],
)
def test_search(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_logs: Callable[[list[Logs]], None],
expression: str,
expected: Callable[[Bodies], set[str]],
) -> None:
"""Four self-naming logs assert keyless/body-scoped search() matches through the
body_v2 JSON column, while resource/attribute/log scopes fan out unchanged."""
body = Bodies(
a="alpha checkout login ok", # service checkout / region useast / tenant acme / INFO
b="bravo declined", # service payment / region euwest / tenant globex / ERROR
c="charlie miss", # service cart / region useast / tenant acme / WARN
d="delta slow", # service checkout / region apac / tenant initech / ERROR
)
# (body, resources, attributes, severity_text)
specs = [
(body.a, {"service.name": "checkout", "region": "useast"}, {"tenant": "acme"}, "INFO"),
(body.b, {"service.name": "payment", "region": "euwest"}, {"tenant": "globex"}, "ERROR"),
(body.c, {"service.name": "cart", "region": "useast"}, {"tenant": "acme"}, "WARN"),
(body.d, {"service.name": "checkout", "region": "apac"}, {"tenant": "initech"}, "ERROR"),
]
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
logs = [Logs(timestamp=now - timedelta(seconds=i + 1), resources=res, attributes=attrs, body=b, severity_text=sev) for i, (b, res, attrs, sev) in enumerate(specs)]
insert_logs(logs)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms=int((now - timedelta(minutes=5)).timestamp() * 1000),
end_ms=int(now.timestamp() * 1000),
request_type="raw",
queries=[
build_raw_query(
"A",
"logs",
filter_expression=expression,
order=[build_order_by("timestamp", "desc"), build_order_by("id", "desc")],
limit=100,
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
# body_v2 comes back parsed; a plain-string body is {"message": <body>}.
assert {row["data"]["body"]["message"] for row in get_rows(response)} == expected(body)
@pytest.mark.parametrize(
"needle",
[
pytest.param("eve@acme.io", id="nested_string_value"),
pytest.param("503", id="nested_numeric_value"),
pytest.param("status", id="nested_key"),
],
)
def test_search_body_reaches_nested_json(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_logs: Callable[[list[Logs]], None],
needle: str,
) -> None:
"""A body-scoped search matches values and keys nested inside the body_v2 JSON."""
# searchable content lives only in nested fields, not a top-level message
nested_body = json.dumps({"user": {"email": "eve@acme.io"}, "http": {"status": 503}}, separators=(",", ":"))
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
insert_logs([Logs(timestamp=now - timedelta(seconds=1), resources={"service.name": "api"}, body=nested_body, severity_text="INFO")])
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms=int((now - timedelta(minutes=5)).timestamp() * 1000),
end_ms=int(now.timestamp() * 1000),
request_type="raw",
queries=[build_raw_query("A", "logs", filter_expression=f"search('{needle}', body)", order=[build_order_by("timestamp", "desc")], limit=100)],
)
assert response.status_code == HTTPStatus.OK, response.text
rows = get_rows(response)
assert len(rows) == 1
assert rows[0]["data"]["body"]["user"]["email"] == "eve@acme.io"

View File

@@ -31,8 +31,8 @@ def signoz_skip_resource_fingerprint(
pytestconfig=pytestconfig,
cache_key="signoz-skip-resource-fingerprint",
env_overrides={
"SIGNOZ_QUERIER_SKIP__RESOURCE__FINGERPRINT_ENABLED": True,
"SIGNOZ_QUERIER_SKIP__RESOURCE__FINGERPRINT_THRESHOLD": 2,
"SIGNOZ_STATEMENTBUILDER_SKIP__RESOURCE__FINGERPRINT_ENABLED": True,
"SIGNOZ_STATEMENTBUILDER_SKIP__RESOURCE__FINGERPRINT_THRESHOLD": 2,
},
)
@@ -66,6 +66,6 @@ def signoz_fingerprint(
pytestconfig=pytestconfig,
cache_key="signoz-skip-resource-fingerprint-baseline",
env_overrides={
"SIGNOZ_QUERIER_SKIP__RESOURCE__FINGERPRINT_ENABLED": False,
"SIGNOZ_STATEMENTBUILDER_SKIP__RESOURCE__FINGERPRINT_ENABLED": False,
},
)

View File

@@ -1,212 +0,0 @@
from collections import namedtuple
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
from http import HTTPStatus
import pytest
from fixtures import types
from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
from fixtures.logs import Logs
from fixtures.querier import build_order_by, build_raw_query, get_column_data_from_response, get_rows, make_query_request
# search(): keyless fans across every field; scoped search('needle', <ctx>...) narrows
# to the named contexts (body/attribute/resource/log). Flag off here, so body matches the
# `body` String column (querier_json_body mirrors this over body_v2). `expected` is a
# lambda over the body markers so cases stay declarative while the dataset lives in one place.
Bodies = namedtuple("Bodies", ["a", "b", "c", "d"])
@pytest.mark.parametrize(
"expression,expected",
[
# ── keyless: fans across every field ────────────────────────────────
pytest.param("search('login')", lambda b: {b.a}, id="keyless_body"),
pytest.param("search('checkout')", lambda b: {b.a, b.d}, id="keyless_body_and_resource"),
pytest.param("search('useast')", lambda b: {b.a, b.c}, id="keyless_resource_value"),
pytest.param("search('acme')", lambda b: {b.a, b.c}, id="keyless_attribute_value"),
pytest.param("search('tenant')", lambda b: {b.a, b.b, b.c, b.d}, id="keyless_attribute_key"),
pytest.param("search('error')", lambda b: {b.b, b.d}, id="keyless_severity_case_insensitive"),
pytest.param("search('CHECKOUT')", lambda b: {b.a, b.d}, id="keyless_needle_case_insensitive"),
# ── scoped: narrows to one context ──────────────────────────────────
pytest.param("search('login', body)", lambda b: {b.a}, id="scope_body"),
pytest.param("search('login', 'body')", lambda b: {b.a}, id="scope_body_quoted"),
pytest.param("search('checkout', body)", lambda b: {b.a}, id="scope_body_excludes_resource"),
pytest.param("search('checkout', resource)", lambda b: {b.a, b.d}, id="scope_resource"),
pytest.param("search('acme', attribute)", lambda b: {b.a, b.c}, id="scope_attribute"),
pytest.param("search('error', log)", lambda b: {b.b, b.d}, id="scope_log_severity"),
pytest.param("search('acme', body)", lambda b: set(), id="scope_body_no_match"),
pytest.param("search('checkout', attribute)", lambda b: set(), id="scope_attribute_no_match"),
# ── multiple scopes: union of the named contexts ────────────────────
pytest.param("search('login', body, resource)", lambda b: {b.a}, id="scopes_body_resource_body_only"),
pytest.param("search('checkout', body, resource)", lambda b: {b.a, b.d}, id="scopes_body_resource_union"),
# ── composition with boolean / field filters ────────────────────────
pytest.param("NOT search('login')", lambda b: {b.b, b.c, b.d}, id="negated"),
pytest.param("search('useast') AND severity_text = 'INFO'", lambda b: {b.a}, id="and_field_filter"),
],
)
def test_search(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_logs: Callable[[list[Logs]], None],
expression: str,
expected: Callable[[Bodies], set[str]],
) -> None:
"""Four self-naming logs, each with a token planted in a distinct place (body,
resource, attribute, severity), assert search() reaches exactly the right ones."""
body = Bodies(
a="alpha checkout login ok", # service checkout / region useast / tenant acme / INFO
b="bravo declined", # service payment / region euwest / tenant globex / ERROR
c="charlie miss", # service cart / region useast / tenant acme / WARN
d="delta slow", # service checkout / region apac / tenant initech / ERROR
)
# (body, resources, attributes, severity_text)
specs = [
(body.a, {"service.name": "checkout", "region": "useast"}, {"tenant": "acme"}, "INFO"),
(body.b, {"service.name": "payment", "region": "euwest"}, {"tenant": "globex"}, "ERROR"),
(body.c, {"service.name": "cart", "region": "useast"}, {"tenant": "acme"}, "WARN"),
(body.d, {"service.name": "checkout", "region": "apac"}, {"tenant": "initech"}, "ERROR"),
]
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
logs = [Logs(timestamp=now - timedelta(seconds=i + 1), resources=res, attributes=attrs, body=b, severity_text=sev) for i, (b, res, attrs, sev) in enumerate(specs)]
insert_logs(logs)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms=int((now - timedelta(minutes=5)).timestamp() * 1000),
end_ms=int(now.timestamp() * 1000),
request_type="raw",
queries=[
build_raw_query(
"A",
"logs",
filter_expression=expression,
order=[build_order_by("timestamp", "desc"), build_order_by("id", "desc")],
limit=100,
)
],
)
assert response.status_code == HTTPStatus.OK, response.text
assert response.json()["status"] == "success"
assert set(get_column_data_from_response(response.json(), "body")) == expected(body)
@pytest.mark.parametrize(
"expression",
[
pytest.param("search('login', bogus)", id="unknown_scope_word"),
pytest.param("search('login', body.message)", id="qualified_field_not_a_scope"),
],
)
def test_search_invalid_scope_rejected(
signoz: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
expression: str,
) -> None:
"""A scope that is not a field context (an unknown word, or a qualified
`context.field`) is rejected at build time with a 400."""
now = datetime.now(tz=UTC)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
response = make_query_request(
signoz,
token,
start_ms=int((now - timedelta(minutes=5)).timestamp() * 1000),
end_ms=int(now.timestamp() * 1000),
request_type="raw",
queries=[build_raw_query("A", "logs", filter_expression=expression, limit=100)],
)
assert response.status_code == HTTPStatus.BAD_REQUEST, response.text
assert "invalid search scope" in response.text
# search() is scan-heavy, so the querier gates it on EXPLAIN ESTIMATE against
# search_max_scan_rows (50000 for this instance, see conftest.py). A broad search over a
# busy range is rejected; the two ways the advisory suggests to recover — add a more
# selective filter, or narrow the time range — both bring the scan under budget.
def test_search_cost_guard_trips_then_passes_with_filter(
signoz_search_scan_budget: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_logs: Callable[[list[Logs]], None],
) -> None:
"""A broad search() over ~61000 logs exceeds the 50000-row budget and is rejected;
the same search narrowed to the 1000-log 'checkout' service scans under budget and
succeeds — the advisory's "add a more selective filter" made real."""
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
# 60000 'catalog' logs in the current bucket + 1000 'checkout' logs ~45m back (an
# earlier ts_bucket bucket), so the checkout fingerprint occupies its own marks and a
# resource filter on it prunes the scan.
logs = [Logs(timestamp=now - timedelta(seconds=1 + i % 30), resources={"service.name": "catalog"}, body="log line") for i in range(60000)]
logs += [Logs(timestamp=now - timedelta(minutes=45, seconds=i % 30), resources={"service.name": "checkout"}, body="log line") for i in range(1000)]
insert_logs(logs)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
start_ms = int((now - timedelta(minutes=60)).timestamp() * 1000)
end_ms = int((now + timedelta(minutes=1)).timestamp() * 1000)
def run(expression: str):
return make_query_request(
signoz_search_scan_budget,
token,
start_ms=start_ms,
end_ms=end_ms,
request_type="raw",
queries=[build_raw_query("A", "logs", filter_expression=expression, order=[build_order_by("timestamp", "desc")], limit=100)],
)
# Broad search over the whole range is over budget -> rejected before executing.
over_budget = run("search('log')")
assert over_budget.status_code == HTTPStatus.BAD_REQUEST, over_budget.text
assert "over the limit" in over_budget.text
# Adding a selective resource filter prunes the scan under budget -> runs.
within_budget = run("search('log') AND resource.service.name = 'checkout'")
assert within_budget.status_code == HTTPStatus.OK, within_budget.text
assert len(get_rows(within_budget)) > 0
def test_search_cost_guard_passes_with_narrower_time_range(
signoz_search_scan_budget: types.SigNoz,
create_user_admin: None, # pylint: disable=unused-argument
get_token: Callable[[str, str], str],
insert_logs: Callable[[list[Logs]], None],
) -> None:
"""A steady stream of ~80000 logs (~4/sec over the last ~5.5h). A search() over the
whole window is over the 50000-row budget and rejected; the same search over the last
15 minutes scans only a few thousand rows and runs — the advisory's "narrow the time
range" made real."""
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
# ~4 logs/sec, oldest ~5.5h back, newest ~now.
logs = [Logs(timestamp=now - timedelta(seconds=i // 4), resources={"service.name": "app"}, body="log line") for i in range(80000)]
insert_logs(logs)
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
def run(lookback_minutes: int):
return make_query_request(
signoz_search_scan_budget,
token,
start_ms=int((now - timedelta(minutes=lookback_minutes)).timestamp() * 1000),
end_ms=int((now + timedelta(minutes=1)).timestamp() * 1000),
request_type="raw",
queries=[build_raw_query("A", "logs", filter_expression="search('log')", order=[build_order_by("timestamp", "desc")], limit=100)],
)
# The last 6 hours cover the whole stream (~80000) -> over budget -> rejected.
wide = run(360)
assert wide.status_code == HTTPStatus.BAD_REQUEST, wide.text
assert "over the limit" in wide.text
# The last 15 minutes hold only ~3600 logs -> under budget -> runs, returning rows.
narrow = run(15)
assert narrow.status_code == HTTPStatus.OK, narrow.text
assert len(get_rows(narrow)) > 0

View File

@@ -1,35 +0,0 @@
import pytest
from testcontainers.core.container import Network
from fixtures import types
from fixtures.signoz import create_signoz
@pytest.fixture(name="signoz_search_scan_budget", scope="package")
def signoz_search_scan_budget(
network: Network,
migrator: types.Operation, # pylint: disable=unused-argument
zeus: types.TestContainerDocker,
gateway: types.TestContainerDocker,
sqlstore: types.TestContainerSQL,
clickhouse: types.TestContainerClickhouse,
request: pytest.FixtureRequest,
pytestconfig: pytest.Config,
) -> types.SigNoz:
"""SigNoz with a conservative search_max_scan_rows (50000) so a broad search() over a
busy range trips the cost guard, while a more selective one (by service or by a
narrower time range) stays under it. Shares the default instance's sqlstore +
clickhouse, so the same admin token and seeded logs work against it."""
return create_signoz(
network=network,
zeus=zeus,
gateway=gateway,
sqlstore=sqlstore,
clickhouse=clickhouse,
request=request,
pytestconfig=pytestconfig,
cache_key="signoz-search-scan-budget-50k",
env_overrides={
"SIGNOZ_QUERIER_SEARCH__MAX__SCAN__ROWS": 50000,
},
)