Compare commits
29 Commits
b3-search
...
issue-5739
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e769b208a5 | ||
|
|
3a6442bd70 | ||
|
|
9685fb6591 | ||
|
|
22359a24b6 | ||
|
|
2c032a1874 | ||
|
|
3acb5e5d68 | ||
|
|
8366a31edd | ||
|
|
b88ee12cd5 | ||
|
|
6f3dd0b7ad | ||
|
|
d32d93cd39 | ||
|
|
28f10b3567 | ||
|
|
5d6ee1b97a | ||
|
|
a61cb1fc15 | ||
|
|
e9a931788c | ||
|
|
e51c464417 | ||
|
|
642c49db37 | ||
|
|
f98d2291fd | ||
|
|
22833ea8fe | ||
|
|
8eb0a24492 | ||
|
|
7e0b19800c | ||
|
|
3bc476c2ed | ||
|
|
6a834ee944 | ||
|
|
ba6a78aea7 | ||
|
|
ca98231b18 | ||
|
|
1644ecd6fc | ||
|
|
f1e2e9f4f7 | ||
|
|
a59e372f07 | ||
|
|
ca368a5b38 | ||
|
|
dab5ceee0d |
@@ -1497,6 +1497,8 @@ components:
|
||||
- cloudsql_postgres
|
||||
- memorystore_redis
|
||||
- computeengine
|
||||
- gke
|
||||
- cloudstorage
|
||||
type: string
|
||||
CloudintegrationtypesServiceMetadata:
|
||||
properties:
|
||||
|
||||
2
frontend/docs/assets/drawer-example.svg
Normal file
|
After Width: | Height: | Size: 139 KiB |
2
frontend/docs/assets/edit-example.svg
Normal file
|
After Width: | Height: | Size: 49 KiB |
2
frontend/docs/assets/list-example.svg
Normal file
|
After Width: | Height: | Size: 86 KiB |
2
frontend/docs/assets/quick-filters-example.svg
Normal file
|
After Width: | Height: | Size: 64 KiB |
136
frontend/docs/authz-guide.md
Normal file
@@ -0,0 +1,136 @@
|
||||
# AuthZ Guide
|
||||
|
||||
How to structure a page so it works with the permission system.
|
||||
|
||||
We are migrating from the `ADMIN | EDITOR | VIEWER` roles to per-action checks. Instead of granting `VIEWER` and
|
||||
exposing everything, a user can now be granted access to a single resource, eg: `Logs` only.
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Core rules](#core-rules)
|
||||
- [Page patterns](#page-patterns)
|
||||
- [What "blocked" means](#what-blocked-means)
|
||||
- [Migration checklist](#migration-checklist)
|
||||
- [Testing](#testing)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Check whether the resource your page represents is supported: see
|
||||
[permissions.type.ts](../src/lib/authz/hooks/useAuthZ/permissions.type.ts) for the current resources and their allowed
|
||||
verbs.
|
||||
|
||||
**If the resource is not listed there, skip authz for now**. The backend does not enforce it yet, so any frontend
|
||||
check would be decorative. Revisit once the resource is generated into that file (it is auto-generated from the
|
||||
backend).
|
||||
|
||||
## Core rules
|
||||
|
||||
These hold for every page. The per-pattern sections below only add to them.
|
||||
|
||||
1. **A page is always reachable, regardless of permission.** The intended action must be known before permission can
|
||||
be checked, so the route renders first and individual pieces are gated.
|
||||
2. **`update` requires both `read` and `update`.** A user who can write but cannot read the current value must not get
|
||||
an edit affordance.
|
||||
3. **`delete` is independent of `read`.** Delete controls stay visible for a user who holds `delete` but not `read`.
|
||||
4. **Never gate per row.** If a user can `list`, render every row. Check `read` only when the row is opened (drawer or
|
||||
detail route).
|
||||
5. **Gate the narrowest thing that works**, a button over a section, a section over a page.
|
||||
6. **A resource may be gated while a sub-resource is not.** A user without `read` on Service Accounts can still hold
|
||||
`create` on API Keys, so blocking the outer container would hide work they are allowed to do.
|
||||
7. **Verbs not covered here** (`attach`, `detach`, `assignee`) behave like `delete`: gate the control that triggers
|
||||
them, not the surrounding content.
|
||||
|
||||
## Page patterns
|
||||
|
||||
### List page
|
||||
|
||||

|
||||
|
||||
Without `list`, but with any of `read` / `create` / `update`, only the table is blocked:
|
||||
|
||||
- Title, description, search filters and action buttons stay visible.
|
||||
- Filters and any control that drives the table are non-interactive.
|
||||
- The create button stays enabled if the user holds `create`.
|
||||
|
||||
### Edit page
|
||||
|
||||

|
||||
|
||||
Without `read`:
|
||||
|
||||
- Block the content with `withAuthZContent`, not the whole route.
|
||||
- Keep delete visible (rule 3).
|
||||
- Keep update blocked (rule 2).
|
||||
|
||||
### Drawer
|
||||
|
||||

|
||||
|
||||
Without `read`:
|
||||
|
||||
- Block the drawer body, not the drawer itself.
|
||||
- Prefer routing that carries the resource ID in the URL, so delete stays reachable without `read`.
|
||||
- Keep update blocked (rule 2).
|
||||
- Watch for sub-resources the user may still be allowed to act on (rule 6).
|
||||
|
||||
### Create
|
||||
|
||||
Without `create`:
|
||||
|
||||
| Entry point | Gate with |
|
||||
| --------------------- | ------------------------------------------------------------- |
|
||||
| Button | `AuthZButton` |
|
||||
| Dedicated create page | `withAuthZPage` |
|
||||
| Create drawer opened directly (deep link) | `withAuthZContent` on the body + `AuthZButton` on footer actions |
|
||||
|
||||
### Delete
|
||||
|
||||
Gate the control with `AuthZButton`, or `AuthZTooltip` for a non-button trigger such as an icon or menu item.
|
||||
|
||||
### Quick filters
|
||||
|
||||

|
||||
|
||||
## What "blocked" means
|
||||
|
||||
Blocking is always a visible denial, never a silent removal. Use the components in
|
||||
[`lib/authz/components`](../src/lib/authz/components/README.md) rather than hand-rolling a check, they carry the
|
||||
denial message and the loading state.
|
||||
|
||||
| Scope | Component | Denied state |
|
||||
| --------------- | ----------------------------------------- | ------------------------------------------- |
|
||||
| Button | `AuthZButton` | Disabled + tooltip |
|
||||
| Any element | `AuthZTooltip` | Disabled child + tooltip |
|
||||
| Section | `withAuthZContent` / `AuthZGuardContent` | Inline `PermissionDeniedCallout` |
|
||||
| Page / route | `withAuthZPage` / `AuthZGuardPage` | `PermissionDeniedFullPage` |
|
||||
| Custom fallback | `withAuthZ` / `AuthZGuard` | Whatever you pass as `fallback` |
|
||||
|
||||
Prefer the HOC (`withAuthZ*`); reach for the JSX guard (`AuthZGuard*`) when the gate depends on conditional rendering
|
||||
and an HOC cannot express it. The components README has the full decision tree and how to build the `checks` array.
|
||||
|
||||
## Migration checklist
|
||||
|
||||
Use the existing components. Create a new one only if none fit.
|
||||
|
||||
- [ ] Can I apply a `withAuthZ*` variant directly, or must I extract the content into a component first?
|
||||
- If extraction is needed, declare the new content component in the same file to keep the diff small, then move it
|
||||
to its own file in a follow-up commit or PR.
|
||||
- [ ] Is the layout structured to respect the [core rules](#core-rules)?
|
||||
- If not, raise it in the frontend Slack channel before reshaping the page.
|
||||
- [ ] Am I gating a shared component rather than a page or section?
|
||||
- If so, it must stay functional with no permission. Example: the Query Builder works without field suggestions when
|
||||
the user cannot read them.
|
||||
|
||||
## Testing
|
||||
|
||||
### Devtool
|
||||
|
||||
Press `Cmd + K` (`Ctrl + K` on Windows/Linux) to open the shortcuts, search for `AuthZ`, and pick the first
|
||||
result. The devtool simulates granted and denied permissions across the UI.
|
||||
|
||||
Available only in local development, it is stripped from production builds.
|
||||
|
||||
### Unit tests
|
||||
|
||||
[`lib/authz/utils/README.md`](../src/lib/authz/utils/README.md) covers the `*.authz.test.tsx` naming convention, the
|
||||
MSW handlers (`setupAuthzAdmin`, `setupAuthzDenyAll`, `setupAuthzDeny`, `setupAuthzAllow`,
|
||||
`setupAuthzGrantByPrefix`), and how to test the loading state.
|
||||
@@ -2816,6 +2816,7 @@ export enum CloudintegrationtypesServiceIDDTO {
|
||||
cloudsql_postgres = 'cloudsql_postgres',
|
||||
memorystore_redis = 'memorystore_redis',
|
||||
computeengine = 'computeengine',
|
||||
gke = 'gke',
|
||||
}
|
||||
export type CloudintegrationtypesCloudIntegrationServiceDTOAnyOf = {
|
||||
/**
|
||||
|
||||
@@ -1 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="1200" height="800" viewBox="-19.2 -28.483 166.401 170.898"><g transform="translate(0 -7.034)"><linearGradient id="a" x1="64" x2="64" y1="7.034" y2="120.789" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="#4387fd"/><stop offset="1" stop-color="#4683ea"/></linearGradient><path fill="url(#a)" d="M27.79 115.217 1.54 69.749a11.5 11.5 0 0 1 0-11.499l26.25-45.467a11.5 11.5 0 0 1 9.96-5.75h52.5a11.5 11.5 0 0 1 9.959 5.75l26.25 45.467a11.5 11.5 0 0 1 0 11.5l-26.25 45.466a11.5 11.5 0 0 1-9.959 5.75h-52.5a11.5 11.5 0 0 1-9.96-5.75z"/></g><g transform="translate(0 -7.034)"><defs><path id="b" d="M27.791 115.217 1.541 69.749a11.5 11.5 0 0 1 0-11.499l26.25-45.467a11.5 11.5 0 0 1 9.959-5.75h52.5a11.5 11.5 0 0 1 9.96 5.75l26.25 45.467a11.5 11.5 0 0 1 0 11.5l-26.25 45.466a11.5 11.5 0 0 1-9.96 5.75h-52.5a11.5 11.5 0 0 1-9.959-5.75z"/></defs><clipPath id="c"><use xlink:href="#b" width="100%" height="100%" overflow="visible"/></clipPath><path d="m49.313 53.875-7.01 6.99 5.957 5.958-5.898 10.476 44.635 44.636 10.816.002L118.936 84 85.489 50.55z" clip-path="url(#c)" opacity=".07"/></g><path fill="#fff" d="M84.7 43.236H43.264c-.667 0-1.212.546-1.212 1.214v8.566c0 .666.546 1.212 1.212 1.212H84.7c.667 0 1.213-.546 1.213-1.212v-8.568c0-.666-.545-1.213-1.212-1.213m-6.416 7.976a2.484 2.484 0 0 1-2.477-2.48 2.475 2.475 0 0 1 2.477-2.477c1.37 0 2.48 1.103 2.48 2.477a2.48 2.48 0 0 1-2.48 2.48m6.415 8.491-41.436.002c-.667 0-1.212.546-1.212 1.214v8.565c0 .666.546 1.213 1.212 1.213H84.7c.667 0 1.213-.547 1.213-1.213v-8.567c0-.666-.545-1.214-1.212-1.214m-6.416 7.976a2.483 2.483 0 0 1-2.477-2.48 2.475 2.475 0 0 1 2.477-2.477 2.48 2.48 0 1 1 0 4.956"/></svg>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" viewBox="0 0 24 24"><defs><style>.cls-1{fill:#aecbfa;}.cls-2{fill:#669df6;}.cls-3{fill:#4285f4;}</style></defs><title>Icon_24px_CloudStorage_Color</title><g data-name="Product Icons"><path class="cls-1" d="M2,4.5H22a0,0,0,0,1,0,0v5a0,0,0,0,1,0,0H2a0,0,0,0,1,0,0v-5A0,0,0,0,1,2,4.5Z"/><path class="cls-2" d="M2,14.5H22a0,0,0,0,1,0,0v5a0,0,0,0,1,0,0H2a0,0,0,0,1,0,0v-5A0,0,0,0,1,2,14.5Z"/><rect class="cls-3" x="4.5" y="6.33" width="3.33" height="1.33"/><rect class="cls-3" x="4.5" y="16.33" width="3.33" height="1.33"/><circle class="cls-3" cx="18.5" cy="7" r="1"/><circle class="cls-3" cx="18.5" cy="17" r="1"/></g></svg>
|
||||
|
||||
|
Before Width: | Height: | Size: 1.7 KiB After Width: | Height: | Size: 689 B |
@@ -1 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24"><path d="M19.73 6.56a1.73 1.73 0 0 0-1.68 1.68 1.83 1.83 0 0 0 .89 1.48v6.9l-5.16 3.06.8 1.28 5.55-3.25a.84.84 0 0 0 .4-.69v-7.3a1.64 1.64 0 0 0 .89-1.48 1.61 1.61 0 0 0-1.69-1.68" style="fill:#aecbfa"/><path d="m18 5.48-5.61-3.16a1.18 1.18 0 0 0-.79 0L5.25 6a1.72 1.72 0 0 0-2.68 1.35A1.73 1.73 0 0 0 4.26 9a1.73 1.73 0 0 0 1.68-1.65L12 3.9l5.15 3ZM11.2 18.5a1.57 1.57 0 0 0-.89.29l-5.16-3V9.92H3.56v6.31a.84.84 0 0 0 .4.69L9.52 20v.09a1.69 1.69 0 0 0 3.37 0 1.65 1.65 0 0 0-1.69-1.59" data-name="Path" style="fill:#aecbfa"/><path d="M16.96 8.63 12.1 5.78 7.13 8.63l4.97 2.77z" data-name="Path" style="fill:#669df6"/><path d="M12.1 12.38 6.84 9.32v2.47l5.26 2.96zM12.1 15.73l-5.26-3.05v2.07l5.26 3.06z" style="fill:#669df6"/><path d="M12.09 12.38v2.37l5.26-3.06V9.33Zm4.32-.94a.38.38 0 1 1 0-.76.38.38 0 0 1 0 .76M12.09 15.73v2.07l5.26-3.05v-2.07Zm4.32-1.07a.38.38 0 1 1 0-.76.38.38 0 0 1 0 .76" style="fill:#4285f4"/></svg>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" viewBox="0 0 24 24"><defs><style>.cls-1{fill:#4285f4;}.cls-1,.cls-2,.cls-4{fill-rule:evenodd;}.cls-2{fill:#669df6;}.cls-3,.cls-4{fill:#aecbfa;}</style></defs><title>Icon_24px_K8Engine_Color</title><g data-name="Product Icons"><g ><polygon class="cls-1" points="14.68 13.06 19.23 15.69 19.23 16.68 14.29 13.83 14.68 13.06"/><polygon class="cls-2" points="9.98 13.65 4.77 16.66 4.45 15.86 9.53 12.92 9.98 13.65"/><rect class="cls-3" x="11.55" y="3.29" width="0.86" height="5.78"/><path class="cls-4" d="M3.25,7V17L12,22l8.74-5V7L12,2Zm15.63,8.89L12,19.78,5.12,15.89V8.11L12,4.22l6.87,3.89v7.78Z"/><polygon class="cls-4" points="11.98 11.5 15.96 9.21 11.98 6.91 8.01 9.21 11.98 11.5"/><polygon class="cls-2" points="11.52 12.3 7.66 10.01 7.66 14.6 11.52 16.89 11.52 12.3"/><polygon class="cls-1" points="12.48 12.3 12.48 16.89 16.34 14.6 16.34 10.01 12.48 12.3"/></g></g></svg>
|
||||
|
Before Width: | Height: | Size: 988 B After Width: | Height: | Size: 941 B |
@@ -143,6 +143,12 @@ export default defineConfig(({ mode }): UserConfig => {
|
||||
plugins,
|
||||
resolve: {
|
||||
alias: {
|
||||
// @grafana/data imports bare CJS `lodash`, whose UMD footer checks for an
|
||||
// AMD loader before assigning module.exports. Any third-party script that
|
||||
// defines window.define.amd first leaves the bundled namespace empty, so
|
||||
// every lodash method reached through it is undefined at runtime. lodash-es
|
||||
// is the same version, real ESM, and tree-shakes.
|
||||
lodash: 'lodash-es',
|
||||
'@': resolve(__dirname, './src'),
|
||||
utils: resolve(__dirname, './src/utils'),
|
||||
types: resolve(__dirname, './src/types'),
|
||||
|
||||
2
go.mod
@@ -4,7 +4,7 @@ go 1.25.7
|
||||
|
||||
require (
|
||||
dario.cat/mergo v1.0.2
|
||||
github.com/AfterShip/clickhouse-sql-parser v0.5.2
|
||||
github.com/AfterShip/clickhouse-sql-parser v0.5.3
|
||||
github.com/ClickHouse/clickhouse-go/v2 v2.44.0
|
||||
github.com/DATA-DOG/go-sqlmock v1.5.2
|
||||
github.com/SigNoz/clickhouse-go-mock v0.14.0
|
||||
|
||||
4
go.sum
@@ -66,8 +66,8 @@ dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
|
||||
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
|
||||
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
|
||||
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
|
||||
github.com/AfterShip/clickhouse-sql-parser v0.5.2 h1:KCxdmvuVawVF4yl0ZS33q7olgmLZwvZG5BjWHp6o414=
|
||||
github.com/AfterShip/clickhouse-sql-parser v0.5.2/go.mod h1:Qi3qvPTfZb/aFwI5V4WFOahgjsLJa4MzVijIAfwOhDw=
|
||||
github.com/AfterShip/clickhouse-sql-parser v0.5.3 h1:6iap8XGjuSjD3w7r1UNrg66ljBugcv2P39s4eo/ZLRw=
|
||||
github.com/AfterShip/clickhouse-sql-parser v0.5.3/go.mod h1:Qi3qvPTfZb/aFwI5V4WFOahgjsLJa4MzVijIAfwOhDw=
|
||||
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible h1:fcYLmCpyNYRnvJbPerq7U0hS+6+I79yEDJBqVNcqUzU=
|
||||
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0 h1:fou+2+WFTib47nS+nz/ozhEBnvU96bKHy6LjRsY4E28=
|
||||
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0/go.mod h1:t76Ruy8AHvUAC8GfMWJMa0ElSbuIcO03NLpynfbgsPA=
|
||||
|
||||
@@ -112,12 +112,15 @@ functionCall
|
||||
;
|
||||
|
||||
/*
|
||||
* Full-text search: search('needle') or scoped search('needle', body, ...).
|
||||
* First param is the needle; the rest are field-context scopes (body/attribute/
|
||||
* resource/log), quoted or bare. Handled in the visitor — no grammar change.
|
||||
* Full-text search call: search('needle')
|
||||
*
|
||||
* Uses the shared functionParamList so future scoped forms like
|
||||
* search(body, 'abc') / search(attribute, 'abc') need no grammar change. Today
|
||||
* only a single needle is supported. Unlike bare/quoted free text (`fullText`),
|
||||
* which only targets the body column, search() fans out across every field.
|
||||
*/
|
||||
searchCall
|
||||
: SEARCH LPAREN valueList RPAREN
|
||||
: SEARCH LPAREN functionParamList RPAREN
|
||||
;
|
||||
|
||||
// Function parameters can be keys, single scalar values, or arrays
|
||||
|
||||
@@ -29,19 +29,13 @@ func New(t *testing.T) flagger.Flagger {
|
||||
|
||||
// WithUseJSONBody returns a Flagger with use_json_body set to the given value.
|
||||
func WithUseJSONBody(t *testing.T, enabled bool) flagger.Flagger {
|
||||
return WithBooleanFlags(t, map[string]bool{
|
||||
flagger.FeatureUseJSONBody.String(): enabled,
|
||||
})
|
||||
}
|
||||
|
||||
// WithBooleanFlags returns a Flagger with the given boolean feature flags set to
|
||||
// the provided values (keyed by feature name, e.g. flagger.FeatureX.String()).
|
||||
func WithBooleanFlags(t *testing.T, flags map[string]bool) flagger.Flagger {
|
||||
t.Helper()
|
||||
registry := flagger.MustNewRegistry()
|
||||
cfg := flagger.Config{}
|
||||
if len(flags) > 0 {
|
||||
cfg.Config.Boolean = flags
|
||||
if enabled {
|
||||
cfg.Config.Boolean = map[string]bool{
|
||||
flagger.FeatureUseJSONBody.String(): true,
|
||||
}
|
||||
}
|
||||
fl, err := flagger.New(
|
||||
context.Background(),
|
||||
|
||||
@@ -0,0 +1,956 @@
|
||||
{
|
||||
"schemaVersion": "v6",
|
||||
"image": "/assets/Logos/gcp-cloud-storage",
|
||||
"name": "",
|
||||
"generateName": true,
|
||||
"tags": [
|
||||
{
|
||||
"key": "tag",
|
||||
"value": "observability"
|
||||
}
|
||||
],
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "GCP Cloud Storage Overview",
|
||||
"description": "Dashboard for GCP Cloud Storage overview"
|
||||
},
|
||||
"variables": [
|
||||
{
|
||||
"kind": "ListVariable",
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "project_id",
|
||||
"description": "GCP Project ID"
|
||||
},
|
||||
"allowAllValue": false,
|
||||
"allowMultiple": false,
|
||||
"customAllValue": "",
|
||||
"capturingRegexp": "",
|
||||
"sort": "none",
|
||||
"plugin": {
|
||||
"kind": "signoz/DynamicVariable",
|
||||
"spec": {
|
||||
"name": "project_id",
|
||||
"signal": "metrics"
|
||||
}
|
||||
},
|
||||
"name": "project_id"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "ListVariable",
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "bucket_name",
|
||||
"description": "GCS bucket name"
|
||||
},
|
||||
"allowAllValue": true,
|
||||
"allowMultiple": true,
|
||||
"customAllValue": "",
|
||||
"capturingRegexp": "",
|
||||
"sort": "none",
|
||||
"plugin": {
|
||||
"kind": "signoz/DynamicVariable",
|
||||
"spec": {
|
||||
"name": "bucket_name",
|
||||
"signal": "metrics"
|
||||
}
|
||||
},
|
||||
"name": "bucket_name"
|
||||
}
|
||||
}
|
||||
],
|
||||
"panels": {
|
||||
"1f0a4d5b-6c2e-4a71-8b3d-9e5c07a41d20": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "Bucket size",
|
||||
"description": "Total size of all objects in the bucket, grouped by storage class."
|
||||
},
|
||||
"plugin": {
|
||||
"kind": "signoz/TablePanel",
|
||||
"spec": {
|
||||
"visualization": {
|
||||
"timePreference": "global_time"
|
||||
},
|
||||
"formatting": {
|
||||
"columnUnits": {
|
||||
"A": "By"
|
||||
},
|
||||
"decimalPrecision": "2"
|
||||
},
|
||||
"thresholds": null
|
||||
}
|
||||
},
|
||||
"queries": [
|
||||
{
|
||||
"kind": "scalar",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"plugin": {
|
||||
"kind": "signoz/BuilderQuery",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"signal": "metrics",
|
||||
"source": "",
|
||||
"aggregations": [
|
||||
{
|
||||
"metricName": "storage.googleapis.com/storage/v2/total_bytes",
|
||||
"temporality": "",
|
||||
"timeAggregation": "latest",
|
||||
"spaceAggregation": "sum",
|
||||
"reduceTo": "last"
|
||||
}
|
||||
],
|
||||
"disabled": false,
|
||||
"filter": {
|
||||
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
|
||||
},
|
||||
"groupBy": [
|
||||
{
|
||||
"name": "bucket_name",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "storage_class",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
}
|
||||
],
|
||||
"order": [],
|
||||
"having": {
|
||||
"expression": ""
|
||||
},
|
||||
"functions": [],
|
||||
"legend": "Size"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
},
|
||||
"29ed9824-21db-44c3-9537-12be458f5c20": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "Total object bytes",
|
||||
"description": "Total size of all objects in the bucket, grouped by storage class and type."
|
||||
},
|
||||
"plugin": {
|
||||
"kind": "signoz/TimeSeriesPanel",
|
||||
"spec": {
|
||||
"visualization": {
|
||||
"timePreference": "global_time",
|
||||
"fillSpans": false
|
||||
},
|
||||
"formatting": {
|
||||
"unit": "By",
|
||||
"decimalPrecision": "2"
|
||||
},
|
||||
"chartAppearance": {
|
||||
"lineInterpolation": "spline",
|
||||
"showPoints": false,
|
||||
"lineStyle": "solid",
|
||||
"fillMode": "none",
|
||||
"spanGaps": {
|
||||
"fillOnlyBelow": false,
|
||||
"fillLessThan": ""
|
||||
}
|
||||
},
|
||||
"axes": {
|
||||
"softMin": 0,
|
||||
"softMax": 0,
|
||||
"isLogScale": false
|
||||
},
|
||||
"legend": {
|
||||
"position": "right",
|
||||
"mode": "list",
|
||||
"customColors": null
|
||||
},
|
||||
"thresholds": null
|
||||
}
|
||||
},
|
||||
"queries": [
|
||||
{
|
||||
"kind": "time_series",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"plugin": {
|
||||
"kind": "signoz/BuilderQuery",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"signal": "metrics",
|
||||
"source": "",
|
||||
"aggregations": [
|
||||
{
|
||||
"metricName": "storage.googleapis.com/storage/v2/total_bytes",
|
||||
"temporality": "",
|
||||
"timeAggregation": "latest",
|
||||
"spaceAggregation": "sum",
|
||||
"reduceTo": "avg"
|
||||
}
|
||||
],
|
||||
"disabled": false,
|
||||
"filter": {
|
||||
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
|
||||
},
|
||||
"groupBy": [
|
||||
{
|
||||
"name": "bucket_name",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "storage_class",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "type",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
}
|
||||
],
|
||||
"order": [],
|
||||
"having": {
|
||||
"expression": ""
|
||||
},
|
||||
"functions": [],
|
||||
"legend": "{{bucket_name}}/{{storage_class}}/{{type}}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
},
|
||||
"cbc53174-5527-4bce-b32d-317403849e48": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "Total object count",
|
||||
"description": "Total number of objects and multipart-uploads per bucket, grouped by storage class and type, where type can be live-object, noncurrent-object, soft-deleted-object or multipart-upload."
|
||||
},
|
||||
"plugin": {
|
||||
"kind": "signoz/TimeSeriesPanel",
|
||||
"spec": {
|
||||
"visualization": {
|
||||
"timePreference": "global_time",
|
||||
"fillSpans": false
|
||||
},
|
||||
"formatting": {
|
||||
"unit": "{count}",
|
||||
"decimalPrecision": "2"
|
||||
},
|
||||
"chartAppearance": {
|
||||
"lineInterpolation": "spline",
|
||||
"showPoints": false,
|
||||
"lineStyle": "solid",
|
||||
"fillMode": "none",
|
||||
"spanGaps": {
|
||||
"fillOnlyBelow": false,
|
||||
"fillLessThan": ""
|
||||
}
|
||||
},
|
||||
"axes": {
|
||||
"softMin": 0,
|
||||
"softMax": 0,
|
||||
"isLogScale": false
|
||||
},
|
||||
"legend": {
|
||||
"position": "right",
|
||||
"mode": "list",
|
||||
"customColors": null
|
||||
},
|
||||
"thresholds": null
|
||||
}
|
||||
},
|
||||
"queries": [
|
||||
{
|
||||
"kind": "time_series",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"plugin": {
|
||||
"kind": "signoz/BuilderQuery",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"signal": "metrics",
|
||||
"source": "",
|
||||
"aggregations": [
|
||||
{
|
||||
"metricName": "storage.googleapis.com/storage/v2/total_count",
|
||||
"temporality": "",
|
||||
"timeAggregation": "latest",
|
||||
"spaceAggregation": "sum",
|
||||
"reduceTo": "avg"
|
||||
}
|
||||
],
|
||||
"disabled": false,
|
||||
"filter": {
|
||||
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
|
||||
},
|
||||
"groupBy": [
|
||||
{
|
||||
"name": "bucket_name",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "storage_class",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "type",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
}
|
||||
],
|
||||
"order": [],
|
||||
"having": {
|
||||
"expression": ""
|
||||
},
|
||||
"functions": [],
|
||||
"legend": "{{bucket_name}}/{{storage_class}}/{{type}}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
},
|
||||
"76e70ce5-47e4-4b5b-9715-49fa21c58b0c": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "Deleted bytes",
|
||||
"description": "Rate of deleted bytes per bucket, grouped by storage class."
|
||||
},
|
||||
"plugin": {
|
||||
"kind": "signoz/TimeSeriesPanel",
|
||||
"spec": {
|
||||
"visualization": {
|
||||
"timePreference": "global_time",
|
||||
"fillSpans": false
|
||||
},
|
||||
"formatting": {
|
||||
"unit": "By/s",
|
||||
"decimalPrecision": "2"
|
||||
},
|
||||
"chartAppearance": {
|
||||
"lineInterpolation": "spline",
|
||||
"showPoints": false,
|
||||
"lineStyle": "solid",
|
||||
"fillMode": "none",
|
||||
"spanGaps": {
|
||||
"fillOnlyBelow": false,
|
||||
"fillLessThan": ""
|
||||
}
|
||||
},
|
||||
"axes": {
|
||||
"softMin": 0,
|
||||
"softMax": 0,
|
||||
"isLogScale": false
|
||||
},
|
||||
"legend": {
|
||||
"position": "right",
|
||||
"mode": "list",
|
||||
"customColors": null
|
||||
},
|
||||
"thresholds": null
|
||||
}
|
||||
},
|
||||
"queries": [
|
||||
{
|
||||
"kind": "time_series",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"plugin": {
|
||||
"kind": "signoz/BuilderQuery",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"signal": "metrics",
|
||||
"source": "",
|
||||
"aggregations": [
|
||||
{
|
||||
"metricName": "storage.googleapis.com/storage/v2/deleted_bytes",
|
||||
"temporality": "",
|
||||
"timeAggregation": "rate",
|
||||
"spaceAggregation": "sum",
|
||||
"reduceTo": "sum"
|
||||
}
|
||||
],
|
||||
"disabled": false,
|
||||
"filter": {
|
||||
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
|
||||
},
|
||||
"groupBy": [
|
||||
{
|
||||
"name": "bucket_name",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "storage_class",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
}
|
||||
],
|
||||
"order": [],
|
||||
"having": {
|
||||
"expression": ""
|
||||
},
|
||||
"functions": [],
|
||||
"legend": "{{bucket_name}}/{{storage_class}}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
},
|
||||
"c5af2813-34ec-41a9-8d53-c9acd67f569f": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "Request count",
|
||||
"description": "Rate of API calls, grouped by the API method name and response code."
|
||||
},
|
||||
"plugin": {
|
||||
"kind": "signoz/TimeSeriesPanel",
|
||||
"spec": {
|
||||
"visualization": {
|
||||
"timePreference": "global_time",
|
||||
"fillSpans": false
|
||||
},
|
||||
"formatting": {
|
||||
"unit": "{count}/s",
|
||||
"decimalPrecision": "2"
|
||||
},
|
||||
"chartAppearance": {
|
||||
"lineInterpolation": "spline",
|
||||
"showPoints": false,
|
||||
"lineStyle": "solid",
|
||||
"fillMode": "none",
|
||||
"spanGaps": {
|
||||
"fillOnlyBelow": false,
|
||||
"fillLessThan": ""
|
||||
}
|
||||
},
|
||||
"axes": {
|
||||
"softMin": 0,
|
||||
"softMax": 0,
|
||||
"isLogScale": false
|
||||
},
|
||||
"legend": {
|
||||
"position": "right",
|
||||
"mode": "list",
|
||||
"customColors": null
|
||||
},
|
||||
"thresholds": null
|
||||
}
|
||||
},
|
||||
"queries": [
|
||||
{
|
||||
"kind": "time_series",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"plugin": {
|
||||
"kind": "signoz/BuilderQuery",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"signal": "metrics",
|
||||
"source": "",
|
||||
"aggregations": [
|
||||
{
|
||||
"metricName": "storage.googleapis.com/api/request_count",
|
||||
"temporality": "",
|
||||
"timeAggregation": "rate",
|
||||
"spaceAggregation": "sum",
|
||||
"reduceTo": "sum"
|
||||
}
|
||||
],
|
||||
"disabled": false,
|
||||
"filter": {
|
||||
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
|
||||
},
|
||||
"groupBy": [
|
||||
{
|
||||
"name": "bucket_name",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "method",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "response_code",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
}
|
||||
],
|
||||
"order": [],
|
||||
"having": {
|
||||
"expression": ""
|
||||
},
|
||||
"functions": [],
|
||||
"legend": "{{bucket_name}}/{{method}} - {{response_code}}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
},
|
||||
"70333210-68c3-4387-a6b7-afa2459d27dd": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "Authentication count",
|
||||
"description": "Rate of HMAC/RSA signed requests, grouped by authentication method, API method name and response code."
|
||||
},
|
||||
"plugin": {
|
||||
"kind": "signoz/TimeSeriesPanel",
|
||||
"spec": {
|
||||
"visualization": {
|
||||
"timePreference": "global_time",
|
||||
"fillSpans": false
|
||||
},
|
||||
"formatting": {
|
||||
"unit": "{count}/s",
|
||||
"decimalPrecision": "2"
|
||||
},
|
||||
"chartAppearance": {
|
||||
"lineInterpolation": "spline",
|
||||
"showPoints": false,
|
||||
"lineStyle": "solid",
|
||||
"fillMode": "none",
|
||||
"spanGaps": {
|
||||
"fillOnlyBelow": false,
|
||||
"fillLessThan": ""
|
||||
}
|
||||
},
|
||||
"axes": {
|
||||
"softMin": 0,
|
||||
"softMax": 0,
|
||||
"isLogScale": false
|
||||
},
|
||||
"legend": {
|
||||
"position": "right",
|
||||
"mode": "list",
|
||||
"customColors": null
|
||||
},
|
||||
"thresholds": null
|
||||
}
|
||||
},
|
||||
"queries": [
|
||||
{
|
||||
"kind": "time_series",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"plugin": {
|
||||
"kind": "signoz/BuilderQuery",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"signal": "metrics",
|
||||
"source": "",
|
||||
"aggregations": [
|
||||
{
|
||||
"metricName": "storage.googleapis.com/authn/authentication_count",
|
||||
"temporality": "",
|
||||
"timeAggregation": "rate",
|
||||
"spaceAggregation": "sum",
|
||||
"reduceTo": "sum"
|
||||
}
|
||||
],
|
||||
"disabled": false,
|
||||
"filter": {
|
||||
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
|
||||
},
|
||||
"groupBy": [
|
||||
{
|
||||
"name": "bucket_name",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "authentication_method",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "method",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "response_code",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
}
|
||||
],
|
||||
"order": [],
|
||||
"having": {
|
||||
"expression": ""
|
||||
},
|
||||
"functions": [],
|
||||
"legend": "{{bucket_name}}/{{authentication_method}}/{{method}} - {{response_code}}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
},
|
||||
"7a98084c-2cd3-4146-a831-f5382162ffe5": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "Received bytes",
|
||||
"description": "Rate of bytes received over the network, grouped by the API method name and response code."
|
||||
},
|
||||
"plugin": {
|
||||
"kind": "signoz/TimeSeriesPanel",
|
||||
"spec": {
|
||||
"visualization": {
|
||||
"timePreference": "global_time",
|
||||
"fillSpans": false
|
||||
},
|
||||
"formatting": {
|
||||
"unit": "By/s",
|
||||
"decimalPrecision": "2"
|
||||
},
|
||||
"chartAppearance": {
|
||||
"lineInterpolation": "spline",
|
||||
"showPoints": false,
|
||||
"lineStyle": "solid",
|
||||
"fillMode": "none",
|
||||
"spanGaps": {
|
||||
"fillOnlyBelow": false,
|
||||
"fillLessThan": ""
|
||||
}
|
||||
},
|
||||
"axes": {
|
||||
"softMin": 0,
|
||||
"softMax": 0,
|
||||
"isLogScale": false
|
||||
},
|
||||
"legend": {
|
||||
"position": "right",
|
||||
"mode": "list",
|
||||
"customColors": null
|
||||
},
|
||||
"thresholds": null
|
||||
}
|
||||
},
|
||||
"queries": [
|
||||
{
|
||||
"kind": "time_series",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"plugin": {
|
||||
"kind": "signoz/BuilderQuery",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"signal": "metrics",
|
||||
"source": "",
|
||||
"aggregations": [
|
||||
{
|
||||
"metricName": "storage.googleapis.com/network/received_bytes_count",
|
||||
"temporality": "",
|
||||
"timeAggregation": "rate",
|
||||
"spaceAggregation": "sum",
|
||||
"reduceTo": "sum"
|
||||
}
|
||||
],
|
||||
"disabled": false,
|
||||
"filter": {
|
||||
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
|
||||
},
|
||||
"groupBy": [
|
||||
{
|
||||
"name": "bucket_name",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "method",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "response_code",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
}
|
||||
],
|
||||
"order": [],
|
||||
"having": {
|
||||
"expression": ""
|
||||
},
|
||||
"functions": [],
|
||||
"legend": "{{bucket_name}}/{{method}} - {{response_code}}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
},
|
||||
"7138bf98-b41a-46d8-90ef-8e7470819cdf": {
|
||||
"kind": "Panel",
|
||||
"spec": {
|
||||
"display": {
|
||||
"name": "Sent bytes",
|
||||
"description": "Rate of bytes sent over the network, grouped by the API method name and response code."
|
||||
},
|
||||
"plugin": {
|
||||
"kind": "signoz/TimeSeriesPanel",
|
||||
"spec": {
|
||||
"visualization": {
|
||||
"timePreference": "global_time",
|
||||
"fillSpans": false
|
||||
},
|
||||
"formatting": {
|
||||
"unit": "By/s",
|
||||
"decimalPrecision": "2"
|
||||
},
|
||||
"chartAppearance": {
|
||||
"lineInterpolation": "spline",
|
||||
"showPoints": false,
|
||||
"lineStyle": "solid",
|
||||
"fillMode": "none",
|
||||
"spanGaps": {
|
||||
"fillOnlyBelow": false,
|
||||
"fillLessThan": ""
|
||||
}
|
||||
},
|
||||
"axes": {
|
||||
"softMin": 0,
|
||||
"softMax": 0,
|
||||
"isLogScale": false
|
||||
},
|
||||
"legend": {
|
||||
"position": "right",
|
||||
"mode": "list",
|
||||
"customColors": null
|
||||
},
|
||||
"thresholds": null
|
||||
}
|
||||
},
|
||||
"queries": [
|
||||
{
|
||||
"kind": "time_series",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"plugin": {
|
||||
"kind": "signoz/BuilderQuery",
|
||||
"spec": {
|
||||
"name": "A",
|
||||
"signal": "metrics",
|
||||
"source": "",
|
||||
"aggregations": [
|
||||
{
|
||||
"metricName": "storage.googleapis.com/network/sent_bytes_count",
|
||||
"temporality": "",
|
||||
"timeAggregation": "rate",
|
||||
"spaceAggregation": "sum",
|
||||
"reduceTo": "sum"
|
||||
}
|
||||
],
|
||||
"disabled": false,
|
||||
"filter": {
|
||||
"expression": "project_id = $project_id AND bucket_name in $bucket_name AND gcp.resource_type = 'gcs_bucket'"
|
||||
},
|
||||
"groupBy": [
|
||||
{
|
||||
"name": "bucket_name",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "method",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
},
|
||||
{
|
||||
"name": "response_code",
|
||||
"signal": "",
|
||||
"fieldContext": "attribute",
|
||||
"fieldDataType": "string"
|
||||
}
|
||||
],
|
||||
"order": [],
|
||||
"having": {
|
||||
"expression": ""
|
||||
},
|
||||
"functions": [],
|
||||
"legend": "{{bucket_name}}/{{method}} - {{response_code}}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
}
|
||||
},
|
||||
"layouts": [
|
||||
{
|
||||
"kind": "Grid",
|
||||
"spec": {
|
||||
"items": [
|
||||
{
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"width": 6,
|
||||
"height": 6,
|
||||
"content": {
|
||||
"$ref": "#/spec/panels/1f0a4d5b-6c2e-4a71-8b3d-9e5c07a41d20"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "Grid",
|
||||
"spec": {
|
||||
"display": {
|
||||
"title": "Storage",
|
||||
"collapse": {
|
||||
"open": true
|
||||
}
|
||||
},
|
||||
"items": [
|
||||
{
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"width": 6,
|
||||
"height": 6,
|
||||
"content": {
|
||||
"$ref": "#/spec/panels/29ed9824-21db-44c3-9537-12be458f5c20"
|
||||
}
|
||||
},
|
||||
{
|
||||
"x": 6,
|
||||
"y": 0,
|
||||
"width": 6,
|
||||
"height": 6,
|
||||
"content": {
|
||||
"$ref": "#/spec/panels/cbc53174-5527-4bce-b32d-317403849e48"
|
||||
}
|
||||
},
|
||||
{
|
||||
"x": 0,
|
||||
"y": 6,
|
||||
"width": 6,
|
||||
"height": 6,
|
||||
"content": {
|
||||
"$ref": "#/spec/panels/76e70ce5-47e4-4b5b-9715-49fa21c58b0c"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "Grid",
|
||||
"spec": {
|
||||
"display": {
|
||||
"title": "API & Authentication",
|
||||
"collapse": {
|
||||
"open": true
|
||||
}
|
||||
},
|
||||
"items": [
|
||||
{
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"width": 6,
|
||||
"height": 6,
|
||||
"content": {
|
||||
"$ref": "#/spec/panels/c5af2813-34ec-41a9-8d53-c9acd67f569f"
|
||||
}
|
||||
},
|
||||
{
|
||||
"x": 6,
|
||||
"y": 0,
|
||||
"width": 6,
|
||||
"height": 6,
|
||||
"content": {
|
||||
"$ref": "#/spec/panels/70333210-68c3-4387-a6b7-afa2459d27dd"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "Grid",
|
||||
"spec": {
|
||||
"display": {
|
||||
"title": "Network",
|
||||
"collapse": {
|
||||
"open": true
|
||||
}
|
||||
},
|
||||
"items": [
|
||||
{
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"width": 6,
|
||||
"height": 6,
|
||||
"content": {
|
||||
"$ref": "#/spec/panels/7a98084c-2cd3-4146-a831-f5382162ffe5"
|
||||
}
|
||||
},
|
||||
{
|
||||
"x": 6,
|
||||
"y": 0,
|
||||
"width": 6,
|
||||
"height": 6,
|
||||
"content": {
|
||||
"$ref": "#/spec/panels/7138bf98-b41a-46d8-90ef-8e7470819cdf"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"duration": "",
|
||||
"refreshInterval": "",
|
||||
"links": []
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" viewBox="0 0 24 24"><defs><style>.cls-1{fill:#aecbfa;}.cls-2{fill:#669df6;}.cls-3{fill:#4285f4;}</style></defs><title>Icon_24px_CloudStorage_Color</title><g data-name="Product Icons"><path class="cls-1" d="M2,4.5H22a0,0,0,0,1,0,0v5a0,0,0,0,1,0,0H2a0,0,0,0,1,0,0v-5A0,0,0,0,1,2,4.5Z"/><path class="cls-2" d="M2,14.5H22a0,0,0,0,1,0,0v5a0,0,0,0,1,0,0H2a0,0,0,0,1,0,0v-5A0,0,0,0,1,2,14.5Z"/><rect class="cls-3" x="4.5" y="6.33" width="3.33" height="1.33"/><rect class="cls-3" x="4.5" y="16.33" width="3.33" height="1.33"/><circle class="cls-3" cx="18.5" cy="7" r="1"/><circle class="cls-3" cx="18.5" cy="17" r="1"/></g></svg>
|
||||
|
After Width: | Height: | Size: 689 B |
@@ -0,0 +1,70 @@
|
||||
{
|
||||
"id": "cloudstorage",
|
||||
"title": "GCP Cloud Storage",
|
||||
"icon": "file://icon.svg",
|
||||
"overview": "file://overview.md",
|
||||
"supportedSignals": {
|
||||
"metrics": true,
|
||||
"logs": true
|
||||
},
|
||||
"dataCollected": {
|
||||
"metrics": [
|
||||
{
|
||||
"name": "storage.googleapis.com/storage/v2/total_bytes",
|
||||
"unit": "Bytes",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "storage.googleapis.com/storage/v2/total_count",
|
||||
"unit": "Count",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "storage.googleapis.com/storage/v2/deleted_bytes",
|
||||
"unit": "Bytes",
|
||||
"type": "Sum",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "storage.googleapis.com/api/request_count",
|
||||
"unit": "Count",
|
||||
"type": "Sum",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "storage.googleapis.com/network/received_bytes_count",
|
||||
"unit": "Bytes",
|
||||
"type": "Sum",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "storage.googleapis.com/network/sent_bytes_count",
|
||||
"unit": "Bytes",
|
||||
"type": "Sum",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "storage.googleapis.com/authn/authentication_count",
|
||||
"unit": "Count",
|
||||
"type": "Sum",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"logs": []
|
||||
},
|
||||
"telemetryCollectionStrategy": {
|
||||
"gcp": {}
|
||||
},
|
||||
"assets": {
|
||||
"dashboards": [
|
||||
{
|
||||
"id": "overview",
|
||||
"title": "GCP Cloud Storage Overview",
|
||||
"description": "Overview of GCP Cloud Storage metrics",
|
||||
"definition": "file://assets/dashboards/overview.json"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
### Monitor GCP Cloud Storage with SigNoz
|
||||
|
||||
Collect key GCP Cloud Storage metrics and view them with an out of the box dashboard.
|
||||
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="24px" height="24px" viewBox="0 0 24 24"><defs><style>.cls-1{fill:#4285f4;}.cls-1,.cls-2,.cls-4{fill-rule:evenodd;}.cls-2{fill:#669df6;}.cls-3,.cls-4{fill:#aecbfa;}</style></defs><title>Icon_24px_K8Engine_Color</title><g data-name="Product Icons"><g ><polygon class="cls-1" points="14.68 13.06 19.23 15.69 19.23 16.68 14.29 13.83 14.68 13.06"/><polygon class="cls-2" points="9.98 13.65 4.77 16.66 4.45 15.86 9.53 12.92 9.98 13.65"/><rect class="cls-3" x="11.55" y="3.29" width="0.86" height="5.78"/><path class="cls-4" d="M3.25,7V17L12,22l8.74-5V7L12,2Zm15.63,8.89L12,19.78,5.12,15.89V8.11L12,4.22l6.87,3.89v7.78Z"/><polygon class="cls-4" points="11.98 11.5 15.96 9.21 11.98 6.91 8.01 9.21 11.98 11.5"/><polygon class="cls-2" points="11.52 12.3 7.66 10.01 7.66 14.6 11.52 16.89 11.52 12.3"/><polygon class="cls-1" points="12.48 12.3 12.48 16.89 16.34 14.6 16.34 10.01 12.48 12.3"/></g></g></svg>
|
||||
|
After Width: | Height: | Size: 941 B |
@@ -0,0 +1,124 @@
|
||||
{
|
||||
"id": "gke",
|
||||
"title": "GCP Kubernetes Engine",
|
||||
"icon": "file://icon.svg",
|
||||
"overview": "file://overview.md",
|
||||
"supportedSignals": {
|
||||
"metrics": true,
|
||||
"logs": true
|
||||
},
|
||||
"dataCollected": {
|
||||
"metrics": [
|
||||
{
|
||||
"name": "kubernetes.io/container/cpu/limit_utilization",
|
||||
"unit": "None",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/container/cpu/request_utilization",
|
||||
"unit": "None",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/container/memory/limit_utilization",
|
||||
"unit": "None",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/container/memory/request_utilization",
|
||||
"unit": "None",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/container/restart_count",
|
||||
"unit": "Count",
|
||||
"type": "Sum",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/pod/latencies/pod_first_ready",
|
||||
"unit": "Seconds",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/pod/network/received_bytes_count",
|
||||
"unit": "Bytes",
|
||||
"type": "Sum",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/pod/network/sent_bytes_count",
|
||||
"unit": "Bytes",
|
||||
"type": "Sum",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/pod/volume/utilization",
|
||||
"unit": "None",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/node/cpu/allocatable_utilization",
|
||||
"unit": "None",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/node/memory/allocatable_utilization",
|
||||
"unit": "None",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/node/ephemeral_storage/used_bytes",
|
||||
"unit": "Bytes",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/node/ephemeral_storage/allocatable_bytes",
|
||||
"unit": "Bytes",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/node/network/received_bytes_count",
|
||||
"unit": "Bytes",
|
||||
"type": "Sum",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/node/network/sent_bytes_count",
|
||||
"unit": "Bytes",
|
||||
"type": "Sum",
|
||||
"description": ""
|
||||
},
|
||||
{
|
||||
"name": "kubernetes.io/node/status_condition",
|
||||
"unit": "None",
|
||||
"type": "Gauge",
|
||||
"description": ""
|
||||
}
|
||||
],
|
||||
"logs": []
|
||||
},
|
||||
"telemetryCollectionStrategy": {
|
||||
"gcp": {}
|
||||
},
|
||||
"assets": {
|
||||
"dashboards": [
|
||||
{
|
||||
"id": "overview",
|
||||
"title": "GCP Kubernetes Engine Overview",
|
||||
"description": "Overview of GCP Kubernetes Engine metrics",
|
||||
"definition": "file://assets/dashboards/overview.json"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
### Monitor GCP Kubernetes Engine with SigNoz
|
||||
|
||||
Collect key GCP Kubernetes Engine metrics and view them with an out of the box dashboard.
|
||||
@@ -35,7 +35,7 @@ func (c *conditionBuilder) ConditionFor(
|
||||
sb *sqlbuilder.SelectBuilder,
|
||||
) ([]string, []string, error) {
|
||||
|
||||
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for rule state history.
|
||||
// has/hasAny/hasAll/hasToken are logs-body-only; reject for rule state history.
|
||||
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
@@ -137,8 +137,8 @@ func filterqueryParserInit() {
|
||||
0, 0, 191, 19, 1, 0, 0, 0, 192, 190, 1, 0, 0, 0, 193, 194, 7, 2, 0, 0,
|
||||
194, 21, 1, 0, 0, 0, 195, 196, 7, 3, 0, 0, 196, 197, 5, 1, 0, 0, 197, 198,
|
||||
3, 26, 13, 0, 198, 199, 5, 2, 0, 0, 199, 23, 1, 0, 0, 0, 200, 201, 5, 27,
|
||||
0, 0, 201, 202, 5, 1, 0, 0, 202, 203, 3, 18, 9, 0, 203, 204, 5, 2, 0, 0,
|
||||
204, 25, 1, 0, 0, 0, 205, 210, 3, 28, 14, 0, 206, 207, 5, 5, 0, 0, 207,
|
||||
0, 0, 201, 202, 5, 1, 0, 0, 202, 203, 3, 26, 13, 0, 203, 204, 5, 2, 0,
|
||||
0, 204, 25, 1, 0, 0, 0, 205, 210, 3, 28, 14, 0, 206, 207, 5, 5, 0, 0, 207,
|
||||
209, 3, 28, 14, 0, 208, 206, 1, 0, 0, 0, 209, 212, 1, 0, 0, 0, 210, 208,
|
||||
1, 0, 0, 0, 210, 211, 1, 0, 0, 0, 211, 27, 1, 0, 0, 0, 212, 210, 1, 0,
|
||||
0, 0, 213, 217, 3, 34, 17, 0, 214, 217, 3, 32, 16, 0, 215, 217, 3, 30,
|
||||
@@ -2945,7 +2945,7 @@ type ISearchCallContext interface {
|
||||
// Getter signatures
|
||||
SEARCH() antlr.TerminalNode
|
||||
LPAREN() antlr.TerminalNode
|
||||
ValueList() IValueListContext
|
||||
FunctionParamList() IFunctionParamListContext
|
||||
RPAREN() antlr.TerminalNode
|
||||
|
||||
// IsSearchCallContext differentiates from other interfaces.
|
||||
@@ -2992,10 +2992,10 @@ func (s *SearchCallContext) LPAREN() antlr.TerminalNode {
|
||||
return s.GetToken(FilterQueryParserLPAREN, 0)
|
||||
}
|
||||
|
||||
func (s *SearchCallContext) ValueList() IValueListContext {
|
||||
func (s *SearchCallContext) FunctionParamList() IFunctionParamListContext {
|
||||
var t antlr.RuleContext
|
||||
for _, ctx := range s.GetChildren() {
|
||||
if _, ok := ctx.(IValueListContext); ok {
|
||||
if _, ok := ctx.(IFunctionParamListContext); ok {
|
||||
t = ctx.(antlr.RuleContext)
|
||||
break
|
||||
}
|
||||
@@ -3005,7 +3005,7 @@ func (s *SearchCallContext) ValueList() IValueListContext {
|
||||
return nil
|
||||
}
|
||||
|
||||
return t.(IValueListContext)
|
||||
return t.(IFunctionParamListContext)
|
||||
}
|
||||
|
||||
func (s *SearchCallContext) RPAREN() antlr.TerminalNode {
|
||||
@@ -3064,7 +3064,7 @@ func (p *FilterQueryParser) SearchCall() (localctx ISearchCallContext) {
|
||||
}
|
||||
{
|
||||
p.SetState(202)
|
||||
p.ValueList()
|
||||
p.FunctionParamList()
|
||||
}
|
||||
{
|
||||
p.SetState(203)
|
||||
|
||||
@@ -20,8 +20,6 @@ import (
|
||||
"github.com/SigNoz/signoz/pkg/valuer"
|
||||
)
|
||||
|
||||
const estimateTimeout = 5 * time.Second
|
||||
|
||||
const traceOutsideRangeWarn = "Query %s references a trace_id that exists between %s and %s (UTC) but lies outside the selected time range; adjust the time range to see results"
|
||||
|
||||
type builderQuery[T any] struct {
|
||||
@@ -249,10 +247,6 @@ func (q *builderQuery[T]) Execute(ctx context.Context) (*qbtypes.Result, error)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := q.enforceEstimate(ctx, stmt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Execute the query with proper context for partial value detection
|
||||
result, err := q.executeWithContext(ctx, stmt.Query, stmt.Args)
|
||||
if err != nil {
|
||||
@@ -264,70 +258,6 @@ func (q *builderQuery[T]) Execute(ctx context.Context) (*qbtypes.Result, error)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// estimateRows runs EXPLAIN ESTIMATE for a cost-guarded statement and returns its
|
||||
// estimated total scan rows. guarded=false means there is nothing to enforce (no
|
||||
// CostGuard or a non-positive budget). A non-nil error means the query must be
|
||||
// rejected: either the estimate itself failed (fail closed — we cannot honor the
|
||||
// budget without it) or the parent context was cancelled (surfaced as-is). Callers
|
||||
// enforce the budget so it can be applied per-statement or cumulatively.
|
||||
func (q *builderQuery[T]) estimateRows(ctx context.Context, stmt *qbtypes.Statement) (int64, bool, error) {
|
||||
if stmt.CostGuard == nil || stmt.CostGuard.MaxScanRows <= 0 {
|
||||
return 0, false, nil
|
||||
}
|
||||
|
||||
estCtx, cancel := context.WithTimeout(ctx, estimateTimeout)
|
||||
defer cancel()
|
||||
|
||||
entries, err := q.telemetryStore.Estimate(estCtx, stmt.Query, stmt.Args...)
|
||||
if err != nil {
|
||||
// Parent cancellation isn't the budget's concern — surface it unchanged.
|
||||
if ctx.Err() != nil {
|
||||
return 0, true, ctx.Err()
|
||||
}
|
||||
// Fail closed: this is a scan-heavy statement and without an estimate we
|
||||
// cannot bound its cost, so running it unbounded is exactly what the guard
|
||||
// exists to prevent.
|
||||
reason := fmt.Sprintf("This query is too broad to plan within %s; narrow the time range or add a more selective filter.", estimateTimeout)
|
||||
if estCtx.Err() != context.DeadlineExceeded {
|
||||
reason = "Could not estimate this query's scan cost; narrow the time range or add a more selective filter."
|
||||
q.logger.WarnContext(ctx, "EXPLAIN ESTIMATE failed; rejecting scan-heavy query (fail-closed)", errors.Attr(err))
|
||||
}
|
||||
return 0, true, errors.NewInvalidInputf(errors.CodeInvalidInput, "%s", withAdvisory(stmt.CostGuard.Warning, reason))
|
||||
}
|
||||
|
||||
var rows int64
|
||||
for _, e := range entries {
|
||||
rows += e.Rows
|
||||
}
|
||||
return rows, true, nil
|
||||
}
|
||||
|
||||
// enforceEstimate rejects a single scan-heavy statement (Statement.CostGuard) whose
|
||||
// EXPLAIN ESTIMATE rows exceed its budget, before executing. Budget 0 disables.
|
||||
func (q *builderQuery[T]) enforceEstimate(ctx context.Context, stmt *qbtypes.Statement) error {
|
||||
rows, guarded, err := q.estimateRows(ctx, stmt)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !guarded {
|
||||
return nil
|
||||
}
|
||||
if budget := stmt.CostGuard.MaxScanRows; rows > budget {
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "%s",
|
||||
withAdvisory(stmt.CostGuard.Warning, fmt.Sprintf("This query would scan about %d rows in this range, over the limit of %d; narrow the time range or add a more selective filter.", rows, budget)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// withAdvisory prefixes reason with the requirement's advisory (e.g. the search()
|
||||
// warning) when present, so the rejection leads with why the query is expensive.
|
||||
func withAdvisory(advisory, reason string) string {
|
||||
if advisory == "" {
|
||||
return reason
|
||||
}
|
||||
return strings.TrimRight(advisory, ". ") + ". " + reason
|
||||
}
|
||||
|
||||
// narrowWindowByTraceID inspects the filter for trace_id predicates and clamps
|
||||
// [fromMS,toMS] to the time range stored in signoz_traces.distributed_trace_summary.
|
||||
// Returns the (possibly narrowed) window, overlap=false when the trace lies
|
||||
@@ -561,11 +491,6 @@ func (q *builderQuery[T]) executeWindowList(ctx context.Context) (*qbtypes.Resul
|
||||
var warnings []string
|
||||
var warningsDocURL string
|
||||
|
||||
// Cost guard is enforced against the cumulative estimate across the buckets we
|
||||
// actually visit — a broad search() that fans every bucket must be bounded by
|
||||
// the per-query budget, not let each bucket pass its slice independently.
|
||||
var estimatedScan int64
|
||||
|
||||
for _, r := range buckets {
|
||||
q.spec.Offset = 0
|
||||
q.spec.Limit = need
|
||||
@@ -576,17 +501,6 @@ func (q *builderQuery[T]) executeWindowList(ctx context.Context) (*qbtypes.Resul
|
||||
}
|
||||
warnings = stmt.Warnings
|
||||
warningsDocURL = stmt.WarningsDocURL
|
||||
rowsEst, guarded, err := q.estimateRows(ctx, stmt)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if guarded {
|
||||
estimatedScan += rowsEst
|
||||
if budget := stmt.CostGuard.MaxScanRows; estimatedScan > budget {
|
||||
return nil, errors.NewInvalidInputf(errors.CodeInvalidInput, "%s",
|
||||
withAdvisory(stmt.CostGuard.Warning, fmt.Sprintf("This query would scan about %d rows across the time range, over the limit of %d; narrow the time range or add a more selective filter.", estimatedScan, budget)))
|
||||
}
|
||||
}
|
||||
// Execute with proper context for partial value detection
|
||||
res, err := q.executeWithContext(ctx, stmt.Query, stmt.Args)
|
||||
if err != nil {
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
|
||||
"github.com/SigNoz/signoz/pkg/errors"
|
||||
"github.com/SigNoz/signoz/pkg/factory"
|
||||
"github.com/SigNoz/signoz/pkg/statementbuilder"
|
||||
)
|
||||
|
||||
const DefaultMaxConcurrentQueries = 8
|
||||
@@ -20,10 +19,6 @@ type Config struct {
|
||||
MaxConcurrentQueries int `yaml:"max_concurrent_queries" mapstructure:"max_concurrent_queries"`
|
||||
// LogTraceIDWindowPadding is the padding added to narrowed down timerange from trace summary to logs with trace_id filter.
|
||||
LogTraceIDWindowPadding time.Duration `yaml:"log_trace_id_window_padding" mapstructure:"log_trace_id_window_padding"`
|
||||
|
||||
// StatementBuilder config is embedded (squashed) so its keys live directly under
|
||||
// the querier namespace, e.g. querier.skip_resource_fingerprint / querier.search_max_scan_rows.
|
||||
statementbuilder.Config `mapstructure:",squash" yaml:",squash"`
|
||||
}
|
||||
|
||||
// NewConfigFactory creates a new config factory for querier.
|
||||
@@ -34,11 +29,10 @@ func NewConfigFactory() factory.ConfigFactory {
|
||||
func newConfig() factory.Config {
|
||||
return Config{
|
||||
// Default values
|
||||
CacheTTL: 168 * time.Hour,
|
||||
FluxInterval: 5 * time.Minute,
|
||||
CacheTTL: 168 * time.Hour,
|
||||
FluxInterval: 5 * time.Minute,
|
||||
MaxConcurrentQueries: DefaultMaxConcurrentQueries,
|
||||
LogTraceIDWindowPadding: 5 * time.Minute,
|
||||
Config: statementbuilder.NewConfig(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -56,10 +50,6 @@ func (c Config) Validate() error {
|
||||
if c.LogTraceIDWindowPadding < 0 {
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "log_trace_id_window_padding must not be negative, got %v", c.LogTraceIDWindowPadding)
|
||||
}
|
||||
// The embedded statement-builder config's own Validate is shadowed by this method; call it explicitly.
|
||||
if err := c.Config.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -9,6 +9,16 @@ import (
|
||||
"github.com/SigNoz/signoz/pkg/errors"
|
||||
)
|
||||
|
||||
var (
|
||||
CodeClickHouseSQLParserPanic = errors.MustNewCode("clickhouse_sql_parser_panic")
|
||||
CodeClickHouseSQLUnparseable = errors.MustNewCode("clickhouse_sql_unparseable")
|
||||
CodeClickHouseSQLNotSingleStatement = errors.MustNewCode("clickhouse_sql_not_single_statement")
|
||||
CodeClickHouseSQLNotSelect = errors.MustNewCode("clickhouse_sql_not_select")
|
||||
CodeClickHouseSQLTableFunction = errors.MustNewCode("clickhouse_sql_table_function")
|
||||
CodeClickHouseSQLInternalDatabase = errors.MustNewCode("clickhouse_sql_internal_database")
|
||||
CodeClickHouseSQLReadonlyOverride = errors.MustNewCode("clickhouse_sql_readonly_override")
|
||||
)
|
||||
|
||||
// internalDatabases hold server metadata, credentials and grants rather than telemetry.
|
||||
var internalDatabases = map[string]struct{}{
|
||||
"system": {},
|
||||
@@ -20,30 +30,30 @@ func ErrIfStatementIsNotValid(query string) (err error) {
|
||||
defer func() {
|
||||
// The parser has a history of panicking on malformed input rather than returning an error.
|
||||
if recovered := recover(); recovered != nil {
|
||||
err = errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid ClickHouse SQL (recovered): %v", recovered)
|
||||
err = errors.NewInvalidInputf(CodeClickHouseSQLParserPanic, "invalid ClickHouse SQL (recovered): %v", recovered)
|
||||
}
|
||||
}()
|
||||
|
||||
stmts, parseErr := chparser.NewParser(query).ParseStmts()
|
||||
if parseErr != nil {
|
||||
// Wrapped rather than formatted in, so that callers can recover the parser's *ParseError and read the position off it.
|
||||
return errors.WrapInvalidInputf(parseErr, errors.CodeInvalidInput, "invalid ClickHouse SQL: %s", parseErr.Error())
|
||||
return errors.WrapInvalidInputf(parseErr, CodeClickHouseSQLUnparseable, "invalid ClickHouse SQL: %s", parseErr.Error())
|
||||
}
|
||||
|
||||
if len(stmts) != 1 {
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "ClickHouse SQL must contain exactly one statement, found %d statements", len(stmts))
|
||||
return errors.NewInvalidInputf(CodeClickHouseSQLNotSingleStatement, "ClickHouse SQL must contain exactly one statement, found %d statements", len(stmts))
|
||||
}
|
||||
|
||||
selectQuery, ok := stmts[0].(*chparser.SelectQuery)
|
||||
if !ok {
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "only SELECT statements are allowed in ClickHouse SQL queries")
|
||||
return errors.NewInvalidInputf(CodeClickHouseSQLNotSelect, "only SELECT statements are allowed in ClickHouse SQL queries")
|
||||
}
|
||||
|
||||
visitor := &chparser.DefaultASTVisitor{Visit: func(node chparser.Expr) error {
|
||||
switch expr := node.(type) {
|
||||
case *chparser.TableFunctionExpr:
|
||||
// Source table functions remain usable in ClickHouse read-only mode.
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "ClickHouse table functions are not allowed in SQL queries: %s", chparser.Format(expr.Name))
|
||||
return errors.NewInvalidInputf(CodeClickHouseSQLTableFunction, "ClickHouse table functions are not allowed in SQL queries: %s", chparser.Format(expr.Name))
|
||||
|
||||
case *chparser.TableIdentifier:
|
||||
// Reading these is unaffected by ClickHouse read-only mode.
|
||||
@@ -52,13 +62,13 @@ func ErrIfStatementIsNotValid(query string) (err error) {
|
||||
}
|
||||
|
||||
if _, ok := internalDatabases[strings.ToLower(expr.Database.Name)]; ok {
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "the ClickHouse %s database is not allowed in SQL queries", expr.Database.Name)
|
||||
return errors.NewInvalidInputf(CodeClickHouseSQLInternalDatabase, "the ClickHouse %s database is not allowed in SQL queries", expr.Database.Name)
|
||||
}
|
||||
|
||||
case *chparser.SettingExpr:
|
||||
// A query-level setting takes precedence over the context setting.
|
||||
if strings.EqualFold(expr.Name.Name, "readonly") {
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "the ClickHouse readonly setting cannot be overridden")
|
||||
return errors.NewInvalidInputf(CodeClickHouseSQLReadonlyOverride, "the ClickHouse readonly setting cannot be overridden")
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -3,6 +3,8 @@ package querybuilder
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/SigNoz/signoz/pkg/errors"
|
||||
|
||||
chparser "github.com/AfterShip/clickhouse-sql-parser/parser"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -30,10 +32,15 @@ func TestErrIfStatementIsNotValid_Pass(t *testing.T) {
|
||||
{"TrailingUnterminatedBlockComment", "SELECT count() FROM t /* unterminated"},
|
||||
// The rule keys on the database, not on the table name.
|
||||
{"TableNamedSystemInTelemetryDatabase", "SELECT * FROM signoz_logs.system"},
|
||||
{"SignedLiteralAfterClosingParen", "SELECT (toUnixTimestamp(now()) - 3600)*1000000000"},
|
||||
{"SignedLiteralAfterClosingParenSpaced", "SELECT (toUnixTimestamp(now()) - 3600)*1000000000"},
|
||||
// order by interval
|
||||
{"OrderByInterval", "SELECT toStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS interval ORDER BY interval"},
|
||||
{"OrderByIntervalAndDirection", "SELECT toStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS `interval` ORDER BY `interval` ASC"},
|
||||
// Unspaced, so rejected until the parser stopped lexing a signed literal after a
|
||||
// closing bracket. The spaced form above no longer needs to be spaced.
|
||||
// https://github.com/AfterShip/clickhouse-sql-parser/issues/286
|
||||
{"SignedLiteralAfterClosingParenUnspaced", "SELECT now() AS ts, toFloat64(count()) AS value FROM ( SELECT attributes_string['TableName'] AS T, attributes_string['MissingId'] AS M, max(fromUnixTimestamp64Nano(timestamp)) AS last_seen, dateDiff('minute', min(fromUnixTimestamp64Nano(timestamp)), max(fromUnixTimestamp64Nano(timestamp))) AS age_min FROM signoz_logs.distributed_logs_v2 WHERE body='missing_map_record' AND timestamp >= (toUnixTimestamp(now())-3600)*1000000000 GROUP BY T, M ) WHERE age_min >= 20 AND last_seen >= now() - toIntervalMinute(8)"},
|
||||
{"SignedLiteralAfterClosingParenMinimal", "SELECT (1)-1"},
|
||||
{"TrimFunction", "SELECT trimBoth('/api/endpoint/', '/');"},
|
||||
}
|
||||
|
||||
@@ -47,49 +54,52 @@ func TestErrIfStatementIsNotValid_Pass(t *testing.T) {
|
||||
|
||||
func TestErrIfStatementIsNotValid_Fail(t *testing.T) {
|
||||
testCases := []struct {
|
||||
name string
|
||||
query string
|
||||
name string
|
||||
query string
|
||||
expectedCode errors.Code
|
||||
}{
|
||||
// Not a single statement, or not a statement at all.
|
||||
{"Empty", ""},
|
||||
{"UnterminatedBlockCommentOnly", "/* x"},
|
||||
{"Unparseable", "SELECT FROM WHERE"},
|
||||
{"MultipleStatements", "SELECT 1; DROP TABLE signoz_logs.logs_v2"},
|
||||
{"Empty", "", CodeClickHouseSQLNotSingleStatement},
|
||||
{"UnterminatedBlockCommentOnly", "/* x", CodeClickHouseSQLUnparseable},
|
||||
{"Unparseable", "SELECT FROM WHERE", CodeClickHouseSQLUnparseable},
|
||||
{"MultipleStatements", "SELECT 1; DROP TABLE signoz_logs.logs_v2", CodeClickHouseSQLNotSingleStatement},
|
||||
// Parses, but is not a SELECT.
|
||||
{"Drop", "DROP TABLE signoz_logs.logs_v2"},
|
||||
{"Insert", "INSERT INTO signoz_logs.logs_v2 SELECT * FROM signoz_logs.logs_v2"},
|
||||
{"AlterDelete", "ALTER TABLE signoz_logs.logs_v2 DELETE WHERE 1 = 1"},
|
||||
{"CreateTable", "CREATE TABLE evil (a Int) ENGINE = Memory"},
|
||||
{"Grant", "GRANT ALL ON *.* TO admin"},
|
||||
{"Set", "SET readonly = 0"},
|
||||
{"Drop", "DROP TABLE signoz_logs.logs_v2", CodeClickHouseSQLNotSelect},
|
||||
{"Insert", "INSERT INTO signoz_logs.logs_v2 SELECT * FROM signoz_logs.logs_v2", CodeClickHouseSQLNotSelect},
|
||||
{"AlterDelete", "ALTER TABLE signoz_logs.logs_v2 DELETE WHERE 1 = 1", CodeClickHouseSQLNotSelect},
|
||||
{"CreateTable", "CREATE TABLE evil (a Int) ENGINE = Memory", CodeClickHouseSQLNotSelect},
|
||||
{"Grant", "GRANT ALL ON *.* TO admin", CodeClickHouseSQLNotSelect},
|
||||
{"Set", "SET readonly = 0", CodeClickHouseSQLNotSelect},
|
||||
// These the parser rejects outright rather than classifying.
|
||||
{"ShowGrants", "SHOW GRANTS"},
|
||||
{"IntoOutfile", "SELECT * FROM t INTO OUTFILE '/tmp/x.csv'"},
|
||||
{"ShowGrants", "SHOW GRANTS", CodeClickHouseSQLUnparseable},
|
||||
{"IntoOutfile", "SELECT * FROM t INTO OUTFILE '/tmp/x.csv'", CodeClickHouseSQLUnparseable},
|
||||
// Table functions, which read through something other than a telemetry table.
|
||||
{"UrlTableFunction", "SELECT * FROM url('http://attacker.example/x', CSV, 'a String')"},
|
||||
{"FileTableFunction", "SELECT * FROM file('/etc/passwd', CSV, 'a String')"},
|
||||
{"ExecutableTableFunction", "SELECT * FROM executable('script.sh', CSV, 'a String')"},
|
||||
{"TableFunctionInJoin", "SELECT * FROM t1 JOIN url('http://x', CSV, 'a String') u ON 1 = 1"},
|
||||
{"TableFunctionInCommonTableExpression", "WITH c AS (SELECT * FROM url('http://x', CSV, 'a String')) SELECT * FROM c"},
|
||||
{"TableFunctionInWhereSubquery", "SELECT * FROM t WHERE a IN (SELECT * FROM file('/etc/passwd', CSV, 'a String'))"},
|
||||
{"TableFunctionInUnion", "SELECT * FROM t UNION ALL SELECT * FROM url('http://x', CSV, 'a String')"},
|
||||
{"UrlTableFunction", "SELECT * FROM url('http://attacker.example/x', CSV, 'a String')", CodeClickHouseSQLTableFunction},
|
||||
{"FileTableFunction", "SELECT * FROM file('/etc/passwd', CSV, 'a String')", CodeClickHouseSQLTableFunction},
|
||||
{"ExecutableTableFunction", "SELECT * FROM executable('script.sh', CSV, 'a String')", CodeClickHouseSQLTableFunction},
|
||||
{"TableFunctionInJoin", "SELECT * FROM t1 JOIN url('http://x', CSV, 'a String') u ON 1 = 1", CodeClickHouseSQLTableFunction},
|
||||
{"TableFunctionInCommonTableExpression", "WITH c AS (SELECT * FROM url('http://x', CSV, 'a String')) SELECT * FROM c", CodeClickHouseSQLTableFunction},
|
||||
{"TableFunctionInWhereSubquery", "SELECT * FROM t WHERE a IN (SELECT * FROM file('/etc/passwd', CSV, 'a String'))", CodeClickHouseSQLTableFunction},
|
||||
{"TableFunctionInUnion", "SELECT * FROM t UNION ALL SELECT * FROM url('http://x', CSV, 'a String')", CodeClickHouseSQLTableFunction},
|
||||
// Internal databases, which hold grants and server metadata rather than telemetry.
|
||||
{"SystemUsers", "SELECT * FROM system.users"},
|
||||
{"SystemUppercase", "SELECT * FROM SYSTEM.USERS"},
|
||||
{"SystemQuoted", "SELECT count() FROM `system`.`tables`"},
|
||||
{"SystemInSubquery", "SELECT * FROM (SELECT name FROM system.parts)"},
|
||||
{"SystemInJoin", "SELECT * FROM signoz_logs.distributed_logs_v2 AS l JOIN system.users AS u ON 1 = 1"},
|
||||
{"SystemInIntersect", "SELECT * FROM t INTERSECT SELECT * FROM system.users"},
|
||||
{"InformationSchema", "SELECT * FROM information_schema.tables"},
|
||||
{"SystemUsers", "SELECT * FROM system.users", CodeClickHouseSQLInternalDatabase},
|
||||
{"SystemUppercase", "SELECT * FROM SYSTEM.USERS", CodeClickHouseSQLInternalDatabase},
|
||||
{"SystemQuoted", "SELECT count() FROM `system`.`tables`", CodeClickHouseSQLInternalDatabase},
|
||||
{"SystemInSubquery", "SELECT * FROM (SELECT name FROM system.parts)", CodeClickHouseSQLInternalDatabase},
|
||||
{"SystemInJoin", "SELECT * FROM signoz_logs.distributed_logs_v2 AS l JOIN system.users AS u ON 1 = 1", CodeClickHouseSQLInternalDatabase},
|
||||
{"SystemInIntersect", "SELECT * FROM t INTERSECT SELECT * FROM system.users", CodeClickHouseSQLInternalDatabase},
|
||||
{"InformationSchema", "SELECT * FROM information_schema.tables", CodeClickHouseSQLInternalDatabase},
|
||||
// A query-level setting takes precedence over the one the caller applies.
|
||||
{"ReadonlySettingOverride", "SELECT * FROM t SETTINGS readonly = 0"},
|
||||
{"ReadonlySettingOverrideAmongOthers", "SELECT * FROM t SETTINGS max_threads = 4, readonly = 0"},
|
||||
{"ReadonlySettingOverride", "SELECT * FROM t SETTINGS readonly = 0", CodeClickHouseSQLReadonlyOverride},
|
||||
{"ReadonlySettingOverrideAmongOthers", "SELECT * FROM t SETTINGS max_threads = 4, readonly = 0", CodeClickHouseSQLReadonlyOverride},
|
||||
}
|
||||
|
||||
for _, testCase := range testCases {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
err := ErrIfStatementIsNotValid(testCase.query)
|
||||
|
||||
assert.Error(t, err)
|
||||
assert.True(t, errors.Asc(err, testCase.expectedCode), "expected code %s, got %v", testCase.expectedCode, err)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -122,18 +132,6 @@ func TestErrIfStatementIsNotValid_ShouldPassButFails(t *testing.T) {
|
||||
expectedStopsAfter: "trim(BOTH '",
|
||||
fix: "SELECT trimBoth(replaceOne( JSONExtractString(body, 'Action'), 'health_status: ', '' ), ' ')",
|
||||
},
|
||||
{
|
||||
name: "SignedLiteralAfterClosingParen",
|
||||
query: "SELECT now() AS ts, toFloat64(count()) AS value FROM ( SELECT attributes_string['TableName'] AS T, attributes_string['MissingId'] AS M, max(fromUnixTimestamp64Nano(timestamp)) AS last_seen, dateDiff('minute', min(fromUnixTimestamp64Nano(timestamp)), max(fromUnixTimestamp64Nano(timestamp))) AS age_min FROM signoz_logs.distributed_logs_v2 WHERE body='missing_map_record' AND timestamp >= (toUnixTimestamp(now())-3600)*1000000000 GROUP BY T, M ) WHERE age_min >= 20 AND last_seen >= now() - toIntervalMinute(8)",
|
||||
expectedStopsAfter: "(toUnixTimestamp(now())",
|
||||
fix: "SELECT (toUnixTimestamp(now()) - 3600) * 1000000000",
|
||||
},
|
||||
{
|
||||
name: "SignedLiteralAfterClosingParenMinimal",
|
||||
query: "SELECT (1)-1",
|
||||
expectedStopsAfter: "(1)",
|
||||
fix: "SELECT (1) - 1",
|
||||
},
|
||||
}
|
||||
|
||||
for _, testCase := range testCases {
|
||||
|
||||
@@ -8,10 +8,6 @@ const (
|
||||
// BodyFullTextSearchDefaultWarning is emitted when a full-text search or "body" searches are hit
|
||||
// with New JSON Body enhancements.
|
||||
BodyFullTextSearchDefaultWarning = "Full text searches default to `body.message:string`. Use `body.<key>` to search a different field inside body"
|
||||
|
||||
// SearchWarning is emitted on every search() call. search() scans all fields,
|
||||
// so it is slow and expensive; a specific field is cheaper.
|
||||
SearchWarning = "search() runs across all fields and can be slow and expensive. Prefer a specific field, e.g. `<context>.<field_key>:<type>`"
|
||||
)
|
||||
|
||||
var (
|
||||
|
||||
@@ -161,16 +161,14 @@ func inferDataTypesFromList(values []any) []telemetrytypes.FieldDataType {
|
||||
return out
|
||||
}
|
||||
|
||||
// NewFunctionUnsupportedError returns the error for a has/hasAny/hasAll/hasToken/search
|
||||
// operator on a builder that doesn't support it (logs only), or nil for other operators.
|
||||
// NewFunctionUnsupportedError returns the error for a has/hasAny/hasAll/hasToken operator
|
||||
// on a builder that doesn't support it (logs body only), or nil for other operators.
|
||||
func NewFunctionUnsupportedError(operator qbtypes.FilterOperator) error {
|
||||
switch operator {
|
||||
case qbtypes.FilterOperatorHasToken:
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "function `hasToken` only supports body field as first parameter").WithUrl(hasTokenFunctionDocURL)
|
||||
case qbtypes.FilterOperatorHas, qbtypes.FilterOperatorHasAny, qbtypes.FilterOperatorHasAll:
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "function `%s` supports only body JSON search", operator.FunctionName()).WithUrl(functionBodyJSONSearchDocURL)
|
||||
case qbtypes.FilterOperatorSearch:
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "function `search` is only supported for logs")
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -43,8 +43,6 @@ type filterExpressionVisitor struct {
|
||||
keysWithWarnings map[string]bool
|
||||
startNs uint64
|
||||
endNs uint64
|
||||
|
||||
requiresCostGuard bool
|
||||
}
|
||||
|
||||
type FilterExprVisitorOpts struct {
|
||||
@@ -83,10 +81,9 @@ func newFilterExpressionVisitor(opts FilterExprVisitorOpts) *filterExpressionVis
|
||||
}
|
||||
|
||||
type PreparedWhereClause struct {
|
||||
WhereClause *sqlbuilder.WhereClause
|
||||
Warnings []string
|
||||
WarningsDocURL string
|
||||
RequiresCostGuard bool
|
||||
WhereClause *sqlbuilder.WhereClause
|
||||
Warnings []string
|
||||
WarningsDocURL string
|
||||
}
|
||||
|
||||
func (p PreparedWhereClause) IsEmpty() bool {
|
||||
@@ -168,12 +165,12 @@ func PrepareWhereClause(query string, opts FilterExprVisitorOpts) (PreparedWhere
|
||||
|
||||
// Return empty where clause so callers can skip the WHERE clause
|
||||
if cond == "" || cond == SkipConditionLiteral {
|
||||
return PreparedWhereClause{WhereClause: nil, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL, RequiresCostGuard: visitor.requiresCostGuard}, nil
|
||||
return PreparedWhereClause{WhereClause: nil, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL}, nil
|
||||
}
|
||||
|
||||
whereClause := sqlbuilder.NewWhereClause().AddWhereExpr(visitor.builder.Args, cond)
|
||||
|
||||
return PreparedWhereClause{WhereClause: whereClause, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL, RequiresCostGuard: visitor.requiresCostGuard}, nil
|
||||
return PreparedWhereClause{WhereClause: whereClause, Warnings: visitor.warnings, WarningsDocURL: visitor.mainWarnURL}, nil
|
||||
}
|
||||
|
||||
// Visit dispatches to the specific visit method based on node type.
|
||||
@@ -779,79 +776,11 @@ func normalizeFunctionValue(operator qbtypes.FilterOperator, functionName string
|
||||
return valueParams, nil
|
||||
}
|
||||
|
||||
// VisitSearchCall handles search('needle') and its scoped forms, e.g.
|
||||
// search('needle', body, resource). The first arg is the case-insensitive needle; the
|
||||
// rest are field-context scopes (bare or quoted). Emits one FilterOperatorSearch per
|
||||
// scope (none = keyless, covering every field) and ORs them.
|
||||
// VisitSearchCall handles search('needle'). The search() function is parsed but
|
||||
// not yet implemented; reject it with a clear invalid-input error.
|
||||
func (v *filterExpressionVisitor) VisitSearchCall(ctx *grammar.SearchCallContext) any {
|
||||
// Flag scan-heavy so the statement builder attaches the cost guard.
|
||||
v.requiresCostGuard = true
|
||||
|
||||
valueList := ctx.ValueList()
|
||||
if valueList == nil {
|
||||
v.errors = append(v.errors, "function `search` expects a needle, e.g. search('error')")
|
||||
return ErrorConditionLiteral
|
||||
}
|
||||
params := valueList.AllValue()
|
||||
if len(params) == 0 {
|
||||
v.errors = append(v.errors, "function `search` expects a needle, e.g. search('error')")
|
||||
return ErrorConditionLiteral
|
||||
}
|
||||
|
||||
searchText, ok := searchParamText(params[0])
|
||||
if !ok {
|
||||
v.errors = append(v.errors, "function `search` expects a needle as its first argument, e.g. search('error')")
|
||||
return ErrorConditionLiteral
|
||||
}
|
||||
|
||||
var fieldContexts []telemetrytypes.FieldContext
|
||||
if len(params) == 1 {
|
||||
fieldContexts = []telemetrytypes.FieldContext{telemetrytypes.FieldContextUnspecified}
|
||||
} else {
|
||||
for _, p := range params[1:] {
|
||||
scopeText, sok := searchParamText(p)
|
||||
if !sok {
|
||||
v.errors = append(v.errors, "function `search` expects each scope to be a context, e.g. search('error', body, resource)")
|
||||
return ErrorConditionLiteral
|
||||
}
|
||||
fc, fok := telemetrytypes.FieldContextFromText(scopeText)
|
||||
if !fok {
|
||||
v.errors = append(v.errors, fmt.Sprintf("invalid search scope %q; expected a field context: body, attribute, resource, or log", scopeText))
|
||||
return ErrorConditionLiteral
|
||||
}
|
||||
fieldContexts = append(fieldContexts, fc)
|
||||
}
|
||||
}
|
||||
|
||||
var conds []string
|
||||
for _, fieldContext := range fieldContexts {
|
||||
key := telemetrytypes.NewTelemetryFieldKey("", fieldContext, telemetrytypes.FieldDataTypeUnspecified)
|
||||
scoped, cok := v.buildConditions(key, nil, qbtypes.FilterOperatorSearch, searchText)
|
||||
if !cok {
|
||||
return ErrorConditionLiteral
|
||||
}
|
||||
conds = append(conds, scoped...)
|
||||
}
|
||||
if len(conds) == 0 {
|
||||
return SkipConditionLiteral
|
||||
}
|
||||
if len(conds) == 1 {
|
||||
return conds[0]
|
||||
}
|
||||
return v.builder.Or(conds...)
|
||||
}
|
||||
|
||||
// searchParamText returns the raw token text of a search() argument (quoted or bare),
|
||||
// not the visited value — so a bare word stays literal and search(1000000) doesn't
|
||||
// become "1e+06".
|
||||
func searchParamText(val grammar.IValueContext) (string, bool) {
|
||||
if val == nil {
|
||||
return "", false
|
||||
}
|
||||
if val.QUOTED_TEXT() != nil {
|
||||
return trimQuotes(val.QUOTED_TEXT().GetText()), true
|
||||
}
|
||||
return val.GetText(), true
|
||||
v.errors = append(v.errors, "function `search` is not yet supported")
|
||||
return ErrorConditionLiteral
|
||||
}
|
||||
|
||||
// VisitFunctionParamList handles the parameter list for function calls.
|
||||
|
||||
@@ -39,6 +39,7 @@ import (
|
||||
"github.com/SigNoz/signoz/pkg/sqlmigrator"
|
||||
"github.com/SigNoz/signoz/pkg/sqlschema"
|
||||
"github.com/SigNoz/signoz/pkg/sqlstore"
|
||||
"github.com/SigNoz/signoz/pkg/statementbuilder"
|
||||
"github.com/SigNoz/signoz/pkg/statsreporter"
|
||||
"github.com/SigNoz/signoz/pkg/telemetrystore"
|
||||
"github.com/SigNoz/signoz/pkg/tokenizer"
|
||||
@@ -94,9 +95,12 @@ type Config struct {
|
||||
// Alertmanager config
|
||||
Alertmanager alertmanager.Config `mapstructure:"alertmanager" yaml:"alertmanager"`
|
||||
|
||||
// Querier config (the statement-builder config is embedded under it)
|
||||
// Querier config
|
||||
Querier querier.Config `mapstructure:"querier"`
|
||||
|
||||
// StatementBuilder config
|
||||
StatementBuilder statementbuilder.Config `mapstructure:"statementbuilder"`
|
||||
|
||||
// Ruler config
|
||||
Ruler ruler.Config `mapstructure:"ruler"`
|
||||
|
||||
@@ -166,6 +170,7 @@ func NewConfig(ctx context.Context, logger *slog.Logger, resolverConfig config.R
|
||||
prometheus.NewConfigFactory(),
|
||||
alertmanager.NewConfigFactory(),
|
||||
querier.NewConfigFactory(),
|
||||
statementbuilder.NewConfigFactory(),
|
||||
ruler.NewConfigFactory(),
|
||||
emailing.NewConfigFactory(),
|
||||
sharder.NewConfigFactory(),
|
||||
|
||||
@@ -122,7 +122,7 @@ func newQueryStack(
|
||||
) {
|
||||
metadataStore := telemetrymetadata.NewTelemetryMetaStore(settings, telemetryStore, fl)
|
||||
|
||||
cfg := config.Querier.Config
|
||||
cfg := config.StatementBuilder
|
||||
traceStmtBuilder, err := tracesstatementbuilder.NewFactory(telemetryStore, metadataStore, fl).New(ctx, settings, cfg)
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, nil, nil, nil, nil, err
|
||||
|
||||
@@ -205,7 +205,7 @@ func (b *auditQueryStatementBuilder) adjustKeys(ctx context.Context, keys map[st
|
||||
}
|
||||
|
||||
for _, action := range actions {
|
||||
b.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
|
||||
b.logger.DebugContext(ctx, "key adjustment action", slog.String("action", action))
|
||||
}
|
||||
|
||||
return query
|
||||
|
||||
@@ -2,6 +2,7 @@ package statementbuilder
|
||||
|
||||
import (
|
||||
"github.com/SigNoz/signoz/pkg/errors"
|
||||
"github.com/SigNoz/signoz/pkg/factory"
|
||||
)
|
||||
|
||||
// SkipResourceFingerprint configures when the resource fingerprint subquery is skipped in favor of main-table filtering.
|
||||
@@ -15,21 +16,19 @@ type SkipResourceFingerprint struct {
|
||||
type Config struct {
|
||||
// SkipResourceFingerprint configures when the resource fingerprint subquery is skipped in favor of main-table filtering.
|
||||
SkipResourceFingerprint SkipResourceFingerprint `yaml:"skip_resource_fingerprint" mapstructure:"skip_resource_fingerprint"`
|
||||
// SearchMaxScanRows caps the rows a search() query may scan; the log statement builder
|
||||
// attaches it to the CostGuard and the querier enforces it via EXPLAIN ESTIMATE (0 disables).
|
||||
SearchMaxScanRows int64 `yaml:"search_max_scan_rows" mapstructure:"search_max_scan_rows"`
|
||||
}
|
||||
|
||||
// NewConfig returns the statement-builder config with its default values. It is
|
||||
// embedded (squashed) into querier.Config rather than registered as a standalone
|
||||
// config section, so its defaults are seeded from querier's newConfig.
|
||||
func NewConfig() Config {
|
||||
// NewConfigFactory creates a new config factory for the statement builders.
|
||||
func NewConfigFactory() factory.ConfigFactory {
|
||||
return factory.NewConfigFactory(factory.MustNewName("statementbuilder"), newConfig)
|
||||
}
|
||||
|
||||
func newConfig() factory.Config {
|
||||
return Config{
|
||||
SkipResourceFingerprint: SkipResourceFingerprint{
|
||||
Enabled: false,
|
||||
Threshold: 100000,
|
||||
},
|
||||
SearchMaxScanRows: 0,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -38,8 +37,5 @@ func (c Config) Validate() error {
|
||||
if c.SkipResourceFingerprint.Enabled && c.SkipResourceFingerprint.Threshold == 0 {
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "skip_resource_fingerprint.threshold must be > 0 when enabled")
|
||||
}
|
||||
if c.SearchMaxScanRows < 0 {
|
||||
return errors.NewInvalidInputf(errors.CodeInvalidInput, "search_max_scan_rows must not be negative, got %v", c.SearchMaxScanRows)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -8,7 +8,6 @@ import (
|
||||
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
|
||||
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
|
||||
"github.com/SigNoz/signoz/pkg/querybuilder"
|
||||
"github.com/SigNoz/signoz/pkg/statementbuilder"
|
||||
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
|
||||
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
|
||||
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
|
||||
@@ -52,8 +51,7 @@ func TestStatementBuilderGroupByUnknownKey(t *testing.T) {
|
||||
statementBuilder := NewLogQueryStatementBuilder(
|
||||
instrumentationtest.New().ToProviderSettings(),
|
||||
mockMetadataStore, fm, cb, aggExprRewriter,
|
||||
logstelemetryschema.DefaultFullTextColumn, fl, nil,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
logstelemetryschema.DefaultFullTextColumn, fl, nil, false, 100000,
|
||||
)
|
||||
|
||||
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
|
||||
|
||||
@@ -11,7 +11,6 @@ import (
|
||||
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
|
||||
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
|
||||
"github.com/SigNoz/signoz/pkg/querybuilder"
|
||||
"github.com/SigNoz/signoz/pkg/statementbuilder"
|
||||
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
|
||||
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
|
||||
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
|
||||
@@ -1347,7 +1346,8 @@ func buildJSONTestStatementBuilder(t *testing.T, addIndexes bool) (*logQueryStat
|
||||
logstelemetryschema.DefaultFullTextColumn,
|
||||
fl,
|
||||
nil,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
false,
|
||||
100000,
|
||||
)
|
||||
|
||||
return statementBuilder, mockMetadataStore
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
package logsstatementbuilder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
|
||||
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
|
||||
"github.com/SigNoz/signoz/pkg/querybuilder"
|
||||
"github.com/SigNoz/signoz/pkg/statementbuilder"
|
||||
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
|
||||
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
|
||||
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
|
||||
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
|
||||
"github.com/SigNoz/signoz/pkg/valuer"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// TestSearchCostGuard covers the search() path end-to-end through Build: the
|
||||
// statement carries a CostGuard with its scan budget and the advisory warning.
|
||||
func TestSearchCostGuard(t *testing.T) {
|
||||
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
|
||||
ctx := context.Background()
|
||||
start := uint64(releaseTime.Add(-5 * time.Minute).UnixMilli())
|
||||
end := uint64(releaseTime.UnixMilli())
|
||||
|
||||
fl := flaggertest.WithBooleanFlags(t, map[string]bool{})
|
||||
fm := logstelemetryschema.NewFieldMapper(fl)
|
||||
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
|
||||
store := telemetrytypestest.NewMockMetadataStore()
|
||||
store.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
|
||||
rewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
|
||||
sb := NewLogQueryStatementBuilder(
|
||||
instrumentationtest.New().ToProviderSettings(),
|
||||
store, fm, cb, rewriter, logstelemetryschema.DefaultFullTextColumn, fl, nil,
|
||||
statementbuilder.Config{SearchMaxScanRows: 100000, SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
)
|
||||
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{
|
||||
Signal: telemetrytypes.SignalLogs,
|
||||
Filter: &qbtypes.Filter{Expression: "search('error')"},
|
||||
Limit: 1,
|
||||
}
|
||||
|
||||
stmt, err := sb.Build(ctx, valuer.UUID{}, start, end, qbtypes.RequestTypeRaw, query, nil)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, stmt.CostGuard)
|
||||
require.Contains(t, stmt.Warnings, querybuilder.SearchWarning)
|
||||
}
|
||||
@@ -41,15 +41,14 @@ type logQueryStatementBuilder struct {
|
||||
fl flagger.Flagger
|
||||
skipResourceFingerprintEnabled bool
|
||||
|
||||
fullTextColumn *telemetrytypes.TelemetryFieldKey
|
||||
searchMaxScanRows int64
|
||||
fullTextColumn *telemetrytypes.TelemetryFieldKey
|
||||
}
|
||||
|
||||
var _ qbtypes.StatementBuilder[qbtypes.LogAggregation] = (*logQueryStatementBuilder)(nil)
|
||||
|
||||
// NewFactory returns a provider factory for the logs statement builder. Its New
|
||||
// internalizes the FieldMapper, ConditionBuilder, and AggExprRewriter, and reads
|
||||
// SkipResourceFingerprint and SearchMaxScanRows from the config.
|
||||
// SkipResourceFingerprint from the config.
|
||||
func NewFactory(
|
||||
telemetryStore telemetrystore.TelemetryStore,
|
||||
metadataStore telemetrytypes.MetadataStore,
|
||||
@@ -63,7 +62,7 @@ func NewFactory(
|
||||
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, logstelemetryschema.DefaultFullTextColumn, fm, cb, fl)
|
||||
return NewLogQueryStatementBuilder(
|
||||
settings, metadataStore, fm, cb, aggExprRewriter, logstelemetryschema.DefaultFullTextColumn,
|
||||
fl, telemetryStore, cfg,
|
||||
fl, telemetryStore, cfg.SkipResourceFingerprint.Enabled, cfg.SkipResourceFingerprint.Threshold,
|
||||
), nil
|
||||
},
|
||||
)
|
||||
@@ -78,7 +77,8 @@ func NewLogQueryStatementBuilder(
|
||||
fullTextColumn *telemetrytypes.TelemetryFieldKey,
|
||||
fl flagger.Flagger,
|
||||
telemetryStore telemetrystore.TelemetryStore,
|
||||
cfg statementbuilder.Config,
|
||||
skipResourceFingerprintEnable bool,
|
||||
skipResourceFingerprintThreshold uint64,
|
||||
) *logQueryStatementBuilder {
|
||||
logsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema")
|
||||
|
||||
@@ -92,10 +92,10 @@ func NewLogQueryStatementBuilder(
|
||||
fullTextColumn,
|
||||
fl,
|
||||
telemetryStore,
|
||||
cfg.SkipResourceFingerprint.Threshold,
|
||||
skipResourceFingerprintThreshold,
|
||||
)
|
||||
|
||||
b := &logQueryStatementBuilder{
|
||||
return &logQueryStatementBuilder{
|
||||
logger: logsSettings.Logger(),
|
||||
metadataStore: metadataStore,
|
||||
fm: fieldMapper,
|
||||
@@ -103,11 +103,9 @@ func NewLogQueryStatementBuilder(
|
||||
resourceFilterResolver: resourceFilterResolver,
|
||||
aggExprRewriter: aggExprRewriter,
|
||||
fl: fl,
|
||||
skipResourceFingerprintEnabled: cfg.SkipResourceFingerprint.Enabled,
|
||||
searchMaxScanRows: cfg.SearchMaxScanRows,
|
||||
skipResourceFingerprintEnabled: skipResourceFingerprintEnable,
|
||||
fullTextColumn: fullTextColumn,
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// Build builds a SQL query for logs based on the given parameters.
|
||||
@@ -153,23 +151,9 @@ func (b *logQueryStatementBuilder) Build(
|
||||
}
|
||||
|
||||
stmt.Warnings = append(stmt.Warnings, warnings...)
|
||||
// The search flag is gated in the condition builder; here the advisory rides on
|
||||
// the statement so the querier can enforce the scan budget from the CostGuard.
|
||||
if stmt.CostGuard != nil && stmt.CostGuard.Warning != "" {
|
||||
stmt.Warnings = append(stmt.Warnings, stmt.CostGuard.Warning)
|
||||
}
|
||||
return stmt, nil
|
||||
}
|
||||
|
||||
// costGuardFor builds the cost guard for a scan-heavy (search()) statement,
|
||||
// pairing the advisory with the configured scan budget. Returns nil otherwise.
|
||||
func (b *logQueryStatementBuilder) costGuardFor(required bool) *qbtypes.CostGuard {
|
||||
if !required {
|
||||
return nil
|
||||
}
|
||||
return &qbtypes.CostGuard{Warning: querybuilder.SearchWarning, MaxScanRows: b.searchMaxScanRows}
|
||||
}
|
||||
|
||||
func getKeySelectors(query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation], bodyJSONEnabled bool) ([]*telemetrytypes.FieldKeySelector, []string) {
|
||||
var keySelectors []*telemetrytypes.FieldKeySelector
|
||||
var warnings []string
|
||||
@@ -290,8 +274,7 @@ func (b *logQueryStatementBuilder) adjustKeys(ctx context.Context, keys map[stri
|
||||
}
|
||||
|
||||
for _, action := range actions {
|
||||
// TODO: change to debug level once we are confident about the behavior
|
||||
b.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
|
||||
b.logger.DebugContext(ctx, "key adjustment action", slog.String("action", action))
|
||||
}
|
||||
|
||||
return query
|
||||
@@ -407,7 +390,6 @@ func (b *logQueryStatementBuilder) buildListQuery(
|
||||
Args: finalArgs,
|
||||
Warnings: preparedWhereClause.Warnings,
|
||||
WarningsDocURL: preparedWhereClause.WarningsDocURL,
|
||||
CostGuard: b.costGuardFor(preparedWhereClause.RequiresCostGuard),
|
||||
}
|
||||
|
||||
return stmt, nil
|
||||
@@ -574,7 +556,6 @@ func (b *logQueryStatementBuilder) buildTimeSeriesQuery(
|
||||
Args: finalArgs,
|
||||
Warnings: preparedWhereClause.Warnings,
|
||||
WarningsDocURL: preparedWhereClause.WarningsDocURL,
|
||||
CostGuard: b.costGuardFor(preparedWhereClause.RequiresCostGuard),
|
||||
}
|
||||
|
||||
return stmt, nil
|
||||
@@ -702,7 +683,6 @@ func (b *logQueryStatementBuilder) buildScalarQuery(
|
||||
Args: finalArgs,
|
||||
Warnings: preparedWhereClause.Warnings,
|
||||
WarningsDocURL: preparedWhereClause.WarningsDocURL,
|
||||
CostGuard: b.costGuardFor(preparedWhereClause.RequiresCostGuard),
|
||||
}
|
||||
|
||||
return stmt, nil
|
||||
|
||||
@@ -11,7 +11,6 @@ import (
|
||||
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
|
||||
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
|
||||
"github.com/SigNoz/signoz/pkg/querybuilder"
|
||||
"github.com/SigNoz/signoz/pkg/statementbuilder"
|
||||
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
|
||||
"github.com/SigNoz/signoz/pkg/telemetrystore"
|
||||
"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
|
||||
@@ -232,7 +231,8 @@ func TestStatementBuilderTimeSeries(t *testing.T) {
|
||||
logstelemetryschema.DefaultFullTextColumn,
|
||||
fl,
|
||||
nil,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
false,
|
||||
100000,
|
||||
)
|
||||
|
||||
for _, c := range cases {
|
||||
@@ -374,7 +374,8 @@ func TestStatementBuilderListQuery(t *testing.T) {
|
||||
logstelemetryschema.DefaultFullTextColumn,
|
||||
fl,
|
||||
nil,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
false,
|
||||
100000,
|
||||
)
|
||||
|
||||
for _, c := range cases {
|
||||
@@ -524,7 +525,8 @@ func TestStatementBuilderListQueryResourceTests(t *testing.T) {
|
||||
logstelemetryschema.DefaultFullTextColumn,
|
||||
fl,
|
||||
nil,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
false,
|
||||
100000,
|
||||
)
|
||||
|
||||
for _, c := range cases {
|
||||
@@ -601,7 +603,8 @@ func TestStatementBuilderTimeSeriesBodyGroupBy(t *testing.T) {
|
||||
logstelemetryschema.DefaultFullTextColumn,
|
||||
fl,
|
||||
nil,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
false,
|
||||
100000,
|
||||
)
|
||||
|
||||
for _, c := range cases {
|
||||
@@ -697,7 +700,8 @@ func TestStatementBuilderListQueryServiceCollision(t *testing.T) {
|
||||
logstelemetryschema.DefaultFullTextColumn,
|
||||
fl,
|
||||
nil,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
false,
|
||||
100000,
|
||||
)
|
||||
|
||||
for _, c := range cases {
|
||||
@@ -922,7 +926,8 @@ func TestAdjustKey(t *testing.T) {
|
||||
logstelemetryschema.DefaultFullTextColumn,
|
||||
fl,
|
||||
nil,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
false,
|
||||
100000,
|
||||
)
|
||||
|
||||
for _, c := range cases {
|
||||
@@ -1068,7 +1073,8 @@ func TestStmtBuilderBodyField(t *testing.T) {
|
||||
logstelemetryschema.DefaultFullTextColumn,
|
||||
fl,
|
||||
nil,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
false,
|
||||
100000,
|
||||
)
|
||||
|
||||
q, err := statementBuilder.Build(context.Background(), valuer.UUID{}, 1747947419000, 1747983448000, c.requestType, c.query, nil)
|
||||
@@ -1168,7 +1174,8 @@ func TestStmtBuilderBodyFullTextSearch(t *testing.T) {
|
||||
logstelemetryschema.DefaultFullTextColumn,
|
||||
fl,
|
||||
nil,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: false, Threshold: 100000}},
|
||||
false,
|
||||
100000,
|
||||
)
|
||||
|
||||
q, err := statementBuilder.Build(context.Background(), valuer.UUID{}, 1747947419000, 1747983448000, c.requestType, c.query, nil)
|
||||
@@ -1289,6 +1296,7 @@ func newSkipResourceFingerprintLogsBuilder(
|
||||
logstelemetryschema.DefaultFullTextColumn,
|
||||
fl,
|
||||
telemetryStore,
|
||||
statementbuilder.Config{SkipResourceFingerprint: statementbuilder.SkipResourceFingerprint{Enabled: skipEnable, Threshold: threshold}},
|
||||
skipEnable,
|
||||
threshold,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -127,8 +127,7 @@ func (b *traceQueryStatementBuilder) Build(
|
||||
}
|
||||
|
||||
for _, action := range adjustTraceKeys(keys, &query, requestType) {
|
||||
// TODO: change to debug level once we are confident about the behavior
|
||||
b.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
|
||||
b.logger.DebugContext(ctx, "key adjustment action", slog.String("action", action))
|
||||
}
|
||||
// Create SQL builder
|
||||
q := sqlbuilder.NewSelectBuilder()
|
||||
|
||||
@@ -223,8 +223,7 @@ func (b *traceOperatorCTEBuilder) buildQueryCTE(ctx context.Context, queryName s
|
||||
// The CTE only selects spans matching the filter. Aggregations, group by
|
||||
// and order by run later in buildFinalQuery, so RequestTypeRaw is fine here.
|
||||
for _, action := range adjustTraceKeys(keys, query, qbtypes.RequestTypeRaw) {
|
||||
// TODO: change to debug level once we are confident about the behavior
|
||||
b.stmtBuilder.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
|
||||
b.stmtBuilder.logger.DebugContext(ctx, "key adjustment action", slog.String("action", action))
|
||||
}
|
||||
|
||||
// Build resource filter CTE for this specific query
|
||||
@@ -576,7 +575,7 @@ func (b *traceOperatorCTEBuilder) adjustOperatorKeys(ctx context.Context, keys m
|
||||
b.operator.Order = tmp.Order
|
||||
|
||||
for _, action := range actions {
|
||||
b.stmtBuilder.logger.InfoContext(ctx, "key adjustment action", slog.String("action", action))
|
||||
b.stmtBuilder.logger.DebugContext(ctx, "key adjustment action", slog.String("action", action))
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -134,7 +134,7 @@ func (c *conditionBuilder) ConditionFor(
|
||||
sb *sqlbuilder.SelectBuilder,
|
||||
) ([]string, []string, error) {
|
||||
|
||||
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for audit.
|
||||
// has/hasAny/hasAll/hasToken are logs-body-only; reject for audit.
|
||||
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
@@ -3,7 +3,6 @@ package logstelemetryschema
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"regexp"
|
||||
|
||||
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
|
||||
"github.com/SigNoz/signoz/pkg/errors"
|
||||
@@ -28,52 +27,6 @@ func NewConditionBuilder(fm qbtypes.FieldMapper, fl flagger.Flagger) *conditionB
|
||||
return &conditionBuilder{fm: fm, fl: fl}
|
||||
}
|
||||
|
||||
// conditionForSearch ORs a case-insensitive match of the needle across the searchable
|
||||
// columns of the key's field context (an unspecified context covers every column).
|
||||
func (c *conditionBuilder) conditionForSearch(
|
||||
ctx context.Context,
|
||||
orgID valuer.UUID,
|
||||
key *telemetrytypes.TelemetryFieldKey,
|
||||
value any,
|
||||
sb *sqlbuilder.SelectBuilder,
|
||||
) ([]string, []string, error) {
|
||||
// Literal case-insensitive substring match: QuoteMeta the needle and LOWER both sides
|
||||
// so the body path can use the LOWER(toString(body_v2)) skip index (a (?i) regex could not).
|
||||
needle := regexp.QuoteMeta(fmt.Sprintf("%v", value))
|
||||
|
||||
useJSONBody := c.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))
|
||||
|
||||
var conditions []string
|
||||
|
||||
for _, col := range searchColumns(key.FieldContext, useJSONBody) {
|
||||
switch col.Type.GetType() {
|
||||
case schema.ColumnTypeEnumMap:
|
||||
keysExpr := fmt.Sprintf("mapKeys(%s)", col.Name)
|
||||
valsExpr := fmt.Sprintf("mapValues(%s)", col.Name)
|
||||
// match() needs a String array; cast non-string map values first.
|
||||
if mc, ok := col.Type.(schema.MapColumnType); ok && mc.ValueType.GetType() != schema.ColumnTypeEnumString {
|
||||
valsExpr = fmt.Sprintf("arrayMap(x -> toString(x), mapValues(%s))", col.Name)
|
||||
}
|
||||
conditions = append(conditions, sb.Or(
|
||||
fmt.Sprintf("arrayExists(x -> match(LOWER(x), LOWER(%s)), %s)", sb.Var(needle), keysExpr),
|
||||
fmt.Sprintf("arrayExists(x -> match(LOWER(x), LOWER(%s)), %s)", sb.Var(needle), valsExpr),
|
||||
))
|
||||
case schema.ColumnTypeEnumJSON:
|
||||
conditions = append(conditions, fmt.Sprintf("match(LOWER(toString(%s)), LOWER(%s))", col.Name, sb.Var(needle)))
|
||||
case schema.ColumnTypeEnumString, schema.ColumnTypeEnumLowCardinality:
|
||||
conditions = append(conditions, fmt.Sprintf("match(LOWER(%s), LOWER(%s))", col.Name, sb.Var(needle)))
|
||||
default:
|
||||
return nil, nil, errors.NewInternalf(errors.CodeInternal, "search does not support the column type of %q", col.Name)
|
||||
}
|
||||
}
|
||||
|
||||
if len(conditions) == 0 {
|
||||
return nil, nil, nil
|
||||
}
|
||||
// The advisory rides on CostGuard (set by the visitor), not warnings.
|
||||
return []string{sb.Or(conditions...)}, nil, nil
|
||||
}
|
||||
|
||||
// isBodyJSONSearch reports whether a key addresses a path within the body JSON. Only
|
||||
// an explicit Body context qualifies; a bare, context-less `body` (e.g. full-text
|
||||
// `count_distinct(body)` or `body EXISTS`) is a full-text match, not a `$.body` path.
|
||||
@@ -455,11 +408,6 @@ func (c *conditionBuilder) ConditionFor(
|
||||
matches := querybuilder.MatchingFieldKeys(key, fieldKeys)
|
||||
skipResourceFilter := options.SkipResourceFilter
|
||||
|
||||
// search() resolves its own (optional) scope; handle it before key resolution.
|
||||
if operator == qbtypes.FilterOperatorSearch {
|
||||
return c.conditionForSearch(ctx, orgID, key, value, sb)
|
||||
}
|
||||
|
||||
keys, warning := querybuilder.ResolveKeys(key, matches)
|
||||
var warnings []string
|
||||
if warning != "" {
|
||||
|
||||
@@ -635,37 +635,3 @@ func (m *fieldMapper) existsExpressionFor(
|
||||
}
|
||||
return querybuilder.ExistsExpression(columns, key, tsStart, tsEnd, fieldExpression, exists)
|
||||
}
|
||||
|
||||
// searchColumns is the single source of truth for the columns search() fans out
|
||||
// across, by field context. Body is body_v2 JSON when useJSONBody, else body string.
|
||||
func searchColumns(fieldContext telemetrytypes.FieldContext, useJSONBody bool) []*schema.Column {
|
||||
switch fieldContext {
|
||||
case telemetrytypes.FieldContextLog:
|
||||
return []*schema.Column{
|
||||
logsV2Columns[LogsV2SeverityTextColumn],
|
||||
logsV2Columns[LogsV2TraceIDColumn],
|
||||
logsV2Columns[LogsV2SpanIDColumn],
|
||||
}
|
||||
case telemetrytypes.FieldContextBody:
|
||||
if useJSONBody {
|
||||
return []*schema.Column{logsV2Columns[LogsV2BodyV2Column]}
|
||||
}
|
||||
return []*schema.Column{logsV2Columns[LogsV2BodyColumn]}
|
||||
case telemetrytypes.FieldContextAttribute:
|
||||
return []*schema.Column{
|
||||
logsV2Columns[LogsV2AttributesStringColumn],
|
||||
logsV2Columns[LogsV2AttributesNumberColumn],
|
||||
logsV2Columns[LogsV2AttributesBoolColumn],
|
||||
}
|
||||
case telemetrytypes.FieldContextResource:
|
||||
return []*schema.Column{
|
||||
logsV2Columns[LogsV2ResourcesStringColumn],
|
||||
}
|
||||
default:
|
||||
columns := searchColumns(telemetrytypes.FieldContextLog, useJSONBody)
|
||||
columns = append(columns, searchColumns(telemetrytypes.FieldContextBody, useJSONBody)...)
|
||||
columns = append(columns, searchColumns(telemetrytypes.FieldContextAttribute, useJSONBody)...)
|
||||
columns = append(columns, searchColumns(telemetrytypes.FieldContextResource, useJSONBody)...)
|
||||
return columns
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,288 +0,0 @@
|
||||
package logstelemetryschema
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/SigNoz/signoz/pkg/flagger"
|
||||
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
|
||||
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
|
||||
"github.com/SigNoz/signoz/pkg/querybuilder"
|
||||
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
|
||||
"github.com/huandu/go-sqlbuilder"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// searchFanOut returns the WHERE fragment search() fans out to. bodyExpr is the
|
||||
// body match expression, which differs between the legacy string body and the
|
||||
// body_v2 JSON column.
|
||||
func searchFanOut(bodyExpr string) string {
|
||||
return "(match(LOWER(severity_text), LOWER(?)) OR match(LOWER(trace_id), LOWER(?)) OR match(LOWER(span_id), LOWER(?)) OR " +
|
||||
bodyExpr + " OR " +
|
||||
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(attributes_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(attributes_string))) OR " +
|
||||
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(attributes_number)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), arrayMap(x -> toString(x), mapValues(attributes_number)))) OR " +
|
||||
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(attributes_bool)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), arrayMap(x -> toString(x), mapValues(attributes_bool)))) OR " +
|
||||
"(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(resources_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(resources_string))))"
|
||||
}
|
||||
|
||||
// searchArgs returns v repeated once per bound parameter search() emits — one per
|
||||
// searchable column expression (currently 12).
|
||||
func searchArgs(v any) []any {
|
||||
const searchColumnParams = 12
|
||||
args := make([]any, searchColumnParams)
|
||||
for i := range args {
|
||||
args[i] = v
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
// TestFilterExprSearch covers the search('needle') function, which fans out
|
||||
// across every searchable column via FilterOperatorSearch.
|
||||
func TestFilterExprSearch(t *testing.T) {
|
||||
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
|
||||
inWindowStart := uint64(releaseTime.Add(-5 * time.Minute).UnixNano())
|
||||
inWindowEnd := uint64(releaseTime.Add(5 * time.Minute).UnixNano())
|
||||
|
||||
legacyBody := "match(LOWER(body), LOWER(?))"
|
||||
jsonBody := "match(LOWER(toString(body_v2)), LOWER(?))"
|
||||
|
||||
// Single-context scope fragments (the fan-out narrowed to one context).
|
||||
logScope := "(match(LOWER(severity_text), LOWER(?)) OR match(LOWER(trace_id), LOWER(?)) OR match(LOWER(span_id), LOWER(?)))"
|
||||
resourceScope := "(arrayExists(x -> match(LOWER(x), LOWER(?)), mapKeys(resources_string)) OR arrayExists(x -> match(LOWER(x), LOWER(?)), mapValues(resources_string)))"
|
||||
|
||||
serviceNameEq := "(multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) = ? " +
|
||||
"AND multiIf(resource.`service.name` IS NOT NULL, resource.`service.name`::String, mapContains(resources_string, 'service.name'), resources_string['service.name'], NULL) IS NOT NULL)"
|
||||
|
||||
testCases := []struct {
|
||||
name string
|
||||
query string
|
||||
jsonBodyEnabled bool
|
||||
fullTextColumn *telemetrytypes.TelemetryFieldKey
|
||||
startNs uint64
|
||||
endNs uint64
|
||||
shouldPass bool
|
||||
expectedQuery string
|
||||
expectedArgs []any
|
||||
expectWarning bool
|
||||
expectedErrorContains string
|
||||
}{
|
||||
{
|
||||
name: "quoted, legacy body",
|
||||
query: "search('error')",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE " + searchFanOut(legacyBody),
|
||||
expectedArgs: searchArgs("error"),
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "quoted, json body",
|
||||
query: "search('error')",
|
||||
jsonBodyEnabled: true,
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE " + searchFanOut(jsonBody),
|
||||
expectedArgs: searchArgs("error"),
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "bare word",
|
||||
query: "search(timeout)",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE " + searchFanOut(legacyBody),
|
||||
expectedArgs: searchArgs("timeout"),
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "negated",
|
||||
query: "NOT search('error')",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE NOT (" + searchFanOut(legacyBody) + ")",
|
||||
expectedArgs: searchArgs("error"),
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "combined with field filter",
|
||||
query: "search('error') AND service.name=\"api\"",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE (" + searchFanOut(legacyBody) + " AND " + serviceNameEq + ")",
|
||||
expectedArgs: append(searchArgs("error"), "api"),
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
// A wide window is allowed at build time; scan cost is bounded by the
|
||||
// querier's EXPLAIN ESTIMATE gate, not a window cap in the builder.
|
||||
name: "wide window builds (estimate gate lives in querier)",
|
||||
query: "search('error')",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: uint64(releaseTime.Add(-10 * time.Hour).UnixNano()),
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE " + searchFanOut(legacyBody),
|
||||
expectedArgs: searchArgs("error"),
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
// search() is keyless and independent of fullTextColumn (which only
|
||||
// governs bare/quoted free text). It must work even when unset.
|
||||
name: "independent of full text column",
|
||||
query: "search('error')",
|
||||
fullTextColumn: nil,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE " + searchFanOut(legacyBody),
|
||||
expectedArgs: searchArgs("error"),
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
// A context-prefixed bare word must be used as the literal needle, not
|
||||
// normalized into a field key (Normalize would strip "resource.").
|
||||
name: "bare word with context prefix is not normalized",
|
||||
query: "search(resource.deployment)",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE " + searchFanOut(legacyBody),
|
||||
expectedArgs: searchArgs("resource\\.deployment"),
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
// A numeric needle must be the literal digits, not the %v rendering of a
|
||||
// parsed float64 (which would make search(1000000) scan for "1e+06").
|
||||
name: "numeric needle is not scientific notation",
|
||||
query: "search(1000000)",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE " + searchFanOut(legacyBody),
|
||||
expectedArgs: searchArgs("1000000"),
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "scoped to body, legacy",
|
||||
query: "search('error', body)",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE (" + legacyBody + ")",
|
||||
expectedArgs: []any{"error"},
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "scoped to body, json",
|
||||
query: "search('error', body)",
|
||||
jsonBodyEnabled: true,
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE (" + jsonBody + ")",
|
||||
expectedArgs: []any{"error"},
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "scoped to resource (quoted scope)",
|
||||
query: "search('error', 'resource')",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE (" + resourceScope + ")",
|
||||
expectedArgs: []any{"error", "error"},
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "scoped to log fields",
|
||||
query: "search('error', log)",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE " + logScope,
|
||||
expectedArgs: []any{"error", "error", "error"},
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "scoped to multiple contexts",
|
||||
query: "search('error', body, resource)",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: true,
|
||||
expectedQuery: "WHERE ((" + legacyBody + ") OR (" + resourceScope + "))",
|
||||
expectedArgs: []any{"error", "error", "error"},
|
||||
expectWarning: true,
|
||||
},
|
||||
{
|
||||
name: "invalid scope",
|
||||
query: "search('error', 'timeout')",
|
||||
fullTextColumn: DefaultFullTextColumn,
|
||||
startNs: inWindowStart,
|
||||
endNs: inWindowEnd,
|
||||
shouldPass: false,
|
||||
expectedErrorContains: "invalid search scope",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range testCases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
fl := flaggertest.WithBooleanFlags(t, map[string]bool{
|
||||
flagger.FeatureUseJSONBody.String(): tc.jsonBodyEnabled,
|
||||
})
|
||||
fm := NewFieldMapper(fl)
|
||||
cb := NewConditionBuilder(fm, fl)
|
||||
keys := BuildCompleteFieldKeyMap(releaseTime)
|
||||
|
||||
opts := querybuilder.FilterExprVisitorOpts{
|
||||
Context: context.Background(),
|
||||
Logger: instrumentationtest.New().Logger(),
|
||||
FieldMapper: fm,
|
||||
ConditionBuilder: cb,
|
||||
FieldKeys: keys,
|
||||
FullTextColumn: tc.fullTextColumn,
|
||||
StartNs: tc.startNs,
|
||||
EndNs: tc.endNs,
|
||||
}
|
||||
|
||||
clause, err := querybuilder.PrepareWhereClause(tc.query, opts)
|
||||
|
||||
if !tc.shouldPass {
|
||||
require.Error(t, err)
|
||||
require.True(t, detailContains(err, tc.expectedErrorContains),
|
||||
"error %v should contain %q", err, tc.expectedErrorContains)
|
||||
return
|
||||
}
|
||||
|
||||
require.NoError(t, err)
|
||||
require.False(t, clause.IsEmpty())
|
||||
|
||||
sql, args := clause.WhereClause.BuildWithFlavor(sqlbuilder.ClickHouse)
|
||||
require.Equal(t, tc.expectedQuery, sql)
|
||||
require.Equal(t, tc.expectedArgs, args)
|
||||
|
||||
if tc.expectWarning {
|
||||
// The visitor only flags the cost guard; the statement builder
|
||||
// materializes the advisory + budget from config downstream.
|
||||
require.True(t, clause.RequiresCostGuard)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -157,7 +157,7 @@ func (c *conditionBuilder) ConditionFor(
|
||||
sb *sqlbuilder.SelectBuilder,
|
||||
) ([]string, []string, error) {
|
||||
|
||||
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for metrics.
|
||||
// has/hasAny/hasAll/hasToken are logs-body-only; reject for metrics.
|
||||
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
@@ -205,7 +205,7 @@ func (c *conditionBuilder) ConditionFor(
|
||||
sb *sqlbuilder.SelectBuilder,
|
||||
) ([]string, []string, error) {
|
||||
|
||||
// has/hasAny/hasAll/hasToken/search are logs-only functions; reject for traces.
|
||||
// has/hasAny/hasAll/hasToken are logs-body-only; reject for traces.
|
||||
if err := querybuilder.NewFunctionUnsupportedError(operator); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
@@ -44,6 +44,8 @@ var (
|
||||
GCPServiceCloudSQLPostgres = ServiceID{valuer.NewString("cloudsql_postgres")}
|
||||
GCPServiceMemorystoreRedis = ServiceID{valuer.NewString("memorystore_redis")}
|
||||
GCPServiceComputeEngine = ServiceID{valuer.NewString("computeengine")}
|
||||
GCPServiceGKE = ServiceID{valuer.NewString("gke")}
|
||||
GCPServiceCloudStorage = ServiceID{valuer.NewString("cloudstorage")}
|
||||
)
|
||||
|
||||
func (ServiceID) Enum() []any {
|
||||
@@ -78,6 +80,8 @@ func (ServiceID) Enum() []any {
|
||||
GCPServiceCloudSQLPostgres,
|
||||
GCPServiceMemorystoreRedis,
|
||||
GCPServiceComputeEngine,
|
||||
GCPServiceGKE,
|
||||
GCPServiceCloudStorage,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -118,6 +122,8 @@ var SupportedServices = map[CloudProviderType][]ServiceID{
|
||||
GCPServiceCloudSQLPostgres,
|
||||
GCPServiceMemorystoreRedis,
|
||||
GCPServiceComputeEngine,
|
||||
GCPServiceGKE,
|
||||
GCPServiceCloudStorage,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -118,10 +118,6 @@ const (
|
||||
FilterOperatorHasToken
|
||||
FilterOperatorHasAny
|
||||
FilterOperatorHasAll
|
||||
|
||||
// FilterOperatorSearch backs search('needle'): a keyless search the condition
|
||||
// builder fans out across every searchable column.
|
||||
FilterOperatorSearch
|
||||
)
|
||||
|
||||
var operatorInverseMapping = map[FilterOperator]FilterOperator{
|
||||
@@ -190,8 +186,7 @@ func (f FilterOperator) IsNegativeOperator() bool {
|
||||
FilterOperatorIn,
|
||||
FilterOperatorExists,
|
||||
FilterOperatorRegexp,
|
||||
FilterOperatorContains,
|
||||
FilterOperatorSearch:
|
||||
FilterOperatorContains:
|
||||
return false
|
||||
}
|
||||
return true
|
||||
@@ -248,11 +243,10 @@ func (f FilterOperator) IsArrayFunctionOperator() bool {
|
||||
}
|
||||
|
||||
// IsFunctionOperator reports whether the operator is a query function
|
||||
// (has/hasAny/hasAll/hasToken/search). These are logs-only; other signals reject
|
||||
// them and the resource-fingerprint builder skips them.
|
||||
// (has/hasAny/hasAll/hasToken); these apply to the logs body column only.
|
||||
func (f FilterOperator) IsFunctionOperator() bool {
|
||||
switch f {
|
||||
case FilterOperatorHas, FilterOperatorHasAny, FilterOperatorHasAll, FilterOperatorHasToken, FilterOperatorSearch:
|
||||
case FilterOperatorHas, FilterOperatorHasAny, FilterOperatorHasAll, FilterOperatorHasToken:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
@@ -271,8 +265,6 @@ func (f FilterOperator) FunctionName() string {
|
||||
return "hasAll"
|
||||
case FilterOperatorHasToken:
|
||||
return "hasToken"
|
||||
case FilterOperatorSearch:
|
||||
return "search"
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
|
||||
@@ -57,12 +57,6 @@ type Statement struct {
|
||||
Args []any
|
||||
Warnings []string
|
||||
WarningsDocURL string
|
||||
CostGuard *CostGuard
|
||||
}
|
||||
|
||||
type CostGuard struct {
|
||||
Warning string
|
||||
MaxScanRows int64
|
||||
}
|
||||
|
||||
// StatementBuilder builds the query.
|
||||
|
||||
@@ -79,14 +79,6 @@ var (
|
||||
}
|
||||
)
|
||||
|
||||
// FieldContextFromText resolves a context word (e.g. "body", "resource") to its
|
||||
// FieldContext, applying the same aliases as key parsing (e.g. "tag" -> attribute). ok
|
||||
// is false for an unknown word, so callers can reject it instead of getting unspecified.
|
||||
func FieldContextFromText(text string) (FieldContext, bool) {
|
||||
fc, ok := fieldContexts[strings.ToLower(strings.TrimSpace(text))]
|
||||
return fc, ok
|
||||
}
|
||||
|
||||
// UnmarshalJSON implements the json.Unmarshaler interface.
|
||||
func (f *FieldContext) UnmarshalJSON(data []byte) error {
|
||||
var str string
|
||||
|
||||
@@ -412,16 +412,10 @@ func (v *variableReplacementVisitor) VisitFunctionCall(ctx *grammar.FunctionCall
|
||||
}
|
||||
|
||||
func (v *variableReplacementVisitor) VisitSearchCall(ctx *grammar.SearchCallContext) any {
|
||||
if ctx.ValueList() == nil {
|
||||
if ctx.FunctionParamList() == nil {
|
||||
return "search()"
|
||||
}
|
||||
// VisitValueList already wraps the args in parens and returns the skip
|
||||
// marker if any arg resolves to __all__.
|
||||
result := v.Visit(ctx.ValueList()).(string)
|
||||
if result == specialSkipMarker {
|
||||
return specialSkipMarker
|
||||
}
|
||||
return "search" + result
|
||||
return "search(" + v.Visit(ctx.FunctionParamList()).(string) + ")"
|
||||
}
|
||||
|
||||
func (v *variableReplacementVisitor) VisitFunctionParamList(ctx *grammar.FunctionParamListContext) any {
|
||||
|
||||
@@ -1,124 +0,0 @@
|
||||
import json
|
||||
from collections import namedtuple
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from http import HTTPStatus
|
||||
|
||||
import pytest
|
||||
|
||||
from fixtures import types
|
||||
from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
|
||||
from fixtures.logs import Logs
|
||||
from fixtures.querier import build_order_by, build_raw_query, get_rows, make_query_request
|
||||
|
||||
# search() with use_json_body on (see conftest.py): body matches run against body_v2 via
|
||||
# LOWER(toString(body_v2)); the map/log fan-out is unchanged from querierlogs/15_search.
|
||||
# The response `body` comes back as parsed JSON, so a plain-string body is {"message": <body>}.
|
||||
|
||||
Bodies = namedtuple("Bodies", ["a", "b", "c", "d"])
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"expression,expected",
|
||||
[
|
||||
# body matches now run through body_v2 (toString of {"message": <body>})
|
||||
pytest.param("search('login')", lambda b: {b.a}, id="keyless_body"),
|
||||
pytest.param("search('checkout')", lambda b: {b.a, b.d}, id="keyless_body_and_resource"),
|
||||
pytest.param("search('CHECKOUT')", lambda b: {b.a, b.d}, id="keyless_case_insensitive"),
|
||||
pytest.param("search('login', body)", lambda b: {b.a}, id="scope_body"),
|
||||
pytest.param("search('checkout', body)", lambda b: {b.a}, id="scope_body_excludes_resource"),
|
||||
# the map / log fan-out is flag-independent — sanity that it still works here
|
||||
pytest.param("search('checkout', resource)", lambda b: {b.a, b.d}, id="scope_resource"),
|
||||
pytest.param("search('acme', attribute)", lambda b: {b.a, b.c}, id="scope_attribute"),
|
||||
pytest.param("search('error', log)", lambda b: {b.b, b.d}, id="scope_log_severity"),
|
||||
pytest.param("search('checkout', body, resource)", lambda b: {b.a, b.d}, id="scopes_union"),
|
||||
pytest.param("NOT search('login')", lambda b: {b.b, b.c, b.d}, id="negated"),
|
||||
],
|
||||
)
|
||||
def test_search(
|
||||
signoz: types.SigNoz,
|
||||
create_user_admin: None, # pylint: disable=unused-argument
|
||||
get_token: Callable[[str, str], str],
|
||||
insert_logs: Callable[[list[Logs]], None],
|
||||
expression: str,
|
||||
expected: Callable[[Bodies], set[str]],
|
||||
) -> None:
|
||||
"""Four self-naming logs assert keyless/body-scoped search() matches through the
|
||||
body_v2 JSON column, while resource/attribute/log scopes fan out unchanged."""
|
||||
body = Bodies(
|
||||
a="alpha checkout login ok", # service checkout / region useast / tenant acme / INFO
|
||||
b="bravo declined", # service payment / region euwest / tenant globex / ERROR
|
||||
c="charlie miss", # service cart / region useast / tenant acme / WARN
|
||||
d="delta slow", # service checkout / region apac / tenant initech / ERROR
|
||||
)
|
||||
# (body, resources, attributes, severity_text)
|
||||
specs = [
|
||||
(body.a, {"service.name": "checkout", "region": "useast"}, {"tenant": "acme"}, "INFO"),
|
||||
(body.b, {"service.name": "payment", "region": "euwest"}, {"tenant": "globex"}, "ERROR"),
|
||||
(body.c, {"service.name": "cart", "region": "useast"}, {"tenant": "acme"}, "WARN"),
|
||||
(body.d, {"service.name": "checkout", "region": "apac"}, {"tenant": "initech"}, "ERROR"),
|
||||
]
|
||||
|
||||
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
|
||||
logs = [Logs(timestamp=now - timedelta(seconds=i + 1), resources=res, attributes=attrs, body=b, severity_text=sev) for i, (b, res, attrs, sev) in enumerate(specs)]
|
||||
insert_logs(logs)
|
||||
|
||||
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
response = make_query_request(
|
||||
signoz,
|
||||
token,
|
||||
start_ms=int((now - timedelta(minutes=5)).timestamp() * 1000),
|
||||
end_ms=int(now.timestamp() * 1000),
|
||||
request_type="raw",
|
||||
queries=[
|
||||
build_raw_query(
|
||||
"A",
|
||||
"logs",
|
||||
filter_expression=expression,
|
||||
order=[build_order_by("timestamp", "desc"), build_order_by("id", "desc")],
|
||||
limit=100,
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
assert response.status_code == HTTPStatus.OK, response.text
|
||||
# body_v2 comes back parsed; a plain-string body is {"message": <body>}.
|
||||
assert {row["data"]["body"]["message"] for row in get_rows(response)} == expected(body)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"needle",
|
||||
[
|
||||
pytest.param("eve@acme.io", id="nested_string_value"),
|
||||
pytest.param("503", id="nested_numeric_value"),
|
||||
pytest.param("status", id="nested_key"),
|
||||
],
|
||||
)
|
||||
def test_search_body_reaches_nested_json(
|
||||
signoz: types.SigNoz,
|
||||
create_user_admin: None, # pylint: disable=unused-argument
|
||||
get_token: Callable[[str, str], str],
|
||||
insert_logs: Callable[[list[Logs]], None],
|
||||
needle: str,
|
||||
) -> None:
|
||||
"""A body-scoped search matches values and keys nested inside the body_v2 JSON."""
|
||||
# searchable content lives only in nested fields, not a top-level message
|
||||
nested_body = json.dumps({"user": {"email": "eve@acme.io"}, "http": {"status": 503}}, separators=(",", ":"))
|
||||
|
||||
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
|
||||
insert_logs([Logs(timestamp=now - timedelta(seconds=1), resources={"service.name": "api"}, body=nested_body, severity_text="INFO")])
|
||||
|
||||
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
response = make_query_request(
|
||||
signoz,
|
||||
token,
|
||||
start_ms=int((now - timedelta(minutes=5)).timestamp() * 1000),
|
||||
end_ms=int(now.timestamp() * 1000),
|
||||
request_type="raw",
|
||||
queries=[build_raw_query("A", "logs", filter_expression=f"search('{needle}', body)", order=[build_order_by("timestamp", "desc")], limit=100)],
|
||||
)
|
||||
|
||||
assert response.status_code == HTTPStatus.OK, response.text
|
||||
rows = get_rows(response)
|
||||
assert len(rows) == 1
|
||||
assert rows[0]["data"]["body"]["user"]["email"] == "eve@acme.io"
|
||||
@@ -31,8 +31,8 @@ def signoz_skip_resource_fingerprint(
|
||||
pytestconfig=pytestconfig,
|
||||
cache_key="signoz-skip-resource-fingerprint",
|
||||
env_overrides={
|
||||
"SIGNOZ_QUERIER_SKIP__RESOURCE__FINGERPRINT_ENABLED": True,
|
||||
"SIGNOZ_QUERIER_SKIP__RESOURCE__FINGERPRINT_THRESHOLD": 2,
|
||||
"SIGNOZ_STATEMENTBUILDER_SKIP__RESOURCE__FINGERPRINT_ENABLED": True,
|
||||
"SIGNOZ_STATEMENTBUILDER_SKIP__RESOURCE__FINGERPRINT_THRESHOLD": 2,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -66,6 +66,6 @@ def signoz_fingerprint(
|
||||
pytestconfig=pytestconfig,
|
||||
cache_key="signoz-skip-resource-fingerprint-baseline",
|
||||
env_overrides={
|
||||
"SIGNOZ_QUERIER_SKIP__RESOURCE__FINGERPRINT_ENABLED": False,
|
||||
"SIGNOZ_STATEMENTBUILDER_SKIP__RESOURCE__FINGERPRINT_ENABLED": False,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -1,212 +0,0 @@
|
||||
from collections import namedtuple
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from http import HTTPStatus
|
||||
|
||||
import pytest
|
||||
|
||||
from fixtures import types
|
||||
from fixtures.auth import USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD
|
||||
from fixtures.logs import Logs
|
||||
from fixtures.querier import build_order_by, build_raw_query, get_column_data_from_response, get_rows, make_query_request
|
||||
|
||||
# search(): keyless fans across every field; scoped search('needle', <ctx>...) narrows
|
||||
# to the named contexts (body/attribute/resource/log). Flag off here, so body matches the
|
||||
# `body` String column (querier_json_body mirrors this over body_v2). `expected` is a
|
||||
# lambda over the body markers so cases stay declarative while the dataset lives in one place.
|
||||
|
||||
Bodies = namedtuple("Bodies", ["a", "b", "c", "d"])
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"expression,expected",
|
||||
[
|
||||
# ── keyless: fans across every field ────────────────────────────────
|
||||
pytest.param("search('login')", lambda b: {b.a}, id="keyless_body"),
|
||||
pytest.param("search('checkout')", lambda b: {b.a, b.d}, id="keyless_body_and_resource"),
|
||||
pytest.param("search('useast')", lambda b: {b.a, b.c}, id="keyless_resource_value"),
|
||||
pytest.param("search('acme')", lambda b: {b.a, b.c}, id="keyless_attribute_value"),
|
||||
pytest.param("search('tenant')", lambda b: {b.a, b.b, b.c, b.d}, id="keyless_attribute_key"),
|
||||
pytest.param("search('error')", lambda b: {b.b, b.d}, id="keyless_severity_case_insensitive"),
|
||||
pytest.param("search('CHECKOUT')", lambda b: {b.a, b.d}, id="keyless_needle_case_insensitive"),
|
||||
# ── scoped: narrows to one context ──────────────────────────────────
|
||||
pytest.param("search('login', body)", lambda b: {b.a}, id="scope_body"),
|
||||
pytest.param("search('login', 'body')", lambda b: {b.a}, id="scope_body_quoted"),
|
||||
pytest.param("search('checkout', body)", lambda b: {b.a}, id="scope_body_excludes_resource"),
|
||||
pytest.param("search('checkout', resource)", lambda b: {b.a, b.d}, id="scope_resource"),
|
||||
pytest.param("search('acme', attribute)", lambda b: {b.a, b.c}, id="scope_attribute"),
|
||||
pytest.param("search('error', log)", lambda b: {b.b, b.d}, id="scope_log_severity"),
|
||||
pytest.param("search('acme', body)", lambda b: set(), id="scope_body_no_match"),
|
||||
pytest.param("search('checkout', attribute)", lambda b: set(), id="scope_attribute_no_match"),
|
||||
# ── multiple scopes: union of the named contexts ────────────────────
|
||||
pytest.param("search('login', body, resource)", lambda b: {b.a}, id="scopes_body_resource_body_only"),
|
||||
pytest.param("search('checkout', body, resource)", lambda b: {b.a, b.d}, id="scopes_body_resource_union"),
|
||||
# ── composition with boolean / field filters ────────────────────────
|
||||
pytest.param("NOT search('login')", lambda b: {b.b, b.c, b.d}, id="negated"),
|
||||
pytest.param("search('useast') AND severity_text = 'INFO'", lambda b: {b.a}, id="and_field_filter"),
|
||||
],
|
||||
)
|
||||
def test_search(
|
||||
signoz: types.SigNoz,
|
||||
create_user_admin: None, # pylint: disable=unused-argument
|
||||
get_token: Callable[[str, str], str],
|
||||
insert_logs: Callable[[list[Logs]], None],
|
||||
expression: str,
|
||||
expected: Callable[[Bodies], set[str]],
|
||||
) -> None:
|
||||
"""Four self-naming logs, each with a token planted in a distinct place (body,
|
||||
resource, attribute, severity), assert search() reaches exactly the right ones."""
|
||||
body = Bodies(
|
||||
a="alpha checkout login ok", # service checkout / region useast / tenant acme / INFO
|
||||
b="bravo declined", # service payment / region euwest / tenant globex / ERROR
|
||||
c="charlie miss", # service cart / region useast / tenant acme / WARN
|
||||
d="delta slow", # service checkout / region apac / tenant initech / ERROR
|
||||
)
|
||||
# (body, resources, attributes, severity_text)
|
||||
specs = [
|
||||
(body.a, {"service.name": "checkout", "region": "useast"}, {"tenant": "acme"}, "INFO"),
|
||||
(body.b, {"service.name": "payment", "region": "euwest"}, {"tenant": "globex"}, "ERROR"),
|
||||
(body.c, {"service.name": "cart", "region": "useast"}, {"tenant": "acme"}, "WARN"),
|
||||
(body.d, {"service.name": "checkout", "region": "apac"}, {"tenant": "initech"}, "ERROR"),
|
||||
]
|
||||
|
||||
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
|
||||
logs = [Logs(timestamp=now - timedelta(seconds=i + 1), resources=res, attributes=attrs, body=b, severity_text=sev) for i, (b, res, attrs, sev) in enumerate(specs)]
|
||||
insert_logs(logs)
|
||||
|
||||
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
response = make_query_request(
|
||||
signoz,
|
||||
token,
|
||||
start_ms=int((now - timedelta(minutes=5)).timestamp() * 1000),
|
||||
end_ms=int(now.timestamp() * 1000),
|
||||
request_type="raw",
|
||||
queries=[
|
||||
build_raw_query(
|
||||
"A",
|
||||
"logs",
|
||||
filter_expression=expression,
|
||||
order=[build_order_by("timestamp", "desc"), build_order_by("id", "desc")],
|
||||
limit=100,
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
assert response.status_code == HTTPStatus.OK, response.text
|
||||
assert response.json()["status"] == "success"
|
||||
assert set(get_column_data_from_response(response.json(), "body")) == expected(body)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"expression",
|
||||
[
|
||||
pytest.param("search('login', bogus)", id="unknown_scope_word"),
|
||||
pytest.param("search('login', body.message)", id="qualified_field_not_a_scope"),
|
||||
],
|
||||
)
|
||||
def test_search_invalid_scope_rejected(
|
||||
signoz: types.SigNoz,
|
||||
create_user_admin: None, # pylint: disable=unused-argument
|
||||
get_token: Callable[[str, str], str],
|
||||
expression: str,
|
||||
) -> None:
|
||||
"""A scope that is not a field context (an unknown word, or a qualified
|
||||
`context.field`) is rejected at build time with a 400."""
|
||||
now = datetime.now(tz=UTC)
|
||||
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
response = make_query_request(
|
||||
signoz,
|
||||
token,
|
||||
start_ms=int((now - timedelta(minutes=5)).timestamp() * 1000),
|
||||
end_ms=int(now.timestamp() * 1000),
|
||||
request_type="raw",
|
||||
queries=[build_raw_query("A", "logs", filter_expression=expression, limit=100)],
|
||||
)
|
||||
assert response.status_code == HTTPStatus.BAD_REQUEST, response.text
|
||||
assert "invalid search scope" in response.text
|
||||
|
||||
|
||||
# search() is scan-heavy, so the querier gates it on EXPLAIN ESTIMATE against
|
||||
# search_max_scan_rows (50000 for this instance, see conftest.py). A broad search over a
|
||||
# busy range is rejected; the two ways the advisory suggests to recover — add a more
|
||||
# selective filter, or narrow the time range — both bring the scan under budget.
|
||||
|
||||
|
||||
def test_search_cost_guard_trips_then_passes_with_filter(
|
||||
signoz_search_scan_budget: types.SigNoz,
|
||||
create_user_admin: None, # pylint: disable=unused-argument
|
||||
get_token: Callable[[str, str], str],
|
||||
insert_logs: Callable[[list[Logs]], None],
|
||||
) -> None:
|
||||
"""A broad search() over ~61000 logs exceeds the 50000-row budget and is rejected;
|
||||
the same search narrowed to the 1000-log 'checkout' service scans under budget and
|
||||
succeeds — the advisory's "add a more selective filter" made real."""
|
||||
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
|
||||
# 60000 'catalog' logs in the current bucket + 1000 'checkout' logs ~45m back (an
|
||||
# earlier ts_bucket bucket), so the checkout fingerprint occupies its own marks and a
|
||||
# resource filter on it prunes the scan.
|
||||
logs = [Logs(timestamp=now - timedelta(seconds=1 + i % 30), resources={"service.name": "catalog"}, body="log line") for i in range(60000)]
|
||||
logs += [Logs(timestamp=now - timedelta(minutes=45, seconds=i % 30), resources={"service.name": "checkout"}, body="log line") for i in range(1000)]
|
||||
insert_logs(logs)
|
||||
|
||||
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
start_ms = int((now - timedelta(minutes=60)).timestamp() * 1000)
|
||||
end_ms = int((now + timedelta(minutes=1)).timestamp() * 1000)
|
||||
|
||||
def run(expression: str):
|
||||
return make_query_request(
|
||||
signoz_search_scan_budget,
|
||||
token,
|
||||
start_ms=start_ms,
|
||||
end_ms=end_ms,
|
||||
request_type="raw",
|
||||
queries=[build_raw_query("A", "logs", filter_expression=expression, order=[build_order_by("timestamp", "desc")], limit=100)],
|
||||
)
|
||||
|
||||
# Broad search over the whole range is over budget -> rejected before executing.
|
||||
over_budget = run("search('log')")
|
||||
assert over_budget.status_code == HTTPStatus.BAD_REQUEST, over_budget.text
|
||||
assert "over the limit" in over_budget.text
|
||||
|
||||
# Adding a selective resource filter prunes the scan under budget -> runs.
|
||||
within_budget = run("search('log') AND resource.service.name = 'checkout'")
|
||||
assert within_budget.status_code == HTTPStatus.OK, within_budget.text
|
||||
assert len(get_rows(within_budget)) > 0
|
||||
|
||||
|
||||
def test_search_cost_guard_passes_with_narrower_time_range(
|
||||
signoz_search_scan_budget: types.SigNoz,
|
||||
create_user_admin: None, # pylint: disable=unused-argument
|
||||
get_token: Callable[[str, str], str],
|
||||
insert_logs: Callable[[list[Logs]], None],
|
||||
) -> None:
|
||||
"""A steady stream of ~80000 logs (~4/sec over the last ~5.5h). A search() over the
|
||||
whole window is over the 50000-row budget and rejected; the same search over the last
|
||||
15 minutes scans only a few thousand rows and runs — the advisory's "narrow the time
|
||||
range" made real."""
|
||||
now = datetime.now(tz=UTC).replace(second=0, microsecond=0)
|
||||
# ~4 logs/sec, oldest ~5.5h back, newest ~now.
|
||||
logs = [Logs(timestamp=now - timedelta(seconds=i // 4), resources={"service.name": "app"}, body="log line") for i in range(80000)]
|
||||
insert_logs(logs)
|
||||
|
||||
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
|
||||
def run(lookback_minutes: int):
|
||||
return make_query_request(
|
||||
signoz_search_scan_budget,
|
||||
token,
|
||||
start_ms=int((now - timedelta(minutes=lookback_minutes)).timestamp() * 1000),
|
||||
end_ms=int((now + timedelta(minutes=1)).timestamp() * 1000),
|
||||
request_type="raw",
|
||||
queries=[build_raw_query("A", "logs", filter_expression="search('log')", order=[build_order_by("timestamp", "desc")], limit=100)],
|
||||
)
|
||||
|
||||
# The last 6 hours cover the whole stream (~80000) -> over budget -> rejected.
|
||||
wide = run(360)
|
||||
assert wide.status_code == HTTPStatus.BAD_REQUEST, wide.text
|
||||
assert "over the limit" in wide.text
|
||||
|
||||
# The last 15 minutes hold only ~3600 logs -> under budget -> runs, returning rows.
|
||||
narrow = run(15)
|
||||
assert narrow.status_code == HTTPStatus.OK, narrow.text
|
||||
assert len(get_rows(narrow)) > 0
|
||||
@@ -1,35 +0,0 @@
|
||||
import pytest
|
||||
from testcontainers.core.container import Network
|
||||
|
||||
from fixtures import types
|
||||
from fixtures.signoz import create_signoz
|
||||
|
||||
|
||||
@pytest.fixture(name="signoz_search_scan_budget", scope="package")
|
||||
def signoz_search_scan_budget(
|
||||
network: Network,
|
||||
migrator: types.Operation, # pylint: disable=unused-argument
|
||||
zeus: types.TestContainerDocker,
|
||||
gateway: types.TestContainerDocker,
|
||||
sqlstore: types.TestContainerSQL,
|
||||
clickhouse: types.TestContainerClickhouse,
|
||||
request: pytest.FixtureRequest,
|
||||
pytestconfig: pytest.Config,
|
||||
) -> types.SigNoz:
|
||||
"""SigNoz with a conservative search_max_scan_rows (50000) so a broad search() over a
|
||||
busy range trips the cost guard, while a more selective one (by service or by a
|
||||
narrower time range) stays under it. Shares the default instance's sqlstore +
|
||||
clickhouse, so the same admin token and seeded logs work against it."""
|
||||
return create_signoz(
|
||||
network=network,
|
||||
zeus=zeus,
|
||||
gateway=gateway,
|
||||
sqlstore=sqlstore,
|
||||
clickhouse=clickhouse,
|
||||
request=request,
|
||||
pytestconfig=pytestconfig,
|
||||
cache_key="signoz-search-scan-budget-50k",
|
||||
env_overrides={
|
||||
"SIGNOZ_QUERIER_SEARCH__MAX__SCAN__ROWS": 50000,
|
||||
},
|
||||
)
|
||||