Compare commits

..

1 Commits

Author SHA1 Message Date
vikrantgupta25
64f9d94e3e chore(claude): add authz skill
Claude Code skill automating the authz contributing workflow: kind/resource
registration, managed-role grants, generated-artifact sync, route wiring with
resource defs, tuple + managed-role JSON backfill migrations, and the authz
integration test patterns.
2026-07-28 19:58:28 +05:30
133 changed files with 1644 additions and 1919 deletions

View File

@@ -0,0 +1,142 @@
---
name: authz
description: Add authorization (authz) to a SigNoz backend feature end-to-end — register kinds/resources, grant managed-role permissions, wire routes with ResourceDefs, backfill tuples for existing orgs, and write the authz integration tests. Use this whenever the user mentions authz, authorization, permissions, access control, managed roles (admin/editor/viewer), OpenFGA, ResourceDef, CheckResources, 403/forbidden behavior on an endpoint, wiring or protecting a new route, registering a new resource/kind, or writing integration tests that assert role-based allow/deny. Also use it when a new API endpoint is being added and its access control hasn't been decided yet — every route needs an authz decision even if the user didn't say the word.
---
# SigNoz Authz Workflow
This skill automates the procedure documented in `docs/contributing/go/authz.md`. Read that doc first — it is the canonical explanation of the concepts (types, verbs, kinds, resources, selectors, tuples, the community/enterprise split). This skill tells you what to do; the doc tells you why it works.
The core model in one paragraph: permissions are never attached to users. They are attached to **roles** as OpenFGA tuples, and users/service accounts are made **assignees** of roles. You declare everything in code registries under `pkg/types/coretypes/`; tuples for new organizations are derived from those registries at org bootstrap. Routes declare what resource+verb they touch via `ResourceDef`s, and middleware turns that into a check that works in both community (role gate) and enterprise (per-resource FGA) editions.
## Before you start
Figure out what shape the work is. Ask the user if unclear:
- **New resource** (new kind + routes) → all steps below.
- **New route on an existing resource** → step 4 only (maybe step 3 if roles need new verbs).
- **Changing what a role can do** → steps 3, 5, 6, 7.
- **Just tests** → steps 6, 7.
Almost every feature resource is a `metaresource` (dashboards, rules, pipelines) or a `telemetryresource` (logs, traces, metrics). If the feature genuinely needs a brand-new FGA **type**, stop and flag it to the user — that requires editing both `pkg/authz/openfgaschema/base.fga` and `ee/authz/openfgaschema/base.fga` together and is almost never the right call. A new **kind** never touches the schema.
## Step 1: Register the kind
Edit `pkg/types/coretypes/registry_kind.go`. Add a var and append it to the `Kinds` slice (both are required — the slice is what gets validated and iterated):
```go
KindThing = MustNewKind("thing")
```
Kind names are lowercase, hyphen-separated (`quick-filter`, `trace-funnel`, `notification-channel`).
Also add the kind to `Kind.Enum()` in `pkg/types/coretypes/kind.go` — a second hand-maintained list mirroring `Kinds` that feeds the OpenAPI enum. It is easy to miss because nothing fails at build time without it; the drift only shows up in the generated spec.
## Step 2: Register the resource
Edit `pkg/types/coretypes/registry_resource.go`. Add a var and append it to the `Resources` slice:
```go
ResourceMetaResourceThing = NewResourceMetaResource(KindThing)
```
Use `NewResourceTelemetryResource(KindThing)` for telemetry data resources. Pass an explicit verb list only when the resource supports fewer verbs than the type default — see `ResourceMetaResourceFactorAPIKey` in that file for the shape.
## Step 3: Grant permissions to managed roles
Edit `pkg/types/coretypes/registry_managed_role.go`. Add `Transaction` entries to `ManagedRoleToTransactions` for each role that may act on the resource:
```go
{Verb: VerbCreate, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindThing}, WildCardSelectorString)},
```
The convention: **admin** gets everything; **editor** gets full CRUD only on day-to-day observability resources (dashboards, rules, saved views — not access control or org settings); **viewer** gets `read`/`list`; **anonymous** gets nothing (public dashboards are the only exception). Match the file's existing style, including its one-line grouping comment per resource block (e.g. `// thing — editors manage, viewers read`) — that file uses grouping comments as structure; do not add comments anywhere else.
This map is the single source of truth: changing it affects only organizations created afterwards (see step 5 for existing orgs), and the integration suite asserts its contents via a golden file (see step 6).
## Step 4: Wire the route
Register routes in the existing domain file under `pkg/apiserver/signozapiserver/` — do not create new per-sub-resource route files. Each route needs three coordinated pieces:
1. **`provider.authzMiddleware.CheckResources(handlerFn, roles...)`** — the role list is the community-edition fallback gate (which managed roles may call this route when per-resource checks are unavailable).
2. **`handler.WithResourceDefs(...)`** — declares resource, verb, audit category, ID extractor, and selector.
3. **`SecuritySchemes: newScopedSecuritySchemes([]string{resource.Scope(verb)})`** — advertises the scope in the OpenAPI spec.
Selector rules (getting these wrong silently over- or under-authorizes):
| Operation | Selector | ID extractor |
|---|---|---|
| create | `WildcardSelector` | `ResponseJSONPath("data.id")` (ID exists only after the handler runs; recorded for audit) |
| list | `WildcardSelector` | none |
| read/update/delete of one instance | `IDSelector` | `PathParam("id")` or `BodyJSONPath(...)` |
| request ID ≠ FGA selector (e.g. route has a role UUID, FGA objects use role names) | custom `SelectorFunc` | as appropriate |
Routes that link two resources check **both** sides. Read `references/route-wiring.md` before wiring any create route, linking route (attach/detach, parent-child), or custom selector — it has complete real examples from `serviceaccount.go`.
Prefer `CheckResources` + `ResourceDef` for anything resource-shaped. The legacy gates `ViewAccess`/`EditAccess`/`AdminAccess` are coarse role-only checks — do not use them for new routes. `OpenAccess` means no authorization (authentication still applies); `CheckWithoutClaims` serves anonymous routes.
## Step 5: Backfill existing organizations (only if needed)
New organizations derive their tuples from the registries at creation — no migration needed for them. If **existing** organizations must get the new permissions, add a migration in `pkg/sqlmigration/` that does two things per org:
1. **Insert the FGA tuples** — follow `083_add_role_crud_tuples.go`: one `tuple` row (+ `changelog` row) per role/verb/object, with `ON CONFLICT ... DO NOTHING` so it stays idempotent, handling both the Postgres and SQLite tuple column layouts as that file does.
2. **Refresh the managed roles' JSON record** — since `099_add_role_transaction_groups.go`, each role row carries a `transaction_groups` JSON column that mirrors its permissions. Tuples and JSON must move together or the stored record drifts from the registry. Follow 099's managed-role loop: marshal `authtypes.NewTransactionGroupsFromTransactions(coretypes.ManagedRoleToTransactions[roleName])` and `UPDATE role SET transaction_groups = ? WHERE org_id = ? AND type = 'managed' AND name = ?`. Serializing from the registry (which already contains your new grants) makes this step self-updating. Custom roles need no changes — only managed roles mirror the registry.
Register the new migration factory in `NewSQLMigrationProviderFactories` in `pkg/signoz/provider.go`, appended after the existing entries. The factory name passed to `factory.MustNewName` must match `^[a-z][a-z0-9_-]{0,30}$` — at most 31 characters, or it panics at startup; prefer a short name like `notification_template_tuples` over `add_notification_template_tuples`.
Ask the user whether backfill is needed rather than assuming — shipping a feature to existing orgs almost always means yes.
## Step 6: Update the golden testdata and regenerate CI-checked artifacts
If you changed `ManagedRoleToTransactions`, update `tests/integration/testdata/role/managed_role_transactions.json` to match — the `role/01_register.py` integration test compares each managed role's verb/type/kind matrix against this file and will fail otherwise.
If you touched the registries or `Kind.Enum()`, regenerate the derived artifacts. The `goci` GitHub workflow reruns each of these and fails the PR on any uncommitted diff:
```bash
go run cmd/enterprise/*.go generate openapi # docs/api/openapi.yml
go run cmd/enterprise/*.go generate authz # frontend/src/lib/authz/hooks/useAuthZ/permissions.type.ts
go run cmd/enterprise/*.go generate config transaction-groups # frontend/src/schemas/generated/transactionGroups.schema.json
```
Commit whatever they change; never hand-edit these generated files.
`generate authz` is allowlist-based: it only emits resources listed in the `allowedResources` map inside `runGenerateAuthz` in `cmd/authz.go`. For the new resource to surface in the UI's permission gating (AuthZButton and friends), add it there —
```go
coretypes.NewResourceRef(coretypes.ResourceMetaResourceThing).String(): true,
```
— then rerun `generate authz` and commit the regenerated `permissions.type.ts`. If the resource has no UI surface yet, skip the allowlist entry and the command produces no diff; that is expected.
## Step 7: Write integration tests
Read `references/integration-tests.md` for the suite layout, fixtures, and canonical allow/deny test shapes. The minimum bar for a new authz'd resource: each managed role's expected 200-vs-403 matrix on the new routes, and (if enterprise/FGA behavior matters) per-object scoping with a custom role — granted instance succeeds, non-granted instance gets 403, revoke flips it back to 403.
Python test conventions that are hard rules in this repo: inline `requests.*` calls directly (never wrap single API calls in helpers), always pass `timeout=`, and run `make py-fmt && make py-lint` before finishing.
## Verification
Run these before declaring the work done:
```bash
make go-build-community
```
Consistency greps — every one of these must hold:
- the new `Kind` var appears in BOTH the `Kinds` slice (`registry_kind.go`) and `Kind.Enum()` (`kind.go`)
- the three `generate` commands from step 6 have been run and leave `git diff` clean when re-run
- if the resource is UI-gated, it appears in `allowedResources` in `cmd/authz.go` and `permissions.type.ts` was regenerated
- the new `Resource` var appears in the `Resources` slice (`registry_resource.go`)
- every `ResourceDef` in the route file references the registered resource and one of its allowed verbs
- every route's `SecuritySchemes` scope matches its `ResourceDef` verb
- if `ManagedRoleToTransactions` changed, `managed_role_transactions.json` changed with it
Then run the relevant integration suites (`make py-test-setup` from the repo root boots the environment; pytest runs from `tests/`):
```bash
uv run pytest --basetemp=./tmp/ -vv --reuse integration/tests/role/
uv run pytest --basetemp=./tmp/ -vv --reuse integration/tests/<your-suite>/
```
General repo conventions apply throughout: no code comments (the grouping comments in `registry_managed_role.go` are the sole exception), complete descriptive names, and never commit without an explicit ask.

View File

@@ -0,0 +1,114 @@
# Authz Integration Tests Reference
Existing authz suites to imitate: `integration/tests/role/` (managed roles, custom-role CRUD, FGA scoping), `integration/tests/serviceaccount/` (API-key auth by role, role assignment), `integration/tests/passwordauthn/04_role.py` (user role changes, editor denied on admin endpoints).
## Where a new test goes
- Suite package: `tests/integration/tests/<suite>/` — pick the existing suite matching the feature's domain, or create a new package when the feature is its own domain.
- File name: two-digit ordered prefix, `NN_<topic>.py` (`03_fga.py`, `04_roles.py`). Files in a suite run in order; setup tests early in the file establish state that later tests use.
- Golden/testdata JSON: `tests/integration/testdata/<suite>/`, loaded via `fixtures.fs.get_testdata_file_path`.
- If the change touched `ManagedRoleToTransactions`, update `tests/integration/testdata/role/managed_role_transactions.json``role/01_register.py` asserts every managed role's verb/type/kind matrix against it.
## Environment and running
`make` targets run from the repo root; direct `uv run pytest` commands run from `tests/`:
```bash
make py-test-setup # repo root — boot the stack once (~4 min), keep it with --reuse
cd tests
uv run pytest --basetemp=./tmp/ -vv --reuse integration/tests/<suite>/
uv run pytest --basetemp=./tmp/ -vv --reuse integration/tests/role/03_fga.py::test_read_scoped_to_granted_role
make py-test-teardown # repo root — destroy when done; never mix --reuse and --teardown
make py-fmt && make py-lint # repo root — mandatory before finishing
```
## Fixtures you'll need
From `tests/fixtures/auth.py` (registered via `pytest_plugins` in `tests/conftest.py`):
- `signoz: types.SigNoz` — the running stack; build URLs with `signoz.self.host_configs["8080"].get("/api/v1/...")`.
- `create_user_admin` — package-scoped; registers the first admin (`USER_ADMIN_EMAIL` / `USER_ADMIN_PASSWORD`). Take it as an unused argument in every test that needs the org to exist.
- `get_token(email, password) -> str` — bearer token for a user.
- `add_license(signoz, make_http_mocks, get_token)` — custom roles are enterprise-only; call it in a `test_apply_license` at the top of any suite that creates custom roles.
- `create_active_user(signoz, admin_token, email=..., role="VIEWER", password=..., name=...) -> user_id` — invite + activate.
- `change_user_role(signoz, admin_token, user_id, old_role, new_role)` — e.g. `"signoz-viewer"` → your custom role name.
From `tests/fixtures/role.py`:
- `create_role(token, name, transaction_groups=None, description="") -> role_id` (fixture).
- `find_role_id(token, name) -> role_id` (fixture).
- `transaction_group(relation, type_name, kind_name, selectors)` — builds one permission entry, e.g. `transaction_group("read", "metaresource", "thing", ["*"])` or with a specific instance name/ID as selector.
From `tests/fixtures/serviceaccount.py`:
- `create_service_account_with_key(signoz, token, name, role="signoz-admin") -> (sa_id, api_key)`.
- Service accounts authenticate with the header `SIGNOZ-API-KEY: <key>`; users with `Authorization: Bearer <token>`.
Hard rule: inline `requests.get/post/put/delete` calls directly in tests — do not wrap single API calls in helper functions or fixtures. Always pass `timeout=5`. Include the response text in assertion messages so failures are diagnosable.
## Shape 1: managed-role matrix (community-safe)
For each role, assert the expected status on each new route. Pattern from `serviceaccount/03_auth.py`:
```python
def test_editor_forbidden_on_admin_endpoint(signoz, create_user_admin, get_token):
token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
_, api_key = create_service_account_with_key(signoz, token, "sa-role-editor", role="signoz-editor")
resp = requests.get(
signoz.self.host_configs["8080"].get("/api/v1/service_accounts"),
headers={"SIGNOZ-API-KEY": api_key},
timeout=5,
)
assert resp.status_code == HTTPStatus.FORBIDDEN, f"expected 403, got {resp.status_code}: {resp.text}"
```
The user-token variant creates an editor/viewer via `create_active_user` and asserts 403 on the route, 200 for admin.
## Shape 2: per-object FGA scoping (enterprise, custom role)
Grant instance verbs on one object only; assert the other stays forbidden. Structure from `role/03_fga.py`:
```python
def test_apply_license(signoz, create_user_admin, make_http_mocks, get_token):
add_license(signoz, make_http_mocks, get_token)
def test_setup_actor(signoz, create_user_admin, get_token, create_role):
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
create_role(
admin_token,
"thing-fga-actor",
[
transaction_group("read", "metaresource", "thing", [_TARGET_A_ID]),
transaction_group("list", "metaresource", "thing", ["*"]),
],
)
user_id = create_active_user(signoz, admin_token, email=_ACTOR_EMAIL, role="VIEWER", password=_ACTOR_PASSWORD, name="thing fga actor")
change_user_role(signoz, admin_token, user_id, "signoz-viewer", "thing-fga-actor")
def test_read_scoped_to_grant(signoz, create_user_admin, get_token):
token = get_token(_ACTOR_EMAIL, _ACTOR_PASSWORD)
resp = requests.get(signoz.self.host_configs["8080"].get(f"/api/v1/things/{_TARGET_A_ID}"), headers={"Authorization": f"Bearer {token}"}, timeout=5)
assert resp.status_code == HTTPStatus.OK, resp.text
resp = requests.get(signoz.self.host_configs["8080"].get(f"/api/v1/things/{_TARGET_B_ID}"), headers={"Authorization": f"Bearer {token}"}, timeout=5)
assert resp.status_code == HTTPStatus.FORBIDDEN, f"expected 403, got {resp.status_code}: {resp.text}"
```
Key semantics to test (they mirror the selector rules on the route side):
- `read`/`update`/`delete` granted on a specific selector authorize only that instance; `*` authorizes all.
- `create` and `list` are collection-scoped — only a `*` grant makes them pass, and `list` on `*` returns every instance including ones the caller can't `read` individually.
- Revoking the grant (update the role's transaction groups, or `change_user_role` back) flips the previously-allowed call to 403.
Module-level constants for names/emails (see `role/03_fga.py``_ACTOR_ROLE_NAME`, `_TARGET_A`) keep the setup test and the assertion tests in sync.
## What "done" looks like
- Every new route appears in at least one allow case and one deny case.
- Suites that create custom roles start with `test_apply_license`.
- `make py-fmt && make py-lint` pass.
- The suite passes against a `--reuse` stack: `uv run pytest --basetemp=./tmp/ -vv --reuse integration/tests/<suite>/`.

View File

@@ -0,0 +1,155 @@
# Route Wiring Reference
All examples below are real code from `pkg/apiserver/signozapiserver/serviceaccount.go`. Match this style exactly.
## Basic route: create (wildcard selector, response-phase ID)
```go
if err := router.Handle("/api/v1/service_accounts", handler.New(
provider.authzMiddleware.CheckResources(provider.serviceAccountHandler.Create, authtypes.SigNozAdminRoleName),
handler.OpenAPIDef{
ID: "CreateServiceAccount",
Tags: []string{"serviceaccount"},
Summary: "Create service account",
Description: "This endpoint creates a service account",
Request: new(serviceaccounttypes.PostableServiceAccount),
Response: new(types.Identifiable),
ResponseContentType: "application/json",
SuccessStatusCode: http.StatusCreated,
ErrorStatusCodes: []int{http.StatusBadRequest, http.StatusConflict},
SecuritySchemes: newScopedSecuritySchemes([]string{coretypes.ResourceServiceAccount.Scope(coretypes.VerbCreate)}),
},
handler.WithResourceDefs(handler.BasicResourceDef{
Resource: coretypes.ResourceServiceAccount,
Verb: coretypes.VerbCreate,
Category: coretypes.ActionCategoryAccessControl,
ID: coretypes.ResponseJSONPath("data.id"),
Selector: coretypes.WildcardSelector,
}),
)).Methods(http.MethodPost).GetError(); err != nil {
return err
}
```
Create checks the wildcard (can the caller create *any* instance?) but still records the new instance's ID for audit via `ResponseJSONPath` — the ID only exists after the handler runs.
Instance routes (read/update/delete) differ only in verb, `ID: coretypes.PathParam("id")`, and `Selector: coretypes.IDSelector`. List routes use `WildcardSelector` and omit `ID` entirely.
Pick the `Category` matching the resource's domain — grep `ActionCategory` in `pkg/types/coretypes/` for the registered values (`ActionCategoryAccessControl` is for access-control resources; observability resources use their own categories).
## ID extractors (`pkg/types/coretypes/extractor.go`)
| Extractor | Phase | Use for |
|---|---|---|
| `PathParam("id")` | request | ID in the URL path |
| `BodyJSONPath("data.id")` | request | ID in the request body (gjson path) |
| `BodyJSONArray("ids")` | request | multiple IDs in a body array |
| `ResponseJSONPath("data.id")` | response | ID known only after the handler (create) |
| `NewResourceIDExtractor(phase, fn)` | either | ID needs a lookup (see below) |
| `OneID(extractor)` | — | lifts a single-ID extractor where a multi-ID one is required (`SourceIDs`/`TargetIDs`/`ChildIDs`) |
Custom extractor example — the route path carries a service-account-role ID but the check needs the underlying role ID, so the extractor resolves it through the module:
```go
func (provider *provider) roleIDExtractor() coretypes.ResourceIDExtractor {
return coretypes.NewResourceIDExtractor(coretypes.PhaseRequest, func(ec coretypes.ExtractorContext) (string, error) {
if ec.Request == nil {
return "", nil
}
claims, err := authtypes.ClaimsFromContext(ec.Request.Context())
if err != nil {
return "", err
}
serviceAccountRoleID, err := valuer.NewUUID(mux.Vars(ec.Request)["id"])
if err != nil {
return "", err
}
serviceAccountRole, err := provider.serviceAccountGetter.GetServiceAccountRole(ec.Request.Context(), valuer.MustNewUUID(claims.OrgID), serviceAccountRoleID)
if err != nil {
return "", err
}
return serviceAccountRole.RoleID.String(), nil
})
}
```
## Custom SelectorFunc
Use one when the extracted ID is not what FGA objects use. Roles are the canonical case: routes receive a role UUID, but FGA role objects are keyed by role *name*. Always include the wildcard alongside the specific selector — authorization on `*` implies authorization on every instance:
```go
func (provider *provider) roleSelector(ctx context.Context, resource coretypes.Resource, id string, orgID valuer.UUID) ([]coretypes.Selector, error) {
roleID, err := valuer.NewUUID(id)
if err != nil {
return nil, err
}
role, err := provider.authzService.Get(ctx, orgID, roleID)
if err != nil {
return nil, err
}
return []coretypes.Selector{
resource.Type().MustSelector(role.Name),
resource.Type().MustSelector(coretypes.WildCardSelectorString),
}, nil
}
```
## Linking routes: siblings (peer-to-peer)
Attaching a role to a service account requires `attach` on **both** resources — the same verb checked on each side. Advertise both scopes:
```go
SecuritySchemes: newScopedSecuritySchemes([]string{coretypes.ResourceServiceAccount.Scope(coretypes.VerbAttach), coretypes.ResourceRole.Scope(coretypes.VerbAttach)}),
// ...
handler.WithResourceDefs(handler.AttachDetachSiblingResourceDef{
Verb: coretypes.VerbAttach,
Category: coretypes.ActionCategoryAccessControl,
SourceResource: coretypes.ResourceServiceAccount,
SourceIDs: coretypes.OneID(coretypes.PathParam("id")),
SourceSelector: coretypes.IDSelector,
TargetResource: coretypes.ResourceRole,
TargetIDs: coretypes.OneID(coretypes.BodyJSONPath("id")),
TargetSelector: provider.roleSelector,
}),
```
The detach route is identical with `VerbDetach` and the target ID coming from the path.
## Linking routes: parent-child
Creating an API key *under* a service account checks two different things: `create` on the child resource, and `attach` on the parent. Declare both defs in the same `WithResourceDefs`:
```go
handler.WithResourceDefs(
handler.BasicResourceDef{
Resource: coretypes.ResourceMetaResourceFactorAPIKey,
Verb: coretypes.VerbCreate,
Category: coretypes.ActionCategoryAccessControl,
ID: coretypes.ResponseJSONPath("data.id"),
Selector: coretypes.WildcardSelector,
},
handler.AttachDetachParentChildResourceDef{
Verb: coretypes.VerbAttach,
Category: coretypes.ActionCategoryAccessControl,
ParentResource: coretypes.ResourceServiceAccount,
ParentID: coretypes.PathParam("id"),
ParentSelector: coretypes.IDSelector,
ChildResource: coretypes.ResourceMetaResourceFactorAPIKey,
ChildIDs: coretypes.OneID(coretypes.ResponseJSONPath("data.id")),
},
),
```
Within the parent-child def, the child is only recorded for audit — the authz check on the child comes from its own `BasicResourceDef`. Deletion mirrors this with `VerbDelete` on the child and `VerbDetach` on the parent (see the `/api/v1/service_accounts/{id}/keys/{fid}` DELETE route).
## Legacy gates — do not use for new routes
`ViewAccess`/`EditAccess`/`AdminAccess` only check "does the caller hold one of these roles". They give up per-resource granularity, produce no audit resources, and don't advertise scopes. They exist for old routes; new routes use `CheckResources` + `ResourceDef`. `OpenAccess` performs no authorization at all (authentication still applies — used for `/me`-style routes); `CheckWithoutClaims` is for anonymous routes such as public dashboards.
Full def types live in `pkg/http/handler/resourcedef.go`; middleware behavior in `pkg/http/middleware/authz.go` and `pkg/http/middleware/resource.go`.

View File

@@ -2,9 +2,9 @@ package main
import (
"github.com/SigNoz/signoz/pkg/metercollector"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
"github.com/SigNoz/signoz/pkg/types/zeustypes"
)
@@ -25,8 +25,8 @@ var meterConfigs = []metercollector.Config{
Name: zeustypes.MeterLogSize,
Unit: zeustypes.MeterUnitBytes,
Aggregation: zeustypes.MeterAggregationSum,
DBName: logstelemetryschema.DBName,
TableName: logstelemetryschema.LogsV2LocalTableName,
DBName: telemetrylogs.DBName,
TableName: telemetrylogs.LogsV2LocalTableName,
DefaultRetentionDays: retentiontypes.DefaultLogsRetentionDays,
},
},
@@ -36,8 +36,8 @@ var meterConfigs = []metercollector.Config{
Name: zeustypes.MeterSpanSize,
Unit: zeustypes.MeterUnitBytes,
Aggregation: zeustypes.MeterAggregationSum,
DBName: tracestelemetryschema.DBName,
TableName: tracestelemetryschema.SpanIndexV3LocalTableName,
DBName: telemetrytraces.DBName,
TableName: telemetrytraces.SpanIndexV3LocalTableName,
DefaultRetentionDays: retentiontypes.DefaultTracesRetentionDays,
},
},
@@ -47,8 +47,8 @@ var meterConfigs = []metercollector.Config{
Name: zeustypes.MeterDatapointCount,
Unit: zeustypes.MeterUnitCount,
Aggregation: zeustypes.MeterAggregationSum,
DBName: metricstelemetryschema.DBName,
TableName: metricstelemetryschema.SamplesV4LocalTableName,
DBName: telemetrymetrics.DBName,
TableName: telemetrymetrics.SamplesV4LocalTableName,
DefaultRetentionDays: retentiontypes.DefaultMetricsRetentionDays,
},
},

View File

@@ -16,7 +16,7 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/metercollector"
"github.com/SigNoz/signoz/pkg/modules/retention"
"github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/licensetypes"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
@@ -153,7 +153,7 @@ func (provider *Provider) Collect(
func buildOriginQuery(meterName string) (string, []any) {
sb := sqlbuilder.NewSelectBuilder()
sb.Select("toInt64(ifNull(min(unix_milli), 0))")
sb.From(metertelemetryschema.DBName + "." + metertelemetryschema.SamplesTableName)
sb.From(telemetrymeter.DBName + "." + telemetrymeter.SamplesTableName)
sb.Where(sb.Equal("metric_name", meterName))
return sb.BuildWithFlavor(sqlbuilder.ClickHouse)
}
@@ -171,7 +171,7 @@ func buildQuery(meterName string, segment *retentiontypes.RetentionPolicySegment
sb := sqlbuilder.NewSelectBuilder()
sb.Select(selects...)
sb.From(metertelemetryschema.DBName + "." + metertelemetryschema.SamplesTableName)
sb.From(telemetrymeter.DBName + "." + telemetrymeter.SamplesTableName)
sb.Where(
sb.Equal("metric_name", meterName),
sb.GTE("unix_milli", segment.StartMs),

View File

@@ -8,16 +8,16 @@ import (
sqlbuilder "github.com/huandu/go-sqlbuilder"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/metricreductionruletypes"
)
var (
reductionRulesTable = metricstelemetryschema.DBName + "." + metricstelemetryschema.ReductionRulesTableName
metadataTable = metricstelemetryschema.DBName + "." + metricstelemetryschema.AttributesMetadataTableName
bufferSeriesTable = metricstelemetryschema.DBName + "." + metricstelemetryschema.TimeseriesV4BufferTableName
reductionRulesTable = telemetrymetrics.DBName + "." + telemetrymetrics.ReductionRulesTableName
metadataTable = telemetrymetrics.DBName + "." + telemetrymetrics.AttributesMetadataTableName
bufferSeriesTable = telemetrymetrics.DBName + "." + telemetrymetrics.TimeseriesV4BufferTableName
)
const timeSeriesBucketMilli = int64(time.Hour / time.Millisecond)
@@ -192,7 +192,7 @@ func (c *clickhouse) ingestedSeriesCount(ctx context.Context, metricNames []stri
sb := sqlbuilder.NewSelectBuilder()
sb.Select("metric_name", "uniq(fingerprint)")
sb.From(metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4BufferTableName)
sb.From(telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4BufferTableName)
conds := []string{
sb.In("metric_name", names...),
sb.GE("unix_milli", startMs),
@@ -229,8 +229,8 @@ func (c *clickhouse) ingestedSeriesCount(ctx context.Context, metricNames []stri
// reduced sample tables.
func (c *clickhouse) reducedSeriesCount(ctx context.Context, metricNames []string, effectiveFrom map[string]int64, startMs, endMs int64) (map[string]uint64, error) {
out := make(map[string]uint64, len(metricNames))
for _, table := range []string{metricstelemetryschema.SamplesV4ReducedLastTableName, metricstelemetryschema.SamplesV4ReducedSumTableName} {
counts, err := c.reducedSeriesCountForTable(ctx, metricstelemetryschema.DBName+"."+table, metricNames, effectiveFrom, startMs, endMs)
for _, table := range []string{telemetrymetrics.SamplesV4ReducedLastTableName, telemetrymetrics.SamplesV4ReducedSumTableName} {
counts, err := c.reducedSeriesCountForTable(ctx, telemetrymetrics.DBName+"."+table, metricNames, effectiveFrom, startMs, endMs)
if err != nil {
return nil, err
}
@@ -300,9 +300,9 @@ func (c *clickhouse) RankByVolume(ctx context.Context, metricNames []string, eff
direction = "DESC"
}
ingestedTable := metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4BufferTableName
reducedLast := metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4ReducedLastTableName
reducedSum := metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4ReducedSumTableName
ingestedTable := telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4BufferTableName
reducedLast := telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4ReducedLastTableName
reducedSum := telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4ReducedSumTableName
sb := sqlbuilder.NewSelectBuilder()
sb.Select("base.metric_name AS metric_name", "ifNull(i.cnt, 0) AS ingested", "ifNull(d.cnt, 0) AS reduced")
@@ -352,16 +352,16 @@ func (c *clickhouse) SampleVolumeByMetric(ctx context.Context, metricNames []str
}
ctx = c.withThreads(ctx)
ingested, err := c.countSamplesByMetric(ctx, metricstelemetryschema.DBName+"."+metricstelemetryschema.SamplesV4BufferTableName, "count()", metricNames, effectiveFrom, startMs, endMs)
ingested, err := c.countSamplesByMetric(ctx, telemetrymetrics.DBName+"."+telemetrymetrics.SamplesV4BufferTableName, "count()", metricNames, effectiveFrom, startMs, endMs)
if err != nil {
return nil, err
}
last, err := c.countSamplesByMetric(ctx, metricstelemetryschema.DBName+"."+metricstelemetryschema.SamplesV4ReducedLastTableName, "uniq(reduced_fingerprint, unix_milli)", metricNames, effectiveFrom, startMs, endMs)
last, err := c.countSamplesByMetric(ctx, telemetrymetrics.DBName+"."+telemetrymetrics.SamplesV4ReducedLastTableName, "uniq(reduced_fingerprint, unix_milli)", metricNames, effectiveFrom, startMs, endMs)
if err != nil {
return nil, err
}
sum, err := c.countSamplesByMetric(ctx, metricstelemetryschema.DBName+"."+metricstelemetryschema.SamplesV4ReducedSumTableName, "uniq(reduced_fingerprint, unix_milli)", metricNames, effectiveFrom, startMs, endMs)
sum, err := c.countSamplesByMetric(ctx, telemetrymetrics.DBName+"."+telemetrymetrics.SamplesV4ReducedSumTableName, "uniq(reduced_fingerprint, unix_milli)", metricNames, effectiveFrom, startMs, endMs)
if err != nil {
return nil, err
}
@@ -419,7 +419,7 @@ func (c *clickhouse) TotalVolume(ctx context.Context, startMs, endMs int64) (uin
sb := sqlbuilder.NewSelectBuilder()
sb.Select("uniq(fingerprint)", "count()")
sb.From(metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4BufferTableName)
sb.From(telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4BufferTableName)
sb.Where(sb.GE("unix_milli", startMs), sb.LT("unix_milli", endMs))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -470,7 +470,7 @@ func (c *clickhouse) SampleTimeseries(ctx context.Context, ruledMetrics []string
func (c *clickhouse) totalSamplesByBucket(ctx context.Context, startMs, endMs int64) (map[int64]uint64, error) {
sb := sqlbuilder.NewSelectBuilder()
sb.Select(sampleBucketExpr, "count()")
sb.From(metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4BufferTableName)
sb.From(telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4BufferTableName)
sb.Where(sb.GE("unix_milli", startMs), sb.LT("unix_milli", endMs))
sb.GroupBy("bucket")
@@ -485,7 +485,7 @@ func (c *clickhouse) ruledIngestedSamplesByBucket(ctx context.Context, metricNam
sb := sqlbuilder.NewSelectBuilder()
sb.Select(sampleBucketExpr, "count()")
sb.From(metricstelemetryschema.DBName + "." + metricstelemetryschema.SamplesV4BufferTableName)
sb.From(telemetrymetrics.DBName + "." + telemetrymetrics.SamplesV4BufferTableName)
conds := []string{sb.In("metric_name", names...), sb.GE("unix_milli", startMs), sb.LT("unix_milli", endMs)}
if len(effectiveFrom) > 0 {
conds = append(conds, strictEffectiveFrom(sb, metricNames, effectiveFrom))
@@ -499,7 +499,7 @@ func (c *clickhouse) ruledIngestedSamplesByBucket(ctx context.Context, metricNam
// reduced 60s rows are versioned by computed_at, so count distinct buckets.
func (c *clickhouse) ruledRetainedSamplesByBucket(ctx context.Context, metricNames []string, effectiveFrom map[string]int64, startMs, endMs int64) (map[int64]uint64, error) {
out := make(map[int64]uint64)
for _, table := range []string{metricstelemetryschema.SamplesV4ReducedLastTableName, metricstelemetryschema.SamplesV4ReducedSumTableName} {
for _, table := range []string{telemetrymetrics.SamplesV4ReducedLastTableName, telemetrymetrics.SamplesV4ReducedSumTableName} {
names := make([]any, len(metricNames))
for i, name := range metricNames {
names[i] = name
@@ -507,7 +507,7 @@ func (c *clickhouse) ruledRetainedSamplesByBucket(ctx context.Context, metricNam
sb := sqlbuilder.NewSelectBuilder()
sb.Select(sampleBucketExpr, "uniq(reduced_fingerprint, unix_milli)")
sb.From(metricstelemetryschema.DBName + "." + table)
sb.From(telemetrymetrics.DBName + "." + table)
conds := []string{sb.In("metric_name", names...), sb.GE("unix_milli", startMs), sb.LT("unix_milli", endMs)}
if len(effectiveFrom) > 0 {
conds = append(conds, strictEffectiveFrom(sb, metricNames, effectiveFrom))

View File

@@ -2,29 +2,6 @@ import { RenderErrorResponseDTO } from 'api/generated/services/sigNoz.schemas';
import { AxiosError } from 'axios';
import APIError from 'types/api/error';
// The wire shape these handlers can actually rely on. The generated
// RenderErrorResponseDTO marks code/message/url/errors as required, but the
// server omits any of them even on valid errors (e.g. a 400 with just a
// message), so a present `error` object is all the guard can guarantee.
type ErrorEnvelope = {
error: {
code?: string;
message?: string;
url?: string;
errors?: { message?: string }[];
};
};
function isErrorEnvelope(data: unknown): data is ErrorEnvelope {
return (
typeof data === 'object' &&
data !== null &&
'error' in data &&
typeof (data as ErrorEnvelope).error === 'object' &&
(data as ErrorEnvelope).error !== null
);
}
// @deprecated Use convertToApiError instead
export function ErrorResponseHandlerForGeneratedAPIs(
error: AxiosError<RenderErrorResponseDTO>,
@@ -33,29 +10,15 @@ export function ErrorResponseHandlerForGeneratedAPIs(
// The request was made and the server responded with a status code
// that falls out of the range of 2xx
if (response) {
// The body isn't guaranteed to be an error envelope — e.g. a gateway 5xx
// with an HTML/empty body during a deploy. Verify the shape before reading
// it; otherwise synthesize a consistent error from the status.
const data: unknown = response.data;
if (isErrorEnvelope(data)) {
const { code, message, url, errors } = data.error;
throw new APIError({
httpStatusCode: response.status || 500,
error: {
code: code ?? '',
message: message ?? '',
url: url ?? '',
errors: (errors ?? []).map((e) => ({ message: e.message ?? '' })),
},
});
}
throw new APIError({
httpStatusCode: response.status || 500,
error: {
code: 'UPSTREAM_UNAVAILABLE',
message: error.message || 'Something went wrong',
url: '',
errors: [],
code: response.data.error.code,
message: response.data.error.message,
url: response.data.error.url ?? '',
errors: (response.data.error.errors ?? []).map((e) => ({
message: e.message ?? '',
})),
},
});
}
@@ -99,7 +62,9 @@ export function convertToApiError(
return new APIError({
httpStatusCode: response?.status || error.status || 500,
error: {
code: errorData?.code || 'UPSTREAM_UNAVAILABLE',
code:
errorData?.code ||
String(response?.status || error.code || 'unknown_error'),
message:
errorData?.message ||
response?.statusText ||

View File

@@ -2,38 +2,19 @@ import { AxiosError } from 'axios';
import { ErrorV2Resp } from 'types/api';
import APIError from 'types/api/error';
function isErrorV2Resp(data: unknown): data is ErrorV2Resp {
return (
typeof data === 'object' &&
data !== null &&
'error' in data &&
typeof (data as ErrorV2Resp).error?.code === 'string'
);
}
// reference - https://axios-http.com/docs/handling_errors
export function ErrorResponseHandlerV2(error: AxiosError<ErrorV2Resp>): never {
const { response, request } = error;
// The request was made and the server responded with a status code
// that falls out of the range of 2xx
if (response) {
// response.data isn't guaranteed to be a V2 envelope (e.g. a gateway 5xx
// with an HTML/empty body during a deploy), so verify the shape first.
const data: unknown = response.data;
if (isErrorV2Resp(data)) {
const { code, message, url, errors } = data.error;
throw new APIError({
httpStatusCode: response.status || 500,
error: { code, message, url, errors },
});
}
throw new APIError({
httpStatusCode: response.status || 500,
error: {
code: 'UPSTREAM_UNAVAILABLE',
message: error.message || 'Something went wrong',
url: '',
errors: [],
code: response.data.error.code,
message: response.data.error.message,
url: response.data.error.url,
errors: response.data.error.errors,
},
});
}

View File

@@ -1,126 +0,0 @@
import { ErrorResponseHandlerV2 } from 'api/ErrorResponseHandlerV2';
import { AxiosError } from 'axios';
import { ErrorV2Resp } from 'types/api';
import APIError from 'types/api/error';
function asAxiosError(partial: Partial<AxiosError>): AxiosError<ErrorV2Resp> {
return partial as AxiosError<ErrorV2Resp>;
}
// ErrorResponseHandlerV2 always throws — capture the APIError so assertions stay
// unconditional.
function runHandler(error: AxiosError<ErrorV2Resp>): APIError {
try {
ErrorResponseHandlerV2(error);
} catch (thrown) {
return thrown as APIError;
}
throw new Error('expected ErrorResponseHandlerV2 to throw');
}
type ExpectedError = {
httpStatusCode: number;
code: string;
message: string;
errors: { message: string }[];
};
// One row per response shape the handler must normalize. New shapes (with
// different bodies) can be added here without a new test block.
const cases: {
name: string;
error: AxiosError<ErrorV2Resp>;
expected: ExpectedError;
}[] = [
{
name: 'well-formed V2 error envelope',
error: asAxiosError({
message: 'Request failed with status code 400',
response: {
status: 400,
data: {
error: {
code: 'bad_request',
message: 'Invalid dashboard payload',
url: 'https://signoz.io/docs',
errors: [{ message: 'name is required' }, { message: 'name too long' }],
},
},
} as AxiosError['response'],
}),
expected: {
httpStatusCode: 400,
code: 'bad_request',
message: 'Invalid dashboard payload',
errors: [{ message: 'name is required' }, { message: 'name too long' }],
},
},
{
// Regression: during a deployment the gateway returns a 5xx with a
// non-envelope body. Reading response.data.error.code used to throw a
// TypeError from inside the handler itself. See engineering-pod#5760.
name: '5xx with a non-envelope HTML body',
error: asAxiosError({
message: 'Request failed with status code 503',
response: {
status: 503,
data: '<html><body>503 Service Temporarily Unavailable</body></html>',
} as AxiosError['response'],
}),
expected: {
httpStatusCode: 503,
code: 'UPSTREAM_UNAVAILABLE',
message: 'Request failed with status code 503',
errors: [],
},
},
{
name: '5xx with an empty body',
error: asAxiosError({
message: 'Request failed with status code 502',
response: {
status: 502,
data: undefined,
} as AxiosError['response'],
}),
expected: {
httpStatusCode: 502,
code: 'UPSTREAM_UNAVAILABLE',
message: 'Request failed with status code 502',
errors: [],
},
},
{
name: 'no response received (network error)',
error: asAxiosError({
message: 'Network Error',
code: 'ERR_NETWORK',
name: 'AxiosError',
request: {},
}),
expected: {
httpStatusCode: 500,
code: 'ERR_NETWORK',
message: 'Network Error',
errors: [],
},
},
];
describe('ErrorResponseHandlerV2', () => {
it.each(cases)(
'normalizes $name into a consistent APIError',
({ error, expected }) => {
const apiError = runHandler(error);
expect(apiError).toBeInstanceOf(APIError);
expect(apiError.getHttpStatusCode()).toBe(expected.httpStatusCode);
expect(apiError.getErrorCode()).toBe(expected.code);
expect(apiError.getErrorMessage()).toBe(expected.message);
// The sub-error messages feed several parts of the UI, so assert them.
expect(apiError.getErrorDetails().error.errors).toStrictEqual(
expected.errors,
);
},
);
});

View File

@@ -92,76 +92,52 @@
color: var(--l3-foreground);
}
.only-btn,
.only-btn {
cursor: not-allowed;
color: var(--l3-foreground);
}
.toggle-btn {
cursor: not-allowed;
color: var(--l3-foreground);
}
}
// Stack "Only"/"All" and "Toggle" in a single cell so revealing
// one swaps in place instead of shifting the row layout.
.value-actions {
display: grid;
align-items: center;
justify-items: end;
> * {
grid-area: 1 / 1;
}
.only-btn {
display: none;
}
.only-btn,
.toggle-btn {
display: none;
}
.toggle-btn:hover {
background-color: unset;
}
.only-btn:hover {
background-color: unset;
}
}
.checkbox-value-section:hover {
.toggle-btn {
display: none;
}
.only-btn {
display: flex;
align-items: center;
justify-content: center;
height: 21px;
opacity: 0;
transform: translateX(4px);
// `display` is animated as a discrete property so the button
// stays mounted through its fade-out on exit.
transition:
opacity 0.16s ease,
transform 0.16s ease,
display 0.16s allow-discrete;
&:hover {
background-color: unset;
}
}
}
// Hovering the label/value area reveals the "Only"/"All" action.
.checkbox-value-section:hover .only-btn {
display: flex;
opacity: 1;
transform: translateX(0);
// Entry animation start state (fires on the display switch).
@starting-style {
opacity: 0;
transform: translateX(4px);
}
}
// Hovering the checkbox reveals the "Toggle" action.
.check-box:hover ~ .checkbox-value-section .toggle-btn {
display: flex;
opacity: 1;
transform: translateX(0);
@starting-style {
opacity: 0;
transform: translateX(4px);
}
}
}
@media (prefers-reduced-motion: reduce) {
.only-btn,
.value:hover {
.toggle-btn {
transition: none;
display: flex;
align-items: center;
justify-content: center;
height: 21px;
}
}
}

View File

@@ -53,14 +53,12 @@ function CheckboxValueRow({
</Typography.Text>
</TooltipSimple>
)}
<div className="value-actions">
<Button type="text" className="only-btn">
{onlyButtonLabel}
</Button>
<Button type="text" className="toggle-btn">
Toggle
</Button>
</div>
<Button type="text" className="only-btn">
{onlyButtonLabel}
</Button>
<Button type="text" className="toggle-btn">
Toggle
</Button>
</div>
</div>
);

View File

@@ -6,7 +6,7 @@ import { Skeleton } from 'antd';
import { Typography } from '@signozhq/ui/typography';
import {
useGetMetricAlerts,
useGetMetricDashboardsV2,
useGetMetricDashboards,
} from 'api/generated/services/metrics';
import { QueryParams } from 'constants/query';
import ROUTES from 'constants/routes';
@@ -38,7 +38,7 @@ function DashboardsAndAlertsPopover({
data: dashboardsData,
isLoading: isLoadingDashboards,
isError: isErrorDashboards,
} = useGetMetricDashboardsV2(
} = useGetMetricDashboards(
{
metricName,
},
@@ -55,8 +55,7 @@ function DashboardsAndAlertsPopover({
const dashboards = useMemo(() => {
const currentDashboards = dashboardsData?.data.dashboards ?? [];
// The API returns one entry per referencing panel, so a dashboard repeats
// once per panel that uses the metric.
// Remove duplicate dashboards
return currentDashboards.filter(
(dashboard, index, self) =>
index === self.findIndex((t) => t.dashboardId === dashboard.dashboardId),

View File

@@ -23,7 +23,7 @@ const useGetMetricAlertsMock = jest.spyOn(
);
const useGetMetricDashboardsMock = jest.spyOn(
metricsExplorerHooks,
'useGetMetricDashboardsV2',
'useGetMetricDashboards',
);
describe('DashboardsAndAlertsPopover', () => {
@@ -153,15 +153,11 @@ describe('DashboardsAndAlertsPopover', () => {
);
});
it('collapses multiple panels of the same dashboard into one entry', async () => {
it('renders unique dashboards even when there are duplicates', async () => {
useGetMetricDashboardsMock.mockReturnValue(
getMockDashboardsData({
data: {
dashboards: [
MOCK_DASHBOARD_1,
MOCK_DASHBOARD_2,
{ ...MOCK_DASHBOARD_1, panelId: '3', panelName: 'Panel 3' },
],
dashboards: [MOCK_DASHBOARD_1, MOCK_DASHBOARD_2, MOCK_DASHBOARD_1],
},
}),
);

View File

@@ -2,7 +2,7 @@ import * as metricsExplorerHooks from 'api/generated/services/metrics';
import {
GetMetricAlerts200,
GetMetricAttributes200,
GetMetricDashboardsV2200,
GetMetricDashboards200,
GetMetricHighlights200,
GetMetricMetadata200,
MetrictypesTemporalityDTO,
@@ -40,14 +40,14 @@ export function getMockMetricHighlightsData(
export const MOCK_DASHBOARD_1 = {
dashboardName: 'Dashboard 1',
dashboardId: '1',
panelId: '1',
panelName: 'Panel 1',
widgetId: '1',
widgetName: 'Widget 1',
};
export const MOCK_DASHBOARD_2 = {
dashboardName: 'Dashboard 2',
dashboardId: '2',
panelId: '2',
panelName: 'Panel 2',
widgetId: '2',
widgetName: 'Widget 2',
};
export const MOCK_ALERT_1 = {
alertName: 'Alert 1',
@@ -59,7 +59,7 @@ export const MOCK_ALERT_2 = {
};
export function getMockDashboardsData(
overrides?: Partial<GetMetricDashboardsV2200>,
overrides?: Partial<GetMetricDashboards200>,
{
isLoading = false,
isError = false,
@@ -67,7 +67,7 @@ export function getMockDashboardsData(
isLoading?: boolean;
isError?: boolean;
} = {},
): ReturnType<typeof metricsExplorerHooks.useGetMetricDashboardsV2> {
): ReturnType<typeof metricsExplorerHooks.useGetMetricDashboards> {
return {
data: {
data: {
@@ -79,7 +79,7 @@ export function getMockDashboardsData(
isLoading,
isError,
} as ReturnType<typeof metricsExplorerHooks.useGetMetricDashboardsV2>;
} as ReturnType<typeof metricsExplorerHooks.useGetMetricDashboards>;
}
export function getMockAlertsData(

View File

@@ -11,9 +11,3 @@
border: 1px solid var(--l3-border) !important;
box-shadow: none !important;
}
/* Highlights the dashboard name inside the delete-confirm title. */
.deleteName {
color: var(--danger-background);
font-weight: var(--font-weight-medium);
}

View File

@@ -27,13 +27,16 @@ import logEvent from 'api/common/logEvent';
import { cloneDashboardV2 } from 'api/generated/services/dashboard';
import type { DashboardtypesGettableDashboardV2DTO } from 'api/generated/services/sigNoz.schemas';
import ROUTES from 'constants/routes';
import { useDeleteDashboard } from 'hooks/dashboard/useDeleteDashboard';
import { useSafeNavigate } from 'hooks/useSafeNavigate';
import history from 'lib/history';
import { DashboardDetailEvents } from 'pages/DashboardPageV2/constants/events';
import { useAppContext } from 'providers/App/App';
import { useErrorModal } from 'providers/ErrorModalProvider';
import APIError from 'types/api/error';
import { USER_ROLES } from 'types/roles';
import ConfirmDeleteDialog from '../../components/ConfirmDeleteDialog/ConfirmDeleteDialog';
import DisabledControlTooltip from '../../components/DisabledControlTooltip/DisabledControlTooltip';
import DisabledMenuItemLabel from '../../components/DisabledMenuItemLabel/DisabledMenuItemLabel';
import DashboardSettings from '../../DashboardSettings';
@@ -42,7 +45,6 @@ import SectionTitleModal from '../../PanelsAndSectionsLayout/Section/SectionTitl
import JsonEditorDrawer from '../JsonEditorDrawer/JsonEditorDrawer';
import SettingsDrawer from '../SettingsDrawer';
import styles from './DashboardActions.module.scss';
import { useDeleteDashboardAction } from './useDeleteDashboardAction';
import { DASHBOARD_LOCKED_REASON } from '../../hooks/useDashboardEditGuard';
import { useDashboardStore } from '../../store/useDashboardStore';
@@ -82,12 +84,8 @@ function DashboardActions({
const [isCloning, setIsCloning] = useState<boolean>(false);
const [isNewSectionOpen, setIsNewSectionOpen] = useState<boolean>(false);
const { contextHolder: deleteConfirmHolder, confirmDeleteDashboard } =
useDeleteDashboardAction({
dashboardId: dashboard.id,
dashboardName: title,
panelCount: Object.keys(dashboard.spec.panels).length,
});
const [isDeleteOpen, setIsDeleteOpen] = useState<boolean>(false);
const deleteDashboardMutation = useDeleteDashboard(dashboard.id);
// Open the settings drawer when something in the tree requests it (e.g. the
// variables bar's "Add variable" button).
@@ -134,6 +132,19 @@ function DashboardActions({
}
}, [dashboard.id, title, safeNavigate, showErrorModal]);
const handleConfirmDelete = useCallback((): void => {
deleteDashboardMutation.mutate(undefined, {
onSuccess: () => {
void logEvent(DashboardDetailEvents.Deleted, {
dashboardId: dashboard.id,
panelCount: Object.keys(dashboard.spec.panels).length,
});
setIsDeleteOpen(false);
history.replace(ROUTES.ALL_DASHBOARD);
},
});
}, [deleteDashboardMutation, dashboard.id, dashboard.spec.panels]);
const handleOpenSettings = useCallback((): void => {
void logEvent(DashboardDetailEvents.SettingsOpened, {
dashboardId: dashboard.id,
@@ -237,7 +248,7 @@ function DashboardActions({
icon: <Trash2 size={14} />,
danger: true,
disabled: isLocked,
onClick: confirmDeleteDashboard,
onClick: (): void => setIsDeleteOpen(true),
},
);
}
@@ -255,7 +266,6 @@ function DashboardActions({
handleClone,
onLockToggle,
handleEnterFullScreen,
confirmDeleteDashboard,
]);
return (
@@ -338,7 +348,14 @@ function DashboardActions({
isOpen={isJsonEditorOpen}
onClose={(): void => setIsJsonEditorOpen(false)}
/>
{deleteConfirmHolder}
<ConfirmDeleteDialog
open={isDeleteOpen}
title={`Delete dashboard"?`}
description={`Are you sure you want to delete this dashboard - "${title}"? This action cannot be undone.`}
isLoading={deleteDashboardMutation.isLoading}
onConfirm={handleConfirmDelete}
onClose={(): void => setIsDeleteOpen(false)}
/>
<SectionTitleModal
open={isNewSectionOpen}
heading="New section"

View File

@@ -1,89 +0,0 @@
import { type ReactNode, useCallback } from 'react';
import { useQueryClient } from 'react-query';
import { toast } from '@signozhq/ui/sonner';
import { Typography } from '@signozhq/ui/typography';
import logEvent from 'api/common/logEvent';
import {
invalidateListDashboardsForUserV2,
useDeleteDashboardV2,
} from 'api/generated/services/dashboard';
import { useDeleteConfirm } from 'components/DeleteConfirmModal/useDeleteConfirm';
import ROUTES from 'constants/routes';
import { useDashboardPreferencesStore } from 'hooks/dashboard/useDashboardPreference';
import history from 'lib/history';
import { DashboardDetailEvents } from 'pages/DashboardPageV2/constants/events';
import { useErrorModal } from 'providers/ErrorModalProvider';
import APIError from 'types/api/error';
import styles from './DashboardActions.module.scss';
interface UseDeleteDashboardActionArgs {
dashboardId: string;
dashboardName: string;
panelCount: number;
}
interface UseDeleteDashboardAction {
/** Must be rendered in the calling component for the modal to appear. */
contextHolder: ReactNode;
confirmDeleteDashboard: () => void;
}
/**
* Deletes the open dashboard through the v2 endpoint behind the shared
* destructive confirmation, then drops its local preferences, refreshes the
* dashboard list cache and sends the user back to the list.
*/
export function useDeleteDashboardAction({
dashboardId,
dashboardName,
panelCount,
}: UseDeleteDashboardActionArgs): UseDeleteDashboardAction {
const queryClient = useQueryClient();
const { showErrorModal } = useErrorModal();
const { contextHolder, confirmDelete } = useDeleteConfirm();
const removePreferences = useDashboardPreferencesStore(
(state) => state.removePreferences,
);
const { mutate: deleteDashboard } = useDeleteDashboardV2({
mutation: {
onSuccess: async (): Promise<void> => {
void logEvent(DashboardDetailEvents.Deleted, { dashboardId, panelCount });
removePreferences(dashboardId);
await invalidateListDashboardsForUserV2(queryClient);
toast.success('Dashboard deleted successfully');
history.replace(ROUTES.ALL_DASHBOARD);
},
onError: (error: unknown): void => {
showErrorModal(error as APIError);
},
},
});
const confirmDeleteDashboard = useCallback((): void => {
confirmDelete({
title: (
<Typography.Title level={5}>
Are you sure you want to delete the
<Typography.Text className={styles.deleteName}>
{' '}
{dashboardName}{' '}
</Typography.Text>
dashboard?
</Typography.Title>
),
content: 'This action cannot be undone.',
// Keeps the Delete button loading until the mutation settles, then closes.
onConfirm: () =>
new Promise<void>((resolve) => {
deleteDashboard(
{ pathParams: { id: dashboardId } },
{ onSettled: () => resolve() },
);
}),
});
}, [confirmDelete, dashboardName, deleteDashboard, dashboardId]);
return { contextHolder, confirmDeleteDashboard };
}

View File

@@ -1,6 +1,5 @@
import { useCallback } from 'react';
import { SolidAlertTriangle, X } from '@signozhq/icons';
import { Badge } from '@signozhq/ui/badge';
import { Button } from '@signozhq/ui/button';
import { DialogWrapper } from '@signozhq/ui/dialog';
import { Divider } from '@signozhq/ui/divider';
@@ -14,7 +13,6 @@ import DisabledControlTooltip from '../../components/DisabledControlTooltip/Disa
import styles from './Header.module.scss';
interface HeaderProps {
/** Unsaved edits exist — shows the "Unsaved Changes" badge and gates the discard confirmation on close (not the Save button). */
isDirty: boolean;
isSaving: boolean;
showSwitchToView?: boolean;
@@ -68,11 +66,6 @@ function Header({
/>
<Divider type="vertical" />
<Typography.Text>Configure panel</Typography.Text>
{isDirty && (
<Badge color="warning" data-testid="panel-editor-v2-unsaved-badge">
Unsaved Changes
</Badge>
)}
</div>
<div className={styles.actions}>
<HeaderRightSection
@@ -95,7 +88,7 @@ function Header({
variant="solid"
color="primary"
data-testid="panel-editor-v2-save"
disabled={readOnly || isSaving}
disabled={readOnly || !isDirty || isSaving}
loading={!readOnly && isSaving}
onClick={readOnly ? undefined : onSave}
>

View File

@@ -1,4 +1,3 @@
import type { ComponentProps } from 'react';
import { MemoryRouter } from 'react-router-dom';
import { render, screen } from '@testing-library/react';
import { TooltipProvider } from '@signozhq/ui/tooltip';
@@ -24,9 +23,7 @@ jest.mock('api/common/logEvent', () => ({
const mockUseIsAIAssistantEnabled = useIsAIAssistantEnabled as jest.Mock;
function renderHeader(
props: Partial<ComponentProps<typeof Header>> = {},
): void {
function renderHeader(): void {
// AppLayout supplies the TooltipProvider in the app; the header is rendered bare here.
render(
<MemoryRouter>
@@ -36,7 +33,6 @@ function renderHeader(
isSaving={false}
onSave={jest.fn()}
onClose={jest.fn()}
{...props}
/>
</TooltipProvider>
</MemoryRouter>,
@@ -70,34 +66,4 @@ describe('PanelEditor Header', () => {
).not.toBeInTheDocument();
expect(screen.getByTestId('panel-editor-v2-save')).toBeInTheDocument();
});
it('keeps Save enabled even when there are no unsaved edits', () => {
mockUseIsAIAssistantEnabled.mockReturnValue(false);
renderHeader({ isDirty: false });
expect(screen.getByTestId('panel-editor-v2-save')).toBeEnabled();
});
it('disables Save only while read-only or saving', () => {
mockUseIsAIAssistantEnabled.mockReturnValue(false);
renderHeader({ isDirty: true, readOnly: true, readOnlyReason: 'Locked' });
expect(screen.getByTestId('panel-editor-v2-save')).toBeDisabled();
});
it('shows the Unsaved Changes badge only when there are unsaved edits', () => {
mockUseIsAIAssistantEnabled.mockReturnValue(false);
renderHeader({ isDirty: false });
expect(
screen.queryByTestId('panel-editor-v2-unsaved-badge'),
).not.toBeInTheDocument();
renderHeader({ isDirty: true });
expect(
screen.getByTestId('panel-editor-v2-unsaved-badge'),
).toBeInTheDocument();
});
});

View File

@@ -1,6 +1,5 @@
import { render, screen, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { toast } from '@signozhq/ui/sonner';
import type { DashboardtypesPanelDTO } from 'api/generated/services/sigNoz.schemas';
import { PANEL_TYPES } from 'constants/queryBuilder';
import { getPanelDefinition } from 'pages/DashboardPageV2/DashboardContainer/Panels/registry';
@@ -101,12 +100,6 @@ jest.mock('@signozhq/ui/resizable', () => ({
jest.mock('@signozhq/ui/sonner', () => ({
toast: { success: jest.fn(), error: jest.fn() },
}));
const mockShowErrorModal = jest.fn();
jest.mock('providers/ErrorModalProvider', () => ({
useErrorModal: (): { showErrorModal: jest.Mock } => ({
showErrorModal: mockShowErrorModal,
}),
}));
// Children mocked to capture props (and expose a Save trigger / footer slot).
const mockHeaderProps = jest.fn();
@@ -268,13 +261,13 @@ describe('PanelEditorContainer composition', () => {
);
});
it('keeps a query-less new panel clean but still serializes its seed query', () => {
it('keeps a query-less new panel unsaveable but still serializes its seed query', () => {
setup(makePanel('signoz/TimeSeriesPanel'), { isNew: true });
expect(mockUseQuerySync).toHaveBeenCalledWith(
expect.objectContaining({ alwaysSerializeQuery: true }),
);
// No query and no edits yet → not dirty, so closing won't prompt to discard.
// No query and no edits yet → nothing to save, so Save stays disabled.
expect(mockHeaderProps).toHaveBeenCalledWith(
expect.objectContaining({ isDirty: false }),
);
@@ -297,7 +290,7 @@ describe('PanelEditorContainer composition', () => {
);
});
it('marks a new panel that already has a query dirty (e.g. list auto-runs one)', () => {
it('marks a new panel that already has a query saveable (e.g. list auto-runs one)', () => {
const seededQuery = {
spec: { plugin: { kind: 'signoz/BuilderQuery', spec: { signal: 'logs' } } },
};
@@ -334,24 +327,6 @@ describe('PanelEditorContainer composition', () => {
expect(mockSave).toHaveBeenCalledWith(panel.spec);
});
it('surfaces a save failure through the error modal', async () => {
// The raw thrown value flows straight to the modal, which normalizes it to an
// APIError itself (that normalization is covered by ErrorModalProvider's tests).
const failure = {
response: {
status: 400,
data: { error: { code: 'INVALID', message: 'Panel name already exists' } },
},
};
mockSave.mockRejectedValueOnce(failure);
setup(makePanel('signoz/TimeSeriesPanel'));
await userEvent.click(screen.getByTestId('editor-save'));
await waitFor(() => expect(mockShowErrorModal).toHaveBeenCalledWith(failure));
expect(toast.error).not.toHaveBeenCalled();
});
it('marks the saved panel to be scrolled into view on the dashboard', async () => {
setup(makePanel('signoz/TimeSeriesPanel'));

View File

@@ -117,49 +117,6 @@ describe('usePanelEditorQuerySync (real query builder)', () => {
},
);
it('a NEW panel stays query-dirty after Stage & Run commits the edited query into the draft', async () => {
// Repro of the P0: new panel → edit → Run commits the query into the draft; the
// dirty baseline must not drift onto it, or Save re-disables.
const editedInUrl: Query = {
...initialQueriesMap[DataSource.METRICS],
id: 'edited-new-panel',
builder: {
...initialQueriesMap[DataSource.METRICS].builder,
queryData: [
{
...initialQueriesMap[DataSource.METRICS].builder.queryData[0],
legend: 'edited-legend',
},
],
},
};
const committed = toPerses(editedInUrl, panelType);
const { result, rerender } = renderHook(
({ draftQueries }: { draftQueries: DashboardtypesQueryDTO[] }) =>
usePanelEditorQuerySync({
draft: makePanel(draftQueries),
panelType,
setSpec: jest.fn(),
refetch: jest.fn(),
alwaysSerializeQuery: true,
signal: DataSource.METRICS as unknown as TelemetrytypesSignalDTO,
// savedQueries omitted — new panel.
}),
{
wrapper: makeUrlWrapper(editedInUrl),
initialProps: { draftQueries: [] as DashboardtypesQueryDTO[] },
},
);
// The edited query (from the URL) diverges from the seeded default → dirty.
await waitFor(() => expect(result.current.isQueryDirty).toBe(true));
// Stage & Run commits the edited query into the draft → must stay dirty.
rerender({ draftQueries: committed });
expect(result.current.isQueryDirty).toBe(true);
});
it('an untouched panel with a minimal/older stored query is NOT dirty (drift fix)', async () => {
// An older saved query carries only a few fields; the builder re-emits many more
// (source, stepInterval, filter, spaceAggregation, …). Comparing raw would read

View File

@@ -79,11 +79,6 @@ export function usePanelEditorQuerySync({
// in-editor edit in the URL survives a refresh / browser Back-Forward.
useShareBuilderUrl({ defaultValue: seedQuery });
// Frozen seed for the new-panel dirty baseline: Stage & Run commits the live query
// into `draft.spec.queries`, so re-reading `seedQuery` would drift the baseline onto
// the run query and Save would re-disable right after a run.
const initialSeedRef = useRef(seedQuery);
// Commit the live query into the draft (what the preview fetches).
const commitQuery = useCallback(
(query: Query): boolean => {
@@ -157,19 +152,16 @@ export function usePanelEditorQuerySync({
// the V5 envelope level. Anchoring to `savedQueries` (not the builder-seed) keeps a
// handed-off / URL-restored edit reading as dirty; routing both sides through the same
// `fromPerses → toPerses` round-trip stops builder-added defaults (absent from an older
// stored query) reading an untouched panel as modified. New panel: fall back to the
// frozen initial seed (see `initialSeedRef`).
// stored query) reading an untouched panel as modified. New panel: fall back to seed.
const baselineEnvelopes = useMemo(
() =>
toQueryEnvelopes(
toPerses(
savedQueries
? fromPerses(savedQueries, panelType)
: initialSeedRef.current,
savedQueries ? fromPerses(savedQueries, panelType) : seedQuery,
panelType,
),
),
[savedQueries, panelType],
[savedQueries, seedQuery, panelType],
);
const isQueryDirty = useMemo(
() =>

View File

@@ -19,7 +19,6 @@ import {
SectionKind,
} from 'pages/DashboardPageV2/DashboardContainer/Panels/types/sections';
import { getBuilderQueries } from 'pages/DashboardPageV2/DashboardContainer/Panels/utils/getBuilderQueries';
import { useErrorModal } from 'providers/ErrorModalProvider';
import { getExecStats } from '../queryV5/v5ResponseData';
import { usePanelInteractions } from '../PanelsAndSectionsLayout/Panel/hooks/usePanelInteractions';
@@ -160,10 +159,9 @@ function PanelEditorContainer({
return section?.controls;
}, [panelDefinition]);
// Unsaved-edits flag driving the discard confirmation on close (Save is always
// enabled). Read the seed `panel`, not the live `draft` — the staged-query sync
// commits the seed into the draft on open, which would falsely dirty an untouched
// query-less new panel.
// New panels are savable once seeded with a query (List auto-seeds one). Read the
// seed `panel`, not the live `draft` — the staged-query sync commits the seed into
// the draft on open, which would falsely dirty an untouched query-less new panel.
const isDirty = useMemo(
() => isSpecDirty || isQueryDirty || (isNew && panel.spec.queries.length > 0),
[isSpecDirty, isQueryDirty, isNew, panel.spec.queries.length],
@@ -228,7 +226,6 @@ function PanelEditorContainer({
});
const setScrollTargetId = useScrollIntoViewStore((s) => s.setScrollTargetId);
const { showErrorModal } = useErrorModal();
const onSave = useCallback(async (): Promise<void> => {
if (!isEditable) {
@@ -239,22 +236,12 @@ function PanelEditorContainer({
const savedPanelId = await save(buildSaveSpec(draft.spec));
// Reveal the saved panel once the dashboard re-renders.
setScrollTargetId(savedPanelId);
toast.success('Panel saved', {
position: 'top-center',
});
toast.success('Panel saved');
onSaved();
} catch (err) {
showErrorModal(err);
} catch {
toast.error('Failed to save panel');
}
}, [
isEditable,
save,
buildSaveSpec,
draft.spec,
setScrollTargetId,
onSaved,
showErrorModal,
]);
}, [isEditable, save, buildSaveSpec, draft.spec, setScrollTargetId, onSaved]);
// Leaving an existing panel's editor (without saving) still returns to it, so
// the dashboard lands on that panel rather than scrolled to the top. A new,

View File

@@ -51,7 +51,7 @@ describe('panelStatusFromError', () => {
it('falls back to the error message when there is no structured body', () => {
expect(panelStatusFromError(new Error('boom'))).toStrictEqual({
code: 'UPSTREAM_UNAVAILABLE',
code: 'unknown_error',
message: 'boom',
docsUrl: undefined,
messages: [],

View File

@@ -12,13 +12,13 @@ import {
deleteDashboardV2,
invalidateListDashboardsForUserV2,
} from 'api/generated/services/dashboard';
import { useDeleteConfirm } from 'components/DeleteConfirmModal/useDeleteConfirm';
import { DashboardListEvents } from 'pages/DashboardsListPageV2/constants/events';
import { useAppContext } from 'providers/App/App';
import { useErrorModal } from 'providers/ErrorModalProvider';
import APIError from 'types/api/error';
import { USER_ROLES } from 'types/roles';
import { useDeleteConfirm } from '../DeleteConfirmModal/useDeleteConfirm';
import styles from './ActionsPopover.module.scss';
interface Props {

View File

@@ -6,11 +6,11 @@ import { Typography } from '@signozhq/ui/typography';
import { Bookmark, PenLine, Plus, Search, Trash2 } from '@signozhq/icons';
import cx from 'classnames';
import { useDeleteConfirm } from 'components/DeleteConfirmModal/useDeleteConfirm';
import { DashboardListEvents } from 'pages/DashboardsListPageV2/constants/events';
import type { SavedView } from '../../types';
import { type BuiltinView } from '../../utils/views';
import { useDeleteConfirm } from '../DeleteConfirmModal/useDeleteConfirm';
import ViewNamePopover from './ViewNamePopover';
import styles from './ViewsRail.module.scss';

View File

@@ -42,12 +42,7 @@ export function toAPIError(
try {
ErrorResponseHandlerForGeneratedAPIs(error);
} catch (apiError) {
// UPSTREAM_UNAVAILABLE means the handler couldn't find a backend error code
// (non-envelope body); prefer the caller's context-specific defaultMessage.
if (
apiError instanceof APIError &&
apiError.getErrorCode() !== 'UPSTREAM_UNAVAILABLE'
) {
if (apiError instanceof APIError) {
return apiError;
}
}

View File

@@ -5,14 +5,14 @@ import (
"fmt"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/valuer"
)
func (m *module) Collect(ctx context.Context, _ valuer.UUID) (map[string]any, error) {
stats := make(map[string]any)
metadataTable := fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName)
metadataTable := fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName)
var (
systemMetricCount uint64
k8sMetricCount uint64

View File

@@ -7,7 +7,7 @@ import (
"strings"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -281,7 +281,7 @@ func (m *module) getPerGroupContainerStatusCounts(
mergedFilterExpr := mergeFilterExpressions(userFilterExpr, buildPageGroupsFilterExpr(pageGroups))
samplesStartMs, flooredEndMs, tsAdjustedStart, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
// Built once; identical across the two fps CTEs (buildFilterClause hits the
// metadata store + parses the expression). AddWhereClause only reads it.
@@ -310,7 +310,7 @@ func (m *module) getPerGroupContainerStatusCounts(
)
}
stateFps.Select(stateFpsCols...)
stateFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
stateFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
stateFps.Where(
stateFps.E("metric_name", containerStatusStateMetricName),
stateFps.GE("unix_milli", tsAdjustedStart),
@@ -342,7 +342,7 @@ func (m *module) getPerGroupContainerStatusCounts(
containerState.Select(containerStateCols...)
containerState.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN state_fps AS fps ON samples.fingerprint = fps.fingerprint",
metricstelemetryschema.DBName, distributedSamplesTable,
telemetrymetrics.DBName, distributedSamplesTable,
))
containerState.Where(
containerState.E("samples.metric_name", containerStatusStateMetricName),
@@ -361,7 +361,7 @@ func (m *module) getPerGroupContainerStatusCounts(
fmt.Sprintf("JSONExtractString(labels, %s) AS container_name", reasonFps.Var(containerNameAttrKey)),
fmt.Sprintf("JSONExtractString(labels, %s) AS reason", reasonFps.Var(containerStatusReasonAttrKey)),
)
reasonFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
reasonFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
reasonFps.Where(
reasonFps.E("metric_name", containerStatusReasonMetricName),
reasonFps.GE("unix_milli", tsAdjustedStart),
@@ -391,7 +391,7 @@ func (m *module) getPerGroupContainerStatusCounts(
)
reasonInner.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN reason_fps AS fps ON samples.fingerprint = fps.fingerprint",
metricstelemetryschema.DBName, distributedSamplesTable,
telemetrymetrics.DBName, distributedSamplesTable,
))
reasonInner.Where(
reasonInner.E("samples.metric_name", containerStatusReasonMetricName),
@@ -546,7 +546,7 @@ func (m *module) getPerGroupContainerRestartCounts(
mergedFilterExpr := mergeFilterExpressions(userFilterExpr, buildPageGroupsFilterExpr(pageGroups))
samplesStartMs, flooredEndMs, tsAdjustedStart, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
var (
filterClause *sqlbuilder.WhereClause
@@ -572,7 +572,7 @@ func (m *module) getPerGroupContainerRestartCounts(
)
}
restartFps.Select(restartFpsCols...)
restartFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
restartFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
restartFps.Where(
restartFps.E("metric_name", containerRestartsMetricName),
restartFps.GE("unix_milli", tsAdjustedStart),
@@ -602,7 +602,7 @@ func (m *module) getPerGroupContainerRestartCounts(
containerRestarts.Select(containerRestartsCols...)
containerRestarts.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN restart_fps AS fps ON samples.fingerprint = fps.fingerprint",
metricstelemetryschema.DBName, distributedSamplesTable,
telemetrymetrics.DBName, distributedSamplesTable,
))
containerRestarts.Where(
containerRestarts.E("samples.metric_name", containerRestartsMetricName),
@@ -690,7 +690,7 @@ func (m *module) getPerGroupContainerReadyCounts(
mergedFilterExpr := mergeFilterExpressions(userFilterExpr, buildPageGroupsFilterExpr(pageGroups))
samplesStartMs, flooredEndMs, tsAdjustedStart, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
var (
filterClause *sqlbuilder.WhereClause
@@ -716,7 +716,7 @@ func (m *module) getPerGroupContainerReadyCounts(
)
}
readyFps.Select(readyFpsCols...)
readyFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
readyFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
readyFps.Where(
readyFps.E("metric_name", containerReadyMetricName),
readyFps.GE("unix_milli", tsAdjustedStart),
@@ -746,7 +746,7 @@ func (m *module) getPerGroupContainerReadyCounts(
containerReady.Select(containerReadyCols...)
containerReady.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN ready_fps AS fps ON samples.fingerprint = fps.fingerprint",
metricstelemetryschema.DBName, distributedSamplesTable,
telemetrymetrics.DBName, distributedSamplesTable,
))
containerReady.Where(
containerReady.E("samples.metric_name", containerReadyMetricName),

View File

@@ -9,7 +9,7 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
@@ -344,11 +344,11 @@ func alignedMetricWindow(startMs, endMs int64) (
flooredEndMs = flooredEndMs - (flooredEndMs % (adjustStep * 1000))
}
tsAdjustedStartMs, _, distributedTSTable, localTSTable := metricstelemetryschema.WhichTSTableToUse(
tsAdjustedStartMs, _, distributedTSTable, localTSTable := telemetrymetrics.WhichTSTableToUse(
samplesAdjustedStartMs, flooredEndMs, false, nil,
)
distributedSamplesTable, localSamplesTable := metricstelemetryschema.WhichSamplesTableToUse(
distributedSamplesTable, localSamplesTable := telemetrymetrics.WhichSamplesTableToUse(
samplesAdjustedStartMs, flooredEndMs,
metrictypes.UnspecifiedType, metrictypes.TimeAggregationUnspecified, false, nil,
)
@@ -362,7 +362,7 @@ func alignedMetricWindow(startMs, endMs int64) (
func (m *module) buildSamplesTblFingerprintSubQuery(metricNames []string, samplesTable string, flooredStart, flooredEnd uint64) *sqlbuilder.SelectBuilder {
fpSB := sqlbuilder.NewSelectBuilder()
fpSB.Select("DISTINCT fingerprint")
fpSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, samplesTable))
fpSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, samplesTable))
fpSB.Where(
fpSB.In("metric_name", sqlbuilder.List(metricNames)),
fpSB.GE("unix_milli", flooredStart),
@@ -376,7 +376,7 @@ func (m *module) buildSamplesTblFingerprintSubQuery(metricNames []string, sample
func (m *module) buildReducedSamplesTblFingerprintSubQuery(metricNames []string, flooredStart, flooredEnd uint64) *sqlbuilder.SelectBuilder {
lastSB := sqlbuilder.NewSelectBuilder()
lastSB.Select("reduced_fingerprint")
lastSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedLastTableName))
lastSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedLastTableName))
lastSB.Where(
lastSB.In("metric_name", sqlbuilder.List(metricNames)),
lastSB.GE("unix_milli", flooredStart),
@@ -385,7 +385,7 @@ func (m *module) buildReducedSamplesTblFingerprintSubQuery(metricNames []string,
sumSB := sqlbuilder.NewSelectBuilder()
sumSB.Select("reduced_fingerprint")
sumSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedSumTableName))
sumSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedSumTableName))
sumSB.Where(
sumSB.In("metric_name", sqlbuilder.List(metricNames)),
sumSB.GE("unix_milli", flooredStart),
@@ -478,7 +478,7 @@ func (m *module) getEarliestMetricTime(ctx context.Context, metricNames []string
sb := sqlbuilder.NewSelectBuilder()
sb.Select("min(first_reported_unix_milli) AS min_first_reported")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.Where(sb.In("metric_name", sqlbuilder.List(metricNames)))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -504,7 +504,7 @@ func (m *module) getMetricsExistence(ctx context.Context, metricNames []string)
sb := sqlbuilder.NewSelectBuilder()
sb.Select("metric_name", "count(*) AS cnt")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.Where(sb.In("metric_name", sqlbuilder.List(metricNames)))
sb.GroupBy("metric_name")
@@ -549,7 +549,7 @@ func (m *module) getAttributesExistence(ctx context.Context, metricNames, attrNa
}
sb := sqlbuilder.NewSelectBuilder()
sb.Select("attr_name", "count(*) AS cnt")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.Where(
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.In("attr_name", sqlbuilder.List(attrNames)),
@@ -656,7 +656,7 @@ func (m *module) getMetadata(
rawSrc := sqlbuilder.NewSelectBuilder()
rawSrc.Select("labels", "unix_milli")
rawSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTableName))
rawSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTableName))
rawSrc.Where(
rawSrc.In("metric_name", sqlbuilder.List(metricNames)),
rawSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -669,7 +669,7 @@ func (m *module) getMetadata(
reducedSrc := sqlbuilder.NewSelectBuilder()
reducedSrc.Select("labels", "unix_milli")
reducedSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedSrc.Where(
reducedSrc.In("metric_name", sqlbuilder.List(metricNames)),
reducedSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -683,7 +683,7 @@ func (m *module) getMetadata(
// Inner query reads over the union of raw + reduced series.
innerSB.From(innerSB.BuilderAs(sqlbuilder.UnionAll(rawSrc, reducedSrc), "series"))
} else {
innerSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTableName))
innerSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTableName))
innerSB.Where(
innerSB.In("metric_name", sqlbuilder.List(metricNames)),
innerSB.GE("unix_milli", tsAdjustedStartMs),
@@ -870,7 +870,7 @@ func (m *module) getPerGroupDistinctCounts(
rawSrc := sqlbuilder.NewSelectBuilder()
rawSrc.Select("labels")
rawSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTimeSeriesTbl))
rawSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTimeSeriesTbl))
rawSrc.Where(
rawSrc.In("metric_name", sqlbuilder.List(metricNames)),
rawSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -883,7 +883,7 @@ func (m *module) getPerGroupDistinctCounts(
reducedSrc := sqlbuilder.NewSelectBuilder()
reducedSrc.Select("labels")
reducedSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedSrc.Where(
reducedSrc.In("metric_name", sqlbuilder.List(metricNames)),
reducedSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -896,7 +896,7 @@ func (m *module) getPerGroupDistinctCounts(
sb.From(sb.BuilderAs(sqlbuilder.UnionAll(rawSrc, reducedSrc), "series"))
} else {
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTimeSeriesTbl))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTimeSeriesTbl))
sb.Where(
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GE("unix_milli", tsAdjustedStartMs),

View File

@@ -7,7 +7,7 @@ import (
"strings"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -72,7 +72,7 @@ func (m *module) getPerGroupHostStatusCounts(
rawSrc := sqlbuilder.NewSelectBuilder()
rawSrc.Select("labels")
rawSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTimeSeriesTableName))
rawSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTimeSeriesTableName))
rawSrc.Where(
rawSrc.In("metric_name", sqlbuilder.List(metricNames)),
rawSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -85,7 +85,7 @@ func (m *module) getPerGroupHostStatusCounts(
reducedSrc := sqlbuilder.NewSelectBuilder()
reducedSrc.Select("labels")
reducedSrc.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedSrc.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedSrc.Where(
reducedSrc.In("metric_name", sqlbuilder.List(metricNames)),
reducedSrc.GE("unix_milli", tsAdjustedStartMs),
@@ -101,7 +101,7 @@ func (m *module) getPerGroupHostStatusCounts(
fpSB := m.buildSamplesTblFingerprintSubQuery(metricNames, localSamplesTable, samplesStartMs, flooredEndMs)
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTimeSeriesTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTimeSeriesTableName))
sb.Where(
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.GE("unix_milli", tsAdjustedStartMs),
@@ -357,7 +357,7 @@ func (m *module) getActiveHostsQuery(metricNames []string, hostNameAttr string,
sb := sqlbuilder.NewSelectBuilder()
sb.Distinct()
sb.Select("attr_string_value")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.Where(
sb.In("metric_name", sqlbuilder.List(metricNames)),
sb.E("attr_name", hostNameAttr),

View File

@@ -9,7 +9,7 @@ import (
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/modules/inframonitoring"
"github.com/SigNoz/signoz/pkg/querier"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
@@ -40,8 +40,8 @@ func NewModule(
providerSettings factory.ProviderSettings,
cfg inframonitoring.Config,
) inframonitoring.Module {
fieldMapper := metricstelemetryschema.NewFieldMapper()
condBuilder := metricstelemetryschema.NewConditionBuilder(fieldMapper)
fieldMapper := telemetrymetrics.NewFieldMapper()
condBuilder := telemetrymetrics.NewConditionBuilder(fieldMapper)
return &module{
telemetryStore: telemetryStore,
telemetryMetadataStore: telemetryMetadataStore,

View File

@@ -7,7 +7,7 @@ import (
"strings"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -186,7 +186,7 @@ func (m *module) getPerGroupNodeConditionCounts(
// Step-floor bounds + resolve tables in one shot to match QB v5 querier.
samplesStartMs, flooredEndMs, tsAdjustedStartMs, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
// ----- timeSeriesFPs -----
timeSeriesFPs := sqlbuilder.NewSelectBuilder()
@@ -200,7 +200,7 @@ func (m *module) getPerGroupNodeConditionCounts(
)
}
timeSeriesFPs.Select(timeSeriesFPsSelectCols...)
timeSeriesFPs.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
timeSeriesFPs.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
timeSeriesFPs.Where(
timeSeriesFPs.E("metric_name", nodeConditionMetricName),
timeSeriesFPs.GE("unix_milli", tsAdjustedStartMs),
@@ -237,7 +237,7 @@ func (m *module) getPerGroupNodeConditionCounts(
latestConditionPerNode.Select(latestConditionPerNodeSelectCols...)
latestConditionPerNode.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN time_series_fps AS tsfp ON samples.fingerprint = tsfp.fingerprint",
metricstelemetryschema.DBName, distributedSamplesTable,
telemetrymetrics.DBName, distributedSamplesTable,
))
latestConditionPerNode.Where(
latestConditionPerNode.E("samples.metric_name", nodeConditionMetricName),

View File

@@ -8,7 +8,7 @@ import (
"time"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -284,7 +284,7 @@ func (m *module) getPerGroupPodStatusCounts(
mergedFilterExpr := mergeFilterExpressions(userFilterExpr, buildPageGroupsFilterExpr(pageGroups))
samplesStartMs, flooredEndMs, tsAdjustedStart, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
// Build the merged filter clause once; it's identical across the three fps
// CTEs, and buildFilterClause hits the metadata store + parses the
@@ -313,7 +313,7 @@ func (m *module) getPerGroupPodStatusCounts(
)
}
phaseFps.Select(phaseFpsCols...)
phaseFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
phaseFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
phaseFps.Where(
phaseFps.E("metric_name", podPhaseMetricName),
phaseFps.GE("unix_milli", tsAdjustedStart),
@@ -340,7 +340,7 @@ func (m *module) getPerGroupPodStatusCounts(
phasePerPod.Select(phasePerPodCols...)
phasePerPod.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN phase_fps AS fps ON samples.fingerprint = fps.fingerprint",
metricstelemetryschema.DBName, distributedSamplesTable,
telemetrymetrics.DBName, distributedSamplesTable,
))
phasePerPod.Where(
phasePerPod.E("samples.metric_name", podPhaseMetricName),
@@ -357,7 +357,7 @@ func (m *module) getPerGroupPodStatusCounts(
"fingerprint",
fmt.Sprintf("JSONExtractString(labels, %s) AS pod_uid", podReasonFps.Var(podUIDAttrKey)),
)
podReasonFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
podReasonFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
podReasonFps.Where(
podReasonFps.E("metric_name", podStatusReasonMetricName),
podReasonFps.GE("unix_milli", tsAdjustedStart),
@@ -377,7 +377,7 @@ func (m *module) getPerGroupPodStatusCounts(
)
podReasonPerPod.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN pod_reason_fps AS fps ON samples.fingerprint = fps.fingerprint",
metricstelemetryschema.DBName, distributedSamplesTable,
telemetrymetrics.DBName, distributedSamplesTable,
))
podReasonPerPod.Where(
podReasonPerPod.E("samples.metric_name", podStatusReasonMetricName),
@@ -396,7 +396,7 @@ func (m *module) getPerGroupPodStatusCounts(
fmt.Sprintf("JSONExtractString(labels, %s) AS container_name", containerReasonFps.Var(containerNameAttrKey)),
fmt.Sprintf("JSONExtractString(labels, %s) AS reason", containerReasonFps.Var(containerStatusReasonAttrKey)),
)
containerReasonFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
containerReasonFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
containerReasonFps.Where(
containerReasonFps.E("metric_name", containerStatusReasonMetricName),
containerReasonFps.GE("unix_milli", tsAdjustedStart),
@@ -432,7 +432,7 @@ func (m *module) getPerGroupPodStatusCounts(
)
containerInner.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN container_reason_fps AS fps ON samples.fingerprint = fps.fingerprint",
metricstelemetryschema.DBName, distributedSamplesTable,
telemetrymetrics.DBName, distributedSamplesTable,
))
containerInner.Where(
containerInner.E("samples.metric_name", containerStatusReasonMetricName),
@@ -611,7 +611,7 @@ func (m *module) getPerGroupPodRestartCounts(
mergedFilterExpr := mergeFilterExpressions(userFilterExpr, buildPageGroupsFilterExpr(pageGroups))
samplesStartMs, flooredEndMs, tsAdjustedStart, _, localTimeSeriesTable, distributedSamplesTable, _ := alignedMetricWindow(start, end)
valueCol := metricstelemetryschema.ValueColumnForSamplesTable(distributedSamplesTable)
valueCol := telemetrymetrics.ValueColumnForSamplesTable(distributedSamplesTable)
var (
filterClause *sqlbuilder.WhereClause
@@ -637,7 +637,7 @@ func (m *module) getPerGroupPodRestartCounts(
)
}
restartFps.Select(restartFpsCols...)
restartFps.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTimeSeriesTable))
restartFps.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTimeSeriesTable))
restartFps.Where(
restartFps.E("metric_name", containerRestartsMetricName),
restartFps.GE("unix_milli", tsAdjustedStart),
@@ -667,7 +667,7 @@ func (m *module) getPerGroupPodRestartCounts(
containerRestarts.Select(containerRestartsCols...)
containerRestarts.From(fmt.Sprintf(
"%s.%s AS samples INNER JOIN restart_fps AS fps ON samples.fingerprint = fps.fingerprint",
metricstelemetryschema.DBName, distributedSamplesTable,
telemetrymetrics.DBName, distributedSamplesTable,
))
containerRestarts.Where(
containerRestarts.E("samples.metric_name", containerRestartsMetricName),

View File

@@ -19,8 +19,8 @@ import (
"github.com/SigNoz/signoz/pkg/modules/dashboard"
"github.com/SigNoz/signoz/pkg/modules/metricsexplorer"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/dashboardtypes"
@@ -49,8 +49,8 @@ type module struct {
// NewModule constructs the metrics module with the provided dependencies.
func NewModule(ts telemetrystore.TelemetryStore, telemetryMetadataStore telemetrytypes.MetadataStore, cache cache.Cache, ruleStore ruletypes.RuleStore, dashboardModule dashboard.Module, fl flagger.Flagger, providerSettings factory.ProviderSettings, cfg metricsexplorer.Config) metricsexplorer.Module {
fieldMapper := metricstelemetryschema.NewFieldMapper()
condBuilder := metricstelemetryschema.NewConditionBuilder(fieldMapper)
fieldMapper := telemetrymetrics.NewFieldMapper()
condBuilder := telemetrymetrics.NewConditionBuilder(fieldMapper)
return &module{
telemetryStore: ts,
fieldMapper: fieldMapper,
@@ -89,7 +89,7 @@ func (m *module) listMeterMetrics(ctx context.Context, params *metricsexplorerty
"argMax(temporality, unix_milli) AS temporality",
"argMax(is_monotonic, unix_milli) AS is_monotonic",
)
sb.From(fmt.Sprintf("%s.%s", metertelemetryschema.DBName, metertelemetryschema.SamplesTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymeter.DBName, telemetrymeter.SamplesTableName))
if params.Start != nil && params.End != nil {
sb.Where(sb.Between("unix_milli", *params.Start, *params.End))
@@ -160,11 +160,11 @@ func (m *module) listMetrics(ctx context.Context, orgID valuer.UUID, params *met
hasRange := params.Start != nil && params.End != nil
if hasRange {
var distributedTsTable string
rangeStart, rangeEnd, distributedTsTable, _ = metricstelemetryschema.WhichTSTableToUse(uint64(*params.Start), uint64(*params.End), false, nil)
rawSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTsTable))
rangeStart, rangeEnd, distributedTsTable, _ = telemetrymetrics.WhichTSTableToUse(uint64(*params.Start), uint64(*params.End), false, nil)
rawSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTsTable))
rawSB.Where(rawSB.Between("unix_milli", rangeStart, rangeEnd))
} else {
rawSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV41weekTableName))
rawSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV41weekTableName))
}
applyListFilters(rawSB)
@@ -174,7 +174,7 @@ func (m *module) listMetrics(ctx context.Context, orgID valuer.UUID, params *met
if reductionEnabled {
reducedSB := sqlbuilder.NewSelectBuilder()
reducedSB.Select("DISTINCT metric_name")
reducedSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
if hasRange {
reducedSB.Where(reducedSB.Between("unix_milli", rangeStart, rangeEnd))
}
@@ -512,7 +512,7 @@ func (m *module) CheckMetricExists(ctx context.Context, orgID valuer.UUID, metri
sb := sqlbuilder.NewSelectBuilder()
sb.Select("count(*) > 0 as metricExists")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.Where(sb.E("metric_name", metricName))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -534,7 +534,7 @@ func (m *module) HasNonSigNozMetrics(ctx context.Context) (bool, error) {
sb := sqlbuilder.NewSelectBuilder()
sb.Select("count(*) > 0")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV41weekTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV41weekTableName))
sb.Where("metric_name NOT LIKE 'signoz_%'")
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -570,10 +570,10 @@ func (m *module) InspectMetrics(
return nil, err
}
tsStart, _, tsTable, _ := metricstelemetryschema.WhichTSTableToUse(start, end, false, nil)
tsStart, _, tsTable, _ := telemetrymetrics.WhichTSTableToUse(start, end, false, nil)
tsSb := sqlbuilder.NewSelectBuilder()
tsSb.Select("fingerprint", "labels")
tsSb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, tsTable))
tsSb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, tsTable))
tsSb.Where(
tsSb.E("metric_name", req.MetricName),
tsSb.GE("unix_milli", tsStart),
@@ -625,7 +625,7 @@ func (m *module) InspectMetrics(
samplesSb := sqlbuilder.NewSelectBuilder()
samplesSb.Select("fingerprint", "unix_milli", "value")
samplesSb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4TableName))
samplesSb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4TableName))
samplesSb.Where(
samplesSb.In("fingerprint", fingerprints...),
samplesSb.E("metric_name", req.MetricName),
@@ -717,7 +717,7 @@ func (m *module) fetchUpdatedMetadata(ctx context.Context, orgID valuer.UUID, me
"argMax(temporality, created_at) AS temporality",
"argMax(is_monotonic, created_at) AS is_monotonic",
)
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.UpdatedMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.UpdatedMetadataTableName))
sb.Where(sb.In("metric_name", args...))
sb.GroupBy("metric_name")
@@ -777,7 +777,7 @@ func (m *module) fetchTimeseriesMetadata(ctx context.Context, orgID valuer.UUID,
"anyLast(temporality) AS temporality",
"argMax(is_monotonic, unix_milli) AS is_monotonic",
)
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4TableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4TableName))
sb.Where(sb.In("metric_name", args...))
sb.GroupBy("metric_name")
@@ -891,7 +891,7 @@ func (m *module) checkForLabelInMetric(ctx context.Context, metricName string, l
sb := sqlbuilder.NewSelectBuilder()
sb.Select("count(*) > 0 AS has_label")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.Where(sb.E("metric_name", metricName))
sb.Where(sb.E("attr_name", label))
sb.Limit(1)
@@ -914,7 +914,7 @@ func (m *module) insertMetricsMetadata(ctx context.Context, orgID valuer.UUID, r
createdAt := time.Now().UnixMilli()
ib := sqlbuilder.NewInsertBuilder()
ib.InsertInto(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.UpdatedMetadataTableName))
ib.InsertInto(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.UpdatedMetadataTableName))
ib.Cols("metric_name", "temporality", "is_monotonic", "type", "description", "unit", "created_at")
ib.Values(
req.MetricName,
@@ -1016,9 +1016,9 @@ func (m *module) fetchMetricsStatsWithSamples(
}
}
start, end, distributedTsTable, localTsTable := metricstelemetryschema.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
distributedSamplesTable, _ := metricstelemetryschema.WhichSamplesTableToUse(uint64(req.Start), uint64(req.End), metrictypes.UnspecifiedType, metrictypes.TimeAggregationUnspecified, false, nil)
countExp := metricstelemetryschema.CountExpressionForSamplesTable(distributedSamplesTable)
start, end, distributedTsTable, localTsTable := telemetrymetrics.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
distributedSamplesTable, _ := telemetrymetrics.WhichSamplesTableToUse(uint64(req.Start), uint64(req.End), metrictypes.UnspecifiedType, metrictypes.TimeAggregationUnspecified, false, nil)
countExp := telemetrymetrics.CountExpressionForSamplesTable(distributedSamplesTable)
// Timeseries counts per metric
tsSB := sqlbuilder.NewSelectBuilder()
@@ -1026,7 +1026,7 @@ func (m *module) fetchMetricsStatsWithSamples(
"metric_name",
"uniq(fingerprint) AS timeseries",
)
tsSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTsTable))
tsSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTsTable))
tsSB.Where(tsSB.Between("unix_milli", start, end))
tsSB.Where("NOT startsWith(metric_name, 'signoz')")
if filterWhereClause != nil {
@@ -1040,7 +1040,7 @@ func (m *module) fetchMetricsStatsWithSamples(
"metric_name",
fmt.Sprintf("%s AS samples", countExp),
)
samplesSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedSamplesTable))
samplesSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedSamplesTable))
samplesSB.Where(samplesSB.Between("unix_milli", req.Start, req.End))
samplesSB.Where("NOT startsWith(metric_name, 'signoz')")
@@ -1053,7 +1053,7 @@ func (m *module) fetchMetricsStatsWithSamples(
if filterWhereClause != nil {
fingerprintSB := sqlbuilder.NewSelectBuilder()
fingerprintSB.Select("fingerprint")
fingerprintSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTsTable))
fingerprintSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTsTable))
fingerprintSB.Where(fingerprintSB.Between("unix_milli", start, end))
fingerprintSB.Where("NOT startsWith(metric_name, 'signoz')")
fingerprintSB.AddWhereClause(sqlbuilder.CopyWhereClause(filterWhereClause))
@@ -1064,7 +1064,7 @@ func (m *module) fetchMetricsStatsWithSamples(
} else {
metricNamesSB := sqlbuilder.NewSelectBuilder()
metricNamesSB.Select("DISTINCT metric_name")
metricNamesSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTsTable))
metricNamesSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTsTable))
metricNamesSB.Where(metricNamesSB.Between("unix_milli", start, end))
metricNamesSB.Where("NOT startsWith(metric_name, 'signoz')")
@@ -1090,18 +1090,18 @@ func (m *module) fetchMetricsStatsWithSamples(
// the data lands either of the _reduced_last/_reduced_sum tables
reducedLastSB := sqlbuilder.NewSelectBuilder()
reducedLastSB.Select("metric_name", "uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedLastSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedLastTableName))
reducedLastSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedLastTableName))
reducedLastSB.Where(reducedLastSB.Between("unix_milli", req.Start, req.End))
reducedLastSB.Where("NOT startsWith(metric_name, 'signoz')")
reducedSumSB := sqlbuilder.NewSelectBuilder()
reducedSumSB.Select("metric_name", "uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedSumSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedSumTableName))
reducedSumSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedSumTableName))
reducedSumSB.Where(reducedSumSB.Between("unix_milli", req.Start, req.End))
reducedSumSB.Where("NOT startsWith(metric_name, 'signoz')")
// separate query for reduced series counts
reducedTsSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedTsSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedTsSB.Where(reducedTsSB.Between("unix_milli", start, end))
reducedTsSB.Where("NOT startsWith(metric_name, 'signoz')")
reducedTsSB.GroupBy("metric_name")
@@ -1112,7 +1112,7 @@ func (m *module) fetchMetricsStatsWithSamples(
// samples uses a separate cte with local table
reducedFpSB := sqlbuilder.NewSelectBuilder()
reducedFpSB.Select("fingerprint")
reducedFpSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedLocalTableName))
reducedFpSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedLocalTableName))
reducedFpSB.Where(reducedFpSB.Between("unix_milli", start, end))
reducedFpSB.Where("NOT startsWith(metric_name, 'signoz')")
reducedFpSB.AddWhereClause(sqlbuilder.CopyWhereClause(filterWhereClause))
@@ -1220,11 +1220,11 @@ func (m *module) computeTimeseriesTreemap(ctx context.Context, orgID valuer.UUID
}
}
start, end, distributedTsTable, _ := metricstelemetryschema.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
start, end, distributedTsTable, _ := telemetrymetrics.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
totalTSBuilder := sqlbuilder.NewSelectBuilder()
totalTSBuilder.Select("uniq(fingerprint) AS total_time_series")
totalTSBuilder.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTsTable))
totalTSBuilder.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTsTable))
totalTSBuilder.Where(totalTSBuilder.Between("unix_milli", start, end))
metricsSB := sqlbuilder.NewSelectBuilder()
@@ -1232,7 +1232,7 @@ func (m *module) computeTimeseriesTreemap(ctx context.Context, orgID valuer.UUID
"metric_name",
"uniq(fingerprint) AS total_value",
)
metricsSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTsTable))
metricsSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTsTable))
metricsSB.Where(metricsSB.Between("unix_milli", start, end))
metricsSB.Where("NOT startsWith(metric_name, 'signoz')")
if filterWhereClause != nil {
@@ -1244,7 +1244,7 @@ func (m *module) computeTimeseriesTreemap(ctx context.Context, orgID valuer.UUID
if reductionEnabled {
reducedTotalTSBuilder := sqlbuilder.NewSelectBuilder()
reducedTotalTSBuilder.Select("uniq(fingerprint) AS total_time_series")
reducedTotalTSBuilder.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedTotalTSBuilder.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedTotalTSBuilder.Where(reducedTotalTSBuilder.Between("unix_milli", start, end))
reducedMetricsSB := sqlbuilder.NewSelectBuilder()
@@ -1252,7 +1252,7 @@ func (m *module) computeTimeseriesTreemap(ctx context.Context, orgID valuer.UUID
"metric_name",
"uniq(fingerprint) AS total_value",
)
reducedMetricsSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedMetricsSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedMetricsSB.Where(reducedMetricsSB.Between("unix_milli", start, end))
reducedMetricsSB.Where("NOT startsWith(metric_name, 'signoz')")
if filterWhereClause != nil {
@@ -1338,15 +1338,15 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
}
}
start, end, distributedTsTable, localTsTable := metricstelemetryschema.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
distributedSamplesTable, _ := metricstelemetryschema.WhichSamplesTableToUse(uint64(req.Start), uint64(req.End), metrictypes.UnspecifiedType, metrictypes.TimeAggregationUnspecified, false, nil)
countExp := metricstelemetryschema.CountExpressionForSamplesTable(distributedSamplesTable)
start, end, distributedTsTable, localTsTable := telemetrymetrics.WhichTSTableToUse(uint64(req.Start), uint64(req.End), false, nil)
distributedSamplesTable, _ := telemetrymetrics.WhichSamplesTableToUse(uint64(req.Start), uint64(req.End), metrictypes.UnspecifiedType, metrictypes.TimeAggregationUnspecified, false, nil)
countExp := telemetrymetrics.CountExpressionForSamplesTable(distributedSamplesTable)
candidateLimit := req.Limit + 50
metricCandidatesSB := sqlbuilder.NewSelectBuilder()
metricCandidatesSB.Select("metric_name")
metricCandidatesSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedTsTable))
metricCandidatesSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedTsTable))
metricCandidatesSB.Where("NOT startsWith(metric_name, 'signoz')")
metricCandidatesSB.Where(metricCandidatesSB.Between("unix_milli", start, end))
if filterWhereClause != nil {
@@ -1364,7 +1364,7 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
if reductionEnabled {
reducedCandidatesSB = sqlbuilder.NewSelectBuilder()
reducedCandidatesSB.Select("metric_name")
reducedCandidatesSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedTableName))
reducedCandidatesSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedTableName))
reducedCandidatesSB.Where("NOT startsWith(metric_name, 'signoz')")
reducedCandidatesSB.Where(reducedCandidatesSB.Between("unix_milli", start, end))
if filterWhereClause != nil {
@@ -1379,7 +1379,7 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
totalSamplesSB := sqlbuilder.NewSelectBuilder()
totalSamplesSB.Select(fmt.Sprintf("%s AS total_samples", countExp))
totalSamplesSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedSamplesTable))
totalSamplesSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedSamplesTable))
totalSamplesSB.Where(totalSamplesSB.Between("unix_milli", req.Start, req.End))
sampleCountsSB := sqlbuilder.NewSelectBuilder()
@@ -1387,7 +1387,7 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
"metric_name",
fmt.Sprintf("%s AS samples", countExp),
)
sampleCountsSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, distributedSamplesTable))
sampleCountsSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, distributedSamplesTable))
sampleCountsSB.Where(sampleCountsSB.Between("unix_milli", req.Start, req.End))
sampleCountsSB.Where("metric_name GLOBAL IN (SELECT metric_name FROM __metric_candidates)")
@@ -1397,13 +1397,13 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
if reductionEnabled {
reducedLastSB = sqlbuilder.NewSelectBuilder()
reducedLastSB.Select("metric_name", "uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedLastSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedLastTableName))
reducedLastSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedLastTableName))
reducedLastSB.Where(reducedLastSB.Between("unix_milli", req.Start, req.End))
reducedLastSB.Where("metric_name GLOBAL IN (SELECT metric_name FROM __reduced_metric_candidates)")
reducedSumSB = sqlbuilder.NewSelectBuilder()
reducedSumSB.Select("metric_name", "uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedSumSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedSumTableName))
reducedSumSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedSumTableName))
reducedSumSB.Where(reducedSumSB.Between("unix_milli", req.Start, req.End))
reducedSumSB.Where("metric_name GLOBAL IN (SELECT metric_name FROM __reduced_metric_candidates)")
}
@@ -1411,7 +1411,7 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
if filterWhereClause != nil {
fingerprintSB := sqlbuilder.NewSelectBuilder()
fingerprintSB.Select("fingerprint")
fingerprintSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTsTable))
fingerprintSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, localTsTable))
fingerprintSB.Where(fingerprintSB.Between("unix_milli", start, end))
fingerprintSB.Where("NOT startsWith(metric_name, 'signoz')")
fingerprintSB.AddWhereClause(sqlbuilder.CopyWhereClause(filterWhereClause))
@@ -1423,7 +1423,7 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
if reductionEnabled {
reducedFingerprintSB := sqlbuilder.NewSelectBuilder()
reducedFingerprintSB.Select("fingerprint")
reducedFingerprintSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4ReducedLocalTableName))
reducedFingerprintSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4ReducedLocalTableName))
reducedFingerprintSB.Where(reducedFingerprintSB.Between("unix_milli", start, end))
reducedFingerprintSB.Where("NOT startsWith(metric_name, 'signoz')")
reducedFingerprintSB.AddWhereClause(sqlbuilder.CopyWhereClause(filterWhereClause))
@@ -1457,12 +1457,12 @@ func (m *module) computeSamplesTreemap(ctx context.Context, orgID valuer.UUID, r
// Grand total includes reduced sample volume so percentages stay consistent.
reducedTotalLastSB := sqlbuilder.NewSelectBuilder()
reducedTotalLastSB.Select("uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedTotalLastSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedLastTableName))
reducedTotalLastSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedLastTableName))
reducedTotalLastSB.Where(reducedTotalLastSB.Between("unix_milli", req.Start, req.End))
reducedTotalSumSB := sqlbuilder.NewSelectBuilder()
reducedTotalSumSB.Select("uniq(reduced_fingerprint, unix_milli) AS cnt")
reducedTotalSumSB.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4ReducedSumTableName))
reducedTotalSumSB.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4ReducedSumTableName))
reducedTotalSumSB.Where(reducedTotalSumSB.Between("unix_milli", req.Start, req.End))
reducedTotalSamplesSB := sqlbuilder.NewSelectBuilder()
@@ -1552,7 +1552,7 @@ func (m *module) getMetricDataPoints(ctx context.Context, metricName string) (ui
sb := sqlbuilder.NewSelectBuilder()
sb.Select("sum(count) AS data_points")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4Agg30mTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4Agg30mTableName))
sb.Where(sb.E("metric_name", metricName))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -1574,7 +1574,7 @@ func (m *module) getMetricLastReceived(ctx context.Context, metricName string) (
sb := sqlbuilder.NewSelectBuilder()
sb.Select("MAX(last_reported_unix_milli) AS last_received_time")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.Where(sb.E("metric_name", metricName))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -1599,7 +1599,7 @@ func (m *module) getTotalTimeSeriesForMetricName(ctx context.Context, metricName
sb := sqlbuilder.NewSelectBuilder()
sb.Select("uniq(fingerprint) AS time_series_count")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV41weekTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV41weekTableName))
sb.Where(sb.E("metric_name", metricName))
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
@@ -1623,7 +1623,7 @@ func (m *module) getActiveTimeSeriesForMetricName(ctx context.Context, metricNam
sb := sqlbuilder.NewSelectBuilder()
sb.Select("uniq(fingerprint) AS active_time_series")
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.TimeseriesV4TableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.TimeseriesV4TableName))
sb.Where(sb.E("metric_name", metricName))
sb.Where(sb.GTE("unix_milli", milli))
@@ -1649,7 +1649,7 @@ func (m *module) fetchMetricAttributes(ctx context.Context, metricName string, s
"groupUniqArray(1000)(attr_string_value) AS values",
"uniq(attr_string_value) AS valueCount",
)
sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.AttributesMetadataTableName))
sb.From(fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.AttributesMetadataTableName))
sb.Where(sb.E("metric_name", metricName))
sb.Where("NOT startsWith(attr_name, '__')")

View File

@@ -9,7 +9,7 @@ import (
schemamigrator "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/modules/promote"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
@@ -52,7 +52,7 @@ func (m *module) ListPromotedAndIndexedPaths(ctx context.Context) ([]promotetype
response := []promotetypes.PromotePath{}
for _, path := range promotedPaths {
fullPath := logstelemetryschema.BodyPromotedColumnPrefix + path
fullPath := telemetrylogs.BodyPromotedColumnPrefix + path
path = telemetrytypes.BodyJSONStringSearchPrefix + path
item := promotetypes.PromotePath{
Path: path,
@@ -68,7 +68,7 @@ func (m *module) ListPromotedAndIndexedPaths(ctx context.Context) ([]promotetype
// add the paths that are not promoted but have indexes
for path, indexes := range aggr {
path := strings.TrimPrefix(path, logstelemetryschema.BodyV2ColumnPrefix)
path := strings.TrimPrefix(path, telemetrylogs.BodyV2ColumnPrefix)
path = telemetrytypes.BodyJSONStringSearchPrefix + path
response = append(response, promotetypes.PromotePath{
Path: path,
@@ -108,8 +108,8 @@ func (m *module) createIndexes(ctx context.Context, indexes []schemamigrator.Ind
for _, index := range indexes {
alterStmt := schemamigrator.AlterTableAddIndex{
Database: logstelemetryschema.DBName,
Table: logstelemetryschema.LogsV2LocalTableName,
Database: telemetrylogs.DBName,
Table: telemetrylogs.LogsV2LocalTableName,
Index: index,
}
op := alterStmt.OnCluster(m.telemetryStore.Cluster())
@@ -153,10 +153,10 @@ func (m *module) PromoteAndIndexPaths(
}
}
if len(it.Indexes) > 0 {
parentColumn := logstelemetryschema.LogsV2BodyV2Column
parentColumn := telemetrylogs.LogsV2BodyV2Column
// if the path is already promoted or is being promoted, add it to the promoted column
if _, promoted := existingPromotedPaths[it.Path]; promoted || it.Promote {
parentColumn = logstelemetryschema.LogsV2BodyPromotedColumn
parentColumn = telemetrylogs.LogsV2BodyPromotedColumn
}
for _, index := range it.Indexes {

View File

@@ -11,8 +11,8 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/modules/services"
"github.com/SigNoz/signoz/pkg/querier"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -39,7 +39,7 @@ func (m *module) FetchTopLevelOperations(ctx context.Context, start time.Time, s
ctx = m.withServicesContext(ctx, "FetchTopLevelOperations")
db := m.TelemetryStore.ClickhouseDB()
query := fmt.Sprintf("SELECT name, serviceName, max(time) as ts FROM %s.%s WHERE time >= @start", tracestelemetryschema.DBName, tracestelemetryschema.TopLevelOperationsTableName)
query := fmt.Sprintf("SELECT name, serviceName, max(time) as ts FROM %s.%s WHERE time >= @start", telemetrytraces.DBName, telemetrytraces.TopLevelOperationsTableName)
args := []any{clickhouse.Named("start", start)}
if len(services) > 0 {
query += " AND serviceName IN @services"

View File

@@ -22,13 +22,6 @@ import (
"github.com/SigNoz/signoz/pkg/variables"
)
type Handler interface {
QueryRange(rw http.ResponseWriter, req *http.Request)
QueryRangePreview(rw http.ResponseWriter, req *http.Request)
QueryRawStream(rw http.ResponseWriter, req *http.Request)
ReplaceVariables(rw http.ResponseWriter, req *http.Request)
}
type handler struct {
set factory.ProviderSettings
analytics analytics.Analytics

View File

@@ -15,14 +15,6 @@ import (
"github.com/SigNoz/signoz/pkg/valuer"
)
// BucketCache is the interface for bucket-based caching.
type BucketCache interface {
// cached portion + list of gaps to fetch
GetMissRanges(ctx context.Context, orgID valuer.UUID, q qbtypes.Query, step qbtypes.Step) (cached *qbtypes.Result, missing []*qbtypes.TimeRange)
// store fresh buckets for future hits
Put(ctx context.Context, orgID valuer.UUID, q qbtypes.Query, step qbtypes.Step, fresh *qbtypes.Result)
}
// bucketCache implements the BucketCache interface.
type bucketCache struct {
cache cache.Cache

View File

@@ -11,8 +11,8 @@ import (
"github.com/ClickHouse/clickhouse-go/v2"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -277,12 +277,12 @@ func (q *builderQuery[T]) narrowWindowByTraceID(ctx context.Context, fromMS, toM
return fromMS, toMS, true, ""
}
traceIDs, found := tracestelemetryschema.ExtractTraceIDsFromFilter(q.spec.Filter.Expression)
traceIDs, found := telemetrytraces.ExtractTraceIDsFromFilter(q.spec.Filter.Expression)
if !found || len(traceIDs) == 0 {
return fromMS, toMS, true, ""
}
finder := tracestelemetryschema.NewTraceTimeRangeFinder(q.telemetryStore)
finder := telemetrytraces.NewTraceTimeRangeFinder(q.telemetryStore)
traceStart, traceEnd, exists, err := finder.GetTraceTimeRangeMulti(ctx, traceIDs)
if err != nil {
return fromMS, toMS, true, ""

View File

@@ -100,20 +100,9 @@ func (q *chSQLQuery) renderVars(query string, vars map[string]qbtypes.VariableIt
return newQuery.String(), nil
}
func (q *chSQLQuery) render(ctx context.Context) (string, error) {
rendered, err := q.renderVars(q.query.Query, q.vars, q.fromMS, q.toMS)
if err != nil {
return "", err
}
querybuilder.LogIfStatementIsNotValid(ctx, q.logger, rendered)
return rendered, nil
}
// Statement renders the SQL without executing it, for the preview path.
func (q *chSQLQuery) Statement(ctx context.Context) (*qbtypes.Statement, error) {
rendered, err := q.render(ctx)
func (q *chSQLQuery) Statement(_ context.Context) (*qbtypes.Statement, error) {
rendered, err := q.renderVars(q.query.Query, q.vars, q.fromMS, q.toMS)
if err != nil {
return nil, err
}
@@ -135,7 +124,7 @@ func (q *chSQLQuery) Execute(ctx context.Context) (*qbtypes.Result, error) {
elapsed += p.Elapsed
}))
query, err := q.render(ctx)
query, err := q.renderVars(q.query.Query, q.vars, q.fromMS, q.toMS)
if err != nil {
return nil, err
}
@@ -146,11 +135,11 @@ func (q *chSQLQuery) Execute(ctx context.Context) (*qbtypes.Result, error) {
}
defer rows.Close()
// TODO: map the errors from ClickHouse to our error types
payload, err := consume(rows, q.kind, nil, qbtypes.Step{}, q.query.Name)
if err != nil {
return nil, err
}
return &qbtypes.Result{
Type: q.kind,
Value: payload,

View File

@@ -6,18 +6,18 @@ import (
"time"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/valuer"
)
func (q *querier) Collect(ctx context.Context, _ valuer.UUID) (map[string]any, error) {
stats := make(map[string]any)
tracesTable := fmt.Sprintf("%s.%s", tracestelemetryschema.DBName, tracestelemetryschema.SpanIndexV3TableName)
logsTable := fmt.Sprintf("%s.%s", logstelemetryschema.DBName, logstelemetryschema.LogsV2TableName)
metricsTable := fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4TableName)
tracesTable := fmt.Sprintf("%s.%s", telemetrytraces.DBName, telemetrytraces.SpanIndexV3TableName)
logsTable := fmt.Sprintf("%s.%s", telemetrylogs.DBName, telemetrylogs.LogsV2TableName)
metricsTable := fmt.Sprintf("%s.%s", telemetrymetrics.DBName, telemetrymetrics.SamplesV4TableName)
var (
traces uint64

View File

@@ -9,6 +9,12 @@ import (
const DefaultMaxConcurrentQueries = 8
type SkipResourceFingerprint struct {
Enabled bool `yaml:"enabled" mapstructure:"enabled"`
// If count of fingerprint is above threshold, skip the fingerprint subquery and filter on main table instead.
Threshold uint64 `yaml:"threshold" mapstructure:"threshold"`
}
// Config represents the configuration for the querier.
type Config struct {
// CacheTTL is the TTL for cached query results
@@ -17,6 +23,8 @@ type Config struct {
FluxInterval time.Duration `yaml:"flux_interval" mapstructure:"flux_interval"`
// MaxConcurrentQueries is the maximum number of concurrent queries for missing ranges
MaxConcurrentQueries int `yaml:"max_concurrent_queries" mapstructure:"max_concurrent_queries"`
// SkipResourceFingerprint configures when the resource fingerprint subquery is skipped in favor of main-table filtering.
SkipResourceFingerprint SkipResourceFingerprint `yaml:"skip_resource_fingerprint" mapstructure:"skip_resource_fingerprint"`
// LogTraceIDWindowPadding is the padding added to narrowed down timerange from trace summary to logs with trace_id filter.
LogTraceIDWindowPadding time.Duration `yaml:"log_trace_id_window_padding" mapstructure:"log_trace_id_window_padding"`
}
@@ -31,7 +39,11 @@ func newConfig() factory.Config {
// Default values
CacheTTL: 168 * time.Hour,
FluxInterval: 5 * time.Minute,
MaxConcurrentQueries: DefaultMaxConcurrentQueries,
MaxConcurrentQueries: DefaultMaxConcurrentQueries,
SkipResourceFingerprint: SkipResourceFingerprint{
Enabled: false,
Threshold: 100000,
},
LogTraceIDWindowPadding: 5 * time.Minute,
}
}
@@ -47,6 +59,9 @@ func (c Config) Validate() error {
if c.MaxConcurrentQueries <= 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "max_concurrent_queries must be positive, got %v", c.MaxConcurrentQueries)
}
if c.SkipResourceFingerprint.Enabled && c.SkipResourceFingerprint.Threshold == 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "skip_resource_fingerprint.threshold must be > 0 when enabled")
}
if c.LogTraceIDWindowPadding < 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "log_trace_id_window_padding must not be negative, got %v", c.LogTraceIDWindowPadding)
}

35
pkg/querier/interfaces.go Normal file
View File

@@ -0,0 +1,35 @@
package querier
import (
"context"
"net/http"
"github.com/SigNoz/signoz/pkg/statsreporter"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
)
// Querier interface defines the contract for querying data.
type Querier interface {
QueryRange(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest) (*qbtypes.QueryRangeResponse, error)
QueryRawStream(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, client *qbtypes.RawStream)
statsreporter.StatsCollector
// QueryRangePreview validates and renders the queries without executing them.
QueryRangePreview(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, opts qbtypes.QueryRangePreviewOptions) (*qbtypes.QueryRangePreviewResponse, error)
}
// BucketCache is the interface for bucket-based caching.
type BucketCache interface {
// cached portion + list of gaps to fetch
GetMissRanges(ctx context.Context, orgID valuer.UUID, q qbtypes.Query, step qbtypes.Step) (cached *qbtypes.Result, missing []*qbtypes.TimeRange)
// store fresh buckets for future hits
Put(ctx context.Context, orgID valuer.UUID, q qbtypes.Query, step qbtypes.Step, fresh *qbtypes.Result)
}
type Handler interface {
QueryRange(rw http.ResponseWriter, req *http.Request)
// QueryRangePreview is the dry-run endpoint: validate and render without executing.
QueryRangePreview(rw http.ResponseWriter, req *http.Request)
QueryRawStream(rw http.ResponseWriter, req *http.Request)
ReplaceVariables(rw http.ResponseWriter, req *http.Request)
}

View File

@@ -21,8 +21,6 @@ import (
"github.com/SigNoz/signoz/pkg/prometheus"
"github.com/SigNoz/signoz/pkg/query-service/utils"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statsreporter"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
@@ -37,15 +35,6 @@ var (
intervalWarn = "Query %s is requesting aggregation interval %v seconds, which is smaller than the minimum allowed interval of %v seconds for selected time range. Using the minimum instead"
)
// Querier interface defines the contract for querying data.
type Querier interface {
QueryRange(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest) (*qbtypes.QueryRangeResponse, error)
QueryRawStream(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, client *qbtypes.RawStream)
statsreporter.StatsCollector
// QueryRangePreview validates and renders the queries without executing them.
QueryRangePreview(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, opts qbtypes.QueryRangePreviewOptions) (*qbtypes.QueryRangePreviewResponse, error)
}
type querier struct {
logger *slog.Logger
fl flagger.Flagger
@@ -71,7 +60,12 @@ func New(
telemetryStore telemetrystore.TelemetryStore,
metadataStore telemetrytypes.MetadataStore,
promEngine prometheus.Prometheus,
builders *statementbuilder.Builders,
traceStmtBuilder qbtypes.StatementBuilder[qbtypes.TraceAggregation],
logStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation],
auditStmtBuilder qbtypes.StatementBuilder[qbtypes.LogAggregation],
metricStmtBuilder qbtypes.StatementBuilder[qbtypes.MetricAggregation],
meterStmtBuilder qbtypes.StatementBuilder[qbtypes.MetricAggregation],
traceOperatorStmtBuilder qbtypes.TraceOperatorStatementBuilder,
bucketCache BucketCache,
flagger flagger.Flagger,
logTraceIDWindowPadding time.Duration,
@@ -81,21 +75,18 @@ func New(
if maxConcurrentQueries <= 0 {
maxConcurrentQueries = DefaultMaxConcurrentQueries
}
if builders == nil {
builders = &statementbuilder.Builders{}
}
return &querier{
logger: querierSettings.Logger(),
fl: flagger,
telemetryStore: telemetryStore,
metadataStore: metadataStore,
promEngine: promEngine,
traceStmtBuilder: builders.Trace,
logStmtBuilder: builders.Log,
auditStmtBuilder: builders.Audit,
metricStmtBuilder: builders.Metric,
meterStmtBuilder: builders.Meter,
traceOperatorStmtBuilder: builders.TraceOperator,
traceStmtBuilder: traceStmtBuilder,
logStmtBuilder: logStmtBuilder,
auditStmtBuilder: auditStmtBuilder,
metricStmtBuilder: metricStmtBuilder,
meterStmtBuilder: meterStmtBuilder,
traceOperatorStmtBuilder: traceOperatorStmtBuilder,
bucketCache: bucketCache,
liveDataRefresh: 5 * time.Second,
builderConfig: builderConfig{

View File

@@ -11,7 +11,6 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
@@ -49,7 +48,12 @@ func TestQueryRange_MetricTypeMissing(t *testing.T) {
nil, // telemetryStore
metadataStore,
nil, // prometheus
nil, // builders
nil, // traceStmtBuilder
nil, // logStmtBuilder
nil, // auditStmtBuilder
nil, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
flaggertest.New(t), // flagger
0, // logTraceIDWindowPadding
@@ -115,12 +119,17 @@ func TestQueryRange_MetricTypeFromStore(t *testing.T) {
providerSettings,
telemetryStore,
metadataStore,
nil, // prometheus
&statementbuilder.Builders{Metric: &mockMetricStmtBuilder{}},
nil, // bucketCache
flaggertest.New(t), // flagger
0, // logTraceIDWindowPadding
0, // maxConcurrentQueries
nil, // prometheus
nil, // traceStmtBuilder
nil, // logStmtBuilder
nil, // auditStmtBuilder
&mockMetricStmtBuilder{}, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
flaggertest.New(t), // flagger
0, // logTraceIDWindowPadding
0, // maxConcurrentQueries
)
req := &qbtypes.QueryRangeRequest{

View File

@@ -1,52 +0,0 @@
package queriertest
import (
"context"
"github.com/SigNoz/signoz/pkg/querier"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
)
// MockQuerier implements querier.Querier for testing. Each behavior can be
// overridden via a function field; unset fields return empty, non-error values.
type MockQuerier struct {
QueryRangeFunc func(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest) (*qbtypes.QueryRangeResponse, error)
QueryRawStreamFunc func(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, client *qbtypes.RawStream)
QueryRangePreviewFunc func(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, opts qbtypes.QueryRangePreviewOptions) (*qbtypes.QueryRangePreviewResponse, error)
CollectFunc func(ctx context.Context, orgID valuer.UUID) (map[string]any, error)
}
var _ querier.Querier = (*MockQuerier)(nil)
// NewMockQuerier creates a new MockQuerier.
func NewMockQuerier() *MockQuerier {
return &MockQuerier{}
}
func (m *MockQuerier) QueryRange(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest) (*qbtypes.QueryRangeResponse, error) {
if m.QueryRangeFunc != nil {
return m.QueryRangeFunc(ctx, orgID, req)
}
return &qbtypes.QueryRangeResponse{}, nil
}
func (m *MockQuerier) QueryRawStream(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, client *qbtypes.RawStream) {
if m.QueryRawStreamFunc != nil {
m.QueryRawStreamFunc(ctx, orgID, req, client)
}
}
func (m *MockQuerier) QueryRangePreview(ctx context.Context, orgID valuer.UUID, req *qbtypes.QueryRangeRequest, opts qbtypes.QueryRangePreviewOptions) (*qbtypes.QueryRangePreviewResponse, error) {
if m.QueryRangePreviewFunc != nil {
return m.QueryRangePreviewFunc(ctx, orgID, req, opts)
}
return &qbtypes.QueryRangePreviewResponse{}, nil
}
func (m *MockQuerier) Collect(ctx context.Context, orgID valuer.UUID) (map[string]any, error) {
if m.CollectFunc != nil {
return m.CollectFunc(ctx, orgID)
}
return map[string]any{}, nil
}

View File

@@ -3,44 +3,192 @@ package signozquerier
import (
"context"
"github.com/SigNoz/signoz/pkg/cache"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/prometheus"
"github.com/SigNoz/signoz/pkg/querier"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetadata"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
)
// NewFactory creates a new factory for the signoz querier provider. The query
// stack (metadata store, statement builders, bucket cache) is assembled once in
// signoz.go and injected here.
// NewFactory creates a new factory for the signoz querier provider.
func NewFactory(
telemetryStore telemetrystore.TelemetryStore,
prometheus prometheus.Prometheus,
metadataStore telemetrytypes.MetadataStore,
builders *statementbuilder.Builders,
bucketCache querier.BucketCache,
cache cache.Cache,
flagger flagger.Flagger,
) factory.ProviderFactory[querier.Querier, querier.Config] {
return factory.NewProviderFactory(
factory.MustNewName("signoz"),
func(
_ context.Context,
ctx context.Context,
settings factory.ProviderSettings,
cfg querier.Config,
) (querier.Querier, error) {
return querier.New(
settings,
telemetryStore,
metadataStore,
prometheus,
builders,
bucketCache,
flagger,
cfg.LogTraceIDWindowPadding,
cfg.MaxConcurrentQueries,
), nil
return newProvider(ctx, settings, cfg, telemetryStore, prometheus, cache, flagger)
},
)
}
func newProvider(
_ context.Context,
settings factory.ProviderSettings,
cfg querier.Config,
telemetryStore telemetrystore.TelemetryStore,
prometheus prometheus.Prometheus,
cache cache.Cache,
flagger flagger.Flagger,
) (querier.Querier, error) {
// Create telemetry metadata store
telemetryMetadataStore := telemetrymetadata.NewTelemetryMetaStore(
settings,
telemetryStore,
telemetrytraces.DBName,
telemetrytraces.TagAttributesV2TableName,
telemetrytraces.SpanAttributesKeysTblName,
telemetrytraces.SpanIndexV3TableName,
telemetrymetrics.DBName,
telemetrymetrics.AttributesMetadataTableName,
telemetrymeter.DBName,
telemetrymeter.SamplesAgg1dTableName,
telemetrylogs.DBName,
telemetrylogs.LogsV2TableName,
telemetrylogs.TagAttributesV2TableName,
telemetrylogs.LogAttributeKeysTblName,
telemetrylogs.LogResourceKeysTblName,
telemetryaudit.DBName,
telemetryaudit.AuditLogsTableName,
telemetryaudit.TagAttributesTableName,
telemetryaudit.LogAttributeKeysTblName,
telemetryaudit.LogResourceKeysTblName,
telemetrymetadata.DBName,
telemetrymetadata.AttributesMetadataTableName,
telemetrymetadata.ColumnEvolutionMetadataTableName,
flagger,
)
// Create trace statement builder
traceFieldMapper := telemetrytraces.NewFieldMapper()
traceConditionBuilder := telemetrytraces.NewConditionBuilder(traceFieldMapper)
traceAggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, traceFieldMapper, traceConditionBuilder, flagger)
traceStmtBuilder := telemetrytraces.NewTraceQueryStatementBuilder(
settings,
telemetryMetadataStore,
traceFieldMapper,
traceConditionBuilder,
traceAggExprRewriter,
telemetryStore,
flagger,
cfg.SkipResourceFingerprint.Enabled,
cfg.SkipResourceFingerprint.Threshold,
)
// Create trace operator statement builder
traceOperatorStmtBuilder := telemetrytraces.NewTraceOperatorStatementBuilder(
settings,
telemetryMetadataStore,
traceFieldMapper,
traceConditionBuilder,
traceStmtBuilder,
traceAggExprRewriter,
flagger,
)
// Create log statement builder
logFieldMapper := telemetrylogs.NewFieldMapper(flagger)
logConditionBuilder := telemetrylogs.NewConditionBuilder(logFieldMapper, flagger)
logAggExprRewriter := querybuilder.NewAggExprRewriter(
settings,
telemetrylogs.DefaultFullTextColumn,
logFieldMapper,
logConditionBuilder,
flagger,
)
logStmtBuilder := telemetrylogs.NewLogQueryStatementBuilder(
settings,
telemetryMetadataStore,
logFieldMapper,
logConditionBuilder,
logAggExprRewriter,
telemetrylogs.DefaultFullTextColumn,
flagger,
telemetryStore,
cfg.SkipResourceFingerprint.Enabled,
cfg.SkipResourceFingerprint.Threshold,
)
// Create audit statement builder
auditFieldMapper := telemetryaudit.NewFieldMapper()
auditConditionBuilder := telemetryaudit.NewConditionBuilder(auditFieldMapper)
auditAggExprRewriter := querybuilder.NewAggExprRewriter(
settings,
telemetryaudit.DefaultFullTextColumn,
auditFieldMapper,
auditConditionBuilder,
flagger,
)
auditStmtBuilder := telemetryaudit.NewAuditQueryStatementBuilder(
settings,
telemetryMetadataStore,
auditFieldMapper,
auditConditionBuilder,
auditAggExprRewriter,
telemetryaudit.DefaultFullTextColumn,
flagger,
)
// Create metric statement builder
metricFieldMapper := telemetrymetrics.NewFieldMapper()
metricConditionBuilder := telemetrymetrics.NewConditionBuilder(metricFieldMapper)
metricStmtBuilder := telemetrymetrics.NewMetricQueryStatementBuilder(
settings,
telemetryMetadataStore,
metricFieldMapper,
metricConditionBuilder,
flagger,
)
// Create meter statement builder
meterStmtBuilder := telemetrymeter.NewMeterQueryStatementBuilder(
settings,
telemetryMetadataStore,
metricFieldMapper,
metricConditionBuilder,
metricStmtBuilder,
)
// Create bucket cache
bucketCache := querier.NewBucketCache(
settings,
cache,
cfg.CacheTTL,
cfg.FluxInterval,
)
// Create and return the querier
return querier.New(
settings,
telemetryStore,
telemetryMetadataStore,
prometheus,
traceStmtBuilder,
logStmtBuilder,
auditStmtBuilder,
metricStmtBuilder,
meterStmtBuilder,
traceOperatorStmtBuilder,
bucketCache,
flagger,
cfg.LogTraceIDWindowPadding,
cfg.MaxConcurrentQueries,
), nil
}

View File

@@ -11,7 +11,6 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/modules/thirdpartyapi"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/queryparser"
"log/slog"
@@ -1093,8 +1092,6 @@ func (aH *APIHandler) queryDashboardVarsV2(w http.ResponseWriter, r *http.Reques
return
}
querybuilder.LogIfStatementIsNotValid(r.Context(), aH.logger, query)
dashboardVars, err := aH.reader.QueryDashboardVars(r.Context(), query)
if err != nil {
RespondError(w, &model.ApiError{Typ: model.ErrorBadData, Err: err}, nil)

View File

@@ -18,8 +18,6 @@ import (
"github.com/SigNoz/signoz/pkg/querier/signozquerier"
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/SigNoz/signoz/pkg/sqlstore/sqlstoretest"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/telemetrymetadata"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
"github.com/stretchr/testify/require"
@@ -107,11 +105,7 @@ func NewTestManager(t *testing.T, testOpts *TestManagerOptions) *Manager {
}
// Create querier with test values
metadataStore := telemetrymetadata.NewTelemetryMetaStore(providerSettings, telemetryStore, flagger)
builders, err := statementbuilder.NewFactory(telemetryStore, metadataStore, flagger).New(context.Background(), providerSettings, statementbuilder.Config{})
require.NoError(t, err)
bucketCache := querier.NewBucketCache(providerSettings, cache, 0, 0)
providerFactory := signozquerier.NewFactory(telemetryStore, prometheus, metadataStore, builders, bucketCache, flagger)
providerFactory := signozquerier.NewFactory(telemetryStore, prometheus, cache, flagger)
mockQuerier, err := providerFactory.New(context.Background(), providerSettings, querier.Config{})
require.NoError(t, err)

View File

@@ -8,11 +8,11 @@ import (
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querier"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/logsstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/metricsstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/tracesstatementbuilder"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
"github.com/stretchr/testify/require"
@@ -24,7 +24,7 @@ func prepareQuerierForMetrics(t *testing.T, telemetryStore telemetrystore.Teleme
providerSettings := instrumentationtest.New().ToProviderSettings()
metadataStore := telemetrytypestest.NewMockMetadataStore()
fl, err := flagger.New(
flagger, err := flagger.New(
context.Background(),
instrumentationtest.New().ToProviderSettings(),
flagger.Config{},
@@ -32,16 +32,29 @@ func prepareQuerierForMetrics(t *testing.T, telemetryStore telemetrystore.Teleme
)
require.NoError(t, err)
metricStmtBuilder := metricsstatementbuilder.New(providerSettings, metadataStore, fl)
metricFieldMapper := telemetrymetrics.NewFieldMapper()
metricConditionBuilder := telemetrymetrics.NewConditionBuilder(metricFieldMapper)
metricStmtBuilder := telemetrymetrics.NewMetricQueryStatementBuilder(
providerSettings,
metadataStore,
metricFieldMapper,
metricConditionBuilder,
flagger,
)
return querier.New(
providerSettings,
telemetryStore,
metadataStore,
nil, // prometheus
&statementbuilder.Builders{Metric: metricStmtBuilder},
nil, // traceStmtBuilder
nil, // logStmtBuilder
nil, // auditStmtBuilder
metricStmtBuilder,
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
fl,
flagger,
0,
0, // maxConcurrentQueries (0 means default)
), metadataStore
@@ -60,15 +73,40 @@ func prepareQuerierForLogs(t *testing.T, telemetryStore telemetrystore.Telemetry
metadataStore.KeysMap = keysMap
fl := flaggertest.New(t)
logStmtBuilder := logsstatementbuilder.New(providerSettings, metadataStore, nil, fl, false, 100000)
logFieldMapper := telemetrylogs.NewFieldMapper(fl)
logConditionBuilder := telemetrylogs.NewConditionBuilder(logFieldMapper, fl)
logAggExprRewriter := querybuilder.NewAggExprRewriter(
providerSettings,
telemetrylogs.DefaultFullTextColumn,
logFieldMapper,
logConditionBuilder,
fl,
)
logStmtBuilder := telemetrylogs.NewLogQueryStatementBuilder(
providerSettings,
metadataStore,
logFieldMapper,
logConditionBuilder,
logAggExprRewriter,
telemetrylogs.DefaultFullTextColumn,
fl,
nil,
false,
100000,
)
return querier.New(
providerSettings,
telemetryStore,
metadataStore,
nil, // prometheus
&statementbuilder.Builders{Log: logStmtBuilder},
nil, // bucketCache
nil, // prometheus
nil, // traceStmtBuilder
logStmtBuilder, // logStmtBuilder
nil, // auditStmtBuilder
nil, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
fl,
5*time.Minute, // logTraceIDWindowPadding
0, // maxConcurrentQueries (0 means default)
@@ -88,16 +126,36 @@ func prepareQuerierForTraces(t *testing.T, telemetryStore telemetrystore.Telemet
}
metadataStore.KeysMap = keysMap
// Create trace statement builder
traceFieldMapper := telemetrytraces.NewFieldMapper()
traceConditionBuilder := telemetrytraces.NewConditionBuilder(traceFieldMapper)
fl := flaggertest.New(t)
traceStmtBuilder, _ := tracesstatementbuilder.New(providerSettings, metadataStore, telemetryStore, fl, false, 100000)
traceAggExprRewriter := querybuilder.NewAggExprRewriter(providerSettings, nil, traceFieldMapper, traceConditionBuilder, fl)
traceStmtBuilder := telemetrytraces.NewTraceQueryStatementBuilder(
providerSettings,
metadataStore,
traceFieldMapper,
traceConditionBuilder,
traceAggExprRewriter,
telemetryStore,
fl,
false,
100000,
)
return querier.New(
providerSettings,
telemetryStore,
metadataStore,
nil, // prometheus
&statementbuilder.Builders{Trace: traceStmtBuilder},
nil, // bucketCache
nil, // prometheus
traceStmtBuilder, // traceStmtBuilder
nil, // logStmtBuilder
nil, // auditStmtBuilder
nil, // metricStmtBuilder
nil, // meterStmtBuilder
nil, // traceOperatorStmtBuilder
nil, // bucketCache
fl,
0,
0, // maxConcurrentQueries (0 means default)

View File

@@ -1,76 +0,0 @@
package querybuilder
import (
"context"
"log/slog"
"strings"
chparser "github.com/AfterShip/clickhouse-sql-parser/parser"
"github.com/SigNoz/signoz/pkg/errors"
)
// internalDatabases hold server metadata, credentials and grants rather than telemetry.
var internalDatabases = map[string]struct{}{
"system": {},
"information_schema": {},
}
// The parser's grammar has gaps against SQL that ClickHouse itself accepts. See TestErrIfStatementIsNotValid_ShouldPassButFails.
func ErrIfStatementIsNotValid(query string) (err error) {
defer func() {
// The parser has a history of panicking on malformed input rather than returning an error.
if recovered := recover(); recovered != nil {
err = errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid ClickHouse SQL (recovered): %v", recovered)
}
}()
stmts, parseErr := chparser.NewParser(query).ParseStmts()
if parseErr != nil {
// Wrapped rather than formatted in, so that callers can recover the parser's *ParseError and read the position off it.
return errors.WrapInvalidInputf(parseErr, errors.CodeInvalidInput, "invalid ClickHouse SQL: %s", parseErr.Error())
}
if len(stmts) != 1 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "ClickHouse SQL must contain exactly one statement, found %d statements", len(stmts))
}
selectQuery, ok := stmts[0].(*chparser.SelectQuery)
if !ok {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "only SELECT statements are allowed in ClickHouse SQL queries")
}
visitor := &chparser.DefaultASTVisitor{Visit: func(node chparser.Expr) error {
switch expr := node.(type) {
case *chparser.TableFunctionExpr:
// Source table functions remain usable in ClickHouse read-only mode.
return errors.NewInvalidInputf(errors.CodeInvalidInput, "ClickHouse table functions are not allowed in SQL queries: %s", chparser.Format(expr.Name))
case *chparser.TableIdentifier:
// Reading these is unaffected by ClickHouse read-only mode.
if expr.Database == nil {
return nil
}
if _, ok := internalDatabases[strings.ToLower(expr.Database.Name)]; ok {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "the ClickHouse %s database is not allowed in SQL queries", expr.Database.Name)
}
case *chparser.SettingExpr:
// A query-level setting takes precedence over the context setting.
if strings.EqualFold(expr.Name.Name, "readonly") {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "the ClickHouse readonly setting cannot be overridden")
}
}
return nil
}}
return selectQuery.Accept(visitor)
}
// TODO(@therealpandey): remove this and move to ErrIfStatementIsNotValid.
func LogIfStatementIsNotValid(ctx context.Context, logger *slog.Logger, query string) {
if err := ErrIfStatementIsNotValid(query); err != nil {
logger.WarnContext(ctx, "clickhouse sql is not valid", errors.Attr(err), slog.String("query", query))
}
}

View File

@@ -1,154 +0,0 @@
package querybuilder
import (
"testing"
chparser "github.com/AfterShip/clickhouse-sql-parser/parser"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestErrIfStatementIsNotValid_Pass(t *testing.T) {
testCases := []struct {
name string
query string
}{
// Shapes a telemetry read is allowed to take.
{"Select", "SELECT region AS r, zone FROM metrics WHERE metric_name = 'cpu' GROUP BY region, zone"},
{"TrailingSemicolon", "SELECT count() FROM signoz_logs.distributed_logs_v2;"},
{"CommonTableExpression", "WITH t AS (SELECT fingerprint FROM signoz_metrics.time_series_v4) SELECT * FROM t"},
{"Join", "SELECT * FROM t1 LEFT JOIN t2 ON t1.a = t2.b"},
{"GlobalIn", "SELECT a FROM t WHERE a GLOBAL IN (SELECT b FROM t2)"},
{"Union", "SELECT * FROM t UNION ALL SELECT * FROM t2"},
{"Intersect", "SELECT * FROM t INTERSECT SELECT * FROM t2"},
{"WindowFunction", "SELECT sum(v) OVER (PARTITION BY a ORDER BY t) FROM t"},
{"UnrelatedSetting", "SELECT * FROM t SETTINGS max_threads = 4"},
{"TerminatedBlockComment", "SELECT /* keep me */ count() FROM t"},
{"BlockCommentMarkerInsideStringLiteral", "SELECT count() FROM t WHERE body = '/* not a comment'"},
// The parser used to loop forever on this; it now reads the comment to the end of
// the input, so this doubles as a canary for that regression.
{"TrailingUnterminatedBlockComment", "SELECT count() FROM t /* unterminated"},
// The rule keys on the database, not on the table name.
{"TableNamedSystemInTelemetryDatabase", "SELECT * FROM signoz_logs.system"},
{"SignedLiteralAfterClosingParen", "SELECT (toUnixTimestamp(now()) - 3600)*1000000000"},
// order by interval
{"OrderByInterval", "SELECT toStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS interval ORDER BY interval"},
{"OrderByIntervalAndDirection", "SELECT toStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS `interval` ORDER BY `interval` ASC"},
{"TrimFunction", "SELECT trimBoth('/api/endpoint/', '/');"},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
err := ErrIfStatementIsNotValid(testCase.query)
assert.NoError(t, err)
})
}
}
func TestErrIfStatementIsNotValid_Fail(t *testing.T) {
testCases := []struct {
name string
query string
}{
// Not a single statement, or not a statement at all.
{"Empty", ""},
{"UnterminatedBlockCommentOnly", "/* x"},
{"Unparseable", "SELECT FROM WHERE"},
{"MultipleStatements", "SELECT 1; DROP TABLE signoz_logs.logs_v2"},
// Parses, but is not a SELECT.
{"Drop", "DROP TABLE signoz_logs.logs_v2"},
{"Insert", "INSERT INTO signoz_logs.logs_v2 SELECT * FROM signoz_logs.logs_v2"},
{"AlterDelete", "ALTER TABLE signoz_logs.logs_v2 DELETE WHERE 1 = 1"},
{"CreateTable", "CREATE TABLE evil (a Int) ENGINE = Memory"},
{"Grant", "GRANT ALL ON *.* TO admin"},
{"Set", "SET readonly = 0"},
// These the parser rejects outright rather than classifying.
{"ShowGrants", "SHOW GRANTS"},
{"IntoOutfile", "SELECT * FROM t INTO OUTFILE '/tmp/x.csv'"},
// Table functions, which read through something other than a telemetry table.
{"UrlTableFunction", "SELECT * FROM url('http://attacker.example/x', CSV, 'a String')"},
{"FileTableFunction", "SELECT * FROM file('/etc/passwd', CSV, 'a String')"},
{"ExecutableTableFunction", "SELECT * FROM executable('script.sh', CSV, 'a String')"},
{"TableFunctionInJoin", "SELECT * FROM t1 JOIN url('http://x', CSV, 'a String') u ON 1 = 1"},
{"TableFunctionInCommonTableExpression", "WITH c AS (SELECT * FROM url('http://x', CSV, 'a String')) SELECT * FROM c"},
{"TableFunctionInWhereSubquery", "SELECT * FROM t WHERE a IN (SELECT * FROM file('/etc/passwd', CSV, 'a String'))"},
{"TableFunctionInUnion", "SELECT * FROM t UNION ALL SELECT * FROM url('http://x', CSV, 'a String')"},
// Internal databases, which hold grants and server metadata rather than telemetry.
{"SystemUsers", "SELECT * FROM system.users"},
{"SystemUppercase", "SELECT * FROM SYSTEM.USERS"},
{"SystemQuoted", "SELECT count() FROM `system`.`tables`"},
{"SystemInSubquery", "SELECT * FROM (SELECT name FROM system.parts)"},
{"SystemInJoin", "SELECT * FROM signoz_logs.distributed_logs_v2 AS l JOIN system.users AS u ON 1 = 1"},
{"SystemInIntersect", "SELECT * FROM t INTERSECT SELECT * FROM system.users"},
{"InformationSchema", "SELECT * FROM information_schema.tables"},
// A query-level setting takes precedence over the one the caller applies.
{"ReadonlySettingOverride", "SELECT * FROM t SETTINGS readonly = 0"},
{"ReadonlySettingOverrideAmongOthers", "SELECT * FROM t SETTINGS max_threads = 4, readonly = 0"},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
err := ErrIfStatementIsNotValid(testCase.query)
assert.Error(t, err)
})
}
}
// Queries the parser cannot read. ClickHouse runs all of them.
func TestErrIfStatementIsNotValid_ShouldPassButFails(t *testing.T) {
testCases := []struct {
name string
query string
// The construct the parser stops after, which is the one it cannot read.
expectedStopsAfter string
// The same construct written so the parser accepts it.
fix string
}{
{
name: "IntervalAliasInOrderBy",
query: "SELECT toStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS interval, resource_string_service$$name AS `service.name`, attributes_string['http.route'] AS `http.route`, quantile(0.95)(duration_nano) / 1000000000 AS value FROM signoz_traces.distributed_signoz_index_v3 WHERE resource_string_service$$name = 'svc-a' AND resources_string['deployment.environment'] = 'dev' AND attributes_string['http.route'] = '/v1' AND http_method = 'POST' AND timestamp BETWEEN toDateTime(1784601720) AND toDateTime(1784602620) AND ts_bucket_start BETWEEN 1784601720 - 1800 AND 1784602620 GROUP BY `service.name`, `http.route`, interval ORDER BY interval ASC",
expectedStopsAfter: "ORDER BY interval ASC",
fix: "SELECT count() AS interval FROM t ORDER BY `interval` ASC",
},
{
name: "IntervalAliasInOrderByDesc",
query: "SELECT count() AS value, toStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS interval, serviceName, resourceTagsMap['deployment.environment'] AS environment, exceptionStacktrace FROM signoz_traces.distributed_signoz_error_index_v2 WHERE exceptionType != 'OSError' AND resourceTagsMap['deployment.environment'] = 'staging' AND timestamp BETWEEN toDateTime(1785186300) AND toDateTime(1785186600) GROUP BY serviceName, interval, environment, exceptionStacktrace ORDER BY interval DESC",
expectedStopsAfter: "ORDER BY interval DESC",
fix: "SELECT count() AS interval FROM t ORDER BY `interval` DESC",
},
{
name: "StandardTrimSyntax",
query: "SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 5 MINUTE) AS interval, resources_string['host.name'] as host_name, toFloat64(countIf( lower(trim(BOTH ' ' FROM replaceOne( JSONExtractString(body, 'Action'), 'health_status: ', '' ))) IN ('unhealthy','starting','failing') )) as value FROM signoz_logs.distributed_logs_v2 WHERE timestamp BETWEEN 1784602320000000000 AND 1784602620000000000 AND ts_bucket_start BETWEEN 1784602320 - 300 AND 1784602620 AND JSONExtractString(body, 'Type') = 'container' AND JSONExtractString(body, 'Actor', 'Attributes', 'name') IS NOT NULL AND resources_string['host.name'] IS NOT NULL AND resources_string['host.name'] = 'aihub-nightly' GROUP BY interval, host_name ORDER BY interval, host_name",
expectedStopsAfter: "trim(BOTH '",
fix: "SELECT trimBoth(replaceOne( JSONExtractString(body, 'Action'), 'health_status: ', '' ), ' ')",
},
{
name: "SignedLiteralAfterClosingParen",
query: "SELECT now() AS ts, toFloat64(count()) AS value FROM ( SELECT attributes_string['TableName'] AS T, attributes_string['MissingId'] AS M, max(fromUnixTimestamp64Nano(timestamp)) AS last_seen, dateDiff('minute', min(fromUnixTimestamp64Nano(timestamp)), max(fromUnixTimestamp64Nano(timestamp))) AS age_min FROM signoz_logs.distributed_logs_v2 WHERE body='missing_map_record' AND timestamp >= (toUnixTimestamp(now())-3600)*1000000000 GROUP BY T, M ) WHERE age_min >= 20 AND last_seen >= now() - toIntervalMinute(8)",
expectedStopsAfter: "(toUnixTimestamp(now())",
fix: "SELECT (toUnixTimestamp(now()) - 3600) * 1000000000",
},
{
name: "SignedLiteralAfterClosingParenMinimal",
query: "SELECT (1)-1",
expectedStopsAfter: "(1)",
fix: "SELECT (1) - 1",
},
}
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
err := ErrIfStatementIsNotValid(testCase.query)
var parseErr *chparser.ParseError
require.ErrorAs(t, err, &parseErr, "expected a parser failure rather than a rule violation")
// The parser reports the offset it stopped at, which sits just past the construct
// it choked on, so the text leading up to it is what needs looking at.
consumed := testCase.query[:parseErr.Pos]
assert.Equal(t, testCase.expectedStopsAfter, consumed[max(0, len(consumed)-len(testCase.expectedStopsAfter)):])
assert.NoError(t, ErrIfStatementIsNotValid(testCase.fix))
})
}
}

1
pkg/querybuilder/init.go Normal file
View File

@@ -0,0 +1 @@
package querybuilder

View File

@@ -39,7 +39,6 @@ import (
"github.com/SigNoz/signoz/pkg/sqlmigrator"
"github.com/SigNoz/signoz/pkg/sqlschema"
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statsreporter"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/tokenizer"
@@ -98,9 +97,6 @@ type Config struct {
// Querier config
Querier querier.Config `mapstructure:"querier"`
// StatementBuilder config
StatementBuilder statementbuilder.Config `mapstructure:"statementbuilder"`
// Ruler config
Ruler ruler.Config `mapstructure:"ruler"`
@@ -170,7 +166,6 @@ func NewConfig(ctx context.Context, logger *slog.Logger, resolverConfig config.R
prometheus.NewConfigFactory(),
alertmanager.NewConfigFactory(),
querier.NewConfigFactory(),
statementbuilder.NewConfigFactory(),
ruler.NewConfigFactory(),
emailing.NewConfigFactory(),
sharder.NewConfigFactory(),

View File

@@ -56,7 +56,6 @@ import (
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/SigNoz/signoz/pkg/sqlstore/sqlitesqlstore"
"github.com/SigNoz/signoz/pkg/sqlstore/sqlstorehook"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statsreporter"
"github.com/SigNoz/signoz/pkg/statsreporter/analyticsstatsreporter"
"github.com/SigNoz/signoz/pkg/statsreporter/noopstatsreporter"
@@ -70,7 +69,6 @@ import (
"github.com/SigNoz/signoz/pkg/types/alertmanagertypes"
"github.com/SigNoz/signoz/pkg/types/dashboardtypes"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/version"
"github.com/SigNoz/signoz/pkg/web"
"github.com/SigNoz/signoz/pkg/web/noopweb"
@@ -287,9 +285,9 @@ func NewStatsReporterProviderFactories(aggregator statsreporter.Aggregator, orgG
)
}
func NewQuerierProviderFactories(telemetryStore telemetrystore.TelemetryStore, prometheus prometheus.Prometheus, metadataStore telemetrytypes.MetadataStore, builders *statementbuilder.Builders, bucketCache querier.BucketCache, flagger flagger.Flagger) factory.NamedMap[factory.ProviderFactory[querier.Querier, querier.Config]] {
func NewQuerierProviderFactories(telemetryStore telemetrystore.TelemetryStore, prometheus prometheus.Prometheus, cache cache.Cache, flagger flagger.Flagger) factory.NamedMap[factory.ProviderFactory[querier.Querier, querier.Config]] {
return factory.MustNewNamedMap(
signozquerier.NewFactory(telemetryStore, prometheus, metadataStore, builders, bucketCache, flagger),
signozquerier.NewFactory(telemetryStore, prometheus, cache, flagger),
)
}

View File

@@ -48,10 +48,14 @@ import (
"github.com/SigNoz/signoz/pkg/sqlmigrator"
"github.com/SigNoz/signoz/pkg/sqlschema"
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/SigNoz/signoz/pkg/statementbuilder"
"github.com/SigNoz/signoz/pkg/statsreporter"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetadata"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
pkgtokenizer "github.com/SigNoz/signoz/pkg/tokenizer"
"github.com/SigNoz/signoz/pkg/types/authtypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -92,29 +96,6 @@ type SigNoz struct {
MeterReporter meterreporter.Reporter
}
// newQueryStack assembles the query stack once: the shared telemetry metadata
// store, the per-signal statement builders, and the bucket cache. The metadata
// store is returned so callers can reuse the single instance downstream.
func newQueryStack(
ctx context.Context,
settings factory.ProviderSettings,
config Config,
telemetryStore telemetrystore.TelemetryStore,
cache cache.Cache,
fl flagger.Flagger,
) (telemetrytypes.MetadataStore, *statementbuilder.Builders, querier.BucketCache, error) {
metadataStore := telemetrymetadata.NewTelemetryMetaStore(settings, telemetryStore, fl)
builders, err := statementbuilder.NewFactory(telemetryStore, metadataStore, fl).New(ctx, settings, config.StatementBuilder)
if err != nil {
return nil, nil, nil, err
}
bucketCache := querier.NewBucketCache(settings, cache, config.Querier.CacheTTL, config.Querier.FluxInterval)
return metadataStore, builders, bucketCache, nil
}
func New(
ctx context.Context,
config Config,
@@ -273,19 +254,12 @@ func New(
return nil, err
}
// Assemble the query stack (metadata store, statement builders, bucket cache) once,
// and reuse the single metadata store everywhere downstream.
telemetryMetadataStore, builders, bucketCache, err := newQueryStack(ctx, providerSettings, config, telemetrystore, cache, flagger)
if err != nil {
return nil, err
}
// Initialize querier from the available querier provider factories
querier, err := factory.NewProviderFromNamedMap(
ctx,
providerSettings,
config.Querier,
NewQuerierProviderFactories(telemetrystore, prometheus, telemetryMetadataStore, builders, bucketCache, flagger),
NewQuerierProviderFactories(telemetrystore, prometheus, cache, flagger),
config.Querier.Provider(),
)
if err != nil {
@@ -453,6 +427,35 @@ func New(
return nil, err
}
// Initialize telemetry metadata store
// TODO: consolidate other telemetrymetadata.NewTelemetryMetaStore initializations to reuse this instance instead.
telemetryMetadataStore := telemetrymetadata.NewTelemetryMetaStore(
providerSettings,
telemetrystore,
telemetrytraces.DBName,
telemetrytraces.TagAttributesV2TableName,
telemetrytraces.SpanAttributesKeysTblName,
telemetrytraces.SpanIndexV3TableName,
telemetrymetrics.DBName,
telemetrymetrics.AttributesMetadataTableName,
telemetrymeter.DBName,
telemetrymeter.SamplesAgg1dTableName,
telemetrylogs.DBName,
telemetrylogs.LogsV2TableName,
telemetrylogs.TagAttributesV2TableName,
telemetrylogs.LogAttributeKeysTblName,
telemetrylogs.LogResourceKeysTblName,
telemetryaudit.DBName,
telemetryaudit.AuditLogsTableName,
telemetryaudit.TagAttributesTableName,
telemetryaudit.LogAttributeKeysTblName,
telemetryaudit.LogResourceKeysTblName,
telemetrymetadata.DBName,
telemetrymetadata.AttributesMetadataTableName,
telemetrymetadata.ColumnEvolutionMetadataTableName,
flagger,
)
global, err := factory.NewProviderFromNamedMap(
ctx,
providerSettings,

View File

@@ -1,26 +0,0 @@
package auditstatementbuilder
import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/audittelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// New builds the audit statement builder, internalizing the FieldMapper,
// ConditionBuilder, and AggExprRewriter.
func New(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fl flagger.Flagger,
) qbtypes.StatementBuilder[qbtypes.LogAggregation] {
fm := audittelemetryschema.NewFieldMapper()
cb := audittelemetryschema.NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, audittelemetryschema.DefaultFullTextColumn, fm, cb, fl)
return NewAuditQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, audittelemetryschema.DefaultFullTextColumn, fl,
)
}

View File

@@ -1,41 +0,0 @@
package statementbuilder
import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
)
// SkipResourceFingerprint configures when the resource fingerprint subquery is skipped in favor of main-table filtering.
type SkipResourceFingerprint struct {
Enabled bool `yaml:"enabled" mapstructure:"enabled"`
// If count of fingerprint is above threshold, skip the fingerprint subquery and filter on main table instead.
Threshold uint64 `yaml:"threshold" mapstructure:"threshold"`
}
// Config represents the configuration for the statement builders.
type Config struct {
// SkipResourceFingerprint configures when the resource fingerprint subquery is skipped in favor of main-table filtering.
SkipResourceFingerprint SkipResourceFingerprint `yaml:"skip_resource_fingerprint" mapstructure:"skip_resource_fingerprint"`
}
// NewConfigFactory creates a new config factory for the statement builders.
func NewConfigFactory() factory.ConfigFactory {
return factory.NewConfigFactory(factory.MustNewName("statementbuilder"), newConfig)
}
func newConfig() factory.Config {
return Config{
SkipResourceFingerprint: SkipResourceFingerprint{
Enabled: false,
Threshold: 100000,
},
}
}
// Validate validates the configuration.
func (c Config) Validate() error {
if c.SkipResourceFingerprint.Enabled && c.SkipResourceFingerprint.Threshold == 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "skip_resource_fingerprint.threshold must be > 0 when enabled")
}
return nil
}

View File

@@ -1,31 +0,0 @@
package logsstatementbuilder
import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// New builds the logs statement builder, internalizing the FieldMapper,
// ConditionBuilder, and AggExprRewriter.
func New(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
telemetryStore telemetrystore.TelemetryStore,
fl flagger.Flagger,
skipResourceFingerprintEnabled bool,
skipResourceFingerprintThreshold uint64,
) qbtypes.StatementBuilder[qbtypes.LogAggregation] {
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, logstelemetryschema.DefaultFullTextColumn, fm, cb, fl)
return NewLogQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, logstelemetryschema.DefaultFullTextColumn,
fl, telemetryStore, skipResourceFingerprintEnabled, skipResourceFingerprintThreshold,
)
}

View File

@@ -1,21 +0,0 @@
package meterstatementbuilder
import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/statementbuilder/metricsstatementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// New builds the meter statement builder. It reuses the metrics FieldMapper and
// ConditionBuilder, and delegates the final SELECT to the provided metrics statement builder.
func New(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
metricsStatementBuilder *metricsstatementbuilder.StatementBuilder,
) qbtypes.StatementBuilder[qbtypes.MetricAggregation] {
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
return NewMeterQueryStatementBuilder(settings, metadataStore, fm, cb, metricsStatementBuilder)
}

View File

@@ -1,20 +0,0 @@
package metricsstatementbuilder
import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// New builds the metrics statement builder, internalizing the FieldMapper and
// ConditionBuilder. It returns the concrete type so the meter builder can reuse it.
func New(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
fl flagger.Flagger,
) *StatementBuilder {
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
return NewMetricQueryStatementBuilder(settings, metadataStore, fm, cb, fl)
}

View File

@@ -1,59 +0,0 @@
package statementbuilder
import (
"context"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/statementbuilder/auditstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/logsstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/meterstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/metricsstatementbuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/tracesstatementbuilder"
"github.com/SigNoz/signoz/pkg/telemetrystore"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// Builders bundles the per-signal statement builders that make up the query stack.
type Builders struct {
Trace qbtypes.StatementBuilder[qbtypes.TraceAggregation]
TraceOperator qbtypes.TraceOperatorStatementBuilder
Log qbtypes.StatementBuilder[qbtypes.LogAggregation]
Audit qbtypes.StatementBuilder[qbtypes.LogAggregation]
Metric qbtypes.StatementBuilder[qbtypes.MetricAggregation]
Meter qbtypes.StatementBuilder[qbtypes.MetricAggregation]
}
// NewFactory returns a provider factory that builds the per-signal statement builders once.
func NewFactory(
telemetryStore telemetrystore.TelemetryStore,
metadataStore telemetrytypes.MetadataStore,
fl flagger.Flagger,
) factory.ProviderFactory[*Builders, Config] {
return factory.NewProviderFactory(
factory.MustNewName("signoz"),
func(_ context.Context, settings factory.ProviderSettings, cfg Config) (*Builders, error) {
traceStmtBuilder, traceOperatorStmtBuilder := tracesstatementbuilder.New(
settings, metadataStore, telemetryStore, fl,
cfg.SkipResourceFingerprint.Enabled, cfg.SkipResourceFingerprint.Threshold,
)
logStmtBuilder := logsstatementbuilder.New(
settings, metadataStore, telemetryStore, fl,
cfg.SkipResourceFingerprint.Enabled, cfg.SkipResourceFingerprint.Threshold,
)
auditStmtBuilder := auditstatementbuilder.New(settings, metadataStore, fl)
metricStmtBuilder := metricsstatementbuilder.New(settings, metadataStore, fl)
meterStmtBuilder := meterstatementbuilder.New(settings, metadataStore, metricStmtBuilder)
return &Builders{
Trace: traceStmtBuilder,
TraceOperator: traceOperatorStmtBuilder,
Log: logStmtBuilder,
Audit: auditStmtBuilder,
Metric: metricStmtBuilder,
Meter: meterStmtBuilder,
}, nil
},
)
}

View File

@@ -1,35 +0,0 @@
package tracesstatementbuilder
import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// New builds the trace and trace-operator statement builders, internalizing the
// FieldMapper, ConditionBuilder, and AggExprRewriter.
func New(
settings factory.ProviderSettings,
metadataStore telemetrytypes.MetadataStore,
telemetryStore telemetrystore.TelemetryStore,
fl flagger.Flagger,
skipResourceFingerprintEnabled bool,
skipResourceFingerprintThreshold uint64,
) (qbtypes.StatementBuilder[qbtypes.TraceAggregation], qbtypes.TraceOperatorStatementBuilder) {
fm := tracestelemetryschema.NewFieldMapper()
cb := tracestelemetryschema.NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(settings, nil, fm, cb, fl)
traceStmtBuilder := NewTraceQueryStatementBuilder(
settings, metadataStore, fm, cb, aggExprRewriter, telemetryStore, fl,
skipResourceFingerprintEnabled, skipResourceFingerprintThreshold,
)
traceOperatorStmtBuilder := NewTraceOperatorStatementBuilder(
settings, metadataStore, fm, cb, traceStmtBuilder, aggExprRewriter, fl,
)
return traceStmtBuilder, traceOperatorStmtBuilder
}

View File

@@ -1,4 +1,4 @@
package audittelemetryschema
package telemetryaudit
import (
"context"

View File

@@ -1,4 +1,4 @@
package audittelemetryschema
package telemetryaudit
import (
schema "github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator"
@@ -127,14 +127,3 @@ var auditLogColumns = map[string]*schema.Column{
"scope_version": {Name: "scope_version", Type: schema.ColumnTypeString},
"scope_string": {Name: "scope_string", Type: schema.MapColumnType{KeyType: schema.LowCardinalityColumnType{ElementType: schema.ColumnTypeString}, ValueType: schema.ColumnTypeString}},
}
const (
DBName = "signoz_audit"
AuditLogsTableName = "distributed_logs"
AuditLogsLocalTableName = "logs"
TagAttributesTableName = "distributed_tag_attributes"
TagAttributesLocalTableName = "tag_attributes"
LogAttributeKeysTblName = "distributed_logs_attribute_keys"
LogResourceKeysTblName = "distributed_logs_resource_keys"
LogsResourceTableName = "distributed_logs_resource"
)

View File

@@ -1,4 +1,4 @@
package audittelemetryschema
package telemetryaudit
import (
"context"

View File

@@ -1,4 +1,4 @@
package auditstatementbuilder
package telemetryaudit
import (
"context"
@@ -10,8 +10,7 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/resourcefilter"
"github.com/SigNoz/signoz/pkg/telemetryschema/audittelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryresourcefilter"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
@@ -39,12 +38,12 @@ func NewAuditQueryStatementBuilder(
fullTextColumn *telemetrytypes.TelemetryFieldKey,
flagger flagger.Flagger,
) *auditQueryStatementBuilder {
auditSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/audittelemetryschema")
auditSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryaudit")
resourceFilterStmtBuilder := resourcefilter.New[qbtypes.LogAggregation](
resourceFilterStmtBuilder := telemetryresourcefilter.New[qbtypes.LogAggregation](
settings,
audittelemetryschema.DBName,
audittelemetryschema.LogsResourceTableName,
DBName,
LogsResourceTableName,
telemetrytypes.SignalLogs,
telemetrytypes.SourceAudit,
metadataStore,
@@ -192,8 +191,8 @@ func (b *auditQueryStatementBuilder) adjustKeys(ctx context.Context, keys map[st
}
func (b *auditQueryStatementBuilder) adjustKey(key *telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) []string {
if _, ok := audittelemetryschema.IntrinsicFields[key.Name]; ok {
intrinsicField := audittelemetryschema.IntrinsicFields[key.Name]
if _, ok := IntrinsicFields[key.Name]; ok {
intrinsicField := IntrinsicFields[key.Name]
return querybuilder.AdjustKey(key, keys, &intrinsicField)
}
return querybuilder.AdjustKey(key, keys, nil)
@@ -220,26 +219,26 @@ func (b *auditQueryStatementBuilder) buildListQuery(
cteArgs = append(cteArgs, args)
}
sb.Select(audittelemetryschema.TimestampColumn)
sb.SelectMore(audittelemetryschema.IDColumn)
sb.Select(TimestampColumn)
sb.SelectMore(IDColumn)
if len(query.SelectFields) == 0 {
sb.SelectMore(audittelemetryschema.TraceIDColumn)
sb.SelectMore(audittelemetryschema.SpanIDColumn)
sb.SelectMore(audittelemetryschema.TraceFlagsColumn)
sb.SelectMore(audittelemetryschema.SeverityTextColumn)
sb.SelectMore(audittelemetryschema.SeverityNumberColumn)
sb.SelectMore(audittelemetryschema.ScopeNameColumn)
sb.SelectMore(audittelemetryschema.ScopeVersionColumn)
sb.SelectMore(audittelemetryschema.BodyColumn)
sb.SelectMore(audittelemetryschema.EventNameColumn)
sb.SelectMore(audittelemetryschema.AttributesStringColumn)
sb.SelectMore(audittelemetryschema.AttributesNumberColumn)
sb.SelectMore(audittelemetryschema.AttributesBoolColumn)
sb.SelectMore(audittelemetryschema.ResourceColumn)
sb.SelectMore(audittelemetryschema.ScopeStringColumn)
sb.SelectMore(TraceIDColumn)
sb.SelectMore(SpanIDColumn)
sb.SelectMore(TraceFlagsColumn)
sb.SelectMore(SeverityTextColumn)
sb.SelectMore(SeverityNumberColumn)
sb.SelectMore(ScopeNameColumn)
sb.SelectMore(ScopeVersionColumn)
sb.SelectMore(BodyColumn)
sb.SelectMore(EventNameColumn)
sb.SelectMore(AttributesStringColumn)
sb.SelectMore(AttributesNumberColumn)
sb.SelectMore(AttributesBoolColumn)
sb.SelectMore(ResourceColumn)
sb.SelectMore(ScopeStringColumn)
} else {
for index := range query.SelectFields {
if query.SelectFields[index].Name == audittelemetryschema.TimestampColumn || query.SelectFields[index].Name == audittelemetryschema.IDColumn {
if query.SelectFields[index].Name == TimestampColumn || query.SelectFields[index].Name == IDColumn {
continue
}
@@ -251,7 +250,7 @@ func (b *auditQueryStatementBuilder) buildListQuery(
}
}
sb.From(fmt.Sprintf("%s.%s", audittelemetryschema.DBName, audittelemetryschema.AuditLogsTableName))
sb.From(fmt.Sprintf("%s.%s", DBName, AuditLogsTableName))
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables)
if err != nil {
@@ -341,7 +340,7 @@ func (b *auditQueryStatementBuilder) buildTimeSeriesQuery(
sb.SelectMore(fmt.Sprintf("%s AS __result_%d", rewritten, i))
}
sb.From(fmt.Sprintf("%s.%s", audittelemetryschema.DBName, audittelemetryschema.AuditLogsTableName))
sb.From(fmt.Sprintf("%s.%s", DBName, AuditLogsTableName))
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables)
if err != nil {
@@ -479,7 +478,7 @@ func (b *auditQueryStatementBuilder) buildScalarQuery(
}
}
sb.From(fmt.Sprintf("%s.%s", audittelemetryschema.DBName, audittelemetryschema.AuditLogsTableName))
sb.From(fmt.Sprintf("%s.%s", DBName, AuditLogsTableName))
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables)
if err != nil {

View File

@@ -1,4 +1,4 @@
package auditstatementbuilder
package telemetryaudit
import (
"context"
@@ -8,7 +8,6 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/audittelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
@@ -55,8 +54,8 @@ func newTestAuditStatementBuilder(t *testing.T) *auditQueryStatementBuilder {
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = auditFieldKeyMap()
fm := audittelemetryschema.NewFieldMapper()
cb := audittelemetryschema.NewConditionBuilder(fm)
fm := NewFieldMapper()
cb := NewConditionBuilder(fm)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
return NewAuditQueryStatementBuilder(
@@ -65,7 +64,7 @@ func newTestAuditStatementBuilder(t *testing.T) *auditQueryStatementBuilder {
fm,
cb,
aggExprRewriter,
audittelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
)
}

View File

@@ -0,0 +1,12 @@
package telemetryaudit
const (
DBName = "signoz_audit"
AuditLogsTableName = "distributed_logs"
AuditLogsLocalTableName = "logs"
TagAttributesTableName = "distributed_tag_attributes"
TagAttributesLocalTableName = "tag_attributes"
LogAttributeKeysTblName = "distributed_logs_attribute_keys"
LogResourceKeysTblName = "distributed_logs_resource_keys"
LogsResourceTableName = "distributed_logs_resource"
)

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"context"

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"context"
@@ -548,7 +548,7 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("bare intrinsic column resolves to the column, not synthesized attributes", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "severity_text"}
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "ERROR", sb)
conds, _, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{},qbtypes.FilterOperatorEqual, "ERROR", sb)
require.NoError(t, err)
require.Len(t, conds, 1)
sb.Where(conds...)
@@ -560,7 +560,7 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("attribute context", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextAttribute}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{},qbtypes.FilterOperatorEqual, "v", sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 2)
@@ -571,7 +571,7 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("resource context", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextResource}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{},qbtypes.FilterOperatorEqual, "v", sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 2)
@@ -582,7 +582,7 @@ func TestConditionForSynthesizedPrefixedKeys(t *testing.T) {
t.Run("log context folds to attributes then body", func(t *testing.T) {
sb := sqlbuilder.NewSelectBuilder()
key := telemetrytypes.TelemetryFieldKey{Name: "custom.key", FieldContext: telemetrytypes.FieldContextLog}
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{}, qbtypes.FilterOperatorEqual, "v", sb)
conds, warnings, err := cb.ConditionFor(ctx, valuer.UUID{}, 0, 0, &key, nil, qbtypes.ConditionBuilderOptions{},qbtypes.FilterOperatorEqual, "v", sb)
require.NoError(t, err)
assert.NotEmpty(t, warnings)
require.Len(t, conds, 4)

View File

@@ -1,6 +1,8 @@
package logstelemetryschema
package telemetrylogs
import (
"fmt"
"github.com/SigNoz/signoz-otel-collector/constants"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -39,8 +41,9 @@ const (
// messageSubColumn is the ClickHouse sub-column that body searches map to
// when use_json_body feature flag is true.
messageSubField = "message"
messageSubColumn = "body_v2.message"
messageSubField = "message"
messageSubColumn = "body_v2.message"
bodySearchDefaultWarning = "body searches default to `body.message:string`. Use `body.<key>` to search a different field inside body"
// bodyMessageField is the field name addressing the message sub-field of the
// body when use_json_body is enabled (i.e. `body` + `.` + `message`). hasToken
@@ -134,16 +137,10 @@ var (
}
)
const (
DBName = "signoz_logs"
LogsV2TableName = "distributed_logs_v2"
LogsV2LocalTableName = "logs_v2"
TagAttributesV2TableName = "distributed_tag_attributes_v2"
TagAttributesV2LocalTableName = "tag_attributes_v2"
LogAttributeKeysTblName = "distributed_logs_attribute_keys"
LogResourceKeysTblName = "distributed_logs_resource_keys"
LogsResourceV2TableName = "distributed_logs_v2_resource"
PathTypesTableName = "distributed_json_path_types"
PromotedPathsTableName = "distributed_json_promoted_paths"
SkipIndexTableName = "system.data_skipping_indices"
)
func bodyAliasExpression(bodyJSONEnabled bool) string {
if !bodyJSONEnabled {
return LogsV2BodyColumn
}
return fmt.Sprintf("%s as body", LogsV2BodyV2Column)
}

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"context"

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"context"

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"context"
@@ -19,7 +19,7 @@ func TestLikeAndILikeWithoutWildcards_Warns(t *testing.T) {
cb := NewConditionBuilder(fm, fl)
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := BuildCompleteFieldKeyMap(releaseTime)
keys := buildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
@@ -57,7 +57,7 @@ func TestLikeAndILikeWithWildcards_NoWarn(t *testing.T) {
cb := NewConditionBuilder(fm, fl)
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := BuildCompleteFieldKeyMap(releaseTime)
keys := buildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"context"
@@ -21,7 +21,7 @@ func TestFilterExprLogsBodyJSON(t *testing.T) {
cb := NewConditionBuilder(fm, fl)
// Define a comprehensive set of field keys to support all test cases
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := BuildCompleteFieldKeyMap(releaseTime)
keys := buildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: context.Background(),

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"context"
@@ -41,7 +41,7 @@ func TestFilterExprLogs(t *testing.T) {
cb := NewConditionBuilder(fm, fl)
// Define a comprehensive set of field keys to support all test cases
keys := BuildCompleteFieldKeyMap(releaseTime)
keys := buildCompleteFieldKeyMap(releaseTime)
opts := querybuilder.FilterExprVisitorOpts{
Context: ctx,
@@ -2476,7 +2476,7 @@ func TestFilterExprLogsConflictNegation(t *testing.T) {
// Define a comprehensive set of field keys to support all test cases
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keys := BuildCompleteFieldKeyMap(releaseTime)
keys := buildCompleteFieldKeyMap(releaseTime)
keys["body"] = []*telemetrytypes.TelemetryFieldKey{
{

View File

@@ -1,4 +1,4 @@
package logsstatementbuilder
package telemetrylogs
import (
"context"
@@ -8,7 +8,6 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
@@ -44,14 +43,14 @@ func TestStatementBuilderGroupByUnknownKey(t *testing.T) {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.useJSONBody)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
statementBuilder := NewLogQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),
mockMetadataStore, fm, cb, aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn, fl, nil, false, 100000,
DefaultFullTextColumn, fl, nil, false, 100000,
)
query := qbtypes.QueryBuilderQuery[qbtypes.LogAggregation]{

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"fmt"

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"strings"

View File

@@ -1,4 +1,4 @@
package logsstatementbuilder
package telemetrylogs
import (
"context"
@@ -11,7 +11,6 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes/telemetrytypestest"
@@ -1226,7 +1225,7 @@ func TestJSONStmtBuilder_OrderBy(t *testing.T) {
func TestResourceAggrAndGroupBy_WithJSONEnabled(t *testing.T) {
statementBuilder, metadataStore := buildJSONTestStatementBuilder(t, false)
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
keysMap := logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
keysMap := buildCompleteFieldKeyMap(releaseTime)
for _, keys := range keysMap {
for _, key := range keys {
metadataStore.SetKey(key)
@@ -1288,7 +1287,7 @@ func TestResourceAggrAndGroupBy_WithJSONEnabled(t *testing.T) {
func buildTestTelemetryMetadataStore(t *testing.T, addIndexes bool) *telemetrytypestest.MockMetadataStore {
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.SetStaticFields(logstelemetryschema.IntrinsicFields)
mockMetadataStore.SetStaticFields(IntrinsicFields)
types, _ := telemetrytypes.TestJSONTypeSet()
for path, fieldDataTypes := range types {
for _, fdt := range fieldDataTypes {
@@ -1308,15 +1307,15 @@ func buildTestTelemetryMetadataStore(t *testing.T, addIndexes bool) *telemetryty
Name: path,
FieldContext: telemetrytypes.FieldContextBody,
FieldDataType: fdt,
BaseColumn: logstelemetryschema.LogsV2BodyV2Column,
IndexExpression: schemamigrator.JSONSubColumnIndexExpr(logstelemetryschema.LogsV2BodyV2Column, path, jsonType.StringValue()),
BaseColumn: LogsV2BodyV2Column,
IndexExpression: schemamigrator.JSONSubColumnIndexExpr(LogsV2BodyV2Column, path, jsonType.StringValue()),
})
}
}
err := key.SetJSONAccessPlan(telemetrytypes.JSONColumnMetadata{
BaseColumn: logstelemetryschema.LogsV2BodyV2Column,
PromotedColumn: logstelemetryschema.LogsV2BodyPromotedColumn,
BaseColumn: LogsV2BodyV2Column,
PromotedColumn: LogsV2BodyPromotedColumn,
}, types)
require.NoError(t, err)
@@ -1332,8 +1331,8 @@ func buildJSONTestStatementBuilder(t *testing.T, addIndexes bool) (*logQueryStat
mockMetadataStore := buildTestTelemetryMetadataStore(t, addIndexes)
fl := flaggertest.WithUseJSONBody(t, true)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -1343,7 +1342,7 @@ func buildJSONTestStatementBuilder(t *testing.T, addIndexes bool) (*logQueryStat
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
nil,
false,

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"context"

View File

@@ -1,4 +1,4 @@
package logsstatementbuilder
package telemetrylogs
import (
"context"
@@ -10,8 +10,7 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/resourcefilter"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryresourcefilter"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -20,22 +19,12 @@ import (
"github.com/huandu/go-sqlbuilder"
)
const bodySearchDefaultWarning = "body searches default to `body.message:string`. Use `body.<key>` to search a different field inside body"
func bodyAliasExpression(bodyJSONEnabled bool) string {
if !bodyJSONEnabled {
return logstelemetryschema.LogsV2BodyColumn
}
return fmt.Sprintf("%s as body", logstelemetryschema.LogsV2BodyV2Column)
}
type logQueryStatementBuilder struct {
logger *slog.Logger
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
resourceFilterResolver *resourcefilter.ResourceFingerprintResolver[qbtypes.LogAggregation]
resourceFilterResolver *telemetryresourcefilter.ResourceFingerprintResolver[qbtypes.LogAggregation]
aggExprRewriter qbtypes.AggExprRewriter
fl flagger.Flagger
skipResourceFingerprintEnabled bool
@@ -57,12 +46,12 @@ func NewLogQueryStatementBuilder(
skipResourceFingerprintEnable bool,
skipResourceFingerprintThreshold uint64,
) *logQueryStatementBuilder {
logsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema")
logsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrylogs")
resourceFilterResolver := resourcefilter.NewResolver[qbtypes.LogAggregation](
resourceFilterResolver := telemetryresourcefilter.NewResolver[qbtypes.LogAggregation](
settings,
logstelemetryschema.DBName,
logstelemetryschema.LogsResourceV2TableName,
DBName,
LogsResourceV2TableName,
telemetrytypes.SignalLogs,
telemetrytypes.SourceUnspecified,
metadataStore,
@@ -185,7 +174,7 @@ func getKeySelectors(query qbtypes.QueryBuilderQuery[qbtypes.LogAggregation], bo
// TODO(Piyush): Setup better for coming FTS support.
if bodyJSONEnabled {
for _, sel := range keySelectors {
if sel.Name == logstelemetryschema.LogsV2BodyColumn {
if sel.Name == LogsV2BodyColumn {
warnings = append(warnings, bodySearchDefaultWarning)
break
}
@@ -261,8 +250,8 @@ func (b *logQueryStatementBuilder) adjustKeys(ctx context.Context, keys map[stri
func (b *logQueryStatementBuilder) adjustKey(key *telemetrytypes.TelemetryFieldKey, keys map[string][]*telemetrytypes.TelemetryFieldKey) []string {
// First check if it matches with any intrinsic fields
var intrinsicOrCalculatedField telemetrytypes.TelemetryFieldKey
if _, ok := logstelemetryschema.IntrinsicFields[key.Name]; ok {
intrinsicOrCalculatedField = logstelemetryschema.IntrinsicFields[key.Name]
if _, ok := IntrinsicFields[key.Name]; ok {
intrinsicOrCalculatedField = IntrinsicFields[key.Name]
return querybuilder.AdjustKey(key, keys, &intrinsicOrCalculatedField)
}
@@ -295,28 +284,28 @@ func (b *logQueryStatementBuilder) buildListQuery(
}
// Select timestamp and id by default
sb.Select(logstelemetryschema.LogsV2TimestampColumn)
sb.SelectMore(logstelemetryschema.LogsV2IDColumn)
sb.Select(LogsV2TimestampColumn)
sb.SelectMore(LogsV2IDColumn)
if len(query.SelectFields) == 0 {
// Select all default columns
sb.SelectMore(logstelemetryschema.LogsV2TraceIDColumn)
sb.SelectMore(logstelemetryschema.LogsV2SpanIDColumn)
sb.SelectMore(logstelemetryschema.LogsV2TraceFlagsColumn)
sb.SelectMore(logstelemetryschema.LogsV2SeverityTextColumn)
sb.SelectMore(logstelemetryschema.LogsV2SeverityNumberColumn)
sb.SelectMore(logstelemetryschema.LogsV2ScopeNameColumn)
sb.SelectMore(logstelemetryschema.LogsV2ScopeVersionColumn)
sb.SelectMore(LogsV2TraceIDColumn)
sb.SelectMore(LogsV2SpanIDColumn)
sb.SelectMore(LogsV2TraceFlagsColumn)
sb.SelectMore(LogsV2SeverityTextColumn)
sb.SelectMore(LogsV2SeverityNumberColumn)
sb.SelectMore(LogsV2ScopeNameColumn)
sb.SelectMore(LogsV2ScopeVersionColumn)
sb.SelectMore(bodyAliasExpression(b.fl.BooleanOrEmpty(ctx, flagger.FeatureUseJSONBody, featuretypes.NewFlaggerEvaluationContext(orgID))))
sb.SelectMore(logstelemetryschema.LogsV2AttributesStringColumn)
sb.SelectMore(logstelemetryschema.LogsV2AttributesNumberColumn)
sb.SelectMore(logstelemetryschema.LogsV2AttributesBoolColumn)
sb.SelectMore(logstelemetryschema.LogsV2ResourcesStringColumn)
sb.SelectMore(logstelemetryschema.LogsV2ScopeStringColumn)
sb.SelectMore(LogsV2AttributesStringColumn)
sb.SelectMore(LogsV2AttributesNumberColumn)
sb.SelectMore(LogsV2AttributesBoolColumn)
sb.SelectMore(LogsV2ResourcesStringColumn)
sb.SelectMore(LogsV2ScopeStringColumn)
} else {
// Select specified columns
for index := range query.SelectFields {
if query.SelectFields[index].Name == logstelemetryschema.LogsV2TimestampColumn || query.SelectFields[index].Name == logstelemetryschema.LogsV2IDColumn {
if query.SelectFields[index].Name == LogsV2TimestampColumn || query.SelectFields[index].Name == LogsV2IDColumn {
continue
}
@@ -329,7 +318,7 @@ func (b *logQueryStatementBuilder) buildListQuery(
}
}
sb.From(fmt.Sprintf("%s.%s", logstelemetryschema.DBName, logstelemetryschema.LogsV2TableName))
sb.From(fmt.Sprintf("%s.%s", DBName, LogsV2TableName))
// Add filter conditions
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables, skipResourceFilter)
@@ -435,7 +424,7 @@ func (b *logQueryStatementBuilder) buildTimeSeriesQuery(
}
// Add FROM clause
sb.From(fmt.Sprintf("%s.%s", logstelemetryschema.DBName, logstelemetryschema.LogsV2TableName))
sb.From(fmt.Sprintf("%s.%s", DBName, LogsV2TableName))
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables, skipResourceFilter)
@@ -603,7 +592,7 @@ func (b *logQueryStatementBuilder) buildScalarQuery(
}
}
sb.From(fmt.Sprintf("%s.%s", logstelemetryschema.DBName, logstelemetryschema.LogsV2TableName))
sb.From(fmt.Sprintf("%s.%s", DBName, LogsV2TableName))
// Add filter conditions
preparedWhereClause, err := b.addFilterCondition(ctx, orgID, sb, start, end, query, keys, variables, skipResourceFilter)

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"testing"

View File

@@ -1,4 +1,4 @@
package logsstatementbuilder
package telemetrylogs
import (
"context"
@@ -11,7 +11,6 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
@@ -213,12 +212,12 @@ func TestStatementBuilderTimeSeries(t *testing.T) {
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keysMap := logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
keysMap := buildCompleteFieldKeyMap(releaseTime)
mockMetadataStore.KeysMap = keysMap
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -228,7 +227,7 @@ func TestStatementBuilderTimeSeries(t *testing.T) {
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
nil,
false,
@@ -356,12 +355,12 @@ func TestStatementBuilderListQuery(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fm := logstelemetryschema.NewFieldMapper(fl)
fm := NewFieldMapper(fl)
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
cb := NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -371,7 +370,7 @@ func TestStatementBuilderListQuery(t *testing.T) {
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
nil,
false,
@@ -508,11 +507,11 @@ func TestStatementBuilderListQueryResourceTests(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fm := logstelemetryschema.NewFieldMapper(fl)
fm := NewFieldMapper(fl)
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
cb := NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -522,7 +521,7 @@ func TestStatementBuilderListQueryResourceTests(t *testing.T) {
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
nil,
false,
@@ -586,11 +585,11 @@ func TestStatementBuilderTimeSeriesBodyGroupBy(t *testing.T) {
ctx := context.Background()
fl := flaggertest.New(t)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fm := logstelemetryschema.NewFieldMapper(fl)
fm := NewFieldMapper(fl)
// Create a test release time
releaseTime := time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(releaseTime)
cb := NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -600,7 +599,7 @@ func TestStatementBuilderTimeSeriesBodyGroupBy(t *testing.T) {
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
nil,
false,
@@ -685,9 +684,9 @@ func TestStatementBuilderListQueryServiceCollision(t *testing.T) {
ctx := context.Background()
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
fl := flaggertest.New(t)
fm := logstelemetryschema.NewFieldMapper(fl)
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMapCollision()
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
fm := NewFieldMapper(fl)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMapCollision()
cb := NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -697,7 +696,7 @@ func TestStatementBuilderListQueryServiceCollision(t *testing.T) {
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
nil,
false,
@@ -741,8 +740,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: logstelemetryschema.IntrinsicFields["severity_text"],
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: IntrinsicFields["severity_text"],
},
{
name: "intrinsic field with other key match - no override",
@@ -778,7 +777,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextBody,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "severity_number",
FieldContext: telemetrytypes.FieldContextBody,
@@ -792,8 +791,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["service.name"][0],
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["service.name"][0],
},
{
name: "single matching key with incorrect context specified - no override",
@@ -802,7 +801,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "service.name",
FieldContext: telemetrytypes.FieldContextAttribute,
@@ -816,8 +815,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["service.name"][0],
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["service.name"][0],
},
{
name: "multiple matching keys - all materialized",
@@ -826,7 +825,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "multi.mat.key",
FieldDataType: telemetrytypes.FieldDataTypeString,
@@ -840,7 +839,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "mixed.materialization.key",
FieldDataType: telemetrytypes.FieldDataTypeString,
@@ -854,8 +853,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["mixed.materialization.key"][0],
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["mixed.materialization.key"][0],
},
{
name: "no matching keys - unknown field",
@@ -864,7 +863,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "unknown.field",
FieldContext: telemetrytypes.FieldContextUnspecified,
@@ -879,7 +878,7 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextAttribute,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: telemetrytypes.TelemetryFieldKey{
Name: "unknown.field",
FieldContext: telemetrytypes.FieldContextAttribute,
@@ -894,8 +893,8 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["mat.key"][0],
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["mat.key"][0],
},
{
name: "non-materialized field",
@@ -904,16 +903,16 @@ func TestAdjustKey(t *testing.T) {
FieldContext: telemetrytypes.FieldContextUnspecified,
FieldDataType: telemetrytypes.FieldDataTypeUnspecified,
},
keysMap: logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime),
expectedKey: *logstelemetryschema.BuildCompleteFieldKeyMap(releaseTime)["user.id"][0],
keysMap: buildCompleteFieldKeyMap(releaseTime),
expectedKey: *buildCompleteFieldKeyMap(releaseTime)["user.id"][0],
},
}
fl := flaggertest.New(t)
fm := logstelemetryschema.NewFieldMapper(fl)
fm := NewFieldMapper(fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMapCollision()
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
mockMetadataStore.KeysMap = buildCompleteFieldKeyMapCollision()
cb := NewConditionBuilder(fm, fl)
aggExprRewriter := querybuilder.NewAggExprRewriter(instrumentationtest.New().ToProviderSettings(), nil, fm, cb, fl)
@@ -923,7 +922,7 @@ func TestAdjustKey(t *testing.T) {
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
nil,
false,
@@ -1055,11 +1054,11 @@ func TestStmtBuilderBodyField(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.enableUseJSONBody)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
// build the key map
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
for _, field := range logstelemetryschema.IntrinsicFields {
for _, field := range IntrinsicFields {
f := field
mockMetadataStore.KeysMap[field.Name] = append(mockMetadataStore.KeysMap[field.Name], &f)
}
@@ -1070,7 +1069,7 @@ func TestStmtBuilderBodyField(t *testing.T) {
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
nil,
false,
@@ -1156,11 +1155,11 @@ func TestStmtBuilderBodyFullTextSearch(t *testing.T) {
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
fl := flaggertest.WithUseJSONBody(t, c.enableUseJSONBody)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
// build the key map
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
for _, field := range logstelemetryschema.IntrinsicFields {
for _, field := range IntrinsicFields {
f := field
mockMetadataStore.KeysMap[field.Name] = append(mockMetadataStore.KeysMap[field.Name], &f)
}
@@ -1171,7 +1170,7 @@ func TestStmtBuilderBodyFullTextSearch(t *testing.T) {
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
nil,
false,
@@ -1274,14 +1273,14 @@ func newSkipResourceFingerprintLogsBuilder(
t.Helper()
fl := flaggertest.New(t)
fm := logstelemetryschema.NewFieldMapper(fl)
cb := logstelemetryschema.NewConditionBuilder(fm, fl)
fm := NewFieldMapper(fl)
cb := NewConditionBuilder(fm, fl)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
mockMetadataStore.KeysMap = logstelemetryschema.BuildCompleteFieldKeyMap(time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC))
mockMetadataStore.KeysMap = buildCompleteFieldKeyMap(time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC))
aggExprRewriter := querybuilder.NewAggExprRewriter(
instrumentationtest.New().ToProviderSettings(),
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fm,
cb,
fl,
@@ -1293,7 +1292,7 @@ func newSkipResourceFingerprintLogsBuilder(
fm,
cb,
aggExprRewriter,
logstelemetryschema.DefaultFullTextColumn,
DefaultFullTextColumn,
fl,
telemetryStore,
skipEnable,

View File

@@ -0,0 +1,15 @@
package telemetrylogs
const (
DBName = "signoz_logs"
LogsV2TableName = "distributed_logs_v2"
LogsV2LocalTableName = "logs_v2"
TagAttributesV2TableName = "distributed_tag_attributes_v2"
TagAttributesV2LocalTableName = "tag_attributes_v2"
LogAttributeKeysTblName = "distributed_logs_attribute_keys"
LogResourceKeysTblName = "distributed_logs_resource_keys"
LogsResourceV2TableName = "distributed_logs_v2_resource"
PathTypesTableName = "distributed_json_path_types"
PromotedPathsTableName = "distributed_json_promoted_paths"
SkipIndexTableName = "system.data_skipping_indices"
)

View File

@@ -1,4 +1,4 @@
package logstelemetryschema
package telemetrylogs
import (
"strings"
@@ -19,7 +19,7 @@ func limitString(s string, maxLen int) string {
}
// Function to build a complete field key map for testing all scenarios.
func BuildCompleteFieldKeyMap(releaseTime time.Time) map[string][]*telemetrytypes.TelemetryFieldKey {
func buildCompleteFieldKeyMap(releaseTime time.Time) map[string][]*telemetrytypes.TelemetryFieldKey {
keysMap := map[string][]*telemetrytypes.TelemetryFieldKey{
"service.name": {
{
@@ -953,7 +953,7 @@ func BuildCompleteFieldKeyMap(releaseTime time.Time) map[string][]*telemetrytype
return keysMap
}
func BuildCompleteFieldKeyMapCollision() map[string][]*telemetrytypes.TelemetryFieldKey {
func buildCompleteFieldKeyMapCollision() map[string][]*telemetrytypes.TelemetryFieldKey {
keysMap := map[string][]*telemetrytypes.TelemetryFieldKey{
"service.name": {
{

View File

@@ -13,7 +13,7 @@ import (
"github.com/SigNoz/signoz-otel-collector/constants"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -139,8 +139,8 @@ func buildListLogsJSONIndexesQuery(cluster string, filters ...string) (string, [
"name", "type_full", "expr", "granularity",
).From(fmt.Sprintf("clusterAllReplicas('%s', %s)", cluster, SkipIndexTableName))
sb.Where(sb.Equal("database", logstelemetryschema.DBName))
sb.Where(sb.Equal("table", logstelemetryschema.LogsV2LocalTableName))
sb.Where(sb.Equal("database", telemetrylogs.DBName))
sb.Where(sb.Equal("table", telemetrylogs.LogsV2LocalTableName))
sb.Where(sb.Or(
sb.ILike("expr", fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyV2ColumnPrefix))),
sb.ILike("expr", fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyPromotedColumnPrefix))),
@@ -189,12 +189,12 @@ func (t *telemetryMetaStore) ListLogsJSONIndexes(ctx context.Context, filters ..
baseColumn := ""
fieldName := ""
switch {
case strings.HasPrefix(columnExpr, logstelemetryschema.BodyV2ColumnPrefix):
baseColumn = logstelemetryschema.BodyV2ColumnPrefix
fieldName = strings.TrimPrefix(columnExpr, logstelemetryschema.BodyV2ColumnPrefix)
case strings.HasPrefix(columnExpr, logstelemetryschema.BodyPromotedColumnPrefix):
baseColumn = logstelemetryschema.BodyPromotedColumnPrefix
fieldName = strings.TrimPrefix(columnExpr, logstelemetryschema.BodyPromotedColumnPrefix)
case strings.HasPrefix(columnExpr, telemetrylogs.BodyV2ColumnPrefix):
baseColumn = telemetrylogs.BodyV2ColumnPrefix
fieldName = strings.TrimPrefix(columnExpr, telemetrylogs.BodyV2ColumnPrefix)
case strings.HasPrefix(columnExpr, telemetrylogs.BodyPromotedColumnPrefix):
baseColumn = telemetrylogs.BodyPromotedColumnPrefix
fieldName = strings.TrimPrefix(columnExpr, telemetrylogs.BodyPromotedColumnPrefix)
}
fieldName = strings.ReplaceAll(fieldName, "`", "")
@@ -228,12 +228,12 @@ func (t *telemetryMetaStore) ListJSONValues(ctx context.Context, path string, li
}
if promoted {
path = logstelemetryschema.BodyPromotedColumnPrefix + path
path = telemetrylogs.BodyPromotedColumnPrefix + path
} else {
path = logstelemetryschema.BodyV2ColumnPrefix + path
path = telemetrylogs.BodyV2ColumnPrefix + path
}
from := fmt.Sprintf("%s.%s", logstelemetryschema.DBName, logstelemetryschema.LogsV2TableName)
from := fmt.Sprintf("%s.%s", telemetrylogs.DBName, telemetrylogs.LogsV2TableName)
colExpr := func(typ telemetrytypes.JSONDataType) string {
return fmt.Sprintf("dynamicElement(%s, '%s')", path, typ.StringValue())
}
@@ -382,7 +382,7 @@ func (t *telemetryMetaStore) IsPathPromoted(ctx context.Context, path string) (b
split := strings.Split(path, telemetrytypes.ArraySep)
pathSegment := split[0]
query := fmt.Sprintf("SELECT 1 FROM %s.%s WHERE signal = ? AND column_name = ? AND field_context = ? AND field_name = ? LIMIT 1", DBName, PromotedPathsTableName)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, query, telemetrytypes.SignalLogs, logstelemetryschema.LogsV2BodyPromotedColumn, telemetrytypes.FieldContextBody, pathSegment)
rows, err := t.telemetrystore.ClickhouseDB().Query(ctx, query, telemetrytypes.SignalLogs, telemetrylogs.LogsV2BodyPromotedColumn, telemetrytypes.FieldContextBody, pathSegment)
if err != nil {
return false, errors.WrapInternalf(err, CodeFailCheckPathPromoted, "failed to check if path %s is promoted", path)
}
@@ -397,7 +397,7 @@ func (t *telemetryMetaStore) GetPromotedPaths(ctx context.Context, paths ...stri
sb := sqlbuilder.Select("field_name").From(fmt.Sprintf("%s.%s", DBName, PromotedPathsTableName))
conditions := []string{
sb.Equal("signal", telemetrytypes.SignalLogs),
sb.Equal("column_name", logstelemetryschema.LogsV2BodyPromotedColumn),
sb.Equal("column_name", telemetrylogs.LogsV2BodyPromotedColumn),
sb.Equal("field_context", telemetrytypes.FieldContextBody),
sb.NotEqual("field_name", "__all__"),
}
@@ -433,8 +433,8 @@ func (t *telemetryMetaStore) GetPromotedPaths(ctx context.Context, paths ...stri
// TODO(Piyush): Remove this function.
func CleanPathPrefixes(path string) string {
path = strings.TrimPrefix(path, telemetrytypes.BodyJSONStringSearchPrefix)
path = strings.TrimPrefix(path, logstelemetryschema.BodyV2ColumnPrefix)
path = strings.TrimPrefix(path, logstelemetryschema.BodyPromotedColumnPrefix)
path = strings.TrimPrefix(path, telemetrylogs.BodyV2ColumnPrefix)
path = strings.TrimPrefix(path, telemetrylogs.BodyPromotedColumnPrefix)
return path
}
@@ -454,7 +454,7 @@ func (t *telemetryMetaStore) PromotePaths(ctx context.Context, paths ...string)
if trimmed == "" {
continue
}
if err := batch.Append(telemetrytypes.SignalLogs, logstelemetryschema.LogsV2BodyPromotedColumn, "JSON()", telemetrytypes.FieldContextBody, trimmed, 0, releaseTime); err != nil {
if err := batch.Append(telemetrytypes.SignalLogs, telemetrylogs.LogsV2BodyPromotedColumn, "JSON()", telemetrytypes.FieldContextBody, trimmed, 0, releaseTime); err != nil {
_ = batch.Abort()
return errors.WrapInternalf(err, CodeFailedToAppendPath, "failed to append path")
}

View File

@@ -6,7 +6,7 @@ import (
"github.com/SigNoz/signoz-otel-collector/constants"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/stretchr/testify/require"
)
@@ -25,8 +25,8 @@ func TestBuildListLogsJSONIndexesQuery(t *testing.T) {
expectedSQL: "SELECT name, type_full, expr, granularity FROM clusterAllReplicas('test-cluster', system.data_skipping_indices) " +
"WHERE database = ? AND table = ? AND (LOWER(expr) LIKE LOWER(?) OR LOWER(expr) LIKE LOWER(?))",
expectedArgs: []any{
logstelemetryschema.DBName,
logstelemetryschema.LogsV2LocalTableName,
telemetrylogs.DBName,
telemetrylogs.LogsV2LocalTableName,
fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyV2ColumnPrefix)),
fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyPromotedColumnPrefix)),
},
@@ -38,8 +38,8 @@ func TestBuildListLogsJSONIndexesQuery(t *testing.T) {
expectedSQL: "SELECT name, type_full, expr, granularity FROM clusterAllReplicas('test-cluster', system.data_skipping_indices) " +
"WHERE database = ? AND table = ? AND (LOWER(expr) LIKE LOWER(?) OR LOWER(expr) LIKE LOWER(?)) AND (LOWER(replaceAll(expr, '`', '')) LIKE LOWER(?) OR LOWER(replaceAll(expr, '`', '')) LIKE LOWER(?))",
expectedArgs: []any{
logstelemetryschema.DBName,
logstelemetryschema.LogsV2LocalTableName,
telemetrylogs.DBName,
telemetrylogs.LogsV2LocalTableName,
fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyV2ColumnPrefix)),
fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains(constants.BodyPromotedColumnPrefix)),
fmt.Sprintf("%%%s%%", querybuilder.FormatValueForContains("foo")),

View File

@@ -14,12 +14,11 @@ import (
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/audittelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/logstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryschema/tracestelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/ctxtypes"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
"github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
@@ -78,6 +77,27 @@ func escapeForLike(s string) string {
func NewTelemetryMetaStore(
settings factory.ProviderSettings,
telemetrystore telemetrystore.TelemetryStore,
tracesDBName string,
tracesFieldsTblName string,
spanAttributesKeysTblName string,
indexV3TblName string,
metricsDBName string,
metricsFieldsTblName string,
meterDBName string,
meterFieldsTblName string,
logsDBName string,
logsV2TblName string,
logsFieldsTblName string,
logAttributeKeysTblName string,
logResourceKeysTblName string,
auditDBName string,
auditLogsTblName string,
auditFieldsTblName string,
auditAttributeKeysTblName string,
auditResourceKeysTblName string,
relatedMetadataDBName string,
relatedMetadataTblName string,
columnEvolutionMetadataTblName string,
fl flagger.Flagger,
) telemetrytypes.MetadataStore {
metadataSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrymetadata")
@@ -88,32 +108,32 @@ func NewTelemetryMetaStore(
t := &telemetryMetaStore{
logger: metadataSettings.Logger(),
telemetrystore: telemetrystore,
tracesDBName: tracestelemetryschema.DBName,
tracesFieldsTblName: tracestelemetryschema.TagAttributesV2TableName,
spanAttributesKeysTblName: tracestelemetryschema.SpanAttributesKeysTblName,
indexV3TblName: tracestelemetryschema.SpanIndexV3TableName,
metricsDBName: metricstelemetryschema.DBName,
metricsFieldsTblName: metricstelemetryschema.AttributesMetadataTableName,
meterDBName: metertelemetryschema.DBName,
meterFieldsTblName: metertelemetryschema.SamplesAgg1dTableName,
logsDBName: logstelemetryschema.DBName,
logsV2TblName: logstelemetryschema.LogsV2TableName,
logsFieldsTblName: logstelemetryschema.TagAttributesV2TableName,
logAttributeKeysTblName: logstelemetryschema.LogAttributeKeysTblName,
logResourceKeysTblName: logstelemetryschema.LogResourceKeysTblName,
auditDBName: audittelemetryschema.DBName,
auditLogsTblName: audittelemetryschema.AuditLogsTableName,
auditFieldsTblName: audittelemetryschema.TagAttributesTableName,
auditAttributeKeysTblName: audittelemetryschema.LogAttributeKeysTblName,
auditResourceKeysTblName: audittelemetryschema.LogResourceKeysTblName,
relatedMetadataDBName: DBName,
relatedMetadataTblName: AttributesMetadataTableName,
columnEvolutionMetadataTblName: ColumnEvolutionMetadataTableName,
tracesDBName: tracesDBName,
tracesFieldsTblName: tracesFieldsTblName,
spanAttributesKeysTblName: spanAttributesKeysTblName,
indexV3TblName: indexV3TblName,
metricsDBName: metricsDBName,
metricsFieldsTblName: metricsFieldsTblName,
meterDBName: meterDBName,
meterFieldsTblName: meterFieldsTblName,
logsDBName: logsDBName,
logsV2TblName: logsV2TblName,
logsFieldsTblName: logsFieldsTblName,
logAttributeKeysTblName: logAttributeKeysTblName,
logResourceKeysTblName: logResourceKeysTblName,
auditDBName: auditDBName,
auditLogsTblName: auditLogsTblName,
auditFieldsTblName: auditFieldsTblName,
auditAttributeKeysTblName: auditAttributeKeysTblName,
auditResourceKeysTblName: auditResourceKeysTblName,
relatedMetadataDBName: relatedMetadataDBName,
relatedMetadataTblName: relatedMetadataTblName,
columnEvolutionMetadataTblName: columnEvolutionMetadataTblName,
jsonColumnMetadata: map[telemetrytypes.Signal]map[telemetrytypes.FieldContext]telemetrytypes.JSONColumnMetadata{
telemetrytypes.SignalLogs: {
telemetrytypes.FieldContextBody: telemetrytypes.JSONColumnMetadata{
BaseColumn: logstelemetryschema.LogsV2BodyV2Column,
PromotedColumn: logstelemetryschema.LogsV2BodyPromotedColumn,
BaseColumn: telemetrylogs.LogsV2BodyV2Column,
PromotedColumn: telemetrylogs.LogsV2BodyPromotedColumn,
},
},
},
@@ -289,8 +309,8 @@ func (t *telemetryMetaStore) getTracesKeys(ctx context.Context, fieldKeySelector
complete := rowCount <= limit
staticKeys := []string{"isRoot", "isEntryPoint"}
staticKeys = append(staticKeys, maps.Keys(tracestelemetryschema.IntrinsicFields)...)
staticKeys = append(staticKeys, maps.Keys(tracestelemetryschema.CalculatedFields)...)
staticKeys = append(staticKeys, maps.Keys(telemetrytraces.IntrinsicFields)...)
staticKeys = append(staticKeys, maps.Keys(telemetrytraces.CalculatedFields)...)
// Add matching intrinsic and matching calculated fields
// These don't count towards the limit
@@ -304,14 +324,14 @@ func (t *telemetryMetaStore) getTracesKeys(ctx context.Context, fieldKeySelector
}
if found {
if field, exists := tracestelemetryschema.IntrinsicFields[key]; exists {
if field, exists := telemetrytraces.IntrinsicFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
continue
}
if field, exists := tracestelemetryschema.CalculatedFields[key]; exists {
if field, exists := telemetrytraces.CalculatedFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
@@ -636,7 +656,7 @@ func (t *telemetryMetaStore) getLogsKeys(ctx context.Context, orgID valuer.UUID,
complete := rowCount <= limit
staticKeys := []string{}
staticKeys = append(staticKeys, maps.Keys(logstelemetryschema.IntrinsicFields)...)
staticKeys = append(staticKeys, maps.Keys(telemetrylogs.IntrinsicFields)...)
// Add matching intrinsic and matching calculated fields
// These don't count towards the limit
@@ -650,7 +670,7 @@ func (t *telemetryMetaStore) getLogsKeys(ctx context.Context, orgID valuer.UUID,
}
if found {
if field, exists := logstelemetryschema.IntrinsicFields[key]; exists {
if field, exists := telemetrylogs.IntrinsicFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
@@ -878,7 +898,7 @@ func (t *telemetryMetaStore) getAuditKeys(ctx context.Context, fieldKeySelectors
complete := rowCount <= limit
// Add intrinsic audit fields (same as logs intrinsics: body, severity_text, etc.)
staticKeys := maps.Keys(audittelemetryschema.IntrinsicFields)
staticKeys := maps.Keys(telemetryaudit.IntrinsicFields)
for _, key := range staticKeys {
found := false
for _, v := range searchTexts {
@@ -889,7 +909,7 @@ func (t *telemetryMetaStore) getAuditKeys(ctx context.Context, fieldKeySelectors
}
if found {
if field, exists := audittelemetryschema.IntrinsicFields[key]; exists {
if field, exists := telemetryaudit.IntrinsicFields[key]; exists {
if _, added := mapOfKeys[field.Name+";"+field.FieldContext.StringValue()+";"+field.FieldDataType.StringValue()]; !added {
keys = append(keys, &field)
}
@@ -1156,7 +1176,7 @@ func enrichWithIntrinsicMetricKeys(keys map[string][]*telemetrytypes.TelemetryFi
continue
}
for name, key := range metricstelemetryschema.IntrinsicMetricFieldDefinitions {
for name, key := range telemetrymetrics.IntrinsicMetricFieldDefinitions {
if !selectorMatchesIntrinsicField(selector, key) {
continue
}
@@ -1872,7 +1892,7 @@ func (t *telemetryMetaStore) getIntrinsicMetricFieldValues(ctx context.Context,
instrumentationtypes.CodeNamespace: "metadata",
instrumentationtypes.CodeFunctionName: "getIntrinsicMetricFieldValues",
})
key, ok := metricstelemetryschema.IntrinsicMetricFieldDefinitions[fieldValueSelector.Name]
key, ok := telemetrymetrics.IntrinsicMetricFieldDefinitions[fieldValueSelector.Name]
if !ok {
return &telemetrytypes.TelemetryFieldValues{}, nil
}
@@ -1902,18 +1922,18 @@ func (t *telemetryMetaStore) getIntrinsicMetricFieldValues(ctx context.Context,
tableName string
extraConds []string
}{
{metricstelemetryschema.TimeseriesV41weekTableName, nil},
{telemetrymetrics.TimeseriesV41weekTableName, nil},
}
if reductionEnabled {
sources = append(sources,
struct {
tableName string
extraConds []string
}{metricstelemetryschema.TimeseriesV4BufferTableName, []string{"is_reduced = false"}},
}{telemetrymetrics.TimeseriesV4BufferTableName, []string{"is_reduced = false"}},
struct {
tableName string
extraConds []string
}{metricstelemetryschema.TimeseriesV4ReducedTableName, nil},
}{telemetrymetrics.TimeseriesV4ReducedTableName, nil},
)
}
@@ -2242,7 +2262,7 @@ func (t *telemetryMetaStore) fetchMetricsTemporalityAndType(ctx context.Context,
instrumentationtypes.CodeNamespace: "metadata",
instrumentationtypes.CodeFunctionName: "fetchMetricsTemporalityAndType",
})
adjustedStartTs, adjustedEndTs, tsTableName, _ := metricstelemetryschema.WhichTSTableToUse(queryTimeRangeStartTs, queryTimeRangeEndTs, false, nil)
adjustedStartTs, adjustedEndTs, tsTableName, _ := telemetrymetrics.WhichTSTableToUse(queryTimeRangeStartTs, queryTimeRangeEndTs, false, nil)
temporalities, types, err := t.fetchTemporalityTypeForTable(ctx, tsTableName, adjustedStartTs, adjustedEndTs, metricNames)
if err != nil {
@@ -2251,7 +2271,7 @@ func (t *telemetryMetaStore) fetchMetricsTemporalityAndType(ctx context.Context,
reduced := make(map[string]bool)
if reductionEnabled {
bufferTemporalities, bufferTypes, err := t.fetchTemporalityTypeForTable(ctx, metricstelemetryschema.TimeseriesV4BufferTableName, adjustedStartTs, adjustedEndTs, metricNames, "is_reduced = false")
bufferTemporalities, bufferTypes, err := t.fetchTemporalityTypeForTable(ctx, telemetrymetrics.TimeseriesV4BufferTableName, adjustedStartTs, adjustedEndTs, metricNames, "is_reduced = false")
if err != nil {
return nil, nil, nil, err
}
@@ -2264,7 +2284,7 @@ func (t *telemetryMetaStore) fetchMetricsTemporalityAndType(ctx context.Context,
}
}
reducedTemporalities, reducedTypes, err := t.fetchTemporalityTypeForTable(ctx, metricstelemetryschema.TimeseriesV4ReducedTableName, adjustedStartTs, adjustedEndTs, metricNames)
reducedTemporalities, reducedTypes, err := t.fetchTemporalityTypeForTable(ctx, telemetrymetrics.TimeseriesV4ReducedTableName, adjustedStartTs, adjustedEndTs, metricNames)
if err != nil {
return nil, nil, nil, err
}
@@ -2596,13 +2616,13 @@ func (t *telemetryMetaStore) GetFirstSeenFromMetricMetadata(ctx context.Context,
}
func (t *telemetryMetaStore) FetchLastSeenInfoMulti(ctx context.Context, orgID valuer.UUID, metricNames ...string) (map[string]int64, error) {
lastSeenInfo, err := t.fetchLastSeenInfoForTable(ctx, metricstelemetryschema.TimeseriesV4TableName, metricNames)
lastSeenInfo, err := t.fetchLastSeenInfoForTable(ctx, telemetrymetrics.TimeseriesV4TableName, metricNames)
if err != nil {
return nil, err
}
if t.fl.BooleanOrEmpty(ctx, flagger.FeatureEnableMetricsReduction, featuretypes.NewFlaggerEvaluationContext(orgID)) {
reducedLastSeen, err := t.fetchLastSeenInfoForTable(ctx, metricstelemetryschema.TimeseriesV4ReducedTableName, metricNames)
reducedLastSeen, err := t.fetchLastSeenInfoForTable(ctx, telemetrymetrics.TimeseriesV4ReducedTableName, metricNames)
if err != nil {
return nil, err
}

View File

@@ -9,8 +9,13 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/telemetryaudit"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/telemetrystore/telemetrystoretest"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -37,6 +42,27 @@ func TestGetFirstSeenFromMetricMetadata(t *testing.T) {
metadata := NewTelemetryMetaStore(
instrumentationtest.New().ToProviderSettings(),
mockTelemetryStore,
telemetrytraces.DBName,
telemetrytraces.TagAttributesV2TableName,
telemetrytraces.SpanAttributesKeysTblName,
telemetrytraces.SpanIndexV3TableName,
telemetrymetrics.DBName,
telemetrymetrics.AttributesMetadataTableName,
telemetrymeter.DBName,
telemetrymeter.SamplesAgg1dTableName,
telemetrylogs.DBName,
telemetrylogs.LogsV2TableName,
telemetrylogs.TagAttributesV2TableName,
telemetrylogs.LogAttributeKeysTblName,
telemetrylogs.LogResourceKeysTblName,
telemetryaudit.DBName,
telemetryaudit.AuditLogsTableName,
telemetryaudit.TagAttributesTableName,
telemetryaudit.LogAttributeKeysTblName,
telemetryaudit.LogResourceKeysTblName,
DBName,
AttributesMetadataTableName,
ColumnEvolutionMetadataTableName,
flaggertest.New(t),
)

View File

@@ -1,4 +1,4 @@
package meterstatementbuilder
package telemetrymeter
import (
"context"
@@ -8,8 +8,7 @@ import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/querybuilder"
"github.com/SigNoz/signoz/pkg/statementbuilder/metricsstatementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -22,7 +21,7 @@ type meterQueryStatementBuilder struct {
metadataStore telemetrytypes.MetadataStore
fm qbtypes.FieldMapper
cb qbtypes.ConditionBuilder
metricsStatementBuilder *metricsstatementbuilder.StatementBuilder
metricsStatementBuilder *telemetrymetrics.MetricQueryStatementBuilder
}
var _ qbtypes.StatementBuilder[qbtypes.MetricAggregation] = (*meterQueryStatementBuilder)(nil)
@@ -32,9 +31,9 @@ func NewMeterQueryStatementBuilder(
metadataStore telemetrytypes.MetadataStore,
fieldMapper qbtypes.FieldMapper,
conditionBuilder qbtypes.ConditionBuilder,
metricsStatementBuilder *metricsstatementbuilder.StatementBuilder,
metricsStatementBuilder *telemetrymetrics.MetricQueryStatementBuilder,
) *meterQueryStatementBuilder {
metricsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetryschema/metertelemetryschema")
metricsSettings := factory.NewScopedProviderSettings(settings, "github.com/SigNoz/signoz/pkg/telemetrymeter")
return &meterQueryStatementBuilder{
logger: metricsSettings.Logger(),
@@ -54,7 +53,7 @@ func (b *meterQueryStatementBuilder) Build(
query qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation],
variables map[string]qbtypes.VariableItem,
) (*qbtypes.Statement, error) {
keySelectors := metricsstatementbuilder.GetKeySelectors(query)
keySelectors := telemetrymetrics.GetKeySelectors(query)
keys, _, err := b.metadataStore.GetKeysMulti(ctx, orgID, keySelectors)
if err != nil {
return nil, err
@@ -134,8 +133,8 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDeltaFastPath(
sb.SelectMore(col)
}
tbl := metertelemetryschema.WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := metertelemetryschema.AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
tbl := WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
if err != nil {
return "", nil, err
}
@@ -144,7 +143,7 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDeltaFastPath(
}
sb.SelectMore(fmt.Sprintf("%s AS value", aggCol))
sb.From(fmt.Sprintf("%s.%s AS points", metertelemetryschema.DBName, tbl))
sb.From(fmt.Sprintf("%s.%s AS points", DBName, tbl))
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
sb.GTE("unix_milli", start),
@@ -223,8 +222,8 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDelta(
sb.SelectMore(col)
}
tbl := metertelemetryschema.WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := metertelemetryschema.AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality,
tbl := WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality,
query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
if err != nil {
return "", nil, err
@@ -235,7 +234,7 @@ func (b *meterQueryStatementBuilder) buildTemporalAggDelta(
sb.SelectMore(fmt.Sprintf("%s AS per_series_value", aggCol))
sb.From(fmt.Sprintf("%s.%s AS points", metertelemetryschema.DBName, tbl))
sb.From(fmt.Sprintf("%s.%s AS points", DBName, tbl))
sb.Where(
sb.In("metric_name", query.Aggregations[0].MetricName),
sb.GTE("unix_milli", start),
@@ -301,14 +300,14 @@ func (b *meterQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
baseSb.SelectMore(col)
}
tbl := metertelemetryschema.WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := metertelemetryschema.AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
tbl := WhichSamplesTableToUse(start, end, query.Aggregations[0].Type, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
aggCol, err := AggregationColumnForSamplesTable(start, end, query.Aggregations[0].Type, query.Aggregations[0].Temporality, query.Aggregations[0].TimeAggregation, query.Aggregations[0].TableHints)
if err != nil {
return "", nil, err
}
baseSb.SelectMore(fmt.Sprintf("%s AS per_series_value", aggCol))
baseSb.From(fmt.Sprintf("%s.%s AS points", metertelemetryschema.DBName, tbl))
baseSb.From(fmt.Sprintf("%s.%s AS points", DBName, tbl))
baseSb.Where(
baseSb.In("metric_name", query.Aggregations[0].MetricName),
baseSb.GTE("unix_milli", start),
@@ -351,7 +350,7 @@ func (b *meterQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
for _, g := range query.GroupBy {
wrapped.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
wrapped.SelectMore(fmt.Sprintf("%s AS per_series_value", metricsstatementbuilder.RateTmpl))
wrapped.SelectMore(fmt.Sprintf("%s AS per_series_value", telemetrymetrics.RateTmpl))
wrapped.From(fmt.Sprintf("(%s) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)", innerQuery))
q, args := wrapped.BuildWithFlavor(sqlbuilder.ClickHouse, innerArgs...)
return fmt.Sprintf("__temporal_aggregation_cte AS (%s)", q), args, nil
@@ -362,7 +361,7 @@ func (b *meterQueryStatementBuilder) buildTemporalAggCumulativeOrUnspecified(
for _, g := range query.GroupBy {
wrapped.SelectMore(fmt.Sprintf("`%s`", g.Name))
}
wrapped.SelectMore(fmt.Sprintf("%s AS per_series_value", metricsstatementbuilder.IncreaseTmpl))
wrapped.SelectMore(fmt.Sprintf("%s AS per_series_value", telemetrymetrics.IncreaseTmpl))
wrapped.From(fmt.Sprintf("(%s) WINDOW rate_window AS (PARTITION BY fingerprint ORDER BY fingerprint, ts)", innerQuery))
q, args := wrapped.BuildWithFlavor(sqlbuilder.ClickHouse, innerArgs...)
return fmt.Sprintf("__temporal_aggregation_cte AS (%s)", q), args, nil

View File

@@ -1,4 +1,4 @@
package meterstatementbuilder
package telemetrymeter
import (
"context"
@@ -7,8 +7,7 @@ import (
"github.com/SigNoz/signoz/pkg/flagger/flaggertest"
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/statementbuilder/metricsstatementbuilder"
"github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
@@ -159,8 +158,8 @@ func TestStatementBuilder(t *testing.T) {
},
}
fm := metricstelemetryschema.NewFieldMapper()
cb := metricstelemetryschema.NewConditionBuilder(fm)
fm := telemetrymetrics.NewFieldMapper()
cb := telemetrymetrics.NewConditionBuilder(fm)
mockMetadataStore := telemetrytypestest.NewMockMetadataStore()
keys, err := telemetrytypestest.LoadFieldKeysFromJSON("testdata/keys_map.json")
if err != nil {
@@ -170,7 +169,7 @@ func TestStatementBuilder(t *testing.T) {
flagger := flaggertest.New(t)
metricStmtBuilder := metricsstatementbuilder.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, fm, cb, flagger)
metricStmtBuilder := telemetrymetrics.NewMetricQueryStatementBuilder(instrumentationtest.New().ToProviderSettings(), mockMetadataStore, fm, cb, flagger)
statementBuilder := NewMeterQueryStatementBuilder(
instrumentationtest.New().ToProviderSettings(),

View File

@@ -1,4 +1,4 @@
package metertelemetryschema
package telemetrymeter
import (
"time"

View File

@@ -1,4 +1,4 @@
package metricstelemetryschema
package telemetrymetrics
import (
"context"

View File

@@ -1,4 +1,4 @@
package metricstelemetryschema
package telemetrymetrics
import (
"context"

Some files were not shown because too many files have changed in this diff Show More