Compare commits

..

7 Commits

Author SHA1 Message Date
Karan Balani
0766ab31c0 feat: meter reporter for new billing infra (#11016)
Some checks are pending
build-staging / prepare (push) Waiting to run
build-staging / js-build (push) Blocked by required conditions
build-staging / go-build (push) Blocked by required conditions
build-staging / staging (push) Blocked by required conditions
Release Drafter / update_release_draft (push) Waiting to run
* feat: meter reporter for new billing infra

* feat(meterreporter): simplify code, add metric meters, dry-run zeus call

* feat(meterreporter): add traces meters

* chore: update interval validation to allow min 5 mins interval for testing

* feat: add telemetry for collect and ship durations & improve comments

* feat(meterreporter): sealed-range catch-up and today-partial ticks

* chore: intermediate commit

* feat: improve retention period queries based on workspace ids for logs only for now

* chore: skip meter checkpoint call temporarily

* feat(meterreporter): bootstrap from data floor, emit sentinel zero-readings

* chore: lower HistoricalBackfillDays

* fix(meterreporter): pin retention type

* refactor(meterreporter): remove unused retry config

* refactor: add retentiontypes

* chore: intermediate commit

* feat(meterreporter): add metric and trace meters

* refactor: cleanup comments

* refactor: remove HistoricalBackfillDays

* refactor: move few things to ee package

* refactor: simplify some sections of tick

* refactor: push meters in batch for each day

* chore: add tracing and logging

* feat: make retention buckets generic

* feat(metercollector): add MeterCollector interface and split type packages

* feat(metercollector/retention): add narrow retention slice loader and SQL helpers

* refactor(meterreporter): wire http collectors

* chore(meterreporter): trim comments

* test(metercollector): add collector coverage

* chore(meterreporter): increase catchup window

* fix: ci lint and flag default value

* refactor(meters): align retention and zeus

* refactor(retention): move ttl types

* refactor(meters): rename platform fee collector

* refactor(meters): add meter constructor

* refactor(meters): add window constructor

* refactor(meters): consolidate zeus meter types

* refactor(meters): centralize meter metadata

* refactor(retention): add getter module

* refactor(retention): consolidate ttl types

* chore: use int64 instead of float64 as meter value

* chore: int64 conversion in clickhouse query too

* chore: error log - make failed meter collection louder

* chore: start sending data to zeus

* chore: add debug statement for logging meter data

* chore: simplify meter query only use org id and retention duration

* chore: remove unused functions from retention module and move sqlbuilder function too

* chore: remove unused code

* chore: switch to info context log for testing

* refactor(meterreporter): consolidate collectors and push origin into source

Replaces six near-duplicate collector packages with two parametrized,
factory-shaped ones: telemetrymetercollector for the ClickHouse-backed
meters (log size, span size, datapoint count) and staticmetercollector
for fixed-value meters (base platform fee). Each meter is now a Config
entry in cmd/enterprise/meter.go, materialized by iterating the factory.

Pushes the catchup floor concept out of the reporter and into each
collector via a new Origin method. Telemetry collectors return per-meter
min(unix_milli) FROM signoz_meter.samples; static collectors return
todayStart. The reporter now computes per-meter next-day-to-report and
only invokes a collector for days at/after its own next, eliminating
the over-emit + dropCheckpointed dance.

Other tightening: typed Meter.MeterName with JSON marshalers; Meter
dimensions built via attribute.Key-based zeustypes.NewDimensions;
license flows into Collect from the reporter (collectors stop fetching
it themselves); providerSettings plumbed into the meterreporter
factory closure for harness-style provider construction.

* refactor(meterreporter): per-collector Origin, simpler tick, semconv metrics

Pushes the catchup-floor concept out of the reporter and into each
collector via MeterCollector.Origin. Telemetry collectors return per-
meter min(unix_milli) FROM signoz_meter.samples; static collectors
return today. The reporter computes per-meter next-reportable-day,
iterates the day-loop globally, and only invokes a collector for days
at/after its own next — eliminating the over-emit + dropCheckpointed
dance entirely.

collectOrg is split into three named helpers: provider.checkpoints
(Zeus call + index), provider.nextDays (per-meter origin + checkpoint
max), and pure backfillRange (start/end clamped to yesterday + cap).
collectOrg itself reads as a five-step recipe.

Provider stores collectors as map[MeterName]MeterCollector keyed by
name; the slice + sort.Slice scaffolding is gone, validation moves
into newProvider. eligibleCollectors and report take the map directly.

Start matches the opaquetokenizer pattern: synchronous select+ticker,
sharder + per-org loop with license check (skipping orgs with no
active license), per-tick span scoped via an IIFE so defer span.End()
fires once per tick. goroutinesWg removed.

Config drops Timeout. CatchupMaxDaysPerTick renamed to MaxBackfillDays.
runPhase renamed to report. telemetryStore injection removed (no
longer used after dataFloor moved into the telemetry collector).

Metrics rebuilt around OTel semconv: signoz.meterreporter.checkpoints,
.reports, .collections, .meters — each bumped on success and failure,
with error.type set on failure via a new errors.TypeAttr helper in
pkg/errors. collections also carries signoz.meter.name.

* refactor(meterreporter): rename base platform fee meter, add metric units

Renames signoz.meter.base.platform.fee to signoz.meter.platform.active.
The new name matches the per-service template signoz.meter.<service>
.active that scales for future per-service billing meters; "active"
fits the billing-eligibility semantic (org's platform subscription
is active for the period) without conflating with operational
liveness conventions like Prometheus's `up`.

Adds UCUM annotated-count units to each reporter counter:
  - signoz.meterreporter.checkpoints  -> {checkpoint}
  - signoz.meterreporter.reports      -> {report}
  - signoz.meterreporter.collections  -> {collection}
  - signoz.meterreporter.meters       -> {meter}

* chore: stop leaking collectors if flag is false  and address comments

* fix(meterreporter): correct startup and retention metadata

* fix(meterreporter): recover static meter backfill

* chore: address review comments

* chore: move flag evaluation into reporter

* refactor: fix retention origin for staticmeter collectors

* fix(meterreporter): gate backfill by license day

Replace max_backfill_days with a backfill switch.
Clamp sealed-day catch-up to the license creation day.

Send retention duration dimensions in seconds.

* fix(meterreporter): anchor backfill to license day

* chore: address review comments

* chore: drop unrelated authz schema diff

---------

Co-authored-by: Karan Balani <29383381+balanikaran@users.noreply.github.com>
Co-authored-by: grandwizard28 <vibhupandey28@gmail.com>
2026-05-11 17:47:29 +00:00
Vinicius Lourenço
3787715fd8 fix(global-time): initialize store with default value instead of 0 (#11257) 2026-05-11 13:08:08 +00:00
shubham yadav
0fbfff353a refactor(icons): replace LoadingOutlined with SigNoz Loader (#10839)
* fix:replace LoadingOutlined to Loader signoz icon

* fix: remove unintended changes

* add correct import for icon and size

* fix test and linting

* fix test and linting

* fix test and linting

* fix test and linting

* fix linting

* fix(src): missing few places after merge main

* fix(hostslisttable): removed file added by accident

---------

Co-authored-by: Vinícius Lourenço <vinicius@signoz.io>
Co-authored-by: Vinicius Lourenço <12551007+H4ad@users.noreply.github.com>
2026-05-11 13:05:31 +00:00
Nikhil Mantri
ca8cc30c34 feat(infra-monitoring): v2 clusters list api (#11132)
* chore: baseline setup

* chore: endpoint detail update

* chore: added logic for hosts v3 api

* fix: bug fix

* chore: disk usage

* chore: added validate function

* chore: added some unit tests

* chore: return status as a string

* chore: yarn generate api

* chore: removed isSendingK8sAgentsMetricsCode

* chore: moved funcs

* chore: added validation on order by

* chore: added pods list logic

* chore: updated openapi yml

* chore: updated spec

* chore: pods api meta start time

* chore: nil pointer check

* chore: nil pointer dereference fix in req.Filter

* chore: added temporalities of metrics

* chore: added pods metrics temporality

* chore: unified composite key function

* chore: code improvements

* chore: added pods list api updates

* chore: hostStatusNone added for clarity that this field can be left empty as well in payload

* chore: yarn generate api

* chore: return errors from getMetadata and lint fix

* chore: return errors from getMetadata and lint fix

* chore: added hostName logic

* chore: modified getMetadata query

* chore: add type for response and files rearrange

* chore: warnings added passing from queryResponse warning to host lists response struct

* chore: added better metrics existence check

* chore: added a TODO remark

* chore: added required metrics check

* chore: distributed samples table to local table change for get metadata

* chore: frontend fix

* chore: endpoint correction

* chore: endpoint modification openapi

* chore: escape backtick to prevent sql injection

* chore: rearrage

* chore: improvements

* chore: validate order by to validate function

* chore: improved description

* chore: added TODOs and made filterByStatus a part of filter struct

* chore: ignore empty string hosts in get active hosts

* feat(infra-monitoring): v2 hosts list - return counts of active & inactive hosts for custom group by attributes (#10956)

* chore: add functionality for showing active and inactive counts in custom group by

* chore: bug fix

* chore: added subquery for active and total count

* chore: ignore empty string hosts in get active hosts

* fix: sinceUnixMilli for determining active hosts compute once per request

* chore: refactor code

* chore: rename HostsList -> ListHosts

* chore: rearrangement

* chore: inframonitoring types renaming

* chore: added types package

* chore: file structure further breakdown for clarity

* chore: comments correction

* chore: removed temporalities

* chore: pods code restructuring

* chore: comments resolve

* chore: added json tag required: true

* chore: removed pod metric temporalities

* chore: removed internal server error

* chore: added status unauthorized

* chore: remove a defensive nil map check, the function ensure non-nil map when err nil

* chore: cleanup and rename

* chore: make sort stable in case of tiebreaker by comparing composite group by keys

* chore: regen api client for inframonitoring

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: added phase counts feature

* chore: added queries for pod phase counts in custom group by

* chore: added required tags

* chore: added support for pod phase unknown

* chore: removed pods - order by phase

* chore: improved api description to document -1 as no data in numeric fields

* fix: rebase fixes

* chore: added unknown phase count

* fix: isPodUIDInGroupBy in buildPodRecords

* chore: 3 cte --> 2 cte

* chore: pod phase with local table of time series as counts

* chore: comment correction

* chore: corrected comment

* chore: value column for samples table added

* chore: removed query G for phase counts

* chore: rename variable

* chore: added PodPhaseNum constants to types

* feat(infra-monitoring): v2 pods list apis - phase counts when custom grouping (#11088)

* chore: added phase counts feature

* chore: added queries for pod phase counts in custom group by

* chore: added unknown phase count

* fix: isPodUIDInGroupBy in buildPodRecords

* chore: 3 cte --> 2 cte

* chore: pod phase with local table of time series as counts

* chore: comment correction

* chore: corrected comment

* chore: value column for samples table added

* chore: removed query G for phase counts

* chore: rename variable

* chore: added PodPhaseNum constants to types

* chore: nodes list v2 full blown

* chore: metadata fix

* chore: updated comment

* chore: namespaces code

* chore: v2 nodes api

* chore: rename

* chore: v2 clusters list api

* chore: namespaces code

* chore: rename

* chore: review clusters PR

* chore: added pod phase counts

* chore: for pods and nodes, replace none with no_data

* chore: node and pod counts structs added

* chore: namespace record uses PodCountsByPhase

* chore: cluster record uses PodCountsByPhase, NodeCountsByReadiness

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Ashwin Bhatkal <ashwin96@gmail.com>
2026-05-11 12:10:42 +00:00
SagarRajput-7
85a8611f2e feat(authz): enable multi role assignment for service account (#11231)
* feat(sa): made service account role selection multiselect, handling multiple api calls parallely

* feat(authz): enable multi role assignemnt for service accounts

* feat(authz): update integration tests

---------

Co-authored-by: vikrantgupta25 <vikrant@signoz.io>
2026-05-11 11:49:32 +00:00
Nikhil Mantri
f0b533b198 feat(infra-monitoring): v2 namespaces list api (#11131)
Some checks failed
build-staging / prepare (push) Has been cancelled
build-staging / js-build (push) Has been cancelled
build-staging / go-build (push) Has been cancelled
build-staging / staging (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
* chore: baseline setup

* chore: endpoint detail update

* chore: added logic for hosts v3 api

* fix: bug fix

* chore: disk usage

* chore: added validate function

* chore: added some unit tests

* chore: return status as a string

* chore: yarn generate api

* chore: removed isSendingK8sAgentsMetricsCode

* chore: moved funcs

* chore: added validation on order by

* chore: added pods list logic

* chore: updated openapi yml

* chore: updated spec

* chore: pods api meta start time

* chore: nil pointer check

* chore: nil pointer dereference fix in req.Filter

* chore: added temporalities of metrics

* chore: added pods metrics temporality

* chore: unified composite key function

* chore: code improvements

* chore: added pods list api updates

* chore: hostStatusNone added for clarity that this field can be left empty as well in payload

* chore: yarn generate api

* chore: return errors from getMetadata and lint fix

* chore: return errors from getMetadata and lint fix

* chore: added hostName logic

* chore: modified getMetadata query

* chore: add type for response and files rearrange

* chore: warnings added passing from queryResponse warning to host lists response struct

* chore: added better metrics existence check

* chore: added a TODO remark

* chore: added required metrics check

* chore: distributed samples table to local table change for get metadata

* chore: frontend fix

* chore: endpoint correction

* chore: endpoint modification openapi

* chore: escape backtick to prevent sql injection

* chore: rearrage

* chore: improvements

* chore: validate order by to validate function

* chore: improved description

* chore: added TODOs and made filterByStatus a part of filter struct

* chore: ignore empty string hosts in get active hosts

* feat(infra-monitoring): v2 hosts list - return counts of active & inactive hosts for custom group by attributes (#10956)

* chore: add functionality for showing active and inactive counts in custom group by

* chore: bug fix

* chore: added subquery for active and total count

* chore: ignore empty string hosts in get active hosts

* fix: sinceUnixMilli for determining active hosts compute once per request

* chore: refactor code

* chore: rename HostsList -> ListHosts

* chore: rearrangement

* chore: inframonitoring types renaming

* chore: added types package

* chore: file structure further breakdown for clarity

* chore: comments correction

* chore: removed temporalities

* chore: pods code restructuring

* chore: comments resolve

* chore: added json tag required: true

* chore: removed pod metric temporalities

* chore: removed internal server error

* chore: added status unauthorized

* chore: remove a defensive nil map check, the function ensure non-nil map when err nil

* chore: cleanup and rename

* chore: make sort stable in case of tiebreaker by comparing composite group by keys

* chore: regen api client for inframonitoring

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: added phase counts feature

* chore: added queries for pod phase counts in custom group by

* chore: added required tags

* chore: added support for pod phase unknown

* chore: removed pods - order by phase

* chore: improved api description to document -1 as no data in numeric fields

* fix: rebase fixes

* chore: added unknown phase count

* fix: isPodUIDInGroupBy in buildPodRecords

* chore: 3 cte --> 2 cte

* chore: pod phase with local table of time series as counts

* chore: comment correction

* chore: corrected comment

* chore: value column for samples table added

* chore: removed query G for phase counts

* chore: rename variable

* chore: added PodPhaseNum constants to types

* feat(infra-monitoring): v2 pods list apis - phase counts when custom grouping (#11088)

* chore: added phase counts feature

* chore: added queries for pod phase counts in custom group by

* chore: added unknown phase count

* fix: isPodUIDInGroupBy in buildPodRecords

* chore: 3 cte --> 2 cte

* chore: pod phase with local table of time series as counts

* chore: comment correction

* chore: corrected comment

* chore: value column for samples table added

* chore: removed query G for phase counts

* chore: rename variable

* chore: added PodPhaseNum constants to types

* chore: nodes list v2 full blown

* chore: metadata fix

* chore: updated comment

* chore: v2 nodes api

* chore: namespaces code

* chore: rename

* chore: added pod phase counts

* chore: for pods and nodes, replace none with no_data

* chore: node and pod counts structs added

* chore: namespace record uses PodCountsByPhase

* chore: merge error resolved

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Ashwin Bhatkal <ashwin96@gmail.com>
2026-05-11 06:35:01 +00:00
Pandey
a2b4a685ad feat(auditor): add file provider (#11252)
* feat(auditor): add file provider for audit logs

* feat(auditor): write payload + newline in a single syscall

Combine the JSON payload and trailing newline into one Write call so a
concurrent reader can never observe a torn JSON object — they see either
the full line or nothing.

* style(auditor): drop wrapping at file-provider export and obvious doc comments
2026-05-11 06:23:42 +00:00
108 changed files with 4584 additions and 559 deletions

View File

@@ -18,16 +18,19 @@ import (
"github.com/SigNoz/signoz/pkg/cache"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/gateway"
"github.com/SigNoz/signoz/pkg/gateway/noopgateway"
"github.com/SigNoz/signoz/pkg/global"
"github.com/SigNoz/signoz/pkg/licensing"
"github.com/SigNoz/signoz/pkg/licensing/nooplicensing"
"github.com/SigNoz/signoz/pkg/meterreporter"
"github.com/SigNoz/signoz/pkg/modules/cloudintegration"
"github.com/SigNoz/signoz/pkg/modules/cloudintegration/implcloudintegration"
"github.com/SigNoz/signoz/pkg/modules/dashboard"
"github.com/SigNoz/signoz/pkg/modules/dashboard/impldashboard"
"github.com/SigNoz/signoz/pkg/modules/organization"
"github.com/SigNoz/signoz/pkg/modules/retention"
"github.com/SigNoz/signoz/pkg/modules/rulestatehistory"
"github.com/SigNoz/signoz/pkg/modules/serviceaccount"
"github.com/SigNoz/signoz/pkg/prometheus"
@@ -109,6 +112,9 @@ func runServer(ctx context.Context, config signoz.Config, logger *slog.Logger) e
func(_ licensing.Licensing) factory.NamedMap[factory.ProviderFactory[auditor.Auditor, auditor.Config]] {
return signoz.NewAuditorProviderFactories()
},
func(_ context.Context, _ factory.ProviderSettings, _ flagger.Flagger, _ licensing.Licensing, _ telemetrystore.TelemetryStore, _ retention.Getter, _ organization.Getter, _ zeus.Zeus) (factory.NamedMap[factory.ProviderFactory[meterreporter.Reporter, meterreporter.Config]], string) {
return signoz.NewMeterReporterProviderFactories(), "noop"
},
func(ps factory.ProviderSettings, q querier.Querier, a analytics.Analytics) querier.Handler {
return querier.NewHandler(ps, q, a)
},

55
cmd/enterprise/meter.go Normal file
View File

@@ -0,0 +1,55 @@
package main
import (
"github.com/SigNoz/signoz/pkg/metercollector"
"github.com/SigNoz/signoz/pkg/telemetrylogs"
"github.com/SigNoz/signoz/pkg/telemetrymetrics"
"github.com/SigNoz/signoz/pkg/telemetrytraces"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
"github.com/SigNoz/signoz/pkg/types/zeustypes"
)
var meterConfigs = []metercollector.Config{
{
Provider: metercollector.ProviderStatic,
Static: metercollector.StaticConfig{
Name: zeustypes.MeterPlatformActive,
Unit: zeustypes.MeterUnitCount,
Aggregation: zeustypes.MeterAggregationMax,
Value: 1,
},
},
{
Provider: metercollector.ProviderTelemetry,
Telemetry: metercollector.TelemetryConfig{
Name: zeustypes.MeterLogSize,
Unit: zeustypes.MeterUnitBytes,
Aggregation: zeustypes.MeterAggregationSum,
DBName: telemetrylogs.DBName,
TableName: telemetrylogs.LogsV2LocalTableName,
DefaultRetentionDays: retentiontypes.DefaultLogsRetentionDays,
},
},
{
Provider: metercollector.ProviderTelemetry,
Telemetry: metercollector.TelemetryConfig{
Name: zeustypes.MeterSpanSize,
Unit: zeustypes.MeterUnitBytes,
Aggregation: zeustypes.MeterAggregationSum,
DBName: telemetrytraces.DBName,
TableName: telemetrytraces.SpanIndexV3LocalTableName,
DefaultRetentionDays: retentiontypes.DefaultTracesRetentionDays,
},
},
{
Provider: metercollector.ProviderTelemetry,
Telemetry: metercollector.TelemetryConfig{
Name: zeustypes.MeterDatapointCount,
Unit: zeustypes.MeterUnitCount,
Aggregation: zeustypes.MeterAggregationSum,
DBName: telemetrymetrics.DBName,
TableName: telemetrymetrics.SamplesV4LocalTableName,
DefaultRetentionDays: retentiontypes.DefaultMetricsRetentionDays,
},
},
}

View File

@@ -8,6 +8,7 @@ import (
"github.com/spf13/cobra"
"github.com/SigNoz/signoz/cmd"
"github.com/SigNoz/signoz/ee/auditor/fileauditor"
"github.com/SigNoz/signoz/ee/auditor/otlphttpauditor"
"github.com/SigNoz/signoz/ee/authn/callbackauthn/oidccallbackauthn"
"github.com/SigNoz/signoz/ee/authn/callbackauthn/samlcallbackauthn"
@@ -17,6 +18,9 @@ import (
"github.com/SigNoz/signoz/ee/gateway/httpgateway"
enterpriselicensing "github.com/SigNoz/signoz/ee/licensing"
"github.com/SigNoz/signoz/ee/licensing/httplicensing"
"github.com/SigNoz/signoz/ee/metercollector/staticmetercollector"
"github.com/SigNoz/signoz/ee/metercollector/telemetrymetercollector"
"github.com/SigNoz/signoz/ee/meterreporter/httpmeterreporter"
"github.com/SigNoz/signoz/ee/modules/cloudintegration/implcloudintegration"
"github.com/SigNoz/signoz/ee/modules/cloudintegration/implcloudintegration/implcloudprovider"
"github.com/SigNoz/signoz/ee/modules/dashboard/impldashboard"
@@ -35,14 +39,17 @@ import (
"github.com/SigNoz/signoz/pkg/cache"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
pkgflagger "github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/gateway"
"github.com/SigNoz/signoz/pkg/global"
"github.com/SigNoz/signoz/pkg/licensing"
"github.com/SigNoz/signoz/pkg/meterreporter"
"github.com/SigNoz/signoz/pkg/modules/cloudintegration"
pkgcloudintegration "github.com/SigNoz/signoz/pkg/modules/cloudintegration/implcloudintegration"
"github.com/SigNoz/signoz/pkg/modules/dashboard"
pkgimpldashboard "github.com/SigNoz/signoz/pkg/modules/dashboard/impldashboard"
"github.com/SigNoz/signoz/pkg/modules/organization"
"github.com/SigNoz/signoz/pkg/modules/retention"
"github.com/SigNoz/signoz/pkg/modules/rulestatehistory"
"github.com/SigNoz/signoz/pkg/modules/serviceaccount"
"github.com/SigNoz/signoz/pkg/prometheus"
@@ -155,8 +162,25 @@ func runServer(ctx context.Context, config signoz.Config, logger *slog.Logger) e
if err := factories.Add(otlphttpauditor.NewFactory(licensing, version.Info)); err != nil {
panic(err)
}
if err := factories.Add(fileauditor.NewFactory(licensing, version.Info)); err != nil {
panic(err)
}
return factories
},
func(ctx context.Context, providerSettings factory.ProviderSettings, flagger pkgflagger.Flagger, licensing licensing.Licensing, telemetryStore telemetrystore.TelemetryStore, retentionGetter retention.Getter, orgGetter organization.Getter, zeus zeus.Zeus) (factory.NamedMap[factory.ProviderFactory[meterreporter.Reporter, meterreporter.Config]], string) {
factories := signoz.NewMeterReporterProviderFactories()
collectorFactories := factory.MustNewNamedMap(
staticmetercollector.NewFactory(),
telemetrymetercollector.NewFactory(telemetryStore, retentionGetter),
)
if err := factories.Add(httpmeterreporter.NewFactory(collectorFactories, meterConfigs, flagger, licensing, orgGetter, zeus)); err != nil {
panic(err)
}
return factories, "http"
},
func(ps factory.ProviderSettings, q querier.Querier, a analytics.Analytics) querier.Handler {
communityHandler := querier.NewHandler(ps, q, a)
return eequerier.NewHandler(ps, q, communityHandler)

View File

@@ -429,3 +429,10 @@ authz:
openfga:
# maximum tuples allowed per openfga write operation.
max_tuples_per_write: 300
##################### Meter Reporter #####################
meterreporter:
# The interval between collection ticks. Minimum 5m.
interval: 6h
# Whether to backfill sealed days from the license creation day.
backfill: true

View File

@@ -2520,6 +2520,65 @@ components:
enabled:
type: boolean
type: object
InframonitoringtypesClusterRecord:
properties:
clusterCPU:
format: double
type: number
clusterCPUAllocatable:
format: double
type: number
clusterMemory:
format: double
type: number
clusterMemoryAllocatable:
format: double
type: number
clusterName:
type: string
meta:
additionalProperties:
type: string
nullable: true
type: object
nodeCountsByReadiness:
$ref: '#/components/schemas/InframonitoringtypesNodeCountsByReadiness'
podCountsByPhase:
$ref: '#/components/schemas/InframonitoringtypesPodCountsByPhase'
required:
- clusterName
- clusterCPU
- clusterCPUAllocatable
- clusterMemory
- clusterMemoryAllocatable
- nodeCountsByReadiness
- podCountsByPhase
- meta
type: object
InframonitoringtypesClusters:
properties:
endTimeBeforeRetention:
type: boolean
records:
items:
$ref: '#/components/schemas/InframonitoringtypesClusterRecord'
nullable: true
type: array
requiredMetricsCheck:
$ref: '#/components/schemas/InframonitoringtypesRequiredMetricsCheck'
total:
type: integer
type:
$ref: '#/components/schemas/InframonitoringtypesResponseType'
warning:
$ref: '#/components/schemas/Querybuildertypesv5QueryWarnData'
required:
- type
- records
- total
- requiredMetricsCheck
- endTimeBeforeRetention
type: object
InframonitoringtypesHostFilter:
properties:
expression:
@@ -2599,6 +2658,54 @@ components:
- requiredMetricsCheck
- endTimeBeforeRetention
type: object
InframonitoringtypesNamespaceRecord:
properties:
meta:
additionalProperties:
type: string
nullable: true
type: object
namespaceCPU:
format: double
type: number
namespaceMemory:
format: double
type: number
namespaceName:
type: string
podCountsByPhase:
$ref: '#/components/schemas/InframonitoringtypesPodCountsByPhase'
required:
- namespaceName
- namespaceCPU
- namespaceMemory
- podCountsByPhase
- meta
type: object
InframonitoringtypesNamespaces:
properties:
endTimeBeforeRetention:
type: boolean
records:
items:
$ref: '#/components/schemas/InframonitoringtypesNamespaceRecord'
nullable: true
type: array
requiredMetricsCheck:
$ref: '#/components/schemas/InframonitoringtypesRequiredMetricsCheck'
total:
type: integer
type:
$ref: '#/components/schemas/InframonitoringtypesResponseType'
warning:
$ref: '#/components/schemas/Querybuildertypesv5QueryWarnData'
required:
- type
- records
- total
- requiredMetricsCheck
- endTimeBeforeRetention
type: object
InframonitoringtypesNodeCondition:
enum:
- ready
@@ -2776,6 +2883,32 @@ components:
- requiredMetricsCheck
- endTimeBeforeRetention
type: object
InframonitoringtypesPostableClusters:
properties:
end:
format: int64
type: integer
filter:
$ref: '#/components/schemas/Querybuildertypesv5Filter'
groupBy:
items:
$ref: '#/components/schemas/Querybuildertypesv5GroupByKey'
nullable: true
type: array
limit:
type: integer
offset:
type: integer
orderBy:
$ref: '#/components/schemas/Querybuildertypesv5OrderBy'
start:
format: int64
type: integer
required:
- start
- end
- limit
type: object
InframonitoringtypesPostableHosts:
properties:
end:
@@ -2802,6 +2935,32 @@ components:
- end
- limit
type: object
InframonitoringtypesPostableNamespaces:
properties:
end:
format: int64
type: integer
filter:
$ref: '#/components/schemas/Querybuildertypesv5Filter'
groupBy:
items:
$ref: '#/components/schemas/Querybuildertypesv5GroupByKey'
nullable: true
type: array
limit:
type: integer
offset:
type: integer
orderBy:
$ref: '#/components/schemas/Querybuildertypesv5OrderBy'
start:
format: int64
type: integer
required:
- start
- end
- limit
type: object
InframonitoringtypesPostableNodes:
properties:
end:
@@ -11672,6 +11831,77 @@ paths:
summary: Health check
tags:
- health
/api/v2/infra_monitoring/clusters:
post:
deprecated: false
description: 'Returns a paginated list of Kubernetes clusters with key aggregated
metrics derived by summing per-node values within the group: CPU usage, CPU
allocatable, memory working set, memory allocatable. Each row also reports
per-group nodeCountsByReadiness ({ ready, notReady } from each node''s latest
k8s.node.condition_ready value) and per-group podCountsByPhase ({ pending,
running, succeeded, failed, unknown } from each pod''s latest k8s.pod.phase
value). Each cluster includes metadata attributes (k8s.cluster.name). The
response type is ''list'' for the default k8s.cluster.name grouping or ''grouped_list''
for custom groupBy keys; in both modes every row aggregates nodes and pods
in the group. Supports filtering via a filter expression, custom groupBy,
ordering by cpu / cpu_allocatable / memory / memory_allocatable, and pagination
via offset/limit. Also reports missing required metrics and whether the requested
time range falls before the data retention boundary. Numeric metric fields
(clusterCPU, clusterCPUAllocatable, clusterMemory, clusterMemoryAllocatable)
return -1 as a sentinel when no data is available for that field.'
operationId: ListClusters
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/InframonitoringtypesPostableClusters'
responses:
"200":
content:
application/json:
schema:
properties:
data:
$ref: '#/components/schemas/InframonitoringtypesClusters'
status:
type: string
required:
- status
- data
type: object
description: OK
"400":
content:
application/json:
schema:
$ref: '#/components/schemas/RenderErrorResponse'
description: Bad Request
"401":
content:
application/json:
schema:
$ref: '#/components/schemas/RenderErrorResponse'
description: Unauthorized
"403":
content:
application/json:
schema:
$ref: '#/components/schemas/RenderErrorResponse'
description: Forbidden
"500":
content:
application/json:
schema:
$ref: '#/components/schemas/RenderErrorResponse'
description: Internal Server Error
security:
- api_key:
- VIEWER
- tokenizer:
- VIEWER
summary: List Clusters for Infra Monitoring
tags:
- inframonitoring
/api/v2/infra_monitoring/hosts:
post:
deprecated: false
@@ -11740,6 +11970,74 @@ paths:
summary: List Hosts for Infra Monitoring
tags:
- inframonitoring
/api/v2/infra_monitoring/namespaces:
post:
deprecated: false
description: 'Returns a paginated list of Kubernetes namespaces with key aggregated
pod metrics: CPU usage and memory working set (summed across pods in the group),
plus per-group podCountsByPhase ({ pending, running, succeeded, failed, unknown
} from each pod''s latest k8s.pod.phase value in the window). Each namespace
includes metadata attributes (k8s.namespace.name, k8s.cluster.name). The response
type is ''list'' for the default k8s.namespace.name grouping or ''grouped_list''
for custom groupBy keys; in both modes every row aggregates pods in the group.
Supports filtering via a filter expression, custom groupBy, ordering by cpu
/ memory, and pagination via offset/limit. Also reports missing required metrics
and whether the requested time range falls before the data retention boundary.
Numeric metric fields (namespaceCPU, namespaceMemory) return -1 as a sentinel
when no data is available for that field.'
operationId: ListNamespaces
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/InframonitoringtypesPostableNamespaces'
responses:
"200":
content:
application/json:
schema:
properties:
data:
$ref: '#/components/schemas/InframonitoringtypesNamespaces'
status:
type: string
required:
- status
- data
type: object
description: OK
"400":
content:
application/json:
schema:
$ref: '#/components/schemas/RenderErrorResponse'
description: Bad Request
"401":
content:
application/json:
schema:
$ref: '#/components/schemas/RenderErrorResponse'
description: Unauthorized
"403":
content:
application/json:
schema:
$ref: '#/components/schemas/RenderErrorResponse'
description: Forbidden
"500":
content:
application/json:
schema:
$ref: '#/components/schemas/RenderErrorResponse'
description: Internal Server Error
security:
- api_key:
- VIEWER
- tokenizer:
- VIEWER
summary: List Namespaces for Infra Monitoring
tags:
- inframonitoring
/api/v2/infra_monitoring/nodes:
post:
deprecated: false

View File

@@ -0,0 +1,33 @@
package fileauditor
import (
"context"
"log/slog"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/types/audittypes"
)
func (provider *provider) export(ctx context.Context, events []audittypes.AuditEvent) error {
logs := audittypes.NewPLogsFromAuditEvents(events, "signoz", provider.build.Version(), "signoz.audit")
payload, err := provider.marshaler.MarshalLogs(logs)
if err != nil {
return err
}
// Combine the payload and trailing newline into one Write call so the line
// is emitted in a single syscall — concurrent readers see either the full
// line or nothing, never a torn JSON object.
payload = append(payload, '\n')
provider.mu.Lock()
defer provider.mu.Unlock()
if _, err := provider.file.Write(payload); err != nil {
provider.settings.Logger().ErrorContext(ctx, "audit export failed", errors.Attr(err), slog.Int("dropped_log_records", len(events)))
return err
}
return provider.file.Sync()
}

View File

@@ -0,0 +1,100 @@
package fileauditor
import (
"context"
"os"
"sync"
"github.com/SigNoz/signoz/pkg/auditor"
"github.com/SigNoz/signoz/pkg/auditor/auditorserver"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/licensing"
"github.com/SigNoz/signoz/pkg/types/audittypes"
"github.com/SigNoz/signoz/pkg/version"
"go.opentelemetry.io/collector/pdata/plog"
)
var _ auditor.Auditor = (*provider)(nil)
type provider struct {
settings factory.ScopedProviderSettings
config auditor.Config
licensing licensing.Licensing
build version.Build
server *auditorserver.Server
marshaler plog.JSONMarshaler
file *os.File
mu sync.Mutex
}
func NewFactory(licensing licensing.Licensing, build version.Build) factory.ProviderFactory[auditor.Auditor, auditor.Config] {
return factory.NewProviderFactory(factory.MustNewName("file"), func(ctx context.Context, providerSettings factory.ProviderSettings, config auditor.Config) (auditor.Auditor, error) {
return newProvider(ctx, providerSettings, config, licensing, build)
})
}
func newProvider(_ context.Context, providerSettings factory.ProviderSettings, config auditor.Config, licensing licensing.Licensing, build version.Build) (auditor.Auditor, error) {
settings := factory.NewScopedProviderSettings(providerSettings, "github.com/SigNoz/signoz/ee/auditor/fileauditor")
file, err := os.OpenFile(config.File.Path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
if err != nil {
return nil, errors.Wrapf(err, errors.TypeInvalidInput, auditor.ErrCodeAuditExportFailed, "failed to open audit file %q", config.File.Path)
}
provider := &provider{
settings: settings,
config: config,
licensing: licensing,
build: build,
marshaler: plog.JSONMarshaler{},
file: file,
}
server, err := auditorserver.New(settings,
auditorserver.Config{
BufferSize: config.BufferSize,
BatchSize: config.BatchSize,
FlushInterval: config.FlushInterval,
},
provider.export,
)
if err != nil {
_ = file.Close()
return nil, err
}
provider.server = server
return provider, nil
}
func (provider *provider) Start(ctx context.Context) error {
return provider.server.Start(ctx)
}
func (provider *provider) Audit(ctx context.Context, event audittypes.AuditEvent) {
if event.PrincipalAttributes.PrincipalOrgID.IsZero() {
provider.settings.Logger().WarnContext(ctx, "audit event dropped as org_id is zero")
return
}
if _, err := provider.licensing.GetActive(ctx, event.PrincipalAttributes.PrincipalOrgID); err != nil {
return
}
provider.server.Add(ctx, event)
}
func (provider *provider) Healthy() <-chan struct{} {
return provider.server.Healthy()
}
func (provider *provider) Stop(ctx context.Context) error {
serverErr := provider.server.Stop(ctx)
fileErr := provider.file.Close()
if serverErr != nil {
return serverErr
}
return fileErr
}

View File

@@ -0,0 +1,61 @@
// Package staticmetercollector emits a fixed-value meter reading per org per window.
package staticmetercollector
import (
"context"
"time"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/metercollector"
"github.com/SigNoz/signoz/pkg/types/licensetypes"
"github.com/SigNoz/signoz/pkg/types/zeustypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
var _ metercollector.MeterCollector = (*Provider)(nil)
type Provider struct {
settings factory.ScopedProviderSettings
config metercollector.StaticConfig
}
func NewFactory() factory.ProviderFactory[metercollector.MeterCollector, metercollector.Config] {
return factory.NewProviderFactory(factory.MustNewName(metercollector.ProviderStatic), func(ctx context.Context, providerSettings factory.ProviderSettings, config metercollector.Config) (metercollector.MeterCollector, error) {
return newProvider(providerSettings, config.Static), nil
},
)
}
func newProvider(providerSettings factory.ProviderSettings, config metercollector.StaticConfig) *Provider {
settings := factory.NewScopedProviderSettings(providerSettings, "github.com/SigNoz/signoz/ee/metercollector/staticmetercollector")
return &Provider{
settings: settings,
config: config,
}
}
func (provider *Provider) Name() zeustypes.MeterName { return provider.config.Name }
func (provider *Provider) Unit() zeustypes.MeterUnit { return provider.config.Unit }
func (provider *Provider) Aggregation() zeustypes.MeterAggregation {
return provider.config.Aggregation
}
func (provider *Provider) Origin(_ context.Context, _ valuer.UUID, license *licensetypes.License, _ time.Time) (time.Time, error) {
if license == nil || license.CreatedAt.IsZero() {
return time.Time{}, nil
}
createdAt := license.CreatedAt.UTC()
return time.Date(createdAt.Year(), createdAt.Month(), createdAt.Day(), 0, 0, 0, 0, time.UTC), nil
}
func (provider *Provider) Collect(_ context.Context, orgID valuer.UUID, license *licensetypes.License, window zeustypes.MeterWindow) ([]zeustypes.Meter, error) {
if license == nil || license.Key == "" {
return nil, nil
}
return []zeustypes.Meter{
zeustypes.NewMeter(provider.config.Name, provider.config.Value, provider.config.Unit, provider.config.Aggregation, window, zeustypes.NewDimensions(zeustypes.OrganizationID.String(orgID.StringValue()))),
}, nil
}

View File

@@ -0,0 +1,247 @@
// Package telemetrymetercollector collects telemetry meters (logs, traces, metrics)
// by retention. One Provider materializes per TelemetryConfig.
package telemetrymetercollector
import (
"context"
"fmt"
"regexp"
"strconv"
"strings"
"time"
"github.com/huandu/go-sqlbuilder"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/metercollector"
"github.com/SigNoz/signoz/pkg/modules/retention"
"github.com/SigNoz/signoz/pkg/telemetrymeter"
"github.com/SigNoz/signoz/pkg/telemetrystore"
"github.com/SigNoz/signoz/pkg/types/licensetypes"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
"github.com/SigNoz/signoz/pkg/types/zeustypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
var (
labelKeyPattern = regexp.MustCompile(`^[A-Za-z0-9_.\-]+$`)
labelValuePattern = regexp.MustCompile(`^[A-Za-z0-9_.\-:]+$`)
)
var _ metercollector.MeterCollector = (*Provider)(nil)
type Provider struct {
settings factory.ScopedProviderSettings
config metercollector.TelemetryConfig
telemetryStore telemetrystore.TelemetryStore
retentionGetter retention.Getter
}
func NewFactory(telemetryStore telemetrystore.TelemetryStore, retentionGetter retention.Getter) factory.ProviderFactory[metercollector.MeterCollector, metercollector.Config] {
return factory.NewProviderFactory(factory.MustNewName(metercollector.ProviderTelemetry), func(ctx context.Context, providerSettings factory.ProviderSettings, config metercollector.Config) (metercollector.MeterCollector, error) {
return newProvider(providerSettings, config.Telemetry, telemetryStore, retentionGetter), nil
},
)
}
func newProvider(
providerSettings factory.ProviderSettings,
config metercollector.TelemetryConfig,
telemetryStore telemetrystore.TelemetryStore,
retentionGetter retention.Getter,
) *Provider {
settings := factory.NewScopedProviderSettings(providerSettings, "github.com/SigNoz/signoz/ee/metercollector/telemetrymetercollector")
return &Provider{
settings: settings,
config: config,
telemetryStore: telemetryStore,
retentionGetter: retentionGetter,
}
}
func (provider *Provider) Name() zeustypes.MeterName { return provider.config.Name }
func (provider *Provider) Unit() zeustypes.MeterUnit { return provider.config.Unit }
func (provider *Provider) Aggregation() zeustypes.MeterAggregation {
return provider.config.Aggregation
}
func (provider *Provider) Origin(ctx context.Context, _ valuer.UUID, _ *licensetypes.License, todayStart time.Time) (time.Time, error) {
query, args := buildOriginQuery(provider.config.Name.String())
var minMs int64
if err := provider.telemetryStore.ClickhouseDB().QueryRow(ctx, query, args...).Scan(&minMs); err != nil {
return time.Time{}, err
}
if minMs == 0 {
return todayStart, nil
}
minDay := time.UnixMilli(minMs).UTC()
return time.Date(minDay.Year(), minDay.Month(), minDay.Day(), 0, 0, 0, 0, time.UTC), nil
}
func (provider *Provider) Collect(
ctx context.Context,
orgID valuer.UUID,
_ *licensetypes.License,
window zeustypes.MeterWindow,
) ([]zeustypes.Meter, error) {
meterName := provider.config.Name.String()
segments, err := provider.retentionGetter.GetRetentionPolicySegments(
ctx,
orgID,
provider.config.DBName,
provider.config.TableName,
provider.config.DefaultRetentionDays,
window.StartUnixMilli,
window.EndUnixMilli,
)
if err != nil {
return nil, err
}
valuesByRetentionDays := make(map[int]int64)
for _, segment := range segments {
query, args, err := buildQuery(meterName, segment)
if err != nil {
return nil, err
}
rows, err := provider.telemetryStore.ClickhouseDB().Query(ctx, query, args...)
if err != nil {
return nil, err
}
if err := func() error {
defer rows.Close()
for rows.Next() {
var retentionDays int32
var value int64
if err := rows.Scan(&retentionDays, &value); err != nil {
return err
}
valuesByRetentionDays[int(retentionDays)] += value
}
if err := rows.Err(); err != nil {
return err
}
return nil
}(); err != nil {
return nil, err
}
}
meters := make([]zeustypes.Meter, 0, len(valuesByRetentionDays))
for retentionDays, value := range valuesByRetentionDays {
meters = append(meters, zeustypes.NewMeter(provider.config.Name, value, provider.config.Unit, provider.config.Aggregation, window, buildDimensions(orgID, retentionDays)))
}
// Empty windows still emit a sentinel so checkpoints can advance.
if len(meters) == 0 && len(segments) > 0 {
meters = append(meters, zeustypes.NewMeter(provider.config.Name, 0, provider.config.Unit, provider.config.Aggregation, window, buildDimensions(orgID, segments[len(segments)-1].DefaultDays)))
}
return meters, nil
}
func buildOriginQuery(meterName string) (string, []any) {
sb := sqlbuilder.NewSelectBuilder()
sb.Select("toInt64(ifNull(min(unix_milli), 0))")
sb.From(telemetrymeter.DBName + "." + telemetrymeter.SamplesTableName)
sb.Where(sb.Equal("metric_name", meterName))
return sb.BuildWithFlavor(sqlbuilder.ClickHouse)
}
func buildQuery(meterName string, segment *retentiontypes.RetentionPolicySegment) (string, []any, error) {
retentionExpr, err := buildRetentionMultiIfSQL(segment.Rules, segment.DefaultDays)
if err != nil {
return "", nil, err
}
selects := []string{
retentionExpr + " AS retention_days",
"toInt64(ifNull(sum(value), 0)) AS value",
}
sb := sqlbuilder.NewSelectBuilder()
sb.Select(selects...)
sb.From(telemetrymeter.DBName + "." + telemetrymeter.SamplesTableName)
sb.Where(
sb.Equal("metric_name", meterName),
sb.GTE("unix_milli", segment.StartMs),
sb.LT("unix_milli", segment.EndMs),
)
sb.GroupBy("retention_days")
query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
return query, args, nil
}
func buildRetentionMultiIfSQL(rules []retentiontypes.CustomRetentionRule, defaultDays int) (string, error) {
if defaultDays <= 0 {
return "", errors.Newf(errors.TypeInvalidInput, metercollector.ErrCodeMeterCollectorInvalidCustomRetentionRule, "non-positive default retention %d", defaultDays)
}
if len(rules) == 0 {
return "toInt32(" + strconv.Itoa(defaultDays) + ")", nil
}
arms := make([]string, 0, 2*len(rules)+1)
for ruleIndex, rule := range rules {
if rule.TTLDays <= 0 {
return "", errors.Newf(errors.TypeInvalidInput, metercollector.ErrCodeMeterCollectorInvalidCustomRetentionRule, "rule %d has non-positive ttl_days %d", ruleIndex, rule.TTLDays)
}
conditionExpr, err := buildRuleConditionSQL(ruleIndex, rule)
if err != nil {
return "", err
}
arms = append(arms, conditionExpr)
arms = append(arms, strconv.Itoa(rule.TTLDays))
}
arms = append(arms, strconv.Itoa(defaultDays))
return "toInt32(multiIf(" + strings.Join(arms, ", ") + "))", nil
}
func buildRuleConditionSQL(ruleIndex int, rule retentiontypes.CustomRetentionRule) (string, error) {
if len(rule.Filters) == 0 {
return "", errors.Newf(errors.TypeInvalidInput, metercollector.ErrCodeMeterCollectorInvalidCustomRetentionRule, "rule %d has no filters", ruleIndex)
}
filterExprs := make([]string, 0, len(rule.Filters))
for filterIndex, filter := range rule.Filters {
if !labelKeyPattern.MatchString(filter.Key) {
return "", errors.Newf(errors.TypeInvalidInput, metercollector.ErrCodeMeterCollectorInvalidCustomRetentionRule, "rule %d filter %d has invalid key %q", ruleIndex, filterIndex, filter.Key)
}
if len(filter.Values) == 0 {
return "", errors.Newf(errors.TypeInvalidInput, metercollector.ErrCodeMeterCollectorInvalidCustomRetentionRule, "rule %d filter %d has no values", ruleIndex, filterIndex)
}
quoted := make([]string, len(filter.Values))
for valueIndex, value := range filter.Values {
if !labelValuePattern.MatchString(value) {
return "", errors.Newf(errors.TypeInvalidInput, metercollector.ErrCodeMeterCollectorInvalidCustomRetentionRule, "rule %d filter %d value %d is invalid %q", ruleIndex, filterIndex, valueIndex, value)
}
quoted[valueIndex] = "'" + value + "'"
}
filterExprs = append(filterExprs, fmt.Sprintf("JSONExtractString(labels, '%s') IN (%s)", filter.Key, strings.Join(quoted, ", ")))
}
return strings.Join(filterExprs, " AND "), nil
}
func buildDimensions(orgID valuer.UUID, retentionDays int) map[string]string {
retentionDurationSeconds := int64(retentionDays) * 24 * 60 * 60 // seconds
return zeustypes.NewDimensions(
zeustypes.OrganizationID.String(orgID.StringValue()),
zeustypes.RetentionDuration.String(strconv.FormatInt(retentionDurationSeconds, 10)),
)
}

View File

@@ -0,0 +1,318 @@
package httpmeterreporter
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"time"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/flagger"
"github.com/SigNoz/signoz/pkg/licensing"
"github.com/SigNoz/signoz/pkg/metercollector"
"github.com/SigNoz/signoz/pkg/meterreporter"
"github.com/SigNoz/signoz/pkg/modules/organization"
"github.com/SigNoz/signoz/pkg/types"
"github.com/SigNoz/signoz/pkg/types/featuretypes"
"github.com/SigNoz/signoz/pkg/types/licensetypes"
"github.com/SigNoz/signoz/pkg/types/zeustypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/SigNoz/signoz/pkg/zeus"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"
"go.opentelemetry.io/otel/trace"
)
var _ factory.ServiceWithHealthy = (*Provider)(nil)
type Provider struct {
settings factory.ScopedProviderSettings
config meterreporter.Config
collectorsByName map[zeustypes.MeterName]metercollector.MeterCollector
flagger flagger.Flagger
licensing licensing.Licensing
orgGetter organization.Getter
zeus zeus.Zeus
healthyC chan struct{}
stopC chan struct{}
metrics *reporterMetrics
}
func NewFactory(collectorFactories factory.NamedMap[factory.ProviderFactory[metercollector.MeterCollector, metercollector.Config]], collectorConfigs []metercollector.Config, flagger flagger.Flagger, licensing licensing.Licensing, orgGetter organization.Getter, zeus zeus.Zeus) factory.ProviderFactory[meterreporter.Reporter, meterreporter.Config] {
return factory.NewProviderFactory(factory.MustNewName("http"), func(ctx context.Context, providerSettings factory.ProviderSettings, config meterreporter.Config) (meterreporter.Reporter, error) {
return newProvider(ctx, providerSettings, config, collectorFactories, collectorConfigs, flagger, licensing, orgGetter, zeus)
},
)
}
func newProvider(
ctx context.Context,
providerSettings factory.ProviderSettings,
config meterreporter.Config,
collectorFactories factory.NamedMap[factory.ProviderFactory[metercollector.MeterCollector, metercollector.Config]],
collectorConfigs []metercollector.Config,
flagger flagger.Flagger,
licensing licensing.Licensing,
orgGetter organization.Getter,
zeus zeus.Zeus,
) (*Provider, error) {
settings := factory.NewScopedProviderSettings(providerSettings, "github.com/SigNoz/signoz/ee/meterreporter/httpmeterreporter")
collectorsByName := map[zeustypes.MeterName]metercollector.MeterCollector{}
for _, collectorConfig := range collectorConfigs {
collector, err := factory.NewProviderFromNamedMap(ctx, providerSettings, collectorConfig, collectorFactories, collectorConfig.Provider)
if err != nil {
return nil, err
}
if _, exists := collectorsByName[collector.Name()]; exists {
return nil, errors.Newf(errors.TypeAlreadyExists, errors.CodeAlreadyExists, "duplicate meter collector %q", collector.Name())
}
collectorsByName[collector.Name()] = collector
}
metrics, err := newReporterMetrics(settings.Meter())
if err != nil {
return nil, err
}
return &Provider{
settings: settings,
config: config,
collectorsByName: collectorsByName,
flagger: flagger,
licensing: licensing,
orgGetter: orgGetter,
zeus: zeus,
healthyC: make(chan struct{}),
stopC: make(chan struct{}),
metrics: metrics,
}, nil
}
func (provider *Provider) Start(ctx context.Context) error {
close(provider.healthyC)
provider.collect(ctx)
ticker := time.NewTicker(provider.config.Interval)
defer ticker.Stop()
for {
select {
case <-provider.stopC:
return nil
case <-ticker.C:
provider.collect(ctx)
}
}
}
func (provider *Provider) collect(ctx context.Context) {
ctx, span := provider.settings.Tracer().Start(ctx, "meterreporter.Collect", trace.WithAttributes(attribute.String("meterreporter.provider", "http")))
defer span.End()
orgs, err := provider.orgGetter.ListByOwnedKeyRange(ctx)
if err != nil {
span.RecordError(err)
provider.settings.Logger().ErrorContext(ctx, "failed to get orgs data", errors.Attr(err))
return
}
for _, org := range orgs {
evalCtx := featuretypes.NewFlaggerEvaluationContext(org.ID)
if !provider.flagger.BooleanOrEmpty(ctx, flagger.FeatureUseMeterReporter, evalCtx) {
provider.settings.Logger().DebugContext(ctx, "meter reporter disabled for org, skipping reporting", slog.String("org_id", org.ID.StringValue()))
continue
}
license, err := provider.licensing.GetActive(ctx, org.ID)
if err != nil {
if errors.Ast(err, errors.TypeNotFound) {
provider.settings.Logger().DebugContext(ctx, "no active license found for org, skipping reporting", slog.String("org_id", org.ID.StringValue()))
continue
}
span.RecordError(err)
provider.settings.Logger().ErrorContext(ctx, "failed to fetch active license for org", errors.Attr(err), slog.String("org_id", org.ID.StringValue()))
return
}
if err := provider.collectOrg(ctx, org, license); err != nil {
span.RecordError(err)
provider.settings.Logger().ErrorContext(ctx, "failed to collect meters", errors.Attr(err), slog.String("org_id", org.ID.StringValue()))
}
}
}
func (provider *Provider) Stop(ctx context.Context) error {
close(provider.stopC)
return nil
}
func (provider *Provider) Healthy() <-chan struct{} {
return provider.healthyC
}
func (provider *Provider) collectOrg(ctx context.Context, org *types.Organization, license *licensetypes.License) error {
now := time.Now().UTC()
// Use one timestamp so a tick cannot straddle midnight.
todayStart := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.UTC)
if provider.config.Backfill {
checkpointsByMeter, err := provider.checkpoints(ctx, license.Key)
if err != nil {
return err
}
nextByCollector := provider.nextDays(license, todayStart, checkpointsByMeter)
start, end, ok := backfillRange(nextByCollector, todayStart)
if ok {
for day := start; !day.After(end); day = day.AddDate(0, 0, 1) {
eligible := eligibleCollectors(provider.collectorsByName, nextByCollector, day)
if len(eligible) == 0 {
continue
}
window, err := zeustypes.NewMeterWindow(day.UnixMilli(), day.AddDate(0, 0, 1).UnixMilli(), true)
if err != nil {
return err
}
if err := provider.report(ctx, org.ID, license, window, eligible); err != nil {
provider.settings.Logger().WarnContext(ctx, "failed to backfill for day", errors.Attr(err), slog.String("date", day.Format("2006-01-02")))
return err
}
}
}
}
// Today's partial window: every collector is always eligible (next <= today).
if now.UnixMilli() > todayStart.UnixMilli() {
todayWindow, err := zeustypes.NewMeterWindow(todayStart.UnixMilli(), now.UnixMilli(), false)
if err != nil {
return err
}
return provider.report(ctx, org.ID, license, todayWindow, provider.collectorsByName)
}
return nil
}
func (provider *Provider) checkpoints(ctx context.Context, licenseKey string) (map[string]time.Time, error) {
list, err := provider.zeus.ListMeterCheckpoints(ctx, licenseKey)
if err != nil {
provider.metrics.checkpoints.Add(ctx, 1, metric.WithAttributes(errors.TypeAttr(err)))
return nil, err
}
provider.metrics.checkpoints.Add(ctx, 1)
checkpointsByMeter := make(map[string]time.Time, len(list))
for _, checkpoint := range list {
checkpointsByMeter[checkpoint.Name] = checkpoint.StartDate.UTC()
}
return checkpointsByMeter, nil
}
func (provider *Provider) nextDays(license *licensetypes.License, todayStart time.Time, checkpointsByMeter map[string]time.Time) map[zeustypes.MeterName]time.Time {
nextByCollector := make(map[zeustypes.MeterName]time.Time, len(provider.collectorsByName))
licenseCreatedAt := license.CreatedAt.UTC()
licenseCreatedAtDay := time.Date(licenseCreatedAt.Year(), licenseCreatedAt.Month(), licenseCreatedAt.Day(), 0, 0, 0, 0, time.UTC)
for _, collector := range provider.collectorsByName {
checkpoint, hasCheckpoint := checkpointsByMeter[collector.Name().String()]
nextByCollector[collector.Name()] = nextReportableDay(licenseCreatedAtDay, todayStart, checkpoint, hasCheckpoint)
}
return nextByCollector
}
func nextReportableDay(licenseCreatedAtDay time.Time, todayStart time.Time, checkpoint time.Time, hasCheckpoint bool) time.Time {
next := licenseCreatedAtDay
if next.IsZero() {
next = todayStart
}
if hasCheckpoint {
checkpointNext := checkpoint.AddDate(0, 0, 1)
if checkpointNext.After(next) {
next = checkpointNext
}
}
return next
}
func (provider *Provider) report(ctx context.Context, orgID valuer.UUID, license *licensetypes.License, window zeustypes.MeterWindow, collectors map[zeustypes.MeterName]metercollector.MeterCollector) error {
date := time.UnixMilli(window.StartUnixMilli).UTC().Format("2006-01-02")
meters := make([]zeustypes.Meter, 0, len(collectors))
for _, collector := range collectors {
meterAttr := attribute.String("signoz.meter.name", collector.Name().String())
collectedReadings, err := collector.Collect(ctx, orgID, license, window)
if err != nil {
provider.metrics.collections.Add(ctx, 1, metric.WithAttributes(meterAttr, errors.TypeAttr(err)))
continue
}
provider.metrics.collections.Add(ctx, 1, metric.WithAttributes(meterAttr))
meters = append(meters, collectedReadings...)
}
if len(meters) == 0 {
return nil
}
idempotencyKey := fmt.Sprintf("meterreporter:%s", date)
body, err := json.Marshal(meters)
if err != nil {
provider.metrics.reports.Add(ctx, 1, metric.WithAttributes(errors.TypeAttr(err)))
return err
}
if err := provider.zeus.PutMetersV3(ctx, license.Key, idempotencyKey, body); err != nil {
provider.metrics.reports.Add(ctx, 1, metric.WithAttributes(errors.TypeAttr(err)))
return err
}
provider.metrics.reports.Add(ctx, 1)
provider.metrics.meters.Add(ctx, int64(len(meters)))
return nil
}
// backfillRange returns the inclusive sealed-day range ending at yesterday.
func backfillRange(nextByCollector map[zeustypes.MeterName]time.Time, todayStart time.Time) (start, end time.Time, ok bool) {
yesterday := todayStart.AddDate(0, 0, -1)
for _, next := range nextByCollector {
if !next.Before(todayStart) {
continue
}
if start.IsZero() || next.Before(start) {
start = next
}
}
if start.IsZero() || start.After(yesterday) {
return time.Time{}, time.Time{}, false
}
return start, yesterday, true
}
func eligibleCollectors(collectors map[zeustypes.MeterName]metercollector.MeterCollector, nextByCollector map[zeustypes.MeterName]time.Time, day time.Time) map[zeustypes.MeterName]metercollector.MeterCollector {
eligible := make(map[zeustypes.MeterName]metercollector.MeterCollector, len(collectors))
for name, collector := range collectors {
if !nextByCollector[name].After(day) {
eligible[name] = collector
}
}
return eligible
}

View File

@@ -0,0 +1,48 @@
package httpmeterreporter
import (
"github.com/SigNoz/signoz/pkg/errors"
"go.opentelemetry.io/otel/metric"
)
type reporterMetrics struct {
checkpoints metric.Int64Counter
reports metric.Int64Counter
collections metric.Int64Counter
meters metric.Int64Counter
}
func newReporterMetrics(meter metric.Meter) (*reporterMetrics, error) {
var errs error
checkpoints, err := meter.Int64Counter("signoz.meterreporter.checkpoints", metric.WithDescription("Zeus meter checkpoint fetches."), metric.WithUnit("{checkpoint}"))
if err != nil {
errs = errors.Join(errs, err)
}
reports, err := meter.Int64Counter("signoz.meterreporter.reports", metric.WithDescription("Meter reports shipped to Zeus."), metric.WithUnit("{report}"))
if err != nil {
errs = errors.Join(errs, err)
}
collections, err := meter.Int64Counter("signoz.meterreporter.collections", metric.WithDescription("Per-meter collect calls."), metric.WithUnit("{collection}"))
if err != nil {
errs = errors.Join(errs, err)
}
meters, err := meter.Int64Counter("signoz.meterreporter.meters", metric.WithDescription("Meter readings shipped to Zeus."), metric.WithUnit("{meter}"))
if err != nil {
errs = errors.Join(errs, err)
}
if errs != nil {
return nil, errs
}
return &reporterMetrics{
checkpoints: checkpoints,
reports: reports,
collections: collections,
meters: meters,
}, nil
}

View File

@@ -150,6 +150,72 @@ func (provider *Provider) PutMetersV2(ctx context.Context, key string, data []by
return err
}
func (provider *Provider) PutMetersV3(ctx context.Context, key string, idempotencyKey string, data []byte) error {
headers := http.Header{}
if idempotencyKey != "" {
headers.Set("X-Idempotency-Key", idempotencyKey)
}
_, err := provider.doWithHeaders(
ctx,
provider.config.URL.JoinPath("/v2/meters"),
http.MethodPost,
key,
data,
headers,
)
return err
}
func (provider *Provider) ListMeterCheckpoints(ctx context.Context, key string) ([]zeustypes.MeterCheckpoint, error) {
response, err := provider.do(
ctx,
provider.config.URL.JoinPath("/v2/meters/checkpoints"),
http.MethodGet,
key,
nil,
)
if err != nil {
return nil, err
}
checkpointValues := gjson.GetBytes(response, "data")
if !checkpointValues.Exists() || checkpointValues.Type == gjson.Null {
return nil, errors.Newf(errors.TypeInternal, zeus.ErrCodeResponseMalformed, "meter checkpoints are required")
}
if !checkpointValues.IsArray() {
return nil, errors.Newf(errors.TypeInternal, zeus.ErrCodeResponseMalformed, "meter checkpoints must be an array")
}
checkpointResults := checkpointValues.Array()
checkpoints := make([]zeustypes.MeterCheckpoint, 0, len(checkpointResults))
for _, checkpointValue := range checkpointResults {
name := checkpointValue.Get("name").String()
if name == "" {
return nil, errors.Newf(errors.TypeInternal, zeus.ErrCodeResponseMalformed, "meter checkpoint name is required")
}
startDateString := checkpointValue.Get("start_date").String()
if startDateString == "" {
return nil, errors.Newf(errors.TypeInternal, zeus.ErrCodeResponseMalformed, "meter checkpoint start_date is required for %q", name)
}
startDate, err := time.Parse("2006-01-02", startDateString)
if err != nil {
return nil, errors.Wrapf(err, errors.TypeInternal, zeus.ErrCodeResponseMalformed, "parse meter checkpoint start_date %q for %q", startDateString, name)
}
checkpoints = append(checkpoints, zeustypes.MeterCheckpoint{
Name: name,
StartDate: startDate,
})
}
return checkpoints, nil
}
func (provider *Provider) PutProfile(ctx context.Context, key string, profile *zeustypes.PostableProfile) error {
body, err := json.Marshal(profile)
if err != nil {
@@ -185,12 +251,21 @@ func (provider *Provider) PutHost(ctx context.Context, key string, host *zeustyp
}
func (provider *Provider) do(ctx context.Context, url *url.URL, method string, key string, requestBody []byte) ([]byte, error) {
return provider.doWithHeaders(ctx, url, method, key, requestBody, nil)
}
func (provider *Provider) doWithHeaders(ctx context.Context, url *url.URL, method string, key string, requestBody []byte, extraHeaders http.Header) ([]byte, error) {
request, err := http.NewRequestWithContext(ctx, method, url.String(), bytes.NewBuffer(requestBody))
if err != nil {
return nil, err
}
request.Header.Set("X-Signoz-Cloud-Api-Key", key)
request.Header.Set("Content-Type", "application/json")
for k, vs := range extraHeaders {
for _, v := range vs {
request.Header.Add(k, v)
}
}
response, err := provider.httpClient.Do(request)
if err != nil {

View File

@@ -12,10 +12,14 @@ import type {
} from 'react-query';
import type {
InframonitoringtypesPostableClustersDTO,
InframonitoringtypesPostableHostsDTO,
InframonitoringtypesPostableNamespacesDTO,
InframonitoringtypesPostableNodesDTO,
InframonitoringtypesPostablePodsDTO,
ListClusters200,
ListHosts200,
ListNamespaces200,
ListNodes200,
ListPods200,
RenderErrorResponseDTO,
@@ -24,6 +28,90 @@ import type {
import { GeneratedAPIInstance } from '../../../generatedAPIInstance';
import type { ErrorType, BodyType } from '../../../generatedAPIInstance';
/**
* Returns a paginated list of Kubernetes clusters with key aggregated metrics derived by summing per-node values within the group: CPU usage, CPU allocatable, memory working set, memory allocatable. Each row also reports per-group nodeCountsByReadiness ({ ready, notReady } from each node's latest k8s.node.condition_ready value) and per-group podCountsByPhase ({ pending, running, succeeded, failed, unknown } from each pod's latest k8s.pod.phase value). Each cluster includes metadata attributes (k8s.cluster.name). The response type is 'list' for the default k8s.cluster.name grouping or 'grouped_list' for custom groupBy keys; in both modes every row aggregates nodes and pods in the group. Supports filtering via a filter expression, custom groupBy, ordering by cpu / cpu_allocatable / memory / memory_allocatable, and pagination via offset/limit. Also reports missing required metrics and whether the requested time range falls before the data retention boundary. Numeric metric fields (clusterCPU, clusterCPUAllocatable, clusterMemory, clusterMemoryAllocatable) return -1 as a sentinel when no data is available for that field.
* @summary List Clusters for Infra Monitoring
*/
export const listClusters = (
inframonitoringtypesPostableClustersDTO: BodyType<InframonitoringtypesPostableClustersDTO>,
signal?: AbortSignal,
) => {
return GeneratedAPIInstance<ListClusters200>({
url: `/api/v2/infra_monitoring/clusters`,
method: 'POST',
headers: { 'Content-Type': 'application/json' },
data: inframonitoringtypesPostableClustersDTO,
signal,
});
};
export const getListClustersMutationOptions = <
TError = ErrorType<RenderErrorResponseDTO>,
TContext = unknown,
>(options?: {
mutation?: UseMutationOptions<
Awaited<ReturnType<typeof listClusters>>,
TError,
{ data: BodyType<InframonitoringtypesPostableClustersDTO> },
TContext
>;
}): UseMutationOptions<
Awaited<ReturnType<typeof listClusters>>,
TError,
{ data: BodyType<InframonitoringtypesPostableClustersDTO> },
TContext
> => {
const mutationKey = ['listClusters'];
const { mutation: mutationOptions } = options
? options.mutation &&
'mutationKey' in options.mutation &&
options.mutation.mutationKey
? options
: { ...options, mutation: { ...options.mutation, mutationKey } }
: { mutation: { mutationKey } };
const mutationFn: MutationFunction<
Awaited<ReturnType<typeof listClusters>>,
{ data: BodyType<InframonitoringtypesPostableClustersDTO> }
> = (props) => {
const { data } = props ?? {};
return listClusters(data);
};
return { mutationFn, ...mutationOptions };
};
export type ListClustersMutationResult = NonNullable<
Awaited<ReturnType<typeof listClusters>>
>;
export type ListClustersMutationBody =
BodyType<InframonitoringtypesPostableClustersDTO>;
export type ListClustersMutationError = ErrorType<RenderErrorResponseDTO>;
/**
* @summary List Clusters for Infra Monitoring
*/
export const useListClusters = <
TError = ErrorType<RenderErrorResponseDTO>,
TContext = unknown,
>(options?: {
mutation?: UseMutationOptions<
Awaited<ReturnType<typeof listClusters>>,
TError,
{ data: BodyType<InframonitoringtypesPostableClustersDTO> },
TContext
>;
}): UseMutationResult<
Awaited<ReturnType<typeof listClusters>>,
TError,
{ data: BodyType<InframonitoringtypesPostableClustersDTO> },
TContext
> => {
const mutationOptions = getListClustersMutationOptions(options);
return useMutation(mutationOptions);
};
/**
* Returns a paginated list of hosts with key infrastructure metrics: CPU usage (%), memory usage (%), I/O wait (%), disk usage (%), and 15-minute load average. Each host includes its current status (active/inactive based on metrics reported in the last 10 minutes) and metadata attributes (e.g., os.type). Supports filtering via a filter expression, filtering by host status, custom groupBy to aggregate hosts by any attribute, ordering by any of the five metrics, and pagination via offset/limit. The response type is 'list' for the default host.name grouping or 'grouped_list' for custom groupBy keys. Also reports missing required metrics and whether the requested time range falls before the data retention boundary. Numeric metric fields (cpu, memory, wait, load15, diskUsage) return -1 as a sentinel when no data is available for that field.
* @summary List Hosts for Infra Monitoring
@@ -108,6 +196,90 @@ export const useListHosts = <
return useMutation(mutationOptions);
};
/**
* Returns a paginated list of Kubernetes namespaces with key aggregated pod metrics: CPU usage and memory working set (summed across pods in the group), plus per-group podCountsByPhase ({ pending, running, succeeded, failed, unknown } from each pod's latest k8s.pod.phase value in the window). Each namespace includes metadata attributes (k8s.namespace.name, k8s.cluster.name). The response type is 'list' for the default k8s.namespace.name grouping or 'grouped_list' for custom groupBy keys; in both modes every row aggregates pods in the group. Supports filtering via a filter expression, custom groupBy, ordering by cpu / memory, and pagination via offset/limit. Also reports missing required metrics and whether the requested time range falls before the data retention boundary. Numeric metric fields (namespaceCPU, namespaceMemory) return -1 as a sentinel when no data is available for that field.
* @summary List Namespaces for Infra Monitoring
*/
export const listNamespaces = (
inframonitoringtypesPostableNamespacesDTO: BodyType<InframonitoringtypesPostableNamespacesDTO>,
signal?: AbortSignal,
) => {
return GeneratedAPIInstance<ListNamespaces200>({
url: `/api/v2/infra_monitoring/namespaces`,
method: 'POST',
headers: { 'Content-Type': 'application/json' },
data: inframonitoringtypesPostableNamespacesDTO,
signal,
});
};
export const getListNamespacesMutationOptions = <
TError = ErrorType<RenderErrorResponseDTO>,
TContext = unknown,
>(options?: {
mutation?: UseMutationOptions<
Awaited<ReturnType<typeof listNamespaces>>,
TError,
{ data: BodyType<InframonitoringtypesPostableNamespacesDTO> },
TContext
>;
}): UseMutationOptions<
Awaited<ReturnType<typeof listNamespaces>>,
TError,
{ data: BodyType<InframonitoringtypesPostableNamespacesDTO> },
TContext
> => {
const mutationKey = ['listNamespaces'];
const { mutation: mutationOptions } = options
? options.mutation &&
'mutationKey' in options.mutation &&
options.mutation.mutationKey
? options
: { ...options, mutation: { ...options.mutation, mutationKey } }
: { mutation: { mutationKey } };
const mutationFn: MutationFunction<
Awaited<ReturnType<typeof listNamespaces>>,
{ data: BodyType<InframonitoringtypesPostableNamespacesDTO> }
> = (props) => {
const { data } = props ?? {};
return listNamespaces(data);
};
return { mutationFn, ...mutationOptions };
};
export type ListNamespacesMutationResult = NonNullable<
Awaited<ReturnType<typeof listNamespaces>>
>;
export type ListNamespacesMutationBody =
BodyType<InframonitoringtypesPostableNamespacesDTO>;
export type ListNamespacesMutationError = ErrorType<RenderErrorResponseDTO>;
/**
* @summary List Namespaces for Infra Monitoring
*/
export const useListNamespaces = <
TError = ErrorType<RenderErrorResponseDTO>,
TContext = unknown,
>(options?: {
mutation?: UseMutationOptions<
Awaited<ReturnType<typeof listNamespaces>>,
TError,
{ data: BodyType<InframonitoringtypesPostableNamespacesDTO> },
TContext
>;
}): UseMutationResult<
Awaited<ReturnType<typeof listNamespaces>>,
TError,
{ data: BodyType<InframonitoringtypesPostableNamespacesDTO> },
TContext
> => {
const mutationOptions = getListNamespacesMutationOptions(options);
return useMutation(mutationOptions);
};
/**
* Returns a paginated list of Kubernetes nodes with key metrics: CPU usage, CPU allocatable, memory working set, memory allocatable, per-group nodeCountsByReadiness ({ ready, notReady } from each node's latest k8s.node.condition_ready in the window) and per-group podCountsByPhase ({ pending, running, succeeded, failed, unknown } for pods scheduled on the listed nodes). Each node includes metadata attributes (k8s.node.uid, k8s.cluster.name). The response type is 'list' for the default k8s.node.name grouping (each row is one node with its current condition string: ready / not_ready / no_data) or 'grouped_list' for custom groupBy keys (each row aggregates nodes in the group; condition stays no_data). Supports filtering via a filter expression, custom groupBy, ordering by cpu / cpu_allocatable / memory / memory_allocatable, and pagination via offset/limit. Also reports missing required metrics and whether the requested time range falls before the data retention boundary. Numeric metric fields (nodeCPU, nodeCPUAllocatable, nodeMemory, nodeMemoryAllocatable) return -1 as a sentinel when no data is available for that field.
* @summary List Nodes for Infra Monitoring

View File

@@ -4568,6 +4568,66 @@ export interface GlobaltypesTokenizerConfigDTO {
enabled?: boolean;
}
/**
* @nullable
*/
export type InframonitoringtypesClusterRecordDTOMeta = {
[key: string]: string;
} | null;
export interface InframonitoringtypesClusterRecordDTO {
/**
* @type number
* @format double
*/
clusterCPU: number;
/**
* @type number
* @format double
*/
clusterCPUAllocatable: number;
/**
* @type number
* @format double
*/
clusterMemory: number;
/**
* @type number
* @format double
*/
clusterMemoryAllocatable: number;
/**
* @type string
*/
clusterName: string;
/**
* @type object
* @nullable true
*/
meta: InframonitoringtypesClusterRecordDTOMeta;
nodeCountsByReadiness: InframonitoringtypesNodeCountsByReadinessDTO;
podCountsByPhase: InframonitoringtypesPodCountsByPhaseDTO;
}
export interface InframonitoringtypesClustersDTO {
/**
* @type boolean
*/
endTimeBeforeRetention: boolean;
/**
* @type array
* @nullable true
*/
records: InframonitoringtypesClusterRecordDTO[] | null;
requiredMetricsCheck: InframonitoringtypesRequiredMetricsCheckDTO;
/**
* @type integer
*/
total: number;
type: InframonitoringtypesResponseTypeDTO;
warning?: Querybuildertypesv5QueryWarnDataDTO;
}
export interface InframonitoringtypesHostFilterDTO {
/**
* @type string
@@ -4653,6 +4713,55 @@ export interface InframonitoringtypesHostsDTO {
warning?: Querybuildertypesv5QueryWarnDataDTO;
}
/**
* @nullable
*/
export type InframonitoringtypesNamespaceRecordDTOMeta = {
[key: string]: string;
} | null;
export interface InframonitoringtypesNamespaceRecordDTO {
/**
* @type object
* @nullable true
*/
meta: InframonitoringtypesNamespaceRecordDTOMeta;
/**
* @type number
* @format double
*/
namespaceCPU: number;
/**
* @type number
* @format double
*/
namespaceMemory: number;
/**
* @type string
*/
namespaceName: string;
podCountsByPhase: InframonitoringtypesPodCountsByPhaseDTO;
}
export interface InframonitoringtypesNamespacesDTO {
/**
* @type boolean
*/
endTimeBeforeRetention: boolean;
/**
* @type array
* @nullable true
*/
records: InframonitoringtypesNamespaceRecordDTO[] | null;
requiredMetricsCheck: InframonitoringtypesRequiredMetricsCheckDTO;
/**
* @type integer
*/
total: number;
type: InframonitoringtypesResponseTypeDTO;
warning?: Querybuildertypesv5QueryWarnDataDTO;
}
export enum InframonitoringtypesNodeConditionDTO {
ready = 'ready',
not_ready = 'not_ready',
@@ -4836,6 +4945,34 @@ export interface InframonitoringtypesPodsDTO {
warning?: Querybuildertypesv5QueryWarnDataDTO;
}
export interface InframonitoringtypesPostableClustersDTO {
/**
* @type integer
* @format int64
*/
end: number;
filter?: Querybuildertypesv5FilterDTO;
/**
* @type array
* @nullable true
*/
groupBy?: Querybuildertypesv5GroupByKeyDTO[] | null;
/**
* @type integer
*/
limit: number;
/**
* @type integer
*/
offset?: number;
orderBy?: Querybuildertypesv5OrderByDTO;
/**
* @type integer
* @format int64
*/
start: number;
}
export interface InframonitoringtypesPostableHostsDTO {
/**
* @type integer
@@ -4864,6 +5001,34 @@ export interface InframonitoringtypesPostableHostsDTO {
start: number;
}
export interface InframonitoringtypesPostableNamespacesDTO {
/**
* @type integer
* @format int64
*/
end: number;
filter?: Querybuildertypesv5FilterDTO;
/**
* @type array
* @nullable true
*/
groupBy?: Querybuildertypesv5GroupByKeyDTO[] | null;
/**
* @type integer
*/
limit: number;
/**
* @type integer
*/
offset?: number;
orderBy?: Querybuildertypesv5OrderByDTO;
/**
* @type integer
* @format int64
*/
start: number;
}
export interface InframonitoringtypesPostableNodesDTO {
/**
* @type integer
@@ -9247,6 +9412,14 @@ export type Healthz503 = {
status: string;
};
export type ListClusters200 = {
data: InframonitoringtypesClustersDTO;
/**
* @type string
*/
status: string;
};
export type ListHosts200 = {
data: InframonitoringtypesHostsDTO;
/**
@@ -9255,6 +9428,14 @@ export type ListHosts200 = {
status: string;
};
export type ListNamespaces200 = {
data: InframonitoringtypesNamespacesDTO;
/**
* @type string
*/
status: string;
};
export type ListNodes200 = {
data: InframonitoringtypesNodesDTO;
/**

View File

@@ -2,8 +2,9 @@ import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
import { useMutation } from 'react-query';
// eslint-disable-next-line no-restricted-imports
import { useSelector } from 'react-redux';
import { LoadingOutlined, SearchOutlined } from '@ant-design/icons';
import { SearchOutlined } from '@ant-design/icons';
import { Color } from '@signozhq/design-tokens';
import { Loader } from '@signozhq/icons';
import {
Button,
Input,
@@ -516,7 +517,9 @@ export default function CeleryOverviewTable({
bordered={false}
loading={{
spinning: isLoading,
indicator: <Spin indicator={<LoadingOutlined size={14} spin />} />,
indicator: (
<Spin indicator={<Loader size={14} className="animate-spin" />} />
),
}}
locale={{
emptyText: isLoading ? null : <Typography.Text>No data</Typography.Text>,

View File

@@ -1,7 +1,8 @@
import { Typography } from '@signozhq/ui/typography';
import { ReactNode, useEffect, useState } from 'react';
import { useHistory } from 'react-router-dom';
import { CaretDownOutlined, LoadingOutlined } from '@ant-design/icons';
import { CaretDownOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Modal, Select, Spin, Tooltip, Tree, TreeDataNode } from 'antd';
import { OnboardingStatusResponse } from 'api/messagingQueues/onboarding/getOnboardingStatus';
import { QueryParams } from 'constants/query';
@@ -222,7 +223,7 @@ function AttributeCheckList({
>
{loading ? (
<div className="loader-container">
<Spin indicator={<LoadingOutlined spin />} size="large" />
<Spin indicator={<Loader className="animate-spin" />} size="large" />
</div>
) : (
<div className="modal-content">

View File

@@ -7,12 +7,9 @@ import React, {
useState,
} from 'react';
import { Virtuoso } from 'react-virtuoso';
import {
DownOutlined,
LoadingOutlined,
ReloadOutlined,
} from '@ant-design/icons';
import { DownOutlined, ReloadOutlined } from '@ant-design/icons';
import { Color } from '@signozhq/design-tokens';
import { Loader } from '@signozhq/icons';
import { Button, Checkbox, Select } from 'antd';
import { Typography } from '@signozhq/ui/typography';
import cx from 'classnames';
@@ -1700,7 +1697,7 @@ const CustomMultiSelect: React.FC<CustomMultiSelectProps> = ({
{loading && (
<div className="navigation-loading">
<div className="navigation-icons">
<LoadingOutlined />
<Loader className="animate-spin" />
</div>
<div className="navigation-text">Refreshing values...</div>
</div>
@@ -1708,7 +1705,7 @@ const CustomMultiSelect: React.FC<CustomMultiSelectProps> = ({
{!loading && waitingMessage && (
<div className="navigation-loading">
<div className="navigation-icons">
<LoadingOutlined />
<Loader className="animate-spin" />
</div>
<div className="navigation-text" title={waitingMessage}>
{waitingMessage}

View File

@@ -6,13 +6,9 @@ import React, {
useRef,
useState,
} from 'react';
import {
CloseOutlined,
DownOutlined,
LoadingOutlined,
ReloadOutlined,
} from '@ant-design/icons';
import { CloseOutlined, DownOutlined, ReloadOutlined } from '@ant-design/icons';
import { Color } from '@signozhq/design-tokens';
import { Loader } from '@signozhq/icons';
import { Select } from 'antd';
import cx from 'classnames';
import TextToolTip from 'components/TextToolTip';
@@ -581,7 +577,7 @@ const CustomSelect: React.FC<CustomSelectProps> = ({
{loading && (
<div className="navigation-loading">
<div className="navigation-icons">
<LoadingOutlined />
<Loader className="animate-spin" />
</div>
<div className="navigation-text">Refreshing values...</div>
</div>
@@ -589,7 +585,7 @@ const CustomSelect: React.FC<CustomSelectProps> = ({
{!loading && waitingMessage && (
<div className="navigation-loading">
<div className="navigation-icons">
<LoadingOutlined />
<Loader className="animate-spin" />
</div>
<div className="navigation-text" title={waitingMessage}>
{waitingMessage}

View File

@@ -16,8 +16,8 @@ interface OverviewTabProps {
account: ServiceAccountRow;
localName: string;
onNameChange: (v: string) => void;
localRole: string;
onRoleChange: (v: string | undefined) => void;
localRoles: string[];
onRolesChange: (v: string[]) => void;
isDisabled: boolean;
availableRoles: AuthtypesRoleDTO[];
rolesLoading?: boolean;
@@ -31,8 +31,8 @@ function OverviewTab({
account,
localName,
onNameChange,
localRole,
onRoleChange,
localRoles,
onRolesChange,
isDisabled,
availableRoles,
rolesLoading,
@@ -95,10 +95,15 @@ function OverviewTab({
{isDisabled ? (
<div className="sa-drawer__input-wrapper sa-drawer__input-wrapper--disabled">
<div className="sa-drawer__disabled-roles">
{localRole ? (
<Badge color="vanilla">
{availableRoles.find((r) => r.id === localRole)?.name ?? localRole}
</Badge>
{localRoles.length > 0 ? (
localRoles.map((roleId) => {
const role = availableRoles.find((r) => r.id === roleId);
return (
<Badge key={roleId} color="vanilla">
{role?.name ?? roleId}
</Badge>
);
})
) : (
<span className="sa-drawer__input-text"></span>
)}
@@ -108,14 +113,15 @@ function OverviewTab({
) : (
<RolesSelect
id="sa-roles"
mode="multiple"
roles={availableRoles}
loading={rolesLoading}
isError={rolesError}
error={rolesErrorObj}
onRefetch={onRefetchRoles}
value={localRole}
onChange={onRoleChange}
placeholder="Select role"
value={localRoles}
onChange={onRolesChange}
placeholder="Select roles"
/>
)}
</div>

View File

@@ -8,9 +8,7 @@ import { ToggleGroup, ToggleGroupItem } from '@signozhq/ui/toggle-group';
import { Pagination, Skeleton } from 'antd';
import { convertToApiError } from 'api/ErrorResponseHandlerForGeneratedAPIs';
import {
getGetServiceAccountRolesQueryKey,
getListServiceAccountsQueryKey,
useDeleteServiceAccountRole,
useGetServiceAccount,
useListServiceAccountKeys,
useUpdateServiceAccount,
@@ -37,7 +35,7 @@ import {
useQueryState,
} from 'nuqs';
import APIError from 'types/api/error';
import { retryOn429, toAPIError } from 'utils/errorUtils';
import { toAPIError } from 'utils/errorUtils';
import AddKeyModal from './AddKeyModal';
import DeleteAccountModal from './DeleteAccountModal';
@@ -92,7 +90,7 @@ function ServiceAccountDrawer({
parseAsBoolean.withDefault(false),
);
const [localName, setLocalName] = useState('');
const [localRole, setLocalRole] = useState('');
const [localRoles, setLocalRoles] = useState<string[]>([]);
const [isSaving, setIsSaving] = useState(false);
const [saveErrors, setSaveErrors] = useState<SaveError[]>([]);
@@ -140,7 +138,7 @@ function ServiceAccountDrawer({
if (!account?.id) {
roleSessionRef.current = null;
} else if (account.id !== roleSessionRef.current && !isRolesLoading) {
setLocalRole(currentRoles[0]?.id ?? '');
setLocalRoles(currentRoles.map((r) => r.id).filter(Boolean) as string[]);
roleSessionRef.current = account.id;
}
}, [account?.id, currentRoles, isRolesLoading]);
@@ -151,7 +149,13 @@ function ServiceAccountDrawer({
const isDirty =
account !== null &&
(localName !== (account.name ?? '') ||
localRole !== (currentRoles[0]?.id ?? ''));
JSON.stringify([...localRoles].sort()) !==
JSON.stringify(
currentRoles
.map((r) => r.id)
.filter(Boolean)
.sort(),
));
const {
roles: availableRoles,
@@ -179,27 +183,6 @@ function ServiceAccountDrawer({
// the retry for this mutation is safe due to the api being idempotent on backend
const { mutateAsync: updateMutateAsync } = useUpdateServiceAccount();
const { mutateAsync: deleteRole } = useDeleteServiceAccountRole({
mutation: {
retry: retryOn429,
},
});
const executeRolesOperation = useCallback(
async (accountId: string): Promise<RoleUpdateFailure[]> => {
if (localRole === '' && currentRoles[0]?.id) {
await deleteRole({
pathParams: { id: accountId, rid: currentRoles[0].id },
});
await queryClient.invalidateQueries(
getGetServiceAccountRolesQueryKey({ id: accountId }),
);
return [];
}
return applyDiff([localRole].filter(Boolean), availableRoles);
},
[localRole, currentRoles, availableRoles, applyDiff, deleteRole, queryClient],
);
const retryNameUpdate = useCallback(async (): Promise<void> => {
if (!account) {
@@ -267,7 +250,7 @@ function ServiceAccountDrawer({
const retryRolesUpdate = useCallback(async (): Promise<void> => {
try {
const failures = await executeRolesOperation(selectedAccountId ?? '');
const failures = await applyDiff([...localRoles], availableRoles);
if (failures.length === 0) {
setSaveErrors((prev) => prev.filter((e) => e.context !== 'Roles update'));
} else {
@@ -283,7 +266,7 @@ function ServiceAccountDrawer({
),
);
}
}, [selectedAccountId, executeRolesOperation, failuresToSaveErrors]);
}, [localRoles, availableRoles, applyDiff, failuresToSaveErrors]);
const handleSave = useCallback(async (): Promise<void> => {
if (!account || !isDirty) {
@@ -302,7 +285,7 @@ function ServiceAccountDrawer({
const [nameResult, rolesResult] = await Promise.allSettled([
namePromise,
executeRolesOperation(account.id),
applyDiff([...localRoles], availableRoles),
]);
const errors: SaveError[] = [];
@@ -343,8 +326,10 @@ function ServiceAccountDrawer({
account,
isDirty,
localName,
localRoles,
availableRoles,
updateMutateAsync,
executeRolesOperation,
applyDiff,
refetchAccount,
onSuccess,
queryClient,
@@ -443,9 +428,9 @@ function ServiceAccountDrawer({
account={account}
localName={localName}
onNameChange={handleNameChange}
localRole={localRole}
onRoleChange={(role): void => {
setLocalRole(role ?? '');
localRoles={localRoles}
onRolesChange={(roles): void => {
setLocalRoles(roles);
clearRoleErrors();
}}
isDisabled={isDeleted}

View File

@@ -151,7 +151,7 @@ describe('ServiceAccountDrawer', () => {
});
});
it('changing roles enables Save; clicking Save sends role add request without delete', async () => {
it('adding a role fires POST for the new role and no DELETE for existing roles', async () => {
const roleSpy = jest.fn();
const deleteSpy = jest.fn();
const user = userEvent.setup({ pointerEventsCheck: 0 });
@@ -171,6 +171,7 @@ describe('ServiceAccountDrawer', () => {
await screen.findByDisplayValue('CI Bot');
// Add signoz-viewer while keeping signoz-admin selected
await user.click(screen.getByLabelText('Roles'));
await user.click(await screen.findByTitle('signoz-viewer'));
@@ -188,6 +189,43 @@ describe('ServiceAccountDrawer', () => {
});
});
it('removing a role fires DELETE for the removed role and no POST', async () => {
const roleSpy = jest.fn();
const deleteSpy = jest.fn();
const user = userEvent.setup({ pointerEventsCheck: 0 });
server.use(
rest.post(SA_ROLES_ENDPOINT, async (req, res, ctx) => {
roleSpy(await req.json());
return res(ctx.status(200), ctx.json({ status: 'success', data: {} }));
}),
rest.delete(SA_ROLE_DELETE_ENDPOINT, (_, res, ctx) => {
deleteSpy();
return res(ctx.status(200), ctx.json({ status: 'success', data: {} }));
}),
);
renderDrawer();
await screen.findByDisplayValue('CI Bot');
// Remove the signoz-admin tag from the multi-select
const adminTag = await screen.findByTitle('signoz-admin');
const removeBtn = adminTag.querySelector(
'.ant-select-selection-item-remove',
) as Element;
await user.click(removeBtn);
const saveBtn = screen.getByRole('button', { name: /Save Changes/i });
await waitFor(() => expect(saveBtn).not.toBeDisabled());
await user.click(saveBtn);
await waitFor(() => {
expect(deleteSpy).toHaveBeenCalled();
expect(roleSpy).not.toHaveBeenCalled();
});
});
it('"Delete Service Account" opens confirm dialog; confirming sends delete request', async () => {
const deleteSpy = jest.fn();
const user = userEvent.setup({ pointerEventsCheck: 0 });

View File

@@ -1,5 +1,5 @@
import { CSSProperties } from 'react';
import { LoadingOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Spin, SpinProps } from 'antd';
import { SpinerStyle } from './styles';
@@ -7,7 +7,14 @@ import { SpinerStyle } from './styles';
function Spinner({ size, tip, height, style }: SpinnerProps): JSX.Element {
return (
<SpinerStyle height={height} style={style}>
<Spin spinning size={size} tip={tip} indicator={<LoadingOutlined spin />} />
<Spin
spinning
size={size}
tip={tip}
indicator={
<Loader className="animate-spin" role="img" aria-label="loading" />
}
/>
</SpinerStyle>
);
}

View File

@@ -10,7 +10,7 @@ import {
} from 'react';
import type { TableComponents } from 'react-virtuoso';
import { TableVirtuoso, TableVirtuosoHandle } from 'react-virtuoso';
import { LoadingOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { DndContext, pointerWithin } from '@dnd-kit/core';
import {
horizontalListSortingStrategy,
@@ -580,7 +580,10 @@ function TanStackTableInner<TData>(
className={viewStyles.tanstackLoadingOverlay}
data-testid="tanstack-infinite-loader"
>
<Spin indicator={<LoadingOutlined spin />} tip="Loading more..." />
<Spin
indicator={<Loader className="animate-spin" />}
tip="Loading more..."
/>
</div>
)}
{showPagination && pagination && (

View File

@@ -398,7 +398,7 @@ describe('useTableParams (selective URL mode — partial config object)', () =>
.filter(Boolean)
.pop();
expect(lastExpanded).toBeDefined();
expect(JSON.parse(lastExpanded!)).toEqual(
expect(JSON.parse(lastExpanded!)).toStrictEqual(
expect.arrayContaining(['row-1', 'row-2']),
);

View File

@@ -1,6 +1,6 @@
import { useMemo, useState } from 'react';
import { QueryFunctionContext, useQueries, useQuery } from 'react-query';
import { LoadingOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Spin, Switch, Table, Tooltip } from 'antd';
import { Typography } from '@signozhq/ui/typography';
import { getQueryRangeV5 } from 'api/v5/queryRange/getQueryRange';
@@ -202,7 +202,9 @@ function TopErrors({
columns={topErrorsColumnsConfig}
loading={{
spinning: isLoading || isRefetching,
indicator: <Spin indicator={<LoadingOutlined size={14} spin />} />,
indicator: (
<Spin indicator={<Loader size={14} className="animate-spin" />} />
),
}}
dataSource={isLoading || isRefetching ? [] : formattedTopErrorsData}
locale={{

View File

@@ -2,7 +2,7 @@ import { useCallback, useEffect, useMemo, useState } from 'react';
import { useQueryClient } from 'react-query';
// eslint-disable-next-line no-restricted-imports
import { useSelector } from 'react-redux';
import { LoadingOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Spin, Table } from 'antd';
import logEvent from 'api/common/logEvent';
import emptyStateUrl from 'assets/Icons/emptyState.svg';
@@ -205,7 +205,9 @@ function DomainList(): JSX.Element {
columns={columnsConfig}
loading={{
spinning: isFetching || isLoading,
indicator: <Spin indicator={<LoadingOutlined size={14} spin />} />,
indicator: (
<Spin indicator={<Loader size={14} className="animate-spin" />} />
),
}}
scroll={{ x: true }}
tableLayout="fixed"

View File

@@ -2,9 +2,8 @@ import { Fragment, useCallback, useEffect, useMemo, useState } from 'react';
import { useTranslation } from 'react-i18next';
import { UseQueryResult } from 'react-query';
import { useInterval } from 'react-use';
import { LoadingOutlined } from '@ant-design/icons';
import { Compass, ScrollText } from '@signozhq/icons';
import { Button } from '@signozhq/ui/button';
import { Button } from '@signozhq/ui';
import { Compass, Loader, ScrollText } from '@signozhq/icons';
import { Modal, Spin } from 'antd';
import setRetentionApi from 'api/settings/setRetention';
import setRetentionApiV2 from 'api/settings/setRetentionV2';
@@ -480,7 +479,11 @@ function GeneralSettings({
saveButtonText:
metricsTtlValuesPayload.status === 'pending' ? (
<span>
<Spin spinning size="small" indicator={<LoadingOutlined spin />} />{' '}
<Spin
spinning
size="small"
indicator={<Loader className="animate-spin" />}
/>{' '}
{t('retention_save_button.pending', { name: 'metrics' })}
</span>
) : (
@@ -523,7 +526,11 @@ function GeneralSettings({
saveButtonText:
tracesTtlValuesPayload.status === 'pending' ? (
<span>
<Spin spinning size="small" indicator={<LoadingOutlined spin />} />{' '}
<Spin
spinning
size="small"
indicator={<Loader className="animate-spin" />}
/>{' '}
{t('retention_save_button.pending', { name: 'traces' })}
</span>
) : (
@@ -565,7 +572,11 @@ function GeneralSettings({
saveButtonText:
logsTtlValuesPayload.status === 'pending' ? (
<span>
<Spin spinning size="small" indicator={<LoadingOutlined spin />} />{' '}
<Spin
spinning
size="small"
indicator={<Loader className="animate-spin" />}
/>{' '}
{t('retention_save_button.pending', { name: 'logs' })}
</span>
) : (

View File

@@ -9,11 +9,8 @@ import React, {
import { useQueryClient } from 'react-query';
// eslint-disable-next-line no-restricted-imports
import { useSelector } from 'react-redux'; // old code, TODO: fix this correctly
import {
LoadingOutlined,
SearchOutlined,
SyncOutlined,
} from '@ant-design/icons';
import { SearchOutlined, SyncOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Button, Input, Spin } from 'antd';
import cx from 'classnames';
import { ToggleGraphProps } from 'components/Graph/types';
@@ -338,7 +335,7 @@ function FullView({
)}
<div className="time-container">
{response.isFetching && (
<Spin spinning indicator={<LoadingOutlined spin />} />
<Spin spinning indicator={<Loader className="animate-spin" />} />
)}
<TimePreference
selectedTime={selectedTime}

View File

@@ -1,6 +1,6 @@
import { useState } from 'react';
import { useTranslation } from 'react-i18next';
import { LoadingOutlined } from '@ant-design/icons';
import { LoaderCircle } from '@signozhq/icons';
import { Button, Input, Space } from 'antd';
import { Typography } from '@signozhq/ui/typography';
import logEvent from 'api/common/logEvent';
@@ -79,7 +79,7 @@ export function RequestDashboardBtn(): JSX.Element {
className="periscope-btn primary"
icon={
isSubmittingRequestForDashboard ? (
<LoadingOutlined />
<LoaderCircle className="animate-spin" size={12} />
) : (
<Check size={12} />
)

View File

@@ -1,6 +1,6 @@
import React, { useMemo } from 'react';
import { LoadingOutlined } from '@ant-design/icons';
import { Color } from '@signozhq/design-tokens';
import { LoaderCircle } from '@signozhq/icons';
import { Spin } from 'antd';
import { CircleCheck } from 'lucide-react';
@@ -21,7 +21,13 @@ export default function QueryStatus(
const content = useMemo((): React.ReactElement => {
if (loading) {
return <Spin spinning size="small" indicator={<LoadingOutlined spin />} />;
return (
<Spin
spinning
size="small"
indicator={<LoaderCircle className="animate-spin" />}
/>
);
}
if (error) {
return (

View File

@@ -1,5 +1,5 @@
import { ReactNode, useCallback, useMemo, useState } from 'react';
import { LoadingOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Button, Popover, Spin } from 'antd';
import { Typography } from '@signozhq/ui/typography';
import { useIsDarkMode } from 'hooks/useDarkMode';
@@ -32,7 +32,13 @@ function FieldItem({
const renderContent = useMemo(() => {
if (isLoading) {
return <Spin spinning size="small" indicator={<LoadingOutlined spin />} />;
return (
<Spin
spinning
size="small"
indicator={<Loader className="animate-spin" />}
/>
);
}
if (isHovered) {

View File

@@ -1,5 +1,5 @@
import { useCallback } from 'react';
import { LoadingOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Spin, Table, TablePaginationConfig, TableProps, Tooltip } from 'antd';
import { Typography } from '@signozhq/ui/typography';
import type { SorterResult } from 'antd/es/table/interface';
@@ -79,7 +79,7 @@ function MetricsTable({
indicator: (
<Spin
data-testid="metrics-table-loading-state"
indicator={<LoadingOutlined size={14} spin />}
indicator={<Loader size={14} className="animate-spin" />}
/>
),
}}

View File

@@ -1,11 +1,8 @@
import { useEffect, useMemo, useState } from 'react';
// eslint-disable-next-line no-restricted-imports
import { useDispatch, useSelector } from 'react-redux';
import {
CheckCircleTwoTone,
CloseCircleTwoTone,
LoadingOutlined,
} from '@ant-design/icons';
import { CheckCircleTwoTone, CloseCircleTwoTone } from '@ant-design/icons';
import { LoaderCircle } from '@signozhq/icons';
import logEvent from 'api/common/logEvent';
import MessagingQueueHealthCheck from 'components/MessagingQueueHealthCheck/MessagingQueueHealthCheck';
import { QueryParams } from 'constants/query';
@@ -341,7 +338,7 @@ export default function ConnectionStatus(): JSX.Element {
<div className="label"> Status </div>
<div className="status">
{isQueryServiceLoading && <LoadingOutlined />}
{isQueryServiceLoading && <LoaderCircle className="animate-spin" />}
{!isQueryServiceLoading &&
isReceivingData &&
(getStartedSource !== 'kafka' ? (

View File

@@ -1,6 +1,6 @@
import React, { useEffect, useState } from 'react';
import { useTranslation } from 'react-i18next';
import { LoadingOutlined } from '@ant-design/icons';
import { LoaderCircle } from '@signozhq/icons';
import { Button, Card, Form, Input, Select, Space } from 'antd';
import { Typography } from '@signozhq/ui/typography';
import logEvent from 'api/common/logEvent';
@@ -272,7 +272,7 @@ export default function DataSource(): JSX.Element {
className="periscope-btn primary"
icon={
isSubmittingRequestForDataSource ? (
<LoadingOutlined />
<LoaderCircle className="animate-spin" />
) : (
<Check size={12} />
)

View File

@@ -1,6 +1,6 @@
import { useState } from 'react';
import { useTranslation } from 'react-i18next';
import { LoadingOutlined } from '@ant-design/icons';
import { LoaderCircle } from '@signozhq/icons';
import { Button, Card, Form, Input, Space } from 'antd';
import { Typography } from '@signozhq/ui/typography';
import logEvent from 'api/common/logEvent';
@@ -185,7 +185,7 @@ export default function EnvironmentDetails(): JSX.Element {
className="periscope-btn primary"
icon={
isSubmittingRequestForEnvironment ? (
<LoadingOutlined />
<LoaderCircle className="animate-spin" />
) : (
<Check size={12} />
)

View File

@@ -1,9 +1,6 @@
import { useEffect, useState } from 'react';
import {
CheckCircleTwoTone,
CloseCircleTwoTone,
LoadingOutlined,
} from '@ant-design/icons';
import { CheckCircleTwoTone, CloseCircleTwoTone } from '@ant-design/icons';
import { LoaderCircle } from '@signozhq/icons';
import logEvent from 'api/common/logEvent';
import { DEFAULT_ENTITY_VERSION } from 'constants/app';
import { PANEL_TYPES } from 'constants/queryBuilder';
@@ -245,7 +242,7 @@ export default function LogsConnectionStatus(): JSX.Element {
<div className="label"> Status </div>
<div className="status">
{(loading || isFetching) && <LoadingOutlined />}
{(loading || isFetching) && <LoaderCircle className="animate-spin" />}
{!(loading || isFetching) && isReceivingData && (
<>
<CheckCircleTwoTone twoToneColor="#52c41a" />

View File

@@ -2,9 +2,9 @@ import {
CheckCircleFilled,
CloseCircleFilled,
ExclamationCircleFilled,
LoadingOutlined,
MinusCircleFilled,
} from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Spin } from 'antd';
export function getDeploymentStage(value: string): string {
@@ -28,7 +28,17 @@ export function getDeploymentStageIcon(value: string): JSX.Element {
switch (value) {
case 'in_progress':
return (
<Spin indicator={<LoadingOutlined style={{ fontSize: 15 }} spin />} />
<Spin
indicator={
<Loader
size="large"
className="animate-spin"
role="img"
aria-label="loading"
data-icon="loading"
/>
}
/>
);
case 'deployed':
return <CheckCircleFilled />;

View File

@@ -1,6 +1,7 @@
import { useCallback, useEffect, useMemo, useState } from 'react';
import { useLocation } from 'react-router-dom';
import { LinkOutlined, LoadingOutlined } from '@ant-design/icons';
import { LinkOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import OverlayScrollbar from 'components/OverlayScrollbar/OverlayScrollbar';
import { QueryParams } from 'constants/query';
import { PANEL_TYPES } from 'constants/queryBuilder';
@@ -230,7 +231,7 @@ const useBaseAggregateOptions = ({
key={key}
icon={
isLoading ? (
<LoadingOutlined spin />
<Loader className="animate-spin" />
) : (
<span style={{ color: aggregateData?.seriesColor }}>{icon}</span>
)

View File

@@ -1,5 +1,5 @@
import { ChangeEvent, useEffect, useMemo, useState } from 'react';
import { LoadingOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Button, Input, Spin } from 'antd';
import cx from 'classnames';
import OverlayScrollbar from 'components/OverlayScrollbar/OverlayScrollbar';
@@ -215,7 +215,7 @@ function AddSpanToFunnelModal({
<Spin
className="add-span-to-funnel-modal__loading-spinner"
spinning={isFunnelDetailsLoading || isFunnelDetailsFetching}
indicator={<LoadingOutlined spin />}
indicator={<Loader className="animate-spin" />}
>
{selectedFunnelId && funnelDetails?.payload && (
<FunnelProvider

View File

@@ -1,6 +1,7 @@
import { useCallback, useState } from 'react';
import { useHistory, useLocation } from 'react-router-dom';
import { InfoCircleOutlined, LoadingOutlined } from '@ant-design/icons';
import { InfoCircleOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Button, Spin, Tooltip } from 'antd';
import { Typography } from '@signozhq/ui/typography';
import { AxiosError } from 'axios';
@@ -188,7 +189,9 @@ function Filters({
/>
</div>
)}
{isFetching && <Spin indicator={<LoadingOutlined spin />} size="small" />}
{isFetching && (
<Spin indicator={<Loader className="animate-spin" />} size="small" />
)}
{error && (
<Tooltip title={(error as AxiosError)?.message || 'Something went wrong'}>
<InfoCircleOutlined size={14} />

View File

@@ -3,6 +3,7 @@ import { useQueryClient } from 'react-query';
import {
getGetServiceAccountRolesQueryKey,
useCreateServiceAccountRole,
useDeleteServiceAccountRole,
useGetServiceAccountRoles,
} from 'api/generated/services/serviceaccount';
import type { AuthtypesRoleDTO } from 'api/generated/services/sigNoz.schemas';
@@ -44,6 +45,9 @@ export function useServiceAccountRoleManager(
const { mutateAsync: createRole } = useCreateServiceAccountRole({
mutation: { retry: retryOn429 },
});
const { mutateAsync: deleteRole } = useDeleteServiceAccountRole({
mutation: { retry: retryOn429 },
});
const invalidateRoles = useCallback(
() =>
@@ -68,14 +72,21 @@ export function useServiceAccountRoleManager(
const addedRoles = availableRoles.filter(
(r) => r.id && desiredRoleIds.has(r.id) && !currentRoleIds.has(r.id),
);
const removedRoles = currentRoles.filter(
(r) => r.id && !desiredRoleIds.has(r.id),
);
// TODO: re-enable deletes once BE for this is streamlined
const allOperations = [
...addedRoles.map((role) => ({
role,
run: (): ReturnType<typeof createRole> =>
createRole({ pathParams: { id: accountId }, data: { id: role.id } }),
})),
...removedRoles.map((role) => ({
role,
run: (): ReturnType<typeof deleteRole> =>
deleteRole({ pathParams: { id: accountId, rid: role.id ?? '' } }),
})),
];
const results = await Promise.allSettled(
@@ -106,7 +117,7 @@ export function useServiceAccountRoleManager(
return failures;
},
[accountId, currentRoles, createRole, invalidateRoles],
[accountId, currentRoles, createRole, deleteRole, invalidateRoles],
);
return {

View File

@@ -25,7 +25,7 @@ describe('computeVisualLayout', () => {
it('should handle empty input', () => {
const layout = computeVisualLayout([]);
expect(layout.totalVisualRows).toBe(0);
expect(layout.visualRows).toEqual([]);
expect(layout.visualRows).toStrictEqual([]);
});
it('should handle single root, no children — 1 visual row', () => {
@@ -36,7 +36,7 @@ describe('computeVisualLayout', () => {
});
const layout = computeVisualLayout([[root]]);
expect(layout.totalVisualRows).toBe(1);
expect(layout.visualRows[0]).toEqual([root]);
expect(layout.visualRows[0]).toStrictEqual([root]);
expect(layout.spanToVisualRow['root']).toBe(0);
});

View File

@@ -44,12 +44,12 @@ function makeSpan(
describe('getVisibleSpans', () => {
it('returns empty array for empty input', () => {
expect(getVisibleSpans([], new Set())).toEqual([]);
expect(getVisibleSpans([], new Set())).toStrictEqual([]);
});
it('returns single root span with no children', () => {
const spans = [makeSpan({ span_id: 'root', level: 0 })];
expect(getVisibleSpans(spans, new Set())).toEqual(spans);
expect(getVisibleSpans(spans, new Set())).toStrictEqual(spans);
});
it('returns all spans for flat tree (all level 0, no children)', () => {
@@ -58,7 +58,7 @@ describe('getVisibleSpans', () => {
makeSpan({ span_id: 'b', level: 0 }),
makeSpan({ span_id: 'c', level: 0 }),
];
expect(getVisibleSpans(spans, new Set())).toEqual(spans);
expect(getVisibleSpans(spans, new Set())).toStrictEqual(spans);
});
it('returns all spans when all parents are expanded', () => {
@@ -68,7 +68,7 @@ describe('getVisibleSpans', () => {
makeSpan({ span_id: 'b', level: 1 }),
];
const uncollapsed = new Set(['root']);
expect(getVisibleSpans(spans, uncollapsed)).toEqual(spans);
expect(getVisibleSpans(spans, uncollapsed)).toStrictEqual(spans);
});
it('hides children when root is collapsed', () => {
@@ -99,7 +99,7 @@ describe('getVisibleSpans', () => {
// root and A expanded, but A is collapsed
const uncollapsed = new Set(['root']); // A not in set → collapsed
const result = getVisibleSpans(spans, uncollapsed);
expect(result.map((s) => s.span_id)).toEqual(['root', 'A']);
expect(result.map((s) => s.span_id)).toStrictEqual(['root', 'A']);
});
it('collapses one subtree while sibling subtree stays visible', () => {
@@ -116,7 +116,7 @@ describe('getVisibleSpans', () => {
// root and childB expanded, childA collapsed
const uncollapsed = new Set(['root', 'childB']);
const result = getVisibleSpans(spans, uncollapsed);
expect(result.map((s) => s.span_id)).toEqual([
expect(result.map((s) => s.span_id)).toStrictEqual([
'root',
'childA', // visible but collapsed
'childB', // visible and expanded
@@ -134,11 +134,11 @@ describe('getVisibleSpans', () => {
// Collapsed
const collapsed = getVisibleSpans(spans, new Set());
expect(collapsed.map((s) => s.span_id)).toEqual(['root']);
expect(collapsed.map((s) => s.span_id)).toStrictEqual(['root']);
// Uncollapsed
const uncollapsed = getVisibleSpans(spans, new Set(['root']));
expect(uncollapsed.map((s) => s.span_id)).toEqual(['root', 'a', 'b']);
expect(uncollapsed.map((s) => s.span_id)).toStrictEqual(['root', 'a', 'b']);
});
it('hides all levels below collapsed span in deeply nested tree', () => {
@@ -154,7 +154,7 @@ describe('getVisibleSpans', () => {
// Expand L0 and L1, collapse L2
const uncollapsed = new Set(['L0', 'L1']);
const result = getVisibleSpans(spans, uncollapsed);
expect(result.map((s) => s.span_id)).toEqual(['L0', 'L1', 'L2']);
expect(result.map((s) => s.span_id)).toStrictEqual(['L0', 'L1', 'L2']);
});
it('shows only root-level spans when all parents are collapsed', () => {
@@ -166,7 +166,7 @@ describe('getVisibleSpans', () => {
];
const uncollapsed = new Set<string>(); // nothing expanded
const result = getVisibleSpans(spans, uncollapsed);
expect(result.map((s) => s.span_id)).toEqual(['root1', 'root2']);
expect(result.map((s) => s.span_id)).toStrictEqual(['root1', 'root2']);
});
it('leaf span in uncollapsed set has no effect', () => {
@@ -177,7 +177,7 @@ describe('getVisibleSpans', () => {
// leaf is in uncollapsed set but has_children=false, should make no difference
const uncollapsed = new Set(['root', 'leaf']);
const result = getVisibleSpans(spans, uncollapsed);
expect(result).toEqual(spans);
expect(result).toStrictEqual(spans);
});
it('handles 10k spans within 50ms', () => {
@@ -218,7 +218,13 @@ describe('getVisibleSpans', () => {
// root and C expanded; A and B collapsed
const uncollapsed = new Set(['root', 'C']);
const result = getVisibleSpans(spans, uncollapsed);
expect(result.map((s) => s.span_id)).toEqual(['root', 'A', 'B', 'C', 'C1']);
expect(result.map((s) => s.span_id)).toStrictEqual([
'root',
'A',
'B',
'C',
'C1',
]);
});
});
@@ -234,7 +240,7 @@ describe('getAncestorSpanIds', () => {
makeSpan({ span_id: 'child', level: 1, parent_span_id: 'root' }),
];
const ancestors = getAncestorSpanIds(spans, 'child');
expect(ancestors).toEqual(new Set(['root']));
expect(ancestors).toStrictEqual(new Set(['root']));
});
it('returns all ancestors for deeply nested span', () => {
@@ -255,7 +261,7 @@ describe('getAncestorSpanIds', () => {
makeSpan({ span_id: 'L3', level: 3, parent_span_id: 'L2' }),
];
const ancestors = getAncestorSpanIds(spans, 'L3');
expect(ancestors).toEqual(new Set(['L2', 'L1', 'L0']));
expect(ancestors).toStrictEqual(new Set(['L2', 'L1', 'L0']));
});
it('returns empty set for unknown span', () => {

View File

@@ -1,5 +1,5 @@
import { useCallback, useMemo, useState } from 'react';
import { LoadingOutlined } from '@ant-design/icons';
import { Loader } from '@signozhq/icons';
import { Empty, Spin } from 'antd';
import {
BarController,
@@ -97,7 +97,7 @@ function FunnelGraph(): JSX.Element {
}
return (
<Spin spinning={isFetching} indicator={<LoadingOutlined spin />}>
<Spin spinning={isFetching} indicator={<Loader className="animate-spin" />}>
<div className={cx('funnel-graph', `funnel-graph--${totalSteps}-columns`)}>
<div className="funnel-graph__chart-container">
<canvas ref={canvasRef} />

View File

@@ -54,9 +54,23 @@ describe('globalTimeStore', () => {
expect(result.current.lastRefreshTimestamp).toBe(0);
});
it('should have lastComputedMinMax with default values', () => {
it('should have lastComputedMinMax computed from initial selectedTime', () => {
const { result } = renderHook(() => useGlobalTimeStore());
expect(result.current.lastComputedMinMax).toStrictEqual({
// Now computes min/max on store creation, no longer starts with 0
expect(result.current.lastComputedMinMax.minTime).toBeGreaterThan(0);
expect(result.current.lastComputedMinMax.maxTime).toBeGreaterThan(0);
});
it('should not crash with invalid relative time format', () => {
// Invalid time formats should not crash store creation
const store = createGlobalTimeStore({
selectedTime: 'invalid_time' as GlobalTimeSelectedTime,
});
// Store should be created successfully
expect(store.getState().selectedTime).toBe('invalid_time');
// Should fallback to {0, 0} when parsing fails
expect(store.getState().lastComputedMinMax).toStrictEqual({
minTime: 0,
maxTime: 0,
});
@@ -724,8 +738,8 @@ describe('globalTimeStore', () => {
const wrapper = createIsolatedWrapper();
const { result } = renderHook(() => useGlobalTime(), { wrapper });
// Initial state has 0 values
expect(result.current.lastComputedMinMax.maxTime).toBe(0);
// Initial state now has computed values (no longer 0)
expect(result.current.lastComputedMinMax.maxTime).toBeGreaterThan(0);
act(() => {
result.current.setSelectedTime('15m');

View File

@@ -134,17 +134,17 @@ describe('useComputedMinMaxSync', () => {
jest.useRealTimers();
});
it('should compute min/max on mount when store has zero values', () => {
it('should have computed min/max on store creation (no longer needs mount sync)', () => {
const contextStore = createGlobalTimeStore({ selectedTime: '15m' });
expect(contextStore.getState().lastComputedMinMax).toStrictEqual({
minTime: 0,
maxTime: 0,
});
// Store now computes min/max on creation, not on mount
expect(contextStore.getState().lastComputedMinMax.minTime).toBeGreaterThan(0);
expect(contextStore.getState().lastComputedMinMax.maxTime).toBeGreaterThan(0);
// Hook still works but is a no-op when values already exist
renderHook(() => useComputedMinMaxSync(contextStore));
// Should have computed values now
// Values remain computed
expect(contextStore.getState().lastComputedMinMax.maxTime).toBeGreaterThan(0);
expect(contextStore.getState().lastComputedMinMax.minTime).toBeGreaterThan(0);
});

View File

@@ -12,6 +12,7 @@ import {
computeRounded5sMinMax,
isCustomTimeRange,
parseSelectedTime,
safeParseSelectedTime,
} from './utils';
export type GlobalTimeStoreApi = StoreApi<GlobalTimeStore>;
@@ -40,7 +41,7 @@ export function createGlobalTimeStore(
refreshInterval,
isRefreshEnabled: computeIsRefreshEnabled(selectedTime, refreshInterval),
lastRefreshTimestamp: 0,
lastComputedMinMax: { minTime: 0, maxTime: 0 },
lastComputedMinMax: safeParseSelectedTime(selectedTime),
setSelectedTime: (
time: GlobalTimeSelectedTime,

View File

@@ -61,6 +61,8 @@ const fallbackDurationInNanoSeconds = 30 * 1000 * NANO_SECOND_MULTIPLIER; // 30s
* Parse the selectedTime string to get min/max time values.
* For relative times, computes fresh values based on Date.now().
* For custom times, extracts the stored min/max values.
*
* @throws Error - When selectedTime is relativeTime and it's invalid
*/
export function parseSelectedTime(selectedTime: string): ParsedTimeRange {
if (isCustomTimeRange(selectedTime)) {
@@ -78,6 +80,18 @@ export function parseSelectedTime(selectedTime: string): ParsedTimeRange {
return getMinMaxForSelectedTime(selectedTime as Time, 0, 0);
}
/**
* The {@ref parseSelectedTime} can throw errors, this handles and fallbacks to 0,0 if invalid selected time is provided
*/
export function safeParseSelectedTime(selectedTime: string): ParsedTimeRange {
try {
return parseSelectedTime(selectedTime);
} catch (e) {
console.error('Error parsing selected time:', e);
return { minTime: 0, maxTime: 0 };
}
}
/**
* @deprecated Use store.getAutoRefreshQueryKey() instead.
* Access via: const getAutoRefreshQueryKey = useGlobalTime((s) => s.getAutoRefreshQueryKey);

View File

@@ -67,5 +67,43 @@ func (provider *provider) addInfraMonitoringRoutes(router *mux.Router) error {
return err
}
if err := router.Handle("/api/v2/infra_monitoring/namespaces", handler.New(
provider.authzMiddleware.ViewAccess(provider.infraMonitoringHandler.ListNamespaces),
handler.OpenAPIDef{
ID: "ListNamespaces",
Tags: []string{"inframonitoring"},
Summary: "List Namespaces for Infra Monitoring",
Description: "Returns a paginated list of Kubernetes namespaces with key aggregated pod metrics: CPU usage and memory working set (summed across pods in the group), plus per-group podCountsByPhase ({ pending, running, succeeded, failed, unknown } from each pod's latest k8s.pod.phase value in the window). Each namespace includes metadata attributes (k8s.namespace.name, k8s.cluster.name). The response type is 'list' for the default k8s.namespace.name grouping or 'grouped_list' for custom groupBy keys; in both modes every row aggregates pods in the group. Supports filtering via a filter expression, custom groupBy, ordering by cpu / memory, and pagination via offset/limit. Also reports missing required metrics and whether the requested time range falls before the data retention boundary. Numeric metric fields (namespaceCPU, namespaceMemory) return -1 as a sentinel when no data is available for that field.",
Request: new(inframonitoringtypes.PostableNamespaces),
RequestContentType: "application/json",
Response: new(inframonitoringtypes.Namespaces),
ResponseContentType: "application/json",
SuccessStatusCode: http.StatusOK,
ErrorStatusCodes: []int{http.StatusBadRequest, http.StatusUnauthorized},
Deprecated: false,
SecuritySchemes: newSecuritySchemes(types.RoleViewer),
})).Methods(http.MethodPost).GetError(); err != nil {
return err
}
if err := router.Handle("/api/v2/infra_monitoring/clusters", handler.New(
provider.authzMiddleware.ViewAccess(provider.infraMonitoringHandler.ListClusters),
handler.OpenAPIDef{
ID: "ListClusters",
Tags: []string{"inframonitoring"},
Summary: "List Clusters for Infra Monitoring",
Description: "Returns a paginated list of Kubernetes clusters with key aggregated metrics derived by summing per-node values within the group: CPU usage, CPU allocatable, memory working set, memory allocatable. Each row also reports per-group nodeCountsByReadiness ({ ready, notReady } from each node's latest k8s.node.condition_ready value) and per-group podCountsByPhase ({ pending, running, succeeded, failed, unknown } from each pod's latest k8s.pod.phase value). Each cluster includes metadata attributes (k8s.cluster.name). The response type is 'list' for the default k8s.cluster.name grouping or 'grouped_list' for custom groupBy keys; in both modes every row aggregates nodes and pods in the group. Supports filtering via a filter expression, custom groupBy, ordering by cpu / cpu_allocatable / memory / memory_allocatable, and pagination via offset/limit. Also reports missing required metrics and whether the requested time range falls before the data retention boundary. Numeric metric fields (clusterCPU, clusterCPUAllocatable, clusterMemory, clusterMemoryAllocatable) return -1 as a sentinel when no data is available for that field.",
Request: new(inframonitoringtypes.PostableClusters),
RequestContentType: "application/json",
Response: new(inframonitoringtypes.Clusters),
ResponseContentType: "application/json",
SuccessStatusCode: http.StatusOK,
ErrorStatusCodes: []int{http.StatusBadRequest, http.StatusUnauthorized},
Deprecated: false,
SecuritySchemes: newSecuritySchemes(types.RoleViewer),
})).Methods(http.MethodPost).GetError(); err != nil {
return err
}
return nil
}

View File

@@ -25,6 +25,8 @@ type Config struct {
FlushInterval time.Duration `mapstructure:"flush_interval"`
OTLPHTTP OTLPHTTPConfig `mapstructure:"otlphttp"`
File FileConfig `mapstructure:"file"`
}
// OTLPHTTPConfig holds configuration for the OTLP HTTP exporter provider.
@@ -46,6 +48,12 @@ type OTLPHTTPConfig struct {
Retry RetryConfig `mapstructure:"retry"`
}
type FileConfig struct {
// Path is the absolute path to the audit log file. The file is opened with
// O_APPEND|O_CREATE|O_WRONLY; existing contents are preserved across runs.
Path string `mapstructure:"path"`
}
// RetryConfig configures exponential backoff for the OTLP HTTP exporter.
type RetryConfig struct {
// Enabled controls whether retries are attempted on transient failures.
@@ -111,5 +119,11 @@ func (c Config) Validate() error {
}
}
if c.Provider == "file" {
if c.File.Path == "" {
return errors.New(errors.TypeInvalidInput, errors.CodeInvalidInput, "auditor::file::path must be set when provider is file")
}
}
return nil
}

View File

@@ -4,6 +4,8 @@ import (
"errors" //nolint:depguard
"fmt"
"log/slog"
"go.opentelemetry.io/otel/attribute"
)
// base is the fundamental struct that implements the error interface.
@@ -253,3 +255,10 @@ func NewTimeoutf(code Code, format string, args ...any) *base {
func Attr(err error) slog.Attr {
return slog.Any("exception", err)
}
// TypeAttr returns an OTel attribute.KeyValue with the "error.type" semconv key
// set to the error's type string.
func TypeAttr(err error) attribute.KeyValue {
t, _, _, _, _, _ := Unwrapb(err)
return attribute.String("error.type", t.String())
}

View File

@@ -8,6 +8,7 @@ var (
FeatureHideRootUser = featuretypes.MustNewName("hide_root_user")
FeatureGetMetersFromZeus = featuretypes.MustNewName("get_meters_from_zeus")
FeaturePutMetersInZeus = featuretypes.MustNewName("put_meters_in_zeus")
FeatureUseMeterReporter = featuretypes.MustNewName("use_meter_reporter")
FeatureUseJSONBody = featuretypes.MustNewName("use_json_body")
)
@@ -53,6 +54,14 @@ func MustNewRegistry() featuretypes.Registry {
DefaultVariant: featuretypes.MustNewName("disabled"),
Variants: featuretypes.NewBooleanVariants(),
},
&featuretypes.Feature{
Name: FeatureUseMeterReporter,
Kind: featuretypes.KindBoolean,
Stage: featuretypes.StageExperimental,
Description: "Controls whether the enterprise meter reporter runs instead of the noop reporter",
DefaultVariant: featuretypes.MustNewName("disabled"),
Variants: featuretypes.NewBooleanVariants(),
},
&featuretypes.Feature{
Name: FeatureUseJSONBody,
Kind: featuretypes.KindBoolean,

View File

@@ -0,0 +1,68 @@
package metercollector
import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/types/zeustypes"
)
const (
ProviderStatic = "static"
ProviderTelemetry = "telemetry"
)
type Config struct {
Provider string `mapstructure:"provider"`
Telemetry TelemetryConfig `mapstructure:"telemetry"`
Static StaticConfig `mapstructure:"static"`
}
func (c Config) Validate() error {
switch c.Provider {
case ProviderStatic:
return c.Static.Validate()
case ProviderTelemetry:
return c.Telemetry.Validate()
default:
return errors.Newf(errors.TypeInvalidInput, ErrCodeInvalidConfig, "meter collector: unknown provider %q", c.Provider)
}
}
type TelemetryConfig struct {
Name zeustypes.MeterName
Unit zeustypes.MeterUnit
Aggregation zeustypes.MeterAggregation
DBName string
TableName string
DefaultRetentionDays int
}
type StaticConfig struct {
Name zeustypes.MeterName
Unit zeustypes.MeterUnit
Aggregation zeustypes.MeterAggregation
Value int64
}
func (c StaticConfig) Validate() error {
if c.Name.IsZero() {
return errors.New(errors.TypeInvalidInput, ErrCodeInvalidConfig, "static meter collector: name must be set")
}
return nil
}
func (c TelemetryConfig) Validate() error {
if c.Name.IsZero() {
return errors.New(errors.TypeInvalidInput, ErrCodeInvalidConfig, "telemetry meter collector: name must be set")
}
if c.DBName == "" || c.TableName == "" {
return errors.New(errors.TypeInvalidInput, ErrCodeInvalidConfig, "telemetry meter collector: db_name and table_name are required")
}
if c.DefaultRetentionDays <= 0 {
return errors.New(errors.TypeInvalidInput, ErrCodeInvalidConfig, "telemetry meter collector: default_retention_days must be positive")
}
return nil
}

View File

@@ -0,0 +1,25 @@
package metercollector
import (
"context"
"time"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/types/licensetypes"
"github.com/SigNoz/signoz/pkg/types/zeustypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
var (
ErrCodeMeterCollectorCollectFailed = errors.MustNewCode("meter_collector_collect_failed")
ErrCodeMeterCollectorInvalidCustomRetentionRule = errors.MustNewCode("meter_collector_invalid_custom_retention_rule")
ErrCodeInvalidConfig = errors.MustNewCode("meter_collector_invalid_config")
)
type MeterCollector interface {
Name() zeustypes.MeterName
Unit() zeustypes.MeterUnit
Aggregation() zeustypes.MeterAggregation
Origin(ctx context.Context, orgID valuer.UUID, license *licensetypes.License, todayStart time.Time) (time.Time, error)
Collect(ctx context.Context, orgID valuer.UUID, license *licensetypes.License, window zeustypes.MeterWindow) ([]zeustypes.Meter, error)
}

View File

@@ -0,0 +1,37 @@
package meterreporter
import (
"time"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
)
var _ factory.Config = (*Config)(nil)
type Config struct {
// Interval is how often the reporter collects and ships meters.
Interval time.Duration `mapstructure:"interval"`
// Backfill enables sealed-day catch-up from the license creation day.
Backfill bool `mapstructure:"backfill"`
}
func newConfig() factory.Config {
return Config{
Interval: 6 * time.Hour,
Backfill: true,
}
}
func NewConfigFactory() factory.ConfigFactory {
return factory.NewConfigFactory(factory.MustNewName("meterreporter"), newConfig)
}
func (c Config) Validate() error {
if c.Interval < 5*time.Minute || c.Interval > 24*time.Hour {
return errors.New(errors.TypeInvalidInput, ErrCodeInvalidInput, "meterreporter::interval must be between 5m and 24h")
}
return nil
}

View File

@@ -0,0 +1,14 @@
package meterreporter
import (
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/factory"
)
var (
ErrCodeInvalidInput = errors.MustNewCode("meterreporter_invalid_input")
)
type Reporter interface {
factory.ServiceWithHealthy
}

View File

@@ -0,0 +1,39 @@
package noopmeterreporter
import (
"context"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/meterreporter"
)
type provider struct {
healthyC chan struct{}
stopC chan struct{}
}
func NewFactory() factory.ProviderFactory[meterreporter.Reporter, meterreporter.Config] {
return factory.NewProviderFactory(factory.MustNewName("noop"), New)
}
func New(_ context.Context, _ factory.ProviderSettings, _ meterreporter.Config) (meterreporter.Reporter, error) {
return &provider{
healthyC: make(chan struct{}),
stopC: make(chan struct{}),
}, nil
}
func (p *provider) Start(_ context.Context) error {
close(p.healthyC)
<-p.stopC
return nil
}
func (p *provider) Stop(_ context.Context) error {
close(p.stopC)
return nil
}
func (p *provider) Healthy() <-chan struct{} {
return p.healthyC
}

View File

@@ -0,0 +1,140 @@
package implinframonitoring
import (
"context"
"slices"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
)
// buildClusterRecords assembles the page records. Node condition counts and
// pod phase counts come from the respective per-group maps in both modes;
// every row is a group of nodes+pods, so there's no per-row "current state"
// concept (analogous to namespaces).
func buildClusterRecords(
resp *qbtypes.QueryRangeResponse,
pageGroups []map[string]string,
groupBy []qbtypes.GroupByKey,
metadataMap map[string]map[string]string,
nodeConditionCountsMap map[string]nodeConditionCounts,
podPhaseCountsMap map[string]podPhaseCounts,
) []inframonitoringtypes.ClusterRecord {
metricsMap := parseFullQueryResponse(resp, groupBy)
records := make([]inframonitoringtypes.ClusterRecord, 0, len(pageGroups))
for _, labels := range pageGroups {
compositeKey := compositeKeyFromLabels(labels, groupBy)
clusterName := labels[clusterNameAttrKey]
record := inframonitoringtypes.ClusterRecord{ // initialize with default values
ClusterName: clusterName,
ClusterCPU: -1,
ClusterCPUAllocatable: -1,
ClusterMemory: -1,
ClusterMemoryAllocatable: -1,
Meta: map[string]string{},
}
if metrics, ok := metricsMap[compositeKey]; ok {
if v, exists := metrics["A"]; exists {
record.ClusterCPU = v
}
if v, exists := metrics["B"]; exists {
record.ClusterCPUAllocatable = v
}
if v, exists := metrics["C"]; exists {
record.ClusterMemory = v
}
if v, exists := metrics["D"]; exists {
record.ClusterMemoryAllocatable = v
}
}
if conditionCountsForGroup, ok := nodeConditionCountsMap[compositeKey]; ok {
record.NodeCountsByReadiness = inframonitoringtypes.NodeCountsByReadiness{
Ready: conditionCountsForGroup.Ready,
NotReady: conditionCountsForGroup.NotReady,
}
}
if phaseCountsForGroup, ok := podPhaseCountsMap[compositeKey]; ok {
record.PodCountsByPhase = inframonitoringtypes.PodCountsByPhase{
Pending: phaseCountsForGroup.Pending,
Running: phaseCountsForGroup.Running,
Succeeded: phaseCountsForGroup.Succeeded,
Failed: phaseCountsForGroup.Failed,
Unknown: phaseCountsForGroup.Unknown,
}
}
if attrs, ok := metadataMap[compositeKey]; ok {
for k, v := range attrs {
record.Meta[k] = v
}
}
records = append(records, record)
}
return records
}
func (m *module) getTopClusterGroups(
ctx context.Context,
orgID valuer.UUID,
req *inframonitoringtypes.PostableClusters,
metadataMap map[string]map[string]string,
) ([]map[string]string, error) {
orderByKey := req.OrderBy.Key.Name
queryNamesForOrderBy := orderByToClustersQueryNames[orderByKey]
rankingQueryName := queryNamesForOrderBy[len(queryNamesForOrderBy)-1]
topReq := &qbtypes.QueryRangeRequest{
Start: uint64(req.Start),
End: uint64(req.End),
RequestType: qbtypes.RequestTypeScalar,
CompositeQuery: qbtypes.CompositeQuery{
Queries: make([]qbtypes.QueryEnvelope, 0, len(queryNamesForOrderBy)),
},
}
for _, envelope := range m.newClustersTableListQuery().CompositeQuery.Queries {
if !slices.Contains(queryNamesForOrderBy, envelope.GetQueryName()) {
continue
}
copied := envelope
if copied.Type == qbtypes.QueryTypeBuilder {
existingExpr := ""
if f := copied.GetFilter(); f != nil {
existingExpr = f.Expression
}
reqFilterExpr := ""
if req.Filter != nil {
reqFilterExpr = req.Filter.Expression
}
merged := mergeFilterExpressions(existingExpr, reqFilterExpr)
copied.SetFilter(&qbtypes.Filter{Expression: merged})
copied.SetGroupBy(req.GroupBy)
}
topReq.CompositeQuery.Queries = append(topReq.CompositeQuery.Queries, copied)
}
resp, err := m.querier.QueryRange(ctx, orgID, topReq)
if err != nil {
return nil, err
}
allMetricGroups := parseAndSortGroups(resp, rankingQueryName, req.GroupBy, req.OrderBy.Direction)
return paginateWithBackfill(allMetricGroups, metadataMap, req.GroupBy, req.Offset, req.Limit), nil
}
func (m *module) getClustersTableMetadata(ctx context.Context, req *inframonitoringtypes.PostableClusters) (map[string]map[string]string, error) {
var nonGroupByAttrs []string
for _, key := range clusterAttrKeysForMetadata {
if !isKeyInGroupByAttrs(req.GroupBy, key) {
nonGroupByAttrs = append(nonGroupByAttrs, key)
}
}
return m.getMetadata(ctx, clustersTableMetricNamesList, req.GroupBy, nonGroupByAttrs, req.Filter, req.Start, req.End)
}

View File

@@ -0,0 +1,140 @@
package implinframonitoring
import (
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
// TODO(nikhilmantri0902): change to k8s.cluster.uid after showing the missing
// data banner. Carried forward from v1 (see k8sClusterUIDAttrKey in
// pkg/query-service/app/inframetrics/clusters.go).
const clusterNameAttrKey = "k8s.cluster.name"
var clusterNameGroupByKey = qbtypes.GroupByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: clusterNameAttrKey,
FieldContext: telemetrytypes.FieldContextResource,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
}
// clustersTableMetricNamesList drives the existence/retention check.
// Includes k8s.node.condition_ready and k8s.pod.phase so the response
// short-circuits cleanly when a cluster doesn't ship those metrics — even
// though they aren't part of the QB composite query (they're queried separately
// via getPerGroupNodeConditionCounts and getPerGroupPodPhaseCounts).
var clustersTableMetricNamesList = []string{
"k8s.node.cpu.usage",
"k8s.node.allocatable_cpu",
"k8s.node.memory.working_set",
"k8s.node.allocatable_memory",
"k8s.node.condition_ready", //TODO(nikhilmantri0902): should these metrics be used to count groups k8s.node.condition_ready and k8s.pod.phase
"k8s.pod.phase",
}
var clusterAttrKeysForMetadata = []string{
"k8s.cluster.name",
}
var orderByToClustersQueryNames = map[string][]string{
inframonitoringtypes.ClustersOrderByCPU: {"A"},
inframonitoringtypes.ClustersOrderByCPUAllocatable: {"B"},
inframonitoringtypes.ClustersOrderByMemory: {"C"},
inframonitoringtypes.ClustersOrderByMemoryAllocatable: {"D"},
}
// newClustersTableListQuery builds the composite QB v5 request for the clusters list.
// Cluster-scope metrics are derived by summing per-node metrics within the
// group (default group: k8s.cluster.name). Node condition counts and pod phase
// counts are derived separately via getPerGroupNodeConditionCounts and
// getPerGroupPodPhaseCounts respectively (works for both list and grouped_list
// modes), so neither is included here. Query letters A/B/C/D mirror the v1
// implementation and the v2 nodes list.
func (m *module) newClustersTableListQuery() *qbtypes.QueryRangeRequest {
queries := []qbtypes.QueryEnvelope{
// Query A: CPU usage — sum of node CPU within the group.
{
Type: qbtypes.QueryTypeBuilder,
Spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Name: "A",
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "k8s.node.cpu.usage",
TimeAggregation: metrictypes.TimeAggregationAvg,
SpaceAggregation: metrictypes.SpaceAggregationSum,
ReduceTo: qbtypes.ReduceToAvg,
},
},
GroupBy: []qbtypes.GroupByKey{clusterNameGroupByKey},
Disabled: false,
},
},
// Query B: CPU allocatable — sum of node allocatable CPU within the group.
// TimeAggregationLatest is the closest v5 equivalent of v1's AnyLast;
// allocatable values change rarely so divergence in practice is negligible.
{
Type: qbtypes.QueryTypeBuilder,
Spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Name: "B",
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "k8s.node.allocatable_cpu",
TimeAggregation: metrictypes.TimeAggregationLatest,
SpaceAggregation: metrictypes.SpaceAggregationSum,
ReduceTo: qbtypes.ReduceToAvg,
},
},
GroupBy: []qbtypes.GroupByKey{clusterNameGroupByKey},
Disabled: false,
},
},
// Query C: Memory working set — sum of node memory within the group.
{
Type: qbtypes.QueryTypeBuilder,
Spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Name: "C",
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "k8s.node.memory.working_set",
TimeAggregation: metrictypes.TimeAggregationAvg,
SpaceAggregation: metrictypes.SpaceAggregationSum,
ReduceTo: qbtypes.ReduceToAvg,
},
},
GroupBy: []qbtypes.GroupByKey{clusterNameGroupByKey},
Disabled: false,
},
},
// Query D: Memory allocatable — sum of node allocatable memory within the group.
// Same Latest caveat as Query B.
{
Type: qbtypes.QueryTypeBuilder,
Spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Name: "D",
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "k8s.node.allocatable_memory",
TimeAggregation: metrictypes.TimeAggregationLatest,
SpaceAggregation: metrictypes.SpaceAggregationSum,
ReduceTo: qbtypes.ReduceToAvg,
},
},
GroupBy: []qbtypes.GroupByKey{clusterNameGroupByKey},
Disabled: false,
},
},
}
return &qbtypes.QueryRangeRequest{
RequestType: qbtypes.RequestTypeScalar,
CompositeQuery: qbtypes.CompositeQuery{
Queries: queries,
},
}
}

View File

@@ -93,3 +93,51 @@ func (h *handler) ListNodes(rw http.ResponseWriter, req *http.Request) {
render.Success(rw, http.StatusOK, result)
}
func (h *handler) ListNamespaces(rw http.ResponseWriter, req *http.Request) {
claims, err := authtypes.ClaimsFromContext(req.Context())
if err != nil {
render.Error(rw, err)
return
}
orgID := valuer.MustNewUUID(claims.OrgID)
var parsedReq inframonitoringtypes.PostableNamespaces
if err := binding.JSON.BindBody(req.Body, &parsedReq); err != nil {
render.Error(rw, err)
return
}
result, err := h.module.ListNamespaces(req.Context(), orgID, &parsedReq)
if err != nil {
render.Error(rw, err)
return
}
render.Success(rw, http.StatusOK, result)
}
func (h *handler) ListClusters(rw http.ResponseWriter, req *http.Request) {
claims, err := authtypes.ClaimsFromContext(req.Context())
if err != nil {
render.Error(rw, err)
return
}
orgID := valuer.MustNewUUID(claims.OrgID)
var parsedReq inframonitoringtypes.PostableClusters
if err := binding.JSON.BindBody(req.Body, &parsedReq); err != nil {
render.Error(rw, err)
return
}
result, err := h.module.ListClusters(req.Context(), orgID, &parsedReq)
if err != nil {
render.Error(rw, err)
return
}
render.Success(rw, http.StatusOK, result)
}

View File

@@ -337,3 +337,194 @@ func (m *module) ListNodes(ctx context.Context, orgID valuer.UUID, req *inframon
return resp, nil
}
func (m *module) ListNamespaces(ctx context.Context, orgID valuer.UUID, req *inframonitoringtypes.PostableNamespaces) (*inframonitoringtypes.Namespaces, error) {
if err := req.Validate(); err != nil {
return nil, err
}
resp := &inframonitoringtypes.Namespaces{}
if req.OrderBy == nil {
req.OrderBy = &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: inframonitoringtypes.NamespacesOrderByCPU,
},
},
Direction: qbtypes.OrderDirectionDesc,
}
}
if len(req.GroupBy) == 0 {
req.GroupBy = []qbtypes.GroupByKey{namespaceNameGroupByKey}
resp.Type = inframonitoringtypes.ResponseTypeList
} else {
resp.Type = inframonitoringtypes.ResponseTypeGroupedList
}
missingMetrics, minFirstReportedUnixMilli, err := m.getMetricsExistenceAndEarliestTime(ctx, namespacesTableMetricNamesList)
if err != nil {
return nil, err
}
if len(missingMetrics) > 0 {
resp.RequiredMetricsCheck = inframonitoringtypes.RequiredMetricsCheck{MissingMetrics: missingMetrics}
resp.Records = []inframonitoringtypes.NamespaceRecord{}
resp.Total = 0
return resp, nil
}
if req.End < int64(minFirstReportedUnixMilli) {
resp.EndTimeBeforeRetention = true
resp.Records = []inframonitoringtypes.NamespaceRecord{}
resp.Total = 0
return resp, nil
}
resp.RequiredMetricsCheck = inframonitoringtypes.RequiredMetricsCheck{MissingMetrics: []string{}}
metadataMap, err := m.getNamespacesTableMetadata(ctx, req)
if err != nil {
return nil, err
}
resp.Total = len(metadataMap)
pageGroups, err := m.getTopNamespaceGroups(ctx, orgID, req, metadataMap)
if err != nil {
return nil, err
}
if len(pageGroups) == 0 {
resp.Records = []inframonitoringtypes.NamespaceRecord{}
return resp, nil
}
filterExpr := ""
if req.Filter != nil {
filterExpr = req.Filter.Expression
}
fullQueryReq := buildFullQueryRequest(req.Start, req.End, filterExpr, req.GroupBy, pageGroups, m.newNamespacesTableListQuery())
queryResp, err := m.querier.QueryRange(ctx, orgID, fullQueryReq)
if err != nil {
return nil, err
}
// Reuse the pods phase-counts CTE function via a temp struct — it reads only
// Start/End/Filter/GroupBy from PostablePods.
phaseCounts, err := m.getPerGroupPodPhaseCounts(ctx, &inframonitoringtypes.PostablePods{
Start: req.Start,
End: req.End,
Filter: req.Filter,
GroupBy: req.GroupBy,
}, pageGroups)
if err != nil {
return nil, err
}
resp.Records = buildNamespaceRecords(queryResp, pageGroups, req.GroupBy, metadataMap, phaseCounts)
resp.Warning = queryResp.Warning
return resp, nil
}
func (m *module) ListClusters(ctx context.Context, orgID valuer.UUID, req *inframonitoringtypes.PostableClusters) (*inframonitoringtypes.Clusters, error) {
if err := req.Validate(); err != nil {
return nil, err
}
resp := &inframonitoringtypes.Clusters{}
if req.OrderBy == nil {
req.OrderBy = &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: inframonitoringtypes.ClustersOrderByCPU,
},
},
Direction: qbtypes.OrderDirectionDesc,
}
}
if len(req.GroupBy) == 0 {
req.GroupBy = []qbtypes.GroupByKey{clusterNameGroupByKey}
resp.Type = inframonitoringtypes.ResponseTypeList
} else {
resp.Type = inframonitoringtypes.ResponseTypeGroupedList
}
missingMetrics, minFirstReportedUnixMilli, err := m.getMetricsExistenceAndEarliestTime(ctx, clustersTableMetricNamesList)
if err != nil {
return nil, err
}
if len(missingMetrics) > 0 {
resp.RequiredMetricsCheck = inframonitoringtypes.RequiredMetricsCheck{MissingMetrics: missingMetrics}
resp.Records = []inframonitoringtypes.ClusterRecord{}
resp.Total = 0
return resp, nil
}
if req.End < int64(minFirstReportedUnixMilli) {
resp.EndTimeBeforeRetention = true
resp.Records = []inframonitoringtypes.ClusterRecord{}
resp.Total = 0
return resp, nil
}
resp.RequiredMetricsCheck = inframonitoringtypes.RequiredMetricsCheck{MissingMetrics: []string{}}
metadataMap, err := m.getClustersTableMetadata(ctx, req)
if err != nil {
return nil, err
}
resp.Total = len(metadataMap)
pageGroups, err := m.getTopClusterGroups(ctx, orgID, req, metadataMap)
if err != nil {
return nil, err
}
if len(pageGroups) == 0 {
resp.Records = []inframonitoringtypes.ClusterRecord{}
return resp, nil
}
filterExpr := ""
if req.Filter != nil {
filterExpr = req.Filter.Expression
}
fullQueryReq := buildFullQueryRequest(req.Start, req.End, filterExpr, req.GroupBy, pageGroups, m.newClustersTableListQuery())
queryResp, err := m.querier.QueryRange(ctx, orgID, fullQueryReq)
if err != nil {
return nil, err
}
// Reuse the nodes condition-counts CTE function via a temp struct — it reads only
// Start/End/Filter/GroupBy from PostableNodes. With default groupBy
// [k8s.cluster.name], counts are bucketed per cluster; with a custom groupBy,
// they aggregate across clusters in that group.
nodeConditionCountsMap, err := m.getPerGroupNodeConditionCounts(ctx, &inframonitoringtypes.PostableNodes{
Start: req.Start,
End: req.End,
Filter: req.Filter,
GroupBy: req.GroupBy,
}, pageGroups)
if err != nil {
return nil, err
}
// Same pattern for pod phase counts via PostablePods shim.
podPhaseCountsMap, err := m.getPerGroupPodPhaseCounts(ctx, &inframonitoringtypes.PostablePods{
Start: req.Start,
End: req.End,
Filter: req.Filter,
GroupBy: req.GroupBy,
}, pageGroups)
if err != nil {
return nil, err
}
resp.Records = buildClusterRecords(queryResp, pageGroups, req.GroupBy, metadataMap, nodeConditionCountsMap, podPhaseCountsMap)
resp.Warning = queryResp.Warning
return resp, nil
}

View File

@@ -0,0 +1,123 @@
package implinframonitoring
import (
"context"
"slices"
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/valuer"
)
// buildNamespaceRecords assembles the page records. Pod phase counts come from
// phaseCounts in both modes; every row is a group of pods, so there's no
// per-row "current phase" concept (unlike pods/nodes list mode).
func buildNamespaceRecords(
resp *qbtypes.QueryRangeResponse,
pageGroups []map[string]string,
groupBy []qbtypes.GroupByKey,
metadataMap map[string]map[string]string,
phaseCounts map[string]podPhaseCounts,
) []inframonitoringtypes.NamespaceRecord {
metricsMap := parseFullQueryResponse(resp, groupBy)
records := make([]inframonitoringtypes.NamespaceRecord, 0, len(pageGroups))
for _, labels := range pageGroups {
compositeKey := compositeKeyFromLabels(labels, groupBy)
namespaceName := labels[namespaceNameAttrKey]
record := inframonitoringtypes.NamespaceRecord{ // initialize with default values
NamespaceName: namespaceName,
NamespaceCPU: -1,
NamespaceMemory: -1,
Meta: map[string]string{},
}
if metrics, ok := metricsMap[compositeKey]; ok {
if v, exists := metrics["A"]; exists {
record.NamespaceCPU = v
}
if v, exists := metrics["D"]; exists {
record.NamespaceMemory = v
}
}
if phaseCountsForGroup, ok := phaseCounts[compositeKey]; ok {
record.PodCountsByPhase = inframonitoringtypes.PodCountsByPhase{
Pending: phaseCountsForGroup.Pending,
Running: phaseCountsForGroup.Running,
Succeeded: phaseCountsForGroup.Succeeded,
Failed: phaseCountsForGroup.Failed,
Unknown: phaseCountsForGroup.Unknown,
}
}
if attrs, ok := metadataMap[compositeKey]; ok {
for k, v := range attrs {
record.Meta[k] = v
}
}
records = append(records, record)
}
return records
}
func (m *module) getTopNamespaceGroups(
ctx context.Context,
orgID valuer.UUID,
req *inframonitoringtypes.PostableNamespaces,
metadataMap map[string]map[string]string,
) ([]map[string]string, error) {
orderByKey := req.OrderBy.Key.Name
queryNamesForOrderBy := orderByToNamespacesQueryNames[orderByKey]
rankingQueryName := queryNamesForOrderBy[len(queryNamesForOrderBy)-1]
topReq := &qbtypes.QueryRangeRequest{
Start: uint64(req.Start),
End: uint64(req.End),
RequestType: qbtypes.RequestTypeScalar,
CompositeQuery: qbtypes.CompositeQuery{
Queries: make([]qbtypes.QueryEnvelope, 0, len(queryNamesForOrderBy)),
},
}
for _, envelope := range m.newNamespacesTableListQuery().CompositeQuery.Queries {
if !slices.Contains(queryNamesForOrderBy, envelope.GetQueryName()) {
continue
}
copied := envelope
if copied.Type == qbtypes.QueryTypeBuilder {
existingExpr := ""
if f := copied.GetFilter(); f != nil {
existingExpr = f.Expression
}
reqFilterExpr := ""
if req.Filter != nil {
reqFilterExpr = req.Filter.Expression
}
merged := mergeFilterExpressions(existingExpr, reqFilterExpr)
copied.SetFilter(&qbtypes.Filter{Expression: merged})
copied.SetGroupBy(req.GroupBy)
}
topReq.CompositeQuery.Queries = append(topReq.CompositeQuery.Queries, copied)
}
resp, err := m.querier.QueryRange(ctx, orgID, topReq)
if err != nil {
return nil, err
}
allMetricGroups := parseAndSortGroups(resp, rankingQueryName, req.GroupBy, req.OrderBy.Direction)
return paginateWithBackfill(allMetricGroups, metadataMap, req.GroupBy, req.Offset, req.Limit), nil
}
func (m *module) getNamespacesTableMetadata(ctx context.Context, req *inframonitoringtypes.PostableNamespaces) (map[string]map[string]string, error) {
var nonGroupByAttrs []string
for _, key := range namespaceAttrKeysForMetadata {
if !isKeyInGroupByAttrs(req.GroupBy, key) {
nonGroupByAttrs = append(nonGroupByAttrs, key)
}
}
return m.getMetadata(ctx, namespacesTableMetricNamesList, req.GroupBy, nonGroupByAttrs, req.Filter, req.Start, req.End)
}

View File

@@ -0,0 +1,92 @@
package implinframonitoring
import (
"github.com/SigNoz/signoz/pkg/types/inframonitoringtypes"
"github.com/SigNoz/signoz/pkg/types/metrictypes"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
)
const (
namespaceNameAttrKey = "k8s.namespace.name"
)
var namespaceNameGroupByKey = qbtypes.GroupByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: namespaceNameAttrKey,
FieldContext: telemetrytypes.FieldContextResource,
FieldDataType: telemetrytypes.FieldDataTypeString,
},
}
// namespacesTableMetricNamesList drives the existence/retention check.
// Includes k8s.pod.phase so the response short-circuits cleanly when a
// cluster doesn't ship the metric — even though phase isn't part of the
// QB composite query (it's queried separately via getPerGroupPodPhaseCounts).
var namespacesTableMetricNamesList = []string{
"k8s.pod.cpu.usage",
"k8s.pod.memory.working_set",
"k8s.pod.phase",
}
var namespaceAttrKeysForMetadata = []string{
"k8s.namespace.name",
"k8s.cluster.name",
}
var orderByToNamespacesQueryNames = map[string][]string{
inframonitoringtypes.NamespacesOrderByCPU: {"A"},
inframonitoringtypes.NamespacesOrderByMemory: {"D"},
}
// newNamespacesTableListQuery builds the composite QB v5 request for the namespaces list.
// Pod phase counts are derived separately via getPerGroupPodPhaseCounts (works for both
// list and grouped_list modes), so no phase query is included here.
// Query letters A and D are kept aligned with the v1 implementation.
func (m *module) newNamespacesTableListQuery() *qbtypes.QueryRangeRequest {
queries := []qbtypes.QueryEnvelope{
// Query A: CPU usage — sum of pod CPU within the group.
{
Type: qbtypes.QueryTypeBuilder,
Spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Name: "A",
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "k8s.pod.cpu.usage",
TimeAggregation: metrictypes.TimeAggregationAvg,
SpaceAggregation: metrictypes.SpaceAggregationSum,
ReduceTo: qbtypes.ReduceToAvg,
},
},
GroupBy: []qbtypes.GroupByKey{namespaceNameGroupByKey},
Disabled: false,
},
},
// Query D: Memory working set — sum of pod memory within the group.
{
Type: qbtypes.QueryTypeBuilder,
Spec: qbtypes.QueryBuilderQuery[qbtypes.MetricAggregation]{
Name: "D",
Signal: telemetrytypes.SignalMetrics,
Aggregations: []qbtypes.MetricAggregation{
{
MetricName: "k8s.pod.memory.working_set",
TimeAggregation: metrictypes.TimeAggregationAvg,
SpaceAggregation: metrictypes.SpaceAggregationSum,
ReduceTo: qbtypes.ReduceToAvg,
},
},
GroupBy: []qbtypes.GroupByKey{namespaceNameGroupByKey},
Disabled: false,
},
},
}
return &qbtypes.QueryRangeRequest{
RequestType: qbtypes.RequestTypeScalar,
CompositeQuery: qbtypes.CompositeQuery{
Queries: queries,
},
}
}

View File

@@ -12,10 +12,14 @@ type Handler interface {
ListHosts(http.ResponseWriter, *http.Request)
ListPods(http.ResponseWriter, *http.Request)
ListNodes(http.ResponseWriter, *http.Request)
ListNamespaces(http.ResponseWriter, *http.Request)
ListClusters(http.ResponseWriter, *http.Request)
}
type Module interface {
ListHosts(ctx context.Context, orgID valuer.UUID, req *inframonitoringtypes.PostableHosts) (*inframonitoringtypes.Hosts, error)
ListPods(ctx context.Context, orgID valuer.UUID, req *inframonitoringtypes.PostablePods) (*inframonitoringtypes.Pods, error)
ListNodes(ctx context.Context, orgID valuer.UUID, req *inframonitoringtypes.PostableNodes) (*inframonitoringtypes.Nodes, error)
ListNamespaces(ctx context.Context, orgID valuer.UUID, req *inframonitoringtypes.PostableNamespaces) (*inframonitoringtypes.Namespaces, error)
ListClusters(ctx context.Context, orgID valuer.UUID, req *inframonitoringtypes.PostableClusters) (*inframonitoringtypes.Clusters, error)
}

View File

@@ -0,0 +1,52 @@
package implretention
import (
"context"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/modules/retention"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
type getter struct {
store retentiontypes.Store
}
// NewGetter creates a retention getter backed by the retention store.
func NewGetter(store retentiontypes.Store) retention.Getter {
return &getter{
store: store,
}
}
// GetRetentionPolicySegments loads successful TTL changes and converts them into retention policy segments.
func (getter *getter) GetRetentionPolicySegments(
ctx context.Context,
orgID valuer.UUID,
dbName string,
tableName string,
fallbackDefaultDays int,
startMs int64,
endMs int64,
) ([]*retentiontypes.RetentionPolicySegment, error) {
if startMs >= endMs {
return nil, nil
}
if dbName == "" {
return nil, errors.New(errors.TypeInvalidInput, errors.CodeInvalidInput, "dbName is empty")
}
if tableName == "" {
return nil, errors.New(errors.TypeInvalidInput, errors.CodeInvalidInput, "tableName is empty")
}
if fallbackDefaultDays <= 0 {
return nil, errors.Newf(errors.TypeInvalidInput, errors.CodeInvalidInput, "non-positive fallbackDefaultDays %d", fallbackDefaultDays)
}
rows, err := getter.store.ListTTLSettingsByTableNameAndBeforeCreatedAt(ctx, orgID, dbName+"."+tableName, endMs)
if err != nil {
return nil, err
}
return retentiontypes.BuildRetentionPolicySegmentsFromRows(rows, fallbackDefaultDays, startMs, endMs)
}

View File

@@ -0,0 +1,41 @@
package implretention
import (
"context"
"time"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
type store struct {
sqlstore sqlstore.SQLStore
}
// NewStore creates a SQL-backed retention store.
func NewStore(sqlstore sqlstore.SQLStore) retentiontypes.Store {
return &store{sqlstore: sqlstore}
}
// ListTTLSettingsByTableNameAndBeforeCreatedAt returns successful TTL settings before the given timestamp.
func (store *store) ListTTLSettingsByTableNameAndBeforeCreatedAt(ctx context.Context, orgID valuer.UUID, tableName string, beforeMs int64) ([]*retentiontypes.TTLSetting, error) {
rows := []*retentiontypes.TTLSetting{}
err := store.
sqlstore.
BunDB().
NewSelect().
Model(&rows).
Where("table_name = ?", tableName).
Where("org_id = ?", orgID.StringValue()).
Where("status = ?", retentiontypes.TTLSettingStatusSuccess).
Where("created_at < ?", time.UnixMilli(beforeMs).UTC()).
OrderExpr("created_at ASC").
Scan(ctx)
if err != nil {
return nil, errors.Wrapf(err, errors.TypeInternal, errors.CodeInternal, "load ttl_setting rows for org %q table %q", orgID.StringValue(), tableName)
}
return rows, nil
}

View File

@@ -0,0 +1,14 @@
package retention
import (
"context"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
"github.com/SigNoz/signoz/pkg/valuer"
)
// Getter resolves retention data and expressions for read paths.
type Getter interface {
// GetRetentionPolicySegments returns retention policy segments active over a half-open meter window.
GetRetentionPolicySegments(ctx context.Context, orgID valuer.UUID, dbName string, tableName string, fallbackDefaultDays int, startMs int64, endMs int64) ([]*retentiontypes.RetentionPolicySegment, error)
}

View File

@@ -377,7 +377,7 @@ func (module *module) getOrGetSetIdentity(ctx context.Context, serviceAccountID
}
func (module *module) setRole(ctx context.Context, orgID valuer.UUID, id valuer.UUID, role *authtypes.Role) error {
serviceAccount, err := module.GetWithRoles(ctx, orgID, id)
serviceAccount, err := module.Get(ctx, orgID, id)
if err != nil {
return err
}
@@ -387,24 +387,12 @@ func (module *module) setRole(ctx context.Context, orgID valuer.UUID, id valuer.
return err
}
err = module.authz.ModifyGrant(ctx, orgID, serviceAccount.RoleNames(), []string{role.Name}, authtypes.MustNewSubject(coretypes.NewResourceServiceAccount(), id.String(), orgID, nil))
err = module.authz.Grant(ctx, orgID, []string{role.Name}, authtypes.MustNewSubject(coretypes.NewResourceServiceAccount(), id.String(), orgID, nil))
if err != nil {
return err
}
err = module.store.RunInTx(ctx, func(ctx context.Context) error {
err = module.store.DeleteServiceAccountRoles(ctx, serviceAccount.ID)
if err != nil {
return err
}
err = module.store.CreateServiceAccountRole(ctx, serviceAccountRole)
if err != nil {
return err
}
return nil
})
err = module.store.CreateServiceAccountRole(ctx, serviceAccountRole)
if err != nil {
return err
}

View File

@@ -207,21 +207,6 @@ func (store *store) CreateServiceAccountRole(ctx context.Context, serviceAccount
return nil
}
func (store *store) DeleteServiceAccountRoles(ctx context.Context, serviceAccountID valuer.UUID) error {
_, err := store.
sqlstore.
BunDBCtx(ctx).
NewDelete().
Model(new(serviceaccounttypes.ServiceAccountRole)).
Where("service_account_id = ?", serviceAccountID).
Exec(ctx)
if err != nil {
return err
}
return nil
}
func (store *store) DeleteServiceAccountRole(ctx context.Context, serviceAccountID valuer.UUID, roleID valuer.UUID) error {
_, err := store.
sqlstore.

View File

@@ -1343,7 +1343,7 @@ func getLocalTableName(tableName string) string {
}
func (r *ClickHouseReader) setTTLLogs(ctx context.Context, orgID string, createdByEmail string, params *model.TTLParams) (*model.SetTTLResponseItem, *model.ApiError) {
func (r *ClickHouseReader) setTTLLogs(ctx context.Context, orgID string, params *retentiontypes.TTLParams) (*retentiontypes.SetTTLResponseItem, *model.ApiError) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.TelemetrySignal: telemetrytypes.SignalLogs.StringValue(),
instrumentationtypes.CodeNamespace: "clickhouse-reader",
@@ -1378,7 +1378,7 @@ func (r *ClickHouseReader) setTTLLogs(ctx context.Context, orgID string, created
if apiErr != nil {
return nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf("error in processing ttl_status check sql query")}
}
if statusItem.Status == constants.StatusPending {
if statusItem.Status == retentiontypes.TTLSettingStatusPending {
return nil, &model.ApiError{Typ: model.ErrorConflict, Err: fmt.Errorf("TTL is already running")}
}
}
@@ -1434,14 +1434,10 @@ func (r *ClickHouseReader) setTTLLogs(ctx context.Context, orgID string, created
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
},
UserAuditable: types.UserAuditable{
CreatedBy: createdByEmail,
UpdatedBy: createdByEmail,
},
TransactionID: uuid,
TableName: tableName,
TTL: int(params.DelDuration),
Status: constants.StatusPending,
Status: retentiontypes.TTLSettingStatusPending,
ColdStorageTTL: coldStorageDuration,
OrgID: orgID,
}
@@ -1467,7 +1463,7 @@ func (r *ClickHouseReader) setTTLLogs(ctx context.Context, orgID string, created
NewUpdate().
Model(new(retentiontypes.TTLSetting)).
Set("updated_at = ?", time.Now()).
Set("status = ?", constants.StatusFailed).
Set("status = ?", retentiontypes.TTLSettingStatusFailed).
Where("id = ?", statusItem.ID.StringValue()).
Exec(ctx)
if dbErr != nil {
@@ -1487,7 +1483,7 @@ func (r *ClickHouseReader) setTTLLogs(ctx context.Context, orgID string, created
NewUpdate().
Model(new(retentiontypes.TTLSetting)).
Set("updated_at = ?", time.Now()).
Set("status = ?", constants.StatusFailed).
Set("status = ?", retentiontypes.TTLSettingStatusFailed).
Where("id = ?", statusItem.ID.StringValue()).
Exec(ctx)
if dbErr != nil {
@@ -1502,7 +1498,7 @@ func (r *ClickHouseReader) setTTLLogs(ctx context.Context, orgID string, created
NewUpdate().
Model(new(retentiontypes.TTLSetting)).
Set("updated_at = ?", time.Now()).
Set("status = ?", constants.StatusSuccess).
Set("status = ?", retentiontypes.TTLSettingStatusSuccess).
Where("id = ?", statusItem.ID.StringValue()).
Exec(ctx)
if dbErr != nil {
@@ -1512,10 +1508,10 @@ func (r *ClickHouseReader) setTTLLogs(ctx context.Context, orgID string, created
}
}(ttlPayload)
return &model.SetTTLResponseItem{Message: "move ttl has been successfully set up"}, nil
return &retentiontypes.SetTTLResponseItem{Message: "move ttl has been successfully set up"}, nil
}
func (r *ClickHouseReader) setTTLTraces(ctx context.Context, orgID string, createdByEmail string, params *model.TTLParams) (*model.SetTTLResponseItem, *model.ApiError) {
func (r *ClickHouseReader) setTTLTraces(ctx context.Context, orgID string, params *retentiontypes.TTLParams) (*retentiontypes.SetTTLResponseItem, *model.ApiError) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.TelemetrySignal: telemetrytypes.SignalTraces.StringValue(),
instrumentationtypes.CodeNamespace: "clickhouse-reader",
@@ -1545,7 +1541,7 @@ func (r *ClickHouseReader) setTTLTraces(ctx context.Context, orgID string, creat
if apiErr != nil {
return nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf("error in processing ttl_status check sql query")}
}
if statusItem.Status == constants.StatusPending {
if statusItem.Status == retentiontypes.TTLSettingStatusPending {
return nil, &model.ApiError{Typ: model.ErrorConflict, Err: fmt.Errorf("TTL is already running")}
}
}
@@ -1576,14 +1572,10 @@ func (r *ClickHouseReader) setTTLTraces(ctx context.Context, orgID string, creat
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
},
UserAuditable: types.UserAuditable{
CreatedBy: createdByEmail,
UpdatedBy: createdByEmail,
},
TransactionID: uuid,
TableName: tableName,
TTL: int(params.DelDuration),
Status: constants.StatusPending,
Status: retentiontypes.TTLSettingStatusPending,
ColdStorageTTL: coldStorageDuration,
OrgID: orgID,
}
@@ -1621,7 +1613,7 @@ func (r *ClickHouseReader) setTTLTraces(ctx context.Context, orgID string, creat
NewUpdate().
Model(new(retentiontypes.TTLSetting)).
Set("updated_at = ?", time.Now()).
Set("status = ?", constants.StatusFailed).
Set("status = ?", retentiontypes.TTLSettingStatusFailed).
Where("id = ?", statusItem.ID.StringValue()).
Exec(ctx)
if dbErr != nil {
@@ -1642,7 +1634,7 @@ func (r *ClickHouseReader) setTTLTraces(ctx context.Context, orgID string, creat
NewUpdate().
Model(new(retentiontypes.TTLSetting)).
Set("updated_at = ?", time.Now()).
Set("status = ?", constants.StatusFailed).
Set("status = ?", retentiontypes.TTLSettingStatusFailed).
Where("id = ?", statusItem.ID.StringValue()).
Exec(ctx)
if dbErr != nil {
@@ -1657,7 +1649,7 @@ func (r *ClickHouseReader) setTTLTraces(ctx context.Context, orgID string, creat
NewUpdate().
Model(new(retentiontypes.TTLSetting)).
Set("updated_at = ?", time.Now()).
Set("status = ?", constants.StatusSuccess).
Set("status = ?", retentiontypes.TTLSettingStatusSuccess).
Where("id = ?", statusItem.ID.StringValue()).
Exec(ctx)
if dbErr != nil {
@@ -1666,7 +1658,7 @@ func (r *ClickHouseReader) setTTLTraces(ctx context.Context, orgID string, creat
}
}(distributedTableName)
}
return &model.SetTTLResponseItem{Message: "move ttl has been successfully set up"}, nil
return &retentiontypes.SetTTLResponseItem{Message: "move ttl has been successfully set up"}, nil
}
func (r *ClickHouseReader) hasCustomRetentionColumn(ctx context.Context) (bool, error) {
@@ -1695,7 +1687,7 @@ func (r *ClickHouseReader) hasCustomRetentionColumn(ctx context.Context) (bool,
return true, nil
}
func (r *ClickHouseReader) SetTTLV2(ctx context.Context, orgID string, createdByEmail string, params *model.CustomRetentionTTLParams) (*model.CustomRetentionTTLResponse, error) {
func (r *ClickHouseReader) SetTTLV2(ctx context.Context, orgID string, params *retentiontypes.CustomRetentionTTLParams) (*retentiontypes.CustomRetentionTTLResponse, error) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.TelemetrySignal: telemetrytypes.SignalLogs.StringValue(),
@@ -1710,7 +1702,7 @@ func (r *ClickHouseReader) SetTTLV2(ctx context.Context, orgID string, createdBy
if !hasCustomRetention {
r.logger.Info("Custom retention not supported, falling back to standard TTL method", "orgID", orgID)
ttlParams := &model.TTLParams{
ttlParams := &retentiontypes.TTLParams{
Type: params.Type,
DelDuration: int64(params.DefaultTTLDays * 24 * 3600),
}
@@ -1726,12 +1718,12 @@ func (r *ClickHouseReader) SetTTLV2(ctx context.Context, orgID string, createdBy
ttlParams.ToColdStorageDuration = 0
}
ttlResult, apiErr := r.SetTTL(ctx, orgID, createdByEmail, ttlParams)
ttlResult, apiErr := r.SetTTL(ctx, orgID, ttlParams)
if apiErr != nil {
return nil, errorsV2.Wrapf(apiErr.Err, errorsV2.TypeInternal, errorsV2.CodeInternal, "failed to set standard TTL")
}
return &model.CustomRetentionTTLResponse{
return &retentiontypes.CustomRetentionTTLResponse{
Message: fmt.Sprintf("Custom retention not supported, applied standard TTL of %d days. %s", params.DefaultTTLDays, ttlResult.Message),
}, nil
}
@@ -1742,7 +1734,7 @@ func (r *ClickHouseReader) SetTTLV2(ctx context.Context, orgID string, createdBy
uuidWithHyphen := valuer.GenerateUUID()
uuid := strings.Replace(uuidWithHyphen.String(), "-", "", -1)
if params.Type != constants.LogsTTL {
if params.Type != retentiontypes.LogsTTL {
return nil, errorsV2.Newf(errorsV2.TypeInternal, errorsV2.CodeInternal, "custom retention TTL only supported for logs")
}
@@ -1773,7 +1765,7 @@ func (r *ClickHouseReader) SetTTLV2(ctx context.Context, orgID string, createdBy
if apiErr != nil {
return nil, errorsV2.Newf(errorsV2.TypeInternal, errorsV2.CodeInternal, "error in processing custom_retention_ttl_status check sql query")
}
if statusItem.Status == constants.StatusPending {
if statusItem.Status == retentiontypes.TTLSettingStatusPending {
return nil, errorsV2.Newf(errorsV2.TypeInternal, errorsV2.CodeInternal, "custom retention TTL is already running")
}
}
@@ -1855,15 +1847,11 @@ func (r *ClickHouseReader) SetTTLV2(ctx context.Context, orgID string, createdBy
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
},
UserAuditable: types.UserAuditable{
CreatedBy: createdByEmail,
UpdatedBy: createdByEmail,
},
TransactionID: uuid,
TableName: tableName,
TTL: params.DefaultTTLDays,
Condition: string(ttlConditionsJSON),
Status: constants.StatusPending,
Status: retentiontypes.TTLSettingStatusPending,
ColdStorageTTL: coldStorageDuration,
OrgID: orgID,
}
@@ -1879,7 +1867,7 @@ func (r *ClickHouseReader) SetTTLV2(ctx context.Context, orgID string, createdBy
err := r.setColdStorage(ctx, tableName, params.ColdStorageVolume)
if err != nil {
r.logger.Error("error in setting cold storage", errorsV2.Attr(err))
r.updateCustomRetentionTTLStatus(ctx, orgID, tableName, constants.StatusFailed)
r.updateCustomRetentionTTLStatus(ctx, orgID, tableName, retentiontypes.TTLSettingStatusFailed)
return nil, errorsV2.Wrapf(err.Err, errorsV2.TypeInternal, errorsV2.CodeInternal, "error setting cold storage for table %s", tableName)
}
}
@@ -1888,21 +1876,21 @@ func (r *ClickHouseReader) SetTTLV2(ctx context.Context, orgID string, createdBy
r.logger.Debug("Executing custom retention TTL request: ", "request", query, "step", i+1)
if err := r.db.Exec(ctx, query); err != nil {
r.logger.Error("error while setting custom retention ttl", errorsV2.Attr(err))
r.updateCustomRetentionTTLStatus(ctx, orgID, tableName, constants.StatusFailed)
r.updateCustomRetentionTTLStatus(ctx, orgID, tableName, retentiontypes.TTLSettingStatusFailed)
return nil, errorsV2.Wrapf(err, errorsV2.TypeInternal, errorsV2.CodeInternal, "error setting custom retention TTL for table %s, query: %s", tableName, query)
}
}
r.updateCustomRetentionTTLStatus(ctx, orgID, tableName, constants.StatusSuccess)
r.updateCustomRetentionTTLStatus(ctx, orgID, tableName, retentiontypes.TTLSettingStatusSuccess)
}
return &model.CustomRetentionTTLResponse{
return &retentiontypes.CustomRetentionTTLResponse{
Message: "custom retention TTL has been successfully set up",
}, nil
}
// New method to build multiIf expressions with support for multiple AND conditions
func (r *ClickHouseReader) buildMultiIfExpression(ttlConditions []model.CustomRetentionRule, defaultTTLDays int, isResourceTable bool) string {
func (r *ClickHouseReader) buildMultiIfExpression(ttlConditions []retentiontypes.CustomRetentionRule, defaultTTLDays int, isResourceTable bool) string {
var conditions []string
for i, rule := range ttlConditions {
@@ -1974,7 +1962,7 @@ func (r *ClickHouseReader) buildMultiIfExpression(ttlConditions []model.CustomRe
return result
}
func (r *ClickHouseReader) GetCustomRetentionTTL(ctx context.Context, orgID string) (*model.GetCustomRetentionTTLResponse, error) {
func (r *ClickHouseReader) GetCustomRetentionTTL(ctx context.Context, orgID string) (*retentiontypes.GetCustomRetentionTTLResponse, error) {
// Check if V2 (custom retention) is supported
hasCustomRetention, err := r.hasCustomRetentionColumn(ctx)
if err != nil {
@@ -1983,7 +1971,7 @@ func (r *ClickHouseReader) GetCustomRetentionTTL(ctx context.Context, orgID stri
hasCustomRetention = false
}
response := &model.GetCustomRetentionTTLResponse{}
response := &retentiontypes.GetCustomRetentionTTLResponse{}
if hasCustomRetention {
// V2 - Custom retention is supported
@@ -2006,19 +1994,19 @@ func (r *ClickHouseReader) GetCustomRetentionTTL(ctx context.Context, orgID stri
if err == sql.ErrNoRows {
// No V2 configuration found, return defaults
response.DefaultTTLDays = 15
response.TTLConditions = []model.CustomRetentionRule{}
response.Status = constants.StatusSuccess
response.DefaultTTLDays = retentiontypes.DefaultLogsRetentionDays
response.TTLConditions = []retentiontypes.CustomRetentionRule{}
response.Status = retentiontypes.TTLSettingStatusSuccess
response.ColdStorageTTLDays = -1
return response, nil
}
// Parse TTL conditions from Condition
var ttlConditions []model.CustomRetentionRule
var ttlConditions []retentiontypes.CustomRetentionRule
if customTTL.Condition != "" {
if err := json.Unmarshal([]byte(customTTL.Condition), &ttlConditions); err != nil {
r.logger.Error("Error parsing TTL conditions", errorsV2.Attr(err))
ttlConditions = []model.CustomRetentionRule{}
ttlConditions = []retentiontypes.CustomRetentionRule{}
}
}
@@ -2032,8 +2020,8 @@ func (r *ClickHouseReader) GetCustomRetentionTTL(ctx context.Context, orgID stri
response.Version = "v1"
// Get V1 TTL configuration
ttlParams := &model.GetTTLParams{
Type: constants.LogsTTL,
ttlParams := &retentiontypes.GetTTLParams{
Type: retentiontypes.LogsTTL,
}
ttlResult, apiErr := r.GetTTL(ctx, orgID, ttlParams)
@@ -2053,7 +2041,7 @@ func (r *ClickHouseReader) GetCustomRetentionTTL(ctx context.Context, orgID stri
}
// For V1, we don't have TTL conditions
response.TTLConditions = []model.CustomRetentionRule{}
response.TTLConditions = []retentiontypes.CustomRetentionRule{}
}
return response, nil
@@ -2093,7 +2081,7 @@ func (r *ClickHouseReader) updateCustomRetentionTTLStatus(ctx context.Context, o
}
// Enhanced validation function with duplicate detection and efficient key validation
func (r *ClickHouseReader) validateTTLConditions(ctx context.Context, ttlConditions []model.CustomRetentionRule) error {
func (r *ClickHouseReader) validateTTLConditions(ctx context.Context, ttlConditions []retentiontypes.CustomRetentionRule) error {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.CodeNamespace: "clickhouse-reader",
instrumentationtypes.CodeFunctionName: "validateTTLConditions",
@@ -2197,24 +2185,24 @@ func (r *ClickHouseReader) validateTTLConditions(ctx context.Context, ttlConditi
// SetTTL sets the TTL for traces or metrics or logs tables.
// This is an async API which creates goroutines to set TTL.
// Status of TTL update is tracked with ttl_status table in sqlite db.
func (r *ClickHouseReader) SetTTL(ctx context.Context, orgID string, userID string, params *model.TTLParams) (*model.SetTTLResponseItem, *model.ApiError) {
func (r *ClickHouseReader) SetTTL(ctx context.Context, orgID string, params *retentiontypes.TTLParams) (*retentiontypes.SetTTLResponseItem, *model.ApiError) {
// Keep only latest 100 transactions/requests
r.deleteTtlTransactions(ctx, orgID, 100)
switch params.Type {
case constants.TraceTTL:
return r.setTTLTraces(ctx, orgID, userID, params)
case constants.MetricsTTL:
return r.setTTLMetrics(ctx, orgID, userID, params)
case constants.LogsTTL:
return r.setTTLLogs(ctx, orgID, userID, params)
case retentiontypes.TraceTTL:
return r.setTTLTraces(ctx, orgID, params)
case retentiontypes.MetricsTTL:
return r.setTTLMetrics(ctx, orgID, params)
case retentiontypes.LogsTTL:
return r.setTTLLogs(ctx, orgID, params)
default:
return nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf("error while setting ttl. ttl type should be <metrics|traces>, got %v", params.Type)}
}
}
func (r *ClickHouseReader) setTTLMetrics(ctx context.Context, orgID string, createdByEmail string, params *model.TTLParams) (*model.SetTTLResponseItem, *model.ApiError) {
func (r *ClickHouseReader) setTTLMetrics(ctx context.Context, orgID string, params *retentiontypes.TTLParams) (*retentiontypes.SetTTLResponseItem, *model.ApiError) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.TelemetrySignal: telemetrytypes.SignalMetrics.StringValue(),
instrumentationtypes.CodeNamespace: "clickhouse-reader",
@@ -2243,7 +2231,7 @@ func (r *ClickHouseReader) setTTLMetrics(ctx context.Context, orgID string, crea
if apiErr != nil {
return nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf("error in processing ttl_status check sql query")}
}
if statusItem.Status == constants.StatusPending {
if statusItem.Status == retentiontypes.TTLSettingStatusPending {
return nil, &model.ApiError{Typ: model.ErrorConflict, Err: fmt.Errorf("TTL is already running")}
}
}
@@ -2256,14 +2244,10 @@ func (r *ClickHouseReader) setTTLMetrics(ctx context.Context, orgID string, crea
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
},
UserAuditable: types.UserAuditable{
CreatedBy: createdByEmail,
UpdatedBy: createdByEmail,
},
TransactionID: uuid,
TableName: tableName,
TTL: int(params.DelDuration),
Status: constants.StatusPending,
Status: retentiontypes.TTLSettingStatusPending,
ColdStorageTTL: coldStorageDuration,
OrgID: orgID,
}
@@ -2301,7 +2285,7 @@ func (r *ClickHouseReader) setTTLMetrics(ctx context.Context, orgID string, crea
NewUpdate().
Model(new(retentiontypes.TTLSetting)).
Set("updated_at = ?", time.Now()).
Set("status = ?", constants.StatusFailed).
Set("status = ?", retentiontypes.TTLSettingStatusFailed).
Where("id = ?", statusItem.ID.StringValue()).
Exec(ctx)
if dbErr != nil {
@@ -2322,7 +2306,7 @@ func (r *ClickHouseReader) setTTLMetrics(ctx context.Context, orgID string, crea
NewUpdate().
Model(new(retentiontypes.TTLSetting)).
Set("updated_at = ?", time.Now()).
Set("status = ?", constants.StatusFailed).
Set("status = ?", retentiontypes.TTLSettingStatusFailed).
Where("id = ?", statusItem.ID.StringValue()).
Exec(ctx)
if dbErr != nil {
@@ -2337,7 +2321,7 @@ func (r *ClickHouseReader) setTTLMetrics(ctx context.Context, orgID string, crea
NewUpdate().
Model(new(retentiontypes.TTLSetting)).
Set("updated_at = ?", time.Now()).
Set("status = ?", constants.StatusSuccess).
Set("status = ?", retentiontypes.TTLSettingStatusSuccess).
Where("id = ?", statusItem.ID.StringValue()).
Exec(ctx)
if dbErr != nil {
@@ -2348,7 +2332,7 @@ func (r *ClickHouseReader) setTTLMetrics(ctx context.Context, orgID string, crea
for _, tableName := range tableNames {
go metricTTL(tableName)
}
return &model.SetTTLResponseItem{Message: "move ttl has been successfully set up"}, nil
return &retentiontypes.SetTTLResponseItem{Message: "move ttl has been successfully set up"}, nil
}
func (r *ClickHouseReader) deleteTtlTransactions(ctx context.Context, orgID string, numberOfTransactionsStore int) {
@@ -2405,7 +2389,7 @@ func (r *ClickHouseReader) checkTTLStatusItem(ctx context.Context, orgID string,
// getTTLQueryStatus fetches ttl_status table status from DB
func (r *ClickHouseReader) getTTLQueryStatus(ctx context.Context, orgID string, tableNameArray []string) (string, *model.ApiError) {
failFlag := false
status := constants.StatusSuccess
status := retentiontypes.TTLSettingStatusSuccess
for _, tableName := range tableNameArray {
statusItem, apiErr := r.checkTTLStatusItem(ctx, orgID, tableName)
emptyStatusStruct := new(retentiontypes.TTLSetting)
@@ -2415,16 +2399,16 @@ func (r *ClickHouseReader) getTTLQueryStatus(ctx context.Context, orgID string,
if apiErr != nil {
return "", &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf("error in processing ttl_status check sql query")}
}
if statusItem.Status == constants.StatusPending && statusItem.UpdatedAt.Unix()-time.Now().Unix() < 3600 {
status = constants.StatusPending
if statusItem.Status == retentiontypes.TTLSettingStatusPending && statusItem.UpdatedAt.Unix()-time.Now().Unix() < 3600 {
status = retentiontypes.TTLSettingStatusPending
return status, nil
}
if statusItem.Status == constants.StatusFailed {
if statusItem.Status == retentiontypes.TTLSettingStatusFailed {
failFlag = true
}
}
if failFlag {
status = constants.StatusFailed
status = retentiontypes.TTLSettingStatusFailed
}
return status, nil
@@ -2477,7 +2461,7 @@ func getLocalTableNameArray(tableNames []string) []string {
}
// GetTTL returns current ttl, expected ttl and past setTTL status for metrics/traces.
func (r *ClickHouseReader) GetTTL(ctx context.Context, orgID string, ttlParams *model.GetTTLParams) (*model.GetTTLResponseItem, *model.ApiError) {
func (r *ClickHouseReader) GetTTL(ctx context.Context, orgID string, ttlParams *retentiontypes.GetTTLParams) (*retentiontypes.GetTTLResponseItem, *model.ApiError) {
ctx = ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
instrumentationtypes.CodeNamespace: "clickhouse-reader",
@@ -2512,8 +2496,8 @@ func (r *ClickHouseReader) GetTTL(ctx context.Context, orgID string, ttlParams *
return delTTL, moveTTL
}
getMetricsTTL := func() (*model.DBResponseTTL, *model.ApiError) {
var dbResp []model.DBResponseTTL
getMetricsTTL := func() (*retentiontypes.DBResponseTTL, *model.ApiError) {
var dbResp []retentiontypes.DBResponseTTL
query := fmt.Sprintf("SELECT engine_full FROM system.tables WHERE name='%v'", signozSampleLocalTableName)
@@ -2530,8 +2514,8 @@ func (r *ClickHouseReader) GetTTL(ctx context.Context, orgID string, ttlParams *
}
}
getTracesTTL := func() (*model.DBResponseTTL, *model.ApiError) {
var dbResp []model.DBResponseTTL
getTracesTTL := func() (*retentiontypes.DBResponseTTL, *model.ApiError) {
var dbResp []retentiontypes.DBResponseTTL
query := fmt.Sprintf("SELECT engine_full FROM system.tables WHERE name='%v' AND database='%v'", r.traceLocalTableName, signozTraceDBName)
@@ -2548,8 +2532,8 @@ func (r *ClickHouseReader) GetTTL(ctx context.Context, orgID string, ttlParams *
}
}
getLogsTTL := func() (*model.DBResponseTTL, *model.ApiError) {
var dbResp []model.DBResponseTTL
getLogsTTL := func() (*retentiontypes.DBResponseTTL, *model.ApiError) {
var dbResp []retentiontypes.DBResponseTTL
query := fmt.Sprintf("SELECT engine_full FROM system.tables WHERE name='%v' AND database='%v'", r.logsLocalTableName, r.logsDB)
@@ -2567,7 +2551,7 @@ func (r *ClickHouseReader) GetTTL(ctx context.Context, orgID string, ttlParams *
}
switch ttlParams.Type {
case constants.TraceTTL:
case retentiontypes.TraceTTL:
tableNameArray := []string{
r.TraceDB + "." + r.traceTableName,
r.TraceDB + "." + r.traceResourceTableV3,
@@ -2595,9 +2579,9 @@ func (r *ClickHouseReader) GetTTL(ctx context.Context, orgID string, ttlParams *
}
delTTL, moveTTL := parseTTL(dbResp.EngineFull)
return &model.GetTTLResponseItem{TracesTime: delTTL, TracesMoveTime: moveTTL, ExpectedTracesTime: ttlQuery.TTL, ExpectedTracesMoveTime: ttlQuery.ColdStorageTTL, Status: status}, nil
return &retentiontypes.GetTTLResponseItem{TracesTime: delTTL, TracesMoveTime: moveTTL, ExpectedTracesTime: ttlQuery.TTL, ExpectedTracesMoveTime: ttlQuery.ColdStorageTTL, Status: status}, nil
case constants.MetricsTTL:
case retentiontypes.MetricsTTL:
tableNameArray := []string{signozMetricDBName + "." + signozSampleTableName}
tableNameArray = getLocalTableNameArray(tableNameArray)
status, apiErr := r.getTTLQueryStatus(ctx, orgID, tableNameArray)
@@ -2618,9 +2602,9 @@ func (r *ClickHouseReader) GetTTL(ctx context.Context, orgID string, ttlParams *
}
delTTL, moveTTL := parseTTL(dbResp.EngineFull)
return &model.GetTTLResponseItem{MetricsTime: delTTL, MetricsMoveTime: moveTTL, ExpectedMetricsTime: ttlQuery.TTL, ExpectedMetricsMoveTime: ttlQuery.ColdStorageTTL, Status: status}, nil
return &retentiontypes.GetTTLResponseItem{MetricsTime: delTTL, MetricsMoveTime: moveTTL, ExpectedMetricsTime: ttlQuery.TTL, ExpectedMetricsMoveTime: ttlQuery.ColdStorageTTL, Status: status}, nil
case constants.LogsTTL:
case retentiontypes.LogsTTL:
tableNameArray := []string{r.logsDB + "." + r.logsTableName}
tableNameArray = getLocalTableNameArray(tableNameArray)
status, apiErr := r.getTTLQueryStatus(ctx, orgID, tableNameArray)
@@ -2641,7 +2625,7 @@ func (r *ClickHouseReader) GetTTL(ctx context.Context, orgID string, ttlParams *
}
delTTL, moveTTL := parseTTL(dbResp.EngineFull)
return &model.GetTTLResponseItem{LogsTime: delTTL, LogsMoveTime: moveTTL, ExpectedLogsTime: ttlQuery.TTL, ExpectedLogsMoveTime: ttlQuery.ColdStorageTTL, Status: status}, nil
return &retentiontypes.GetTTLResponseItem{LogsTime: delTTL, LogsMoveTime: moveTTL, ExpectedLogsTime: ttlQuery.TTL, ExpectedLogsMoveTime: ttlQuery.ColdStorageTTL, Status: status}, nil
default:
return nil, &model.ApiError{Typ: model.ErrorExec, Err: fmt.Errorf("error while getting ttl. ttl type should be metrics|traces, got %v",

View File

@@ -34,6 +34,7 @@ import (
"github.com/SigNoz/signoz/pkg/query-service/app/cloudintegrations/services"
"github.com/SigNoz/signoz/pkg/query-service/app/integrations"
"github.com/SigNoz/signoz/pkg/signoz"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/gorilla/mux"
@@ -1655,7 +1656,7 @@ func (aH *APIHandler) setTTL(w http.ResponseWriter, r *http.Request) {
}
// Context is not used here as TTL is long duration DB operation
result, apiErr := aH.reader.SetTTL(context.Background(), claims.OrgID, claims.Email, ttlParams)
result, apiErr := aH.reader.SetTTL(context.Background(), claims.OrgID, ttlParams)
if apiErr != nil {
if apiErr.Typ == model.ErrorConflict {
aH.HandleError(w, apiErr.Err, http.StatusConflict)
@@ -1677,14 +1678,14 @@ func (aH *APIHandler) setCustomRetentionTTL(w http.ResponseWriter, r *http.Reque
return
}
var params model.CustomRetentionTTLParams
var params retentiontypes.CustomRetentionTTLParams
if err := json.NewDecoder(r.Body).Decode(&params); err != nil {
render.Error(w, errorsV2.Newf(errorsV2.TypeInvalidInput, errorsV2.CodeInvalidInput, "Invalid data"))
return
}
// Context is not used here as TTL is long duration DB operation
result, apiErr := aH.reader.SetTTLV2(context.Background(), claims.OrgID, claims.Email, &params)
result, apiErr := aH.reader.SetTTLV2(context.Background(), claims.OrgID, &params)
if apiErr != nil {
render.Error(w, errorsV2.New(errorsV2.TypeInvalidInput, errorsV2.CodeInternal, apiErr.Error()))
return

View File

@@ -40,6 +40,7 @@ import (
"github.com/SigNoz/signoz/pkg/query-service/postprocess"
"github.com/SigNoz/signoz/pkg/query-service/utils"
querytemplate "github.com/SigNoz/signoz/pkg/query-service/utils/queryTemplate"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
chVariables "github.com/SigNoz/signoz/pkg/variables/clickhouse"
)
@@ -419,7 +420,7 @@ func parseTime(param string, r *http.Request) (*time.Time, error) {
}
func parseTTLParams(r *http.Request) (*model.TTLParams, error) {
func parseTTLParams(r *http.Request) (*retentiontypes.TTLParams, error) {
// make sure either of the query params are present
typeTTL := r.URL.Query().Get("type")
@@ -432,7 +433,7 @@ func parseTTLParams(r *http.Request) (*model.TTLParams, error) {
}
// Validate the type parameter
if typeTTL != baseconstants.TraceTTL && typeTTL != baseconstants.MetricsTTL && typeTTL != baseconstants.LogsTTL {
if typeTTL != retentiontypes.TraceTTL && typeTTL != retentiontypes.MetricsTTL && typeTTL != retentiontypes.LogsTTL {
return nil, fmt.Errorf("type param should be metrics|traces|logs, got %v", typeTTL)
}
@@ -455,7 +456,7 @@ func parseTTLParams(r *http.Request) (*model.TTLParams, error) {
}
}
return &model.TTLParams{
return &retentiontypes.TTLParams{
Type: typeTTL,
DelDuration: int64(durationParsed.Seconds()),
ColdStorageVolume: coldStorage,
@@ -463,7 +464,7 @@ func parseTTLParams(r *http.Request) (*model.TTLParams, error) {
}, nil
}
func parseGetTTL(r *http.Request) (*model.GetTTLParams, error) {
func parseGetTTL(r *http.Request) (*retentiontypes.GetTTLParams, error) {
typeTTL := r.URL.Query().Get("type")
@@ -471,12 +472,12 @@ func parseGetTTL(r *http.Request) (*model.GetTTLParams, error) {
return nil, fmt.Errorf("type param cannot be empty from the query")
} else {
// Validate the type parameter
if typeTTL != baseconstants.TraceTTL && typeTTL != baseconstants.MetricsTTL && typeTTL != baseconstants.LogsTTL {
if typeTTL != retentiontypes.TraceTTL && typeTTL != retentiontypes.MetricsTTL && typeTTL != retentiontypes.LogsTTL {
return nil, fmt.Errorf("type param should be metrics|traces|logs, got %v", typeTTL)
}
}
return &model.GetTTLParams{Type: typeTTL}, nil
return &retentiontypes.GetTTLParams{Type: typeTTL}, nil
}
func parseAggregateAttributeRequest(r *http.Request) (*v3.AggregateAttributeRequest, error) {

View File

@@ -19,10 +19,6 @@ const (
const MaxAllowedPointsInTimeSeries = 300
const TraceTTL = "traces"
const MetricsTTL = "metrics"
const LogsTTL = "logs"
const SpanSearchScopeRoot = "isroot"
const SpanSearchScopeEntryPoint = "isentrypoint"
const OrderBySpanCount = "span_count"

View File

@@ -7,6 +7,7 @@ import (
"github.com/SigNoz/signoz/pkg/query-service/model"
v3 "github.com/SigNoz/signoz/pkg/query-service/model/v3"
"github.com/SigNoz/signoz/pkg/query-service/querycache"
"github.com/SigNoz/signoz/pkg/types/retentiontypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/prometheus/prometheus/promql"
"github.com/prometheus/prometheus/util/stats"
@@ -23,8 +24,8 @@ type Reader interface {
GetServicesList(ctx context.Context) (*[]string, error)
GetDependencyGraph(ctx context.Context, query *model.GetServicesParams) (*[]model.ServiceMapDependencyResponseItem, error)
GetTTL(ctx context.Context, orgID string, ttlParams *model.GetTTLParams) (*model.GetTTLResponseItem, *model.ApiError)
GetCustomRetentionTTL(ctx context.Context, orgID string) (*model.GetCustomRetentionTTLResponse, error)
GetTTL(ctx context.Context, orgID string, ttlParams *retentiontypes.GetTTLParams) (*retentiontypes.GetTTLResponseItem, *model.ApiError)
GetCustomRetentionTTL(ctx context.Context, orgID string) (*retentiontypes.GetCustomRetentionTTLResponse, error)
// GetDisks returns a list of disks configured in the underlying DB. It is supported by
// clickhouse only.
@@ -46,8 +47,8 @@ type Reader interface {
GetFlamegraphSpansForTrace(ctx context.Context, orgID valuer.UUID, traceID string, req *model.GetFlamegraphSpansForTraceParams) (*model.GetFlamegraphSpansForTraceResponse, error)
// Setter Interfaces
SetTTL(ctx context.Context, orgID string, userID string, ttlParams *model.TTLParams) (*model.SetTTLResponseItem, *model.ApiError)
SetTTLV2(ctx context.Context, orgID string, userID string, params *model.CustomRetentionTTLParams) (*model.CustomRetentionTTLResponse, error)
SetTTL(ctx context.Context, orgID string, ttlParams *retentiontypes.TTLParams) (*retentiontypes.SetTTLResponseItem, *model.ApiError)
SetTTLV2(ctx context.Context, orgID string, params *retentiontypes.CustomRetentionTTLParams) (*retentiontypes.CustomRetentionTTLResponse, error)
FetchTemporality(ctx context.Context, orgID valuer.UUID, metricNames []string) (map[string]map[v3.Temporality]bool, error)
GetMetricAggregateAttributes(ctx context.Context, orgID valuer.UUID, req *v3.AggregateAttributeRequest, skipSignozMetrics bool) (*v3.AggregateAttributeResponse, error)

View File

@@ -404,56 +404,6 @@ type TagKey struct {
Type TagDataType `json:"type"`
}
type TTLParams struct {
Type string // It can be one of {traces, metrics}.
ColdStorageVolume string // Name of the cold storage volume.
ToColdStorageDuration int64 // Seconds after which data will be moved to cold storage.
DelDuration int64 // Seconds after which data will be deleted.
}
type CustomRetentionTTLParams struct {
Type string `json:"type"`
DefaultTTLDays int `json:"defaultTTLDays"`
TTLConditions []CustomRetentionRule `json:"ttlConditions"`
ColdStorageVolume string `json:"coldStorageVolume,omitempty"`
ToColdStorageDurationDays int64 `json:"coldStorageDurationDays,omitempty"`
}
type CustomRetentionRule struct {
Filters []FilterCondition `json:"conditions"`
TTLDays int `json:"ttlDays"`
}
type FilterCondition struct {
Key string `json:"key"`
Values []string `json:"values"`
}
type GetCustomRetentionTTLResponse struct {
Version string `json:"version"`
Status string `json:"status"`
// V1 fields
// LogsTime int `json:"logs_ttl_duration_hrs,omitempty"`
// LogsMoveTime int `json:"logs_move_ttl_duration_hrs,omitempty"`
ExpectedLogsTime int `json:"expected_logs_ttl_duration_hrs,omitempty"`
ExpectedLogsMoveTime int `json:"expected_logs_move_ttl_duration_hrs,omitempty"`
// V2 fields
DefaultTTLDays int `json:"default_ttl_days,omitempty"`
TTLConditions []CustomRetentionRule `json:"ttl_conditions,omitempty"`
ColdStorageVolume string `json:"cold_storage_volume,omitempty"`
ColdStorageTTLDays int `json:"cold_storage_ttl_days,omitempty"`
}
type CustomRetentionTTLResponse struct {
Message string `json:"message"`
}
type GetTTLParams struct {
Type string
}
type ListErrorsParams struct {
StartStr string `json:"start"`
EndStr string `json:"end"`

View File

@@ -150,16 +150,6 @@ type RuleResponseItem struct {
Data string `json:"data" db:"data"`
}
type TTLStatusItem struct {
Id int `json:"id" db:"id"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
TableName string `json:"table_name" db:"table_name"`
TTL int `json:"ttl" db:"ttl"`
Status string `json:"status" db:"status"`
ColdStorageTtl int `json:"cold_storage_ttl" db:"cold_storage_ttl"`
}
type ChannelItem struct {
Id int `json:"id" db:"id"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
@@ -462,35 +452,11 @@ type SpanAggregatesDBResponseItem struct {
GroupBy string `ch:"groupBy"`
}
type SetTTLResponseItem struct {
Message string `json:"message"`
}
type DiskItem struct {
Name string `json:"name,omitempty" ch:"name"`
Type string `json:"type,omitempty" ch:"type"`
}
type DBResponseTTL struct {
EngineFull string `ch:"engine_full"`
}
type GetTTLResponseItem struct {
MetricsTime int `json:"metrics_ttl_duration_hrs,omitempty"`
MetricsMoveTime int `json:"metrics_move_ttl_duration_hrs,omitempty"`
TracesTime int `json:"traces_ttl_duration_hrs,omitempty"`
TracesMoveTime int `json:"traces_move_ttl_duration_hrs,omitempty"`
LogsTime int `json:"logs_ttl_duration_hrs,omitempty"`
LogsMoveTime int `json:"logs_move_ttl_duration_hrs,omitempty"`
ExpectedMetricsTime int `json:"expected_metrics_ttl_duration_hrs,omitempty"`
ExpectedMetricsMoveTime int `json:"expected_metrics_move_ttl_duration_hrs,omitempty"`
ExpectedTracesTime int `json:"expected_traces_ttl_duration_hrs,omitempty"`
ExpectedTracesMoveTime int `json:"expected_traces_move_ttl_duration_hrs,omitempty"`
ExpectedLogsTime int `json:"expected_logs_ttl_duration_hrs,omitempty"`
ExpectedLogsMoveTime int `json:"expected_logs_move_ttl_duration_hrs,omitempty"`
Status string `json:"status"`
}
type DBResponseServiceName struct {
ServiceName string `ch:"serviceName"`
Count uint64 `ch:"count"`

View File

@@ -23,6 +23,7 @@ import (
"github.com/SigNoz/signoz/pkg/global"
"github.com/SigNoz/signoz/pkg/identn"
"github.com/SigNoz/signoz/pkg/instrumentation"
"github.com/SigNoz/signoz/pkg/meterreporter"
"github.com/SigNoz/signoz/pkg/modules/cloudintegration"
"github.com/SigNoz/signoz/pkg/modules/inframonitoring"
"github.com/SigNoz/signoz/pkg/modules/metricsexplorer"
@@ -135,6 +136,9 @@ type Config struct {
// Auditor config
Auditor auditor.Config `mapstructure:"auditor"`
// MeterReporter config
MeterReporter meterreporter.Config `mapstructure:"meterreporter"`
// CloudIntegration config
CloudIntegration cloudintegration.Config `mapstructure:"cloudintegration"`
@@ -175,6 +179,7 @@ func NewConfig(ctx context.Context, logger *slog.Logger, resolverConfig config.R
identn.NewConfigFactory(),
serviceaccount.NewConfigFactory(),
auditor.NewConfigFactory(),
meterreporter.NewConfigFactory(),
cloudintegration.NewConfigFactory(),
tracedetail.NewConfigFactory(),
authz.NewConfigFactory(),

View File

@@ -16,6 +16,7 @@ import (
"github.com/SigNoz/signoz/pkg/instrumentation/instrumentationtest"
"github.com/SigNoz/signoz/pkg/modules/dashboard/impldashboard"
"github.com/SigNoz/signoz/pkg/modules/organization/implorganization"
"github.com/SigNoz/signoz/pkg/modules/retention/implretention"
"github.com/SigNoz/signoz/pkg/modules/user/impluser"
"github.com/SigNoz/signoz/pkg/querier"
"github.com/SigNoz/signoz/pkg/queryparser"
@@ -52,7 +53,8 @@ func TestNewHandlers(t *testing.T) {
userRoleStore := impluser.NewUserRoleStore(sqlstore, providerSettings)
userGetter := impluser.NewGetter(impluser.NewStore(sqlstore, providerSettings), userRoleStore, flagger)
modules := NewModules(sqlstore, tokenizer, emailing, providerSettings, orgGetter, alertmanager, nil, nil, nil, nil, nil, nil, nil, queryParser, Config{}, dashboardModule, userGetter, userRoleStore, nil, nil, flagger)
retentionGetter := implretention.NewGetter(implretention.NewStore(sqlstore))
modules := NewModules(sqlstore, tokenizer, emailing, providerSettings, orgGetter, alertmanager, nil, nil, nil, nil, nil, nil, nil, queryParser, Config{}, dashboardModule, userGetter, userRoleStore, nil, nil, retentionGetter, flagger)
querierHandler := querier.NewHandler(providerSettings, nil, nil)
registryHandler := factory.NewHandler(nil)

View File

@@ -29,6 +29,7 @@ import (
"github.com/SigNoz/signoz/pkg/modules/quickfilter/implquickfilter"
"github.com/SigNoz/signoz/pkg/modules/rawdataexport"
"github.com/SigNoz/signoz/pkg/modules/rawdataexport/implrawdataexport"
"github.com/SigNoz/signoz/pkg/modules/retention"
"github.com/SigNoz/signoz/pkg/modules/rulestatehistory"
"github.com/SigNoz/signoz/pkg/modules/rulestatehistory/implrulestatehistory"
"github.com/SigNoz/signoz/pkg/modules/savedview"
@@ -63,6 +64,7 @@ type Modules struct {
Preference preference.Module
UserSetter user.Setter
UserGetter user.Getter
RetentionGetter retention.Getter
SavedView savedview.Module
Apdex apdex.Module
Dashboard dashboard.Module
@@ -103,6 +105,7 @@ func NewModules(
userRoleStore authtypes.UserRoleStore,
serviceAccount serviceaccount.Module,
cloudIntegrationModule cloudintegration.Module,
retentionGetter retention.Getter,
fl flagger.Flagger,
) Modules {
quickfilter := implquickfilter.NewModule(implquickfilter.NewStore(sqlstore))
@@ -119,6 +122,7 @@ func NewModules(
Dashboard: dashboard,
UserSetter: userSetter,
UserGetter: userGetter,
RetentionGetter: retentionGetter,
QuickFilter: quickfilter,
TraceFunnel: impltracefunnel.NewModule(impltracefunnel.NewStore(sqlstore)),
RawDataExport: implrawdataexport.NewModule(querier),

View File

@@ -16,6 +16,7 @@ import (
"github.com/SigNoz/signoz/pkg/modules/cloudintegration/implcloudintegration"
"github.com/SigNoz/signoz/pkg/modules/dashboard/impldashboard"
"github.com/SigNoz/signoz/pkg/modules/organization/implorganization"
"github.com/SigNoz/signoz/pkg/modules/retention/implretention"
"github.com/SigNoz/signoz/pkg/modules/serviceaccount"
"github.com/SigNoz/signoz/pkg/modules/serviceaccount/implserviceaccount"
"github.com/SigNoz/signoz/pkg/modules/user/impluser"
@@ -56,7 +57,8 @@ func TestNewModules(t *testing.T) {
serviceAccount := implserviceaccount.NewModule(implserviceaccount.NewStore(sqlstore), nil, nil, nil, providerSettings, serviceaccount.Config{})
modules := NewModules(sqlstore, tokenizer, emailing, providerSettings, orgGetter, alertmanager, nil, nil, nil, nil, nil, nil, nil, queryParser, Config{}, dashboardModule, userGetter, userRoleStore, serviceAccount, implcloudintegration.NewModule(), flagger)
retentionGetter := implretention.NewGetter(implretention.NewStore(sqlstore))
modules := NewModules(sqlstore, tokenizer, emailing, providerSettings, orgGetter, alertmanager, nil, nil, nil, nil, nil, nil, nil, queryParser, Config{}, dashboardModule, userGetter, userRoleStore, serviceAccount, implcloudintegration.NewModule(), retentionGetter, flagger)
reflectVal := reflect.ValueOf(modules)
for i := 0; i < reflectVal.NumField(); i++ {

View File

@@ -28,6 +28,8 @@ import (
"github.com/SigNoz/signoz/pkg/identn/apikeyidentn"
"github.com/SigNoz/signoz/pkg/identn/impersonationidentn"
"github.com/SigNoz/signoz/pkg/identn/tokenizeridentn"
"github.com/SigNoz/signoz/pkg/meterreporter"
"github.com/SigNoz/signoz/pkg/meterreporter/noopmeterreporter"
"github.com/SigNoz/signoz/pkg/modules/authdomain/implauthdomain"
"github.com/SigNoz/signoz/pkg/modules/organization"
"github.com/SigNoz/signoz/pkg/modules/organization/implorganization"
@@ -196,7 +198,6 @@ func NewSQLMigrationProviderFactories(
sqlmigration.NewDropUserDeletedAtFactory(sqlstore, sqlschema),
sqlmigration.NewMigrateAWSAllRegionsFactory(sqlstore),
sqlmigration.NewAddServiceAccountManagedRoleTransactionsFactory(sqlstore),
sqlmigration.NewUpdateTTLSettingUserAuditFactory(sqlstore, sqlschema),
)
}
@@ -320,6 +321,12 @@ func NewAuditorProviderFactories() factory.NamedMap[factory.ProviderFactory[audi
)
}
func NewMeterReporterProviderFactories() factory.NamedMap[factory.ProviderFactory[meterreporter.Reporter, meterreporter.Config]] {
return factory.MustNewNamedMap(
noopmeterreporter.NewFactory(),
)
}
func NewFlaggerProviderFactories(registry featuretypes.Registry) factory.NamedMap[factory.ProviderFactory[flagger.FlaggerProvider, flagger.Config]] {
return factory.MustNewNamedMap(
configflagger.NewFactory(registry),

View File

@@ -22,10 +22,13 @@ import (
"github.com/SigNoz/signoz/pkg/identn"
"github.com/SigNoz/signoz/pkg/instrumentation"
"github.com/SigNoz/signoz/pkg/licensing"
"github.com/SigNoz/signoz/pkg/meterreporter"
"github.com/SigNoz/signoz/pkg/modules/cloudintegration"
"github.com/SigNoz/signoz/pkg/modules/dashboard"
"github.com/SigNoz/signoz/pkg/modules/organization"
"github.com/SigNoz/signoz/pkg/modules/organization/implorganization"
"github.com/SigNoz/signoz/pkg/modules/retention"
"github.com/SigNoz/signoz/pkg/modules/retention/implretention"
"github.com/SigNoz/signoz/pkg/modules/rulestatehistory"
"github.com/SigNoz/signoz/pkg/modules/serviceaccount"
"github.com/SigNoz/signoz/pkg/modules/serviceaccount/implserviceaccount"
@@ -84,6 +87,7 @@ type SigNoz struct {
Flagger flagger.Flagger
Gateway gateway.Gateway
Auditor auditor.Auditor
MeterReporter meterreporter.Reporter
}
func New(
@@ -104,6 +108,7 @@ func New(
dashboardModuleCallback func(sqlstore.SQLStore, factory.ProviderSettings, analytics.Analytics, organization.Getter, queryparser.QueryParser, querier.Querier, licensing.Licensing) dashboard.Module,
gatewayProviderFactory func(licensing.Licensing) factory.ProviderFactory[gateway.Gateway, gateway.Config],
auditorProviderFactories func(licensing.Licensing) factory.NamedMap[factory.ProviderFactory[auditor.Auditor, auditor.Config]],
meterReporterProviderFactories func(context.Context, factory.ProviderSettings, flagger.Flagger, licensing.Licensing, telemetrystore.TelemetryStore, retention.Getter, organization.Getter, zeus.Zeus) (factory.NamedMap[factory.ProviderFactory[meterreporter.Reporter, meterreporter.Config]], string),
querierHandlerCallback func(factory.ProviderSettings, querier.Querier, analytics.Analytics) querier.Handler,
cloudIntegrationCallback func(sqlstore.SQLStore, global.Global, zeus.Zeus, gateway.Gateway, licensing.Licensing, serviceaccount.Module, cloudintegration.Config) (cloudintegration.Module, error),
rulerProviderFactories func(cache.Cache, alertmanager.Alertmanager, sqlstore.SQLStore, telemetrystore.TelemetryStore, telemetrytypes.MetadataStore, prometheus.Prometheus, organization.Getter, rulestatehistory.Module, querier.Querier, queryparser.QueryParser) factory.NamedMap[factory.ProviderFactory[ruler.Ruler, ruler.Config]],
@@ -228,6 +233,8 @@ func New(
return nil, err
}
retentionGetter := implretention.NewGetter(implretention.NewStore(sqlstore))
// Initialize prometheus from the available prometheus provider factories
prometheus, err := factory.NewProviderFromNamedMap(
ctx,
@@ -386,6 +393,13 @@ func New(
return nil, err
}
// Initialize meter reporter from the variant-specific provider factories
meterReporterFactories, meterReporterProvider := meterReporterProviderFactories(ctx, providerSettings, flagger, licensing, telemetrystore, retentionGetter, orgGetter, zeus)
meterReporter, err := factory.NewProviderFromNamedMap(ctx, providerSettings, config.MeterReporter, meterReporterFactories, meterReporterProvider)
if err != nil {
return nil, err
}
// Initialize authns
store := sqlauthnstore.NewStore(sqlstore)
authNs, err := authNsCallback(ctx, providerSettings, store, licensing)
@@ -441,7 +455,7 @@ func New(
}
// Initialize all modules
modules := NewModules(sqlstore, tokenizer, emailing, providerSettings, orgGetter, alertmanager, analytics, querier, telemetrystore, telemetryMetadataStore, authNs, authz, cache, queryParser, config, dashboard, userGetter, userRoleStore, serviceAccount, cloudIntegrationModule, flagger)
modules := NewModules(sqlstore, tokenizer, emailing, providerSettings, orgGetter, alertmanager, analytics, querier, telemetrystore, telemetryMetadataStore, authNs, authz, cache, queryParser, config, dashboard, userGetter, userRoleStore, serviceAccount, cloudIntegrationModule, retentionGetter, flagger)
// Initialize ruler from the variant-specific provider factories
rulerInstance, err := factory.NewProviderFromNamedMap(ctx, providerSettings, config.Ruler, rulerProviderFactories(cache, alertmanager, sqlstore, telemetrystore, telemetryMetadataStore, prometheus, orgGetter, modules.RuleStateHistory, querier, queryParser), "signoz")
@@ -501,6 +515,7 @@ func New(
factory.NewNamedService(factory.MustNewName("authz"), authz),
factory.NewNamedService(factory.MustNewName("user"), userService, factory.MustNewName("authz")),
factory.NewNamedService(factory.MustNewName("auditor"), auditor),
factory.NewNamedService(factory.MustNewName("meterreporter"), meterReporter, factory.MustNewName("licensing")),
factory.NewNamedService(factory.MustNewName("ruler"), rulerInstance),
)
if err != nil {
@@ -550,5 +565,6 @@ func New(
Flagger: flagger,
Gateway: gateway,
Auditor: auditor,
MeterReporter: meterReporter,
}, nil
}

View File

@@ -1,84 +0,0 @@
package sqlmigration
import (
"context"
"github.com/SigNoz/signoz/pkg/factory"
"github.com/SigNoz/signoz/pkg/sqlschema"
"github.com/SigNoz/signoz/pkg/sqlstore"
"github.com/uptrace/bun"
"github.com/uptrace/bun/migrate"
)
type updateTTLSettingUserAudit struct {
sqlstore sqlstore.SQLStore
sqlschema sqlschema.SQLSchema
}
func NewUpdateTTLSettingUserAuditFactory(sqlstore sqlstore.SQLStore, sqlschema sqlschema.SQLSchema) factory.ProviderFactory[SQLMigration, Config] {
return factory.NewProviderFactory(factory.MustNewName("update_ttl_setting_user_audit"), func(ctx context.Context, providerSettings factory.ProviderSettings, config Config) (SQLMigration, error) {
return newUpdateTTLSettingUserAudit(ctx, providerSettings, config, sqlstore, sqlschema)
})
}
func newUpdateTTLSettingUserAudit(_ context.Context, _ factory.ProviderSettings, _ Config, sqlstore sqlstore.SQLStore, sqlschema sqlschema.SQLSchema) (SQLMigration, error) {
return &updateTTLSettingUserAudit{
sqlstore: sqlstore,
sqlschema: sqlschema,
}, nil
}
func (migration *updateTTLSettingUserAudit) Register(migrations *migrate.Migrations) error {
if err := migrations.Register(migration.Up, migration.Down); err != nil {
return err
}
return nil
}
func (migration *updateTTLSettingUserAudit) Up(ctx context.Context, db *bun.DB) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer func() {
_ = tx.Rollback()
}()
table, uniqueConstraints, err := migration.sqlschema.GetTable(ctx, sqlschema.TableName("ttl_setting"))
if err != nil {
return err
}
columns := []*sqlschema.Column{
{
Name: sqlschema.ColumnName("created_by"),
DataType: sqlschema.DataTypeText,
Nullable: true,
},
{
Name: sqlschema.ColumnName("updated_by"),
DataType: sqlschema.DataTypeText,
Nullable: true,
},
}
for _, column := range columns {
sqls := migration.sqlschema.Operator().AddColumn(table, uniqueConstraints, column, nil)
for _, sql := range sqls {
if _, err := tx.ExecContext(ctx, string(sql)); err != nil {
return err
}
}
}
if err := tx.Commit(); err != nil {
return err
}
return nil
}
func (migration *updateTTLSettingUserAudit) Down(ctx context.Context, db *bun.DB) error {
return nil
}

View File

@@ -0,0 +1,105 @@
package inframonitoringtypes
import (
"encoding/json"
"slices"
"github.com/SigNoz/signoz/pkg/errors"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
)
type Clusters struct {
Type ResponseType `json:"type" required:"true"`
Records []ClusterRecord `json:"records" required:"true"`
Total int `json:"total" required:"true"`
RequiredMetricsCheck RequiredMetricsCheck `json:"requiredMetricsCheck" required:"true"`
EndTimeBeforeRetention bool `json:"endTimeBeforeRetention" required:"true"`
Warning *qbtypes.QueryWarnData `json:"warning,omitempty"`
}
type ClusterRecord struct {
// TODO(nikhilmantri0902): once the underlying attr key is migrated to
// k8s.cluster.uid (see clusterNameAttrKey TODO in implinframonitoring),
// surface ClusterUID alongside (or replace) ClusterName.
ClusterName string `json:"clusterName" required:"true"`
ClusterCPU float64 `json:"clusterCPU" required:"true"`
ClusterCPUAllocatable float64 `json:"clusterCPUAllocatable" required:"true"`
ClusterMemory float64 `json:"clusterMemory" required:"true"`
ClusterMemoryAllocatable float64 `json:"clusterMemoryAllocatable" required:"true"`
NodeCountsByReadiness NodeCountsByReadiness `json:"nodeCountsByReadiness" required:"true"`
PodCountsByPhase PodCountsByPhase `json:"podCountsByPhase" required:"true"`
Meta map[string]string `json:"meta" required:"true"`
}
// PostableClusters is the request body for the v2 clusters list API.
type PostableClusters struct {
Start int64 `json:"start" required:"true"`
End int64 `json:"end" required:"true"`
Filter *qbtypes.Filter `json:"filter"`
GroupBy []qbtypes.GroupByKey `json:"groupBy"`
OrderBy *qbtypes.OrderBy `json:"orderBy"`
Offset int `json:"offset"`
Limit int `json:"limit" required:"true"`
}
// Validate ensures PostableClusters contains acceptable values.
func (req *PostableClusters) Validate() error {
if req == nil {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "request is nil")
}
if req.Start <= 0 {
return errors.NewInvalidInputf(
errors.CodeInvalidInput,
"invalid start time %d: start must be greater than 0",
req.Start,
)
}
if req.End <= 0 {
return errors.NewInvalidInputf(
errors.CodeInvalidInput,
"invalid end time %d: end must be greater than 0",
req.End,
)
}
if req.Start >= req.End {
return errors.NewInvalidInputf(
errors.CodeInvalidInput,
"invalid time range: start (%d) must be less than end (%d)",
req.Start,
req.End,
)
}
if req.Limit < 1 || req.Limit > 5000 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "limit must be between 1 and 5000")
}
if req.Offset < 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "offset cannot be negative")
}
if req.OrderBy != nil {
if !slices.Contains(ClustersValidOrderByKeys, req.OrderBy.Key.Name) {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid order by key: %s", req.OrderBy.Key.Name)
}
if req.OrderBy.Direction != qbtypes.OrderDirectionAsc && req.OrderBy.Direction != qbtypes.OrderDirectionDesc {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid order by direction: %s", req.OrderBy.Direction)
}
}
return nil
}
// UnmarshalJSON validates input immediately after decoding.
func (req *PostableClusters) UnmarshalJSON(data []byte) error {
type raw PostableClusters
var decoded raw
if err := json.Unmarshal(data, &decoded); err != nil {
return err
}
*req = PostableClusters(decoded)
return req.Validate()
}

View File

@@ -0,0 +1,15 @@
package inframonitoringtypes
const (
ClustersOrderByCPU = "cpu"
ClustersOrderByCPUAllocatable = "cpu_allocatable"
ClustersOrderByMemory = "memory"
ClustersOrderByMemoryAllocatable = "memory_allocatable"
)
var ClustersValidOrderByKeys = []string{
ClustersOrderByCPU,
ClustersOrderByCPUAllocatable,
ClustersOrderByMemory,
ClustersOrderByMemoryAllocatable,
}

View File

@@ -0,0 +1,291 @@
package inframonitoringtypes
import (
"testing"
"github.com/SigNoz/signoz/pkg/errors"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/require"
)
func TestPostableClusters_Validate(t *testing.T) {
tests := []struct {
name string
req *PostableClusters
wantErr bool
}{
{
name: "valid request",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
},
wantErr: false,
},
{
name: "nil request",
req: nil,
wantErr: true,
},
{
name: "start time zero",
req: &PostableClusters{
Start: 0,
End: 2000,
Limit: 100,
Offset: 0,
},
wantErr: true,
},
{
name: "start time negative",
req: &PostableClusters{
Start: -1000,
End: 2000,
Limit: 100,
Offset: 0,
},
wantErr: true,
},
{
name: "end time zero",
req: &PostableClusters{
Start: 1000,
End: 0,
Limit: 100,
Offset: 0,
},
wantErr: true,
},
{
name: "start time greater than end time",
req: &PostableClusters{
Start: 2000,
End: 1000,
Limit: 100,
Offset: 0,
},
wantErr: true,
},
{
name: "start time equal to end time",
req: &PostableClusters{
Start: 1000,
End: 1000,
Limit: 100,
Offset: 0,
},
wantErr: true,
},
{
name: "limit zero",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 0,
Offset: 0,
},
wantErr: true,
},
{
name: "limit negative",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: -10,
Offset: 0,
},
wantErr: true,
},
{
name: "limit exceeds max",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 5001,
Offset: 0,
},
wantErr: true,
},
{
name: "offset negative",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: -5,
},
wantErr: true,
},
{
name: "orderBy nil is valid",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
},
wantErr: false,
},
{
name: "orderBy with valid key cpu and direction asc",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: ClustersOrderByCPU,
},
},
Direction: qbtypes.OrderDirectionAsc,
},
},
wantErr: false,
},
{
name: "orderBy with valid key cpu_allocatable and direction desc",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: ClustersOrderByCPUAllocatable,
},
},
Direction: qbtypes.OrderDirectionDesc,
},
},
wantErr: false,
},
{
name: "orderBy with valid key memory and direction desc",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: ClustersOrderByMemory,
},
},
Direction: qbtypes.OrderDirectionDesc,
},
},
wantErr: false,
},
{
name: "orderBy with valid key memory_allocatable and direction asc",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: ClustersOrderByMemoryAllocatable,
},
},
Direction: qbtypes.OrderDirectionAsc,
},
},
wantErr: false,
},
{
name: "orderBy with condition key is rejected",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "condition",
},
},
Direction: qbtypes.OrderDirectionDesc,
},
},
wantErr: true,
},
{
name: "orderBy with pod_phase key is rejected",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "pod_phase",
},
},
Direction: qbtypes.OrderDirectionDesc,
},
},
wantErr: true,
},
{
name: "orderBy with invalid key",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "unknown",
},
},
Direction: qbtypes.OrderDirectionDesc,
},
},
wantErr: true,
},
{
name: "orderBy with valid key but invalid direction",
req: &PostableClusters{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: ClustersOrderByMemory,
},
},
Direction: qbtypes.OrderDirection{String: valuer.NewString("invalid")},
},
},
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := tt.req.Validate()
if tt.wantErr {
require.Error(t, err)
require.True(t, errors.Ast(err, errors.TypeInvalidInput), "expected error to be of type InvalidInput")
} else {
require.NoError(t, err)
}
})
}
}

View File

@@ -0,0 +1,99 @@
package inframonitoringtypes
import (
"encoding/json"
"slices"
"github.com/SigNoz/signoz/pkg/errors"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
)
type Namespaces struct {
Type ResponseType `json:"type" required:"true"`
Records []NamespaceRecord `json:"records" required:"true"`
Total int `json:"total" required:"true"`
RequiredMetricsCheck RequiredMetricsCheck `json:"requiredMetricsCheck" required:"true"`
EndTimeBeforeRetention bool `json:"endTimeBeforeRetention" required:"true"`
Warning *qbtypes.QueryWarnData `json:"warning,omitempty"`
}
type NamespaceRecord struct {
NamespaceName string `json:"namespaceName" required:"true"`
NamespaceCPU float64 `json:"namespaceCPU" required:"true"`
NamespaceMemory float64 `json:"namespaceMemory" required:"true"`
PodCountsByPhase PodCountsByPhase `json:"podCountsByPhase" required:"true"`
Meta map[string]string `json:"meta" required:"true"`
}
// PostableNamespaces is the request body for the v2 namespaces list API.
type PostableNamespaces struct {
Start int64 `json:"start" required:"true"`
End int64 `json:"end" required:"true"`
Filter *qbtypes.Filter `json:"filter"`
GroupBy []qbtypes.GroupByKey `json:"groupBy"`
OrderBy *qbtypes.OrderBy `json:"orderBy"`
Offset int `json:"offset"`
Limit int `json:"limit" required:"true"`
}
// Validate ensures PostableNamespaces contains acceptable values.
func (req *PostableNamespaces) Validate() error {
if req == nil {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "request is nil")
}
if req.Start <= 0 {
return errors.NewInvalidInputf(
errors.CodeInvalidInput,
"invalid start time %d: start must be greater than 0",
req.Start,
)
}
if req.End <= 0 {
return errors.NewInvalidInputf(
errors.CodeInvalidInput,
"invalid end time %d: end must be greater than 0",
req.End,
)
}
if req.Start >= req.End {
return errors.NewInvalidInputf(
errors.CodeInvalidInput,
"invalid time range: start (%d) must be less than end (%d)",
req.Start,
req.End,
)
}
if req.Limit < 1 || req.Limit > 5000 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "limit must be between 1 and 5000")
}
if req.Offset < 0 {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "offset cannot be negative")
}
if req.OrderBy != nil {
if !slices.Contains(NamespacesValidOrderByKeys, req.OrderBy.Key.Name) {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid order by key: %s", req.OrderBy.Key.Name)
}
if req.OrderBy.Direction != qbtypes.OrderDirectionAsc && req.OrderBy.Direction != qbtypes.OrderDirectionDesc {
return errors.NewInvalidInputf(errors.CodeInvalidInput, "invalid order by direction: %s", req.OrderBy.Direction)
}
}
return nil
}
// UnmarshalJSON validates input immediately after decoding.
func (req *PostableNamespaces) UnmarshalJSON(data []byte) error {
type raw PostableNamespaces
var decoded raw
if err := json.Unmarshal(data, &decoded); err != nil {
return err
}
*req = PostableNamespaces(decoded)
return req.Validate()
}

View File

@@ -0,0 +1,11 @@
package inframonitoringtypes
const (
NamespacesOrderByCPU = "cpu"
NamespacesOrderByMemory = "memory"
)
var NamespacesValidOrderByKeys = []string{
NamespacesOrderByCPU,
NamespacesOrderByMemory,
}

View File

@@ -0,0 +1,237 @@
package inframonitoringtypes
import (
"testing"
"github.com/SigNoz/signoz/pkg/errors"
qbtypes "github.com/SigNoz/signoz/pkg/types/querybuildertypes/querybuildertypesv5"
"github.com/SigNoz/signoz/pkg/types/telemetrytypes"
"github.com/SigNoz/signoz/pkg/valuer"
"github.com/stretchr/testify/require"
)
func TestPostableNamespaces_Validate(t *testing.T) {
tests := []struct {
name string
req *PostableNamespaces
wantErr bool
}{
{
name: "valid request",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
},
wantErr: false,
},
{
name: "nil request",
req: nil,
wantErr: true,
},
{
name: "start time zero",
req: &PostableNamespaces{
Start: 0,
End: 2000,
Limit: 100,
Offset: 0,
},
wantErr: true,
},
{
name: "start time negative",
req: &PostableNamespaces{
Start: -1000,
End: 2000,
Limit: 100,
Offset: 0,
},
wantErr: true,
},
{
name: "end time zero",
req: &PostableNamespaces{
Start: 1000,
End: 0,
Limit: 100,
Offset: 0,
},
wantErr: true,
},
{
name: "start time greater than end time",
req: &PostableNamespaces{
Start: 2000,
End: 1000,
Limit: 100,
Offset: 0,
},
wantErr: true,
},
{
name: "start time equal to end time",
req: &PostableNamespaces{
Start: 1000,
End: 1000,
Limit: 100,
Offset: 0,
},
wantErr: true,
},
{
name: "limit zero",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: 0,
Offset: 0,
},
wantErr: true,
},
{
name: "limit negative",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: -10,
Offset: 0,
},
wantErr: true,
},
{
name: "limit exceeds max",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: 5001,
Offset: 0,
},
wantErr: true,
},
{
name: "offset negative",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: 100,
Offset: -5,
},
wantErr: true,
},
{
name: "orderBy nil is valid",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
},
wantErr: false,
},
{
name: "orderBy with valid key cpu and direction asc",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: NamespacesOrderByCPU,
},
},
Direction: qbtypes.OrderDirectionAsc,
},
},
wantErr: false,
},
{
name: "orderBy with valid key memory and direction desc",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: NamespacesOrderByMemory,
},
},
Direction: qbtypes.OrderDirectionDesc,
},
},
wantErr: false,
},
{
name: "orderBy with pod_phase key is rejected",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "pod_phase",
},
},
Direction: qbtypes.OrderDirectionDesc,
},
},
wantErr: true,
},
{
name: "orderBy with invalid key",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: "unknown",
},
},
Direction: qbtypes.OrderDirectionDesc,
},
},
wantErr: true,
},
{
name: "orderBy with valid key but invalid direction",
req: &PostableNamespaces{
Start: 1000,
End: 2000,
Limit: 100,
Offset: 0,
OrderBy: &qbtypes.OrderBy{
Key: qbtypes.OrderByKey{
TelemetryFieldKey: telemetrytypes.TelemetryFieldKey{
Name: NamespacesOrderByMemory,
},
},
Direction: qbtypes.OrderDirection{String: valuer.NewString("invalid")},
},
},
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := tt.req.Validate()
if tt.wantErr {
require.Error(t, err)
require.True(t, errors.Ast(err, errors.TypeInvalidInput), "expected error to be of type InvalidInput")
} else {
require.NoError(t, err)
}
})
}
}

View File

@@ -0,0 +1,12 @@
package retentiontypes
import (
"context"
"github.com/SigNoz/signoz/pkg/valuer"
)
type Store interface {
// ListTTLSettingsByTableNameAndBeforeCreatedAt returns successful TTL settings before the given timestamp.
ListTTLSettingsByTableNameAndBeforeCreatedAt(ctx context.Context, orgID valuer.UUID, tableName string, beforeMs int64) ([]*TTLSetting, error)
}

View File

@@ -1,15 +1,145 @@
package retentiontypes
import (
"encoding/json"
"time"
"github.com/SigNoz/signoz/pkg/errors"
"github.com/SigNoz/signoz/pkg/types"
"github.com/uptrace/bun"
)
const secondsPerDay = 24 * 60 * 60
const (
DefaultLogsRetentionDays = 15
DefaultMetricsRetentionDays = 30
DefaultTracesRetentionDays = 15
)
const (
TraceTTL = "traces"
MetricsTTL = "metrics"
LogsTTL = "logs"
)
const (
TTLSettingStatusPending = "pending"
TTLSettingStatusFailed = "failed"
TTLSettingStatusSuccess = "success"
)
// RetentionPolicySegment is a half-open time range using one retention policy.
type RetentionPolicySegment struct {
StartMs int64
EndMs int64
Rules []CustomRetentionRule
DefaultDays int
}
// NewRetentionPolicySegment creates a retention policy segment for a half-open time range.
func NewRetentionPolicySegment(startMs int64, endMs int64, rules []CustomRetentionRule, defaultDays int) *RetentionPolicySegment {
return &RetentionPolicySegment{
StartMs: startMs,
EndMs: endMs,
Rules: rules,
DefaultDays: defaultDays,
}
}
// BuildRetentionPolicySegmentsFromRows converts successful TTL settings into retention policy segments.
func BuildRetentionPolicySegmentsFromRows(rows []*TTLSetting, fallbackDefaultDays int, startMs, endMs int64) ([]*RetentionPolicySegment, error) {
if startMs >= endMs {
return nil, nil
}
var activeAtStart *TTLSetting
inWindow := make([]*TTLSetting, 0, len(rows))
for _, row := range rows {
rowMs := row.CreatedAt.UnixMilli()
if rowMs <= startMs {
activeAtStart = row
continue
}
if rowMs >= endMs {
continue
}
inWindow = append(inWindow, row)
}
activeRules, activeDefault, err := parseTTLSetting(activeAtStart, fallbackDefaultDays)
if err != nil {
return nil, err
}
segments := make([]*RetentionPolicySegment, 0, len(inWindow)+1)
cursor := startMs
for _, row := range inWindow {
rowMs := row.CreatedAt.UnixMilli()
if rowMs <= cursor {
activeRules, activeDefault, err = parseTTLSetting(row, fallbackDefaultDays)
if err != nil {
return nil, err
}
continue
}
segments = append(segments, NewRetentionPolicySegment(cursor, rowMs, activeRules, activeDefault))
cursor = rowMs
activeRules, activeDefault, err = parseTTLSetting(row, fallbackDefaultDays)
if err != nil {
return nil, err
}
}
if cursor < endMs {
segments = append(segments, NewRetentionPolicySegment(cursor, endMs, activeRules, activeDefault))
}
return segments, nil
}
func parseTTLSetting(row *TTLSetting, fallbackDefaultDays int) ([]CustomRetentionRule, int, error) {
if row == nil {
return nil, fallbackDefaultDays, nil
}
defaultDays := row.TTL
if row.Condition == "" {
defaultDays = (row.TTL + secondsPerDay - 1) / secondsPerDay
}
if defaultDays <= 0 {
defaultDays = fallbackDefaultDays
}
if row.Condition == "" {
return nil, defaultDays, nil
}
var rules []CustomRetentionRule
if err := json.Unmarshal([]byte(row.Condition), &rules); err != nil {
return nil, 0, errors.Wrapf(err, errors.TypeInternal, errors.CodeInternal, "parse ttl_setting condition for row %q", row.ID.StringValue())
}
return rules, defaultDays, nil
}
// CustomRetentionRule is one custom retention rule as stored in ttl_setting.condition.
// Rules are evaluated in declaration order; the first matching rule wins.
type CustomRetentionRule struct {
Filters []FilterCondition `json:"conditions"`
TTLDays int `json:"ttlDays"`
}
// FilterCondition is one label-key, allowed-values condition inside a retention rule.
type FilterCondition struct {
Key string `json:"key"`
Values []string `json:"values"`
}
type TTLSetting struct {
bun.BaseModel `bun:"table:ttl_setting"`
types.Identifiable
types.TimeAuditable
types.UserAuditable
TransactionID string `bun:"transaction_id,type:text,notnull"`
TableName string `bun:"table_name,type:text,notnull"`
TTL int `bun:"ttl,notnull,default:0"`
@@ -18,3 +148,73 @@ type TTLSetting struct {
OrgID string `json:"-" bun:"org_id,notnull"`
Condition string `bun:"condition,type:text"`
}
type TTLParams struct {
Type string
ColdStorageVolume string
ToColdStorageDuration int64
DelDuration int64
}
type CustomRetentionTTLParams struct {
Type string `json:"type"`
DefaultTTLDays int `json:"defaultTTLDays"`
TTLConditions []CustomRetentionRule `json:"ttlConditions"`
ColdStorageVolume string `json:"coldStorageVolume,omitempty"`
ToColdStorageDurationDays int64 `json:"coldStorageDurationDays,omitempty"`
}
type GetTTLParams struct {
Type string
}
type GetCustomRetentionTTLResponse struct {
Version string `json:"version"`
Status string `json:"status"`
ExpectedLogsTime int `json:"expected_logs_ttl_duration_hrs,omitempty"`
ExpectedLogsMoveTime int `json:"expected_logs_move_ttl_duration_hrs,omitempty"`
DefaultTTLDays int `json:"default_ttl_days,omitempty"`
TTLConditions []CustomRetentionRule `json:"ttl_conditions,omitempty"`
ColdStorageVolume string `json:"cold_storage_volume,omitempty"`
ColdStorageTTLDays int `json:"cold_storage_ttl_days,omitempty"`
}
type CustomRetentionTTLResponse struct {
Message string `json:"message"`
}
type TTLStatusItem struct {
Id int `json:"id" db:"id"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at"`
CreatedAt time.Time `json:"created_at" db:"created_at"`
TableName string `json:"table_name" db:"table_name"`
TTL int `json:"ttl" db:"ttl"`
Status string `json:"status" db:"status"`
ColdStorageTtl int `json:"cold_storage_ttl" db:"cold_storage_ttl"`
}
type SetTTLResponseItem struct {
Message string `json:"message"`
}
type DBResponseTTL struct {
EngineFull string `ch:"engine_full"`
}
type GetTTLResponseItem struct {
MetricsTime int `json:"metrics_ttl_duration_hrs,omitempty"`
MetricsMoveTime int `json:"metrics_move_ttl_duration_hrs,omitempty"`
TracesTime int `json:"traces_ttl_duration_hrs,omitempty"`
TracesMoveTime int `json:"traces_move_ttl_duration_hrs,omitempty"`
LogsTime int `json:"logs_ttl_duration_hrs,omitempty"`
LogsMoveTime int `json:"logs_move_ttl_duration_hrs,omitempty"`
ExpectedMetricsTime int `json:"expected_metrics_ttl_duration_hrs,omitempty"`
ExpectedMetricsMoveTime int `json:"expected_metrics_move_ttl_duration_hrs,omitempty"`
ExpectedTracesTime int `json:"expected_traces_ttl_duration_hrs,omitempty"`
ExpectedTracesMoveTime int `json:"expected_traces_move_ttl_duration_hrs,omitempty"`
ExpectedLogsTime int `json:"expected_logs_ttl_duration_hrs,omitempty"`
ExpectedLogsMoveTime int `json:"expected_logs_move_ttl_duration_hrs,omitempty"`
Status string `json:"status"`
}

View File

@@ -0,0 +1,83 @@
package retentiontypes
import (
"encoding/json"
"testing"
"time"
"github.com/SigNoz/signoz/pkg/types"
"github.com/stretchr/testify/require"
)
func TestBuildRetentionPolicySegmentsFromRows(t *testing.T) {
start := time.Date(2026, 5, 4, 0, 0, 0, 0, time.UTC)
end := start.AddDate(0, 0, 1)
ruleA := CustomRetentionRule{
Filters: []FilterCondition{{Key: "service.name", Values: []string{"api"}}},
TTLDays: 7,
}
ruleB := CustomRetentionRule{
Filters: []FilterCondition{{Key: "env", Values: []string{"prod"}}},
TTLDays: 15,
}
t.Run("row before window is active at start", func(t *testing.T) {
segments, err := BuildRetentionPolicySegmentsFromRows(
[]*TTLSetting{
ttlSetting(t, start.Add(-time.Hour), 45, []CustomRetentionRule{ruleA}),
},
30,
start.UnixMilli(),
end.UnixMilli(),
)
require.NoError(t, err)
require.Equal(t, []*RetentionPolicySegment{
NewRetentionPolicySegment(start.UnixMilli(), end.UnixMilli(), []CustomRetentionRule{ruleA}, 45),
}, segments)
})
t.Run("row inside window splits segments", func(t *testing.T) {
firstChange := start.Add(6 * time.Hour)
secondChange := start.Add(18 * time.Hour)
segments, err := BuildRetentionPolicySegmentsFromRows(
[]*TTLSetting{
ttlSetting(t, firstChange, 21, []CustomRetentionRule{ruleA}),
ttlSetting(t, secondChange, 14, []CustomRetentionRule{ruleB}),
},
30,
start.UnixMilli(),
end.UnixMilli(),
)
require.NoError(t, err)
require.Equal(t, []*RetentionPolicySegment{
NewRetentionPolicySegment(start.UnixMilli(), firstChange.UnixMilli(), nil, 30),
NewRetentionPolicySegment(firstChange.UnixMilli(), secondChange.UnixMilli(), []CustomRetentionRule{ruleA}, 21),
NewRetentionPolicySegment(secondChange.UnixMilli(), end.UnixMilli(), []CustomRetentionRule{ruleB}, 14),
}, segments)
})
t.Run("no rows uses fallback", func(t *testing.T) {
segments, err := BuildRetentionPolicySegmentsFromRows(nil, 30, start.UnixMilli(), end.UnixMilli())
require.NoError(t, err)
require.Equal(t, []*RetentionPolicySegment{
NewRetentionPolicySegment(start.UnixMilli(), end.UnixMilli(), nil, 30),
}, segments)
})
}
func ttlSetting(t *testing.T, createdAt time.Time, ttlDays int, rules []CustomRetentionRule) *TTLSetting {
t.Helper()
condition, err := json.Marshal(rules)
require.NoError(t, err)
return &TTLSetting{
TimeAuditable: types.TimeAuditable{
CreatedAt: createdAt,
},
TTL: ttlDays,
Condition: string(condition),
}
}

View File

@@ -245,7 +245,6 @@ type Store interface {
// Service Account Role
CreateServiceAccountRole(context.Context, *ServiceAccountRole) error
DeleteServiceAccountRoles(context.Context, valuer.UUID) error
DeleteServiceAccountRole(context.Context, valuer.UUID, valuer.UUID) error
// Service Account Factor API Key

View File

@@ -0,0 +1,20 @@
package zeustypes
import "go.opentelemetry.io/otel/attribute"
var (
// Identifies the organization.
OrganizationID = attribute.Key("signoz.organization.id")
// Identifies the retention bucket a meter belongs to.
RetentionDuration = attribute.Key("signoz.retention.duration")
)
func NewDimensions(kvs ...attribute.KeyValue) map[string]string {
dimensions := map[string]string{}
for _, kv := range kvs {
dimensions[string(kv.Key)] = kv.Value.AsString()
}
return dimensions
}

View File

@@ -0,0 +1,29 @@
package zeustypes
import (
"encoding/json"
)
type GettableDeployment struct {
ID string `json:"id"`
Name string `json:"name"`
Cluster struct {
ID string `json:"id"`
Name string `json:"name"`
Region struct {
ID string `json:"id"`
Name string `json:"name"`
DNS string `json:"dns"`
} `json:"region"`
} `json:"cluster"`
}
func NewGettableDeployment(data []byte) (*GettableDeployment, error) {
deployment := new(GettableDeployment)
err := json.Unmarshal(data, deployment)
if err != nil {
return nil, err
}
return deployment, nil
}

View File

@@ -0,0 +1,46 @@
package zeustypes
import (
"net/url"
"github.com/tidwall/gjson"
)
type Host struct {
Name string `json:"name" required:"true"`
IsDefault bool `json:"is_default" required:"true"`
URL string `json:"url" required:"true"`
}
type GettableHost struct {
Name string `json:"name" required:"true"`
State string `json:"state" required:"true"`
Tier string `json:"tier" required:"true"`
Hosts []Host `json:"hosts" required:"true"`
}
type PostableHost struct {
Name string `json:"name" required:"true"`
}
func NewGettableHost(data []byte) *GettableHost {
parsed := gjson.ParseBytes(data)
dns := parsed.Get("cluster.region.dns").String()
hostResults := parsed.Get("hosts").Array()
hosts := make([]Host, len(hostResults))
for i, h := range hostResults {
name := h.Get("name").String()
hosts[i].Name = name
hosts[i].IsDefault = h.Get("is_default").Bool()
hosts[i].URL = (&url.URL{Scheme: "https", Host: name + "." + dns}).String()
}
return &GettableHost{
Name: parsed.Get("name").String(),
State: parsed.Get("state").String(),
Tier: parsed.Get("tier").String(),
Hosts: hosts,
}
}

View File

@@ -0,0 +1,54 @@
package zeustypes
import "time"
type MeterCheckpoint struct {
Name string
StartDate time.Time
}
type Meter struct {
// MeterName is the fully-qualified meter identifier.
MeterName MeterName `json:"name"`
// Value is the aggregated integer scalar for this meter over the reporting window.
Value int64 `json:"value"`
// Unit is the metric unit for this meter.
Unit MeterUnit `json:"unit"`
// Aggregation names the aggregation applied to produce Value.
Aggregation MeterAggregation `json:"aggregation"`
// StartUnixMilli is the inclusive window start in epoch milliseconds.
StartUnixMilli int64 `json:"start_unix_milli"`
// EndUnixMilli is the exclusive window end in epoch milliseconds.
EndUnixMilli int64 `json:"end_unix_milli"`
// IsCompleted is false for the current day's partial value.
IsCompleted bool `json:"is_completed"`
// Dimensions is the per-meter label set.
Dimensions map[string]string `json:"dimensions"`
}
func NewMeter(
name MeterName,
value int64,
unit MeterUnit,
aggregation MeterAggregation,
window MeterWindow,
dimensions map[string]string,
) Meter {
return Meter{
MeterName: name,
Value: value,
Unit: unit,
Aggregation: aggregation,
StartUnixMilli: window.StartUnixMilli,
EndUnixMilli: window.EndUnixMilli,
IsCompleted: window.IsCompleted,
Dimensions: dimensions,
}
}

View File

@@ -0,0 +1,12 @@
package zeustypes
import "github.com/SigNoz/signoz/pkg/valuer"
type MeterAggregation struct {
valuer.String
}
var (
MeterAggregationSum = MeterAggregation{valuer.NewString("sum")}
MeterAggregationMax = MeterAggregation{valuer.NewString("max")}
)

View File

@@ -0,0 +1,66 @@
package zeustypes
import (
"encoding/json"
"regexp"
"github.com/SigNoz/signoz/pkg/errors"
)
var (
MeterSpanSize = MustNewMeterName("signoz.meter.span.size")
MeterLogSize = MustNewMeterName("signoz.meter.log.size")
MeterDatapointCount = MustNewMeterName("signoz.meter.metric.datapoint.count")
MeterPlatformActive = MustNewMeterName("signoz.meter.platform.active")
)
var meterNameRegex = regexp.MustCompile(`^[a-z][a-z0-9_.]+$`)
// MeterName is a validated dotted Zeus meter name.
type MeterName struct {
s string
}
func NewMeterName(s string) (MeterName, error) {
if !meterNameRegex.MatchString(s) {
return MeterName{}, errors.Newf(errors.TypeInvalidInput, errors.CodeInvalidInput, "invalid meter name: %s", s)
}
return MeterName{s: s}, nil
}
func MustNewMeterName(s string) MeterName {
name, err := NewMeterName(s)
if err != nil {
panic(err)
}
return name
}
func (n MeterName) String() string {
return n.s
}
func (n MeterName) IsZero() bool {
return n.s == ""
}
func (n MeterName) MarshalJSON() ([]byte, error) {
return json.Marshal(n.s)
}
func (n *MeterName) UnmarshalJSON(data []byte) error {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
name, err := NewMeterName(s)
if err != nil {
return err
}
*n = name
return nil
}

View File

@@ -0,0 +1,30 @@
package zeustypes
import (
"testing"
"time"
"github.com/stretchr/testify/require"
)
func TestNewMeterWindow(t *testing.T) {
start := time.Date(2026, 5, 4, 0, 0, 0, 0, time.UTC)
window, err := NewMeterWindow(start.UnixMilli(), start.AddDate(0, 0, 1).UnixMilli(), true)
require.NoError(t, err)
require.Equal(t, start.UnixMilli(), window.StartUnixMilli)
require.Equal(t, start.AddDate(0, 0, 1).UnixMilli(), window.EndUnixMilli)
require.True(t, window.IsCompleted)
_, err = NewMeterWindow(0, start.UnixMilli(), true)
require.Error(t, err)
_, err = NewMeterWindow(start.UnixMilli(), start.UnixMilli(), false)
require.Error(t, err)
}
func TestMustNewMeterWindowPanicsForInvalidWindow(t *testing.T) {
require.Panics(t, func() {
MustNewMeterWindow(0, 0, true)
})
}

Some files were not shown because too many files have changed in this diff Show More