<!--A few plain bullets saying what changed and why, for a reviewer
skimming it - not a wall of text, not a restatement of the diff, not
generated boilerplate.-->
#### Description
The agent only saw the current list of enabled regions, so it couldn’t
tell which regions had been removed. To find stacks to clean up, it
checked unrelated AWS regions, causing unnecessary calls and permission
errors. Sync state keeps track of regions sent to the agent and pending
removals until the agent acknowledges cleanup.
Please check
[comment](https://github.com/SigNoz/keystone-pod/issues/101#issuecomment-5832865898)
for approach
<!--Reference issues using `Closes #issue-number` to enable automatic
closure on merge. -->
#### Issues closed by this PR
Contributes to https://github.com/SigNoz/keystone-pod/issues/101
<!--Anything reviewers should keep in mind while reviewing -->
#### Additional Information
This PR should be merged before changes for cloud-integration repo.
<!--Please delete paragraphs that you did not use before submitting.-->
<!--A few plain bullets saying what changed and why, for a reviewer
skimming it - not a wall of text, not a restatement of the diff, not
generated boilerplate.-->
#### Description
- Adding GCP integration MySQL service
- Related fix: adding formula to convert CPU utilization fraction into
percentage for Postgres dashboard
<!--Reference issues using `Closes #issue-number` to enable automatic
closure on merge. -->
#### Issues closed by this PR
https://github.com/SigNoz/platform-pod/issues/2942
* feat: adding gcp memorystore redis service
* refactor: updating dashboard title
* refactor: extending width of uptime gauge panel
* refactor: updating cpu utilization panel
* refactor: updating dashboard panel to use rate function instead of hack
* feat: adding compute engine service
* refactor: updating dashboard panels to use rate aggregation
* fix: correct typo and unit in compute engine dashboard
* refactor: migrating dashboard to v6
* chore: add migration script from current to perses dashboard
* fix: move WrapInV5Envelope to types package
* fix: generate internal name from title
* fix: match with lower case signal for variables
* fix: sanitize tags instead of throwing error
* fix: extract row and widget positions to build expanded sections
* chore: add a catch all panic check to log migration error
* chore: catch typecast errors
* fix: remove nil nil return
* test: fix ut to check for v2 internal name
* chore: remove spec md file
* chore: separate error type for migration
* feat: note down all errors in migration
* fix: allow zero value for threshold values
* fix: ignore react placeholders in layout
* fix: note error in query parsing
* fix: read threshold value properly during migration
* fix: adjust to schema changes in variables
* test: fix unit tests
* fix: catch error from decodeTelemetryFields
* fix: make threshold value non-nullable
* fix: handle list of widget IDs in panel maps
* fix: add better error note down for failed conversions
* fix: normalize having array from v4 schema
* fix: remove metric aggregation from non metric queries (malformed json)
* fix: normalize telemetry field keys in query
* chore: add temporary doc for other malformation handlers needed
* fix: add more malformed query handling
* fix: add more malformation handling
* chore: reuse transition package (adds dependency, need to see what to do)
* fix: fix go lint issue
* revert: remove md file
* fix: make none the default fill mode during migration
* fix: add nil check on limit value when normalizing page size
* fix: validate dashboard spec after migration
* fix: remove unused method
* fix: match shape safe version to existing version
* chore: improve variable names
* chore: better comments and names in layout migrator
* chore: add explanation comment for continuing on empty id widget
* fix: ditch dynamic vars with no attr, rearrange overlapping layouts
* fix: add default aggregation count() if none is present in logs/traces
* fix: add empty list tolerance to v1 variables
* fix: add fixes based on backup migration testing
* chore: skip over unreferenced widgets
* chore: remove widgets that have a single query that has no metric name
* chore: assign missing formula names like ui does
* chore: assign query names to queries if missing
* chore: drop legacy filter fields that ui also ignores
* chore: make migration drop widgets that have no aggregations for metrics
* chore: normalize group by
* chore: remove expressions from metric aggregations
* chore: add more error notes
* chore: replace em dash with hyphen in widget id
* chore: add more panel id sanitization
* chore: replace faulty order by keys with aggregation expression
* chore: fill valid span gaps
* chore: handle misformed threshold operators
* chore: remove repeat occurences of widgets in layouts
* chore: remove limits that cross max limit
* chore: restore function args malformed by v4->v5 migration
* chore: remove invalid functions
* chore: skip variables in wrong type
* chore: add wdiget id type check in layout migration
* chore: add a.count to malformed order by keys list
* chore: remove bad aggregations that accompany valid ones
* chore: drop trace operators with no expression
* chore: drop invalid formulas
* chore: skip unknown panel kind
* chore: drop widgets with >1 and all aggregations as malformed
* chore: skip empty type variables
* chore: pass threshold format raw without typecasting
* chore: add handling for trace operators around wrapinv5env
* chore: handle no string value for variable sort
* test: fix span gaps test
* chore: undo schema change in threshold value
* chore: final set of error fixes/hacks
* fix: fix build errors post merge conflicts
* test: remove test for floating pt value
* test: use appendf instead of sprintf
* chore: migrate integration dashboards to v2
* feat: add scaffolding to migrate dashboards
* fix: fill in ref path when building grid layout
* feat: handle v1 payloads in v2 create api
* fix: return empty objects instead of null if no variables/panels/layouts are there
* test: fix unit test for empty layouts
* fix: add empty links array in migration
* chore: rerun migration after links and signal change
* test: add unit test to show v5 takes precedence over v4
* fix: change EXISTS to exists so that migrateCommon handles it properly
* test: add unit test for count with attributes
* fix: run order by handling after default aggregation handling
* fix: restore formula fields that wrapinv5envelope removes
* fix: keep thresholds that do not have an optional label
* fix: force allowAll to be false if allowMultiple is false
* chore: rerun migration
* fix: migrate name column even if data column migration fails
* fix: use slog for logging
* refactor(dashboard): re-host ConvertAllV1ToV2 off the Module interface
Mirror base's removal of ConvertAllV1ToV2 from the shared files so future
base→branch merges stay conflict-free, and re-host the capability in dedicated
branch-only files (V1ToV2Migrator interface + module method + result types).
* refactor(dashboard): drop ConvertAllV1ToV2 module scaffolding from base
Remove the temporary v1→v2 bulk-conversion module method (ConvertAllV1ToV2 +
migrateOneV1ToV2), its result types (V1ToV2MigrationResult / V1ToV2MigrationItem),
the Module interface entry, and the EE delegation. This scaffolding lives on the
migration feature branches, not the base branch that merges to main.
V1Name and store.UpdateName are kept (small helpers; UpdateName is on the shared
Store interface).
* fix: add image validation and base64 to asset conversion
* fix: remove unneeded interface
* chore: rerun migration
* fix: make ReencodeInlineSVGImage so that migration script can call it
* chore: remove temp script
* fix: remove local laptop default value
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* feat: adding gcp memorystore redis service
* refactor: updating dashboard title
* refactor: extending width of uptime gauge panel
* refactor: updating cpu utilization panel
* refactor: updating dashboard panel to use rate function instead of hack
* chore: mark required and nullable in json tag, renamed methods, and added more functionality
* fix: unit test
* fix: cast clickhouse exceptions
* fix: go mod tidy
* fix: telemetrystore now returns explicit base errors
* fix: typo
* fix: added all changes
* fix: added nil check
* fix: update test files
* fix: addressed comments
* fix: change errors and suggestions to be non-nullable
* feat: adding support for Azure SQL DB
* refactor: adding missing metrics in data collected and panels in dash
* feat: adding support for Azure SQL DB Managed Instances
* feat: generating openapi spec
* feat: adding nosql and cache services in azure integration
* feat: extend error responses with new error struct
* fix: enriched error for dashboard api
* fix: merge issues
* fix: reverted dashboards changes and add for cloud integrations
* fix: delete file
* fix: add back file
* fix: added a helper
* fix: removed invlaid referencess
* fix: generate openapi
* fix: keeping additional along with suggestion
* Revert "fix: keeping additional along with suggestion"
This reverts commit be30e2ffd2.
* fix: added suggestions per additonal error
* fix: generate openapi
* fix: remove valid references
* fix: removeg valid references for select and group by and only did you mean is kept
* fix: unit test
* fix: use binding for deconding for both ee and community
* fix: trim down suggestions methods
* fix: added renamed methods and moved stuff around
* fix: typo
* fix: removed json decoder
* fix: added empty check
* fix: retain addtional
* fix: reverted re-structing of file
* feat: adding support for Azure AKS
* refactor: updating metric names in integration config
* chore: generating OpenAPI spec
* chore: adding note in overview
* refactor: updating spatial aggregation and other names
* refactor: updating reduceTo to empty value
* refactor: updating title
* chore: added migration setup
* feat(sqlmigration): add integration_dashboards table (migration 079)
Adds the `integration_dashboards` relations table that stores the
integration-specific identity for dashboards provisioned from cloud
or builtin integrations. Columns: id, org_id, dashboard_id, provider,
slug, created_at, updated_at. Includes a unique index on dashboard_id.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(sqlmigration): backfill cloud integration dashboards to DB (migration 080)
One-time idempotent migration that provisions dashboard rows for all
orgs with existing cloud integration services where metrics are enabled.
Each dashboard is inserted into the `dashboard` table with
source="integration" and locked=true, and a companion row is added to
`integration_dashboards` with provider="cloud_integrations" and
slug="{provider}-{service}-{dashboard}" (e.g. aws-alb-overview).
Idempotency is enforced by checking (org_id, provider, slug) on
integration_dashboards before each insert.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(sqlmigration): clean up stale 079 artifacts, add 079 schema migration
Remove the pre-rename 079_migrate_cloud_integration_dashboards.go and
079_cloud_integration_dashboards/ directory that were left behind when
the backfill migration was renumbered to 080. Add the missing
079_add_integration_dashboards.go (schema-only migration creating the
integration_dashboards table) which provider.go already references.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: adding comment for fk
* refactor: renaming table name
* refactor: rename and restructure cloud integration dashboard migration types
* chore: file rename
* refactor: dashboard creation and listing flow change
* refactor: removing loose strings
* refactor: adding DeleteBySource on dashboard module
* refactor: review changes and update service flow change
* refactor: simplify comments
* ci: lint staticcheck fix
* refactor: renaming migration and adding integration tests
* ci: py fmt lint fixes
* feat: adding ListSharedServices store method
* ci: golangci-lint fix
* feat(integrations): persist installed integration dashboards in DB
Provisions dashboard DB rows when an integration is installed and
deprovisions them on uninstall. Adds a backfill migration (087) for
users with already-installed integrations. Removes the on-the-fly
filesystem serving path from http_handler in favor of the standard
dashboard module.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor: changing dashboard ID and other cleanup
* chore: update code structure for better readability and maintainability
* refactor: removing deprecated cloud integrations and merging
integration types
* refactor: renaming migration files and removing deprecated tests
* refactor: using BunDBCtx method instead
* ci: fix py fmt lint
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: added migration setup
* feat(sqlmigration): add integration_dashboards table (migration 079)
Adds the `integration_dashboards` relations table that stores the
integration-specific identity for dashboards provisioned from cloud
or builtin integrations. Columns: id, org_id, dashboard_id, provider,
slug, created_at, updated_at. Includes a unique index on dashboard_id.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(sqlmigration): backfill cloud integration dashboards to DB (migration 080)
One-time idempotent migration that provisions dashboard rows for all
orgs with existing cloud integration services where metrics are enabled.
Each dashboard is inserted into the `dashboard` table with
source="integration" and locked=true, and a companion row is added to
`integration_dashboards` with provider="cloud_integrations" and
slug="{provider}-{service}-{dashboard}" (e.g. aws-alb-overview).
Idempotency is enforced by checking (org_id, provider, slug) on
integration_dashboards before each insert.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(sqlmigration): clean up stale 079 artifacts, add 079 schema migration
Remove the pre-rename 079_migrate_cloud_integration_dashboards.go and
079_cloud_integration_dashboards/ directory that were left behind when
the backfill migration was renumbered to 080. Add the missing
079_add_integration_dashboards.go (schema-only migration creating the
integration_dashboards table) which provider.go already references.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: adding comment for fk
* refactor: renaming table name
* refactor: rename and restructure cloud integration dashboard migration types
* chore: file rename
* refactor: dashboard creation and listing flow change
* refactor: removing loose strings
* refactor: adding DeleteBySource on dashboard module
* refactor: review changes and update service flow change
* refactor: simplify comments
* ci: lint staticcheck fix
* refactor: renaming migration and adding integration tests
* ci: py fmt lint fixes
* feat: adding ListSharedServices store method
* ci: golangci-lint fix
* refactor: code cleanup
* chore: revert changed due to js lint
* refactor: test assertion changes
* refactor: using bindparam for sql generation
* chore: migrate integration dashboards json to v5 (#11419)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* feat: adding cloud integration type for refactor
* refactor: store interfaces to use local types and error
* feat: adding sql store implementation
* refactor: removing interface check
* feat: adding updated types for cloud integration
* refactor: using struct for map
* refactor: update cloud integration types and module interface
* fix: correct GetService signature and remove shadowed Data field
* feat: implement cloud integration store
* refactor: adding comments and removed wrong code
* refactor: streamlining types
* refactor: add comments for backward compatibility in PostableAgentCheckInRequest
* refactor: update Dashboard struct comments and remove unused fields
* refactor: split upsert store method
* feat: adding integration test
* refactor: clean up types
* refactor: renaming service type to service id
* refactor: using serviceID type
* feat: adding method for service id creation
* refactor: updating store methods
* refactor: clean up
* refactor: clean up
* refactor: review comments
* refactor: clean up
* feat: adding handlers
* fix: lint and ci issues
* fix: lint issues
* fix: update error code for service not found
* feat: adding handler skeleton
* chore: removing todo comment
* feat: adding frontend openapi schema
* refactor: making review changes
* feat: regenerating openapi specs
* feat: adding cloud integration type for refactor
* refactor: store interfaces to use local types and error
* feat: adding sql store implementation
* refactor: removing interface check
* feat: adding updated types for cloud integration
* refactor: using struct for map
* refactor: update cloud integration types and module interface
* fix: correct GetService signature and remove shadowed Data field
* feat: implement cloud integration store
* refactor: adding comments and removed wrong code
* refactor: streamlining types
* refactor: add comments for backward compatibility in PostableAgentCheckInRequest
* refactor: update Dashboard struct comments and remove unused fields
* refactor: split upsert store method
* feat: adding integration test
* refactor: clean up types
* refactor: renaming service type to service id
* refactor: using serviceID type
* feat: adding method for service id creation
* refactor: updating store methods
* refactor: clean up
* refactor: clean up
* refactor: review comments
* refactor: clean up
* fix: update error code for service not found
* refactor: returning error only for create methods
* refactor: method chaining formatting
* feat: adding cloud integration type for refactor
* refactor: store interfaces to use local types and error
* feat: adding updated types for cloud integration
* refactor: using struct for map
* refactor: update cloud integration types and module interface
* fix: correct GetService signature and remove shadowed Data field
* refactor: adding comments and removed wrong code
* refactor: streamlining types
* refactor: add comments for backward compatibility in PostableAgentCheckInRequest
* refactor: update Dashboard struct comments and remove unused fields
* refactor: clean up types
* refactor: renaming service type to service id
* refactor: using serviceID type
* feat: adding method for service id creation
* refactor: updating store methods
* refactor: clean up
* refactor: review comments
