feat: span mapper test endpoint

deploy/MIGRATION.md

@@ -1,119 +0,0 @@
-# Migrating from Docker Compose to Foundry
-
-This guide walks you through migrating an existing SigNoz deployment running via
-Docker Compose to [Foundry](https://signoz.io/docs/install/docker/).
-
-## Overview
-SigNoz has developed a CLI to make installation and deployment much easier. Foundry is the evolution of how SigNoz should be installed and managed going forward.
-
-The install script is now deprecated and will no longer receive updates.
-
-To stay up to date on new installation platforms and patterns, please refer to [Foundry](https://github.com/SigNoz/foundry).
-
-Two `foundryctl` commands are used throughout this guide:
-- **`forge`** — generates deployment manifests from your `casting.yaml`. It does not touch running containers, so it is safe to re-run while you iterate.
-- **`cast`** — applies the generated manifests: it creates and starts the containers (and pulls new images).
-
-## Prerequisites
-- [ ] Install Foundry - `curl -fsSL https://signoz.io/foundry.sh | bash`
-
-## Migration Steps
-> [!WARNING]
-> **Before proceeding, back up both:**
-> - **Your docker volumes** — these hold your data.
-> - **Your existing `docker-compose.yaml` (and any config it references)** — keep a copy somewhere safe. The compose manifests are no longer distributed by SigNoz, so this backup is your only way to roll back to your previous setup.
-
-1. Make a note of the volume names used by your existing deployment for the following components:
-- ClickHouse
-- SigNoz
-- ZooKeeper
-
-> If you used the docker compose file we provided, the volumes will be `signoz-clickhouse`, `signoz-sqlite`, and `signoz-zookeeper-1`.
-
-2. Generate your `casting.yaml`. Based on internal testing, the following casting should generate the manifests that mimic the legacy docker compose setup (compare against your backed-up `docker-compose.yaml`). Once created, run `foundryctl forge -f casting.yaml`.
-
-> [!NOTE]
-> The casting contains `patches` and `config` overrides to ensure that the newly generated manifests use the existing volumes and configurations.
-
-> [!WARNING]
-> If your deployment had more than 1 shard or replica, you will need to adjust your manifest volumes accordingly. Additionally, if you had specific container images, you need to include them in your casting.
-
-> [!IMPORTANT]
-> The `replica` and `shard` macros below are placeholders. Replace them with the values from your existing ClickHouse configuration (check the `macros` section of your current ClickHouse config, e.g. `config.xml`/`metrika.xml`), otherwise the generated manifests will not match your existing data.
-
-```yaml
-apiVersion: v1alpha1
-kind: Installation
-metadata:
-  name: signoz
-spec:
-  deployment:
-    flavor: compose
-    mode: docker
-  metastore:
-    kind: sqlite
-  telemetrykeeper:
-    kind: zookeeper
-  telemetrystore:
-    spec:
-      config:
-        data:
-          config-0-0.yaml: |
-            macros:
-              replica: "example01-01-1" # replace with your existing ClickHouse replica macro (see legacy configuration files for reference)
-              shard: "01"               # replace with your existing ClickHouse shard macro (see legacy configuration files for reference)
-  patches:
-    - target: "deployment/compose.yaml"
-      operations:
-        - op: replace
-          path: /volumes/dev-telemetrykeeper-0-data/name
-          value: signoz-zookeeper-1
-        - op: replace
-          path: /volumes/dev-telemetrystore-0-0-data/name
-          value: signoz-clickhouse
-        - op: replace
-          path: /volumes/dev-metastore-sqlite-0-data/name
-          value: signoz-sqlite
-        - op: add
-          path: /services/dev-telemetrykeeper-zookeeper-0/user
-          value: root
-```
-
-> [!NOTE]
-> The `user: root` patch on the ZooKeeper service is required so the container can read/write the data in your reused ZooKeeper volume, which was created with `root`-owned files by the legacy compose setup. Without it, ZooKeeper may fail to start with permission errors.
-
-3. Validate the manifests in `pours/deployment`. Pay special attention to `compose.yaml` — it should mimic the legacy manifest and the configuration files needed for `clickhouse`. **Do note that these are now in YAML instead of XML.**
-
-   If you had custom settings for features like SMTP or ingestion processors/receivers, you will need to include those in your casting file.
-
-4. Execute a `docker compose down`. **Do not** include parameters such as `--volumes` (or `-v`), as it will wipe the volumes we need to maintain and reuse to avoid data loss. **NOTE: this will generate downtime so please plan accordingly**.
-
-5. Validate the SigNoz containers are down with `docker ps -a`. Multiple containers cannot bind the same volume.
-
-6. Run `foundryctl cast -f casting.yaml`. This will recreate the containers based on the spec. This process will download new container images.
-
-> [!NOTE]
-> When `cast` is run, the migration container will execute its migrations.
-
-## Verifying the Migration
-- SigNoz containers will be up and running.
-- Log in to the SigNoz UI and verify that data is present.
-- Validate that your data ingestion is receiving data.
-- Review the logs from both ClickHouse and ZooKeeper; no errors should be present.
-
-## Rolling Back
-Because step 4 brought the legacy stack down *without* `-v`, your original volumes
-are untouched and still hold your data. To roll back:
-
-- Stop and remove the containers created by Foundry (`docker compose down`, again without `-v`).
-- Confirm the containers are gone with `docker ps -a` so nothing else is bound to the volumes.
-- Reapply your original docker compose file (`docker compose up -d`). It will reattach to the
-  existing volumes and restore your prior state.
-
-## Troubleshooting
-- Please reach out to our community on [Slack](https://signoz.io/slack).
-
-## References
-- [SigNoz Docker installation docs](https://signoz.io/docs/install/docker/)
-- [SigNoz documentation](https://signoz.io/docs)
-- [Foundry](https://github.com/SigNoz/foundry)

deploy/README.md

@@ -3,18 +3,77 @@
 Check that you have cloned [signoz/signoz](https://github.com/signoz/signoz)
 and currently are in `signoz/deploy` folder.
 
-## Installation
+## Docker
 
-> **Note:** The `install.sh` script and the `docker-compose` manifests have been deprecated.
+If you don't have docker set up, please follow [this guide](https://docs.docker.com/engine/install/)
+to set up docker before proceeding with the next steps.
 
-SigNoz now installs and runs through [Foundry](https://signoz.io/docs/install/docker/).
+### Using Install Script
 
-> **Already running SigNoz via Docker Compose?** See the [Migration Guide](./MIGRATION.md) to transition your existing deployment to Foundry.
+Now run the following command to install:
 
-Please follow the latest installation instructions at [signoz.io/docs/install/docker](https://signoz.io/docs/install/docker/).
-Foundry has support for **different platforms and architectures**, please review the project documentation for more details.
+```sh
+./install.sh
+```
 
-> You will need Docker installed first. If you don't have it set up, follow [this guide](https://docs.docker.com/engine/install/).
+### Using Docker Compose
+
+If you don't have docker compose set up, please follow [this guide](https://docs.docker.com/compose/install/)
+to set up docker compose before proceeding with the next steps.
+
+```sh
+cd deploy/docker
+docker compose up -d
+```
+
+Open http://localhost:8080 in your favourite browser.
+
+To start collecting logs and metrics from your infrastructure, run the following command:
+
+```sh
+cd generator/infra
+docker compose up -d
+```
+
+To start generating sample traces, run the following command:
+
+```sh
+cd generator/hotrod
+docker compose up -d
+```
+
+In a couple of minutes, you should see the data generated from hotrod in SigNoz UI.
+
+For more details, please refer to the [SigNoz documentation](https://signoz.io/docs/install/docker/).
+
+## Docker Swarm
+
+To install SigNoz using Docker Swarm, run the following command:
+
+```sh
+cd deploy/docker-swarm
+docker stack deploy -c docker-compose.yaml signoz
+```
+
+Open http://localhost:8080 in your favourite browser.
+
+To start collecting logs and metrics from your infrastructure, run the following command:
+
+```sh
+cd generator/infra
+docker stack deploy -c docker-compose.yaml infra
+```
+
+To start generating sample traces, run the following command:
+
+```sh
+cd generator/hotrod
+docker stack deploy -c docker-compose.yaml hotrod
+```
+
+In a couple of minutes, you should see the data generated from hotrod in SigNoz UI.
+
+For more details, please refer to the [SigNoz documentation](https://signoz.io/docs/install/docker-swarm/).
 
 ## Uninstall/Troubleshoot?
 

deploy/common/clickhouse/cluster.ha.xml

@@ -0,0 +1,75 @@
+<?xml version="1.0"?>
+<clickhouse>
+    <!-- ZooKeeper is used to store metadata about replicas, when using Replicated tables.
+         Optional. If you don't use replicated tables, you could omit that.
+
+         See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/
+      -->
+    <zookeeper>
+        <node index="1">
+            <host>zookeeper-1</host>
+            <port>2181</port>
+        </node>
+        <node index="2">
+            <host>zookeeper-2</host>
+            <port>2181</port>
+        </node>
+        <node index="3">
+            <host>zookeeper-3</host>
+            <port>2181</port>
+        </node>
+    </zookeeper>
+
+    <!-- Configuration of clusters that could be used in Distributed tables.
+         https://clickhouse.com/docs/en/operations/table_engines/distributed/
+      -->
+    <remote_servers>
+        <cluster>
+            <!-- Inter-server per-cluster secret for Distributed queries
+                 default: no secret (no authentication will be performed)
+
+                 If set, then Distributed queries will be validated on shards, so at least:
+                 - such cluster should exist on the shard,
+                 - such cluster should have the same secret.
+
+                 And also (and which is more important), the initial_user will
+                 be used as current user for the query.
+
+                 Right now the protocol is pretty simple and it only takes into account:
+                 - cluster name
+                 - query
+
+                 Also it will be nice if the following will be implemented:
+                 - source hostname (see interserver_http_host), but then it will depends from DNS,
+                   it can use IP address instead, but then the you need to get correct on the initiator node.
+                 - target hostname / ip address (same notes as for source hostname)
+                 - time-based security tokens
+            -->
+            <!-- <secret></secret> -->
+            <shard>
+                <!-- Optional. Whether to write data to just one of the replicas. Default: false (write data to all replicas). -->
+                <!-- <internal_replication>false</internal_replication> -->
+                <!-- Optional. Shard weight when writing data. Default: 1. -->
+                <!-- <weight>1</weight> -->
+                <replica>
+                    <host>clickhouse</host>
+                    <port>9000</port>
+                    <!-- Optional. Priority of the replica for load_balancing. Default: 1 (less value has more priority). -->
+                    <!-- <priority>1</priority> -->
+                </replica>
+            </shard>
+            <shard>
+                <replica>
+                    <host>clickhouse-2</host>
+                    <port>9000</port>
+                </replica>
+            </shard>
+            <shard>
+                <replica>
+                    <host>clickhouse-3</host>
+                    <port>9000</port>
+                </replica>
+            </shard>
+        </cluster>
+    </remote_servers>
+</clickhouse>

deploy/common/clickhouse/cluster.xml

@@ -0,0 +1,75 @@
+<?xml version="1.0"?>
+<clickhouse>
+    <!-- ZooKeeper is used to store metadata about replicas, when using Replicated tables.
+         Optional. If you don't use replicated tables, you could omit that.
+
+         See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/
+      -->
+    <zookeeper>
+        <node index="1">
+            <host>zookeeper-1</host>
+            <port>2181</port>
+        </node>
+        <!-- <node index="2">
+            <host>zookeeper-2</host>
+            <port>2181</port>
+        </node>
+        <node index="3">
+            <host>zookeeper-3</host>
+            <port>2181</port>
+        </node> -->
+    </zookeeper>
+
+    <!-- Configuration of clusters that could be used in Distributed tables.
+         https://clickhouse.com/docs/en/operations/table_engines/distributed/
+      -->
+    <remote_servers>
+        <cluster>
+            <!-- Inter-server per-cluster secret for Distributed queries
+                 default: no secret (no authentication will be performed)
+
+                 If set, then Distributed queries will be validated on shards, so at least:
+                 - such cluster should exist on the shard,
+                 - such cluster should have the same secret.
+
+                 And also (and which is more important), the initial_user will
+                 be used as current user for the query.
+
+                 Right now the protocol is pretty simple and it only takes into account:
+                 - cluster name
+                 - query
+
+                 Also it will be nice if the following will be implemented:
+                 - source hostname (see interserver_http_host), but then it will depends from DNS,
+                   it can use IP address instead, but then the you need to get correct on the initiator node.
+                 - target hostname / ip address (same notes as for source hostname)
+                 - time-based security tokens
+            -->
+            <!-- <secret></secret> -->
+            <shard>
+                <!-- Optional. Whether to write data to just one of the replicas. Default: false (write data to all replicas). -->
+                <!-- <internal_replication>false</internal_replication> -->
+                <!-- Optional. Shard weight when writing data. Default: 1. -->
+                <!-- <weight>1</weight> -->
+                <replica>
+                    <host>clickhouse</host>
+                    <port>9000</port>
+                    <!-- Optional. Priority of the replica for load_balancing. Default: 1 (less value has more priority). -->
+                    <!-- <priority>1</priority> -->
+                </replica>
+            </shard>
+            <!-- <shard>
+                <replica>
+                    <host>clickhouse-2</host>
+                    <port>9000</port>
+                </replica>
+            </shard>
+            <shard>
+                <replica>
+                    <host>clickhouse-3</host>
+                    <port>9000</port>
+                </replica>
+            </shard> -->
+        </cluster>
+    </remote_servers>
+</clickhouse>

deploy/common/clickhouse/custom-function.xml

@@ -0,0 +1,21 @@
+<functions>
+    <function>
+        <type>executable</type>
+        <name>histogramQuantile</name>
+        <return_type>Float64</return_type>
+        <argument>
+            <type>Array(Float64)</type>
+            <name>buckets</name>
+        </argument>
+        <argument>
+            <type>Array(Float64)</type>
+            <name>counts</name>
+        </argument>
+        <argument>
+            <type>Float64</type>
+            <name>quantile</name>
+        </argument>
+        <format>CSV</format>
+        <command>./histogramQuantile</command>
+    </function>
+</functions>

deploy/common/clickhouse/storage.xml

@@ -0,0 +1,41 @@
+<?xml version="1.0"?>
+<clickhouse>
+<storage_configuration>
+    <disks>
+        <default>
+            <keep_free_space_bytes>10485760</keep_free_space_bytes>
+        </default>
+        <s3>
+            <type>s3</type>
+            <!-- For S3 cold storage,
+                    if region is us-east-1, endpoint can be https://<bucket-name>.s3.amazonaws.com
+                    if region is not us-east-1, endpoint should be https://<bucket-name>.s3-<region>.amazonaws.com
+                For GCS cold storage,
+                    endpoint should be https://storage.googleapis.com/<bucket-name>/data/
+                -->
+            <endpoint>https://BUCKET-NAME.s3-REGION-NAME.amazonaws.com/data/</endpoint>
+            <access_key_id>ACCESS-KEY-ID</access_key_id>
+            <secret_access_key>SECRET-ACCESS-KEY</secret_access_key>
+            <!-- In case of S3, uncomment the below configuration in case you want to read
+                AWS credentials from the Environment variables if they exist. -->
+            <!-- <use_environment_credentials>true</use_environment_credentials> -->
+            <!-- In case of GCS, uncomment the below configuration, since GCS does
+                not support batch deletion and result in error messages in logs. -->
+            <!-- <support_batch_delete>false</support_batch_delete> -->
+        </s3>
+   </disks>
+   <policies>
+       <tiered>
+           <volumes>
+                <default>
+                    <disk>default</disk>
+                </default>
+                <s3>
+                    <disk>s3</disk>
+                    <perform_ttl_move_on_insert>0</perform_ttl_move_on_insert>
+                </s3>
+            </volumes>
+        </tiered>
+    </policies>
+</storage_configuration>
+</clickhouse>

deploy/common/clickhouse/users.xml

@@ -0,0 +1,123 @@
+<?xml version="1.0"?>
+<clickhouse>
+    <!-- See also the files in users.d directory where the settings can be overridden. -->
+
+    <!-- Profiles of settings. -->
+    <profiles>
+        <!-- Default settings. -->
+        <default>
+            <!-- Maximum memory usage for processing single query, in bytes. -->
+            <max_memory_usage>10000000000</max_memory_usage>
+
+            <!-- How to choose between replicas during distributed query processing.
+                 random - choose random replica from set of replicas with minimum number of errors
+                 nearest_hostname - from set of replicas with minimum number of errors, choose replica
+                  with minimum number of different symbols between replica's hostname and local hostname
+                  (Hamming distance).
+                 in_order - first live replica is chosen in specified order.
+                 first_or_random - if first replica one has higher number of errors, pick a random one from replicas with minimum number of errors.
+            -->
+            <load_balancing>random</load_balancing>
+        </default>
+
+        <!-- Profile that allows only read queries. -->
+        <readonly>
+            <readonly>1</readonly>
+        </readonly>
+    </profiles>
+
+    <!-- Users and ACL. -->
+    <users>
+        <!-- If user name was not specified, 'default' user is used. -->
+        <default>
+            <!-- See also the files in users.d directory where the password can be overridden.
+
+                 Password could be specified in plaintext or in SHA256 (in hex format).
+
+                 If you want to specify password in plaintext (not recommended), place it in 'password' element.
+                 Example: <password>qwerty</password>.
+                 Password could be empty.
+
+                 If you want to specify SHA256, place it in 'password_sha256_hex' element.
+                 Example: <password_sha256_hex>65e84be33532fb784c48129675f9eff3a682b27168c0ea744b2cf58ee02337c5</password_sha256_hex>
+                 Restrictions of SHA256: impossibility to connect to ClickHouse using MySQL JS client (as of July 2019).
+
+                 If you want to specify double SHA1, place it in 'password_double_sha1_hex' element.
+                 Example: <password_double_sha1_hex>e395796d6546b1b65db9d665cd43f0e858dd4303</password_double_sha1_hex>
+
+                 If you want to specify a previously defined LDAP server (see 'ldap_servers' in the main config) for authentication,
+                  place its name in 'server' element inside 'ldap' element.
+                 Example: <ldap><server>my_ldap_server</server></ldap>
+
+                 If you want to authenticate the user via Kerberos (assuming Kerberos is enabled, see 'kerberos' in the main config),
+                  place 'kerberos' element instead of 'password' (and similar) elements.
+                 The name part of the canonical principal name of the initiator must match the user name for authentication to succeed.
+                 You can also place 'realm' element inside 'kerberos' element to further restrict authentication to only those requests
+                  whose initiator's realm matches it.
+                 Example: <kerberos />
+                 Example: <kerberos><realm>EXAMPLE.COM</realm></kerberos>
+
+                 How to generate decent password:
+                 Execute: PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha256sum | tr -d '-'
+                 In first line will be password and in second - corresponding SHA256.
+
+                 How to generate double SHA1:
+                 Execute: PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha1sum | tr -d '-' | xxd -r -p | sha1sum | tr -d '-'
+                 In first line will be password and in second - corresponding double SHA1.
+            -->
+            <password></password>
+
+            <!-- List of networks with open access.
+
+                 To open access from everywhere, specify:
+                    <ip>::/0</ip>
+
+                 To open access only from localhost, specify:
+                    <ip>::1</ip>
+                    <ip>127.0.0.1</ip>
+
+                 Each element of list has one of the following forms:
+                 <ip> IP-address or network mask. Examples: 213.180.204.3 or 10.0.0.1/8 or 10.0.0.1/255.255.255.0
+                     2a02:6b8::3 or 2a02:6b8::3/64 or 2a02:6b8::3/ffff:ffff:ffff:ffff::.
+                 <host> Hostname. Example: server01.clickhouse.com.
+                     To check access, DNS query is performed, and all received addresses compared to peer address.
+                 <host_regexp> Regular expression for host names. Example, ^server\d\d-\d\d-\d\.clickhouse\.com$
+                     To check access, DNS PTR query is performed for peer address and then regexp is applied.
+                     Then, for result of PTR query, another DNS query is performed and all received addresses compared to peer address.
+                     Strongly recommended that regexp is ends with $
+                 All results of DNS requests are cached till server restart.
+            -->
+            <networks>
+                <ip>::/0</ip>
+            </networks>
+
+            <!-- Settings profile for user. -->
+            <profile>default</profile>
+
+            <!-- Quota for user. -->
+            <quota>default</quota>
+
+            <!-- User can create other users and grant rights to them. -->
+            <!-- <access_management>1</access_management> -->
+        </default>
+    </users>
+
+    <!-- Quotas. -->
+    <quotas>
+        <!-- Name of quota. -->
+        <default>
+            <!-- Limits for time interval. You could specify many intervals with different limits. -->
+            <interval>
+                <!-- Length of interval. -->
+                <duration>3600</duration>
+
+                <!-- No limits. Just calculate resource usage for time interval. -->
+                <queries>0</queries>
+                <errors>0</errors>
+                <result_rows>0</result_rows>
+                <read_rows>0</read_rows>
+                <execution_time>0</execution_time>
+            </interval>
+        </default>
+    </quotas>
+</clickhouse>

deploy/common/locust-scripts/locustfile.py

@@ -0,0 +1,16 @@
+from locust import HttpUser, task, between
+class UserTasks(HttpUser):
+    wait_time = between(5, 15)
+
+    @task
+    def rachel(self):
+        self.client.get("/dispatch?customer=123&nonse=0.6308392664170006")
+    @task
+    def trom(self):
+        self.client.get("/dispatch?customer=392&nonse=0.015296363321630757")
+    @task
+    def japanese(self):
+        self.client.get("/dispatch?customer=731&nonse=0.8022286220408668")
+    @task
+    def coffee(self):
+        self.client.get("/dispatch?customer=567&nonse=0.0022220379420636593")

deploy/common/signoz/otel-collector-opamp-config.yaml

@@ -0,0 +1 @@
+server_endpoint: ws://signoz:4320/v1/opamp

deploy/common/signoz/prometheus.yml

@@ -0,0 +1,25 @@
+# my global config
+global:
+  scrape_interval:     5s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
+  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
+  # scrape_timeout is set to the global default (10s).
+
+# Alertmanager configuration
+alerting:
+  alertmanagers:
+  - static_configs:
+    - targets:
+      - alertmanager:9093
+
+# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
+rule_files: []
+  # - "first_rules.yml"
+  # - "second_rules.yml"
+  # - 'alerts.yml'
+
+# A scrape configuration containing exactly one endpoint to scrape:
+# Here it's Prometheus itself.
+scrape_configs: []
+
+remote_read:
+  - url: tcp://clickhouse:9000/signoz_metrics

deploy/docker-swarm/docker-compose.ha.yaml

@@ -0,0 +1,288 @@
+version: "3"
+x-common: &common
+  networks:
+    - signoz-net
+  deploy:
+    restart_policy:
+      condition: on-failure
+  logging:
+    options:
+      max-size: 50m
+      max-file: "3"
+x-clickhouse-defaults: &clickhouse-defaults
+  !!merge <<: *common
+  image: clickhouse/clickhouse-server:25.5.6
+  tty: true
+  deploy:
+    labels:
+      signoz.io/scrape: "true"
+      signoz.io/port: "9363"
+      signoz.io/path: "/metrics"
+  depends_on:
+    - zookeeper-1
+    - zookeeper-2
+    - zookeeper-3
+  healthcheck:
+    test:
+      - CMD
+      - wget
+      - --spider
+      - -q
+      - 0.0.0.0:8123/ping
+    interval: 30s
+    timeout: 5s
+    retries: 3
+  ulimits:
+    nproc: 65535
+    nofile:
+      soft: 262144
+      hard: 262144
+  environment:
+    - CLICKHOUSE_SKIP_USER_SETUP=1
+x-zookeeper-defaults: &zookeeper-defaults
+  !!merge <<: *common
+  image: signoz/zookeeper:3.7.1
+  user: root
+  deploy:
+    labels:
+      signoz.io/scrape: "true"
+      signoz.io/port: "9141"
+      signoz.io/path: "/metrics"
+  healthcheck:
+    test:
+      - CMD-SHELL
+      - curl -s -m 2 http://localhost:8080/commands/ruok | grep error | grep null
+    interval: 30s
+    timeout: 5s
+    retries: 3
+x-db-depend: &db-depend
+  !!merge <<: *common
+  depends_on:
+    - clickhouse
+    - clickhouse-2
+    - clickhouse-3
+services:
+  init-clickhouse:
+    !!merge <<: *common
+    image: clickhouse/clickhouse-server:25.5.6
+    command:
+      - bash
+      - -c
+      - |
+        version="v0.0.1"
+        node_os=$$(uname -s | tr '[:upper:]' '[:lower:]')
+        node_arch=$$(uname -m | sed s/aarch64/arm64/ | sed s/x86_64/amd64/)
+        echo "Fetching histogram-binary for $${node_os}/$${node_arch}"
+        cd /tmp
+        wget -O histogram-quantile.tar.gz "https://github.com/SigNoz/signoz/releases/download/histogram-quantile%2F$${version}/histogram-quantile_$${node_os}_$${node_arch}.tar.gz"
+        tar -xvzf histogram-quantile.tar.gz
+        mv histogram-quantile /var/lib/clickhouse/user_scripts/histogramQuantile
+    deploy:
+      restart_policy:
+        condition: on-failure
+    volumes:
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+  zookeeper-1:
+    !!merge <<: *zookeeper-defaults
+    # ports:
+    #   - "2181:2181"
+    #   - "2888:2888"
+    #   - "3888:3888"
+    volumes:
+      - ./clickhouse-setup/data/zookeeper-1:/bitnami/zookeeper
+    environment:
+      - ZOO_SERVER_ID=1
+      - ZOO_SERVERS=0.0.0.0:2888:3888,zookeeper-2:2888:3888,zookeeper-3:2888:3888
+      - ALLOW_ANONYMOUS_LOGIN=yes
+      - ZOO_AUTOPURGE_INTERVAL=1
+      - ZOO_ENABLE_PROMETHEUS_METRICS=yes
+      - ZOO_PROMETHEUS_METRICS_PORT_NUMBER=9141
+  zookeeper-2:
+    !!merge <<: *zookeeper-defaults
+    # ports:
+    #   - "2182:2181"
+    #   - "2889:2888"
+    #   - "3889:3888"
+    volumes:
+      - ./clickhouse-setup/data/zookeeper-2:/bitnami/zookeeper
+    environment:
+      - ZOO_SERVER_ID=2
+      - ZOO_SERVERS=zookeeper-1:2888:3888,0.0.0.0:2888:3888,zookeeper-3:2888:3888
+      - ALLOW_ANONYMOUS_LOGIN=yes
+      - ZOO_AUTOPURGE_INTERVAL=1
+      - ZOO_ENABLE_PROMETHEUS_METRICS=yes
+      - ZOO_PROMETHEUS_METRICS_PORT_NUMBER=9141
+  zookeeper-3:
+    !!merge <<: *zookeeper-defaults
+    # ports:
+    #   - "2183:2181"
+    #   - "2890:2888"
+    #   - "3890:3888"
+    volumes:
+      - ./clickhouse-setup/data/zookeeper-3:/bitnami/zookeeper
+    environment:
+      - ZOO_SERVER_ID=3
+      - ZOO_SERVERS=zookeeper-1:2888:3888,zookeeper-2:2888:3888,0.0.0.0:2888:3888
+      - ALLOW_ANONYMOUS_LOGIN=yes
+      - ZOO_AUTOPURGE_INTERVAL=1
+      - ZOO_ENABLE_PROMETHEUS_METRICS=yes
+      - ZOO_PROMETHEUS_METRICS_PORT_NUMBER=9141
+  clickhouse:
+    !!merge <<: *clickhouse-defaults
+    # TODO: needed for schema-migrator to work, remove this redundancy once we have a better solution
+    hostname: clickhouse
+    # ports:
+    #   - "9000:9000"
+    #   - "8123:8123"
+    #   - "9181:9181"
+    configs:
+      - source: clickhouse-config
+        target: /etc/clickhouse-server/config.xml
+      - source: clickhouse-users
+        target: /etc/clickhouse-server/users.xml
+      - source: clickhouse-custom-function
+        target: /etc/clickhouse-server/custom-function.xml
+      - source: clickhouse-cluster
+        target: /etc/clickhouse-server/config.d/cluster.ha.xml
+    volumes:
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+      - ./clickhouse-setup/data/clickhouse/:/var/lib/clickhouse/
+      # - ../common/clickhouse/storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  clickhouse-2:
+    !!merge <<: *clickhouse-defaults
+    hostname: clickhouse-2
+    # ports:
+    #   - "9001:9000"
+    #   - "8124:8123"
+    #   - "9182:9181"
+    configs:
+      - source: clickhouse-config
+        target: /etc/clickhouse-server/config.xml
+      - source: clickhouse-users
+        target: /etc/clickhouse-server/users.xml
+      - source: clickhouse-custom-function
+        target: /etc/clickhouse-server/custom-function.xml
+      - source: clickhouse-cluster
+        target: /etc/clickhouse-server/config.d/cluster.ha.xml
+    volumes:
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+      - ./clickhouse-setup/data/clickhouse-2/:/var/lib/clickhouse/
+      # - ../common/clickhouse/storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  clickhouse-3:
+    !!merge <<: *clickhouse-defaults
+    hostname: clickhouse-3
+    # ports:
+    #   - "9002:9000"
+    #   - "8125:8123"
+    #   - "9183:9181"
+    configs:
+      - source: clickhouse-config
+        target: /etc/clickhouse-server/config.xml
+      - source: clickhouse-users
+        target: /etc/clickhouse-server/users.xml
+      - source: clickhouse-custom-function
+        target: /etc/clickhouse-server/custom-function.xml
+      - source: clickhouse-cluster
+        target: /etc/clickhouse-server/config.d/cluster.ha.xml
+    volumes:
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+      - ./clickhouse-setup/data/clickhouse-3/:/var/lib/clickhouse/
+      # - ../common/clickhouse/storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  signoz:
+    !!merge <<: *db-depend
+    image: signoz/signoz:v0.129.0
+    ports:
+      - "8080:8080" # signoz port
+    #   - "6060:6060"     # pprof port
+    volumes:
+      - ./clickhouse-setup/data/signoz/:/var/lib/signoz/
+    environment:
+      - SIGNOZ_ALERTMANAGER_PROVIDER=signoz
+      - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db
+      - SIGNOZ_TOKENIZER_JWT_SECRET=secret
+    healthcheck:
+      test:
+        - CMD
+        - wget
+        - --spider
+        - -q
+        - localhost:8080/api/v1/health
+      interval: 30s
+      timeout: 5s
+      retries: 3
+  otel-collector:
+    !!merge <<: *db-depend
+    image: signoz/signoz-otel-collector:v0.144.5
+    entrypoint:
+      - /bin/sh
+    command:
+      - -c
+      - |
+        /signoz-otel-collector migrate sync check &&
+        /signoz-otel-collector --config=/etc/otel-collector-config.yaml --manager-config=/etc/manager-config.yaml --copy-path=/var/tmp/collector-config.yaml
+    configs:
+      - source: otel-collector-config
+        target: /etc/otel-collector-config.yaml
+      - source: otel-manager-config
+        target: /etc/manager-config.yaml
+    environment:
+      - OTEL_RESOURCE_ATTRIBUTES=host.name={{.Node.Hostname}},os.type={{.Node.Platform.OS}}
+      - LOW_CARDINAL_EXCEPTION_GROUPING=false
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true
+      - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m
+    ports:
+      # - "1777:1777"     # pprof extension
+      - "4317:4317" # OTLP gRPC receiver
+      - "4318:4318" # OTLP HTTP receiver
+    deploy:
+      replicas: 3
+  signoz-telemetrystore-migrator:
+    !!merge <<: *db-depend
+    image: signoz/signoz-otel-collector:v0.144.5
+    environment:
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true
+      - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m
+    entrypoint:
+      - /bin/sh
+    command:
+      - -c
+      - |
+        /signoz-otel-collector migrate bootstrap &&
+        /signoz-otel-collector migrate sync up &&
+        /signoz-otel-collector migrate async up
+networks:
+  signoz-net:
+    name: signoz-net
+volumes:
+  clickhouse:
+    name: signoz-clickhouse
+  clickhouse-2:
+    name: signoz-clickhouse-2
+  clickhouse-3:
+    name: signoz-clickhouse-3
+  sqlite:
+    name: signoz-sqlite
+  zookeeper-1:
+    name: signoz-zookeeper-1
+  zookeeper-2:
+    name: signoz-zookeeper-2
+  zookeeper-3:
+    name: signoz-zookeeper-3
+configs:
+  clickhouse-config:
+    file: ../common/clickhouse/config.xml
+  clickhouse-users:
+    file: ../common/clickhouse/users.xml
+  clickhouse-custom-function:
+    file: ../common/clickhouse/custom-function.xml
+  clickhouse-cluster:
+    file: ../common/clickhouse/cluster.ha.xml
+  otel-collector-config:
+    file: ./otel-collector-config.yaml
+  otel-manager-config:
+    file: ../common/signoz/otel-collector-opamp-config.yaml

deploy/docker-swarm/docker-compose.yaml

@@ -0,0 +1,206 @@
+version: "3"
+x-common: &common
+  networks:
+    - signoz-net
+  deploy:
+    restart_policy:
+      condition: on-failure
+  logging:
+    options:
+      max-size: 50m
+      max-file: "3"
+x-clickhouse-defaults: &clickhouse-defaults
+  !!merge <<: *common
+  image: clickhouse/clickhouse-server:25.5.6
+  tty: true
+  deploy:
+    labels:
+      signoz.io/scrape: "true"
+      signoz.io/port: "9363"
+      signoz.io/path: "/metrics"
+  depends_on:
+    - init-clickhouse
+    - zookeeper-1
+  healthcheck:
+    test:
+      - CMD
+      - wget
+      - --spider
+      - -q
+      - 0.0.0.0:8123/ping
+    interval: 30s
+    timeout: 5s
+    retries: 3
+  ulimits:
+    nproc: 65535
+    nofile:
+      soft: 262144
+      hard: 262144
+  environment:
+    - CLICKHOUSE_SKIP_USER_SETUP=1
+x-zookeeper-defaults: &zookeeper-defaults
+  !!merge <<: *common
+  image: signoz/zookeeper:3.7.1
+  user: root
+  deploy:
+    labels:
+      signoz.io/scrape: "true"
+      signoz.io/port: "9141"
+      signoz.io/path: "/metrics"
+  healthcheck:
+    test:
+      - CMD-SHELL
+      - curl -s -m 2 http://localhost:8080/commands/ruok | grep error | grep null
+    interval: 30s
+    timeout: 5s
+    retries: 3
+x-db-depend: &db-depend
+  !!merge <<: *common
+  depends_on:
+    - clickhouse
+services:
+  init-clickhouse:
+    !!merge <<: *common
+    image: clickhouse/clickhouse-server:25.5.6
+    command:
+      - bash
+      - -c
+      - |
+        version="v0.0.1"
+        node_os=$$(uname -s | tr '[:upper:]' '[:lower:]')
+        node_arch=$$(uname -m | sed s/aarch64/arm64/ | sed s/x86_64/amd64/)
+        echo "Fetching histogram-binary for $${node_os}/$${node_arch}"
+        cd /tmp
+        wget -O histogram-quantile.tar.gz "https://github.com/SigNoz/signoz/releases/download/histogram-quantile%2F$${version}/histogram-quantile_$${node_os}_$${node_arch}.tar.gz"
+        tar -xvzf histogram-quantile.tar.gz
+        mv histogram-quantile /var/lib/clickhouse/user_scripts/histogramQuantile
+    deploy:
+      restart_policy:
+        condition: on-failure
+    volumes:
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+  zookeeper-1:
+    !!merge <<: *zookeeper-defaults
+    # ports:
+    #   - "2181:2181"
+    #   - "2888:2888"
+    #   - "3888:3888"
+    volumes:
+      - zookeeper-1:/bitnami/zookeeper
+    environment:
+      - ZOO_SERVER_ID=1
+      - ALLOW_ANONYMOUS_LOGIN=yes
+      - ZOO_AUTOPURGE_INTERVAL=1
+      - ZOO_ENABLE_PROMETHEUS_METRICS=yes
+      - ZOO_PROMETHEUS_METRICS_PORT_NUMBER=9141
+  clickhouse:
+    !!merge <<: *clickhouse-defaults
+    # TODO: needed for clickhouse TCP connectio
+    hostname: clickhouse
+    # ports:
+    #   - "9000:9000"
+    #   - "8123:8123"
+    #   - "9181:9181"
+
+    configs:
+      - source: clickhouse-config
+        target: /etc/clickhouse-server/config.xml
+      - source: clickhouse-users
+        target: /etc/clickhouse-server/users.xml
+      - source: clickhouse-custom-function
+        target: /etc/clickhouse-server/custom-function.xml
+      - source: clickhouse-cluster
+        target: /etc/clickhouse-server/config.d/cluster.xml
+    volumes:
+      - clickhouse:/var/lib/clickhouse/
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+      # - ../common/clickhouse/storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  signoz:
+    !!merge <<: *db-depend
+    image: signoz/signoz:v0.129.0
+    ports:
+      - "8080:8080" # signoz port
+    volumes:
+      - sqlite:/var/lib/signoz/
+    environment:
+      - SIGNOZ_ALERTMANAGER_PROVIDER=signoz
+      - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db
+      - SIGNOZ_TOKENIZER_JWT_SECRET=secret
+    healthcheck:
+      test:
+        - CMD
+        - wget
+        - --spider
+        - -q
+        - localhost:8080/api/v1/health
+      interval: 30s
+      timeout: 5s
+      retries: 3
+  otel-collector:
+    !!merge <<: *db-depend
+    image: signoz/signoz-otel-collector:v0.144.5
+    entrypoint:
+      - /bin/sh
+    command:
+      - -c
+      - |
+        /signoz-otel-collector migrate sync check &&
+        /signoz-otel-collector --config=/etc/otel-collector-config.yaml --manager-config=/etc/manager-config.yaml --copy-path=/var/tmp/collector-config.yaml
+    configs:
+      - source: otel-collector-config
+        target: /etc/otel-collector-config.yaml
+      - source: otel-manager-config
+        target: /etc/manager-config.yaml
+    environment:
+      - OTEL_RESOURCE_ATTRIBUTES=host.name={{.Node.Hostname}},os.type={{.Node.Platform.OS}}
+      - LOW_CARDINAL_EXCEPTION_GROUPING=false
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true
+      - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m
+    ports:
+      # - "1777:1777"     # pprof extension
+      - "4317:4317" # OTLP gRPC receiver
+      - "4318:4318" # OTLP HTTP receiver
+    deploy:
+      replicas: 3
+  signoz-telemetrystore-migrator:
+    !!merge <<: *db-depend
+    image: signoz/signoz-otel-collector:v0.144.5
+    environment:
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true
+      - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m
+    entrypoint:
+      - /bin/sh
+    command:
+      - -c
+      - |
+        /signoz-otel-collector migrate bootstrap &&
+        /signoz-otel-collector migrate sync up &&
+        /signoz-otel-collector migrate async up
+networks:
+  signoz-net:
+    name: signoz-net
+volumes:
+  clickhouse:
+    name: signoz-clickhouse
+  sqlite:
+    name: signoz-sqlite
+  zookeeper-1:
+    name: signoz-zookeeper-1
+configs:
+  clickhouse-config:
+    file: ../common/clickhouse/config.xml
+  clickhouse-users:
+    file: ../common/clickhouse/users.xml
+  clickhouse-custom-function:
+    file: ../common/clickhouse/custom-function.xml
+  clickhouse-cluster:
+    file: ../common/clickhouse/cluster.xml
+  otel-collector-config:
+    file: ./otel-collector-config.yaml
+  otel-manager-config:
+    file: ../common/signoz/otel-collector-opamp-config.yaml

deploy/docker-swarm/otel-collector-config.yaml

@@ -0,0 +1,118 @@
+connectors:
+  signozmeter:
+    metrics_flush_interval: 1h
+    dimensions:
+      - name: service.name
+      - name: deployment.environment
+      - name: host.name
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+  prometheus:
+    config:
+      global:
+        scrape_interval: 60s
+      scrape_configs:
+        - job_name: otel-collector
+          static_configs:
+          - targets:
+              - localhost:8888
+            labels:
+              job_name: otel-collector
+processors:
+  batch:
+    send_batch_size: 10000
+    send_batch_max_size: 11000
+    timeout: 10s
+  batch/meter:
+    send_batch_max_size: 25000
+    send_batch_size: 20000
+    timeout: 1s
+  resourcedetection:
+    # Using OTEL_RESOURCE_ATTRIBUTES envvar, env detector adds custom labels.
+    detectors: [env, system]
+    timeout: 2s
+  signozspanmetrics/delta:
+    metrics_exporter: signozclickhousemetrics
+    metrics_flush_interval: 60s
+    latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
+    dimensions_cache_size: 100000
+    aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA
+    enable_exp_histogram: true
+    dimensions:
+      - name: service.namespace
+        default: default
+      - name: deployment.environment
+        default: default
+      # This is added to ensure the uniqueness of the timeseries
+      # Otherwise, identical timeseries produced by multiple replicas of
+      # collectors result in incorrect APM metrics
+      - name: signoz.collector.id
+      - name: service.version
+      - name: browser.platform
+      - name: browser.mobile
+      - name: k8s.cluster.name
+      - name: k8s.node.name
+      - name: k8s.namespace.name
+      - name: host.name
+      - name: host.type
+      - name: container.name
+extensions:
+  health_check:
+    endpoint: 0.0.0.0:13133
+  pprof:
+    endpoint: 0.0.0.0:1777
+exporters:
+  clickhousetraces:
+    datasource: tcp://clickhouse:9000/signoz_traces
+    low_cardinal_exception_grouping: ${env:LOW_CARDINAL_EXCEPTION_GROUPING}
+    use_new_schema: true
+  signozclickhousemetrics:
+    dsn: tcp://clickhouse:9000/signoz_metrics
+  clickhouselogsexporter:
+    dsn: tcp://clickhouse:9000/signoz_logs
+    timeout: 10s
+    use_new_schema: true
+  signozclickhousemeter:
+    dsn: tcp://clickhouse:9000/signoz_meter
+    timeout: 45s
+    sending_queue:
+      enabled: false
+  metadataexporter:
+    cache:
+      provider: in_memory
+    dsn: tcp://clickhouse:9000/signoz_metadata
+    enabled: true
+    timeout: 45s
+service:
+  telemetry:
+    logs:
+      encoding: json
+  extensions:
+    - health_check
+    - pprof
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [signozspanmetrics/delta, batch]
+      exporters: [clickhousetraces, metadataexporter, signozmeter]
+    metrics:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [signozclickhousemetrics, metadataexporter, signozmeter]
+    metrics/prometheus:
+      receivers: [prometheus]
+      processors: [batch]
+      exporters: [signozclickhousemetrics, metadataexporter, signozmeter]
+    logs:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [clickhouselogsexporter, metadataexporter, signozmeter]
+    metrics/meter:
+      receivers: [signozmeter]
+      processors: [batch/meter]
+      exporters: [signozclickhousemeter]

deploy/docker/.env

@@ -0,0 +1 @@
+COMPOSE_PROJECT_NAME=signoz

deploy/docker/clickhouse-setup/.deprecated

@@ -0,0 +1,3 @@
+This data directory is deprecated and will be removed in the future.
+Please use the migration script under `scripts/volume-migration` to migrate data from bind mounts to Docker volumes.
+The script also renames the project name to `signoz` and the network name to `signoz-net` (if not already in place).

deploy/docker/clickhouse-setup/user_scripts/.deprecated

@@ -0,0 +1,2 @@
+This directory is deprecated and will be removed in the future.
+Please use the new directory for Clickhouse setup scripts: `scripts/clickhouse` instead.

deploy/docker/docker-compose.ha.yaml

@@ -0,0 +1,265 @@
+version: "3"
+x-common: &common
+  networks:
+    - signoz-net
+  restart: unless-stopped
+  logging:
+    options:
+      max-size: 50m
+      max-file: "3"
+x-clickhouse-defaults: &clickhouse-defaults
+  !!merge <<: *common
+  # addding non LTS version due to this fix https://github.com/ClickHouse/ClickHouse/commit/32caf8716352f45c1b617274c7508c86b7d1afab
+  image: clickhouse/clickhouse-server:25.5.6
+  tty: true
+  labels:
+    signoz.io/scrape: "true"
+    signoz.io/port: "9363"
+    signoz.io/path: "/metrics"
+  depends_on:
+    init-clickhouse:
+      condition: service_completed_successfully
+    zookeeper-1:
+      condition: service_healthy
+    zookeeper-2:
+      condition: service_healthy
+    zookeeper-3:
+      condition: service_healthy
+  healthcheck:
+    test:
+      - CMD
+      - wget
+      - --spider
+      - -q
+      - 0.0.0.0:8123/ping
+    interval: 30s
+    timeout: 5s
+    retries: 3
+  ulimits:
+    nproc: 65535
+    nofile:
+      soft: 262144
+      hard: 262144
+  environment:
+    - CLICKHOUSE_SKIP_USER_SETUP=1
+x-zookeeper-defaults: &zookeeper-defaults
+  !!merge <<: *common
+  image: signoz/zookeeper:3.7.1
+  user: root
+  labels:
+    signoz.io/scrape: "true"
+    signoz.io/port: "9141"
+    signoz.io/path: "/metrics"
+  healthcheck:
+    test:
+      - CMD-SHELL
+      - curl -s -m 2 http://localhost:8080/commands/ruok | grep error | grep null
+    interval: 30s
+    timeout: 5s
+    retries: 3
+x-db-depend: &db-depend
+  !!merge <<: *common
+  depends_on:
+    clickhouse:
+      condition: service_healthy
+    clickhouse-2:
+      condition: service_healthy
+    clickhouse-3:
+      condition: service_healthy
+services:
+  init-clickhouse:
+    !!merge <<: *common
+    image: clickhouse/clickhouse-server:25.5.6
+    container_name: signoz-init-clickhouse
+    command:
+      - bash
+      - -c
+      - |
+        version="v0.0.1"
+        node_os=$$(uname -s | tr '[:upper:]' '[:lower:]')
+        node_arch=$$(uname -m | sed s/aarch64/arm64/ | sed s/x86_64/amd64/)
+        echo "Fetching histogram-binary for $${node_os}/$${node_arch}"
+        cd /tmp
+        wget -O histogram-quantile.tar.gz "https://github.com/SigNoz/signoz/releases/download/histogram-quantile%2F$${version}/histogram-quantile_$${node_os}_$${node_arch}.tar.gz"
+        tar -xvzf histogram-quantile.tar.gz
+        mv histogram-quantile /var/lib/clickhouse/user_scripts/histogramQuantile
+    restart: on-failure
+    volumes:
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+  zookeeper-1:
+    !!merge <<: *zookeeper-defaults
+    container_name: signoz-zookeeper-1
+    # ports:
+    #   - "2181:2181"
+    #   - "2888:2888"
+    #   - "3888:3888"
+    volumes:
+      - zookeeper-1:/bitnami/zookeeper
+    environment:
+      - ZOO_SERVER_ID=1
+      - ZOO_SERVERS=0.0.0.0:2888:3888,zookeeper-2:2888:3888,zookeeper-3:2888:3888
+      - ALLOW_ANONYMOUS_LOGIN=yes
+      - ZOO_AUTOPURGE_INTERVAL=1
+      - ZOO_ENABLE_PROMETHEUS_METRICS=yes
+      - ZOO_PROMETHEUS_METRICS_PORT_NUMBER=9141
+  zookeeper-2:
+    !!merge <<: *zookeeper-defaults
+    container_name: signoz-zookeeper-2
+    # ports:
+    #   - "2182:2181"
+    #   - "2889:2888"
+    #   - "3889:3888"
+    volumes:
+      - zookeeper-2:/bitnami/zookeeper
+    environment:
+      - ZOO_SERVER_ID=2
+      - ZOO_SERVERS=zookeeper-1:2888:3888,0.0.0.0:2888:3888,zookeeper-3:2888:3888
+      - ALLOW_ANONYMOUS_LOGIN=yes
+      - ZOO_AUTOPURGE_INTERVAL=1
+      - ZOO_ENABLE_PROMETHEUS_METRICS=yes
+      - ZOO_PROMETHEUS_METRICS_PORT_NUMBER=9141
+  zookeeper-3:
+    !!merge <<: *zookeeper-defaults
+    container_name: signoz-zookeeper-3
+    # ports:
+    #   - "2183:2181"
+    #   - "2890:2888"
+    #   - "3890:3888"
+    volumes:
+      - zookeeper-3:/bitnami/zookeeper
+    environment:
+      - ZOO_SERVER_ID=3
+      - ZOO_SERVERS=zookeeper-1:2888:3888,zookeeper-2:2888:3888,0.0.0.0:2888:3888
+      - ALLOW_ANONYMOUS_LOGIN=yes
+      - ZOO_AUTOPURGE_INTERVAL=1
+      - ZOO_ENABLE_PROMETHEUS_METRICS=yes
+      - ZOO_PROMETHEUS_METRICS_PORT_NUMBER=9141
+  clickhouse:
+    !!merge <<: *clickhouse-defaults
+    container_name: signoz-clickhouse
+    # ports:
+    #   - "9000:9000"
+    #   - "8123:8123"
+    #   - "9181:9181"
+    volumes:
+      - ../common/clickhouse/config.xml:/etc/clickhouse-server/config.xml
+      - ../common/clickhouse/users.xml:/etc/clickhouse-server/users.xml
+      - ../common/clickhouse/custom-function.xml:/etc/clickhouse-server/custom-function.xml
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+      - ../common/clickhouse/cluster.ha.xml:/etc/clickhouse-server/config.d/cluster.xml
+      - clickhouse:/var/lib/clickhouse/
+      # - ../common/clickhouse/storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  clickhouse-2:
+    !!merge <<: *clickhouse-defaults
+    container_name: signoz-clickhouse-2
+    # ports:
+    #   - "9001:9000"
+    #   - "8124:8123"
+    #   - "9182:9181"
+    volumes:
+      - ../common/clickhouse/config.xml:/etc/clickhouse-server/config.xml
+      - ../common/clickhouse/users.xml:/etc/clickhouse-server/users.xml
+      - ../common/clickhouse/custom-function.xml:/etc/clickhouse-server/custom-function.xml
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+      - ../common/clickhouse/cluster.ha.xml:/etc/clickhouse-server/config.d/cluster.xml
+      - clickhouse-2:/var/lib/clickhouse/
+      # - ../common/clickhouse/storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  clickhouse-3:
+    !!merge <<: *clickhouse-defaults
+    container_name: signoz-clickhouse-3
+    # ports:
+    #   - "9002:9000"
+    #   - "8125:8123"
+    #   - "9183:9181"
+    volumes:
+      - ../common/clickhouse/config.xml:/etc/clickhouse-server/config.xml
+      - ../common/clickhouse/users.xml:/etc/clickhouse-server/users.xml
+      - ../common/clickhouse/custom-function.xml:/etc/clickhouse-server/custom-function.xml
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+      - ../common/clickhouse/cluster.ha.xml:/etc/clickhouse-server/config.d/cluster.xml
+      - clickhouse-3:/var/lib/clickhouse/
+      # - ../common/clickhouse/storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  signoz:
+    !!merge <<: *db-depend
+    image: signoz/signoz:${VERSION:-v0.129.0}
+    container_name: signoz
+    ports:
+      - "8080:8080" # signoz port
+    volumes:
+      - sqlite:/var/lib/signoz/
+    environment:
+      - SIGNOZ_ALERTMANAGER_PROVIDER=signoz
+      - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db
+      - SIGNOZ_TOKENIZER_JWT_SECRET=secret
+    healthcheck:
+      test:
+        - CMD
+        - wget
+        - --spider
+        - -q
+        - localhost:8080/api/v1/health
+      interval: 30s
+      timeout: 5s
+      retries: 3
+  otel-collector:
+    !!merge <<: *db-depend
+    image: signoz/signoz-otel-collector:${OTELCOL_TAG:-v0.144.5}
+    container_name: signoz-otel-collector
+    entrypoint:
+      - /bin/sh
+    command:
+      - -c
+      - |
+        /signoz-otel-collector migrate sync check &&
+        /signoz-otel-collector --config=/etc/otel-collector-config.yaml --manager-config=/etc/manager-config.yaml --copy-path=/var/tmp/collector-config.yaml
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
+      - ../common/signoz/otel-collector-opamp-config.yaml:/etc/manager-config.yaml
+    environment:
+      - OTEL_RESOURCE_ATTRIBUTES=host.name=signoz-host,os.type=linux
+      - LOW_CARDINAL_EXCEPTION_GROUPING=false
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true
+      - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m
+    ports:
+      # - "1777:1777"     # pprof extension
+      - "4317:4317" # OTLP gRPC receiver
+      - "4318:4318" # OTLP HTTP receiver
+  signoz-telemetrystore-migrator:
+    !!merge <<: *db-depend
+    image: signoz/signoz-otel-collector:${OTELCOL_TAG:-v0.144.5}
+    container_name: signoz-telemetrystore-migrator
+    environment:
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true
+      - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m
+    entrypoint:
+      - /bin/sh
+    command:
+      - -c
+      - |
+        /signoz-otel-collector migrate bootstrap &&
+        /signoz-otel-collector migrate sync up &&
+        /signoz-otel-collector migrate async up
+    restart: on-failure
+networks:
+  signoz-net:
+    name: signoz-net
+volumes:
+  clickhouse:
+    name: signoz-clickhouse
+  clickhouse-2:
+    name: signoz-clickhouse-2
+  clickhouse-3:
+    name: signoz-clickhouse-3
+  sqlite:
+    name: signoz-sqlite
+  zookeeper-1:
+    name: signoz-zookeeper-1
+  zookeeper-2:
+    name: signoz-zookeeper-2
+  zookeeper-3:
+    name: signoz-zookeeper-3

deploy/docker/docker-compose.yaml

@@ -0,0 +1,185 @@
+version: "3"
+x-common: &common
+  networks:
+    - signoz-net
+  restart: unless-stopped
+  logging:
+    options:
+      max-size: 50m
+      max-file: "3"
+x-clickhouse-defaults: &clickhouse-defaults
+  !!merge <<: *common
+  image: clickhouse/clickhouse-server:25.5.6
+  tty: true
+  labels:
+    signoz.io/scrape: "true"
+    signoz.io/port: "9363"
+    signoz.io/path: "/metrics"
+  depends_on:
+    init-clickhouse:
+      condition: service_completed_successfully
+    zookeeper-1:
+      condition: service_healthy
+  healthcheck:
+    test:
+      - CMD
+      - wget
+      - --spider
+      - -q
+      - 0.0.0.0:8123/ping
+    interval: 30s
+    timeout: 5s
+    retries: 3
+  ulimits:
+    nproc: 65535
+    nofile:
+      soft: 262144
+      hard: 262144
+  environment:
+    - CLICKHOUSE_SKIP_USER_SETUP=1
+x-zookeeper-defaults: &zookeeper-defaults
+  !!merge <<: *common
+  image: signoz/zookeeper:3.7.1
+  user: root
+  labels:
+    signoz.io/scrape: "true"
+    signoz.io/port: "9141"
+    signoz.io/path: "/metrics"
+  healthcheck:
+    test:
+      - CMD-SHELL
+      - curl -s -m 2 http://localhost:8080/commands/ruok | grep error | grep null
+    interval: 30s
+    timeout: 5s
+    retries: 3
+x-db-depend: &db-depend
+  !!merge <<: *common
+  depends_on:
+    clickhouse:
+      condition: service_healthy
+services:
+  init-clickhouse:
+    !!merge <<: *common
+    image: clickhouse/clickhouse-server:25.5.6
+    container_name: signoz-init-clickhouse
+    command:
+      - bash
+      - -c
+      - |
+        version="v0.0.1"
+        node_os=$$(uname -s | tr '[:upper:]' '[:lower:]')
+        node_arch=$$(uname -m | sed s/aarch64/arm64/ | sed s/x86_64/amd64/)
+        echo "Fetching histogram-binary for $${node_os}/$${node_arch}"
+        cd /tmp
+        wget -O histogram-quantile.tar.gz "https://github.com/SigNoz/signoz/releases/download/histogram-quantile%2F$${version}/histogram-quantile_$${node_os}_$${node_arch}.tar.gz"
+        tar -xvzf histogram-quantile.tar.gz
+        mv histogram-quantile /var/lib/clickhouse/user_scripts/histogramQuantile
+    restart: on-failure
+    volumes:
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+  zookeeper-1:
+    !!merge <<: *zookeeper-defaults
+    container_name: signoz-zookeeper-1
+    # ports:
+    #   - "2181:2181"
+    #   - "2888:2888"
+    #   - "3888:3888"
+    volumes:
+      - zookeeper-1:/bitnami/zookeeper
+    environment:
+      - ZOO_SERVER_ID=1
+      - ALLOW_ANONYMOUS_LOGIN=yes
+      - ZOO_AUTOPURGE_INTERVAL=1
+      - ZOO_ENABLE_PROMETHEUS_METRICS=yes
+      - ZOO_PROMETHEUS_METRICS_PORT_NUMBER=9141
+  clickhouse:
+    !!merge <<: *clickhouse-defaults
+    container_name: signoz-clickhouse
+    # ports:
+    #   - "9000:9000"
+    #   - "8123:8123"
+    #   - "9181:9181"
+    volumes:
+      - ../common/clickhouse/config.xml:/etc/clickhouse-server/config.xml
+      - ../common/clickhouse/users.xml:/etc/clickhouse-server/users.xml
+      - ../common/clickhouse/custom-function.xml:/etc/clickhouse-server/custom-function.xml
+      - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/
+      - ../common/clickhouse/cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
+      - clickhouse:/var/lib/clickhouse/
+      # - ../common/clickhouse/storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  signoz:
+    !!merge <<: *db-depend
+    image: signoz/signoz:${VERSION:-v0.129.0}
+    container_name: signoz
+    ports:
+      - "8080:8080" # signoz port
+    volumes:
+      - sqlite:/var/lib/signoz/
+    environment:
+      - SIGNOZ_ALERTMANAGER_PROVIDER=signoz
+      - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db
+      - SIGNOZ_TOKENIZER_JWT_SECRET=secret
+    healthcheck:
+      test:
+        - CMD
+        - wget
+        - --spider
+        - -q
+        - localhost:8080/api/v1/health
+      interval: 30s
+      timeout: 5s
+      retries: 3
+  otel-collector:
+    !!merge <<: *db-depend
+    image: signoz/signoz-otel-collector:${OTELCOL_TAG:-v0.144.5}
+    container_name: signoz-otel-collector
+    entrypoint:
+      - /bin/sh
+    command:
+      - -c
+      - |
+        /signoz-otel-collector migrate sync check &&
+        /signoz-otel-collector --config=/etc/otel-collector-config.yaml --manager-config=/etc/manager-config.yaml --copy-path=/var/tmp/collector-config.yaml
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
+      - ../common/signoz/otel-collector-opamp-config.yaml:/etc/manager-config.yaml
+    environment:
+      - OTEL_RESOURCE_ATTRIBUTES=host.name=signoz-host,os.type=linux
+      - LOW_CARDINAL_EXCEPTION_GROUPING=false
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true
+      - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m
+    ports:
+      # - "1777:1777"     # pprof extension
+      - "4317:4317" # OTLP gRPC receiver
+      - "4318:4318" # OTLP HTTP receiver
+  signoz-telemetrystore-migrator:
+    !!merge <<: *db-depend
+    image: signoz/signoz-otel-collector:${OTELCOL_TAG:-v0.144.5}
+    container_name: signoz-telemetrystore-migrator
+    environment:
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_DSN=tcp://clickhouse:9000
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_CLUSTER=cluster
+      - SIGNOZ_OTEL_COLLECTOR_CLICKHOUSE_REPLICATION=true
+      - SIGNOZ_OTEL_COLLECTOR_TIMEOUT=10m
+    entrypoint:
+      - /bin/sh
+    command:
+      - -c
+      - |
+        /signoz-otel-collector migrate bootstrap &&
+        /signoz-otel-collector migrate sync up &&
+        /signoz-otel-collector migrate async up
+    restart: on-failure
+networks:
+  signoz-net:
+    name: signoz-net
+volumes:
+  clickhouse:
+    name: signoz-clickhouse
+  sqlite:
+    name: signoz-sqlite
+  zookeeper-1:
+    name: signoz-zookeeper-1

deploy/docker/otel-collector-config.yaml

@@ -0,0 +1,118 @@
+connectors:
+  signozmeter:
+    metrics_flush_interval: 1h
+    dimensions:
+      - name: service.name
+      - name: deployment.environment
+      - name: host.name
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+  prometheus:
+    config:
+      global:
+        scrape_interval: 60s
+      scrape_configs:
+        - job_name: otel-collector
+          static_configs:
+          - targets:
+              - localhost:8888
+            labels:
+              job_name: otel-collector
+processors:
+  batch:
+    send_batch_size: 10000
+    send_batch_max_size: 11000
+    timeout: 10s
+  batch/meter:
+    send_batch_max_size: 25000
+    send_batch_size: 20000
+    timeout: 1s
+  resourcedetection:
+    # Using OTEL_RESOURCE_ATTRIBUTES envvar, env detector adds custom labels.
+    detectors: [env, system]
+    timeout: 2s
+  signozspanmetrics/delta:
+    metrics_exporter: signozclickhousemetrics
+    metrics_flush_interval: 60s
+    latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
+    dimensions_cache_size: 100000
+    aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA
+    enable_exp_histogram: true
+    dimensions:
+      - name: service.namespace
+        default: default
+      - name: deployment.environment
+        default: default
+      # This is added to ensure the uniqueness of the timeseries
+      # Otherwise, identical timeseries produced by multiple replicas of
+      # collectors result in incorrect APM metrics
+      - name: signoz.collector.id
+      - name: service.version
+      - name: browser.platform
+      - name: browser.mobile
+      - name: k8s.cluster.name
+      - name: k8s.node.name
+      - name: k8s.namespace.name
+      - name: host.name
+      - name: host.type
+      - name: container.name
+extensions:
+  health_check:
+    endpoint: 0.0.0.0:13133
+  pprof:
+    endpoint: 0.0.0.0:1777
+exporters:
+  clickhousetraces:
+    datasource: tcp://clickhouse:9000/signoz_traces
+    low_cardinal_exception_grouping: ${env:LOW_CARDINAL_EXCEPTION_GROUPING}
+    use_new_schema: true
+  signozclickhousemetrics:
+    dsn: tcp://clickhouse:9000/signoz_metrics
+  clickhouselogsexporter:
+    dsn: tcp://clickhouse:9000/signoz_logs
+    timeout: 10s
+    use_new_schema: true
+  signozclickhousemeter:
+    dsn: tcp://clickhouse:9000/signoz_meter
+    timeout: 45s
+    sending_queue:
+      enabled: false
+  metadataexporter:
+    cache:
+      provider: in_memory
+    dsn: tcp://clickhouse:9000/signoz_metadata
+    enabled: true
+    timeout: 45s
+service:
+  telemetry:
+    logs:
+      encoding: json
+  extensions:
+    - health_check
+    - pprof
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [signozspanmetrics/delta, batch]
+      exporters: [clickhousetraces, metadataexporter, signozmeter]
+    metrics:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [signozclickhousemetrics, metadataexporter, signozmeter]
+    metrics/prometheus:
+      receivers: [prometheus]
+      processors: [batch]
+      exporters: [signozclickhousemetrics, metadataexporter, signozmeter]
+    logs:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [clickhouselogsexporter, metadataexporter, signozmeter]
+    metrics/meter:
+      receivers: [signozmeter]
+      processors: [batch/meter]
+      exporters: [signozclickhousemeter]

deploy/install.sh

@@ -0,0 +1,563 @@
+#!/bin/bash
+
+set -o errexit
+
+# Variables
+BASE_DIR="$(dirname "$(readlink -f "$0")")"
+DOCKER_STANDALONE_DIR="docker"
+DOCKER_SWARM_DIR="docker-swarm" # TODO: Add docker swarm support
+
+# Regular Colors
+Black='\033[0;30m'        # Black
+Red='\[\e[0;31m\]'        # Red
+Green='\033[0;32m'        # Green
+Yellow='\033[0;33m'       # Yellow
+Blue='\033[0;34m'         # Blue
+Purple='\033[0;35m'       # Purple
+Cyan='\033[0;36m'         # Cyan
+White='\033[0;37m'        # White
+NC='\033[0m' # No Color
+
+is_command_present() {
+    type "$1" >/dev/null 2>&1
+}
+
+# Check whether 'wget' command exists.
+has_wget() {
+    has_cmd wget
+}
+
+# Check whether 'curl' command exists.
+has_curl() {
+    has_cmd curl
+}
+
+# Check whether the given command exists.
+has_cmd() {
+    command -v "$1" > /dev/null 2>&1
+}
+
+# Check if docker compose plugin is present
+has_docker_compose_plugin() {
+    docker compose version > /dev/null 2>&1
+}
+
+is_mac() {
+    [[ $OSTYPE == darwin* ]]
+}
+
+is_arm64(){
+    [[ `uname -m` == 'arm64' || `uname -m` == 'aarch64' ]]
+}
+
+check_os() {
+    if is_mac; then
+        package_manager="brew"
+        desired_os=1
+        os="Mac"
+        return
+    fi
+
+    if is_arm64; then
+        arch="arm64"
+        arch_official="aarch64"
+    else
+        arch="amd64"
+        arch_official="x86_64"
+    fi
+
+    platform=$(uname -s | tr '[:upper:]' '[:lower:]')
+
+    os_name="$(cat /etc/*-release | awk -F= '$1 == "NAME" { gsub(/"/, ""); print $2; exit }')"
+
+    case "$os_name" in
+        Ubuntu*|Pop!_OS)
+            desired_os=1
+            os="ubuntu"
+            package_manager="apt-get"
+            ;;
+        Amazon\ Linux*)
+            desired_os=1
+            os="amazon linux"
+            package_manager="yum"
+            ;;
+        Debian*)
+            desired_os=1
+            os="debian"
+            package_manager="apt-get"
+            ;;
+        Linux\ Mint*)
+            desired_os=1
+            os="linux mint"
+            package_manager="apt-get"
+            ;;
+        Red\ Hat*)
+            desired_os=1
+            os="rhel"
+            package_manager="yum"
+            ;;
+        CentOS*)
+            desired_os=1
+            os="centos"
+            package_manager="yum"
+            ;;
+        Rocky*)
+            desired_os=1
+            os="centos"
+            package_manager="yum"
+            ;;
+        SLES*)
+            desired_os=1
+            os="sles"
+            package_manager="zypper"
+            ;;
+        openSUSE*)
+            desired_os=1
+            os="opensuse"
+            package_manager="zypper"
+            ;;
+        *)
+            desired_os=0
+            os="Not Found: $os_name"
+    esac
+}
+
+
+# This function checks if the relevant ports required by SigNoz are available or not
+# The script should error out in case they aren't available
+check_ports_occupied() {
+    local port_check_output
+    local ports_pattern="8080|4317"
+
+    if is_mac; then
+        port_check_output="$(netstat -anp tcp | awk '$6 == "LISTEN" && $4 ~ /^.*\.('"$ports_pattern"')$/')"
+    elif is_command_present ss; then
+        # The `ss` command seems to be a better/faster version of `netstat`, but is not available on all Linux
+        # distributions by default. Other distributions have `ss` but no `netstat`. So, we try for `ss` first, then
+        # fallback to `netstat`.
+        port_check_output="$(ss --all --numeric --tcp | awk '$1 == "LISTEN" && $4 ~ /^.*:('"$ports_pattern"')$/')"
+    elif is_command_present netstat; then
+        port_check_output="$(netstat --all --numeric --tcp | awk '$6 == "LISTEN" && $4 ~ /^.*:('"$ports_pattern"')$/')"
+    fi
+
+    if [[ -n $port_check_output ]]; then
+        send_event "port_not_available"
+
+        echo "+++++++++++ ERROR ++++++++++++++++++++++"
+        echo "SigNoz requires ports 8080 & 4317 to be open. Please shut down any other service(s) that may be running on these ports."
+        echo "You can run SigNoz on another port following this guide https://signoz.io/docs/install/troubleshooting/"
+        echo "++++++++++++++++++++++++++++++++++++++++"
+        echo ""
+        exit 1
+    fi
+}
+
+install_docker() {
+    echo "++++++++++++++++++++++++"
+    echo "Setting up docker repos"
+
+
+    if [[ $package_manager == apt-get ]]; then
+        apt_cmd="$sudo_cmd apt-get --yes --quiet"
+        $apt_cmd update
+        $apt_cmd install software-properties-common gnupg-agent
+        curl -fsSL "https://download.docker.com/linux/$os/gpg" | $sudo_cmd apt-key add -
+        $sudo_cmd add-apt-repository \
+            "deb [arch=$arch] https://download.docker.com/linux/$os $(lsb_release -cs) stable"
+        $apt_cmd update
+        echo "Installing docker"
+        $apt_cmd install docker-ce docker-ce-cli containerd.io
+    elif [[ $package_manager == zypper ]]; then
+        zypper_cmd="$sudo_cmd zypper --quiet --no-gpg-checks --non-interactive"
+        echo "Installing docker"
+        if [[ $os == sles ]]; then
+            os_sp="$(cat /etc/*-release | awk -F= '$1 == "VERSION_ID" { gsub(/"/, ""); print $2; exit }')"
+            os_arch="$(uname -i)"
+            SUSEConnect -p sle-module-containers/$os_sp/$os_arch -r ''
+        fi
+        $zypper_cmd install docker docker-runc containerd
+        $sudo_cmd systemctl enable docker.service
+    elif [[ $package_manager == yum && $os == 'amazon linux' ]]; then
+        echo
+        echo "Amazon Linux detected ... "
+        echo
+        # yum install docker
+        # service docker start
+        $sudo_cmd yum install -y amazon-linux-extras
+        $sudo_cmd amazon-linux-extras enable docker
+        $sudo_cmd yum install -y docker
+    else
+
+        yum_cmd="$sudo_cmd yum --assumeyes --quiet"
+        $yum_cmd install yum-utils
+        $sudo_cmd yum-config-manager --add-repo https://download.docker.com/linux/$os/docker-ce.repo
+        echo "Installing docker"
+        $yum_cmd install docker-ce docker-ce-cli containerd.io
+    fi
+}
+
+compose_version () {
+    local compose_version
+    compose_version="$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep 'tag_name' | cut -d\" -f4)"
+    echo "${compose_version:-v2.18.1}"
+}
+
+install_docker_compose() {
+    if [[ $package_manager == "apt-get" || $package_manager == "zypper" || $package_manager == "yum" ]]; then
+        if [[ ! -f /usr/bin/docker-compose ]];then
+            echo "++++++++++++++++++++++++"
+            echo "Installing docker-compose"
+            compose_url="https://github.com/docker/compose/releases/download/$(compose_version)/docker-compose-$platform-$arch_official"
+            echo "Downloading docker-compose from $compose_url"
+            $sudo_cmd curl -L "$compose_url" -o /usr/local/bin/docker-compose
+            $sudo_cmd chmod +x /usr/local/bin/docker-compose
+            $sudo_cmd ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
+            echo "docker-compose installed!"
+            echo ""
+        fi
+    else
+        send_event "docker_compose_not_found"
+
+        echo "+++++++++++ IMPORTANT READ ++++++++++++++++++++++"
+        echo "docker-compose not found! Please install docker-compose first and then continue with this installation."
+        echo "Refer https://docs.docker.com/compose/install/ for installing docker-compose."
+        echo "+++++++++++++++++++++++++++++++++++++++++++++++++"
+        exit 1
+    fi
+}
+
+start_docker() {
+    echo -e "🐳 Starting Docker ...\n"
+    if [[ $os == "Mac" ]]; then
+        open --background -a Docker && while ! docker system info > /dev/null 2>&1; do sleep 1; done
+    else
+        if ! $sudo_cmd systemctl is-active docker.service > /dev/null; then
+            echo "Starting docker service"
+            $sudo_cmd systemctl start docker.service
+        fi
+        if [[ -z $sudo_cmd ]]; then
+            if ! docker ps > /dev/null && true; then
+                request_sudo
+            fi
+        fi
+    fi
+}
+
+wait_for_containers_start() {
+    local timeout=$1
+
+    # The while loop is important because for-loops don't work for dynamic values
+    while [[ $timeout -gt 0 ]]; do
+        status_code="$(curl -s -o /dev/null -w "%{http_code}" "http://localhost:8080/api/v1/health?live=1" || true)"
+        if [[ status_code -eq 200 ]]; then
+            break
+        else
+            echo -ne "Waiting for all containers to start. This check will timeout in $timeout seconds ...\r\c"
+        fi
+        ((timeout--))
+        sleep 1
+    done
+
+    echo ""
+}
+
+bye() {  # Prints a friendly good bye message and exits the script.
+    # Switch back to the original directory
+    popd > /dev/null 2>&1
+    if [[ "$?" -ne 0 ]]; then
+        set +o errexit
+
+        echo "🔴 The containers didn't seem to start correctly. Please run the following command to check containers that may have errored out:"
+        echo ""
+        echo -e "cd ${DOCKER_STANDALONE_DIR}"
+        echo -e "$sudo_cmd $docker_compose_cmd ps -a"
+
+        echo "Please read our troubleshooting guide https://signoz.io/docs/install/troubleshooting/"
+        echo "or reach us for support in #help channel in our Slack Community https://signoz.io/slack"
+        echo "++++++++++++++++++++++++++++++++++++++++"
+
+        if [[ $email == "" ]]; then
+            echo -e "\n📨 Please share your email to receive support with the installation"
+            read -rp 'Email: ' email
+
+            while [[ $email == "" ]]
+            do
+                read -rp 'Email: ' email
+            done
+        fi
+
+        send_event "installation_support"
+
+
+        echo ""
+        echo -e "\nWe will reach out to you at the email provided shortly, Exiting for now. Bye! 👋 \n"
+        exit 0
+    fi
+}
+
+request_sudo() {
+    if hash sudo 2>/dev/null; then
+        echo -e "\n\n🙇 We will need sudo access to complete the installation."
+        if (( $EUID != 0 )); then
+            sudo_cmd="sudo"
+            echo -e "Please enter your sudo password, if prompted."
+            if ! $sudo_cmd -l | grep -e "NOPASSWD: ALL" > /dev/null && ! $sudo_cmd -v; then
+                echo "Need sudo privileges to proceed with the installation."
+                exit 1;
+            fi
+
+            echo -e "Got it! Thanks!! 🙏\n"
+            echo -e "Okay! We will bring up the SigNoz cluster from here 🚀\n"
+        fi
+	fi
+}
+
+echo ""
+echo -e "👋 Thank you for trying out SigNoz! "
+echo ""
+
+sudo_cmd=""
+docker_compose_cmd=""
+
+# Check sudo permissions
+if (( $EUID != 0 )); then
+    echo "🟡 Running installer with non-sudo permissions."
+    echo "   In case of any failure or prompt, please consider running the script with sudo privileges."
+    echo ""
+else
+    sudo_cmd="sudo"
+fi
+
+# Checking OS and assigning package manager
+desired_os=0
+os=""
+email=""
+echo -e "🌏 Detecting your OS ...\n"
+check_os
+
+# Obtain unique installation id
+# sysinfo="$(uname -a)"
+# if [[ $? -ne 0 ]]; then
+#     uuid="$(uuidgen)"
+#     uuid="${uuid:-$(cat /proc/sys/kernel/random/uuid)}"
+#     sysinfo="${uuid:-$(cat /proc/sys/kernel/random/uuid)}"
+# fi
+if ! sysinfo="$(uname -a)"; then
+    uuid="$(uuidgen)"
+    uuid="${uuid:-$(cat /proc/sys/kernel/random/uuid)}"
+    sysinfo="${uuid:-$(cat /proc/sys/kernel/random/uuid)}"
+fi
+
+digest_cmd=""
+if hash shasum 2>/dev/null; then
+    digest_cmd="shasum -a 256"
+elif hash sha256sum 2>/dev/null; then
+    digest_cmd="sha256sum"
+elif hash openssl 2>/dev/null; then
+    digest_cmd="openssl dgst -sha256"
+fi
+
+if [[ -z $digest_cmd ]]; then
+    SIGNOZ_INSTALLATION_ID="$sysinfo"
+else
+    SIGNOZ_INSTALLATION_ID=$(echo "$sysinfo" | $digest_cmd | grep -E -o '[a-zA-Z0-9]{64}')
+fi
+
+setup_type='clickhouse'
+
+# Run bye if failure happens
+trap bye EXIT
+
+URL="https://api.segment.io/v1/track"
+HEADER_1="Content-Type: application/json"
+HEADER_2="Authorization: Basic OWtScko3b1BDR1BFSkxGNlFqTVBMdDVibGpGaFJRQnI="
+
+send_event() {
+    error=""
+
+    case "$1" in
+        'install_started')
+            event="Installation Started"
+            ;;
+        'os_not_supported')
+            event="Installation Error"
+            error="OS Not Supported"
+            ;;
+        'docker_not_installed')
+            event="Installation Error"
+            error="Docker not installed"
+            ;;
+        'docker_compose_not_found')
+            event="Installation Error"
+            event="Docker Compose not found"
+            ;;
+        'port_not_available')
+            event="Installation Error"
+            error="port not available"
+            ;;
+        'installation_error_checks')
+            event="Installation Error - Checks"
+            error="Containers not started"
+            others='"data": "some_checks",'
+            ;;
+        'installation_support')
+            event="Installation Support"
+            others='"email": "'"$email"'",'
+            ;;
+        'installation_success')
+            event="Installation Success"
+            ;;
+        'identify_successful_installation')
+            event="Identify Successful Installation"
+            others='"email": "'"$email"'",'
+            ;;
+        *)
+            print_error "unknown event type: $1"
+            exit 1
+            ;;
+    esac
+
+    if [[ "$error" != "" ]]; then
+        error='"error": "'"$error"'", '
+    fi
+
+    DATA='{ "anonymousId": "'"$SIGNOZ_INSTALLATION_ID"'", "event": "'"$event"'", "properties": { "os": "'"$os"'", '"$error $others"' "setup_type": "'"$setup_type"'" } }'
+
+    if has_curl; then
+        curl -sfL -d "$DATA" --header "$HEADER_1" --header "$HEADER_2" "$URL" > /dev/null 2>&1
+    elif has_wget; then
+        wget -q --post-data="$DATA" --header "$HEADER_1" --header "$HEADER_2" "$URL" > /dev/null 2>&1
+    fi
+}
+
+send_event "install_started"
+
+if [[ $desired_os -eq 0 ]]; then
+    send_event "os_not_supported"
+fi
+
+# Check is Docker daemon is installed and available. If not, the install & start Docker for Linux machines. We cannot automatically install Docker Desktop on Mac OS
+if ! is_command_present docker; then
+
+    if [[ $package_manager == "apt-get" || $package_manager == "zypper" || $package_manager == "yum" ]]; then
+        request_sudo
+        install_docker
+        # enable docker without sudo from next reboot
+        sudo usermod -aG docker "${USER}"
+    elif is_mac; then
+        echo ""
+        echo "+++++++++++ IMPORTANT READ ++++++++++++++++++++++"
+        echo "Docker Desktop must be installed manually on Mac OS to proceed. Docker can only be installed automatically on Ubuntu / openSUSE / SLES / Redhat / Cent OS"
+        echo "https://docs.docker.com/docker-for-mac/install/"
+        echo "++++++++++++++++++++++++++++++++++++++++++++++++"
+
+        send_event "docker_not_installed"
+        exit 1
+    else
+        echo ""
+        echo "+++++++++++ IMPORTANT READ ++++++++++++++++++++++"
+        echo "Docker must be installed manually on your machine to proceed. Docker can only be installed automatically on Ubuntu / openSUSE / SLES / Redhat / Cent OS"
+        echo "https://docs.docker.com/get-docker/"
+        echo "++++++++++++++++++++++++++++++++++++++++++++++++"
+
+        send_event "docker_not_installed"
+        exit 1
+    fi
+fi
+
+if has_docker_compose_plugin; then
+    echo "docker compose plugin is present, using it"
+    docker_compose_cmd="docker compose"
+# Install docker-compose
+else
+    docker_compose_cmd="docker-compose"
+    if ! is_command_present docker-compose; then
+        request_sudo
+        install_docker_compose
+    fi
+fi
+
+start_docker
+
+# Switch to the Docker Standalone directory
+pushd "${BASE_DIR}/${DOCKER_STANDALONE_DIR}" > /dev/null 2>&1
+
+# check for open ports, if signoz is not installed
+if is_command_present docker-compose; then
+    if $sudo_cmd $docker_compose_cmd ps | grep "signoz" | grep -q "healthy" > /dev/null 2>&1; then
+        echo "SigNoz already installed, skipping the occupied ports check"
+    else
+        check_ports_occupied
+    fi
+fi
+
+echo ""
+echo -e "\n🟡 Pulling the latest container images for SigNoz.\n"
+$sudo_cmd $docker_compose_cmd pull
+
+echo ""
+echo "🟡 Starting the SigNoz containers. It may take a few minutes ..."
+echo
+# The $docker_compose_cmd command does some nasty stuff for the `--detach` functionality. So we add a `|| true` so that the
+# script doesn't exit because this command looks like it failed to do it's thing.
+$sudo_cmd $docker_compose_cmd up --detach --remove-orphans || true
+
+wait_for_containers_start 60
+echo ""
+
+if [[ $status_code -ne 200 ]]; then
+    echo "+++++++++++ ERROR ++++++++++++++++++++++"
+    echo "🔴 The containers didn't seem to start correctly. Please run the following command to check containers that may have errored out:"
+    echo ""
+
+    echo "cd ${DOCKER_STANDALONE_DIR}"
+    echo "$sudo_cmd $docker_compose_cmd ps -a"
+    echo ""
+
+    echo "Try bringing down the containers and retrying the installation"
+    echo "cd ${DOCKER_STANDALONE_DIR}"
+    echo "$sudo_cmd $docker_compose_cmd down -v"
+    echo ""
+
+    echo "Please read our troubleshooting guide https://signoz.io/docs/install/troubleshooting/"
+    echo "or reach us on SigNoz for support https://signoz.io/slack"
+    echo "++++++++++++++++++++++++++++++++++++++++"
+
+    send_event "installation_error_checks"
+    exit 1
+
+else
+    send_event "installation_success"
+
+    echo "++++++++++++++++++ SUCCESS ++++++++++++++++++++++"
+    echo ""
+    echo "🟢 Your installation is complete!"
+    echo ""
+    echo -e "🟢 SigNoz is running on http://localhost:8080"
+    echo ""
+    echo "ℹ️  By default, retention period is set to 15 days for logs and traces, and 30 days for metrics."
+    echo -e "To change this, navigate to the General tab on the Settings page of SigNoz UI. For more details, refer to https://signoz.io/docs/userguide/retention-period \n"
+
+    echo "ℹ️  To bring down SigNoz and clean volumes:"
+    echo ""
+    echo "cd ${DOCKER_STANDALONE_DIR}"
+    echo "$sudo_cmd $docker_compose_cmd down -v"
+
+    echo ""
+    echo "+++++++++++++++++++++++++++++++++++++++++++++++++"
+    echo ""
+    echo "👉 Need help in Getting Started?"
+    echo -e "Join us on Slack https://signoz.io/slack"
+    echo ""
+    echo -e "\n📨 Please share your email to receive support & updates about SigNoz!"
+    read -rp 'Email: ' email
+
+    while [[ $email == "" ]]
+    do
+        read -rp 'Email: ' email
+    done
+
+    send_event "identify_successful_installation"
+fi
+
+echo -e "\n🙏 Thank you!\n"

docs/api/openapi.yml

@@ -6992,6 +6992,16 @@ components:
       required:
       - items
       type: object
+    SpantypesGettableSpanMapperTest:
+      properties:
+        spans:
+          items:
+            $ref: '#/components/schemas/SpantypesSpanMapperTestSpan'
+          nullable: true
+          type: array
+      required:
+      - spans
+      type: object
     SpantypesGettableTraceAggregations:
       properties:
         aggregations:
@@ -7079,6 +7089,39 @@ components:
       - name
       - condition
       type: object
+    SpantypesPostableSpanMapperTest:
+      properties:
+        groups:
+          items:
+            $ref: '#/components/schemas/SpantypesPostableSpanMapperTestGroup'
+          nullable: true
+          type: array
+        spans:
+          items:
+            $ref: '#/components/schemas/SpantypesSpanMapperTestSpan'
+          nullable: true
+          type: array
+      required:
+      - spans
+      - groups
+      type: object
+    SpantypesPostableSpanMapperTestGroup:
+      properties:
+        condition:
+          $ref: '#/components/schemas/SpantypesSpanMapperGroupCondition'
+        enabled:
+          type: boolean
+        mappers:
+          items:
+            $ref: '#/components/schemas/SpantypesPostableSpanMapper'
+          nullable: true
+          type: array
+        name:
+          type: string
+      required:
+      - name
+      - condition
+      type: object
     SpantypesPostableTraceAggregations:
       properties:
         aggregations:
@@ -7240,6 +7283,17 @@ components:
       - operation
       - priority
       type: object
+    SpantypesSpanMapperTestSpan:
+      properties:
+        attributes:
+          additionalProperties: {}
+          nullable: true
+          type: object
+        resource:
+          additionalProperties: {}
+          nullable: true
+          type: object
+      type: object
     SpantypesUpdatableSpanMapper:
       properties:
         config:
@@ -12797,6 +12851,69 @@ paths:
       summary: Update a span mapper
       tags:
       - spanmapper
+  /api/v1/span_mapper_groups/test:
+    post:
+      deprecated: false
+      description: Tests how span mappers would transform sample spans
+      operationId: TestSpanMappers
+      requestBody:
+        content:
+          application/json:
+            schema:
+              $ref: '#/components/schemas/SpantypesPostableSpanMapperTest'
+      responses:
+        "200":
+          content:
+            application/json:
+              schema:
+                properties:
+                  data:
+                    $ref: '#/components/schemas/SpantypesGettableSpanMapperTest'
+                  status:
+                    type: string
+                required:
+                - status
+                - data
+                type: object
+          description: OK
+        "400":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/RenderErrorResponse'
+          description: Bad Request
+        "401":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/RenderErrorResponse'
+          description: Unauthorized
+        "403":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/RenderErrorResponse'
+          description: Forbidden
+        "404":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/RenderErrorResponse'
+          description: Not Found
+        "500":
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/RenderErrorResponse'
+          description: Internal Server Error
+      security:
+      - api_key:
+        - VIEWER
+      - tokenizer:
+        - VIEWER
+      summary: Test span mappers against sample spans
+      tags:
+      - spanmapper
   /api/v1/stats:
     get:
       deprecated: false

frontend/src/api/generated/services/sigNoz.schemas.ts

@@ -8135,6 +8135,44 @@ export interface SpantypesGettableSpanMapperGroupsDTO {
 	items: SpantypesSpanMapperGroupDTO[];
 }
 
+export type SpantypesSpanMapperTestSpanDTOAttributesAnyOf = {
+	[key: string]: unknown;
+};
+
+/**
+ * @nullable
+ */
+export type SpantypesSpanMapperTestSpanDTOAttributes =
+	SpantypesSpanMapperTestSpanDTOAttributesAnyOf | null;
+
+export type SpantypesSpanMapperTestSpanDTOResourceAnyOf = {
+	[key: string]: unknown;
+};
+
+/**
+ * @nullable
+ */
+export type SpantypesSpanMapperTestSpanDTOResource =
+	SpantypesSpanMapperTestSpanDTOResourceAnyOf | null;
+
+export interface SpantypesSpanMapperTestSpanDTO {
+	/**
+	 * @type object,null
+	 */
+	attributes?: SpantypesSpanMapperTestSpanDTOAttributes;
+	/**
+	 * @type object,null
+	 */
+	resource?: SpantypesSpanMapperTestSpanDTOResource;
+}
+
+export interface SpantypesGettableSpanMapperTestDTO {
+	/**
+	 * @type array,null
+	 */
+	spans: SpantypesSpanMapperTestSpanDTO[] | null;
+}
+
 export enum SpantypesSpanAggregationTypeDTO {
 	span_count = 'span_count',
 	execution_time_percentage = 'execution_time_percentage',
@@ -8430,6 +8468,33 @@ export interface SpantypesPostableSpanMapperGroupDTO {
 	name: string;
 }
 
+export interface SpantypesPostableSpanMapperTestGroupDTO {
+	condition: SpantypesSpanMapperGroupConditionDTO | null;
+	/**
+	 * @type boolean
+	 */
+	enabled?: boolean;
+	/**
+	 * @type array,null
+	 */
+	mappers?: SpantypesPostableSpanMapperDTO[] | null;
+	/**
+	 * @type string
+	 */
+	name: string;
+}
+
+export interface SpantypesPostableSpanMapperTestDTO {
+	/**
+	 * @type array,null
+	 */
+	groups: SpantypesPostableSpanMapperTestGroupDTO[] | null;
+	/**
+	 * @type array,null
+	 */
+	spans: SpantypesSpanMapperTestSpanDTO[] | null;
+}
+
 export interface SpantypesSpanAggregationDTO {
 	aggregation: SpantypesSpanAggregationTypeDTO;
 	field: TelemetrytypesTelemetryFieldKeyDTO;
@@ -9798,6 +9863,14 @@ export type UpdateSpanMapperPathParameters = {
 	groupId: string;
 	mapperId: string;
 };
+export type TestSpanMappers200 = {
+	data: SpantypesGettableSpanMapperTestDTO;
+	/**
+	 * @type string
+	 */
+	status: string;
+};
+
 export type GetStats200Data = { [key: string]: unknown };
 
 export type GetStats200 = {

frontend/src/api/generated/services/spanmapper/index.ts

@@ -30,8 +30,10 @@ import type {
 	RenderErrorResponseDTO,
 	SpantypesPostableSpanMapperDTO,
 	SpantypesPostableSpanMapperGroupDTO,
+	SpantypesPostableSpanMapperTestDTO,
 	SpantypesUpdatableSpanMapperDTO,
 	SpantypesUpdatableSpanMapperGroupDTO,
+	TestSpanMappers200,
 	UpdateSpanMapperGroupPathParameters,
 	UpdateSpanMapperPathParameters,
 } from '../sigNoz.schemas';
@@ -780,3 +782,86 @@ export const useUpdateSpanMapper = <
 > => {
 	return useMutation(getUpdateSpanMapperMutationOptions(options));
 };
+/**
+ * Tests how span mappers would transform sample spans
+ * @summary Test span mappers against sample spans
+ */
+export const testSpanMappers = (
+	spantypesPostableSpanMapperTestDTO?: BodyType<SpantypesPostableSpanMapperTestDTO>,
+	signal?: AbortSignal,
+) => {
+	return GeneratedAPIInstance<TestSpanMappers200>({
+		url: `/api/v1/span_mapper_groups/test`,
+		method: 'POST',
+		headers: { 'Content-Type': 'application/json' },
+		data: spantypesPostableSpanMapperTestDTO,
+		signal,
+	});
+};
+
+export const getTestSpanMappersMutationOptions = <
+	TError = ErrorType<RenderErrorResponseDTO>,
+	TContext = unknown,
+>(options?: {
+	mutation?: UseMutationOptions<
+		Awaited<ReturnType<typeof testSpanMappers>>,
+		TError,
+		{ data?: BodyType<SpantypesPostableSpanMapperTestDTO> },
+		TContext
+	>;
+}): UseMutationOptions<
+	Awaited<ReturnType<typeof testSpanMappers>>,
+	TError,
+	{ data?: BodyType<SpantypesPostableSpanMapperTestDTO> },
+	TContext
+> => {
+	const mutationKey = ['testSpanMappers'];
+	const { mutation: mutationOptions } = options
+		? options.mutation &&
+			'mutationKey' in options.mutation &&
+			options.mutation.mutationKey
+			? options
+			: { ...options, mutation: { ...options.mutation, mutationKey } }
+		: { mutation: { mutationKey } };
+
+	const mutationFn: MutationFunction<
+		Awaited<ReturnType<typeof testSpanMappers>>,
+		{ data?: BodyType<SpantypesPostableSpanMapperTestDTO> }
+	> = (props) => {
+		const { data } = props ?? {};
+
+		return testSpanMappers(data);
+	};
+
+	return { mutationFn, ...mutationOptions };
+};
+
+export type TestSpanMappersMutationResult = NonNullable<
+	Awaited<ReturnType<typeof testSpanMappers>>
+>;
+export type TestSpanMappersMutationBody =
+	| BodyType<SpantypesPostableSpanMapperTestDTO>
+	| undefined;
+export type TestSpanMappersMutationError = ErrorType<RenderErrorResponseDTO>;
+
+/**
+ * @summary Test span mappers against sample spans
+ */
+export const useTestSpanMappers = <
+	TError = ErrorType<RenderErrorResponseDTO>,
+	TContext = unknown,
+>(options?: {
+	mutation?: UseMutationOptions<
+		Awaited<ReturnType<typeof testSpanMappers>>,
+		TError,
+		{ data?: BodyType<SpantypesPostableSpanMapperTestDTO> },
+		TContext
+	>;
+}): UseMutationResult<
+	Awaited<ReturnType<typeof testSpanMappers>>,
+	TError,
+	{ data?: BodyType<SpantypesPostableSpanMapperTestDTO> },
+	TContext
+> => {
+	return useMutation(getTestSpanMappersMutationOptions(options));
+};

pkg/apiserver/signozapiserver/spanmapper.go

@@ -51,6 +51,26 @@ func (provider *provider) addSpanMapperRoutes(router *mux.Router) error {
 		return err
 	}
 
+	if err := router.Handle("/api/v1/span_mapper_groups/test", handler.New(
+		provider.authzMiddleware.ViewAccess(provider.spanMapperHandler.TestMappers),
+		handler.OpenAPIDef{
+			ID:                  "TestSpanMappers",
+			Tags:                []string{"spanmapper"},
+			Summary:             "Test span mappers against sample spans",
+			Description:         "Tests how span mappers would transform sample spans",
+			Request:             new(spantypes.PostableSpanMapperTest),
+			RequestContentType:  "application/json",
+			Response:            new(spantypes.GettableSpanMapperTest),
+			ResponseContentType: "application/json",
+			SuccessStatusCode:   http.StatusOK,
+			ErrorStatusCodes:    []int{http.StatusBadRequest, http.StatusNotFound},
+			Deprecated:          false,
+			SecuritySchemes:     newSecuritySchemes(types.RoleViewer),
+		},
+	)).Methods(http.MethodPost).GetError(); err != nil {
+		return err
+	}
+
 	if err := router.Handle("/api/v1/span_mapper_groups/{groupId}", handler.New(
 		provider.authzMiddleware.AdminAccess(provider.spanMapperHandler.UpdateGroup),
 		handler.OpenAPIDef{

pkg/modules/spanmapper/implspanmapper/handler.go

@@ -273,6 +273,35 @@ func (h *handler) DeleteMapper(rw http.ResponseWriter, r *http.Request) {
 	render.Success(rw, http.StatusNoContent, nil)
 }
 
+// TestMappers handles POST /api/v1/span_mapper_groups/test.
+func (h *handler) TestMappers(rw http.ResponseWriter, r *http.Request) {
+	ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
+	defer cancel()
+
+	claims, err := authtypes.ClaimsFromContext(ctx)
+	if err != nil {
+		render.Error(rw, err)
+		return
+	}
+
+	orgID := valuer.MustNewUUID(claims.OrgID)
+
+	req := new(spantypes.PostableSpanMapperTest)
+	if err := binding.JSON.BindBody(r.Body, req); err != nil {
+		render.Error(rw, err)
+		return
+	}
+
+	groups := spantypes.NewSpanMapperGroupsWithMappersFromPostable(orgID, req.Groups)
+	out, err := h.module.TestMappers(ctx, orgID, req.Spans, groups)
+	if err != nil {
+		render.Error(rw, err)
+		return
+	}
+
+	render.Success(rw, http.StatusOK, &spantypes.GettableSpanMapperTest{Spans: out})
+}
+
 // groupIDFromPath extracts and validates the {id} or {groupId} path variable.
 func groupIDFromPath(r *http.Request) (valuer.UUID, error) {
 	vars := mux.Vars(r)

pkg/modules/spanmapper/implspanmapper/module.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"encoding/json"
 
+	"github.com/SigNoz/signoz/pkg/errors"
 	"github.com/SigNoz/signoz/pkg/modules/spanmapper"
 	"github.com/SigNoz/signoz/pkg/query-service/agentConf"
 	"github.com/SigNoz/signoz/pkg/types/opamptypes"
@@ -102,6 +103,54 @@ func (module *module) DeleteMapper(ctx context.Context, orgID, groupID, id value
 	return nil
 }
 
+func (module *module) TestMappers(ctx context.Context, orgID valuer.UUID, spans []spantypes.SpanMapperTestSpan, groups []*spantypes.SpanMapperGroupWithMappers) ([]spantypes.SpanMapperTestSpan, error) {
+	if len(spans) == 0 {
+		return nil, errors.New(errors.TypeInvalidInput, spantypes.ErrCodeMappingInvalidInput, "'spans' must contain at least one span")
+	}
+
+	_, err := module.backfillMappers(ctx, orgID, groups)
+	if err != nil {
+		return nil, err
+	}
+
+	// out, _, err := spantypes.SimulateSpanMappersProcessing(ctx, resolved, spans)
+	// if err != nil {
+	// 	return nil, err
+	// }
+	// return out, nil
+	return nil, nil
+}
+
+// backfillMappers loads saved mappers for any group whose Mappers is nil.
+func (module *module) backfillMappers(ctx context.Context, orgID valuer.UUID, groups []*spantypes.SpanMapperGroupWithMappers) ([]*spantypes.SpanMapperGroupWithMappers, error) {
+	// Load all the saved groups for this org, so we can look up by name.
+	savedGroups, err := module.store.ListGroups(ctx, orgID, &spantypes.ListSpanMapperGroupsQuery{})
+	if err != nil {
+		return nil, err
+	}
+	savedByName := make(map[string]*spantypes.SpanMapperGroup, len(savedGroups))
+	for _, g := range savedGroups {
+		savedByName[g.Name] = g
+	}
+
+	// For each group in the request, if Mappers is nil, load the saved mappers for that group name.
+	for _, g := range groups {
+		if g.Mappers != nil {
+			continue
+		}
+		saved, ok := savedByName[g.Group.Name]
+		if !ok {
+			return nil, errors.Newf(errors.TypeInvalidInput, spantypes.ErrCodeMappingGroupNotFound, "no saved group named %q to load mappers from; send 'mappers' for new or edited groups", g.Group.Name)
+		}
+		loaded, err := module.store.ListMappers(ctx, orgID, saved.ID)
+		if err != nil {
+			return nil, err
+		}
+		g.Mappers = loaded
+	}
+	return groups, nil
+}
+
 func (module *module) AgentFeatureType() agentConf.AgentFeatureType {
 	return spantypes.SpanAttrMappingFeatureType
 }

pkg/modules/spanmapper/spanmapper.go

@@ -27,6 +27,7 @@ type Module interface {
 	CreateMapper(ctx context.Context, orgID, groupID valuer.UUID, mapper *spantypes.SpanMapper) error
 	UpdateMapper(ctx context.Context, orgID, groupID, id valuer.UUID, fieldContext spantypes.FieldContext, config *spantypes.SpanMapperConfig, enabled *bool, updatedBy string) error
 	DeleteMapper(ctx context.Context, orgID, groupID, id valuer.UUID) error
+	TestMappers(ctx context.Context, orgID valuer.UUID, spans []spantypes.SpanMapperTestSpan, groups []*spantypes.SpanMapperGroupWithMappers) ([]spantypes.SpanMapperTestSpan, error)
 }
 
 // Handler defines the HTTP handler interface for mapping group and mapper endpoints.
@@ -42,4 +43,5 @@ type Handler interface {
 	CreateMapper(rw http.ResponseWriter, r *http.Request)
 	UpdateMapper(rw http.ResponseWriter, r *http.Request)
 	DeleteMapper(rw http.ResponseWriter, r *http.Request)
+	TestMappers(rw http.ResponseWriter, r *http.Request)
 }

pkg/types/spantypes/spanmappersimulator.go

@@ -0,0 +1,151 @@
+package spantypes
+
+import (
+	"github.com/SigNoz/signoz/pkg/errors"
+	"go.opentelemetry.io/collector/pdata/ptrace"
+)
+
+var (
+	ErrCodeProcessorFactoryMapFailed  = errors.MustNewCode("processor_factory_map_failed")
+	ErrCodeSpanMapperSimulationFailed = errors.MustNewCode("span_mapper_simulation_failed")
+)
+
+// spanInputOrderAttr tags each input span with its index so the simulator
+// output can be sorted back into input order. The collector simulator does
+// not guarantee that traces come out in the order they went in.
+const spanInputOrderAttr = "__signoz_input_idx__"
+
+// SimulateSpanMappersProcessing runs the given spans through an in-memory
+// collector pipeline that hosts signozspanmapperprocessor configured by the
+// supplied groups, and returns the transformed spans. Mirrors
+// SimulatePipelinesProcessing in pkg/query-service/app/logparsingpipeline.
+// func SimulateSpanMappersProcessing(
+// 	ctx context.Context,
+// 	groups []*SpanMapperGroupWithMappers,
+// 	spans []SpanMapperTestSpan,
+// ) (
+// 	[]SpanMapperTestSpan, []string, error,
+// ) {
+// 	enabled := filterEnabledGroupsWithMappers(groups)
+// 	if len(enabled) < 1 {
+// 		return spans, nil, nil
+// 	}
+
+// 	for i := range spans {
+// 		if spans[i].Attributes == nil {
+// 			spans[i].Attributes = map[string]any{}
+// 		}
+// 		spans[i].Attributes[spanInputOrderAttr] = int64(i)
+// 	}
+// 	simulatorInput := SpansToPTraces(spans)
+
+// 	processorFactories, err := otelcol.MakeFactoryMap(signozspanmapperprocessor.NewFactory())
+// 	if err != nil {
+// 		return nil, nil, errors.WrapInternalf(err, ErrCodeProcessorFactoryMapFailed, "could not construct processor factory map")
+// 	}
+
+// 	configGenerator := func(baseConf []byte) ([]byte, error) {
+// 		return GenerateCollectorConfigWithSpanMapperProcessor(baseConf, enabled)
+// 	}
+
+// 	// signozspanmapperprocessor does no batching; spans flow through immediately.
+// 	timeout := 200 * time.Millisecond
+
+// 	outputTraces, collectorErrs, simErr := collectorsimulator.SimulateTracesProcessing(
+// 		ctx,
+// 		processorFactories,
+// 		configGenerator,
+// 		simulatorInput,
+// 		timeout,
+// 	)
+// 	if simErr != nil {
+// 		if errors.Is(simErr, collectorsimulator.ErrInvalidConfig) {
+// 			return nil, nil, errors.WrapInvalidInputf(simErr, errors.CodeInvalidInput, "invalid config")
+// 		}
+// 		return nil, nil, errors.WrapInternalf(simErr, ErrCodeSpanMapperSimulationFailed, "could not simulate span mapper processing")
+// 	}
+
+// 	outputSpans := PTracesToSpans(outputTraces)
+
+// 	sort.Slice(outputSpans, func(i, j int) bool {
+// 		iIdx, _ := outputSpans[i].Attributes[spanInputOrderAttr].(int64)
+// 		jIdx, _ := outputSpans[j].Attributes[spanInputOrderAttr].(int64)
+// 		return iIdx < jIdx
+// 	})
+// 	for _, s := range outputSpans {
+// 		delete(s.Attributes, spanInputOrderAttr)
+// 	}
+
+// 	collectorWarnAndErrorLogs := []string{}
+// 	for _, log := range collectorErrs {
+// 		if log == "" || strings.Contains(log, "featuregate.go") {
+// 			continue
+// 		}
+// 		collectorWarnAndErrorLogs = append(collectorWarnAndErrorLogs, log)
+// 	}
+
+// 	return outputSpans, collectorWarnAndErrorLogs, nil
+// }
+
+// SpansToPTraces packs each input span into its own ptrace.Traces with one
+// ResourceSpans / ScopeSpans / Span carrying its attribute and resource maps.
+func SpansToPTraces(spans []SpanMapperTestSpan) []ptrace.Traces {
+	result := make([]ptrace.Traces, 0, len(spans))
+	for _, s := range spans {
+		td := ptrace.NewTraces()
+		rs := td.ResourceSpans().AppendEmpty()
+		if s.Resource != nil {
+			_ = rs.Resource().Attributes().FromRaw(s.Resource)
+		}
+		sl := rs.ScopeSpans().AppendEmpty()
+		span := sl.Spans().AppendEmpty()
+		if s.Attributes != nil {
+			_ = span.Attributes().FromRaw(s.Attributes)
+		}
+		result = append(result, td)
+	}
+	return result
+}
+
+// PTracesToSpans flattens simulator output back into SpanMapperTestSpan: one
+// entry per individual Span across all ResourceSpans / ScopeSpans.
+func PTracesToSpans(traces []ptrace.Traces) []SpanMapperTestSpan {
+	result := []SpanMapperTestSpan{}
+	for _, td := range traces {
+		rss := td.ResourceSpans()
+		for i := 0; i < rss.Len(); i++ {
+			rs := rss.At(i)
+			resourceAttrs := rs.Resource().Attributes().AsRaw()
+			ilss := rs.ScopeSpans()
+			for j := 0; j < ilss.Len(); j++ {
+				spans := ilss.At(j).Spans()
+				for k := 0; k < spans.Len(); k++ {
+					result = append(result, SpanMapperTestSpan{
+						Attributes: spans.At(k).Attributes().AsRaw(),
+						Resource:   resourceAttrs,
+					})
+				}
+			}
+		}
+	}
+	return result
+}
+
+func filterEnabledGroupsWithMappers(groups []*SpanMapperGroupWithMappers) []*SpanMapperGroupWithMappers {
+	out := make([]*SpanMapperGroupWithMappers, 0, len(groups))
+	for _, gm := range groups {
+		if gm == nil || gm.Group == nil || !gm.Group.Enabled {
+			continue
+		}
+		enabled := make([]*SpanMapper, 0, len(gm.Mappers))
+		for _, m := range gm.Mappers {
+			if m != nil && m.Enabled {
+				enabled = append(enabled, m)
+			}
+		}
+		if len(enabled) > 0 {
+			out = append(out, &SpanMapperGroupWithMappers{Group: gm.Group, Mappers: enabled})
+		}
+	}
+	return out
+}

pkg/types/spantypes/spanmappertest.go

@@ -0,0 +1,53 @@
+package spantypes
+
+import (
+	"github.com/SigNoz/signoz/pkg/valuer"
+)
+
+type SpanMapperTestSpan struct {
+	Attributes map[string]any `json:"attributes"`
+	Resource   map[string]any `json:"resource"`
+}
+
+// Mappers is optional because the module can backfill from the store by Group.Name.
+type PostableSpanMapperTestGroup struct {
+	PostableSpanMapperGroup
+	Mappers []PostableSpanMapper `json:"mappers"`
+}
+
+type PostableSpanMapperTest struct {
+	Spans  []SpanMapperTestSpan          `json:"spans"  required:"true"`
+	Groups []PostableSpanMapperTestGroup `json:"groups" required:"true"`
+}
+
+type GettableSpanMapperTest struct {
+	Spans []SpanMapperTestSpan `json:"spans" required:"true"`
+}
+
+func NewSpanMapperGroupsWithMappersFromPostable(orgID valuer.UUID, in []PostableSpanMapperTestGroup) []*SpanMapperGroupWithMappers {
+	out := make([]*SpanMapperGroupWithMappers, 0, len(in))
+	for _, pg := range in {
+		var mappers []*SpanMapper
+		if pg.Mappers != nil {
+			mappers = make([]*SpanMapper, 0, len(pg.Mappers))
+			for _, pm := range pg.Mappers {
+				mappers = append(mappers, &SpanMapper{
+					Name:         pm.Name,
+					FieldContext: pm.FieldContext,
+					Config:       pm.Config,
+					Enabled:      pm.Enabled,
+				})
+			}
+		}
+		out = append(out, &SpanMapperGroupWithMappers{
+			Group: &SpanMapperGroup{
+				OrgID:     orgID,
+				Name:      pg.Name,
+				Condition: pg.Condition,
+				Enabled:   pg.Enabled,
+			},
+			Mappers: mappers,
+		})
+	}
+	return out
+}

pkg/types/telemetrytypes/field.go

@@ -4,7 +4,6 @@ import (
 	"fmt"
 	"strings"
 
-	"github.com/SigNoz/signoz-otel-collector/exporter/jsontypeexporter"
 	"github.com/SigNoz/signoz/pkg/valuer"
 )
 
@@ -22,7 +21,7 @@ const (
 	// BodyJSONStringSearchPrefix is the prefix used for body JSON search queries.
 	// e.g., "body.status" where "body." is the prefix.
 	BodyJSONStringSearchPrefix = "body."
-	ArraySep                   = jsontypeexporter.ArraySeparator
+	ArraySep                   = "[]."
 	ArraySepSuffix             = "[]"
 	// TODO(Piyush): Remove once we've migrated to the new array syntax.
 	ArrayAnyIndex       = "[*]."

pkg/types/telemetrytypes/json_access_plan.go

@@ -6,7 +6,6 @@ import (
 	"slices"
 	"strings"
 
-	"github.com/SigNoz/signoz-otel-collector/exporter/jsontypeexporter"
 	"github.com/SigNoz/signoz/pkg/errors"
 	"github.com/SigNoz/signoz/pkg/valuer"
 )
@@ -76,7 +75,7 @@ func (n *JSONAccessNode) Alias() string {
 	parentAlias := strings.TrimLeft(n.Parent.Alias(), "`")
 	parentAlias = strings.TrimRight(parentAlias, "`")
 
-	sep := jsontypeexporter.ArraySeparator
+	sep := "[]."
 	if n.Parent.isRoot {
 		sep = "."
 	}