mirror of
https://github.com/SigNoz/signoz.git
synced 2026-10-29 12:30:42 +00:00
Compare commits
4 Commits
docs/e2e-a
...
issue_nerv
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5c24c882d9 | ||
|
|
dbd318feaf | ||
|
|
3c263501b4 | ||
|
|
d04c0a2366 |
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: playwright-test-generator
|
||||
description: Use this agent to convert a SigNoz E2E test plan into Playwright spec files under `tests/e2e/tests/<feature>/`, adding or extending the feature's driver under `tests/e2e/drivers/`. Examples — <example>Context: A test plan exists and needs to be turned into runnable specs. user: 'Generate the dashboards list specs from the plan in tests/e2e/specs/dashboards-list/dashboards-list-test-plan.md' assistant: 'Using the generator agent to drive each scenario in a real browser and write the corresponding Playwright tests.'</example>
|
||||
tools: Glob, Grep, Read, Write, Edit, Bash, mcp__playwright-test__browser_click, mcp__playwright-test__browser_drag, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_file_upload, mcp__playwright-test__browser_handle_dialog, mcp__playwright-test__browser_hover, mcp__playwright-test__browser_navigate, mcp__playwright-test__browser_press_key, mcp__playwright-test__browser_select_option, mcp__playwright-test__browser_snapshot, mcp__playwright-test__browser_type, mcp__playwright-test__browser_verify_element_visible, mcp__playwright-test__browser_verify_list_visible, mcp__playwright-test__browser_verify_text_visible, mcp__playwright-test__browser_verify_value, mcp__playwright-test__browser_wait_for, mcp__playwright-test__generator_read_log, mcp__playwright-test__generator_setup_page, mcp__playwright-test__generator_write_test
|
||||
description: Use this agent to convert a SigNoz E2E test plan into Playwright spec files under `tests/e2e/tests/<feature>/`. Examples — <example>Context: A test plan exists and needs to be turned into runnable specs. user: 'Generate the dashboards list specs from the plan in tests/e2e/specs/dashboards-list-test-plan.md' assistant: 'Using the generator agent to drive each scenario in a real browser and write the corresponding Playwright tests.'</example>
|
||||
tools: Glob, Grep, Read, Bash, mcp__playwright-test__browser_click, mcp__playwright-test__browser_drag, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_file_upload, mcp__playwright-test__browser_handle_dialog, mcp__playwright-test__browser_hover, mcp__playwright-test__browser_navigate, mcp__playwright-test__browser_press_key, mcp__playwright-test__browser_select_option, mcp__playwright-test__browser_snapshot, mcp__playwright-test__browser_type, mcp__playwright-test__browser_verify_element_visible, mcp__playwright-test__browser_verify_list_visible, mcp__playwright-test__browser_verify_text_visible, mcp__playwright-test__browser_verify_value, mcp__playwright-test__browser_wait_for, mcp__playwright-test__generator_read_log, mcp__playwright-test__generator_setup_page, mcp__playwright-test__generator_write_test
|
||||
model: sonnet
|
||||
color: blue
|
||||
---
|
||||
@@ -10,23 +10,17 @@ You are the Playwright Test Generator for the SigNoz frontend. You take a plan w
|
||||
|
||||
# Repo conventions you must follow
|
||||
|
||||
- **Spec location:** `tests/e2e/tests/<feature>/<spec-name>.spec.ts`. One file per resource; cross-resource concerns get their own file. Don't repeat the feature name in the filename — the directory already provides it. `dashboards/list/search.spec.ts`, not `dashboards/list/dashboards-list-search.spec.ts`.
|
||||
- **Drivers first.** New specs import `test`, `expect` and constants from one driver module in [tests/e2e/drivers/](../../tests/e2e/drivers/) and receive the feature's driver as a fixture. A spec never passes `page` around.
|
||||
- **Spec location:** `tests/e2e/tests/<feature>/<spec-name>.spec.ts`. One file per resource; cross-resource concerns get their own file. Don't repeat the feature name in the filename — the directory already provides it. `dashboards/list.spec.ts`, not `dashboards/dashboards-list.spec.ts`.
|
||||
- **Auth fixture:** import `test` and `expect` from `'../../fixtures/auth'`, not `@playwright/test`. Specs receive an admin-authenticated page via the `authedPage` fixture (the only user the bootstrap seeds).
|
||||
```ts
|
||||
import { expect, PATH, test } from '../../drivers/routing';
|
||||
import { test, expect } from '../../fixtures/auth';
|
||||
|
||||
test('TC-09 RouteTab switches the url', async ({ routing }) => {
|
||||
await routing.goto(PATH.metricsSummary);
|
||||
await expect(routing.activeRouteTab(PATH.metricsSummary)).toBeVisible();
|
||||
test('TC-01 alerts page — tabs render', async ({ authedPage: page }) => {
|
||||
await page.goto('/alerts');
|
||||
await expect(page.getByRole('tab', { name: /alert rules/i })).toBeVisible();
|
||||
});
|
||||
```
|
||||
- A driver is a function closing over the page that returns `Locator` getters and async actions. Actions wait inside themselves (e.g. `run()` pairs the click with `waitForResponse(/query_range/)`), so specs don't.
|
||||
- `driver(name, build, dispose?)` in `drivers/kit.ts` makes it a test-scoped fixture on `authedPage`; `dispose` deletes what it seeded. `seeded(name, open, close)` makes a worker-scoped seed.
|
||||
- Layers compose: `withNewPanel(withEditor(panelsTest))`. Extend the closest existing driver instead of starting a parallel one.
|
||||
- When no driver covers the feature, add `drivers/<feature>.ts` with the spec. Locators live in the driver, not the spec. `drivers/` is AI-owned: keep the banner, and change a driver only alongside a spec that needs it.
|
||||
- Read [tests/e2e/tests/dashboards/editor/query.spec.ts](../../tests/e2e/tests/dashboards/editor/query.spec.ts) and [drivers/panel-editor.ts](../../tests/e2e/drivers/panel-editor.ts) for the canonical shape.
|
||||
- **Plain auth fixture** only for a spec no driver fits: import `test` and `expect` from `'../../fixtures/auth'` (never `@playwright/test`) and use the `authedPage` fixture (the admin, the only user the bootstrap seeds).
|
||||
- **Extended fixtures:** older suites import domain fixtures that extend `auth`. See [docs/contributing/tests/e2e.md](../../docs/contributing/tests/e2e.md).
|
||||
- **Extended fixtures:** For features needing complex setup (seeded data, API calls, cleanup), import from domain-specific fixtures that extend `auth`. See [docs/contributing/tests/e2e.md](../../docs/contributing/tests/e2e.md) for the full pattern.
|
||||
- `fixtures/alerts/alert-rules` — worker-scoped rule list + test-scoped rule factory
|
||||
- `fixtures/alerts/alert-history` — extends alert-rules, adds history fixtures (waits on ruler evaluation)
|
||||
```ts
|
||||
@@ -37,18 +31,15 @@ You are the Playwright Test Generator for the SigNoz frontend. You take a plan w
|
||||
import { test, expect } from '../../../fixtures/alerts/alert-history';
|
||||
```
|
||||
- **Test titles:** `TC-NN <short description>` — matches the planner's IDs.
|
||||
- **Self-contained state.** The bootstrap creates a fresh stack with **zero** dashboards / alerts / etc. — never assume pre-existing resources. Cleanup shapes, in order of preference:
|
||||
- **A driver's `dispose`** — the driver tracks what it seeded and deletes it after the test (`dashboards.seed(...)` in `drivers/dashboards.ts`). The spec has no cleanup code.
|
||||
- **A worker-scoped `seeded(...)` layer** — read-only data shared by a worker's tests.
|
||||
- **Per-test `try / finally`** or **suite-level `afterAll` with an id registry** — specs without a driver. Suite hooks can't see `authedPage`; use `newAdminContext(browser)` from `helpers/auth.ts`.
|
||||
- **Golden telemetry is pre-seeded.** `bootstrap/global.setup.ts` loads logs, traces and metrics over the last 6h (constants in `GOLDEN`, `helpers/dashboard-spec.ts`). Query within ≤ 6h and assert structure (series, rows, request payload), never exact counts: per-test seeds from parallel workers land in the same tenant. Seed missing shapes through the seeder with unique service names (`helpers/alerts/seeding.ts`).
|
||||
- **Reuse helpers from `tests/e2e/helpers/`** inside drivers. Don't reinvent:
|
||||
- [`helpers/common.ts`](../../tests/e2e/helpers/common.ts) — `authToken`, `watchConsole`, `collectRequests`, `requestUrl`, `seederUrl`.
|
||||
- [`helpers/auth.ts`](../../tests/e2e/helpers/auth.ts) — `newAdminContext(browser)`.
|
||||
- [`helpers/query-range-mock.ts`](../../tests/e2e/helpers/query-range-mock.ts) — `query_range` mocks for error / no-data / warning / loading states only; install before `goto`.
|
||||
- **Seed via API when the UI flow is multi-step or brittle**, through the driver (`dashboards.seed`, `seedAndOpen`, `seedAndEdit`). `page.request.*` does **not** auto-attach `Authorization`; drivers add `authToken(page)`.
|
||||
- **Static payloads live in [tests/e2e/testdata/](../../tests/e2e/testdata/)** (`dashboards/panels.ts`, `traces/large-trace.json`).
|
||||
- **Resource names:** short and descriptive, no timestamps. Where parallel workers share a list, add a random suffix as `uniqueName` in `drivers/dashboard-list.ts` does.
|
||||
- **Self-contained state.** The bootstrap creates a fresh stack with **zero** dashboards / alerts / etc. — never assume pre-existing data. Two cleanup shapes are valid; pick based on the spec size:
|
||||
- **Per-test `try / finally`** — small specs (~ <10 scenarios) where each test owns its data.
|
||||
- **Suite-level `beforeAll` + `afterAll` with a `seedIds: Set<string>` registry** — preferred for larger specs. Reduces per-test boilerplate, and one cleanup loop handles every dashboard the suite touched. See [tests/e2e/tests/dashboards/list.spec.ts](../../tests/e2e/tests/dashboards/list.spec.ts) for the canonical shape.
|
||||
- **Reuse helpers from `tests/e2e/helpers/`.** Don't reinvent. The current set:
|
||||
- [`helpers/auth.ts`](../../tests/e2e/helpers/auth.ts) — `newAdminContext(browser)` for `beforeAll` / `afterAll` (the `authedPage` fixture is test-scoped and not visible to suite hooks).
|
||||
- [`helpers/dashboards.ts`](../../tests/e2e/helpers/dashboards.ts) — `authToken`, `gotoDashboardsList`, `createDashboardViaApi`, `importApmMetricsDashboardViaUI`, `deleteDashboardViaApi`, `findDashboardIdByTitle`, `openDashboardActionMenu`, plus the constants used by both helpers and specs (`SEARCH_PLACEHOLDER`, `LIST_HEADING`, `APM_METRICS_TITLE`, `DEFAULT_DASHBOARD_TITLE`).
|
||||
- **Seed via API when the UI flow is multi-step or brittle.** Implementation lives in `createDashboardViaApi` — use it. `page.request.*` does **not** auto-attach `Authorization`; the helpers handle that for you. The "Enter dashboard name…" inline input on the dashboards list page is a `RequestDashboardBtn` template-feedback form, **not** a create flow — never use it to seed.
|
||||
- **Reusable JSON fixtures live in [tests/e2e/fixtures/](../../tests/e2e/fixtures/).** `apm-metrics.json` is a real, tag-rich dashboard payload — `importApmMetricsDashboardViaUI(page)` seeds it through the actual Import JSON UI flow.
|
||||
- **Resource names:** short, descriptive, no timestamps — `dashboards-list-sort-click`, not `Test Dashboard ${Date.now()}`. Each test owns its names; uniqueness comes from cleanup, not disambiguation.
|
||||
- **Serial mode** when tests in a file mutate the same list page:
|
||||
```ts
|
||||
test.describe.configure({ mode: 'serial' });
|
||||
@@ -73,14 +64,13 @@ For each scenario in the plan:
|
||||
- For verifications, use the dedicated `browser_verify_*` tools — they capture the assertion as Playwright code in the log.
|
||||
4. **Read the log.** Call `generator_read_log` immediately after the last step. Don't intersperse other tool calls.
|
||||
5. **Write the spec.** Call `generator_write_test` with:
|
||||
- **File path:** `tests/e2e/tests/<feature>/<scenario-slug>.spec.ts` — fs-friendly slug from the scenario title. Drop the feature prefix when it duplicates the directory (`dashboards/list/search.spec.ts`, not `dashboards/list/dashboards-list-search.spec.ts`).
|
||||
- **File path:** `tests/e2e/tests/<feature>/<scenario-slug>.spec.ts` — fs-friendly slug from the scenario title. Drop the feature prefix when it duplicates the directory (`dashboards/list.spec.ts`, not `dashboards/dashboards-list.spec.ts`).
|
||||
- **Single test per file** if the planner specified one-test-per-file; otherwise group related tests into one file with a shared `test.describe('<Feature>', () => { … })`.
|
||||
- **`describe` block** matches the top-level plan section.
|
||||
- **Title** matches `TC-NN <description>` exactly.
|
||||
- **Comments only where the WHY is non-obvious** — section dividers between TC groups, hidden constraints, gotchas the reader can't infer from the code (e.g. "Monaco swallows Escape — click the title to blur first"). **Do not narrate steps** by pasting the plan's bullets back as `// 1. Navigate…` `// 2. Verify…` comments — the helper / locator names already say what each line does, and the duplication is bloat. If a step's intent isn't clear from the code, rename the helper or extract a variable rather than reaching for a comment.
|
||||
- **Imports** from the feature's driver module (or `../../fixtures/auth` when none fits). **Do not** import from `@playwright/test` directly.
|
||||
- **Cleanup** through the driver's `dispose`; `try / finally` only without a driver.
|
||||
- **Move recorded locators and waits into the driver.** The log gives raw `page.*` calls; the spec should read as driver actions.
|
||||
- **Imports** from `../../fixtures/auth`. **Do not** import from `@playwright/test` directly.
|
||||
- **Try / finally** cleanup using the API (delete the resources you seeded).
|
||||
|
||||
# Quality bar — what to write, what to skip
|
||||
|
||||
@@ -98,57 +88,81 @@ When you cut, merge, or renumber TCs vs the plan, note it in your final summary.
|
||||
For a plan section:
|
||||
|
||||
```markdown
|
||||
### 1. Search
|
||||
### 1. Page Load
|
||||
|
||||
#### TC-01 searching by name writes the query to the URL and lists the match
|
||||
**Preconditions:** one dashboard with a unique name (seed via the `dashboards` driver).
|
||||
#### TC-01 page chrome and core controls render
|
||||
**Steps:**
|
||||
1. Open the dashboards list.
|
||||
2. Search `name CONTAINS '<name>'`.
|
||||
**Expected:** the `query` URL param carries the search; only the seeded dashboard is listed.
|
||||
1. Navigate to `/dashboard`
|
||||
2. Verify the page title is "SigNoz | All Dashboards"
|
||||
3. Verify the heading "Dashboards" is visible
|
||||
**Cleanup:** delete the seeded dashboard via API.
|
||||
```
|
||||
|
||||
You produce:
|
||||
You produce (suite-level shape, preferred for files with multiple scenarios):
|
||||
|
||||
```ts
|
||||
// tests/e2e/tests/dashboards/list/search.spec.ts
|
||||
// tests/e2e/tests/dashboards/list.spec.ts
|
||||
import type { Page } from '@playwright/test';
|
||||
|
||||
import { expect, test } from '../../fixtures/auth';
|
||||
import { newAdminContext } from '../../helpers/auth';
|
||||
import {
|
||||
expect,
|
||||
listDashboard,
|
||||
nameContains,
|
||||
test,
|
||||
uniqueName,
|
||||
} from '../../../drivers/dashboard-list';
|
||||
authToken,
|
||||
createDashboardViaApi,
|
||||
deleteDashboardViaApi,
|
||||
gotoDashboardsList,
|
||||
} from '../../helpers/dashboards';
|
||||
|
||||
test.describe('Dashboards — list search', () => {
|
||||
test('TC-01 searching by name writes the query to the URL and lists the match', async ({
|
||||
test.describe.configure({ mode: 'serial' });
|
||||
|
||||
const seedIds = new Set<string>();
|
||||
|
||||
async function seed(page: Page, title: string): Promise<string> {
|
||||
const id = await createDashboardViaApi(page, title);
|
||||
seedIds.add(id);
|
||||
return id;
|
||||
}
|
||||
|
||||
test.afterAll(async ({ browser }) => {
|
||||
if (seedIds.size === 0) return;
|
||||
const ctx = await newAdminContext(browser);
|
||||
const page = await ctx.newPage();
|
||||
try {
|
||||
const token = await authToken(page);
|
||||
for (const id of [...seedIds]) {
|
||||
await deleteDashboardViaApi(ctx.request, id, token);
|
||||
seedIds.delete(id);
|
||||
}
|
||||
} finally {
|
||||
await ctx.close();
|
||||
}
|
||||
});
|
||||
|
||||
test.describe('Dashboards List Page', () => {
|
||||
test('TC-01 page chrome and core controls render', async ({
|
||||
authedPage: page,
|
||||
dashboards,
|
||||
dashboardList,
|
||||
}) => {
|
||||
const name = uniqueName('search');
|
||||
await dashboards.seed(listDashboard(name));
|
||||
await dashboardList.open();
|
||||
await seed(page, 'list-chrome');
|
||||
|
||||
await dashboardList.runSearch(nameContains(name));
|
||||
await gotoDashboardsList(page);
|
||||
|
||||
expect(new URL(page.url()).searchParams.get('query')).toBe(
|
||||
nameContains(name),
|
||||
);
|
||||
await expect(dashboardList.rows()).toHaveText([name]);
|
||||
await expect(page).toHaveTitle('SigNoz | All Dashboards');
|
||||
await expect(
|
||||
page.getByRole('heading', { name: 'Dashboards', level: 1 }),
|
||||
).toBeVisible();
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
Note: no cleanup code (the `dashboards` driver deletes its seed), no raw locators (they live in `drivers/dashboard-list.ts`), and no `// 1. …` step narration. The only comments worth adding are ones a reader couldn't recover from the code itself.
|
||||
Note how the example carries no `// 1. …` `// 2. …` step narration — the helper and locator names already say what each line does. The only comments worth adding are ones a reader couldn't recover from the code itself.
|
||||
|
||||
# Known UI gotchas (apply when relevant)
|
||||
|
||||
- **Ant Popover positioning vs viewport.** Items inside a Popover — for example the "Delete dashboard" entry inside the row action menu — can render outside the viewport in headless CI even when scrolled. `click({ force: true })` skips actionability checks but Playwright still requires the click coordinates to land inside the viewport. Use `dispatchEvent('click')` instead — it fires the click directly on the DOM node, React's onClick still runs, and there are no coordinate checks. Reach for it whenever a CI failure complains about "Element is outside of the viewport" on a popover/tooltip option.
|
||||
- **Sticky-header rows below the fold.** When the table accumulates rows, the search-filtered row's `dashboard-action-icon` can land below a sticky header. Always `await actionIcon.scrollIntoViewIfNeeded()` before clicking, inside the driver action.
|
||||
- **Sticky-header rows below the fold.** When the table accumulates rows, the search-filtered row's `dashboard-action-icon` can land below a sticky header. Always `await actionIcon.scrollIntoViewIfNeeded()` before clicking. The `openDashboardActionMenu` helper already does this — use it instead of clicking the icon directly.
|
||||
- **React Query mutations vs navigation.** UI delete clicks fire an async DELETE through React Query. Navigating away before the mutation completes cancels it. Pair the click with `page.waitForResponse((r) => r.request().method() === 'DELETE' && /\/dashboards\//.test(r.url()))` and `await expect(dialog).not.toBeVisible()` before the next `page.goto(...)`.
|
||||
- **Monaco editor swallows Escape.** Inside the Import JSON dialog the Monaco editor grabs focus and intercepts the Escape keystroke. Click the modal title first to blur Monaco; `dashboardList.dismissWithEscape()` does this.
|
||||
- **Empty zero-state hides controls.** With no dashboards in the workspace, the list controls and `new-dashboard-cta` are absent. Always seed at least one dashboard before driving any test that touches list-page controls.
|
||||
- **Monaco editor swallows Escape.** Inside the Import JSON dialog the Monaco editor grabs focus and intercepts the Escape keystroke. Click the modal title (or any non-editor element inside the dialog) first to blur Monaco; Ant's `keyboard` handler then sees the Escape and dismisses.
|
||||
- **Empty zero-state hides controls.** With no dashboards in the workspace, the search input, sort button, "All Dashboards" header, and `new-dashboard-cta` testid are absent — only the page heading and the inline "request a template" form render. Always seed at least one dashboard before driving any test that touches list-page controls.
|
||||
|
||||
# Quality bar
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: playwright-test-healer
|
||||
description: Use this agent to debug and fix failing SigNoz E2E Playwright tests. Examples — <example>Context: A spec is red. user: 'tests/e2e/tests/dashboards/list/search.spec.ts is failing, fix it' assistant: 'Using the healer agent to debug each failing scenario and adjust the spec.'</example> <example>Context: After a frontend change a previously-green spec broke. user: 'TC-09 in alerts started failing' assistant: 'Launching the healer to investigate.'</example>
|
||||
description: Use this agent to debug and fix failing SigNoz E2E Playwright tests. Examples — <example>Context: A spec is red. user: 'tests/e2e/tests/dashboards/list.spec.ts is failing, fix it' assistant: 'Using the healer agent to debug each failing scenario and adjust the spec.'</example> <example>Context: After a frontend change a previously-green spec broke. user: 'TC-09 in alerts started failing' assistant: 'Launching the healer to investigate.'</example>
|
||||
tools: Glob, Grep, Read, Write, Edit, Bash, mcp__playwright-test__browser_console_messages, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_generate_locator, mcp__playwright-test__browser_network_requests, mcp__playwright-test__browser_snapshot, mcp__playwright-test__test_debug, mcp__playwright-test__test_list, mcp__playwright-test__test_run
|
||||
model: sonnet
|
||||
color: red
|
||||
@@ -27,11 +27,11 @@ Don't try to start the stack yourself — it can take ~4 minutes on a cold build
|
||||
- `browser_network_requests` for API failures (the SigNoz API requires `Authorization: Bearer <localStorage.AUTH_TOKEN>`; 401s usually mean the test bypassed the fixture).
|
||||
- `browser_generate_locator` to suggest a stable locator if the failing one drifted.
|
||||
4. **Root-cause.** Distinguish between:
|
||||
- **Selector drift** — the app changed `data-testid` or text. Fix the locator where it lives: in the driver under `tests/e2e/drivers/` when the spec uses one, so every spec sharing it heals at once. Prefer `data-testid` (grep `frontend/src/<feature-dir>/` for the new one).
|
||||
- **Timing** — the test races a load. Replace `waitForTimeout` with `await expect(locator).toBe…()` or `page.waitForResponse(...)` on the triggering action — inside the driver action when there is one.
|
||||
- **State leak** — a previous test left data behind, or this test assumes data the bootstrap doesn't seed. Seed through the driver (its `dispose` cleans up) or via API with `try / finally`. The bootstrap creates a fresh stack with **zero** dashboards / alerts and only the golden telemetry; parallel per-test seeds share the tenant, so an exact-count assertion on telemetry is a test bug — assert structure instead.
|
||||
- **Selector drift** — the app changed `data-testid` or text. Fix the locator. Prefer `data-testid` (grep `frontend/src/<feature-dir>/` for the new one).
|
||||
- **Timing** — the test races a load. Replace `waitForTimeout` with `await expect(locator).toBe…()` or `page.waitForResponse(...)` on the triggering action.
|
||||
- **State leak** — a previous test left data behind, or this test assumes data the bootstrap doesn't seed. Ensure the test seeds via API and cleans up in `try / finally`. The bootstrap creates a fresh stack with **zero** dashboards / alerts.
|
||||
- **Genuine app bug** — the app is broken, not the test. Mark the test with `test.fixme(...)` and add a one-line `// known: <description>` comment. Don't silently change the assertion to make it pass.
|
||||
5. **Fix.** Edit the spec or its driver. Preserve TC-NN titles, the driver / `authedPage` fixtures, cleanup, and serial mode if present. A driver change must keep every other spec importing it green — run them too. If you renumber, retitle, or `test.fixme(...)` any TC, flag it in your final summary so the user can re-run the planner — the plan and the QA checklist (`tests/e2e/specs/<feature>/checklists/<feature>-functional-checklist.md`) re-derive from the current `.spec.ts` and will otherwise drift.
|
||||
5. **Fix.** Edit the spec. Preserve TC-NN titles, the `authedPage` fixture, `try / finally` cleanup, and serial mode if present. If you renumber, retitle, or `test.fixme(...)` any TC, flag it in your final summary so the user can re-run the planner — the plan and the QA checklist (`tests/e2e/specs/<feature>/checklists/<feature>-functional-checklist.md`) re-derive from the current `.spec.ts` and will otherwise drift.
|
||||
6. **Re-run only the fixed test** before moving to the next failure. Three options:
|
||||
- `npx playwright test -g 'TC-09' --project=chromium` — target a single TC by title
|
||||
- `npx playwright test --last-failed --project=chromium` — replay everything that failed last run
|
||||
@@ -41,13 +41,14 @@ Don't try to start the stack yourself — it can take ~4 minutes on a cold build
|
||||
|
||||
# Repo-specific signals
|
||||
|
||||
- **Reuse drivers and helpers before adding new code.** [`tests/e2e/drivers/`](../../tests/e2e/drivers/) holds the per-feature seed, cleanup, navigation and action methods (`kit.ts` explains `driver` / `seeded`); [`helpers/common.ts`](../../tests/e2e/helpers/common.ts) and [`helpers/auth.ts`](../../tests/e2e/helpers/auth.ts) the plain plumbing. Prefer them over re-inlining auth/login/POST code in the spec.
|
||||
- **Reuse helpers before adding new code.** [`tests/e2e/helpers/dashboards.ts`](../../tests/e2e/helpers/dashboards.ts) and [`tests/e2e/helpers/auth.ts`](../../tests/e2e/helpers/auth.ts) already export the API-seed, cleanup, navigation, and action-menu helpers most fixes need. Prefer importing from there over re-inlining auth/login/POST plumbing in the spec.
|
||||
- **Ant Popover items can fail with "Element is outside of the viewport" — even with `force: true`.** `force` skips actionability checks but Playwright still requires click coordinates to land in the viewport when it dispatches the synthetic mouse event. The robust fix is `tooltip.getByText('…').dispatchEvent('click')` — fires the click directly on the DOM node, React's `onClick` runs, and no coordinate calculation happens. Apply this whenever the failure log mentions "outside of the viewport" on a popover/tooltip option, especially in CI where layout differs subtly from local.
|
||||
- **Action-icon rows below the fold.** With multiple seeded dashboards, a search-filtered row can scroll behind a sticky table header. Add `scrollIntoViewIfNeeded` before the click, in the driver action.
|
||||
- **Action-icon rows below the fold.** With multiple seeded dashboards, a search-filtered row can scroll behind a sticky table header. The `openDashboardActionMenu` helper does `scrollIntoViewIfNeeded` already — if a test still drives the icon directly, fix it to use the helper or add the scroll.
|
||||
- **React Query mutations vs page.goto.** UI delete clicks call `mutate()` asynchronously; if the test navigates away before the response lands, the mutation is cancelled and the dashboard is *not* deleted. Wait for the DELETE response and the dialog dismissal explicitly: `page.waitForResponse((r) => r.request().method() === 'DELETE' && /\/dashboards\//.test(r.url()))` plus `await expect(dialog).not.toBeVisible()`.
|
||||
- **Monaco editor swallows Escape inside the Import JSON dialog.** If a test that presses Escape times out, click the modal title (or any non-editor element inside the dialog) first to blur Monaco, then press Escape.
|
||||
- **The list pages render zero-state when the workspace is empty.** List controls and the `new-dashboard-cta` testid are absent in zero-state. A timeout on those usually means the workspace was empty — seed first via `dashboards.seed(...)`.
|
||||
- **Auth.** `tests/e2e/fixtures/auth.ts` logs in once per worker and caches `storageState` (cookies + localStorage with `AUTH_TOKEN`). For API-driven seeding/cleanup, use `authToken(page)` from `helpers/common.ts` and pass `Authorization: Bearer <token>`. Never re-implement login.
|
||||
- **The list pages render zero-state when the workspace is empty.** Many locators (search input, sort button, `new-dashboard-cta` testid, "All Dashboards" header) are absent in zero-state. A 30s timeout on those usually means the workspace was empty — seed first via `createDashboardViaApi`.
|
||||
- **The "Enter dashboard name…" inline field is a `RequestDashboardBtn` (template-request feedback form), not a create flow.** Tests that try to use it to create a named dashboard will silently no-op. The only UI create paths are the "New dashboard" dropdown → "Create dashboard" (default name "Sample Title", see `DEFAULT_DASHBOARD_TITLE`) or "Import JSON".
|
||||
- **Auth.** `tests/e2e/fixtures/auth.ts` logs in once per worker and caches `storageState` (cookies + localStorage with `AUTH_TOKEN`). For API-driven seeding/cleanup, use `authToken(page)` from `helpers/dashboards.ts` and pass `Authorization: Bearer <token>`. Never re-implement login.
|
||||
- **Extended fixtures.** Domain-specific fixtures extend `auth` and add seeded data. Alerts uses `fixtures/alerts/alert-rules` (worker-scoped rule list, test-scoped factory) and `fixtures/alerts/alert-history` (extends alert-rules, waits on ruler evaluation). See [docs/contributing/tests/e2e.md](../../docs/contributing/tests/e2e.md) for the pattern. When a test fails on missing data, check if it imports the wrong fixture level.
|
||||
- **Ant Design popovers** (sort menu, action menu) are click-toggle. The trigger element is often an inline `<svg>` with a `data-testid` — clicking it opens the popover; clicking it again closes. After selecting an option, the popover auto-closes. If a test interacts with the popover twice, wait for the menu items to be visible explicitly between toggles.
|
||||
- **Artifacts.** Every failed test writes to `tests/e2e/artifacts/results/<test-slug>/` — the `error-context.md` accessibility snapshot is the fastest way to see what the page actually looked like when it failed.
|
||||
|
||||
@@ -20,10 +20,8 @@ You will:
|
||||
- The frontend stores its JWT in `localStorage` under `AUTH_TOKEN` and the API requires `Authorization: Bearer <token>` for protected endpoints. Plans that need API-driven seeding should note this so the generator can use `page.request.*`.
|
||||
|
||||
2. **Check what's already wired up.**
|
||||
- [tests/e2e/drivers/](../../tests/e2e/drivers/) holds one driver per feature (e.g. `dashboards.ts`, `dashboard-list.ts`, `panel-editor.ts`, `routing.ts`, `channels.ts`). A driver is a fixture exposing the feature's locators and actions, including API seeding that cleans itself up (`dashboards.seed(...)`). When one covers the feature, write steps in terms of its actions by name so the generator reuses them. When none does, say so: the generator adds one.
|
||||
- [tests/e2e/helpers/](../../tests/e2e/helpers/) holds plain helpers: `common.ts` (`authToken`, `watchConsole`, `collectRequests`, `seederUrl`), `auth.ts` (`newAdminContext`), `query-range-mock.ts` (mock `query_range` for error, no-data, warning and loading states only).
|
||||
- **Telemetry is pre-seeded.** `bootstrap/global.setup.ts` loads a golden OTel-demo dataset (logs, traces, metrics over the last 6h; constants in `GOLDEN` in `helpers/dashboard-spec.ts`). Plans that query data use a window ≤ 6h and assert structure, not exact counts. Data the golden set lacks is seeded per test through the seeder (`helpers/alerts/seeding.ts` shows how).
|
||||
- [tests/e2e/testdata/](../../tests/e2e/testdata/) holds static payloads (`dashboards/panels.ts`, `traces/large-trace.json`) — note in the plan if a scenario benefits from one.
|
||||
- [tests/e2e/helpers/dashboards.ts](../../tests/e2e/helpers/dashboards.ts) and [tests/e2e/helpers/auth.ts](../../tests/e2e/helpers/auth.ts) hold reusable helpers (`createDashboardViaApi`, `gotoDashboardsList`, `openDashboardActionMenu`, `newAdminContext`, `importApmMetricsDashboardViaUI`, etc.). When the plan touches dashboards, reference these by name in the steps so the generator can reuse them rather than reinvent.
|
||||
- [tests/e2e/fixtures/apm-metrics.json](../../tests/e2e/fixtures/apm-metrics.json) is a real-world dashboard payload (rich tags, panels, description) suitable as a seed fixture — note in the plan if a scenario benefits from it.
|
||||
|
||||
3. **Navigate and explore**
|
||||
- Invoke `planner_setup_page` once before any other browser tool.
|
||||
@@ -41,7 +39,7 @@ You will:
|
||||
|
||||
Each scenario must include:
|
||||
- **TC-NN** title — `TC-NN <short description>` (matches the naming this repo uses for test titles).
|
||||
- Preconditions (what state the test expects — note that the bootstrap creates a fresh stack with **zero dashboards / alerts / etc.** (only the golden telemetry), so plans must seed their own resources).
|
||||
- Preconditions (what state the test expects — note that the bootstrap creates a fresh stack with **zero dashboards / alerts / etc.**, so plans must seed their own data).
|
||||
- Step-by-step user actions.
|
||||
- Expected outcomes per step.
|
||||
- Cleanup notes (what gets created and how to remove it — usually via API).
|
||||
@@ -66,7 +64,7 @@ The dashboards list page (`/dashboard`) lists all dashboards in the workspace. F
|
||||
- Open per-row actions: View, Open in New Tab, Copy Link, Export JSON, Delete dashboard
|
||||
- Create a new dashboard via the "New dashboard" dropdown (Create dashboard / Import JSON / View templates)
|
||||
|
||||
**Bootstrap state:** the pytest harness creates a fresh stack with no pre-seeded dashboards. Every test seeds its own through the `dashboards` driver (`dashboards.seed(...)`, deleted when the test ends).
|
||||
**Bootstrap state:** the pytest harness creates a fresh stack with no pre-seeded dashboards. Every test must seed its own data. The "Enter dashboard name…" inline input is a "request a new template" feedback form — **not** a create flow. The only UI create path is the dropdown.
|
||||
|
||||
**Known limitations:**
|
||||
- Ascending sort is not yet implemented — repeated clicks on the sort button keep `order=descend`.
|
||||
@@ -89,7 +87,7 @@ The dashboards list page (`/dashboard`) lists all dashboards in the workspace. F
|
||||
- All Dashboards section header rendered.
|
||||
- Search input, sort button, and at least one dashboard thumbnail visible.
|
||||
|
||||
**Cleanup:** none in the spec — the `dashboards` driver deletes what it seeded.
|
||||
**Cleanup:** delete the seeded dashboard via `DELETE /api/v1/dashboards/<id>`.
|
||||
|
||||
#### TC-02 ...
|
||||
</example-spec>
|
||||
@@ -97,7 +95,7 @@ The dashboards list page (`/dashboard`) lists all dashboards in the workspace. F
|
||||
**Quality bar:**
|
||||
- Steps must be specific enough that any tester (or the generator agent) can follow without ambiguity.
|
||||
- Include negative scenarios — empty state, no-match search, validation errors.
|
||||
- Each scenario must own its preconditions and cleanup. **Do not invent cross-file global fixtures** — they break parallel-by-file execution. Prefer a driver's self-cleaning seed (`dispose`) or worker-scoped `seeded(...)` layer; without a driver, suite-level `beforeAll` / `afterAll` within one spec file or per-test `try / finally` both work.
|
||||
- Each scenario must own its preconditions and cleanup. **Do not invent cross-file global fixtures** — they break parallel-by-file execution. Suite-level `beforeAll` / `afterAll` *within* a single spec file is fine and is the preferred shape for files with > ~10 scenarios; per-test `try / finally` is fine for smaller specs.
|
||||
- Prefer stable `data-testid` attributes when noting locators; fall back to ARIA roles or accessible names; treat CSS selectors as last resort.
|
||||
- **Don't pad coverage.** Every TC must catch a regression that would actually ship if the test were missing — a real branch in the source, a real user-visible failure mode. A TC that asserts a hard-coded string is still rendered, or that a button is still a button, adds nothing but maintenance cost: it inflates the suite, slows CI, and trains readers to skim past the directory. Before writing a scenario, ask "what code change would break this?" — if the only answer is "deleting the literal under test," cut it or fold it into a richer scenario as one assertion among many.
|
||||
- **Collapse near-duplicates.** Two TCs that differ only in input value (search by title vs search by description, when the underlying code path is the same) should merge into one parameterised scenario unless each input genuinely exercises a distinct branch. Prefer one assertion-rich TC over three thin ones.
|
||||
|
||||
@@ -3932,6 +3932,7 @@ components:
|
||||
- logs-field
|
||||
- traces-field
|
||||
- metrics-field
|
||||
- llm-pricing-rule
|
||||
type: string
|
||||
CoretypesObject:
|
||||
properties:
|
||||
@@ -15759,9 +15760,9 @@ paths:
|
||||
description: Internal Server Error
|
||||
security:
|
||||
- api_key:
|
||||
- VIEWER
|
||||
- llm-pricing-rule:list
|
||||
- tokenizer:
|
||||
- VIEWER
|
||||
- llm-pricing-rule:list
|
||||
summary: List pricing rules
|
||||
tags:
|
||||
- llmpricingrules
|
||||
@@ -15806,9 +15807,9 @@ paths:
|
||||
description: Internal Server Error
|
||||
security:
|
||||
- api_key:
|
||||
- ADMIN
|
||||
- llm-pricing-rule:update
|
||||
- tokenizer:
|
||||
- ADMIN
|
||||
- llm-pricing-rule:update
|
||||
summary: Create or update pricing rules
|
||||
tags:
|
||||
- llmpricingrules
|
||||
@@ -15854,9 +15855,9 @@ paths:
|
||||
description: Internal Server Error
|
||||
security:
|
||||
- api_key:
|
||||
- ADMIN
|
||||
- llm-pricing-rule:delete
|
||||
- tokenizer:
|
||||
- ADMIN
|
||||
- llm-pricing-rule:delete
|
||||
summary: Delete a pricing rule
|
||||
tags:
|
||||
- llmpricingrules
|
||||
@@ -15912,9 +15913,9 @@ paths:
|
||||
description: Internal Server Error
|
||||
security:
|
||||
- api_key:
|
||||
- VIEWER
|
||||
- llm-pricing-rule:read
|
||||
- tokenizer:
|
||||
- VIEWER
|
||||
- llm-pricing-rule:read
|
||||
summary: Get a pricing rule
|
||||
tags:
|
||||
- llmpricingrules
|
||||
@@ -15967,9 +15968,9 @@ paths:
|
||||
description: Internal Server Error
|
||||
security:
|
||||
- api_key:
|
||||
- VIEWER
|
||||
- llm-pricing-rule:list
|
||||
- tokenizer:
|
||||
- VIEWER
|
||||
- llm-pricing-rule:list
|
||||
summary: List unmapped models
|
||||
tags:
|
||||
- llmpricingrules
|
||||
|
||||
@@ -173,10 +173,9 @@ Each spec follows these principles:
|
||||
1. **Directory per feature**: `tests/e2e/tests/<feature>/*.spec.ts`. Cross-resource junction concerns (e.g. cascade-delete) go in their own file, not packed into one giant spec.
|
||||
2. **Test titles use `TC-NN`**: `test('TC-01 alerts page — tabs render', ...)`. Preserves ordering at a glance and maps to external coverage tracking.
|
||||
3. **UI-first**: drive flows through the UI. Playwright traces capture every BE request/response the UI triggers, so asserting on UI outcomes implicitly validates BE contracts. Reach for direct `page.request.*` only when the test's *purpose* is asserting a response contract (use `page.waitForResponse` on a UI click) or when a specific UI step is structurally flaky (e.g. Ant DatePicker calendar-cell indices) — and even then try UI first.
|
||||
4. **Self-contained state**: each spec seeds its own data and cleans up at suite teardown. The pytest harness creates a fresh stack with **zero** dashboards / alerts / etc. — never assume pre-existing data. Patterns, in order of preference:
|
||||
- **Driver `dispose`** — the driver tracks what it seeded and deletes it after the test, so the spec has no cleanup code. See `dashboards.seed(...)` in `tests/e2e/drivers/dashboards.ts`.
|
||||
- **Per-test seed + cleanup in `try / finally`** — small specs without a driver, where each test owns its data.
|
||||
- **Suite-level seed + `afterAll` teardown** — larger specs without a driver. One `test.afterAll(...)` deletes everything the suite seeded. See `tests/e2e/tests/llm-o11y/llm-pricing.spec.ts`. `test.beforeAll` / `test.afterAll` cannot use `authedPage` directly (it's test-scoped); use `newAdminContext(browser)` from `helpers/auth.ts` instead — it performs one fresh login per suite hook.
|
||||
4. **Self-contained state**: each spec seeds its own data and cleans up at suite teardown. The pytest harness creates a fresh stack with **zero** dashboards / alerts / etc. — never assume pre-existing data. Two patterns work:
|
||||
- **Per-test seed + cleanup in `try / finally`** — small specs where each test owns its data.
|
||||
- **Suite-level seed + `afterAll` teardown** — preferred for larger specs. Each `createDashboard(...)` call adds the resulting ID to a module-level `Set<string>`, and one `test.afterAll(...)` deletes everything in the set. See `tests/e2e/tests/dashboards/list.spec.ts` for the full pattern. `test.beforeAll` / `test.afterAll` cannot use `authedPage` directly (it's test-scoped); use `newAdminContext(browser)` from `helpers/auth.ts` instead — it performs one fresh login per suite hook.
|
||||
5. **Seed via API when the UI flow is multi-step or brittle.** The frontend stores its JWT in `localStorage` under `AUTH_TOKEN`; `page.request.*` inherits the auth fixture's storage state. A typical pattern:
|
||||
```ts
|
||||
const token = await page.evaluate(
|
||||
|
||||
@@ -3579,6 +3579,7 @@ export enum CoretypesKindDTO {
|
||||
'logs-field' = 'logs-field',
|
||||
'traces-field' = 'traces-field',
|
||||
'metrics-field' = 'metrics-field',
|
||||
'llm-pricing-rule' = 'llm-pricing-rule',
|
||||
}
|
||||
export enum CoretypesTypeDTO {
|
||||
user = 'user',
|
||||
|
||||
@@ -74,7 +74,8 @@
|
||||
"meter-metrics",
|
||||
"logs-field",
|
||||
"traces-field",
|
||||
"metrics-field"
|
||||
"metrics-field",
|
||||
"llm-pricing-rule"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
|
||||
@@ -4,14 +4,15 @@ import (
|
||||
"net/http"
|
||||
|
||||
"github.com/SigNoz/signoz/pkg/http/handler"
|
||||
"github.com/SigNoz/signoz/pkg/types"
|
||||
"github.com/SigNoz/signoz/pkg/types/authtypes"
|
||||
"github.com/SigNoz/signoz/pkg/types/coretypes"
|
||||
"github.com/SigNoz/signoz/pkg/types/llmpricingruletypes"
|
||||
"github.com/gorilla/mux"
|
||||
)
|
||||
|
||||
func (provider *provider) addLLMPricingRuleRoutes(router *mux.Router) error {
|
||||
if err := router.Handle("/api/v1/llm_pricing_rules", handler.New(
|
||||
provider.authzMiddleware.ViewAccess(provider.llmPricingRuleHandler.List),
|
||||
provider.authzMiddleware.CheckResources(provider.llmPricingRuleHandler.List, authtypes.SigNozAdminRoleName, authtypes.SigNozEditorRoleName, authtypes.SigNozViewerRoleName),
|
||||
handler.OpenAPIDef{
|
||||
ID: "ListLLMPricingRules",
|
||||
Tags: []string{"llmpricingrules"},
|
||||
@@ -25,14 +26,20 @@ func (provider *provider) addLLMPricingRuleRoutes(router *mux.Router) error {
|
||||
SuccessStatusCode: http.StatusOK,
|
||||
ErrorStatusCodes: []int{http.StatusBadRequest},
|
||||
Deprecated: false,
|
||||
SecuritySchemes: newSecuritySchemes(types.RoleViewer),
|
||||
SecuritySchemes: newScopedSecuritySchemes([]string{coretypes.ResourceMetaResourceLLMPricingRule.Scope(coretypes.VerbList)}),
|
||||
},
|
||||
handler.WithResourceDefs(handler.BasicResourceDef{
|
||||
Resource: coretypes.ResourceMetaResourceLLMPricingRule,
|
||||
Verb: coretypes.VerbList,
|
||||
Category: coretypes.ActionCategoryDataAccess,
|
||||
Selector: coretypes.WildcardSelector,
|
||||
}),
|
||||
)).Methods(http.MethodGet).GetError(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := router.Handle("/api/v1/llm_pricing_rules", handler.New(
|
||||
provider.authzMiddleware.AdminAccess(provider.llmPricingRuleHandler.CreateOrUpdate),
|
||||
provider.authzMiddleware.CheckResources(provider.llmPricingRuleHandler.CreateOrUpdate, authtypes.SigNozAdminRoleName),
|
||||
handler.OpenAPIDef{
|
||||
ID: "CreateOrUpdateLLMPricingRules",
|
||||
Tags: []string{"llmpricingrules"},
|
||||
@@ -43,14 +50,20 @@ func (provider *provider) addLLMPricingRuleRoutes(router *mux.Router) error {
|
||||
SuccessStatusCode: http.StatusNoContent,
|
||||
ErrorStatusCodes: []int{http.StatusBadRequest},
|
||||
Deprecated: false,
|
||||
SecuritySchemes: newSecuritySchemes(types.RoleAdmin),
|
||||
SecuritySchemes: newScopedSecuritySchemes([]string{coretypes.ResourceMetaResourceLLMPricingRule.Scope(coretypes.VerbUpdate)}),
|
||||
},
|
||||
handler.WithResourceDefs(handler.TelemetryResourceDef{
|
||||
Verb: coretypes.VerbUpdate,
|
||||
Category: coretypes.ActionCategoryConfigurationChange,
|
||||
Selector: llmpricingruletypes.RuleSelector,
|
||||
Resources: llmpricingruletypes.UpdatableRulesResources,
|
||||
}),
|
||||
)).Methods(http.MethodPut).GetError(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := router.Handle("/api/v1/llm_pricing_rules/unmapped_models", handler.New(
|
||||
provider.authzMiddleware.ViewAccess(provider.llmPricingRuleHandler.ListUnmappedModels),
|
||||
provider.authzMiddleware.CheckResources(provider.llmPricingRuleHandler.ListUnmappedModels, authtypes.SigNozAdminRoleName, authtypes.SigNozEditorRoleName, authtypes.SigNozViewerRoleName),
|
||||
handler.OpenAPIDef{
|
||||
ID: "ListUnmappedLLMModels",
|
||||
Tags: []string{"llmpricingrules"},
|
||||
@@ -63,14 +76,20 @@ func (provider *provider) addLLMPricingRuleRoutes(router *mux.Router) error {
|
||||
SuccessStatusCode: http.StatusOK,
|
||||
ErrorStatusCodes: []int{http.StatusBadRequest},
|
||||
Deprecated: false,
|
||||
SecuritySchemes: newSecuritySchemes(types.RoleViewer),
|
||||
SecuritySchemes: newScopedSecuritySchemes([]string{coretypes.ResourceMetaResourceLLMPricingRule.Scope(coretypes.VerbList)}),
|
||||
},
|
||||
handler.WithResourceDefs(handler.BasicResourceDef{
|
||||
Resource: coretypes.ResourceMetaResourceLLMPricingRule,
|
||||
Verb: coretypes.VerbList,
|
||||
Category: coretypes.ActionCategoryDataAccess,
|
||||
Selector: coretypes.WildcardSelector,
|
||||
}),
|
||||
)).Methods(http.MethodGet).GetError(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := router.Handle("/api/v1/llm_pricing_rules/{id}", handler.New(
|
||||
provider.authzMiddleware.ViewAccess(provider.llmPricingRuleHandler.Get),
|
||||
provider.authzMiddleware.CheckResources(provider.llmPricingRuleHandler.Get, authtypes.SigNozAdminRoleName, authtypes.SigNozEditorRoleName, authtypes.SigNozViewerRoleName),
|
||||
handler.OpenAPIDef{
|
||||
ID: "GetLLMPricingRule",
|
||||
Tags: []string{"llmpricingrules"},
|
||||
@@ -83,14 +102,21 @@ func (provider *provider) addLLMPricingRuleRoutes(router *mux.Router) error {
|
||||
SuccessStatusCode: http.StatusOK,
|
||||
ErrorStatusCodes: []int{http.StatusNotFound},
|
||||
Deprecated: false,
|
||||
SecuritySchemes: newSecuritySchemes(types.RoleViewer),
|
||||
SecuritySchemes: newScopedSecuritySchemes([]string{coretypes.ResourceMetaResourceLLMPricingRule.Scope(coretypes.VerbRead)}),
|
||||
},
|
||||
handler.WithResourceDefs(handler.BasicResourceDef{
|
||||
Resource: coretypes.ResourceMetaResourceLLMPricingRule,
|
||||
Verb: coretypes.VerbRead,
|
||||
Category: coretypes.ActionCategoryDataAccess,
|
||||
ID: coretypes.PathParam("id"),
|
||||
Selector: coretypes.IDSelector,
|
||||
}),
|
||||
)).Methods(http.MethodGet).GetError(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := router.Handle("/api/v1/llm_pricing_rules/{id}", handler.New(
|
||||
provider.authzMiddleware.AdminAccess(provider.llmPricingRuleHandler.Delete),
|
||||
provider.authzMiddleware.CheckResources(provider.llmPricingRuleHandler.Delete, authtypes.SigNozAdminRoleName),
|
||||
handler.OpenAPIDef{
|
||||
ID: "DeleteLLMPricingRule",
|
||||
Tags: []string{"llmpricingrules"},
|
||||
@@ -103,8 +129,15 @@ func (provider *provider) addLLMPricingRuleRoutes(router *mux.Router) error {
|
||||
SuccessStatusCode: http.StatusNoContent,
|
||||
ErrorStatusCodes: []int{http.StatusNotFound},
|
||||
Deprecated: false,
|
||||
SecuritySchemes: newSecuritySchemes(types.RoleAdmin),
|
||||
SecuritySchemes: newScopedSecuritySchemes([]string{coretypes.ResourceMetaResourceLLMPricingRule.Scope(coretypes.VerbDelete)}),
|
||||
},
|
||||
handler.WithResourceDefs(handler.BasicResourceDef{
|
||||
Resource: coretypes.ResourceMetaResourceLLMPricingRule,
|
||||
Verb: coretypes.VerbDelete,
|
||||
Category: coretypes.ActionCategoryConfigurationChange,
|
||||
ID: coretypes.PathParam("id"),
|
||||
Selector: coretypes.IDSelector,
|
||||
}),
|
||||
)).Methods(http.MethodDelete).GetError(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -266,6 +266,7 @@ func NewSQLMigrationProviderFactories(
|
||||
sqlmigration.NewAddMetricsFieldTuplesFactory(sqlstore),
|
||||
sqlmigration.NewAddGCPConnectionMethodFactory(),
|
||||
sqlmigration.NewDeleteOrphanSARolesFactory(),
|
||||
sqlmigration.NewAddLLMPricingRuleTuplesFactory(sqlstore),
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
162
pkg/sqlmigration/140_add_llm_pricing_rule_tuples.go
Normal file
162
pkg/sqlmigration/140_add_llm_pricing_rule_tuples.go
Normal file
@@ -0,0 +1,162 @@
|
||||
package sqlmigration
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"time"
|
||||
|
||||
"github.com/SigNoz/signoz/pkg/factory"
|
||||
"github.com/SigNoz/signoz/pkg/sqlstore"
|
||||
"github.com/SigNoz/signoz/pkg/types/authtypes"
|
||||
"github.com/SigNoz/signoz/pkg/types/coretypes"
|
||||
"github.com/oklog/ulid/v2"
|
||||
"github.com/uptrace/bun"
|
||||
"github.com/uptrace/bun/dialect"
|
||||
"github.com/uptrace/bun/migrate"
|
||||
)
|
||||
|
||||
type addLLMPricingRuleTuples struct {
|
||||
sqlstore sqlstore.SQLStore
|
||||
}
|
||||
|
||||
func NewAddLLMPricingRuleTuplesFactory(sqlstore sqlstore.SQLStore) factory.ProviderFactory[SQLMigration, Config] {
|
||||
return factory.NewProviderFactory(factory.MustNewName("add_llm_pricing_rule_tuples"), func(ctx context.Context, ps factory.ProviderSettings, c Config) (SQLMigration, error) {
|
||||
return &addLLMPricingRuleTuples{sqlstore: sqlstore}, nil
|
||||
})
|
||||
}
|
||||
|
||||
func (migration *addLLMPricingRuleTuples) Register(migrations *migrate.Migrations) error {
|
||||
return migrations.Register(migration.Up, migration.Down)
|
||||
}
|
||||
|
||||
func (migration *addLLMPricingRuleTuples) Up(ctx context.Context, db *bun.DB) error {
|
||||
tx, err := db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
var storeID string
|
||||
err = tx.QueryRowContext(ctx, `SELECT id FROM store WHERE name = ? LIMIT 1`, "signoz").Scan(&storeID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var orgIDs []string
|
||||
err = tx.NewSelect().
|
||||
Table("organizations").
|
||||
Column("id").
|
||||
Scan(ctx, &orgIDs)
|
||||
if err != nil && err != sql.ErrNoRows {
|
||||
return err
|
||||
}
|
||||
|
||||
isPG := migration.sqlstore.BunDB().Dialect().Name() == dialect.PG
|
||||
|
||||
tuples := []migrationTuple{
|
||||
{authtypes.SigNozAdminRoleName, "metaresource", "llm-pricing-rule", "read"},
|
||||
{authtypes.SigNozAdminRoleName, "metaresource", "llm-pricing-rule", "update"},
|
||||
{authtypes.SigNozAdminRoleName, "metaresource", "llm-pricing-rule", "delete"},
|
||||
{authtypes.SigNozAdminRoleName, "metaresource", "llm-pricing-rule", "list"},
|
||||
{authtypes.SigNozEditorRoleName, "metaresource", "llm-pricing-rule", "read"},
|
||||
{authtypes.SigNozEditorRoleName, "metaresource", "llm-pricing-rule", "list"},
|
||||
{authtypes.SigNozViewerRoleName, "metaresource", "llm-pricing-rule", "read"},
|
||||
{authtypes.SigNozViewerRoleName, "metaresource", "llm-pricing-rule", "list"},
|
||||
}
|
||||
|
||||
for _, orgID := range orgIDs {
|
||||
for _, tuple := range tuples {
|
||||
entropy := ulid.DefaultEntropy()
|
||||
now := time.Now().UTC()
|
||||
tupleID := ulid.MustNew(ulid.Timestamp(now), entropy).String()
|
||||
|
||||
objectID := "organization/" + orgID + "/" + tuple.objectName + "/*"
|
||||
roleSubject := "organization/" + orgID + "/role/" + tuple.roleName
|
||||
|
||||
if isPG {
|
||||
user := "role:" + roleSubject + "#assignee"
|
||||
result, err := tx.ExecContext(ctx, `
|
||||
INSERT INTO tuple (store, object_type, object_id, relation, _user, user_type, ulid, inserted_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (store, object_type, object_id, relation, _user) DO NOTHING`,
|
||||
storeID, tuple.objectType, objectID, tuple.relation, user, "userset", tupleID, now,
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rowsAffected, err := result.RowsAffected()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if rowsAffected == 0 {
|
||||
continue
|
||||
}
|
||||
_, err = tx.ExecContext(ctx, `
|
||||
INSERT INTO changelog (store, object_type, object_id, relation, _user, operation, ulid, inserted_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (store, ulid, object_type) DO NOTHING`,
|
||||
storeID, tuple.objectType, objectID, tuple.relation, user, 0, tupleID, now,
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
result, err := tx.ExecContext(ctx, `
|
||||
INSERT INTO tuple (store, object_type, object_id, relation, user_object_type, user_object_id, user_relation, user_type, ulid, inserted_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (store, object_type, object_id, relation, user_object_type, user_object_id, user_relation) DO NOTHING`,
|
||||
storeID, tuple.objectType, objectID, tuple.relation, "role", roleSubject, "assignee", "userset", tupleID, now,
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rowsAffected, err := result.RowsAffected()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if rowsAffected == 0 {
|
||||
continue
|
||||
}
|
||||
_, err = tx.ExecContext(ctx, `
|
||||
INSERT INTO changelog (store, object_type, object_id, relation, user_object_type, user_object_id, user_relation, operation, ulid, inserted_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (store, ulid, object_type) DO NOTHING`,
|
||||
storeID, tuple.objectType, objectID, tuple.relation, "role", roleSubject, "assignee", 0, tupleID, now,
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
managedRoleGroups := make(map[string]string, len(coretypes.ManagedRoleToTransactions))
|
||||
for roleName, transactions := range coretypes.ManagedRoleToTransactions {
|
||||
data, err := json.Marshal(authtypes.NewTransactionGroupsFromTransactions(transactions))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
managedRoleGroups[roleName] = string(data)
|
||||
}
|
||||
|
||||
for _, orgID := range orgIDs {
|
||||
for roleName, data := range managedRoleGroups {
|
||||
if _, err := tx.NewUpdate().
|
||||
Model(new(roles)).
|
||||
Set("transaction_groups = ?", data).
|
||||
Where("org_id = ?", orgID).
|
||||
Where("type = ?", authtypes.RoleTypeManaged.StringValue()).
|
||||
Where("name = ?", roleName).
|
||||
Exec(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
func (migration *addLLMPricingRuleTuples) Down(context.Context, *bun.DB) error {
|
||||
return nil
|
||||
}
|
||||
@@ -80,6 +80,7 @@ func (name Kind) Enum() []any {
|
||||
KindLogsField,
|
||||
KindTracesField,
|
||||
KindMetricsField,
|
||||
KindLLMPricingRule,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -40,6 +40,7 @@ var Kinds = []Kind{
|
||||
KindLogsField,
|
||||
KindTracesField,
|
||||
KindMetricsField,
|
||||
KindLLMPricingRule,
|
||||
}
|
||||
|
||||
var (
|
||||
@@ -82,4 +83,5 @@ var (
|
||||
KindLogsField = MustNewKind("logs-field")
|
||||
KindTracesField = MustNewKind("traces-field")
|
||||
KindMetricsField = MustNewKind("metrics-field")
|
||||
KindLLMPricingRule = MustNewKind("llm-pricing-rule")
|
||||
)
|
||||
|
||||
@@ -218,6 +218,11 @@ var ManagedRoleToTransactions = map[string][]Transaction{
|
||||
{Verb: VerbRead, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindMetricsField}, WildCardSelectorString)},
|
||||
{Verb: VerbUpdate, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindMetricsField}, WildCardSelectorString)},
|
||||
{Verb: VerbList, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindMetricsField}, WildCardSelectorString)},
|
||||
// llm-pricing-rule — admin writes, viewer reads
|
||||
{Verb: VerbRead, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindLLMPricingRule}, WildCardSelectorString)},
|
||||
{Verb: VerbUpdate, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindLLMPricingRule}, WildCardSelectorString)},
|
||||
{Verb: VerbDelete, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindLLMPricingRule}, WildCardSelectorString)},
|
||||
{Verb: VerbList, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindLLMPricingRule}, WildCardSelectorString)},
|
||||
},
|
||||
SigNozEditorRoleName: {
|
||||
// cloud-integration — editor can read accounts
|
||||
@@ -317,6 +322,9 @@ var ManagedRoleToTransactions = map[string][]Transaction{
|
||||
{Verb: VerbRead, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindMetricsField}, WildCardSelectorString)},
|
||||
{Verb: VerbUpdate, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindMetricsField}, WildCardSelectorString)},
|
||||
{Verb: VerbList, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindMetricsField}, WildCardSelectorString)},
|
||||
// llm-pricing-rule — read only (admin writes)
|
||||
{Verb: VerbRead, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindLLMPricingRule}, WildCardSelectorString)},
|
||||
{Verb: VerbList, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindLLMPricingRule}, WildCardSelectorString)},
|
||||
},
|
||||
SigNozViewerRoleName: {
|
||||
// cloud-integration — viewer can read accounts
|
||||
@@ -381,6 +389,9 @@ var ManagedRoleToTransactions = map[string][]Transaction{
|
||||
// metrics-field — viewer reads
|
||||
{Verb: VerbRead, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindMetricsField}, WildCardSelectorString)},
|
||||
{Verb: VerbList, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindMetricsField}, WildCardSelectorString)},
|
||||
// llm-pricing-rule — read only
|
||||
{Verb: VerbRead, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindLLMPricingRule}, WildCardSelectorString)},
|
||||
{Verb: VerbList, Object: *MustNewObject(ResourceRef{Type: TypeMetaResource, Kind: KindLLMPricingRule}, WildCardSelectorString)},
|
||||
},
|
||||
SigNozAnonymousRoleName: {
|
||||
// public-dashboard — anonymous read
|
||||
|
||||
@@ -40,6 +40,7 @@ var Resources = []Resource{
|
||||
ResourceMetaResourceLogsField,
|
||||
ResourceMetaResourceTracesField,
|
||||
ResourceMetaResourceMetricsField,
|
||||
ResourceMetaResourceLLMPricingRule,
|
||||
}
|
||||
|
||||
var (
|
||||
@@ -82,4 +83,5 @@ var (
|
||||
ResourceMetaResourceLogsField = NewResourceMetaResource(KindLogsField)
|
||||
ResourceMetaResourceTracesField = NewResourceMetaResource(KindTracesField)
|
||||
ResourceMetaResourceMetricsField = NewResourceMetaResource(KindMetricsField, VerbList, VerbRead, VerbUpdate)
|
||||
ResourceMetaResourceLLMPricingRule = NewResourceMetaResource(KindLLMPricingRule, VerbList, VerbRead, VerbUpdate, VerbDelete)
|
||||
)
|
||||
|
||||
65
pkg/types/llmpricingruletypes/resource.go
Normal file
65
pkg/types/llmpricingruletypes/resource.go
Normal file
@@ -0,0 +1,65 @@
|
||||
package llmpricingruletypes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"slices"
|
||||
|
||||
"github.com/SigNoz/signoz/pkg/errors"
|
||||
"github.com/SigNoz/signoz/pkg/types/coretypes"
|
||||
"github.com/SigNoz/signoz/pkg/valuer"
|
||||
)
|
||||
|
||||
// UpdatableRulesResources resolves the pricing rules an upsert body writes to.
|
||||
//
|
||||
// A rule with isOverride and an id is an edit, checked by that id. Everything
|
||||
// else resolves to the wildcard: isOverride without an id is a create, sourceId
|
||||
// without isOverride is a sync row, and an empty list has nothing to check.
|
||||
func UpdatableRulesResources(ec coretypes.ExtractorContext) ([]coretypes.ResourceWithID, error) {
|
||||
// encoding/json with the handler's tags, so duplicate or case-variant id
|
||||
// keys resolve to the id that is actually written.
|
||||
var body struct {
|
||||
Rules []*struct {
|
||||
// Normalised through valuer so an uppercase id still matches the
|
||||
// lowercase selector regex of the metaresource type.
|
||||
ID *valuer.UUID `json:"id"`
|
||||
IsOverride *bool `json:"isOverride"`
|
||||
} `json:"rules"`
|
||||
}
|
||||
if err := json.Unmarshal(ec.RequestBody, &body); err != nil {
|
||||
return nil, errors.Wrapf(err, errors.TypeInvalidInput, ErrCodePricingRuleInvalidInput, "invalid pricing rules body")
|
||||
}
|
||||
|
||||
ids := make([]string, 0, len(body.Rules))
|
||||
wildcard := len(body.Rules) == 0
|
||||
for _, rule := range body.Rules {
|
||||
if rule == nil || rule.ID == nil || rule.IsOverride == nil {
|
||||
wildcard = true
|
||||
continue
|
||||
}
|
||||
|
||||
if !slices.Contains(ids, rule.ID.StringValue()) {
|
||||
ids = append(ids, rule.ID.StringValue())
|
||||
}
|
||||
}
|
||||
if wildcard {
|
||||
ids = append(ids, coretypes.WildCardSelectorString)
|
||||
}
|
||||
|
||||
resources := make([]coretypes.ResourceWithID, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
resources = append(resources, coretypes.ResourceWithID{Resource: coretypes.ResourceMetaResourceLLMPricingRule, ID: id})
|
||||
}
|
||||
|
||||
return resources, nil
|
||||
}
|
||||
|
||||
// RuleSelector exists because IDSelector would turn the wildcard entry into
|
||||
// the wildcard twice; an id still resolves to the id plus the wildcard.
|
||||
func RuleSelector(ctx context.Context, resource coretypes.Resource, id string, orgID valuer.UUID) ([]coretypes.Selector, error) {
|
||||
if id == coretypes.WildCardSelectorString {
|
||||
return coretypes.WildcardSelector(ctx, resource, id, orgID)
|
||||
}
|
||||
|
||||
return coretypes.IDSelector(ctx, resource, id, orgID)
|
||||
}
|
||||
116
pkg/types/llmpricingruletypes/resource_test.go
Normal file
116
pkg/types/llmpricingruletypes/resource_test.go
Normal file
@@ -0,0 +1,116 @@
|
||||
package llmpricingruletypes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/SigNoz/signoz/pkg/types/coretypes"
|
||||
"github.com/SigNoz/signoz/pkg/valuer"
|
||||
)
|
||||
|
||||
func TestUpdatableRulesResources(t *testing.T) {
|
||||
testCases := []struct {
|
||||
name string
|
||||
body string
|
||||
want []string
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "EmptyRules_Wildcard",
|
||||
body: `{"rules":[]}`,
|
||||
want: []string{"*"},
|
||||
},
|
||||
{
|
||||
name: "Edit_ID",
|
||||
body: `{"rules":[{"id":"0b6d1a4e-5c1f-4d2a-9e3b-1f2a3b4c5d6e","isOverride":true}]}`,
|
||||
want: []string{"0b6d1a4e-5c1f-4d2a-9e3b-1f2a3b4c5d6e"},
|
||||
},
|
||||
{
|
||||
name: "Create_Wildcard",
|
||||
body: `{"rules":[{"modelName":"gpt","isOverride":true}]}`,
|
||||
want: []string{"*"},
|
||||
},
|
||||
{
|
||||
name: "SyncRow_Wildcard",
|
||||
body: `{"rules":[{"sourceId":"4e9a0b12-3c4d-4e5f-8a6b-7c8d9e0f1a2b"}]}`,
|
||||
want: []string{"*"},
|
||||
},
|
||||
{
|
||||
name: "Mixed_DedupedIDsThenWildcardOnce",
|
||||
body: `{"rules":[{"id":"6abc2d34-5e6f-4a7b-8c8d-9e0f1a2b3c4d","isOverride":true},{"modelName":"new","isOverride":true},{"id":"6abc2d34-5e6f-4a7b-8c8d-9e0f1a2b3c4d","isOverride":false},{"sourceId":"7bcd3e45-6f7a-4b8c-9d9e-0f1a2b3c4d5e"}]}`,
|
||||
want: []string{"6abc2d34-5e6f-4a7b-8c8d-9e0f1a2b3c4d", "*"},
|
||||
},
|
||||
{
|
||||
name: "UppercaseID_Lowercased",
|
||||
body: `{"rules":[{"id":"8CDE4F56-7A8B-4C9D-8E0F-1A2B3C4D5E6F","isOverride":true}]}`,
|
||||
want: []string{"8cde4f56-7a8b-4c9d-8e0f-1a2b3c4d5e6f"},
|
||||
},
|
||||
{
|
||||
name: "DuplicateIDKey_LastWins",
|
||||
body: `{"rules":[{"id":"0b6d1a4e-5c1f-4d2a-9e3b-1f2a3b4c5d6e","id":"6abc2d34-5e6f-4a7b-8c8d-9e0f1a2b3c4d","isOverride":true}]}`,
|
||||
want: []string{"6abc2d34-5e6f-4a7b-8c8d-9e0f1a2b3c4d"},
|
||||
},
|
||||
{
|
||||
name: "CaseVariantIDKey_LastWins",
|
||||
body: `{"rules":[{"id":"0b6d1a4e-5c1f-4d2a-9e3b-1f2a3b4c5d6e","ID":"6abc2d34-5e6f-4a7b-8c8d-9e0f1a2b3c4d","isOverride":true}]}`,
|
||||
want: []string{"6abc2d34-5e6f-4a7b-8c8d-9e0f1a2b3c4d"},
|
||||
},
|
||||
{
|
||||
name: "InvalidID_Rejected",
|
||||
body: `{"rules":[{"id":"not-a-uuid","isOverride":true}]}`,
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, testCase := range testCases {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
resources, err := UpdatableRulesResources(coretypes.ExtractorContext{RequestBody: []byte(testCase.body)})
|
||||
if testCase.wantErr {
|
||||
require.Error(t, err)
|
||||
return
|
||||
}
|
||||
|
||||
require.NoError(t, err)
|
||||
got := make([]string, 0, len(resources))
|
||||
for _, resource := range resources {
|
||||
got = append(got, resource.ID)
|
||||
}
|
||||
assert.Equal(t, testCase.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRuleSelector(t *testing.T) {
|
||||
testCases := []struct {
|
||||
name string
|
||||
id string
|
||||
want []string
|
||||
}{
|
||||
{
|
||||
name: "Wildcard_WildcardOnly",
|
||||
id: "*",
|
||||
want: []string{"*"},
|
||||
},
|
||||
{
|
||||
name: "ID_IDThenWildcard",
|
||||
id: "a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d",
|
||||
want: []string{"a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", "*"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, testCase := range testCases {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
selectors, err := RuleSelector(context.Background(), coretypes.ResourceMetaResourceLLMPricingRule, testCase.id, valuer.GenerateUUID())
|
||||
require.NoError(t, err)
|
||||
|
||||
got := make([]string, 0, len(selectors))
|
||||
for _, selector := range selectors {
|
||||
got = append(got, selector.String())
|
||||
}
|
||||
assert.Equal(t, testCase.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -297,6 +297,16 @@
|
||||
"list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "metaresource",
|
||||
"kind": "llm-pricing-rule",
|
||||
"verbs": [
|
||||
"read",
|
||||
"update",
|
||||
"delete",
|
||||
"list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "metaresource",
|
||||
"kind": "ttl-setting",
|
||||
@@ -528,6 +538,14 @@
|
||||
"list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "metaresource",
|
||||
"kind": "llm-pricing-rule",
|
||||
"verbs": [
|
||||
"read",
|
||||
"list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "metaresource",
|
||||
"kind": "ttl-setting",
|
||||
@@ -707,6 +725,14 @@
|
||||
"list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "metaresource",
|
||||
"kind": "llm-pricing-rule",
|
||||
"verbs": [
|
||||
"read",
|
||||
"list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "metaresource",
|
||||
"kind": "ttl-setting",
|
||||
|
||||
267
tests/integration/tests/llmpricingrules/02_authz.py
Normal file
267
tests/integration/tests/llmpricingrules/02_authz.py
Normal file
@@ -0,0 +1,267 @@
|
||||
from collections.abc import Callable
|
||||
from http import HTTPStatus
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
from wiremock.resources.mappings import Mapping
|
||||
|
||||
from fixtures import types
|
||||
from fixtures.auth import (
|
||||
USER_ADMIN_EMAIL,
|
||||
USER_ADMIN_PASSWORD,
|
||||
add_license,
|
||||
change_user_role,
|
||||
create_active_user,
|
||||
)
|
||||
from fixtures.llmpricingrules import (
|
||||
LLM_PRICING_RULES_URL,
|
||||
delete_all_llm_pricing_rules,
|
||||
list_llm_pricing_rules,
|
||||
upsert_llm_pricing_rules,
|
||||
)
|
||||
from fixtures.role import find_role_by_name, transaction_group
|
||||
|
||||
user_password = "password123Z$"
|
||||
editor_email = "llm-pricing-editor@integration.test"
|
||||
viewer_email = "llm-pricing-viewer@integration.test"
|
||||
updater_role = "llm-pricing-updater"
|
||||
updater_email = "llm-pricing-updater@integration.test"
|
||||
scoped_role = "llm-pricing-scoped"
|
||||
scoped_email = "llm-pricing-scoped@integration.test"
|
||||
|
||||
PROBE_RULE = {
|
||||
"modelName": "authz-probe",
|
||||
"provider": "OpenAI",
|
||||
"modelPattern": ["authz-probe*"],
|
||||
"unit": "per_million_tokens",
|
||||
"pricing": {"input": 1, "output": 2},
|
||||
"isOverride": True,
|
||||
"enabled": True,
|
||||
}
|
||||
|
||||
OTHER_RULE = {
|
||||
"modelName": "authz-probe-other",
|
||||
"provider": "Anthropic",
|
||||
"modelPattern": ["authz-probe-other*"],
|
||||
"unit": "per_million_tokens",
|
||||
"pricing": {"input": 3, "output": 4},
|
||||
"isOverride": True,
|
||||
"enabled": True,
|
||||
}
|
||||
|
||||
|
||||
def test_setup(
|
||||
signoz: types.SigNoz,
|
||||
create_user_admin: types.Operation, # pylint: disable=unused-argument
|
||||
make_http_mocks: Callable[[types.TestContainerDocker, list[Mapping]], None],
|
||||
get_token: Callable[[str, str], str],
|
||||
create_role: Callable[..., str],
|
||||
) -> None:
|
||||
add_license(signoz, make_http_mocks, get_token)
|
||||
|
||||
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
delete_all_llm_pricing_rules(signoz, admin_token)
|
||||
assert upsert_llm_pricing_rules(signoz, admin_token, [PROBE_RULE, OTHER_RULE]).status_code == HTTPStatus.NO_CONTENT
|
||||
probe_id = next(r["id"] for r in list_llm_pricing_rules(signoz, admin_token) if r["modelName"] == PROBE_RULE["modelName"])
|
||||
|
||||
create_active_user(signoz, admin_token, email=editor_email, role="signoz-editor", password=user_password)
|
||||
create_active_user(signoz, admin_token, email=viewer_email, role="signoz-viewer", password=user_password)
|
||||
|
||||
create_role(admin_token, updater_role, [transaction_group("update", "metaresource", "llm-pricing-rule", ["*"])])
|
||||
updater = create_active_user(signoz, admin_token, email=updater_email, role="signoz-viewer", password=user_password)
|
||||
change_user_role(signoz, admin_token, updater, "signoz-viewer", updater_role)
|
||||
|
||||
create_role(
|
||||
admin_token,
|
||||
scoped_role,
|
||||
[
|
||||
transaction_group("read", "metaresource", "llm-pricing-rule", [probe_id]),
|
||||
transaction_group("update", "metaresource", "llm-pricing-rule", [probe_id]),
|
||||
transaction_group("delete", "metaresource", "llm-pricing-rule", [probe_id]),
|
||||
],
|
||||
)
|
||||
scoped = create_active_user(signoz, admin_token, email=scoped_email, role="signoz-viewer", password=user_password)
|
||||
change_user_role(signoz, admin_token, scoped, "signoz-viewer", scoped_role)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path",
|
||||
[LLM_PRICING_RULES_URL, f"{LLM_PRICING_RULES_URL}/unmapped_models"],
|
||||
ids=["list", "unmapped_models"],
|
||||
)
|
||||
@pytest.mark.parametrize(
|
||||
("email", "expected"),
|
||||
[
|
||||
(USER_ADMIN_EMAIL, HTTPStatus.OK),
|
||||
(editor_email, HTTPStatus.OK),
|
||||
(viewer_email, HTTPStatus.OK),
|
||||
(updater_email, HTTPStatus.FORBIDDEN),
|
||||
(scoped_email, HTTPStatus.FORBIDDEN),
|
||||
],
|
||||
ids=["admin", "editor", "viewer", "updater", "scoped"],
|
||||
)
|
||||
def test_reads_need_list_grant(
|
||||
signoz: types.SigNoz,
|
||||
get_token: Callable[[str, str], str],
|
||||
path: str,
|
||||
email: str,
|
||||
expected: HTTPStatus,
|
||||
) -> None:
|
||||
token = get_token(email, USER_ADMIN_PASSWORD if email == USER_ADMIN_EMAIL else user_password)
|
||||
|
||||
response = requests.get(
|
||||
signoz.self.host_configs["8080"].get(path),
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
timeout=10,
|
||||
)
|
||||
assert response.status_code == expected, response.text
|
||||
if expected == HTTPStatus.FORBIDDEN:
|
||||
assert "llm-pricing-rule:list" in response.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("email", "model_name", "expected"),
|
||||
[
|
||||
(USER_ADMIN_EMAIL, PROBE_RULE["modelName"], HTTPStatus.OK),
|
||||
(editor_email, PROBE_RULE["modelName"], HTTPStatus.OK),
|
||||
(viewer_email, PROBE_RULE["modelName"], HTTPStatus.OK),
|
||||
(updater_email, PROBE_RULE["modelName"], HTTPStatus.FORBIDDEN),
|
||||
(scoped_email, PROBE_RULE["modelName"], HTTPStatus.OK),
|
||||
(scoped_email, OTHER_RULE["modelName"], HTTPStatus.FORBIDDEN),
|
||||
],
|
||||
ids=["admin", "editor", "viewer", "updater", "scoped_granted", "scoped_other"],
|
||||
)
|
||||
def test_get_needs_read_grant(
|
||||
signoz: types.SigNoz,
|
||||
get_token: Callable[[str, str], str],
|
||||
email: str,
|
||||
model_name: str,
|
||||
expected: HTTPStatus,
|
||||
) -> None:
|
||||
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
rule_id = next(r["id"] for r in list_llm_pricing_rules(signoz, admin_token) if r["modelName"] == model_name)
|
||||
token = get_token(email, USER_ADMIN_PASSWORD if email == USER_ADMIN_EMAIL else user_password)
|
||||
|
||||
response = requests.get(
|
||||
signoz.self.host_configs["8080"].get(f"{LLM_PRICING_RULES_URL}/{rule_id}"),
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
timeout=10,
|
||||
)
|
||||
assert response.status_code == expected, response.text
|
||||
if expected == HTTPStatus.FORBIDDEN:
|
||||
assert "llm-pricing-rule:read" in response.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("email", "expected"),
|
||||
[
|
||||
(editor_email, HTTPStatus.FORBIDDEN),
|
||||
(viewer_email, HTTPStatus.FORBIDDEN),
|
||||
(scoped_email, HTTPStatus.NO_CONTENT),
|
||||
(updater_email, HTTPStatus.NO_CONTENT),
|
||||
(USER_ADMIN_EMAIL, HTTPStatus.NO_CONTENT),
|
||||
],
|
||||
ids=["editor", "viewer", "scoped", "updater", "admin"],
|
||||
)
|
||||
def test_upsert_needs_update_grant(
|
||||
signoz: types.SigNoz,
|
||||
get_token: Callable[[str, str], str],
|
||||
email: str,
|
||||
expected: HTTPStatus,
|
||||
) -> None:
|
||||
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
rule_id = next(r["id"] for r in list_llm_pricing_rules(signoz, admin_token) if r["modelName"] == PROBE_RULE["modelName"])
|
||||
token = get_token(email, USER_ADMIN_PASSWORD if email == USER_ADMIN_EMAIL else user_password)
|
||||
|
||||
response = upsert_llm_pricing_rules(signoz, token, [{**PROBE_RULE, "id": rule_id}])
|
||||
assert response.status_code == expected, response.text
|
||||
if expected == HTTPStatus.FORBIDDEN:
|
||||
assert "llm-pricing-rule:update" in response.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"shape",
|
||||
["other_rule", "create", "sync_row", "own_and_other"],
|
||||
ids=["other_rule", "create", "sync_row", "own_and_other"],
|
||||
)
|
||||
def test_upsert_scoped_to_granted_rule(
|
||||
signoz: types.SigNoz,
|
||||
get_token: Callable[[str, str], str],
|
||||
shape: str,
|
||||
) -> None:
|
||||
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
rules = {r["modelName"]: r for r in list_llm_pricing_rules(signoz, admin_token)}
|
||||
own = {**PROBE_RULE, "id": rules[PROBE_RULE["modelName"]]["id"]}
|
||||
other = {**OTHER_RULE, "id": rules[OTHER_RULE["modelName"]]["id"]}
|
||||
sync_row = {**{k: v for k, v in PROBE_RULE.items() if k != "isOverride"}, "sourceId": "11111111-1111-4111-8111-111111111199"}
|
||||
bodies = {"other_rule": [other], "create": [PROBE_RULE], "sync_row": [sync_row], "own_and_other": [own, other]}
|
||||
|
||||
response = upsert_llm_pricing_rules(signoz, get_token(scoped_email, user_password), bodies[shape])
|
||||
assert response.status_code == HTTPStatus.FORBIDDEN, response.text
|
||||
assert "llm-pricing-rule:update" in response.text
|
||||
|
||||
|
||||
def test_revoke_update_grant(
|
||||
signoz: types.SigNoz,
|
||||
get_token: Callable[[str, str], str],
|
||||
) -> None:
|
||||
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
rule_id = next(r["id"] for r in list_llm_pricing_rules(signoz, admin_token) if r["modelName"] == PROBE_RULE["modelName"])
|
||||
|
||||
response = requests.put(
|
||||
signoz.self.host_configs["8080"].get(f"/api/v1/roles/{find_role_by_name(signoz, admin_token, scoped_role)}"),
|
||||
json={
|
||||
"description": "",
|
||||
"transactionGroups": [
|
||||
transaction_group("read", "metaresource", "llm-pricing-rule", [rule_id]),
|
||||
transaction_group("delete", "metaresource", "llm-pricing-rule", [rule_id]),
|
||||
],
|
||||
},
|
||||
headers={"Authorization": f"Bearer {admin_token}"},
|
||||
timeout=10,
|
||||
)
|
||||
assert response.status_code == HTTPStatus.NO_CONTENT, response.text
|
||||
|
||||
response = upsert_llm_pricing_rules(signoz, get_token(scoped_email, user_password), [{**PROBE_RULE, "id": rule_id}])
|
||||
assert response.status_code == HTTPStatus.FORBIDDEN, response.text
|
||||
assert "llm-pricing-rule:update" in response.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("email", "model_name", "expected"),
|
||||
[
|
||||
(editor_email, PROBE_RULE["modelName"], HTTPStatus.FORBIDDEN),
|
||||
(viewer_email, PROBE_RULE["modelName"], HTTPStatus.FORBIDDEN),
|
||||
(updater_email, PROBE_RULE["modelName"], HTTPStatus.FORBIDDEN),
|
||||
(scoped_email, OTHER_RULE["modelName"], HTTPStatus.FORBIDDEN),
|
||||
(scoped_email, PROBE_RULE["modelName"], HTTPStatus.NO_CONTENT),
|
||||
(USER_ADMIN_EMAIL, OTHER_RULE["modelName"], HTTPStatus.NO_CONTENT),
|
||||
],
|
||||
ids=["editor", "viewer", "updater", "scoped_other", "scoped_granted", "admin"],
|
||||
)
|
||||
def test_delete_needs_delete_grant(
|
||||
signoz: types.SigNoz,
|
||||
get_token: Callable[[str, str], str],
|
||||
email: str,
|
||||
model_name: str,
|
||||
expected: HTTPStatus,
|
||||
) -> None:
|
||||
admin_token = get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD)
|
||||
rule_id = next(r["id"] for r in list_llm_pricing_rules(signoz, admin_token) if r["modelName"] == model_name)
|
||||
token = get_token(email, USER_ADMIN_PASSWORD if email == USER_ADMIN_EMAIL else user_password)
|
||||
|
||||
response = requests.delete(
|
||||
signoz.self.host_configs["8080"].get(f"{LLM_PRICING_RULES_URL}/{rule_id}"),
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
timeout=10,
|
||||
)
|
||||
assert response.status_code == expected, response.text
|
||||
if expected == HTTPStatus.FORBIDDEN:
|
||||
assert "llm-pricing-rule:delete" in response.text
|
||||
|
||||
|
||||
def test_cleanup(
|
||||
signoz: types.SigNoz,
|
||||
get_token: Callable[[str, str], str],
|
||||
) -> None:
|
||||
delete_all_llm_pricing_rules(signoz, get_token(USER_ADMIN_EMAIL, USER_ADMIN_PASSWORD))
|
||||
Reference in New Issue
Block a user