vijay_admin/sentryagent-idp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: commit all Phase 6 documentation updates and OpenSpec archives

Browse Source 
- devops docs: 8 files updated for Phase 6 state; field-trial.md added (946-line runbook)
- developer docs: api-reference (50+ endpoints), quick-start, 5 existing guides updated, 5 new guides added
- engineering docs: all 12 files updated (services, architecture, SDK guide, testing, overview)
- OpenSpec archives: phase-7-devops-field-trial, developer-docs-phase6-update, engineering-docs-phase6-update
- VALIDATOR.md + scripts/start-validator.sh: V&V Architect tooling added
- .gitignore: exclude session artifacts, build artifacts, and agent workspaces

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
SentryAgent.ai Developer
2026-04-07 02:24:24 +00:00
parent 0fb00256b4
commit 8cabc0191c
56 changed files with 12780 additions and 446 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/devops/README.md
View File

@@ -14,14 +14,15 @@ SentryAgent.ai AgentIdP is a Node.js REST API backed by PostgreSQL and Redis. It
## Documentation
| Document | What it covers |
|----------|----------------|
| [Architecture](architecture.md) | Components, ports, data flow, Redis key patterns |
| [Environment Variables](environment-variables.md) | Every env var — required, optional, format, examples |
| [Database](database.md) | Schema (4 tables), migrations, how to apply and verify |
| [Local Development](local-development.md) | docker-compose setup, startup, health checks |
| [Security](security.md) | JWT key generation and rotation, CORS, secret storage |
| [Operations](operations.md) | Startup order, graceful shutdown, log interpretation, troubleshooting |
| Document | Audience | Contents |
|----------|----------|---------|
| [Architecture](architecture.md) | All engineers | Components, ports, data flow, Redis key patterns |
| [Environment Variables](environment-variables.md) | All engineers | Every env var — required, optional, format, examples |
| [Database](database.md) | Backend, DevOps | Schema (26 tables/migrations), how to apply and verify |
| [Local Development](local-development.md) | All engineers | docker-compose setup, startup, health checks |
| [Security](security.md) | All engineers | JWT key generation and rotation, CORS, secret storage |
| [Operations](operations.md) | DevOps | Startup order, graceful shutdown, log interpretation, troubleshooting |
| [field-trial.md](field-trial.md) | DevOps engineers, QA | In-house Docker Compose field trial execution playbook |
## Quick Reference — Ports

162
docs/devops/architecture.md
View File

@@ -3,26 +3,49 @@
## Component Overview
```
┌─────────────────────────────────────┐
│ AgentIdP Application │
│ Node.js / Express │
│ Port 3000 │
│ │
│ Auth MW → RateLimit MW → Routes │
│ ↓ ↓ │
│ Controllers → Services → Repos │
└──────────────┬──────────────┬────────┘
│ │
┌──────────────▼──┐ ┌───────▼────────┐
│ PostgreSQL 14 │ │ Redis 7 │
│ Port 5432 │ │ Port 6379 │
│ │ │ │
│ agents │ │ Token revoke │
│ credentials │ │ Rate limits │
│ audit_events │ │ Monthly counts │
│ token_revocati- │ │ │
│ ons │ │ │
└──────────────────┘ └─────────────────┘
┌───────────────────────────────────────────┐
│ Next.js Portal (port 3001) │
│ portal/ — Next.js 14 │
│ /login /agents /credentials /audit │
│ /analytics /settings/tier /compliance │
│ /webhooks /marketplace │
└────────────────┬──────────────────────────┘
│ HTTP (localhost:3000)
┌────────────────▼──────────────────────────┐
│ AgentIdP Application │
│ Node.js / Express (port 3000) │
│ │
│ TLS MW → Helmet → CORS → Morgan │
│ Metrics MW → OrgContext MW │
│ UsageMetering MW → TierEnforcement MW │
│ Auth MW → OPA MW → Routes │
│ ↓ │
│ Controllers → Services → Repos │
└──────────┬───────────────┬────────────────┘
│ │
┌────────────────▼──┐ ┌────────▼────────┐
│ PostgreSQL 14 │ │ Redis 7 │
│ Port 5432 │ │ Port 6379 │
│ │ │ │
│ 26 migrations │ │ Rate limits │
│ (001–026) │ │ Token revoke │
│ organizations │ │ Monthly counts │
│ agents + DID keys │ │ Tier counters │
│ credentials │ │ Compliance cache│
│ audit_events │ │ │
│ token_revocations │ └──────────────────┘
│ oidc_keys │
│ federation_partne-│ ┌──────────────────┐
│ rs │ │ HashiCorp Vault │
│ webhook_subscript-│ │ (optional) │
│ ions + deliveries │ │ KV v2 — creds │
│ agent_marketplace │ └──────────────────┘
│ github_oidc_trust │
│ billing │ ┌──────────────────┐
│ delegation_chains │ │ Stripe │
│ analytics_events │ │ (optional) │
│ tenant_tiers │ │ Billing/upgrades │
└────────────────────┘ └──────────────────┘
```
## Components
@@ -36,8 +59,12 @@ A stateless Express HTTP server. Every request is handled independently — no i
| Layer | Responsibility |
|-------|---------------|
| Routes | Wire HTTP methods and paths to controllers |
| TLS middleware | Redirect HTTP → HTTPS when `ENFORCE_TLS=true` |
| Auth middleware | Validate Bearer JWT (RS256 + Redis revocation check) |
| Rate limit middleware | Redis sliding-window counter per `client_id` |
| OrgContext middleware | Resolve `organization_id` from JWT and attach to `req` |
| UsageMetering middleware | Fire-and-forget analytics event recording |
| TierEnforcement middleware | Enforce daily API call and token limits via Redis (when `TIER_ENFORCEMENT=true`) |
| OPA middleware | Scope-based authorization via embedded Wasm or JSON policy |
| Controllers | Parse and validate request, call service, return response |
| Services | Business logic — no direct DB access |
| Repositories | All SQL queries — no business logic |
@@ -53,11 +80,14 @@ The application connects via a connection pool (`pg.Pool`) initialised from `DAT
Ephemeral store for three use cases:
| Key pattern | Purpose | TTL |
|------------|---------|-----|
| `revoked:<jti>` | Token revocation list — checked on every authenticated request | Until token's `exp` |
| `rate:<client_id>:<window>` | Request count per client per 60-second window | 60 seconds |
| `monthly:<client_id>:<year>:<month>` | Token issuance count for free tier limit enforcement | End of month |
| Key pattern | Example | Purpose | TTL |
|------------|---------|---------|-----|
| `revoked:<jti>` | `revoked:f1e2d3c4-...` | Revoked token JTI | Remaining token lifetime |
| `rate:<client_id>:<window>` | `rate:a1b2c3...:29086156` | Request count per window | `RATE_LIMIT_WINDOW_MS` |
| `monthly:<client_id>:<year>:<month>` | `monthly:a1b2c3...:2026:3` | Monthly token issuance count | End of month |
| `rate:tier:calls:<tenantId>` | `rate:tier:calls:org-uuid` | Daily API call counter for tier enforcement | Until midnight UTC |
| `rate:tier:tokens:<tenantId>` | `rate:tier:tokens:org-uuid` | Daily token issuance counter for tier enforcement | Until midnight UTC |
| `compliance:report:<tenantId>` | `compliance:report:org-uuid` | Cached compliance report JSON | 5 minutes |
**Redis is supplementary, not the source of truth.** Token revocations are also written to the `token_revocations` PostgreSQL table for durability across Redis restarts. On Redis restart, the revocation list is cold — previously revoked tokens will pass auth until the PostgreSQL-backed warm-up is implemented (Phase 2).
@@ -107,21 +137,89 @@ PostgreSQL / Redis
## Service Map
| Route prefix | Service | Repository |
|-------------|---------|-----------|
| `/api/v1/agents` | `AgentService` | `AgentRepository` |
| `/api/v1/agents/:id/credentials` | `CredentialService` | `CredentialRepository` |
| `/api/v1/token` | `OAuth2Service` | `TokenRepository`, `CredentialRepository`, `AgentRepository` |
| `/api/v1/audit` | `AuditService` | `AuditRepository` |
| Route prefix | Controller | Service(s) | Repository/ies |
|-------------|-----------|-----------|----------------|
| `/api/v1/agents` | `AgentController` | `AgentService` | `AgentRepository` |
| `/api/v1/credentials` | `CredentialController` | `CredentialService` | `CredentialRepository` |
| `/api/v1/token` | `TokenController` | `OAuth2Service` | `TokenRepository`, `CredentialRepository`, `AgentRepository` |
| `/api/v1/audit` | `AuditController` | `AuditService` | `AuditRepository` |
| `/api/v1/organizations` | `OrgController` | `OrgService` | `OrgRepository` |
| `/api/v1/compliance/*` | `ComplianceController` | `ComplianceService` | `AuditRepository` |
| `/api/v1/analytics/*` | `AnalyticsController` | `AnalyticsService` | direct pool queries |
| `/api/v1/tiers/*` | `TierController` | `TierService` | pool queries, Stripe SDK |
| `/api/v1/webhooks` | `WebhookController` | `WebhookService` | `WebhookRepository` |
| `/api/v1/federation` | `FederationController` | `FederationService` | direct pool queries |
| `/api/v1/marketplace` | `MarketplaceController` | `MarketplaceService` | direct pool queries |
| `/api/v1/billing` | `BillingController` | `BillingService` | direct pool queries |
| `/.well-known/did.json`, `/api/v1/did/*` | `DIDController` | `DIDService` | `AgentRepository` |
| `/.well-known/openid-configuration`, `/api/v1/oidc/*` | `OIDCController` | `OIDCKeyService`, `IDTokenService` | direct pool queries |
| `/api/v1/oidc/trust-policies` | `OIDCTrustPolicyController` | `OIDCTrustPolicyService` | direct pool queries |
| `/api/v1/delegation` | `DelegationController` | `DelegationService` | direct pool queries |
| `/api/v1/scaffold` | `ScaffoldController` | `ScaffoldService` | — |
| `/health` | inline | — | pool, redis |
| `/metrics` | inline | — | prom-client |
## New Services (Phases 3–6)
| Service | Source file | Responsibility |
|---------|------------|----------------|
| `AnalyticsService` | `src/services/AnalyticsService.ts` | Fire-and-forget `recordEvent`, time-series `getTokenTrend`, heatmap `getAgentActivity`, per-agent `getAgentUsageSummary` |
| `TierService` | `src/services/TierService.ts` | `getStatus` (reads `tenant_tiers`), `initiateUpgrade` (creates Stripe Checkout Session), `applyUpgrade` (handles Stripe webhook), `enforceAgentLimit` |
| `ComplianceService` | `src/services/ComplianceService.ts` | `generateReport` (Redis-cached 5 min), `exportAgentCards` (AGNTCY format) |
| `DelegationService` | `src/services/DelegationService.ts` | A2A delegation chain creation and verification |
| `DIDService` | `src/services/DIDService.ts` | `did:web` identifier generation and DID document management |
| `OIDCKeyService` | `src/services/OIDCKeyService.ts` | OIDC key rotation, JWKS endpoint |
| `IDTokenService` | `src/services/IDTokenService.ts` | OIDC ID token issuance |
| `FederationService` | `src/services/FederationService.ts` | Cross-tenant agent identity federation |
| `WebhookService` | `src/services/WebhookService.ts` | Event subscriptions, delivery with retry, dead-letter queue |
| `VaultService` | `src/services/VaultService.ts` | HashiCorp Vault KV v2 read/write for credential storage |
| `BillingService` | `src/services/BillingService.ts` | Stripe customer and subscription management |
| `MarketplaceService` | `src/services/MarketplaceService.ts` | Agent listing and discovery |
| `OIDCTrustPolicyService` | `src/services/OIDCTrustPolicyService.ts` | GitHub OIDC trust policy management |
| `EventPublisher` | `src/services/EventPublisher.ts` | Routes domain events to webhook delivery and Kafka (if configured) |
## Ports
| Service | Internal port | Exposed port (local dev) |
|---------|--------------|--------------------------|
| AgentIdP app | 3000 | 3000 |
| Next.js portal | 3001 | 3001 |
| PostgreSQL | 5432 | 5432 |
| Redis | 6379 | 6379 |
## API Routes (Phase 6 complete)
Base path: `/api/v1`
| Route | Method(s) | Auth | Feature flag |
|-------|----------|------|-------------|
| `/api/v1/agents` | GET, POST, PATCH, DELETE | Bearer JWT | always on |
| `/api/v1/credentials` | GET, POST, DELETE | Bearer JWT | always on |
| `/api/v1/token` | POST | none (client credentials) | always on |
| `/api/v1/audit` | GET | Bearer JWT | always on |
| `/api/v1/audit/verify` | GET | Bearer JWT | always on |
| `/api/v1/organizations` | GET, POST | Bearer JWT | always on |
| `/api/v1/compliance/controls` | GET | none | always on |
| `/api/v1/compliance/report` | GET | Bearer JWT | `COMPLIANCE_ENABLED=true` |
| `/api/v1/compliance/agent-cards` | GET | Bearer JWT | `COMPLIANCE_ENABLED=true` |
| `/api/v1/analytics/token-trend` | GET | Bearer JWT | `ANALYTICS_ENABLED=true` |
| `/api/v1/analytics/agent-activity` | GET | Bearer JWT | `ANALYTICS_ENABLED=true` |
| `/api/v1/analytics/usage-summary` | GET | Bearer JWT | `ANALYTICS_ENABLED=true` |
| `/api/v1/tiers/status` | GET | Bearer JWT | always on |
| `/api/v1/tiers/upgrade` | POST | Bearer JWT | always on |
| `/api/v1/webhooks` | GET, POST, DELETE | Bearer JWT | always on |
| `/api/v1/federation` | GET, POST | Bearer JWT | always on |
| `/api/v1/delegation` | GET, POST | Bearer JWT | always on |
| `/api/v1/marketplace` | GET | none | always on |
| `/api/v1/billing` | GET, POST | Bearer JWT | always on |
| `/api/v1/did/*` | GET | none | always on |
| `/api/v1/oidc/*` | GET, POST | mixed | always on |
| `/.well-known/openid-configuration` | GET | none | always on |
| `/.well-known/jwks.json` | GET | none | always on |
| `/.well-known/did.json` | GET | none | always on |
| `/health` | GET | none | always on |
| `/metrics` | GET | none | always on |
## Graceful Shutdown
The server listens for `SIGTERM` and `SIGINT`. On receipt:

275
docs/devops/database.md
View File

@@ -1,18 +1,28 @@
# Database
AgentIdP uses PostgreSQL 14+ as its primary data store. The schema consists of four tables managed by a custom migration runner.
AgentIdP uses PostgreSQL 14+ as its primary data store. The schema consists of 26 migrations managed by a custom migration runner.
---
## Schema Overview
```
agents
└── credentials (FK: client_id → agents.agent_id, CASCADE DELETE)
audit_events (no FK — append-only, agent_id is informational)
organizations
├── agents (FK: organization_id → organizations.org_id)
│ ├── credentials (FK: client_id → agents.agent_id, CASCADE DELETE)
│ └── agent_did_keys (FK: agent_id → agents.agent_id)
└── audit_events (FK: organization_id — informational, no cascade)
token_revocations (no FK — independent revocation store)
oidc_keys (standalone — OIDC signing key rotation)
federation_partners (standalone — cross-tenant identity)
webhook_subscriptions → webhook_deliveries (FK: subscription_id)
agent_marketplace (standalone — agent discovery catalog)
github_oidc_trust_policies (standalone — CI/CD trust)
billing (FK: org_id → organizations.org_id — one row per org)
delegation_chains (standalone — A2A delegation records)
analytics_events (FK: organization_id — append-only)
tenant_tiers (FK: org_id → organizations.org_id — one row per org)
```
---
@@ -134,6 +144,234 @@ Durable record of revoked JWT tokens. Supplements Redis for durability across Re
---
### `organizations`
Created by migration `006_create_organizations_table.sql`.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `org_id` | `UUID` | No | Primary key |
| `name` | `VARCHAR(255)` | No | Organisation display name |
| `slug` | `VARCHAR(64)` | No | URL-safe unique identifier |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |
---
### `agent_did_keys`
Created by migration `012_create_agent_did_keys_table.sql`.
Stores the DID document key material for each agent. One agent may have multiple keys for
rotation purposes.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `agent_id` | `UUID` | No | FK → `agents.agent_id` |
| `key_id` | `VARCHAR(255)` | No | DID key fragment identifier |
| `public_key_jwk` | `JSONB` | No | Public key in JWK format |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |
---
### DID columns on `agents`
Added by migration `013_add_did_columns_to_agents.sql`:
- `did` — `VARCHAR(512)` nullable — the `did:web` identifier for this agent
- `did_document` — `JSONB` nullable — full DID document
---
### `oidc_keys`
Created by migration `014_create_oidc_keys_table.sql`.
Stores RSA key pairs used for OIDC ID token signing. Supports key rotation — active key is
determined by the most recently created row.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `kid` | `VARCHAR(128)` | No | Key ID — referenced in JWKS |
| `private_key_pem` | `TEXT` | No | Encrypted RSA private key (pgcrypto) |
| `public_key_pem` | `TEXT` | No | RSA public key |
| `algorithm` | `VARCHAR(16)` | No | Always `RS256` |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |
---
### `federation_partners`
Created by migration `015_create_federation_partners_table.sql`.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `org_id` | `UUID` | No | Owning organisation |
| `partner_name` | `VARCHAR(255)` | No | Display name |
| `partner_jwks_url` | `TEXT` | No | URL to partner's JWKS endpoint |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |
---
### `webhook_subscriptions`
Created by migration `016_create_webhook_subscriptions_table.sql`.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `org_id` | `UUID` | No | Owning organisation |
| `event_type` | `VARCHAR(128)` | No | Event type filter (e.g. `agent.created`) |
| `target_url` | `TEXT` | No | HTTPS delivery endpoint |
| `secret` | `VARCHAR(255)` | Yes | HMAC signing secret for delivery verification |
| `active` | `BOOLEAN` | No | Default: `true` |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |
---
### `webhook_deliveries`
Created by migration `017_create_webhook_deliveries_table.sql`.
Records each delivery attempt for a webhook event, including the dead-letter queue entries.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `subscription_id` | `UUID` | No | FK → `webhook_subscriptions.id` |
| `event_type` | `VARCHAR(128)` | No | Event type delivered |
| `payload` | `JSONB` | No | Full event payload |
| `status` | `VARCHAR(32)` | No | `pending`, `delivered`, `failed`, `dead_letter` |
| `response_status` | `INTEGER` | Yes | HTTP status from delivery endpoint |
| `attempt_count` | `INTEGER` | No | Default: `0` |
| `last_attempted_at` | `TIMESTAMPTZ` | Yes | |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |
**Dead-letter queue:** After 3 failed delivery attempts, the row status is set to `dead_letter`
and the `agentidp_webhook_dead_letters_total` Prometheus counter is incremented. The Prometheus
metric label is `event_type`.
---
### pgcrypto extension
Enabled by migration `018_enable_pgcrypto.sql`. Used for encrypting sensitive columns in
`oidc_keys` and credential data.
---
### `agent_marketplace`
Created by migration `021_add_agent_marketplace.sql`.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `agent_id` | `UUID` | No | FK → `agents.agent_id` |
| `listing_name` | `VARCHAR(255)` | No | Display name in marketplace |
| `description` | `TEXT` | Yes | Markdown description |
| `tags` | `TEXT[]` | No | Searchable tags. Default: `{}` |
| `published` | `BOOLEAN` | No | Default: `false` |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |
---
### `github_oidc_trust_policies`
Created by migration `022_add_github_oidc_trust_policies.sql`.
Maps GitHub Actions OIDC claims to agent identities for CI/CD token exchange.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `org_id` | `UUID` | No | Owning organisation |
| `repository` | `VARCHAR(512)` | No | GitHub repository slug (`owner/repo`) |
| `branch` | `VARCHAR(255)` | Yes | Branch filter (null = any branch) |
| `agent_id` | `UUID` | No | Agent to issue a token for on match |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |
---
### `billing`
Created by migration `023_add_billing.sql`.
One row per organisation. Tracks the org's Stripe customer and subscription state.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `org_id` | `UUID` | No | FK → `organizations.org_id` (UNIQUE) |
| `stripe_customer_id` | `VARCHAR(255)` | Yes | Stripe Customer ID |
| `stripe_subscription_id` | `VARCHAR(255)` | Yes | Stripe Subscription ID |
| `status` | `VARCHAR(64)` | No | Stripe subscription status or `none` |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |
---
### `delegation_chains`
Created by migration `024_add_delegation_chains.sql`.
Records A2A delegation grants created via the delegation API.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `delegator_agent_id` | `UUID` | No | Agent granting the delegation |
| `delegate_agent_id` | `UUID` | No | Agent receiving the delegation |
| `scopes` | `TEXT[]` | No | Scopes being delegated |
| `expires_at` | `TIMESTAMPTZ` | Yes | Optional expiry |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |
---
### `analytics_events`
Created by migration `025_add_analytics_events.sql`.
Append-only event store for analytics. Supports token trend, agent activity, and usage summary
queries.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `organization_id` | `UUID` | No | Owning organisation |
| `date` | `DATE` | No | Calendar date of the event (UTC) |
| `metric_type` | `VARCHAR(64)` | No | e.g. `token_issued`, `agent_called` |
| `count` | `INTEGER` | No | Event count for this date+type |
**Index:** `(organization_id, date DESC)` for fast time-series queries.
---
### `tenant_tiers`
Created by migration `026_add_tenant_tiers.sql`.
One row per organisation. Stores the current tier and enforces tier limits via the
`tierEnforcement` middleware.
| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `org_id` | `UUID` | No | FK → `organizations.org_id` (UNIQUE) |
| `tier` | `ENUM('free','pro','enterprise')` | No | Current tier. Default: `free` |
| `updated_at` | `TIMESTAMPTZ` | No | Last tier change. Default: `NOW()` |
**Tier limits** (from `src/config/tiers.ts`):
| Tier | Max Agents | Max API Calls/Day | Max Tokens/Day |
|------|-----------|-------------------|----------------|
| free | 10 | 1,000 | 1,000 |
| pro | 100 | 50,000 | 50,000 |
| enterprise | unlimited | unlimited | unlimited |
---
## Migration Runner
Migrations are managed by `scripts/migrate.ts`. It reads `.sql` files from `src/db/migrations/` in alphabetical order, tracks applied migrations in a `schema_migrations` table, and executes only unapplied migrations — each in its own transaction.
@@ -160,10 +398,11 @@ Expected output (first run):
Running database migrations...
✓ Applied: 001_create_agents.sql
✓ Applied: 002_create_credentials.sql
✓ Applied: 003_create_audit_events.sql
✓ Applied: 004_create_tokens.sql
...
✓ Applied: 025_add_analytics_events.sql
✓ Applied: 026_add_tenant_tiers.sql
Migrations complete. 4 migration(s) applied.
Migrations complete. 26 migration(s) applied.
```
Expected output (already applied):
@@ -191,9 +430,10 @@ Expected output:
-----------------------------------+-------------------------------
001_create_agents.sql | 2026-03-28 09:00:00.000000+00
002_create_credentials.sql | 2026-03-28 09:00:00.000000+00
003_create_audit_events.sql | 2026-03-28 09:00:00.000000+00
004_create_tokens.sql | 2026-03-28 09:00:00.000000+00
(4 rows)
...
025_add_analytics_events.sql | 2026-04-04 09:00:00.000000+00
026_add_tenant_tiers.sql | 2026-04-04 09:00:00.000000+00
(26 rows)
```
### Adding a new migration
@@ -214,6 +454,15 @@ There is no automated rollback. To undo a migration:
## Connection Pool
The application uses `pg.Pool` with default settings (max 10 connections). The pool is a singleton — one pool per process instance.
The application uses `pg.Pool` with settings read from environment variables. The pool is a
singleton — one pool per process instance.
To override pool size, modify `src/db/pool.ts`. In production, ensure `DATABASE_URL` includes connection pool parameters if using PgBouncer or a managed connection pooler.
| Variable | Default | Description |
|----------|---------|-------------|
| `DB_POOL_MAX` | `20` | Maximum connections |
| `DB_POOL_MIN` | `2` | Minimum idle connections |
| `DB_POOL_IDLE_TIMEOUT_MS` | `30000` | Idle eviction timeout (ms) |
| `DB_POOL_CONNECTION_TIMEOUT_MS` | `5000` | Acquisition timeout (ms) |
Pool size is exposed as Prometheus metrics: `agentidp_db_pool_active_connections` and
`agentidp_db_pool_waiting_requests`. Monitor these in production to detect pool exhaustion.

18
docs/devops/deployment.md
View File

@@ -543,6 +543,24 @@ All environment variables injected into the AgentIdP container are documented in
| `VAULT_ADDR` | No | Task definition env var | Cloud Run env var |
| `VAULT_TOKEN` | No | Secrets Manager: `/<project>/<env>/vault-token` | Secret Manager: `<name-prefix>-vault-token` |
| `VAULT_MOUNT` | No | Task definition env var (default: `secret`) | Cloud Run env var (default: `secret`) |
| `BILLING_ENABLED` | No | Task definition env var | Cloud Run env var |
| `STRIPE_SECRET_KEY` | Only if billing enabled | Secrets Manager: `/<project>/<env>/stripe-secret-key` | Secret Manager: `<name-prefix>-stripe-secret-key` |
| `STRIPE_WEBHOOK_SECRET` | Only if billing enabled | Secrets Manager: `/<project>/<env>/stripe-webhook-secret` | Secret Manager: `<name-prefix>-stripe-webhook-secret` |
| `STRIPE_PRICE_ID` | Only if billing enabled | Task definition env var | Cloud Run env var |
| `ANALYTICS_ENABLED` | No | Task definition env var (default: `true`) | Cloud Run env var |
| `TIER_ENFORCEMENT` | No | Task definition env var (default: `true`) | Cloud Run env var |
| `COMPLIANCE_ENABLED` | No | Task definition env var (default: `true`) | Cloud Run env var |
| `REDIS_RATE_LIMIT_ENABLED` | No | Task definition env var | Cloud Run env var |
| `RATE_LIMIT_WINDOW_MS` | No | Task definition env var (default: `60000`) | Cloud Run env var |
| `RATE_LIMIT_MAX_REQUESTS` | No | Task definition env var (default: `100`) | Cloud Run env var |
| `DB_POOL_MAX` | No | Task definition env var (default: `20`) | Cloud Run env var |
| `DB_POOL_MIN` | No | Task definition env var (default: `2`) | Cloud Run env var |
| `DB_POOL_IDLE_TIMEOUT_MS` | No | Task definition env var (default: `30000`) | Cloud Run env var |
| `DB_POOL_CONNECTION_TIMEOUT_MS` | No | Task definition env var (default: `5000`) | Cloud Run env var |
| `KAFKA_BROKERS` | No | Task definition env var | Cloud Run env var |
| `ENFORCE_TLS` | No | Task definition env var | Cloud Run env var |
| `OPA_URL` | No | Task definition env var | Cloud Run env var |
| `VAULT_KV_MOUNT` | No | Task definition env var (default: `secret`) | Cloud Run env var |
### Updating a Secret

320
docs/devops/environment-variables.md
View File

@@ -20,7 +20,7 @@ PostgreSQL connection string.
| **Format** | `postgresql://<user>:<password>@<host>:<port>/<database>` |
| **Example** | `postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp` |
The application uses `pg.Pool` with this connection string. Connection pool size uses the `pg` default (10 connections).
The application uses `pg.Pool` with this connection string. Pool sizing is controlled by the optional `DB_POOL_*` variables documented below.
---
@@ -72,6 +72,10 @@ Every authenticated request verifies the JWT signature using this key. If this k
---
> **Note on Billing:** `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, and `STRIPE_PRICE_ID` are
> required when `BILLING_ENABLED=true`. For local development, set `BILLING_ENABLED=false` and
> use placeholder values.
## Optional Variables
These variables have defaults and do not need to be set for local development.
@@ -117,6 +121,257 @@ KV v2 secrets engine mount path.
---
### `BILLING_ENABLED`
| | |
|-|-|
| **Required** | No |
| **Default** | `false` |
| **Values** | `true`, `false` |
| **Example** | `BILLING_ENABLED=false` |
Gates Stripe billing integration and free-tier agent limit enforcement. When `false`, no Stripe
API calls are made and all tier limits are unenforced. Set to `false` for in-house testing.
---
### `STRIPE_SECRET_KEY`
| | |
|-|-|
| **Required** | Only when `BILLING_ENABLED=true` |
| **Format** | Stripe secret key string (`sk_live_*` or `sk_test_*`) |
| **Example** | `STRIPE_SECRET_KEY=sk_test_placeholder` |
Stripe API key used to create Checkout Sessions for tier upgrades. Never use a live key in
development.
---
### `STRIPE_WEBHOOK_SECRET`
| | |
|-|-|
| **Required** | Only when `BILLING_ENABLED=true` |
| **Format** | Stripe webhook signing secret (`whsec_*`) |
| **Example** | `STRIPE_WEBHOOK_SECRET=whsec_placeholder` |
Used to verify the HMAC signature on incoming Stripe webhook events. Without this, the billing
webhook endpoint will reject all events.
---
### `STRIPE_PRICE_ID`
| | |
|-|-|
| **Required** | Only when `BILLING_ENABLED=true` |
| **Format** | Stripe Price ID string (`price_*`) |
| **Example** | `STRIPE_PRICE_ID=price_placeholder` |
The Stripe Price object used when creating a Checkout Session for the Pro tier upgrade.
---
### `ANALYTICS_ENABLED`
| | |
|-|-|
| **Required** | No |
| **Default** | `true` |
| **Values** | `true`, `false` |
| **Example** | `ANALYTICS_ENABLED=true` |
Feature flag that gates the `/api/v1/analytics/*` routes. When `false`, the analytics router is
not mounted and all analytics endpoints return 404. Events are still recorded internally
regardless of this flag.
---
### `TIER_ENFORCEMENT`
| | |
|-|-|
| **Required** | No |
| **Default** | `true` |
| **Values** | `true`, `false` |
| **Example** | `TIER_ENFORCEMENT=true` |
Enables Redis-backed tier limit enforcement per tenant. When `true`, the `tierEnforcement`
middleware checks daily API call and token counts against per-tier limits defined in
`src/config/tiers.ts`. Enterprise tenants with `maxCallsPerDay: Infinity` bypass enforcement.
When `false`, no tier limits are enforced.
---
### `COMPLIANCE_ENABLED`
| | |
|-|-|
| **Required** | No |
| **Default** | `true` |
| **Values** | `true`, `false` |
| **Example** | `COMPLIANCE_ENABLED=true` |
Feature flag that gates the report and agent-card export endpoints under
`/api/v1/compliance/*`. When `false`, those endpoints return 404. The SOC2 controls endpoint
(`/api/v1/compliance/controls`) and audit chain verification (`/api/v1/audit/verify`) are
always enabled regardless of this flag.
---
### `REDIS_RATE_LIMIT_ENABLED`
| | |
|-|-|
| **Required** | No |
| **Default** | `false` |
| **Values** | `true`, `false` |
| **Example** | `REDIS_RATE_LIMIT_ENABLED=true` |
When `true`, rate limiting uses a Redis-backed sliding-window counter per `client_id`. When
`false`, rate limiting uses an in-process `RateLimiterMemory` store (does not share state
across multiple app instances).
---
### `RATE_LIMIT_WINDOW_MS`
| | |
|-|-|
| **Required** | No |
| **Default** | `60000` |
| **Format** | Integer (milliseconds) |
| **Example** | `RATE_LIMIT_WINDOW_MS=60000` |
Duration of the sliding-window rate limit period in milliseconds. Only effective when
`REDIS_RATE_LIMIT_ENABLED=true`.
---
### `RATE_LIMIT_MAX_REQUESTS`
| | |
|-|-|
| **Required** | No |
| **Default** | `100` |
| **Format** | Integer |
| **Example** | `RATE_LIMIT_MAX_REQUESTS=100` |
Maximum number of requests allowed per `client_id` within `RATE_LIMIT_WINDOW_MS`. Requests
exceeding this limit receive `429 RATE_LIMIT_EXCEEDED`.
---
### `DB_POOL_MAX`
| | |
|-|-|
| **Required** | No |
| **Default** | `20` |
| **Format** | Integer |
| **Example** | `DB_POOL_MAX=20` |
Maximum number of PostgreSQL connections in the pool. Increase for high-throughput production
deployments. Ensure your PostgreSQL instance's `max_connections` is set to at least
`DB_POOL_MAX × number_of_app_instances + 5`.
---
### `DB_POOL_MIN`
| | |
|-|-|
| **Required** | No |
| **Default** | `2` |
| **Format** | Integer |
| **Example** | `DB_POOL_MIN=2` |
Minimum number of idle connections kept alive in the pool.
---
### `DB_POOL_IDLE_TIMEOUT_MS`
| | |
|-|-|
| **Required** | No |
| **Default** | `30000` |
| **Format** | Integer (milliseconds) |
| **Example** | `DB_POOL_IDLE_TIMEOUT_MS=30000` |
Milliseconds a connection can sit idle before being evicted from the pool.
---
### `DB_POOL_CONNECTION_TIMEOUT_MS`
| | |
|-|-|
| **Required** | No |
| **Default** | `5000` |
| **Format** | Integer (milliseconds) |
| **Example** | `DB_POOL_CONNECTION_TIMEOUT_MS=5000` |
Milliseconds the pool waits for a connection to become available before throwing a connection
timeout error.
---
### `VAULT_KV_MOUNT`
| | |
|-|-|
| **Required** | No |
| **Default** | `secret` |
| **Format** | String (no leading or trailing slash) |
| **Example** | `VAULT_KV_MOUNT=agentidp` |
KV v2 secrets engine mount path used by `VaultService`. Equivalent to the existing `VAULT_MOUNT`
variable — note that `.env.example` uses `VAULT_KV_MOUNT`; the underlying service reads either.
---
### `OPA_URL`
| | |
|-|-|
| **Required** | No |
| **Format** | URL string |
| **Example** | `OPA_URL=http://localhost:8181` |
URL of a running OPA server for external policy evaluation. When unset, the application falls
back to the embedded Wasm or JSON policy in `POLICY_DIR`. Used for health check reporting.
---
### `KAFKA_BROKERS`
| | |
|-|-|
| **Required** | No |
| **Format** | Comma-separated broker addresses |
| **Example** | `KAFKA_BROKERS=localhost:9092` |
When set, the `KafkaAdapter` publishes domain events to Kafka. When unset, Kafka publishing is
disabled and events are only delivered via the `WebhookService`.
---
### `ENFORCE_TLS`
| | |
|-|-|
| **Required** | No |
| **Default** | `false` |
| **Values** | `true`, `false` |
| **Example** | `ENFORCE_TLS=true` |
When `true`, the `tlsEnforcementMiddleware` redirects all HTTP requests to HTTPS. Enable in
production deployments where TLS termination is handled at the application layer.
---
### `POLICY_DIR`
Directory containing OPA policy files (`authz.rego`, `authz.wasm`, `data/scopes.json`).
@@ -178,33 +433,53 @@ In production, set this to the specific origin(s) that should be permitted to ca
## Complete `.env` Example
```
# Database
DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp
# Redis
REDIS_URL=redis://localhost:6379
# Application
PORT=3000
# ── Server ──────────────────────────────────────────────────────────────────
NODE_ENV=development
CORS_ORIGIN=*
PORT=3000
CORS_ORIGIN=http://localhost:3001
# JWT Keys (generate with openssl — see docs/devops/security.md)
JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA...
-----END RSA PRIVATE KEY-----"
# ── Database ─────────────────────────────────────────────────────────────────
DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp
DB_POOL_MAX=20
DB_POOL_MIN=2
DB_POOL_IDLE_TIMEOUT_MS=30000
DB_POOL_CONNECTION_TIMEOUT_MS=5000
JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----
MIIBIjANBgkq...
-----END PUBLIC KEY-----"
# ── Redis ────────────────────────────────────────────────────────────────────
REDIS_URL=redis://localhost:6379
REDIS_RATE_LIMIT_ENABLED=true
RATE_LIMIT_WINDOW_MS=60000
RATE_LIMIT_MAX_REQUESTS=100
# HashiCorp Vault (Phase 2 — optional, omit to use bcrypt mode)
# ── JWT Keys (generate with openssl — see docs/devops/security.md) ──────────
JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\nMIIEow...\n-----END RSA PRIVATE KEY-----"
JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\nMIIBIj...\n-----END PUBLIC KEY-----"
# ── Billing (Stripe) — set BILLING_ENABLED=false for local/in-house testing ─
BILLING_ENABLED=false
STRIPE_SECRET_KEY=sk_test_placeholder
STRIPE_WEBHOOK_SECRET=whsec_placeholder
STRIPE_PRICE_ID=price_placeholder
# ── Phase 6 Feature Flags ─────────────────────────────────────────────────────
ANALYTICS_ENABLED=true
TIER_ENFORCEMENT=true
COMPLIANCE_ENABLED=true
# ── HashiCorp Vault (optional) ────────────────────────────────────────────────
# VAULT_ADDR=http://127.0.0.1:8200
# VAULT_TOKEN=hvs.XXXXXXXXXXXXXXXXXXXXXX
# VAULT_MOUNT=secret
# VAULT_KV_MOUNT=secret
# OPA Policy Engine (Phase 2 — optional, defaults to <cwd>/policies)
# ── OPA (optional) ───────────────────────────────────────────────────────────
# POLICY_DIR=/etc/sentryagent/policies
# OPA_URL=http://localhost:8181
# ── Kafka (optional) ─────────────────────────────────────────────────────────
# KAFKA_BROKERS=localhost:9092
# ── TLS ──────────────────────────────────────────────────────────────────────
# ENFORCE_TLS=true
```
> Do not commit `.env` to version control. Add it to `.gitignore`.
@@ -220,3 +495,8 @@ The application validates required variables at startup in this order:
3. `REDIS_URL` — checked when `getRedisClient()` is first called (during `createApp()`)
If any required variable is missing, the process exits with an error before binding to any port.
> **Feature flags** (`BILLING_ENABLED`, `ANALYTICS_ENABLED`, `TIER_ENFORCEMENT`,
> `COMPLIANCE_ENABLED`) are read at startup. `ANALYTICS_ENABLED` and `COMPLIANCE_ENABLED`
> determine whether their respective routers are mounted — changing these values requires a
> process restart.

946
docs/devops/field-trial.md Normal file
View File

@@ -0,0 +1,946 @@
# SentryAgent.ai AgentIdP — In-House Field Trial Guide
This guide is the execution playbook for in-house Docker Compose field trials of SentryAgent.ai
AgentIdP. Follow each phase in order. All commands are exact — copy and paste them directly.
Estimated time to complete all phases: 45–60 minutes.
Prerequisites must be satisfied before Section 0.
## Prerequisites
**Docker 24+ and Docker Compose 2.20+**
```bash
docker --version
# Expected: Docker version 24.x.x or higher
docker compose version
# Expected: Docker Compose version v2.20.x or higher
```
**Node.js 18+ via nvm**
```bash
export NVM_DIR="$HOME/.nvm" && source "$NVM_DIR/nvm.sh"
node --version
# Expected: v18.x.x or higher
```
**openssl**
```bash
openssl version
# Expected: OpenSSL 1.1.x or higher (any version)
```
**Git repo cloned**
```bash
git clone https://git.sentryagent.ai/vijay_admin/sentryagent-idp.git
cd sentryagent-idp
```
**Ports free**
The following ports must be free on the machine before starting:
| Port | Service |
|------|---------|
| 3000 | AgentIdP backend |
| 3001 | Next.js portal |
| 5432 | PostgreSQL |
| 6379 | Redis |
Check all ports:
```bash
lsof -i :3000 -i :3001 -i :5432 -i :6379
# Expected: no output (all ports free)
```
If any port is in use, kill the occupying process:
```bash
lsof -ti:<port> | xargs kill
```
---
## Section 0 — Environment Setup
This section guides the engineer through creating a valid `.env` file for field trial use.
**Step 0.1 — Copy `.env.example`**
```bash
cp .env.example .env
```
**Step 0.2 — Generate RSA-2048 keypair**
Generate the JWT signing keys:
```bash
openssl genrsa -out private.pem 2048
openssl rsa -in private.pem -pubout -out public.pem
```
Verify the keys are valid:
```bash
openssl rsa -in private.pem -check -noout
# Expected: RSA key ok
openssl rsa -in public.pem -pubin -noout -text 2>&1 | head -3
# Expected: Public-Key: (2048 bit)
```
**Step 0.3 — Write keys into `.env`**
Write the private key as a single-line PEM with `\n` separators:
```bash
PRIVATE_KEY_LINE=$(awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' private.pem)
sed -i "s|JWT_PRIVATE_KEY=.*|JWT_PRIVATE_KEY=\"${PRIVATE_KEY_LINE}\"|" .env
```
Write the public key:
```bash
PUBLIC_KEY_LINE=$(awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' public.pem)
sed -i "s|JWT_PUBLIC_KEY=.*|JWT_PUBLIC_KEY=\"${PUBLIC_KEY_LINE}\"|" .env
```
Verify both keys are present and non-empty:
```bash
grep -c "BEGIN RSA PRIVATE KEY" .env
# Expected: 1
grep -c "BEGIN PUBLIC KEY" .env
# Expected: 1
```
**Step 0.4 — Configure field trial values**
Set the following values in `.env`. These are the correct values for an in-house field trial
(no real Stripe, no Kafka, no Vault):
```bash
# Disable real Stripe billing for field trial
sed -i "s|BILLING_ENABLED=.*|BILLING_ENABLED=false|" .env
sed -i "s|STRIPE_SECRET_KEY=.*|STRIPE_SECRET_KEY=sk_test_placeholder|" .env
sed -i "s|STRIPE_WEBHOOK_SECRET=.*|STRIPE_WEBHOOK_SECRET=whsec_placeholder|" .env
sed -i "s|STRIPE_PRICE_ID=.*|STRIPE_PRICE_ID=price_placeholder|" .env
# Keep feature flags at defaults
sed -i "s|ANALYTICS_ENABLED=.*|ANALYTICS_ENABLED=true|" .env
sed -i "s|TIER_ENFORCEMENT=.*|TIER_ENFORCEMENT=true|" .env
sed -i "s|COMPLIANCE_ENABLED=.*|COMPLIANCE_ENABLED=true|" .env
# Allow portal CORS
sed -i "s|CORS_ORIGIN=.*|CORS_ORIGIN=http://localhost:3001|" .env
```
**Step 0.5 — Verify final `.env`**
```bash
grep -E "^(DATABASE_URL|REDIS_URL|JWT_PRIVATE_KEY|JWT_PUBLIC_KEY|BILLING_ENABLED|ANALYTICS_ENABLED|TIER_ENFORCEMENT|COMPLIANCE_ENABLED|CORS_ORIGIN)=" .env
```
Expected output (values abbreviated):
```
DATABASE_URL=postgresql://agentidp:password@localhost:5432/agentidp
REDIS_URL=redis://localhost:6379
JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...
JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\n...
BILLING_ENABLED=false
ANALYTICS_ENABLED=true
TIER_ENFORCEMENT=true
COMPLIANCE_ENABLED=true
CORS_ORIGIN=http://localhost:3001
```
---
## Phase A — Stack Startup
**Step A.1 — Build and start the full stack**
```bash
docker compose up --build -d
```
This builds the `app` container image and starts all three services. The `app` service waits
for `postgres` and `redis` to pass their health checks before starting.
**Step A.2 — Verify all services are healthy**
```bash
docker compose ps
```
Expected output — all three services must show `healthy`:
```
NAME IMAGE STATUS
sentryagent-idp-app-1 sentryagent-idp-app running (healthy)
sentryagent-idp-postgres-1 postgres:14-alpine running (healthy)
sentryagent-idp-redis-1 redis:7-alpine running (healthy)
```
If any service shows `starting` or `unhealthy`, wait 15 seconds and run `docker compose ps`
again. If a service remains unhealthy after 60 seconds, see Troubleshooting.
**Step A.3 — Run database migrations**
```bash
docker compose exec app npm run db:migrate
```
Expected output:
```
Running database migrations...
✓ Applied: 001_create_agents.sql
✓ Applied: 002_create_credentials.sql
...
✓ Applied: 025_add_analytics_events.sql
✓ Applied: 026_add_tenant_tiers.sql
Migrations complete. 26 migration(s) applied.
```
All 26 migrations must apply without error before proceeding.
**Step A.4 — Verify application health**
```bash
curl -s http://localhost:3000/health | jq .
```
Expected response:
```json
{"status":"ok"}
```
**Step A.5 — Verify Prometheus metrics**
```bash
curl -s http://localhost:3000/metrics | head -20
```
Expected: Prometheus text output beginning with `# HELP` lines. Verify these specific metrics
are present:
```bash
curl -s http://localhost:3000/metrics | grep -E "^# HELP agentidp_"
```
Expected: at least 19 lines matching `# HELP agentidp_*`.
---
## Phase B — Core Product Journeys
This phase tests the end-to-end agent identity lifecycle. Run each step in order. Each step
depends on the output of the previous step.
> **Note on tokens:** The steps below use shell variables to pass values between commands. Run
> all commands in the same terminal session.
**Step B.1 — Create an organisation**
```bash
ORG_RESPONSE=$(curl -s -X POST http://localhost:3000/api/v1/organizations \
-H "Content-Type: application/json" \
-d '{"name":"Field Trial Org","slug":"field-trial"}')
echo $ORG_RESPONSE | jq .
ORG_ID=$(echo $ORG_RESPONSE | jq -r '.org_id')
echo "ORG_ID: $ORG_ID"
```
Expected: HTTP 201 response body containing an `org_id` UUID. `ORG_ID` must be a non-empty UUID.
**Step B.2 — Register an agent**
```bash
AGENT_RESPONSE=$(curl -s -X POST http://localhost:3000/api/v1/agents \
-H "Content-Type: application/json" \
-d "{
\"email\": \"trial-agent@field-trial.sentryagent.ai\",
\"agent_type\": \"classifier\",
\"version\": \"1.0.0\",
\"capabilities\": [\"documents:read\", \"documents:classify\"],
\"owner\": \"field-trial-team\",
\"deployment_env\": \"development\",
\"organization_id\": \"$ORG_ID\"
}")
echo $AGENT_RESPONSE | jq .
AGENT_ID=$(echo $AGENT_RESPONSE | jq -r '.agent_id')
echo "AGENT_ID: $AGENT_ID"
```
Expected: HTTP 201 response body containing an `agent_id` UUID.
**Step B.3 — Generate credentials**
```bash
CRED_RESPONSE=$(curl -s -X POST http://localhost:3000/api/v1/credentials \
-H "Content-Type: application/json" \
-d "{\"agent_id\": \"$AGENT_ID\"}")
echo $CRED_RESPONSE | jq .
CLIENT_ID=$(echo $CRED_RESPONSE | jq -r '.client_id')
CLIENT_SECRET=$(echo $CRED_RESPONSE | jq -r '.client_secret')
echo "CLIENT_ID: $CLIENT_ID"
echo "CLIENT_SECRET: $CLIENT_SECRET"
```
Expected: HTTP 201 response body containing `client_id` and `client_secret`. The `client_secret`
is only returned once — save it now.
**Step B.4 — Issue an OAuth 2.0 access token**
```bash
TOKEN_RESPONSE=$(curl -s -X POST http://localhost:3000/api/v1/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&client_id=$CLIENT_ID&client_secret=$CLIENT_SECRET&scope=read")
echo $TOKEN_RESPONSE | jq .
ACCESS_TOKEN=$(echo $TOKEN_RESPONSE | jq -r '.access_token')
echo "ACCESS_TOKEN obtained: ${ACCESS_TOKEN:0:30}..."
```
Expected: HTTP 200 response body with `access_token`, `token_type: "Bearer"`, `expires_in: 3600`,
`scope: "read"`.
**Step B.5 — Use the token on a protected endpoint**
```bash
curl -s -H "Authorization: Bearer $ACCESS_TOKEN" \
http://localhost:3000/api/v1/agents | jq .
```
Expected: HTTP 200 with a JSON array of agents including the agent registered in Step B.2.
**Step B.6 — Inspect JWT claims**
Decode and inspect the access token structure (without verifying signature):
```bash
echo $ACCESS_TOKEN | cut -d. -f2 | base64 -d 2>/dev/null | jq .
```
Expected claims:
```json
{
"sub": "<client_id>",
"iss": "https://sentryagent.ai",
"aud": "sentryagent-api",
"scope": "read",
"agent_id": "<agent_id>",
"organization_id": "<org_id>",
"iat": "<issued-at-timestamp>",
"exp": "<expiry-timestamp>",
"jti": "<unique-jwt-id>"
}
```
Verify `exp - iat = 3600` (1 hour TTL).
**Step B.7 — Rotate credentials and verify old token is rejected**
Rotate the credentials (generates a new client_secret, revokes the old one):
```bash
ROTATE_RESPONSE=$(curl -s -X POST http://localhost:3000/api/v1/credentials \
-H "Content-Type: application/json" \
-d "{\"agent_id\": \"$AGENT_ID\"}")
NEW_CLIENT_ID=$(echo $ROTATE_RESPONSE | jq -r '.client_id')
NEW_CLIENT_SECRET=$(echo $ROTATE_RESPONSE | jq -r '.client_secret')
echo "New credential: $NEW_CLIENT_ID"
```
Attempt to use the old token (must be rejected):
```bash
curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer $ACCESS_TOKEN" \
http://localhost:3000/api/v1/agents
# Expected: 401
```
Issue a new token with the new credentials:
```bash
NEW_TOKEN_RESPONSE=$(curl -s -X POST http://localhost:3000/api/v1/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&client_id=$NEW_CLIENT_ID&client_secret=$NEW_CLIENT_SECRET&scope=read")
NEW_ACCESS_TOKEN=$(echo $NEW_TOKEN_RESPONSE | jq -r '.access_token')
echo "New token obtained."
```
Verify the new token works:
```bash
curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer $NEW_ACCESS_TOKEN" \
http://localhost:3000/api/v1/agents
# Expected: 200
```
**Step B.8 — Check audit log**
```bash
curl -s -H "Authorization: Bearer $NEW_ACCESS_TOKEN" \
"http://localhost:3000/api/v1/audit?limit=10" | jq .
```
Expected: JSON array of audit events. Verify these action types are present from Steps B.1–B.7:
`agent.created`, `credential.generated`, `token.issued`, `credential.rotated`, `token.revoked`.
---
## Phase C — Guardrails
This phase tests security boundaries. Each test case must be run with the exact command shown
and must produce the specified HTTP status code.
> **Setup:** Ensure `$NEW_ACCESS_TOKEN` is still set from Phase B. Use `export NEW_ACCESS_TOKEN`
> if switching terminals.
**Test C.1 — No Authorization header → 401**
```bash
curl -s -o /dev/null -w "%{http_code}" \
http://localhost:3000/api/v1/agents
```
Expected HTTP status: `401`
**Test C.2 — Malformed JWT → 401**
```bash
curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer notavalidjwt" \
http://localhost:3000/api/v1/agents
```
Expected HTTP status: `401`
**Test C.3 — Expired JWT → 401**
Use a known-expired token. Generate one with a 1-second TTL (requires a test helper or
manually craft an expired JWT). For field trial purposes, use this pre-constructed expired token
(signed with a different key — will fail signature verification and return 401):
```bash
EXPIRED_TOKEN="eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0IiwiZXhwIjoxfQ.invalid"
curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer $EXPIRED_TOKEN" \
http://localhost:3000/api/v1/agents
```
Expected HTTP status: `401`
**Test C.4 — Valid JWT, wrong scope → 403**
Issue a token with scope `read`, then attempt to access an endpoint requiring scope `write`:
```bash
# The NEW_ACCESS_TOKEN has scope "read"
# Attempt an action requiring "write" scope (create agent)
curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer $NEW_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-X POST http://localhost:3000/api/v1/agents \
-d '{"email":"scope-test@example.com","agent_type":"custom","version":"1.0.0","capabilities":[],"owner":"test","deployment_env":"development"}'
```
Expected HTTP status: `403`
**Test C.5 — Rate limit: 101 requests → 429 on the 101st**
Send 101 requests in rapid succession. The 101st must return 429.
```bash
for i in $(seq 1 101); do
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer $NEW_ACCESS_TOKEN" \
http://localhost:3000/api/v1/agents)
if [ "$STATUS" = "429" ]; then
echo "Request $i returned 429 (PASS)"
break
fi
done
```
Expected: Output shows `Request 101 returned 429 (PASS)` (or earlier if previous requests in
the session have already counted toward the window).
After this test, wait 60 seconds for the rate limit window to reset, or use a fresh
`client_id` for subsequent tests.
**Test C.6 — Tier limit: exceed free-tier API call limit → 429 with `tier_limit_exceeded`**
The free tier allows 1,000 API calls per day. For field trial, manually set the counter to the
limit value to trigger the guard without making 1,000 real requests:
```bash
# Get the org_id from the token
ORG_ID=$(echo $NEW_ACCESS_TOKEN | cut -d. -f2 | base64 -d 2>/dev/null | jq -r '.organization_id')
# Force the counter to the limit via Redis CLI
docker compose exec redis redis-cli SET "rate:tier:calls:$ORG_ID" 1001 EX 86400
# The next API call must be rejected
TIER_RESPONSE=$(curl -s -w "\n%{http_code}" \
-H "Authorization: Bearer $NEW_ACCESS_TOKEN" \
http://localhost:3000/api/v1/agents)
echo "$TIER_RESPONSE"
```
Expected: HTTP status `429`. Response body must contain `"code":"tier_limit_exceeded"`.
Reset the counter after this test:
```bash
docker compose exec redis redis-cli DEL "rate:tier:calls:$ORG_ID"
```
**Test C.7 — Tenant isolation: Org A token cannot access Org B agents → 403**
Create a second organisation and agent:
```bash
ORG_B_RESPONSE=$(curl -s -X POST http://localhost:3000/api/v1/organizations \
-H "Content-Type: application/json" \
-d '{"name":"Org B","slug":"org-b"}')
ORG_B_ID=$(echo $ORG_B_RESPONSE | jq -r '.org_id')
echo "ORG_B_ID: $ORG_B_ID"
AGENT_B_RESPONSE=$(curl -s -X POST http://localhost:3000/api/v1/agents \
-H "Content-Type: application/json" \
-d "{
\"email\": \"org-b-agent@org-b.sentryagent.ai\",
\"agent_type\": \"monitor\",
\"version\": \"1.0.0\",
\"capabilities\": [],
\"owner\": \"org-b\",
\"deployment_env\": \"development\",
\"organization_id\": \"$ORG_B_ID\"
}")
AGENT_B_ID=$(echo $AGENT_B_RESPONSE | jq -r '.agent_id')
echo "AGENT_B_ID: $AGENT_B_ID"
```
Attempt to access Org B's agent using Org A's token:
```bash
curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer $NEW_ACCESS_TOKEN" \
http://localhost:3000/api/v1/agents/$AGENT_B_ID
```
Expected HTTP status: `403`
---
## Phase D — Portal
**Step D.1 — Install portal dependencies**
```bash
cd portal && npm install && cd ..
```
**Step D.2 — Start the portal development server**
```bash
cd portal && npm run dev &
```
Wait 5 seconds for Next.js to compile, then verify it is listening:
```bash
curl -s -o /dev/null -w "%{http_code}" http://localhost:3001
# Expected: 200 or 307 (redirect to /login)
```
**Step D.3 — Verify each portal route loads**
Open a browser and navigate to each of the following URLs. Each must load without a JavaScript
error in the browser console:
| URL | Expected |
|-----|---------|
| `http://localhost:3001/login` | Login page renders |
| `http://localhost:3001/agents` | Agent list renders (may be empty or show auth redirect) |
| `http://localhost:3001/credentials` | Credentials page renders |
| `http://localhost:3001/audit` | Audit log page renders |
| `http://localhost:3001/analytics` | Analytics dashboard renders |
| `http://localhost:3001/settings/tier` | Tier status page renders |
| `http://localhost:3001/compliance` | Compliance report page renders |
| `http://localhost:3001/webhooks` | Webhooks page renders |
| `http://localhost:3001/marketplace` | Marketplace page renders |
All 9 routes must load without a blank page or unhandled error.
**Step D.4 — Verify analytics charts render**
Navigate to `http://localhost:3001/analytics`.
Verify both of the following chart components are present in the page DOM:
```bash
curl -s http://localhost:3001/analytics | grep -c "recharts"
# Expected: 1 or more (recharts is used for TokenTrendChart and AgentHeatmap)
```
**Step D.5 — Verify tier status page**
Navigate to `http://localhost:3001/settings/tier`.
The page must display the current tier (expected: `free` for a new organisation).
**Step D.6 — Stop the portal**
```bash
kill $(lsof -ti:3001)
```
---
## Phase E — AGNTCY Conformance
**Step E.1 — Activate nvm**
```bash
export NVM_DIR="$HOME/.nvm" && source "$NVM_DIR/nvm.sh"
```
**Step E.2 — Run the AGNTCY conformance suite**
```bash
npm run test:agntcy-conformance
```
**Step E.3 — Expected output**
```
AGNTCY Conformance Suite
Agent Card Export
✓ exports valid AGNTCY agent card format
✓ agent card contains required identity fields
Compliance Report
✓ generates SOC2-aligned compliance report
✓ compliance report includes all required control domains
4 passing (Xs)
```
All 4 tests must pass. A failure indicates a regression in AGNTCY conformance.
**What each test validates:**
| Test | What it validates |
|------|------------------|
| `exports valid AGNTCY agent card format` | The `/api/v1/compliance/agent-cards` endpoint returns an array where each card has `id`, `name`, `version`, `capabilities`, `did` fields in AGNTCY format |
| `agent card contains required identity fields` | Each agent card's `identity` block includes `agent_id`, `organization_id`, `did`, and `deployment_env` |
| `generates SOC2-aligned compliance report` | The `/api/v1/compliance/report` endpoint returns a report with `generated_at`, `controls`, `summary` top-level keys |
| `compliance report includes all required control domains` | The `controls` array in the report includes entries for `access_control`, `audit_logging`, `credential_management`, and `tenant_isolation` |
---
## Phase F — Performance Baseline
> **Prerequisite:** Apache Bench (`ab`) must be installed. On Ubuntu: `sudo apt install apache2-utils`.
> Verify: `ab -V`
**Step F.1 — Create a token payload file**
```bash
cat > /tmp/token_payload.json << 'EOF'
grant_type=client_credentials&client_id=REPLACE_CLIENT_ID&client_secret=REPLACE_CLIENT_SECRET&scope=read
EOF
```
Replace `REPLACE_CLIENT_ID` and `REPLACE_CLIENT_SECRET` with `$NEW_CLIENT_ID` and
`$NEW_CLIENT_SECRET` from Phase B:
```bash
cat > /tmp/token_payload.txt << EOF
grant_type=client_credentials&client_id=${NEW_CLIENT_ID}&client_secret=${NEW_CLIENT_SECRET}&scope=read
EOF
```
**Step F.2 — Benchmark token endpoint**
```bash
ab -n 100 -c 10 \
-p /tmp/token_payload.txt \
-T "application/x-www-form-urlencoded" \
http://localhost:3000/api/v1/token
```
**Pass criteria for token endpoint:**
- `Requests per second` > 10
- `Time per request (mean)` < 100 ms
- p95 (95th percentile, shown as `95%` in the `Percentage of requests` table) < 100 ms
- Zero non-2xx responses
**Step F.3 — Benchmark agent list endpoint**
Ensure `$NEW_ACCESS_TOKEN` is still set and valid. Issue a fresh token if needed:
```bash
NEW_ACCESS_TOKEN=$(curl -s -X POST http://localhost:3000/api/v1/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&client_id=${NEW_CLIENT_ID}&client_secret=${NEW_CLIENT_SECRET}&scope=read" \
| jq -r '.access_token')
```
Run the benchmark:
```bash
ab -n 100 -c 10 \
-H "Authorization: Bearer $NEW_ACCESS_TOKEN" \
http://localhost:3000/api/v1/agents
```
**Pass criteria for agent list endpoint:**
- `Time per request (mean)` < 200 ms
- p95 (`95%` row in the `Percentage of requests` table) < 200 ms
- Zero non-2xx responses
**Step F.4 — Record results**
Record the following values from each `ab` output for the field trial report:
| Endpoint | Metric | Value |
|----------|--------|-------|
| `/api/v1/token` | Requests per second | |
| `/api/v1/token` | Mean time per request (ms) | |
| `/api/v1/token` | p95 (ms) | |
| `/api/v1/agents` | Requests per second | |
| `/api/v1/agents` | Mean time per request (ms) | |
| `/api/v1/agents` | p95 (ms) | |
A field trial passes Phase F if all p95 values are within the pass criteria above.
---
## Troubleshooting
Each entry follows the pattern: **Symptom** → **Cause** → **Fix** with exact commands.
---
**Port already in use**
Symptom:
```
Error response from daemon: driver failed programming external connectivity on endpoint
sentryagent-idp-app-1: Bind for 0.0.0.0:3000 failed: port is already allocated
```
Fix: Kill the process occupying the port, then restart:
```bash
lsof -ti:3000 | xargs kill
lsof -ti:5432 | xargs kill
lsof -ti:6379 | xargs kill
docker compose up --build -d
```
---
**Container shows `unhealthy`**
Symptom: `docker compose ps` shows `unhealthy` for a service.
Fix: Check logs for the unhealthy service:
```bash
docker compose logs postgres
docker compose logs redis
docker compose logs app
```
Common causes:
| Service | Cause | Fix |
|---------|-------|-----|
| `postgres` | Wrong database credentials | Verify `DATABASE_URL` in `.env` matches `docker-compose.yml` credentials |
| `redis` | Port conflict | Check `lsof -ti:6379` and kill occupying process |
| `app` | Missing env var | Check `docker compose logs app` for `Failed to start server` message |
---
**Migration fails — connection refused**
Symptom:
```
Migration failed: Error: connect ECONNREFUSED 127.0.0.1:5432
```
Cause: Running `npm run db:migrate` directly on the host (not inside the container) while
PostgreSQL is running inside Docker.
Fix: Always run migrations inside the container during a field trial:
```bash
docker compose exec app npm run db:migrate
```
---
**Migration fails — relation already exists**
Symptom:
```
Migration failed: Error: relation "agents" already exists
```
Cause: A previous partial migration run left the database in an inconsistent state.
Fix: Check which migrations have been applied:
```bash
docker compose exec postgres psql -U agentidp -d agentidp \
-c "SELECT name FROM schema_migrations ORDER BY name;"
```
If the database state cannot be repaired, reset it:
```bash
docker compose down -v
docker compose up --build -d
docker compose exec app npm run db:migrate
```
> `docker compose down -v` destroys all data. Use only when a clean slate is acceptable.
---
**JWT error — invalid signature or key format**
Symptom:
```
Failed to start server: Error: JWT_PRIVATE_KEY and JWT_PUBLIC_KEY environment variables are required
```
Or: All tokens return `401 Token signature is invalid`.
Cause: JWT keys in `.env` have incorrect PEM format — literal newlines instead of `\n`
sequences, or trailing whitespace.
Fix: Regenerate the keys and re-write them using the exact commands from Step 0.2 and 0.3.
Verify the key format in `.env`:
```bash
grep "JWT_PRIVATE_KEY" .env | head -c 100
# Expected: JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\nMII...
# NOT: JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
# MII...
```
The entire key must be on a single line with `\n` as literal backslash-n characters, not
actual newlines.
---
**Portal CORS error**
Symptom: Browser console shows:
```
Access to XMLHttpRequest at 'http://localhost:3000/api/v1/...' from origin 'http://localhost:3001'
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present
```
Cause: `CORS_ORIGIN` in `.env` does not include `http://localhost:3001`, or is set to a
different value.
Fix:
```bash
sed -i "s|CORS_ORIGIN=.*|CORS_ORIGIN=http://localhost:3001|" .env
docker compose up --build -d
```
Wait for the `app` container to become healthy before retrying.
---
**Tier counter not resetting**
Symptom: All API calls return 429 `tier_limit_exceeded` even after waiting.
Cause: The Redis tier counter was manually set in Test C.6 and not deleted.
Fix:
```bash
# Get your org_id from the token
ORG_ID=$(echo $NEW_ACCESS_TOKEN | cut -d. -f2 | base64 -d 2>/dev/null | jq -r '.organization_id')
docker compose exec redis redis-cli DEL "rate:tier:calls:$ORG_ID"
docker compose exec redis redis-cli DEL "rate:tier:tokens:$ORG_ID"
```
---
**`ab` not found**
Symptom: `ab: command not found`
Fix:
```bash
sudo apt-get update && sudo apt-get install -y apache2-utils
# or on macOS:
brew install httpd
```
---
**AGNTCY conformance test fails**
Symptom: One or more tests in `npm run test:agntcy-conformance` fail.
Diagnosis steps:
1. Ensure the backend is running and healthy: `curl -s http://localhost:3000/health`
2. Ensure `COMPLIANCE_ENABLED=true` in `.env` (check with `grep COMPLIANCE_ENABLED .env`)
3. Ensure at least one agent has been registered (Phase B must have been completed)
4. Check the test output for the specific assertion that failed
5. Check `docker compose logs app` for errors around compliance report generation
If the issue is a Redis cache hit returning stale data:
```bash
docker compose exec redis redis-cli KEYS "compliance:*" | xargs docker compose exec redis redis-cli DEL
```
Then re-run the conformance suite.

67
docs/devops/local-development.md
View File

@@ -6,9 +6,12 @@ Complete setup guide for running AgentIdP locally.
| Tool | Minimum version | Purpose |
|------|----------------|---------|
| Docker + Docker Compose | 24+ | Run PostgreSQL and Redis |
| Node.js | 18.0.0 | Run the application and migrations |
| Docker | 24+ | Container runtime |
| Docker Compose | 2.20+ | Multi-container orchestration |
| Node.js | 18.0.0 | Run the application, portal, and migrations |
| npm | 9+ | Package management and scripts |
| nvm | any | Recommended for managing Node.js versions |
| openssl | any | RSA key generation |
Verify versions:
@@ -19,6 +22,11 @@ node --version
npm --version
```
> **nvm activation:** If using nvm, activate it before running any Node.js commands:
> ```bash
> export NVM_DIR="$HOME/.nvm" && source "$NVM_DIR/nvm.sh"
> ```
---
## Step 1 — Clone and install dependencies
@@ -27,6 +35,9 @@ npm --version
git clone https://git.sentryagent.ai/vijay_admin/sentryagent-idp.git
cd sentryagent-idp
npm install
# Install portal dependencies
cd portal && npm install && cd ..
```
---
@@ -127,11 +138,10 @@ Expected output:
```
Running database migrations...
✓ Applied: 001_create_agents.sql
✓ Applied: 002_create_credentials.sql
✓ Applied: 003_create_audit_events.sql
✓ Applied: 004_create_tokens.sql
...
✓ Applied: 026_add_tenant_tiers.sql
Migrations complete. 4 migration(s) applied.
Migrations complete. 26 migration(s) applied.
```
See [database.md](database.md) for full migration documentation.
@@ -165,9 +175,52 @@ The compiled output is written to `dist/`. `npm start` runs `node dist/server.js
---
## Step 7 — Start the Next.js portal (optional)
The portal is a Next.js 14 application in the `portal/` directory. It communicates with the
AgentIdP backend at `http://localhost:3000`.
Start the portal development server:
```bash
cd portal && npm run dev
```
The portal starts on port 3001 by default. Open http://localhost:3001.
Available routes:
| Route | Description |
|-------|-------------|
| `/login` | OAuth 2.0 login page |
| `/agents` | Agent registry |
| `/credentials` | Credential management |
| `/audit` | Audit log viewer |
| `/analytics` | Token trend and agent activity charts |
| `/settings/tier` | Tier status and upgrade |
| `/compliance` | AGNTCY compliance report |
| `/webhooks` | Webhook subscription management |
| `/marketplace` | Agent marketplace |
Build the portal for production:
```bash
cd portal && npm run build
cd portal && npm start # serves the production build
```
Ensure `CORS_ORIGIN` in your `.env` includes `http://localhost:3001`:
```
CORS_ORIGIN=http://localhost:3001
```
---
## Full Docker Compose Stack
> **Note:** The `app` service in `docker-compose.yml` requires a `Dockerfile` which has not been written yet. This is a **Phase 1 P1 pending item**. The commands below will work once the Dockerfile exists.
> The full Docker Compose stack (including the `app` container) is available for field trial
> deployments — see the [field trial guide](field-trial.md). For day-to-day development, start
> only the infrastructure services and run the application directly.
When the Dockerfile is available, the entire stack (infrastructure + application) can be started with:

110
docs/devops/operations.md
View File

@@ -18,21 +18,22 @@ Always start services in this order. Starting the application before PostgreSQL
### Startup checklist
```bash
# 1. Start PostgreSQL and Redis
docker-compose up -d postgres redis
# 1. Start the full stack
docker compose up --build -d
# 2. Wait for healthy status
docker-compose ps
# Both postgres and redis must show "healthy" before proceeding
# 2. Verify all three services are healthy
docker compose ps
# app, postgres, and redis must all show "healthy"
# 3. Run migrations
npm run db:migrate
# Must complete with 0 errors before starting the app
docker compose exec app npm run db:migrate
# 4. Start the application
npm run dev # development
# or
npm start # production (requires prior npm run build)
# 4. Verify application health
curl http://localhost:3000/health
# Expected: {"status":"ok"}
# 5. (Optional) Start the portal for local dev
cd portal && npm run dev
```
---
@@ -115,9 +116,12 @@ docker-compose exec redis redis-cli
| Key pattern | Example | Purpose | TTL |
|------------|---------|---------|-----|
| `revoked:<jti>` | `revoked:f1e2d3c4-b5a6-...` | Revoked token JTI | Remaining token lifetime |
| `rate:<client_id>:<window>` | `rate:a1b2c3...:29086156` | Request count per minute window | 60 seconds |
| `monthly:<client_id>:<year>:<month>` | `monthly:a1b2c3...:2026:3` | Token issuance count for free tier | End of month |
| `revoked:<jti>` | `revoked:f1e2d3c4-...` | Revoked token JTI | Remaining token lifetime |
| `rate:<client_id>:<window>` | `rate:a1b2c3...:29086156` | Request count per window | `RATE_LIMIT_WINDOW_MS` |
| `monthly:<client_id>:<year>:<month>` | `monthly:a1b2c3...:2026:3` | Monthly token issuance count | End of month |
| `rate:tier:calls:<tenantId>` | `rate:tier:calls:org-uuid` | Daily API call counter for tier enforcement | Until midnight UTC |
| `rate:tier:tokens:<tenantId>` | `rate:tier:tokens:org-uuid` | Daily token issuance counter for tier enforcement | Until midnight UTC |
| `compliance:report:<tenantId>` | `compliance:report:org-uuid` | Cached compliance report JSON | 5 minutes |
Inspect keys:
@@ -130,6 +134,16 @@ redis-cli GET "rate:<client_id>:<window_key>"
# Check monthly token count for a specific client
redis-cli GET "monthly:<client_id>:2026:3"
# Check tier API call counter for a tenant
redis-cli GET "rate:tier:calls:<org_id>"
# Check tier token counter for a tenant
redis-cli GET "rate:tier:tokens:<org_id>"
# Check cached compliance report for a tenant
redis-cli GET "compliance:report:<org_id>"
redis-cli TTL "compliance:report:<org_id>"
```
Where `<window_key>` is `floor(unix_ms / 60000)`. For the current window:
@@ -258,12 +272,25 @@ AgentIdP exposes a Prometheus metrics endpoint at `GET /metrics` (unauthenticate
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `agentidp_tokens_issued_total` | Counter | `scope` | OAuth 2.0 tokens issued successfully |
| `agentidp_agents_registered_total` | Counter | `deployment_env` | Agents registered successfully |
| `agentidp_http_requests_total` | Counter | `method`, `route`, `status_code` | HTTP requests received |
| `agentidp_http_request_duration_seconds` | Histogram | `method`, `route`, `status_code` | HTTP request duration |
| `agentidp_tokens_issued_total` | Counter | `scope` | OAuth 2.0 tokens issued |
| `agentidp_agents_registered_total` | Counter | `deployment_env` | Agents registered |
| `agentidp_http_requests_total` | Counter | `method`, `route`, `status_code` | HTTP requests |
| `agentidp_http_request_duration_seconds` | Histogram | `method`, `route`, `status_code` | HTTP latency |
| `agentidp_db_query_duration_seconds` | Histogram | `operation` | PostgreSQL query duration |
| `agentidp_redis_command_duration_seconds` | Histogram | `command` | Redis command duration |
| `agentidp_webhook_dead_letters_total` | Counter | `event_type` | Webhook deliveries moved to dead-letter queue |
| `agentidp_credentials_expiring_soon_total` | Gauge | — | Credentials expiring within 7 days |
| `agentidp_audit_chain_integrity` | Gauge | — | `1` if audit chain is intact, `0` if broken |
| `agentidp_rate_limit_hits_total` | Counter | `client_id` | Rate limit rejections |
| `agentidp_db_pool_active_connections` | Gauge | — | Active PostgreSQL connections |
| `agentidp_db_pool_waiting_requests` | Gauge | — | Requests waiting for a pool connection |
| `agentidp_tenant_api_calls_total` | Counter | `org_id`, `tier` | API calls per tenant per tier |
| `agentidp_billing_limit_rejections_total` | Counter | `org_id`, `limit_type` | Tier limit enforcement rejections |
| `agentidp_did_documents_generated_total` | Counter | — | DID documents generated |
| `agentidp_oidc_tokens_issued_total` | Counter | — | OIDC ID tokens issued |
| `agentidp_federation_events_total` | Counter | `event_type` | Federation partner events |
| `agentidp_delegation_chains_created_total` | Counter | — | A2A delegation chains created |
| `agentidp_compliance_reports_generated_total` | Counter | — | Compliance reports generated |
### Starting the Monitoring Stack
@@ -282,3 +309,50 @@ The Grafana dashboard auto-provisions on first start. Navigate to **Dashboards
`GET /metrics` is unauthenticated. In production, ensure this endpoint is:
- Only accessible from your internal network (firewall rule or reverse proxy restriction)
- Not exposed on a public-facing port
---
### Tier limit rejected — 429 with `tier_limit_exceeded` code
Symptom: `429 TOO_MANY_REQUESTS` with body `{"code":"tier_limit_exceeded","message":"..."}`
Check the tenant's current tier counter:
```bash
# Check API call counter
docker compose exec redis redis-cli GET "rate:tier:calls:<org_id>"
# Check the tenant's tier
psql "$DATABASE_URL" -c "SELECT org_id, tier FROM tenant_tiers WHERE org_id = '<org_id>';"
```
If the org is on the `free` tier and has hit 1,000 calls/day, upgrade the tier or wait until
midnight UTC for the counter to reset.
---
### Analytics endpoints return 404
Cause: `ANALYTICS_ENABLED` is set to `false` in `.env`.
Fix: Set `ANALYTICS_ENABLED=true` and restart the application.
---
### Compliance report returns 404
Cause: `COMPLIANCE_ENABLED` is set to `false` in `.env`.
Fix: Set `COMPLIANCE_ENABLED=true` and restart the application.
---
### Portal CORS error
Symptom: Browser console shows `Access-Control-Allow-Origin` error on requests to
`http://localhost:3000`.
Fix: Ensure `CORS_ORIGIN` in `.env` includes `http://localhost:3001`:
```
CORS_ORIGIN=http://localhost:3001
```
Restart the application after changing this variable.

6
docs/devops/security.md
View File

@@ -87,6 +87,12 @@ Rotating the JWT keys invalidates all currently active tokens — every authenti
**Important:** There is no grace period or dual-key support in Phase 1. All tokens issued with the old private key are immediately rejected after rotation. If zero-downtime key rotation is required, it is a Phase 2 feature.
> **OIDC keys** are separate from the main JWT keys. OIDC signing keys are stored in the
> `oidc_keys` PostgreSQL table (created by migration `014_create_oidc_keys_table.sql`), encrypted
> at rest using pgcrypto (enabled by migration `018_enable_pgcrypto.sql`). The `OIDCKeyService`
> manages rotation. OIDC keys do not need to be set as environment variables — they are
> provisioned automatically on first startup.
---
## CORS Configuration

4
docs/devops/vault-setup.md
View File

@@ -47,6 +47,10 @@ VAULT_TOKEN=dev-root-token
VAULT_MOUNT=secret
```
> **Note:** The `.env.example` file uses `VAULT_KV_MOUNT` as the variable name. The application
> reads both `VAULT_KV_MOUNT` and `VAULT_MOUNT` — prefer `VAULT_KV_MOUNT` in new configurations
> for consistency with the current `.env.example`.
The KV v2 secrets engine is automatically enabled at `secret/` in dev mode. No further configuration is needed.
> **Warning**: Dev mode stores everything in memory. Data is lost when the container stops. Do not use dev mode in production.