sentryagent-idp/docs/devops/database.md

# Database

AgentIdP uses PostgreSQL 14+ as its primary data store. The schema consists of 26 migrations managed by a custom migration runner.

---

## Schema Overview

```
organizations
  ├── agents (FK: organization_id → organizations.org_id)
  │     ├── credentials (FK: client_id → agents.agent_id, CASCADE DELETE)
  │     └── agent_did_keys (FK: agent_id → agents.agent_id)
  └── audit_events (FK: organization_id — informational, no cascade)

token_revocations (no FK — independent revocation store)
oidc_keys (standalone — OIDC signing key rotation)
federation_partners (standalone — cross-tenant identity)
webhook_subscriptions → webhook_deliveries (FK: subscription_id)
agent_marketplace (standalone — agent discovery catalog)
github_oidc_trust_policies (standalone — CI/CD trust)
billing (FK: org_id → organizations.org_id — one row per org)
delegation_chains (standalone — A2A delegation records)
analytics_events (FK: organization_id — append-only)
tenant_tiers (FK: org_id → organizations.org_id — one row per org)
```

---

## Tables

### `agents`

The Agent Registry. One row per registered AI agent identity.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `agent_id` | `UUID` | No | Primary key — system-assigned, immutable |
| `email` | `VARCHAR(255)` | No | Unique email-format identifier |
| `agent_type` | `VARCHAR(32)` | No | Enum: `screener`, `classifier`, `orchestrator`, `extractor`, `summarizer`, `router`, `monitor`, `custom` |
| `version` | `VARCHAR(64)` | No | Semantic version string |
| `capabilities` | `TEXT[]` | No | Array of `resource:action` strings |
| `owner` | `VARCHAR(128)` | No | Owning team or organisation |
| `deployment_env` | `VARCHAR(16)` | No | Enum: `development`, `staging`, `production` |
| `status` | `VARCHAR(24)` | No | Enum: `active`, `suspended`, `decommissioned`. Default: `active` |
| `created_at` | `TIMESTAMPTZ` | No | Registration timestamp. Default: `NOW()` |
| `updated_at` | `TIMESTAMPTZ` | No | Last update timestamp. Default: `NOW()` |

**Indexes:**

| Index | Column | Purpose |
|-------|--------|---------|
| `idx_agents_email` | `email` | Unique lookup on registration and conflict check |
| `idx_agents_status` | `status` | Filter by lifecycle status |
| `idx_agents_owner` | `owner` | Filter by owner |
| `idx_agents_agent_type` | `agent_type` | Filter by type |
| `idx_agents_created_at` | `created_at DESC` | Default sort for list queries |

**Constraints:**
- `email` is UNIQUE — one registration per email address
- `agent_type` and `deployment_env` and `status` have CHECK constraints enforcing the enum values

---

### `credentials`

OAuth 2.0 client credentials. One agent can have multiple credentials.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `credential_id` | `UUID` | No | Primary key — system-assigned |
| `client_id` | `UUID` | No | FK → `agents.agent_id` (CASCADE DELETE) |
| `secret_hash` | `VARCHAR(255)` | No | bcrypt hash of the client secret. Plaintext is never stored. |
| `status` | `VARCHAR(16)` | No | Enum: `active`, `revoked`. Default: `active` |
| `created_at` | `TIMESTAMPTZ` | No | Creation timestamp |
| `expires_at` | `TIMESTAMPTZ` | Yes | Optional expiry. NULL = no expiry. |
| `revoked_at` | `TIMESTAMPTZ` | Yes | Revocation timestamp. NULL = not revoked. |

**Indexes:**

| Index | Column | Purpose |
|-------|--------|---------|
| `idx_credentials_client_id` | `client_id` | List credentials for an agent |
| `idx_credentials_status` | `status` | Filter active/revoked |
| `idx_credentials_created_at` | `created_at DESC` | Default sort |

**Cascade behaviour:** Deleting an agent record cascades and deletes all associated credentials. In practice, agents are soft-deleted (status → `decommissioned`) not hard-deleted, so this cascade is a safety net.

---

### `audit_events`

Immutable audit log. Append-only by design — no application-layer UPDATE or DELETE is ever issued against this table.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `event_id` | `UUID` | No | Primary key — system-assigned |
| `agent_id` | `UUID` | No | Agent that triggered the event (informational, no FK) |
| `action` | `VARCHAR(32)` | No | Enum — see values below |
| `outcome` | `VARCHAR(16)` | No | Enum: `success`, `failure` |
| `ip_address` | `VARCHAR(64)` | No | Client IP address (IPv4 or IPv6) |
| `user_agent` | `TEXT` | No | HTTP User-Agent from the request |
| `metadata` | `JSONB` | No | Action-specific data. Default: `{}` |
| `timestamp` | `TIMESTAMPTZ` | No | Event timestamp. Default: `NOW()` |

**`action` enum values:** `agent.created`, `agent.updated`, `agent.decommissioned`, `agent.suspended`, `agent.reactivated`, `token.issued`, `token.revoked`, `token.introspected`, `credential.generated`, `credential.rotated`, `credential.revoked`, `auth.failed`

**Indexes:**

| Index | Column | Purpose |
|-------|--------|---------|
| `idx_audit_events_agent_id` | `agent_id` | Filter events by agent |
| `idx_audit_events_action` | `action` | Filter by action type |
| `idx_audit_events_outcome` | `outcome` | Filter successes/failures |
| `idx_audit_events_timestamp` | `timestamp DESC` | Default sort, date range queries |

**Why no FK on `agent_id`?** Audit records must be retained even after an agent is decommissioned. A FK would prevent decommission or cascade-delete history. The `agent_id` is stored as an informational reference only.

**Free tier retention:** The application enforces a 90-day retention window at the query layer. Purging old records is not yet automated — it is a Phase 2 task.

---

### `token_revocations`

Durable record of revoked JWT tokens. Supplements Redis for durability across Redis restarts.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `jti` | `UUID` | No | Primary key — the JWT ID claim from the revoked token |
| `expires_at` | `TIMESTAMPTZ` | No | When the token would have expired naturally |
| `revoked_at` | `TIMESTAMPTZ` | No | When the token was revoked. Default: `NOW()` |

**Indexes:**

| Index | Column | Purpose |
|-------|--------|---------|
| `idx_token_revocations_expires_at` | `expires_at` | Enables future cleanup of expired revocation records |

**Dual-store design:** When a token is revoked, the `jti` is written to both:
1. Redis key `revoked:<jti>` with TTL set to the token's remaining lifetime — fast O(1) lookup on every authenticated request
2. This PostgreSQL table — durable record if Redis is restarted

**Note:** On Redis restart, the in-memory revocation cache is cold. Tokens revoked before the restart will pass auth until Phase 2 implements a warm-up that loads active revocations from PostgreSQL into Redis on startup.

---

### `organizations`

Created by migration `006_create_organizations_table.sql`.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `org_id` | `UUID` | No | Primary key |
| `name` | `VARCHAR(255)` | No | Organisation display name |
| `slug` | `VARCHAR(64)` | No | URL-safe unique identifier |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |

---

### `agent_did_keys`

Created by migration `012_create_agent_did_keys_table.sql`.

Stores the DID document key material for each agent. One agent may have multiple keys for
rotation purposes.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `agent_id` | `UUID` | No | FK → `agents.agent_id` |
| `key_id` | `VARCHAR(255)` | No | DID key fragment identifier |
| `public_key_jwk` | `JSONB` | No | Public key in JWK format |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |

---

### DID columns on `agents`

Added by migration `013_add_did_columns_to_agents.sql`:

- `did` — `VARCHAR(512)` nullable — the `did:web` identifier for this agent
- `did_document` — `JSONB` nullable — full DID document

---

### `oidc_keys`

Created by migration `014_create_oidc_keys_table.sql`.

Stores RSA key pairs used for OIDC ID token signing. Supports key rotation — active key is
determined by the most recently created row.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `kid` | `VARCHAR(128)` | No | Key ID — referenced in JWKS |
| `private_key_pem` | `TEXT` | No | Encrypted RSA private key (pgcrypto) |
| `public_key_pem` | `TEXT` | No | RSA public key |
| `algorithm` | `VARCHAR(16)` | No | Always `RS256` |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |

---

### `federation_partners`

Created by migration `015_create_federation_partners_table.sql`.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `org_id` | `UUID` | No | Owning organisation |
| `partner_name` | `VARCHAR(255)` | No | Display name |
| `partner_jwks_url` | `TEXT` | No | URL to partner's JWKS endpoint |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |

---

### `webhook_subscriptions`

Created by migration `016_create_webhook_subscriptions_table.sql`.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `org_id` | `UUID` | No | Owning organisation |
| `event_type` | `VARCHAR(128)` | No | Event type filter (e.g. `agent.created`) |
| `target_url` | `TEXT` | No | HTTPS delivery endpoint |
| `secret` | `VARCHAR(255)` | Yes | HMAC signing secret for delivery verification |
| `active` | `BOOLEAN` | No | Default: `true` |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |

---

### `webhook_deliveries`

Created by migration `017_create_webhook_deliveries_table.sql`.

Records each delivery attempt for a webhook event, including the dead-letter queue entries.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `subscription_id` | `UUID` | No | FK → `webhook_subscriptions.id` |
| `event_type` | `VARCHAR(128)` | No | Event type delivered |
| `payload` | `JSONB` | No | Full event payload |
| `status` | `VARCHAR(32)` | No | `pending`, `delivered`, `failed`, `dead_letter` |
| `response_status` | `INTEGER` | Yes | HTTP status from delivery endpoint |
| `attempt_count` | `INTEGER` | No | Default: `0` |
| `last_attempted_at` | `TIMESTAMPTZ` | Yes | |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |

**Dead-letter queue:** After 3 failed delivery attempts, the row status is set to `dead_letter`
and the `agentidp_webhook_dead_letters_total` Prometheus counter is incremented. The Prometheus
metric label is `event_type`.

---

### pgcrypto extension

Enabled by migration `018_enable_pgcrypto.sql`. Used for encrypting sensitive columns in
`oidc_keys` and credential data.

---

### `agent_marketplace`

Created by migration `021_add_agent_marketplace.sql`.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `agent_id` | `UUID` | No | FK → `agents.agent_id` |
| `listing_name` | `VARCHAR(255)` | No | Display name in marketplace |
| `description` | `TEXT` | Yes | Markdown description |
| `tags` | `TEXT[]` | No | Searchable tags. Default: `{}` |
| `published` | `BOOLEAN` | No | Default: `false` |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |

---

### `github_oidc_trust_policies`

Created by migration `022_add_github_oidc_trust_policies.sql`.

Maps GitHub Actions OIDC claims to agent identities for CI/CD token exchange.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `org_id` | `UUID` | No | Owning organisation |
| `repository` | `VARCHAR(512)` | No | GitHub repository slug (`owner/repo`) |
| `branch` | `VARCHAR(255)` | Yes | Branch filter (null = any branch) |
| `agent_id` | `UUID` | No | Agent to issue a token for on match |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |

---

### `billing`

Created by migration `023_add_billing.sql`.

One row per organisation. Tracks the org's Stripe customer and subscription state.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `org_id` | `UUID` | No | FK → `organizations.org_id` (UNIQUE) |
| `stripe_customer_id` | `VARCHAR(255)` | Yes | Stripe Customer ID |
| `stripe_subscription_id` | `VARCHAR(255)` | Yes | Stripe Subscription ID |
| `status` | `VARCHAR(64)` | No | Stripe subscription status or `none` |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |

---

### `delegation_chains`

Created by migration `024_add_delegation_chains.sql`.

Records A2A delegation grants created via the delegation API.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `delegator_agent_id` | `UUID` | No | Agent granting the delegation |
| `delegate_agent_id` | `UUID` | No | Agent receiving the delegation |
| `scopes` | `TEXT[]` | No | Scopes being delegated |
| `expires_at` | `TIMESTAMPTZ` | Yes | Optional expiry |
| `created_at` | `TIMESTAMPTZ` | No | Default: `NOW()` |

---

### `analytics_events`

Created by migration `025_add_analytics_events.sql`.

Append-only event store for analytics. Supports token trend, agent activity, and usage summary
queries.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `organization_id` | `UUID` | No | Owning organisation |
| `date` | `DATE` | No | Calendar date of the event (UTC) |
| `metric_type` | `VARCHAR(64)` | No | e.g. `token_issued`, `agent_called` |
| `count` | `INTEGER` | No | Event count for this date+type |

**Index:** `(organization_id, date DESC)` for fast time-series queries.

---

### `tenant_tiers`

Created by migration `026_add_tenant_tiers.sql`.

One row per organisation. Stores the current tier and enforces tier limits via the
`tierEnforcement` middleware.

| Column | Type | Nullable | Description |
|--------|------|----------|-------------|
| `id` | `UUID` | No | Primary key |
| `org_id` | `UUID` | No | FK → `organizations.org_id` (UNIQUE) |
| `tier` | `ENUM('free','pro','enterprise')` | No | Current tier. Default: `free` |
| `updated_at` | `TIMESTAMPTZ` | No | Last tier change. Default: `NOW()` |

**Tier limits** (from `src/config/tiers.ts`):

| Tier | Max Agents | Max API Calls/Day | Max Tokens/Day |
|------|-----------|-------------------|----------------|
| free | 10 | 1,000 | 1,000 |
| pro | 100 | 50,000 | 50,000 |
| enterprise | unlimited | unlimited | unlimited |

---

## Migration Runner

Migrations are managed by `scripts/migrate.ts`. It reads `.sql` files from `src/db/migrations/` in alphabetical order, tracks applied migrations in a `schema_migrations` table, and executes only unapplied migrations — each in its own transaction.

### `schema_migrations` table

Created automatically on first run if it does not exist.

| Column | Type | Description |
|--------|------|-------------|
| `name` | `VARCHAR(255)` | Migration filename (primary key) |
| `applied_at` | `TIMESTAMPTZ` | When the migration was applied |

### Running migrations

```bash
# Set DATABASE_URL in environment or .env first
npm run db:migrate
```

Expected output (first run):

```
Running database migrations...
  ✓ Applied: 001_create_agents.sql
  ✓ Applied: 002_create_credentials.sql
  ...
  ✓ Applied: 025_add_analytics_events.sql
  ✓ Applied: 026_add_tenant_tiers.sql

Migrations complete. 26 migration(s) applied.
```

Expected output (already applied):

```
Running database migrations...
  - Skipped (already applied): 001_create_agents.sql
  - Skipped (already applied): 002_create_credentials.sql
  - Skipped (already applied): 003_create_audit_events.sql
  - Skipped (already applied): 004_create_tokens.sql

Migrations complete. 0 migration(s) applied.
```

### Verifying applied migrations

```bash
psql "$DATABASE_URL" -c "SELECT name, applied_at FROM schema_migrations ORDER BY name;"
```

Expected output:

```
               name                |          applied_at
-----------------------------------+-------------------------------
 001_create_agents.sql             | 2026-03-28 09:00:00.000000+00
 002_create_credentials.sql        | 2026-03-28 09:00:00.000000+00
 ...
 025_add_analytics_events.sql      | 2026-04-04 09:00:00.000000+00
 026_add_tenant_tiers.sql          | 2026-04-04 09:00:00.000000+00
(26 rows)
```

### Adding a new migration

1. Create a new `.sql` file in `src/db/migrations/` with the next numeric prefix (e.g. `005_add_column.sql`)
2. Write idempotent SQL using `IF NOT EXISTS` / `IF EXISTS` guards where possible
3. Run `npm run db:migrate`

Migrations are run in alphabetical filename order. The prefix ensures correct ordering.

### Rollback

There is no automated rollback. To undo a migration:
1. Write and apply a compensating migration (e.g. `005_rollback_add_column.sql`)
2. Or connect directly to PostgreSQL and run the reverse SQL manually

---

## Connection Pool

The application uses `pg.Pool` with settings read from environment variables. The pool is a
singleton — one pool per process instance.

| Variable | Default | Description |
|----------|---------|-------------|
| `DB_POOL_MAX` | `20` | Maximum connections |
| `DB_POOL_MIN` | `2` | Minimum idle connections |
| `DB_POOL_IDLE_TIMEOUT_MS` | `30000` | Idle eviction timeout (ms) |
| `DB_POOL_CONNECTION_TIMEOUT_MS` | `5000` | Acquisition timeout (ms) |

Pool size is exposed as Prometheus metrics: `agentidp_db_pool_active_connections` and
`agentidp_db_pool_waiting_requests`. Monitor these in production to detect pool exhaustion.