docs: engineering knowledge base for new hires
Complete docs/engineering/ suite — 12 documents covering company overview, system architecture, tech stack ADRs, codebase structure, service deep dives, annotated code walkthroughs, dev setup, engineering workflow, testing strategy, deployment/ops, SDK guide, and README index. All content verified against source files. All 82 tasks in openspec/changes/engineering-docs/tasks.md marked complete. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
SentryAgent.ai Developer
115
docs/engineering/01-overview.md
Normal file
115
docs/engineering/01-overview.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# SentryAgent.ai — Company and Product Overview
|
||||
|
||||
---
|
||||
|
||||
## 1. Company Mission
|
||||
|
||||
SentryAgent.ai is building the world's first free, open-source Agent Identity Provider
|
||||
(AgentIdP) — democratising AI agent authentication, authorisation, and governance for
|
||||
developers worldwide. The core problem it solves is one that did not have a standard
|
||||
answer until now: when an AI agent needs to call an API, how does it prove who it is?
|
||||
How does it obtain a short-lived token? How does a security team revoke its access the
|
||||
moment it is compromised? How does compliance require a full, tamper-proof record of
|
||||
everything that agent ever did? Traditional identity infrastructure — built for humans
|
||||
and static service accounts — was not designed for the fluid lifecycle of AI agents.
|
||||
|
||||
AgentIdP is the answer. It is a REST API server that acts as an identity provider
|
||||
designed specifically for non-human AI agents. Agents register with a stable UUID
|
||||
identity, authenticate via the OAuth 2.0 Client Credentials grant (RFC 6749), receive
|
||||
short-lived RS256 JWTs, and are governed by an OPA policy engine that enforces
|
||||
capability-based access control at runtime. Every significant event is written to an
|
||||
immutable audit log. The entire system is built on open standards: OAuth 2.0, RFC 7662
|
||||
(introspection), RFC 7009 (revocation), OpenAPI 3.0, and the AGNTCY interoperability
|
||||
standard from the Linux Foundation.
|
||||
|
||||
The market context is one of rapid proliferation. Enterprises are deploying dozens,
|
||||
then hundreds, then thousands of autonomous AI agents — each one acting on behalf of
|
||||
the organisation, calling APIs, reading sensitive data, and making decisions. Without
|
||||
standardised identity infrastructure, there is no way to audit what happened, no way
|
||||
to revoke a compromised agent cleanly, and no standard protocol for agents from
|
||||
different vendors to authenticate to each other. SentryAgent.ai fills this gap,
|
||||
providing every developer — from a student working alone to a global enterprise — the
|
||||
same enterprise-grade identity layer for free.
|
||||
|
||||
---
|
||||
|
||||
## 2. What is AGNTCY?
|
||||
|
||||
AGNTCY (pronounced "agency") is an open interoperability standard for AI agents,
|
||||
maintained under the Linux Foundation. Its central premise is that AI agents must be
|
||||
treated as first-class identities — with stable identifiers, standard authentication
|
||||
protocols, lifecycle management, and accountability mechanisms — in the same way that
|
||||
human users and cloud service accounts are today.
|
||||
|
||||
AgentIdP is the first production IdP implementing AGNTCY-aligned agent identity across
|
||||
all six AGNTCY domains:
|
||||
|
||||
| AGNTCY Domain | How AgentIdP Implements It |
|
||||
|---------------|---------------------------|
|
||||
| Non-Human Identity | Every agent receives an immutable UUID (`agentId`) assigned at registration. The identifier is DID-ready — structured to be portable into W3C DID documents in Phase 3. |
|
||||
| Agent Registry | `POST /api/v1/agents` registers an agent. `GET /api/v1/agents` lists all agents. `GET /api/v1/agents/:id` retrieves a single agent by UUID. |
|
||||
| Credential Management | Each agent holds one or more `(client_id, client_secret)` credential pairs. Secrets are bcrypt-hashed in PostgreSQL or stored in HashiCorp Vault KV v2. Credentials can be rotated and revoked independently. |
|
||||
| Authentication | OAuth 2.0 Client Credentials grant per RFC 6749. Agents POST `grant_type=client_credentials` with their `client_id` and `client_secret` to receive a signed RS256 JWT. |
|
||||
| Lifecycle | Agents transition through `active`, `suspended`, and `decommissioned` states. Decommissioning is a soft delete that cascades to revoke all active credentials. Suspended agents cannot obtain new tokens. |
|
||||
| Audit | Every significant platform event is written to an immutable `audit_events` table in PostgreSQL. Events carry `agentId`, `action`, `outcome`, `ipAddress`, `userAgent`, `metadata`, and `timestamp`. The free tier retains 90 days of history. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Product Features
|
||||
|
||||
| Feature | Endpoint(s) | Notes |
|
||||
|---------|-------------|-------|
|
||||
| Agent Registry | `POST /api/v1/agents`, `GET /api/v1/agents`, `GET /api/v1/agents/:id`, `PATCH /api/v1/agents/:id`, `DELETE /api/v1/agents/:id` | Full CRUD with lifecycle; free tier capped at 100 registered agents |
|
||||
| OAuth 2.0 Token Issuance | `POST /api/v1/token` | Client Credentials flow (RFC 6749); issues RS256 JWTs; free tier capped at 10,000 tokens/month |
|
||||
| Token Introspection | `POST /api/v1/token/introspect` | RFC 7662 compliant; always returns 200, check `active` field |
|
||||
| Token Revocation | `POST /api/v1/token/revoke` | RFC 7009 compliant; idempotent; agents may only revoke their own tokens |
|
||||
| Credential Management | `POST /api/v1/agents/:id/credentials`, `GET /api/v1/agents/:id/credentials`, `DELETE /api/v1/agents/:id/credentials/:credId` | `client_secret` returned once on creation; never retrievable again |
|
||||
| Credential Rotation | `POST /api/v1/agents/:id/credentials/:credId/rotate` | Generates new secret; old secret immediately invalidated; atomic |
|
||||
| Audit Log | `GET /api/v1/audit`, `GET /api/v1/audit/:id` | Immutable, filterable by `agentId`, `action`, `outcome`, date range; paginated |
|
||||
| Web Dashboard | `/dashboard` | React 18 SPA — agents list, agent detail, credentials management, audit log, health views |
|
||||
| OPA Policy Engine | (middleware on all protected routes) | Dynamic scope-based authorisation; Rego policy in `policies/authz.rego`; hot-reload via SIGHUP |
|
||||
| Prometheus Metrics | `GET /metrics` | prom-client; all HTTP routes instrumented with request counter and duration histogram |
|
||||
| HashiCorp Vault | (opt-in, via `VAULT_ADDR` + `VAULT_TOKEN`) | KV v2 secret storage; constant-time comparison; bcrypt fallback when Vault is not configured |
|
||||
| Health Check | `GET /health` | Checks PostgreSQL and Redis connectivity; unauthenticated; used by load balancers |
|
||||
|
||||
---
|
||||
|
||||
## 4. Phase Roadmap
|
||||
|
||||
| Phase | Status | Key Deliverables |
|
||||
|-------|--------|-----------------|
|
||||
| Phase 1 — MVP | COMPLETE | Agent registry, OAuth 2.0 Client Credentials (RS256 JWTs), credential management (bcrypt), immutable audit log, Node.js SDK, Dockerfile, Docker Compose, AGNTCY alignment documentation, >80% test coverage |
|
||||
| Phase 2 — Production-Ready | COMPLETE | HashiCorp Vault opt-in integration, Python SDK (sync + async), Go SDK (context-aware), Java SDK (builder + CompletableFuture), OPA policy engine (Rego + Wasm + TypeScript fallback), React 18 + Vite 5 web dashboard, Prometheus metrics + Grafana dashboards, Terraform multi-region deployment (AWS ECS + RDS + ElastiCache; GCP Cloud Run + Cloud SQL + Memorystore) |
|
||||
| Phase 3 — Enterprise | PLANNED | AGNTCY federation (cross-IdP agent identity), W3C Decentralised Identifiers (DIDs), agent marketplace, advanced compliance reporting, SOC 2 Type II certification, enterprise tier (custom retention, SLAs, advanced RBAC) |
|
||||
|
||||
---
|
||||
|
||||
## 5. Virtual Engineering Team
|
||||
|
||||
SentryAgent.ai uses a Virtual Engineering Team (VET) model — all engineering work is
|
||||
designed, implemented, tested, and reviewed by Claude Code instances fulfilling defined
|
||||
engineering roles. The CEO (human) is the sole business decision-maker. The CTO (Claude)
|
||||
owns technical architecture and manages the engineering team autonomously. The team
|
||||
follows a strict spec-first workflow governed by the OpenSpec change management process:
|
||||
no implementation begins until an OpenAPI specification is approved by the CTO.
|
||||
|
||||
| Role | Responsibility | Approval Gate |
|
||||
|------|---------------|---------------|
|
||||
| CEO | Business priorities, scope approval, architecture approval | All scope changes, new dependencies, git push to main |
|
||||
| Virtual CTO | Architecture, technical standards, engineering team coordination, risk management | Reports to CEO; approves all implementation before commit; approves all QA sign-offs before merge |
|
||||
| Virtual Architect | OpenAPI specs, ADRs, system design, database schemas | CTO review required before implementation begins |
|
||||
| Virtual Principal Developer | TypeScript implementation per approved spec; JSDoc; zero `any` types | CTO review required before QA begins |
|
||||
| Virtual QA Engineer | Jest + Supertest test suites; >80% coverage; all quality gates | All gates must pass before CTO signs off for merge |
|
||||
|
||||
---
|
||||
|
||||
## 6. Free Tier Limits
|
||||
|
||||
| Limit | Value |
|
||||
|-------|-------|
|
||||
| Max agents | 100 |
|
||||
| Max credentials per agent | No hard cap enforced in code (5 is the documented recommendation) |
|
||||
| Max tokens in flight | 10,000 per agent per calendar month |
|
||||
| Token TTL | 3,600 seconds (1 hour) |
|
||||
| Audit log retention | 90 days |
|
||||
| API rate limit | 100 requests per minute per IP address |
|
||||
140
docs/engineering/02-architecture.md
Normal file
140
docs/engineering/02-architecture.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# System Architecture
|
||||
|
||||
---
|
||||
|
||||
## 1. Component Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
Client["Client (AI Agent / Browser / CI)"]
|
||||
|
||||
Client -->|HTTPS| ExpressApp["Express App (AgentIdP)"]
|
||||
|
||||
subgraph ExpressApp["Express App — src/app.ts"]
|
||||
Router["Router (src/routes/)"]
|
||||
AuthMW["authMiddleware (src/middleware/auth.ts)"]
|
||||
OpaMW["opaMiddleware (src/middleware/opa.ts)"]
|
||||
Controller["Controller (src/controllers/)"]
|
||||
Service["Service (src/services/)"]
|
||||
Repository["Repository (src/repositories/)"]
|
||||
Router --> AuthMW --> OpaMW --> Controller --> Service --> Repository
|
||||
end
|
||||
|
||||
Repository -->|parameterized SQL| PG["PostgreSQL 14\n(agents, credentials, audit_events, token_revocations)"]
|
||||
Service -->|Redis commands| Redis["Redis 7\n(token revocation list, monthly counts, rate-limit counters)"]
|
||||
Service -->|KV v2 read/write| Vault["HashiCorp Vault\n(opt-in — when VAULT_ADDR is set)"]
|
||||
|
||||
ExpressApp -->|evaluate input| OPA["OPA Policy Engine\n(policies/authz.rego + data/scopes.json)"]
|
||||
ExpressApp -->|expose| Metrics["/metrics (prom-client)"]
|
||||
|
||||
Dashboard["Dashboard SPA (React 18 + Vite 5)\ndashboard/dist/ served from /dashboard"]
|
||||
Client -->|browser| Dashboard
|
||||
Dashboard -->|REST API calls| ExpressApp
|
||||
|
||||
Grafana["Grafana (port 3001)"] -->|scrapes| Metrics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. HTTP Request Lifecycle
|
||||
|
||||
Every authenticated API request travels through the following sequence. Understanding
|
||||
this sequence end-to-end is essential for debugging and for writing new endpoints
|
||||
correctly.
|
||||
|
||||
1. HTTP request arrives at the Node.js HTTP listener — configured in `src/server.ts`, which calls `app.listen(PORT)` after `createApp()` resolves.
|
||||
2. App-level middleware runs in registration order: `helmet()` sets security headers, `cors()` applies CORS policy from `CORS_ORIGIN`, `morgan('combined')` logs the request line (skipped in `NODE_ENV=test`), `express.json()` and `express.urlencoded()` parse the body, `metricsMiddleware` (`src/middleware/metrics.ts`) starts the request timer and records `agentidp_http_requests_total` and `agentidp_http_request_duration_seconds` on response finish.
|
||||
3. The Express router matches the path to a route definition in `src/routes/*.ts` and hands off to the appropriate middleware chain.
|
||||
4. `authMiddleware` (`src/middleware/auth.ts`) validates the Bearer JWT: extracts the token from the `Authorization` header, calls `verifyToken()` for RS256 signature and expiry, then calls `redis.get('revoked:{jti}')` to check the revocation list. On success, attaches the decoded `ITokenPayload` to `req.user`.
|
||||
5. `opaMiddleware` (`src/middleware/opa.ts`) evaluates the OPA policy: builds an `OpaInput` object from `req.method`, `req.baseUrl + req.path`, and `req.user.scope.split(' ')`, then calls `evaluate(input)`. Uses the Wasm bundle (`policies/authz.wasm`) when present, or the TypeScript fallback reading `policies/data/scopes.json`. Calls `next(new AuthorizationError())` if the policy denies.
|
||||
6. The controller (`src/controllers/*.ts`) receives the validated request, extracts and validates path params and body using Joi schemas, then delegates to the service layer.
|
||||
7. The service (`src/services/*.ts`) executes all business logic — enforces free-tier limits, resolves domain rules, and calls repositories. The service has no knowledge of HTTP.
|
||||
8. The repository (`src/repositories/*.ts`) executes parameterized SQL against PostgreSQL via `node-postgres`, or issues Redis commands via the `redis` client. No business logic lives here.
|
||||
9. The controller serialises the service result and calls `res.status(xxx).json(payload)`.
|
||||
10. `AuditService.logEvent()` is called — for high-throughput paths (token issuance, introspection, revocation) this is fire-and-forget (`void` — not awaited); for CRUD operations it is awaited. The audit event is written as an immutable row to the `audit_events` table in PostgreSQL.
|
||||
|
||||
---
|
||||
|
||||
## 3. OAuth 2.0 Client Credentials Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor Agent
|
||||
participant AgentIdP
|
||||
participant PostgreSQL
|
||||
participant Redis
|
||||
participant Vault as Vault (optional)
|
||||
|
||||
Agent->>AgentIdP: POST /api/v1/token<br/>grant_type=client_credentials<br/>client_id=<agentId><br/>client_secret=sk_live_...&<br/>scope=agents:read agents:write
|
||||
|
||||
AgentIdP->>PostgreSQL: SELECT * FROM agents WHERE agent_id = $1
|
||||
PostgreSQL-->>AgentIdP: agent row (status, etc.)
|
||||
|
||||
AgentIdP->>PostgreSQL: SELECT * FROM credentials WHERE agent_id = $1 AND status = 'active'
|
||||
PostgreSQL-->>AgentIdP: active credential rows
|
||||
|
||||
alt Vault path (vaultPath IS NOT NULL and VAULT_ADDR is set)
|
||||
AgentIdP->>Vault: readSecret(agentId, credentialId)
|
||||
Vault-->>AgentIdP: plain-text secret
|
||||
AgentIdP->>AgentIdP: crypto.timingSafeEqual(stored, candidate)
|
||||
else bcrypt path (fallback)
|
||||
AgentIdP->>AgentIdP: bcrypt.compare(clientSecret, secretHash)
|
||||
end
|
||||
|
||||
AgentIdP->>Redis: GET monthly:tokens:{agentId}:{yyyy-mm}
|
||||
Redis-->>AgentIdP: current monthly count
|
||||
|
||||
AgentIdP->>AgentIdP: signToken(payload, privateKey) — RS256 JWT
|
||||
|
||||
AgentIdP->>Redis: INCR monthly:tokens:{agentId}:{yyyy-mm} (fire-and-forget)
|
||||
|
||||
AgentIdP-->>Agent: 200 OK<br/>{ access_token, token_type: "Bearer", expires_in: 3600, scope }
|
||||
|
||||
Note over Agent,AgentIdP: Subsequent protected API call
|
||||
|
||||
Agent->>AgentIdP: GET /api/v1/agents<br/>Authorization: Bearer <access_token>
|
||||
AgentIdP->>AgentIdP: verifyToken(token, publicKey) — RS256 verify + expiry
|
||||
AgentIdP->>Redis: GET revoked:{jti}
|
||||
Redis-->>AgentIdP: null (not revoked)
|
||||
AgentIdP->>AgentIdP: OPA evaluate({method, path, scopes})
|
||||
AgentIdP-->>Agent: 200 OK — agents list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Multi-Region Deployment Topology
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
TFRoot["Terraform Root Module\nterraform/"]
|
||||
TFRoot --> AWSMod["AWS Module\nterraform/environments/aws/"]
|
||||
TFRoot --> GCPMod["GCP Module\nterraform/environments/gcp/"]
|
||||
|
||||
subgraph AWS["AWS (us-east-1 default)"]
|
||||
AWSVPC["VPC"] --> ECSCluster["ECS Cluster (Fargate)"]
|
||||
ECSCluster --> ECSTask["ECS Task — AgentIdP container"]
|
||||
ECSTask --> RDS["RDS PostgreSQL 14 (Multi-AZ)"]
|
||||
ECSTask --> Elasticache["ElastiCache Redis 7"]
|
||||
ALB["Application Load Balancer"] --> ECSCluster
|
||||
end
|
||||
|
||||
subgraph GCP["GCP (us-central1 default)"]
|
||||
GCPVPC["VPC"] --> CloudRun["Cloud Run service — AgentIdP"]
|
||||
CloudRun --> CloudSQL["Cloud SQL PostgreSQL 14"]
|
||||
CloudRun --> Memorystore["Memorystore Redis 7"]
|
||||
GCPLB["Cloud Load Balancer"] --> CloudRun
|
||||
end
|
||||
|
||||
AWSMod --> AWS
|
||||
GCPMod --> GCP
|
||||
|
||||
ECR["ECR / Artifact Registry\n(container image)"] --> ECSTask
|
||||
ECR --> CloudRun
|
||||
```
|
||||
|
||||
Each region is an independent deployment with its own PostgreSQL and Redis instances.
|
||||
The Terraform root module sets `aws_region` (default `us-east-1`) and `gcp_region`
|
||||
(default `us-central1`) as input variables. Infrastructure modules live under
|
||||
`terraform/modules/` (agentidp, lb, rds, redis) with environment-specific configuration
|
||||
under `terraform/environments/aws/` and `terraform/environments/gcp/`. Cross-region
|
||||
data replication and federation are Phase 3 goals.
|
||||
255
docs/engineering/03-tech-stack.md
Normal file
255
docs/engineering/03-tech-stack.md
Normal file
@@ -0,0 +1,255 @@
|
||||
# Technology Stack and Architecture Decision Records
|
||||
|
||||
Every technology choice in AgentIdP was made deliberately. This document records the
|
||||
decision, rationale, and alternatives considered for each major technology. New engineers
|
||||
should read this before making any technology additions or changes — the pattern here is
|
||||
the template for future ADRs.
|
||||
|
||||
---
|
||||
|
||||
### ADR-1: Node.js 18 LTS
|
||||
|
||||
**Status**: Adopted
|
||||
**Component**: AgentIdP server runtime and Node.js SDK runtime
|
||||
|
||||
**Decision**: Use Node.js 18 LTS as the server runtime.
|
||||
|
||||
**Rationale**: Node.js 18 LTS provides native `fetch`, native ESM support, and a
|
||||
stable V8 engine with long-term security updates. The ecosystem for Express, PostgreSQL
|
||||
(`pg`), Redis (`redis`), JWT (`jsonwebtoken`), and bcrypt (`bcryptjs`) is mature and
|
||||
well-maintained on this version. The non-blocking I/O model is well-suited for an IdP
|
||||
that handles many concurrent short-lived authentication requests. The `engines.node`
|
||||
field in `package.json` enforces `>=18.0.0`.
|
||||
|
||||
**Alternatives considered**:
|
||||
- Deno — rejected because the npm ecosystem compatibility layer introduced friction with key dependencies (`pg`, `bcryptjs`), and the production deployment story on ECS and Cloud Run was less mature at the time of the decision.
|
||||
- Bun — rejected because it lacked LTS stability guarantees at the time of the decision, which is not acceptable for a security-critical authentication service.
|
||||
|
||||
**Consequences**: All Dockerfiles and Terraform ECS/Cloud Run task definitions must
|
||||
target Node.js 18 or a compatible LTS release. Upgrading the Node.js version requires
|
||||
CTO approval and a QA sign-off on the full test suite.
|
||||
|
||||
---
|
||||
|
||||
### ADR-2: TypeScript 5.3 Strict Mode
|
||||
|
||||
**Status**: Adopted
|
||||
**Component**: All source files — server, all SDKs, dashboard
|
||||
|
||||
**Decision**: TypeScript 5.3 with `strict: true` and every additional strictness flag enabled in `tsconfig.json`.
|
||||
|
||||
**Rationale**: AgentIdP handles authentication tokens and cryptographic secrets. Type
|
||||
errors in this domain can cause security vulnerabilities — a value that should be
|
||||
`string | null` treated as `string` can produce silent authentication bypasses. Strict
|
||||
TypeScript with `noImplicitAny`, `strictNullChecks`, `noUnusedLocals`, `noUnusedParameters`,
|
||||
and `noImplicitReturns` makes these classes of bug a compile-time error rather than a
|
||||
runtime failure in production.
|
||||
|
||||
**Alternatives considered**:
|
||||
- Plain JavaScript — rejected because a security-critical IdP with no type safety is not a system this team is willing to ship. Every public method, every error boundary, and every data transformation must be typed.
|
||||
|
||||
**Consequences**: All new code must compile cleanly under `tsc --strict`. Zero `any`
|
||||
types — ever. No exceptions granted without CTO approval. The `tsconfig.json` enables
|
||||
`noImplicitAny`, `strictNullChecks`, `strictFunctionTypes`, `strictBindCallApply`,
|
||||
`strictPropertyInitialization`, `noImplicitThis`, `alwaysStrict`, `noUnusedLocals`,
|
||||
`noUnusedParameters`, `noImplicitReturns`, and `noFallthroughCasesInSwitch`.
|
||||
|
||||
---
|
||||
|
||||
### ADR-3: Express 4.18
|
||||
|
||||
**Status**: Adopted
|
||||
**Component**: HTTP server framework
|
||||
|
||||
**Decision**: Use Express 4.18 as the HTTP framework.
|
||||
|
||||
**Rationale**: Express is the most widely understood Node.js HTTP framework. Its
|
||||
middleware model (`(req, res, next)`) maps directly to the IdP's layered architecture:
|
||||
`helmet` → `cors` → `metricsMiddleware` → `authMiddleware` → `opaMiddleware` →
|
||||
controller → service → repository → `errorHandler`. The ecosystem for Express
|
||||
middleware (`helmet`, `cors`, `morgan`) is mature. For a spec-first project, Express's
|
||||
lack of convention about code structure is a feature — the architecture is explicit and
|
||||
fully visible in `src/app.ts`.
|
||||
|
||||
**Alternatives considered**:
|
||||
- Fastify — rejected because the team's familiarity was lower and the performance gains would be negligible for a token service whose latency is dominated by PostgreSQL queries and bcrypt comparisons.
|
||||
- NestJS — rejected because its decorator-heavy convention-over-configuration style adds complexity not appropriate for the current team size and project scope.
|
||||
- Koa — rejected because its ecosystem is smaller and fewer engineers are familiar with it.
|
||||
|
||||
**Consequences**: All HTTP concerns (routing, middleware, error handling) use the
|
||||
Express 4 API. The `errorHandler` middleware must remain the last `app.use()` call in
|
||||
`src/app.ts`.
|
||||
|
||||
---
|
||||
|
||||
### ADR-4: PostgreSQL 14
|
||||
|
||||
**Status**: Adopted
|
||||
**Component**: Primary data store for agents, credentials, and audit events
|
||||
|
||||
**Decision**: Use PostgreSQL 14 as the primary relational database.
|
||||
|
||||
**Rationale**: The audit log requires ACID guarantees — partial writes or uncommitted
|
||||
reads are not acceptable for a compliance-grade append-only event store. PostgreSQL's
|
||||
`JSONB` column type is used for the `metadata` field in `audit_events`, allowing
|
||||
structured context data without schema changes for each new event type. PostgreSQL's
|
||||
row-level security is available for multi-tenancy if that becomes a Phase 3 requirement.
|
||||
|
||||
**Alternatives considered**:
|
||||
- MySQL — rejected because its JSON support is weaker than PostgreSQL's `JSONB` with GIN indexing, and its default transaction isolation level has historically produced surprises.
|
||||
- MongoDB — rejected because the audit log must be append-only and ACID-safe. MongoDB's document model requires explicit multi-document transactions for ACID behaviour, and the schema flexibility is not needed here.
|
||||
|
||||
**Consequences**: All schema changes go through numbered SQL migration files in
|
||||
`src/db/migrations/`. Migration files are append-only — never modify an existing
|
||||
migration. New tables require a new numbered file (e.g. `005_create_agent_groups.sql`).
|
||||
|
||||
---
|
||||
|
||||
### ADR-5: Redis 7
|
||||
|
||||
**Status**: Adopted
|
||||
**Component**: Token revocation list, monthly usage counters, rate-limit sliding window
|
||||
|
||||
**Decision**: Use Redis 7 as the in-memory data store.
|
||||
|
||||
**Rationale**: Token revocation requires O(1) key lookup with TTL-based automatic
|
||||
expiry. `SET revoked:{jti} 1 EX {seconds_until_expiry}` stores a revocation entry
|
||||
that expires precisely when the token itself would have expired — zero manual cleanup
|
||||
required. The monthly token counter uses Redis `INCR`, which is atomic and O(1). The
|
||||
rate-limiter uses a Redis sorted set for the sliding-window algorithm.
|
||||
|
||||
**Alternatives considered**:
|
||||
- Memcached — rejected because Memcached does not support per-key TTL on sorted-set structures, which is required for the sliding-window rate-limiter.
|
||||
- PostgreSQL for revocation — rejected because the token verification path is the hot path in every authenticated request. A PostgreSQL round-trip adds 5–15 ms compared to a Redis `GET` at sub-millisecond latency.
|
||||
|
||||
**Consequences**: Redis is a required infrastructure dependency. A Redis instance must
|
||||
be running and reachable via `REDIS_URL` before the server starts. `docker-compose.yml`
|
||||
provides a Redis 7 Alpine container for local development on port 6379.
|
||||
|
||||
---
|
||||
|
||||
### ADR-6: HashiCorp Vault
|
||||
|
||||
**Status**: Adopted (opt-in)
|
||||
**Component**: Credential secret storage — alternative to bcrypt in PostgreSQL
|
||||
|
||||
**Decision**: Integrate HashiCorp Vault KV v2 as an opt-in secret storage backend for agent credentials.
|
||||
|
||||
**Rationale**: The Phase 1 bcrypt approach stores hashes in PostgreSQL. While bcrypt
|
||||
hashes cannot be reversed, some enterprises require that secrets never touch a relational
|
||||
database — even in hashed form. Vault provides a dedicated secrets management plane with
|
||||
HSM backing and an independent audit trail at the secrets level. The `verifySecret`
|
||||
method in `VaultClient` uses `crypto.timingSafeEqual` to prevent timing-based
|
||||
side-channel attacks when comparing stored and candidate secrets.
|
||||
|
||||
**Alternatives considered**:
|
||||
- AWS Secrets Manager — rejected because it introduces cloud-vendor lock-in. AgentIdP must run identically on AWS, GCP, and on-premises; a Vault-based approach works in all environments.
|
||||
- Plain bcrypt only — retained as the fallback path. When `VAULT_ADDR` is not set, `createVaultClientFromEnv()` returns `null` and the server operates identically to Phase 1.
|
||||
|
||||
**Consequences**: Vault is controlled by `VAULT_ADDR` (required), `VAULT_TOKEN`
|
||||
(required), and `VAULT_MOUNT` (optional, defaults to `secret`). When these are not set,
|
||||
bcrypt is used unchanged. Credential rows carry a nullable `vault_path` column: `null`
|
||||
means bcrypt; a non-null path means Vault verification is used.
|
||||
|
||||
---
|
||||
|
||||
### ADR-7: OPA (Open Policy Agent)
|
||||
|
||||
**Status**: Adopted
|
||||
**Component**: Request authorisation — scope enforcement on all protected endpoints
|
||||
|
||||
**Decision**: Use Open Policy Agent with a Rego policy compiled to a Wasm bundle for runtime authorisation.
|
||||
|
||||
**Rationale**: Hard-coded scope checks in middleware would require a code deployment
|
||||
for every policy change. OPA decouples the policy (`policies/authz.rego`) from the
|
||||
server code. The policy can be updated, re-compiled to Wasm, and hot-reloaded via
|
||||
`SIGHUP` without restarting the server. The `@open-policy-agent/opa-wasm` package
|
||||
evaluates the compiled Wasm bundle in-process with microsecond latency. When no Wasm
|
||||
bundle is present (development, CI), the middleware falls back to a TypeScript
|
||||
implementation that reads `policies/data/scopes.json`.
|
||||
|
||||
**Alternatives considered**:
|
||||
- Custom middleware with hard-coded scope checks — rejected because policy changes require code changes and a full deployment cycle. As the endpoint surface grows this becomes unmanageable.
|
||||
- Casbin — rejected because its RBAC/ABAC model is less expressive than Rego for the compound `method + path + scope-intersection` pattern AgentIdP requires.
|
||||
|
||||
**Consequences**: All authorisation rules live in `policies/authz.rego` and
|
||||
`policies/data/scopes.json`. Adding a new endpoint requires adding its scope
|
||||
requirement to `scopes.json`. A policy change is deployed by updating `scopes.json`
|
||||
(or `authz.wasm`) and sending `SIGHUP` to the running process — no redeployment needed.
|
||||
|
||||
---
|
||||
|
||||
### ADR-8: React 18 + Vite 5
|
||||
|
||||
**Status**: Adopted
|
||||
**Component**: Web dashboard SPA (`dashboard/`)
|
||||
|
||||
**Decision**: Use React 18 with Vite 5 as the web dashboard framework and build tool.
|
||||
|
||||
**Rationale**: React 18's concurrent rendering model handles the dashboard's async data
|
||||
fetching patterns cleanly. The `@sentryagent/idp-sdk` Node.js package is reused
|
||||
directly in the dashboard via `TokenManager` for authentication, avoiding duplicated
|
||||
API client code. Vite 5 provides sub-second HMR in development and a fast production
|
||||
build with tree-shaking. The dashboard is built to `dashboard/dist/` and served as
|
||||
static files from Express at `/dashboard`, keeping the deployment footprint to a
|
||||
single container.
|
||||
|
||||
**Alternatives considered**:
|
||||
- Next.js — rejected because server-side rendering is not needed for an internal operator dashboard, and the added complexity of a Next.js server is not justified.
|
||||
- Vue — rejected because the broader SentryAgent.ai ecosystem is React-first; consistency reduces context-switching overhead.
|
||||
|
||||
**Consequences**: The dashboard must be built (`cd dashboard && npm run build`) before
|
||||
Express can serve it. In local development, run `cd dashboard && npm run dev` to use
|
||||
Vite's dev server with HMR; the Vite proxy forwards `/api/` calls to Express at
|
||||
`localhost:3000`.
|
||||
|
||||
---
|
||||
|
||||
### ADR-9: Prometheus + Grafana
|
||||
|
||||
**Status**: Adopted
|
||||
**Component**: Operational metrics collection and visualisation
|
||||
|
||||
**Decision**: Use Prometheus for metrics collection and Grafana for dashboards.
|
||||
|
||||
**Rationale**: Prometheus is the industry standard for metrics in container
|
||||
environments. The `prom-client` npm package integrates natively with Express and
|
||||
provides `Counter` and `Histogram` metric types that cover all observability needs for
|
||||
AgentIdP. Grafana's YAML provisioning in `monitoring/grafana/provisioning/` makes
|
||||
dashboards reproducible and version-controlled. The monitoring stack runs as a Docker
|
||||
Compose overlay (`docker-compose.monitoring.yml`) without interfering with the base dev
|
||||
environment.
|
||||
|
||||
**Alternatives considered**:
|
||||
- Datadog — rejected because SaaS cost and vendor lock-in are not acceptable for a free, open-source product. Operators who self-host AgentIdP should not be required to pay for monitoring.
|
||||
- StatsD — rejected because StatsD's flat metric model lacks label/dimension support, which is essential for distinguishing metrics by `method`, `route`, and `status_code`.
|
||||
|
||||
**Consequences**: All metric definitions live exclusively in `src/metrics/registry.ts`.
|
||||
No other file may instantiate a `Counter` or `Histogram` — all other files import
|
||||
specific metrics from that registry. Grafana is available at port 3001 when the
|
||||
monitoring overlay is running.
|
||||
|
||||
---
|
||||
|
||||
### ADR-10: Terraform
|
||||
|
||||
**Status**: Adopted
|
||||
**Component**: Infrastructure as code — multi-region AWS + GCP deployment
|
||||
|
||||
**Decision**: Use Terraform with HCL for all infrastructure provisioning across AWS and GCP.
|
||||
|
||||
**Rationale**: Terraform's HCL syntax is readable and its provider ecosystem covers
|
||||
both AWS and GCP with the same toolchain. Reusable modules in `terraform/modules/`
|
||||
(agentidp, lb, rds, redis) are composed in environment-specific configurations under
|
||||
`terraform/environments/aws/` and `terraform/environments/gcp/`. All infrastructure
|
||||
changes go through `terraform plan` review before `terraform apply`, providing a
|
||||
diff-based approval workflow.
|
||||
|
||||
**Alternatives considered**:
|
||||
- Pulumi — rejected because the Pulumi provider ecosystem for AWS and GCP was less mature than Terraform's at the time of the Phase 2 decision, and HCL is more readable for non-engineers reviewing infrastructure changes.
|
||||
- AWS CDK — rejected because it is AWS-only. AgentIdP must deploy identically to both AWS and GCP.
|
||||
|
||||
**Consequences**: All infrastructure changes must go through Terraform. No manual edits
|
||||
via the AWS console or GCP console are permitted — they will be overwritten on the next
|
||||
`terraform apply`. Terraform state is stored in a remote backend and must not be edited
|
||||
manually.
|
||||
146
docs/engineering/04-codebase-structure.md
Normal file
146
docs/engineering/04-codebase-structure.md
Normal file
@@ -0,0 +1,146 @@
|
||||
# Codebase Structure
|
||||
|
||||
---
|
||||
|
||||
## 1. Annotated Directory Tree
|
||||
|
||||
```
|
||||
sentryagent-idp/
|
||||
├── src/ # Express application source — controllers, services, middleware, repositories, routes
|
||||
│ ├── app.ts # Express app factory — creates and configures the app; does NOT call listen
|
||||
│ ├── server.ts # Entry point — calls listen, handles SIGTERM/SIGINT/SIGHUP
|
||||
│ ├── types/ # Canonical TypeScript interfaces and type definitions
|
||||
│ ├── controllers/ # HTTP layer — extract/validate inputs, call services, build responses
|
||||
│ ├── services/ # Business logic — pure domain operations, no HTTP knowledge
|
||||
│ ├── repositories/ # Database and Redis access — parameterized SQL, no logic
|
||||
│ ├── middleware/ # Cross-cutting request concerns — auth, OPA, rate-limit, metrics, error handling
|
||||
│ ├── routes/ # Express router definitions — wiring only, no logic
|
||||
│ ├── utils/ # Shared pure utilities — errors, validators, crypto, JWT helpers
|
||||
│ ├── vault/ # HashiCorp Vault KV v2 client
|
||||
│ ├── metrics/ # Prometheus metrics registry — all Counter and Histogram definitions
|
||||
│ ├── db/ # PostgreSQL pool factory and SQL migration files
|
||||
│ └── cache/ # Redis client factory
|
||||
├── tests/ # Jest test suite — mirrors src/ structure (unit/ and integration/)
|
||||
├── dashboard/ # React 18 + Vite 5 web dashboard SPA
|
||||
│ ├── src/ # Dashboard source — pages, components, auth, API client
|
||||
│ └── dist/ # Built dashboard — served by Express at /dashboard (git-ignored)
|
||||
├── sdk/ # Node.js SDK (@sentryagent/idp-sdk) — TypeScript, auto token refresh
|
||||
├── sdk-python/ # Python SDK (sentryagent-idp) — sync + async clients
|
||||
├── sdk-go/ # Go SDK (github.com/sentryagent/idp-sdk-go) — context-aware, goroutine-safe
|
||||
├── sdk-java/ # Java SDK (ai.sentryagent:idp-sdk) — builder pattern, CompletableFuture
|
||||
├── policies/ # OPA policy files
|
||||
│ ├── authz.rego # Rego policy — normalise_path + scope-intersection allow rule
|
||||
│ └── data/scopes.json # Endpoint permission map — used by Rego and TypeScript fallback
|
||||
├── terraform/ # Terraform infrastructure as code
|
||||
│ ├── modules/ # Reusable modules: agentidp, lb, rds, redis
|
||||
│ └── environments/ # Environment configs: aws/ (ECS+RDS+ElastiCache), gcp/ (Cloud Run+SQL+Memorystore)
|
||||
├── monitoring/ # Prometheus and Grafana configuration
|
||||
│ ├── prometheus/ # prometheus.yml scrape configuration
|
||||
│ └── grafana/ # Grafana provisioning YAML and dashboard JSON files
|
||||
├── docs/ # All project documentation
|
||||
│ ├── engineering/ # Internal engineering knowledge base (this directory)
|
||||
│ ├── developers/ # End-user API reference and developer guides
|
||||
│ ├── devops/ # Operator runbooks and environment variable reference
|
||||
│ ├── agntcy/ # AGNTCY alignment documentation
|
||||
│ └── openapi/ # OpenAPI 3.0 specification files
|
||||
├── openspec/ # OpenSpec change management — proposals, designs, specs, tasks, archives
|
||||
├── Dockerfile # Multi-stage production build (build + runtime stages)
|
||||
├── docker-compose.yml # Local development: PostgreSQL 14 (port 5432) + Redis 7 (port 6379)
|
||||
├── docker-compose.monitoring.yml # Monitoring overlay: Prometheus (port 9090) + Grafana (port 3001)
|
||||
├── package.json # Node.js dependencies and npm scripts
|
||||
├── tsconfig.json # TypeScript strict configuration — compiled to dist/
|
||||
└── jest.config.ts # Jest configuration — ts-jest, test timeouts, coverage thresholds
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. src/ Subdirectory Roles
|
||||
|
||||
| Directory | Role | Rule |
|
||||
|-----------|------|------|
|
||||
| `src/controllers/` | Receive HTTP requests, extract and validate inputs using Joi, call service methods, serialise responses | No business logic — controllers are thin wrappers that translate HTTP into service calls |
|
||||
| `src/services/` | All business logic — free-tier limit enforcement, domain rule evaluation, orchestration of repository calls and audit events | Never import from controllers or routes; never know about `req` or `res` |
|
||||
| `src/repositories/` | All database and Redis queries — parameterized SQL via `node-postgres`, Redis commands via `redis` client | Only called from services; never called directly from controllers; no business logic |
|
||||
| `src/middleware/` | Cross-cutting request concerns — `authMiddleware`, `opaMiddleware`, `rateLimitMiddleware`, `metricsMiddleware`, `errorHandler` | Applied at router or app level in `src/app.ts`; never import from controllers |
|
||||
| `src/routes/` | Map HTTP paths and methods to middleware chains and controller methods | Wiring only — no logic, no validation, no business rules |
|
||||
| `src/utils/` | Shared pure utilities — `errors.ts`, `validators.ts`, `crypto.ts`, `jwt.ts`, `asyncHandler.ts` | No side effects; no imports from services or controllers |
|
||||
| `src/types/` | All TypeScript type definitions, interfaces, and enums — the single source of truth for all shared types | Imported everywhere; never imports from anywhere else in `src/` |
|
||||
| `src/vault/` | `VaultClient` — wraps HashiCorp Vault KV v2 operations; constant-time secret verification | Only instantiated by `createVaultClientFromEnv()` in `src/app.ts`; passed to services via constructor injection |
|
||||
| `src/metrics/` | Prometheus metrics registry — all `Counter` and `Histogram` definitions in one place | Only file that calls `new Counter()` or `new Histogram()`; all other files import from here |
|
||||
| `src/db/` | PostgreSQL connection pool factory (`pool.ts`) and numbered SQL migration files in `migrations/` | Pool is a singleton created once in `src/app.ts` and passed to repositories |
|
||||
| `src/cache/` | Redis client factory — creates and caches a single `redis` client instance | Client is a singleton created once in `src/app.ts` and passed to repositories |
|
||||
|
||||
---
|
||||
|
||||
## 3. Where to Add New Code
|
||||
|
||||
| I need to add... | Where it goes | Example |
|
||||
|-----------------|---------------|---------|
|
||||
| A new API endpoint | `src/routes/` (wire it), `src/controllers/` (HTTP layer), `src/services/` (business logic), `src/repositories/` (data access) | Adding `DELETE /api/v1/agents/:id/credentials/:credId/bulk` |
|
||||
| A new business rule | `src/services/[relevant]Service.ts` | Enforcing a maximum of 5 credentials per agent |
|
||||
| A new database table | `src/db/migrations/` — new numbered SQL file (append-only) | Adding an `agent_groups` table as `005_create_agent_groups.sql` |
|
||||
| A new authorisation policy rule | `policies/authz.rego` + `policies/data/scopes.json` | Adding a new scope `reports:read` for a `GET /api/v1/reports` endpoint |
|
||||
| A new shared error type | `src/utils/errors.ts` | `VaultUnavailableError` extending `SentryAgentError` |
|
||||
| A new environment variable | `src/utils/config.ts` (if it exists) or the relevant consumer file + `docs/devops/environment-variables.md` | `RATE_LIMIT_MAX` controlling the rate-limit ceiling |
|
||||
| A new Prometheus metric | `src/metrics/registry.ts` | A `Histogram` for Vault lookup duration |
|
||||
| A new TypeScript type used in 2+ files | `src/types/index.ts` | A new `AgentGroupMembership` interface |
|
||||
|
||||
---
|
||||
|
||||
## 4. Key Files
|
||||
|
||||
**`src/app.ts`**
|
||||
Creates and configures the Express application. Registers all middleware (helmet, cors,
|
||||
morgan, body parsers, metricsMiddleware), instantiates all infrastructure singletons
|
||||
(PostgreSQL pool, Redis client, VaultClient), constructs the full dependency graph
|
||||
(repositories → services → controllers), and mounts all routers. Returns the configured
|
||||
`Application` without calling `listen`. Tests import `createApp()` directly — this is
|
||||
the design decision that makes integration tests possible without binding a port.
|
||||
|
||||
**`src/server.ts`**
|
||||
The only file that calls `app.listen()`. Loads environment variables via `dotenv.config()`,
|
||||
calls `createApp()`, binds the port from `PORT` env var (default 3000), and registers
|
||||
`SIGTERM`, `SIGINT` (graceful shutdown), and `SIGHUP` (OPA policy hot-reload via
|
||||
`reloadOpaPolicy()`) signal handlers. This file is never imported by tests.
|
||||
|
||||
**`src/types/index.ts`**
|
||||
The canonical type definition file for the entire project. Contains all exported
|
||||
interfaces (`IAgent`, `ICredential`, `ITokenPayload`, `IAuditEvent`, etc.), union types
|
||||
(`AgentStatus`, `AgentType`, `AuditAction`, etc.), and the global Express `Request`
|
||||
augmentation that adds `req.user?: ITokenPayload`. If a type is needed in two or more
|
||||
files, it lives here — never redefined inline.
|
||||
|
||||
**`src/utils/errors.ts`**
|
||||
The `SentryAgentError` base class and all typed error subclasses. Every error thrown
|
||||
in the application must extend `SentryAgentError` — never `throw new Error('string')`.
|
||||
The `errorHandler` middleware in `src/middleware/errorHandler.ts` maps
|
||||
`SentryAgentError` subclasses to their `httpStatus` codes and serialises the response
|
||||
as `IErrorResponse { code, message, details }`.
|
||||
|
||||
**`docker-compose.yml`**
|
||||
Starts PostgreSQL 14 (Alpine) on port 5432 with database `sentryagent_idp` and
|
||||
Redis 7 (Alpine) on port 6379. Used for local development only. Both services have
|
||||
health checks so `depends_on` conditions work correctly. The `app` service mounts
|
||||
`./src` as a read-only volume for live code reloading.
|
||||
|
||||
**`tsconfig.json`**
|
||||
TypeScript compiler configuration. `strict: true` enables the full suite of strictness
|
||||
checks. `target: ES2022`, `module: commonjs` (the project compiles to CommonJS for
|
||||
Node.js compatibility). `outDir: ./dist`, `rootDir: ./src`. The `noUnusedLocals` and
|
||||
`noUnusedParameters` flags are enabled — unused code is a compile error. Never disable
|
||||
these flags.
|
||||
|
||||
---
|
||||
|
||||
## 5. DRY Enforcement
|
||||
|
||||
Every piece of logic lives in exactly one place. Violations are CTO-blocking.
|
||||
|
||||
| Concern | Single source of truth | Violation pattern to reject |
|
||||
|---------|----------------------|----------------------------|
|
||||
| Business logic | One service method — called from multiple controllers if needed | Business logic duplicated in a route handler or controller |
|
||||
| Database queries | One repository method — never repeated inline | SQL written directly in a service or controller |
|
||||
| Error types | `src/utils/errors.ts` — imported wherever errors are thrown | `new Error('AGENT_NOT_FOUND')` instead of `new AgentNotFoundError()` |
|
||||
| TypeScript types | `src/types/index.ts` — imported in every consumer file | An interface defined inline in a service file |
|
||||
| Validation logic | `src/utils/validators.ts` — Joi schemas used in controllers | Validation logic duplicated across multiple controllers |
|
||||
| Prometheus metrics | `src/metrics/registry.ts` — one definition per metric | A second `new Counter({ name: 'agentidp_tokens_issued_total' })` anywhere |
|
||||
342
docs/engineering/05-services.md
Normal file
342
docs/engineering/05-services.md
Normal file
@@ -0,0 +1,342 @@
|
||||
# Service Deep Dives
|
||||
|
||||
---
|
||||
|
||||
### AgentService
|
||||
|
||||
**Purpose**: Manages the full lifecycle of AI agent identities — registration, retrieval, updates, and decommissioning.
|
||||
|
||||
**Responsibility boundary**: AgentService does not handle HTTP, credential secrets,
|
||||
token issuance, or audit log queries. It delegates all data access to
|
||||
`AgentRepository` and `CredentialRepository`, and all audit logging to `AuditService`.
|
||||
It enforces free-tier limits and domain rules before any data is written.
|
||||
|
||||
**Public interface** (key methods):
|
||||
|
||||
| Method | Parameters | Returns | Description |
|
||||
|--------|-----------|---------|-------------|
|
||||
| `registerAgent` | `data: ICreateAgentRequest, ipAddress: string, userAgent: string` | `Promise<IAgent>` | Checks the free-tier 100-agent limit, enforces email uniqueness, creates the agent record, writes an `agent.created` audit event, increments `agentidp_agents_registered_total` Prometheus counter |
|
||||
| `getAgentById` | `agentId: string` | `Promise<IAgent>` | Retrieves a single agent by UUID; throws `AgentNotFoundError` if not found |
|
||||
| `listAgents` | `filters: IAgentListFilters` | `Promise<IPaginatedAgentsResponse>` | Returns a paginated, optionally filtered list; filters include `owner`, `agentType`, `status`, `page`, `limit` |
|
||||
| `updateAgent` | `agentId: string, data: IUpdateAgentRequest, ipAddress: string, userAgent: string` | `Promise<IAgent>` | Partially updates agent metadata; rejects updates to decommissioned agents; determines the correct audit action (`agent.updated`, `agent.suspended`, `agent.reactivated`, `agent.decommissioned`) based on status transition |
|
||||
| `decommissionAgent` | `agentId: string, ipAddress: string, userAgent: string` | `Promise<void>` | Soft-deletes the agent (sets `status = 'decommissioned'`); revokes all active credentials by calling `credentialRepository.revokeAllForAgent(agentId)` before decommissioning |
|
||||
|
||||
**Database / storage schema**:
|
||||
- Table `agents`: `agent_id` (UUID PK), `email` (UNIQUE), `agent_type`, `version`, `capabilities` (text array), `owner`, `deployment_env`, `status`, `created_at`, `updated_at`.
|
||||
- No Redis usage — AgentService is PostgreSQL-only.
|
||||
|
||||
**Error types**:
|
||||
- `FreeTierLimitError` (403) — 100-agent limit reached
|
||||
- `AgentAlreadyExistsError` (409) — email already registered
|
||||
- `AgentNotFoundError` (404) — agent UUID not found
|
||||
- `AgentAlreadyDecommissionedError` (409) — agent is already decommissioned
|
||||
|
||||
**Configuration**: None — AgentService reads no environment variables. The free-tier limit (`FREE_TIER_MAX_AGENTS = 100`) is a module-level constant.
|
||||
|
||||
---
|
||||
|
||||
### OAuth2Service
|
||||
|
||||
**Purpose**: Issues, introspects, and revokes RS256 JWT access tokens via the OAuth 2.0 Client Credentials grant.
|
||||
|
||||
**Responsibility boundary**: OAuth2Service does not know about HTTP or routing. It
|
||||
receives already-extracted values (`clientId`, `clientSecret`, `scope`) from the
|
||||
controller, resolves credential verification (Vault or bcrypt), enforces the 10,000
|
||||
tokens/month free-tier limit, and returns a typed `ITokenResponse`. All audit writes
|
||||
on high-throughput paths (issue, introspect, revoke) are fire-and-forget (`void`) to
|
||||
keep token endpoint latency low.
|
||||
|
||||
**Public interface** (key methods):
|
||||
|
||||
| Method | Parameters | Returns | Description |
|
||||
|--------|-----------|---------|-------------|
|
||||
| `issueToken` | `clientId: string, clientSecret: string, scope: string, ipAddress: string, userAgent: string` | `Promise<ITokenResponse>` | Verifies credentials (Vault or bcrypt), checks agent status, enforces 10k/month limit, signs RS256 JWT, increments monthly counter and audit event as fire-and-forget |
|
||||
| `introspectToken` | `token: string, callerPayload: ITokenPayload, ipAddress: string, userAgent: string` | `Promise<IIntrospectResponse>` | Verifies JWT signature and checks Redis revocation list; always returns 200 with `active: true/false` per RFC 7662 |
|
||||
| `revokeToken` | `token: string, callerPayload: ITokenPayload, ipAddress: string, userAgent: string` | `Promise<void>` | Decodes token without verification; enforces that caller can only revoke their own tokens (`decoded.sub === callerPayload.sub`); adds JTI to Redis revocation list with TTL matching token expiry |
|
||||
|
||||
**Database / storage schema**:
|
||||
- Redis key `revoked:{jti}` — value `1`, TTL = seconds until token expiry. Written on revocation; read on every authenticated request via `authMiddleware`.
|
||||
- Redis key `monthly:tokens:{agentId}:{yyyy-mm}` — integer counter, incremented on every successful token issuance. Read to enforce the 10k/month free-tier limit.
|
||||
|
||||
**Error types**:
|
||||
- `AuthenticationError` (401) — agent not found, or no active credential matches the provided secret
|
||||
- `AuthorizationError` (403) — agent is suspended or decommissioned; or caller attempts to revoke another agent's token
|
||||
- `FreeTierLimitError` (403) — 10,000 tokens/month limit reached
|
||||
|
||||
**Configuration**:
|
||||
- `JWT_PRIVATE_KEY` — PEM-encoded RSA private key, required, read at app startup in `src/app.ts`
|
||||
- `JWT_PUBLIC_KEY` — PEM-encoded RSA public key, required, read at app startup and in `authMiddleware`
|
||||
- `VAULT_ADDR`, `VAULT_TOKEN`, `VAULT_MOUNT` — optional; when set, Vault is used for credential verification instead of bcrypt
|
||||
|
||||
---
|
||||
|
||||
### CredentialService
|
||||
|
||||
**Purpose**: Manages the full lifecycle of agent credentials — generation, listing, rotation, and revocation.
|
||||
|
||||
**Responsibility boundary**: CredentialService does not know about HTTP or token
|
||||
issuance. It enforces that credentials can only be generated for `active` agents. It
|
||||
delegates secret storage to either `VaultClient` (Phase 2) or bcrypt (Phase 1 fallback).
|
||||
The plain-text `clientSecret` is generated here, returned once in the response, and
|
||||
never stored or logged — only the bcrypt hash or Vault path is persisted.
|
||||
|
||||
**Public interface** (key methods):
|
||||
|
||||
| Method | Parameters | Returns | Description |
|
||||
|--------|-----------|---------|-------------|
|
||||
| `generateCredential` | `agentId: string, data: IGenerateCredentialRequest, ipAddress: string, userAgent: string` | `Promise<ICredentialWithSecret>` | Verifies agent exists and is `active`; generates a cryptographically random secret via `generateClientSecret()`; writes to Vault (when configured) or hashes with bcrypt; returns `ICredentialWithSecret` — the only time the plain-text secret is returned |
|
||||
| `listCredentials` | `agentId: string, filters: ICredentialListFilters` | `Promise<IPaginatedCredentialsResponse>` | Returns paginated credentials for an agent; `clientSecret` is never included in list responses |
|
||||
| `rotateCredential` | `agentId: string, credentialId: string, data: IGenerateCredentialRequest, ipAddress: string, userAgent: string` | `Promise<ICredentialWithSecret>` | Generates a new secret for the same `credentialId`; overwrites Vault entry (new KV v2 version) or updates bcrypt hash; old secret is immediately invalidated; returns new `ICredentialWithSecret` once |
|
||||
| `revokeCredential` | `agentId: string, credentialId: string, ipAddress: string, userAgent: string` | `Promise<void>` | Sets credential `status = 'revoked'`; permanently deletes the Vault secret via `vaultClient.deleteSecret()` when Vault is configured; idempotent rejection of already-revoked credentials with `CredentialAlreadyRevokedError` |
|
||||
|
||||
**Database / storage schema**:
|
||||
- Table `credentials`: `credential_id` (UUID PK), `client_id` (= `agentId`, FK to `agents`), `secret_hash` (bcrypt hash; empty string when Vault path is set), `vault_path` (nullable — KV v2 data path), `status`, `created_at`, `expires_at` (nullable), `revoked_at` (nullable).
|
||||
|
||||
**Error types**:
|
||||
- `AgentNotFoundError` (404) — agent UUID not found
|
||||
- `CredentialError` (400) — agent is not in `active` status (code: `AGENT_NOT_ACTIVE`)
|
||||
- `CredentialNotFoundError` (404) — credential not found or belongs to a different agent
|
||||
- `CredentialAlreadyRevokedError` (409) — credential is already revoked
|
||||
|
||||
**Configuration**:
|
||||
- `VAULT_ADDR`, `VAULT_TOKEN`, `VAULT_MOUNT` — optional; when set, new credentials are stored in Vault KV v2 instead of bcrypt. Existing bcrypt-based credentials continue to work unchanged.
|
||||
|
||||
---
|
||||
|
||||
### AuditService
|
||||
|
||||
**Purpose**: Creates and queries immutable audit events for compliance and observability.
|
||||
|
||||
**Responsibility boundary**: AuditService does not know about HTTP, tokens, or agents.
|
||||
It receives already-assembled event data from other services and delegates all
|
||||
persistence to `AuditRepository`. It enforces the 90-day free-tier retention window
|
||||
on all query and retrieval operations — events older than 90 days are treated as
|
||||
non-existent.
|
||||
|
||||
**Public interface** (key methods):
|
||||
|
||||
| Method | Parameters | Returns | Description |
|
||||
|--------|-----------|---------|-------------|
|
||||
| `logEvent` | `agentId: string, action: AuditAction, outcome: AuditOutcome, ipAddress: string, userAgent: string, metadata: Record<string, unknown>` | `Promise<IAuditEvent>` | Writes an immutable audit row to PostgreSQL. For token endpoints, callers use `void` (fire-and-forget). For CRUD operations, callers `await` this method. |
|
||||
| `queryEvents` | `filters: IAuditListFilters` | `Promise<IPaginatedAuditEventsResponse>` | Returns paginated, filtered audit events; enforces the 90-day retention window by computing the cutoff date and rejecting queries with `fromDate` before the cutoff; validates that `fromDate <= toDate` |
|
||||
| `getEventById` | `eventId: string` | `Promise<IAuditEvent>` | Retrieves a single event by UUID; returns `AuditEventNotFoundError` for both genuinely missing events and events outside the 90-day retention window (indistinguishable by design) |
|
||||
|
||||
**Database / storage schema**:
|
||||
- Table `audit_events`: `event_id` (UUID PK), `agent_id` (text FK to agents), `action` (text — one of the `AuditAction` union type values), `outcome` (`success` or `failure`), `ip_address` (text), `user_agent` (text), `metadata` (JSONB), `timestamp` (timestamptz, NOT NULL, indexed).
|
||||
- No Redis usage — AuditService is PostgreSQL-only.
|
||||
|
||||
**Error types**:
|
||||
- `AuditEventNotFoundError` (404) — event not found or outside retention window
|
||||
- `RetentionWindowError` (400) — query `fromDate` is before the 90-day retention cutoff
|
||||
- `ValidationError` (400) — `fromDate` is after `toDate`
|
||||
|
||||
**Configuration**: None — the retention window (`FREE_TIER_RETENTION_DAYS = 90`) is a module-level constant.
|
||||
|
||||
---
|
||||
|
||||
### VaultClient
|
||||
|
||||
**Purpose**: Wraps HashiCorp Vault KV v2 operations for credential secret storage and verification.
|
||||
|
||||
**Responsibility boundary**: VaultClient is a client adapter — it knows only about
|
||||
Vault API calls. It has no knowledge of business rules, HTTP, or PostgreSQL. It is
|
||||
injected into `CredentialService` and `OAuth2Service` via constructor injection. When
|
||||
`VAULT_ADDR` is not set, `createVaultClientFromEnv()` returns `null` and the bcrypt
|
||||
code path is used unchanged.
|
||||
|
||||
**Public methods**:
|
||||
|
||||
| Method | Parameters | Returns | Description |
|
||||
|--------|-----------|---------|-------------|
|
||||
| `writeSecret` | `agentId: string, credentialId: string, plainSecret: string` | `Promise<string>` | Writes the plain-text secret to the KV v2 data path; returns the path; creates a new KV v2 version on subsequent calls (used for rotation) |
|
||||
| `readSecret` | `agentId: string, credentialId: string` | `Promise<string>` | Reads and returns the plain-text secret from Vault; throws `CredentialError` if the path is not found or the read fails |
|
||||
| `verifySecret` | `agentId: string, credentialId: string, candidateSecret: string` | `Promise<boolean>` | Reads the stored secret via `readSecret`, then compares using `crypto.timingSafeEqual` to prevent timing-based side-channel attacks; returns `false` on any Vault error rather than throwing |
|
||||
| `deleteSecret` | `agentId: string, credentialId: string` | `Promise<void>` | Permanently deletes all versions of a credential secret by calling the KV v2 metadata path (`DELETE {mount}/metadata/agentidp/agents/{agentId}/credentials/{credentialId}`) |
|
||||
|
||||
**KV v2 path structure**:
|
||||
- Data path: `{mount}/data/agentidp/agents/{agentId}/credentials/{credentialId}`
|
||||
- Metadata path (for permanent deletion): `{mount}/metadata/agentidp/agents/{agentId}/credentials/{credentialId}`
|
||||
- Default mount: `secret` (overridable via `VAULT_MOUNT`)
|
||||
|
||||
**Opt-in configuration**:
|
||||
- `VAULT_ADDR` — Vault server address (e.g. `http://127.0.0.1:8200`) — required to enable Vault mode
|
||||
- `VAULT_TOKEN` — Vault authentication token — required to enable Vault mode
|
||||
- `VAULT_MOUNT` — KV v2 mount path — optional, defaults to `secret`
|
||||
|
||||
**Constant-time comparison rationale**: The `verifySecret` method uses Node.js
|
||||
`crypto.timingSafeEqual` instead of `===` to prevent attackers from inferring the
|
||||
length or content of stored secrets by measuring how long the comparison takes. When
|
||||
the stored and candidate secrets differ in length, a dummy `timingSafeEqual` call is
|
||||
still performed to eliminate the timing signal from the early-exit path.
|
||||
|
||||
---
|
||||
|
||||
### OPA Policy Engine
|
||||
|
||||
**Purpose**: Enforces scope-based authorisation on every protected HTTP request without requiring a code deployment to change access rules.
|
||||
|
||||
**Responsibility boundary**: The OPA policy engine (`src/middleware/opa.ts`) is a
|
||||
middleware layer — it does not know about business rules, credentials, or audit events.
|
||||
It receives the HTTP method, full request path, and caller scopes from `req.user`, and
|
||||
returns allow or deny. All policy logic lives in `policies/authz.rego` and
|
||||
`policies/data/scopes.json`.
|
||||
|
||||
**Policy file locations**:
|
||||
- `policies/authz.rego` — Rego policy defining `normalise_path`, `lookup_key`, and the `allow` rule. Evaluated by the Wasm bundle when compiled; replicated in TypeScript for the fallback path.
|
||||
- `policies/data/scopes.json` — JSON map of `"METHOD:/path/pattern"` → `[required_scopes]`. Loaded as data into the Wasm policy and used directly by the TypeScript fallback.
|
||||
- `policies/authz.wasm` — compiled Wasm bundle (not committed to source control; built from `authz.rego` using the OPA CLI). When present, the Wasm path is used; when absent, the TypeScript fallback reads `scopes.json`.
|
||||
|
||||
**How `opaMiddleware` evaluates input**:
|
||||
|
||||
1. `createOpaMiddleware()` is called once at app startup in `src/app.ts`.
|
||||
2. It attempts to load `policies/authz.wasm`. If found, `loadPolicy(wasmBuffer)` is called and `scopes.json` data is injected via `loaded.setData(parsed)`.
|
||||
3. If no Wasm bundle is found, `scopes.json` is loaded into `scopesMap` as the TypeScript fallback.
|
||||
4. On every request, the middleware builds an `OpaInput` object: `{ method: req.method, path: req.baseUrl + req.path, scopes: req.user.scope.split(' ') }`.
|
||||
5. `evaluate(input)` checks the Wasm policy (if loaded) or applies `normalisePath` + scope-intersection logic against `scopesMap`. Returns `false` if neither is loaded (fail-closed).
|
||||
6. If `evaluate` returns `false`, the middleware calls `next(new AuthorizationError())`.
|
||||
|
||||
**How to write a new policy rule**:
|
||||
|
||||
1. Add the new endpoint's scope requirement to `policies/data/scopes.json`:
|
||||
```json
|
||||
"GET:/api/v1/reports": ["reports:read"]
|
||||
```
|
||||
2. Add `"reports:read"` to the `OAuthScope` union type in `src/types/index.ts`.
|
||||
3. If Wasm mode is in use, recompile `authz.rego` to `authz.wasm` using the OPA CLI: `opa build policies/authz.rego -d policies/data/ -o policies/authz.wasm`.
|
||||
4. Send `SIGHUP` to the running process to hot-reload: `kill -HUP <pid>`.
|
||||
|
||||
**How to test a policy rule**:
|
||||
```bash
|
||||
# Using the OPA CLI directly
|
||||
opa eval --data policies/data/scopes.json \
|
||||
--input '{"method":"GET","path":"/api/v1/agents","scopes":["agents:read"]}' \
|
||||
--bundle policies/ \
|
||||
'data.authz.allow'
|
||||
```
|
||||
Expected output: `true`. Replace method/path/scopes to test deny cases.
|
||||
|
||||
**Hot-reload via SIGHUP**: When `SIGHUP` is received by the Node.js process,
|
||||
`server.ts` calls `reloadOpaPolicy()`. This re-executes the same startup loading logic:
|
||||
tries to load the Wasm bundle, falls back to `scopes.json`. The in-memory `wasmPolicy`
|
||||
and `scopesMap` module-level variables are replaced atomically. No requests are dropped.
|
||||
|
||||
---
|
||||
|
||||
### Web Dashboard
|
||||
|
||||
**Purpose**: Provides a browser-based UI for human operators to manage agents, credentials, and audit logs without writing API calls directly.
|
||||
|
||||
**Responsibility boundary**: The dashboard is a pure client-side React SPA. It has no
|
||||
server-side logic. It calls the AgentIdP REST API using the `@sentryagent/idp-sdk`
|
||||
`TokenManager` for authentication and a typed `ApiClient` from `dashboard/src/lib/client.ts`
|
||||
for all API calls. It never stores the `access_token` in localStorage — only
|
||||
`client_id`, `client_secret`, and `baseUrl` are stored in `sessionStorage` (cleared
|
||||
on tab close).
|
||||
|
||||
**React component structure**:
|
||||
|
||||
```
|
||||
dashboard/src/
|
||||
├── main.tsx # React root — mounts App into #root, wraps with BrowserRouter
|
||||
├── App.tsx # Route definitions — AuthProvider, RequireAuth, AppShell
|
||||
├── lib/
|
||||
│ ├── auth.tsx # AuthContext, AuthProvider, useAuth hook, sessionStorage helpers
|
||||
│ └── client.ts # Typed ApiClient class — wraps fetch with TokenManager token injection
|
||||
├── components/
|
||||
│ ├── RequireAuth.tsx # Route guard — redirects to /dashboard/login if not authenticated
|
||||
│ └── layout/AppShell.tsx # Persistent sidebar navigation + Outlet for page content
|
||||
└── pages/
|
||||
├── Login.tsx # Login form — calls auth.login(), redirects to /dashboard/agents
|
||||
├── Agents.tsx # Paginated agents list with status filter and search
|
||||
├── AgentDetail.tsx # Single agent view — status, metadata, update, decommission actions
|
||||
├── Credentials.tsx # Credential list for an agent — generate, rotate, revoke actions
|
||||
├── AuditLog.tsx # Paginated audit log with date range and action filters
|
||||
└── Health.tsx # /health endpoint response — PostgreSQL and Redis status display
|
||||
```
|
||||
|
||||
**Authentication flow with sessionStorage**:
|
||||
1. On `Login.tsx` form submit, `auth.login(creds)` is called.
|
||||
2. `validateCredentials(creds)` creates a `TokenManager` and calls `getToken()` — if this succeeds, the credentials are valid.
|
||||
3. `saveCredentials(creds)` stores `{ clientId, clientSecret, baseUrl }` in `sessionStorage` under key `agentidp_credentials`.
|
||||
4. On every subsequent API call, `getClient()` in `lib/client.ts` reads credentials from `sessionStorage`, creates a `TokenManager`, and injects the current `access_token` into the `Authorization: Bearer` header. The `TokenManager` handles automatic token refresh when the token is expired.
|
||||
5. `auth.logout()` calls `clearCredentials()` (removes the `sessionStorage` key) and navigates to `/dashboard/login`.
|
||||
|
||||
**Main views and their API calls**:
|
||||
- **Agents** — `GET /api/v1/agents?page=N&limit=20` — paginated list with `status` filter
|
||||
- **AgentDetail** — `GET /api/v1/agents/:id`, `PATCH /api/v1/agents/:id`, `DELETE /api/v1/agents/:id`
|
||||
- **Credentials** — `GET /api/v1/agents/:id/credentials`, `POST /api/v1/agents/:id/credentials`, `POST /api/v1/agents/:id/credentials/:credId/rotate`, `DELETE /api/v1/agents/:id/credentials/:credId`
|
||||
- **AuditLog** — `GET /api/v1/audit?page=N&limit=20&fromDate=...&toDate=...`
|
||||
- **Health** — `GET /health`
|
||||
|
||||
**Local development**:
|
||||
```bash
|
||||
cd dashboard
|
||||
npm install
|
||||
npm run dev # Vite dev server with HMR — dashboard available at http://localhost:5173/dashboard
|
||||
```
|
||||
The Vite dev server proxies `/api/` calls to the Express server at `http://localhost:3000`.
|
||||
The Express server must be running separately for API calls to work.
|
||||
|
||||
---
|
||||
|
||||
### Prometheus/Grafana Monitoring
|
||||
|
||||
**Purpose**: Provides operational visibility into AgentIdP's HTTP traffic, token issuance rates, agent registration rates, database latency, and Redis command latency.
|
||||
|
||||
**Responsibility boundary**: The metrics middleware (`src/middleware/metrics.ts`) and
|
||||
the metrics registry (`src/metrics/registry.ts`) are observability concerns only — they
|
||||
do not affect business logic. Metrics are exposed at `GET /metrics` via
|
||||
`createMetricsRouter()` using `metricsRegistry.metrics()` from `prom-client`. The
|
||||
`/metrics` endpoint is unauthenticated, intended for scraping by Prometheus only and
|
||||
not exposed to the public internet.
|
||||
|
||||
**Key metrics with labels**:
|
||||
|
||||
| Metric Name | Type | Labels | Description |
|
||||
|-------------|------|--------|-------------|
|
||||
| `agentidp_http_requests_total` | Counter | `method`, `route`, `status_code` | Total HTTP requests received; route is normalised (UUIDs replaced with `:id`) |
|
||||
| `agentidp_http_request_duration_seconds` | Histogram | `method`, `route`, `status_code` | HTTP request duration; buckets from 5ms to 2.5s |
|
||||
| `agentidp_tokens_issued_total` | Counter | `scope` | Total OAuth 2.0 access tokens successfully issued |
|
||||
| `agentidp_agents_registered_total` | Counter | `deployment_env` | Total AI agents successfully registered |
|
||||
| `agentidp_db_query_duration_seconds` | Histogram | `operation` | PostgreSQL query duration; buckets from 1ms to 1s |
|
||||
| `agentidp_redis_command_duration_seconds` | Histogram | `command` | Redis command duration; buckets from 0.5ms to 250ms |
|
||||
|
||||
**How to add a new Counter**:
|
||||
1. Open `src/metrics/registry.ts`.
|
||||
2. Add a new `Counter` export:
|
||||
```typescript
|
||||
export const myNewCounter = new Counter({
|
||||
name: 'agentidp_my_new_counter_total',
|
||||
help: 'Description of what this counts.',
|
||||
labelNames: ['label_one'] as const,
|
||||
registers: [metricsRegistry],
|
||||
});
|
||||
```
|
||||
3. Import and call `myNewCounter.inc({ label_one: value })` in the service or middleware where the event occurs.
|
||||
|
||||
**How to add a new Histogram**:
|
||||
1. Open `src/metrics/registry.ts`.
|
||||
2. Add a new `Histogram` export with appropriate buckets:
|
||||
```typescript
|
||||
export const myDurationHistogram = new Histogram({
|
||||
name: 'agentidp_my_operation_duration_seconds',
|
||||
help: 'Duration of my operation in seconds.',
|
||||
labelNames: ['operation'] as const,
|
||||
buckets: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1],
|
||||
registers: [metricsRegistry],
|
||||
});
|
||||
```
|
||||
3. Use `const end = myDurationHistogram.startTimer({ operation: 'name' }); ... end();` around the operation being measured.
|
||||
|
||||
**Grafana access in local Docker**:
|
||||
|
||||
Start the monitoring overlay:
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up
|
||||
```
|
||||
- Prometheus: `http://localhost:9090`
|
||||
- Grafana: `http://localhost:3001` — default credentials: `admin` / `agentidp`
|
||||
|
||||
Grafana is pre-provisioned with a Prometheus data source pointing to `http://prometheus:9090`
|
||||
and dashboard JSON files from `monitoring/grafana/dashboards/`. No manual configuration
|
||||
is needed after startup.
|
||||
717
docs/engineering/06-walkthroughs.md
Normal file
717
docs/engineering/06-walkthroughs.md
Normal file
@@ -0,0 +1,717 @@
|
||||
# 06 — Code Walkthroughs
|
||||
|
||||
Last verified against commit: `1f95cfe89d1f45fa43b9fb7cff237f07bf9e889e`
|
||||
|
||||
These walkthroughs trace three real production code paths from the HTTP request
|
||||
to the database and back. Every step includes a `file:line` reference and a
|
||||
"why" annotation explaining the design decision.
|
||||
|
||||
---
|
||||
|
||||
## Walkthrough 1 — Token Issuance
|
||||
|
||||
**Request:** `POST /api/v1/token` with `grant_type=client_credentials`
|
||||
|
||||
This is the most security-critical path in the codebase. An AI agent calling this
|
||||
endpoint is proving its identity and receiving a token that grants access to the
|
||||
entire API for one hour.
|
||||
|
||||
---
|
||||
|
||||
### Step 1 — Express middleware stack
|
||||
|
||||
**File:** `src/app.ts` lines 57–83
|
||||
|
||||
```
|
||||
helmet() → security headers
|
||||
cors() → CORS headers
|
||||
morgan() → access log line (skipped in test env)
|
||||
express.json() → parse JSON bodies
|
||||
express.urlencoded({ extended: false }) → parse form-encoded bodies
|
||||
metricsMiddleware → start request timer, record counters on finish
|
||||
```
|
||||
|
||||
**Why `extended: false`?** The token endpoint receives `application/x-www-form-urlencoded`
|
||||
bodies (RFC 6749 mandates this format for OAuth 2.0). The `express.urlencoded`
|
||||
middleware parses them into `req.body`. `extended: false` uses the native `querystring`
|
||||
parser, which is sufficient and avoids `qs` library complexity for flat key-value data.
|
||||
|
||||
---
|
||||
|
||||
### Step 2 — Route dispatch
|
||||
|
||||
**File:** `src/routes/token.ts` line 24
|
||||
|
||||
```typescript
|
||||
router.post('/', asyncHandler(rateLimitMiddleware), asyncHandler(tokenController.issueToken.bind(tokenController)));
|
||||
```
|
||||
|
||||
**Why no `authMiddleware` here?** The token endpoint is where the agent _gets_ its
|
||||
token — it cannot present a Bearer token to authenticate. Instead, credentials go
|
||||
in the request body (`client_id`, `client_secret`). `POST /token` is deliberately
|
||||
unauthenticated at the transport layer; authentication happens inside the controller.
|
||||
|
||||
**Why `asyncHandler`?** Express does not natively support async middleware. `asyncHandler`
|
||||
wraps the async function and calls `next(err)` if the promise rejects, routing the
|
||||
error to `errorHandler`.
|
||||
|
||||
---
|
||||
|
||||
### Step 3 — Rate limit check
|
||||
|
||||
**File:** `src/middleware/rateLimit.ts`
|
||||
|
||||
The rate limiter checks a Redis sliding-window counter for the client's IP address.
|
||||
If the counter exceeds 100 requests/minute, it throws `RateLimitError` (429).
|
||||
|
||||
**Why Redis, not in-memory?** If the server restarts or scales horizontally to multiple
|
||||
instances, an in-memory counter would reset. Redis maintains the counter across
|
||||
instances and restarts.
|
||||
|
||||
---
|
||||
|
||||
### Step 4 — Controller: validate grant_type
|
||||
|
||||
**File:** `src/controllers/TokenController.ts` lines 84–103
|
||||
|
||||
```typescript
|
||||
issueToken = async (req: Request, res: Response, _next: NextFunction): Promise<void> => {
|
||||
const body = req.body as ITokenRequest;
|
||||
|
||||
if (!body.grant_type) { ... return res.status(400).json({error: 'invalid_request', ...}) }
|
||||
if (body.grant_type !== 'client_credentials') { ... return res.status(400).json(...) }
|
||||
```
|
||||
|
||||
**Why does this method catch errors itself instead of calling `next(err)`?** The token
|
||||
endpoint must return errors in the **OAuth 2.0 error format** (`{ error, error_description }`)
|
||||
per RFC 6749 §5.2, not the standard SentryAgent.ai format (`{ code, message }`). The
|
||||
`mapToOAuth2Error()` helper translates `AuthenticationError` and `AuthorizationError`
|
||||
into OAuth2 error codes. The `_next` parameter is intentionally unused for the error path.
|
||||
|
||||
---
|
||||
|
||||
### Step 5 — Controller: Joi validation and credential extraction
|
||||
|
||||
**File:** `src/controllers/TokenController.ts` lines 106–138
|
||||
|
||||
```typescript
|
||||
const { error, value } = tokenRequestSchema.validate(body, { abortEarly: false });
|
||||
// ...
|
||||
// Support HTTP Basic auth fallback (RFC 6749 §2.3.1)
|
||||
const authHeader = req.headers['authorization'];
|
||||
if (authHeader?.startsWith('Basic ')) {
|
||||
const base64 = authHeader.slice(6);
|
||||
const decoded = Buffer.from(base64, 'base64').toString('utf-8');
|
||||
const colonIndex = decoded.indexOf(':');
|
||||
clientId = decoded.slice(0, colonIndex);
|
||||
clientSecret = decoded.slice(colonIndex + 1);
|
||||
}
|
||||
```
|
||||
|
||||
**Why `abortEarly: false`?** This returns all validation errors at once, so the
|
||||
client can fix all problems in one round trip.
|
||||
|
||||
**Why Basic auth support?** RFC 6749 §2.3.1 specifies that client credentials MAY
|
||||
be sent via HTTP Basic authentication. Some OAuth libraries default to this method.
|
||||
|
||||
---
|
||||
|
||||
### Step 6 — Controller: scope validation
|
||||
|
||||
**File:** `src/controllers/TokenController.ts` lines 141–151
|
||||
|
||||
```typescript
|
||||
const requestedScope = tokenBody.scope ?? 'agents:read';
|
||||
const validScopes = ['agents:read', 'agents:write', 'tokens:read', 'audit:read'];
|
||||
const scopeList = requestedScope.split(' ');
|
||||
const invalidScope = scopeList.find((s) => !validScopes.includes(s));
|
||||
if (invalidScope) { return res.status(400).json({error: 'invalid_scope', ...}) }
|
||||
```
|
||||
|
||||
**Why validate scopes here?** Scope validation at the controller layer provides an
|
||||
RFC 6749-compliant `invalid_scope` error before we even look up the agent. This is
|
||||
faster and gives the client a clearer error message.
|
||||
|
||||
---
|
||||
|
||||
### Step 7 — Service: agent lookup
|
||||
|
||||
**File:** `src/services/OAuth2Service.ts` lines 83–94
|
||||
|
||||
```typescript
|
||||
const agent = await this.agentRepository.findById(clientId);
|
||||
if (!agent) {
|
||||
void this.auditService.logEvent(clientId, 'auth.failed', 'failure', ..., { reason: 'agent_not_found', clientId });
|
||||
throw new AuthenticationError('Client authentication failed...');
|
||||
}
|
||||
```
|
||||
|
||||
**Why log auth failures?** Failed authentication attempts may indicate a brute-force
|
||||
attack or a misconfigured client. Having them in the audit log enables incident
|
||||
investigation and alerting.
|
||||
|
||||
**Why not distinguish between "agent not found" and "wrong secret" in the error message?**
|
||||
Revealing which is wrong gives an attacker information — they can enumerate valid
|
||||
`client_id` values by checking whether they get "agent not found" vs "wrong secret".
|
||||
Both cases return the same message.
|
||||
|
||||
---
|
||||
|
||||
### Step 8 — Service: credential verification
|
||||
|
||||
**File:** `src/services/OAuth2Service.ts` lines 97–131
|
||||
|
||||
```typescript
|
||||
const { credentials } = await this.credentialRepository.findByAgentId(clientId, { status: 'active', page: 1, limit: 100 });
|
||||
|
||||
for (const cred of credentials) {
|
||||
const credRow = await this.credentialRepository.findById(cred.credentialId);
|
||||
if (credRow) {
|
||||
if (credRow.expiresAt !== null && credRow.expiresAt < new Date()) { continue; }
|
||||
|
||||
let matches: boolean;
|
||||
if (credRow.vaultPath !== null && this.vaultClient !== null) {
|
||||
matches = await this.vaultClient.verifySecret(clientId, credRow.credentialId, clientSecret);
|
||||
} else {
|
||||
matches = await verifySecret(clientSecret, credRow.secretHash);
|
||||
}
|
||||
if (matches) { credentialVerified = true; break; }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Why iterate over multiple credentials?** An agent can have multiple active
|
||||
credentials (e.g. one per service that calls it). The agent rotates credentials
|
||||
one at a time — if credential A is rotated while service X is still using it,
|
||||
service X will fail. By checking all active credentials, we allow overlapping rotation.
|
||||
|
||||
**Why check expiry before hashing?** Bcrypt is intentionally slow (~100ms). Checking
|
||||
expiry first is a cheap early exit that avoids the bcrypt computation on expired
|
||||
credentials.
|
||||
|
||||
---
|
||||
|
||||
### Step 9 — Service: status and monthly limit checks
|
||||
|
||||
**File:** `src/services/OAuth2Service.ts` lines 144–176
|
||||
|
||||
```typescript
|
||||
if (agent.status === 'suspended') { throw new AuthorizationError(...) }
|
||||
if (agent.status === 'decommissioned') { throw new AuthorizationError(...) }
|
||||
|
||||
const monthlyCount = await this.tokenRepository.getMonthlyCount(clientId);
|
||||
if (monthlyCount >= FREE_TIER_MAX_MONTHLY_TOKENS) { throw new FreeTierLimitError(...) }
|
||||
```
|
||||
|
||||
**Why check status after credential verification?** We verify credentials first so
|
||||
a suspended agent with a wrong secret gets `AuthenticationError` (401) not
|
||||
`AuthorizationError` (403). This prevents leaking which agents are suspended to
|
||||
unauthenticated callers.
|
||||
|
||||
---
|
||||
|
||||
### Step 10 — Service: sign the JWT
|
||||
|
||||
**File:** `src/services/OAuth2Service.ts` lines 179–190
|
||||
|
||||
```typescript
|
||||
const jti = uuidv4();
|
||||
const payload: Omit<ITokenPayload, 'iat' | 'exp'> = { sub: clientId, client_id: clientId, scope, jti };
|
||||
const accessToken = signToken(payload, this.privateKey);
|
||||
```
|
||||
|
||||
**File:** `src/utils/jwt.ts` lines 19–31
|
||||
|
||||
```typescript
|
||||
export function signToken(payload: Omit<ITokenPayload, 'iat' | 'exp'>, privateKey: string): string {
|
||||
const now = Math.floor(Date.now() / 1000);
|
||||
const fullPayload: ITokenPayload = { ...payload, iat: now, exp: now + TOKEN_EXPIRES_IN };
|
||||
return jwt.sign(fullPayload, privateKey, { algorithm: 'RS256' });
|
||||
}
|
||||
```
|
||||
|
||||
**Why RS256 instead of HS256?** RS256 (RSA asymmetric) allows any consumer of the
|
||||
token to verify it using the public key without needing the private signing key.
|
||||
HS256 (HMAC symmetric) would require sharing the secret with every service that
|
||||
verifies tokens.
|
||||
|
||||
**Why `jti` (JWT ID)?** The `jti` is a unique identifier for this specific token.
|
||||
It is used as the key in the Redis revocation list. Without `jti`, you cannot
|
||||
revoke a single token without revoking all tokens for the agent.
|
||||
|
||||
---
|
||||
|
||||
### Step 11 — Service: fire-and-forget operations
|
||||
|
||||
**File:** `src/services/OAuth2Service.ts` lines 193–207
|
||||
|
||||
```typescript
|
||||
void this.tokenRepository.incrementMonthlyCount(clientId);
|
||||
void this.auditService.logEvent(clientId, 'token.issued', 'success', ..., { scope, expiresAt });
|
||||
tokensIssuedTotal.inc({ scope });
|
||||
```
|
||||
|
||||
**Why `void` (fire-and-forget)?** The token has been signed and is ready to return.
|
||||
Waiting for the Redis increment and audit write would add ~5–10ms to every token
|
||||
request. These operations are best-effort — if they fail, the token is still valid.
|
||||
|
||||
**Why is the Prometheus `.inc()` call synchronous?** Prometheus counters are
|
||||
in-process memory operations — they do not write to Redis or PostgreSQL. They are
|
||||
O(1) and sub-microsecond.
|
||||
|
||||
---
|
||||
|
||||
### Step 12 — Response
|
||||
|
||||
**File:** `src/controllers/TokenController.ts` lines 163–167
|
||||
|
||||
```typescript
|
||||
res.setHeader('Cache-Control', 'no-store');
|
||||
res.setHeader('Pragma', 'no-cache');
|
||||
res.status(200).json(tokenResponse);
|
||||
```
|
||||
|
||||
**Why `Cache-Control: no-store`?** RFC 6749 §5.1 mandates that token responses
|
||||
must not be cached. Without this header, a shared proxy or CDN could cache the
|
||||
response and replay it to another client.
|
||||
|
||||
Final response:
|
||||
```json
|
||||
{
|
||||
"access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...",
|
||||
"token_type": "Bearer",
|
||||
"expires_in": 3600,
|
||||
"scope": "agents:read agents:write"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Walkthrough 2 — Agent Registration
|
||||
|
||||
**Request:** `POST /api/v1/agents` with Bearer token and agent data JSON body
|
||||
|
||||
After token issuance, registering an agent is the second most common operation.
|
||||
This walkthrough shows a request that goes through all three auth middleware layers.
|
||||
|
||||
---
|
||||
|
||||
### Step 1 — Middleware stack
|
||||
|
||||
**File:** `src/app.ts` lines 57–83 (same security and parsing middleware as Walkthrough 1)
|
||||
|
||||
---
|
||||
|
||||
### Step 2 — Route dispatch
|
||||
|
||||
**File:** `src/routes/agents.ts` lines 22–27
|
||||
|
||||
```typescript
|
||||
router.use(asyncHandler(authMiddleware));
|
||||
router.use(opaMiddleware);
|
||||
router.use(asyncHandler(rateLimitMiddleware));
|
||||
router.post('/', asyncHandler(agentController.registerAgent.bind(agentController)));
|
||||
```
|
||||
|
||||
All three middleware run on every request to the agents router before the handler.
|
||||
|
||||
---
|
||||
|
||||
### Step 3 — Auth middleware: Bearer token verification
|
||||
|
||||
**File:** `src/middleware/auth.ts` lines 28–77
|
||||
|
||||
```typescript
|
||||
const authHeader = req.headers['authorization'];
|
||||
if (!authHeader || !authHeader.startsWith('Bearer ')) { throw new AuthenticationError(...) }
|
||||
|
||||
const token = authHeader.slice(7).trim();
|
||||
const publicKey = process.env['JWT_PUBLIC_KEY'];
|
||||
let payload: ITokenPayload;
|
||||
try {
|
||||
payload = verifyToken(token, publicKey);
|
||||
} catch (err) {
|
||||
if (err instanceof TokenExpiredError) { throw new AuthenticationError('Token has expired.') }
|
||||
if (err instanceof JsonWebTokenError) { throw new AuthenticationError('Token signature is invalid.') }
|
||||
}
|
||||
|
||||
const redis = await getRedisClient();
|
||||
const revocationKey = `revoked:${payload.jti}`;
|
||||
const isRevoked = await redis.get(revocationKey);
|
||||
if (isRevoked !== null) { throw new AuthenticationError('Token has been revoked.') }
|
||||
|
||||
req.user = payload;
|
||||
next();
|
||||
```
|
||||
|
||||
**Why check Redis after signature verification?** Signature verification is a pure
|
||||
cryptographic operation (no I/O). If the token is expired or has a bad signature,
|
||||
there is no need to hit Redis. The fast path exits early; Redis is the slower
|
||||
secondary check.
|
||||
|
||||
**Why `await getRedisClient()` instead of storing the client?** `getRedisClient()`
|
||||
returns the same singleton every time — the connection is created once and reused.
|
||||
The `await` is fast (no I/O after the first call).
|
||||
|
||||
---
|
||||
|
||||
### Step 4 — OPA middleware: scope enforcement
|
||||
|
||||
**File:** `src/middleware/opa.ts` lines 230–257
|
||||
|
||||
```typescript
|
||||
const input: OpaInput = {
|
||||
method: req.method, // "POST"
|
||||
path: req.baseUrl + req.path, // "/api/v1/agents"
|
||||
scopes: req.user.scope.split(' '), // ["agents:read", "agents:write"]
|
||||
};
|
||||
|
||||
if (!evaluate(input)) {
|
||||
next(new AuthorizationError());
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
For `POST /api/v1/agents`, the policy requires `["agents:write"]`. If `agents:write`
|
||||
is not in the token's scope, the request is rejected with 403 before the controller
|
||||
runs.
|
||||
|
||||
**Why reconstruct the full path with `req.baseUrl + req.path`?** The OPA policy
|
||||
uses full paths (`/api/v1/agents/:id`). Inside a nested router, `req.path` is
|
||||
relative to the router's mount point (e.g. `/`). `req.baseUrl` is the mount prefix
|
||||
(`/api/v1/agents`). Concatenating them gives the full path the policy expects.
|
||||
|
||||
---
|
||||
|
||||
### Step 5 — Controller: validation
|
||||
|
||||
**File:** `src/controllers/AgentController.ts` lines 37–60
|
||||
|
||||
```typescript
|
||||
registerAgent = async (req: Request, res: Response, next: NextFunction): Promise<void> => {
|
||||
if (!req.user) { throw new AuthorizationError() }
|
||||
|
||||
const { error, value } = createAgentSchema.validate(req.body, { abortEarly: false });
|
||||
if (error) {
|
||||
throw new ValidationError('Request validation failed.', {
|
||||
details: error.details.map((d) => ({ field: d.path.join('.'), reason: d.message })),
|
||||
});
|
||||
}
|
||||
|
||||
const data = value as ICreateAgentRequest;
|
||||
const ipAddress = req.ip ?? '0.0.0.0';
|
||||
const userAgent = req.headers['user-agent'] ?? 'unknown';
|
||||
|
||||
const agent = await this.agentService.registerAgent(data, ipAddress, userAgent);
|
||||
res.status(201).json(agent);
|
||||
```
|
||||
|
||||
**Why check `req.user` in the controller when `authMiddleware` already set it?**
|
||||
TypeScript's type system marks `req.user` as `ITokenPayload | undefined`. The check
|
||||
at line 39 narrows the type so subsequent code can use `req.user` without null
|
||||
assertions. It is a guard, not redundant authentication.
|
||||
|
||||
**Why pass `ipAddress` and `userAgent` to the service?** The service logs audit events.
|
||||
Audit events include the client IP and User-Agent for forensic value. These values
|
||||
come from the HTTP request, which the service has no access to — so the controller
|
||||
extracts them and passes them down.
|
||||
|
||||
---
|
||||
|
||||
### Step 6 — Service: free-tier limit check
|
||||
|
||||
**File:** `src/services/AgentService.ts` lines 59–65
|
||||
|
||||
```typescript
|
||||
const currentCount = await this.agentRepository.countActive();
|
||||
if (currentCount >= FREE_TIER_MAX_AGENTS) {
|
||||
throw new FreeTierLimitError('Free tier limit of 100 registered agents has been reached.', ...);
|
||||
}
|
||||
```
|
||||
|
||||
**Why count before checking email uniqueness?** If the limit is reached, there is
|
||||
no point checking whether the email already exists. Doing the cheaper check (count)
|
||||
first avoids an unnecessary query.
|
||||
|
||||
---
|
||||
|
||||
### Step 7 — Service: email uniqueness check
|
||||
|
||||
**File:** `src/services/AgentService.ts` lines 68–71
|
||||
|
||||
```typescript
|
||||
const existing = await this.agentRepository.findByEmail(data.email);
|
||||
if (existing !== null) { throw new AgentAlreadyExistsError(data.email) }
|
||||
```
|
||||
|
||||
**Why not rely on the database UNIQUE constraint?** We could, but catching a
|
||||
PostgreSQL `23505` error code in the repository would be less readable and would
|
||||
not produce a typed `AgentAlreadyExistsError` with a structured `details` field.
|
||||
The explicit check gives better error messages and keeps the repository layer clean.
|
||||
|
||||
---
|
||||
|
||||
### Step 8 — Repository: INSERT
|
||||
|
||||
**File:** `src/repositories/AgentRepository.ts` lines 67–85
|
||||
|
||||
```typescript
|
||||
async create(data: ICreateAgentRequest): Promise<IAgent> {
|
||||
const agentId = uuidv4();
|
||||
const result: QueryResult<AgentRow> = await this.pool.query(
|
||||
`INSERT INTO agents (agent_id, email, agent_type, version, capabilities, owner, deployment_env, status, created_at, updated_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, 'active', NOW(), NOW())
|
||||
RETURNING *`,
|
||||
[agentId, data.email, data.agentType, data.version, data.capabilities, data.owner, data.deploymentEnv],
|
||||
);
|
||||
return mapRowToAgent(result.rows[0]);
|
||||
}
|
||||
```
|
||||
|
||||
**Why generate `agentId` in application code instead of relying on `gen_random_uuid()`?**
|
||||
Because we use the UUID as the OAuth 2.0 `client_id`. We need the UUID before writing
|
||||
to the database so we can use it in the audit event and the response. Having it in
|
||||
application code avoids a separate SELECT after the INSERT.
|
||||
|
||||
**Why `RETURNING *`?** PostgreSQL's `RETURNING` clause sends back the inserted row
|
||||
in the same round trip as the INSERT. This avoids a second SELECT to fetch the
|
||||
newly created record.
|
||||
|
||||
---
|
||||
|
||||
### Step 9 — Service: audit event
|
||||
|
||||
**File:** `src/services/AgentService.ts` lines 76–83
|
||||
|
||||
```typescript
|
||||
await this.auditService.logEvent(
|
||||
agent.agentId,
|
||||
'agent.created',
|
||||
'success',
|
||||
ipAddress,
|
||||
userAgent,
|
||||
{ agentType: agent.agentType, owner: agent.owner },
|
||||
);
|
||||
```
|
||||
|
||||
**Why `await` here but `void` for token audit events?** Agent registration is a
|
||||
database write operation that happens once. Adding ~5ms for the audit write is
|
||||
acceptable and ensures the audit event is recorded before the 201 response is sent.
|
||||
Token issuance happens far more frequently — audit is fire-and-forget there.
|
||||
|
||||
---
|
||||
|
||||
### Step 10 — Response
|
||||
|
||||
**File:** `src/controllers/AgentController.ts` line 56
|
||||
|
||||
```typescript
|
||||
res.status(201).json(agent);
|
||||
```
|
||||
|
||||
Returns the full `IAgent` object with HTTP 201 Created.
|
||||
|
||||
---
|
||||
|
||||
## Walkthrough 3 — Credential Rotation
|
||||
|
||||
**Request:** `POST /api/v1/agents/:agentId/credentials/:credentialId/rotate`
|
||||
|
||||
Credential rotation is the process of replacing an existing client secret with a
|
||||
new one without changing the `credentialId`. This is the recommended security
|
||||
practice — rotate periodically and rotate immediately after suspected compromise.
|
||||
|
||||
---
|
||||
|
||||
### Step 1 — Route dispatch
|
||||
|
||||
**File:** `src/routes/credentials.ts` line 34
|
||||
|
||||
```typescript
|
||||
router.post('/:credentialId/rotate', asyncHandler(credentialController.rotateCredential.bind(credentialController)));
|
||||
```
|
||||
|
||||
The credentials router is mounted at `/api/v1/agents/:agentId/credentials` in `app.ts`.
|
||||
The full path becomes `POST /api/v1/agents/:agentId/credentials/:credentialId/rotate`.
|
||||
|
||||
---
|
||||
|
||||
### Step 2 — Auth middleware
|
||||
|
||||
Same as Walkthrough 2, Step 3. Bearer token is verified via RS256 and Redis revocation check.
|
||||
`req.user` is populated with the JWT payload.
|
||||
|
||||
---
|
||||
|
||||
### Step 3 — OPA middleware
|
||||
|
||||
The path `/api/v1/agents/:agentId/credentials/:credId/rotate` is normalised to
|
||||
`/api/v1/agents/:id/credentials/:credId/rotate`. The policy requires `["agents:write"]`.
|
||||
|
||||
---
|
||||
|
||||
### Step 4 — Controller: ownership check
|
||||
|
||||
**File:** `src/controllers/CredentialController.ts` lines 127–137
|
||||
|
||||
```typescript
|
||||
rotateCredential = async (req: Request, res: Response, next: NextFunction): Promise<void> => {
|
||||
if (!req.user) { throw new AuthenticationError() }
|
||||
|
||||
const { agentId, credentialId } = req.params;
|
||||
|
||||
if (req.user.sub !== agentId) {
|
||||
throw new AuthorizationError('You do not have permission to manage credentials for this agent.');
|
||||
}
|
||||
```
|
||||
|
||||
**Why check `req.user.sub !== agentId`?** An agent's token contains its own
|
||||
`agentId` as the `sub` claim. This check enforces that an agent can only manage
|
||||
its own credentials. Even if an agent has `agents:write` scope, it cannot rotate
|
||||
another agent's credentials. This is Phase 1 behaviour — there is no admin scope yet.
|
||||
|
||||
---
|
||||
|
||||
### Step 5 — Controller: request validation
|
||||
|
||||
**File:** `src/controllers/CredentialController.ts` lines 139–157
|
||||
|
||||
```typescript
|
||||
const { error, value } = generateCredentialSchema.validate(req.body ?? {}, { abortEarly: false });
|
||||
// generateCredentialSchema validates optional `expiresAt` field
|
||||
const data = value as IGenerateCredentialRequest;
|
||||
const result = await this.credentialService.rotateCredential(agentId, credentialId, data, ipAddress, userAgent);
|
||||
res.status(200).json(result);
|
||||
```
|
||||
|
||||
**Why `req.body ?? {}`?** The rotation body is optional — an agent may rotate a
|
||||
credential without an expiry date, in which case the body may be empty. Passing
|
||||
`undefined` to Joi would cause a different error than passing `{}`.
|
||||
|
||||
---
|
||||
|
||||
### Step 6 — Service: existence checks
|
||||
|
||||
**File:** `src/services/CredentialService.ts` lines 163–177
|
||||
|
||||
```typescript
|
||||
const agent = await this.agentRepository.findById(agentId);
|
||||
if (!agent) { throw new AgentNotFoundError(agentId) }
|
||||
|
||||
const existing = await this.credentialRepository.findById(credentialId);
|
||||
if (!existing || existing.clientId !== agentId) { throw new CredentialNotFoundError(credentialId) }
|
||||
|
||||
if (existing.status === 'revoked') {
|
||||
throw new CredentialAlreadyRevokedError(credentialId, existing.revokedAt?.toISOString() ?? ...);
|
||||
}
|
||||
```
|
||||
|
||||
**Why check `existing.clientId !== agentId`?** Even though OPA restricts the agent
|
||||
to its own credentials, a malicious actor could craft a request with a valid
|
||||
`agentId` in the path but a `credentialId` belonging to another agent. This check
|
||||
ensures that a credential is only accessible to the agent it was created for.
|
||||
|
||||
---
|
||||
|
||||
### Step 7 — Service: generate new secret and write to Vault or bcrypt
|
||||
|
||||
**File:** `src/services/CredentialService.ts` lines 180–192
|
||||
|
||||
```typescript
|
||||
const expiresAt = data.expiresAt !== undefined ? new Date(data.expiresAt) : null;
|
||||
const plainSecret = generateClientSecret(); // sk_live_<64 hex chars>
|
||||
|
||||
let updated: ICredential | null;
|
||||
|
||||
if (this.vaultClient !== null) {
|
||||
// Phase 2: overwrite the existing Vault secret (KV v2 creates a new version)
|
||||
const vaultPath = await this.vaultClient.writeSecret(agentId, credentialId, plainSecret);
|
||||
updated = await this.credentialRepository.updateVaultPath(credentialId, vaultPath, expiresAt);
|
||||
} else {
|
||||
// Phase 1: use bcrypt
|
||||
const newHash = await hashSecret(plainSecret);
|
||||
updated = await this.credentialRepository.updateHash(credentialId, newHash, expiresAt);
|
||||
}
|
||||
```
|
||||
|
||||
**Why does Vault rotation write to the same path?** Vault KV v2 is versioned — writing
|
||||
to an existing path creates a new version without overwriting previous versions.
|
||||
This preserves an audit trail in Vault itself.
|
||||
|
||||
**Why does the Vault path stay the same after rotation?** The `vault_path` column
|
||||
stores the path, not the secret. The path is deterministic:
|
||||
`{mount}/data/agentidp/agents/{agentId}/credentials/{credentialId}`. Since the
|
||||
`credentialId` does not change on rotation, the path does not change either.
|
||||
Only the Vault version at that path changes.
|
||||
|
||||
---
|
||||
|
||||
### Step 8 — Repository: UPDATE the credential
|
||||
|
||||
**File:** `src/repositories/CredentialRepository.ts` lines 180–218
|
||||
|
||||
```typescript
|
||||
// Bcrypt path (updateHash):
|
||||
UPDATE credentials
|
||||
SET secret_hash = $1, vault_path = NULL, expires_at = $2, status = 'active', revoked_at = NULL
|
||||
WHERE credential_id = $3
|
||||
RETURNING *
|
||||
|
||||
// Vault path (updateVaultPath):
|
||||
UPDATE credentials
|
||||
SET vault_path = $1, secret_hash = '', expires_at = $2, status = 'active', revoked_at = NULL
|
||||
WHERE credential_id = $3
|
||||
RETURNING *
|
||||
```
|
||||
|
||||
**Why `status = 'active'` in the UPDATE?** A credential could theoretically be
|
||||
in any state when rotated. The UPDATE explicitly sets it to active. This handles
|
||||
edge cases where a revoked credential is being "un-revoked" by rotation (though
|
||||
the service layer prevents this — revoked credentials throw `CredentialAlreadyRevokedError`).
|
||||
The belt-and-suspenders approach at the SQL layer ensures data integrity.
|
||||
|
||||
---
|
||||
|
||||
### Step 9 — Service: audit event
|
||||
|
||||
**File:** `src/services/CredentialService.ts` lines 199–206
|
||||
|
||||
```typescript
|
||||
await this.auditService.logEvent(
|
||||
agentId,
|
||||
'credential.rotated',
|
||||
'success',
|
||||
ipAddress,
|
||||
userAgent,
|
||||
{ credentialId },
|
||||
);
|
||||
```
|
||||
|
||||
The audit event records which credential was rotated. Combined with the timestamp,
|
||||
this gives a complete rotation history for each credential.
|
||||
|
||||
---
|
||||
|
||||
### Step 10 — Response
|
||||
|
||||
**File:** `src/controllers/CredentialController.ts` line 161
|
||||
|
||||
```typescript
|
||||
res.status(200).json(result);
|
||||
```
|
||||
|
||||
Returns `ICredentialWithSecret` — the updated credential including the new
|
||||
`clientSecret`. This is the only time the new secret is ever returned. The caller
|
||||
must store it securely.
|
||||
|
||||
```json
|
||||
{
|
||||
"credentialId": "d4e5f6a7-...",
|
||||
"clientId": "a1b2c3d4-...",
|
||||
"status": "active",
|
||||
"clientSecret": "sk_live_4f8a2e9b...",
|
||||
"createdAt": "2026-01-15T10:00:00Z",
|
||||
"expiresAt": "2027-01-15T10:00:00Z",
|
||||
"revokedAt": null
|
||||
}
|
||||
```
|
||||
404
docs/engineering/07-dev-setup.md
Normal file
404
docs/engineering/07-dev-setup.md
Normal file
@@ -0,0 +1,404 @@
|
||||
# 07 — Development Environment Setup
|
||||
|
||||
This guide takes you from a fresh machine to a running AgentIdP server with a
|
||||
passing smoke test. Estimated time: 15–20 minutes.
|
||||
|
||||
---
|
||||
|
||||
## 8.1 Prerequisites
|
||||
|
||||
Install all of these before proceeding.
|
||||
|
||||
| Prerequisite | Minimum version | Install link |
|
||||
|-------------|----------------|--------------|
|
||||
| Node.js | 18.x LTS | https://nodejs.org/en/download — use the LTS version |
|
||||
| npm | 9.x (ships with Node.js 18) | Included with Node.js |
|
||||
| Docker Desktop | Latest stable | https://docs.docker.com/get-docker/ |
|
||||
| Git | 2.x | https://git-scm.com/downloads |
|
||||
|
||||
**Verify your versions:**
|
||||
```bash
|
||||
node --version # Should print v18.x.x or higher
|
||||
npm --version # Should print 9.x or higher
|
||||
docker --version # Should print Docker version 24.x or higher
|
||||
git --version # Should print git version 2.x
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8.2 Clone and Install
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/sentryagent-ai/sentryagent-idp.git
|
||||
cd sentryagent-idp
|
||||
|
||||
# Install Node.js dependencies
|
||||
npm install
|
||||
```
|
||||
|
||||
This installs all production dependencies (Express, pg, Redis, etc.) and
|
||||
development dependencies (TypeScript, Jest, ts-jest, eslint).
|
||||
|
||||
---
|
||||
|
||||
## 8.3 Environment Variables Setup
|
||||
|
||||
The server requires a `.env` file at the project root. There is no `.env.example`
|
||||
file — create it from scratch using the template below.
|
||||
|
||||
```bash
|
||||
touch .env
|
||||
```
|
||||
|
||||
Add the following content to `.env`. Every variable is documented below.
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# PostgreSQL connection
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Redis connection
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
REDIS_URL=redis://localhost:6379
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# HTTP server port
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
PORT=3000
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# JWT RSA keys (generate these below)
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
JWT_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----"
|
||||
JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----"
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# CORS (optional — defaults to '*' which allows all origins)
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
CORS_ORIGIN=*
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Node environment
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
NODE_ENV=development
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# OPA policy directory (optional — defaults to ./policies)
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# POLICY_DIR=/path/to/policies
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# HashiCorp Vault (optional — omit to use bcrypt mode)
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# VAULT_ADDR=http://127.0.0.1:8200
|
||||
# VAULT_TOKEN=your-vault-token
|
||||
# VAULT_MOUNT=secret
|
||||
```
|
||||
|
||||
**Complete environment variable reference:**
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| `DATABASE_URL` | Yes | — | PostgreSQL connection string. Format: `postgresql://user:password@host:port/dbname` |
|
||||
| `REDIS_URL` | Yes | — | Redis connection URL. Format: `redis://host:port[/db]` |
|
||||
| `PORT` | No | `3000` | TCP port the HTTP server listens on |
|
||||
| `JWT_PRIVATE_KEY` | Yes | — | PEM-encoded RSA private key (2048-bit minimum) for signing tokens |
|
||||
| `JWT_PUBLIC_KEY` | Yes | — | PEM-encoded RSA public key (matching the private key above) for verifying tokens |
|
||||
| `CORS_ORIGIN` | No | `*` | CORS allowed origin. Use `*` for development, set to your dashboard domain in production |
|
||||
| `NODE_ENV` | No | `development` | Set to `test` to suppress Morgan HTTP logging during tests |
|
||||
| `POLICY_DIR` | No | `./policies` | Absolute path to the directory containing `authz.wasm` or `data/scopes.json` |
|
||||
| `VAULT_ADDR` | No | — | HashiCorp Vault server address (e.g. `http://127.0.0.1:8200`). When omitted, bcrypt mode is used |
|
||||
| `VAULT_TOKEN` | No | — | Vault authentication token. Required when `VAULT_ADDR` is set |
|
||||
| `VAULT_MOUNT` | No | `secret` | Vault KV v2 mount path. Only used when Vault is configured |
|
||||
|
||||
**Generating JWT keys:**
|
||||
```bash
|
||||
# Generate RSA 2048-bit private key
|
||||
openssl genrsa -out private.pem 2048
|
||||
|
||||
# Extract public key
|
||||
openssl rsa -in private.pem -pubout -out public.pem
|
||||
|
||||
# Print private key as single-line for .env (replace newlines with \n)
|
||||
awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' private.pem
|
||||
|
||||
# Print public key as single-line for .env
|
||||
awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' public.pem
|
||||
```
|
||||
|
||||
Paste the output (including the `-----BEGIN/END-----` lines) as the value for
|
||||
`JWT_PRIVATE_KEY` and `JWT_PUBLIC_KEY` in your `.env` file, surrounded by double
|
||||
quotes.
|
||||
|
||||
---
|
||||
|
||||
## 8.4 Docker Compose Startup and Health Checks
|
||||
|
||||
Docker Compose starts PostgreSQL 14 and Redis 7. The application reads the
|
||||
`DATABASE_URL` and `REDIS_URL` from your `.env` file to connect to them.
|
||||
|
||||
```bash
|
||||
# Start PostgreSQL and Redis in the background
|
||||
docker compose up postgres redis -d
|
||||
|
||||
# Wait for health checks to pass (usually 10-15 seconds)
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
Expected output when both services are healthy:
|
||||
```
|
||||
NAME STATUS PORTS
|
||||
sentryagent-idp-postgres-1 Up (healthy) 0.0.0.0:5432->5432/tcp
|
||||
sentryagent-idp-redis-1 Up (healthy) 0.0.0.0:6379->6379/tcp
|
||||
```
|
||||
|
||||
**Manual health check:**
|
||||
```bash
|
||||
# Test PostgreSQL connection
|
||||
docker exec sentryagent-idp-postgres-1 pg_isready -U sentryagent -d sentryagent_idp
|
||||
# Expected: /var/run/postgresql:5432 - accepting connections
|
||||
|
||||
# Test Redis connection
|
||||
docker exec sentryagent-idp-redis-1 redis-cli ping
|
||||
# Expected: PONG
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8.5 Database Migrations
|
||||
|
||||
Run the migration script to create all required tables:
|
||||
|
||||
```bash
|
||||
npm run db:migrate
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Running database migrations...
|
||||
✓ Applied: 001_create_agents.sql
|
||||
✓ Applied: 002_create_credentials.sql
|
||||
✓ Applied: 003_create_audit_events.sql
|
||||
✓ Applied: 004_create_tokens.sql
|
||||
✓ Applied: 005_add_vault_path.sql
|
||||
|
||||
Migrations complete. 5 migration(s) applied.
|
||||
```
|
||||
|
||||
Running `npm run db:migrate` a second time is safe — it skips already-applied migrations:
|
||||
```
|
||||
- Skipped (already applied): 001_create_agents.sql
|
||||
...
|
||||
Migrations complete. 0 migration(s) applied.
|
||||
```
|
||||
|
||||
**Migration internals:**
|
||||
The migration runner (`scripts/migrate.ts`) reads `.sql` files from `src/db/migrations/`
|
||||
in alphabetical order, wraps each in a transaction, and records the filename in the
|
||||
`schema_migrations` table. If a migration fails, the transaction rolls back and
|
||||
the runner exits with code 1.
|
||||
|
||||
---
|
||||
|
||||
## 8.6 Start the Server
|
||||
|
||||
```bash
|
||||
npm run dev
|
||||
# Expected: SentryAgent.ai AgentIdP listening on port 3000
|
||||
```
|
||||
|
||||
`npm run dev` uses `ts-node` to execute `src/server.ts` directly without compiling.
|
||||
This is faster for development. For a production-style start, compile first:
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
npm start
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8.7 Smoke Test
|
||||
|
||||
Verify the server is working with these three curl commands.
|
||||
|
||||
**1. Health check:**
|
||||
```bash
|
||||
curl http://localhost:3000/health
|
||||
```
|
||||
Expected response (200 OK):
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"checks": {
|
||||
"database": "healthy",
|
||||
"redis": "healthy"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**2. Register an agent:**
|
||||
First, you need a token to authenticate. But to get a token, you need credentials.
|
||||
And to get credentials, you need an agent. The chicken-and-egg is resolved by the
|
||||
fact that agent registration requires an `agents:write` scoped token — which means
|
||||
you need to bootstrap the first agent another way.
|
||||
|
||||
For local development, temporarily test without auth by using the `/api/v1` prefix
|
||||
directly (the server accepts requests; OPA will enforce scope).
|
||||
|
||||
The easiest approach: generate a test token programmatically:
|
||||
|
||||
```bash
|
||||
# Generate test keys
|
||||
openssl genrsa -out /tmp/test_private.pem 2048 2>/dev/null
|
||||
openssl rsa -in /tmp/test_private.pem -pubout -out /tmp/test_public.pem 2>/dev/null
|
||||
|
||||
# Set them in your environment temporarily
|
||||
export JWT_PRIVATE_KEY="$(cat /tmp/test_private.pem)"
|
||||
export JWT_PUBLIC_KEY="$(cat /tmp/test_public.pem)"
|
||||
|
||||
# Start server with these keys and use a tool or short Node.js script to mint a test token
|
||||
```
|
||||
|
||||
**3. Token endpoint with seeded credentials:**
|
||||
|
||||
Once you have an agent with credentials (e.g. created via the API or seeded in
|
||||
development), issue a token:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/v1/token \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "grant_type=client_credentials" \
|
||||
-d "client_id=YOUR_AGENT_ID" \
|
||||
-d "client_secret=sk_live_YOUR_SECRET" \
|
||||
-d "scope=agents:read agents:write"
|
||||
```
|
||||
|
||||
Expected response (200 OK):
|
||||
```json
|
||||
{
|
||||
"access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...",
|
||||
"token_type": "Bearer",
|
||||
"expires_in": 3600,
|
||||
"scope": "agents:read agents:write"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8.8 Troubleshooting
|
||||
|
||||
### Error: `connection refused` on PostgreSQL or Redis
|
||||
**Cause:** Docker services not running or not yet healthy.
|
||||
**Fix:**
|
||||
```bash
|
||||
docker compose ps # Check status
|
||||
docker compose up postgres redis -d # Start if not running
|
||||
docker compose logs postgres # Check for startup errors
|
||||
```
|
||||
|
||||
### Error: `DATABASE_URL environment variable is required`
|
||||
**Cause:** `.env` file missing or not being loaded.
|
||||
**Fix:** Ensure `.env` exists at the project root. `npm run dev` loads it via `dotenv.config()` in `src/server.ts`.
|
||||
|
||||
### Error: `JWT_PRIVATE_KEY and JWT_PUBLIC_KEY environment variables are required`
|
||||
**Cause:** JWT keys not in `.env`, or newlines in the PEM keys are not properly escaped.
|
||||
**Fix:** Ensure the keys are wrapped in double quotes and newlines are represented as `\n`. Use the `awk` command from section 8.3 to format them correctly.
|
||||
|
||||
### Error: `Cannot find module 'ts-node'`
|
||||
**Cause:** `npm install` was not run, or ran against a different Node.js version.
|
||||
**Fix:**
|
||||
```bash
|
||||
node --version # Confirm Node.js 18+
|
||||
rm -rf node_modules package-lock.json
|
||||
npm install
|
||||
```
|
||||
|
||||
### Error: `Cannot connect to Redis` (during migration or server start)
|
||||
**Cause:** Redis container not running or `REDIS_URL` is incorrect.
|
||||
**Fix:**
|
||||
```bash
|
||||
docker exec sentryagent-idp-redis-1 redis-cli ping
|
||||
# If container is not running:
|
||||
docker compose up redis -d
|
||||
```
|
||||
|
||||
### Port 3000 already in use
|
||||
**Cause:** Another process is listening on port 3000.
|
||||
**Fix:**
|
||||
```bash
|
||||
lsof -i :3000 # Find the process
|
||||
kill <PID> # Kill it
|
||||
# Or: set PORT=3001 in .env and restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8.9 Running Tests Locally
|
||||
|
||||
```bash
|
||||
# Run all tests (unit + integration)
|
||||
npm test
|
||||
|
||||
# Run tests with coverage report
|
||||
npm run test:unit -- --coverage
|
||||
# Coverage report: coverage/lcov-report/index.html
|
||||
|
||||
# Run only unit tests
|
||||
npm run test:unit
|
||||
|
||||
# Run only integration tests (requires running PostgreSQL and Redis)
|
||||
npm run test:integration
|
||||
|
||||
# Run a single test file
|
||||
npx jest tests/unit/services/AgentService.test.ts
|
||||
|
||||
# Run tests matching a pattern
|
||||
npx jest --testNamePattern="registerAgent"
|
||||
|
||||
# Watch mode (re-runs on file changes)
|
||||
npx jest --watch
|
||||
```
|
||||
|
||||
**Integration test requirements:**
|
||||
Integration tests connect to real PostgreSQL and Redis. Set these environment
|
||||
variables before running integration tests:
|
||||
```bash
|
||||
TEST_DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp_test
|
||||
TEST_REDIS_URL=redis://localhost:6379/1
|
||||
```
|
||||
|
||||
The integration tests create their own tables (using `CREATE TABLE IF NOT EXISTS`)
|
||||
and clean up after themselves with `DELETE FROM` statements in `afterAll`.
|
||||
|
||||
---
|
||||
|
||||
## 8.10 Web Dashboard Local Development
|
||||
|
||||
The web dashboard is a separate Vite project in the `dashboard/` directory.
|
||||
|
||||
```bash
|
||||
# From the project root
|
||||
cd dashboard
|
||||
npm install
|
||||
|
||||
# Start the Vite development server
|
||||
npm run dev
|
||||
# Dashboard available at http://localhost:5173
|
||||
```
|
||||
|
||||
The Vite dev server proxies all `/api/*` requests to `http://localhost:3000`,
|
||||
so the API server must be running concurrently (in a separate terminal).
|
||||
|
||||
**Build for production:**
|
||||
```bash
|
||||
cd dashboard
|
||||
npm run build
|
||||
# Output: dashboard/dist/ (served by Express at /dashboard)
|
||||
```
|
||||
|
||||
After building, the Express server serves the built dashboard at
|
||||
`http://localhost:3000/dashboard`. You do not need to run the Vite dev server
|
||||
for this — the static files are served directly by Express.
|
||||
372
docs/engineering/08-workflow.md
Normal file
372
docs/engineering/08-workflow.md
Normal file
@@ -0,0 +1,372 @@
|
||||
# 08 — Engineering Workflow
|
||||
|
||||
---
|
||||
|
||||
## 9.1 OpenSpec Spec-First Workflow
|
||||
|
||||
Every feature in this codebase was designed before it was implemented.
|
||||
The OpenSpec workflow enforces this order without exception.
|
||||
|
||||
### The Full Sequence
|
||||
|
||||
```
|
||||
1. CEO identifies a feature or change
|
||||
└── Documents it in the backlog
|
||||
|
||||
2. CEO approves the feature for the current sprint
|
||||
└── Creates an OpenSpec change document
|
||||
|
||||
3. Virtual Architect designs the API
|
||||
└── Writes the OpenAPI 3.0 spec BEFORE any implementation
|
||||
└── Produces spec file in docs/openapi/ or openspec/changes/<name>/specs/
|
||||
└── Every endpoint in the spec must have:
|
||||
- Summary and description
|
||||
- Request body schema (with all validation rules)
|
||||
- All response schemas (every status code)
|
||||
- Error response schemas
|
||||
- Authentication requirements
|
||||
- Example requests and responses
|
||||
|
||||
4. Virtual CTO reviews the spec
|
||||
└── Checks DRY, SOLID, AGNTCY compliance, completeness
|
||||
└── Either approves or returns with corrections
|
||||
|
||||
5. CEO approves the spec
|
||||
└── Only after CTO approval
|
||||
└── Scope changes require re-running from step 1
|
||||
|
||||
6. Virtual Principal Developer implements the spec
|
||||
└── Implementation must match the spec exactly
|
||||
└── TypeScript strict mode, DRY, SOLID, JSDoc on all public methods
|
||||
└── Zero any types
|
||||
└── All errors typed and handled
|
||||
|
||||
7. Virtual QA Engineer writes and runs tests
|
||||
└── Unit tests: >80% coverage on all services
|
||||
└── Integration tests: every endpoint in the spec tested
|
||||
└── Edge cases: null, empty, invalid inputs
|
||||
└── Performance: token endpoints <100ms, all others <200ms
|
||||
└── Verifies spec matches implementation exactly
|
||||
|
||||
8. Virtual CTO reviews the implementation and QA report
|
||||
└── If quality gates not met: returns to step 6 or 7
|
||||
└── If approved: notifies CEO
|
||||
|
||||
9. CEO approves → code is merged to develop
|
||||
└── No code ever goes to main without CEO awareness
|
||||
```
|
||||
|
||||
### Rule: No Code Without a Spec
|
||||
|
||||
If you find yourself implementing something without an approved OpenAPI spec,
|
||||
stop. Write the spec first, get it reviewed, then implement. This is not
|
||||
bureaucracy — it is how you avoid building the wrong thing.
|
||||
|
||||
---
|
||||
|
||||
## 9.2 OpenSpec CLI Commands Reference
|
||||
|
||||
OpenSpec is the change management workflow built into this project. Changes
|
||||
are tracked in `openspec/changes/`.
|
||||
|
||||
```bash
|
||||
# Create a new change (starts the design process)
|
||||
openspec new change <name>
|
||||
# Creates: openspec/changes/<name>/proposal.md, design.md, specs/, tasks.md
|
||||
|
||||
# Check the status of all active changes
|
||||
openspec status
|
||||
# Shows: change name, phase (design/implementation/review), completion %
|
||||
|
||||
# List all active changes
|
||||
openspec list
|
||||
|
||||
# Get implementation instructions for a change
|
||||
openspec instructions <name>
|
||||
# Outputs the tasks.md formatted for implementation
|
||||
|
||||
# Archive a completed change
|
||||
openspec archive <name>
|
||||
# Moves: openspec/changes/<name>/ → openspec/archive/<name>/
|
||||
```
|
||||
|
||||
### Change Lifecycle
|
||||
|
||||
```
|
||||
openspec/changes/<name>/
|
||||
├── proposal.md — Business case and feature description (CEO-authored)
|
||||
├── design.md — Technical design decisions (Architect-authored)
|
||||
├── specs/ — OpenAPI specs and interface contracts
|
||||
│ └── *.md or *.yaml
|
||||
└── tasks.md — Implementation tasks checklist (checked off as work completes)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9.3 Branching Strategy
|
||||
|
||||
### Branch naming
|
||||
|
||||
```
|
||||
feature/<short-description> — New features (from develop)
|
||||
fix/<short-description> — Bug fixes (from develop)
|
||||
docs/<short-description> — Documentation changes only (from develop)
|
||||
```
|
||||
|
||||
### Workflow
|
||||
|
||||
```
|
||||
main — Production. Only CTO-approved, CEO-aware merges.
|
||||
└── develop — Integration branch. All feature branches merge here.
|
||||
└── feature/my-feature — Your work branch
|
||||
```
|
||||
|
||||
1. Create your branch from `develop`:
|
||||
```bash
|
||||
git checkout develop
|
||||
git pull origin develop
|
||||
git checkout -b feature/my-feature
|
||||
```
|
||||
|
||||
2. Work on your branch, committing as you go.
|
||||
|
||||
3. When ready, push and open a PR targeting `develop`:
|
||||
```bash
|
||||
git push -u origin feature/my-feature
|
||||
gh pr create --base develop --title "feat: my feature" --body "..."
|
||||
```
|
||||
|
||||
4. Virtual QA reviews the PR (all quality gates must pass).
|
||||
|
||||
5. Virtual CTO approves the PR.
|
||||
|
||||
6. Merge to `develop`.
|
||||
|
||||
7. `develop` → `main` requires an explicit CEO decision.
|
||||
|
||||
**Rule:** Never push directly to `main` or `develop`. Always work through a PR.
|
||||
|
||||
---
|
||||
|
||||
## 9.4 TypeScript and Code Standards
|
||||
|
||||
### Strict Mode
|
||||
|
||||
All compiler strictness flags are enabled in `tsconfig.json`. These are
|
||||
non-negotiable:
|
||||
|
||||
```json
|
||||
{
|
||||
"strict": true,
|
||||
"noImplicitAny": true,
|
||||
"strictNullChecks": true,
|
||||
"noUnusedLocals": true,
|
||||
"noUnusedParameters": true,
|
||||
"noImplicitReturns": true
|
||||
}
|
||||
```
|
||||
|
||||
**Consequence of violation:** The TypeScript compiler (`npm run build`) will fail.
|
||||
PRs that cause build failures are rejected automatically.
|
||||
|
||||
### No `any` Types
|
||||
|
||||
Never use `any`. If a third-party library returns `unknown` or `any`, cast it to
|
||||
a specific interface you define:
|
||||
|
||||
```typescript
|
||||
// BAD
|
||||
const result: any = await vault.read(path);
|
||||
const secret = result.data.data.clientSecret;
|
||||
|
||||
// GOOD
|
||||
interface KvV2ReadResponse {
|
||||
data: { data: Record<string, string>; metadata: { version: number; } };
|
||||
}
|
||||
const result = (await vault.read(path)) as KvV2ReadResponse;
|
||||
const secret = result.data.data.clientSecret;
|
||||
```
|
||||
|
||||
### DRY (Don't Repeat Yourself)
|
||||
|
||||
Zero code duplication. See `04-codebase-structure.md` section 5.5 for the complete
|
||||
mapping of what lives where. Before writing a utility function, check whether
|
||||
it already exists in `src/utils/`.
|
||||
|
||||
### SOLID Principles
|
||||
|
||||
Each service has a single, clear responsibility. If you find yourself adding a
|
||||
method to `AgentService` that queries the `audit_events` table, stop — that
|
||||
belongs in `AuditService`. If you find yourself adding SQL to a controller, stop —
|
||||
that belongs in a repository.
|
||||
|
||||
### JSDoc on All Public Methods
|
||||
|
||||
Every public class, method, and interface must have a JSDoc comment that includes:
|
||||
- `@param` for every parameter
|
||||
- `@returns` describing the return value
|
||||
- `@throws` for every error that can be thrown
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Registers a new AI agent identity.
|
||||
*
|
||||
* @param data - Agent registration request data.
|
||||
* @param ipAddress - Client IP for audit logging.
|
||||
* @param userAgent - Client User-Agent for audit logging.
|
||||
* @returns The newly created agent record.
|
||||
* @throws FreeTierLimitError if the 100-agent limit is reached.
|
||||
* @throws AgentAlreadyExistsError if the email is already registered.
|
||||
*/
|
||||
async registerAgent(data: ICreateAgentRequest, ipAddress: string, userAgent: string): Promise<IAgent>
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
Always throw a typed error from the `SentryAgentError` hierarchy. Never throw
|
||||
raw `Error` objects with string messages in service or controller code:
|
||||
|
||||
```typescript
|
||||
// BAD
|
||||
throw new Error('Agent not found');
|
||||
|
||||
// GOOD
|
||||
throw new AgentNotFoundError(agentId);
|
||||
```
|
||||
|
||||
The `errorHandler` middleware maps `SentryAgentError` subclasses to HTTP status
|
||||
codes automatically. Adding a new error type only requires adding a class in
|
||||
`src/utils/errors.ts`.
|
||||
|
||||
---
|
||||
|
||||
## 9.5 PR Checklist
|
||||
|
||||
Every pull request must pass all items before it can be merged.
|
||||
|
||||
```
|
||||
Code quality:
|
||||
- [ ] TypeScript builds without errors: npm run build
|
||||
- [ ] No any types introduced
|
||||
- [ ] All new public methods have JSDoc
|
||||
- [ ] ESLint passes: npm run lint
|
||||
- [ ] No code duplication — logic extracted to utils/services
|
||||
|
||||
Testing:
|
||||
- [ ] Unit tests added for all new service methods
|
||||
- [ ] Integration tests added for all new API endpoints
|
||||
- [ ] Coverage threshold maintained: npm run test:unit -- --coverage
|
||||
(>80% statements, branches, functions, lines)
|
||||
- [ ] Integration tests pass against real PostgreSQL and Redis
|
||||
|
||||
Spec compliance:
|
||||
- [ ] Implementation matches the approved OpenAPI spec exactly
|
||||
- [ ] If the spec needs updating, the spec was updated BEFORE the code was changed
|
||||
|
||||
Documentation:
|
||||
- [ ] docs/engineering/ updated if the change affects service interfaces or workflows
|
||||
- [ ] CHANGELOG.md updated with a summary of the change
|
||||
- [ ] Any new environment variables documented in docs/engineering/07-dev-setup.md
|
||||
|
||||
Database:
|
||||
- [ ] If a new table or column is added, a migration file exists in src/db/migrations/
|
||||
- [ ] Migration file is numbered correctly and tested locally
|
||||
|
||||
Review:
|
||||
- [ ] Virtual CTO reviewed the implementation
|
||||
- [ ] Virtual QA signed off on tests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9.6 Virtual Engineering Team Roles for Contributors
|
||||
|
||||
External contributors operate within the same team structure as the internal
|
||||
Virtual Engineering Team. Here is how to interact with each role:
|
||||
|
||||
**Virtual CTO (architecture gate)**
|
||||
Opens architectural discussions by filing a GitHub issue labeled `architecture`.
|
||||
The Virtual CTO must approve any change to:
|
||||
- The layered architecture (adding a direct DB call in a controller)
|
||||
- The error hierarchy
|
||||
- The authentication or authorisation flow
|
||||
- Any new dependency (package)
|
||||
- Multi-region deployment topology
|
||||
|
||||
**Virtual Architect (spec owner)**
|
||||
All API changes require an updated OpenAPI spec. If you are adding an endpoint,
|
||||
file an issue labeled `spec-change` with your proposed additions before writing
|
||||
any code. The Architect will review and approve the spec.
|
||||
|
||||
**Virtual Principal Developer (code reviewer)**
|
||||
All implementation PRs are reviewed for TypeScript compliance, DRY violations,
|
||||
SOLID violations, JSDoc completeness, and correctness against the spec.
|
||||
|
||||
**Virtual QA Engineer (quality gate)**
|
||||
All PRs require >80% coverage and passing integration tests. The QA Engineer
|
||||
will review test completeness and flag edge cases that need coverage.
|
||||
|
||||
---
|
||||
|
||||
## 9.7 Commit Message Conventions
|
||||
|
||||
This project uses **Conventional Commits** (https://www.conventionalcommits.org).
|
||||
|
||||
### Format
|
||||
|
||||
```
|
||||
<type>(<optional scope>): <short description>
|
||||
|
||||
<optional body — why this change was made>
|
||||
|
||||
<optional footer — breaking changes, issue references>
|
||||
```
|
||||
|
||||
### Types
|
||||
|
||||
| Type | When to use | Example |
|
||||
|------|-------------|---------|
|
||||
| `feat` | A new feature | `feat(agents): add agent status filter to list endpoint` |
|
||||
| `fix` | A bug fix | `fix(auth): handle TokenExpiredError separately from JsonWebTokenError` |
|
||||
| `docs` | Documentation only | `docs(engineering): add credential rotation walkthrough` |
|
||||
| `test` | Adding or updating tests | `test(oauth2): add monthly token limit integration test` |
|
||||
| `chore` | Build, tooling, dependencies | `chore: update jest to 29.7.0` |
|
||||
| `refactor` | Code change with no behaviour change | `refactor(credential): extract secret storage to VaultClient` |
|
||||
| `perf` | Performance improvement | `perf(token): make audit log write fire-and-forget` |
|
||||
|
||||
### Rules
|
||||
|
||||
- Keep the description under 72 characters
|
||||
- Use the imperative mood: "add" not "added" or "adds"
|
||||
- Include the scope in parentheses when the change is limited to one area
|
||||
- Reference issues in the footer: `Closes #123`
|
||||
|
||||
### Examples
|
||||
|
||||
```
|
||||
feat(vault): add optional HashiCorp Vault credential backend
|
||||
|
||||
Adds VaultClient wrapping node-vault for KV v2 operations.
|
||||
When VAULT_ADDR and VAULT_TOKEN are set, new credentials are
|
||||
stored in Vault instead of as bcrypt hashes in PostgreSQL.
|
||||
|
||||
Backwards compatible: existing bcrypt credentials continue to work.
|
||||
|
||||
Closes #45
|
||||
```
|
||||
|
||||
```
|
||||
fix(opa): normalise /token/revoke path before OPA lookup
|
||||
|
||||
The path /api/v1/token/revoke was not in the normalisation
|
||||
switch, causing OPA to deny all revocation requests even with
|
||||
the correct scope.
|
||||
```
|
||||
|
||||
```
|
||||
docs: add engineering knowledge base for new hires
|
||||
|
||||
Adds docs/engineering/ with 11 documents covering architecture,
|
||||
service deep-dives, code walkthroughs, dev setup, workflow,
|
||||
testing, deployment, and SDK guide.
|
||||
```
|
||||
424
docs/engineering/09-testing.md
Normal file
424
docs/engineering/09-testing.md
Normal file
@@ -0,0 +1,424 @@
|
||||
# 09 — Testing Strategy
|
||||
|
||||
---
|
||||
|
||||
## 10.1 Test Types and Purposes
|
||||
|
||||
This codebase uses two types of tests. Understanding when to use each prevents
|
||||
you from writing integration tests for things that should be unit tests (slow)
|
||||
and unit tests for things that need a real database (misleading).
|
||||
|
||||
### Unit Tests
|
||||
|
||||
**Location:** `tests/unit/`
|
||||
|
||||
**What they test:** A single class or function in complete isolation. All
|
||||
dependencies (repositories, services, external clients) are replaced with Jest mocks.
|
||||
|
||||
**When to use:**
|
||||
- Testing service business logic (free-tier limits, status transitions, error cases)
|
||||
- Testing utility functions (crypto, jwt, validators)
|
||||
- Testing error hierarchy behaviour
|
||||
- Any code that has conditional logic you want to test exhaustively
|
||||
|
||||
**What they do NOT test:**
|
||||
- Whether the SQL queries are correct
|
||||
- Whether the HTTP routing works
|
||||
- Whether middleware chains execute in the right order
|
||||
|
||||
**Speed:** Milliseconds. Hundreds of unit tests should complete in under 10 seconds.
|
||||
|
||||
### Integration Tests
|
||||
|
||||
**Location:** `tests/integration/`
|
||||
|
||||
**What they test:** A full HTTP request through the Express application against
|
||||
a real PostgreSQL database and real Redis instance.
|
||||
|
||||
**When to use:**
|
||||
- Testing that a route is correctly wired to the right controller method
|
||||
- Testing authentication and authorisation middleware in combination
|
||||
- Testing database operations end-to-end (INSERT → read back → verify)
|
||||
- Testing response shapes match the OpenAPI spec exactly
|
||||
|
||||
**What they require:**
|
||||
- Running PostgreSQL (at `TEST_DATABASE_URL` or default)
|
||||
- Running Redis (at `TEST_REDIS_URL` or default)
|
||||
- The test creates its own tables and cleans up after every test case
|
||||
|
||||
**Speed:** Seconds. Expect 2–5 seconds per integration test file.
|
||||
|
||||
---
|
||||
|
||||
## 10.2 Test Framework Stack
|
||||
|
||||
| Tool | Role |
|
||||
|------|------|
|
||||
| **Jest 29.7** | Test runner. `describe`, `it`, `expect`, `beforeEach`, `afterAll`. Also provides mocking via `jest.mock()`, `jest.fn()`, `jest.spyOn()`. |
|
||||
| **ts-jest** | Transforms TypeScript test files for Jest without a separate compilation step. Configured in `jest.config.ts`. |
|
||||
| **Supertest 6.3** | HTTP testing library. Used in integration tests to make real HTTP requests against the Express app without opening a network port. Works by passing the `Application` object directly. |
|
||||
|
||||
**Jest configuration** (`jest.config.ts`):
|
||||
```typescript
|
||||
export default {
|
||||
preset: 'ts-jest',
|
||||
testEnvironment: 'node',
|
||||
roots: ['<rootDir>/tests'],
|
||||
testPathPattern: ['tests/unit', 'tests/integration'],
|
||||
collectCoverageFrom: ['src/**/*.ts', '!src/server.ts'],
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10.3 Coverage Gates
|
||||
|
||||
All four coverage metrics must be above 80% before a feature is considered complete:
|
||||
|
||||
| Metric | Gate | What it means |
|
||||
|--------|------|---------------|
|
||||
| Statements | >80% | Each statement was executed at least once |
|
||||
| Branches | >80% | Each `if`/`else`/`switch` branch was taken at least once |
|
||||
| Functions | >80% | Each function was called at least once |
|
||||
| Lines | >80% | Each line was executed at least once |
|
||||
|
||||
**Enforcement:**
|
||||
|
||||
Coverage is checked in the PR process:
|
||||
```bash
|
||||
npm run test:unit -- --coverage
|
||||
# Fails if any metric is below 80%
|
||||
```
|
||||
|
||||
Coverage reports are output to `coverage/lcov-report/index.html` for visual inspection.
|
||||
|
||||
The coverage threshold configuration is in `jest.config.ts`:
|
||||
```typescript
|
||||
coverageThreshold: {
|
||||
global: {
|
||||
statements: 80,
|
||||
branches: 80,
|
||||
functions: 80,
|
||||
lines: 80,
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10.4 How to Run the Test Suite
|
||||
|
||||
```bash
|
||||
# Run all tests (unit + integration)
|
||||
npm test
|
||||
|
||||
# Run only unit tests
|
||||
npm run test:unit
|
||||
|
||||
# Run only integration tests
|
||||
npm run test:integration
|
||||
|
||||
# Run unit tests with coverage report
|
||||
npm run test:unit -- --coverage
|
||||
# HTML report: coverage/lcov-report/index.html
|
||||
|
||||
# Run a single test file
|
||||
npx jest tests/unit/services/AgentService.test.ts
|
||||
|
||||
# Run tests matching a name pattern
|
||||
npx jest --testNamePattern="should throw FreeTierLimitError"
|
||||
|
||||
# Run tests in watch mode (re-runs on file changes)
|
||||
npx jest --watch
|
||||
|
||||
# Run with verbose output (shows each test name)
|
||||
npx jest --verbose
|
||||
```
|
||||
|
||||
**Integration test environment variables:**
|
||||
```bash
|
||||
export TEST_DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp_test
|
||||
export TEST_REDIS_URL=redis://localhost:6379/1
|
||||
npm run test:integration
|
||||
```
|
||||
|
||||
Using database index `/1` for Redis in tests prevents test runs from polluting
|
||||
the main database (index `0`) used for local development.
|
||||
|
||||
---
|
||||
|
||||
## 10.5 Unit Test Writing Conventions
|
||||
|
||||
Unit tests follow a strict pattern. Study this example carefully — it shows every
|
||||
convention in use.
|
||||
|
||||
**Real example from `tests/unit/services/AgentService.test.ts`:**
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Unit tests for src/services/AgentService.ts
|
||||
*/
|
||||
|
||||
import { AgentService } from '../../../src/services/AgentService';
|
||||
import { AgentRepository } from '../../../src/repositories/AgentRepository';
|
||||
import { CredentialRepository } from '../../../src/repositories/CredentialRepository';
|
||||
import { AuditService } from '../../../src/services/AuditService';
|
||||
import {
|
||||
AgentAlreadyExistsError,
|
||||
FreeTierLimitError,
|
||||
} from '../../../src/utils/errors';
|
||||
import { IAgent, ICreateAgentRequest } from '../../../src/types/index';
|
||||
|
||||
// Mock all dependencies — none of them execute real code
|
||||
jest.mock('../../../src/repositories/AgentRepository');
|
||||
jest.mock('../../../src/repositories/CredentialRepository');
|
||||
jest.mock('../../../src/services/AuditService');
|
||||
|
||||
// Get typed mock constructors so we can call .mockResolvedValue() on them
|
||||
const MockAgentRepository = AgentRepository as jest.MockedClass<typeof AgentRepository>;
|
||||
const MockCredentialRepository = CredentialRepository as jest.MockedClass<typeof CredentialRepository>;
|
||||
const MockAuditService = AuditService as jest.MockedClass<typeof AuditService>;
|
||||
|
||||
// Define a complete test fixture — reuse this instead of duplicating object literals
|
||||
const MOCK_AGENT: IAgent = {
|
||||
agentId: 'a1b2c3d4-e5f6-7890-abcd-ef1234567890',
|
||||
email: 'agent@sentryagent.ai',
|
||||
agentType: 'screener',
|
||||
version: '1.0.0',
|
||||
capabilities: ['resume:read'],
|
||||
owner: 'team-a',
|
||||
deploymentEnv: 'production',
|
||||
status: 'active',
|
||||
createdAt: new Date('2026-03-28T09:00:00Z'),
|
||||
updatedAt: new Date('2026-03-28T09:00:00Z'),
|
||||
};
|
||||
|
||||
describe('AgentService', () => {
|
||||
let agentService: AgentService;
|
||||
let agentRepo: jest.Mocked<AgentRepository>;
|
||||
let credentialRepo: jest.Mocked<CredentialRepository>;
|
||||
let auditService: jest.Mocked<AuditService>;
|
||||
|
||||
beforeEach(() => {
|
||||
// Clear all mocks before each test — prevents state leakage
|
||||
jest.clearAllMocks();
|
||||
// Create fresh mock instances for each test
|
||||
agentRepo = new MockAgentRepository({} as never) as jest.Mocked<AgentRepository>;
|
||||
credentialRepo = new MockCredentialRepository({} as never) as jest.Mocked<CredentialRepository>;
|
||||
auditService = new MockAuditService({} as never) as jest.Mocked<AuditService>;
|
||||
// Inject mocks into the system under test
|
||||
agentService = new AgentService(agentRepo, credentialRepo, auditService);
|
||||
});
|
||||
|
||||
describe('registerAgent()', () => {
|
||||
const createData: ICreateAgentRequest = {
|
||||
email: 'agent@sentryagent.ai',
|
||||
agentType: 'screener',
|
||||
version: '1.0.0',
|
||||
capabilities: ['resume:read'],
|
||||
owner: 'team-a',
|
||||
deploymentEnv: 'production',
|
||||
};
|
||||
|
||||
it('should create and return a new agent', async () => {
|
||||
// Arrange — set up mock return values
|
||||
agentRepo.countActive.mockResolvedValue(0);
|
||||
agentRepo.findByEmail.mockResolvedValue(null);
|
||||
agentRepo.create.mockResolvedValue(MOCK_AGENT);
|
||||
auditService.logEvent.mockResolvedValue({} as never);
|
||||
|
||||
// Act — call the method under test
|
||||
const result = await agentService.registerAgent(createData, '127.0.0.1', 'test/1.0');
|
||||
|
||||
// Assert — verify the result
|
||||
expect(result).toEqual(MOCK_AGENT);
|
||||
// Also verify the mock was called with the right arguments
|
||||
expect(agentRepo.create).toHaveBeenCalledWith(createData);
|
||||
});
|
||||
|
||||
it('should throw FreeTierLimitError when 100 agents already registered', async () => {
|
||||
// Arrange — simulate limit reached
|
||||
agentRepo.countActive.mockResolvedValue(100);
|
||||
|
||||
// Assert error — rejects.toThrow checks the error type
|
||||
await expect(agentService.registerAgent(createData, '127.0.0.1', 'test/1.0'))
|
||||
.rejects.toThrow(FreeTierLimitError);
|
||||
});
|
||||
|
||||
it('should throw AgentAlreadyExistsError if email is already registered', async () => {
|
||||
agentRepo.countActive.mockResolvedValue(0);
|
||||
agentRepo.findByEmail.mockResolvedValue(MOCK_AGENT); // Simulate existing agent
|
||||
|
||||
await expect(agentService.registerAgent(createData, '127.0.0.1', 'test/1.0'))
|
||||
.rejects.toThrow(AgentAlreadyExistsError);
|
||||
});
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Conventions explained:
|
||||
|
||||
1. **One test file per source file.** `AgentService.test.ts` tests `AgentService.ts`.
|
||||
2. **`jest.mock()` before any imports from the mocked module.** Jest hoists mock declarations.
|
||||
3. **`jest.clearAllMocks()` in `beforeEach`.** Prevents mock call counts from leaking between tests.
|
||||
4. **AAA pattern (Arrange, Act, Assert).** Every `it` block follows this order.
|
||||
5. **Test both the happy path and every error case.** A service with 3 error conditions
|
||||
needs at least 4 tests (1 success + 3 failures).
|
||||
6. **Verify mock calls for side effects.** Use `.toHaveBeenCalledWith()` to verify that
|
||||
`auditService.logEvent` was called with the right arguments, not just that it was called.
|
||||
7. **Use typed error assertions.** `.rejects.toThrow(FreeTierLimitError)` verifies the
|
||||
error type, not just a message string.
|
||||
|
||||
---
|
||||
|
||||
## 10.6 Integration Test Writing Conventions
|
||||
|
||||
Integration tests use Supertest to make real HTTP requests against a live Express app.
|
||||
|
||||
**Real example from `tests/integration/agents.test.ts`:**
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Integration tests for Agent Registry endpoints.
|
||||
*/
|
||||
|
||||
import crypto from 'crypto';
|
||||
import request from 'supertest';
|
||||
import { Application } from 'express';
|
||||
import { v4 as uuidv4 } from 'uuid';
|
||||
import { Pool } from 'pg';
|
||||
|
||||
// Generate RSA keys for test tokens — done once per test module
|
||||
const { privateKey, publicKey } = crypto.generateKeyPairSync('rsa', {
|
||||
modulusLength: 2048,
|
||||
publicKeyEncoding: { type: 'spki', format: 'pem' },
|
||||
privateKeyEncoding: { type: 'pkcs8', format: 'pem' },
|
||||
});
|
||||
|
||||
// Set environment variables BEFORE importing the app
|
||||
process.env['DATABASE_URL'] = process.env['TEST_DATABASE_URL'] ?? 'postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp_test';
|
||||
process.env['REDIS_URL'] = process.env['TEST_REDIS_URL'] ?? 'redis://localhost:6379/1';
|
||||
process.env['JWT_PRIVATE_KEY'] = privateKey;
|
||||
process.env['JWT_PUBLIC_KEY'] = publicKey;
|
||||
process.env['NODE_ENV'] = 'test';
|
||||
|
||||
import { createApp } from '../../src/app';
|
||||
import { signToken } from '../../src/utils/jwt';
|
||||
import { closePool } from '../../src/db/pool';
|
||||
import { closeRedisClient } from '../../src/cache/redis';
|
||||
|
||||
// Helper: mint a valid test token
|
||||
function makeToken(sub: string = uuidv4(), scope: string = 'agents:read agents:write'): string {
|
||||
return signToken({ sub, client_id: sub, scope, jti: uuidv4() }, privateKey);
|
||||
}
|
||||
|
||||
describe('Agent Registry Integration Tests', () => {
|
||||
let app: Application;
|
||||
let pool: Pool;
|
||||
|
||||
beforeAll(async () => {
|
||||
// Boot the real Express app
|
||||
app = await createApp();
|
||||
pool = new Pool({ connectionString: process.env['DATABASE_URL'] });
|
||||
|
||||
// Create test tables (idempotent)
|
||||
await pool.query(`CREATE TABLE IF NOT EXISTS agents (...)`);
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
// Clean up after each test — order matters (foreign key constraints)
|
||||
await pool.query('DELETE FROM audit_events');
|
||||
await pool.query('DELETE FROM credentials');
|
||||
await pool.query('DELETE FROM agents');
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
// Close all connections — prevents Jest from hanging
|
||||
await pool.end();
|
||||
await closePool();
|
||||
await closeRedisClient();
|
||||
});
|
||||
|
||||
describe('POST /api/v1/agents', () => {
|
||||
it('should register a new agent and return 201', async () => {
|
||||
const token = makeToken();
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/v1/agents')
|
||||
.set('Authorization', `Bearer ${token}`)
|
||||
.send({
|
||||
email: 'test-agent@sentryagent.ai',
|
||||
agentType: 'screener',
|
||||
version: '1.0.0',
|
||||
capabilities: ['resume:read'],
|
||||
owner: 'test-team',
|
||||
deploymentEnv: 'development',
|
||||
});
|
||||
|
||||
expect(res.status).toBe(201);
|
||||
expect(res.body.agentId).toBeDefined();
|
||||
expect(res.body.email).toBe('test-agent@sentryagent.ai');
|
||||
expect(res.body.status).toBe('active');
|
||||
});
|
||||
|
||||
it('should return 401 without a token', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/v1/agents')
|
||||
.send({ email: 'test@sentryagent.ai' });
|
||||
|
||||
expect(res.status).toBe(401);
|
||||
});
|
||||
|
||||
it('should return 409 for duplicate email', async () => {
|
||||
const token = makeToken();
|
||||
const body = { email: 'dup@sentryagent.ai', agentType: 'screener', version: '1.0', capabilities: [], owner: 'team', deploymentEnv: 'development' };
|
||||
|
||||
await request(app).post('/api/v1/agents').set('Authorization', `Bearer ${token}`).send(body);
|
||||
const res = await request(app).post('/api/v1/agents').set('Authorization', `Bearer ${token}`).send(body);
|
||||
|
||||
expect(res.status).toBe(409);
|
||||
expect(res.body.code).toBe('AGENT_ALREADY_EXISTS');
|
||||
});
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Conventions explained:
|
||||
|
||||
1. **Set `process.env` before importing the app.** The app reads env vars at import
|
||||
time (`getPool()`, JWT keys). Setting them after import does nothing.
|
||||
2. **`afterEach` cleanup.** Delete all rows after each test so tests are independent.
|
||||
Always delete in child-to-parent order (audit_events → credentials → agents)
|
||||
to respect foreign key constraints.
|
||||
3. **`afterAll` close connections.** Always close the pool and Redis client at the end
|
||||
of the suite. Jest will hang if connections remain open.
|
||||
4. **Test both success and failure status codes.** Every endpoint test must include
|
||||
an unauthenticated request (401) and an invalid request (400).
|
||||
5. **Verify response body shape.** Check `res.body.code` for error responses to
|
||||
verify the correct error type, not just the status code.
|
||||
6. **Use `makeToken()` for test tokens.** A helper function keeps token creation
|
||||
consistent across all integration test files.
|
||||
|
||||
---
|
||||
|
||||
## 10.7 OWASP Top 10 Security Testing Reference
|
||||
|
||||
These are the security concerns most relevant to an identity provider. For each,
|
||||
here is what AgentIdP does to mitigate the risk and how to test it.
|
||||
|
||||
| OWASP Category | Relevant risk | Mitigation | Test approach |
|
||||
|---------------|--------------|-----------|---------------|
|
||||
| **A01 Broken Access Control** | Agent A accesses agent B's credentials | `req.user.sub !== agentId` check in all credential endpoints | Test: send credential request with a token for agent A but agentId for agent B in the path — expect 403 |
|
||||
| **A02 Cryptographic Failures** | Weak credential secrets or JWT algorithm | `sk_live_<64 hex>` = 256-bit entropy; RS256 signing; bcrypt 10 rounds | Test: verify generated secrets are 72 chars; verify JWT header shows `alg: RS256` |
|
||||
| **A03 Injection** | SQL injection via input fields | Parameterised queries (`$1, $2, ...`) in all repositories | Test: send `'; DROP TABLE agents; --` as `owner` field — expect 400 from Joi validation |
|
||||
| **A05 Security Misconfiguration** | Server leaking stack traces | `errorHandler` returns generic 500 for unknown errors | Test: trigger an unexpected error (mock a repository to throw `new Error()`) — verify response body does not contain stack trace |
|
||||
| **A06 Vulnerable Components** | Outdated dependencies with CVEs | Regular `npm audit` | Run: `npm audit` in CI; fail on high/critical findings |
|
||||
| **A07 Auth Failures** | Timing attack on credential verification | `crypto.timingSafeEqual` in `VaultClient.verifySecret()`; bcrypt inherently timing-safe | Test: measure multiple failed verification attempts with wrong secrets of varying lengths — timing should not increase linearly with shared prefix length |
|
||||
| **A08 Integrity Failures** | Forged JWT tokens | RS256 verification rejects tokens signed with wrong key | Test: create a token signed with a different private key — expect 401 |
|
||||
| **A09 Logging Failures** | Auth failures not logged | `auth.failed` audit events written for every authentication failure | Test: attempt token issuance with wrong secret — verify `auth_events` table contains `auth.failed` row |
|
||||
| **A10 SSRF** | Not applicable to current API surface | No outbound HTTP from user-supplied URLs | N/A — no URL-accepting fields in current API |
|
||||
|
||||
**JWT algorithm confusion (bonus):**
|
||||
Test that the server rejects tokens with `alg: none` or `alg: HS256`. The
|
||||
`verifyToken()` function specifies `algorithms: ['RS256']`, which causes jsonwebtoken
|
||||
to reject any token with a different algorithm header.
|
||||
273
docs/engineering/10-deployment.md
Normal file
273
docs/engineering/10-deployment.md
Normal file
@@ -0,0 +1,273 @@
|
||||
# 10 — Deployment and Operations
|
||||
|
||||
This document covers building and running AgentIdP in production: Docker, environment variables, database migrations, Terraform multi-region deployment, Prometheus/Grafana monitoring, and operational runbooks for common incidents.
|
||||
|
||||
---
|
||||
|
||||
## 1. Docker Build and Run
|
||||
|
||||
The Dockerfile uses a two-stage build:
|
||||
|
||||
- **Stage 1 (builder):** `node:18-alpine` — installs all dependencies (including dev) and compiles TypeScript to `dist/`.
|
||||
- **Stage 2 (production):** `node:18-alpine` — copies `dist/` and `node_modules` (production only), runs as the built-in non-root `node` user.
|
||||
|
||||
```bash
|
||||
# Build
|
||||
docker build -t sentryagent-idp:latest .
|
||||
|
||||
# Run (supply required env vars)
|
||||
docker run -d \
|
||||
-p 3000:3000 \
|
||||
-e DATABASE_URL=postgresql://sentryagent:sentryagent@<host>:5432/sentryagent_idp \
|
||||
-e REDIS_URL=redis://<host>:6379 \
|
||||
-e JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n..." \
|
||||
-e JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\n..." \
|
||||
sentryagent-idp:latest
|
||||
```
|
||||
|
||||
The container exposes port `3000`. Override with `PORT` environment variable if needed.
|
||||
|
||||
For local full-stack development, use Docker Compose instead:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
The `docker-compose.yml` starts the app, PostgreSQL 14, and Redis 7 with health checks and data volumes.
|
||||
|
||||
---
|
||||
|
||||
## 2. Environment Variables Reference
|
||||
|
||||
All variables are loaded at startup via `dotenv`. In production, inject them directly into the process environment — do not commit `.env` to version control.
|
||||
|
||||
| Variable | Required | Default | Purpose |
|
||||
|----------|----------|---------|---------|
|
||||
| `DATABASE_URL` | Yes | — | PostgreSQL connection string. Format: `postgresql://<user>:<password>@<host>:<port>/<db>` |
|
||||
| `REDIS_URL` | Yes | — | Redis connection URL. Format: `redis://<host>:<port>` |
|
||||
| `JWT_PRIVATE_KEY` | Yes | — | PEM-encoded RSA-2048 private key for signing RS256 JWT tokens |
|
||||
| `JWT_PUBLIC_KEY` | Yes | — | PEM-encoded RSA-2048 public key for verifying tokens on every authenticated request |
|
||||
| `PORT` | No | `3000` | HTTP port the Express server listens on |
|
||||
| `NODE_ENV` | No | `undefined` | Set to `production` in production, `test` in test (disables Morgan logging in test) |
|
||||
| `CORS_ORIGIN` | No | `*` | Allowed CORS origin(s). Set to specific URL in production (e.g. `https://app.mycompany.ai`) |
|
||||
| `VAULT_ADDR` | No | — | HashiCorp Vault server address. When set with `VAULT_TOKEN`, new credentials are stored in Vault KV v2 instead of bcrypt |
|
||||
| `VAULT_TOKEN` | No | — | Vault authentication token. Required when `VAULT_ADDR` is set |
|
||||
| `VAULT_MOUNT` | No | `secret` | KV v2 secrets engine mount path |
|
||||
| `POLICY_DIR` | No | `<cwd>/policies` | Directory containing OPA policy files (`authz.wasm` or `data/scopes.json`) |
|
||||
|
||||
**Validation at startup:** `JWT_PRIVATE_KEY` and `JWT_PUBLIC_KEY` are checked in `createApp()` (see `src/app.ts:117–121`). If missing, the process exits before binding to any port. `DATABASE_URL` and `REDIS_URL` are validated when their respective singletons are first initialised.
|
||||
|
||||
---
|
||||
|
||||
## 3. Database Migrations
|
||||
|
||||
Migrations are plain SQL files in `src/db/migrations/`. They are append-only — never modify an existing migration file. Always create a new numbered file.
|
||||
|
||||
Current migration files:
|
||||
|
||||
| File | What it creates |
|
||||
|------|----------------|
|
||||
| `001_create_agents.sql` | `agents` table with UUID primary key, email unique constraint, status enum |
|
||||
| `002_create_credentials.sql` | `credentials` table linked to `agents` by `client_id` foreign key |
|
||||
| `003_create_audit_events.sql` | `audit_events` table with JSONB `metadata` column |
|
||||
| `004_create_tokens.sql` | `token_monthly_counts` table for free-tier token limit tracking |
|
||||
| `005_add_vault_path.sql` | Adds `vault_path VARCHAR(512)` column to the `credentials` table |
|
||||
|
||||
**Run migrations:**
|
||||
|
||||
```bash
|
||||
npm run db:migrate
|
||||
```
|
||||
|
||||
This executes `scripts/migrate.ts` which applies all SQL files that have not yet been recorded in the `schema_migrations` tracking table.
|
||||
|
||||
**Adding a new migration:**
|
||||
|
||||
1. Create `src/db/migrations/006_<description>.sql`
|
||||
2. Write idempotent SQL (use `CREATE TABLE IF NOT EXISTS`, `ADD COLUMN IF NOT EXISTS`, etc.)
|
||||
3. Run `npm run db:migrate`
|
||||
|
||||
---
|
||||
|
||||
## 4. Terraform Multi-Region Deployment
|
||||
|
||||
The `terraform/` directory contains reusable modules and two environment configurations.
|
||||
|
||||
**Directory structure:**
|
||||
|
||||
```
|
||||
terraform/
|
||||
modules/
|
||||
agentidp/ # Core AgentIdP compute resources
|
||||
lb/ # Load balancer (ALB/Cloud Load Balancer)
|
||||
rds/ # RDS PostgreSQL (AWS)
|
||||
redis/ # ElastiCache Redis (AWS) / Memorystore (GCP)
|
||||
environments/
|
||||
aws/ # AWS deployment (ECS Fargate, ALB, RDS, ElastiCache)
|
||||
gcp/ # GCP deployment (Cloud Run, Cloud SQL, Memorystore)
|
||||
```
|
||||
|
||||
### AWS Deployment
|
||||
|
||||
Architecture: `Internet → Route 53 → ALB (public subnets, HTTPS) → ECS Fargate tasks (private subnets) → RDS PostgreSQL 14 (Multi-AZ) + ElastiCache Redis 7`
|
||||
|
||||
All secrets are stored in AWS Secrets Manager and injected into ECS task definitions at launch time.
|
||||
|
||||
```bash
|
||||
cd terraform/environments/aws
|
||||
terraform init
|
||||
terraform plan -var="aws_region=us-east-1"
|
||||
terraform apply
|
||||
```
|
||||
|
||||
Resources provisioned:
|
||||
- VPC with public and private subnets across multiple availability zones
|
||||
- ECS Cluster and Fargate task definition (running `sentryagent-idp` container)
|
||||
- Application Load Balancer with HTTPS listener and health check target group
|
||||
- RDS PostgreSQL 14 (Multi-AZ for high availability)
|
||||
- ElastiCache Redis 7 (primary + replica)
|
||||
- IAM roles and instance profiles for ECS task permissions
|
||||
- Security groups enforcing least-privilege network access
|
||||
|
||||
### GCP Deployment
|
||||
|
||||
Architecture: `Internet → Cloud Run (Google-managed TLS, auto-scaling) → Cloud SQL PostgreSQL 14 (REGIONAL HA) + Memorystore Redis 7 (STANDARD_HA)`
|
||||
|
||||
All secrets are stored in GCP Secret Manager and mounted into the Cloud Run service at startup.
|
||||
|
||||
```bash
|
||||
cd terraform/environments/gcp
|
||||
terraform init
|
||||
terraform plan -var="gcp_region=us-central1"
|
||||
terraform apply
|
||||
```
|
||||
|
||||
Resources provisioned:
|
||||
- VPC network with Serverless VPC Access connector (Cloud Run → private databases)
|
||||
- Cloud Run service (auto-scales to zero, Google-managed TLS)
|
||||
- Cloud Load Balancer with global anycast IP
|
||||
- Cloud SQL PostgreSQL 14 with regional high-availability
|
||||
- Memorystore Redis 7 (STANDARD_HA with in-transit encryption)
|
||||
- IAM service accounts and bindings
|
||||
|
||||
**Important:** All infrastructure changes must go through Terraform. Never make manual edits in the AWS console or GCP Cloud Console — they will be overwritten on the next `terraform apply` and will not be tracked in state.
|
||||
|
||||
---
|
||||
|
||||
## 5. Prometheus and Grafana
|
||||
|
||||
**Metrics endpoint:** `GET /metrics` (unauthenticated — restrict in production to internal network or scrape from within the cluster)
|
||||
|
||||
The metrics endpoint is served by the `prom-client` library using a dedicated registry (`metricsRegistry`) defined in `src/metrics/registry.ts`. The registry is isolated from the default global registry to prevent conflicts in tests.
|
||||
|
||||
### Metric Definitions
|
||||
|
||||
All 6 metrics are defined in `src/metrics/registry.ts`:
|
||||
|
||||
| Metric name | Type | Labels | What it measures |
|
||||
|-------------|------|--------|-----------------|
|
||||
| `agentidp_tokens_issued_total` | Counter | `scope` | Total OAuth 2.0 access tokens issued successfully |
|
||||
| `agentidp_agents_registered_total` | Counter | `deployment_env` | Total AI agents registered successfully |
|
||||
| `agentidp_http_requests_total` | Counter | `method`, `route`, `status_code` | Total HTTP requests received |
|
||||
| `agentidp_http_request_duration_seconds` | Histogram | `method`, `route`, `status_code` | HTTP request duration in seconds (buckets: 5ms–2.5s) |
|
||||
| `agentidp_db_query_duration_seconds` | Histogram | `operation` | PostgreSQL query duration in seconds |
|
||||
| `agentidp_redis_command_duration_seconds` | Histogram | `command` | Redis command duration in seconds |
|
||||
|
||||
The HTTP metrics (`agentidp_http_requests_total` and `agentidp_http_request_duration_seconds`) are populated by `metricsMiddleware` in `src/middleware/metrics.ts`, which is registered before all routes in `src/app.ts`. Route labels are normalised to replace UUIDs with `:id` to prevent high cardinality (e.g. `/api/v1/agents/:id` rather than `/api/v1/agents/a1b2c3...`).
|
||||
|
||||
### Local Grafana
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d
|
||||
```
|
||||
|
||||
- Prometheus: http://localhost:9090
|
||||
- Grafana: http://localhost:3001 (admin password: `agentidp`)
|
||||
|
||||
The monitoring compose overlay starts `prom/prometheus:v2.53.0` and `grafana/grafana:11.2.0`. Grafana dashboards and datasource provisioning are loaded from `monitoring/grafana/provisioning/`.
|
||||
|
||||
### Adding a New Metric
|
||||
|
||||
1. Define the metric in `src/metrics/registry.ts` using the shared `metricsRegistry` (not the default prom-client registry).
|
||||
2. Export it from that file.
|
||||
3. Import it in the file where the instrumentation point lives.
|
||||
4. Call `.inc(labels)` for Counters or `.observe(labels, value)` for Histograms at the instrumentation point.
|
||||
5. Verify it appears in `GET /metrics` after starting the server.
|
||||
|
||||
---
|
||||
|
||||
## 6. Operational Runbook
|
||||
|
||||
### Health Check
|
||||
|
||||
```bash
|
||||
curl http://<host>/health
|
||||
```
|
||||
|
||||
Expected response:
|
||||
|
||||
```json
|
||||
{"status":"ok","postgres":"connected","redis":"connected"}
|
||||
```
|
||||
|
||||
Troubleshooting:
|
||||
- If `postgres: "error"` — verify `DATABASE_URL` is correct and PostgreSQL is reachable. Check `docker compose logs postgres` for local dev.
|
||||
- If `redis: "error"` — verify `REDIS_URL` is correct and Redis is reachable. Check `docker compose logs redis` for local dev.
|
||||
- If the health endpoint returns 502 or times out — the app process has crashed; check application logs.
|
||||
|
||||
---
|
||||
|
||||
### Rotate the JWT Signing Key
|
||||
|
||||
All active tokens become invalid after a key rotation — agents must re-authenticate.
|
||||
|
||||
1. Generate a new RSA-2048 key pair:
|
||||
```bash
|
||||
openssl genrsa -out new-private.pem 2048
|
||||
openssl rsa -in new-private.pem -pubout -out new-public.pem
|
||||
```
|
||||
2. Update `JWT_PRIVATE_KEY` and `JWT_PUBLIC_KEY` in your deployment environment (AWS Secrets Manager, GCP Secret Manager, or `.env`).
|
||||
3. Perform a rolling restart:
|
||||
- **ECS:** trigger a new task deployment — ECS drains existing tasks and starts new ones with the updated secret values.
|
||||
- **Cloud Run:** deploy a new revision — Cloud Run gradually shifts traffic to the new revision.
|
||||
4. Tokens signed with the old key will fail verification immediately after all instances have restarted.
|
||||
|
||||
---
|
||||
|
||||
### Revoke All Tokens for a Compromised Agent
|
||||
|
||||
Suspend the agent to stop new token issuance immediately:
|
||||
|
||||
```bash
|
||||
curl -X PATCH http://<host>/api/v1/agents/<agentId> \
|
||||
-H "Authorization: Bearer <admin_token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"status": "suspended"}'
|
||||
```
|
||||
|
||||
This prevents any new `POST /api/v1/token` requests for that agent. Active tokens remain valid until their TTL (1 hour). To invalidate active tokens immediately, also revoke all credentials for the agent:
|
||||
|
||||
```bash
|
||||
# List credentials
|
||||
curl http://<host>/api/v1/agents/<agentId>/credentials \
|
||||
-H "Authorization: Bearer <admin_token>"
|
||||
|
||||
# Revoke each active credential
|
||||
curl -X DELETE http://<host>/api/v1/agents/<agentId>/credentials/<credentialId> \
|
||||
-H "Authorization: Bearer <admin_token>"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Read Audit Logs for an Incident
|
||||
|
||||
Query the audit log with date range and agent filter:
|
||||
|
||||
```bash
|
||||
curl "http://<host>/api/v1/audit?agentId=<agentId>&startDate=2026-01-01T00:00:00Z&endDate=2026-01-31T23:59:59Z" \
|
||||
-H "Authorization: Bearer <admin_token>"
|
||||
```
|
||||
|
||||
Events are returned newest-first. Audit log retention is 90 days on the free tier. Each event includes: `eventId`, `agentId`, `action`, `outcome`, `ipAddress`, `userAgent`, `metadata`, `timestamp`.
|
||||
|
||||
Common `action` values: `token.issued`, `token.revoked`, `token.introspected`, `agent.created`, `agent.updated`, `agent.suspended`, `agent.decommissioned`, `credential.generated`, `credential.rotated`, `credential.revoked`, `auth.failed`.
|
||||
456
docs/engineering/11-sdk-guide.md
Normal file
456
docs/engineering/11-sdk-guide.md
Normal file
@@ -0,0 +1,456 @@
|
||||
# 11 — SDK Integration Guide
|
||||
|
||||
AgentIdP ships four official client SDKs — Node.js, Python, Go, and Java. All four expose an identical API surface, handle OAuth 2.0 token acquisition automatically, and throw typed errors. This document covers installation, complete working examples, error handling, and the contribution guide for adding new endpoints.
|
||||
|
||||
---
|
||||
|
||||
## 1. SDK Architecture Overview
|
||||
|
||||
Every SDK composes the same four service clients:
|
||||
|
||||
| Service client | Node.js | Python | Go | Java |
|
||||
|---------------|---------|--------|----|------|
|
||||
| Agent Registry | `AgentRegistryClient` | `AgentRegistryClient` | `AgentsClient` | `AgentServiceClient` |
|
||||
| Credential Management | `CredentialClient` | `CredentialClient` | `CredentialsClient` | `CredentialServiceClient` |
|
||||
| Token Operations | `TokenClient` | `TokenClient` | `TokenServiceClient` | `TokenServiceClient` |
|
||||
| Audit Log | `AuditClient` | `AuditClient` | `AuditClient` | `AuditServiceClient` |
|
||||
|
||||
All four SDKs also implement:
|
||||
|
||||
- **`AgentIdPClient`** — the top-level client that composes all four service clients and wires them to a shared `TokenManager`.
|
||||
- **`TokenManager`** — fetches and caches the OAuth 2.0 access token. Automatically requests a new token when the cached one is within 60 seconds of expiry. Thread-safe / goroutine-safe.
|
||||
- **Typed error class** — `AgentIdPError` (Node.js, Python, Go) or `AgentIdPException` (Java) — with `code`, `httpStatus`, and `details` fields.
|
||||
|
||||
This consistency is a maintained standard. When a new API endpoint is added to the server, it must be added to all four SDKs simultaneously.
|
||||
|
||||
---
|
||||
|
||||
## 2. Node.js SDK
|
||||
|
||||
**Install:**
|
||||
|
||||
```bash
|
||||
npm install @sentryagent/idp-sdk
|
||||
```
|
||||
|
||||
**Requirements:** Node.js 18+ (uses native `fetch`).
|
||||
|
||||
**Complete example:**
|
||||
|
||||
```typescript
|
||||
import { AgentIdPClient, AgentIdPError } from '@sentryagent/idp-sdk';
|
||||
|
||||
const client = new AgentIdPClient({
|
||||
baseUrl: 'http://localhost:3000',
|
||||
clientId: 'a1b2c3d4-e5f6-7890-abcd-ef1234567890',
|
||||
clientSecret: 'sk_live_...',
|
||||
// Optional: restrict scopes. Defaults to all four.
|
||||
// scopes: ['agents:read', 'tokens:read'],
|
||||
});
|
||||
|
||||
// Register a new agent
|
||||
const agent = await client.agents.registerAgent({
|
||||
email: 'classifier-v2@myorg.ai',
|
||||
agentType: 'classifier',
|
||||
version: '2.0.0',
|
||||
capabilities: ['resume:read', 'classify'],
|
||||
owner: 'platform-team',
|
||||
deploymentEnv: 'production',
|
||||
});
|
||||
console.log('Registered:', agent.agentId);
|
||||
|
||||
// List active agents (token acquired automatically)
|
||||
const { data: agents } = await client.agents.listAgents({ status: 'active' });
|
||||
console.log('Active agents:', agents.length);
|
||||
|
||||
// Generate credentials for an agent
|
||||
const cred = await client.credentials.generateCredential(agent.agentId);
|
||||
console.log('Client secret (store this — shown once):', cred.clientSecret);
|
||||
|
||||
// Rotate credentials
|
||||
const newCred = await client.credentials.rotateCredential(agent.agentId, cred.credentialId);
|
||||
console.log('New secret:', newCred.clientSecret);
|
||||
|
||||
// Introspect a token
|
||||
const introspection = await client.tokens.introspectToken('eyJ...');
|
||||
console.log('Active:', introspection.active);
|
||||
|
||||
// Error handling
|
||||
try {
|
||||
await client.agents.getAgent('non-existent-id');
|
||||
} catch (err) {
|
||||
if (err instanceof AgentIdPError) {
|
||||
console.error(err.code); // e.g. AGENT_NOT_FOUND
|
||||
console.error(err.httpStatus); // e.g. 404
|
||||
console.error(err.details); // optional structured context
|
||||
}
|
||||
}
|
||||
|
||||
// Force a fresh token on the next call (e.g. after credential rotation)
|
||||
client.clearTokenCache();
|
||||
```
|
||||
|
||||
**Token manager behaviour:** `TokenManager` in `sdk/src/token-manager.ts` caches the token and requests a new one when fewer than 60 seconds remain before expiry.
|
||||
|
||||
**Service clients are accessible at:**
|
||||
- `client.agents` — `AgentRegistryClient` (register, list, get, update, decommission)
|
||||
- `client.credentials` — `CredentialClient` (generate, list, rotate, revoke)
|
||||
- `client.tokens` — `TokenClient` (introspect, revoke)
|
||||
- `client.audit` — `AuditClient` (query, get event)
|
||||
|
||||
---
|
||||
|
||||
## 3. Python SDK
|
||||
|
||||
**Install:**
|
||||
|
||||
```bash
|
||||
pip install sentryagent-idp
|
||||
```
|
||||
|
||||
**Requirements:** Python 3.9+. Synchronous client uses `requests`; asynchronous client uses `httpx`.
|
||||
|
||||
### Synchronous example
|
||||
|
||||
```python
|
||||
from sentryagent_idp import AgentIdPClient, AgentIdPError, RegisterAgentRequest
|
||||
|
||||
client = AgentIdPClient(
|
||||
base_url="http://localhost:3000",
|
||||
client_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
client_secret="sk_live_...",
|
||||
# scopes=["agents:read", "tokens:read"], # optional
|
||||
)
|
||||
|
||||
# Register an agent
|
||||
agent = client.agents.register_agent(RegisterAgentRequest(
|
||||
email="screener@myorg.ai",
|
||||
agent_type="screener",
|
||||
version="1.0.0",
|
||||
capabilities=["resume:read"],
|
||||
owner="recruiting-team",
|
||||
deployment_env="production",
|
||||
))
|
||||
print("Registered:", agent.agent_id)
|
||||
|
||||
# List agents
|
||||
result = client.agents.list_agents(status="active", page=1, limit=20)
|
||||
for a in result.data:
|
||||
print(a.agent_id, a.status)
|
||||
|
||||
# Generate credentials
|
||||
cred = client.credentials.generate_credential(agent.agent_id)
|
||||
print("Client secret (shown once):", cred.client_secret)
|
||||
|
||||
# Error handling
|
||||
try:
|
||||
client.agents.get_agent("non-existent-id")
|
||||
except AgentIdPError as e:
|
||||
print(e.code) # e.g. AGENT_NOT_FOUND
|
||||
print(e.http_status) # e.g. 404
|
||||
print(e.details) # optional dict
|
||||
```
|
||||
|
||||
### Asynchronous example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from sentryagent_idp import AsyncAgentIdPClient, AgentIdPError
|
||||
|
||||
async def main() -> None:
|
||||
client = AsyncAgentIdPClient(
|
||||
base_url="http://localhost:3000",
|
||||
client_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
client_secret="sk_live_...",
|
||||
)
|
||||
|
||||
result = await client.agents.list_agents(status="active")
|
||||
print(f"Found {result.total} active agents")
|
||||
|
||||
# Rotate a credential
|
||||
new_cred = await client.credentials.rotate_credential(
|
||||
"agent-uuid", "credential-uuid"
|
||||
)
|
||||
print("New secret:", new_cred.client_secret)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
`AsyncAgentIdPClient` uses an `AsyncTokenManager` backed by `httpx.AsyncClient`. Both sync and async clients are available from the `sentryagent_idp` top-level package.
|
||||
|
||||
---
|
||||
|
||||
## 4. Go SDK
|
||||
|
||||
**Install:**
|
||||
|
||||
```bash
|
||||
go get github.com/sentryagent/idp-sdk-go
|
||||
```
|
||||
|
||||
**Requirements:** Go 1.21+.
|
||||
|
||||
**Complete example:**
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
|
||||
agentidp "github.com/sentryagent/idp-sdk-go"
|
||||
)
|
||||
|
||||
func main() {
|
||||
ctx := context.Background()
|
||||
|
||||
client := agentidp.NewAgentIdPClient(agentidp.AgentIdPClientConfig{
|
||||
BaseURL: "http://localhost:3000",
|
||||
ClientID: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
ClientSecret: "sk_live_...",
|
||||
// Scope: "agents:read agents:write", // optional
|
||||
})
|
||||
|
||||
// Register an agent
|
||||
agent, err := client.Agents.RegisterAgent(ctx, agentidp.RegisterAgentRequest{
|
||||
Email: "screener@myorg.ai",
|
||||
AgentType: "screener",
|
||||
Version: "1.0.0",
|
||||
Capabilities: []string{"resume:read"},
|
||||
Owner: "recruiting-team",
|
||||
DeploymentEnv: "production",
|
||||
})
|
||||
if err != nil {
|
||||
// Type-assert for structured error information
|
||||
var idpErr *agentidp.AgentIdPError
|
||||
if errors.As(err, &idpErr) {
|
||||
log.Fatalf("API error: code=%s status=%d", idpErr.Code, idpErr.HTTPStatus)
|
||||
}
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Println("Registered:", agent.AgentID)
|
||||
|
||||
// List agents with filters
|
||||
list, err := client.Agents.ListAgents(ctx, &agentidp.ListAgentsParams{
|
||||
Status: "active",
|
||||
Page: 1,
|
||||
Limit: 20,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Printf("Found %d agents\n", list.Total)
|
||||
|
||||
// Generate credentials
|
||||
cred, err := client.Credentials.GenerateCredential(ctx, agent.AgentID, nil)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Println("Client secret (shown once):", cred.ClientSecret)
|
||||
|
||||
// Rotate credentials
|
||||
newCred, err := client.Credentials.RotateCredential(ctx, agent.AgentID, cred.CredentialID, nil)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Println("New secret:", newCred.ClientSecret)
|
||||
}
|
||||
```
|
||||
|
||||
`context.Context` is the first parameter of every method — use `context.Background()` for simple cases or a derived context with deadline/cancellation for production code. The `TokenManager` is goroutine-safe and the client is safe for concurrent use.
|
||||
|
||||
---
|
||||
|
||||
## 5. Java SDK
|
||||
|
||||
**Maven dependency:**
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>ai.sentryagent</groupId>
|
||||
<artifactId>idp-sdk</artifactId>
|
||||
<version>1.0.0</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
**Gradle:**
|
||||
|
||||
```groovy
|
||||
implementation 'ai.sentryagent:idp-sdk:1.0.0'
|
||||
```
|
||||
|
||||
**Requirements:** Java 17+.
|
||||
|
||||
### Synchronous example
|
||||
|
||||
```java
|
||||
import ai.sentryagent.idp.AgentIdPClient;
|
||||
import ai.sentryagent.idp.AgentIdPException;
|
||||
import ai.sentryagent.idp.models.*;
|
||||
|
||||
// Builder pattern — scope is optional (defaults to all four scopes)
|
||||
AgentIdPClient client = new AgentIdPClient(
|
||||
"http://localhost:3000",
|
||||
"a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
"sk_live_..."
|
||||
);
|
||||
|
||||
// Register an agent
|
||||
Agent agent = client.agents().registerAgent(
|
||||
RegisterAgentRequest.builder()
|
||||
.email("screener@myorg.ai")
|
||||
.agentType("screener")
|
||||
.version("1.0.0")
|
||||
.capabilities(List.of("resume:read"))
|
||||
.owner("recruiting-team")
|
||||
.deploymentEnv("production")
|
||||
.build()
|
||||
);
|
||||
System.out.println("Registered: " + agent.getAgentId());
|
||||
|
||||
// List agents
|
||||
PaginatedAgents result = client.agents().listAgents(
|
||||
ListAgentsParams.builder().status("active").page(1).limit(20).build()
|
||||
);
|
||||
System.out.println("Total: " + result.getTotal());
|
||||
|
||||
// Generate credentials
|
||||
CredentialWithSecret cred = client.credentials().generateCredential(agent.getAgentId());
|
||||
System.out.println("Client secret (shown once): " + cred.getClientSecret());
|
||||
|
||||
// Rotate credentials
|
||||
CredentialWithSecret newCred = client.credentials().rotateCredential(
|
||||
agent.getAgentId(), cred.getCredentialId()
|
||||
);
|
||||
System.out.println("New secret: " + newCred.getClientSecret());
|
||||
|
||||
// Error handling
|
||||
try {
|
||||
client.agents().getAgent("non-existent-id");
|
||||
} catch (AgentIdPException ex) {
|
||||
System.out.printf("code=%s status=%d%n", ex.getCode(), ex.getHttpStatus());
|
||||
// e.g. code=AGENT_NOT_FOUND status=404
|
||||
}
|
||||
```
|
||||
|
||||
### Async example (CompletableFuture)
|
||||
|
||||
```java
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
|
||||
// Every sync method has an async counterpart
|
||||
CompletableFuture<Agent> future = client.agents().getAgentAsync("agent-uuid");
|
||||
future.thenAccept(a -> System.out.println(a.getAgentId()));
|
||||
|
||||
// Compose multiple async calls
|
||||
client.agents().getAgentAsync("agent-uuid")
|
||||
.thenCompose(a -> client.credentials().generateCredentialAsync(a.getAgentId()))
|
||||
.thenAccept(cred -> System.out.println("New secret: " + cred.getClientSecret()))
|
||||
.exceptionally(ex -> {
|
||||
if (ex.getCause() instanceof AgentIdPException idpEx) {
|
||||
System.err.printf("code=%s%n", idpEx.getCode());
|
||||
}
|
||||
return null;
|
||||
});
|
||||
```
|
||||
|
||||
The `TokenManager` is thread-safe. `AgentIdPClient` is safe for concurrent use from multiple threads.
|
||||
|
||||
---
|
||||
|
||||
## 6. SDK Contribution Guide — Adding a New Endpoint
|
||||
|
||||
When the server adds a new API endpoint, update all four SDKs. The checklist below covers each SDK.
|
||||
|
||||
### Node.js SDK (`sdk/`)
|
||||
|
||||
```
|
||||
src/
|
||||
services/
|
||||
agents.ts # AgentRegistryClient
|
||||
credentials.ts # CredentialClient
|
||||
token.ts # TokenClient
|
||||
audit.ts # AuditClient
|
||||
types.ts # All request/response type definitions
|
||||
token-manager.ts # TokenManager
|
||||
client.ts # AgentIdPClient (top-level)
|
||||
errors.ts # AgentIdPError
|
||||
```
|
||||
|
||||
Checklist:
|
||||
- [ ] Add method to the appropriate service client in `src/services/<client>.ts`
|
||||
- [ ] Add TypeScript request/response types in `src/types.ts`
|
||||
- [ ] Add JSDoc with `@param`, `@returns`, and `@throws`
|
||||
- [ ] Add unit test in `tests/<client>.test.ts`
|
||||
- [ ] Verify `npx tsc --strict` exits 0
|
||||
|
||||
### Python SDK (`sdk-python/`)
|
||||
|
||||
```
|
||||
src/sentryagent_idp/
|
||||
services/
|
||||
agents.py # AgentRegistryClient + AsyncAgentRegistryClient
|
||||
credentials.py # CredentialClient + AsyncCredentialClient
|
||||
token.py # TokenClient + AsyncTokenClient
|
||||
audit.py # AuditClient + AsyncAuditClient
|
||||
client.py # AgentIdPClient + AsyncAgentIdPClient
|
||||
token_manager.py # TokenManager (sync)
|
||||
async_token_manager.py # AsyncTokenManager
|
||||
errors.py # AgentIdPError
|
||||
types.py # TypedDict / dataclass definitions
|
||||
```
|
||||
|
||||
Checklist:
|
||||
- [ ] Add method to both the sync and async service clients
|
||||
- [ ] Add type hints (all parameters and return types)
|
||||
- [ ] Verify `mypy --strict` passes
|
||||
- [ ] Add unit test in `tests/`
|
||||
- [ ] Verify `pytest` passes with >80% coverage
|
||||
|
||||
### Go SDK (`sdk-go/`)
|
||||
|
||||
```
|
||||
agentidp/
|
||||
client.go # AgentIdPClient + AgentIdPClientConfig
|
||||
agents.go # AgentsClient
|
||||
credentials.go # CredentialsClient
|
||||
token_service.go # TokenServiceClient
|
||||
audit.go # AuditClient
|
||||
token_manager.go # TokenManager (goroutine-safe)
|
||||
errors.go # AgentIdPError
|
||||
types.go # All request/response struct types
|
||||
request.go # Shared HTTP request helper
|
||||
```
|
||||
|
||||
Checklist:
|
||||
- [ ] Add method to the appropriate `*Client` type
|
||||
- [ ] Use `context.Context` as the first parameter
|
||||
- [ ] Add godoc comment above the method
|
||||
- [ ] Add request/response struct types in `types.go` if needed
|
||||
- [ ] Add unit test in `<file>_test.go`
|
||||
- [ ] Verify `go vet ./... && staticcheck ./...` pass
|
||||
|
||||
### Java SDK (`sdk-java/`)
|
||||
|
||||
```
|
||||
src/main/java/ai/sentryagent/idp/
|
||||
AgentIdPClient.java # Top-level client
|
||||
services/
|
||||
AgentServiceClient.java # Agent Registry
|
||||
CredentialServiceClient.java
|
||||
TokenServiceClient.java
|
||||
AuditServiceClient.java
|
||||
models/ # Request/response POJOs (@JsonProperty)
|
||||
TokenManager.java # Thread-safe token caching
|
||||
AgentIdPException.java # Typed exception
|
||||
```
|
||||
|
||||
Checklist:
|
||||
- [ ] Add sync method to the appropriate service client
|
||||
- [ ] Add `CompletableFuture<T>` async counterpart with the `Async` suffix
|
||||
- [ ] Add request/response POJO in `models/` with `@JsonProperty` annotations
|
||||
- [ ] Add Javadoc on the method
|
||||
- [ ] Add JUnit 5 test in `src/test/java/`
|
||||
- [ ] Verify `mvn verify` passes (compiles, tests, and checks coverage)
|
||||
56
docs/engineering/README.md
Normal file
56
docs/engineering/README.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# SentryAgent.ai — Engineering Knowledge Base
|
||||
|
||||
> Internal reference for engineers contributing to AgentIdP. Read in order if you're new. Jump to the relevant document if you know what you need.
|
||||
|
||||
---
|
||||
|
||||
## Reading Order (New Engineers Start Here)
|
||||
|
||||
| # | Document | What you'll learn | Time |
|
||||
|---|---------|------------------|------|
|
||||
| 1 | [Company and Product Overview](01-overview.md) | What SentryAgent.ai builds, why it exists, the product feature set, Phase roadmap | 15 min |
|
||||
| 2 | [System Architecture](02-architecture.md) | Component diagram, HTTP request lifecycle, OAuth 2.0 data flow, multi-region topology | 20 min |
|
||||
| 3 | [Technology Stack and ADRs](03-tech-stack.md) | Why each technology was chosen — rationale and alternatives considered | 20 min |
|
||||
| 4 | [Codebase Structure](04-codebase-structure.md) | Directory map, where to add new code, DRY enforcement rules | 15 min |
|
||||
| 5 | [Service Deep Dives](05-services.md) | All 8 services/components — purpose, interface, schema, error types | 30 min |
|
||||
| 6 | [Annotated Code Walkthroughs](06-walkthroughs.md) | Step-by-step traces of token issuance, agent registration, credential rotation | 30 min |
|
||||
| 7 | [Development Environment Setup](07-dev-setup.md) | Clone to running local stack — under 30 minutes | 30 min |
|
||||
| 8 | [Engineering Workflow](08-workflow.md) | OpenSpec spec-first workflow, branching, PR checklist, commit conventions | 20 min |
|
||||
| 9 | [Testing Strategy](09-testing.md) | Unit vs integration, coverage gates, how to write tests, OWASP reference | 20 min |
|
||||
| 10 | [Deployment and Operations](10-deployment.md) | Docker, Terraform, Prometheus/Grafana, operational runbook | 20 min |
|
||||
| 11 | [SDK Integration Guide](11-sdk-guide.md) | All 4 SDKs — installation, examples, contribution guide | 20 min |
|
||||
|
||||
**Total estimated reading time for new engineers: ~3.5 hours**
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| I need to... | Go to |
|
||||
|-------------|-------|
|
||||
| Understand the codebase layout | [04-codebase-structure.md](04-codebase-structure.md) |
|
||||
| Run the project locally | [07-dev-setup.md](07-dev-setup.md) |
|
||||
| Understand how token issuance works end-to-end | [06-walkthroughs.md](06-walkthroughs.md) |
|
||||
| Add a new API endpoint | [08-workflow.md](08-workflow.md) + [04-codebase-structure.md](04-codebase-structure.md) |
|
||||
| Write tests | [09-testing.md](09-testing.md) |
|
||||
| Deploy to production | [10-deployment.md](10-deployment.md) |
|
||||
| Integrate with the SDK | [11-sdk-guide.md](11-sdk-guide.md) |
|
||||
| Understand why a technology was chosen | [03-tech-stack.md](03-tech-stack.md) |
|
||||
|
||||
---
|
||||
|
||||
## Document Conventions
|
||||
|
||||
- **File paths** are always relative to the project root unless otherwise noted.
|
||||
- **Line numbers** in [06-walkthroughs.md](06-walkthroughs.md) were verified against commit `1f95cfe`.
|
||||
- **Code examples** are complete and runnable — no ellipses, no placeholders.
|
||||
- **ADR** stands for Architecture Decision Record — a short document recording a technology choice.
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/developers/` — End-user API reference (for agents calling the AgentIdP API)
|
||||
- `docs/devops/` — Operator runbooks and environment variable reference
|
||||
- `docs/agntcy/` — AGNTCY alignment documentation
|
||||
- `openspec/` — OpenSpec change management (proposals, designs, specs, tasks, archives)
|
||||
Reference in New Issue
Block a user