f9a6a8aafb
- Replace all docker-compose.yml/docker-compose.monitoring.yml references with compose.yaml/compose.monitoring.yaml (modern Compose Spec naming) - Replace all `docker-compose` CLI commands with `docker compose` (plugin syntax) - Update Dockerfile stage descriptions: node:18-alpine → node:20.11-bookworm-slim, built-in node user → explicit nodeapp:1001 non-root user - Update image version references: postgres:14-alpine → postgres:14.12-alpine3.19, redis:7-alpine → redis:7.2-alpine3.19 - Externalize postgres credentials: hardcoded values → POSTGRES_USER/PASSWORD/DB env vars - Externalize Grafana admin password: hardcoded 'agentidp' → GF_ADMIN_PASSWORD env var - Add Docker Compose Variables section to environment-variables.md (POSTGRES_*, GF_ADMIN_PASSWORD) - Update local-development.md Step 3: cp .env.example .env, document POSTGRES_* purpose - Update quick-start.md: cp .env.example .env, use awk/sed for JWT key injection - Update 07-dev-setup.md: remove 'no .env.example' claim, reference cp .env.example - Update docker-compose.yml key file description in 04-codebase-structure.md - Update monitoring overlay launch commands across all docs (compose.yaml + compose.monitoring.yaml) - Update volume names to kebab-case: postgres_data → postgres-data, redis_data → redis-data - Fix compliance encryption-runbook: docker-compose restart agentidp → docker compose restart app All docs now consistent with compose.yaml in repo root. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
167 lines
14 KiB
Markdown
167 lines
14 KiB
Markdown
# Codebase Structure
|
|
|
|
---
|
|
|
|
## 1. Annotated Directory Tree
|
|
|
|
```
|
|
sentryagent-idp/
|
|
├── src/ # Express application source — controllers, services, middleware, repositories, routes
|
|
│ ├── app.ts # Express app factory — creates and configures the app; does NOT call listen
|
|
│ ├── server.ts # Entry point — calls listen, handles SIGTERM/SIGINT/SIGHUP
|
|
│ ├── types/ # Canonical TypeScript interfaces and type definitions
|
|
│ ├── controllers/ # HTTP layer — extract/validate inputs, call services, build responses
|
|
│ ├── services/ # Business logic — pure domain operations, no HTTP knowledge
|
|
│ ├── repositories/ # Database and Redis access — parameterized SQL, no logic
|
|
│ ├── middleware/ # Cross-cutting request concerns — auth, OPA, rate-limit, metrics, error handling
|
|
│ ├── routes/ # Express router definitions — wiring only, no logic
|
|
│ ├── utils/ # Shared pure utilities — errors, validators, crypto, JWT helpers
|
|
│ ├── vault/ # HashiCorp Vault KV v2 client
|
|
│ ├── metrics/ # Prometheus metrics registry — all Counter and Histogram definitions
|
|
│ ├── db/ # PostgreSQL pool factory and SQL migration files
|
|
│ └── cache/ # Redis client factory
|
|
├── tests/ # Jest test suite — mirrors src/ structure (unit/ and integration/)
|
|
├── dashboard/ # React 18 + Vite 5 web dashboard SPA
|
|
│ ├── src/ # Dashboard source — pages, components, auth, API client
|
|
│ └── dist/ # Built dashboard — served by Express at /dashboard (git-ignored)
|
|
├── sdk/ # Node.js SDK (@sentryagent/idp-sdk) — TypeScript, auto token refresh
|
|
├── sdk-python/ # Python SDK (sentryagent-idp) — sync + async clients
|
|
├── sdk-go/ # Go SDK (github.com/sentryagent/idp-sdk-go) — context-aware, goroutine-safe
|
|
├── sdk-java/ # Java SDK (ai.sentryagent:idp-sdk) — builder pattern, CompletableFuture
|
|
├── sdk-rust/ # Rust SDK (sentryagent-idp crate) — async, tokio, reqwest, typed errors
|
|
├── policies/ # OPA policy files
|
|
│ ├── authz.rego # Rego policy — normalise_path + scope-intersection allow rule
|
|
│ └── data/scopes.json # Endpoint permission map — used by Rego and TypeScript fallback
|
|
├── portal/ # Developer Portal — Next.js 14 App Router, Tailwind CSS
|
|
│ ├── app/ # Next.js App Router pages (get-started, pricing, sdks, analytics, settings, login)
|
|
│ ├── components/ # Shared UI components (Nav.tsx, SwaggerExplorer.tsx, GetStartedWizard.tsx)
|
|
│ ├── hooks/ # React hooks (useAuth.ts)
|
|
│ └── types/ # TypeScript type definitions for portal-only types
|
|
├── terraform/ # Terraform infrastructure as code
|
|
│ ├── modules/ # Reusable modules: agentidp, lb, rds, redis
|
|
│ └── environments/ # Environment configs: aws/ (ECS+RDS+ElastiCache), gcp/ (Cloud Run+SQL+Memorystore)
|
|
├── monitoring/ # Prometheus and Grafana configuration
|
|
│ ├── prometheus/ # prometheus.yml scrape configuration
|
|
│ └── grafana/ # Grafana provisioning YAML and dashboard JSON files
|
|
├── docs/ # All project documentation
|
|
│ ├── engineering/ # Internal engineering knowledge base (this directory)
|
|
│ ├── developers/ # End-user API reference and developer guides
|
|
│ ├── devops/ # Operator runbooks and environment variable reference
|
|
│ ├── agntcy/ # AGNTCY alignment documentation
|
|
│ └── openapi/ # OpenAPI 3.0 specification files
|
|
├── openspec/ # OpenSpec change management — proposals, designs, specs, tasks, archives
|
|
├── tests/ # Jest test suite — mirrors src/ structure
|
|
│ ├── unit/ # Unit tests (mocked dependencies) — mirrors src/
|
|
│ ├── integration/ # Integration tests (real DB + Redis)
|
|
│ ├── agntcy-conformance/ # AGNTCY conformance test suite (separate Jest config)
|
|
│ └── load/ # k6 load test scripts
|
|
├── Dockerfile # Multi-stage production build (build + runtime stages)
|
|
├── compose.yaml # Local development: PostgreSQL 14.12 (port 5432) + Redis 7.2 (port 6379)
|
|
├── compose.monitoring.yaml # Monitoring overlay: Prometheus (port 9090) + Grafana (port 3001)
|
|
├── package.json # Node.js dependencies and npm scripts
|
|
├── tsconfig.json # TypeScript strict configuration — compiled to dist/
|
|
└── jest.config.ts # Jest configuration — ts-jest, test timeouts, coverage thresholds
|
|
```
|
|
|
|
---
|
|
|
|
## 2. src/ Subdirectory Roles
|
|
|
|
| Directory | Role | Rule |
|
|
|-----------|------|------|
|
|
| `src/controllers/` | Receive HTTP requests, extract and validate inputs using Joi, call service methods, serialise responses | No business logic — controllers are thin wrappers that translate HTTP into service calls |
|
|
| `src/services/` | All business logic — free-tier limit enforcement, domain rule evaluation, orchestration of repository calls and audit events | Never import from controllers or routes; never know about `req` or `res` |
|
|
| `src/repositories/` | All database and Redis queries — parameterized SQL via `node-postgres`, Redis commands via `redis` client | Only called from services; never called directly from controllers; no business logic |
|
|
| `src/middleware/` | Cross-cutting request concerns — `authMiddleware`, `opaMiddleware`, `rateLimitMiddleware`, `metricsMiddleware`, `errorHandler` | Applied at router or app level in `src/app.ts`; never import from controllers |
|
|
| `src/routes/` | Map HTTP paths and methods to middleware chains and controller methods | Wiring only — no logic, no validation, no business rules |
|
|
| `src/utils/` | Shared pure utilities — `errors.ts`, `validators.ts`, `crypto.ts`, `jwt.ts`, `asyncHandler.ts` | No side effects; no imports from services or controllers |
|
|
| `src/types/` | All TypeScript type definitions, interfaces, and enums — the single source of truth for all shared types | Imported everywhere; never imports from anywhere else in `src/` |
|
|
| `src/vault/` | `VaultClient` — wraps HashiCorp Vault KV v2 operations; constant-time secret verification | Only instantiated by `createVaultClientFromEnv()` in `src/app.ts`; passed to services via constructor injection |
|
|
| `src/metrics/` | Prometheus metrics registry — all `Counter` and `Histogram` definitions in one place | Only file that calls `new Counter()` or `new Histogram()`; all other files import from here |
|
|
| `src/db/` | PostgreSQL connection pool factory (`pool.ts`) and numbered SQL migration files in `migrations/` | Pool is a singleton created once in `src/app.ts` and passed to repositories |
|
|
| `src/cache/` | Redis client factory — creates and caches a single `redis` client instance | Client is a singleton created once in `src/app.ts` and passed to repositories |
|
|
| `src/config/` | Configuration constants — `tiers.ts` exports `TIER_CONFIG`, `TIER_RANK`, `TierName`, and `isTierName()` type guard | Imported by `TierService` and `tierMiddleware`; never imports from services |
|
|
| `src/middleware/tier.ts` | Tier enforcement middleware — reads org tier from `TierService`, checks daily call counter in Redis, throws `TierLimitError` (429) when limit is exceeded, increments counter on pass | Applied only to API routes; skips `/health`, `/metrics`, and static file routes |
|
|
|
|
---
|
|
|
|
## 3. Where to Add New Code
|
|
|
|
| I need to add... | Where it goes | Example |
|
|
|-----------------|---------------|---------|
|
|
| A new API endpoint | `src/routes/` (wire it), `src/controllers/` (HTTP layer), `src/services/` (business logic), `src/repositories/` (data access) | Adding `DELETE /api/v1/agents/:id/credentials/:credId/bulk` |
|
|
| A new business rule | `src/services/[relevant]Service.ts` | Enforcing a maximum of 5 credentials per agent |
|
|
| A new database table | `src/db/migrations/` — new numbered SQL file (append-only) | Adding an `agent_groups` table as `005_create_agent_groups.sql` |
|
|
| A new authorisation policy rule | `policies/authz.rego` + `policies/data/scopes.json` | Adding a new scope `reports:read` for a `GET /api/v1/reports` endpoint |
|
|
| A new shared error type | `src/utils/errors.ts` | `VaultUnavailableError` extending `SentryAgentError` |
|
|
| A new environment variable | `src/utils/config.ts` (if it exists) or the relevant consumer file + `docs/devops/environment-variables.md` | `RATE_LIMIT_MAX` controlling the rate-limit ceiling |
|
|
| A new Prometheus metric | `src/metrics/registry.ts` | A `Histogram` for Vault lookup duration |
|
|
| A new TypeScript type used in 2+ files | `src/types/index.ts` | A new `AgentGroupMembership` interface |
|
|
| A new tier-gated feature | `src/config/tiers.ts` (add limit field) + `src/middleware/tier.ts` (add check) + service (enforce) | Adding a `maxWebhooksPerOrg` tier limit |
|
|
| A webhook event handler | `src/services/WebhookService.ts` (add event type to `WebhookEventType`) + the producer that calls `void webhookService.dispatch(orgId, eventType, payload)` | Emitting `agent.decommissioned` events to subscriber URLs |
|
|
| A new analytics metric type | `src/services/AnalyticsService.ts` (call `recordEvent(tenantId, 'new_metric')` in the relevant service using `void`) | Recording `credential_rotated` events for analytics |
|
|
| A new DID endpoint | `src/controllers/DIDController.ts` + `src/routes/did.ts` + `src/services/DIDService.ts` (if new method needed) + `policies/data/scopes.json` | Adding `GET /api/v1/agents/:id/did/rotate-key` |
|
|
|
|
---
|
|
|
|
## 4. Key Files
|
|
|
|
**`src/app.ts`**
|
|
Creates and configures the Express application. Registers all middleware (helmet, cors,
|
|
morgan, body parsers, metricsMiddleware), instantiates all infrastructure singletons
|
|
(PostgreSQL pool, Redis client, VaultClient), constructs the full dependency graph
|
|
(repositories → services → controllers), and mounts all routers. Returns the configured
|
|
`Application` without calling `listen`. Tests import `createApp()` directly — this is
|
|
the design decision that makes integration tests possible without binding a port.
|
|
|
|
**`src/server.ts`**
|
|
The only file that calls `app.listen()`. Loads environment variables via `dotenv.config()`,
|
|
calls `createApp()`, binds the port from `PORT` env var (default 3000), and registers
|
|
`SIGTERM`, `SIGINT` (graceful shutdown), and `SIGHUP` (OPA policy hot-reload via
|
|
`reloadOpaPolicy()`) signal handlers. This file is never imported by tests.
|
|
|
|
**`src/types/index.ts`**
|
|
The canonical type definition file for the entire project. Contains all exported
|
|
interfaces (`IAgent`, `ICredential`, `ITokenPayload`, `IAuditEvent`, etc.), union types
|
|
(`AgentStatus`, `AgentType`, `AuditAction`, etc.), and the global Express `Request`
|
|
augmentation that adds `req.user?: ITokenPayload`. If a type is needed in two or more
|
|
files, it lives here — never redefined inline.
|
|
|
|
**`src/utils/errors.ts`**
|
|
The `SentryAgentError` base class and all typed error subclasses. Every error thrown
|
|
in the application must extend `SentryAgentError` — never `throw new Error('string')`.
|
|
The `errorHandler` middleware in `src/middleware/errorHandler.ts` maps
|
|
`SentryAgentError` subclasses to their `httpStatus` codes and serialises the response
|
|
as `IErrorResponse { code, message, details }`.
|
|
|
|
**`compose.yaml`**
|
|
Starts PostgreSQL 14.12 (Alpine) on port 5432 and Redis 7.2 (Alpine) on port 6379.
|
|
All services use a dedicated `app-tier` bridge network, `restart: unless-stopped`,
|
|
and `deploy.resources.limits` per DockerSpec standards. Both infrastructure services
|
|
have health checks so `depends_on` conditions work correctly. The `app` service mounts
|
|
`./src` as a read-only bind volume for live code reloading and has its own
|
|
`healthcheck` probe via `curl /health`. Postgres credentials and Grafana admin
|
|
password are externalized to environment variables — see `docs/devops/environment-variables.md`.
|
|
|
|
**`tsconfig.json`**
|
|
TypeScript compiler configuration. `strict: true` enables the full suite of strictness
|
|
checks. `target: ES2022`, `module: commonjs` (the project compiles to CommonJS for
|
|
Node.js compatibility). `outDir: ./dist`, `rootDir: ./src`. The `noUnusedLocals` and
|
|
`noUnusedParameters` flags are enabled — unused code is a compile error. Never disable
|
|
these flags.
|
|
|
|
---
|
|
|
|
## 5. DRY Enforcement
|
|
|
|
Every piece of logic lives in exactly one place. Violations are CTO-blocking.
|
|
|
|
| Concern | Single source of truth | Violation pattern to reject |
|
|
|---------|----------------------|----------------------------|
|
|
| Business logic | One service method — called from multiple controllers if needed | Business logic duplicated in a route handler or controller |
|
|
| Database queries | One repository method — never repeated inline | SQL written directly in a service or controller |
|
|
| Error types | `src/utils/errors.ts` — imported wherever errors are thrown | `new Error('AGENT_NOT_FOUND')` instead of `new AgentNotFoundError()` |
|
|
| TypeScript types | `src/types/index.ts` — imported in every consumer file | An interface defined inline in a service file |
|
|
| Validation logic | `src/utils/validators.ts` — Joi schemas used in controllers | Validation logic duplicated across multiple controllers |
|
|
| Prometheus metrics | `src/metrics/registry.ts` — one definition per metric | A second `new Counter({ name: 'agentidp_tokens_issued_total' })` anywhere |
|