sentryagent-idp/docs/engineering/04-codebase-structure.md

Codebase Structure

---

1. Annotated Directory Tree

sentryagent-idp/
├── src/                       # Express application source — controllers, services, middleware, repositories, routes
│   ├── app.ts                 # Express app factory — creates and configures the app; does NOT call listen
│   ├── server.ts              # Entry point — calls listen, handles SIGTERM/SIGINT/SIGHUP
│   ├── types/                 # Canonical TypeScript interfaces and type definitions
│   ├── controllers/           # HTTP layer — extract/validate inputs, call services, build responses
│   ├── services/              # Business logic — pure domain operations, no HTTP knowledge
│   ├── repositories/          # Database and Redis access — parameterized SQL, no logic
│   ├── middleware/             # Cross-cutting request concerns — auth, OPA, rate-limit, metrics, error handling
│   ├── routes/                # Express router definitions — wiring only, no logic
│   ├── utils/                 # Shared pure utilities — errors, validators, crypto, JWT helpers
│   ├── vault/                 # HashiCorp Vault KV v2 client
│   ├── metrics/               # Prometheus metrics registry — all Counter and Histogram definitions
│   ├── db/                    # PostgreSQL pool factory and SQL migration files
│   └── cache/                 # Redis client factory
├── tests/                     # Jest test suite — mirrors src/ structure (unit/ and integration/)
├── dashboard/                 # React 18 + Vite 5 web dashboard SPA
│   ├── src/                   # Dashboard source — pages, components, auth, API client
│   └── dist/                  # Built dashboard — served by Express at /dashboard (git-ignored)
├── sdk/                       # Node.js SDK (@sentryagent/idp-sdk) — TypeScript, auto token refresh
├── sdk-python/                # Python SDK (sentryagent-idp) — sync + async clients
├── sdk-go/                    # Go SDK (github.com/sentryagent/idp-sdk-go) — context-aware, goroutine-safe
├── sdk-java/                  # Java SDK (ai.sentryagent:idp-sdk) — builder pattern, CompletableFuture
├── sdk-rust/                  # Rust SDK (sentryagent-idp crate) — async, tokio, reqwest, typed errors
├── policies/                  # OPA policy files
│   ├── authz.rego             # Rego policy — normalise_path + scope-intersection allow rule
│   └── data/scopes.json       # Endpoint permission map — used by Rego and TypeScript fallback
├── portal/                    # Developer Portal — Next.js 14 App Router, Tailwind CSS
│   ├── app/                   # Next.js App Router pages (get-started, pricing, sdks, analytics, settings, login)
│   ├── components/            # Shared UI components (Nav.tsx, SwaggerExplorer.tsx, GetStartedWizard.tsx)
│   ├── hooks/                 # React hooks (useAuth.ts)
│   └── types/                 # TypeScript type definitions for portal-only types
├── terraform/                 # Terraform infrastructure as code
│   ├── modules/               # Reusable modules: agentidp, lb, rds, redis
│   └── environments/          # Environment configs: aws/ (ECS+RDS+ElastiCache), gcp/ (Cloud Run+SQL+Memorystore)
├── monitoring/                # Prometheus and Grafana configuration
│   ├── prometheus/            # prometheus.yml scrape configuration
│   └── grafana/               # Grafana provisioning YAML and dashboard JSON files
├── docs/                      # All project documentation
│   ├── engineering/           # Internal engineering knowledge base (this directory)
│   ├── developers/            # End-user API reference and developer guides
│   ├── devops/                # Operator runbooks and environment variable reference
│   ├── agntcy/                # AGNTCY alignment documentation
│   └── openapi/               # OpenAPI 3.0 specification files
├── openspec/                  # OpenSpec change management — proposals, designs, specs, tasks, archives
├── tests/                     # Jest test suite — mirrors src/ structure
│   ├── unit/                  # Unit tests (mocked dependencies) — mirrors src/
│   ├── integration/           # Integration tests (real DB + Redis)
│   ├── agntcy-conformance/    # AGNTCY conformance test suite (separate Jest config)
│   └── load/                  # k6 load test scripts
├── Dockerfile                 # Multi-stage production build (build + runtime stages)
├── compose.yaml               # Local development: PostgreSQL 14.12 (port 5432) + Redis 7.2 (port 6379)
├── compose.monitoring.yaml    # Monitoring overlay: Prometheus (port 9090) + Grafana (port 3001)
├── package.json               # Node.js dependencies and npm scripts
├── tsconfig.json              # TypeScript strict configuration — compiled to dist/
└── jest.config.ts             # Jest configuration — ts-jest, test timeouts, coverage thresholds

---

2. src/ Subdirectory Roles

Directory	Role	Rule
src/controllers/	Receive HTTP requests, extract and validate inputs using Joi, call service methods, serialise responses	No business logic — controllers are thin wrappers that translate HTTP into service calls
src/services/	All business logic — free-tier limit enforcement, domain rule evaluation, orchestration of repository calls and audit events	Never import from controllers or routes; never know about req or res
src/repositories/	All database and Redis queries — parameterized SQL via node-postgres, Redis commands via redis client	Only called from services; never called directly from controllers; no business logic
src/middleware/	Cross-cutting request concerns — authMiddleware, opaMiddleware, rateLimitMiddleware, metricsMiddleware, errorHandler	Applied at router or app level in src/app.ts; never import from controllers
src/routes/	Map HTTP paths and methods to middleware chains and controller methods	Wiring only — no logic, no validation, no business rules
src/utils/	Shared pure utilities — errors.ts, validators.ts, crypto.ts, jwt.ts, asyncHandler.ts	No side effects; no imports from services or controllers
src/types/	All TypeScript type definitions, interfaces, and enums — the single source of truth for all shared types	Imported everywhere; never imports from anywhere else in src/
src/vault/	VaultClient — wraps HashiCorp Vault KV v2 operations; constant-time secret verification	Only instantiated by createVaultClientFromEnv() in src/app.ts; passed to services via constructor injection
src/metrics/	Prometheus metrics registry — all Counter and Histogram definitions in one place	Only file that calls new Counter() or new Histogram(); all other files import from here
src/db/	PostgreSQL connection pool factory (pool.ts) and numbered SQL migration files in migrations/	Pool is a singleton created once in src/app.ts and passed to repositories
src/cache/	Redis client factory — creates and caches a single redis client instance	Client is a singleton created once in src/app.ts and passed to repositories
src/config/	Configuration constants — tiers.ts exports TIER_CONFIG, TIER_RANK, TierName, and isTierName() type guard	Imported by TierService and tierMiddleware; never imports from services
src/middleware/tier.ts	Tier enforcement middleware — reads org tier from TierService, checks daily call counter in Redis, throws TierLimitError (429) when limit is exceeded, increments counter on pass	Applied only to API routes; skips /health, /metrics, and static file routes

---

3. Where to Add New Code

I need to add...	Where it goes	Example
A new API endpoint	src/routes/ (wire it), src/controllers/ (HTTP layer), src/services/ (business logic), src/repositories/ (data access)	Adding DELETE /api/v1/agents/:id/credentials/:credId/bulk
A new business rule	src/services/[relevant]Service.ts	Enforcing a maximum of 5 credentials per agent
A new database table	src/db/migrations/ — new numbered SQL file (append-only)	Adding an agent_groups table as 005_create_agent_groups.sql
A new authorisation policy rule	policies/authz.rego + policies/data/scopes.json	Adding a new scope reports:read for a GET /api/v1/reports endpoint
A new shared error type	src/utils/errors.ts	VaultUnavailableError extending SentryAgentError
A new environment variable	src/utils/config.ts (if it exists) or the relevant consumer file + docs/devops/environment-variables.md	RATE_LIMIT_MAX controlling the rate-limit ceiling
A new Prometheus metric	src/metrics/registry.ts	A Histogram for Vault lookup duration
A new TypeScript type used in 2+ files	src/types/index.ts	A new AgentGroupMembership interface
A new tier-gated feature	src/config/tiers.ts (add limit field) + src/middleware/tier.ts (add check) + service (enforce)	Adding a maxWebhooksPerOrg tier limit
A webhook event handler	src/services/WebhookService.ts (add event type to WebhookEventType) + the producer that calls void webhookService.dispatch(orgId, eventType, payload)	Emitting agent.decommissioned events to subscriber URLs
A new analytics metric type	src/services/AnalyticsService.ts (call recordEvent(tenantId, 'new_metric') in the relevant service using void)	Recording credential_rotated events for analytics
A new DID endpoint	src/controllers/DIDController.ts + src/routes/did.ts + src/services/DIDService.ts (if new method needed) + policies/data/scopes.json	Adding GET /api/v1/agents/:id/did/rotate-key

---

4. Key Files

src/app.ts Creates and configures the Express application. Registers all middleware (helmet, cors, morgan, body parsers, metricsMiddleware), instantiates all infrastructure singletons (PostgreSQL pool, Redis client, VaultClient), constructs the full dependency graph (repositories → services → controllers), and mounts all routers. Returns the configured Application without calling listen. Tests import createApp() directly — this is the design decision that makes integration tests possible without binding a port.

src/server.ts The only file that calls app.listen(). Loads environment variables via dotenv.config(), calls createApp(), binds the port from PORT env var (default 3000), and registers SIGTERM, SIGINT (graceful shutdown), and SIGHUP (OPA policy hot-reload via reloadOpaPolicy()) signal handlers. This file is never imported by tests.

src/types/index.ts The canonical type definition file for the entire project. Contains all exported interfaces (IAgent, ICredential, ITokenPayload, IAuditEvent, etc.), union types (AgentStatus, AgentType, AuditAction, etc.), and the global Express Request augmentation that adds req.user?: ITokenPayload. If a type is needed in two or more files, it lives here — never redefined inline.

src/utils/errors.ts The SentryAgentError base class and all typed error subclasses. Every error thrown in the application must extend SentryAgentError — never throw new Error('string'). The errorHandler middleware in src/middleware/errorHandler.ts maps SentryAgentError subclasses to their httpStatus codes and serialises the response as IErrorResponse { code, message, details }.

compose.yaml Starts PostgreSQL 14.12 (Alpine) on port 5432 and Redis 7.2 (Alpine) on port 6379. All services use a dedicated app-tier bridge network, restart: unless-stopped, and deploy.resources.limits per DockerSpec standards. Both infrastructure services have health checks so depends_on conditions work correctly. The app service mounts ./src as a read-only bind volume for live code reloading and has its own healthcheck probe via curl /health. Postgres credentials and Grafana admin password are externalized to environment variables — see docs/devops/environment-variables.md.

tsconfig.json TypeScript compiler configuration. strict: true enables the full suite of strictness checks. target: ES2022, module: commonjs (the project compiles to CommonJS for Node.js compatibility). outDir: ./dist, rootDir: ./src. The noUnusedLocals and noUnusedParameters flags are enabled — unused code is a compile error. Never disable these flags.

---

5. DRY Enforcement

Every piece of logic lives in exactly one place. Violations are CTO-blocking.

Concern	Single source of truth	Violation pattern to reject
Business logic	One service method — called from multiple controllers if needed	Business logic duplicated in a route handler or controller
Database queries	One repository method — never repeated inline	SQL written directly in a service or controller
Error types	src/utils/errors.ts — imported wherever errors are thrown	new Error('AGENT_NOT_FOUND') instead of new AgentNotFoundError()
TypeScript types	src/types/index.ts — imported in every consumer file	An interface defined inline in a service file
Validation logic	src/utils/validators.ts — Joi schemas used in controllers	Validation logic duplicated across multiple controllers
Prometheus metrics	src/metrics/registry.ts — one definition per metric	A second new Counter({ name: 'agentidp_tokens_issued_total' }) anywhere