docs(devops): update all documentation for DockerSpec compliance

- Replace all docker-compose.yml/docker-compose.monitoring.yml references with
  compose.yaml/compose.monitoring.yaml (modern Compose Spec naming)
- Replace all `docker-compose` CLI commands with `docker compose` (plugin syntax)
- Update Dockerfile stage descriptions: node:18-alpine → node:20.11-bookworm-slim,
  built-in node user → explicit nodeapp:1001 non-root user
- Update image version references: postgres:14-alpine → postgres:14.12-alpine3.19,
  redis:7-alpine → redis:7.2-alpine3.19
- Externalize postgres credentials: hardcoded values → POSTGRES_USER/PASSWORD/DB env vars
- Externalize Grafana admin password: hardcoded 'agentidp' → GF_ADMIN_PASSWORD env var
- Add Docker Compose Variables section to environment-variables.md (POSTGRES_*, GF_ADMIN_PASSWORD)
- Update local-development.md Step 3: cp .env.example .env, document POSTGRES_* purpose
- Update quick-start.md: cp .env.example .env, use awk/sed for JWT key injection
- Update 07-dev-setup.md: remove 'no .env.example' claim, reference cp .env.example
- Update docker-compose.yml key file description in 04-codebase-structure.md
- Update monitoring overlay launch commands across all docs (compose.yaml + compose.monitoring.yaml)
- Update volume names to kebab-case: postgres_data → postgres-data, redis_data → redis-data
- Fix compliance encryption-runbook: docker-compose restart agentidp → docker compose restart app

All docs now consistent with compose.yaml in repo root.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
SentryAgent.ai Developer
2026-04-08 08:27:37 +00:00
parent 6fada694bb
commit f9a6a8aafb
12 changed files with 160 additions and 74 deletions

View File

@@ -68,7 +68,7 @@ The `EncryptionService` caches the key in process memory. A restart forces a re-
kubectl rollout restart deployment/agentidp
# Docker Compose
docker-compose restart agentidp
docker compose restart app
# PM2
pm2 restart agentidp

View File

@@ -6,7 +6,7 @@ This guide gets you from zero to a working agent identity inside an organization
You need two tools installed:
- **Docker** (includes `docker-compose`) — to run PostgreSQL and Redis
- **Docker** (with Compose plugin, v2.20+) — to run PostgreSQL and Redis
- **Node.js 18+** (includes `npm`) — to run the server
- **curl** — to call the API
@@ -32,16 +32,19 @@ openssl genrsa -out private.pem 2048
openssl rsa -in private.pem -pubout -out public.pem
```
Create your `.env` file:
Copy the environment template and fill in your JWT keys:
```bash
cat > .env << 'EOF'
DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp
REDIS_URL=redis://localhost:6379
PORT=3000
JWT_PRIVATE_KEY="$(cat private.pem)"
JWT_PUBLIC_KEY="$(cat public.pem)"
EOF
cp .env.example .env
```
Write your JWT keys into `.env`:
```bash
PRIVATE_KEY_LINE=$(awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' private.pem)
PUBLIC_KEY_LINE=$(awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' public.pem)
sed -i "s|JWT_PRIVATE_KEY=.*|JWT_PRIVATE_KEY=\"${PRIVATE_KEY_LINE}\"|" .env
sed -i "s|JWT_PUBLIC_KEY=.*|JWT_PUBLIC_KEY=\"${PUBLIC_KEY_LINE}\"|" .env
```
> **Note**: The `.env` file stores your private key. Do not commit it to version control.
@@ -53,7 +56,7 @@ EOF
Start PostgreSQL and Redis using Docker Compose (infrastructure services only):
```bash
docker-compose up -d postgres redis
docker compose up -d postgres redis
```
Expected output:

View File

@@ -19,7 +19,7 @@ SentryAgent.ai AgentIdP is a Node.js REST API backed by PostgreSQL and Redis. It
| [Architecture](architecture.md) | All engineers | Components, ports, data flow, Redis key patterns |
| [Environment Variables](environment-variables.md) | All engineers | Every env var — required, optional, format, examples |
| [Database](database.md) | Backend, DevOps | Schema (26 tables/migrations), how to apply and verify |
| [Local Development](local-development.md) | All engineers | docker-compose setup, startup, health checks |
| [Local Development](local-development.md) | All engineers | Docker Compose setup (`compose.yaml`), startup, health checks |
| [Security](security.md) | All engineers | JWT key generation and rotation, CORS, secret storage |
| [Operations](operations.md) | DevOps | Startup order, graceful shutdown, log interpretation, troubleshooting |
| [field-trial.md](field-trial.md) | DevOps engineers, QA | In-house Docker Compose field trial execution playbook |

View File

@@ -6,6 +6,62 @@ Variables are loaded from a `.env` file at startup via `dotenv`. In production,
---
## Docker Compose Variables
These variables are read by `compose.yaml` — not by the application itself. They are required when running the stack via `docker compose up`.
### `POSTGRES_USER`
PostgreSQL superuser name — used to configure the `postgres` container and construct `DATABASE_URL`.
| | |
|-|-|
| **Required for Compose** | Yes |
| **Default in `.env.example`** | `sentryagent` |
| **Example** | `POSTGRES_USER=sentryagent` |
---
### `POSTGRES_PASSWORD`
PostgreSQL superuser password.
| | |
|-|-|
| **Required for Compose** | Yes |
| **Default in `.env.example`** | `change-me-in-production` |
| **Example** | `POSTGRES_PASSWORD=strongpassword` |
> Never use the default value in production. Generate a strong random password.
---
### `POSTGRES_DB`
PostgreSQL database name to create on first startup.
| | |
|-|-|
| **Required for Compose** | Yes |
| **Default in `.env.example`** | `sentryagent_idp` |
| **Example** | `POSTGRES_DB=sentryagent_idp` |
---
### `GF_ADMIN_PASSWORD`
Grafana admin panel password — used by `compose.monitoring.yaml`.
| | |
|-|-|
| **Required for monitoring stack** | Yes |
| **Default in `.env.example`** | `change-me-in-production` |
| **Example** | `GF_ADMIN_PASSWORD=strongpassword` |
> Never use the default value in production.
---
## Required Variables
These variables must be set. The server will throw and exit immediately if any are missing.
@@ -438,6 +494,12 @@ NODE_ENV=development
PORT=3000
CORS_ORIGIN=http://localhost:3001
# ── Docker Compose (postgres container + monitoring) ─────────────────────────
POSTGRES_USER=sentryagent
POSTGRES_PASSWORD=change-me-in-production
POSTGRES_DB=sentryagent_idp
GF_ADMIN_PASSWORD=change-me-in-production
# ── Database ─────────────────────────────────────────────────────────────────
DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp
DB_POOL_MAX=20

View File

@@ -152,7 +152,10 @@ grep -E "^(DATABASE_URL|REDIS_URL|JWT_PRIVATE_KEY|JWT_PUBLIC_KEY|BILLING_ENABLED
Expected output (values abbreviated):
```
DATABASE_URL=postgresql://agentidp:password@localhost:5432/agentidp
POSTGRES_USER=sentryagent
POSTGRES_PASSWORD=sentryagent
POSTGRES_DB=sentryagent_idp
DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp
REDIS_URL=redis://localhost:6379
JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...
JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\n...
@@ -187,8 +190,8 @@ Expected output — all three services must show `healthy`:
```
NAME IMAGE STATUS
sentryagent-idp-app-1 sentryagent-idp-app running (healthy)
sentryagent-idp-postgres-1 postgres:14-alpine running (healthy)
sentryagent-idp-redis-1 redis:7-alpine running (healthy)
sentryagent-idp-postgres-1 postgres:14.12-alpine3.19 running (healthy)
sentryagent-idp-redis-1 redis:7.2-alpine3.19 running (healthy)
```
If any service shows `starting` or `unhealthy`, wait 15 seconds and run `docker compose ps`
@@ -787,7 +790,7 @@ Common causes:
| Service | Cause | Fix |
|---------|-------|-----|
| `postgres` | Wrong database credentials | Verify `DATABASE_URL` in `.env` matches `docker-compose.yml` credentials |
| `postgres` | Wrong database credentials | Verify `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` in `.env` match values in `compose.yaml` |
| `redis` | Port conflict | Check `lsof -ti:6379` and kill occupying process |
| `app` | Missing env var | Check `docker compose logs app` for `Failed to start server` message |
@@ -825,7 +828,7 @@ Cause: A previous partial migration run left the database in an inconsistent sta
Fix: Check which migrations have been applied:
```bash
docker compose exec postgres psql -U agentidp -d agentidp \
docker compose exec postgres psql -U sentryagent -d sentryagent_idp \
-c "SELECT name FROM schema_migrations ORDER BY name;"
```

View File

@@ -17,7 +17,7 @@ Verify versions:
```bash
docker --version
docker-compose --version
docker compose version
node --version
npm --version
```
@@ -57,18 +57,29 @@ Keep these files in the project root. They are used only locally and should not
## Step 3 — Configure environment
Create a `.env` file in the project root:
Copy the template and fill in your values:
```bash
cat > .env << 'ENVEOF'
cp .env.example .env
```
The template already includes all required variables. At minimum, verify these are set correctly for local development:
```
POSTGRES_USER=sentryagent
POSTGRES_PASSWORD=sentryagent
POSTGRES_DB=sentryagent_idp
DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp
REDIS_URL=redis://localhost:6379
PORT=3000
NODE_ENV=development
CORS_ORIGIN=*
ENVEOF
```
> **Note:** `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `POSTGRES_DB` are used by `compose.yaml`
> to configure the PostgreSQL container and construct `DATABASE_URL`. They are not read by
> the application directly — only `DATABASE_URL` is.
Append the JWT keys to `.env`:
```bash
@@ -86,10 +97,10 @@ grep -E "^(DATABASE_URL|REDIS_URL|JWT_PRIVATE_KEY|JWT_PUBLIC_KEY)" .env
## Step 4 — Start infrastructure services
The `docker-compose.yml` defines three services: `postgres`, `redis`, and `app`. For local development, start only the infrastructure services — the application runs directly via Node.js.
The `compose.yaml` defines three services: `postgres`, `redis`, and `app`. For local development, start only the infrastructure services — the application runs directly via Node.js.
```bash
docker-compose up -d postgres redis
docker compose up -d postgres redis
```
Expected output:
@@ -100,7 +111,7 @@ Expected output:
✔ Container sentryagent-idp-redis-1 Healthy
```
Both services must show `Healthy` before proceeding. If they show `Starting`, wait a few seconds and run `docker-compose ps` to recheck.
Both services must show `Healthy` before proceeding. If they show `Starting`, wait a few seconds and run `docker compose ps` to recheck.
### Service ports
@@ -112,18 +123,18 @@ Both services must show `Healthy` before proceeding. If they show `Starting`, wa
Verify manually:
```bash
docker-compose exec postgres pg_isready -U sentryagent -d sentryagent_idp
docker-compose exec redis redis-cli ping
docker compose exec postgres pg_isready -U sentryagent -d sentryagent_idp
docker compose exec redis redis-cli ping
```
### Docker volumes
Data is persisted in named Docker volumes:
Data is persisted in named Docker volumes (kebab-case per Compose Spec standard):
| Volume | Service | Contents |
|--------|---------|---------|
| `sentryagent-idp_postgres_data` | PostgreSQL | All database data |
| `sentryagent-idp_redis_data` | Redis | Redis persistence (if enabled) |
| `sentryagent-idp_postgres-data` | PostgreSQL | All database data |
| `sentryagent-idp_redis-data` | Redis | Redis persistence (if enabled) |
---
@@ -222,15 +233,13 @@ CORS_ORIGIN=http://localhost:3001
> deployments — see the [field trial guide](field-trial.md). For day-to-day development, start
> only the infrastructure services and run the application directly.
When the Dockerfile is available, the entire stack (infrastructure + application) can be started with:
The entire stack (infrastructure + application) can be started with:
```bash
docker-compose up -d
docker compose up --build -d
```
The `app` service depends on `postgres` and `redis` with health check conditions, so it will not start until both services are healthy.
Environment variables for the container are loaded from `.env` via the `env_file` directive in `docker-compose.yml`.
The `app` service depends on `postgres` and `redis` with health check conditions, so it will not start until both services are healthy. Environment variables are loaded from `.env` via the `env_file` directive in `compose.yaml` (`required: false` — the file is optional if env vars are injected directly).
---
@@ -239,19 +248,19 @@ Environment variables for the container are loaded from `.env` via the `env_file
Stop infrastructure only (preserves volumes):
```bash
docker-compose stop postgres redis
docker compose stop postgres redis
```
Stop and remove containers (preserves volumes):
```bash
docker-compose down
docker compose down
```
Stop and remove containers AND volumes (destroys all data):
```bash
docker-compose down -v
docker compose down -v
```
> Use `-v` only when you want a clean slate. This deletes all PostgreSQL data and Redis data permanently.

View File

@@ -111,7 +111,7 @@ Three key patterns are used in Redis. Useful for debugging and manual inspection
```bash
# Connect to Redis CLI
docker-compose exec redis redis-cli
docker compose exec redis redis-cli
```
| Key pattern | Example | Purpose | TTL |
@@ -192,10 +192,10 @@ Error: connect ECONNREFUSED 127.0.0.1:5432
| Cause | Fix |
|-------|-----|
| PostgreSQL container not started | Run `docker-compose up -d postgres` |
| PostgreSQL container not yet healthy | Wait and run `docker-compose ps` — wait for `healthy` |
| PostgreSQL container not started | Run `docker compose up -d postgres` |
| PostgreSQL container not yet healthy | Wait and run `docker compose ps` — wait for `healthy` |
| Wrong `DATABASE_URL` host/port | Check `DATABASE_URL` matches the PostgreSQL port (5432) |
| PostgreSQL container exited | Run `docker-compose logs postgres` to see why it exited |
| PostgreSQL container exited | Run `docker compose logs postgres` to see why it exited |
---
@@ -210,8 +210,8 @@ Redis client error Error: connect ECONNREFUSED 127.0.0.1:6379
| Cause | Fix |
|-------|-----|
| Redis container not started | Run `docker-compose up -d redis` |
| Redis container not yet healthy | Run `docker-compose ps` — wait for `healthy` |
| Redis container not started | Run `docker compose up -d redis` |
| Redis container not yet healthy | Run `docker compose ps` — wait for `healthy` |
| Wrong `REDIS_URL` | Check `REDIS_URL` matches the Redis port (6379) |
---
@@ -257,7 +257,7 @@ If a migration is listed there but the table is inconsistent, manually inspect a
# Find the current window key
WINDOW=$(node -e "console.log(Math.floor(Date.now() / 60000))")
# Check count for a specific client
docker-compose exec redis redis-cli GET "rate:<client_id>:$WINDOW"
docker compose exec redis redis-cli GET "rate:<client_id>:$WINDOW"
```
**Fix:** Wait until `X-RateLimit-Reset` (Unix timestamp in the response header) before retrying. The window resets every 60 seconds.
@@ -296,10 +296,10 @@ AgentIdP exposes a Prometheus metrics endpoint at `GET /metrics` (unauthenticate
```bash
# Start the full stack with monitoring
docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d
docker compose -f compose.yaml -f compose.monitoring.yaml up -d
# Prometheus: http://localhost:9090
# Grafana: http://localhost:3001 (admin / agentidp)
# Grafana: http://localhost:3001 (admin / <GF_ADMIN_PASSWORD from .env>)
```
The Grafana dashboard auto-provisions on first start. Navigate to **Dashboards → AgentIdP → SentryAgent.ai — AgentIdP**.

View File

@@ -123,8 +123,8 @@ rate-limiter uses a Redis sorted set for the sliding-window algorithm.
- PostgreSQL for revocation — rejected because the token verification path is the hot path in every authenticated request. A PostgreSQL round-trip adds 515 ms compared to a Redis `GET` at sub-millisecond latency.
**Consequences**: Redis is a required infrastructure dependency. A Redis instance must
be running and reachable via `REDIS_URL` before the server starts. `docker-compose.yml`
provides a Redis 7 Alpine container for local development on port 6379.
be running and reachable via `REDIS_URL` before the server starts. `compose.yaml`
provides a Redis 7.2 Alpine container for local development on port 6379.
---
@@ -217,7 +217,7 @@ environments. The `prom-client` npm package integrates natively with Express and
provides `Counter` and `Histogram` metric types that cover all observability needs for
AgentIdP. Grafana's YAML provisioning in `monitoring/grafana/provisioning/` makes
dashboards reproducible and version-controlled. The monitoring stack runs as a Docker
Compose overlay (`docker-compose.monitoring.yml`) without interfering with the base dev
Compose overlay (`compose.monitoring.yaml`) without interfering with the base dev
environment.
**Alternatives considered**:

View File

@@ -56,8 +56,8 @@ sentryagent-idp/
│ ├── agntcy-conformance/ # AGNTCY conformance test suite (separate Jest config)
│ └── load/ # k6 load test scripts
├── Dockerfile # Multi-stage production build (build + runtime stages)
├── docker-compose.yml # Local development: PostgreSQL 14 (port 5432) + Redis 7 (port 6379)
├── docker-compose.monitoring.yml # Monitoring overlay: Prometheus (port 9090) + Grafana (port 3001)
├── compose.yaml # Local development: PostgreSQL 14.12 (port 5432) + Redis 7.2 (port 6379)
├── compose.monitoring.yaml # Monitoring overlay: Prometheus (port 9090) + Grafana (port 3001)
├── package.json # Node.js dependencies and npm scripts
├── tsconfig.json # TypeScript strict configuration — compiled to dist/
└── jest.config.ts # Jest configuration — ts-jest, test timeouts, coverage thresholds
@@ -134,11 +134,14 @@ The `errorHandler` middleware in `src/middleware/errorHandler.ts` maps
`SentryAgentError` subclasses to their `httpStatus` codes and serialises the response
as `IErrorResponse { code, message, details }`.
**`docker-compose.yml`**
Starts PostgreSQL 14 (Alpine) on port 5432 with database `sentryagent_idp` and
Redis 7 (Alpine) on port 6379. Used for local development only. Both services have
health checks so `depends_on` conditions work correctly. The `app` service mounts
`./src` as a read-only volume for live code reloading.
**`compose.yaml`**
Starts PostgreSQL 14.12 (Alpine) on port 5432 and Redis 7.2 (Alpine) on port 6379.
All services use a dedicated `app-tier` bridge network, `restart: unless-stopped`,
and `deploy.resources.limits` per DockerSpec standards. Both infrastructure services
have health checks so `depends_on` conditions work correctly. The `app` service mounts
`./src` as a read-only bind volume for live code reloading and has its own
`healthcheck` probe via `curl /health`. Postgres credentials and Grafana admin
password are externalized to environment variables — see `docs/devops/environment-variables.md`.
**`tsconfig.json`**
TypeScript compiler configuration. `strict: true` enables the full suite of strictness

View File

@@ -332,10 +332,10 @@ not exposed to the public internet.
Start the monitoring overlay:
```bash
docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up
docker compose -f compose.yaml -f compose.monitoring.yaml up
```
- Prometheus: `http://localhost:9090`
- Grafana: `http://localhost:3001` — default credentials: `admin` / `agentidp`
- Grafana: `http://localhost:3001` — credentials: `admin` / `<GF_ADMIN_PASSWORD from .env>`
Grafana is pre-provisioned with a Prometheus data source pointing to `http://prometheus:9090`
and dashboard JSON files from `monitoring/grafana/dashboards/`. No manual configuration

View File

@@ -44,18 +44,24 @@ development dependencies (TypeScript, Jest, ts-jest, eslint).
## 8.3 Environment Variables Setup
The server requires a `.env` file at the project root. There is no `.env.example`
file — create it from scratch using the template below.
The server requires a `.env` file at the project root. Copy the template:
```bash
touch .env
cp .env.example .env
```
Add the following content to `.env`. Every variable is documented below.
The template includes all required variables with sensible local defaults. Edit `.env` to set your values. Key variables are documented below.
```bash
# ─────────────────────────────────────────────────────────────
# PostgreSQL connection
# PostgreSQL — individual credentials for compose.yaml
# ─────────────────────────────────────────────────────────────
POSTGRES_USER=sentryagent
POSTGRES_PASSWORD=sentryagent
POSTGRES_DB=sentryagent_idp
# ─────────────────────────────────────────────────────────────
# PostgreSQL connection (application reads this directly)
# ─────────────────────────────────────────────────────────────
DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp

View File

@@ -8,12 +8,12 @@ This document covers building and running AgentIdP in production: Docker, enviro
The Dockerfile uses a two-stage build:
- **Stage 1 (builder):** `node:18-alpine` — installs all dependencies (including dev) and compiles TypeScript to `dist/`.
- **Stage 2 (production):** `node:18-alpine` — copies `dist/` and `node_modules` (production only), runs as the built-in non-root `node` user.
- **Stage 1 (build):** `node:20.11-bookworm-slim` — installs all dependencies (including dev) and compiles TypeScript to `dist/`.
- **Stage 2 (final):** `node:20.11-bookworm-slim` — copies `dist/` and `node_modules` (production only), installs `curl` for healthcheck, and runs as the created non-root `nodeapp` user (UID 1001).
```bash
# Build
docker build -t sentryagent-idp:latest .
docker build -t sentryagent-idp:1.0.0 .
# Run (supply required env vars)
docker run -d \
@@ -22,18 +22,18 @@ docker run -d \
-e REDIS_URL=redis://<host>:6379 \
-e JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n..." \
-e JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\n..." \
sentryagent-idp:latest
sentryagent-idp:1.0.0
```
The container exposes port `3000`. Override with `PORT` environment variable if needed.
The container exposes port `3000`. Override with `PORT` environment variable if needed. The container runs as non-root user `nodeapp` (UID 1001) — do not mount volumes requiring root ownership.
For local full-stack development, use Docker Compose instead:
```bash
docker compose up -d
docker compose up --build -d
```
The `docker-compose.yml` starts the app, PostgreSQL 14, and Redis 7 with health checks and data volumes.
The `compose.yaml` starts the app, PostgreSQL 14.12, and Redis 7.2 with health checks, resource limits, restart policies, and data volumes — per DockerSpec standards.
---
@@ -178,11 +178,11 @@ The HTTP metrics (`agentidp_http_requests_total` and `agentidp_http_request_dura
### Local Grafana
```bash
docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d
docker compose -f compose.yaml -f compose.monitoring.yaml up -d
```
- Prometheus: http://localhost:9090
- Grafana: http://localhost:3001 (admin password: `agentidp`)
- Grafana: http://localhost:3001 (admin password: `GF_ADMIN_PASSWORD` value from `.env`)
The monitoring compose overlay starts `prom/prometheus:v2.53.0` and `grafana/grafana:11.2.0`. Grafana dashboards and datasource provisioning are loaded from `monitoring/grafana/provisioning/`.