diff --git a/docs/compliance/encryption-runbook.md b/docs/compliance/encryption-runbook.md index b4ccf0c..ef3f790 100644 --- a/docs/compliance/encryption-runbook.md +++ b/docs/compliance/encryption-runbook.md @@ -68,7 +68,7 @@ The `EncryptionService` caches the key in process memory. A restart forces a re- kubectl rollout restart deployment/agentidp # Docker Compose -docker-compose restart agentidp +docker compose restart app # PM2 pm2 restart agentidp diff --git a/docs/developers/quick-start.md b/docs/developers/quick-start.md index 4cf2fcd..a582412 100644 --- a/docs/developers/quick-start.md +++ b/docs/developers/quick-start.md @@ -6,7 +6,7 @@ This guide gets you from zero to a working agent identity inside an organization You need two tools installed: -- **Docker** (includes `docker-compose`) — to run PostgreSQL and Redis +- **Docker** (with Compose plugin, v2.20+) — to run PostgreSQL and Redis - **Node.js 18+** (includes `npm`) — to run the server - **curl** — to call the API @@ -32,16 +32,19 @@ openssl genrsa -out private.pem 2048 openssl rsa -in private.pem -pubout -out public.pem ``` -Create your `.env` file: +Copy the environment template and fill in your JWT keys: ```bash -cat > .env << 'EOF' -DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp -REDIS_URL=redis://localhost:6379 -PORT=3000 -JWT_PRIVATE_KEY="$(cat private.pem)" -JWT_PUBLIC_KEY="$(cat public.pem)" -EOF +cp .env.example .env +``` + +Write your JWT keys into `.env`: + +```bash +PRIVATE_KEY_LINE=$(awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' private.pem) +PUBLIC_KEY_LINE=$(awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' public.pem) +sed -i "s|JWT_PRIVATE_KEY=.*|JWT_PRIVATE_KEY=\"${PRIVATE_KEY_LINE}\"|" .env +sed -i "s|JWT_PUBLIC_KEY=.*|JWT_PUBLIC_KEY=\"${PUBLIC_KEY_LINE}\"|" .env ``` > **Note**: The `.env` file stores your private key. Do not commit it to version control. @@ -53,7 +56,7 @@ EOF Start PostgreSQL and Redis using Docker Compose (infrastructure services only): ```bash -docker-compose up -d postgres redis +docker compose up -d postgres redis ``` Expected output: diff --git a/docs/devops/README.md b/docs/devops/README.md index 99b0344..373eb52 100644 --- a/docs/devops/README.md +++ b/docs/devops/README.md @@ -19,7 +19,7 @@ SentryAgent.ai AgentIdP is a Node.js REST API backed by PostgreSQL and Redis. It | [Architecture](architecture.md) | All engineers | Components, ports, data flow, Redis key patterns | | [Environment Variables](environment-variables.md) | All engineers | Every env var — required, optional, format, examples | | [Database](database.md) | Backend, DevOps | Schema (26 tables/migrations), how to apply and verify | -| [Local Development](local-development.md) | All engineers | docker-compose setup, startup, health checks | +| [Local Development](local-development.md) | All engineers | Docker Compose setup (`compose.yaml`), startup, health checks | | [Security](security.md) | All engineers | JWT key generation and rotation, CORS, secret storage | | [Operations](operations.md) | DevOps | Startup order, graceful shutdown, log interpretation, troubleshooting | | [field-trial.md](field-trial.md) | DevOps engineers, QA | In-house Docker Compose field trial execution playbook | diff --git a/docs/devops/environment-variables.md b/docs/devops/environment-variables.md index 62d9ea7..246e471 100644 --- a/docs/devops/environment-variables.md +++ b/docs/devops/environment-variables.md @@ -6,6 +6,62 @@ Variables are loaded from a `.env` file at startup via `dotenv`. In production, --- +## Docker Compose Variables + +These variables are read by `compose.yaml` — not by the application itself. They are required when running the stack via `docker compose up`. + +### `POSTGRES_USER` + +PostgreSQL superuser name — used to configure the `postgres` container and construct `DATABASE_URL`. + +| | | +|-|-| +| **Required for Compose** | Yes | +| **Default in `.env.example`** | `sentryagent` | +| **Example** | `POSTGRES_USER=sentryagent` | + +--- + +### `POSTGRES_PASSWORD` + +PostgreSQL superuser password. + +| | | +|-|-| +| **Required for Compose** | Yes | +| **Default in `.env.example`** | `change-me-in-production` | +| **Example** | `POSTGRES_PASSWORD=strongpassword` | + +> Never use the default value in production. Generate a strong random password. + +--- + +### `POSTGRES_DB` + +PostgreSQL database name to create on first startup. + +| | | +|-|-| +| **Required for Compose** | Yes | +| **Default in `.env.example`** | `sentryagent_idp` | +| **Example** | `POSTGRES_DB=sentryagent_idp` | + +--- + +### `GF_ADMIN_PASSWORD` + +Grafana admin panel password — used by `compose.monitoring.yaml`. + +| | | +|-|-| +| **Required for monitoring stack** | Yes | +| **Default in `.env.example`** | `change-me-in-production` | +| **Example** | `GF_ADMIN_PASSWORD=strongpassword` | + +> Never use the default value in production. + +--- + ## Required Variables These variables must be set. The server will throw and exit immediately if any are missing. @@ -438,6 +494,12 @@ NODE_ENV=development PORT=3000 CORS_ORIGIN=http://localhost:3001 +# ── Docker Compose (postgres container + monitoring) ───────────────────────── +POSTGRES_USER=sentryagent +POSTGRES_PASSWORD=change-me-in-production +POSTGRES_DB=sentryagent_idp +GF_ADMIN_PASSWORD=change-me-in-production + # ── Database ───────────────────────────────────────────────────────────────── DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp DB_POOL_MAX=20 diff --git a/docs/devops/field-trial.md b/docs/devops/field-trial.md index 9f4c875..803eb42 100644 --- a/docs/devops/field-trial.md +++ b/docs/devops/field-trial.md @@ -152,7 +152,10 @@ grep -E "^(DATABASE_URL|REDIS_URL|JWT_PRIVATE_KEY|JWT_PUBLIC_KEY|BILLING_ENABLED Expected output (values abbreviated): ``` -DATABASE_URL=postgresql://agentidp:password@localhost:5432/agentidp +POSTGRES_USER=sentryagent +POSTGRES_PASSWORD=sentryagent +POSTGRES_DB=sentryagent_idp +DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp REDIS_URL=redis://localhost:6379 JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n... JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\n... @@ -185,10 +188,10 @@ docker compose ps Expected output — all three services must show `healthy`: ``` -NAME IMAGE STATUS -sentryagent-idp-app-1 sentryagent-idp-app running (healthy) -sentryagent-idp-postgres-1 postgres:14-alpine running (healthy) -sentryagent-idp-redis-1 redis:7-alpine running (healthy) +NAME IMAGE STATUS +sentryagent-idp-app-1 sentryagent-idp-app running (healthy) +sentryagent-idp-postgres-1 postgres:14.12-alpine3.19 running (healthy) +sentryagent-idp-redis-1 redis:7.2-alpine3.19 running (healthy) ``` If any service shows `starting` or `unhealthy`, wait 15 seconds and run `docker compose ps` @@ -787,7 +790,7 @@ Common causes: | Service | Cause | Fix | |---------|-------|-----| -| `postgres` | Wrong database credentials | Verify `DATABASE_URL` in `.env` matches `docker-compose.yml` credentials | +| `postgres` | Wrong database credentials | Verify `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` in `.env` match values in `compose.yaml` | | `redis` | Port conflict | Check `lsof -ti:6379` and kill occupying process | | `app` | Missing env var | Check `docker compose logs app` for `Failed to start server` message | @@ -825,7 +828,7 @@ Cause: A previous partial migration run left the database in an inconsistent sta Fix: Check which migrations have been applied: ```bash -docker compose exec postgres psql -U agentidp -d agentidp \ +docker compose exec postgres psql -U sentryagent -d sentryagent_idp \ -c "SELECT name FROM schema_migrations ORDER BY name;" ``` diff --git a/docs/devops/local-development.md b/docs/devops/local-development.md index 7d8aaac..67112f9 100644 --- a/docs/devops/local-development.md +++ b/docs/devops/local-development.md @@ -17,7 +17,7 @@ Verify versions: ```bash docker --version -docker-compose --version +docker compose version node --version npm --version ``` @@ -57,18 +57,29 @@ Keep these files in the project root. They are used only locally and should not ## Step 3 — Configure environment -Create a `.env` file in the project root: +Copy the template and fill in your values: ```bash -cat > .env << 'ENVEOF' +cp .env.example .env +``` + +The template already includes all required variables. At minimum, verify these are set correctly for local development: + +``` +POSTGRES_USER=sentryagent +POSTGRES_PASSWORD=sentryagent +POSTGRES_DB=sentryagent_idp DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp REDIS_URL=redis://localhost:6379 PORT=3000 NODE_ENV=development CORS_ORIGIN=* -ENVEOF ``` +> **Note:** `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `POSTGRES_DB` are used by `compose.yaml` +> to configure the PostgreSQL container and construct `DATABASE_URL`. They are not read by +> the application directly — only `DATABASE_URL` is. + Append the JWT keys to `.env`: ```bash @@ -86,10 +97,10 @@ grep -E "^(DATABASE_URL|REDIS_URL|JWT_PRIVATE_KEY|JWT_PUBLIC_KEY)" .env ## Step 4 — Start infrastructure services -The `docker-compose.yml` defines three services: `postgres`, `redis`, and `app`. For local development, start only the infrastructure services — the application runs directly via Node.js. +The `compose.yaml` defines three services: `postgres`, `redis`, and `app`. For local development, start only the infrastructure services — the application runs directly via Node.js. ```bash -docker-compose up -d postgres redis +docker compose up -d postgres redis ``` Expected output: @@ -100,7 +111,7 @@ Expected output: ✔ Container sentryagent-idp-redis-1 Healthy ``` -Both services must show `Healthy` before proceeding. If they show `Starting`, wait a few seconds and run `docker-compose ps` to recheck. +Both services must show `Healthy` before proceeding. If they show `Starting`, wait a few seconds and run `docker compose ps` to recheck. ### Service ports @@ -112,18 +123,18 @@ Both services must show `Healthy` before proceeding. If they show `Starting`, wa Verify manually: ```bash -docker-compose exec postgres pg_isready -U sentryagent -d sentryagent_idp -docker-compose exec redis redis-cli ping +docker compose exec postgres pg_isready -U sentryagent -d sentryagent_idp +docker compose exec redis redis-cli ping ``` ### Docker volumes -Data is persisted in named Docker volumes: +Data is persisted in named Docker volumes (kebab-case per Compose Spec standard): | Volume | Service | Contents | |--------|---------|---------| -| `sentryagent-idp_postgres_data` | PostgreSQL | All database data | -| `sentryagent-idp_redis_data` | Redis | Redis persistence (if enabled) | +| `sentryagent-idp_postgres-data` | PostgreSQL | All database data | +| `sentryagent-idp_redis-data` | Redis | Redis persistence (if enabled) | --- @@ -222,15 +233,13 @@ CORS_ORIGIN=http://localhost:3001 > deployments — see the [field trial guide](field-trial.md). For day-to-day development, start > only the infrastructure services and run the application directly. -When the Dockerfile is available, the entire stack (infrastructure + application) can be started with: +The entire stack (infrastructure + application) can be started with: ```bash -docker-compose up -d +docker compose up --build -d ``` -The `app` service depends on `postgres` and `redis` with health check conditions, so it will not start until both services are healthy. - -Environment variables for the container are loaded from `.env` via the `env_file` directive in `docker-compose.yml`. +The `app` service depends on `postgres` and `redis` with health check conditions, so it will not start until both services are healthy. Environment variables are loaded from `.env` via the `env_file` directive in `compose.yaml` (`required: false` — the file is optional if env vars are injected directly). --- @@ -239,19 +248,19 @@ Environment variables for the container are loaded from `.env` via the `env_file Stop infrastructure only (preserves volumes): ```bash -docker-compose stop postgres redis +docker compose stop postgres redis ``` Stop and remove containers (preserves volumes): ```bash -docker-compose down +docker compose down ``` Stop and remove containers AND volumes (destroys all data): ```bash -docker-compose down -v +docker compose down -v ``` > Use `-v` only when you want a clean slate. This deletes all PostgreSQL data and Redis data permanently. diff --git a/docs/devops/operations.md b/docs/devops/operations.md index 5383bd7..ebe53e5 100644 --- a/docs/devops/operations.md +++ b/docs/devops/operations.md @@ -111,7 +111,7 @@ Three key patterns are used in Redis. Useful for debugging and manual inspection ```bash # Connect to Redis CLI -docker-compose exec redis redis-cli +docker compose exec redis redis-cli ``` | Key pattern | Example | Purpose | TTL | @@ -192,10 +192,10 @@ Error: connect ECONNREFUSED 127.0.0.1:5432 | Cause | Fix | |-------|-----| -| PostgreSQL container not started | Run `docker-compose up -d postgres` | -| PostgreSQL container not yet healthy | Wait and run `docker-compose ps` — wait for `healthy` | +| PostgreSQL container not started | Run `docker compose up -d postgres` | +| PostgreSQL container not yet healthy | Wait and run `docker compose ps` — wait for `healthy` | | Wrong `DATABASE_URL` host/port | Check `DATABASE_URL` matches the PostgreSQL port (5432) | -| PostgreSQL container exited | Run `docker-compose logs postgres` to see why it exited | +| PostgreSQL container exited | Run `docker compose logs postgres` to see why it exited | --- @@ -210,8 +210,8 @@ Redis client error Error: connect ECONNREFUSED 127.0.0.1:6379 | Cause | Fix | |-------|-----| -| Redis container not started | Run `docker-compose up -d redis` | -| Redis container not yet healthy | Run `docker-compose ps` — wait for `healthy` | +| Redis container not started | Run `docker compose up -d redis` | +| Redis container not yet healthy | Run `docker compose ps` — wait for `healthy` | | Wrong `REDIS_URL` | Check `REDIS_URL` matches the Redis port (6379) | --- @@ -257,7 +257,7 @@ If a migration is listed there but the table is inconsistent, manually inspect a # Find the current window key WINDOW=$(node -e "console.log(Math.floor(Date.now() / 60000))") # Check count for a specific client -docker-compose exec redis redis-cli GET "rate::$WINDOW" +docker compose exec redis redis-cli GET "rate::$WINDOW" ``` **Fix:** Wait until `X-RateLimit-Reset` (Unix timestamp in the response header) before retrying. The window resets every 60 seconds. @@ -296,10 +296,10 @@ AgentIdP exposes a Prometheus metrics endpoint at `GET /metrics` (unauthenticate ```bash # Start the full stack with monitoring -docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d +docker compose -f compose.yaml -f compose.monitoring.yaml up -d # Prometheus: http://localhost:9090 -# Grafana: http://localhost:3001 (admin / agentidp) +# Grafana: http://localhost:3001 (admin / ) ``` The Grafana dashboard auto-provisions on first start. Navigate to **Dashboards → AgentIdP → SentryAgent.ai — AgentIdP**. diff --git a/docs/engineering/03-tech-stack.md b/docs/engineering/03-tech-stack.md index dd24f8b..0406926 100644 --- a/docs/engineering/03-tech-stack.md +++ b/docs/engineering/03-tech-stack.md @@ -123,8 +123,8 @@ rate-limiter uses a Redis sorted set for the sliding-window algorithm. - PostgreSQL for revocation — rejected because the token verification path is the hot path in every authenticated request. A PostgreSQL round-trip adds 5–15 ms compared to a Redis `GET` at sub-millisecond latency. **Consequences**: Redis is a required infrastructure dependency. A Redis instance must -be running and reachable via `REDIS_URL` before the server starts. `docker-compose.yml` -provides a Redis 7 Alpine container for local development on port 6379. +be running and reachable via `REDIS_URL` before the server starts. `compose.yaml` +provides a Redis 7.2 Alpine container for local development on port 6379. --- @@ -217,7 +217,7 @@ environments. The `prom-client` npm package integrates natively with Express and provides `Counter` and `Histogram` metric types that cover all observability needs for AgentIdP. Grafana's YAML provisioning in `monitoring/grafana/provisioning/` makes dashboards reproducible and version-controlled. The monitoring stack runs as a Docker -Compose overlay (`docker-compose.monitoring.yml`) without interfering with the base dev +Compose overlay (`compose.monitoring.yaml`) without interfering with the base dev environment. **Alternatives considered**: diff --git a/docs/engineering/04-codebase-structure.md b/docs/engineering/04-codebase-structure.md index 7a705f9..cc919f9 100644 --- a/docs/engineering/04-codebase-structure.md +++ b/docs/engineering/04-codebase-structure.md @@ -56,8 +56,8 @@ sentryagent-idp/ │ ├── agntcy-conformance/ # AGNTCY conformance test suite (separate Jest config) │ └── load/ # k6 load test scripts ├── Dockerfile # Multi-stage production build (build + runtime stages) -├── docker-compose.yml # Local development: PostgreSQL 14 (port 5432) + Redis 7 (port 6379) -├── docker-compose.monitoring.yml # Monitoring overlay: Prometheus (port 9090) + Grafana (port 3001) +├── compose.yaml # Local development: PostgreSQL 14.12 (port 5432) + Redis 7.2 (port 6379) +├── compose.monitoring.yaml # Monitoring overlay: Prometheus (port 9090) + Grafana (port 3001) ├── package.json # Node.js dependencies and npm scripts ├── tsconfig.json # TypeScript strict configuration — compiled to dist/ └── jest.config.ts # Jest configuration — ts-jest, test timeouts, coverage thresholds @@ -134,11 +134,14 @@ The `errorHandler` middleware in `src/middleware/errorHandler.ts` maps `SentryAgentError` subclasses to their `httpStatus` codes and serialises the response as `IErrorResponse { code, message, details }`. -**`docker-compose.yml`** -Starts PostgreSQL 14 (Alpine) on port 5432 with database `sentryagent_idp` and -Redis 7 (Alpine) on port 6379. Used for local development only. Both services have -health checks so `depends_on` conditions work correctly. The `app` service mounts -`./src` as a read-only volume for live code reloading. +**`compose.yaml`** +Starts PostgreSQL 14.12 (Alpine) on port 5432 and Redis 7.2 (Alpine) on port 6379. +All services use a dedicated `app-tier` bridge network, `restart: unless-stopped`, +and `deploy.resources.limits` per DockerSpec standards. Both infrastructure services +have health checks so `depends_on` conditions work correctly. The `app` service mounts +`./src` as a read-only bind volume for live code reloading and has its own +`healthcheck` probe via `curl /health`. Postgres credentials and Grafana admin +password are externalized to environment variables — see `docs/devops/environment-variables.md`. **`tsconfig.json`** TypeScript compiler configuration. `strict: true` enables the full suite of strictness diff --git a/docs/engineering/05-services.md b/docs/engineering/05-services.md index d6bd30e..d31875b 100644 --- a/docs/engineering/05-services.md +++ b/docs/engineering/05-services.md @@ -332,10 +332,10 @@ not exposed to the public internet. Start the monitoring overlay: ```bash -docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up +docker compose -f compose.yaml -f compose.monitoring.yaml up ``` - Prometheus: `http://localhost:9090` -- Grafana: `http://localhost:3001` — default credentials: `admin` / `agentidp` +- Grafana: `http://localhost:3001` — credentials: `admin` / `` Grafana is pre-provisioned with a Prometheus data source pointing to `http://prometheus:9090` and dashboard JSON files from `monitoring/grafana/dashboards/`. No manual configuration diff --git a/docs/engineering/07-dev-setup.md b/docs/engineering/07-dev-setup.md index 496e77d..5676f94 100644 --- a/docs/engineering/07-dev-setup.md +++ b/docs/engineering/07-dev-setup.md @@ -44,18 +44,24 @@ development dependencies (TypeScript, Jest, ts-jest, eslint). ## 8.3 Environment Variables Setup -The server requires a `.env` file at the project root. There is no `.env.example` -file — create it from scratch using the template below. +The server requires a `.env` file at the project root. Copy the template: ```bash -touch .env +cp .env.example .env ``` -Add the following content to `.env`. Every variable is documented below. +The template includes all required variables with sensible local defaults. Edit `.env` to set your values. Key variables are documented below. ```bash # ───────────────────────────────────────────────────────────── -# PostgreSQL connection +# PostgreSQL — individual credentials for compose.yaml +# ───────────────────────────────────────────────────────────── +POSTGRES_USER=sentryagent +POSTGRES_PASSWORD=sentryagent +POSTGRES_DB=sentryagent_idp + +# ───────────────────────────────────────────────────────────── +# PostgreSQL connection (application reads this directly) # ───────────────────────────────────────────────────────────── DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp diff --git a/docs/engineering/10-deployment.md b/docs/engineering/10-deployment.md index faf3630..fa84c05 100644 --- a/docs/engineering/10-deployment.md +++ b/docs/engineering/10-deployment.md @@ -8,12 +8,12 @@ This document covers building and running AgentIdP in production: Docker, enviro The Dockerfile uses a two-stage build: -- **Stage 1 (builder):** `node:18-alpine` — installs all dependencies (including dev) and compiles TypeScript to `dist/`. -- **Stage 2 (production):** `node:18-alpine` — copies `dist/` and `node_modules` (production only), runs as the built-in non-root `node` user. +- **Stage 1 (build):** `node:20.11-bookworm-slim` — installs all dependencies (including dev) and compiles TypeScript to `dist/`. +- **Stage 2 (final):** `node:20.11-bookworm-slim` — copies `dist/` and `node_modules` (production only), installs `curl` for healthcheck, and runs as the created non-root `nodeapp` user (UID 1001). ```bash # Build -docker build -t sentryagent-idp:latest . +docker build -t sentryagent-idp:1.0.0 . # Run (supply required env vars) docker run -d \ @@ -22,18 +22,18 @@ docker run -d \ -e REDIS_URL=redis://:6379 \ -e JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n..." \ -e JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\n..." \ - sentryagent-idp:latest + sentryagent-idp:1.0.0 ``` -The container exposes port `3000`. Override with `PORT` environment variable if needed. +The container exposes port `3000`. Override with `PORT` environment variable if needed. The container runs as non-root user `nodeapp` (UID 1001) — do not mount volumes requiring root ownership. For local full-stack development, use Docker Compose instead: ```bash -docker compose up -d +docker compose up --build -d ``` -The `docker-compose.yml` starts the app, PostgreSQL 14, and Redis 7 with health checks and data volumes. +The `compose.yaml` starts the app, PostgreSQL 14.12, and Redis 7.2 with health checks, resource limits, restart policies, and data volumes — per DockerSpec standards. --- @@ -178,11 +178,11 @@ The HTTP metrics (`agentidp_http_requests_total` and `agentidp_http_request_dura ### Local Grafana ```bash -docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d +docker compose -f compose.yaml -f compose.monitoring.yaml up -d ``` - Prometheus: http://localhost:9090 -- Grafana: http://localhost:3001 (admin password: `agentidp`) +- Grafana: http://localhost:3001 (admin password: `GF_ADMIN_PASSWORD` value from `.env`) The monitoring compose overlay starts `prom/prometheus:v2.53.0` and `grafana/grafana:11.2.0`. Grafana dashboards and datasource provisioning are loaded from `monitoring/grafana/provisioning/`.