docs: engineering knowledge base for new hires
Complete docs/engineering/ suite — 12 documents covering company overview, system architecture, tech stack ADRs, codebase structure, service deep dives, annotated code walkthroughs, dev setup, engineering workflow, testing strategy, deployment/ops, SDK guide, and README index. All content verified against source files. All 82 tasks in openspec/changes/engineering-docs/tasks.md marked complete. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
SentryAgent.ai Developer
404
docs/engineering/07-dev-setup.md
Normal file
404
docs/engineering/07-dev-setup.md
Normal file
@@ -0,0 +1,404 @@
|
||||
# 07 — Development Environment Setup
|
||||
|
||||
This guide takes you from a fresh machine to a running AgentIdP server with a
|
||||
passing smoke test. Estimated time: 15–20 minutes.
|
||||
|
||||
---
|
||||
|
||||
## 8.1 Prerequisites
|
||||
|
||||
Install all of these before proceeding.
|
||||
|
||||
| Prerequisite | Minimum version | Install link |
|
||||
|-------------|----------------|--------------|
|
||||
| Node.js | 18.x LTS | https://nodejs.org/en/download — use the LTS version |
|
||||
| npm | 9.x (ships with Node.js 18) | Included with Node.js |
|
||||
| Docker Desktop | Latest stable | https://docs.docker.com/get-docker/ |
|
||||
| Git | 2.x | https://git-scm.com/downloads |
|
||||
|
||||
**Verify your versions:**
|
||||
```bash
|
||||
node --version # Should print v18.x.x or higher
|
||||
npm --version # Should print 9.x or higher
|
||||
docker --version # Should print Docker version 24.x or higher
|
||||
git --version # Should print git version 2.x
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8.2 Clone and Install
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/sentryagent-ai/sentryagent-idp.git
|
||||
cd sentryagent-idp
|
||||
|
||||
# Install Node.js dependencies
|
||||
npm install
|
||||
```
|
||||
|
||||
This installs all production dependencies (Express, pg, Redis, etc.) and
|
||||
development dependencies (TypeScript, Jest, ts-jest, eslint).
|
||||
|
||||
---
|
||||
|
||||
## 8.3 Environment Variables Setup
|
||||
|
||||
The server requires a `.env` file at the project root. There is no `.env.example`
|
||||
file — create it from scratch using the template below.
|
||||
|
||||
```bash
|
||||
touch .env
|
||||
```
|
||||
|
||||
Add the following content to `.env`. Every variable is documented below.
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# PostgreSQL connection
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Redis connection
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
REDIS_URL=redis://localhost:6379
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# HTTP server port
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
PORT=3000
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# JWT RSA keys (generate these below)
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
JWT_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----"
|
||||
JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----"
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# CORS (optional — defaults to '*' which allows all origins)
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
CORS_ORIGIN=*
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Node environment
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
NODE_ENV=development
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# OPA policy directory (optional — defaults to ./policies)
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# POLICY_DIR=/path/to/policies
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# HashiCorp Vault (optional — omit to use bcrypt mode)
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# VAULT_ADDR=http://127.0.0.1:8200
|
||||
# VAULT_TOKEN=your-vault-token
|
||||
# VAULT_MOUNT=secret
|
||||
```
|
||||
|
||||
**Complete environment variable reference:**
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| `DATABASE_URL` | Yes | — | PostgreSQL connection string. Format: `postgresql://user:password@host:port/dbname` |
|
||||
| `REDIS_URL` | Yes | — | Redis connection URL. Format: `redis://host:port[/db]` |
|
||||
| `PORT` | No | `3000` | TCP port the HTTP server listens on |
|
||||
| `JWT_PRIVATE_KEY` | Yes | — | PEM-encoded RSA private key (2048-bit minimum) for signing tokens |
|
||||
| `JWT_PUBLIC_KEY` | Yes | — | PEM-encoded RSA public key (matching the private key above) for verifying tokens |
|
||||
| `CORS_ORIGIN` | No | `*` | CORS allowed origin. Use `*` for development, set to your dashboard domain in production |
|
||||
| `NODE_ENV` | No | `development` | Set to `test` to suppress Morgan HTTP logging during tests |
|
||||
| `POLICY_DIR` | No | `./policies` | Absolute path to the directory containing `authz.wasm` or `data/scopes.json` |
|
||||
| `VAULT_ADDR` | No | — | HashiCorp Vault server address (e.g. `http://127.0.0.1:8200`). When omitted, bcrypt mode is used |
|
||||
| `VAULT_TOKEN` | No | — | Vault authentication token. Required when `VAULT_ADDR` is set |
|
||||
| `VAULT_MOUNT` | No | `secret` | Vault KV v2 mount path. Only used when Vault is configured |
|
||||
|
||||
**Generating JWT keys:**
|
||||
```bash
|
||||
# Generate RSA 2048-bit private key
|
||||
openssl genrsa -out private.pem 2048
|
||||
|
||||
# Extract public key
|
||||
openssl rsa -in private.pem -pubout -out public.pem
|
||||
|
||||
# Print private key as single-line for .env (replace newlines with \n)
|
||||
awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' private.pem
|
||||
|
||||
# Print public key as single-line for .env
|
||||
awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' public.pem
|
||||
```
|
||||
|
||||
Paste the output (including the `-----BEGIN/END-----` lines) as the value for
|
||||
`JWT_PRIVATE_KEY` and `JWT_PUBLIC_KEY` in your `.env` file, surrounded by double
|
||||
quotes.
|
||||
|
||||
---
|
||||
|
||||
## 8.4 Docker Compose Startup and Health Checks
|
||||
|
||||
Docker Compose starts PostgreSQL 14 and Redis 7. The application reads the
|
||||
`DATABASE_URL` and `REDIS_URL` from your `.env` file to connect to them.
|
||||
|
||||
```bash
|
||||
# Start PostgreSQL and Redis in the background
|
||||
docker compose up postgres redis -d
|
||||
|
||||
# Wait for health checks to pass (usually 10-15 seconds)
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
Expected output when both services are healthy:
|
||||
```
|
||||
NAME STATUS PORTS
|
||||
sentryagent-idp-postgres-1 Up (healthy) 0.0.0.0:5432->5432/tcp
|
||||
sentryagent-idp-redis-1 Up (healthy) 0.0.0.0:6379->6379/tcp
|
||||
```
|
||||
|
||||
**Manual health check:**
|
||||
```bash
|
||||
# Test PostgreSQL connection
|
||||
docker exec sentryagent-idp-postgres-1 pg_isready -U sentryagent -d sentryagent_idp
|
||||
# Expected: /var/run/postgresql:5432 - accepting connections
|
||||
|
||||
# Test Redis connection
|
||||
docker exec sentryagent-idp-redis-1 redis-cli ping
|
||||
# Expected: PONG
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8.5 Database Migrations
|
||||
|
||||
Run the migration script to create all required tables:
|
||||
|
||||
```bash
|
||||
npm run db:migrate
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Running database migrations...
|
||||
✓ Applied: 001_create_agents.sql
|
||||
✓ Applied: 002_create_credentials.sql
|
||||
✓ Applied: 003_create_audit_events.sql
|
||||
✓ Applied: 004_create_tokens.sql
|
||||
✓ Applied: 005_add_vault_path.sql
|
||||
|
||||
Migrations complete. 5 migration(s) applied.
|
||||
```
|
||||
|
||||
Running `npm run db:migrate` a second time is safe — it skips already-applied migrations:
|
||||
```
|
||||
- Skipped (already applied): 001_create_agents.sql
|
||||
...
|
||||
Migrations complete. 0 migration(s) applied.
|
||||
```
|
||||
|
||||
**Migration internals:**
|
||||
The migration runner (`scripts/migrate.ts`) reads `.sql` files from `src/db/migrations/`
|
||||
in alphabetical order, wraps each in a transaction, and records the filename in the
|
||||
`schema_migrations` table. If a migration fails, the transaction rolls back and
|
||||
the runner exits with code 1.
|
||||
|
||||
---
|
||||
|
||||
## 8.6 Start the Server
|
||||
|
||||
```bash
|
||||
npm run dev
|
||||
# Expected: SentryAgent.ai AgentIdP listening on port 3000
|
||||
```
|
||||
|
||||
`npm run dev` uses `ts-node` to execute `src/server.ts` directly without compiling.
|
||||
This is faster for development. For a production-style start, compile first:
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
npm start
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8.7 Smoke Test
|
||||
|
||||
Verify the server is working with these three curl commands.
|
||||
|
||||
**1. Health check:**
|
||||
```bash
|
||||
curl http://localhost:3000/health
|
||||
```
|
||||
Expected response (200 OK):
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"checks": {
|
||||
"database": "healthy",
|
||||
"redis": "healthy"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**2. Register an agent:**
|
||||
First, you need a token to authenticate. But to get a token, you need credentials.
|
||||
And to get credentials, you need an agent. The chicken-and-egg is resolved by the
|
||||
fact that agent registration requires an `agents:write` scoped token — which means
|
||||
you need to bootstrap the first agent another way.
|
||||
|
||||
For local development, temporarily test without auth by using the `/api/v1` prefix
|
||||
directly (the server accepts requests; OPA will enforce scope).
|
||||
|
||||
The easiest approach: generate a test token programmatically:
|
||||
|
||||
```bash
|
||||
# Generate test keys
|
||||
openssl genrsa -out /tmp/test_private.pem 2048 2>/dev/null
|
||||
openssl rsa -in /tmp/test_private.pem -pubout -out /tmp/test_public.pem 2>/dev/null
|
||||
|
||||
# Set them in your environment temporarily
|
||||
export JWT_PRIVATE_KEY="$(cat /tmp/test_private.pem)"
|
||||
export JWT_PUBLIC_KEY="$(cat /tmp/test_public.pem)"
|
||||
|
||||
# Start server with these keys and use a tool or short Node.js script to mint a test token
|
||||
```
|
||||
|
||||
**3. Token endpoint with seeded credentials:**
|
||||
|
||||
Once you have an agent with credentials (e.g. created via the API or seeded in
|
||||
development), issue a token:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/v1/token \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "grant_type=client_credentials" \
|
||||
-d "client_id=YOUR_AGENT_ID" \
|
||||
-d "client_secret=sk_live_YOUR_SECRET" \
|
||||
-d "scope=agents:read agents:write"
|
||||
```
|
||||
|
||||
Expected response (200 OK):
|
||||
```json
|
||||
{
|
||||
"access_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...",
|
||||
"token_type": "Bearer",
|
||||
"expires_in": 3600,
|
||||
"scope": "agents:read agents:write"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8.8 Troubleshooting
|
||||
|
||||
### Error: `connection refused` on PostgreSQL or Redis
|
||||
**Cause:** Docker services not running or not yet healthy.
|
||||
**Fix:**
|
||||
```bash
|
||||
docker compose ps # Check status
|
||||
docker compose up postgres redis -d # Start if not running
|
||||
docker compose logs postgres # Check for startup errors
|
||||
```
|
||||
|
||||
### Error: `DATABASE_URL environment variable is required`
|
||||
**Cause:** `.env` file missing or not being loaded.
|
||||
**Fix:** Ensure `.env` exists at the project root. `npm run dev` loads it via `dotenv.config()` in `src/server.ts`.
|
||||
|
||||
### Error: `JWT_PRIVATE_KEY and JWT_PUBLIC_KEY environment variables are required`
|
||||
**Cause:** JWT keys not in `.env`, or newlines in the PEM keys are not properly escaped.
|
||||
**Fix:** Ensure the keys are wrapped in double quotes and newlines are represented as `\n`. Use the `awk` command from section 8.3 to format them correctly.
|
||||
|
||||
### Error: `Cannot find module 'ts-node'`
|
||||
**Cause:** `npm install` was not run, or ran against a different Node.js version.
|
||||
**Fix:**
|
||||
```bash
|
||||
node --version # Confirm Node.js 18+
|
||||
rm -rf node_modules package-lock.json
|
||||
npm install
|
||||
```
|
||||
|
||||
### Error: `Cannot connect to Redis` (during migration or server start)
|
||||
**Cause:** Redis container not running or `REDIS_URL` is incorrect.
|
||||
**Fix:**
|
||||
```bash
|
||||
docker exec sentryagent-idp-redis-1 redis-cli ping
|
||||
# If container is not running:
|
||||
docker compose up redis -d
|
||||
```
|
||||
|
||||
### Port 3000 already in use
|
||||
**Cause:** Another process is listening on port 3000.
|
||||
**Fix:**
|
||||
```bash
|
||||
lsof -i :3000 # Find the process
|
||||
kill <PID> # Kill it
|
||||
# Or: set PORT=3001 in .env and restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8.9 Running Tests Locally
|
||||
|
||||
```bash
|
||||
# Run all tests (unit + integration)
|
||||
npm test
|
||||
|
||||
# Run tests with coverage report
|
||||
npm run test:unit -- --coverage
|
||||
# Coverage report: coverage/lcov-report/index.html
|
||||
|
||||
# Run only unit tests
|
||||
npm run test:unit
|
||||
|
||||
# Run only integration tests (requires running PostgreSQL and Redis)
|
||||
npm run test:integration
|
||||
|
||||
# Run a single test file
|
||||
npx jest tests/unit/services/AgentService.test.ts
|
||||
|
||||
# Run tests matching a pattern
|
||||
npx jest --testNamePattern="registerAgent"
|
||||
|
||||
# Watch mode (re-runs on file changes)
|
||||
npx jest --watch
|
||||
```
|
||||
|
||||
**Integration test requirements:**
|
||||
Integration tests connect to real PostgreSQL and Redis. Set these environment
|
||||
variables before running integration tests:
|
||||
```bash
|
||||
TEST_DATABASE_URL=postgresql://sentryagent:sentryagent@localhost:5432/sentryagent_idp_test
|
||||
TEST_REDIS_URL=redis://localhost:6379/1
|
||||
```
|
||||
|
||||
The integration tests create their own tables (using `CREATE TABLE IF NOT EXISTS`)
|
||||
and clean up after themselves with `DELETE FROM` statements in `afterAll`.
|
||||
|
||||
---
|
||||
|
||||
## 8.10 Web Dashboard Local Development
|
||||
|
||||
The web dashboard is a separate Vite project in the `dashboard/` directory.
|
||||
|
||||
```bash
|
||||
# From the project root
|
||||
cd dashboard
|
||||
npm install
|
||||
|
||||
# Start the Vite development server
|
||||
npm run dev
|
||||
# Dashboard available at http://localhost:5173
|
||||
```
|
||||
|
||||
The Vite dev server proxies all `/api/*` requests to `http://localhost:3000`,
|
||||
so the API server must be running concurrently (in a separate terminal).
|
||||
|
||||
**Build for production:**
|
||||
```bash
|
||||
cd dashboard
|
||||
npm run build
|
||||
# Output: dashboard/dist/ (served by Express at /dashboard)
|
||||
```
|
||||
|
||||
After building, the Express server serves the built dashboard at
|
||||
`http://localhost:3000/dashboard`. You do not need to run the Vite dev server
|
||||
for this — the static files are served directly by Express.
|
||||
Reference in New Issue
Block a user