Web APIs
Project Planning
Scope, phase, and document the Final Project before writing a single line of code — so you build the right thing in the right order with a clear definition of done.
Most developers start coding immediately and plan as they go. That works for small scripts. For an API with auth, versioning, rate limiting, pagination, webhooks, and documentation — it produces something half-finished, inconsistently designed, and impossible to test systematically. The planning phase is not bureaucracy. It is the work that makes the coding fast.
The APIForge Product team is about to build the final version of the APIForge platform API — a production-ready system that combines everything covered in this course. Before the Engineering team writes anything, Product runs a planning session: define the scope, break it into phases, write the API contract, and set the acceptance criteria that will tell everyone when the project is done.
This lesson walks through that planning session step by step. By the end, you will have a complete project plan for the Final Project in Lesson 37 — and a planning framework you can apply to any API project you build after this course.
Step 1 — Define the Scope: What Is In and What Is Out
Every project plan starts with a scope decision, and scope decisions are always about what you are deliberately leaving out. An API with no defined boundaries grows without limit until the deadline forces a scramble. The APIForge team uses a simple rule: if a feature is not in the v1 plan, it is not being built in Lesson 37. It goes on the v2 backlog.
The team uses a two-column exercise: write everything the API could possibly do on the left, then move only the items that are strictly necessary for a working v1 to the right. Anything on the left that is not moved becomes a documented out-of-scope item — not forgotten, just scheduled for later.
Step 2 — Break the Work Into Phases
Phasing is about ordering work so that each phase produces something testable and usable on its own. If Phase 1 is not working, Phase 2 does not start. This constraint forces the team to build foundations before features — authentication before any protected route, database schema before any data endpoint.
The APIForge team identifies four phases. Each phase has a clear deliverable and a definition of done — a specific test that can be run to confirm the phase is complete before moving on.
Step 3 — Write the API Contract
An API contract is a document that specifies every endpoint before any code is written. It names the method, the path, what authentication is required, what the request body looks like, and what the response looks like for both success and error cases. It is the single source of truth that the backend developer, frontend developer, and QA engineer all refer to.
The contract is written in plain language first — not OpenAPI YAML, not code. Once everyone agrees on the design, it gets formalised. Trying to write OpenAPI spec before the design is settled is like writing a legal contract before agreeing on the deal.
# WHAT: APIForge Platform API v1 — Full API Contract
# Format: METHOD /path | Auth | Request body | Success response | Error cases
# This document is agreed before any code is written
# ── AUTH ENDPOINTS ─────────────────────────────────────────────────────────
POST /auth/register
Auth: None
Body: { email, password }
Success: 201 { id, email, createdAt }
Errors: 400 missing fields | 409 email already registered
POST /auth/login
Auth: None
Body: { email, password }
Success: 200 { token, refreshToken, expiresIn }
Errors: 400 missing fields | 401 invalid credentials
POST /auth/refresh
Auth: Bearer refreshToken
Body: None
Success: 200 { token, expiresIn }
Errors: 401 invalid/expired refresh token
GET /auth/me
Auth: Bearer token
Body: None
Success: 200 { id, email, createdAt, role }
Errors: 401 not authenticated
# ── PROJECT ENDPOINTS ──────────────────────────────────────────────────────
GET /api/v1/projects
Auth: Bearer token
Query: limit (default 20) | starting_after (cursor) | status filter
Success: 200 { object:"list", data:[], has_more, url }
Errors: 401 | 400 invalid query params
POST /api/v1/projects
Auth: Bearer token
Headers: Idempotency-Key (optional)
Body: { name, description?, status? }
Success: 201 { id, name, description, status, createdAt, owner }
Errors: 400 missing name | 401 | 409 idempotency key conflict
GET /api/v1/projects/:id
Auth: Bearer token
Success: 200 { id, name, description, status, members[], createdAt }
Errors: 401 | 404 project not found
PATCH /api/v1/projects/:id
Auth: Bearer token
Body: { name?, description?, status? } (partial update)
Success: 200 updated project object
Errors: 400 | 401 | 403 not owner | 404
DELETE /api/v1/projects/:id
Auth: Bearer token
Success: 200 { deleted: true, id }
Errors: 401 | 403 not owner | 404
# ── MEMBER ENDPOINTS ───────────────────────────────────────────────────────
GET /api/v1/projects/:id/members
Auth: Bearer token
Success: 200 { object:"list", data:[], has_more }
Errors: 401 | 404
POST /api/v1/projects/:id/members
Auth: Bearer token
Body: { email, role }
Success: 201 { id, email, role, joinedAt }
Errors: 400 | 401 | 403 not owner | 404 | 409 already a member
DELETE /api/v1/projects/:id/members/:memberId
Auth: Bearer token
Success: 200 { deleted: true }
Errors: 401 | 403 | 404
# ── SEARCH ─────────────────────────────────────────────────────────────────
GET /api/v1/search
Auth: Bearer token
Query: q (required) | status | limit | offset
Success: 200 { query, hits:[], total, processingMs }
Errors: 400 missing q | 401
# ── FILE ATTACHMENTS ───────────────────────────────────────────────────────
POST /api/v1/projects/:id/attachments/upload-url
Auth: Bearer token
Body: { filename, contentType, fileSizeBytes }
Success: 200 { uploadUrl, attachmentId, expiresAt }
Errors: 400 | 401 | 413 file too large | 415 unsupported type
POST /api/v1/attachments/:id/confirm
Auth: Bearer token
Success: 200 { id, filename, url, status:"ready", sizeBytes }
Errors: 401 | 404 | 409 already confirmed
# ── WEBHOOKS ───────────────────────────────────────────────────────────────
POST /api/v1/webhooks
Auth: Bearer token
Body: { url, events:[], secret }
Success: 201 { id, url, events, createdAt }
Errors: 400 | 401
GET /api/v1/webhooks
Auth: Bearer token
Success: 200 { object:"list", data:[] }
Errors: 401
DELETE /api/v1/webhooks/:id
Auth: Bearer token
Success: 200 { deleted: true }
Errors: 401 | 40419 endpoints, fully specified, before any code exists. Notice what the contract captures that a simple list of routes would not: the authentication requirement on each endpoint, the exact field names in every request and response, the specific HTTP status codes for each error case, and which endpoints require idempotency keys.
The PATCH vs PUT decision for project updates is explicit in the contract — PATCH for partial updates, which is the correct choice when clients should not need to send the full object to change one field. Making this decision now prevents a debate in a code review at 11pm the night before deadline.
Step 4 — Design the Database Schema
The API contract drives the database schema. Every resource in the contract needs a table. Every relationship between resources — a project has many members, a member belongs to many projects — needs a join table or a foreign key. Getting the schema wrong requires a migration that breaks existing data, so the APIForge team designs it carefully before Phase 1 starts.
Two decisions get made now that would be painful to change later: all primary keys use prefixed string IDs (like proj_8f3k2m) rather than auto-increment integers, and all timestamps are stored as UTC Unix integers rather than database timestamp types. Both choices make the API responses more consistent and easier to work with across time zones.
# WHAT: APIForge Platform API v1 — PostgreSQL database schema
# Designed from the API contract — every endpoint's data needs are covered
-- Users table (auth)
CREATE TABLE users (
id VARCHAR(20) PRIMARY KEY, -- format: usr_xxxxxxx
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
role VARCHAR(20) NOT NULL DEFAULT 'user',
created_at BIGINT NOT NULL, -- unix timestamp
updated_at BIGINT NOT NULL
);
CREATE INDEX idx_users_email ON users(email);
-- Projects table
CREATE TABLE projects (
id VARCHAR(20) PRIMARY KEY, -- format: proj_xxxxxxx
name VARCHAR(255) NOT NULL,
description TEXT,
status VARCHAR(20) NOT NULL DEFAULT 'active',
owner_id VARCHAR(20) NOT NULL REFERENCES users(id),
created_at BIGINT NOT NULL,
updated_at BIGINT NOT NULL
);
CREATE INDEX idx_projects_owner ON projects(owner_id);
CREATE INDEX idx_projects_status ON projects(status);
CREATE INDEX idx_projects_created ON projects(created_at DESC); -- cursor pagination
-- Project members (many-to-many)
CREATE TABLE project_members (
id VARCHAR(20) PRIMARY KEY,
project_id VARCHAR(20) NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
user_id VARCHAR(20) NOT NULL REFERENCES users(id),
role VARCHAR(20) NOT NULL DEFAULT 'member',
joined_at BIGINT NOT NULL,
UNIQUE (project_id, user_id) -- one membership per user per project
);
-- File attachments
CREATE TABLE attachments (
id VARCHAR(20) PRIMARY KEY, -- format: att_xxxxxxx
project_id VARCHAR(20) NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
s3_key VARCHAR(500) NOT NULL,
filename VARCHAR(255) NOT NULL,
content_type VARCHAR(100) NOT NULL,
size_bytes INTEGER NOT NULL,
status VARCHAR(20) NOT NULL DEFAULT 'pending', -- pending | ready | failed
uploaded_by VARCHAR(20) NOT NULL REFERENCES users(id),
created_at BIGINT NOT NULL
);
-- Webhook registrations
CREATE TABLE webhooks (
id VARCHAR(20) PRIMARY KEY, -- format: wh_xxxxxxx
user_id VARCHAR(20) NOT NULL REFERENCES users(id),
url VARCHAR(500) NOT NULL,
events TEXT[] NOT NULL, -- array: ['project.created', 'member.added']
secret VARCHAR(255) NOT NULL, -- HMAC signing secret, stored encrypted
created_at BIGINT NOT NULL
);
-- Idempotency key store
CREATE TABLE idempotency_keys (
key_hash VARCHAR(64) PRIMARY KEY, -- SHA256 of user_id + key
status_code INTEGER NOT NULL,
response JSONB NOT NULL,
created_at BIGINT NOT NULL,
expires_at BIGINT NOT NULL
);
CREATE INDEX idx_idempotency_expires ON idempotency_keys(expires_at);Six tables covering all 19 endpoints. The indexes are not decoration — each one was placed to support a specific query pattern defined in the API contract. The idx_projects_created index exists specifically because the cursor pagination query on projects is WHERE created_at < cursor ORDER BY created_at DESC — without that index, every paginated request is a full table scan.
The UNIQUE (project_id, user_id) constraint on project_members is a database-level enforcement of the "409 already a member" error case in the API contract. The API checks for duplicates first, but the constraint is the safety net — no race condition can produce duplicate memberships even under concurrent requests.
Step 5 — Write Acceptance Criteria
Acceptance criteria are the specific, testable conditions that must be true for a feature to be considered complete. Not "auth works" — that is a hope. "POST /auth/register returns 201 with a user object, duplicate email returns 409, missing password returns 400" — those are acceptance criteria. You either pass them or you do not.
The APIForge team writes acceptance criteria as a checklist. During the build in Lesson 37, this checklist is the test plan. Each item gets a checkbox — either it passes or it gets flagged for a fix before the lesson is marked complete.
# WHAT: APIForge Platform API v1 — Acceptance Criteria Checklist
# Each item is a specific HTTP request and expected response
# All items must pass before the Final Project is considered complete
# ── PHASE 1: FOUNDATION ────────────────────────────────────────────────────
[ ] GET /health returns 200 { status: "ok", version: "1.0.0", timestamp }
[ ] Server boots and validates all required env vars — crashes with clear error if any missing
[ ] Every response includes X-Request-ID header
[ ] Unhandled route returns 404 with structured error object
[ ] Unhandled server error returns 500 with structured error (no stack trace in response)
# ── PHASE 2: AUTHENTICATION ────────────────────────────────────────────────
[ ] POST /auth/register with valid data returns 201
[ ] POST /auth/register with duplicate email returns 409
[ ] POST /auth/register with missing password returns 400
[ ] POST /auth/login with correct credentials returns token and refreshToken
[ ] POST /auth/login with wrong password returns 401 "Invalid credentials"
[ ] POST /auth/login with unknown email returns 401 "Invalid credentials" (same message)
[ ] GET /api/v1/projects without token returns 401
[ ] GET /api/v1/projects with expired token returns 401 "Token has expired"
[ ] GET /api/v1/projects with valid token returns 200
# ── PHASE 3: CORE RESOURCES ────────────────────────────────────────────────
[ ] POST /api/v1/projects creates project and returns 201 with all fields
[ ] GET /api/v1/projects returns list with has_more and cursor support
[ ] GET /api/v1/projects?starting_after=ID returns correct next page
[ ] GET /api/v1/projects?starting_after=(last item ID) returns has_more: false
[ ] PATCH /api/v1/projects/:id updates only the provided fields
[ ] DELETE /api/v1/projects/:id returns 200 { deleted: true }
[ ] DELETE /api/v1/projects/:id (already deleted) returns 404
[ ] POST /api/v1/projects/:id/members adds member and returns 201
[ ] POST /api/v1/projects/:id/members (duplicate) returns 409
[ ] GET /api/v1/search?q=gateway returns hits with processingMs
[ ] POST /api/v1/projects/:id/attachments/upload-url returns signed URL
[ ] File uploaded to signed URL returns 200 from S3 directly
[ ] POST /api/v1/attachments/:id/confirm updates status to "ready"
# ── PHASE 4: PRODUCTION FEATURES ───────────────────────────────────────────
[ ] POST /api/v1/projects with Idempotency-Key — second request returns same response
[ ] Every response includes X-RateLimit-Remaining header
[ ] 101st request in one minute returns 429 with Retry-After header
[ ] POST /api/v1/webhooks registers endpoint and returns 201
[ ] project.created event fires webhook within 500ms of project creation
[ ] Webhook request includes APIForge-Signature header with valid HMAC
[ ] Tampered webhook payload returns 400 on verification
[ ] All error responses follow { error: { type, message, code, param, request_id } } format
[ ] APIForge-Version header accepted and echoed back in response
[ ] gzip compression active — Content-Encoding: gzip on responses over 1KB40 acceptance criteria — each one a specific HTTP request with a specific expected response. Notice that the auth error criteria deliberately test that a wrong password and an unknown email return the same error message. That is not an accident — it is a security requirement from the Lesson 34 case study, now enforced as a testable criterion.
The split between happy paths (24) and edge cases (9) reflects a common planning mistake: teams write acceptance criteria only for the success case and discover error handling gaps in production. Writing the edge case criteria now means the error paths get built and tested during the planned build window, not as emergency hotfixes later.
Try this: Before starting any API feature, write at least 3 acceptance criteria for it — one happy path, one missing-field error, one auth error. This takes 5 minutes and saves an hour of debugging later.Step 6 — Environment Configuration and Project Structure
The final planning deliverable is the project structure and environment variable specification. These two decisions affect every developer who touches the project. An inconsistent folder structure means developers spend time navigating instead of building. Missing environment variables documented nowhere means every new developer loses an hour figuring out why the server will not start.
# WHAT: APIForge Final Project — folder structure and environment variable spec
# ── FOLDER STRUCTURE ───────────────────────────────────────────────────────
apiforge-api/
├── src/
│ ├── controllers/ # Route handlers — one file per resource
│ │ ├── auth.controller.js
│ │ ├── projects.controller.js
│ │ ├── members.controller.js
│ │ ├── search.controller.js
│ │ ├── attachments.controller.js
│ │ └── webhooks.controller.js
│ ├── middleware/ # Reusable middleware
│ │ ├── authenticate.js # JWT verification
│ │ ├── rateLimiter.js # Redis rate limiting
│ │ ├── idempotency.js # Idempotency key handling
│ │ ├── requestLogger.js # Morgan structured logging
│ │ └── errorHandler.js # Global error handler
│ ├── routes/ # Express router definitions
│ │ ├── auth.routes.js
│ │ ├── projects.routes.js
│ │ ├── search.routes.js
│ │ └── webhooks.routes.js
│ ├── services/ # Business logic — called by controllers
│ │ ├── auth.service.js
│ │ ├── projects.service.js
│ │ └── webhook.service.js # Fires outbound webhooks
│ ├── utils/ # Shared utilities
│ │ ├── generateId.js # Prefixed ID generator
│ │ ├── buildError.js # Structured error builder
│ │ └── crypto.js # HMAC, encrypt, decrypt helpers
│ └── server.js # App entry point
├── db/
│ ├── schema.sql # Full database schema (from Step 4)
│ └── seed.sql # Test data for development
├── tests/
│ └── acceptance/ # One file per acceptance criteria group
├── .env.example # Template — committed to repo
├── .env # Real values — NEVER committed
└── package.json
# ── ENVIRONMENT VARIABLES (.env.example) ──────────────────────────────────
# Server
PORT=3000
NODE_ENV=development
# Database
DATABASE_URL=postgresql://user:password@localhost:5432/apiforge
# Auth
JWT_SECRET=replace-with-64-char-random-string
JWT_EXPIRES_IN=2h
JWT_REFRESH_EXPIRES_IN=7d
# Redis (rate limiting + idempotency)
REDIS_URL=redis://localhost:6379
# AWS S3 (file attachments)
AWS_REGION=ap-south-1
AWS_ACCESS_KEY_ID=replace-with-real-key
AWS_SECRET_ACCESS_KEY=replace-with-real-secret
S3_BUCKET=apiforge-attachments-dev
# Search
MEILISEARCH_URL=http://localhost:7700
MEILISEARCH_KEY=replace-with-master-key
# Webhooks
WEBHOOK_SIGNING_SECRET=replace-with-32-char-random-string
# API Versioning
API_VERSION=2024-11-01The controllers/services split is the most important structural decision here. Controllers handle the HTTP layer — parsing request bodies, setting status codes, sending responses. Services handle business logic — querying the database, calling external APIs, applying rules. Keeping them separate means you can test the business logic without spinning up an HTTP server, and you can change the HTTP framework without rewriting any business logic.
The boot-time environment variable check means the server refuses to start with a clear error message rather than failing silently on the first request that needs a missing variable. One line of startup validation saves every new developer from 30 minutes of debugging a mysterious undefined error in a database connection.
Try this: Create the folder structure above right now, before Lesson 37. Create empty files in each location. Having the scaffolding in place means Lesson 37 is filling in code — not making structural decisions under time pressure.Before and After: Unplanned vs Planned Build
The planning session covered in this lesson takes two to four hours for a project of this size. That investment pays back within the first day of building. Here is what the build experience looks like with and without it.
| Planning Step | Output | Prevents |
|---|---|---|
| 1. Scope definition | In-scope list + documented backlog | Scope creep, mid-project feature debates |
| 2. Phase breakdown | 4 phases, each with definition of done | Building features before foundations are solid |
| 3. API contract | 19 endpoints fully specified | Naming inconsistencies, design debates in code review |
| 4. Database schema | 6 tables, 6 indexes, cascade rules | Mid-build migrations, missing indexes in production |
| 5. Acceptance criteria | 40 testable conditions | Untested edge cases, vague definition of done |
| 6. Project structure | Folder layout + env var spec | Navigation friction, missing config bugs |
Quiz
1. During the APIForge planning session, the team decides that real-time WebSocket events are a valuable feature but not needed for v1. What is the correct way to handle this decision?
2. The APIForge schema includes UNIQUE (project_id, user_id) on the project_members table. The API contract also checks for duplicates in the controller. Why does the schema constraint still matter?
3. The APIForge project structure separates src/controllers from src/services. A junior developer asks why both are needed instead of putting all logic in the controllers. What is the correct explanation?