# Steps.md - Process and Workflow Documentation ## Autonomy Tiers Every step in this document carries a tier telling you who may execute it. The same runbook then serves a person working at 3 AM and an agent working unattended, without pretending an agent should do everything. | Tag | Tier | Meaning | |-----|------|---------| | `[auto]` | agent-auto | An agent may run this unattended. Reversible, or read-only, or fully covered by an automated check. Log the result; do not ask. | | `[approve]` | agent-with-approval | An agent may prepare and propose this, and must get a named human's approval before it takes effect. Anything that changes production state, costs money, or is awkward to undo. | | `[human]` | human-only | A person performs this. Irreversible, judgement-heavy, legally attributable, or requiring a decision the runbook cannot pre-answer. | Rules: - **Untagged means `[human]`.** Absence of a tag is not permission. - **Tiers do not inherit downward.** A `[auto]` section can contain an `[approve]` step, and the step wins. - **An agent that hits an `[approve]` or `[human]` step stops there and reports.** It does not proceed and ask forgiveness, and it does not work around the gate. - **Escalate a tier during an active P0/P1 incident, never relax one.** Pressure is exactly when the gates matter. - **Review the tiers quarterly.** They should loosen as your automated verification gets better. That is the point of measuring them. ### Writing Agent-Executable Procedures A step is executable by an agent when it names an exact command and an exact expected result. A step that says "check that it looks right" is not executable by anything, including a tired human at 3 AM. | Not executable | Executable | |----------------|------------| | Make sure the tests pass | Run `npm test` - expect all green, 0 failures | | Check the build worked | Run `npm run build` - expect exit code 0, no new warnings | | Verify the service is up | `curl -fsS https://api.yourapp.com/health` - expect `{"status": "ok"}` | | Confirm the migration applied | `npm run db:migrate:status` - expect 0 pending | | Wait for DNS | Wait 15-30 minutes, then `dig +short api.yourapp.com` - expect the new address | Two habits carry the whole idea: **number every step** so a run can report exactly where it stopped, and **give every step a verification command** so "done" is a fact rather than an opinion. Steps written this way are better for humans too - the agent-readability is a side effect of writing them precisely. ## Deployment Checklist ### Pre-Deployment (Before Merging to Main) **Developer responsibilities** - complete before requesting deployment approval: - [ ] `[auto]` **Code review approved** - at least 1 approval from a code owner (an agent may confirm the state; it may not be the approver) - [ ] `[auto]` **All CI checks pass** - lint, type-check, unit tests, integration tests, build - [ ] `[human]` **Manual testing completed** - tested the feature locally against acceptance criteria - [ ] `[approve]` **Database migration tested** - ran migration on a copy of staging data, verified rollback works - [ ] `[auto]` **Environment variables documented** - any new env vars added to `.env.example` and deployment docs - [ ] `[approve]` **Feature flag configured** - new features behind a flag for gradual rollout (if applicable) - [ ] `[auto]` **Monitoring updated** - dashboards and alerts cover the new functionality - [ ] `[auto]` **Documentation updated** - API docs, README, runbook, or user-facing docs as needed - [ ] `[auto]` **Changelog entry added** - description of changes for the release notes - [ ] `[approve]` **Rollback plan written** - specific steps to undo this change if something goes wrong ### Deployment Execution **Deployer responsibilities** - the person who triggers the production deployment: 1. `[auto]` **Announce deployment** in #deploys: "Starting deployment of [PR #/feature name]. ETA: 15 minutes." 2. `[approve]` **Merge PR to main** - squash merge, verify the commit lands on main 3. `[auto]` **Monitor CI/CD pipeline** - watch GitHub Actions for build and deploy stages 4. `[auto]` **Verify staging deployment** (if applicable): - Run smoke tests against staging: `npm run test:smoke -- --env staging` - expect all green - Spot-check the feature manually in the staging environment - Verify no new errors in Datadog/Sentry for staging 5. `[human]` **Promote to production** (manual gate or auto-deploy depending on config) 6. `[auto]` **Monitor production** for 15 minutes after deployment: - Watch error rates in Datadog (should stay below 0.1%) - Check latency percentiles (p50, p95 should not increase by more than 10%) - Verify the deployed feature works with a manual check - Monitor the #incidents channel for user reports 7. `[auto]` **Announce completion** in #deploys: "Deployment complete. [PR #/feature name] is live." An agent may run steps 1, 3, 4, 6, and 7 end to end and stop at 5 with a summary. That covers most of the wall-clock time in a deployment and leaves the irreversible moment with a person. ### Post-Deployment Verification - [ ] `[auto]` Feature accessible in production - [ ] `[auto]` Error rates stable (compare to pre-deployment baseline) - [ ] `[auto]` Performance metrics within acceptable range - [ ] `[approve]` Feature flag toggled for initial rollout group (if applicable) - [ ] `[auto]` No user-facing error reports in first 30 minutes ### Rollback Procedure `[approve]` - an agent may prepare the revert commit and the rollback command, and must get a human to trigger it. Exception: if your team has pre-authorized automatic rollback on a specific alert, that alert-driven path is `[auto]` and must be written down here as such. If something goes wrong after deployment: ```bash # Step 1: Revert the commit on main git revert [commit-hash] --no-edit git push origin main # Step 2: CI/CD auto-deploys the revert (monitor pipeline) # Step 3: If database migration was involved: npm run db:migrate:rollback # Verify application works with the previous schema # Step 4: Announce in #deploys and #incidents: # "Rolled back [feature]. Investigating. No user action needed." # Step 5: Create a post-mortem ticket if the issue affected users ``` **Decision framework for rollback**: Roll back immediately if any of these are true: - Error rate > 1% (normal baseline is < 0.1%) - P95 latency > 2x normal - Any user-facing data integrity issue - Security vulnerability discovered - Payment processing affected ## Incident Response ### Severity Levels | Level | Description | Response Time | Examples | |-------|-------------|---------------|---------| | P0 - Critical | Service down, data loss, security breach | 15 minutes | Full outage, payment processing broken, data leak | | P1 - High | Major feature broken, significant user impact | 30 minutes | Dashboard not loading, authentication failing, search broken | | P2 - Medium | Feature degraded, workaround available | 4 hours | Slow queries, intermittent errors, export failing | | P3 - Low | Minor issue, minimal user impact | Next business day | UI glitch, non-critical error in logs, cosmetic bug | ### Incident Response Steps #### Step 1: Detect and Alert (0-5 minutes) `[auto]` - Automated alert fires (Datadog, Sentry, PagerDuty) OR user report received - On-call engineer acknowledges the alert - Create incident channel: #incident-YYYY-MM-DD-brief-description #### Step 2: Assess Severity (5-10 minutes) `[human]` ``` Questions to answer: 1. How many users are affected? (all / subset / single) 2. Is data being lost or corrupted? (yes / no / unknown) 3. Is there a workaround? (yes / no) 4. Is this a security issue? (yes / no) 5. What changed recently? (deployment / config change / external) ``` Assign severity level based on answers. Escalate P0/P1 to engineering lead. An agent may draft the answers from telemetry and recent commits, but a human owns the severity call - it sets the response obligations for everyone else. #### Step 3: Investigate (10-30 minutes) `[auto]` Read-only. An agent may run this whole step unattended and should, because it is the slow part. It gathers evidence; it does not change anything. An agent that edits code or config while investigating has contaminated the reproduction and must say so. ```bash # Check recent deployments git log --oneline -10 # Check application logs # Datadog: Filter by service, time range, error level # Check infrastructure # AWS Console: EC2/ECS health, RDS metrics, Redis metrics # Check external dependencies # Status pages: Stripe, Auth0, AWS, Cloudflare ``` #### Step 4: Mitigate (As Soon as Root Cause Identified) `[human]` - **If caused by recent deployment**: Roll back immediately (see rollback procedure above) - **If caused by external service**: Enable fallback mode, notify affected users - **If caused by data issue**: Stop the bleeding first (disable writes if needed), then fix - **If cause is unknown**: Add more logging, monitor closely, prepare to escalate Every branch here changes production state under time pressure with incomplete information. That combination is the worst possible case for unattended action, so mitigation stays human-only regardless of how good your automation is elsewhere. An agent may stage the exact commands and hold them ready. #### Step 5: Resolve and Verify (Once Fix is Applied) `[auto]` - Verify error rates return to normal - Confirm affected users can use the feature - Update incident channel with resolution - Stand down from incident response - `[human]`, the on-call engineer makes this call #### Step 6: Post-Mortem (Within 48 Hours) `[approve]` An agent can assemble a strong first draft: the timeline from alert and deploy logs, the commands that were run, the diff that caused it. It cannot write the "What Could Be Improved" section honestly, and a post-mortem nobody wrote is a post-mortem nobody read. A named human edits and publishes it. ```markdown # Post-Mortem: [Incident Title] Date: YYYY-MM-DD Duration: [Start time] - [End time] ([X] minutes) Severity: [P0-P3] Authored by: [On-call engineer] ## Summary [1-2 sentence description of what happened] ## Impact - Users affected: [number or percentage] - Duration: [X minutes/hours] - Revenue impact: [if applicable] ## Timeline - [HH:MM] Alert fired / user report received - [HH:MM] On-call acknowledged - [HH:MM] Root cause identified - [HH:MM] Fix deployed - [HH:MM] Verified resolution ## Root Cause [Detailed technical explanation] ## What Went Well - [Thing that helped] ## What Could Be Improved - [Thing that slowed us down] ## Action Items - [ ] [Preventive action] - Owner: [Name] - Due: [Date] - [ ] [Monitoring improvement] - Owner: [Name] - Due: [Date] - [ ] [Process improvement] - Owner: [Name] - Due: [Date] ``` ## Feature Development Workflow ### From Idea to Production ``` 1. Product Brief (PM) -> Word.md / product requirements 2. Technical Design (Engineer) -> Spider.md / SPEC + PSEUDOCODE 3. Estimation (Team) -> Story points in sprint planning 4. Implementation (Engineer) -> Feature branch + PR 5. Code Review (Team) -> PR review + approval 6. QA Testing (QA/Engineer) -> Staging deployment + test plan 7. Deployment (Engineer) -> Production deployment (this doc) 8. Monitoring (Team) -> 48-hour watch period 9. Retrospective (Team) -> Lessons learned ``` ### Sprint Ceremonies | Ceremony | When | Duration | Purpose | |----------|------|----------|---------| | Sprint Planning | Monday 10 AM | 1 hour | Select and estimate work for the sprint | | Daily Standup | Daily 10 AM | 15 min | Sync on progress and blockers | | Mid-Sprint Check | Wednesday 2 PM | 30 min | Course-correct if behind schedule | | Sprint Review | Friday 3 PM (every 2 weeks) | 30 min | Demo completed work to stakeholders | | Retrospective | Friday 3:30 PM (every 2 weeks) | 30 min | Process improvement | ## Release Process ### Semantic Versioning ``` MAJOR.MINOR.PATCH MAJOR: Breaking API changes, major UI redesign, database incompatibility MINOR: New features, non-breaking API additions PATCH: Bug fixes, performance improvements, documentation updates Examples: 2.0.0 - Redesigned dashboard, new GraphQL API (breaking) 1.5.0 - Added team billing feature 1.4.3 - Fixed timezone bug in event timestamps ``` ### Release Steps 1. **Create release branch**: `git checkout -b release/v1.5.0` 2. **Update version**: Bump version in `package.json` 3. **Update changelog**: Add release notes to `CHANGELOG.md` 4. **Final testing**: Run full test suite on release branch 5. **Create PR**: Release branch -> main 6. **Get approval**: Release manager + 1 engineer approve 7. **Merge and tag**: Merge PR, create git tag `v1.5.0` 8. **Deploy**: Tag triggers production deployment via CI/CD 9. **Announce**: Post release notes in #releases and email stakeholders 10. **Monitor**: Watch production metrics for 24 hours ### Hotfix Process For urgent production fixes that cannot wait for the next release: ```bash # 1. Branch from main (not from develop/staging) git checkout main git pull origin main git checkout -b hotfix/v1.4.4-fix-payment-webhook # 2. Make the minimal fix (smallest possible change) # 3. Add a test that covers the bug # 4. Create PR with [HOTFIX] prefix in the title # 5. Fast-track review (1 approval, can be from any engineer) # 6. Merge to main AND cherry-pick to develop/staging # 7. Deploy immediately # 8. Write post-mortem if it was a P0/P1 incident ``` ## Code Review Steps ### Author Checklist (Before Requesting Review) - [ ] PR title follows commit convention: `feat/fix/refactor: description` - [ ] PR description explains WHAT changed and WHY - [ ] Self-reviewed the diff - no debug code, no commented-out blocks - [ ] Tests added or updated for the change - [ ] No unrelated changes included (keep PRs focused) - [ ] Screenshots included for UI changes - [ ] Breaking changes documented in PR description ### Reviewer Workflow 1. **Read the PR description first** - understand the goal before reading code 2. **Check the test plan** - are the right scenarios covered? 3. **Review the diff file by file** - start with the most important files 4. **Leave comments with prefixes** - nit/suggestion/concern/blocker/praise 5. **Approve or request changes** - do not leave reviews in "comment only" state 6. **Respond within 4 hours** - unblocking teammates is a priority ### Review Turnaround Expectations | PR Size | Expected Review Time | |---------|---------------------| | Small (<50 lines) | Same day | | Medium (50-200 lines) | Within 4 hours | | Large (200-500 lines) | Within 1 business day | | XL (500+ lines) | Schedule a walkthrough first | ## On-Call Procedures ### On-Call Rotation - **Rotation**: Weekly, Monday 9 AM to Monday 9 AM - **Schedule**: [Link to PagerDuty/OpsGenie schedule] - **Handoff**: Outgoing on-call briefs incoming on any ongoing issues - **Escalation**: If on-call cannot resolve within 30 minutes, escalate to engineering lead ### On-Call Responsibilities 1. **Respond to alerts** within the SLA for each severity level 2. **Triage user reports** in #support and #incidents channels 3. **Keep stakeholders updated** during active incidents (every 30 minutes for P0/P1) 4. **Write post-mortems** for any P0/P1 incidents during your shift 5. **Hand off cleanly** - document any ongoing issues for the next on-call ### On-Call Toolkit ```bash # Quick health check curl https://api.yourapp.com/health # Should return: {"status": "ok", "version": "1.5.0"} # Check application logs (last 30 minutes) # Datadog: [link to saved search] # Check database health # RDS Console: [link to dashboard] # Check deployment status # ArgoCD: [link to dashboard] # Emergency contact list # Engineering Lead: [phone] # VP Engineering: [phone] (P0 only) # AWS Support: [case portal link] ``` ## Database Migration Procedures ### Standard Migration ```bash # 1. [auto] Create the migration pnpm db:migrate:create --name add_team_billing_tables # 2. [auto] Write the migration SQL (up and down) # Edit the generated file in prisma/migrations/ # 3. [auto] Test locally - a throwaway local database, never a shared one pnpm db:migrate # Apply pnpm db:migrate:rollback # Rollback pnpm db:migrate # Re-apply to verify both directions work pnpm db:migrate:status # Expect 0 pending # 4. [auto] Test on staging data copy - the copy, not staging itself pg_dump staging_db > staging_copy.sql psql test_db < staging_copy.sql pnpm db:migrate # Apply migration to staging data copy # Verify data integrity # 5. [human] Include in PR - a named human reviews the SQL before merge. # Migration runs automatically during deployment, which means merge # approval IS the approval to alter production schema. Treat it that way. ``` An agent may own steps 1 through 4 completely. What it must never do is point any migration command at a shared or production database - that is the single most damaging unattended action available to it, and no amount of local test coverage authorizes it. ### High-Risk Migration (Large Tables, Schema Changes) For migrations that affect tables with 1M+ rows or change column types. This entire procedure is `[human]`. An agent may prepare every artifact and measure everything, but a person runs the window. 1. `[approve]` **Schedule maintenance window** - notify users 48 hours in advance 2. `[human]` **Take a full backup** before running the migration 3. `[auto]` **Run migration on a staging copy** first and measure execution time 4. `[approve]` **Use online schema change tools** if migration takes > 5 minutes 5. `[human]` **Monitor during execution** - watch lock waits, replication lag, disk I/O 6. `[auto]` **Verify application compatibility** - the app must work with both old and new schema during rollout 7. `[approve]` **Keep rollback script ready** - tested and verified before the migration window ## Process Improvement ### Retrospective Template ```markdown # Sprint [X] Retrospective Date: YYYY-MM-DD Facilitator: [Name] Attendees: [Names] ## What Went Well - [Item 1] - [Item 2] ## What Could Be Improved - [Item 1] - [Item 2] ## Action Items (max 3 per retro) - [ ] [Specific, actionable item] - Owner: [Name] - Due: [Date] - [ ] [Specific, actionable item] - Owner: [Name] - Due: [Date] ## Follow-up on Previous Action Items - [x] [Completed item from last retro] - [ ] [Still in progress - update on status] ``` ### Continuous Improvement Metrics Track these monthly to measure process health: | Metric | Target | How to Measure | |--------|--------|---------------| | Deployment frequency | 5+/week | Count merges to main | | Lead time for changes | < 3 days | PR open to production | | Change failure rate | < 5% | Rollbacks / total deployments | | Mean time to recovery | < 30 min | Incident start to resolution | | PR review turnaround | < 4 hours | Time from review request to first review | | Steps at `[auto]` | Trending up | Count of `[auto]` steps in this document | | Agent runs stopped at a gate | Trending down | Runs that halted at `[approve]`/`[human]` vs total runs | The last two are the ones to watch together. A step moves from `[approve]` to `[auto]` only when its verification command got good enough to trust unattended - so the count going up is evidence your gates improved, not evidence you got braver. If it goes up while the change failure rate also goes up, you loosened too early. Move the step back.