diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..fb78262 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,67 @@ +# Contributing to OpenClaw Mission Control + +Thanks for your interest in improving Mission Control. + +This repo welcomes contributions in three broad categories: + +- **Issues**: bug reports, feature requests, and design discussions +- **Documentation**: improvements to clarity, correctness, onboarding, and runbooks +- **Code**: fixes, features, tests, and refactors + +## Where to start + +- Docs landing page: [Docs landing](./docs/README.md) +- Development workflow: [Development workflow](./docs/03-development.md) +- Testing guide: [Testing guide](./docs/testing/README.md) + +## Filing issues + +When opening an issue, please include: + +- What you expected vs what happened +- Steps to reproduce (commands, env vars, links) +- Logs and screenshots where helpful +- Your environment (OS, Docker version, Node/Python versions) + +## Pull requests + +### Expectations + +- Keep PRs **small and focused** when possible. +- Include a clear description of the change and why it’s needed. +- Add/adjust tests when behavior changes. +- Update docs when contributor-facing or operator-facing behavior changes. + +### Local checks + +From repo root, the closest “CI parity” command is: + +```bash +make check +``` + +If you’re iterating on a specific area, the Makefile also provides targeted commands (lint, typecheck, unit tests, etc.). See `make help`. + +## Docs contribution guidelines + +- The numbered pages under `docs/` are **entrypoints**. Prefer linking to deeper pages instead of duplicating large blocks of content. +- Use concise language and concrete examples. +- When documenting operational behavior, call out risk areas (secrets, data loss, migrations). + +## Security and vulnerability reporting + +If you believe you’ve found a security vulnerability: + +- **Do not** open a public issue. +- Prefer GitHub’s private reporting flow: + - https://github.com/abhi1693/openclaw-mission-control/security/advisories/new + +If that’s not available in your environment, contact the maintainers privately. + +## Code of conduct + +If this repository adopts a Code of Conduct, we will link it here. + +## License + +By contributing, you agree that your contributions will be licensed under the MIT License. See [`LICENSE`](./LICENSE). diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..adc0ffe --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 OpenClaw Mission Control + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 237fcbd..615cc27 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![CI](https://github.com/abhi1693/openclaw-mission-control/actions/workflows/ci.yml/badge.svg)](https://github.com/abhi1693/openclaw-mission-control/actions/workflows/ci.yml) -Web UI + API for operating OpenClaw: managing boards, tasks, agents, approvals, and gateway connections. +Mission Control is the **web UI and HTTP API** for operating OpenClaw. It’s designed for teams that want a clear control plane for managing **boards**, **tasks**, **agents**, **approvals**, and (optionally) **gateway connections**. ## Active development @@ -11,53 +11,65 @@ OpenClaw Mission Control is under active development. Expect breaking changes an - Use at your own risk for production workloads. - We welcome **bug reports**, **feature requests**, and **PRs** — see GitHub Issues: https://github.com/abhi1693/openclaw-mission-control/issues -- **Frontend:** Next.js app (default http://localhost:3000) -- **Backend:** FastAPI service (default http://localhost:8000) -- **Data:** Postgres -- **Gateway integration:** see [`docs/openclaw_gateway_ws.md`](./docs/openclaw_gateway_ws.md) -- **Gateway base config (local):** see [`docs/openclaw_gateway_base_config.md`](./docs/openclaw_gateway_base_config.md) +## Architecture (high level) -> Note on auth (Clerk) -> -> Clerk is **optional** for local/self-host. The frontend enables Clerk **only** when -> `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` is set. If you don’t want to configure Clerk, -> make sure that variable is **unset/blank**. +Mission Control is a small, service-oriented stack: -## Quick start (self-host with Docker Compose) +- **Frontend:** Next.js (default http://localhost:3000) +- **Backend:** FastAPI (default http://localhost:8000) +- **Database:** Postgres +- **Gateway integration (optional):** WebSocket protocol documented in [Gateway WebSocket protocol](./docs/openclaw_gateway_ws.md) -### Prerequisites +## Documentation -- Docker + Docker Compose v2 (`docker compose`) +Start with the docs landing page: +- [Docs landing](./docs/README.md) -### Run +Operational deep dives: +- Deployment: [Deployment guide](./docs/deployment/README.md) +- Production notes: [Production notes](./docs/production/README.md) +- Troubleshooting: [Troubleshooting](./docs/troubleshooting/README.md) + +## Authentication (Clerk) + +**Clerk is currently required**. + +You must configure Clerk keys for: +- the frontend (`NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY`, `CLERK_SECRET_KEY`) +- the backend (`CLERK_SECRET_KEY`) + +See: [Deployment guide](./docs/deployment/README.md#clerk-auth-notes). + +## Deployment modes + +### 1) Self-host (Docker Compose) + +**Prerequisites:** Docker + Docker Compose v2 (`docker compose`) ```bash cp .env.example .env -# REQUIRED: ensure the browser can reach the backend API. +# REQUIRED: the browser must be able to reach the backend. # NEXT_PUBLIC_API_URL must be reachable from the *browser* (host), not an internal Docker network name. -# If you change ports/hosts, update NEXT_PUBLIC_API_URL in .env accordingly. -# (Missing/blank NEXT_PUBLIC_API_URL will break frontend API calls like Activity feed.) +# Missing/blank NEXT_PUBLIC_API_URL will break frontend API calls (e.g. Activity feed). -# IMPORTANT: if you are not configuring Clerk, disable it by ensuring -# NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY is NOT set. -# (The default `frontend/.env.example` contains placeholders that you should delete/blank.) +# REQUIRED: Clerk config. +# Provide real Clerk values via frontend/.env (recommended) and backend/.env. docker compose -f compose.yml --env-file .env up -d --build ``` Open: - - Frontend: http://localhost:3000 - Backend health: http://localhost:8000/healthz -### Stop +Stop: ```bash docker compose -f compose.yml --env-file .env down ``` -### Common Compose commands +Useful ops: ```bash # Tail logs @@ -70,133 +82,25 @@ docker compose -f compose.yml --env-file .env up -d --build backend docker compose -f compose.yml --env-file .env down -v ``` -## Quick start (local development) +### 2) Contributor local dev loop (DB in Docker, apps on host) This is the fastest workflow for contributors: run Postgres via Docker, and run the backend + frontend in dev mode. -### Prerequisites +See: [Development workflow](./docs/03-development.md) -- Docker + Docker Compose v2 -- Python **3.12+** + [`uv`](https://github.com/astral-sh/uv) -- Node.js (recommend 18+) + npm +## Testing and CI parity -### 1) Start Postgres - -```bash -cp .env.example .env - -docker compose -f compose.yml --env-file .env up -d db -``` - -### 2) Backend (FastAPI) - -```bash -cd backend - -cp .env.example .env - -# deps -uv sync --extra dev - -# run API on :8000 -uv run uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 -``` - -Notes: - -- If you run the DB container, the backend should use the default in `backend/.env` (`localhost:5432`). -- Database migrations: - - ```bash - cd backend - uv run alembic upgrade head - ``` - -### 3) Frontend (Next.js) - -```bash -cd frontend - -# Configure API URL (REQUIRED) and optionally disable Clerk for local dev by removing/blanking Clerk env vars -cp .env.example .env.local - -# If you run the backend locally on :8000, this should be: -# NEXT_PUBLIC_API_URL=http://localhost:8000 - -npm install -npm run dev -``` - -Open http://localhost:3000. - -### Cypress E2E (local) - -When running Cypress (`cd frontend && npm run e2e`), make sure `NEXT_PUBLIC_API_URL` is set (either in `frontend/.env.local` or your shell env). In CI we run the frontend on `http://localhost:3000`, so `NEXT_PUBLIC_API_URL` is set to `http://localhost:3000` for the E2E job. - -## Key concepts / high-level architecture - -- **Mission Control backend** exposes a REST API at `/api/v1/*` and also hosts health endpoints (`/healthz`, `/readyz`). -- **Mission Control frontend** calls the backend via `NEXT_PUBLIC_API_URL`. -- **Postgres** stores boards/tasks/agents/etc. -- **OpenClaw Gateway** connectivity is over WebSockets; protocol details live in [`docs/openclaw_gateway_ws.md`](./docs/openclaw_gateway_ws.md). - -## Common commands - -- Testing notes: [`docs/testing/README.md`](./docs/testing/README.md) - -### CI test suites (canonical commands) - -CI (`.github/workflows/ci.yml`) runs these suites on PRs: - -- **Backend**: `make backend-coverage` (includes `pytest` + coverage artifacts) -- **Frontend**: `make frontend-test` (Vitest + coverage) -- **E2E (Cypress)**: runs only when a `cypress.config.*` is present in the repo (job `e2e`). - -### Coverage policy - -CI enforces a **scoped 100% coverage gate** (statements + branches) for a small set of unit-testable modules. -See `docs/coverage-policy.md`. +- Testing guide: [Testing guide](./docs/testing/README.md) +- Coverage policy: [Coverage policy](./docs/coverage-policy.md) From repo root: ```bash make help make setup -make lint -make typecheck -make test make check ``` -## Troubleshooting +## License -More: [`docs/troubleshooting/README.md`](./docs/troubleshooting/README.md) - -### Frontend keeps redirecting / Clerk errors - -You likely have `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` set (even to a placeholder). To run without Clerk: - -- Remove the `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` line from `frontend/.env.local`, **or** set it to an empty value. - -### Backend can’t connect to Postgres - -- Confirm containers are up: - - ```bash - docker compose -f compose.yml --env-file .env ps - ``` - -- If you’re running backend locally (not in compose), make sure `backend/.env` points to `localhost`: - - `DATABASE_URL=postgresql+psycopg://postgres:postgres@localhost:5432/mission_control` - -### Port already in use - -Adjust ports in `.env` (copied from `.env.example`): - -- `FRONTEND_PORT` -- `BACKEND_PORT` -- `POSTGRES_PORT` - -## Star History - -[![Star History Chart](https://api.star-history.com/svg?repos=abhi1693/openclaw-mission-control&type=date&legend=top-left)](https://www.star-history.com/#abhi1693/openclaw-mission-control&type=date&legend=top-left) +This project is licensed under the MIT License. See [`LICENSE`](./LICENSE). diff --git a/docs/01-overview.md b/docs/01-overview.md new file mode 100644 index 0000000..b776fef --- /dev/null +++ b/docs/01-overview.md @@ -0,0 +1,45 @@ +# Overview + +Mission Control is the **web UI + HTTP API** for operating OpenClaw. + +It’s where you manage **boards**, **tasks**, **agents**, **approvals**, and (optionally) **gateway connections**. + +## Problem statement + +OpenClaw can execute work (tools/skills) and converse across channels, but real operations need a place to: + +- **Coordinate** work across people + agents (what’s next, what’s blocked, who owns what) +- **Track evidence** of what happened (commands run, links, logs, artifacts) +- **Control risk** (approvals, guardrails, isolation) +- **Operate reliably** (deployment, configuration, troubleshooting) + +Mission Control provides that control plane. + +## Who uses it + +- **Maintainers / operators**: keep Mission Control + gateways healthy, deploy upgrades, respond to incidents. +- **Contributors**: develop backend/frontend changes, run tests, ship docs. +- **Automation authors**: define agent identities, skills, and task workflows. + +## Key concepts (glossary-lite) + +- **Board**: a workspace containing tasks, memory, and agents. +- **Task**: a unit of work on a board (status + comments/evidence). +- **Agent**: an automated worker that can execute tasks and post evidence. +- **Approval**: a structured “allow/deny” checkpoint for risky actions. +- **Gateway**: the OpenClaw runtime host that executes tools/skills and runs heartbeats/cron. +- **Heartbeat**: periodic agent check-in loop for incremental work. +- **Cron job**: scheduled execution (recurring or one-shot), often isolated from conversational context. + +## Out of scope + +- Not a general-purpose project management suite (we optimize for AI-assisted operations, not every PM feature). +- Not a full observability platform (we integrate with logs/metrics rather than replacing them). +- Not a secrets manager (we reference secret sources; don’t store secrets in docs/tasks/comments). + +## Where to go next + +- Want it running? → [Quickstart](02-quickstart.md) +- Want to contribute? → [Development](03-development.md) +- Want to understand internals? → [Architecture](05-architecture.md) +- Operating it? → [Ops / runbooks](09-ops-runbooks.md) diff --git a/docs/02-quickstart.md b/docs/02-quickstart.md new file mode 100644 index 0000000..f49407d --- /dev/null +++ b/docs/02-quickstart.md @@ -0,0 +1,14 @@ +# Quickstart (self-host with Docker Compose) + +This page is a pointer to the canonical quickstart in the repo root README. + +- Canonical quickstart: [`README.md#quick-start-self-host-with-docker-compose`](../README.md#quick-start-self-host-with-docker-compose) + +## Verify it works +After `docker compose up`: +- Backend health: `http://localhost:8000/healthz` returns `{ "ok": true }` +- Frontend: `http://localhost:3000` + +## Common gotchas +- `NEXT_PUBLIC_API_URL` must be reachable from your browser (host), not just from inside Docker. +- Clerk auth is required; ensure Clerk keys are configured (see [Deployment guide](deployment/README.md)). diff --git a/docs/03-development.md b/docs/03-development.md new file mode 100644 index 0000000..4a5f785 --- /dev/null +++ b/docs/03-development.md @@ -0,0 +1,182 @@ +# Development + +## Deep dives + +- [Testing guide](testing/README.md) +- [Troubleshooting deep dive](troubleshooting/README.md) + +How we develop Mission Control locally, with a workflow that stays close to CI. + + +## Prerequisites + +- Docker + Docker Compose v2 (`docker compose`) +- Python **3.12+** + `uv` +- Node.js + npm + - CI pins **Node 20** via GitHub Actions (`actions/setup-node@v4` with `node-version: "20"`). + +## Repo structure (where to run commands) + +- Repo root: `Makefile` contains canonical targets. +- Backend code: `backend/` (FastAPI) +- Frontend code: `frontend/` (Next.js) + +## “One command” setup + +From repo root: + +```bash +make setup +``` + +What it does: +- Syncs backend deps with `uv`. +- Syncs frontend deps with `npm` via the node wrapper. + +## Canonical checks (CI parity) + +### Run everything locally (closest to CI) + +From repo root: + +```bash +make check +``` + +CI runs two jobs: +- `check` (lint/typecheck/tests/coverage/build) +- `e2e` (Cypress) + +## Backend workflow + +### Install/sync deps + +```bash +cd backend +uv sync --extra dev +``` + +### Run the API (dev) + +```bash +cd backend +cp .env.example .env +uv run uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 +``` + +### Backend checks + +From repo root: + +```bash +make backend-lint # flake8 +make backend-typecheck # mypy --strict +make backend-test # pytest +make backend-coverage # pytest + scoped coverage gate +``` + +### DB migrations + +From repo root: + +```bash +make backend-migrate +``` + +## Frontend workflow + +### Install deps + +```bash +cd frontend +npm install +``` + +(or from repo root: `make frontend-sync`) + +### Run the UI (dev) + +```bash +cd frontend +cp .env.example .env.local +# Ensure NEXT_PUBLIC_API_URL is correct for the browser: +# NEXT_PUBLIC_API_URL=http://localhost:8000 +npm run dev +``` + +### Frontend checks + +From repo root: + +```bash +make frontend-lint # eslint +make frontend-typecheck # tsc +make frontend-test # vitest +make frontend-build # next build +``` + +## Local dev loops + +### Loop A (recommended): DB via Compose, backend + frontend in dev mode + +1) Start Postgres only: + +```bash +cp .env.example .env +docker compose -f compose.yml --env-file .env up -d db +``` + +2) Backend (local): + +```bash +cd backend +cp .env.example .env +uv sync --extra dev +uv run uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 +``` + +3) Frontend (local): + +```bash +cd frontend +cp .env.example .env.local +npm install +npm run dev +``` + +### Loop B: all-in-one Docker Compose + +```bash +cp .env.example .env + +docker compose -f compose.yml --env-file .env up -d --build +``` + +Useful ops: + +```bash +docker compose -f compose.yml --env-file .env logs -f --tail=200 +docker compose -f compose.yml --env-file .env up -d --build backend +# destructive reset (drops Postgres volume): +docker compose -f compose.yml --env-file .env down -v +``` + +## Cypress E2E workflow (high level) + +See the deep dive: [docs/testing/README.md](testing/README.md). + +Notes: +- E2E uses Clerk (official `@clerk/testing` integration); CI injects Clerk env vars. + +## Tooling notes + +### Node wrapper (`scripts/with_node.sh`) + +Many Make targets run frontend commands via `bash scripts/with_node.sh`. +It checks `node`/`npm`/`npx` and can use `nvm` if present. + +## Quick troubleshooting + +- UI loads but API calls fail / activity feed blank: + - confirm `NEXT_PUBLIC_API_URL` is set and browser-reachable. + - see [Troubleshooting](troubleshooting/README.md). diff --git a/docs/04-repo-tour.md b/docs/04-repo-tour.md new file mode 100644 index 0000000..6e4aeb8 --- /dev/null +++ b/docs/04-repo-tour.md @@ -0,0 +1,35 @@ +# Repo tour + +High-level map of the codebase so you can quickly find where to change things. + +## Top-level +- `backend/` — FastAPI backend (API server) +- `frontend/` — Next.js frontend (web UI) +- `docs/` — documentation +- `compose.yml` — local/self-host stack (db + backend + frontend) +- `scripts/` — helper scripts + +## Backend: where to look +- App entrypoint + router wiring: `backend/app/main.py` +- Routers: `backend/app/api/*` +- Settings/config: `backend/app/core/config.py` +- Auth (Clerk + agent token): `backend/app/core/auth.py`, `backend/app/core/agent_auth.py` +- Models: `backend/app/models/*` +- Services/domain logic: `backend/app/services/*` + +## Frontend: where to look +- Routes (App Router): `frontend/src/app/*` +- Components: `frontend/src/components/*` +- API utilities: `frontend/src/lib/*` and `frontend/src/api/*` +- Auth (Clerk gating/wrappers): `frontend/src/auth/*` + +## Where to change X + +| You want to… | Start here | +|---|---| +| Add/modify an API endpoint | `backend/app/api/*` + `backend/app/main.py` | +| Change auth behavior | `backend/app/core/auth.py` + `frontend/src/auth/*` | +| Change a UI page | `frontend/src/app/*` | +| Update Compose topology | `compose.yml` | + +Next: see [Architecture](05-architecture.md) for system-level flows. diff --git a/docs/05-architecture.md b/docs/05-architecture.md new file mode 100644 index 0000000..8ec794c --- /dev/null +++ b/docs/05-architecture.md @@ -0,0 +1,93 @@ +# Architecture + +## Deep dives + +- [Architecture deep dive](architecture/README.md) +- [Gateway protocol](openclaw_gateway_ws.md) + +Mission Control is the **web UI + HTTP API** for operating OpenClaw. It’s where you manage boards, tasks, agents, approvals, and (optionally) gateway connections. + +> Auth note: **Clerk is required for production**. The codebase includes gating so CI/local can run without “real” keys, but real deployments should configure Clerk. + +## Components + +- **Frontend**: Next.js app used by humans + - Location: `frontend/` + - Routes/pages: `frontend/src/app/*` (Next.js App Router) + - API client: generated + custom fetch (see `frontend/src/api/*`, `frontend/src/lib/api-base.ts`) +- **Backend**: FastAPI service exposing REST endpoints + - Location: `backend/` + - Entrypoint: `backend/app/main.py` + - API prefix: `/api/v1/*` +- **Database**: Postgres (see `compose.yml`) +- **Gateway integration (optional)**: backend may call into OpenClaw Gateways over WebSockets + - Client/protocol list: `backend/app/services/openclaw/gateway_rpc.py` + +## Diagram (conceptual) + +```mermaid +flowchart LR + U[User / Browser] -->|HTTP| FE[Next.js Frontend :3000] + FE -->|HTTP /api/v1/*| BE[FastAPI Backend :8000] + + BE -->|SQL| PG[(Postgres :5432)] + + BE -->|WebSocket (optional)| GW[OpenClaw Gateway] + GW --> OC[OpenClaw runtime] +``` + +## How requests flow + +### 1) A human uses the UI + +1. Browser loads the Next.js frontend (`frontend/`). +2. Frontend calls backend endpoints using `NEXT_PUBLIC_API_URL`. +3. Backend routes under `/api/v1/*` (`backend/app/main.py`) and reads/writes Postgres. + +Common UI-driven data shapes: +- “boards/tasks” views → board/task CRUD + streams. +- “activity feed” → activity/events endpoints. + +### 2) Authentication (Clerk) + +- **Frontend**: Clerk is enabled only when a publishable key is present/valid. + - Gating/wrappers: `frontend/src/auth/clerkKey.ts`, `frontend/src/auth/clerk.tsx`. +- **Frontend → backend**: API calls attach `Authorization: Bearer ` when available. + - Token injection: `frontend/src/api/mutator.ts` (uses `window.Clerk.session.getToken()`). +- **Backend**: validates inbound auth and resolves a user context. + - Implementation: `backend/app/core/auth.py` (uses `clerk_backend_api` SDK with `CLERK_SECRET_KEY`). + +### 3) Agent automation surface (`/api/v1/agent/*`) + +Agents can call a dedicated API surface: + +- Router: `backend/app/api/agent.py` (prefix `/agent` → mounted under `/api/v1/agent/*`). +- Authentication: `X-Agent-Token` header (or agent-only Authorization bearer parsing). + - Implementation: `backend/app/core/agent_auth.py`. + +Typical agent flows: +- Heartbeat/presence updates +- Task comment posting (evidence) +- Board memory updates +- Lead coordination actions (if board-lead agent) + +### 4) Streaming/feeds (server-sent events) + +Some endpoints support streaming via SSE (`text/event-stream`). +Notes: +- Uses `sse-starlette` in backend routes (e.g. task/activity/memory routers). + +### 5) Gateway integration (optional) + +Mission Control can coordinate with OpenClaw Gateways over WebSockets. + +- Protocol methods/events list: `backend/app/services/openclaw/gateway_rpc.py`. +- Operator-facing protocol docs: [Gateway WebSocket protocol](openclaw_gateway_ws.md). + +## Where to start reading code + +- Backend entrypoint + router wiring: `backend/app/main.py` +- Auth dependencies + access enforcement: `backend/app/api/deps.py` +- User auth: `backend/app/core/auth.py` +- Agent auth: `backend/app/core/agent_auth.py` +- Agent API surface: `backend/app/api/agent.py` diff --git a/docs/06-configuration.md b/docs/06-configuration.md new file mode 100644 index 0000000..855f024 --- /dev/null +++ b/docs/06-configuration.md @@ -0,0 +1,105 @@ +# Configuration + +This page documents how Mission Control is configured across local dev, self-host, and production. + +## Deep dives + +- Deployment: [docs/deployment/README.md](deployment/README.md) +- Production notes: [docs/production/README.md](production/README.md) + +## Config sources & precedence + +Mission Control is a 3-service stack (`compose.yml`): Postgres (`db`), FastAPI backend (`backend`), and Next.js frontend (`frontend`). Configuration comes from a mix of **compose env files**, **service env vars**, and **app-specific env files**. + +### Docker Compose (recommended local/self-host) + +Precedence (highest → lowest): + +1) **Explicit runtime environment** passed to Compose +- `docker compose ... -e NAME=value` (or exported in your shell) + +2) **Compose env-file** used for interpolation +- `docker compose -f compose.yml --env-file .env up ...` +- Suggested workflow: copy repo root `.env.example` → `.env` and edit. + +3) **Compose defaults** embedded in `compose.yml` +- e.g. `${BACKEND_PORT:-8000}`. + +4) **Backend container env** +- `compose.yml` sets backend `env_file: ./backend/.env.example` (defaults) +- plus overrides in `compose.yml: services.backend.environment`. + +5) **Frontend container env** +- `compose.yml` sets `NEXT_PUBLIC_API_URL` via `environment:` and also as a **build arg**. +- `compose.yml` optionally loads `frontend/.env` (user-managed), *not* `frontend/.env.example`. + +### Backend env-file loading behavior (non-Compose) + +When running the backend directly (e.g., `uvicorn`), settings load from env vars and from these files: +- `backend/.env` (always attempted) +- `.env` (repo root; optional) + +This is intentional so running from repo root still picks up backend config. + +### Frontend env-file behavior (non-Compose) + +- Next.js uses `NEXT_PUBLIC_*` variables for browser-visible configuration. +- For local dev you typically create `frontend/.env.local` (Next.js convention) or `frontend/.env` (if you want Compose to read it). + +## Environment variables + +This table is based on `backend/app/core/config.py`, `.env.example`, `backend/.env.example`, `frontend/.env.example`, and `compose.yml`. + +### Compose / shared (repo root `.env`) + +| Variable | Used by | Purpose | Default / example | Footguns | +|---|---|---|---|---| +| `FRONTEND_PORT` | compose | Host port for frontend container | `3000` | Port conflicts on host are common | +| `BACKEND_PORT` | compose | Host port for backend container | `8000` | If changed, ensure frontend points at the new port | +| `POSTGRES_DB` | db/compose | Postgres database name | `mission_control` | Changing requires new DB or migration plan | +| `POSTGRES_USER` | db/compose | Postgres user | `postgres` | — | +| `POSTGRES_PASSWORD` | db/compose | Postgres password | `postgres` | Don’t use defaults in real deployments | +| `POSTGRES_PORT` | compose | Host port for Postgres | `5432` | Port conflicts on host are common | +| `CORS_ORIGINS` | backend/compose | Backend CORS allowlist | `http://localhost:3000` | Must include the real frontend origin | +| `DB_AUTO_MIGRATE` | backend/compose | Auto-run Alembic migrations at backend startup | `true` (in `.env.example`) | Can be risky in prod; see notes below | +| `NEXT_PUBLIC_API_URL` | frontend (build+runtime) | Browser-reachable backend URL | `http://localhost:8000` | Must be reachable from the **browser**, not just Docker | + +### Backend (FastAPI) + +> Settings are defined in `backend/app/core/config.py` and typically configured via `backend/.env`. + +| Variable | Required? | Purpose | Default / example | Notes | +|---|---:|---|---|---| +| `ENVIRONMENT` | no | Environment name (drives defaults) | `dev` | In `dev`, `DB_AUTO_MIGRATE` defaults to true **if not explicitly set** | +| `DATABASE_URL` | no | Postgres connection string | `postgresql+psycopg://...@localhost:5432/...` | In Compose, overridden to use `db:5432` | +| `CORS_ORIGINS` | no | Comma-separated CORS origins | empty | Compose supplies a sane default | +| `BASE_URL` | no | External base URL for this service | empty | Used for absolute links/callbacks if needed | +| `CLERK_SECRET_KEY` | **yes** | Clerk secret key (backend auth) | (none) | `backend/app/core/config.py` enforces non-empty | +| `CLERK_API_URL` | no | Clerk API base | `https://api.clerk.com` | — | +| `CLERK_VERIFY_IAT` | no | Verify issued-at claims | `true` | — | +| `CLERK_LEEWAY` | no | JWT timing leeway seconds | `10.0` | — | +| `LOG_LEVEL` | no | Logging level | `INFO` | — | +| `LOG_FORMAT` | no | Log format | `text` | — | +| `LOG_USE_UTC` | no | Use UTC timestamps | `false` | — | +| `DB_AUTO_MIGRATE` | no | Auto-migrate DB on startup | `false` in backend `.env.example` | In `dev`, backend may flip this to true if unset | + +### Frontend (Next.js) + +| Variable | Required? | Purpose | Default / example | Footguns | +|---|---:|---|---|---| +| `NEXT_PUBLIC_API_URL` | **yes** | Backend base URL used by the browser | `http://localhost:8000` | Must be browser-reachable | +| `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` | **yes** | Enables Clerk in the frontend | (none) | Must be a real publishable key | +| `CLERK_SECRET_KEY` | **yes** | Clerk secret key used by the frontend (server-side) and E2E | (none) | Do not commit; required for Clerk-enabled operation | +| `NEXT_PUBLIC_CLERK_SIGN_IN_FORCE_REDIRECT_URL` | optional | Post-login redirect | `/boards` | — | +| `NEXT_PUBLIC_CLERK_SIGN_UP_FORCE_REDIRECT_URL` | optional | Post-signup redirect | `/boards` | — | +| `NEXT_PUBLIC_CLERK_SIGN_IN_FALLBACK_REDIRECT_URL` | optional | Fallback redirect | `/boards` | — | +| `NEXT_PUBLIC_CLERK_SIGN_UP_FALLBACK_REDIRECT_URL` | optional | Fallback redirect | `/boards` | — | +| `NEXT_PUBLIC_CLERK_AFTER_SIGN_OUT_URL` | optional | Post-logout redirect | `/` | — | + +## Operational footguns + +- **Clerk placeholder keys**: `frontend/.env.example` contains non-empty Clerk placeholders. `compose.yml` intentionally does **not** load it, because it can accidentally flip Clerk “on”. Prefer user-managed `frontend/.env` (for Compose) or `frontend/.env.local` (for Next dev). +- **`DB_AUTO_MIGRATE`**: + - In `ENVIRONMENT=dev`, backend defaults `DB_AUTO_MIGRATE=true` if you didn’t set it explicitly. + - In production, consider disabling auto-migrate and running migrations as an explicit step. +- **`NEXT_PUBLIC_API_URL` reachability**: must work from the browser’s network context (host), not only from within the Docker network. diff --git a/docs/07-api-reference.md b/docs/07-api-reference.md new file mode 100644 index 0000000..346ffc2 --- /dev/null +++ b/docs/07-api-reference.md @@ -0,0 +1,290 @@ +# API reference + +## Deep dives + +- [Architecture](05-architecture.md) +- [Gateway protocol](openclaw_gateway_ws.md) + +This page summarizes the **HTTP API surface** exposed by the FastAPI backend. +It is derived from `backend/app/main.py` (router registration) and `backend/app/api/*` (route modules). + +## Base +- API prefix: `/api/v1/*` (see `backend/app/main.py`) + +## Auth model (recap) +- **Clerk (user auth)**: used by the human web UI; frontend enables Clerk when `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` is set; backend verifies requests using `CLERK_SECRET_KEY` via the Clerk SDK (see `backend/app/core/auth.py`). +- **X-Agent-Token (agent auth)**: used by automation/agents; send header `X-Agent-Token: ` to `/api/v1/agent/*` endpoints (see `backend/app/core/agent_auth.py`). + +## Route groups (modules) + +| Module | Prefix (under `/api/v1`) | Purpose | +|---|---|---| +| `activity.py` | `/activity` | Activity listing and task-comment feed endpoints. | +| `agent.py` | `/agent` | Agent-scoped API routes for board operations and gateway coordination. | +| `agents.py` | `/agents` | Thin API wrappers for async agent lifecycle operations. | +| `approvals.py` | `/boards/{board_id}/approvals` | Approval listing, streaming, creation, and update endpoints. | +| `auth.py` | `/auth` | Authentication bootstrap endpoints for the Mission Control API. | +| `board_group_memory.py` | `` | Board-group memory CRUD and streaming endpoints. | +| `board_groups.py` | `/board-groups` | Board group CRUD, snapshot, and heartbeat endpoints. | +| `board_memory.py` | `/boards/{board_id}/memory` | Board memory CRUD and streaming endpoints. | +| `board_onboarding.py` | `/boards/{board_id}/onboarding` | Board onboarding endpoints for user/agent collaboration. | +| `boards.py` | `/boards` | Board CRUD and snapshot endpoints. | +| `gateway.py` | `/gateways` | Thin gateway session-inspection API wrappers. | +| `gateways.py` | `/gateways` | Thin API wrappers for gateway CRUD and template synchronization. | +| `metrics.py` | `/metrics` | Dashboard metric aggregation endpoints. | +| `organizations.py` | `/organizations` | Organization management endpoints and membership/invite flows. | +| `souls_directory.py` | `/souls-directory` | API routes for searching and fetching souls-directory markdown entries. | +| `tasks.py` | `/boards/{board_id}/tasks` | Task API routes for listing, streaming, and mutating board tasks. | +| `users.py` | `/users` | User self-service API endpoints for profile retrieval and updates. | + +## `/activity` — `activity.py` +*Activity listing and task-comment feed endpoints.* + +### router (prefix `/activity`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/activity` | `list_activity()` | List activity events visible to the calling actor. | +| `GET` | `/api/v1/activity/task-comments` | `list_task_comment_feed()` | List task-comment feed items for accessible boards. | +| `GET` | `/api/v1/activity/task-comments/stream` | `stream_task_comment_feed()` | Stream task-comment events for accessible boards. | + +## `/agent` — `agent.py` +*Agent-scoped API routes for board operations and gateway coordination.* + +### Agent automation API (`/api/v1/agent/*`) + +Auth: +- Header: `X-Agent-Token: ` +- See: `backend/app/core/agent_auth.py` and `backend/app/api/deps.py` + +High-signal endpoint index (from `backend/app/api/agent.py`): + +| Method | Path | Purpose | +|---|---|---| +| `POST` | `/api/v1/agent/heartbeat` | Agent check-in / heartbeat status | +| `GET` | `/api/v1/agent/boards` | List boards visible to the agent | +| `GET` | `/api/v1/agent/boards/{board_id}/tasks` | List tasks with filters (status, assignment, etc.) | +| `PATCH` | `/api/v1/agent/boards/{board_id}/tasks/{task_id}` | Update task fields (status/assignment/etc.) | +| `GET` | `/api/v1/agent/boards/{board_id}/tasks/{task_id}/comments` | List task comments | +| `POST` | `/api/v1/agent/boards/{board_id}/tasks/{task_id}/comments` | Create task comment (note: request body uses `message`) | +| `GET` | `/api/v1/agent/boards/{board_id}/memory` | List board memory entries | +| `POST` | `/api/v1/agent/boards/{board_id}/memory` | Create board memory entry | +| `POST` | `/api/v1/agent/boards/{board_id}/gateway/main/ask-user` | Route an “ask user” message through gateway-main | +| `POST` | `/api/v1/agent/gateway/leads/broadcast` | Broadcast a gateway-main message to multiple board leads | + +### router (prefix `/agent`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/agent/agents` | `list_agents()` | List agents, optionally filtered to a board. | +| `POST` | `/api/v1/agent/agents` | `create_agent()` | Create an agent on the caller's board. | +| `GET` | `/api/v1/agent/boards` | `list_boards()` | List boards visible to the authenticated agent. | +| `POST` | `/api/v1/agent/heartbeat` | `agent_heartbeat()` | Record heartbeat status for the authenticated agent. | +| `GET` | `/api/v1/agent/boards/{board_id}` | `get_board()` | Return a board if the authenticated agent can access it. | +| `GET` | `/api/v1/agent/boards/{board_id}/tasks` | `list_tasks()` | List tasks on a board with optional status and assignment filters. | +| `POST` | `/api/v1/agent/boards/{board_id}/tasks` | `create_task()` | Create a task on the board as the lead agent. | +| `POST` | `/api/v1/agent/gateway/leads/broadcast` | `broadcast_gateway_lead_message()` | Broadcast a gateway-main message to multiple board leads. | +| `GET` | `/api/v1/agent/boards/{board_id}/memory` | `list_board_memory()` | List board memory entries with optional chat filtering. | +| `POST` | `/api/v1/agent/boards/{board_id}/memory` | `create_board_memory()` | Create a board memory entry. | + +## `/agents` — `agents.py` +*Thin API wrappers for async agent lifecycle operations.* + +### router (prefix `/agents`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/agents` | `list_agents()` | List agents visible to the active organization admin. | +| `POST` | `/api/v1/agents` | `create_agent()` | Create and provision an agent. | +| `GET` | `/api/v1/agents/stream` | `stream_agents()` | Stream agent updates as SSE events. | +| `POST` | `/api/v1/agents/heartbeat` | `heartbeat_or_create_agent()` | Heartbeat an existing agent or create/provision one if needed. | +| `DELETE` | `/api/v1/agents/{agent_id}` | `delete_agent()` | Delete an agent and clean related task state. | +| `GET` | `/api/v1/agents/{agent_id}` | `get_agent()` | Get a single agent by id. | +| `PATCH` | `/api/v1/agents/{agent_id}` | `update_agent()` | Update agent metadata and optionally reprovision. | +| `POST` | `/api/v1/agents/{agent_id}/heartbeat` | `heartbeat_agent()` | Record a heartbeat for a specific agent. | + +## `/boards/{board_id}/approvals` — `approvals.py` +*Approval listing, streaming, creation, and update endpoints.* + +### router (prefix `/boards/{board_id}/approvals`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/boards/{board_id}/approvals` | `list_approvals()` | List approvals for a board, optionally filtering by status. | +| `POST` | `/api/v1/boards/{board_id}/approvals` | `create_approval()` | Create an approval for a board. | +| `GET` | `/api/v1/boards/{board_id}/approvals/stream` | `stream_approvals()` | Stream approval updates for a board using server-sent events. | +| `PATCH` | `/api/v1/boards/{board_id}/approvals/{approval_id}` | `update_approval()` | Update an approval's status and resolution timestamp. | + +## `/auth` — `auth.py` +*Authentication bootstrap endpoints for the Mission Control API.* + +### router (prefix `/auth`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `POST` | `/api/v1/auth/bootstrap` | `bootstrap_user()` | Return the authenticated user profile from token claims. | + +## `` — `board_group_memory.py` +*Board-group memory CRUD and streaming endpoints.* + +### board_router (prefix `/boards/{board_id}/group-memory`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/boards/{board_id}/group-memory` | `list_board_group_memory_for_board()` | List memory entries for the board's linked group. | +| `POST` | `/api/v1/boards/{board_id}/group-memory` | `create_board_group_memory_for_board()` | Create a group memory entry from a board context and notify recipients. | +| `GET` | `/api/v1/boards/{board_id}/group-memory/stream` | `stream_board_group_memory_for_board()` | Stream memory entries for the board's linked group. | + +### group_router (prefix `/board-groups/{group_id}/memory`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/board-groups/{group_id}/memory` | `list_board_group_memory()` | List board-group memory entries for a specific group. | +| `POST` | `/api/v1/board-groups/{group_id}/memory` | `create_board_group_memory()` | Create a board-group memory entry and notify chat recipients. | +| `GET` | `/api/v1/board-groups/{group_id}/memory/stream` | `stream_board_group_memory()` | Stream memory entries for a board group via server-sent events. | + +## `/board-groups` — `board_groups.py` +*Board group CRUD, snapshot, and heartbeat endpoints.* + +### router (prefix `/board-groups`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/board-groups` | `list_board_groups()` | List board groups in the active organization. | +| `POST` | `/api/v1/board-groups` | `create_board_group()` | Create a board group in the active organization. | +| `DELETE` | `/api/v1/board-groups/{group_id}` | `delete_board_group()` | Delete a board group. | +| `GET` | `/api/v1/board-groups/{group_id}` | `get_board_group()` | Get a board group by id. | +| `PATCH` | `/api/v1/board-groups/{group_id}` | `update_board_group()` | Update a board group. | +| `GET` | `/api/v1/board-groups/{group_id}/snapshot` | `get_board_group_snapshot()` | Get a snapshot across boards in a group. | +| `POST` | `/api/v1/board-groups/{group_id}/heartbeat` | `apply_board_group_heartbeat()` | Apply heartbeat settings to agents in a board group. | + +## `/boards/{board_id}/memory` — `board_memory.py` +*Board memory CRUD and streaming endpoints.* + +### router (prefix `/boards/{board_id}/memory`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/boards/{board_id}/memory` | `list_board_memory()` | List board memory entries, optionally filtering chat entries. | +| `POST` | `/api/v1/boards/{board_id}/memory` | `create_board_memory()` | Create a board memory entry and notify chat targets when needed. | +| `GET` | `/api/v1/boards/{board_id}/memory/stream` | `stream_board_memory()` | Stream board memory events over server-sent events. | + +## `/boards/{board_id}/onboarding` — `board_onboarding.py` +*Board onboarding endpoints for user/agent collaboration.* + +### router (prefix `/boards/{board_id}/onboarding`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/boards/{board_id}/onboarding` | `get_onboarding()` | Get the latest onboarding session for a board. | +| `POST` | `/api/v1/boards/{board_id}/onboarding/agent` | `agent_onboarding_update()` | Store onboarding updates submitted by the gateway agent. | +| `POST` | `/api/v1/boards/{board_id}/onboarding/start` | `start_onboarding()` | Start onboarding and send instructions to the gateway agent. | +| `POST` | `/api/v1/boards/{board_id}/onboarding/answer` | `answer_onboarding()` | Send a user onboarding answer to the gateway agent. | +| `POST` | `/api/v1/boards/{board_id}/onboarding/confirm` | `confirm_onboarding()` | Confirm onboarding results and provision the board lead agent. | + +## `/boards` — `boards.py` +*Board CRUD and snapshot endpoints.* + +### router (prefix `/boards`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/boards` | `list_boards()` | List boards visible to the current organization member. | +| `POST` | `/api/v1/boards` | `create_board()` | Create a board in the active organization. | +| `DELETE` | `/api/v1/boards/{board_id}` | `delete_board()` | Delete a board and all dependent records. | +| `GET` | `/api/v1/boards/{board_id}` | `get_board()` | Get a board by id. | +| `PATCH` | `/api/v1/boards/{board_id}` | `update_board()` | Update mutable board properties. | +| `GET` | `/api/v1/boards/{board_id}/snapshot` | `get_board_snapshot()` | Get a board snapshot view model. | +| `GET` | `/api/v1/boards/{board_id}/group-snapshot` | `get_board_group_snapshot()` | Get a grouped snapshot across related boards. | + +## `/gateways` — `gateway.py` +*Thin gateway session-inspection API wrappers.* + +### router (prefix `/gateways`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/gateways/status` | `gateways_status()` | Return gateway connectivity and session status. | +| `GET` | `/api/v1/gateways/commands` | `gateway_commands()` | Return supported gateway protocol methods and events. | +| `GET` | `/api/v1/gateways/sessions` | `list_gateway_sessions()` | List sessions for a gateway associated with a board. | +| `GET` | `/api/v1/gateways/sessions/{session_id}` | `get_gateway_session()` | Get a specific gateway session by key. | +| `GET` | `/api/v1/gateways/sessions/{session_id}/history` | `get_session_history()` | Fetch chat history for a gateway session. | +| `POST` | `/api/v1/gateways/sessions/{session_id}/message` | `send_gateway_session_message()` | Send a message into a specific gateway session. | + +## `/gateways` — `gateways.py` +*Thin API wrappers for gateway CRUD and template synchronization.* + +### router (prefix `/gateways`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/gateways` | `list_gateways()` | List gateways for the caller's organization. | +| `POST` | `/api/v1/gateways` | `create_gateway()` | Create a gateway and provision or refresh its main agent. | +| `DELETE` | `/api/v1/gateways/{gateway_id}` | `delete_gateway()` | Delete a gateway in the caller's organization. | +| `GET` | `/api/v1/gateways/{gateway_id}` | `get_gateway()` | Return one gateway by id for the caller's organization. | +| `PATCH` | `/api/v1/gateways/{gateway_id}` | `update_gateway()` | Patch a gateway and refresh the main-agent provisioning state. | +| `POST` | `/api/v1/gateways/{gateway_id}/templates/sync` | `sync_gateway_templates()` | Sync templates for a gateway and optionally rotate runtime settings. | + +## `/metrics` — `metrics.py` +*Dashboard metric aggregation endpoints.* + +### router (prefix `/metrics`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/metrics/dashboard` | `dashboard_metrics()` | Return dashboard KPIs and time-series data for accessible boards. | + +## `/organizations` — `organizations.py` +*Organization management endpoints and membership/invite flows.* + +### router (prefix `/organizations`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `POST` | `/api/v1/organizations` | `create_organization()` | Create an organization and assign the caller as owner. | +| `DELETE` | `/api/v1/organizations/me` | `delete_my_org()` | Delete the active organization and related entities. | +| `GET` | `/api/v1/organizations/me` | `get_my_org()` | Return the caller's active organization. | +| `GET` | `/api/v1/organizations/me/list` | `list_my_organizations()` | List organizations where the current user is a member. | +| `PATCH` | `/api/v1/organizations/me/active` | `set_active_org()` | Set the caller's active organization. | +| `GET` | `/api/v1/organizations/me/member` | `get_my_membership()` | Get the caller's membership record in the active organization. | +| `GET` | `/api/v1/organizations/me/invites` | `list_org_invites()` | List pending invites for the active organization. | +| `POST` | `/api/v1/organizations/me/invites` | `create_org_invite()` | Create an organization invite for an email address. | +| `GET` | `/api/v1/organizations/me/members` | `list_org_members()` | List members for the active organization. | +| `POST` | `/api/v1/organizations/invites/accept` | `accept_org_invite()` | Accept an invite and return resulting membership. | + +## `/souls-directory` — `souls_directory.py` +*API routes for searching and fetching souls-directory markdown entries.* + +### router (prefix `/souls-directory`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/souls-directory/search` | `search()` | Search souls-directory entries by handle/slug query text. | +| `GET` | `/api/v1/souls-directory/{handle}/{slug}` | `get_markdown()` | Fetch markdown content for a validated souls-directory handle and slug. | +| `GET` | `/api/v1/souls-directory/{handle}/{slug}.md` | `get_markdown()` | Fetch markdown content for a validated souls-directory handle and slug. | + +## `/boards/{board_id}/tasks` — `tasks.py` +*Task API routes for listing, streaming, and mutating board tasks.* + +### router (prefix `/boards/{board_id}/tasks`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `GET` | `/api/v1/boards/{board_id}/tasks` | `list_tasks()` | List board tasks with optional status and assignment filters. | +| `POST` | `/api/v1/boards/{board_id}/tasks` | `create_task()` | Create a task and initialize dependency rows. | +| `GET` | `/api/v1/boards/{board_id}/tasks/stream` | `stream_tasks()` | Stream task and task-comment events as SSE payloads. | +| `DELETE` | `/api/v1/boards/{board_id}/tasks/{task_id}` | `delete_task()` | Delete a task and related records. | +| `PATCH` | `/api/v1/boards/{board_id}/tasks/{task_id}` | `update_task()` | Update task status, assignment, comment, and dependency state. | +| `GET` | `/api/v1/boards/{board_id}/tasks/{task_id}/comments` | `list_task_comments()` | List comments for a task in chronological order. | +| `POST` | `/api/v1/boards/{board_id}/tasks/{task_id}/comments` | `create_task_comment()` | Create a task comment and notify relevant agents. | + +## `/users` — `users.py` +*User self-service API endpoints for profile retrieval and updates.* + +### router (prefix `/users`) + +| Method | Path | Handler | Notes | +|---|---|---|---| +| `DELETE` | `/api/v1/users/me` | `delete_me()` | Delete the authenticated account and any personal-only organizations. | +| `GET` | `/api/v1/users/me` | `get_me()` | Return the authenticated user's current profile payload. | +| `PATCH` | `/api/v1/users/me` | `update_me()` | Apply partial profile updates for the authenticated user. | diff --git a/docs/08-agents-and-skills.md b/docs/08-agents-and-skills.md new file mode 100644 index 0000000..1712154 --- /dev/null +++ b/docs/08-agents-and-skills.md @@ -0,0 +1,28 @@ +# Agents & skills + +## Deep dives + +- [Gateway protocol](openclaw_gateway_ws.md) +- [Gateway base config](openclaw_gateway_base_config.md) + +This page explains the automation model as it appears in Mission Control. + +## Agent lifecycle (conceptual) +- An **agent** checks in to Mission Control (often on a schedule) and posts work results as task comments. +- In OpenClaw terms, agents can run: + - **heartbeats** (periodic loops) + - **cron jobs** (scheduled runs; better for exact timing / isolation) + +## Heartbeats vs cron +- Use **heartbeat** for batched checks and context-aware incremental work. +- Use **cron** for exact timing and isolated, standalone actions. + +## Skills (how to think about them) +- A skill is a packaged workflow/tooling instruction set that agents can follow. +- Skills typically define: + - when to use them + - required binaries/services + - command patterns + +## Next +- Add repo-specific guidance for authoring skills and where they live (once standardized). diff --git a/docs/09-ops-runbooks.md b/docs/09-ops-runbooks.md new file mode 100644 index 0000000..1094e05 --- /dev/null +++ b/docs/09-ops-runbooks.md @@ -0,0 +1,38 @@ +# Ops / runbooks + +## Deep dives + +- [Deployment](deployment/README.md) +- [Production](production/README.md) +- [Troubleshooting](troubleshooting/README.md) + +This page is the operator entrypoint. It points to the existing deep-dive runbooks and adds a short “first 30 minutes” checklist. + +## First 30 minutes (incident checklist) + +1. **Confirm impact** + - What’s broken: UI, API, auth, or gateway integration? + - All users or a subset? + +2. **Check service health** + - Backend: `/healthz` and `/readyz` + - Frontend: can it load? does it reach the API? + +3. **Check auth (Clerk)** + - Frontend: did Clerk get enabled unintentionally? (publishable key set) + - Backend: is `CLERK_SECRET_KEY` configured correctly? + +4. **Check DB connectivity** + - Can backend connect to Postgres (`DATABASE_URL`)? + +5. **Check logs** + - Backend logs for 5xx spikes or auth failures. + - Frontend logs for API URL/proxy misconfig. + +6. **Stabilize** + - Roll back the last change if you can. + - Temporarily disable optional integrations (gateway) to isolate. + +## Backups / restore + +See [Production](production/README.md). If you run Mission Control in production, treat backup/restore as a regular drill, not a one-time setup. diff --git a/docs/10-troubleshooting.md b/docs/10-troubleshooting.md new file mode 100644 index 0000000..6e837bd --- /dev/null +++ b/docs/10-troubleshooting.md @@ -0,0 +1,24 @@ +# Troubleshooting + +## Deep dives + +- [Troubleshooting deep dive](troubleshooting/README.md) + +This is the “quick triage” page. For detailed playbooks and diagnostics, use the deep dive. + +## Quick triage + +### Frontend loads but shows API errors +- Confirm `NEXT_PUBLIC_API_URL` points to a backend your browser can reach. +- Check backend `/healthz`. + +### Frontend keeps redirecting / Clerk errors +- Verify your Clerk keys are set correctly in the frontend environment. +- See: [Deployment guide](deployment/README.md) (Clerk auth notes). + +### Backend returns 5xx +- Check DB connectivity (`DATABASE_URL`) and migrations. +- Check backend logs. + +## Next +- Promote the most common issues from [Troubleshooting deep dive](troubleshooting/README.md) into this page once we see repeated incidents. diff --git a/docs/11-contributing.md b/docs/11-contributing.md new file mode 100644 index 0000000..f5b2ac9 --- /dev/null +++ b/docs/11-contributing.md @@ -0,0 +1,18 @@ +# Contributing + +## Deep dives + +- [Coverage policy](coverage-policy.md) +- [Testing guide](testing/README.md) + +## How to contribute +- Follow the repo’s existing PR and review conventions. +- Prefer small PRs. +- Update docs when behavior changes. + +## Adding/changing docs +- Canonical docs live in `docs/`. +- Follow the IA in [Docs landing](README.md). + +## Testing expectations +- See [docs/testing/README.md](testing/README.md). diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..8f75bbf --- /dev/null +++ b/docs/README.md @@ -0,0 +1,34 @@ +# Mission Control docs + +This folder is the canonical documentation set for Mission Control. + +## Start here (by role) + +- **Contributor**: start with [Quickstart](../README.md#quick-start-self-host-with-docker-compose) → [Development](development.md) → [Contributing](contributing.md) +- **Maintainer**: start with [Architecture](05-architecture.md) → [Repo tour](04-repo-tour.md) → [API reference](07-api-reference.md) +- **Operator/SRE**: start with [Ops / runbooks](09-ops-runbooks.md) → [Troubleshooting](10-troubleshooting.md) + +## Table of contents (IA) + +1. [Overview](01-overview.md) +2. [Quickstart](02-quickstart.md) +3. [Development](03-development.md) +4. [Repo tour](04-repo-tour.md) +5. [Architecture](05-architecture.md) +6. [Configuration](06-configuration.md) +7. [API reference](07-api-reference.md) +8. [Agents & skills](08-agents-and-skills.md) +9. [Ops / runbooks](09-ops-runbooks.md) +10. [Troubleshooting](10-troubleshooting.md) +11. [Contributing](11-contributing.md) + +## Existing deep-dive docs + +These deeper references already exist under `docs/` directories: +- [Architecture deep dive](architecture/README.md) +- [Deployment guide](deployment/README.md) +- [Production notes](production/README.md) +- [Testing guide](testing/README.md) +- [Troubleshooting](troubleshooting/README.md) +- [Gateway WebSocket protocol](openclaw_gateway_ws.md) +- [Gateway base config](openclaw_gateway_base_config.md) diff --git a/docs/architecture/README.md b/docs/architecture/README.md index f77dfb8..183de6f 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -29,9 +29,11 @@ flowchart LR - Routes/pages: `frontend/src/app/*` (Next.js App Router) - API utilities: `frontend/src/lib/*` and `frontend/src/api/*` -**Auth (Clerk, required for now)** -- The codebase includes gating so CI/local can run without secrets, but real deployments should configure Clerk. -- See `frontend/src/auth/clerkKey.ts`, `frontend/src/auth/clerk.tsx`, and `frontend/src/proxy.ts`. +**Auth (Clerk, required)** +- Clerk is required for real deployments and currently required by backend config (see `backend/app/core/config.py`). +- Frontend uses Clerk when keys are configured; see `frontend/src/auth/clerkKey.ts` and `frontend/src/auth/clerk.tsx`. +- Backend authenticates requests using the Clerk SDK and `CLERK_SECRET_KEY`; see `backend/app/core/auth.py`. + ### Backend (FastAPI) - Location: `backend/` @@ -52,8 +54,8 @@ flowchart LR ### Gateway integration (optional) Mission Control can call into an OpenClaw Gateway over WebSockets. - Client + protocol: `backend/app/services/openclaw/gateway_rpc.py` -- Protocol doc: `docs/openclaw_gateway_ws.md` -- Base gateway config (getting started): `docs/openclaw_gateway_base_config.md` +- Protocol doc: [Gateway WebSocket protocol](../openclaw_gateway_ws.md) +- Base gateway config (getting started): [Gateway base config](../openclaw_gateway_base_config.md) ## Request flows @@ -62,11 +64,9 @@ Mission Control can call into an OpenClaw Gateway over WebSockets. 2. Frontend calls backend endpoints under `/api/v1/*`. 3. Backend reads/writes Postgres. -### Auth (Clerk — required for now) -- **Frontend** enables Clerk when a publishable key is present/valid. -- **Backend** uses `fastapi-clerk-auth` when `CLERK_JWKS_URL` is configured. - - See `backend/app/core/auth.py`. - +### Auth (Clerk — required) +- **Frontend** uses Clerk when keys are configured (see `frontend/src/auth/*`). +- **Backend** authenticates requests using the Clerk SDK and `CLERK_SECRET_KEY` (see `backend/app/core/auth.py`). ### Agent access (X-Agent-Token) Automation/agents can use the “agent” API surface: - Endpoints under `/api/v1/agent/*` (router: `backend/app/api/agent.py`). @@ -110,9 +110,9 @@ Frontend: ## Related docs - Self-host (Docker Compose): see repo root README: [Quick start (self-host with Docker Compose)](../../README.md#quick-start-self-host-with-docker-compose) -- Production-ish deployment: [`docs/production/README.md`](../production/README.md) -- Testing (Cypress/Clerk): [`docs/testing/README.md`](../testing/README.md) -- Troubleshooting: [`docs/troubleshooting/README.md`](../troubleshooting/README.md) +- Production-ish deployment: [Production notes](../production/README.md) +- Testing (Cypress/Clerk): [Testing guide](../testing/README.md) +- Troubleshooting: [Troubleshooting](../troubleshooting/README.md) ## Notes / gotchas - Mermaid rendering depends on the markdown renderer. diff --git a/docs/deployment/README.md b/docs/deployment/README.md index 5224335..0d5aba6 100644 --- a/docs/deployment/README.md +++ b/docs/deployment/README.md @@ -15,7 +15,7 @@ When running Compose, you get: - Health check: `GET /healthz` - **Frontend UI** (Next.js) on `http://localhost:${FRONTEND_PORT:-3000}` -Auth (Clerk) is **required** right now. You must configure Clerk keys for the frontend, and configure `CLERK_JWKS_URL` for the backend so protected API routes can verify JWTs. +Auth (Clerk) is **required** right now. You must configure Clerk keys for the frontend and backend (`CLERK_SECRET_KEY`). ## Requirements @@ -133,18 +133,24 @@ NEXT_PUBLIC_CLERK_SIGN_IN_FALLBACK_REDIRECT_URL=/boards NEXT_PUBLIC_CLERK_SIGN_UP_FALLBACK_REDIRECT_URL=/boards ``` -### Backend (JWT verification) +### Backend (auth) -The backend verifies Clerk JWTs using **`CLERK_JWKS_URL`**. +The backend authenticates requests using the Clerk SDK and **`CLERK_SECRET_KEY`** (see `backend/app/core/auth.py`). -- Compose loads `backend/.env.example` by default, where `CLERK_JWKS_URL` is empty. -- For a real deployment, provide a real value either by: - 1) creating `backend/.env` and updating `compose.yml` to load it (preferred), **or** - 2) adding `CLERK_JWKS_URL: ${CLERK_JWKS_URL}` under `backend.environment` and setting it in root `.env`. +Create `backend/.env` (this file is **not** committed) with at least: -Related backend env vars (optional tuning): -- `CLERK_VERIFY_IAT` (default: true) -- `CLERK_LEEWAY` (default: 10.0) +```env +CLERK_SECRET_KEY=sk_test_your_real_key + +# Optional tuning +CLERK_API_URL=https://api.clerk.com +CLERK_VERIFY_IAT=true +CLERK_LEEWAY=10.0 +``` + +Then either: +1) update `compose.yml` to load `backend/.env` (recommended), or +2) pass the values via `services.backend.environment`. **Security:** treat `CLERK_SECRET_KEY` like a password. Do not commit it. diff --git a/docs/openclaw_gateway_base_config.md b/docs/openclaw_gateway_base_config.md index 7bdb22d..38b7b0a 100644 --- a/docs/openclaw_gateway_base_config.md +++ b/docs/openclaw_gateway_base_config.md @@ -3,8 +3,8 @@ This document explains a "base" OpenClaw Gateway config intended for local development and LAN use, and how to connect it to Mission Control. Related: -- Gateway WebSocket protocol: `docs/openclaw_gateway_ws.md` (default URL `ws://127.0.0.1:18789`) -- Mission Control architecture (gateway integration): `docs/architecture/README.md` +- Gateway WebSocket protocol: [Gateway WebSocket protocol](openclaw_gateway_ws.md) (default URL `ws://127.0.0.1:18789`) +- Mission Control architecture (gateway integration): [Architecture deep dive](architecture/README.md) ## Who This Is For diff --git a/docs/testing/README.md b/docs/testing/README.md index 23bf76c..a788126 100644 --- a/docs/testing/README.md +++ b/docs/testing/README.md @@ -30,7 +30,7 @@ Implementation in this repo: - `cy.clerkLoaded()` - `cy.clerkSignIn(...)` / `cy.clerkSignOut(...)` -See also: [`docs/e2e-auth.md`](../e2e-auth.md). +See also: [E2E auth notes](../e2e-auth.md). ### Test user (non-secret)