OpsNexus

Architecture

The services that make up OpsNexus, how requests and background jobs flow between them, and where data is stored.

On this page

OpsNexus is a web app (React) in front of a Python API (FastAPI), with Celery workers for background jobs, PostgreSQL for data, Redis for queues and live updates, and HashiCorp Vault for secret values. Build agents run on your own machines and talk to the API over HTTPS. Everything except the agents runs as containers from one docker-compose.yml.

Diagram

                        Browser
                           │  HTTPS (via your reverse proxy)
                           ▼
                 ┌───────────────────┐
                 │ web (Nginx :8080) │  serves the React app
                 └─────────┬─────────┘
                           │ /api, /api/ws (WebSocket)
                           ▼
 Build agents ──HTTPS──► ┌──────────────────┐ ──► vault (secret values)
 (Linux, Windows,        │ api (FastAPI     │ ──► guacd ──RDP──► Windows servers
  macOS, Docker)         │      :8000)      │ ──SSH──────────► Linux servers
                         └──┬───────────┬───┘
                            │           │
                            ▼           ▼
                     ┌──────────┐  ┌──────────────────────────┐
                     │ db       │  │ redis                    │
                     │ Postgres │  │ job queue, pub/sub,      │
                     └──────────┘  │ rate limits              │
                            ▲      └──────┬────────────▲──────┘
                            │             │ jobs       │ schedule
                            │             ▼            │
                            │      ┌────────────┐  ┌───┴────────┐
                            └──────┤ worker     │  │ scheduler  │
                                   │ (Celery)   │  │ (Celery    │
                                   └─────┬──────┘  │  beat)     │
                                         │         └────────────┘
                                         ▼
           Jenkins, GitLab, Azure DevOps, Argo CD, GitHub Actions,
           AWS, Azure, Google Cloud, Oracle Cloud, Kubernetes APIs,
           container registries, scanners, notification channels

Services

Service Image What it does
web built from frontend/ (Nginx, unprivileged) Serves the React app on port 8080 inside the container, published on WEB_PORT. Proxies /api and the WebSocket to api. Adds security headers and a Content Security Policy.
api built from backend/ FastAPI on port 8000 (not published). On start it runs database migrations and first-run setup, then serves the REST API, the WebSocket and the API docs at /api/docs.
worker same backend image Celery worker. Runs background jobs from the Redis queue: integration and cloud syncs, domain checks, notification delivery with retries, report generation and more.
scheduler same backend image Celery beat. Puts scheduled jobs on the queue. Must run as exactly one container.
db postgres:16-alpine PostgreSQL. All application data.
redis redis:7-alpine with append-only persistence Job queue and result store for Celery, publish/subscribe for live updates, and rate limiting.
vault hashicorp/vault:1.18 Stores secret values in a KV v2 engine. UI on 127.0.0.1:${VAULT_HOST_PORT} (default 18200).
vault-init hashicorp/vault:1.18 One-shot step on every start: initializes and unseals Vault, enables KV v2 at secret/, writes the OpsNexus policy, and creates a least-privilege token for the backend. Exits when done.
guacd guacamole/guacd:1.5.5 Apache Guacamole proxy for Remote Desktop (RDP) in the browser. Only the API talks to it; no published port.
runner built from agent/ Optional build agent in Docker, started with --profile runner.

The api, worker and scheduler containers share one image. The entrypoint picks the role: api, worker or scheduler.

How a request flows

  1. The browser loads the app from web and calls /api/... with an access token (Authorization: Bearer ...), or a script calls it with an API key (X-API-Key).
  2. web forwards the request to api.
  3. The API checks the token, checks that the session has not been revoked, and loads the user's effective permissions: the highest level per module across the user's role and team roles, then their project role for project modules.
  4. If the level is too low for the action, the API returns 403.
  5. The route calls a service, which reads and writes PostgreSQL (and Vault when secrets change) and records an audit event.
  6. Every query is limited to the caller's organization and active project.

How live updates flow

  1. A service or background job publishes an event (for example, an agent status change or a finished sync) on the organization's Redis channel.
  2. The browser keeps a WebSocket open to /api/ws.
  3. The API forwards events from Redis to the browser, which refreshes the affected data.

How background jobs flow

  1. The scheduler puts due jobs on the Redis queue at fixed intervals.
  2. The API also queues jobs directly, for example a sync you start by hand or a notification to deliver.
  3. A worker takes each job, does the work (usually calling an external API) and writes the result to PostgreSQL.

If Redis cannot be reached when the API queues a sync or a pipeline scan, the API can run it directly instead.

Scheduled jobs

Job Interval Purpose
Sync integrations every INTEGRATION_SYNC_MINUTES (5) Import pipelines and recent builds from external CI/CD tools.
Mark offline agents 30 seconds Mark agents silent for longer than AGENT_OFFLINE_SECONDS as disconnected or offline.
CI tick 30 seconds OpsNexus pipelines: fail runs whose agent stopped responding, start scheduled and SCM-polling builds, scan folders.
Check domains 30 seconds Run the domain checks that are due.
Evaluate alerts 1 minute Evaluate alert rules against current data.
Sync Kubernetes 1 minute Collect cluster data.
Refresh deployments 2 minutes Update the status of marketplace tool deployments.
Notification digests 2 minutes Send notification summaries that are due.
Report schedules 5 minutes Generate scheduled reports.
Scheduled backup 10 minutes Run in-app backups that are due.
Sync registries 30 minutes Sync container registries.
Sync cloud accounts every CLOUD_SYNC_MINUTES (60) Import cloud resources and costs.
Prune metrics hourly Delete agent metrics, domain checks, traffic and report runs past their retention.
Sync scanners hourly Import findings from connected scanners such as SonarQube.
Apply retention policies daily Apply container registry retention policies (only for registries that allow write actions).
Refresh GeoIP daily Refresh the country database, only when a domain uses the tracking snippet.
Renew Vault token daily Keep the periodic Vault token alive.

How build agents work

Agents only make outbound HTTPS requests to OpsNexus. OpsNexus never connects to them.

  1. Register. The agent exchanges a single-use registration token for an agent ID and secret, which it saves in a config file readable only by its service account.
  2. Heartbeat. About every 15 seconds it sends host health and receives settings and commands (such as restart or upgrade).
  3. Run pipelines. When it has executors, it claims pipeline runs from the API, runs the steps in its workspace, streams logs and events back, uploads artifacts, and reports the result.
  4. Infra designer deployments run as pipelines on agents. The agent downloads the generated Terraform, runs plan and apply in the hashicorp/terraform image, and uses OpsNexus as the Terraform HTTP state backend.

How remote access works

  • SSH terminal. The browser opens a WebSocket to /api/ws/terminal/... with a single-use, 60-second ticket. The API opens the SSH connection itself, with credentials from the secret store, so passwords and keys never reach the browser.
  • Remote Desktop. The API connects to guacd, does the Guacamole handshake and fills in the password. guacd opens the RDP connection, so it must be able to reach your servers.

Where data is stored

Store What it holds
PostgreSQL Organizations, users, roles, teams, projects, sessions and API keys (hashed); integrations, pipelines, runs and logs; agents and their metrics; cloud accounts, resources and costs; Kubernetes, monitoring, domains, notifications, dashboards and reports; secret metadata; the audit log.
Vault Secret values: integration tokens, cloud credentials, server passwords and keys, pipeline secrets. Rows in PostgreSQL keep only the path.
Redis The job queue and job results, live-update channels and rate-limit counters. Persisted with append-only mode, but not a system of record.
artifacts volume Files archived by pipeline runs.
backups volume In-app backup files.
geoip volume The DB-IP Lite country database.

Some values in PostgreSQL are encrypted with a key derived from SECRET_KEY (or LOCAL_SECRET_KEY): MFA seeds, Terraform state, pipeline webhook tokens and secret masks for build logs. The audit log is append-only and each entry is hash-chained to the previous one.

With SECRET_BACKEND=local (development only), secret values are stored encrypted in PostgreSQL instead of Vault.

Scaling

  • api is stateless. Raise API_WORKERS, or run more replicas behind a load balancer.
  • worker scales horizontally. Raise WORKER_CONCURRENCY or run more containers.
  • scheduler must run as exactly one container.
  • Builds run on agents, not on the server. Add agents or executors to run more builds at once.

See Running in production and Environment variables.