OpsNexus

Troubleshooting

Common problems with agents, sign-in, cloud sync, remote access, Vault, backups and deployments, and how to fix them.

On this page

Each entry starts with what you see, then explains the cause and the fix. Messages in quotes are the exact text OpsNexus shows.

Tip

Most server-side errors are also written to the container logs. Run docker compose logs --tail 200 api worker on the OpsNexus host when an error message isn't enough.

Build agents

The agent log repeats "Claim failed: HTTP 401" and the container keeps restarting

Cause. The agent was unregistered in OpsNexus (or its config belongs to another OpsNexus database), so the server rejects its ID and secret. Agents before 1.3 exited on a 401. Docker's --restart unless-stopped then started them again, forever. Running the install command again with a new token didn't help, because the old config in the data volume won.

Fix. Upgrade to agent 1.3 or later. A 1.3+ agent no longer exits. It logs one message that starts with "OpsNexus at ... rejected this agent", pauses the pipeline runner and checks again every 5 minutes. Then re-register it:

  • Docker: create a new token under Build agents → Download & install and run the Docker command again. The command removes the old container first. The agent sees a registration token it didn't register with and registers again, even with the same volume.
  • Installed service: run register again with the new token, then restart the service.
bash
python onx_agent.py register --url https://opsnexus.example.com --token onx_reg_... --name build-01

See Build agents for upgrade steps.

"Registration failed: An agent named X already exists"

Cause. An agent with that name is still registered in the organization.

Fix. Unregister the old agent under Build agents, or pick another name.

"This registration token is invalid, used or expired"

Cause. Registration tokens are single use and expire (24 hours by default, at most 7 days).

Fix. Create a new token. Since agent 1.3.2, a container started with an old, already used token logs "Registration with the given token failed ... using the saved registration" and keeps working with its saved registration instead of exiting.

"WARNING: http://localhost:8080 only works on the OpsNexus machine itself"

Cause. The agent was given localhost or host.docker.internal as the server address. Those addresses only work on the OpsNexus host. Agents on other machines can't reach them.

Fix. Use an address every agent can reach:

  1. Set AGENT_URL in .env to the server's network address, for example http://192.168.1.10:8080, and restart the api service. Install commands then use it. Leave PUBLIC_URL alone if SSO is set up, because SSO providers need that exact URL.
  2. For a one-off install, edit the OpsNexus address field on the install page.
  3. Open WEB_PORT (default 8080) in the server's firewall.
  4. Test from the agent machine: curl http://<address>/api/health/live.

An agent that is already registered doesn't need to register again after OpsNexus moves:

bash
# Container: recreate it with the new address (ONX_URL always wins over the saved one)
docker run ... -e ONX_URL=https://new-address ...

# Installed service
python onx_agent.py set-url https://new-address

"Registration failed: can't reach OpsNexus" or "Server unreachable"

Cause. Wrong address, DNS, or a firewall between the agent and OpsNexus.

Fix. Same steps as above. The agent only needs outbound HTTP(S) to OpsNexus.

Builds stay queued: "No build agent runs pipelines yet"

Cause. No online agent has executors, or no agent has the labels the pipeline asks for. Builds wait in the queue until an agent with free executors and matching labels picks them up.

Fix. Install an agent with executors, for example --executors 2 --labels linux, or start the bundled one:

bash
# Set RUNNER_TOKEN=onx_reg_... in .env first
docker compose --profile runner up -d runner

Check that the pipeline's agent: { labels: [...] } matches labels an online agent has.

Checkout fails with "remote: Repository not found"

Cause. GitHub and GitLab answer "not found" for private repositories the credential can't see. Agent 1.3.1 and later explain this as "Can't access the repository". Common reasons:

  • The pipeline has no Git credential and the repository is private.
  • The token is valid but has no access to this repository. GitHub fine-grained tokens only reach repositories of their own resource owner (the account or organization chosen when the token was created) and need Contents: read.
  • The token expired, or the repository name is wrong.

Fix. Add or change the credential under the pipeline's Source code management settings. For another account's repository, use a token created by that account, a classic token with repo scope from a collaborator, or an SSH deploy key. Use Verify connection to check the credential, repository, branch and pipeline file before saving.

Sign-in, MFA and SSO

"Wrong sign-in details"

Cause. The email, username or password is wrong. OpsNexus never says which part was wrong, so it doesn't reveal whether an account exists.

Fix. Check the details. If you use Account ID sign-in, check the Account ID too.

"This account is locked after too many failed attempts"

Cause. The password policy locks an account after a number of failed attempts (5 by default) for a set time.

Fix. Wait for the lock to end, or ask an admin to unlock the account under Access control → Users. Admins can change the lockout settings in the password policy.

"That code is not valid. Check your authenticator app and try again."

Cause. The 6-digit code is wrong or out of date. Codes are accepted one step either side of the current one, so a phone clock that is off by more than about 30 seconds fails.

Fix. Turn on automatic time on the phone and try a fresh code.

"The sign-in step expired. Start again."

Cause. After the password step you have 5 minutes to enter the MFA code.

Fix. Sign in again.

Sign-in answers with HTTP 429

Cause. Sign-in allows 10 attempts per minute for each address and account, and MFA 10 per minute for each address.

Fix. Wait a minute and try again.

SSO: "Google sign-in is not set up"

Cause. The provider's client ID and secret are not in .env.

Fix. Set GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET (or the MICROSOFT_* settings) and restart the api service. See SSO and MFA.

SSO: the provider shows a redirect URI mismatch

Cause. The redirect URI registered with Google or Microsoft doesn't match PUBLIC_URL exactly.

Fix. Register these redirect URIs, built from your PUBLIC_URL:

https://opsnexus.example.com/api/auth/sso/google/callback
https://opsnexus.example.com/api/auth/sso/microsoft/callback

Don't change PUBLIC_URL to reach agents. Use AGENT_URL for that.

SSO: "No active OpsNexus account uses this email"

Cause. SSO only signs in existing users. It matches the verified email from the provider to a user in OpsNexus.

Fix. Ask an admin to add you with the same email you use at the provider.

Cause. The sign-in started in another tab, or took too long.

Fix. Start the sign-in again from the OpsNexus login page.

"Your session expired" or "Your session was ended"

Cause. The session reached its end (REFRESH_TOKEN_DAYS, 7 by default), or someone signed it out under My profile → Sessions.

Fix. Sign in again.

Cloud accounts

An account shows an error after sync

The message on the account says what failed. The most common ones:

Message Fix
"AWS sign-in failed" Check the access key and secret, or the role to assume and its external ID.
"Azure sign-in failed" Check the tenant ID, client ID and client secret of the service principal.
"Azure denied access to subscription ... Grant the Reader role." Give the service principal Reader on that subscription.
"Google sign-in failed" or "The service account key isn't valid JSON" Paste the whole JSON key file again.
"Cloud Asset API (project): HTTP 403 ..." Enable the Cloud Asset API in the project and grant Cloud Asset Viewer.
"OCI refused the request ... Check the OCIDs, fingerprint and policies." Check the tenancy and user OCIDs, the key fingerprint and the IAM policy.
"The OCI private key isn't a valid PEM key without a passphrase" Use an API signing key without a passphrase.

OpsNexus only reads from clouds. Grant read-only roles: AWS ReadOnlyAccess plus ce:GetCostAndUsage, Azure Reader and Cost Management Reader, Google Cloud Asset Viewer plus BigQuery access to the billing dataset, OCI inspect all-resources and read usage-reports.

Resources appear but cost is empty

Cause. Cost comes from a different API than inventory, with its own permission. The account shows a cost status such as:

  • "Cost Explorer unavailable" (AWS)
  • "Cost Management: HTTP 403 (grant Cost Management Reader)" (Azure)
  • "Set the billing export table to see GCP cost" (Google Cloud)
  • "OCI Usage API: HTTP ... (policy: allow group ... to read usage-reports in tenancy)" (Oracle)

Fix. Grant the cost permission named in the message. For Google Cloud, turn on the BigQuery billing export and enter the table name on the account. See Cost and budgets.

Servers, SSH and Remote Desktop

"No answer from host:22 within 15 seconds"

Cause. A firewall drops the traffic, the address is wrong, or the server is off. The OpsNexus api container makes the connection, so the path from the OpsNexus host is what matters.

Fix. Check the address and the firewall from the OpsNexus host. TERMINAL_CONNECT_TIMEOUT_SECONDS changes the timeout.

"refused the connection: nothing listens for SSH on port 3389"

Cause. Port 3389 is Windows Remote Desktop, not SSH.

Fix. Edit the server and set its type to Remote Desktop (RDP), or keep SSH and use port 22 with the Windows OpenSSH Server feature. When a terminal fails on port 3389, the terminal page offers Switch to Remote Desktop.

"The server refused the sign-in. Check the username and the password or key."

Cause. Wrong username, password or key. On Windows, the domain may be missing.

Fix. Check the credential. For RDP, enter the domain or use DOMAIN\user; OpsNexus splits it for you. Remote Desktop needs a password; a private key doesn't work.

"The server's host key changed"

Cause. OpsNexus pins each server's SSH host key on first use. A different key blocks the connection. This happens when a server is rebuilt, or when something intercepts the connection.

Fix. If the change is expected, someone with full access to Cloud resets the trusted key on the server's page.

"The remote desktop gateway (guacd ...) isn't reachable"

Cause. Remote Desktop goes through the guacd service, and it is not running.

Fix.

bash
docker compose up -d guacd

"You already have N open terminals. Close some first."

Cause. Each person can have a limited number of terminals and desktops open at once.

Fix. Close terminals or desktops you don't use. Sessions also close after TERMINAL_IDLE_MINUTES (30) without input and after TERMINAL_MAX_HOURS (8).

Secrets and Vault

System status shows the secret store as "sealed"

Cause. Vault restarted and is sealed. With the bundled compose setup, the one-shot vault-init service unseals it on every docker compose up.

Fix.

bash
docker compose up -d vault vault-init
docker compose logs vault-init   # should end with "Vault is ready."
Warning

The bundled vault-init keeps the unseal key in its volume on the same host. That is fine for evaluation, not for production. Use your own Vault cluster with auto-unseal. See Production.

"Vault read failed" or "Vault write failed"

Cause. OpsNexus can't reach Vault, or its token is missing or lacks permission.

Fix. Check System status, then docker compose logs vault vault-init api. If you use your own Vault, check VAULT_ADDR and VAULT_TOKEN or VAULT_ROLE_ID and VAULT_SECRET_ID.

"Stored value could not be decrypted. Was SECRET_KEY changed?"

Cause. Some values, such as MFA secrets, are encrypted with SECRET_KEY. Changing it makes them unreadable.

Fix. Put the previous SECRET_KEY back. Never change it on a running installation.

Backups and restore

"This backup is from database revision X, but this server runs Y"

Cause. A backup can only be restored on the OpsNexus version that made it.

Fix. Restore it on the matching version, then upgrade. See Backup and restore.

"This backup is encrypted. Enter its passphrase." or "Wrong passphrase, or the file is damaged"

Cause. Backups that include secret values are encrypted with a passphrase.

Fix. Enter the passphrase used when the backup was made. OpsNexus can't recover it.

"Set a backup passphrase to include secret values"

Cause. Secret values are only ever stored in backups encrypted.

Fix. Set a backup passphrase in the backup settings, or make the backup without secret values.

"The backup file is no longer on the server"

Cause. Old backups are pruned to the number you chose to keep.

Fix. Use a newer backup, or upload the file from your S3 or Azure Blob copy.

Disk fills up

The large items are backups (backups volume), build artifacts (artifacts volume) and the database. To limit them:

  • Lower the number of backups kept (14 by default), and send copies to S3 or Azure Blob.
  • Lower ARTIFACT_MAX_MB (per file, default 100) and ARTIFACT_MAX_FILES (per run, default 200). Artifacts are deleted with their build or pipeline.
  • METRIC_RETENTION_DAYS and DOMAIN_CHECK_RETENTION_DAYS (30 each) control metric and check history.

Pipelines and plugins

A plugin step disappeared after saving

Cause. Before the 2026-11-08 release, picking a plugin could reset the step, and the empty step was dropped on save.

Fix. Upgrade. The step header now shows Plugin · <name>, or a warning while no plugin is chosen.

Trivy prints "rego_type_error ... requestedamis"

Cause. Checks downloaded at run time were newer than the Trivy version in the plugin.

Fix. Use the Trivy plugin 1.1.1 or later. It skips the check update and removes a bad bundle left in the cache by older runs.

Infra designer deployments

"A deploy of this stack is already ... Wait for it or cancel it."

Cause. Each stack allows one active deployment at a time.

Fix. Wait for it to finish, or cancel it from the stack's Deploy tab.

"... Add them to the credential's IAM policy, then deploy again."

Cause. Before each deploy, OpsNexus simulates the deploy credential's IAM permissions. Some actions the stack needs are missing.

Fix. Add the missing actions listed in the message. The stack's permission policy endpoint gives a ready policy. See Infra designer.

"Fix the N problem(s) in the design before deploying"

Cause. The stack has blocking validation problems.

Fix. Open the Design tab and fix the problems shown inline. Recommendations don't block a deploy.

The web app

502 Bad Gateway after recreating the api container

Cause. Before 2026-11-11, nginx in the web container kept the old address of api.

Fix. Upgrade. The web container now re-resolves the API's address every 10 seconds. On an older version, restart web.

worker or scheduler shows "unhealthy" but works

Cause. Before 2026-11-09, they inherited the API's health check, which calls port 8000.

Fix. Upgrade and rebuild. The worker now answers a Celery ping, and the scheduler checks that its schedule file is recent.

A page shows an error with "Try again" and "Reload"

Cause. A page crashed in the browser. The rest of the app keeps working.

Fix. Try again or reload. If it keeps happening, note the error text and check the api logs for the same time.