Deployment Troubleshooting

Most deployment problems show up either while a CLI command runs or as a Runtime that never becomes ready. This page works through both: collect the diagnostics first, then jump to the matching symptom.

Everything below runs on the host that holds the deployment, inside the deployment directory (/opt/soln-tenant-runtime on a server, $HOME/soln-tenant-runtime on this computer).

Step 1: Collect the Diagnostics

cd /opt/soln-tenant-runtime
soln status --dir .

• Prints Release:, Runtime: ready|not ready, Tunnel: …, and Operation: when a lifecycle operation is still pending
• Add --json for the full record: container states, the tunnel state, the update check, and deploymentId
• DEPLOYMENT_NOT_INSTALLED — this directory does not hold a deployment; check the path

soln doctor --dir .

• Shows one line per check: docker, compose, buildx, cosign, files, environment, tunnel, storage-security, disk, compose-config
• Every failure names the variable, file, or tool it needs; fix the ✗ lines before looking further

docker compose ps

• Lists every service: tenant-runtime-service, tenant-runtime-web, tenant-workflow-worker, reverse-proxy, and — in single-machine mode — postgres, temporal, minio, document-parser
• State should be running (healthy where a health check exists) for the services in your mode

docker compose logs --tail=200 tenant-runtime-service tenant-workflow-worker

• The Runtime Service log holds the heartbeat and Lease messages; the Workflow Worker log holds Temporal connection problems
• Add reverse-proxy to see TLS or upstream errors, or tenant-database-migrator for migration output

docker compose exec tenant-runtime-service cat /tmp/soln-agent-health.json

• The Runtime writes its own state there: businessRuntimeReadyAt, lunaTemporalReadyAt, lastHeartbeatAt, lastHeartbeatError, runtimeLeaseExpiresAt, stoppingAt, and the last release notice
• A missing lastHeartbeatAt or a non-empty lastHeartbeatError explains an offline Runtime faster than reading logs

Step 2: Match the Symptom

What you see	Start with
soln install or soln upgrade exits with an error code	Installation and Upgrade Errors
soln status prints Runtime: not ready	The Runtime Is Not Ready
The Control Plane shows the Runtime offline, but /health/ready answers 200	The Runtime Is Offline in the Control Plane
A lifecycle operation is stuck (Operation: in the status)	Upgrade or Rollback Is Stuck
The browser cannot open the Runtime Origin	The Browser Cannot Open the Address
Tunnel state is not ready	The Tunnel Is Not Ready
Everything worked before, then the host rebooted	Host-Level Problems

Installation and Upgrade Errors

Code	Meaning	What to do
DEPLOYMENT_ALREADY_INSTALLED	the directory already holds a deployment	use it as it is, or install into an empty directory; never install over a deployment
DEPLOYMENT_ENV_INCOMPLETE	.env still misses required or placeholder values	the message lists the variables; see Environment Variables
DEPLOYMENT_ENV_NOT_FOUND	.env does not exist	run the command inside the deployment directory, so .env is created from .env.example
LOCAL_ENCRYPTED_STORAGE_CONFIRMATION_REQUIRED	a this-computer install without --confirm-encrypted-storage	enable FileVault, BitLocker, or LUKS, then add the flag
LOCAL_RUNTIME_ORIGIN_INVALID	the local Origin is not https://*.localhost:8443	re-run soln prepare --deployment-target local and use the generated address
DEPLOYMENT_BINDING_MISMATCH	tenant, Runtime, or agent differ from the recorded state	copy the command from the Control Plane again instead of reusing an old one
DEPLOYMENT_DISK_INSUFFICIENT	fewer than 10 GiB free	free space; an upgrade keeps two releases plus a database backup at once
DEPLOYMENT_COMMAND_NOT_FOUND	Docker, Compose, Buildx, or Cosign is missing	install it as described in Server Environment
DEPLOYMENT_COMMAND_TIMEOUT	a Compose command did not finish in time	check docker compose ps for a stuck container, then run the command again
REGISTRY_ACCESS_TOKEN_FAILED	the Control Plane did not issue a Registry credential	check outbound HTTPS to the Control Plane, then verify the Agent secret is the current one
RELEASE_MANIFEST_SIGNATURE_INVALID	the release signature did not verify	do not bypass it; the release is not authorized for this tenant — ask your administrator
MISSING_ARGUMENT	a required option is missing from the pasted command	copy the whole command block again
DEPLOYMENT_MODE_INVALID / DEPLOYMENT_TARGET_INVALID	.env says something other than singleMachine or externalServices, or local and server	re-run soln prepare with the matching flags
DEPLOYMENT_SETUP_FILE_MISMATCH	the mode in .env and in the installer disagree	keep both on the same value

When an installation stops in the middle, the database has not been migrated yet: fix the reported cause and run the same command again. soln install is not resumable after the first successful installation — use soln upgrade from then on.

The Runtime Is Not Ready

/health/ready answers 200 only when the Runtime is initialized, holds a valid Lease, and is not in maintenance mode. Check the three conditions in order:

curl -fsS https://runtime.customer.example.com/health/live
curl -fsS https://runtime.customer.example.com/health/ready

• /health/live returns {"status":"ok"} while the process is up; a connection refused means the container or the reverse proxy is down
• /health/ready returning an error body names the condition: TENANT_RUNTIME_INITIALIZING, TENANT_RUNTIME_MAINTENANCE_MODE, or a Lease error

Read the Runtime's own state file next:

docker compose exec tenant-runtime-service cat /tmp/soln-agent-health.json

Field	Meaning
businessRuntimeReadyAt	the business runtime finished initializing; missing means the service is still starting or database access fails
lunaTemporalReadyAt	the Assistant task queue is connected to Temporal; missing means TEMPORAL_ADDRESS or the namespace is wrong
lastHeartbeatAt	the last successful heartbeat to the Control Plane
lastHeartbeatError	why the last heartbeat failed; a Lease or origin problem shows up here
runtimeLeaseExpiresAt	when the current Lease expires; an expired value keeps /health/ready failing
stoppingAt	set while the process is shutting down

What to do:

• TENANT_RUNTIME_INITIALIZING for more than a few minutes: check database connectivity (DATABASE_URL) and docker compose logs --tail=200 tenant-runtime-service
• lunaTemporalReadyAt missing: verify TEMPORAL_ADDRESS, the namespace, and — with TLS — that the CA and client certificates are mounted under /run/secrets with mode 0600
• runtimeLeaseExpiresAt in the past: continue with the next section
• Wait 20 to 30 seconds after a fix and refresh the Control Plane; the status there follows the heartbeat

The Runtime Is Offline in the Control Plane

The heartbeat is rejected with a typed error. These are the usual causes:

Cause	How to confirm	What to do
RUNTIME_ORIGIN does not match the registration (AGENT_RUNTIME_ORIGIN_MISMATCH)	compare RUNTIME_ORIGIN in .env with the origin shown by sl deployment status --json	make them identical, then re-run soln install/soln upgrade so the value reaches the container
the Agent secret was rotated or is stale	lastHeartbeatError mentions the agent or secret	plan a replacement with sl deployment register --force, regenerate configuration, and reconnect Runtime; do not rotate for ordinary check failures
the release is not authorized for this tenant (AGENT_RUNTIME_RELEASE_NOT_AUTHORIZED)	the Runtime log names the release version	ask your administrator to publish or authorize that release
the release is below the enforced minimum (AGENT_RUNTIME_RELEASE_UNSUPPORTED)	soln status prints Update required	upgrade to the release the Control Plane enforces
the Runtime is not registered (AGENT_RUNTIME_RUNTIME_NOT_REGISTERED)	the registration ID is unknown to the Control Plane	check sl deployment status --json, then register with sl deployment register and prepare installation
the host clock drifted	date -u differs from real time by minutes	enable time synchronization (timedatectl status) and restart the deployment
the host cannot reach CP_URL	curl -I "$CP_URL" from the host	open outbound HTTPS to the Control Plane and to the image Registry

Never work around a Lease failure by editing .soln/deployment.json, disabling checks, or copying state between hosts. Fix the cause, then run soln install, soln upgrade, or soln rollback.

Upgrade or Rollback Is Stuck

soln status --dir .

• Operation: upgrade -> <version> or Operation: rollback -> <version> means an earlier lifecycle command did not finish. The Runtime stays in recovery mode until the operation is cleared
• The pending operation is also recorded in .soln/deployment.json (operation.phase, startedAt) and the pre-migration backup sits in .soln/backups

soln rollback --dir .

• Restores the previous release and its database backup, then clears the operation. This is the supported recovery; DEPLOYMENT_RECOVERY_REQUIRED from soln upgrade means the same thing
• DEPLOYMENT_ROLLBACK_UNAVAILABLE — there is no previous release with a backup. Check whether an earlier rollback already consumed it, then reinstall only if you accept the data loss

If the migrator itself failed, its output is kept:

docker compose run --rm tenant-database-migrator

• Success: it prints the applied migrations and exits 0
• A failure here is a database problem, not a release problem: check the migration name in the output and the database permissions

The Browser Cannot Open the Address

What you see	What to do
nothing loads, connection refused	check docker compose ps for reverse-proxy and tenant-runtime-service; start them with docker compose up -d
502 Bad Gateway from the reverse proxy	the upstream Runtime is not ready — work through The Runtime Is Not Ready
DNS_PROBE_FINISHED_NXDOMAIN on a server deployment	create the DNS record for the Runtime Origin and wait for it to propagate
SSL certificate problem or a certificate warning	use a certificate that covers the exact Runtime Origin hostname, and import the local root certificate on Linux as described in Linux
the page loads but requests fail with 401 or 403	sign in again on the Runtime Origin; the session cookie belongs to that exact host
the port is not reachable from outside	open the port in the host firewall or security group; the reverse proxy binds loopback and needs your TLS terminator or tunnel in front of it

On a this-computer deployment the address is https://soln-<tenant-subdomain>.localhost:8443; only that machine can reach it, and 8443 must be free. Another process on 8443 makes the reverse proxy fail with port is already allocated.

The Tunnel Is Not Ready

soln status --dir .
docker compose logs --tail=100 cloudflared

• Tunnel: not ready with TUNNEL_CREDENTIALS_INVALID — the credentials JSON is missing or malformed; place the Named Tunnel credentials at secrets/cloudflared-credentials.json with mode 0600
• TUNNEL_ID_INVALID — CLOUDFLARE_TUNNEL_ID is not a UUID; copy it from the Cloudflare dashboard
• TUNNEL_SERVICE_MISSING — the Compose file has no cloudflared service; re-run soln prepare to regenerate it
• Public traffic failing while /health/ready answers locally is expected when the connector is down: the Control Plane never relays business traffic through itself

Host-Level Problems

df -h /opt/soln-tenant-runtime
timedatectl status
systemctl is-active docker

• Less than 10 GiB free: images, backups, and the coexistence window during an upgrade all live there. Remove old backups that do not belong to the running release
• System clock synchronized: no: fix time synchronization; Lease validation and TLS handshakes both fail on clock skew
• Docker not active: start the daemon. Containers stop when the host reboots unless Docker starts on boot
• Docker Hub returns i/o timeout while pulling public dependency images: check whether the Docker daemon can reach registry-1.docker.io and auth.docker.io. If using a cloud-provider image accelerator, use the address issued to your account. Keep the original setup file and retry sl install after restoring access; do not register again
• After a host reboot, confirm findmnt /srv/soln/data shows the encrypted volume mounted before starting Docker. Then check the containers and run soln status --dir .; do not let data be written into the unmounted directory

What to Collect Before Asking for Help

1. the exact command you ran and its full output, including the error code
2. soln status --dir . --json
3. soln doctor --dir .
4. docker compose ps
5. /tmp/soln-agent-health.json from inside tenant-runtime-service
6. the tail of the Runtime Service and Workflow Worker logs
7. the tenant ID, Runtime ID, deployment target, deployment mode, and release version

Never include .env, the Agent secret, the database password, or a database backup in a report. .soln/deployment.json is safe to share: it contains no secrets.

Rules

• Do not edit .soln/deployment.json or .soln/docker-compose.release.yml; they record the installed release, the image digests, and any pending operation
• Do not bypass signature or Lease validation, and do not hand-edit .env values that the Control Plane also records
• Reuse the same deployment directory; never install a second deployment over an existing one
• Page and API errors carry code, reason, and domain; see Error Codes for what they mean