Skip to main content

Server Upgrade and Rollback

Every upgrade moves the four products and the database schema forward together and backs up the tenant database first; rollback restores the previous release and that backup.

Step 1: Check What Is Installed​

soln status --dir /opt/soln-tenant-runtime
  • Success: the installed release version, the container state of the four products, and ready
  • DEPLOYMENT_NOT_INSTALLED — this directory is not a deployment directory; check --dir
soln upgrade --check --dir /opt/soln-tenant-runtime
  • Soln <version> is up to date. — nothing to do
  • Update available: <current> -> <next> — continue with step 2
  • Update required: <current> -> <next> — the Control Plane enforced this release; upgrade to keep the Runtime connected

Step 2: Check Storage and Capacity​

sl storage status --dir /opt/soln-tenant-runtime --json
sl upgrade --dir /opt/soln-tenant-runtime --check --json

Verify the data mount, its encryption, and free space on each filesystem. The CLI budgets new images, database backups, and migration writes separately, combining allocations on shared filesystems. Insufficient space stops execution before services or migrations are changed. Configure the actual containerd image-store path when enabled; see Storage and Capacity Management.

Backups are encrypted and stored in the deployment’s .soln/backups, created by the first upgrade. They are not automatically placed on the encrypted tenant volume and do not include MinIO attachments. Establish independent backups and recovery procedures.

Step 3: Upgrade​

soln upgrade --dir /opt/soln-tenant-runtime

Add --yes for a non-interactive shell, and --release <version> only when you must install one exact release. In order, the CLI:

  1. checks the environment and encrypted data mount, and validates deployment bindings
  2. obtains and verifies release information and temporary Registry credentials
  3. checks backup, Docker image, and tenant data capacity budgets
  4. verifies image signatures and pulls the new images
  5. records the operation, stops business services, and writes the encrypted database backup
  6. records the migration phase and runs the Database Migrator
  7. starts the new services; direct CLI upgrades wait for health, while the tenant page reconnects and confirms the target release
  8. saves the new deployment state and previous release for rollback
  • Success: Soln upgraded to <version>. Backup: <deployment>/.soln/backups/postgres-tenant-<timestamp>.sql.gz.enc
  • DEPLOYMENT_ALREADY_CURRENT — that release is already installed
  • DEPLOYMENT_RECOVERY_REQUIRED — a previous upgrade did not finish; run rollback first (step 5)
  • A failed signature check or doctor check stops the upgrade before any migration runs

Step 4: Verify the New Version​

soln status --dir /opt/soln-tenant-runtime
  • Success: the new release version, and ready
curl -fsS -o /dev/null -w '%{http_code}\n' \
  https://runtime.customer.example.com/health/ready
  • Success: 200
  • TENANT_RUNTIME_NOT_READY or a non-200 answer: inspect the heartbeat and Lease error in Deployment Troubleshooting before touching anything else

Step 5: Roll Back​

Data impact: a normal rollback restores the pre-upgrade PostgreSQL backup. Database records created, changed, or deleted after that backup are not retained. It does not restore object storage files or rewind Temporal state, so you may need to reconcile those services with the restored database. Stop business writes and preserve current database, object storage, and external service backups before approving a rollback.

If the upgrade stopped in the backup_pending phase, the CLI recovery branch restarts the original release without restoring the database. Follow the CLI's operation prompt; do not edit the unfinished-operation marker.

soln rollback --dir /opt/soln-tenant-runtime

Use it when a migration or a health check failed, or when the new release must be undone. The CLI:

  1. resolves the previous release and its backup (DEPLOYMENT_ROLLBACK_UNAVAILABLE when there is none)
  2. stops the business services
  3. restores the PostgreSQL backup
  4. runs the Database Migrator for the previous release
  5. starts the previous release and waits for /health/ready
  6. makes the previous release current and drops the rollback target
  • Success: Soln rolled back to <version>.
  • In external services mode the backup is restored into the tenant's own PostgreSQL, so make sure the account may still drop and recreate the schema objects the previous release needs
  • Only one rollback target is kept: after a rollback, the next upgrade takes a fresh backup

Operational Rules​

  • Do not edit .soln/deployment.json or .soln/docker-compose.release.yml; they record the installed release, the digests, and any pending operation
  • Do not bypass Lease validation when /health/ready fails; read the heartbeat and the Lease error instead
  • Keep free capacity on every filesystem according to the upgrade budget and business growth; never delete the backup that belongs to the running release
  • Reuse the same deployment directory; never install over an existing deployment (DEPLOYMENT_ALREADY_INSTALLED)
  • Rotate the Agent secret only when you intend to reconnect the Runtime, then run a lifecycle operation so the new value reaches the container

See also Deployment Troubleshooting.

To change an installed Runtime’s public address, see Change the Tenant Domain.