Upgrade the CosmicAC stack

Upgrade a running CosmicAC deployment to a new release tag. One tag covers every image in the Docker Compose stack.

This guide doesn't upgrade the images that jobs run in your Kubernetes cluster. To change those, update their tags under containerImages in the Kubernetes worker's config/common.json.

Prerequisites

Before you start, make sure that you have the following.

A running CosmicAC deployment. See Deploy CosmicAC.
The deployment repository on your host machine.
The release tag that you want to deploy.
A root shell in the deployment directory. The stack writes its configuration and state files as root.

Steps

Back up the current state

Save a snapshot that you can restore if the upgrade fails.

task backup

The command copies each service's configuration and state directories to backups/<timestamp>.

Set the new tag

In .env, set TAG to the new release tag, as in the following line.

TAG=<new-tag>

If .env also sets a tag for one service, such as APPNODE_TAG, that tag overrides TAG for that service.

Run the update

Run the following command.

task update

The command pulls the new images, adds new configuration files and keys, and recreates the services at the new tag. Your existing configuration values and state remain unchanged.

Verify the deployment

Check that every service is running.

task ps

For the full health checks, see Verify the deployment.

Upgrade one service

To upgrade one service instead of the whole stack, set SERVICES to the service name.

task update SERVICES="cosmicac-app-node"

When you upgrade cosmicac-app-node, the command also recreates cosmicac-ui and caddy.

Upgrade without editing `.env`

To use a tag without editing .env, pass the tag on the command line. A command-line tag also overrides the tags that .env sets for individual services.

task update TAG=<new-tag>

CosmicAC doesn't save a command-line tag to .env. A later command that doesn't pass TAG, such as task restart, uses the tag in .env.

Change a configuration value

task update sets each new configuration key to the release default. To use a different value, do the following.

Find the key in the service's config/common.json.
Set the key in .env. The following example sets pagination.limit to 100 for cosmicac-app-node.
```
APPNODE_COMMON_CONFIG__pagination__limit=100
```
Apply the change.
```
task apply-app-node-common-config
```

Restart the service.

task restart SERVICES="cosmicac-app-node"

For each service's prefix and apply command, see Task deployment commands.

Roll back

To roll back to the previous release, do the following. A rollback keeps your configuration and state.

Back up the current state.
```
task backup
```
Deploy the previous tag.
```
task update TAG=<previous-tag>
```

Help and troubleshooting

A worker doesn't connect after the upgrade

Refresh the shared runtime keys.
```
task wire
```

Recreate the worker so that it reads the new keys.

task restart SERVICES="cosmicac-wrk-server-k8s-nvidia"