Containers

The Ansible container is published as two image variants, each in its own image repository under the ghcr.io/sapphire-health/ansible-epic/ namespace. A single image serves both AWS and Azure: it ships both clouds' CLIs, Python SDKs, inventories, and Ansible configs, and cloud selection happens at runtime via ANSIBLE_CONFIG — the repo-root ansible.cfg targets AWS by default; set ANSIBLE_CONFIG=./ansible-azure.cfg to target Azure.

Image	VS Code pre-installed	Registry
`base`	No	`ghcr.io/sapphire-health/ansible-epic/base`
`web`	Yes	`ghcr.io/sapphire-health/ansible-epic/web`

Every build is published with an immutable CalVer version tag (e.g. 2026.07.18) plus the moving latest tag — see Versioning. The examples below use latest; substitute a version tag to pin a specific build.

The web variant extends the base image with a pre-installed VS Code Server and the Ansible VS Code extension. Use it in environments with restricted outbound internet access where VS Code cannot be downloaded at runtime. See VS Code Web Server.

When running as a Coder workspace, the image is chosen by the template's Container Image parameter — it defaults to the current production image (base), with alternate images selectable per workspace for testing candidates before a rollout. The Coder agent binary is downloaded at container startup directly from the Coder server, so the image never needs to be rebuilt when the Coder server is upgraded.

The container supports several runtime configurations. It can be accessed over SSH, via a VS Code tunnel, or as a Coder workspace, and optionally run a GitHub Actions self-hosted runner in the background concurrently with any access method.

Environment Variables

Variable	Required	Default	Description
`AUTHORIZED_KEYS`	No	—	SSH public key(s). When set, starts `sshd` and accepts connections on port 22.
`ANSIBLE_REPO_URL`	No	—	HTTPS URL of the Ansible repository to clone on first start.
`GITHUB_TOKEN`	No	—	GitHub PAT used to clone a private repository. See PAT Requirements.
`TUNNEL_NAME`	No	`ansible_container`	VS Code tunnel display name. Used when `AUTHORIZED_KEYS` is not set.
`GITHUB_RUNNER_URL`	No	—	Repository or organization URL to register the GitHub Actions runner against, e.g. `https://github.com/org/repo`.
`GITHUB_RUNNER_PAT`	No	—	Long-lived GitHub PAT. The container exchanges this for a fresh registration token at startup. Recommended for automated deployments.
`GITHUB_RUNNER_TOKEN`	No	—	Short-lived GitHub Actions runner registration token. Use for manual or one-off deployments.
`GITHUB_RUNNER_NAME`	No	ECS task ID or hostname	Runner display name shown in GitHub. When running in ECS the task ID is used automatically so each container gets a unique name. On other platforms the hostname is used.
`GITHUB_RUNNER_LABELS`	No	`self-hosted,linux`	Comma-separated labels assigned to the runner.
`RUNNER_DEBUG`	No	—	When set, runner output goes to stdout. By default runner output is written to `~/runner.log`.
`CODER_AGENT_TOKEN`	No	—	Coder workspace agent token. When set, starts the Coder agent instead of SSH or VS Code tunnel. See Coder Workspace.
`CODER_AGENT_URL`	No	—	URL of the Coder server, e.g. `https://coder.example.com`. Required when `CODER_AGENT_TOKEN` is set.
`VSCODE_WEB`	No	—	When set to any non-empty value, starts the VS Code web server on startup. Intended for use with the `web` image variant. See VS Code Web Server.
`VSCODE_WEB_PORT`	No	`8080`	Port the VS Code web server listens on. Only used when `VSCODE_WEB` is set.
`USER_SETUP`	No	—	When set to any non-empty value, enables per-user setup for standalone deployments: venv auto-activation, persona selection, and per-user persistence. See Per-User Setup and Persistence.
`USER_SETUP_ROOT`	No	`/home/ansible/source`	Root directory personas persist under. Mount the deployment's persistent storage here. Only used when `USER_SETUP` is set.
`USER_SETUP_USER`	No	—	Preset persona name: skips the interactive picker and applies persistence at container start, before any server launches. Only used when `USER_SETUP` is set.

Access Modes

SSH

When AUTHORIZED_KEYS is set the container generates SSH host keys, writes the provided key(s) to ~/.ssh/authorized_keys, and starts sshd on port 22.

BashPowerShell

docker run -d \
  -p 22:22 \
  -e AUTHORIZED_KEYS="ssh-ed25519 AAAA... user@host" \
  ghcr.io/sapphire-health/ansible-epic/base:latest

docker run -d `
  -p 22:22 `
  -e AUTHORIZED_KEYS="ssh-ed25519 AAAA... user@host" `
  ghcr.io/sapphire-health/ansible-epic/base:latest

VS Code Tunnel

When AUTHORIZED_KEYS is not set the container launches a VS Code tunnel. A device login code is printed to the container logs each time the container starts until Microsoft credentials are cached inside the container.

BashPowerShell

docker run -d \
  -e TUNNEL_NAME="my-ansible-tunnel" \
  ghcr.io/sapphire-health/ansible-epic/base:latest

docker run -d `
  -e TUNNEL_NAME="my-ansible-tunnel" `
  ghcr.io/sapphire-health/ansible-epic/base:latest

Note

Check the container logs for a device login URL and code. Navigate to https://login.microsoft.com/device and enter the code to authenticate with your Microsoft account. The code refreshes every 15 minutes until authentication completes.

VS Code Web Server

When VSCODE_WEB is set, the container starts code serve-web on startup without requiring a Microsoft login or outbound tunnel. This mode is intended for the web image variant, which pre-installs the VS Code Server and the Ansible extension so no downloads are needed at runtime.

The server binds to 0.0.0.0 when running standalone, making it accessible on the container's published port. When CODER_AGENT_TOKEN is also set the server binds to 127.0.0.1 instead, and the Coder agent proxies access through the Coder UI.

BashPowerShell

docker run -d \
  -p 8080:8080 \
  -e VSCODE_WEB=1 \
  ghcr.io/sapphire-health/ansible-epic/web:latest

docker run -d `
  -p 8080:8080 `
  -e VSCODE_WEB=1 `
  ghcr.io/sapphire-health/ansible-epic/web:latest

To use a custom port:

BashPowerShell

docker run -d \
  -p 8080:8080 \
  -e VSCODE_WEB=1 \
  -e VSCODE_WEB_PORT=8080 \
  ghcr.io/sapphire-health/ansible-epic/web:latest

docker run -d `
  -p 8080:8080 `
  -e VSCODE_WEB=1 `
  -e VSCODE_WEB_PORT=8080 `
  ghcr.io/sapphire-health/ansible-epic/web:latest

The GitHub Actions runner works alongside the VS Code web server when GITHUB_RUNNER_URL is also set.

Note

The VS Code web server has no authentication when started with --without-connection-token. Ensure the container port is not exposed to untrusted networks, or place a reverse proxy with authentication in front of it.

Coder Workspace

When CODER_AGENT_TOKEN is set the container downloads the Coder agent binary from CODER_AGENT_URL at startup, then starts it. The agent connects back to the Coder server and makes the container available as a workspace. VS Code (via code serve-web) and a terminal are accessible through the Coder UI without any additional configuration. Use the standard base image — no separate Coder-specific image is needed.

BashPowerShell

docker run -d \
  -e CODER_AGENT_TOKEN="<token>" \
  -e CODER_AGENT_URL="https://coder.example.com" \
  ghcr.io/sapphire-health/ansible-epic/base:latest

docker run -d `
  -e CODER_AGENT_TOKEN="<token>" `
  -e CODER_AGENT_URL="https://coder.example.com" `
  ghcr.io/sapphire-health/ansible-epic/base:latest

The GitHub Actions runner works alongside the Coder agent when GITHUB_RUNNER_URL is also set:

BashPowerShell

docker run -d \
  -e CODER_AGENT_TOKEN="<token>" \
  -e CODER_AGENT_URL="https://coder.example.com" \
  -e GITHUB_RUNNER_URL="https://github.com/Sapphire-Health/ansible-epic" \
  -e GITHUB_RUNNER_PAT="<pat>" \
  ghcr.io/sapphire-health/ansible-epic/base:latest

docker run -d `
  -e CODER_AGENT_TOKEN="<token>" `
  -e CODER_AGENT_URL="https://coder.example.com" `
  -e GITHUB_RUNNER_URL="https://github.com/Sapphire-Health/ansible-epic" `
  -e GITHUB_RUNNER_PAT="<pat>" `
  ghcr.io/sapphire-health/ansible-epic/base:latest

Obtaining the Agent Token

The agent token is generated when a Coder workspace is created from a template. To retrieve it for manual use:

BashPowerShell

coder state pull <workspace-name> | python3 -c "
import sys, json
state = json.load(sys.stdin)
print(state['outputs']['agent_token']['value'])
"

coder state pull <workspace-name> | python3 -c "import sys, json; state = json.load(sys.stdin); print(state['outputs']['agent_token']['value'])"

Coder Template

The Coder workspace templates for this container are in the ansible-epic repository under coder/aws/dev-container/main.tf (general-purpose development container) and coder/aws/gh-runner/main.tf (development container with a GitHub Actions self-hosted runner). Both provision an AWS ECS Fargate task running the base image (pulled through the ECR pull-through cache), backed by EFS for persistent storage, and wire up the Coder agent, VS Code web access, and optional tooling automatically.

This template is deployed to the Sapphire Health Coder instance at https://coder.sapphirehealth.org. Access requires a Sapphire Health account — sign in using Microsoft Entra ID (single sign-on) from the Coder login page.

Per-User Setup and Persistence (Standalone Deployments)

Coder workspaces get venv activation, git identity, and per-user persistence from the workspace templates. Standalone deployments (SSH, VS Code tunnel, VS Code web) can opt into the same conveniences via the image's baked-in user-setup feature, gated by the USER_SETUP environment variable — when it is unset, container behavior is completely unchanged.

With USER_SETUP set:

Interactive shells activate the Ansible venv automatically and flush shell history after every command. Non-interactive shells (scp, ssh host command, Ansible connections) are always silent no-ops.
Persona selection runs in the first terminal on a real TTY: it lists existing persona directories under USER_SETUP_ROOT, lets the user pick one by number, or creates a new one from a typed name. If the terminal opens inside a persona's directory — e.g. VS Code reconnecting to a recent workspace whose folder lives under <root>/<name>/ — the persona is inferred from the path and the picker is skipped. Setting USER_SETUP_USER skips the picker and applies that persona at container start, before the VS Code server or sshd launches — the same links-before-server sequencing Coder workspaces get.
Persistence: ~/.ssh, ~/.claude, ~/.claude.json, ~/.bash_history, and ~/.gitconfig are symlinked into <root>/<name>/.persist/, and <root>/<name>/repos/ is created — the same layout the Coder templates use on EFS / Azure Files.
GitHub auth persists per persona: gh auth login stores its token in ~/.config/gh, and credential.helper store users get ~/.git-credentials — both are linked into the persist store, so a fresh container on the same storage is still logged in. (Unlike the tunnel token, these are only ever created inside a shell — after persona selection — so persona scope is correct.) SSH-based git access persists via ~/.ssh as before.
VS Code state: extensions and user settings (~/.vscode-server/extensions, ~/.vscode-server/data/User) persist per persona; server binaries stay ephemeral, so corrupt or version-mismatched server state never outlives a container. One set of links covers all three modes: Remote-SSH uses these directories by default, startup.sh pins tunnel mode to them, and the code-server that code serve-web downloads defaults to the same layout (verified — only its server builds live elsewhere, under ~/.vscode/cli/serve-web/, which stays ephemeral). Extensions baked into the web image seed each new persona's store, so first-run personas start with the pre-installed set and additions persist from there. The tunnel's Microsoft device login is deliberately not persisted: tunnel identity is container-scoped and established at startup, before any persona is known — expect to re-authenticate once per container start in tunnel mode.
Git identity is prompted for once per persona (name and email, on a TTY only) and stored in the persisted .gitconfig, so it travels with the persona across container recreations.
Switching: run user-setup (on PATH) in any terminal to switch personas; open new terminals afterwards, and if VS Code is connected, reload the window so it sees the new persona.

Mount real storage at the persistence root

The default root /home/ansible/source exists inside the image, so the feature works even without a mounted volume — but personas then live only as long as the container. For durable persistence, mount the deployment's persistent storage (EFS, Azure Files, or a bind mount) at USER_SETUP_ROOT, as the documented customer deployments already do.

Azure Files (SMB) mounts

The design follows the SMB-safe patterns proven by the Coder Azure template: symlinks live in the local home pointing into the mount (never created on SMB), VS Code server binaries and runtime files stay local, and only the SMB-safe data/User subset persists. Two SMB accommodations are built in: the image's sshd runs with StrictModes no (SMB reports permissive modes on authorized_keys, which would otherwise break SSH-mode logins after persona linking), and personas get git safe.directory = * (SMB doesn't preserve ownership, which git otherwise refuses). For outbound SSH keys, prefer the agent-based flow in SSH Authentication — the ssh client and ssh-add enforce key-file permissions that SMB cannot express. To validate against a real SMB mount (docker volumes can't reproduce its semantics), use the azure-standalone-test harness template — see Testing.

A third accommodation is baked in: the image sets ANSIBLE_CONFIG=./ansible.cfg (and startup.sh propagates it to SSH sessions via /etc/profile.d, since sshd scrubs the environment). SMB reports every directory as world-writable (ACI hardcodes 0777 mount options), and Ansible refuses to auto-load ansible.cfg from a world-writable working directory — without the explicit path, repos on the mount emit [WARNING]: Ansible is being run in a world writable directory ... ignoring it as an ansible.cfg source and silently run without roles_path, inventory, or vault configuration. An env-specified path is Ansible's documented escape hatch for permission-emulating filesystems; the relative path resolves against the working directory each run (same semantics as auto-discovery), a missing file degrades gracefully to config file = None, and a deployment-level ANSIBLE_CONFIG (e.g. the Coder Azure template's ./ansible-azure.cfg) overrides the image default. Note the profile.d propagation covers login shells — a bare non-interactive ssh host 'ansible ...' doesn't source it (the same limitation as the USER_SETUP env vars). Hosts that mount SMB themselves (fstab/cifs rather than ACI) can also mount with dir_mode=0755,file_mode=0644, which eliminates the entire fixed-0777 category. Images published before 2026-07-24 need ANSIBLE_CONFIG=./ansible.cfg set in the deployment env instead.

Behavior details

A persona is a directory containing .persist. The root routinely holds non-persona content — the shared files/ directory, repo clones, and the image's baked repo copy (docker seeds fresh named volumes from the image) — which is never listed and cannot be converted by typing its name.
Deploy-time AUTHORIZED_KEYS always remain valid: after linking a persona's persisted .ssh (whose stored authorized_keys may predate a key rotation), the container's deploy keys are merged back in.
Selecting or switching personas while VS Code is already connected relinks correctly on disk, but the running session keeps watching the old locations until the window is reloaded.

GitHub Actions Runner

Setting GITHUB_RUNNER_URL and either GITHUB_RUNNER_PAT or GITHUB_RUNNER_TOKEN causes the container to configure and start a GitHub Actions self-hosted runner in the background before starting SSH or the VS Code tunnel. The runner and the access method run concurrently within the same container.

Automated Deployment (Terraform / ECS)

For automated deployments, supply a long-lived GitHub PAT via GITHUB_RUNNER_PAT. The container calls the GitHub API at startup to generate a fresh registration token each time the container starts, so no manual token rotation is required.

Store the PAT as an AWS Secrets Manager secret and inject it into the ECS task definition using the secrets field. ECS fetches the secret value and injects it as an environment variable before the container starts.

resource "aws_secretsmanager_secret" "github_runner_pat" {
  name = "ansible/github-runner-pat"
}

resource "aws_secretsmanager_secret_version" "github_runner_pat" {
  secret_id     = aws_secretsmanager_secret.github_runner_pat.id
  secret_string = var.github_runner_pat
}

Note

Store the secret as a plain string (the raw token value), not as a JSON object. The valueFrom field in the task definition references the secret ARN directly, which instructs ECS to inject the entire secret value as the environment variable. If the secret is stored as JSON (e.g. {"GITHUB_RUNNER_PAT":"github_pat_..."}), ECS will inject the JSON string rather than the token.

Reference the secret in the ECS task definition alongside the other runner environment variables:

resource "aws_ecs_task_definition" "ansible" {
  ...
  container_definitions = jsonencode([{
    ...
    secrets = [
      {
        name      = "GITHUB_RUNNER_PAT"
        valueFrom = aws_secretsmanager_secret.github_runner_pat.arn
      }
    ]
    environment = [
      { name = "GITHUB_RUNNER_URL",    value = "https://github.com/Sapphire-Health/ansible-epic" },
      { name = "GITHUB_RUNNER_NAME",   value = "ansible-container" },
      { name = "GITHUB_RUNNER_LABELS", value = "self-hosted,linux,aws" }
    ]
  }])
}

Note

The ECS task execution role must have secretsmanager:GetSecretValue on the secret ARN. See AWS Container Requirements.

Manual Token

For testing or one-off deployments, supply a short-lived registration token directly via GITHUB_RUNNER_TOKEN:

BashPowerShell

docker run -d \
  -p 22:22 \
  -e AUTHORIZED_KEYS="ssh-ed25519 AAAA... user@host" \
  -e GITHUB_RUNNER_URL="https://github.com/Sapphire-Health/ansible-epic" \
  -e GITHUB_RUNNER_TOKEN="<registration-token>" \
  ghcr.io/sapphire-health/ansible-epic/base:latest

docker run -d `
  -p 22:22 `
  -e AUTHORIZED_KEYS="ssh-ed25519 AAAA... user@host" `
  -e GITHUB_RUNNER_URL="https://github.com/Sapphire-Health/ansible-epic" `
  -e GITHUB_RUNNER_TOKEN="<registration-token>" `
  ghcr.io/sapphire-health/ansible-epic/base:latest

To obtain a registration token from GitHub:

Navigate to the repository or organization Settings.
Select Actions → Runners → New self-hosted runner.
Copy the token shown in the configuration step, e.g. --token ABCD1234....

Note

Runner registration tokens expire after one hour. Start the container before the token expires.

Runner Logs

By default the runner writes its output to ~/runner.log inside the container rather than stdout, keeping container logs uncluttered. To follow the log over SSH or a VS Code terminal:

BashPowerShell

tail -f ~/runner.log

Get-Content ~/runner.log -Wait

To send runner output to stdout instead (visible via docker logs), set RUNNER_DEBUG to any non-empty value:

BashPowerShell

docker run -d \
  -e GITHUB_RUNNER_URL="..." \
  -e GITHUB_RUNNER_PAT="..." \
  -e RUNNER_DEBUG="1" \
  ghcr.io/sapphire-health/ansible-epic/base:latest

docker run -d `
  -e GITHUB_RUNNER_URL="..." `
  -e GITHUB_RUNNER_PAT="..." `
  -e RUNNER_DEBUG="1" `
  ghcr.io/sapphire-health/ansible-epic/base:latest

Deregistration on Shutdown

When GITHUB_RUNNER_PAT is set, the container automatically deregisters the runner from GitHub when it stops. This keeps the runner list clean when containers are frequently rebuilt or when multiple containers are running simultaneously — each container registers with a unique name and removes itself on exit.

Note

Automatic deregistration requires GITHUB_RUNNER_PAT. If only GITHUB_RUNNER_TOKEN was supplied, the runner will appear offline in GitHub and be removed automatically after 30 days.

GitHub PAT Requirements

Repository Clone Token (`GITHUB_TOKEN`)

GITHUB_TOKEN is only required when ANSIBLE_REPO_URL points to a private repository. The token is used exclusively to clone the repository and is removed from the git remote URL immediately after cloning.

PAT Type	Required Permission
Classic	`repo`
Fine-grained	Contents → Read

Runner PAT (`GITHUB_RUNNER_PAT`)

GITHUB_RUNNER_PAT is a long-lived PAT used to generate a fresh runner registration token at container startup. The container determines whether to use the repository or organization API endpoint based on the number of path segments in GITHUB_RUNNER_URL.

PAT Type	Runner Scope	Required Permission
Classic	Repository	`repo`
Classic	Organization	`admin:org`
Fine-grained	Repository	Administration → Write
Fine-grained	Organization	Organization self-hosted runners → Write

Publishing Images

The container images are built locally and pushed to the GitHub Container Registry (ghcr.io). Each image is published as its own package under the ghcr.io/sapphire-health/ansible-epic/ namespace: base and web.

Versioning

Every build of every image is pushed with two tags:

An immutable CalVer version tag — YYYY.MM.DD (e.g. 2026.07.18). For a second build on the same day, append a serial: 2026.07.18.2. A version tag is never overwritten; it always refers to the exact same build.
The moving latest tag — always points at the most recent build of that image.

Runtime consumers (the Coder templates, the ECS task definitions, the ACI container group) track latest, so redeploying picks up the newest build automatically. Version tags exist for provenance and rollback: to roll back, retag a known-good version as latest and push, then redeploy.

Each image also carries OCI metadata labels stamped at build time by the build script:

Label	Value
`org.opencontainers.image.version`	The CalVer version tag
`org.opencontainers.image.revision`	The git commit the image was built from
`org.opencontainers.image.created`	The build timestamp (UTC)
`org.opencontainers.image.source`	The repository URL (also links the package to the repo — see below)

Inspect them on any pulled image with docker image inspect <image> under .Config.Labels.

Migration from earlier layouts

Two earlier layouts are frozen and superseded. Single package (oldest): tags of one package — ghcr.io/sapphire-health/ansible-epic:{aws,azure,aws-web,azure-web} — mirrored by manual push to 271851283454.dkr.ecr.us-west-2.amazonaws.com/ansible-epic and coderdevcontainer.azurecr.io/ansible-epic; the manual mirror flows were later replaced by the automatic caches, and the coderdevcontainer ACR by coderdevcontainerreg. Per-cloud packages: ansible-epic/{aws,azure,aws-web,azure-web}, replaced by the combined base/web images when the AWS and Azure Dockerfiles were merged — a base image serves both clouds. Once nothing references the old names, the old GHCR packages, old ECR repositories (including cached ghcr/sapphire-health/ansible-epic/{aws,aws-web} entries), the old ansible-epic/azure ACR cache rule and repository, and the old ACR registry can be deleted.

Package Visibility and Repository Association

The Dockerfiles set the org.opencontainers.image.source label to this repository's URL. GHCR reads that label on first push and automatically links each package to the Sapphire-Health/ansible-epic repository — no manual association step is needed. Linked packages appear in the repository's Packages sidebar.

Package visibility is a one-time manual setting per package: packages are created private by default. To make one public, open the package page → Package settings → Danger Zone → Change package visibility → Public. Public packages can be pulled by anyone without authentication.

Public packages expose the image contents

These images contain the full ansible-epic repository source (including the encrypted group_vars/all/vault.yml), the private Sapphire Health role repositories installed under ~/.ansible/roles, and internal details such as AWS account IDs and hostnames. Making a package public makes all of that world-readable, and once an image has been pulled it cannot be un-published. Before making a package public, ensure the vault password is long and random (the ciphertext becomes available for offline attack) and treat everything baked into the image as public information.

Prerequisites

Git — to clone or pull the repository
Docker — to build and push images
- On Windows: Docker Desktop must be installed and running in Linux container mode
GitHub account with write access to the Sapphire-Health/ansible-epic repository and the write:packages permission

Authenticate to GHCR

Use the GitHub CLI to authenticate Docker against ghcr.io:

BashPowerShell

gh auth token | docker login ghcr.io -u $(gh api user --jq .login) --password-stdin

gh auth token | docker login ghcr.io -u $(gh api user --jq .login) --password-stdin

If your current gh token does not have write:packages scope, add it and re-authenticate. Use gh auth refresh rather than gh auth login — refresh reliably adds the scope to your existing token, whereas re-running login may not:

BashPowerShell

gh auth refresh -h github.com -s write:packages -s read:packages
gh auth token | docker login ghcr.io -u $(gh api user --jq .login) --password-stdin

gh auth refresh -h github.com -s write:packages -s read:packages
gh auth token | docker login ghcr.io -u $(gh api user --jq .login) --password-stdin

Push fails with permission_denied: The token provided does not match expected scopes

This error usually does not mean your gh token is missing a scope. The more common cause is a broken Docker credential: docker login will silently store an empty password if gh auth token returns nothing at the moment it runs (for example, when GH_TOKEN is unset or hasn't been populated yet). Docker then authenticates to GHCR with a blank token, which the registry reports as a scope mismatch.

This is especially easy to hit inside the Coder dev containers, where GH_TOKEN is populated from Coder's external-auth on shell startup — running the login before that token is fetched poisons the stored credential.

First confirm your gh token is actually healthy — the Token scopes: line should include write:packages:

gh auth status

Then clear the bad credential and log in again, guarding against an empty token so it can't happen silently:

BashPowerShell

docker logout ghcr.io
TOKEN=$(gh auth token)
[ -n "$TOKEN" ] || { echo "no gh token — run: gh auth login"; exit 1; }
echo "$TOKEN" | docker login ghcr.io -u "$(gh api user --jq .login)" --password-stdin

docker logout ghcr.io
$token = gh auth token
if (-not $token) { throw "no gh token — run: gh auth login" }
$token | docker login ghcr.io -u "$(gh api user --jq .login)" --password-stdin

A successful login reports Login Succeeded. If pushes still fail after this, verify your account has write access to the package and — if the org enforces SAML SSO — that your token is authorized for Sapphire-Health.

Authenticate to ECR

Use the AWS CLI to authenticate Docker against ECR. Your AWS credentials must have ecr:GetAuthorizationToken permission. Unlike GHCR (where the public packages pull anonymously), ECR always requires this login — including for pulls through the pull-through cache, which additionally need ecr:BatchImportUpstreamImage and ecr:CreateRepository on the ghcr/* repositories to trigger cache refreshes.

BashPowerShell

aws ecr get-login-password --region us-west-2 | \
  docker login --username AWS --password-stdin \
  271851283454.dkr.ecr.us-west-2.amazonaws.com

aws ecr get-login-password --region us-west-2 | `
  docker login --username AWS --password-stdin `
  271851283454.dkr.ecr.us-west-2.amazonaws.com

ECR tokens expire after 12 hours. Re-run this command if Docker reports an authentication error during a push.

Authenticate to ACR

Authentication is only needed for manual pushes — the ACR cache rule mirrors the base image from GHCR without any push. Use the Azure CLI to authenticate Docker against ACR. Your Azure account must have at least the AcrPush role on the registry.

BashPowerShell

az acr login --name coderdevcontainerreg

az acr login --name coderdevcontainerreg

If you are not already signed in to the Azure CLI, run az login first.

Build the Images

Run from the root of the repository.

Use the build script

scripts/build-images.sh (Bash) and scripts/build-images.ps1 (PowerShell) run the entire build sequence below in one step — both images, version + latest tags, and OCI version labels:

BashPowerShell

./scripts/build-images.sh              # build only
./scripts/build-images.sh -p           # build and push to ghcr.io
./scripts/build-images.sh -v 2026.07.16.2 -p   # same-day rebuild
./scripts/build-images.sh -n           # force --no-cache

./scripts/build-images.ps1                              # build only
./scripts/build-images.ps1 -Push                        # build and push to ghcr.io
./scripts/build-images.ps1 -Version 2026.07.16.2 -Push  # same-day rebuild
./scripts/build-images.ps1 -NoCache                     # force --no-cache

The manual commands below document what the script does.

The build requires a GitHub token to clone private Sapphire Health role repositories during the builder stage. The token is passed as a Docker build secret — it is never baked into any image layer.

Set the token and version before building:

BashPowerShell

export GITHUB_TOKEN=$(gh auth token)
VERSION=$(date +%Y.%m.%d)   # append a serial for a same-day rebuild, e.g. 2026.07.16.2

$env:GITHUB_TOKEN = $(gh auth token)
$VERSION = Get-Date -Format yyyy.MM.dd   # append a serial for a same-day rebuild, e.g. 2026.07.16.2

The token must have at least repo (classic PAT) or Contents → Read (fine-grained PAT) access to the Sapphire Health organization's private repositories.

Tip

Add --no-cache to any build command to force Docker to pull the latest versions of all packages and tools, ignoring the layer cache. This is useful when you want to pick up updated apt packages, a newer VS Code CLI, or a newer GitHub Actions runner without bumping the pinned version arguments. Omitting it (the default) reuses cached layers for a faster incremental build when only source files changed.

Base image:

BashPowerShell

docker build \
  --secret id=github_token,env=GITHUB_TOKEN \
  --build-arg IMAGE_VERSION=$VERSION \
  -f Dockerfile \
  -t ghcr.io/sapphire-health/ansible-epic/base:$VERSION \
  -t ghcr.io/sapphire-health/ansible-epic/base:latest .

docker build `
  --secret id=github_token,env=GITHUB_TOKEN `
  --build-arg IMAGE_VERSION=$VERSION `
  -f Dockerfile `
  -t ghcr.io/sapphire-health/ansible-epic/base:$VERSION `
  -t ghcr.io/sapphire-health/ansible-epic/base:latest .

Web image:

BashPowerShell

docker build -f Dockerfile.Web \
  --build-arg BASE_IMAGE=ghcr.io/sapphire-health/ansible-epic/base:$VERSION \
  --build-arg IMAGE_VERSION=$VERSION \
  -t ghcr.io/sapphire-health/ansible-epic/web:$VERSION \
  -t ghcr.io/sapphire-health/ansible-epic/web:latest .

docker build -f Dockerfile.Web `
  --build-arg BASE_IMAGE=ghcr.io/sapphire-health/ansible-epic/base:$VERSION `
  --build-arg IMAGE_VERSION=$VERSION `
  -t ghcr.io/sapphire-health/ansible-epic/web:$VERSION `
  -t ghcr.io/sapphire-health/ansible-epic/web:latest .

Note

The Dockerfiles also accept IMAGE_REVISION (git commit) and IMAGE_CREATED (UTC timestamp) build args for the corresponding OCI labels. The build script stamps them automatically; for manual builds they are optional and default to unknown.

GitHub Actions pipeline

The Build Container Images workflow runs this entire build-and-push sequence in CI — trigger it from the Actions tab instead of building locally.

Note

The Web image layers on top of the base image via BASE_IMAGE. Build base for the same $VERSION on the same machine first (the build script does this automatically) — Docker then resolves BASE_IMAGE from the local image cache. A push is only required if the web image is built on a different machine than its base.

Note

The base Dockerfile contains an ARG BUILD_DATE line near the top of the final stage. Updating this value forces Docker to invalidate the cache from that layer onward while keeping the expensive builder-stage cache warm. This is a more targeted alternative to --no-cache when you want to pick up updated apt packages or tool downloads (VS Code CLI, GitHub runner, AWS/Azure CLI).

Push the Images

Push both the version tag and latest for each variant. The image layers upload once — the second tag is metadata only.

BashPowerShell

docker push ghcr.io/sapphire-health/ansible-epic/base:$VERSION
docker push ghcr.io/sapphire-health/ansible-epic/base:latest
docker push ghcr.io/sapphire-health/ansible-epic/web:$VERSION
docker push ghcr.io/sapphire-health/ansible-epic/web:latest

docker push ghcr.io/sapphire-health/ansible-epic/base:$VERSION
docker push ghcr.io/sapphire-health/ansible-epic/base:latest
docker push ghcr.io/sapphire-health/ansible-epic/web:$VERSION
docker push ghcr.io/sapphire-health/ansible-epic/web:latest

Note

On the first push of each image, GHCR creates the package automatically and links it to this repository via the org.opencontainers.image.source label. Remember to set the desired visibility on newly created packages — they start private — and to grant the repository Write access under Manage Actions access so the CI workflow can push to them.

CI Pipeline (GitHub Actions)

The Build Container Images workflow (.github/workflows/build-containers.yml) runs the full build-and-push sequence on a GitHub-hosted runner. It wraps scripts/build-images.sh, so CI and local builds share a single code path and produce identically tagged and labeled images.

Trigger: workflow_dispatch only — run it from the repository's Actions tab.
Version input: optional. Left empty, the workflow uses today's date and automatically appends a .N serial if that tag already exists on GHCR (so re-running on the same day is safe). An explicitly supplied version that already exists fails the run — immutable version tags are never overwritten.
Runner: GitHub-hosted ubuntu-latest. The self-hosted runner cannot build images — it runs on ECS Fargate, which has no Docker daemon and does not allow privileged containers.
Authentication uses two separate tokens, because the built-in one can't do both jobs:
- The workflow's built-in GITHUB_TOKEN (with permissions: packages: write) authenticates the push to GHCR.
- A short-lived installation token minted from the org's container-build GitHub App is passed to the build as the github_token secret to clone the private ansible-role-* repositories — the built-in token is scoped to this repository only and cannot read the role repos.

One-Time Setup

GitHub App — in the org: Settings → Developer settings → GitHub Apps → New GitHub App. Repository permissions: Contents: Read-only (nothing else); webhook disabled. Install the app on the organization, granting it the ansible-role-* repositories (or all repositories). Generate a private key and note the App ID.
Repository configuration — in this repo's Settings → Secrets and variables → Actions: create the variable CONTAINER_BUILD_APP_ID (the App ID) and the secret CONTAINER_BUILD_APP_PRIVATE_KEY (the full private-key PEM).
Package access — for each package (base, web): Package settings → Manage Actions access → Add repository → this repository with the Write role. Packages created by manual docker push do not grant workflow access automatically; without this step the CI push fails with denied: permission_denied: write_package even though the token scope is correct.

Mirror to ECR (Automatic — Pull-Through Cache)

On AWS the image is served to ECS from Amazon ECR, so tasks pull over AWS's internal network rather than from ghcr.io. No manual push is required: the ECR private registry in account 271851283454 (us-west-2) has a pull through cache rule (defined in the Terraform-Github-AWS repo) that mirrors images from GHCR automatically.

Unlike the ACR cache rule (which maps to a target repository without the org segment), the ECR rule prefixes the full upstream path with ghcr/:

ghcr.io/sapphire-health/ansible-epic/base:latest
→ 271851283454.dkr.ecr.us-west-2.amazonaws.com/ghcr/sapphire-health/ansible-epic/base:latest

ECR acts as a pull-through cache: the first pull of a tag creates the cached repository (if needed), fetches the image from GHCR, stores it, and serves it; subsequent pulls are served directly from ECR. The rule covers the whole ghcr/ prefix, so every image (base, web, …) is mirrored on demand with no per-repository setup.

Authentication differs from the ACR setup: ECR private registries always require auth, even though the upstream GHCR package is public. The ECS task execution role (IAM-CoderExecutionAWSRole-C, defined as CoderExecution in Terraform-Github-AWS) is already covered — it has the standard AmazonECSTaskExecutionRolePolicy pull permissions plus an ECRPullThroughCachePolicy granting ecr:BatchImportUpstreamImage and ecr:CreateRepository on arn:aws:ecr:us-west-2:271851283454:repository/ghcr/* (required to trigger cache refreshes and create cached repositories). Any other principal pulling through the cache needs the same two permissions, and non-IAM docker clients must authenticate to ECR first.

Caching is pull-triggered, and latest can lag up to 24 hours

ECR does not watch GHCR — a build lands in the cache when something pulls it through ECR. New (previously uncached) tags such as CalVer versions are imported immediately on first pull. For an already-cached mutable tag like latest, ECR revalidates against GHCR at most once per 24 hours, so a freshly re-published latest can take up to a day to propagate. To force an immediate refresh, either pin the new immutable version tag, or delete the cached latest image from the ghcr/sapphire-health/ansible-epic/base repository so the next pull re-imports it from GHCR.

So the complete AWS release flow is: push to GHCR (./scripts/build-images.sh -p), then redeploy the container — the cache rule handles the rest, subject to the 24-hour note above.

Manual Tag and Push

For pushing an image to an ECR repository directly — useful for repositories not covered by the pull-through cache — authenticate to ECR, create the repository if it doesn't exist, then tag and push:

BashPowerShell

aws ecr create-repository \
  --repository-name <repository> \
  --region us-west-2
docker tag ghcr.io/sapphire-health/ansible-epic/base:$VERSION \
  271851283454.dkr.ecr.us-west-2.amazonaws.com/<repository>:$VERSION
docker push 271851283454.dkr.ecr.us-west-2.amazonaws.com/<repository>:$VERSION

aws ecr create-repository `
  --repository-name <repository> `
  --region us-west-2
docker tag ghcr.io/sapphire-health/ansible-epic/base:$VERSION `
  271851283454.dkr.ecr.us-west-2.amazonaws.com/<repository>:$VERSION
docker push 271851283454.dkr.ecr.us-west-2.amazonaws.com/<repository>:$VERSION

Warning

Do not push into the ghcr/* namespace — those repositories are managed by the pull-through cache rule, and ECR rejects direct pushes to them. Use a different repository path for manual pushes.

Update the Task Definition

The Coder templates (coder/aws/dev-container/main.tf and coder/aws/gh-runner/main.tf) select the image via a Container Image parameter whose options reference the pull-through cache's latest tags:

data "coder_parameter" "image" {
  default = "271851283454.dkr.ecr.us-west-2.amazonaws.com/ghcr/sapphire-health/ansible-epic/base:latest"
  option {
    name  = "Production (base)"
    value = "271851283454.dkr.ecr.us-west-2.amazonaws.com/ghcr/sapphire-health/ansible-epic/base:latest"
  }
  option {
    name  = "Testing (base:testing candidate slot)"
    value = "271851283454.dkr.ecr.us-west-2.amazonaws.com/ghcr/sapphire-health/ansible-epic/base:testing"
  }
}

The parameter defaults to the production image (base:latest). The Testing option is a standing candidate slot pointing at the base:testing tag, so a new image can be tested per workspace without affecting other developers and without a template push. Because all options track moving tags, no template change is needed after publishing a new build — redeploying the workspace (below) pulls it once the cache has revalidated.

To publish a candidate to the testing slot, tag any build as base:testing on GHCR (both registry caches cover all tags of the repository, so no cache changes are needed):

docker pull ghcr.io/sapphire-health/ansible-epic/base:<candidate-version>
docker tag ghcr.io/sapphire-health/ansible-epic/base:<candidate-version> ghcr.io/sapphire-health/ansible-epic/base:testing
docker push ghcr.io/sapphire-health/ansible-epic/base:testing

Testers then flip their workspace's Container Image parameter to the Testing option and restart. Selecting the Testing option while no candidate is published fails the workspace build with an image-pull error — pick it only during an active test window. Promoting a candidate to everyone means re-tagging it as latest (or letting the next CI build become latest).

Redeploy the AWS Container

After pushing a new build to GHCR, force ECS to pull the image again:

BashPowerShell

aws ecs update-service \
  --cluster <cluster-name> \
  --service <service-name> \
  --force-new-deployment

aws ecs update-service `
  --cluster <cluster-name> `
  --service <service-name> `
  --force-new-deployment

ECS will start a new task using the updated image and stop the old one. The runner in the old container deregisters itself during shutdown before the new container registers a replacement. Remember the pull-through cache's 24-hour revalidation window for latest — if the new build must go out immediately, delete the cached latest from ECR first or use the version tag.

Mirror to ACR (Automatic — Cache Rule)

On Azure the image is served to Azure Container Instances from Azure Container Registry at coderdevcontainerreg.azurecr.io/ansible-epic/base, so containers pull over Azure's internal network rather than from ghcr.io. As with the ECR mirror, no manual push is required: the coderdevcontainerreg registry has an artifact cache rule that mirrors the image from GHCR automatically.

The cache rule maps the upstream repository ghcr.io/sapphire-health/ansible-epic/base to the local repository ansible-epic/base, making ACR act as a pull-through cache:

When a client pulls a tag from coderdevcontainerreg.azurecr.io/ansible-epic/base that isn't cached yet (or whose upstream tag has moved, like latest after a new build), ACR fetches the image from GHCR, stores it in the registry, and serves it.
Subsequent pulls of the same tag are served directly from ACR — no internet egress.

Unlike ECR's prefix-wide rule, an ACR cache rule maps one repository, so renaming the image means creating a new cache rule (a rule's target repository must not already exist, so the old rule can't be edited in place): registry Cache blade → Create rule → source ghcr.io/sapphire-health/ansible-epic/base, destination ansible-epic/base, no credentials.

Because the GHCR package is public, the cache rule needs no upstream credentials. Client-side authentication is unchanged: pullers still authenticate to ACR normally (the prod-ansible_container-eastus2-uai managed identity needs the AcrPull role on coderdevcontainerreg).

Caching is pull-triggered, not push-triggered

ACR does not watch GHCR or prefetch in the background — a new build lands in the cache the first time something pulls it through ACR (e.g. a workspace start), not when it is pushed to GHCR. That first pull after a new build traverses the internet once; every pull after that is served locally.

So the complete Azure release flow is: push to GHCR (./scripts/build-images.sh -p), then redeploy the container — the cache rule handles the rest.

Manual Tag and Push

For pushing an image to ACR directly — useful for registries or repositories not covered by a cache rule — authenticate to ACR, then tag and push. ACR creates the repository automatically on first push; no create step is needed.

BashPowerShell

docker tag ghcr.io/sapphire-health/ansible-epic/base:$VERSION \
  coderdevcontainerreg.azurecr.io/<repository>:$VERSION
docker tag ghcr.io/sapphire-health/ansible-epic/base:$VERSION \
  coderdevcontainerreg.azurecr.io/<repository>:latest
docker push coderdevcontainerreg.azurecr.io/<repository>:$VERSION
docker push coderdevcontainerreg.azurecr.io/<repository>:latest

docker tag ghcr.io/sapphire-health/ansible-epic/base:$VERSION `
  coderdevcontainerreg.azurecr.io/<repository>:$VERSION
docker tag ghcr.io/sapphire-health/ansible-epic/base:$VERSION `
  coderdevcontainerreg.azurecr.io/<repository>:latest
docker push coderdevcontainerreg.azurecr.io/<repository>:$VERSION
docker push coderdevcontainerreg.azurecr.io/<repository>:latest

Warning

Do not push into the ansible-epic/base repository — that namespace is managed by the cache rule (a cache rule's target repository must not already exist when the rule is created, and manual pushes into it can conflict with cached content). Use a different repository path for manual pushes.

Redeploy the Azure Container

After pushing a new build to GHCR, stop and start the Coder workspace from the Coder dashboard. Stopping the workspace destroys the container group and starting it recreates it; ACI pulls latest through the ACR cache rule, which fetches the new build from GHCR and caches it for subsequent pulls.

Updating the GitHub Actions Runner Version

The runner binary version is pinned in both base Dockerfiles via the GITHUB_RUNNER_VERSION build argument. To update it, change the value in both files and rebuild:

ARG GITHUB_RUNNER_VERSION=2.322.0

Find the latest release version at github.com/actions/runner/releases.

Updating the Coder Agent

The Coder agent binary is not baked into the image. At container startup, startup.sh downloads the agent directly from $CODER_AGENT_URL/bin/coder-linux-amd64, so it always matches the running Coder server version automatically. No image rebuild is required when the Coder server is upgraded.

Containers

Environment Variables

Access Modes

SSH

VS Code Tunnel

VS Code Web Server

Coder Workspace

Obtaining the Agent Token

Coder Template

Per-User Setup and Persistence (Standalone Deployments)

GitHub Actions Runner

Automated Deployment (Terraform / ECS)

Manual Token

Runner Logs

Deregistration on Shutdown

GitHub PAT Requirements

Repository Clone Token (GITHUB_TOKEN)

Runner PAT (GITHUB_RUNNER_PAT)

Publishing Images

Versioning

Package Visibility and Repository Association

Prerequisites

Authenticate to GHCR

Authenticate to ECR

Authenticate to ACR

Build the Images

Push the Images

CI Pipeline (GitHub Actions)

One-Time Setup

Mirror to ECR (Automatic — Pull-Through Cache)

Manual Tag and Push

Update the Task Definition

Redeploy the AWS Container

Mirror to ACR (Automatic — Cache Rule)

Manual Tag and Push

Redeploy the Azure Container

Updating the GitHub Actions Runner Version

Updating the Coder Agent

Repository Clone Token (`GITHUB_TOKEN`)

Runner PAT (`GITHUB_RUNNER_PAT`)