Container Image Testing
The combined base/web images have a test suite covering the customer deployment
surface (standalone Docker: SSH, VS Code web, VS Code tunnel) and the internal dev
surface (Coder workspaces on AWS and Azure).
The local tiers (1, 2, 2u) are implemented in scripts/test-images.sh and run in two
places: the Build Container Images workflow executes the script between build and
push, so a failing check blocks the publish and latest can never move to a broken
build (the workflow's push input can be disabled entirely for a dry run — build + test,
publish nothing); and the /image-tests skill (.claude/skills/image-tests/SKILL.md)
runs the same script on demand, plus the credentialed cloud tiers CI can't reach:
/image-tests # local tiers via the script, then offers Tier 3
/image-tests quick # local tiers only
/image-tests full # local tiers + Tier 3 (Coder workspaces)
/image-tests standalone # + Tier 4 (cloud standalone harness, real EFS/SMB)
/image-tests 2026.07.21 # test a specific version tag instead of latest
Passing a version tag makes the suite useful for pre-promotion checks: test the exact
CalVer build you are about to roll out, not whatever latest points at.
Test matrix
Tier 1 — image content (local, no credentials, ~2 min)
| Check | Verifies |
|---|---|
| venv symlinks | ~/venv/aws and ~/venv/azure resolve to ~/venv/ansible (compat paths for persisted bashrc / ENV_SETUP_SCRIPT) |
| Both cloud SDKs | boto3 and azure.mgmt.* import from the single venv |
| ansible-core version | Within the pinned >=2.18,<2.19 range |
| molecule baked | Pinned molecule imports from the venv; ansible-core pin survives its dependency set |
| Collections | amazon.aws, community.aws, azure.azcollection, microsoft.ad, ansible.windows, community.windows present |
| Roles | Every entry in roles/requirements.yml installed under ~/.ansible/roles |
| CLIs | aws, az, session-manager-plugin, gh, git, code, terraform all execute |
| Cloud configs | Both inventories + both ansible*.cfg present; ansible-config dump parses under each |
| No context junk | No .terraform or dev role clones baked into the image |
| OCI labels | version / revision / source populated |
| Release stamp | /etc/ansible-epic-release present; matches the intended version when one is passed (-V) — lets running deployments prove which build they are from inside |
| Size guard | Uncompressed image within ~15% of baseline: base < 6.5 GB, web < 8 GB (catches build-context bloat pre-push, where no registry manifest exists to read compressed sizes from) |
Tier 2 — access modes (local, ~3 min)
| Check | Verifies |
|---|---|
| SSH mode | AUTHORIZED_KEYS → sshd → key auth → run a command end to end |
| VS Code web mode | web image serves HTTP 200 on the configured port |
| VS Code tunnel smoke | Container reaches the Microsoft device-login prompt |
| ssh-agent socket | startup.sh's empty agent exists for non-Coder deployments |
Tier 2u — standalone user setup (local, ~5 min)
Exercises the USER_SETUP personalization feature against a docker volume standing in
for the customer mount, driving the interactive flows over ssh -tt:
| Check | Verifies |
|---|---|
| Ungated no-op | Without USER_SETUP, behavior is byte-identical to today |
| Boot preset | USER_SETUP_USER applies marker + links at container start, before any terminal; deploy key authenticates |
| Git identity | First TTY terminal prompts once; value lands in the persisted .gitconfig; no re-prompt |
| Picker | Lists only real personas (.persist present), creates new from a typed name |
| Non-interactive silence | ssh host cmd output is byte-clean — no prompts, no hangs |
| CWD inference | A shell opened inside a persona dir auto-selects that persona; a non-persona dir still gets the picker |
| VS Code links | Extensions + data/User are persist-store symlinks; server binaries local; tunnel token NOT persisted |
| GitHub auth links | ~/.config/gh + ~/.git-credentials are persist-store symlinks; a real gh auth login --with-token survives into a fresh container |
| Round-trip | Fresh container on the same volume: identity, SSH files, history, extensions, and gh auth persist |
| Web-mode persistence | web image: baked extension seeds the persona store; an extension installed via the serve-web server survives into a fresh container; server builds stay ephemeral |
No zero-byte .claude.json |
Fresh persona's link dangles until Claude Code writes it (an empty file reads as corrupt JSON); stale empty persisted copies are healed at boot |
ANSIBLE_CONFIG on 0777 dirs |
The baked ./ansible.cfg default reaches SSH login shells and loads config from world-writable dirs (SMB simulation via chmod 777) |
Tier 3 — Coder integration (gated, creates real workspaces, ~15–25 min)
Creates a disposable imgtest-* workspace per cloud from aws-dev-container and
azure-dev-container, asserts inside it, and deletes it. Guard rails: a stale-imgtest-*
sweep at the start of every run, a 1-hour autostop on test workspaces so orphans
self-terminate, run-unique names so concurrent runs cannot collide, and deletion
restricted to imgtest-* workspaces owned by the invoking user.
| Check | Verifies |
|---|---|
| Workspace builds | Template + registry cache + image boot on real infra |
| venv auto-activated | Persisted ~/.bashrc chain: source ~/venv/aws\|azure/bin/activate → symlink → combined venv |
| SSH agent key loaded | ssh_agent coder_script fetched the managed-host key from Secrets Manager / Key Vault |
| Vault password chain | scripts/aws-vault-pass.sh / azure-vault-pass.sh return a secret in-workspace |
| Live inventory | ansible-inventory --list returns hosts (task role / managed identity + correct ANSIBLE_CONFIG) |
| Linux ping | Full SSH credential chain against one running os_linux host |
| Windows ping | Full WinRM + vault credential chain against one os_windows host |
| Azure specifics | ANSIBLE_CONFIG=./ansible-azure.cfg set; git safe.directory applied |
Tier 4 — cloud standalone harness (gated, creates real infra, ~20–30 min)
Runs the Tier 2u user-setup scenarios against customer-shaped cloud deployments using
the aws-standalone-test and azure-standalone-test templates (coder/<cloud>/standalone-test/).
These templates are deliberately agent-less: no coder_agent, no Coder conveniences —
the container runs exactly the customer path, and the workspace showing "agent timeout" in
the Coder UI is expected. A test_mode parameter switches between the three customer access
workflows (ssh, web, tunnel — mutable, so one workspace can cover all three across
restarts), and each workspace gets isolated storage (a dedicated Azure Files share / EFS
access-point directory) so tests never touch dev data.
The Azure run is the only place real SMB semantics are validated (StrictModes no after
persona linking, safe.directory on the share, symlink-in-local-home layout) — run it
before the first Azure customer deployment uses USER_SETUP, and after changes to
user-setup.sh or startup.sh sequencing.
| Check | Verifies |
|---|---|
| Boot preset on real storage | Marker + links applied at container start against EFS / real SMB |
| SSH login after persona link | StrictModes no accommodation on SMB's fixed 0777 modes (Azure's highest-value check) |
| Git on the mount | safe.directory=* suppresses dubious-ownership on SMB |
| Stop/start round-trip | Persistence across a real container replacement on the same cloud storage |
| Web mode reachable | serve-web on 8080 with the web image, switched via the mutable test_mode |
Not automated
- Full VS Code tunnel connectivity — requires an interactive Microsoft device login;
the suite only verifies the container reaches the login prompt (the harness templates'
tunnelmode surfaces the prompt in the cloud logs for a manual run). Test manually when the tunnel path matters for a customer engagement. - Workspace persistence round-trip (file survives a workspace restart) — doubles Tier 3 runtime; run manually when the persist scripts change.
- GitHub runner registration (
aws-gh-runner) — registers real runners against the org; verify manually after runner-related changes.
CI integration
Implemented (2026-08): the Build Container Images workflow shellchecks the container
scripts, builds, runs scripts/test-images.sh against the just-built images, and only
then pushes — every published build passed the local tiers before anything could pull it.
Dispatching with the push input disabled turns the run into a dry run (build + full
test battery, nothing published), useful for validating a branch. Tiers 3–4 remain
on-demand via /image-tests because they need Coder credentials and VPC/VNet
reachability the hosted runner doesn't have.