Skip to content

Molecule Testing

Molecule tests a role by actually running it: it provisions fresh machines, applies the role, applies it again to prove idempotence, runs assertions against the end state, and tears everything down. In this repo, "fresh machines" are real EC2 VMs — the same self-service test VMs developers create by hand — so a passing run means the role worked on a genuine, fleet-shaped target: real systemd, real SELinux, real firewalld, no public IP, downloads through the squid proxy.

Whether a role must have a scenario, and what a green check means for PR review, is policy — see Development Standards → Molecule testing. This page is the mechanics: how the plumbing works and how to run it.

How it works

Molecule normally drives Docker or Vagrant. Neither fits here (no Docker daemon in the Fargate workspaces, and containers can't honestly exercise systemd/SELinux/reboots), so our scenarios use Molecule's delegated ("default") driver: Molecule delegates provisioning to a pair of shared playbooks in molecule/shared/ at the repo root, and they do what a developer would do by hand:

  1. create.yml — one coder create --template aws-test-vm per platform in molecule.yml (async, so multi-VM scenarios build in parallel — the wall-clock cost is the slowest single build, ~2–3 min). Platform names are prefixed mol- and the owner's name is worked into the hostname, so concurrent developers never collide. It then resolves each VM's private IP from EC2, waits for SSH, and hands Molecule the connection details. Existing workspaces are reused: a second molecule converge against a living VM skips the build entirely.
  2. Molecule runs the scenario's playbooks over plain SSH (the fleet key your workspace ssh-agent already holds): prepare.yml once per VM if present, converge.yml (applies the role), the idempotence re-run (must report zero changes), optional side_effect.yml, then verify.yml (assertions about the end state — ports listening, services running, files owned correctly).
  3. destroy.yml — coder delete per platform, then polls coder list until they're really gone (a transient CLI error mid-delete doesn't fail a green run).

In CI the same flow runs on the self-hosted runner with MOLECULE_RUN_SUFFIX=<scenario><run id> appended to every workspace and hostname, so a CI run and your interactive run of the same role never share a VM, and neither do two scenarios of the same run.

Running it

Prerequisites: a dev workspace (the coder/aws CLIs, fleet SSH key, and molecule itself are all preinstalled), with the role cloned inside an ansible-epic checkout at roles/<name>/ — the scenarios resolve the shared playbooks and the role code by relative path, so a standalone role clone outside the monorepo won't work.

Always run from the role's root directory (Molecule discovers ./molecule/ relative to where you run it):

cd roles/<name>
molecule test        # the full cycle: destroy → create → converge → idempotence → verify → destroy
molecule converge    # create (or reuse) VMs and apply the role — the iteration loop
molecule verify      # run the assertions against the current VMs
molecule destroy     # tear the VMs down

Two behaviors worth internalizing:

  • molecule test starts with a destroy — it guarantees a clean slate, so it always builds fresh VMs. That's what you want for a final proof, and not what you want while iterating.
  • molecule test also destroys on failure. If you're debugging, iterate with converge (VMs survive, failures leave the machine inspectable over SSH) and save test for the end. Don't start a converge while a failed test's destroy is still running — it races the dying VMs' IPs.

The VMs are ordinary test-VM workspaces: they show up in coder list, the 8-hour autostop backstops aborted runs, and you can SSH to them mid-scenario (ssh ec2-user@<ip> with the IP from coder list or the EC2 console — or just read it out of Molecule's inventory below).

Running other playbooks against Molecule's VMs

Molecule writes a normal Ansible inventory for its VMs, which means anything that takes -i can run against them — a role's regression test suite, ad hoc checks:

ansible-playbook -i ~/.ansible/tmp/molecule.*/inventory roles/iris/tests/main.yml -e @extra_vars/users.yml

Writing a scenario for a role

The copy-paste boilerplate — with commentary on every required line — lives in molecule/README.md at the repo root; start there rather than from upstream examples, because several of its lines are load-bearing in this environment specifically:

  • ANSIBLE_ROLES_PATH (provisioner env) must put the checkout first and keep the default locations after it. Without the pin, Molecule silently tests the image-baked copy of the role instead of your working tree; without the appended defaults, roles that depend on other roles (meta dependencies, include_role) fail syntax-check.
  • Per-platform ansible_ssh_common_args host_vars — Molecule's delegated defaults set IdentitiesOnly=yes, which disables ssh-agent keys, and the fleet key exists only in the agent.
  • Proxy vars in both shapes (flat http_proxy/… plus an env dict) — test VMs have no direct egress, and some roles' task-level environment: blocks shadow the play-level one.
  • Tag-gated roles need ansible_args: [--tags, install], and every scenario playbook plus the shared create/destroy carry play-level tags: always so the filter can't cut them off.

Multi-host scenarios (e.g. a PRD/DR pair) are just more entries in platforms: — each becomes its own VM, built in parallel — with per-host config under provisioner.inventory.host_vars.<platform>, exactly like production host_vars/ files. The iris role's scenario is the reference example, including composing per-host variables over a shared base with combine() and pointing paired hosts at each other by platform name.

Platform sizing: t3.medium is the default and fits most roles — but watch for software that sizes itself from system RAM (IRIS's InstanceType: OLTP demands 9 GB of shared memory on a 16 GB box; the scenario uses NONPROD). Declare data disks per platform in the same YAML structure the template takes.

CI

Scenario runs in CI are label-gated, never per-push — each run builds real VMs, so they're deliberate. Apply the cloud's label (molecule-aws or molecule-azure) to the role PR. Whether a later push re-runs the check or the label has to be re-applied depends on how that repo wired the trigger; both wirings are documented in molecule/README.md. What a green check is worth — and that it must be green on the PR's current head before approval — is the standards page's territory.

The run executes on a self-hosted runner created for that job and deleted when it finishes, so scenarios run in parallel and nothing idles in between. A repo opts in with a CODER_SESSION_TOKEN secret and secrets: inherit; without them, AWS falls back to the shared long-lived runner (if a run sits queued, that workspace is stopped or outdated — coder start / coder update it), while Azure fails fast, having no long-lived runner to fall back to.

Troubleshooting

Symptom Likely cause
Role changes have no effect on converge The scenario is missing the ANSIBLE_ROLES_PATH pin and is testing the image-baked role copy — check the error paths for ~/.ansible/roles/<name>
Permission denied (publickey) connecting to VMs Missing per-platform ansible_ssh_common_args override (IdentitiesOnly=yes is blocking the ssh-agent key), or the workspace's ssh-agent isn't loaded (ssh-add -l)
Downloads time out on the VMs Proxy vars missing from scenario group_vars (both flat and dict shapes), or the URL's host isn't in the squid allowlist
Syntax stage fails on a dependency role ANSIBLE_ROLES_PATH replaced the default path instead of appending to it
create fails at flag parsing, no workspace made JSON/YAML passed through --parameter (CSV parser) — the shared playbooks use --rich-parameter-file; so must any manual create
A failed run left mol-* workspaces behind coder delete mol-<platform> — or just run molecule destroy from the role