Skip to content

SSH Authentication — Development Environment

Sapphire's own Coder-managed dev workspaces (coder/aws/dev-container, coder/aws/gh-runner on AWS; coder/azure/dev-container on Azure) need two things at every start: the ansible-vault password, and an SSH private key to reach the aws_ec2 / azure_rm_linux managed hosts. Neither is stored on disk or in git — both are fetched fresh from a cloud secret store using the container's own cloud identity, and the SSH key is piped straight into a shared ssh-agent so it never touches disk either.

Which config file is active

ansible.cfg at the repo root is AWS-flavored by default (it's the implicit default everywhere else in this repo — the GitHub Actions runners are AWS-only and never override it). The Azure Coder template sets ANSIBLE_CONFIG=./ansible-azure.cfg as a container environment variable, pointing Ansible at a separate, complete config file instead:

Cloud Config file Selected by
AWS ansible.cfg Default — no override needed
Azure ansible-azure.cfg ANSIBLE_CONFIG env var in coder/azure/dev-container/main.tf

Why ansible-azure.cfg, not ansible.cfg.azure

Ansible resolves a config file's type from its final suffix only. ansible.cfg.azure splits to extension .azure, which Ansible doesn't recognize — it fails with Unsupported configuration file extension. Hence the hyphenated name.

Ansible doesn't merge or include config files, so ansible-azure.cfg is a complete standalone file duplicating the shared settings (collections_on_ansible_version_mismatch, host_key_checking, roles_path) rather than inheriting them from ansible.cfg. If a shared setting changes, update both files.

AWS and Azure side by side

Coder templates: coder/aws/dev-container/main.tf, coder/aws/gh-runner/main.tf

Vault password — scripts/aws-vault-pass.sh reads the ansible-vault-password secret from AWS Secrets Manager:

aws secretsmanager get-secret-value \
  --region us-west-2 \
  --secret-id ansible-vault-password \
  --query SecretString --output text

SSH key — the ssh_agent coder_script (run_on_start) pulls the aws-ec2-linux-ssh-key secret and pipes it straight into ssh-add:

aws secretsmanager get-secret-value \
  --region us-west-2 \
  --secret-id aws-ec2-linux-ssh-key \
  --query SecretString --output text | ssh-add -

No login step is needed for either — the ECS task role's credentials are available automatically via the container credentials endpoint.

IAM permissions go on the task role, not the execution role

IAM-CoderTaskAWSRole-C needs secretsmanager:GetSecretValue on both secret ARNs — this is the role the running container actually assumes, confirmed via aws sts get-caller-identity inside the workspace. The execution role (IAM-CoderExecutionAWSRole-C) is unrelated; it only pulls the image and injects task-definition-level secrets like GITHUB_RUNNER_PAT. Granting the permission to the execution role instead of the task role is the most likely cause of an AccessDeniedException here.

Rotate the key or password — put-secret-value creates a new version and makes it current, no need to delete/recreate the secret:

aws secretsmanager put-secret-value \
  --secret-id <arn> \
  --secret-string file:///path/to/new/key

Coder template: coder/azure/dev-container/main.tf

Vault password — scripts/azure-vault-pass.sh logs in with whatever managed identity is attached to the container and fetches the ansible-vault-password secret from the prod-sapphire-vault Key Vault. No --client-id: the container group has exactly one identity, so IMDS returns it by default, and a pinned GUID would go stale whenever that identity is recreated or the workspace is rebuilt from a different template.

az login --identity -o none

az keyvault secret show \
  --vault-name prod-sapphire-vault \
  --name ansible-vault-password \
  --query value -o tsv

SSH key — the ssh_agent coder_script does the same login, then pulls the azure-rm-linux-ssh-key secret and pipes it straight into ssh-add:

az keyvault secret show --vault-name prod-sapphire-vault \
  --name azure-rm-linux-ssh-key --query value -o tsv | ssh-add -

az login is run with -o none so it can't accidentally leak anything into stdout ahead of the actual secret value.

The agent socket must stay on tmpfs

~/.ssh is a symlink onto the Azure Files SMB mount, and unix sockets cannot be bound on SMB/CIFS — the container will fail with Operation not permitted if the socket is placed there. It lives in /tmp instead, which is local container storage.

Rotate the key or password — update the corresponding Key Vault secret and restart the workspace.

Common mechanics

In both clouds the agent binds a fixed socket at /tmp/ansible-ssh-agent.sock, and the persist coder_script appends export SSH_AUTH_SOCK=/tmp/ansible-ssh-agent.sock to ~/.bashrc so every terminal inherits it — no ansible_ssh_private_key_file setting needed anywhere.

These ssh_agent scripts live in the Coder templates, not the image's startup.sh, so they re-run on every workspace start (the container's $HOME is ephemeral) and ship via coder template push with no image rebuild required.

EFS (AWS) supports unix sockets natively, so putting the socket in /tmp there is just for consistency with Azure and to avoid a stale socket persisting across restarts — not a hard requirement the way it is on Azure Files.

Why we're not using keychain

keychain solves one specific problem: a plain eval "$(ssh-agent)" starts a new agent bound to a random socket that only that one shell knows about — open a second terminal and there's no agent; close the shell and the agent leaks or dies. keychain fixes this by ensuring one long-lived agent per user, recording its socket in ~/.keychain/, and making every new shell re-attach to that same agent instead of starting its own.

This was tried in an earlier iteration of the dev environment, wired into ~/.bashrc directly. It works well on a personal machine, but doesn't hold up in a Coder container: the manual ~/.bashrc edit disappeared on the next restart, because $HOME is ephemeral and the persist coder_script regenerates .bashrc from scratch on every start.

We dropped it because something else already does keychain's job here, at a layer that survives restarts: the ssh_agent coder_script starts a single agent on the fixed /tmp/ansible-ssh-agent.sock socket on every boot, and the persist coder_script exports SSH_AUTH_SOCK to ~/.bashrc on every boot too — so every terminal attaches to that one agent without keychain's find-or-start logic. Adding keychain on top would start a second, competing agent and trigger a redundant Secrets Manager / Key Vault fetch for no benefit, since keychain's core value — persisting across logins — is moot in a container whose $HOME doesn't persist between restarts anyway.

keychain is still the right tool on a standalone machine that never runs under Coder — a personal dev box, laptop, or bare VM with no orchestrator managing the agent for you, or WSL, where a systemd --user agent service (the other common way to get persistence) is often unreliable and keychain sidesteps systemd entirely. In those cases, add it to ~/.bashrc as documented in the root CLAUDE.md.