SSH Authentication — Production (Customers)
This page is about outbound authentication — how the Ansible container reaches the Linux hosts
it manages (EC2 instances, Azure VMs) once it's running. That's a different concern from the
inbound AUTHORIZED_KEYS mechanism documented in Containers → SSH,
which controls who can connect into the Ansible container itself.
Unlike the development environment, a customer's production container
has no Coder coder_script hook to fetch secrets on every start — it runs via startup.sh under
plain docker run, ECS, or ACI. So the model here splits into two phases: an empty agent
initialized at container start, and a playbook you run once to populate it with the key.
Implementation status
The two playbooks below (playbooks/load-ssh-key.yml, playbooks/load-ssh-key-vault.yml) exist
in the repo. The empty-agent initialization is shown as a startup.sh snippet below and is the
one remaining wiring step for non-Coder customer containers.
How it works: empty agent at startup, populated by a playbook
The design leans on one property: the agent socket is a fixed rendezvous point. If the socket
path is established before VS Code launches, every terminal inherits SSH_AUTH_SOCK from the start.
Whatever key gets added to the agent behind that socket later is then instantly usable in every
terminal — including ones opened before the key was loaded — because ssh reconnects to the socket
on each invocation.
So the two phases are cleanly separated:
- Startup starts an empty agent on the fixed socket and exports
SSH_AUTH_SOCK. No secret handling happens at boot — the container starts clean. - A playbook, run once, fetches the key and pipes it into the already-running agent. Loading the key is an explicit, auditable action, not something that happens silently at boot.
The empty agent (startup.sh)
Add this early in startup.sh, before VS Code / the tunnel launches, so they inherit the socket.
The CODER_AGENT_TOKEN guard keeps it from competing with the dev environment's ssh_agent
coder_script, which already does this under Coder:
# Non-Coder containers: start an EMPTY ssh-agent on a fixed socket so every
# terminal inherits it. The key is loaded later by playbooks/load-ssh-key*.yml.
if [ -z "${CODER_AGENT_TOKEN:-}" ]; then
SOCK=/tmp/ansible-ssh-agent.sock
[ -S "$SOCK" ] || { rm -f "$SOCK"; ssh-agent -a "$SOCK" >/dev/null; }
export SSH_AUTH_SOCK="$SOCK" # VS Code server + its terminals inherit this
# Login shells (SSH sessions) get the container agent ONLY if they don't
# already have one, so `ssh -A` agent forwarding is preserved, not clobbered.
cat <<'PROFILE' | sudo tee /etc/profile.d/ssh-agent.sh >/dev/null
[ -z "${SSH_AUTH_SOCK:-}" ] && [ -S /tmp/ansible-ssh-agent.sock ] && export SSH_AUTH_SOCK=/tmp/ansible-ssh-agent.sock
PROFILE
fi
Don't clobber SSH agent forwarding
The /etc/profile.d export must be guarded with [ -z "${SSH_AUTH_SOCK:-}" ]. When a client
connects with ssh -A, sshd sets SSH_AUTH_SOCK to the forwarded socket before the login shell
runs; an unconditional export would overwrite it and silently disable agent forwarding. The guard
lets a forwarded agent win and only falls back to the container agent when the session has none.
The inline export above is fine — sshd builds a clean per-session environment, so it never
inherits startup.sh's value.
The agent socket must stay on tmpfs
Keep the socket in /tmp (local container storage). Unix sockets cannot be bound on SMB/CIFS —
on Azure Files, ~/.ssh is a symlink onto the SMB mount and binding there fails with
Operation not permitted. This is the same constraint noted for the
dev environment.
The playbooks below are also self-healing: if the socket doesn't exist yet, they start the empty
agent themselves, so they work whether or not startup.sh pre-created it.
Choosing a playbook
| Playbook | Key source | Use when |
|---|---|---|
playbooks/load-ssh-key.yml |
Cloud secret store (Key Vault / Secrets Manager), cloud auto-detected | The environment has a cloud secrets manager and a workload identity (managed identity / instance or task role) |
playbooks/load-ssh-key-vault.yml |
ansible-vault-encrypted file in extra_vars/ |
Air-gapped / on-prem environments with no cloud secrets manager |
load-ssh-key.yml auto-detects the cloud. It checks the managed-identity / task-credential
environment markers first (IDENTITY_HEADER / IDENTITY_ENDPOINT / MSI_ENDPOINT for Azure ACI &
App Service; AWS_CONTAINER_CREDENTIALS_* / ECS_CONTAINER_METADATA_URI_V4 for AWS ECS/Fargate),
then falls back to probing the instance metadata service (Azure IMDS vs. AWS IMDSv2 at
169.254.169.254) for bare VMs and EC2 instances. The env-marker check matters because container
platforms like ACI and Fargate do not expose the VM/EC2 instance-metadata endpoint even though
their identity mechanism works. Override detection entirely by exporting CLOUD=aws or
CLOUD=azure.
Both playbooks pipe the key straight into ssh-add via stdin with no_log: true, so it never lands
on disk, never appears in a registered variable, and never shows up in ps.
Storing the key
Use a per-customer key and (for the ansible-vault route) a per-customer vault password, so a
single compromise never spans customers. Prefer a passphrase-less key — the secret store or the
vault encryption is the protection, and ssh-add - cannot read a passphrase non-interactively.
The vault and secret names shown below are the defaults in group_vars/all/vars.yml, which are
Sapphire's own. A customer environment will have a different vault, and may name the secrets
differently — every one of these is overridable, see
Pointing at a different vault or secret.
Store the private key under the secret name load-ssh-key.yml expects
(azure-rm-linux-ssh-key). --file preserves the multi-line key intact:
az keyvault secret set \
--vault-name prod-sapphire-vault \
--name azure-rm-linux-ssh-key \
--file ~/.ssh/customer_key \
--encoding utf-8
Grant the container's managed identity get on this secret. Which command applies depends on
the vault's authorization model — check with
az keyvault show --name prod-sapphire-vault --query properties.enableRbacAuthorization:
# Classic access policies (enableRbacAuthorization: false)
az keyvault set-policy \
--name prod-sapphire-vault \
--object-id <managed-identity-object-id> \
--secret-permissions get
# Azure RBAC authorization (enableRbacAuthorization: true)
az role assignment create \
--role "Key Vault Secrets User" \
--assignee <managed-identity-object-id> \
--scope <key-vault-resource-id>
Rotate by updating the secret with the same az keyvault secret set command.
Store the key under aws-ec2-linux-ssh-key (create once, then put-secret-value to rotate):
aws secretsmanager create-secret \
--region us-west-2 \
--name aws-ec2-linux-ssh-key \
--secret-string file://~/.ssh/customer_key
Grant the container's task role / instance profile (not the ECS execution role, if using
ECS) secretsmanager:GetSecretValue on the secret ARN — add a statement like this:
{
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:us-west-2:<account-id>:secret:aws-ec2-linux-ssh-key-*"
}
Fetch the role's current policy document first (aws iam get-policy-version), add the
statement, then publish a new version (aws iam create-policy-version ... --set-as-default) —
don't overwrite the existing document. No login step is needed once granted — boto3 resolves the
role automatically. Rotate the key with aws secretsmanager put-secret-value.
Encrypt the key into its own file in extra_vars/, rather than inline in
group_vars/all/vault.yml, so it can be passed explicitly at run time and kept separate
per-customer. Use ansible-vault encrypt_string --stdin-name, which reads the raw key from
stdin and writes out the complete vault_ssh_private_key: !vault | block itself — paste the
key exactly as it is, left-aligned, with no manual indentation to get right:
ansible-vault encrypt_string --stdin-name vault_ssh_private_key > extra_vars/ssh_key.yml
# paste the key, then press Ctrl-D
(Or non-interactively, from a file you already have: add < ~/.ssh/customer_key after
--stdin-name vault_ssh_private_key.) The tool prompts for the vault password if one isn't
already resolvable, encrypts the pasted content, and generates the indentation on the encrypted
ciphertext itself — nothing about the key's own line breaks matters.
The vault password itself is resolved by vault_password_file (see
Secret Management) or --ask-vault-pass. The encrypted file lives in
git, so its security rests entirely on the vault password and AES256 — keep the password
per-customer, and use a separate file (e.g. extra_vars/<customer>_ssh_key.yml) per customer if
managing several from one repo checkout.
Rotate by re-running the same command to overwrite the file with a freshly encrypted key.
Running the playbook
Run once per container start (the loaded key persists for the container's lifetime, and becomes usable in every open terminal the instant it's added):
Pointing at a different vault or secret
load-ssh-key.yml defaults to Sapphire's own secret store. Override these in
group_vars/all/vars.yml, which is where they are defined:
| Variable | Environment fallback | Default |
|---|---|---|
azure_key_vault |
AZURE_KEY_VAULT |
prod-sapphire-vault |
azure_ssh_key_secret |
AZURE_SSH_KEY_SECRET |
azure-rm-linux-ssh-key |
azure_msi_client_id |
AZURE_MSI_CLIENT_ID |
(empty — system-assigned identity) |
aws_ssh_key_secret |
AWS_SSH_KEY_SECRET |
aws-ec2-linux-ssh-key |
aws_region |
AWS_REGION |
us-west-2 |
# group_vars/all/vars.yml — a customer's own Key Vault and secret name
azure_key_vault: contoso-epic-kv
azure_ssh_key_secret: linux-ssh-key
The uppercase environment variable is a fallback for cases where editing inventory isn't practical — a one-off run, or a container that injects configuration purely through its environment:
Why these live in group_vars/all and not in the play
The play targets localhost, which belongs to no groups, so group_vars/all is the only
group scope that reaches it — an environment-scoped group such as
ire_copier_azure_platform will not apply. They are also deliberately not declared in
the play's own vars: block: a play var outranks both group_vars and host_vars, so
declaring them there would silently shadow every override.
Verify, then confirm end-to-end connectivity:
Debugging a failed load
The load tasks use no_log: true to keep key material out of the logs, which also hides error
detail. If a fetch fails, temporarily remove no_log from the relevant task (or check the
secret name, region, and IAM/identity grant) to see the underlying error.
Security notes
- No secret access at boot. The container starts with an empty agent; credentials only enter when an operator deliberately runs the playbook.
- Key never on disk. Fetched value is piped to
ssh-add -via stdin; nothing is written to a key file, a variable, or the process arguments. - Per-customer isolation. Separate key per customer; for the vault route, a separate vault password per customer. Grant each container's identity access only to its own secret.
- A stepping stone to short-lived certs. This same empty-agent + populate-later plumbing is what
a short-lived-certificate model (Vault SSH CA / Teleport) wants — swap
ssh-add <key>forssh-add <short-lived-cert>and the rest is unchanged.