Secret Management

Roles in this repository accept secrets two ways: as a lowercase Ansible variable (most often stored encrypted in Ansible Vault) or as a matching uppercase environment variable. Both exist side by side so the same role works whether secrets live in a shared team vault (this repo's own dev environment) or are injected per-run by a customer's own CI/CD or container secrets manager, with no vault file required at all.

Global (Non-Role) Secrets: Per-Group Vaults

Not every secret belongs to a role. Ansible's own connection variables — ansible_user, ansible_password — are secrets too, but they're scoped to an inventory group rather than a role, since they authenticate the connection itself before any role runs. Getting these right is what lets Ansible reach a host at all, before any role-specific secret matters.

Rather than cramming every secret into group_vars/all/vault.yml, split a group's variables across two files in group_vars/<group>/:

vars.yml — plaintext, everything that isn't sensitive
vault.yml — encrypted, only the actual secret(s) for that group

Ansible loads every file inside a group_vars/<group>/ directory automatically and merges them, so this split requires no extra wiring — it's the same mechanism as a single group_vars/<group>.yml file, just spread across two files by sensitivity instead of one.

group_vars/os_windows/ is the concrete example:

# group_vars/os_windows/vars.yml (plaintext)
ansible_shell_type: powershell
ansible_connection: psrp
ansible_psrp_negotiate_hostname_override: "{{ inventory_hostname }}"
ansible_port: 5985

# group_vars/os_windows/vault.yml (encrypted)
ansible_password: <encrypted>

Only ansible_password is sensitive, so it's the only thing in the encrypted vault.yml; the non-sensitive connection settings stay in plaintext vars.yml. Both files load automatically for every host in os_windows. This is the OS tier of the group taxonomy — it carries connection defaults common to every Windows host regardless of cloud. ansible_user, which can differ by cloud, is set at the platform_* tier (or, during the group migration, still by the legacy _Windows / _Red_Hat_Enterprise_Linux groups).

The os_linux group has no equivalent vault.yml — Linux hosts authenticate over SSH via a shared ssh-agent instead of a password variable. See SSH Authentication for that mechanism.

Encrypt, edit, or view a group vault the same way as any other vault file (see Ansible Vault below for the full command reference), just point at the group-specific path:

ansible-vault edit group_vars/os_windows/vault.yml

When to use group_vars/all/vault.yml vs. a per-group vault: all/vault.yml holds secrets tied to a role that can run against any host (kuiper_install_password, microsoft_sql_sa_password, etc.) — they aren't specific to one inventory group. A per-group vault.yml holds secrets tied to the connection itself for hosts in that specific group. If a secret is a connection credential, put it in that group's vault.yml; if it's a role parameter, put it in all/vault.yml (or override per-host in host_vars if it varies host to host) — unless it is a per-environment value, which is the case covered next.

Per-Environment Global Secrets

Some secrets aren't tied to a role or to a connection, but to an environment: the AADDS domain admin, the Windows local admin, and the Azure SQL server admin for the Azure IRE dev environment are one set of credentials, and their AWS equivalents are another. group_vars/all/vault.yml cannot hold both — it applies to every host on both clouds, and the AWS-only GitHub Actions runners in .github/workflows/prep-environment.yml would pick up the Azure values.

They live in the environment+platform group instead, <env>_azure_platform (see Application & Environment Groups for how membership is derived):

group_vars/ire_copier_azure_platform/
├── vars.yml    # plaintext, non-secret environment-wide settings
└── vault.yml   # encrypted

group_vars/ire_copier_azure_platform/vault.yml holds:

Variable	Purpose
`domain_admin_user` / `domain_admin_password`	AADDS domain admin
`server_admin_user` / `server_admin_password`	Windows connection credentials
`azure_sql_admin_user` / `azure_sql_admin_password`	Azure SQL server admin
`microsoft_sql_sa_password`	`microsoft_sql` role `sa` login
`microsoft_sql_spuser_password`	`microsoft_sql` role `spuser` login

The two microsoft_sql_* entries are also in group_vars/all/vault.yml, deliberately duplicated with the same value so the IRE copy can be rotated without touching the AWS one. microsoft_sql_spadmin_password is only in all/vault.yml. The domain, server and Azure SQL admin credentials are only in the IRE group — on AWS those variable names are undefined and the playbooks fall back to their uppercase environment variables, supplied by the runner's ENV_SETUP_SCRIPT secret, exactly as described in The Convention below.

Adding a second Azure environment means adding group_vars/<env>_azure_platform/ and a matching ansible_group_priority entry in inventory.group_priority.yml — see Group priority for why the pin is required.

The Convention

For a variable named <role>_<name>, the matching environment variable is <ROLE>_<NAME> (uppercased, same word boundaries). For example:

Ansible Variable	Environment Variable
`kuiper_install_password`	`KUIPER_INSTALL_PASSWORD`
`kuiper_gmsa_user`	`KUIPER_GMSA_USER`
`kuiper_aes_password`	`KUIPER_AES_PASSWORD`
`microsoft_sql_sa_password`	`MICROSOFT_SQL_SA_PASSWORD`
`microsoft_sql_<user>_password`	`MICROSOFT_SQL_<USER>_PASSWORD`

Precedence: the Ansible variable wins if it is set; the environment variable is only used as a fallback. This is deliberate — the vault is the source of truth when one is configured, and the environment variable exists purely as an alternate way to supply the same value when no vault entry is present.

Implementing It in a Role

roles/kuiper/tasks/install_kuiper.yml and roles/kuiper/tasks/dependencies.yml show the canonical pattern:

vars:
  domain_user: "{{ kuiper_install_user | default(lookup('ansible.builtin.env', 'KUIPER_INSTALL_USER'), true) | default(false, true) }}"
  domain_password: "{{ kuiper_install_password | default(lookup('ansible.builtin.env', 'KUIPER_INSTALL_PASSWORD'), true) | default(false, true) }}"
  gmsa_user: "{{ kuiper_gmsa_user | default(lookup('ansible.builtin.env', 'KUIPER_GMSA_USER'), true) | default(false, true) }}"

Read kuiper_install_password | default(lookup('ansible.builtin.env', 'KUIPER_INSTALL_PASSWORD'), true) right to left: if kuiper_install_password is unset or empty, fall back to the KUIPER_INSTALL_PASSWORD environment variable; the trailing , true tells Jinja's default filter to treat an empty string as "unset" too, not just an undefined variable — without it, a role default of "" (rather than leaving the variable undefined) would never fall through to the environment variable. The final | default(false, true) is a safety net so the expression always resolves to something falsy rather than raising an undefined-variable error when neither is set.

When adding this pattern to a new role, copy that shape exactly:

<role>_<name>: "{{ <role>_<name> | default(lookup('ansible.builtin.env', '<ROLE>_<NAME>'), true) }}"

Formerly deviating — system_pulse_sql_password

roles/system_pulse/defaults/main.yml's system_pulse_sql_password used to invert the precedence, checking the environment variable first and falling back to the vault variable. It has been normalized to the chain above — the microsoft_sql_spadmin_password Ansible variable wins, MICROSOFT_SQL_SPADMIN_PASSWORD is the fallback — matching this convention and Microsoft SQL → Tags and Variables. There is no sanctioned exception left to copy.

Handling Secrets Safely in Tasks

Two rules apply to any role that touches a secret, on top of the variable/env-var convention above:

Set no_log: true on any task whose arguments or return value contain a secret. Without it, the password (or the whole module invocation) can be printed to the console and captured in logs. If such a task legitimately needs to surface failure detail, wrap it in a block/rescue and have the rescue filter or redact before printing — roles/kuiper/tasks/install/install_kuiper.yml runs its win_package install with no_log: true and reads only fail|error lines from the MSI log on failure.

Never hardcode or commit a secret value. Defaults for secret variables are left empty/undefined in defaults/main.yml; the real value arrives only at runtime, from the vault or the matching uppercase environment variable (per the convention above). Nothing sensitive belongs in defaults/, vars/, or a task literal — not even as a "temporary" placeholder.

Setting a Secret

Via the vault (this repo's shared dev environment): add the lowercase variable directly to group_vars/all/vault.yml (or the relevant group's vault.yml — see Global Secrets above), using the exact variable name the role expects — ansible-vault edit group_vars/all/vault.yml and add kuiper_install_password: 'Sql@dm1n5!'. Group vars take precedence over role defaults automatically, so no other wiring is needed.

Via an environment variable (CI/CD, a customer's own pipeline, or any run without a shared vault): set the uppercase variable before invoking ansible-playbook, e.g. export KUIPER_INSTALL_PASSWORD=..., or inject it as a secret in whatever automation runs the playbook. This only takes effect when the vault variable is unset — leave the vault entirely absent if you want every secret supplied this way.

Disable bash history before pasting a secret into an interactive shell

Typing or pasting an export SOME_PASSWORD=... command directly into an interactive shell writes it to ~/.bash_history in plaintext by default — it persists on disk long after the session ends and is readable by anyone with access to that file. Disable history first, run the export, then re-enable it:

set +o history
export KUIPER_INSTALL_PASSWORD='...'
set -o history

This has to happen before the sensitive command — there's no way to scrub a line that's already been recorded. CI/CD secret injection (masked pipeline variables, task-definition secrets, etc.) doesn't have this problem, since the value never passes through an interactive shell in the first place.

Ansible Vault

Ansible Vault encrypts sensitive variables (passwords, license keys, etc.) so they can be safely committed to the repository. This project keeps the secrets that apply everywhere in group_vars/all/vault.yml, where they are loaded automatically for every host, and the rest in a per-group vault.yml — see Global Secrets and Per-Environment Global Secrets above.

Use a secure password generator to create strong passwords.

Create

Create the vault file with the secrets you want to encrypt:

cat <<EOF > group_vars/all/vault.yml
server_admin_user: Administrator
server_admin_password: USE_PASSWORD_GENERATOR
microsoft_sql_sa_password: USE_PASSWORD_GENERATOR
microsoft_sql_spadmin_password: USE_PASSWORD_GENERATOR
microsoft_sql_spuser_password: USE_PASSWORD_GENERATOR
EOF

Encrypt

Encrypt the file in place. You will be prompted for a vault password:

ansible-vault encrypt group_vars/all/vault.yml

An encrypted file begins with a header like $ANSIBLE_VAULT;1.1;AES256 and is safe to commit.

Edit

Edit an encrypted file without manually decrypting it. The file is decrypted into your editor and re-encrypted on save:

ansible-vault edit group_vars/all/vault.yml

View

Print the decrypted contents to the terminal without opening an editor:

ansible-vault view group_vars/all/vault.yml

Rekey

Change the password protecting the file:

ansible-vault rekey group_vars/all/vault.yml

Encrypt a Single String

To store an encrypted value inline (e.g. directly in host_vars) rather than encrypting an entire file:

ansible-vault encrypt_string 'USE_PASSWORD_GENERATOR' --name 'microsoft_sql_sa_password'

Paste the resulting !vault block into the relevant YAML file.

Automating Vault Password Entry

Rather than typing the vault password interactively on every run (--ask-vault-pass) or storing it in a plaintext file, point vault_password_file at an executable script instead. Ansible runs the script and reads the password from its stdout:

[defaults]
vault_password_file = ./scripts/my-vault-pass.sh

The only contract the script must honor: print only the password to stdout, and nothing else — no login banners, progress output, or trailing text. Beyond that, the script can fetch the password however makes sense for where it runs: a cloud secrets manager, a local password manager CLI, a hardware token, etc.

This repo's own dev environment implements this by fetching the password from a cloud secret store (AWS Secrets Manager or Azure Key Vault) using whatever identity the container already has — no credentials to type or rotate manually.

Setting It Up: AWS or Azure

AWSAzure

1. Create the secret, holding the vault password as a plain string:

aws secretsmanager create-secret \
  --region us-west-2 \
  --name ansible-vault-password \
  --secret-string '<the vault password>'

2. Grant the container's IAM role read access. Whatever role the container actually assumes at runtime needs a policy statement like:

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:us-west-2:271851283454:secret:ansible-vault-password-*"
}

This has to go on the ECS task role (IAM-CoderTaskAWSRole-C), not the execution role — see SSH Authentication → Development Environment for why that distinction matters and how to confirm which role a container is actually assuming. Attach the statement by fetching the role's current policy document first (aws iam get-policy-version), adding the statement, then publishing a new version (aws iam create-policy-version ... --set-as-default) — don't overwrite the existing document.

3. Point vault_password_file at scripts/aws-vault-pass.sh (already in this repo). It calls aws secretsmanager get-secret-value for the ansible-vault-password secret and prints the result to stdout — no login step needed, since the task role's credentials are available automatically via the container credentials endpoint.

1. Create the secret, holding the vault password as a plain string:

az keyvault secret set \
  --vault-name prod-sapphire-vault \
  --name ansible-vault-password \
  --value '<the vault password>'

2. Grant the container's managed identity read access. Which command applies depends on the vault's authorization model — check with az keyvault show --name prod-sapphire-vault --query properties.enableRbacAuthorization:

# Classic access policies (enableRbacAuthorization: false)
az keyvault set-policy \
  --name prod-sapphire-vault \
  --object-id <managed-identity-object-id> \
  --secret-permissions get

# Azure RBAC authorization (enableRbacAuthorization: true)
az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee <managed-identity-object-id> \
  --scope <key-vault-resource-id>

3. Point vault_password_file at scripts/azure-vault-pass.sh (already in this repo). It runs az login --identity, then az keyvault secret show for the ansible-vault-password secret and prints the result to stdout.

The SSH private key used to reach managed Linux hosts is set up the same way, in the same secret stores, using the same two scripts' sibling logic — see SSH Authentication → Development Environment for that walkthrough, and SSH Authentication → Production (Customers) for the equivalent guidance when deploying this repo into a customer's own environment.

Protect the vault password

Never commit a vault password to the repository in plaintext, and never have a vault_password_file script print anything besides the password itself — script output can end up in CI logs. Rotate the password with ansible-vault rekey if it is ever exposed.