Tips and Tricks

Get Host IPs

ansible all -m debug -a 'msg="{{ ansible_host }}"'

Get AWS Dev Container IP

hostname -i

Show the Content of all group_vars Ansible Vaults

find group_vars -mindepth 2 -maxdepth 2 -name 'vault.yml' | while read f; do
  echo "=== $f ==="
  ansible-vault view "$f"
  echo
done

Download a File from a Host to the Ansible Controller

Use the fetch module to pull a file off a managed host. flat=yes writes the file directly to dest instead of nesting it under a <hostname>/<full/source/path> directory tree — if dest ends in a / the source file name is preserved, otherwise dest is treated as the target file name.

LinuxWindows

ansible linux_host -m ansible.builtin.fetch -a "dest=~/source/files/ flat=yes src=/path/to/file.txt"

ansible windows_host -m ansible.builtin.fetch -a "dest=~/source/files/ flat=yes src=C:\\path\\to\\file.txt"

Note

ansible.builtin.fetch works against Windows hosts as-is — there is no separate win_fetch module.

Upload a File from the Ansible Controller to a Host

Use the copy module to push a file from the controller to a managed host. dest must be the full destination path including the file name (a trailing / or \ means "into this directory", keeping the source file name).

LinuxWindows

ansible linux_host -m ansible.builtin.copy -a "src=~/source/files/file.txt dest=/path/to/destination/"

ansible windows_host -m ansible.builtin.copy -a "src=~/source/files/file.txt dest=C:\\path\\to\\destination\\"

Tip

Writing to a system-owned directory on Linux needs elevation — add --become to the ansible command.

Delete and Redeploy Instances (Windows)

The script below demonstrates a copy/pasteable script to delete AWS instances and disks, remove computer objects from Active Directory, redeploy infrastructure with Terraform, and run Ansible playbooks to configure the new instances. Be sure to update the script with any environment specific values before running.

source /dev/stdin << 'EOF'
# Fetch a fresh gh CLI token from Coder so this script works from any terminal
export GH_TOKEN=$(/tmp/coder-agent external-auth access-token primary-github)

# remove multiple computer objects from Active Directory
ansible-playbook -e ad_management_host=epic-kpr-sapph1.sapphire.dev -e '{"computers":["epic-msql-sapph", "epic-kpr-sapph2", "epic-kpr-sapph1"]}' playbooks/remove-computer-ad.yml

# The example below demonstrates the same as the previous command, but removing only a single computer object from the domain
# ansible-playbook -e ad_management_host=epic-kpr-sapph1.sapphire.dev -e computers=epic-cog-sapph1 playbooks/remove-computer-ad.yml

# Delete AWS instances and disks
ansible-playbook --limit=app_kuiper,app_msql -e continue=true playbooks/delete-aws-instance-and-disks.yml

# Trigger Terraform GitHub Action workflow to redeploy infrastructure
echo '{"environment":"dev"}' | gh workflow run terraform-apply-dev-6am.yml -R sapphire-health/terraform-github-aws --ref main --json

# Wait for new instances to appear in inventory and respond to WinRM
until ansible -m ansible.windows.win_ping --limit=app_msql,app_kuiper all; do
  echo "Waiting for instances... retrying in 60 seconds"
  sleep 60
done

# Configure Windows Proxy on new instances
ansible-playbook -i inventory.aws_ec2.yml --limit='os_windows' playbooks/configure-windows-proxy.yml

# Create GMSAs in Active Directory with new computer accounts
ansible-playbook playbooks/create-gmsa.yml --limit=app_kuiper[0] -e ad_management_host=epic-kpr-sapph1.sapphire.dev -e domain_admin_user=admin -e domain_admin_password=Loc@lPassw0rd -e domain_name=SAPPHIRE.DEV -e @extra_vars/gmsas.yml

# Reboot instances to apply changes and prepare for SQL and Kuiper installations
ansible -m win_reboot all --limit=app_msql,app_kuiper

# Provision storage for new instances with AWS Windows Storage playbook
ansible-playbook --limit=app_msql playbooks/storage/provision-aws-windows.yml
EOF

Delete and Redeploy Instances (Linux)

The script below demonstrates a copy/pasteable script to delete AWS instances and disks, remove computer objects from Active Directory, redeploy infrastructure with Terraform, and run Ansible playbooks to configure the new instances. Be sure to update the script with any environment specific values before running.

source /dev/stdin << 'EOF'
# Fetch a fresh gh CLI token from Coder so this script works from any terminal
export GH_TOKEN=$(/tmp/coder-agent external-auth access-token primary-github)

# remove multiple computer objects from Active Directory (only needs to be run if linux machines were joined to the domain)
# ansible-playbook -e ad_management_host=epic-kpr-sapph1.sapphire.dev -e '{"computers":["tstodb", "prdodb", "relodb"]}' playbooks/remove-computer-ad.yml

# Delete AWS instances and disks
ansible-playbook --limit=*TSTODB* -e continue=true playbooks/delete-aws-instance-and-disks.yml

# Trigger Terraform GitHub Action workflow to redeploy infrastructure
echo '{"environment":"dev"}' | gh workflow run terraform-apply-dev-6am.yml -R sapphire-health/terraform-github-aws --ref main --json

# Remove old SSH host keys from known_hosts file to avoid SSH key verification errors when new instances are redeployed
rm -f /home/ansible/.ssh/known_hosts

# Wait for new instances to appear in inventory and respond to WinRM
until ansible -m ansible.builtin.ping --limit=*TSTODB* all; do
  echo "Waiting for instances... retrying in 60 seconds"
  sleep 60
done

EOF

Delete and Redeploy Instances (Azure IRE — Kuiper / System Pulse)

The script below deletes a Kuiper or System Pulse VM (with its disks) and its Azure SQL database, redeploys both with Terraform, and re-runs the Ansible playbooks that configure the new instance. It assumes the Azure Coder workspace (ANSIBLE_CONFIG=./ansible-azure.cfg is set automatically) and an az session with delete rights on the ire-* resource groups — the container managed identity holds the custom "IRE Contributor" role there (assignments managed in copier-terraform-azure-ire src/ansible_dev_connection.tf, recreated on every apply because the nightly destroy deletes them with the resource groups); az login --use-device-code as yourself is the fallback if those grants are missing. The delete/recreate portion (everything up to and including the WinRM wait) is also packaged as a Claude Code skill: /redeploy-azure-app, backed by .claude/skills/redeploy-azure-app/redeploy.sh. Domain credentials come from group_vars/ire_copier_azure_platform/vault.yml (see Secret Management). Be sure to update the script with any environment specific values before running.

source /dev/stdin << 'EOF'
# GH_TOKEN is set on every shell open in the Azure Coder workspace; refresh it manually if needed
export GH_TOKEN=$(/tmp/coder-agent external-auth access-token primary-github)

# Pick the app to rebuild (uncomment one line)
VM=azcusikpr001; DB=Kuiper;      DEPLOY=playbooks/deploy-kuiper.yml
# VM=azcusisp001; DB=SystemPulse; DEPLOY=playbooks/deploy-system-pulse.yml

# Remove the computer object from AADDS. AD operations delegate to
# ad_management_host, which defaults to the first Kuiper server — the very VM
# being deleted when rebuilding Kuiper — so pin it to a different Windows host.
ansible-playbook -e computers=$VM -e ad_management_host=azcusihsw001 playbooks/remove-computer-ad.yml

# Delete the Azure VM, its OS disk, and any data disks (the NIC is kept so
# Terraform reuses it and the private IP does not change)
ansible-playbook --limit=$VM -e continue=true playbooks/delete-azure-instance-and-disks.yml

# Delete the app database and its diagnostic settings — the Terraform apply
# below recreates both. These are management-plane calls, so they work even
# though the SQL server only allows data connections through its private
# endpoint. The diagnostic settings must be deleted explicitly: they survive
# database deletion as orphans, re-attach to the recreated database, and make
# the Terraform apply fail with "already exists" when it recreates them.
DB_ID="/subscriptions/$(az account show --query id -o tsv)/resourceGroups/ire-internal-centralus-rg/providers/Microsoft.Sql/servers/sph-ire-epicsql-centralus-sql/databases/$DB"
for ds in $(az monitor diagnostic-settings list --resource "$DB_ID" --query '[].name' -o tsv); do
  az monitor diagnostic-settings delete --resource "$DB_ID" --name "$ds"
done
az sql db delete --resource-group ire-internal-centralus-rg --server sph-ire-epicsql-centralus-sql --name $DB --yes

# Trigger the Terraform GitHub Actions workflow to redeploy the VM and database
gh workflow run terraform-apply-dev-6am.yml -R Sapphire-Health/copier-terraform-azure-ire --ref main -f confirm=apply

# Wait for the new instance to appear in inventory and respond to WinRM
until ansible -m ansible.windows.win_ping --limit=$VM all; do
  echo "Waiting for instance... retrying in 60 seconds"
  sleep 60
done

# Rejoin the domain (reboots the host; the computer object lands in its OU via the computer_ou group var)
ansible-playbook -l $VM playbooks/windows-join-domain.yml \
  -e 'domain_join_user="{{ domain_admin_user }}"' \
  -e 'domain_join_password="{{ domain_admin_password }}"'

# Recreate the contained SQL user in the fresh database (the azure_sql_* desired
# state lives in the app's group_vars; SQL work is delegated to
# azure_sql_management_host, which defaults to the Kuiper server)
ansible-playbook -l $VM playbooks/ire/configure-azure-sql-db-user.yml

# Kuiper only: re-add the new computer object to KuiperAllowedHosts and verify gMSA retrieval
[ "$VM" = azcusikpr001 ] && ansible-playbook -l $VM playbooks/create-gmsa.yml -e @extra_vars/gmsas.yml

# Install the application
ansible-playbook -l $VM $DEPLOY --tags install

# System Pulse only: the dev HTTPS binding uses a manually added self-signed
# certificate — re-add it after the rebuild (System Pulse section of the dev
# branch's ire-dev-readme.md)
EOF

Configure Ansible Container for Kerberos Authentication

Reference for what each step does, plus SPN troubleshooting, is in KERBEROS.md at the repo root.

# allow all traffic from the container networks (10.197.0.0/25) to the AWS directory services security group
ansible-playbook --limit 'app_kuiper[0]' playbooks/configure-ansible-for-kerberos-auth.yml
# the playbook writes krb5.conf to KRB5_CONFIG if it is already set in the environment, else the
# persistent source root (/home/ansible/source) — a stable path outside any git repo, reused across
# our ephemeral containers; override explicitly with -e krb5_conf_path=<path>. Note the ordering: if
# KRB5_CONFIG is exported *before* the run, that is also where the file is written. The dev-container
# templates auto-export KRB5_CONFIG at the default file once it exists, so new terminals pick it up
# automatically; the export below only affects the current shell (or non-container controllers)
export KRB5_CONFIG="/home/ansible/source/krb5.conf"
# ensure all VMs have been configured to allow WinRM Kerberos authentication (equivalent to winrm quickconfig -quiet)
ansible-playbook --limit=_Windows playbooks/configure-winrm-kerberos.yml
# unset or comment out any `server_admin_user` and `server_admin_password` variables so that Kerberos will be used
kinit admin@SAPPHIRE.DEV
# the SPN is built from ansible_psrp_negotiate_service (HTTP) + ansible_psrp_negotiate_hostname_override
# (inventory_hostname), both set in group_vars/os_windows/vars.yml — inventory_hostname must be a name the
# target's SPNs cover (its AD short name or FQDN, not a DNS alias). Verify on the host with `setspn -L <name>`

From a VS Code shell

kill -9 $(pidof code)