Tips and Tricks

Get Host IPs

ansible all -m debug -a 'msg="{{ ansible_host }}"'

Delete and Redeploy Instances (Windows)

The script below demonstrates a copy/pasteable script to delete AWS instances and disks, remove computer objects from Active Directory, redeploy infrastructure with Terraform, and run Ansible playbooks to configure the new instances. Be sure to update the script with any environment specific values before running.

source /dev/stdin << 'EOF'
# Fetch a fresh gh CLI token from Coder so this script works from any terminal
export GH_TOKEN=$(/tmp/coder-agent external-auth access-token primary-github)

# remove multiple computer objects from Active Directory
ansible-playbook --limit=epic-kpr-sapph1.sapphire.dev -e '{"computers":["epic-msql-sapph", "epic-kpr-sapph1", "epic-kpr-sapph2"]}' playbooks/remove-computer-ad.yml

# Delete AWS instances and disks
ansible-playbook --limit=epic-kpr-sapph*,epic-msql-sapph.sapphire.dev -e continue=true playbooks/delete-aws-instance-and-disks.yml

# The example below demonstrates the same as the previous command, but removing only a single computer object from the domain
# ansible-playbook --limit=epic-kpr-sapph1.sapphire.dev -e computers=epic-cog-sapph1 playbooks/remove-computer-ad.yml

# Trigger Terraform GitHub Action workflow to redeploy infrastructure
echo '{"environment":"dev"}' | gh workflow run terraform-apply-dev-6am.yml -R sapphire-health/terraform-github-aws --ref main --json

# Wait for new instances to appear in inventory and respond to WinRM
until ansible -m ansible.windows.win_ping --limit=epic-msql-sapph.sapphire.dev,epic-kpr-sapph* all; do
  echo "Waiting for instances... retrying in 60 seconds"
  sleep 60
done

# Configure Windows Proxy on new instances
ansible-playbook -i inventory.aws_ec2.yml --limit='_Windows' playbooks/configure-windows-proxy.yml

# Create GMSAs in Active Directory with new computer accounts
ansible-playbook playbooks/create-gmsa.yml --limit=epic-msql-sapph.sapphire.dev -e domain_admin_user=admin -e domain_admin_password=Loc@lPassw0rd -e domain_name=SAPPHIRE.DEV -e @extra_vars/gmsas.yml

# Reboot instances to apply changes and prepare for SQL and Kuiper installations
ansible -m win_reboot all --limit=epic-msql-sapph.sapphire.dev,epic-kpr-sapph*

# Provision storage for new instances with AWS Windows Storage playbook
ansible-playbook --limit=epic-msql-sapph.sapphire.dev playbooks/storage/provision-aws-windows.yml
EOF

Delete and Redeploy Instances (Linux)

The script below demonstrates a copy/pasteable script to delete AWS instances and disks, remove computer objects from Active Directory, redeploy infrastructure with Terraform, and run Ansible playbooks to configure the new instances. Be sure to update the script with any environment specific values before running.

source /dev/stdin << 'EOF'
# Fetch a fresh gh CLI token from Coder so this script works from any terminal
export GH_TOKEN=$(/tmp/coder-agent external-auth access-token primary-github)

# remove multiple computer objects from Active Directory (only needs to be run if linux machines were joined to the domain)
# ansible-playbook --limit=epic-kpr-sapph1.sapphire.dev -e '{"computers":["tstodb", "prdodb", "relodb"]}' playbooks/remove-computer-ad.yml

# Delete AWS instances and disks
ansible-playbook --limit=*ODB* -e continue=true playbooks/delete-aws-instance-and-disks.yml

# Trigger Terraform GitHub Action workflow to redeploy infrastructure
echo '{"environment":"dev"}' | gh workflow run terraform-apply-dev-6am.yml -R sapphire-health/terraform-github-aws --ref main --json

# Remove old SSH host keys from known_hosts file to avoid SSH key verification errors when new instances are redeployed
rm -f /home/ansible/.ssh/known_hosts

# Wait for new instances to appear in inventory and respond to WinRM
until ansible -m ansible.builtin.ping --limit=*ODB* all; do
  echo "Waiting for instances... retrying in 60 seconds"
  sleep 60
done

EOF

Configure Ansible Container for Kerberos Authentication

# allow all traffic from the container networks (10.197.0.0/25) to the AWS directory services security group
ansible-playbook --limit=epic-kpr-sapph1.sapphire.dev playbooks/configure-ansible-for-kerberos-auth.yml
# ensure all VMs have been configured to allow WinRM Kerberos authentication (equivalent to winrm quickconfig -quiet)
ansible-playbook --limit=_Windows playbooks/configure-winrm-kerberos.yml
# unset or comment out any `server_admin_user` and `server_admin_password` variables so that Kerberos will be used
kinit admin@SAPPHIRE.DEV
# ensure `ansible_psrp_negotiate_hostname_override` is defined in the inventory for the target hosts, and that the value matches the SPN registered in Active Directory for the host