Skip to content

AWS Container

Everything a customer AWS environment must provide to run the ansible-epic control node as an ECS Fargate service — the compute itself plus the registry, storage, identity, secret, and network dependencies around it. This page is the full deployment picture behind the summary bullets in Requirements → AWS Container; it slots into steps 1–2 of the New Customer Checklist. Access-mode and image details live in Containers.

The reference implementation is the environments/ansible Terraform environment in the Terraform-Github-AWS repository, which deploys this stack on top of an existing shared-infrastructure VPC and subnet with its own state file.

Compute

  • An ECS cluster, task definition, and service (desired_count = 1) running ansible-epic/base:latest on the Fargate launch type with the awsvpc network mode — 2 vCPU / 4 GB (cpu = "2048", memory = "4096") is the working default — with:
    • a private IP in the container subnet (assign_public_ip = false — see Network for what that implies about endpoints);
    • TCP 22 in portMappings if the SSH access mode is used;
    • the EFS volume mounted at /home/ansible/source;
    • both the execution role and the task role attached (below);
    • an awslogs log configuration pointing at the log group (below);
    • environment variables per the chosen access mode (e.g. AUTHORIZED_KEYS for SSH, TUNNEL_NAME for a VS Code tunnel), USER_SETUP=1 if the per-user setup and persistence feature is wanted, and KRB5_CONFIG=/home/ansible/source/krb5.conf if Kerberos authentication to Windows hosts is used, so the realm config lands on persistent storage.
  • A CloudWatch log group for the container. Optional in principle, but required in practice for the VS Code tunnel access mode — the device login code is only ever printed to the container logs.
  • Container Insights can stay disabled on the cluster; container logs still flow to the log group, and enabling it costs a few dollars a month in CloudWatch metrics. Turn it on temporarily if CPU/memory data is needed for right-sizing the task.

Container registry

  • An ECR pull-through cache rule for the ghcr.io upstream, using the ghcr/ prefix so the full upstream path is preserved. The image is then referenced as <account>.dkr.ecr.<region>.amazonaws.com/ghcr/sapphire-health/ansible-epic/base:latest and mirrors on demand — no manual push pipeline. See Mirror to ECR.

  • Unlike the Azure cache rule, an ECR rule for a ghcr.io upstream requires a credential_arn — a Secrets Manager secret holding a GitHub PAT. This is an ECR requirement for the upstream registry type, not a consequence of the packages being private; they are public.

    Creating the credential secret

    ECR only accepts a secret that follows its own conventions, and a rule pointed at a secret that breaks any of them fails at creation:

    • the name must begin with ecr-pullthroughcache/. This is not just a console filter — the credentialArn API parameter is regex-validated against ^arn:aws(-\w+)*:secretsmanager:...:secret:ecr\-pullthroughcache\/...$, so a differently-named secret is rejected at rule creation;
    • it must be in the same account and Region as the pull-through cache rule;
    • it must be encrypted with the default aws/secretsmanager KMS key — ECR cannot decrypt a secret encrypted with a customer managed key;
    • the value is JSON with exactly two keys, username and accessToken. The secret string to use for customer deployments is stored in passbolt as the password in AWS ECR Pull-Through Cache GHCR PAT.

    Creating it needs no local tooling — the AWS CloudShell in the target account's Region already has the CLI and your console credentials:

    aws secretsmanager create-secret \
  --region us-west-2 \
  --name ecr-pullthroughcache/ghcr \
  --secret-string '{"username":"<github-username>","accessToken":"<pat>"}'

    The PAT itself needs only read access to packages — see Pull-Through Cache PAT. To rotate, update the secret's value in place with aws secretsmanager put-secret-value; the rule references the secret by ARN and picks the new value up on its next upstream fetch.

    • Leave the cached repositories' tag mutability MUTABLE so re-pulls of a moving tag like latest can refresh the cached image.
    • Be aware of the cache's latest-tag revalidation lag: for an already-cached mutable tag, ECR revalidates against GHCR at most once per 24 hours, so a freshly published latest can take up to a day to propagate. Pin the immutable CalVer tag when a build must go out immediately.
    • Optionally attach a lifecycle policy to the auto-created cache repositories — expiring untagged manifests after 14 days keeps the registry tidy.
    • The execution role needs the standard ECR pull permissions plus ecr:BatchImportUpstreamImage and ecr:CreateRepository on the ghcr/* repositories, which are what trigger the initial import and later refreshes.

Persistent storage

  • An EFS filesystem with an access point and mount targets for /home/ansible/source — playbooks, inventory, and per-user state survive task restarts. A mount target is needed in every subnet the service places tasks in, so a two-AZ service needs two.

  • The access point must map to the image's ansible user, or nothing on the share is writable:

    Setting Required value
    posix_user.uid / posix_user.gid 1001 / 1001
    root_directory.path /ansible-persist
    creation_info.owner_uid / owner_gid 1001 / 1001
    creation_info.permissions 0750

  • The mount targets need a security group allowing TCP 2049 (NFS) whose source is the task's security group as a referenced security group, not a CIDR — the task's IP changes on every restart.

  • Mount the volume with IAM authorization (authorization_config.iam = "ENABLED" on the task definition volume). The mount is then signed with the task role, which must hold elasticfilesystem:ClientMount and elasticfilesystem:ClientWrite on the filesystem ARN.
  • The container must be able to resolve the EFS endpoint — DNS resolution and DNS hostnames enabled on the VPC.

Secrets Manager and the vault password

  • A Secrets Manager secret named ansible-vault-password, which the container reads on every playbook run (scripts/aws-vault-pass.sh → the task role's credentials → secretsmanager:GetSecretValue; see Secret Management). No static keys are involved — the task role's credentials are available automatically through the container credentials endpoint.

  • The secret's value should be generated, never typed or committed — and must be retrievable only from Secrets Manager (long and random; the encrypted vault files' ciphertext is only as strong as this password).

    The helper script hardcodes its region and secret id

    scripts/aws-vault-pass.sh calls aws secretsmanager get-secret-value --region us-west-2 --secret-id ansible-vault-password. A customer deployment outside us-west-2, or one that names the secret differently, has to adjust the script.

  • Optionally a second secret holding the SSH private key for Linux hosts, read the same way — see SSH Authentication.

  • These runtime reads belong to the task role. The one exception is GITHUB_RUNNER_PAT, which ECS injects via the task definition's secrets block before the container starts — that one is granted on the execution role, and must be stored as a plain string rather than a JSON object.

Identity and permissions

Two IAM roles, both with a trust policy allowing ecs-tasks.amazonaws.com to assume them:

Role Grant Purpose
Execution AmazonECSTaskExecutionRolePolicy Pull the image, write to the log group
Execution ecr:BatchImportUpstreamImage, ecr:CreateRepository on ghcr/* Trigger the pull-through cache import and refresh
Execution secretsmanager:GetSecretValue on the PAT ARN Only if GITHUB_RUNNER_PAT is used
Task ec2:DescribeInstances The aws_ec2 dynamic inventory enumerates VMs
Task ec2:DescribeRegions Only if the inventory does not pin regions:
Task ec2:DescribeVolumes Only for playbooks that enumerate attached storage
Task secretsmanager:GetSecretValue on each secret ARN Vault password, SSH key
Task elasticfilesystem:ClientMount, ClientWrite on the filesystem ARN IAM-authorized EFS mount
Task sts:AssumeRole Only when managing other accounts — the role in each target account needs ec2:DescribeInstances

The distinction matters and is easy to get backwards: the task role is what the running container actually assumes, confirmable with aws sts get-caller-identity inside it. The execution role is used by ECS itself, before and around the container's lifetime.

Whoever performs the deployment needs, beyond resource-creation rights in the account:

  • rights to create IAM roles and policies for the two roles above;
  • secretsmanager:PutSecretValue (or CreateSecret) to seed the ansible-vault-password secret.

Network

  • A VPC with one or more container subnets. These may be colocated on the shared-infrastructure subnet or dedicated to the Ansible container. A service spanning two AZs needs a subnet — and an EFS mount target — in each.
  • Routes from the container subnet(s) to every subnet with managed hosts.
  • Reachability from the container to every managed host: 22 to Linux, 5985/5986 to Windows — see Requirements → Network. The task's security group is used as the referenced security group in those rules. Fargate tasks get a new IP on every restart, so never scope these rules to the task's current IP.
  • The task security group needs egress to the paths below. Inbound is optional and access-mode dependent: 22 for SSH, 8000 for the VS Code web server, from the customer's management ranges only.
  • A separate NFS security group on the mount targets admitting 2049 from the task security group.

  • With no public IP, the container subnet reaches AWS services through a NAT gateway or interface VPC endpoints. The paths actually needed are:

    Destination Why
    ecr.api and ecr.dkr endpoints Image pull
    S3 gateway endpoint ECR image layers are served from S3 — a pull fails with only the two ECR endpoints
    secretsmanager endpoint Vault password and SSH key reads
    logs endpoint Container logs to CloudWatch
    sts endpoint Only for cross-account sts:AssumeRole

  • No ghcr.io egress is needed — the pull-through cache fetches upstream on ECR's side.

  • If the GitHub Actions runner is used, the subnet must additionally reach the GitHub API (api.github.com) to generate and remove runner tokens at startup and shutdown.

Validating a deployment

From inside the running container:

  1. aws sts get-caller-identity returns the task role, not the execution role (identity wired correctly);
  2. ./scripts/aws-vault-pass.sh prints the secret (Secrets Manager grant + network path);
  3. ansible-inventory --list enumerates the customer's VMs (ec2:Describe* grants + inventory config);
  4. a file written under /home/ansible/source survives aws ecs update-service --force-new-deployment (share mounted through the access point, with write permission);
  5. ansible -m ping / -m win_ping reach a Linux and a Windows host (network reachability rules).

Then continue the New Customer Checklist from step 3 (repository, inventory, vault contents, first playbook run).