{"id":352,"date":"2022-08-03T14:42:00","date_gmt":"2022-08-03T14:42:00","guid":{"rendered":"https:\/\/www.simplybusiness.co.uk\/implementing-container-shell-system-ecs\/"},"modified":"2025-10-30T14:14:31","modified_gmt":"2025-10-30T14:14:31","slug":"implementing-container-shell-system-ecs","status":"publish","type":"post","link":"https:\/\/www.simplybusiness.co.uk\/about-us\/tech\/2022\/08\/implementing-container-shell-system-ecs","title":{"rendered":"How we implemented container shell access on ECS"},"content":{"rendered":"\n

At Simply Business, our developers often need terminal emulation access to production-like environments. In the DevOps team, we’re keen to support this but also need to cater for the security implications, as well as the need to audit activity on interactive container sessions. In this post, we share how in the DevOps team, we’ve addressed these needs by creating a custom solution for container shell access on ECS Exec<\/a>, integrated with our in-house CI\/CD system built with Github Actions<\/a> and Codebuild<\/a>.<\/p>\n\n\n\n

Recently the Simply Business DevOps team has been looking into replacing our existing solution – butlerx\/wetty<\/a>, which we use to grant shell access for developers to Docker containers running in our ECS<\/a> cluster. Just to make it clear, this post is not a philosophical discussion on shell access to Docker containers. There\u2019s plenty of material out there if you\u2019re looking to forge an opinion on that. Rather, it’s about how we’re handling very legitimate use cases for granting shell access to containers, given that turning off our existing solution isn’t really an option.<\/em><\/p>\n\n\n\n

Why build a container shell access system?<\/h2>\n\n\n\n

We’ve been using butlerx\/wetty<\/a> for several years, and although it has served us well, we were looking to improve on a number of issues identified with the existing solution, including tight coupling to production deployment pipelines, logging and observability, security and costs.<\/p>\n\n\n\n

In designing a replacement, it was also important that the new solution integrated well with two other systems that we’ve introduced to improve our deployment pipeline and logging needs, namely our custom-built CI\/CD platform Huxley<\/a>, which uses Github Actions<\/a> and Codebuild<\/a> to replace our previous system on Jenkins<\/a>, and Elastic SIEM<\/a>, an information security logging system.<\/p>\n\n\n\n

As chance would have it, AWS had enough customer feedback from the developer community wanting something similar to K8\u2019s kubectl exec<\/a> for accessing ECS containers that they released ECS Exec<\/a>.<\/p>\n\n\n\n

After some investigation, it turned out that ECS Exec looked like it might offer us the types of improvement we were looking for, particularly in regard to security. With ECS Exec, we could handle all authorisation through Identity and Access Management (IAM<\/a>), and it could also be set up to use attribute-based access control (ABAC<\/a>) authorisation, and secure access to services and resources via IAM policies that fit with our existing IAM authorisation model, all on a per-teams access basis. By granting authorisation through IAM, we’d be able to log all activity and calls in CloudTrail<\/a>. Tick!<\/p>\n\n\n\n

One of the security requirements for our container shell access system was that the containers providing shell access should not be the same containers used for production workload computation. So the DevOps team decided to provision a new container on ECS as a one-off task with sleep running under the initialisation<\/a> process of the container, that would provide shell access until sleep ended. This implementation also solved a cost issue we had with the initial solution we’d proposed, which required an ECS service<\/a> and an application load balancer (ALB<\/a>) to be provisioned to run and provide access to the shell.<\/p>\n\n\n\n

However, this implementation presented another issue with ECS Exec\u2019s logging, whereby logs would be lost if the sleep timeout was reached and the container was deprovisioned before any user who was logged into the shell had exited that shell.<\/p>\n\n\n\n

\"sb-tech-site-banner\"<\/a><\/figure>\n\n\n\n

Wilson – waiting for cats with pipes<\/h2>\n\n\n\n

After some thinking around alternative ideas on how to improve on this, we ended up writing a small program in Go<\/a>, which we’ve named Wilson<\/em>. The main goal of implementing Wilson was to capture the standard i\/o of the shell sessions and log it to standard output<\/a> of the container.<\/p>\n\n\n\n

We’ve implemented this in Wilson as follows.<\/p>\n\n\n\n