{"id":332,"date":"2020-09-28T14:42:00","date_gmt":"2020-09-28T14:42:00","guid":{"rendered":"https:\/\/www.simplybusiness.co.uk\/docker-build-observability\/"},"modified":"2024-05-30T11:03:03","modified_gmt":"2024-05-30T11:03:03","slug":"docker-build-observability","status":"publish","type":"post","link":"https:\/\/www.simplybusiness.co.uk\/about-us\/tech\/2020\/09\/docker-build-observability","title":{"rendered":"Improving monitoring and observability of Docker build times"},"content":{"rendered":"\n

It can be difficult to understand what is going on under the hood in the Docker Build process, especially in complex multi-stage builds. There are many solutions available for observing running Docker containers, but limited options for observing Docker builds.<\/p>\n\n\n\n

Context<\/h2>\n\n\n\n

The time required to build the Docker image for our core software product at Simply Business has been slowly creeping up over the years. The application tooling team was tasked with optimising our Docker images for faster build times. This would reduce the time it takes to deploy code to production, speed up feedback cycles for code under development, and we thought it might impact developer happiness too.<\/p>\n\n\n\n

The problem<\/h2>\n\n\n\n

Measuring improvement is not straightforward<\/h3>\n\n\n\n

There are several reasons why simply measuring total build time for Docker images is not granular enough for our use case. We use several different caching strategies, and have network variability and other random sources of latency to account for, which combined can cause significant fluctuations in build times.<\/p>\n\n\n\n

Let’s look at a super simple example that illustrates this:<\/p>\n\n\n\n

Original Dockerfile<\/p>\n\n\n\n

FROM scratch\nCOPY \/ .\n# do some other stuff...\nCMD [\u201cecho\u201d,\u201dImage created\u201d]<\/code><\/pre>\n\n\n\n
Optimised Dockerfile<\/p>\n\n\n\n
FROM scratch\nCOPY \/lib .\n# do some other stuff...\nCMD [\u201cecho\u201d,\u201dImage created\u201d]<\/code><\/pre>\n\n\n\n
Instead of copying the whole directory COPY \/ .<\/code>, we are now only copying a subdirectory lib<\/code> with the files we care about onto our final image. The time wasted copying over tests, scripts and assets has been saved. However, since we are using Docker’s built-in layer caching, we’ll get cache misses starting from the second line all the way to the end of the file. This means on the first run, the optimised image may take longer to build than the original! No problem, we can just run the build a second time; we’ll get all the advantages of caching this time. But wait, the build now takes almost no time at all, because we have not made any changes to our image on rerun, so we are getting cache hits for every single step. This is the optimal scenario but doesn’t reflect the much more common scenario where some part of the image has changed. We’ve made an optimisation that we think should improve our build speeds, but can’t actually measure or even validate this improvement.<\/p>\n\n\n\n
The strategy<\/h2>\n\n\n\n
Improve monitoring and observability<\/h3>\n\n\n\n
We wanted a more granular view of our Docker build times. Instead of looking at the overall time to build a complete image, we care much more about comparing the time to execute a single layer. And rather than looking at the times for a single build run, we are much more interested in the normative average across several production builds. Fortunately, Docker BuildKit offers some snazzy features<\/a> for improved observability using Jaeger and the OpenTracing specification. However, many parts of BuildKit are still under development, and implementation was not straightforward.<\/p>\n\n\n\n
Gotcha #1: using buildctl<\/h4>\n\n\n\n
At the time of writing, OpenTracing in BuildKit is supported only with buildctl<\/code> or gRPC calls to the BuildKit daemon. We were not able to view traces using our docker-compose<\/code> configuration or even the standard docker build .<\/code>.<\/p>\n\n\n\n
Gotcha #2: running the BuildKit daemon<\/h4>\n\n\n\n
OpenTracing also requires that the BuildKit daemon (buildkitd<\/code>) is running. However, buildkitd<\/code> is currently only runnable on Linux systems. The majority of Simply Business engineers do their local development on MacOS, so this was our biggest barrier to successfully accessing Opentraces. We hacked around this by using Docker in Docker<\/a> as a substitute Linux VM.<\/p>\n\n\n\n
Gotcha #3: Jaeger tight coupling<\/h4>\n\n\n\n
The OpenTracing BuildKit feature is tightly coupled to Jaeger<\/a> as a visualisation tool (even though the OpenTracing specification<\/a> is vendor agnostic). At Simply Business, we prefer to consolidate our monitoring services as much as possible for simplicity. To extract the data and import it in a different visualisation tool, we used the Jaeger JSON API.<\/p>\n\n\n\n
Going forward<\/h4>\n\n\n\n
Keep an eye on the BuildKit Docs<\/a> for updates. As BuildKit features mature, many of these pain points may get resolved.<\/p>\n\n\n\n
\"sb-tech-site-technology\"<\/a><\/figure>\n\n\n\n
The Solution<\/h2>\n\n\n\n
A step-by-step guide for viewing Opentraces from Docker builds in Jaeger<\/h3>\n\n\n\n
1) Run Linux<\/h4>\n\n\n\n
If you’re already running on a Linux system (most distributions are fine), congratulations! You can skip this step.<\/p>\n\n\n\n
If you’re running Windows or MacOS, you can use the Docker in Docker image.<\/p>\n\n\n\n
docker run -it --privileged --name docker-in-docker -d -p 16686:16686 docker:dind\ncontainer_id=`docker ps -aqf \"name=docker-in-docker\"`\n\n# copy the project into that container\ndocker cp . $container_id:\/\n\n\n# enter the docker-in-docker container\ndocker exec -it $container_id \/bin\/sh<\/code><\/pre>\n\n\n\n
Notice we need to expose port 16686<\/code> to access the Jaeger dashboard. This will make more sense after Step 2.<\/p>\n\n\n\n
All the steps going forward should be run from inside the Docker in Docker container unless stated otherwise.<\/p>\n\n\n\n
2) Run Jaeger<\/h4>\n\n\n\n
docker run -e JAEGER_DISABLED=true -d -p6831:6831\/udp -p16686:16686 jaegertracing\/all-in-one:latest\nexport JAEGER_TRACE=0.0.0.0:6831<\/code><\/pre>\n\n\n\n
The JAEGER_DISABLED=true<\/code> flag prevents Jaeger from creating its own traces that will clutter up the dashboard. The -p6831:6831\/udp<\/code> flag sets the listening port in Jaeger for receiving OpenTrace data, and -p16686:16686<\/code> sets the output port for viewing the Jaeger dashboard. For a quick general overview of Docker binding ports, see this guide<\/a>. The export JAEGER_TRACE=0.0.0.0:6831<\/code> command specifies the same listening port to tell BuildKit where to publish the data.<\/p>\n\n\n\n
3) Install Docker BuildKit and run the BuildKit daemon<\/h4>\n\n\n\n
# install BuildKit\nwget https:\/\/github.com\/moby\/buildkit\/releases\/download\/v0.7.2\/buildkit-v0.7.2.linux-amd64.tar.gz\ntar xvfz buildkit-v0.7.2.linux-amd64.tar.gz\n\n# run BuildKit daemon (in the background)\n.\/bin\/buildkitd &<\/code><\/pre>\n\n\n\n
4) Build the image<\/h4>\n\n\n\n
.\/bin\/buildctl build \\\n    --frontend=dockerfile.v0 \\\n    --local context=. \\\n    --local dockerfile=.<\/code><\/pre>\n\n\n\n
The --frontend<\/code> flag<\/a> specifies how to convert the build definition into the intermediate binary (called an LLB<\/a>) that BuildKit expects. The --local<\/code> flag<\/a> exposes local files from the source to the builder, and both context<\/code> and dockerfile<\/code> are required. It is also possible to optionally pass build arguments with the --opt<\/code> flag:<\/p>\n\n\n\n
--opt build-arg:foo=bar<\/code><\/pre>\n\n\n\n
5) View results<\/h4>\n\n\n\n
View the traces in the browser at localhost:16686<\/code> or extract them via the JSON API. If you’re running Docker in Docker, this is accessible from outside<\/em> the container.<\/p>\n\n\n\n
curl http:\/\/localhost:16686\/api\/traces?service=buildkitd | jq '.data'<\/code><\/pre>\n\n\n\n
The complete script<\/h3>\n\n\n\n
# run parent docker container (hack so we have a linux system)\ndocker run -it --privileged --name docker-in-docker -d -p 16686:16686 docker:dind\ncontainer_id=`docker ps -aqf \"name=docker-in-docker\"`\n\n# copy the project into that container\ndocker cp . $container_id:\/\n\n# enter that container\ndocker exec -it $container_id \/bin\/sh\n\n# install and run jaeger listener\ndocker run -e JAEGER_DISABLED=true -d -p6831:6831\/udp -p16686:16686 jaegertracing\/all-in-one:latest\nexport JAEGER_TRACE=0.0.0.0:6831\n\n# install BuildKit\nwget https:\/\/github.com\/moby\/buildkit\/releases\/download\/v0.7.2\/buildkit-v0.7.2.linux-amd64.tar.gz\ntar xvfz buildkit-v0.7.2.linux-amd64.tar.gz\n\n# run BuildKit daemon (in the background)\n.\/bin\/buildkitd &\n\n# build the Dockerfile using buildctl\n.\/bin\/buildctl build \\\n    --frontend=dockerfile.v0 \\\n    --local context=. \\\n    --local dockerfile=.<\/code><\/pre>\n\n\n\n
The results<\/h2>\n\n\n\n
Analyzing Opentraces<\/h3>\n\n\n\n
Revisiting the original example, we can now view the Jaeger dashboard at http:\/\/localhost:16686\/<\/code> and understand exactly how much impact our optimisations have. Just to illustrate the point, we used a project with a very small lib\/<\/code> folder and included an additional 1 Gigabyte of cat photos in a different subdirectory*<\/sup>.<\/p>\n\n\n\n
\"project\"<\/figure>\n\n\n\n
In the original Dockerfile, Jaeger tells us that the COPY . \/<\/code> layer took 4.5 seconds to build, and was only able to start after the build context had loaded, which took 12 seconds. The overall build of this ultra-simple 3 line Dockerfile took about 17 seconds.<\/p>\n\n\n\n
\"original-jaeger-trace\"<\/figure>\n\n\n\n
After reducing the volume of data to copy over, Jaeger traces show the new command COPY lib\/ \/<\/code> took only 26 milliseconds. Even more significant is that loading the build context took less than 0.2 seconds. Total build time was 0.3 seconds.<\/p>\n\n\n\n
\"optimized-jaeger-trace\"<\/figure>\n\n\n\n
Both of these builds were run after clearing all Docker internal caches (Docker in Docker made it easy to restart the container and start from scratch). Therefore, these numbers represent worst-case-scenario build times.<\/p>\n\n\n\n
Obviously, for this trivial example we don’t need<\/em> the granularity of OpenTracing to see that we’ve made improvements. However, it does help us measure and understand exactly what has improved and by how much. In a significantly more complex Docker configuration with multiple stages and many more layers, these insights become invaluable.<\/p>\n\n\n\n
* In real life these might be replaced by testing scripts or assets. Or you may just have a lot of cat photos. We don’t judge.<\/p>\n\n\n\n
\n
Ready to start your career at Simply Business?<\/h3>\n\n\n\n
Want to know more about what it’s like to work in tech at Simply Business? Read about our approach to tech, then check out our current vacancies.<\/p>\n\n\n\n
\n
Find out more<\/a><\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
We create this content for general information purposes and it should not be taken as advice. Always take professional advice. Read our full disclaimer<\/a><\/p>\n<\/div>\n\n\n\n
\n
A free monthly newsletter with the latest news and tools for small businesses and landlords.<\/p>\n\n\n\n
\n
Subscribe to our newsletter<\/a><\/div>\n<\/div>\n\n\n\n