Design Considerations for CI/CD Pipelines
When developing a new application, teams often take a close look to design application architecture, mitigate security concerns, address non-functional requirements, and plan delivery around critical business timelines. Designing the CI/CD process risks becoming something bolted-on to an existing application, rather than a critical consideration at the outset. I want to explore some of the considerations that can be made early in the lifecycle of an application to align the application for rapid change and consistent feature delivery.
Align Infrastructure and Application Deployment
In modern cloud platforms, the application infrastructure can be defined in code, just as the application itself is. Managing updates to the infrastructure and the application code in a synchronized manner can help reduce complications caused by the natural drift that occurs when these components are managed separately. Some simple use cases for synchronizing these changes are:
- Passing configuration values at runtime to an application based on terraform generated output
- Managing migration to new architectural components in code, through the stack of environments
- Running the same set of validations within the pipeline for infrastructure changes and application changes
In the end, what matters within these pipelines is that your application can continue to meet the functional and non-functional requirements to which it adheres. Running validation across the entire stack for every single change can help avoid common errors that can bubble up due to misconfiguration and identify errors earlier within the pipeline process.
Align the Test Pyramid
Martin Fowler has a great blog post on a practical test pyramid that outlines a disciplined testing philosophy, originally defined by Mike Cohn in his book Succeeding with Agile. I like to visualize the test pyramid laid out horizontally across the application delivery pipeline. The bottom of the test pyramid has little cost to run and since it is made up of unit tests, can be executed in almost any environment. Run these tests as early as possible and in as many situations as you can, every pushed commit to every branch of the repository should be possible for most development teams. Get used to using these tests as your early red flags, highlighting a potentially risky pull request or feature that often breaks the build before being integrated to master. Also, use these tests to keep the master branch as clean as possible, there should be no reason for master to ever turn red based on unit test failures given all the opportunity for validation before merge.
The more complex tests then lay themselves out along the deployment pipeline as the application is deployed to integration environments. Some end-to-end tests may not work in some lower environments due to missing dependencies or validations for non-functional requirements, so it can be helpful to categorize these tests in code and execute them in separate steps of the pipeline. Use the pipeline design to help improve the flow of the application. If bugs are consistently caught at a later stage, identify the key reasons for this delay and bring those forward within the pipeline.
Testing is critical for a CI/CD pipeline. Without good testing, these pipelines can become automated paths to trashing production and very sad customers. Pairing good testing practices and a solid pipeline approach can help reduce the time teams spend debugging costly bugs in production and non-production alike.
The final point I want to address is something that happens regularly in the JVM space. If an application has components that need to be warmed up or take significant time in initialization, dropping new versions out into a hot production environment can have a significant negative impact as the application struggles to catch up with the existing load. There are many strategies for addressing this challenge, let’s look at a high level at a couple approaches that can work:
Internalize Initialization Routines
Within many IoC systems like Spring Boot, you can create startup routines that report back to a central start up control operation to ensure that all operations are complete. A pattern that can work for Spring Boot is using an
ApplicationRunner component and report successful completion of that to a
HealthIndicator interface. Then, use the orchestration interface to wait until all the Spring Boot health interfaces return OK before shifting traffic. Within Kubernetes, this can be accomplished by creating a readiness probe that calls the
/health Actuator endpoint and waits for a 200 response. Actuator will not return 200 until all
HealthIndicator beans are reporting as healthy.
If we have unified our infrastructure with the application code, it can be very easy to stand up testing and warm up components within an integrated environment and use that to initialize the application. Often these end to end tests will hit most common use cases, so if we are able to create new instances without joining them to the load balancer or running service, we can run end to end tests against each new instance that is stood up and validate it before bringing it into the load balancer member list or behind a Kubernetes service endpoint.
There are challenges with this approach when an application development team has less control of the infrastructure. If teams cannot stand up these additional testing components or have fine grained control of the networking stack for the application, it can be very difficult to carry out this approach in a meaningful way. On the other hand, having the application initialization as just another step in the pipeline can provide another way to validate a code change before affecting live customers.
In this case, the platform is responsible for warming up the new instance before it activates it. A great example of this can be found in the Azure App Service deployment slots functionality. The service has a built in initialization feature, and will wait for warm up requests to be successful before swapping the active instance with the updated code. If you can rely on the platform to execute this process before activating the application, it becomes another consideration we can offload to the platform and reduce cognitive overhead during testing.
These three considerations are just a subset of all the challenges teams can face when building great CI/CD pipelines. Addressing these early in the development lifecycle can help eliminate toil from development teams during active development cycles and make it easier for all engineers to manage change within the application. When answers to these questions get placed in pipeline code rather than placed in the hands of a couple members of the team, siloed knowledge is replaced with a practice of codifying the team’s operating principals. This can reduce day-to-day operational burden and encourage all team members to participate in the consistent, successful delivery of features to production environments.