Continuous Integration and Automated Deployment with Personal and Shared Environments

Siddharth Malani Posted 03 September 2021

The Scenario

A Modis client in the resources sector uses machine learning to deliver a capability that helps improve the downstream processing of their product. Instead of having development environments for geoscientists that are running on local desktops and laptops, the customer wanted to make use of automatically generated, production like environments, to facilitate further investigation, development and delivery of the machine learning models, data pipelines, workflows, and configuration in the cloud environment.

The aim of using the AWS (Amazon Web Services) Cloud was to give the customer the ability to dynamically create multiple deployment environments that were the same as production, allowing data sets, configuration, and other resources to be shared. Additional benefits included:

  • Ability to scale down, or terminate these environments when no longer needed
  • Deployed on demand, helping the rapid development of features and ML (Machine Learning) models
  • Consistency in build, deployment, and test processes
  • Traceability of code and versioning
  • Environments for developers to test their code prior to merging into the project’s mainline repository.

The Solution

The customer imposed a few constraints on the solution, mandating the use of:

  • Gitlab
  • Terraform
  • AWS


Recommended best practice is to separate your CI/CD pipelines from your workloads by utilizing separate AWS accounts for each. The “DevOps” account would have access to push to each of the workload accounts. Each separate workload account other than production was multi-tenanted to support more than one developer concurrently.

This structure is depicted in the below diagram:

Continuous Integration and Automated Deployment with Personal and Shared Environments

Code Deployment Pipeline

A uniform approach to the CI/CD pipeline was used:

  1. Code gets pushed into the code repository
  2. Trigger code build
  3. Code is then deployed into the Development account
  4. When required, code gets promoted to higher environments.

AWS CodePipeline is utilized to orchestrate the build and deployment process.

Continuous Integration and Automated Deployment with Personal and Shared Environments

The built artifacts are stored in S3 and copied into a separate S3 bucket that is aligned to each deployable environment. S3 replication is used to copy the built artifacts into the relevant workload account, triggering a Lambda function to deploy to the respective microservice.

Infrastructure Deployment Pipeline

Gitlab and Terraform Cloud were used to deploy applications into AWS. All the AWS resources including AWS Lambda microservices, Step Functions, SageMaker, and managed databases (RDS) were defined in Terraform and deployed using Terraform Cloud into the AWS environments.

Each artifact that gets created has the checksum of the commit ID which triggered the build in its filename. The file that gets copied into S3 has the following naming convention:

  • <microservice-name>-<SHA>.zip

The same checksum is used in the Terraform scripts as variables to deploy for full traceability.

Sandpit environments

The scripts used to perform the deployments to each environment are parameterized to support several parallel deployments or biomes (sub-environments). This approach enables the Geoscientists to build and test their code independently. These scripts are also optimized to share the same RDS instance and SageMaker endpoints. The SageMaker endpoints use multi-model deployments for cost-optimization.


The Outcome

The solution uses Infrastructure as Code to define the environments, enabling rapid development and deployment of microservices and models across all the environments. The manual steps are reduced with the adoption of AWS CodePipeline and AWS CodeBuild, resulting in a consistent, simplified, repeatable developer experience.

The solution is also cost optimized and gives all the benefits required to the client under budget.



About Modis

At Modis we connect people, technology and businesses to the opportunities they need to thrive in a rapidly advancing world. With 1,300+ technology professionals across six locations (Sydney, Melbourne, Brisbane, Adelaide, Canberra and Perth), we work with our clients to deliver solutions and talent to transform technology portfolios, streamline business functions, drive innovation or enhance organisational capability.

Find out how Modis can provide you with innovative AWS cloud based solutions and servicesModis has been an AWS Advanced Tier Partner since 2014. Modis' AWS Cloud Consulting services encompasses fundamentals of cyber security, fault tolerant digital system architecture, modernisation, traditional virtual machine or through to modern Serverless approaches, commercial off-the-shelf software operation to bespoke software development, delivered with high throughput, repeatable DevOps approaches to operations. With over half a decade of running critical authoritative government data sets that affects the lives of millions of citizens and the economies of the state, Modis has one of the most mature, experienced and recognised consulting service providers in the world. More importantly, we like to work very closely with our customers, not providing something to purchase, but taking a deep understanding of their business, and providing the recommendations and implementations to ensure a modern, efficient, reliable and secure environment for digital business systems.Contact us
Modis Australia | Animated map showing global locations
We operate around the world. Would you like to find out more about your local office?Find out about Modis