AWS Cloud Operations Review

James Bromberger Posted 11 March 2020

In late 2019, a multi-national enterprise asked Modis to review its AWS Cloud operations

Having started their AWS Cloud journey in early 2018 and applying a large amount of automation to their VPC environment creation, there were concerns about a number of factors of the AWS Well-Architected principles, particularly around security.

The Challenge

With a growing landscape, they had over 150 AWS Accounts, with over 100 Virtual Private Cloud environments. The organisation ran their corporate DNS domain in split-horizon – with an internal (on-premises) view that all cloud environments had to have the ability to resolve names for.

With the point of view of 2018, they had established a shared-services VPC, and a pair of DNS resolvers running atop of two small EC2 virtual machines. In each connected VPC, they had updated the DHCP Options to point all clients at these two EC2 instances, which in turn would either query DNS on premises, or resolve to the outside world.

As they had started to use VPC Endpoints for various services in their VPC architectures, they were finding these were not being used in each workload account.

It also became apparent that Guard Duty, enabled for all environment, was not getting the visibility of the DNS queries in the originating source environment: they we only appearing in the shared-services environment, which made problem determination more difficult.

Modis has had extensive experience with the Guard Duty service since its inception, and its history in improving and adding new security findings over time is a key piece of workload visibility and security operations.

The Solution

Modis routinely runs minor improvements of its customer environment when delivering a DevOps ongoing engagement, and as such, pays close attention to the changes that fundamentally improve the capability of the AWS Cloud environment.

One key release from November 2018 was the ability to define an outbound DNS resolver rule that would support the split-horizon view for the VPC DNS resolver. Called Route 53 Outbound Resolver, it replaces the current architecture, removing the need for a customer to run (and maintain) their own DNS services for the purpose of split-horizon.

This service also allows the Outbound rule and endpoints to be shared across account using the Resource Access manager (RAM). Thus, only one Outbound Resolver needs to be defined for an enterprise, with the Resolver having two Endpoints that originate queries towards the on-premises resolver from different Availability Zones of the existing shared-services VPC.

With this Rule shared across all organisations in the AWS Organization, each workload could then attach this rule to their VPC(s), adjust the VPC DHCP Options to reference the AmazonProvidedDNS (local per VPC) and wait for hosts to refresh their DHCP leases.

Outcomes and Results

There are several improvements the customer now sees:

  • No maintenance on the DNS server software (VPC resolver is fully managed component of the VPC)
  • No maintenance on the DNS server hosts (OS patching)
  • No instance maintenance of the DNS server host (instance family changes)
  • No VPC peering (or other connectivity) from workload accounts to shared services purely for DNS traffic
  • All instances can have an EGRESS Security Group rule clean up that removes outbound UDP and TCP 53 completely, for a better security posture.
  • Guard Duty now sees security issues on the source account for errant or unwanted DNS traffic
  • Local VPC Endpoints per VPC now resolve correctly
  • Increased DNS capacity: not relying on two DNS servers, but a scalable VPC cache as the EC2 fleet grows across all accounts.
  • No interruption to service during the host maintenance; previously during maintenance/reboot of DNS hosts, 50% of DNS queries would hit a 2 seconds timeout before attempting to query the second DNS host (UDP transport default timeout)

The change replaces two EC2 instances with one VPC Resolve Outbound of similar cost.

Find out how Modis can provide you with innovative AWS cloud based solutions and servicesModis has been an AWS Advanced Tier Partner since 2014. Modis' AWS Cloud Consulting services encompasses fundamentals of cyber security, fault tolerant digital system architecture, modernisation, traditional virtual machine or through to modern Serverless approaches, commercial off-the-shelf software operation to bespoke software development, delivered with high throughput, repeatable DevOps approaches to operations. With over half a decade of running critical authoritative government data sets that affects the lives of millions of citizens and the economies of the state, Modis has one of the most mature, experienced and recognised consulting service providers in the world. More importantly, we like to work very closely with our customers, not providing something to purchase, but taking a deep understanding of their business, and providing the recommendations and implementations to ensure a modern, efficient, reliable and secure environment for digital business systems.Contact us
Modis Australia | Animated map showing global locations
We operate around the world. Would you like to find out more about your local office?Find out about Modis