AWS Analytics Use Case - Data Pipeline and RedShift

Peter Brammer Posted 15 October 2019

Modis has been delivering data engineering and analytics services for several clients;

In this post, we’ll speak about an industrial commercial organisation who needed assistance to understand the opportunities and implementation around their industrial data.

The Challenge

Our customer was challenged to be able to present a consolidated view of data from up to 3,000 industrial sensors that produce data every second from their manufacturing environment. The management of the organisation wished to view historical plant operations data in a precise and consistent manner.

Why Amazon Web Services

The company required a scalable cost-effective platform to securely store, process and visualise performance data from their manufacturing plant. The data needed to be encrypted at rest, during transmission, and eventually stored in a secure scalable data warehouse structure. Amazon S3 is a scalable, cost effective storage platform, proven for over 15 years to be durable, available, and horizontally scalable for high performance. More recently this is being used for massive data lakes – with data staged into suitable formats before being imported into data warehousing for querying.

Modis helped worked with our customer to architect, and deliver a managed pipeline for moving raw data with AWS Data Pipeline into S3, and then taking raw data, reformatting it using scale out processing on an Amazon Elastic Map Reduce, restaging this back to S3, and then importing it into Amazon RedShift to be ready to be efficiently queried. From here, visualisation dashboards permitted the client to interrogate their data.

Outcome and Results

By leveraging these managed components, Modis implemented a solution that kept direct costs and overheads as low as possible. Furthermore, the initial requirement for an hourly refresh was implemented with a 15 minute refresh cycle, giving the team higher confidence in the evidence of operation they were looking at.

Modis Australia | AWS Analytics Use Case - Data Pipeline and Redshift

Find out how Modis can provide you with innovative AWS cloud based solutions and servicesModis has been an AWS Advanced Tier Partner since 2014. Modis' AWS Cloud Consulting services encompasses fundamentals of cyber security, fault tolerant digital system architecture, modernisation, traditional virtual machine or through to modern Serverless approaches, commercial off-the-shelf software operation to bespoke software development, delivered with high throughput, repeatable DevOps approaches to operations. With over half a decade of running critical authoritative government data sets that affects the lives of millions of citizens and the economies of the state, Modis has one of the most mature, experienced and recognised consulting service providers in the world. More importantly, we like to work very closely with our customers, not providing something to purchase, but taking a deep understanding of their business, and providing the recommendations and implementations to ensure a modern, efficient, reliable and secure environment for digital business systems.Contact us
Modis Australia | Animated map showing global locations
We operate around the world. Would you like to find out more about your local office?Find out about Modis