The Challenge

 
 

Procogia’s Approach

 

  • Our Data Engineers setup multiple environments to support the need for a volatile development environment and stable production environment. This was done using a sophisticated CI/CD process which involved running automated tests, approval gates and automated delivery pipelines for pushing code to Amazon Managed Workflows for Apache Airflow (AWS MWAA).
  • As part of the workflows that run on AWS MWAA, ProCogia leveraged the following:

    S3 as a scalable storage layer for both structured / unstructured data

    Redshift for the warehouse layer to connect BI tools to

    AWS Secrets Manager for securing database credentials.

  • To enable ease of debugging and promote long-term maintainability, system logs were funnelled into CloudWatch.

The Results

 

  • We delivered a stable, highly flexible solution that can also be re-purposed for other workflow applications.
  • Entire deployment was automated (infrastructure setup, unit testing, building, deployment) making it easily portable.
  • Users can make key business decisions regarding clinical survey targets within a week of the data becoming available.
  • The infrastructure is highly scalable, so if the volume/workload increases per threshold, its easy to scale the Airflow as well as Redshift environment with auto-scaling policies.

 
 

Related Blogs

 
 

Let’s Connect