- Our Data Engineers setup multiple environments to support the need for a volatile development environment and stable production environment. This was done using a sophisticated CI/CD process which involved running automated tests, approval gates and automated delivery pipelines for pushing code to Amazon Managed Workflows for Apache Airflow (AWS MWAA).
- As part of the workflows that run on AWS MWAA, ProCogia leveraged the following:
S3 as a scalable storage layer for both structured / unstructured data
Redshift for the warehouse layer to connect BI tools to
AWS Secrets Manager for securing database credentials.
- To enable ease of debugging and promote long-term maintainability, system logs were funnelled into CloudWatch.
- We delivered a stable, highly flexible solution that can also be re-purposed for other workflow applications.
- Entire deployment was automated (infrastructure setup, unit testing, building, deployment) making it easily portable.
- Users can make key business decisions regarding clinical survey targets within a week of the data becoming available.
- The infrastructure is highly scalable, so if the volume/workload increases per threshold, its easy to scale the Airflow as well as Redshift environment with auto-scaling policies.