The Challenge
Design and develop a solution used to aggregate clinical data across multiple data providers for consumption by the clinical surveying team.
Procogia’s Approach
- Our Data Engineers setup multiple environments to support the need for a volatile development environment and stable production environment. This was done using a sophisticated CI/CD process which involved running automated tests, approval gates and automated delivery pipelines for pushing code to Amazon Managed Workflows for Apache Airflow (AWS MWAA).
- As part of the workflows that run on AWS MWAA, ProCogia leveraged the following:
S3 as a scalable storage layer for both structured / unstructured data
Redshift for the warehouse layer to connect BI tools to
AWS Secrets Manager for securing database credentials.
- To enable ease of debugging and promote long-term maintainability, system logs were funnelled into CloudWatch.
The Results
- We delivered a stable, highly flexible solution that can also be re-purposed for other workflow applications.
- Entire deployment was automated (infrastructure setup, unit testing, building, deployment) making it easily portable.
- Users can make key business decisions regarding clinical survey targets within a week of the data becoming available.
- The infrastructure is highly scalable, so if the volume/workload increases per threshold, its easy to scale the Airflow as well as Redshift environment with auto-scaling policies.
Services Used
Data Consultancy
We provide Data Consultancy to organizations to optimize your investment in people, processes, and technology.
Data Engineering
ProCogia is partnered with the leading cloud providers, enabling our agnostic approach to focus on delivering tailored game-changing solutions for our clients.
Related Blogs
Let’s Connect
What can we help you with?
T: +1 425-624-7532
Alternatively, simply fill in this form and we’ll be in touch.
