ProCogia enabled a T-Mobile
team to reduce their
Amazon S3 costs
40%
Reduction in Amazon S3 Costs
19%
Immediate Cost Savings After Implementing S3 Lifecycle Policies
Company Information
T-Mobile’s data team faced challenges with managing their rapidly growing Data Lake, which had reached 1.87 PB and was expanding by 0.15 PB monthly. This growth was leading to escalating Amazon S3 costs, with a projected year-on-year increase of 32%. To address this, ProCogia was brought in to implement cost-saving measures and improve data management efficiency.
The Challenge
T-Mobile needed help managing their expanding Data Lake. The growing volume of data correlated with rising Amazon S3 expenses, with the S3 costs increasing by 7% in the first month of the year. This trend was set to compound, potentially causing a significant impact on the team’s budget. The client required quick and effective solutions to reduce their cloud storage costs.
Our Approach
ProCogia consultants engaged with key stakeholders to understand the Data Lake's contents, how this data was written, and how it was consumed by their downstream customers.
An object-deletion policy was implemented, first bulk deleting all unnecessary objects and then applying S3 'Lifecycle Policies' in order to automatically delete objects once they passed an agreed retention period.
For objects that could not be deleted, S3 'Lifecycle Policies' were applied to transition infrequently-accessed objects from hotter to colder storage classes (S3 'Infrequent Access' and 'Glacier Instant Retrieval'). Other objects with undetermined access patterns were managed & transitioned to the S3 'Intelligent Tiering' storage class.
The daily Elastic Map Reduce jobs were updated to use the 'EMR File System' (EMRFS). This replaced the existing solution of HDFS and the 'S3a' client which can write a huge volume of temporary files to S3 as part of each ETL job.
The Results
Initial Cost Reduction
Following implementation of new S3 Lifecycle Policies in the third month of the year, the monthly S3 bill was immediately down 19% (versus the forecasted bill without intervention).
Further Savings
Following the upgrade of EMR jobs to EMRFS (and S3 Intelligent Tiering beginning to self-manage object classes), the monthly S3 bill was further reduced by an additional 26% (a 40% total saving for the month versus the forecasted bill without intervention).
Conclusion
ProCogia’s strategic interventions, including the application of S3 Lifecycle Policies and the optimization of EMR workflows, led to a substantial 40% reduction in T-Mobile’s monthly S3 costs. By implementing automated solutions and transitioning data to more efficient storage classes, T-Mobile successfully curbed the escalating cloud expenses and ensured more sustainable data management practices moving forward.
Explore more stories
Dig deeper into data development by browsing our blogs…
How ProCogia Reduced Azure Storage Costs by 95% for a Marine & Logistics Client
Introduction Storage costs in cloud environments can quickly become unmanageable. For a Marine and Logistics client, ProCogia implemented a Delta Lake vacuuming strategy on Azure,
Ensuring Code Quality in Cloud Transformations: A Marine Industry Case Study
Introduction At ProCogia, we prioritize quality and resilience in cloud transformations. For a marine industry client, we transitioned transformation code from local development to Azure
Counterfeit Detection and Legal Compliance: Optimizing Data Analytics with Redshift
Information This case study showcases the expertise of ProCogia’s Senior Data Analytics Consultant, Jens Sommerfeld, in optimizing data analysis with Redshift to support counterfeit detection
Get in Touch
Let us leverage your data so that you can make smarter decisions. Talk to our team of data experts today.
