ProCogia enabled a T-Mobile
team to reduce their
Amazon S3 costs
40%
Reduction in Amazon S3 Costs
19%
Immediate Cost Savings After Implementing S3 Lifecycle Policies
Company Information
T-Mobile’s data team faced challenges with managing their rapidly growing Data Lake, which had reached 1.87 PB and was expanding by 0.15 PB monthly. This growth was leading to escalating Amazon S3 costs, with a projected year-on-year increase of 32%. To address this, ProCogia was brought in to implement cost-saving measures and improve data management efficiency.
The Challenge
T-Mobile needed help managing their expanding Data Lake. The growing volume of data correlated with rising Amazon S3 expenses, with the S3 costs increasing by 7% in the first month of the year. This trend was set to compound, potentially causing a significant impact on the team’s budget. The client required quick and effective solutions to reduce their cloud storage costs.
Our Approach
ProCogia consultants engaged with key stakeholders to understand the Data Lake's contents, how this data was written, and how it was consumed by their downstream customers.
An object-deletion policy was implemented, first bulk deleting all unnecessary objects and then applying S3 'Lifecycle Policies' in order to automatically delete objects once they passed an agreed retention period.
For objects that could not be deleted, S3 'Lifecycle Policies' were applied to transition infrequently-accessed objects from hotter to colder storage classes (S3 'Infrequent Access' and 'Glacier Instant Retrieval'). Other objects with undetermined access patterns were managed & transitioned to the S3 'Intelligent Tiering' storage class.
The daily Elastic Map Reduce jobs were updated to use the 'EMR File System' (EMRFS). This replaced the existing solution of HDFS and the 'S3a' client which can write a huge volume of temporary files to S3 as part of each ETL job.
The Results
Initial Cost Reduction
Following implementation of new S3 Lifecycle Policies in the third month of the year, the monthly S3 bill was immediately down 19% (versus the forecasted bill without intervention).
Further Savings
Following the upgrade of EMR jobs to EMRFS (and S3 Intelligent Tiering beginning to self-manage object classes), the monthly S3 bill was further reduced by an additional 26% (a 40% total saving for the month versus the forecasted bill without intervention).
Conclusion
ProCogia’s strategic interventions, including the application of S3 Lifecycle Policies and the optimization of EMR workflows, led to a substantial 40% reduction in T-Mobile’s monthly S3 costs. By implementing automated solutions and transitioning data to more efficient storage classes, T-Mobile successfully curbed the escalating cloud expenses and ensured more sustainable data management practices moving forward.
Explore more stories
Dig deeper into data development by browsing our blogs…
How QueryIQ is Revolutionizing Financial Reporting
How QueryIQ is Revolutionizing Financial Reporting Introduction In the high-octane realm of accounting and financial services, having immediate access to precise data isn’t just
How a Leading Retailer Boosted Sales by 18% with QueryIQ
How a Leading Retailer Boosted Sales by 18% with QueryIQ 18% Increase in Sales Introduction In today’s fast-paced retail landscape, the ability to make
Improving Weekly ETL Dependencies and Redshift Cluster Runtime for a Global Retailer
80% Reduction in Run Time Information This case study showcases the expertise of ProCogia’s Senior Data Analytics Consultant, Jens Sommerfeld, in optimizing ETL processes
Get in Touch
Let us leverage your data so that you can make smarter decisions. Talk to our team of data experts today or fill in this form and we’ll be in touch.
