Company Information
T-Mobile’s data team faced challenges with managing their rapidly-growing Data Lake, which had reached 1.87 PB and was expanding by 0.15 PB per month. This growth was driving an acceleration in Amazon S3 costs, with annual expenditure forecasted to climb by 32%. To address this, ProCogia was trusted to implement cost-saving measures and improve data-management efficiency.
The Challenge
T-Mobile needed help managing its expanding Data Lake. The growing volume of data correlated with rising Amazon S3 expenses, with S3 costs increasing by 7% in the first month of the year. This trend was set to compound, potentially causing a significant impact on the team’s budget. The client required quick and effective solutions to reduce its cloud storage costs.
Our Approach
ProCogia consultants engaged with key stakeholders to understand the Data Lake's contents, how this data was written, and how it was consumed by T-Mobile's downstream customers.
An object-deletion policy was implemented, first bulk deleting all unnecessary objects and then applying S3 'Lifecycle Policies' in order to automatically delete objects once they passed an agreed retention period.
For objects that could not be deleted, S3 'Lifecycle Policies' were applied to transition infrequently-accessed objects from hotter to colder storage classes (S3 'Infrequent Access' and 'Glacier Instant Retrieval'). Other objects with undetermined access patterns were managed & transitioned to the S3 'Intelligent Tiering' storage class.
The daily Elastic Map Reduce jobs were updated to use the 'EMR File System' (EMRFS). This replaced the existing solution of HDFS and the 'S3a' client which can write a huge volume of temporary files to S3 as part of each ETL job.
The Results
Initial Cost Reduction
Following implementation of new S3 Lifecycle Policies in the third month of the year, the monthly S3 bill was immediately down 19% (versus the forecasted bill without intervention).
Further Savings
Following the upgrade of EMR jobs to EMRFS (and S3 Intelligent Tiering beginning to self-manage object classes), the monthly S3 bill was further reduced by an additional 26% (a 40% total saving for the month versus the forecasted bill without intervention).
Conclusion
ProCogia’s strategic interventions, including the application of S3 Lifecycle Policies and the optimization of EMR workflows, led to a substantial 40% reduction in T-Mobile’s monthly S3 costs. By implementing automated solutions and transitioning data to more efficient storage classes, T-Mobile reduced and stabilized its cloud expenses, while future-proofing its data management strategy.
