ProCogia migrated legacy telemetry solution to a new Azure data lake platform
Company Information
A global shipping giant faced the challenge of modernizing its vehicle telemetry data management system. The existing system, based on an SQL database, was becoming increasingly inadequate for the company’s evolving data needs. ProCogia was tasked with migrating this legacy system to a new, more flexible, and scalable Azure data lake platform, leveraging the Delta format for data storage. This transition was aimed at enhancing data accessibility, processing efficiency, and integration with the company’s broader data platform.
The Challenge
The primary challenge was to replace the outdated telemetry solution with a system that not only accommodated the complex nature of the telemetry data, including vessel sensor readings like engine RPM, fuel burn rates, and GPS locations but also aligned with the company’s current data storage and processing protocols. The solution needed to be cost-effective, scalable, and capable of handling the intricacies of XML-formatted telemetry data efficiently.
Procogia’s Approach
The client had asked ProCogia to help replace their existing vehicle telemetry solution, which persisted data in an SQL database. This solution was to be deprecated in favour of one that aligned with their existing Data Platform, where data was stored using the Delta format.
ProCogia built a new Azure Data Factory pipeline to facilitate procurement of the raw XML formatted telemetry data and convert it to a tabular (parquet file) form.
The telemetry comprised of vessel sensor reading which included engine RPM, fuel burn rates and GPS locations amongst others.
The pipeline employed an Azure function app, again written in Python/Pandas. With each row of XML delivering a different timestamped sensor reading, data processing was non-trivial and called for both time-based grouping and pivoting.
Pandas was chosen as it provided a natural and easy solution to wrangling XML data. Furthermore, when deployed as a function app, Azure offers a low cost, pay-per-use serverless solution, perfect for batch-based ingestion pipelines.
A PySpark script, deployed into a Databricks cluster completed the curation process by persisting the data as Delta formatted parquet files. ProCogia demonstrated industry best practices by developing and unit testing PySpark code in a local development environment. To date all Spark code was built interactively in notebooks on a running cluster.
The Results
ProCogia were able to rebuild a new telemetry ingestion mechanism from the ground up that delivered data straight into the client’s Data Lake hosted in Azure, making it available as alongside other operational data.efficient size standardization solution and an extensive library of industry-standard size ranges for over 1.2 Million Shelf Keeping Units (SKU).
Using Databricks, we were able to materialize results from complex joins, preparing metrics for rapid and easy consumption at the presentation layer.
Our solution helped the client gain a deeper understanding of their KPIs including distance travelled, RPM (Revs per minute), fuel consumption rates, journey duration and GPS tracking.
Services Used
Data Engineering
We partner with all major cloud providers, allowing us to adopt a data-agnostic approach focused on delivering tailored game-changing solutions.
Data Consultancy
We provide data consultancy to organizations to optimize your investment in people, processes, and technology. This is typically through data strategy engagements, roadmaps, transformations, and independent technology advice.
Conclusion
ProCogia’s migration of the legacy telemetry solution to a new Azure data lake platform represents a pivotal enhancement in data management for the global shipping company. This project not only streamlined data ingestion and storage but also facilitated deeper insights into operational KPIs, driving efficiencies and strategic decision-making. ProCogia’s adept use of cloud technologies, combined with its commitment to best practices in data processing and analysis, has positioned the client for continued success in a data-driven future, underscoring ProCogia’s role as a key partner in digital transformation.
Explore more stories
Dig deeper into data development by browsing our blogs…
Implemented Bookmarking Apps
A leading investment consulting firm faced challenges in managing and sharing complex data analyses through their internal tools developed with Shiny. As the firm sought
Optimizing and Scaling RESTful API for an Investment Consulting Firm
Optimizing and Scaling RESTful API for an Investment Consulting Firm Company Information A prominent investment consulting firm sought to elevate the performance and maintainability
Low-code solutions using Azure & Snowflake
ProCogia implemented low-code solutions using Azure Data Factory and Snowflake for the Performance Data Science team of a Leading Sports Organization
Get in Touch
Let us leverage your data so that you can make smarter decisions. Talk to our team of data experts today or fill in this form and we’ll be in touch.
