ProCogia migrated legacy telemetry solution to a new Azure data lake platform

Company Information

A global shipping giant faced the challenge of modernizing its vehicle telemetry data management system. The existing system, based on an SQL database, was becoming increasingly inadequate for the company’s evolving data needs. ProCogia was tasked with migrating this legacy system to a new, more flexible, and scalable Azure data lake platform, leveraging the Delta format for data storage. This transition was aimed at enhancing data accessibility, processing efficiency, and integration with the company’s broader data platform.

The Challenge

The primary challenge was to replace the outdated telemetry solution with a system that not only accommodated the complex nature of the telemetry data, including vessel sensor readings like engine RPM, fuel burn rates, and GPS locations but also aligned with the company’s current data storage and processing protocols. The solution needed to be cost-effective, scalable, and capable of handling the intricacies of XML-formatted telemetry data efficiently.

Procogia’s Approach

ProCogia collaborated with leaders in the client’s organization to build the solution. Our process included:

The client had asked ProCogia to help replace their existing vehicle telemetry solution, which persisted data in an SQL database. This solution was to be deprecated in favour of one that aligned with their existing Data Platform, where data was stored using the Delta format.

ProCogia built a new Azure Data Factory pipeline to facilitate procurement of the raw XML formatted telemetry data and convert it to a tabular (parquet file) form.

The telemetry comprised of vessel sensor reading which included engine RPM, fuel burn rates and GPS locations amongst others.

The pipeline employed an Azure function app, again written in Python/Pandas. With each row of XML delivering a different timestamped sensor reading, data processing was non-trivial and called for both time-based grouping and pivoting.

Pandas was chosen as it provided a natural and easy solution to wrangling XML data. Furthermore, when deployed as a function app, Azure offers a low cost, pay-per-use serverless solution, perfect for batch-based ingestion pipelines.

A PySpark script, deployed into a Databricks cluster completed the curation process by persisting the data as Delta formatted parquet files. ProCogia demonstrated industry best practices by developing and unit testing PySpark code in a local development environment. To date all Spark code was built interactively in notebooks on a running cluster.

The Results

ProCogia were able to rebuild a new telemetry ingestion mechanism from the ground up that delivered data straight into the client’s Data Lake hosted in Azure, making it available as alongside other operational data.efficient size standardization solution and an extensive library of industry-standard size ranges for over 1.2 Million Shelf Keeping Units (SKU).

Using Databricks, we were able to materialize results from complex joins, preparing metrics for rapid and easy consumption at the presentation layer.

Our solution helped the client gain a deeper understanding of their KPIs including distance travelled, RPM (Revs per minute), fuel consumption rates, journey duration and GPS tracking.

Services Used

Data Engineering

We partner with all major cloud providers, allowing us to adopt a data-agnostic approach focused on delivering tailored game-changing solutions.

Data Consultancy

We provide data consultancy to organizations to optimize your investment in people, processes, and technology. This is typically through data strategy engagements, roadmaps, transformations, and independent technology advice.

Conclusion

ProCogia’s migration of the legacy telemetry solution to a new Azure data lake platform represents a pivotal enhancement in data management for the global shipping company. This project not only streamlined data ingestion and storage but also facilitated deeper insights into operational KPIs, driving efficiencies and strategic decision-making. ProCogia’s adept use of cloud technologies, combined with its commitment to best practices in data processing and analysis, has positioned the client for continued success in a data-driven future, underscoring ProCogia’s role as a key partner in digital transformation.

Explore more stories

Dig deeper into data development by browsing our blogs…

Custom Logging System for Plumber API and Shiny Applications A futuristic dashboard showing a central server connected to multiple API endpoints. Lines representing data flow between endpoints show user actions and responses, with real-time monitoring screens and storage for log files. On the left, warning icons indicate error tracking, while on the right, a clock symbolizes automated log management. The background has code snippets and data logs, creating a technical, blue-toned atmosphere that reflects system performance monitoring and troubleshooting.

Custom Logging Plumber API

Introduction Our client, a leading investment consulting firm, aimed to improve system monitoring, troubleshooting, and reliability by implementing a logging system that tracks user actions

Watch now ->

Get in Touch

Let us leverage your data so that you can make smarter decisions. Talk to our team of data experts today or fill in this form and we’ll be in touch.

Author