Developed a reliable data platform that processes and transforms raw data from multiple internal software systems (CLI, Accordex, UKG, Paylocity, Oracle) and external sources (public data, competitor data, prices of goods). This platform creates a unified source of truth, ensuring data consistency and accessibility across the organization.
By implementing robust data cleaning and transformation processes, we enable various stakeholders—such as management, operations, and finance—to make informed decisions based on accurate and up-to-date information. Ultimately, this empowers the organization to optimize logistics operations, enhance strategic planning, and improve overall efficiency.
Before Implementation
Delays in Delivery. New pipelines or modifications were slow, especially with Oracle Custom Functions.
Poor Modeling Practices. Best practices for data modeling were not followed.
Communication Issues. Ineffective and delayed communication between the Diamond Line team and the previous Azure consultant.
Key Improvements
Team Collaboration. Enhanced communication and collaboration through daily Slack updates and weekly meetings.
Documentation. Continuous documentation updates following best practices.
Pipeline Enhancements. Improvements in data pipelines, including parameterization, incremental loading, and API analysis.
Project Overview
1.
Pipelines Improvements
Pipelines were improved for Oracle with JSON parsing, date parameterization, and incremental load. Accordex saw enhancements in file ingestion and duplication checks.
2.
Data integration
Data centralization involved consolidating data from internal systems (CLI, Accordex, UKG, Paylocity, Oracle) and external sources (competitor data, product prices) into a single source.
3.
Data Transformation
Data transformation involved applying modeling techniques for data cleaning and unification, resulting in a well-organized data structure that facilitates the creation of metrics and visualizations.
4.
Dashboarding
Two dashboards were developed: the Shipments dashboard for delivery and driver performance metrics, and the Operational dashboard summarizing key efficiency KPIs.
Oracle
Power BI
Azure Data Factory
SQL
Technologies used
Cody Clarke
Director of IT - Diamond Line Logistics
"This project has transformed our operations, streamlining processes and providing clear insights into delivery performance, which enhances decision-making and resource optimization."
