Building a Scalable and Agile Modern Data Architecture: Lessons from Deliverect

The Growing Data Challenge

Data is growing exponentially, with organisations generating vast amounts of data. However, research shows that many companies are struggling to extract measurable value from their data. This highlights the need for a robust data platform able to effectively leverage data and turn it into actionable insights.

Before delving into the technical aspects, the alignment between the business and data strategies is crucial. There must be an evaluation of whether to build or buy solutions for your data architecture. While off-the-shelf solutions like Customer Data Platforms (CDPs) can provide interesting analytics, building a customised data architecture allows you to get the full potential out of your data.

Essential Components of a Modern Data Architecture:

• Data Lakehouse

The concept of a data lakehouse combines the benefits of data warehouses and data lakes. It provides a unified approach to storing and querying data, leveraging technologies like Google Cloud BigQuery, and BigLake. By eliminating the need to move data between different storage solutions, you can streamline data management and access.

• Data Ingestion

Efficiently bringing data from various sources into the data lakehouse is essential. Google Cloud offers native connectors that simplify the data ingestion process. Leveraging these connectors reduces development and maintenance efforts. In cases where native connectors are unavailable, you can choose between building custom pipelines, using open-source frameworks or third-party platforms like Fivetran.

• Data Transformation and Orchestration

Transforming raw data into valuable insights requires robust data transformation and orchestration capabilities. Google Cloud’s BigQuery enables scalable data transformations, allowing it to handle semi-structured data efficiently. For orchestration, options like Dataform and DBT (Data Build Tool) offer powerful templating and automation features, each with its strengths and community support.

• Business Intelligence 

To address the challenges of bottlenecks and data chaos, there is a need for self-service analytics capabilities. Google Cloud’s Looker offers a unique architecture directly connecting to BigQuery, allowing users to explore and drill down into data without creating multiple cubes. This approach fosters consistent data definitions and empowers users with flexible and comprehensive analytics.

• AI and Machine Learning

Google Cloud’s AI capabilities provide organisations with out-of-the-box solutions for tasks like image recognition and natural language processing through Vertex AI. You can also leverage Google’s machine learning models trained on their data by utilising SQL-like queries. For more complex scenarios, you can choose to build their pipelines using frameworks like TensorFlow or Kubeflow.

• Reverse ETL

It allows you to leverage your data in various tools such as Salesforce, HubSpot, or Facebook. Transferring data from your data warehouse to a performance database, like a NoSQL, you achieve faster response times and real-time insights. Reverse ETL simplifies this process by facilitating the movement of data from your data warehouse to an application database or third-party integration in just three simple steps: connecting the source and destination, selecting relevant tables with filters, and mapping fields to the target system’s API. 

• Governance and Data Management

Ensuring data security, privacy, and governance is vital for maintaining control and compliance. Google Cloud’s Dataplex offerings provide unified data management layers that enable quality checks, data profiling, lineage tracking, and business glossaries. 

Takeaways

• Leverage Google’s wide ecosystem of Data & AI services
• KISS, don’t reinvent the wheel, use frameworks
• Balance development & maintenance vs licensing
• Don’t be too dependent on tooling, go for an open architecture
• Enable the people & processes, not only the technology
• Leverage the expertise of a trusted partner

Deliverect’s Data Stack

Challenges Faced and the Need for a Solution

Managing multiple brands across various aggregators and cloud kitchens proved to be a significant challenge for the company. With over 50 virtual brands operating in Kuwait, Dubai, and Qatar, the complexity of coordinating orders and streamlining kitchen operations became apparent. Additionally, the lack of centralised data management and collaboration hindered decision-making and impacted the overall customer experience.

Deliverect, a comprehensive solution for order management, played a vital role in helping the company address their operational difficulties. By integrating with multiple aggregators and providing direct order transmission from customers to kitchens, Deliverect significantly reduced overall kitchen time and improved order accuracy. This streamlined process allowed the team to focus on other operational aspects and enhance the customer experience.

Menu Engineering and Better Decision-Making

Deliverect provides support in menu engineering. By leveraging data insights and analytics, the company gained a deeper understanding of the impact of menu changes across different aggregators. This knowledge allowed them to make informed decisions to cater to their customers’ preferences, ensuring maximum satisfaction and revenue generation.

The company’s journey towards modernising its data management architecture began with a shift towards a hosted database. As the services expanded and more developers joined the team, the need for Google Kubernetes Engine (GKE) and separate environments became apparent. The company also recognized the importance of adopting a modern data stack to optimise data usage and address challenges arising from rapid development.

Implementing a Modern Data Warehouse

To overcome the challenges of scattered knowledge and ineffective collaboration, the company started building a modern data warehouse. This architectural change allowed for better data governance, improved collaboration, and enhanced discoverability of data. By adopting a common code base and implementing separate environments, the team achieved greater agility and ensured the scalability of their data operations.

The successful implementation of the modern data stack brought significant benefits to the company. The most notable impact was the elimination of time-consuming data searches and the ability to find data reliably. This enhanced data accessibility and reliability resulted in improved decision-making and productivity. Furthermore, the company witnessed a doubling of queries and dashboards that took less than 4 seconds to load, while queries taking more than 300 seconds were reduced to a minimum.

Future Focus and Expansion

With the solid foundation of its modern data stack in place, the company is now focusing on key areas for future growth. Data security and quality are top priorities as the company continues to scale. Automation of data quality auditing and improved data governance will ensure that data remains accurate, reliable, and compliant. The integration of acquired companies and expansion into machine learning and real-time data analysis is also on the horizon.