56% Reduction in Data Processing Costs

At Glance

• 56% decrease in data processing costs

• $170,000 USD in annualized savings

• Faster pipeline refresh times

• Simplified and scalable architecture

Industry: Retail - Jewellery

Tech Stack: Google Cloud Platform · dbt · Dagster · Looker

The Problem: Rising Processing Costs

Mejuri, a Canadian Retailer with more than 30 stores had already set up data processing and analytical processes with over five years of maturity. As complexity increases, so does the processing costs that were heavily impacting the Tech & Engineering budget expending +1000 USD per day on refreshing data. The client set several project objectives:

• Reduce data processing costs. 

• Reduce refresh time of slowest pipelines.

• Simplify the architecture

• Train data team to continue the efforts

The Solution: Data Transformation and BI layer refactoring

After an initial assessment to understand usage and patterns, we defined the solution in five workstreams: 

• Query refactoring: Applied SQL best practices to optimize the most expensive queries and reduce the data scanned.

• Incrementality: Added incremental refreshes for large fact tables to avoid full table reloads. 

• Partitioning & clustering: Implement partitioning and clustering strategies based on usage patterns. 

• Orchestration optimization: Reduce unnecessary refreshes and duplicated jobs. 

• BI dashboards improvements: Improve dashboards by refactoring queries and building aggregated models for faster performance.

  
  

Results

Outcome (in just 2 months): 

• 56% cost reduction in data processing costs (accounting multiple environments)

• $170k in annualized savings

• Improved data processing times

• Scalable data infrastructure

BigQuery Admin Console June Period for one of the environments

BigQuery Admin Console May Period for one of the environments

BigQuery console expanded for one of the environments