Why? 

Not only does fast growth in multiple countries bring impressive business results, but it also rapidly increases data volume. Pieces of valuable raw data were continuously flowing down into the Client’s servers. There were many ideas on how they could be used by the company's Analysis and Strategy team. Such conditions led to frequent changes in the platform, so most solutions we read-hoc. The lack of automation, standardisation, and proper monitoring techniques was getting more and more problematic. That’s when Datumo stepped in - the challenge was to stabilise and replace some solutions with well-fitted GCP services, share Cloud and data engineering best practices and prepare generic data pipelines to seamlessly handle future analytical needs.

What & how?

The first step was to introduce the core tool for the solution - Apache Airflow. It replaced a bunch of crontab jobs with automated and reliable service. Due to Client requirements deployment on GKE was used instead of Cloud Composer. Datumo experts prepared pipelines utilising Airflow, BigQuery, GCS and other services to process and handle the data. Data backups were configured to ensure that the platform was prepared for any eventuality. Monitoring alerts were added to draw attention to such issues as quickly as possible. Chosen solutions and implementation were generic, allowing us to easily introduce new data pipelines addressing complicated business requirements.

Benefits 

With improvements introduced by Datumo, creating and managing core data pipelines became reliable and easy - for most common tasks providing a few configuration values is sufficient to get a robust, multi-service solution out of the box. All data reaches BigQuery uninterruptedly, hence the Client’s analysts can perform complex BI tasks, and developers can focus on improving the platform. With monitored orchestration based on Apache Airflow, all processes are properly executed. In case of any issues, the supporting team receives transparent and clear information allowing it to quickly detect the cause.