Big data project for
The Client is a global commercial real estate service company. As rental contracts are long-standing, the demand for office space has to be accurately predicted in order to maximise rental potential of their properties. The Client needs a tool to manage bookings and plan demand for all corporate locations for their tenants in the APAC region. As the legacy solution was built using Google Spreadsheets, integrating it with BigQuery and Google Cloud Platform was a natural choice.
What & How?
The legacy system used Google Spreadsheets for manual planning of supply and demand. It was difficult to maintain data quality. Report generation was restricted to a monthly basis. The proposed solution overcomes this limitation through planned automation resulting in weekly reporting which improved planning capabilities. Our goal was to automate the process and keep the data in one central location. The data is populated into Google Spreadsheets and then transferred into BigQuery tables using Plex technology. Next, it is orchestrated with Cloud Composer. Stored Procedures (routines) are run on top of the data to keep the quality and mappings in line with master data. When the ETL process ends, partitioned BigQuery tables are visualised with Data Studio for further analytics and reporting purposes. BigQuery is also an important input to machine learning algorithms which run as Docker containers inside Google Kubernetes Engine.
- reporting weekly instead of monthly improved planning capabilities
- automation of the demand planning process
- centralization of data which can be further analysed to improve business processes.
We provided consulting and shared best practices for the following areas:
- business requirements analysis and gathering
- introduction to DevOps practices (Terraform usage for resource management)
- introduction to GitOps practices and releases
- Cloud Composer orchestration and DAGs creation
- running legacy SQL routines inside DAGs and automating the process
- running quality checks on the data and notifying the relevant teams in case of any anomalies
- partitioning the data in BigQuery.
Get to know us, discover our interests, projects and training courses.