In this article, we delve deep into Spark 3's AQE framework, focusing on the coalesce and caching mechanisms of its shuffle partitions.

When delving into crafting a new and efficient Spark job, or optimising an existing one, multiple implementation and design choices may have a significant impact on the job’s performance. One of the most prominent aspects influencing a Spark job’s efficiency is the fundamental difference between actions and transformations.

Dive into the world of MLOps on Azure with our comprehensive guide. This article serves as a gateway to understanding the symbiotic relationship between building MLOps platform and Azure.

Today, we aim to share our firsthand experience with transitioning existing data platforms to GCP (Google Cloud Platform). Our narrative will not only highlight the advantages of cloud migration but also shed light on the potential challenges encountered throughout the process.

Dive into the success story of how Datumo utilized automation to accelerate recruitment processes, making it seamless and comfortable for all departments and employees. Explore the transformative power of our in-house solution and witness the remarkable time savings and enhanced productivity it brings to the table.

In the digital realm where data reigns supreme, managing cloud expenses is no longer just an option, it is absolutely vital. With this in mind, have you evaluated your cloud spending recently? Are you fully aware of where your resources are allocated and confident that no hidden costs are draining your budget? If these issues have crossed your mind, our recent webinar may hold the answers you seek.

Explore the dynamic realm of tech troubleshooting with our latest post comparing ChatGPT and Stack Overflow. Uncover the strengths of each platform and find the perfect ally for your tech challenges.

Big Data Engineers must stay up-to-date with the latest skills and trends. Based on our experience implementing many Big Data projects, this blog will explore some essential data engineering skills for 2024.‍

In this article we will present some key areas that affect the future of Big Data analytics, while focusing on such topics as cloud migration, the development of native data platforms for cloud environments, real-time data processing and more.

dataset-logger, our open-source tool for Apache Spark, allows tracking and analyzing how particular data records change during Spark job execution.