Название: Delta Lake: Up and Running: Modern Data Lakehouse Architectures with Delta Lake (5th Early Release)
Автор: Bennie Haelen, Dan Davis
Издательство: O’Reilly Media, Inc.
Год: 2023-07-25
Страниц: 157
Язык: английский
Формат: epub
Размер: 10.2 MB

With the rapid growth of big data and AI, organizations are quickly building data products and solutions in an ad-hoc manner. But as these data organizations mature, it's apparent that their analysis and Machine Learning (ML) models are only as reliable as the data they're built upon. The solution? Delta Lake, an open-source format that enables building a lakehouse architecture on top of existing storage systems such as S3, ADLS, and GCS.

In this practical book, author Bennie Haelen shows data engineers, data scientists, and data analysts how to get Delta Lake and its unique features up and running. The ultimate goal of building data pipelines and applications is to query processed data and gain insights from it. You'll learn how the choice of storage solution determines the robustness and performance of the data pipeline, from raw data to insights.

Delta Lake brings capabilities such as transactional reliability and support for UPSERTs and MERGEs to data lakes while maintaining the dynamic horizontal scalability and separation of storage and compute of data lakes. Delta Lake is one the enablers for building Data lakehouses, an open data architecture that combines the best of data warehouses and data lakes.

The goal of this book is to provide experienced data practitioners with practical instructions on how to set up Delta Lake and start using its unique features. First, we’ll discuss why Delta Lake is an important tool for building modern enterprise data platforms and data science and AI solutions, followed by instructions on how to set up Delta Lake with Spark. Each of the subsequent chapters will walk you through the fundamental functions and operations of Delta Lake using step-by-step instructions and real-world examples.

With this book, you will:

Use modern data management and data engineering techniques
Understand how ACID transactions bring reliability to data lakes at scale
Run streaming and batch jobs against your data lake concurrently
Execute update, delete, and merge commands against your data lake
Use time travel to roll back and examine previous versions of your data
Build a streaming data quality pipeline following the medallion architecture