Data Digest: lakeFS, One Big Table, and Data Usability
Welcome to this week’s edition of our data newsletter. This week, we delve into the world of data lakes with lakeFS, explore the concept of One Big Table (OBT), and discuss how to build better data products. These articles provide a comprehensive view of the current trends and practices in data management and usability. Let’s dive in!
lakeFS is an open-source project that brings software engineering best practices to data engineering. It provides version control over the data lake, using Git-like semantics to create and access versions. If you’re familiar with Git, you’ll feel right at home with lakeFS. It supports managing data in AWS S3, Azure Blob Storage, Google Cloud Storage (GCS), and any other object storage with an S3 interface. It integrates seamlessly with popular data frameworks such as Spark, Hive Metastore, dbt, Trino, Presto, and many others.
One Big Table (OBT) is a data modeling technique where all the data attributes needed for analytics are stored into one, wide, denormalized table. This approach contrasts with traditional data models such as star schema and snowflake. The article provides a deep dive into the concept of OBT, its use cases, and its advantages and disadvantages.
Data Usability: How to Build Better Data Products
This article introduces a ‘data to impact framework,’ which illustrates how humans leverage data products — originating from AI and data pipelines — to transform raw data into tangible outcomes. It emphasizes the importance of data usability in the construction of better data products. By incorporating data usability within your organization’s data strategy, you can initiate the journey towards augmenting the impact derived from data-driven practices.
Interesting Jobs and Opportunities
Senior Data Analyst- Masco (Remote)
Data Engineer - Amur (Remote)
Sr. Data Engineer - Amplify (Remote)
Upcoming Events
Implementing end-to-end CDC in the open data lakehouse - OneHouse
Thu, Jan 25, 2024, 12:00 PM - 1:00 PM
How to get Data Skilled in 2024 - Adaptive US Inc.
Wed, Jan 17, 2024, 11:00 AM - 12:00 PM
Not too Serious … 🤣
As we conclude this week's data expedition, I invite you to explore, engage, and share your thoughts on this enriching journey. Stay tuned for more captivating insights in the weeks to come.
Cheers to a data-filled week ahead!