Notes

Data Lake "A data lake is a collection of storage instances of various data assets additional to the originating data sources. These assets are stored in a near-exact, copy of the source format. The purpose of a data lake is to present an unrefined view of data to only the most highly skilled analysts, to help them explore their data refinement and analysis techniques independent of any of the system-of-record compromises that may exist in a traditional analytic data store (such as a data mart or data warehouse)".

Raw Data

Processed Data The data lake also contains a vast amount of processed (and/or semi-processed) data.

Technologies The Data Lake is mostly based on Hadoop & Spark.

Users

Data Lake Rezzani's Definition:

• A data lake is a repository that holds:
  • A vast amount of raw data in its native format, including structured, semi-structured and unstructured data.
  • A vast amount of processed (and/or semi-processed) data.
• The data structure and requirements are not defined until the data is needed (schema on read).
• The data lake provides data for as reporting, analytics and machine learning tasks.
• The data lake is mostly based on the Hadoop & Spark platforms.

Low Cost Repository

Four steps of the Data Lake adoption

• Staging area & raw-data zone. At the first level, the data lake can separate collect data from operational systems and servers as a low-cost, scalable, staging environment.
• Data Science Environment. As a second step, companies may start to use the data lake as a platform for experimentation. The Data Lake becomes the place where the data scientist run their machine learning experiments.
• Offloading system for data warehouses. As a further step, data lakes are integrated with existing Data Warehouses. Taking advantage of the low storage costs associated with a data lake, companies can store "cold" (rarely used) data.
• Main component of the information system. The data lake becomes a core part of the data infrastructure, as it's involved in most of the data flows.

Traditional Business Intelligence vs Data Lake

Data Lake Architecture: Data Flows Data from the traditional sources can still be processed by the ETL processes, but some (or all) the ETL procedures could moved to the data lake (DL).

• So the DL becomes the repository of raw data and the place where some ETL transformations happens. Data from the new sources go directly to the DL.
• Those data can be processed or used in complex analytical tasks (like machine learning).
• And, once we get the results, they can be moved to the data warehouse.

Data Lake Architecture: the DWH Role The data warehouse still exists. It contains:

• The "hot" data coming from the ETL processes (whether they grab the data directly from the sources or from the DL).
• The "distilled" data coming from the analytical processes that run over the data lake. The data warehouse allows for low-latency (interactive) queries. It's part of the Business Intelligence system that still carries out its job in the data lake era. Keeping the data warehouse means preserving the querying performances and all the reporting systems that are built on top of it.

Data Lake Architecture: the DL role The data lake contains:

• (All) the raw data, both structured and unstructured.
• The final processed data (which are structured)
  • Some of those data is pushed to the data warehouse (the hot data).
  • The "cold" data are stored in the data lake and are available with a longer latency. It eventually takes care of all the data transformations (ETL + Analytical tasks). In the DL we can store more data: more details + greater historical depth. Supplies data:
• To the data warehouse.
• To the Data Scientists. The data scientists can use both the raw and the processed data to build both descriptive reports and predictive models.
• To the applications. The data lake could provide data services for corporate applications.

Data Lake Architecture: Data Distillation The data lake allows more detailed and more complex analysis on a larger amount of data. Often the results of the computations are small enough to be transferred to the Data Warehouse. So, the results are injected in the Business Intelligence workflow.

Critical Success Factors for the Data Lake A big data lake will take whatever data you give it, and that's going to be a problem. How to address it?

• Data Governance
• Data Quality Process Many users with different profiles and needs may access the Data Lake. How can we make the usage easy for any user profile?
• Metadata management (part of the Data Governance)
• User skills improvement (i.e., training)
  • Data Scientists
  • End Users

Data Governance Policy

• Data Sources mapping (applications, input services)
• Data Flows mapping: How data is loaded into the Data Lake?
• Output services mapping:
  • Data Warehouse
  • Applications
  • Output Data Services
• Metadata Management: the Metadata make it easy to search for data and to interpret the content of the data lake.

Data Quality Process The data quality rules should be applied on the output data. If we apply data quality filters on the input data, we would prevent some data to enter the data lake.

• Data that have a medium quality level could be useful for certain applications. The best way to proceed is putting a "quality label" to the data and let the final user decide whether or not to use it.