Notes

Data Types

• Structured Data
• Unstructured Data
• Semi-Structured Data

Without a structure (fields or columns). We can't conveniently put them into a table:

• Documents (Word, Excel, PowerPoint, PDF, ...)
• Images
• E-Mails
• Web 2.0 (blog posts, forum, wikis, ...)

Big Data Data that have one or more of the following features:

• Volume (Potentially huge)
• Velocity (Fast data)
• Variety (Structured, Semi-Structured, Unstructured).

A better definition:

• Data that we can't analyze using a traditional technology (like RDBMS)
• Data that could be analyzed using an RDBMS, but at a huge cost.

For huge amounts of data we need more than one machine: we need a distributed system that can split the data among its nodes and can perform distributed computations. For the unstructured data we need a system that works with non tabular data structures. For complex calculations we need a system that can handle (for instance) machine learning tasks in a distributed manner. Moreover, we need a tool that is easy to use (at least for the final user). Does such a system exist?

Big Data Tools We have two main tools in the big data panorama:

• Hadoop is a complete platform for Big Data.
• Spark is an high performance calculation engine.

Hadoop It is a distributed calculation system based on two components:

• HDFS: Hadoop Distributed File System.
• MapReduce: a distributed calculation framework.

Key Points:

• Open Source
• Distributed (Up to thousands of nodes)
• Fault Tolerant
  • Hadoop is built to run on commodity hardware.
  • The fault tolerance is guaranteed using replication.
• Scalable
  • Up to thousands of nodes.
• Can handle both structured and unstructured data.

Definitions

• Node = Single Server.
• Rack = Case with a bunch of Servers.
• Cluster = Collection of Servers that behaves as if it was a single machine. Can be made of one or more racks.

How HDFS works

Distribution

• Each file in the distributed file system is made of parts called blocks (generally 64 or 128 MB).
• A server, called Name Node, contains all the file system metadata (i.e., the list of all the files, the list if each block and their locations).

Replicas

• Each block is replicated (the default replication factor is 3, but it can be changed).
• Replication ensures the fault tolerance (at least until all the nodes with the replica of the same block fail).
• HDFS is rack aware: the replication ensures that even if an entire rack fails, all the data is still available.
• The Name Node is the master node in the Hadoop cluster.
  • It handles metadata operations (like creating a new file, renaming file, deleting a file).
  • The name node also takes care of the replication.
• The other nodes in the Hadoop cluster are called Data Nodes.
  • They are slave nodes.
  • Data nodes take care of the read and write operations.

Replication and Hadoop v3 The replication mechanism ensures a certain level of fault tolerance. But it requires more disk space to store the data (n times the actual file size, where n is the replication factor). In Hadoop 3.x the replication has been replaced by a more space efficient algorithm called "Erasure Coding" which reduces to 50% the Storage overhead:

• The EC algorithm adds some data to each block.
• That data is encoded in such a way that the system can use it to rebuild other blocks (in case of their unavailability.

Map Reduce MR is a distributed computation engine that works with data stored in HDFS. Although today we have other (and more efficient) engines in Hadoop, Map Reduce is the one from which all the other system have been derived.

How Map Reduce works Map Reduce only has a Java API: the programmer must split the problem in two phases:

• The Map phase, in which the data are prepared and "mapped" to a key-value structure. The "mapper" program output is a key-value data structure.
• The Reduce phase, where the actual calculation is performed. The "reducer" output is the final result that is saved to an HDFS file.

The system receives the mapper and the reducer programs and performs the following operations:

• It executed the mapper program on each node where the input file parts are stored.
• It sorts, shuffles and aggregates (puts the values of the same key together) the outputs of the mapper programs.
• It executes the reducer program on each node.

Word count is the first task we accomplish when performing a text analysis.

"The Apache Hadoop software library is a framework that allows for the distributed processing of large datasets across clusters of computers using simple programming models."

Map Reduce: Batch Workload Map Reduce has been designed for batch workloads: Computations on a huge amount of data, that complete in minutes or even hours. Map Reduce isn't suitable for interactive queries (like the ones we issue against an RDBMS, i.e., many queries on a small amount of data): for this type of workload we expect the query to return a result in seconds or even in tenths of a second. Map reduce is hard to program and most problems are very difficult to design with a Map Reduce logic.

Pig and Hive Pig and Hive are high level tools that simplify the usage of Map Reduce.
They provide an high level language:

• Pig Latin: procedural language whose instructions resemble the SQL keywords.
• HiveQL: an SQL dialect with implements the ANSI SQL syntax + some built-in functions.

They translate the programs in the high level language into Map Reduce Jobs. The main Hadoop distributions provide ODBC Drivers that allow the user to connect to hive like if it was an RDBMS: in this way all the Business intelligence & reporting tools can connect to Hadoop!

The Hadoop Ecosystem (simplified)

Core Service

• HDFS: Hadoop Distributed File System
• YARN: A resource negotiator, that takes care of allocating the resources for each Hadoop task.
• Map Reduce, Tez and Spark: the distributed calculation Engines.

Data Services

• Hive: the SQL interface to Hadoop. It translates HiveQL (SQL) programs into Map Reduce, Tez or Spark programs.
• Pig: another service that translates Pig Latin programs into Map Reduce, Tez or Spark programs.
• Hbase: the NoSQL columnar database that only works inside Hadoop.
• Storm & Flume: data ingestion tools used to read data that is produced in streams. (e.g., servers logs, tweets, sensors data from IOT devices, ...)
• Sqoop: it's a sort of ETL tool that acts as an interface between the RBDMS and Hadoop. By means of Sqoop we can import data from a relational database to Hadoop or export data from Hadoop to an RBDMS.

Operational Services

• Ambari: web tool used to manage the Hadoop cluster.
• Oozie: workflow tool that the developer can use to orchestrate the execution of different jobs (Map Reduce, Hive, Pig, Sqoop, Flume, Storm, ...).
• Zookeeper: it shares and keep synchronized the configuration files for each application.

Analytical tools Tools that only work inside an Hadoop cluster:

• Giraph: a graph database tool.
• Mahout: a machine learning library.
• Kylin: OLAP tool on Hadoop. It exploit the columnar storage provided by HBase for saving aggregations and using them to support Hive queries.

Other distributed tools:

• Elastic Search: a very powerful search engine.
• Spark: a fast calculation engine.

Other analytic tools that can connect/interact with Hadoop.

• SAS: it provides components for interacting with Hadoop and Spark.
• KNIME: it provides components for interacting with Hadoop and Spark.
• R (The Microsoft R version is full integrated with Hadoop).

The Hadoop Distributions Installing a cluster using the Apache Hadoop distribution can be very hard. There are other Hadoop distributions that provide a semi-automatic installer that make it easy to deploy an Hadoop cluster. The most famous one is Cloudera. In 2018 the merged with Hortonworks, another company that provides a Hadoop distribution.

Hadoop pros & Cons Pros

• Can handle every data type
• Can store and compute huge amounts of data

Cons

• Complex system.
• Difficult to install and maintain.
• Slow for a certain kind of workloads (like interactive queries).

Other tools in the Big Data Scenario

• NoSQL Databases
• Distributed databases: they work on more than 1 server simultaneously
• Mostly open-source
• Horizontally scalable: adding one or more servers we increase the computation power and/or the available storage.
• Schema free: there's no table definition and can host data with variable structures.
• They have a simple programming interface ... but most of times they have SQL interfaces.

NoSQL Taxonomy

• Key-Value stores (DynamoDB), data are stored in key-value pairs.
• Document Oriented DB (MongoDB), the typical data structure is a JSON or XML fragment (both contain both metadata (the keys) and the data (the values)).
• Column Oriented DB (Cassandra), the main data structure is the column.
• Graph oriented DB, based on the graph concept (nodes + relationships).