Notes

Spark Apache Spark is a fast, in-memory data processing engine. APIs in Scala, Java, Python & R. Spark on Apache Hadoop YARN enables deep integration with Hadoop and other YARN enabled workloads in the enterprise. Spark powers a stack of high-level tools including Spark SQL, MLlib for machine learning, GraphX and Spark Streaming.

Spark History

• 2009: Hadoop Subproject, by AMPLab (UCLA)
• 2010: Open Source
• 2013: Apache Project
• 2016: Spark 1.6.2 & 2.0.0
• 2018: Spark 2.3.x
• 2018: Spark 2.4.x

Unified Framework

Spark scales well Some examples of real usage:

Spark Performance

What makes Spark fast?

How Spark works The Driver program contains all the instructions that Spark must execute (for example: read a file, aggregate the data, save the aggregated data to another file). In the Driver program the programmer creates the SparkSession object that takes care of:

• Connecting to the Spark cluster (it connects to the cluster manager).
• Sending the program to the cluster and make it execute.

Spark have different cluster managers:

• It can use YARN as cluster manager if Spark lives on the same cluster as Hadoop.
• It can use another Apache cluster manager (Mesos).
• Spark itself contains a cluster manager tool (standalone cluster mode).

Once the Driver program is connected to the cluster manager it sends lines of code that contain the Spark commands. The Worker Nodes (generally one for each physical node of the cluster) receive the commands and allocate one or more Executors. The Executor is a JVM service that takes in charge:

• The creation of one or more tasks that physically execute the Spark instructions.
• The allocation of enough memory to execute each task.

Executors in different Worker Nodes can exchange data if needed (Data shuffle operations). Shuffles occur when the Spark program:

• Aggregates data (data with the same aggregation key may be located in different Worker Nodes).
• Sorting data.

Architecture

Spark Distributions The Apache Spark distribution is relatively easy to install. It works on all the operating systems. There's another Spark distribution: Databricks that also provides a cloud version of Spark. Cloud services:

• Microsoft HDInsight
• Google Cloud
• IBM Blumix

Connecting to the Spark cluster The master parameter for a SparkContext determines which cluster to use:

Resilient Distributed Datasets (RDD) Resilient Distributed Datasets (RDD) are the primarily abstraction in Spark - a fault-tolerant collection of elements that can be operated on in parallel.

• An RDD is comprised of partitions.
• Each partition is managed by an executor (generally one executor per node).
• Each executor can manage more than one partition of the same RDD.

RDD Examples There are different types of RDD. For example we can have:

• RDD of strings (created when we read a file).
• RDD of pairs (key/value pairs, programmatically created).

How Spark works The RDD is an immutable collection of objects that is partitioned and distributed across multiple physical nodes of a YARN cluster and that can be operated in parallel. There are currently two types:

• Hadoop (or other FS) datasets - Typically, RDDs are instantiated by loading data from a shared file system, HDFS, HBase, or any data source offering a Hadoop InputFormat.
• Parallelized collections - take an existing Scala/Python collection and run functions n it in parallel. Once an RDD is instantiated, you can apply a series of operations. All operations fall into one of two types: transformations or actions.
• Transformation operations create new datasets from an existing RDD and build out the processing DAG that can then be applied on the partitioned dataset across the YARN cluster.
• An Action operation, on the other hand, executes DAG and returns a value. DAG = Directed Acyclic Graph. It's the execution workflow of a Spark program.

Example

DAG, Stages & Shuffle

Spark SQL Spark SQL is a Spark module for structured data processing. It provides a programming abstraction called DataFrames and can also act as distributed SQL query engine. A DataFrame is a distributed collection of data organized into named columns.

• It is conceptually equivalent to a table in a relational database or a data frame in R/Python, but which richer optimizations under the hood.

DataFrames A distributed collection of data organized into named columns. Similar to RDD with schema. Conceptually equivalent to tables in relational database, or to DataFrames in R/Python. With domain-specific functions designed for common tasks:

• Metadata
• Sampling
• Project, filter, aggregation and join.
• UDFs.

Creating DataFrames from data sources A Spark data source can read in-data to create DataFrames, which has a schema that Spark understands. Examples include: JSON files, JDBC source, Parquet (*).

(*) Parquet is an optimized file format. It contains both data and metadata (column names and data types). It stores data in an efficient columnar format.

DataFrame API DataFrames provide a domain-specific language for structured data manipulation in Scala, Java and Python.

Dataframe registered as a table A DataFrame can be registered as a table that can then be used in SQL queries.

Persistent Tables DataFrames can also be saved as persistent tables into Hive metastore using the saveAsTable command.

• An existing Hive deployment is not necessary to use this feature. Spark will create a default local Hive metastore (using Derby) for you. SaveAsTable will materialize the contents of the DataFrame and create a pointer to the data in the Hive metastore. Persistent tables will still exist even after your Spark program has restarted, as long as you maintain your connection to the same metastore.

Spark Streaming

• Streaming Data is data that is continuously generated by a data source. Such data should be processed incrementally.
• Spark Structured Streaming: enables scalable, high-throughput, fault-tolerant stream processing live data streams.
  • Built on the Spark SQL engine.
  • You can express your streaming computation the same way you would express a batch computation on static data.
  • The Spark SQL engine will take care of running it incrementally and continuously and updating the final result as streaming data continues to arrive.

Streaming

Stream Exercise

• Loading sample data.
• Perform some interactive analysis (on the static data).
• Read the files as a stream.
• Do the same analysis as in point 2.
• See the streaming result.

Spark GraphX Graphs

• The graph data structure is comprised of:
  • Nodes are the entities in the graph.
    • They can hold any number of attributes (key-value pairs) called properties.
    • Nodes can be tagged with labels, representing their different roles in your domain.
  • Relationships provide directed, named, semantically-relevant connections between two nodes.
    • A relationship always has a direction, a type, a start node, and an end node.
  • Such a graph is called Property Graph.

Graph X

• Graph X is a new component in Spark for graphs and graph-parallel computation.
• At a high-level, GraphX extends the Spark RDD by introducing a new Graph abstraction:
  • A directed multi graph with properties attached to each vertex and edge.
• In addition, GraphX includes a collection of graph algorithms and builders to simplify graph analytics tasks.

Built-In Algorithms

GraphX Exercise

• Load the library into the Spark environment.
• Create the graph data structures.
• Basic calculations.
• Built-in Algorithms.

Spark Machine Learning Typical ML Process

• Understand the data
• Split the data into training set and test set
• Prepare the data
  • Data cleansing
  • Specific data preparation for the algorithms
    • Normalization
    • Conversion of categorical variables to numerical variables
• Train the algorithm using the training set
• Test the model on the test set

Spark MLlib is Spark's machine learning library. Its goal is to make practical machine learning scalable and easy. MLlib tools:

• ML Algorithms: common learning algorithms such as classification, regression, clustering, and collaborative filtering.
• Featurization: feature extraction, transformation, dimensionality reduction and selection.
• Pipelines: tools for constructing, evaluating and tuning ML workflows.
• Utilities: linear algebra, statistics, data handling, etc.

Spark ML Algorithms Regressions:

• Linear Regression
• Generalized Linear Regression
• Decision Tree Regression
• Random Forest Regression
• Gradient-boosted Tree Regression
• Survival Regression
• Isotonic Regression

Classifications:

• Logistic Regression
• Decision Tree Classifier
• Random Forest Classifier
• Gradient-boosted Tree Classifier
• Multilayer Perceptron Classifier
• Linear Support Vector Machine
• One-vs-Rest Classifier (a.k.a One-vs-All)
• Naïve Bayes

Clustering & Other Supervised Algos:

• K-Means
• Latent Dirichlet Allocation (LDA)
• Bisecting K-Means
• Gaussian Mixture Model (GMM)

Recommendation System:

• Alternating Least Squares (ALS)
• FP-Growth
• PrefixSpan

Spark ML Pipeline ML Pipeline API lets Spark users quickly and easily assemble and configure practical distributed Machine Learning pipelines by standardizing the APIs for different Machine Learning concepts. The nice thing of using Spark ML is that the ML Dataset is simply a DataFrame.

Pipeline Components A transformer is a ML Pipeline component that transforms a DataFrame into another DataFrame:

• OneHotEncoder
• Binarizer
• Tokenizer

An estimator is an abstraction of a learning algorithm that fits a model on a dataset.

• An estimator fits a model to the input DataFrame and ParamMap to produce a Transformer (a Model) that can calculate predictions for any DataFrame - based input datasets.

An evaluator is a transformer that takes DataFrames and computes metrics indicating how good a model is.

• BinaryClassificationEvaluator
• MulticlassClassificationEvaluator
• RegressionEvaluator

Spark ML Exercise Example 1 - Logistic Regression

• Creating a toy dataset
• Creating the pipeline
• Training the model
• Evaluating against the test set

Example 2 - Linear Regression

• Loading sample data
• Exploring the data
• Creating the pipeline
• Training the model
• Evaluating against the test set.