This article is a little unconventional. I have compiled it using information from my actual hard-copy notebook which I use to write notes for the classes I've attended at school. At the time of this writing, I am studying towards a Masters degree in Software Engineering. This article has definitions related to Data Science and Big Data Analytics, and could be helpful to Scholars or professionals.

Machine Learning Approaches

Supervised Learning

A machine learning approach defined by its use of labeled datasets to train algorithms to classify data and predict outcomes. Examples are Decision Trees and Machine Learning-Based Sentiment Analysis.

Unsupervised Learning

A type of machine learning in which the algorithms are provided with data that does not contain any labels or explicit instructions about what to do with it. The goal is for the learning algorithm to find structure in the input data on its own. Examples of unsupervised learning are K-means Clustering and sentiment dictionaries like VADER.

Descriptive Analytics

This comprises of analysing past data to present it in a summarised form that can be easily interpreted. Descriptive analytics aims to answer "what has happened?" A major portion of analytics done today is descriptive analytics through use of statistics functions such as counts, maximum, minimum, mean, top-N, and percentage.

Diagnostics Analytics

The analysis of past/historical data with the aim of investigating the reasons that something happened. Diagnostic analytics try answer "why did something happen?" For example, a system may collect and analyse sensor data from machines for monitoring their health and predicting failures.

Predictive Analytics

Comprises of predicting the occurrence of an event or the likely outcome of an event by forecasting future values using prediction models. They aim to answer "what is likely to happen?" For example, predictive analytics can be used to determine the probability of a price on the stock exchange market to go up📈 or down📉.

Prescriptive Analytics

Uses multiple prediction models to predict various outcomes and the best course of action for each. Prescriptive analytics aims to answer "what can we do to make it happen?" They help to select the best from all the available options.

Big Data

Hadoop is a a framework and tool used for processing and storing large amounts of data or "big data". Hadoop comes pre-installed with a system that facilitates read and write operations called HDFS (Hadoop Distributed File System).

Using the HDFS, the main data is stored in multiple data nodes, and also replicated. When one data node is down, the data can be accessed through any other data node which contains the replication. HDFS is very cost effective as it uses very simple commodity hardware to process and store data blocks.

Data Blocks

This is the minimum size of data that can be read or written in one shot. The default size of an HDFS block is 128 MB. The block size is changeable. The data is divided into different blocks and stored in the clusters.

NameNode

This is like the master node in the HDFS and it stores the metadata of all files. The NameNode serves as the controller or the manager of the HDFS by facilitating all operations. The NameNode is also responsible for:

• Managing the file system namespace.
• Controlling the access of different clients in the data blocks.
• Periodically checking the availability of the DataNodes.
• Managing the replication factor of the data blocks.

DataNode

These are the worker nodes of HDFS and they store the data blocks. DataNodes periodically report back different information to the NameNode about the data blocks such as the list of data being stored. DataNodes are generally the commodity hardware. They also create blocks, create replication or delete data blocks as instructed by the NameNode.

HDFS Architecture

It uses a Master-Worker architecture where the NameNode is the master and the workers are the low cost community hardware. In this architecture, there is a single NameNode and multiple DataNodes where the actual data is stored. The image below illustrates the architecture used by the HDFS.

Data Analytics Lifecycle (DAL)

A step-by-step methodology that organises the activities and tasks involved with acquiring, processing, analysing, and repurposing data. The key roles for a successful analytics project are:

• Business User👤
• Project Sponsor👤
• Project Manager👤
• BI (Business Intelligence) Analyst👤
• Database Administrator👤
• Data Engineer👤
• Data Scientist👤

Discovery (Phase 1)

This includes understanding the business domain, writing the problem statement, and assessing the skills and roles of all users involved. This phase formulates an initial hypothesis to test with data.

Data Preparation (Phase 2)

This step involves collecting, processing, and cleaning the accumulated data. The analyst essentially creates an environment to explore the data separate from the production environment. This helps them understand the data in detail. Some tools used during this phase include Hadoop and Alpine Miner.

Model Planning (Phase 3)

This phase aims to identify the candidate models to apply to the data in order to solve the business problem. Model Planning assesses the structure of the data in order to select the right tools and methods. The Analyst tries to understand how others generally approach similar problems.

Model Building (Phase 4)

This step of the DAL comprises of developing data sets for testing, training, and production purposes. The data analytics experts meticulously build and operate the model that they had designed in the previous step. They rely on tools and several techniques like decision tree, regression techniques, and neural networks for building and executing the model. The experts also perform a trial run of the model to observe if the model corresponds to the dataset.

Communicating Results (Phase 5)

After executing the model, the team needs to compare the outcomes of the modeling to the criteria established for success and failure. The team should identify key findings, quantify the business value and develop a narrative to summarise and convey findings to stakeholders.

Operationalise (Phase 6)

In the final phase, the team communicates the benefits of the project more broadly and sets up a pilot project to deploy the work in a controlled way before broadening the work to a full enterprise or ecosystem of users. The team delivers final reports, briefings, code, and technical documents.

Decision Tree

A decision tree is a non-parametric supervised learning algorithm, which is utilised for both classification and regression tasks. It has a hierarchical, tree structure, which consists of a root node, branches, internal nodes and leaf nodes.

K-Means Clustering

An unsupervised machine learning algorithm that groups unlabeled data into 'K' distinct clusters. It works by iteratively assigning data points to the nearest cluster (centroid) and then recalculating the centroids until the clusters stop changing and optimise for similarity (convergence).

Apriori Algorithm

A foundational data mining technique used for association rule learning to identify frequent itemsets in transactional databases. It is most famously used in retail for market basket analysis to uncover hidden patterns like "customers who buy bread and milk also tend to buy butter." The Apriori algorithm calculates Support, Confidence, and Lift.

Support measures how frequently an item or itemset appears in the total dataset.

Support(A) = No. of transactions containing A / Total transactions

Confidence measures how often items in B appear in transactions that contain A. It quantifies the operational reliability of the if-then statement.

Confidence (A->B) = Support(A u B) / Support(A)

Lift measures how much more likely item B is purchased relative to its standard average purchase rate when item A is purchased.

Life(A->B) = Support(A u B) / Support(A) x Support(B)

• Lift > 1: Positive correlation; items are highly likely to be bought together.

• Lift = 1: No correlation; items are independent of each other.

• Lift < 1: Negative correlation; items substitute or avoid each other.

Bayes Theorem

A powerful theorem in probability. It uses Bayesian inference. The formula is P(A|B) = (P(B|A) x P(A)) / P(B). This can be translated to "The probability of A given B is equal to the probability of B given A multiplied by the probability of A, all divided by the probability of B".