ניהול פרויקטים וניתוח מערכות בעולם ה-Big Data

Module Description

Module Title

• What is Business Intelligence
• How Important is BI
• From data to knowledge
• BI Life cycle
• Requirements in BI project 
• Front end / Dashboards / Data visualization
• Data modeling
• ETL
• What if
• Prediction
• In memory db
• Data mining

From BI to Big Data

Introduction to Big Data

• Big Data Characteristics and Use-Cases
• Big Data from Business Perspective Challenges and limitations of Big Data Implementation

The Data in Big Data

• Structured vs. Semi-structured vs. Unstructured Data
• Various Phases of Data Processing
• Problems with O/R Mapping

 Introduction to NoSQL

• NoSQL Definition
• Business Drivers for NoSQL
• New NoSQL Paradigms
• NoSQL Categories:
  • Key-value Store
  • Column Family
  • Document DBs
  • Graph DBs

 Hadoop Overview

• Apache Hadoop Architecture: HDFS and MapReduce
• MapReduce and Big Data – Real Life Examples

 Introduction to Big Data Analytics

• Data and Analytical Complexity
• Real-Time Analytics
• Big Data Visualization

Introduction to Big Data

• Making sense of Big Data
• Turning Big Data into something useful
• Storage
• Security
• Data reconciliation
• Information extraction
• Insight distribution
• Successfully Navigating Big Data

Big Data: What it Means to IT Managers on the Front Lines

1. Data exploration
2. Data cleansing
3. Data preparation
4. Data analyze
5. Data display

1+3 data exploration and data preparation

Introduction

There are no shortcuts for data exploration. If you are in a state of mind, that machine learning can sail you away from every data storm, trust me, it won’t. After some point of time, you’ll realize that you are struggling at improving model’s accuracy. In such situation, data exploration techniques will come to your rescue.

Steps of Data Exploration and Preparation

• Variable Identification
• Univariate Analysis
• Bi-variate Analysis
• Missing values treatment
• Outlier treatment
• Variable transformation
• Variable creation

Missing Value Treatment

• Why missing value treatment is required ?
• Why data has missing values?
• Which are the methods to treat missing value ?

Techniques of Outlier Detection and Treatment

• What is an outlier?
• What are the types of outliers ?
• What are the causes of outliers ?
• What is the impact of outliers on dataset ?
• How to detect outlier ?
• How to remove outlier ?

The Art of Feature Engineering

• What is Feature Engineering ?
• What is the process of Feature Engineering ?
• What is Variable Transformation ?
• When should we use variable transformation ?
• What are the common methods of variable transformation ?
• What is feature variable creation

Project management

(project steps) 

in big data

• Introduction
• Motivation
• Data quality
  • Validity:
  • DE cleansing
  • Accuracy:
  • Completeness:
  • Consistency:
  • Uniformity:
• The process of data cleansing
  • Data auditing:
  • Workflow specification:
  • Workflow execution:
  • Post-processing and controlling:
• De-cleanse
  • Parsing:
  • Data transformation
  • Duplicate elimination:
  • Statistical methods:
• Challenges and problems

Data Cleansing

• The ETL Bottleneck in Big Data Analytics
• Apache Hadoop for Big Data
• ETL, ELT, and ETLT with Apache Hadoop:
• Choosing the Physical Infrastructure for ETL with Hadoop:
  • Compute
  • Memory
  • Storage
  • Network
  • Software

Data exploration with ETL, ELT, and ETLT

• Introduction
• Hadoop distributors (Cloudera, Hortonworks, MapR)
• Building blocks of Hadoop (NameNode, DataNode…)
• Introduction to HDFS
• Map-reduce pattern
• Distribute cache
• Introduction to Hive for ad-hoc queries
  • Hive basics
  • Hive data types
  • HiveQL
• Pig:
  • Introduction to Pig as data flow language
  • Pig Latin basic expressions
  • Operators for data processing
•  YARN (Map-Reduce 2)
  • Motivation for YARN.
  • Architecture.
  • Features.

Data collection and integration from outside resources

• Saving data methods:
• Hadoop , Files, XML, CSV, RDBMS,OLAP, Tabular,  No-SQL db
• RDBMS challenge in Big-data world
• No-SQL vs traditional relational data
• Scaling vs. consistency
• No-Sql database types:
  • Key Value  DB
  • Column-Family  DB
  • Document DB
  • graph  DB
• Transaction in No-SQL
• Applying map-reduce
• No-SQL leading implementations

Data modeling in Big data world

• Hive
  • Introduction to Hive for ad-hoc queries
  • Hive basics
• Hbase
  • Introduction to Pig as data flow language
  • Introduction to Hbase for processing huge tables
  • Hbase data model
  • Hbase vs. RDBMS
•  Data mining , using R

Date inquiry in Big data

• 1:2:3 Method to data visualization
• What are the changes that should be made in our methods
• Alerts & exceptions
• Links & networks
• Hue – the new gate to big data

Data visualization

• Presentation/ project