Welcome to my Learning Apache Spark with Python note! In this note, you will learn a wide array of concepts about PySpark in Data Mining, Text Mining, Machine Learning and Deep Learning. The PDF version can be downloaded from HERE.

Contents

• 1. Preface
  • 1.1. About
  • 1.2. Motivation for this tutorial
  • 1.3. Copyright notice and license info
  • 1.4. Acknowledgement
  • 1.5. Feedback and suggestions
• 2. Why Spark with Python ?
  • 2.1. Why Spark?
  • 2.2. Why Spark with Python (PySpark)?
• 3. Configure Running Platform
  • 3.1. Run on Databricks Community Cloud
  • 3.2. Configure Spark on Mac and Ubuntu
  • 3.3. Configure Spark on Windows
  • 3.4. PySpark With Text Editor or IDE
  • 3.5. PySparkling Water: Spark + H2O
  • 3.6. Set up Spark on Cloud
  • 3.7. PySpark on Colaboratory
  • 3.8. Demo Code in this Section
• 4. An Introduction to Apache Spark
  • 4.1. Core Concepts
  • 4.2. Spark Components
  • 4.3. Architecture
  • 4.4. How Spark Works?
• 5. Programming with RDDs
  • 5.1. Create RDD
  • 5.2. Spark Operations
  • 5.3. rdd.DataFrame vs pd.DataFrame
• 6. Statistics and Linear Algebra Preliminaries
  • 6.1. Notations
  • 6.2. Linear Algebra Preliminaries
  • 6.3. Measurement Formula
  • 6.4. Confusion Matrix
  • 6.5. Statistical Tests
• 7. Data Exploration
  • 7.1. Univariate Analysis
  • 7.2. Multivariate Analysis
• 8. Data Manipulation: Features
  • 8.1. Feature Extraction
  • 8.2. Feature Transform
  • 8.3. Feature Selection
  • 8.4. Unbalanced data: Undersampling
• 9. Regression
  • 9.1. Linear Regression
  • 9.2. Generalized linear regression
  • 9.3. Decision tree Regression
  • 9.4. Random Forest Regression
  • 9.5. Gradient-boosted tree regression
• 10. Regularization
  • 10.1. Ordinary least squares regression
  • 10.2. Ridge regression
  • 10.3. Least Absolute Shrinkage and Selection Operator (LASSO)
  • 10.4. Elastic net
• 11. Classification
  • 11.1. Binomial logistic regression
  • 11.2. Multinomial logistic regression
  • 11.3. Decision tree Classification
  • 11.4. Random forest Classification
  • 11.5. Gradient-boosted tree Classification
  • 11.6. XGBoost: Gradient-boosted tree Classification
  • 11.7. Naive Bayes Classification
• 12. Clustering
  • 12.1. K-Means Model
• 13. RFM Analysis
  • 13.1. RFM Analysis Methodology
  • 13.2. Demo
  • 13.3. Extension
• 14. Text Mining
  • 14.1. Text Collection
  • 14.2. Text Preprocessing
  • 14.3. Text Classification
  • 14.4. Sentiment analysis
  • 14.5. N-grams and Correlations
  • 14.6. Topic Model: Latent Dirichlet Allocation
• 15. Social Network Analysis
  • 15.1. Introduction
  • 15.2. Co-occurrence Network
  • 15.3. Appendix: matrix multiplication in PySpark
  • 15.4. Correlation Network
• 16. ALS: Stock Portfolio Recommendations
  • 16.1. Recommender systems
  • 16.2. Alternating Least Squares
  • 16.3. Demo
• 17. Monte Carlo Simulation
  • 17.1. Simulating Casino Win
  • 17.2. Simulating a Random Walk
• 18. Markov Chain Monte Carlo
  • 18.1. Metropolis algorithm
  • 18.2. A Toy Example of Metropolis
  • 18.3. Demos
• 19. Neural Network
  • 19.1. Feedforward Neural Network
• 20. Automation for Cloudera Distribution Hadoop
  • 20.1. Automation Pipeline
  • 20.2. Data Clean and Manipulation Automation
  • 20.3. ML Pipeline Automation
  • 20.4. Save and Load PipelineModel
  • 20.5. Ingest Results Back into Hadoop
• 21. Wrap PySpark Package
  • 21.1. Package Wrapper
  • 21.2. Pacakge Publishing on PyPI
• 22. PySpark Data Audit Library
  • 22.1. Install with pip
  • 22.2. Install from Repo
  • 22.3. Uninstall
  • 22.4. Test
  • 22.5. Auditing on Big Dataset
• 23. Zeppelin to jupyter notebook
  • 23.1. How to Install
  • 23.2. Converting Demos
• 24. My Cheat Sheet
• 25. JDBC Connection
  • 25.1. JDBC Driver
  • 25.2. JDBC read
  • 25.3. JDBC write
  • 25.4. JDBC temp_view
• 26. Databricks Tips
  • 26.1. Display samples
  • 26.2. Auto files download
  • 26.3. Working with AWS S3
  • 26.4. delta format
  • 26.5. mlflow
• 27. PySpark API
  • 27.1. Stat API
  • 27.2. Regression API
  • 27.3. Classification API
  • 27.4. Clustering API
  • 27.5. Recommendation API
  • 27.6. Pipeline API
  • 27.7. Tuning API
  • 27.8. Evaluation API
• 28. Main Reference