Welcome to my Python Tips for Data Scientist notes! In those notes, you will learn some useful tips for Data Scientist daily work. The PDF version can be downloaded from HERE.

Contents

• 1. Preface
  • 1.1. About
    • 1.1.1. About this note
    • 1.1.2. About the authors
  • 1.2. Motivation for this note
  • 1.3. Feedback and suggestions
• 2. Python Environment
  • 2.1. Use python on local computer
  • 2.2. Use python on the cloud (zero setup)
    • 2.2.1. Google Colab (FREE)
    • 2.2.2. Kaggle Kernels (FREE)
    • 2.2.3. Databricks Community Edition (FREE)
• 3. Notebooks
  • 3.1. Nteract
  • 3.2. Jupyter Notebook Viewer
  • 3.3. Apache Zeppelin
  • 3.4. Jupyter Notebook
• 4. Confidential Information
• 5. Primer Functions
  • 5.1. *
  • 5.2. range
  • 5.3. random
    • 5.3.1. random.random
    • 5.3.2. np.random
  • 5.4. round
  • 5.5. TODO..
• 6. Data Structures
  • 6.1. List
    • 6.1.1. Create empty list
    • 6.1.2. Unpack list
    • 6.1.3. Methods of list objects
    • 6.1.4. list.append(x) vs. list.extend(iterable)
  • 6.2. Tuple
    • 6.2.1. Create Tuple
    • 6.2.2. Assigning Multiple Values At Once with Tuple
  • 6.3. Dictionary
    • 6.3.1. Create dict from lists
    • 6.3.2. dict.get()
    • 6.3.3. Looping Techniques
    • 6.3.4. Update Values in Dict
    • 6.3.5. Update Keys in Dict
  • 6.4. One line if-else statement
    • 6.4.1. With filter
    • 6.4.2. Without filter
• 7. Data Read and Ingestion with DataBase
  • 7.1. Data Ingestion from Local to DataBase
  • 7.2. Data Read from DataBase to Local
  • 7.3. Connect to various DataBases ()
  • 7.4. Hive and Impala Table Ingestion
• 8. Working with AWS S3
  • 8.1. Credentials
  • 8.2. File Upload to S3
    • 8.2.1. s3_file_upload Function
    • 8.2.2. s3_content_type Function
    • 8.2.3. Examples
  • 8.3. File Download from S3
    • 8.3.1. s3_file_download Function
    • 8.3.2. Examples
  • 8.4. File Management in S3
    • 8.4.1. s3_fs Function
    • 8.4.2. Examples
• 9. pd.DataFrame vs PySpark DataFrame
  • 9.1. Create DataFrame
    • 9.1.1. From List
    • 9.1.2. From Dict
  • 9.2. Convert between pandas and pyspark DataFrame
    • 9.2.1. From pandas to pyspark DataFrame
    • 9.2.2. From pyspark to pandas DataFrame
  • 9.3. Load DataFrame
    • 9.3.1. From DataBase
    • 9.3.2. From .csv
    • 9.3.3. From .json
  • 9.4. First n Rows
  • 9.5. Column Names
  • 9.6. Data types
  • 9.7. Replace Data types
  • 9.8. Fill Null
  • 9.9. Replace Values
  • 9.10. Rename Columns
    • 9.10.1. Rename all columns
    • 9.10.2. Rename one or more columns
  • 9.11. Drop Columns
  • 9.12. Filter
  • 9.13. With New Column
  • 9.14. Join
    • 9.14.1. Left Join
    • 9.14.2. Right Join
    • 9.14.3. Inner Join
    • 9.14.4. Full Join
  • 9.15. Concat Columns
  • 9.16. GroupBy
  • 9.17. Pivot
  • 9.18. Unixtime to Date
• 10. pd.DataFrame manipulation
  • 10.1. TODO..
• 11. rdd.DataFrame manipulation
  • 11.1. TODO..
• 12. Online Courses and Useful Websites
  • 12.1. Recommended online courses
  • 12.2. Recommended Online Resources
  • 12.3. Free Online E-books
• 13. Package Wrapper
  • 13.1. Hierarchical Structure
  • 13.2. Set Up
  • 13.3. Requirements
  • 13.4. ReadMe
• 14. Publish Package to PyPI
  • 14.1. Register PyPI account
  • 14.2. Install twine
  • 14.3. Build Your Package
  • 14.4. Upload Your Package
  • 14.5. Package at PyPI
• 15. Model Deployment with Flask
  • 15.1. Install flask
  • 15.2. Train and Save your model
  • 15.3. Deplyment with Flask
  • 15.4. Lunch your app on server
    • 15.4.1. 1. Lunch the APP
    • 15.4.2. 2. Run the APP
• 16. API Book
  • 16.1. Basics Module
    • 16.1.1. rnorm
    • 16.1.2. dnorm
    • 16.1.3. runif
  • 16.2. Tests Module
    • 16.2.1. T-test
• 17. Main Reference