Getting Started

This page summarizes the basic steps required to setup and get started with PySpark. There are more guides shared with other languages such as Quick Start in Programming Guides at the Spark documentation.

There are live notebooks where you can try PySpark out without any other step:

• Live Notebook: DataFrame

• Live Notebook: Spark Connect

• Live Notebook: Pandas API on Spark

The list below is the contents of this quickstart page:

• Installation
  • Python Versions Supported
  • Using PyPI
  • Using Conda
  • Using venv
  • Manually Downloading
  • Installing from Source
  • Dependencies
• Quickstart: DataFrame
  • DataFrame Creation
  • Viewing Data
  • Selecting and Accessing Data
  • Applying a Function
  • Grouping Data
  • Getting Data In/Out
  • Working with SQL
• Quickstart: Spark Connect
  • Launch Spark server with Spark Connect
  • Connect to Spark Connect server
  • Create DataFrame
• Quickstart: Pandas API on Spark
  • Object Creation
  • Missing Data
  • Operations
  • Grouping
  • Plotting
  • Getting data in/out
• Testing PySpark
  • Build a PySpark Application
  • Testing your PySpark Application
  • Putting It All Together!