Create temp tables/views

---

createOrReplaceTempView()

What it does

It allows you to register a DataFrame as a temporary table/view and reference it in Spark using SQL queries:

# Register the DataFrame
df_green.createOrReplaceTempView('trips_data')

# Now you can query it with SQL
result = spark.sql("""
    SELECT * 
    FROM trips_data 
    WHERE fare_amount > 10
""")

Types of temp tables/views:

1. Temporary View (session-scoped):

df.createOrReplaceTempView('my_table')
# Available only in current Spark session

2. Global Temporary View (application-scoped):

df.createGlobalTempView('my_table')
# Available across all sessions in the application
# Must query with: spark.sql("SELECT * FROM global_temp.my_table")

Key characteristics:

• Temporary - exists only for the duration of the Spark session

• In-memory metadata - doesn't write data to disk

• SQL interface - lets you use SQL instead of DataFrame API

• Replaces if exists - createOrReplaceTempView() overwrites existing view with same name

Example workflow:

# Read data
df_green = spark.read.parquet('green_tripdata.parquet')
df_yellow = spark.read.parquet('yellow_tripdata.parquet')

# Register both as temp views
df_green.createOrReplaceTempView('green_trips')
df_yellow.createOrReplaceTempView('yellow_trips')

# Now use SQL to join them
result = spark.sql("""
    SELECT * FROM green_trips
    UNION ALL
    SELECT * FROM yellow_trips
""")

---