Polars

---

Index - Polars user guidedocs.pola.rs

Python API reference

https://www.confessionsofadataguy.com/replacing-pandas-with-polars-a-practical-guide/

---

Polars: The "Rust-Powered" DataFrame Library

While PyArrow provides the foundation (memory format) and DuckDB acts as an SQL engine, Polars is a DataFrame library built from scratch in Rust specifically to replace Pandas.

It uses Apache Arrow as its native memory format (just like PyArrow), but unlike Pandas, it is multithreaded by default and supports Lazy Evaluation.

The Big Differences: Polars vs. Pandas vs. PyArrow

Feature	Pandas (Standard)	PyArrow	Polars
Primary Language	Python (C backend)	C++	Rust
Memory Format	NumPy (Row/Array)	Arrow (Columnar)	Arrow (Columnar)
Execution	Eager (Line-by-line)	Eager	Lazy or Eager
Parallelism	Single Core (mostly)	Single Core	All Cores (Multithreaded)
Syntax Style	Index-based (df.loc)	Low-level	Expression-based (col("a"))

---

Key Feature 1: Lazy Evaluation (The "Query Plan")

This is Polars' superpower. In Pandas, every line of code runs immediately. In Polars, you can write a "Lazy" query that doesn't run until you say collect().

This allows Polars to look at your entire query and optimize it before running.

Example: Reading a file and filtering

• Pandas: Reads the entire CSV into RAM → Filters rows → Selects columns.

• Polars (Lazy): Scans metadata → Sees you only want columns "A" and "B" → Sees you filter where "A > 100" → Only reads the specific chunks of the file that match.

import polars as pl

# 1. Scan a file lazily (No data loaded yet!)
#    'scan_csv' creates a LazyFrame, unlike 'read_csv'
lazy_df = pl.scan_csv("large_data.csv")

# 2. Build the query plan
#    Nothing happens here except building a graph of operations
query = (
    lazy_df
    .filter(pl.col("sales_amount") > 500)
    .group_by("region")
    .agg([
        pl.col("sales_amount").sum().alias("total_sales"),
        pl.col("product_id").count().alias("items_sold")
    ])
)

# 3. Execute! (The Query Optimizer takes over)
#    Polars optimizes the plan (e.g., predicate pushdown) and runs it on all cores
result = query.collect()

print(result)

Key Feature 2: Multithreaded Speed

If you have an 8-core CPU:

• Pandas generally uses 1 core for string processing.

• Polars will divide the data into chunks and process them on all 8 cores simultaneously.

Key Feature 3: The Expression API

Polars moves away from the confusing Pandas syntax (indexes, .loc, .iloc) and uses a "verb-based" syntax that reads like English (or SQL).

Comparison: Adding a column based on logic

Pandas:

# Often requires slow 'apply' or complex numpy.where
import numpy as np
df['category'] = np.where(df['sales'] > 100, 'High', 'Low')

Polars:

# Pure expressions, runs in parallel
df = df.with_columns(
    pl.when(pl.col("sales") > 100)
    .then(pl.lit("High"))
    .otherwise(pl.lit("Low"))
    .alias("category")
)

Summary of the Ecosystem

You now have a complete toolkit for high-performance Python data:

1. PyArrow: The Storage Layer. Use it to read/write Parquet files efficiently and share memory between tools.

2. Pandas 2.0: The Generalist. Use it if you need legacy compatibility or specific libraries (like plotting) that expect Pandas objects. Enable the PyArrow backend for a speed boost.

3. DuckDB: The SQL Analyst. Use it if you prefer writing SQL queries on your dataframes or need to aggregate data that is larger than RAM.

4. Polars: The Speed Specialist. Use it for heavy ETL (Extract, Transform, Load) jobs, massive data cleaning, or when Pandas is simply too slow.

---