Polars Python: Faster ETL and Automation Scripts with Lazy Evaluation

Polars is a DataFrame library for Python built in Rust that processes large datasets 3–11x faster than pandas on most analytical operations. It runs on every available CPU core by default, uses Apache Arrow's columnar memory format, and evaluates queries lazily, pushing predicates directly into file reads before any data enters memory.

With 575M+ downloads, 38,000+ GitHub stars, and a €18M Series A from Accel in September 2025, Polars has moved well past the experimental phase.

For Python automation scripts and data pipelines, the difference shows up immediately. Import time alone is ~70ms vs ~520ms for Polars vs pandas: meaningful when scheduled jobs run thousands of times a day on cold containers.

This guide covers installation through production deployment, with real 2026 benchmarks, side-by-side pandas migration patterns, and case studies from teams running Polars at scale.

Key Takeaways

• Use scan_csv() instead of read_csv() for large files: the lazy path pushes filters into the reader before any data loads
• The expression API (pl.col() inside select(), with_columns(), filter(), group_by().agg()) is the core concept; once it clicks, everything else follows
• Polars 1.18 lazy mode filters 11x faster and runs group-by 10x faster than Pandas 2.2 on a 240M-row dataset
• There is no row index (intentional: eliminates a class of index-alignment bugs that pandas creates)
• pip install polars works on Python 3.10+; add [numpy,pandas,pyarrow] extras for interoperability

What Is Polars?

Polars is a high-performance DataFrame library for Python, Rust, Node.js, and R, built on Apache Arrow's columnar memory format. Ritchie Vink created it in 2020 as a side project after running into three pandas bottlenecks repeatedly: single-threaded execution, row-oriented memory with Python object overhead, and no lazy execution path for large files.

Polars runs all available CPU cores by default and uses SIMD vectorization on Arrow's columnar layout. A query optimizer prunes columns and pushes predicates before any bytes enter memory. Polars 1.0 shipped in July 2024, the first stable API milestone with a commitment to fewer breaking changes going forward.

The latest stable release is py-1.42.0 (June 24, 2026), which adds naive out-of-core spilling, strict mode, and SQL implicit JOIN support.

Why Polars Matters for Python Automation in 2026

Polars BV raised €18M from Accel and Bain Capital Ventures in September 2025. The library has 575M+ total downloads and 23M+ monthly users. In Q1 2026 alone, the team shipped 12 releases and merged 778 pull requests from 95 contributors.

For scheduled jobs and serverless functions, the cold-start import time gap compounds at scale. Polars loads in ~70ms vs ~520ms. On a pipeline that runs 1,000 times per day, that is 450 seconds of pure import overhead per day in the pandas version.

Automation scripts and scheduled pipelines are where Polars' advantages land hardest: fast imports cut cold-start overhead, multi-core execution saturates the host by default, and lazy file scanning avoids loading gigabytes that filters would discard anyway. Existing Python tutorials rarely frame Polars from this angle.

Installing Polars

Polars requires Python 3.10+ and installs via pip:

# Minimal
pip install polars

# With interoperability extras
pip install "polars[numpy,pandas,pyarrow]"

# All optional dependencies
pip install "polars[all]"

# For datasets exceeding 4.2 billion rows (64-bit row indices)
pip install polars-u64-idx

# For CPUs without AVX2 instructions (older hardware or some cloud VMs)
pip install polars-lts-cpu

The standard import convention is import polars as pl. Verify with `pl.version`.

Conda installs via conda install -c conda-forge polars, though the pip path is preferred to avoid Arrow version conflicts with other packages.

How Polars Works: Expressions, Contexts, and Frames

The Polars data model has three core structures: Series (typed 1-D arrays), DataFrame (in-memory table, eager execution), and LazyFrame (deferred query plan, executes on .collect()).

DataFrames and Series

A DataFrame is a table of named Series. Every Series is homogenously typed: Polars displays column dtypes alongside names (str, f64, i64, date) in every print output, a concrete signal that type safety is enforced at the column level. There is no row index.

import polars as pl

df = pl.DataFrame({
    "name": ["Alice Archer", "Ben Brown", "Chloe Cooper", "Daniel Donovan"],
    "birthdate": ["1997-01-10", "1985-02-15", "1983-03-22", "1981-04-30"],
    "weight": [57.9, 72.5, 53.6, 83.1],   # kg
    "height": [1.56, 1.77, 1.65, 1.75],    # m
})

The Expressions System

Expressions are the defining abstraction in Polars. An expression describes what to compute; the engine decides how to compute it efficiently across cores. Expressions compose inside one of four contexts:

# select — compute BMI, return only name + birth_year + bmi
result = df.select(
    pl.col("name"),
    pl.col("birthdate").dt.year().alias("birth_year"),
    (pl.col("weight") / (pl.col("height") ** 2)).alias("bmi"),
)

# with_columns — add columns without dropping originals
result = df.with_columns(
    birth_year=pl.col("birthdate").dt.year(),
    bmi=pl.col("weight") / (pl.col("height") ** 2),
)

# filter — rows born before 1990
result = df.filter(pl.col("birthdate").dt.year() < 1990)

# group_by + agg — average weight by birth decade
result = df.group_by(
    (pl.col("birthdate").dt.year() // 10 * 10).alias("decade"),
).agg(
    pl.len().alias("sample_size"),
    pl.col("weight").mean().round(2).alias("avg_weight"),
)

Jeroen Janssens, co-author of Python Polars: The Definitive Guide, put it plainly in his PyData crash course: "When you understand expressions, you're halfway there. You're halfway there." He planned one chapter on expressions; the book ended up needing three.

Pandas users migrating to Polars often hit this wall first. Jeroen adds in the same talk: "If you do have pandas experience, it's not always an advantage. There are some things that you need to unlearn." (Jeroen Janssens at PyData)

LazyFrames: Deferred Execution

A LazyFrame records operations without running them. You build the full pipeline, then call .collect() once at the end. The query optimizer runs between plan-building and execution: it pushes filters into file reads (predicate pushdown), prunes columns not referenced downstream (projection pushdown), folds constants, and parallelizes independent branches.

lf = pl.scan_csv("events.csv", try_parse_dates=True)

out = (
    lf
    .filter(pl.col("ts").is_between(pl.date(2026, 1, 1), pl.date(2026, 3, 31)))
    .group_by(["user_id", "event_type"])
    .agg(pl.col("value").mean())
    .sort("value", descending=True)
    .collect()
)

Inspect the optimizer's plan at any stage:

lf.explain()      # print logical plan as text
lf.show_graph()   # visualize plan (requires graphviz)

Lazy Evaluation Is the Default, Not the Advanced Mode

Most Polars tutorials treat lazy evaluation as a late "advanced" section. That ordering is backwards.

The practical rule for data analysis pipelines: default to scan_csv() / scan_parquet() + .collect() for any file you wouldn't print in a REPL. Switch to eager read_csv() only for small, interactive DataFrames where you want to inspect intermediate results mid-pipeline.

# Preferred pattern for production pipelines
out = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("country") == "US")
    .select(["user_id", "revenue", "ts"])
    .with_columns([
        pl.col("revenue").fill_null(0),
        pl.col("ts").dt.date().alias("day"),
    ])
    .group_by(["user_id", "day"])
    .agg(pl.col("revenue").sum().alias("total_revenue"))
    .collect()
)

Matt Harrison, who presented Getting Started with Polars at PyCon US, draws the memory model distinction clearly. Pandas requires all data in memory before any operation runs; Polars can also stream. As Harrison puts it: "Polars is similar in that generally you will have your data in memory, but it can also do some streaming." (Matt Harrison, PyCon US)

The lazy optimizer delivers an additional 30–60% speedup over eager Polars on the same operations. The gain comes not just from parallelism but from reordering filters and pushing predicates into the Parquet reader, so fewer bytes ever enter memory.

Streaming for Larger-Than-RAM Files

When a dataset exceeds available RAM, .collect(engine="streaming") processes the query in bounded memory chunks. sink_parquet() and sink_csv() write streaming output without materializing the full result.

# Streaming collect — bounded memory
out = (
    pl.scan_csv("huge_log.csv")
    .filter(pl.col("severity") == "ERROR")
    .group_by("service")
    .agg(pl.len().alias("error_count"))
    .collect(engine="streaming")
)

# Streaming write — never materializes the full dataset
(
    pl.scan_parquet("very_large.parquet")
    .filter(pl.col("region") == "EMEA")
    .group_by("product_id")
    .agg(pl.col("revenue").sum())
    .sink_parquet("emea_revenue.parquet")
)

This pattern runs on memory-constrained CI runners and small cloud instances without modification.

Polars vs Pandas: Benchmarks and Migration

A 2026 benchmark from Danilchenko.dev tested Polars 1.18 against Pandas 2.2 on a 240M-row / 14 GB Parquet file on an Apple M2 Pro:

The honest caveat: string-heavy operations show only a 1.3x improvement. If your pipeline is primarily regex, .str.contains(), or text parsing on large columns, the gain is narrower. For analytical workloads (filter, aggregate, join, sort), Polars wins by an order of magnitude at scale.

A complete ETL pipeline benchmark measured load + clean + aggregate + export as a single workflow: pandas finished in 62.37 s, Polars lazy in 19.10 s, a 3.3x speedup on a realistic end-to-end job.

On r/Python in June 2026, u/GunZinn captured the switch-day experience: "I was parsing a 4GB csv file last week. Polars was nearly 18x faster than using pandas. First time I used polars."

Side-by-Side Syntax

Polars is declarative and immutable: every operation returns a new DataFrame. Pandas is imperative and mutable: .loc[] modifies in place and indexes accumulate. The syntax differences follow from this architectural split.

# --- FILTER + SELECT ---
# pandas
result = pdf[pdf["country"] == "US"][["user_id", "revenue"]]

# polars
result = pldf.filter(pl.col("country") == "US").select(["user_id", "revenue"])


# --- GROUP-BY + AGGREGATE ---
# pandas
rev = pdf.groupby("user_id", as_index=False)["revenue"].sum()

# polars
rev = pldf.group_by("user_id").agg(pl.col("revenue").sum())


# --- CONDITIONAL COLUMN (no in-place assignment) ---
# pandas
df.loc[df["score"] > 90, "grade"] = "A"

# polars (immutable — returns a new column)
df = df.with_columns(
    grade=pl.when(pl.col("score") > 90).then(pl.lit("A")).otherwise(pl.col("grade"))
)


# --- FULL PIPELINE COMPARISON ---
# pandas
df = pd.read_parquet("events.parquet")
df = df[df["country"] == "US"][["user_id", "revenue", "ts"]]
df["revenue"] = df["revenue"].fillna(0)
df["day"] = pd.to_datetime(df["ts"]).dt.date
out = df.groupby(["user_id", "day"], as_index=False).agg(
    total_revenue=("revenue", "sum")
)

# polars lazy
out = (
    pl.scan_parquet("events.parquet")
    .filter(pl.col("country") == "US")
    .select(["user_id", "revenue", "ts"])
    .with_columns([
        pl.col("revenue").fill_null(0),
        pl.col("ts").dt.date().alias("day"),
    ])
    .group_by(["user_id", "day"])
    .agg(pl.col("revenue").sum().alias("total_revenue"))
    .collect()
)

Migrating an Existing Codebase

Rewriting everything at once introduces risk. The recommended path: use pl.from_pandas() and .to_pandas() to swap one hot-path pipeline step at a time.

import pandas as pd
import polars as pl

# Existing pandas pipeline
pdf = pd.read_parquet("events.parquet")
pdf = pdf[pdf["country"] == "US"]

# Swap just the slow aggregation into Polars
pldf = pl.from_pandas(pdf)
result = pldf.group_by("user_id").agg(pl.col("revenue").sum())

# Convert back if downstream code still expects pandas
pdf_out = result.to_pandas()

Conversion is zero-copy where dtypes and nulls allow. Four patterns to unlearn during migration:

• Replace df["col"] = value with with_columns(pl.when(...).then(...).otherwise(...))
• Replace .apply(axis=1) with native expressions under `str., dt., list.*`
• Replace pd.read_csv() in loops with pl.scan_csv() + lazy pipeline
• For positional access: Polars has no .iloc[]. Use df.with_row_index() to add an explicit integer column

u/corey_sheerer in r/learnpython on the API trajectory: "Polars has vastly surpassed pandas in performance. I would utilize Polars for new projects. Unfortunately, with pandas new 3.0 API, it is clear pandas will not catch Polars in performance (or syntax) due to pandas desire to maintain backwards capabilities."

Where Pandas Still Wins

Pandas fits small exploratory analysis (under ~1 GB), scikit-learn and statsmodels pipelines (which expect DataFrames natively), and Matplotlib/seaborn visualizations where the accessor chain is already written. On python-statistics workflows that feed directly into statsmodels, the conversion cost outweighs the gain. For those cases, use .to_pandas() at the boundary and keep the rest of the pipeline in Polars.

The No-Index Design: Bug-Eliminator, Not a Quirk

Every Polars tutorial mentions the missing row index. Most frame it as a limitation. The accurate framing is the opposite.

Pandas' row index creates a specific class of bugs: index-alignment errors (pandas silently aligns on index before binary operations), .reset_index() footguns after groupby(), and multi-index confusion in hierarchical data. None of those bugs exist in Polars. The index simply isn't there to mismanage.

Matt Harrison described the adjustment at PyCon US: "If you're familiar with pandas, there is no index here. That causes changes or effects that if you're coming from pandas you just need to be aware of." (Getting Started with Polars, Matt Harrison, PyCon US)

On r/learnpython, the adjustment follows a predictable arc. u/likethevegetable: "I think you need to reframe your approach. I struggled with this a bit at first, but now I've come to appreciate the lack of index in polars: in pandas, I was messing around far too often with reindexing."

For positional access where you genuinely need row numbers, add an explicit column:

df = df.with_row_index("row_nr")
# df now has an integer "row_nr" column starting at 0

Essential I/O Patterns

# EAGER — small files, interactive exploration
df = pl.read_csv("data.csv")
df = pl.read_parquet("data.parquet")
df = pl.read_excel("data.xlsx", schema_overrides={"date": pl.Datetime})

# LAZY — large files, any production pipeline (preferred)
lf = pl.scan_csv("data.csv")
lf = pl.scan_parquet("data/*.parquet")       # glob scans multiple files

# WRITE
df.write_csv("output.csv")
df.write_parquet("output.parquet")

# STREAMING WRITE — never materializes the full dataset in memory
(
    pl.scan_csv("very_large.csv")
    .filter(pl.col("region") == "EMEA")
    .group_by("product_id")
    .agg(pl.col("revenue").sum())
    .sink_parquet("emea_revenue.parquet")
)

Parquet is the recommended format for large data analysis pipelines: column-pruning on read means Polars loads only the columns referenced in the query, not the full file.

Joins and Concatenation

# Inner join
result = df1.join(df2, on="id", how="inner")

# Left join
result = df1.join(df2, on="name", how="left")

# Multi-key join
result = df1.join(df2, on=["user_id", "date"], how="inner")

# Full outer join
result = df1.join(df2, on="id", how="full")

# Stack rows
result = pl.concat([df_a, df_b], how="vertical")

# Side-by-side columns
result = pl.concat([df_a, df_b], how="horizontal")

Pre-sort join inputs and Polars activates sort-merge joins automatically, yielding up to 18x speedup on join-intensive workloads.

Polars in Production: Case Studies

Check Technologies

Check Technologies runs a Dutch mobility platform serving 300,000 daily users. After hitting out-of-memory errors on Kubernetes, the team migrated 100+ Apache Airflow DAGs to Polars in a single two-week sprint.

The initial test DAG ran 3.3x faster. Within weeks, nearly all DAGs had doubled in speed.

The team cut cloud costs 25% after scaling down Kubernetes infrastructure. Senior Data Engineer Paul Duvenage: "Polars not only solved our initial problem but opened the door to new possibilities."

METRO.digital

METRO.digital supports 623 METRO wholesale stores across 30+ countries. The team migrated 50 SQL files per KPI across 19 countries to Python + Polars, with the largest dataset holding 930 million invoice rows.

After migration, preprocessing on the 930M-row dataset went from minutes to seconds. Selective lazy evaluation on join bottlenecks cut 128 GB RAM and 16 vCPUs from the infrastructure, enabling a smaller, cheaper machine type.

The Polars Ecosystem in 2026

Polars integrates with the Python data stack:

Jensen Huang named Polars at NVIDIA GTC 2026: "Now we will have AI use structured data. And we are going to accelerate the living daylights out of it." (Polars on LinkedIn, March 2026)

GPU acceleration via collect(engine="gpu") is in Open Beta as of June 2026, requiring NVIDIA RAPIDS cuDF. Multi-GPU execution is available via RayEngine. The CPU fallback is automatic when the engine encounters an unsupported operation.

For workloads that exceed a single machine, Distributed Polars on Kubernetes launched in mid-2026. A benchmark against PySpark 4.0.1 on TPC-H (1 TB scale) showed Polars averaging 3.2x faster across queries, peaking at 7.8x on individual queries.

Polars also consumes 63% of pandas' energy on TPC-H benchmarks and 8x less energy on synthetic data, per an academic study from EASE '24 (ACM). For cloud workloads billed by CPU-hour, energy savings translate directly to infrastructure cost.

Common Polars Mistakes to Avoid

Calling .collect() After Every Step

The query optimizer needs the full lazy plan to work. Calling .collect() mid-pipeline forces execution before the optimizer can push predicates or prune columns.

# Wrong — defeats lazy optimization
filtered = lf.filter(pl.col("country") == "US").collect()
grouped = pl.from_pandas(filtered.to_pandas()).group_by("user_id").agg(...)

# Correct — one collect at the end
result = (
    lf
    .filter(pl.col("country") == "US")
    .group_by("user_id")
    .agg(pl.col("revenue").sum())
    .collect()
)

Using `read_ Instead of scan_` for Large Files

pl.read_csv("large.csv") loads the entire file into memory before any filter runs. pl.scan_csv("large.csv").filter(...) pushes the filter into the reader. For GB-range files in Python automation scripts, this is the difference between a job that completes and one that triggers an OOM kill.

Row-by-Row Python Loops

# Slow — Python-level iteration defeats vectorization
results = []
for row in df.iter_rows(named=True):
    results.append(row["value"] * 2)

# Fast — vectorized expression
df = df.with_columns(value_doubled=pl.col("value") * 2)

The expression API covers virtually every per-row operation: string methods under `pl.col().str., datetime operations under .dt., list operations under .list.*, and conditional logic via pl.when().then().otherwise(). Reaching for .apply()` or a Python loop almost always signals that a native Polars expression handles the operation more efficiently.

Expecting In-Place Mutation

Polars DataFrames are immutable. df["col"] = value raises a TypeError. Assign the result of every operation:

# Raises TypeError
df["grade"] = "A"

# Correct
df = df.with_columns(grade=pl.lit("A"))

Passing Strings Where Expressions Are Expected

In Polars contexts (select(), with_columns(), filter(), group_by().agg()), column references must be expressions: pl.col("name"). Passing a bare string "name" where an expression is expected is a common beginner error that produces a confusing type error rather than a clear message.