python-pandas

Python Pandas

Standards for working with Pandas DataFrames and Series. Extends the "prefer library idioms" rule in python-code-style with Pandas specifics.

Vectorise

• Operate on whole Series and DataFrames; don't loop with .apply, .iterrows, or a Python for over rows. Reach for Series methods — .where/.mask/.map/.clip/ .str.*, .between, .isin — and df.eval/df.query for arithmetic and filtering.
• Prefer Series.map(mapping) to .apply(lambda x: mapping[x]), and use &/| only for row-wise boolean masks — keep and/or for scalar conditionals.
• When you genuinely must iterate, use .itertuples() (named, typed, fast), never .iterrows().
• Stay in Pandas types all the way to the function boundary — don't drop to Python lists or NumPy arrays mid-pipeline and convert back.
• Suffix DataFrame variables _df so the type is visible at the call site; name Series for their contents (e.g. unit_price, not s or data).

Don't mix NumPy into Pandas

• Use pd.NA/pd.NaT for missing values, not np.nan, and prefer the nullable extension dtypes (Int64, boolean, string) so missingness is first-class — NumPy float columns silently coerce pd.NA to NaN.
• Prefer Series methods over np.* functions on a Series (including via .apply), and never use .values (the PD011 lint flags it) — reach for .to_numpy() only at a boundary needing a raw array.
• Don't store NaN as a sentinel; model "missing" explicitly with a nullable dtype.
• Keeping everything Pandas-native also keeps a future move to Polars tractable.

Pandera schemas

• Lean on Pandera: type-hint every DataFrame parameter and return with DataFrame[Model]. The payoff is readability — the schema becomes explicit at every reference — so use it ubiquitously, not sparingly.
• Give each schema model its own module, and keep the raw (as-ingested) schema in a separate module from the processed (validated or derived) one.
• Back categorical columns with a Category dtype built from an Enum's values (iterate the Enum to build the categories), and annotate timestamp columns as pd.Timestamp.