Data-cleaning and profiling scripts in Python you can audit
The first pass on a new dataset is always the same grind: check the shape, dtypes, missing values, duplicates, and outliers before any real analysis can start. Analysts now describe the dataset's columns and hand AI the profiling and cleaning code, cutting a task that took an afternoon to well under an hour — provided the script is auditable and no raw data leaves the building.
You are a data analyst fluent in pandas. Write Python to profile and clean a dataset. I will run it myself against the real data. Columns and dtypes (no live data, just the structure): {{data_description}} My cleaning objective: {{objective}} Rules: - Work from the described structure only. Do not assume distributions, category values, or ranges you cannot see. Where a decision depends on the actual data, emit a printed summary or an assert, not a hard-coded assumption. - Every transformation must be reversible and logged: print the before/after row and null counts for each step, and never drop rows silently — flag them into a separate frame I can inspect. - Handle the classic traps explicitly: text-formatted numbers, mixed date formats, whitespace and case in categoricals, and division-by-zero in any derived metric. - Put every threshold I might want to change (outlier cutoff, null strategy) in clearly named variables at the top. - End with a short data-quality report: rows in vs. out, columns affected, and anything I should decide on manually. - Add a one-line comment on every non-obvious step. Keep it readable over clever.
Fill in your details and the prompt updates live — then copy.
# --- knobs you can change --- NULL_STRATEGY = "flag" # flag, drop, or fill OUTLIER_Z = 3.0 before = len(df) df["plan"] = df["plan"].str.strip().str.lower().map( {"free": "Free", "pro": "Pro"}).fillna(df["plan"]) df["mrr"] = pd.to_numeric( df["mrr"].str.replace(r"[$,]", "", regex=True), errors="coerce") # rows where mrr failed to parse are kept and flagged, not dropped: bad_mrr = df[df["mrr"].isna()] print(f"rows in={before} unparseable mrr={len(bad_mrr)}") dupes = df[df.duplicated("customer_id", keep=False)] print(f"duplicate customer_ids: {dupes['customer_id'].nunique()} ids, {len(dupes)} rows") # Data-quality report: review bad_mrr and dupes before analysis.
The full workflow
- Describe the columns and dtypes from your schema, not by pasting real rows
- Run the script on a copy of the data and read the printed before/after counts at each step
- Inspect every flagged frame (bad parses, duplicates, outliers) instead of trusting an auto-drop
- Recompute any derived metric by hand on a few rows before building analysis on top of it
Watch out for
AI is at its most dangerous on numbers: it produces confident, wrong answers on time-series and derived metrics rather than admitting uncertainty. Verify year-over-year math and any calculation with gaps or zeros yourself.
Do not upload files of real customer or employee data to a consumer AI tool to clean them. Use the described-schema pattern, or a synthetic or fully anonymized sample, on an approved instance — PII in a public tool can breach data-protection law and your own governance policy.
A cleaning script that changes row counts changes your results. Never accept a silent drop or fill — every transformation needs a logged before/after you actually read.
Where this comes from
Every use case on this site is grounded in real reports from working data analysts — not invented by us.