Pandasdrilldrop_duplicates
df.drop_duplicates() removes rows that are exact copies of an earlier row, keeping the first occurrence. subset=["col"] treats rows as duplicates when only those columns match, and keep="last" or keep=False changes which copies survive.
Syntax
df.drop_duplicates(subset=None, keep="first", inplace=False, ignore_index=False)| Part | What it is |
|---|---|
subset | Column(s) used to decide what counts as a duplicate; default all columns |
keep | "first" (default), "last", or False to drop every copy |
ignore_index | True renumbers the result 0..n-1 |
Examples
import pandas as pd
df = pd.DataFrame({"email": ["a@x.com", "b@x.com", "a@x.com", "c@x.com"],
"plan": ["free", "pro", "pro", "free"]})
df.drop_duplicates()Result: email plan 0 a@x.com free 1 b@x.com pro 2 a@x.com pro 3 c@x.com free
No full-row duplicates, so nothing is removed: rows 0 and 2 differ in plan.
df.drop_duplicates(subset=["email"], keep="last")Result: email plan 1 b@x.com pro 2 a@x.com pro 3 c@x.com free
df[df.duplicated(subset=["email"], keep=False)]Result: email plan 0 a@x.com free 2 a@x.com pro
duplicated with keep=False flags every copy, useful for inspecting before dropping.
Common mistakes
- Duplicates remain: values differ in case or trailing spaces; normalize with str.strip().str.lower() before deduplicating.
- Too many rows removed with keep=False: it drops all copies including the first; use keep="first" to keep one.
- The original frame still has duplicates: drop_duplicates returns a copy; assign the result or pass inplace=True.
Practice drop_duplicates
These drills check your answer on live data, free, no account needed.
- Remove duplicate recordsThe `leads` table has repeated emails. Keep only the most recent row for each email, which is the last one in the table. Keep the original row labels.
Questions
How do I count duplicate rows?
df.duplicated().sum() counts rows that are copies of an earlier row; add subset for specific columns.
How do I keep the row with the latest date per key?
Sort by date first, then drop: df.sort_values("date").drop_duplicates("id", keep="last").
How do I get unique values of one column?
df["col"].unique() returns an array; df["col"].drop_duplicates() returns a Series keeping the index.
Related
- unique and nuniqueHow to get distinct values in pandas with unique, count them with nunique, count distinct per group with groupby, handle NaN, and see counts for all columns.
- dropnaHow to drop NaN rows in pandas with dropna: any or all missing, only certain columns with subset, drop columns with axis=1, thresh for a minimum of values.
- value_countsHow to count values in a pandas column with value_counts: counts sorted by frequency, percentages with normalize=True, including NaN, and two-column combos.
- sort_valuesHow to sort a pandas DataFrame with sort_values: ascending and descending, several columns with mixed order, NaN first, sort by index and reset the index.