shift and diff
s.shift(1) moves values down one row so each row sees the previous value (NaN in the first row); s.diff() is s - s.shift(1), the change from the previous row, and s.pct_change() is the relative change. Negative periods look forward instead.
Syntax
Series.shift(periods=1) | Series.diff(periods=1) | Series.pct_change(periods=1)| Part | What it is |
|---|---|
periods | How many rows to shift or compare across; negative looks ahead |
fill_value | shift only: value used instead of NaN for the vacated rows |
Examples
import pandas as pd
df = pd.DataFrame({"month": ["Jan", "Feb", "Mar", "Apr"], "sales": [100, 120, 90, 135]})
df["prev"] = df["sales"].shift(1)
df["change"] = df["sales"].diff()
dfResult: month sales prev change 0 Jan 100 NaN NaN 1 Feb 120 100.0 20.0 2 Mar 90 120.0 -30.0 3 Apr 135 90.0 45.0
df["growth"] = df["sales"].pct_change().round(3)
df["growth"].tolist()Result: [nan, 0.2, -0.25, 0.5]
Feb is 20% above Jan, Mar is 25% below Feb, Apr is 50% above Mar.
df["sales"].shift(-1).tolist()Result: [120.0, 90.0, 135.0, nan]
Negative periods bring the next row up.
Common mistakes
- Integer column becomes float after shift: NaN forces float; pass fill_value=0 if that is acceptable.
- Lags cross customer or product boundaries: use groupby(...).shift() instead of a plain shift.
- Wrong results because the rows are not in time order: sort by date before shifting.
Practice shift and diff
These drills check your answer on live data, free, no account needed.
- Month over month growthReturn the month over month change in `revenue` as a fraction (0.25 means 25 percent growth). The first month has no prior month and is NaN.
- Change from the previous periodReturn the change in `revenue` from the previous month (this month minus last month), using `shift`. The first month is NaN.
Questions
How do I shift by time instead of rows?
On a DatetimeIndex use s.shift(freq="D") to move the index by one day, keeping values attached to their new timestamps.
How do I compute year-over-year change on monthly data?
s.pct_change(12) compares each month with the same month a year earlier, assuming one row per month with no gaps.
Related
- rollingHow to compute a moving average in pandas with rolling: window size, min_periods to avoid leading NaN, centered windows, rolling sum and std, time windows.
- cumsumHow to compute a running total in pandas with cumsum, cumulative sums per group with groupby, cumulative max and count, cumulative percentages, and resets.
- resampleHow to resample a time series in pandas: monthly, weekly or daily buckets with sum or mean, upsample with ffill, use a date column with on, and new aliases.
- groupbyHow groupby works in pandas: group by one or more columns, apply sum, mean, count or size, get a flat table with as_index=False, and iterate over groups.