This guide is for students learning regression in high school statistics or data science classes, university students fitting calibration lines in lab reports, and anyone who wants to check a spreadsheet result by hand.
“The letters a and b keep swapping places.” “I cannot explain the difference between r and R².” “Excel gave me a slope and intercept, but I want to verify them.” This guide walks through the least squares method with a sum table and the working shown. The datasets are teaching examples, not measured experimental results.
The three formulas you need
For data (x₁, y₁), …, (xₙ, yₙ), write the regression line as y = ax + b (a: slope, b: intercept). Compute in this order, where x̄ and ȳ are the means of x and y.
| Quantity | Formula | Meaning |
|---|---|---|
| Slope a | a = Sxy / Sxx | Change in y per unit increase in x |
| Intercept b | b = ȳ − a·x̄ | Where the line crosses the y axis |
| Correlation r | r = Sxy / √(Sxx·Syy) | Strength and direction of the linear relationship (−1 to 1) |
| Coefficient of determination R² | R² = r² | Share of the variation in y explained by the line (0 to 1) |
Sxx, Sxy and Syy are sums of squared deviations and products. You can compute them from the raw sums without subtracting the mean point by point:
- Sxx = Σx² − (Σx)² / n
- Sxy = Σxy − (Σx·Σy) / n
- Syy = Σy² − (Σy)² / n
The workflow is: ① add x² and xy columns → ② sum each column → ③ compute the means → ④ compute Sxx and Sxy → ⑤ find a and b → ⑥ find r and R². The first example follows every step. For means and standard deviations themselves, see the statistics guide; for rounding measured values, see the significant figures guide.
Example 1: fitting a line to five points
Use (x, y) = (1, 2), (2, 4), (3, 5), (4, 4), (5, 5).
| # | x | y | x² | xy |
|---|---|---|---|---|
| 1 | 1 | 2 | 1 | 2 |
| 2 | 2 | 4 | 4 | 8 |
| 3 | 3 | 5 | 9 | 15 |
| 4 | 4 | 4 | 16 | 16 |
| 5 | 5 | 5 | 25 | 25 |
| Sum | 15 | 20 | 55 | 66 |
- Means: x̄ = 15 ÷ 5 = 3, ȳ = 20 ÷ 5 = 4
- Sxx = 55 − 15² ÷ 5 = 55 − 45 = 10
- Sxy = 66 − 15×20 ÷ 5 = 66 − 60 = 6
- Slope: a = 6 ÷ 10 = 0.6
- Intercept: b = 4 − 0.6×3 = 2.2
- Regression line: y = 0.6x + 2.2
For the correlation, Syy = Σy² − (Σy)²/n. Since Σy² = 4+16+25+16+25 = 86, Syy = 86 − 80 = 6. Therefore
- r = 6 ÷ √(10×6) = 6 ÷ √60 ≈ 0.7746
- R² = 0.7746² ≈ 0.6
R² = 0.6 means the line explains roughly 60% of the variation in y. The residuals also sum to zero, as shown below.
Example 2: a calibration line
For (0, 0.1), (1, 1.1), (2, 1.9), (3, 3.2), (4, 4.0): Σx = 10, Σy = 10.3, Σx² = 30, Σxy = 30.5, n = 5.
- x̄ = 2, ȳ = 2.06
- Sxx = 30 − 10² ÷ 5 = 10
- Sxy = 30.5 − 10×10.3 ÷ 5 = 30.5 − 20.6 = 9.9
- a = 9.9 ÷ 10 = 0.99
- b = 2.06 − 0.99×2 = 0.08
- Regression line: y = 0.99x + 0.08
Syy = 31.07 − 21.218 = 9.852, so r = 9.9 ÷ √(10×9.852) ≈ 0.9974 and R² ≈ 0.9948. At x = 5 the predicted value is 0.99×5 + 0.08 = 5.03, but x = 5 lies outside the data range (0 to 4), so this prediction is an extrapolation.
Check the residuals
A residual is the observed value minus the predicted value, y − ŷ. For example 1 (y = 0.6x + 2.2):
| x | y | Predicted ŷ | Residual y − ŷ |
|---|---|---|---|
| 1 | 2 | 2.8 | −0.8 |
| 2 | 4 | 3.4 | 0.6 |
| 3 | 5 | 4.0 | 1.0 |
| 4 | 4 | 4.6 | −0.6 |
| 5 | 5 | 5.2 | −0.2 |
| Sum | 0 |
For a least squares line with an intercept, the residuals always sum to zero. If yours do not, check the arithmetic or confirm that you did not fit a line through the origin. The residual table also shows which points sit far from the line, i.e. outlier candidates.
The a/b notation trap
This guide and the Tools Hub calculator display y = ax + b (a: slope, b: intercept). Many statistics textbooks, graphing calculators and OpenStax write ŷ = a + bx (a: intercept, b: slope), which swaps the letters. The numbers are the same either way, but always state which letter is the slope and which is the intercept in your answer.
Common mistakes
- Swapping x and y: the line for predicting y from x is not the same as the line for predicting x from y unless r = ±1. Decide which variable is explanatory and which is the response first
- Forgetting to square r: the “share explained” is R² = r². If r = 0.7, then R² = 0.49, about 49%
- Removing outliers without a reason: least squares is strongly affected by outliers. If you exclude one, record the justification (for example a measurement error)
- Fitting a straight line to curved data: check the scatter plot first; a line through a curve gives a low r and R² and poor predictions
- Predicting outside the data range: extrapolation assumes the linear relationship continues, which is not guaranteed
- Reading correlation as causation: a strong correlation is not proof of cause and effect, and unrelated variables can show a spurious correlation
Check your work with the calculator
The Tools Hub least squares calculator mirrors both examples: paste the pairs and it shows the sums, Sxx, Sxy, a, b, r and R² with the working, plus a scatter plot and a residual table.
Check example 2
Press the "Calibration example (5 points)" preset to get y = 0.99x + 0.08, slope 0.99, intercept 0.08, r = 0.9974 and R² = 0.9948. The working shows Sxx = 10, Sxy = 9.9 and a = 9.9 ÷ 10 = 0.99, exactly as in this guide.
Predict and see the extrapolation warning
Enter 5 under "Predict y from x" to get ŷ = 5.03 with an extrapolation notice, because 5 is outside the data range. Enter 2 instead to get ŷ = 2.06 with no warning.
Inspect the residuals
Open "Data and residuals" to see the predicted value ŷ and the residual y − ŷ for every point, and compare them with your hand calculation. You can also paste your own pairs in the x, y format (for example 1, 3).
Tool used in this guide
Least Squares Regression
Paste data to get the regression line, r, R², a scatter plot and residuals with the working shown. Free, no sign-up, and everything runs in your browser.
Applicability and limits
- Least squares assumes a linear relationship between x and y. Plot the data first and check that a straight line is reasonable (OpenStax 12.2)
- Ordinary least squares minimises the vertical (y-direction) residuals. If x also carries non-negligible measurement error, this method may not be appropriate (NIST)
- The fit is strongly influenced by outliers; check the scatter plot and residual table
- Predicting outside the observed range (extrapolation) is less reliable because the linear relationship is not guaranteed to continue
- Correlation does not imply causation. Do not judge a relationship from the coefficient alone; inspect the scatter plot and the context (Statistics Bureau of Japan)
- R² = r² holds for a simple linear regression with an intercept. It can fail for curved fits or lines forced through the origin
FAQ
What is the least squares method?
It chooses the line that minimises the sum of squared vertical differences (residuals) between the data points and the line. Squaring stops positive and negative differences from cancelling out. The resulting line is the regression line.
Why are the slope and intercept letters different from my textbook?
This guide and calculator use y = ax + b (a: slope, b: intercept). Textbooks and calculators often write ŷ = a + bx (a: intercept, b: slope), so the same letters carry the opposite meaning. Whatever notation you use, state clearly which letter is the slope.
What is the difference between r and R²?
r measures the strength and direction of the linear relationship from −1 to 1. R² = r² estimates how much of the variation in y the line explains, from 0 to 1. If r = 0.7, then R² = 0.49, so about 49% is explained.
Can I fit a line with only two points?
Two points determine a line, so the arithmetic works, but you cannot judge scatter, outliers or measurement error, and r is always ±1. Use a scatter plot and enough data to see the trend.
How should I handle outliers?
Check the residual table for points that deviate strongly and keep them unless there is a documented reason to exclude them (for example a known measurement error). Removing inconvenient points without justification distorts the result.
Does least squares work for curved relationships?
The idea (minimising the sum of squared residuals) extends to curves, but this guide and the calculator fit straight lines only. A line through curved data gives a low r and R² and poor predictions.
How this guide was checked
On 5 October 2026 the definitions and formulas were checked against the sources above, and the sums, Sxx, Sxy, slope, intercept, r, R² and residuals for both examples were recalculated by hand. The datasets are teaching examples, not measured experimental results. The calculator inputs and outputs described here (presets, prediction, residual table) were also verified.
