Linear Regression in R: lm(), summary() and predict()

The Idea: the Least-Squares Line

Linear regression fits a straight line through a cloud of points: y = intercept + slope × x. Out of every possible line, lm() picks the one minimizing the sum of squared residuals - a residual being the vertical gap between a point and the line. Squaring makes big misses count disproportionately, which is why one wild outlier can tilt the whole fit.

Where correlation gives you one unitless number for "how tightly do these move together," regression gives you an equation - with units, a slope you can interpret, and the machinery to predict.

fit <- lm(mpg ~ wt, data = mtcars)
print(coef(fit))

Read the formula as "model mpg as a function of wt." The two coefficients are the fitted line: mpg ≈ 37.3 − 5.3 × weight. The slope has real units - each extra 1000 lbs of car (wt is in 1000-lb units) costs about 5.3 miles per gallon. The intercept (37.3 mpg at weight zero) is just where the line crosses zero; no car weighs nothing, so don't over-read it.

The summary() Walkthrough

summary(fit) is the output every stats course asks you to interpret. Run it, then take it block by block:

fit <- lm(mpg ~ wt, data = mtcars)
summary(fit)

Call - echoes the model you fit. Trivial now, a lifesaver when you're juggling six model objects.

Residuals - the five-number summary of the leftovers (actual − predicted). You want the median near 0 and rough symmetry between Min/Max and 1Q/3Q; strong asymmetry hints the straight-line model is missing something.

Coefficients - the heart of the output, one row per term:

• Estimate - the fitted value. For wt, −5.34: each additional 1000 lbs is associated with about 5.3 fewer mpg. Always translate the slope into a sentence with units; that sentence is the model's entire practical content.
• Std. Error - how much the estimate would wobble across repeated samples. Estimates within a couple of standard errors of zero are shaky.
• t value - Estimate ÷ Std. Error: how many standard errors the coefficient sits from zero (−9.56 here).
• Pr(>|t|) - the p-value for "could this coefficient really be zero?" For wt it's about 1.3e-10: if weight truly had no linear relationship with mpg, a slope this steep would essentially never appear in a sample of 32. Small p-value = evidence the association exists - not proof the model is correct, and not a measure of importance (a tiny, precisely-estimated effect gets a tiny p-value too).
• Signif. codes / stars - a visual shorthand for the p-value column. Convenient; adds no information.

Residual standard error: 3.05 on 30 degrees of freedom - the typical size of a prediction miss, in the response's own units: predictions are typically off by about 3 mpg. Judge it against the scale of mpg (which ranges about 10-34).

Multiple R-squared: 0.75 - weight explains about 75% of the variance in mpg. Adjusted R-squared (0.74) re-computes that with a penalty per predictor, because the raw version can only increase as you add variables - even random noise. When comparing models with different numbers of predictors, adjusted is the honest one. And resist the reflex that "good model = high R²": a genuinely useful effect can live in a low-R² model (noisy outcome, one of many drivers), while a high R² can come from an overfit or a leaked variable.

F-statistic: 91.4 ... p-value: 1.29e-10 - the whole-model test: does this model beat "just predict the mean for everyone"? With one predictor it duplicates the slope's t-test (note 9.56² ≈ 91.4); with several predictors it becomes the joint test that at least one coefficient is nonzero. Its machinery is the same variance decomposition as ANOVA.

Multiple Regression: Holding Others Constant

Add predictors with +:

fit2 <- lm(mpg ~ wt + hp, data = mtcars)
summary(fit2)

The interpretation changes in one crucial way. Each Estimate is now the effect of that predictor holding the others constant: the wt coefficient (about −3.9, down from −5.3) is the mpg cost of extra weight comparing cars with the same horsepower. The simple regression's −5.3 silently bundled in the fact that heavier cars also tend to be more powerful; the multiple regression unbundles it. This is also why coefficients shift when you add variables - if the new predictor correlates with an old one, the old one's job description changes. R-squared climbs to about 0.83, and here the adjusted version is the fair comparison against the single-predictor model.

Predictions: predict()

The fitted model is a function; predict() evaluates it. Build a newdata data frame whose column names match the predictors exactly:

fit <- lm(mpg ~ wt, data = mtcars)

new_cars <- data.frame(wt = c(2.5, 3.5))
print(predict(fit, newdata = new_cars))

print(predict(fit, newdata = new_cars, interval = "confidence"))
print(predict(fit, newdata = new_cars, interval = "prediction"))

The two interval types answer different questions, and mixing them up is a classic exam mistake:

• interval = "confidence" - uncertainty about the average: "for all cars weighing 2500 lbs, where is the mean mpg?" Narrow, and shrinks as data grows.
• interval = "prediction" - the range where an individual new car of that weight is likely to land. Much wider, because a single car carries its own scatter around the line - scatter that no amount of data averages away.

Reporting a confidence interval when the question is about one new observation dramatically overstates your precision.

Diagnostics and the Extrapolation Trap

summary() tells you what the model estimates; the residual plots tell you whether to believe it. In an interactive session:

par(mfrow = c(2, 2))
plot(fit)   # four diagnostic plots

What to look for: Residuals vs Fitted should be a shapeless cloud - a curve means the relationship isn't straight; a funnel (spread growing with fitted values) means non-constant variance, and your standard errors are off. Q-Q plot points should hug the line - heavy tails mean outliers are distorting the fit. Scale-Location is the funnel check again. Residuals vs Leverage flags influential points - observations that, alone, drag the coefficients (in mtcars, exotic cars like the Chrysler Imperial tend to show up here). A quick scatter plot of the raw data before fitting catches most of this early.

Finally, the trap that no diagnostic catches: extrapolation. The model learned from cars weighing roughly 1500-5400 lbs. Feed predict() a wt of 8 and it will cheerfully return a negative mpg - the math extends the line forever, but the evidence stops at the edge of the data. Predict only within (or near) the range you fit on.

What You Take Away

• fit <- lm(y ~ x, data = df) fits the least-squares line; coef(fit) is the equation, summary(fit) the full report.
• Read Estimates as sentences with units; Pr(>|t|) asks "could this be zero?", not "does this matter?"
• Residual standard error is the typical miss in real units; adjusted R-squared is the fair model-comparison number.
• In multiple regression every coefficient means "holding the others constant" - and coefficients shift when correlated predictors join.
• predict(fit, newdata, interval = ...): "confidence" for the mean, "prediction" for one new case - the wide one.
• Check plot(fit) for curves, funnels and influential points; never trust predictions outside the data's range.

Next up: when the outcome is a yes/no instead of a number - logistic regression with glm().