Predicting with Regression: Assumptions and Influential Points

Suppose an economics department at a Delhi college has years of data linking students’ attendance percentage with their final exam marks. A professor with a new batch’s attendance figures wants to know: can this relationship actually predict marks for a specific student? Correlation only tells you two variables move together. Prediction requires a working formula, a set of conditions for trusting it, and an honest look at whether a few unusual data points are quietly distorting the whole picture.

Turning correlation into a working formula 🔗

Once you know the correlation coefficient (r) along with the means and standard deviations of both variables, the regression coefficients can be rewritten into a practical, ready-to-use form. To predict Y from X, the equation is:

Y − Ȳ = r (σY / σX) (X − X̄)

Here, Ȳ and X̄ are the means of Y and X, while σY and σX are their standard deviations. Plug in a specific X value, and the equation returns the predicted Y. To predict X from Y instead, the roles simply reverse:

X − X̄ = r (σX / σY) (Y − Ȳ)

Go back to the attendance-and-marks example. If the average attendance is 75%, average marks are 60, and the correlation is a healthy 0.7, a student with 90% attendance can get a specific predicted mark just by substituting the numbers. This is exactly the logic used in far larger applications too, such as regression-based models that help forecast agricultural commodity prices in markets across Odisha, where past price and yield patterns are used to anticipate future values for farmers and traders.

Why the two equations aren’t the same line 🔗

A common assumption is that “the regression line” is a single, fixed line. It isn’t. The line for predicting Y from X and the line for predicting X from Y are generally different, because each one minimises errors in a different direction: one minimises vertical distances (errors in Y), the other minimises horizontal distances (errors in X). The two lines coincide only when the correlation is perfect, that is, r equals 1 or −1. This distinction matters practically. A line built to predict a student’s marks from attendance shouldn’t be flipped around and used to predict attendance from marks; a fresh calculation is needed for that direction.

The assumptions that keep predictions trustworthy 🔗

A regression equation can be computed from almost any dataset, but that doesn’t automatically make its predictions reliable. Three conditions need to broadly hold before the numbers coming out of the formula can be trusted.

Y values should cluster normally around the line 🔗

For any fixed value of X, the corresponding Y values are assumed to be normally distributed around the regression line, forming a bell-shaped spread rather than a lopsided or erratic one. This is closely tied to how residuals, the gaps between actual and predicted values, are expected to behave. When residuals depart sharply from a normal distribution, it is often a signal, as regression diagnostics literature notes, that something else is wrong with the model or that a few unusual points need closer inspection.

Homoscedasticity: consistent spread across the range 🔗

The spread, or standard deviation, of Y should stay roughly the same across every value of X. This property is called homoscedasticity. If predictions are tight and accurate for small X values but scatter wildly for larger ones, the assumption is being violated, a pattern known as heteroscedasticity. Checking this is usually as simple as plotting residuals against fitted values: a random, evenly spread band suggests the assumption holds, while a funnel or cone shape suggests it doesn’t, a diagnostic step commonly used in regression teaching.

Bivariate normal distribution 🔗

Beyond how Y behaves for a given X, both variables together should follow a bivariate normal distribution, meaning their joint spread across a scatterplot forms an elliptical, cloud-like shape rather than a skewed or clumped pattern. This assumption underlies the statistical tests used to judge whether the correlation itself is meaningful, and not just a quirk of one particular sample.

When these three conditions are reasonably met, predictions from the regression equation carry real statistical weight. When they are seriously violated, say, Y’s spread balloons for higher X values, or the joint distribution is heavily skewed, predictions can look precise on paper while being misleading in practice.

When a handful of points quietly change the story 🔗

Even with sound data, a dataset will often contain a point or two that looks out of place compared with the rest. These are outliers. Most outliers are harmless quirks that don’t meaningfully shift the regression line. A smaller subset, however, called influential points, can pull the entire line toward themselves, changing the slope, the intercept, or both.

Not all outliers pull equally hard 🔗

The direction of the outlier matters more than most people expect. A point that is unusual mainly in its Y value, sitting far above or below the general trend for its X value, is a vertical outlier. It typically produces a large residual but doesn’t drag the line very far, since the point still sits near the middle of the X range. A point that is unusual in its X value, sitting far to the left or right of the rest of the cloud, is a different matter altogether. Statisticians call this a high-leverage point, and because it sits at an extreme end of the X-axis, it has far more power to swing the line’s slope, even if it doesn’t look dramatically unusual in the Y direction. This is precisely why horizontal outliers are considered more likely to be influential than vertical ones.

Testing for influence directly 🔗

The most intuitive way to check whether a suspicious point is actually influential is to compute the regression line twice: once with the point included, and once with it removed. If the slope and intercept barely move, the point isn’t influential, regardless of how unusual it looked on the scatterplot. If the line shifts noticeably, the point is influential and deserves a closer look before it’s treated as routine data. As explained in resources on distinguishing outliers from high-leverage observations, a point can have high leverage without being influential, and a point can be a clear outlier without swinging the line much; influence only shows up when leverage and a large residual combine. A related walkthrough on identifying influential observations illustrates this with side-by-side comparisons of regression lines fitted with and without a suspect point, making the visual difference easy to spot.

Deciding what to do with an influential point 🔗

Finding an influential point isn’t the end of the analysis; it’s the start of a judgment call. Two questions guide that call: is the point a genuine, valid observation, or is it the result of a data entry mistake, faulty instrument, or a case that simply doesn’t belong to the population being studied?

Consider a dataset tracking rainfall against crop yield across districts, where one district reports an implausibly high yield figure due to a reporting error in the agricultural survey. If that figure is traced back to a clerical mistake, excluding it from the final regression is reasonable and should be noted transparently. But if a similar outlier represents a real district that genuinely had unusual irrigation infrastructure or an exceptional monsoon year, removing it just because it’s inconvenient would be dishonest. In agricultural forecasting studies across Indian states, researchers routinely flag such genuine extreme years rather than discarding them, since real-world price and yield data often carries legitimate volatility that a forecasting model needs to account for, not hide.

The responsible approach, then, is threefold: check whether the point is a leverage point, a vertical outlier, or both; test whether removing it meaningfully changes the regression line; and, if it stays in the analysis, report its influence openly rather than presenting the final regression equation as if that single point didn’t matter. Predictions built this way are more defensible, because anyone reviewing the analysis can see exactly how much of the result rests on ordinary data and how much rests on one unusual case.

Bringing it together 🔗

Regression equations turn a correlation coefficient into something usable: an actual predicted value for Y given X, or X given Y. But that predictive power only holds up when the underlying assumptions, normally distributed Y values for each X, homoscedasticity, and joint bivariate normality, are reasonably satisfied. And even when they are, a single high-leverage point sitting far out on the X-axis can quietly rewrite the slope of the line unless it’s specifically tested and either justified or excluded with a clear reason.

What do you think? If you were analysing a dataset and found one data point dramatically changing your regression line, how would you decide whether it reflects genuine variation worth keeping, or an error worth removing? And in situations like exam scores or sales figures, where would you expect horizontal outliers to be more common than vertical ones?

References

1. https://pmc.ncbi.nlm.nih.gov/articles/PMC9258887/
2. https://people.duke.edu/~rnau/testing.htm
3. https://stats.oarc.ucla.edu/spss/seminars/introduction-to-regression-with-spss/introreg-lesson2/
4. https://www.sciencedirect.com/topics/mathematics/leverage-point
5. https://online.stat.psu.edu/stat501/lesson/11/11.1
6. https://www.bookdown.org/rwnahhas/RMPH/mlr-influence.html