Least squares

In regression analysis, least squares is a good way to find the best model for a set of data points. It works by making the total of the squares of the differences between the real data and the model's predictions as small as possible. These differences are called residuals.

There are two main types of least squares problems. One type is called linear or ordinary least squares. This happens when the model is straight or linear. The other type is nonlinear least squares, which is solved step-by-step.

Polynomial least squares helps us understand how much a prediction can change based on the values we use. This method is important in science and math because it helps us make good guesses from data.

History

The method of least squares started from many ideas in the 1700s. Early thinkers like Isaac Newton and Roger Cotes suggested using many observations to get better estimates. Others, such as Tobias Mayer and Pierre-Simon Laplace, used these ideas to study the movements of planets like Jupiter and Saturn.

The method became clear when Legendre published it in 1805. Soon after, it was used a lot in astronomy and mapping in France, Italy, and Prussia. Carl Friedrich Gauss improved the method by linking it to probability, showing that it could predict positions very well — like finding the asteroid Ceres after it disappeared behind the Sun.

Problem statement

The goal is to make a model match a set of data points as well as possible. Each data point has an input, like hours studied, and an output, like a test score. We want to find the best way to link these inputs and outputs with a model.

The least-squares method helps by looking at the differences between what we see and what the model predicts. We square these differences and add them up. Then we pick the model that makes this total as small as possible. This gives us the best-fit model for our data.

For simple models, like a straight line, this method gives us the average of the data. For more complex models, we may need to adjust more parts to get the best fit.

Limitations

This method only looks at errors in what we are measuring. Another method, called total least squares, can look at errors in all variables.

There are two main reasons to use this method:

• For making predictions. We use the data we have to create a rule to guess future values.
• For finding a true relationship. Normally, we think there are no errors in the data we start with. But when there are errors, special methods can help us get better results. Using total least squares is one way to balance different types of errors.

Main articles: Regression analysis, Independent variable, Models of measurement error, Parameter estimates, Hypothesis testing, Confidence intervals

Solving the least squares problem

Least squares is a way to find the best fit for data. It makes the total of the squared differences between what we observe and what our model predicts as small as possible. We do this by setting a special math rule, called the gradient, to zero. This helps us find the best values for our model.

There are two main types of least squares problems: linear and nonlinear. In linear least squares, the model is a straight combination of simple pieces, and we can find the answer directly using math formulas. In nonlinear least squares, the model is more complex, and we usually need to start with guesses and improve them step by step.

Example

Imagine you have a spring, and you want to know how much it stretches when you push on it with different amounts of force. Scientists use something called Hooke's law. This law says the stretch of the spring, called "y", depends on the force, called "F", and a special number called "k".

To find this number "k", they test the spring many times with different forces and measure how much it stretches each time. But each test has a tiny bit of error. So, they use a method called least squares. This method looks at all the tests and finds the value of "k" that makes the total of the squares of these tiny errors as small as possible. Once they have this best guess for "k", they can predict how much the spring will stretch for any force using Hooke's law.

Uncertainty quantification

In least squares calculations, we try to understand how sure we can be about our results. We look at how spread out the possible values might be for each part of our answer. This helps us know how reliable our findings are.

We use a special number called the error variance to estimate this spread. We replace the true error variance with an estimate based on how well our model fits the data. This gives us a way to measure the uncertainty in our results. The number of data points we have compared to the number of things we’re trying to find also affects this uncertainty.

Statistical testing

When we know how the numbers in our data are spread out, we can find ranges for our guesses and test our results. If we assume that the small differences between what we measure and what we predict are spread out in a certain way, it makes our guesses more reliable.

A key idea is that these small differences are spread out evenly. This helps us know that our best guesses are the ones with the least changes. Even if these differences aren’t spread out perfectly, with enough data, our guesses will still be close to the right answer.

Weighted least squares

Main article: Weighted least squares

Weighted least squares is a special way to find the best fit for data. It is used when the spread of the data points changes depending on their value. Weighted least squares helps adjust for this to give a better result.

Relationship to principal components

The first principal component shows the line that comes closest to a group of points. It measures how far each point is from the line, looking at all directions equally.

Linear least squares only focuses on reducing the distance in one specific direction. So, while both methods try to reduce errors, linear least squares treats the data differently than principal components.

Relationship to measure theory

A well-known statistician named Sara van de Geer used ideas from empirical process theory and the Vapnik–Chervonenkis dimension to show that the least-squares method is a special kind of measure on a space of square-integrable functions. This helps explain the math behind finding the best fits in data.

Regularization

Main article: Regularized least squares

Main article: Tikhonov regularization

Sometimes, we use a special version of the least squares method to get better results. This version is called Tikhonov regularization or ridge regression. It adds a rule to keep the numbers in our model small. This helps stop the model from becoming too complicated.

Another method is called Lasso. It also keeps the numbers in our model small, but in a different way. Unlike ridge regression, Lasso can make some numbers become zero. This helps us focus on the most important parts of our data. Lasso is useful for finding which pieces of information are most important in our model.