“Ordinary Least Squares” regression, or OLS for short, is a method for finding a best-fit line, given a set of data points. When people refer to linear regression without additional context or qualifiers, they’re probably referring to ordinary least squares regression.
So, what does “ordinary” mean in this context? Is it a mathematical term with a precise technical meaning?
Well, no. In this case “ordinary” means exactly what you might expect: standard, vanilla, lacking any bells and whistles. Since least squares regression is so common, there are lots of variants. Weighted Least Squares (in which each data point is given its own weight) is a such a variant. So, in OLS, the O for ordinary simply means we’re not referring to one of these variants.
What about “ordinary differential equations” (ODEs)? What does ordinary refer to there?
In that context, ordinary refers to ordinary derivatives, as opposed to partial derivatives. So again, it’s not a mathematical term like a reference to ordinal numbers.
In both cases, ordinary seems to take on its ordinary English definition – its OED, if I may.