The Regression analysis is a statistical procedure for estimating a relationship between a dependent variable and one or more independent variables using data from observations.
Many types of regression methods exist depending on the nature of the variables and the functional form of the assumed relationship between them.
The methods of regression analysis play a central role in the statistical analysis of data, almost in every branch of basic science and engineering. The regression analysis forms one of the pillers of machine learning in the modern age.
In this section, we will confine ourselves to the description of four regression methods namely Simple Linear Regression, Multiple Linear Legression,
Logistic Regression and Nonlinear Regression .
For each type of regression analysis, we will develope the basic theory leading to the final formulae for regression. These derivations are best done employing the tools of linear algebra like matrices. After establishing the best parameters for the assumed functions from the analysis, we will learn to perform appropriate statistical tests under the null hypothesis that the original variables were uncorrelated to get an accidental probability of observing the values of the parameters we obtined. We will also learn to compute other parameters like $r^2$ which gives an estimate of the extent to which our assumed curve passes cloer to the observed data points.
Finally, for each regression method, we give a specific example calculation for an actual data followed by performing the same regression analysis in R statistical package.
Our aim is to understand the theoretical ideas of regression methods from first principles and to acquire the skills of performing the analysis in R statistical package.
The theoretical ideas behind each method is very important in understading the purpose, strength and limitation of the algorithm and the assumptions behind their usage.
A familiarity with the fundamentals of calculus and linear algebra (vectors and matrices) is a prerequisite for understanding the theory of linear regression presented here.
We hope that this set of lesseons will be very useful to enter the world of regression analysis, which is an important component of machine learning and AI.