Linear Regression Vs Closed Form Ordinary Least Squares In Python

I am trying to apply Linear Regression method for a dataset of 9 sample with around 50 features using python. I have tried different methodology for Linear Regression i.e Closed fo

Solution 1:

Check out the docs of sklearn's LinearRegression again.

By default (like you call it), it also fits an intercept term!

Demo:

import numpy as np
from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression

X, y = load_boston(return_X_y=True)

""" OLS custom """
w = np.dot(np.linalg.pinv(X), y)
print('custom')
print(w)

""" sklearn's LinearRegression (default) """
clf = LinearRegression()
print('sklearn default')
print(clf.fit(X, y).coef_)


""" sklearn's LinearRegression (no intercept-fitting) """
print('sklearn fit_intercept=False')
clf = LinearRegression(fit_intercept=False)
print(clf.fit(X, y).coef_)

Output:

custom
[ -9.16297843e-02   4.86751203e-02  -3.77930006e-03   2.85636751e+00
  -2.88077933e+00   5.92521432e+00  -7.22447929e-03  -9.67995240e-01
   1.70443393e-01  -9.38925373e-03  -3.92425680e-01   1.49832102e-02
  -4.16972624e-01]
sklearn default
[ -1.07170557e-01   4.63952195e-02   2.08602395e-02   2.68856140e+00
  -1.77957587e+01   3.80475246e+00   7.51061703e-04  -1.47575880e+00
   3.05655038e-01  -1.23293463e-02  -9.53463555e-01   9.39251272e-03
  -5.25466633e-01]
sklearn fit_intercept=False
[ -9.16297843e-02   4.86751203e-02  -3.77930006e-03   2.85636751e+00
  -2.88077933e+00   5.92521432e+00  -7.22447929e-03  -9.67995240e-01
   1.70443393e-01  -9.38925373e-03  -3.92425680e-01   1.49832102e-02
  -4.16972624e-01]