Exercise 4.13

This is similar to exercise 4.11. We define a new variable with the value of 1 if the crime rate is above the median and 0 if it is below.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns 

from sklearn.cross_validation import train_test_split
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis 
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis 
from sklearn.linear_model import LogisticRegression 
from sklearn.neighbors import KNeighborsClassifier 
from sklearn.metrics import accuracy_score

from sklearn.datasets import load_boston

%matplotlib inline

boston = load_boston()
df = pd.DataFrame(boston.data, columns=boston.feature_names)
df.head()

df['CRIM01'] = np.where(df['CRIM'] > df['CRIM'].median(), 1, 0)
df.head()

df = df.drop('CRIM', axis=1)
df.head()

As in 4.11, we try to assess which variables are more associated with 'CRIM01' by looking at correlations between the variables.

df.corr()

x_all = df.iloc[:,:-1].values
x_6 = df[['INDUS','NOX','AGE','DIS','RAD','TAX']].values
y = df['CRIM01'].values

x_all_train, x_all_test, y_train, y_test = train_test_split(x_all, y, random_state=1)
x_6_train, x_6_test, y_train, y_test = train_test_split(x_6, y, random_state=1)

lr = LogisticRegression()
lr.fit(x_6_train, y_train)
print("Accuracy with x_6  :",accuracy_score(y_test, lr.predict(x_6_test)))
lr.fit(x_all_train, y_train)
print("Accuracy with x_all:",accuracy_score(y_test, lr.predict(x_all_test)))

Accuracy with x_6  : 0.842519685039
Accuracy with x_all: 0.858267716535

lda = LinearDiscriminantAnalysis()
lda.fit(x_6_train, y_train)
print("Accuracy with x_6  :",accuracy_score(y_test, lda.predict(x_6_test)))
lda.fit(x_all_train, y_train)
print("Accuracy with x_all:",accuracy_score(y_test, lda.predict(x_all_test)))

Accuracy with x_6  : 0.818897637795
Accuracy with x_all: 0.834645669291

qda = QuadraticDiscriminantAnalysis()
qda.fit(x_6_train, y_train)
print("Accuracy with x_6  :",accuracy_score(y_test, qda.predict(x_6_test)))
qda.fit(x_all_train, y_train)
print("Accuracy with x_all:",accuracy_score(y_test, qda.predict(x_all_test)))

Accuracy with x_6  : 0.874015748031
Accuracy with x_all: 0.905511811024

best_acc = 0
best_k = 0
for K in range(1,101):
    knn = KNeighborsClassifier(n_neighbors=K)
    knn.fit(x_all_train, y_train)
    acc = accuracy_score(y_test, knn.predict(x_all_test))
    if acc > best_acc:
        best_acc, best_k = acc, K

print('Best accuracy = {:.4f} with K = {:3}'.format(best_acc, best_k))

Best accuracy = 0.9370 with K =   1

best_acc = 0
best_k = 0
for K in range(1,101):
    knn = KNeighborsClassifier(n_neighbors=K)
    knn.fit(x_6_train, y_train)
    acc = accuracy_score(y_test, knn.predict(x_6_test))
    if acc > best_acc:
        best_acc, best_k = acc, K

print('Best accuracy = {:.4f} with K = {:3}'.format(best_acc, best_k))

Best accuracy = 0.9370 with K =   1

We used two subsets of the predictors: all of them and the 6 most correlated with the response. From the results above, we can see that including all the predictors improves the accuracy (except for KNN) but not by much (less than 3%). QDA has better accuracy test performance than LDA, indicating a non-linear Bayes decision boundary. The best model was KNN with K = 1. For a more robust analysis we should perform cross-validation. Which leads us to the next chapter :)