Exercise 2.7

import numpy as np
import pandas as pd

d = {'X1': pd.Series([0,2,0,0,-1,1]),
'X2': pd.Series([3,0,1,1,0,1]),
'X3': pd.Series([0,0,3,2,1,1]),
'Y': pd.Series(['Red','Red','Red','Green','Green','Red'])}
df = pd.DataFrame(d)
df.index = np.arange(1, len(df) + 1)
df

X1 X2 X3 Y
1 0 3 0 Red
2 2 0 0 Red
3 0 1 3 Red
4 0 1 2 Green
5 -1 0 1 Green
6 1 1 1 Red

(a) Euclidian distance

from math import sqrt
df['distance']=np.sqrt(df['X1']**2+df['X2']**2+df['X3']**2)
df

X1 X2 X3 Y distance
1 0 3 0 Red 3.000000
2 2 0 0 Red 2.000000
3 0 1 3 Red 3.162278
4 0 1 2 Green 2.236068
5 -1 0 1 Green 1.414214
6 1 1 1 Red 1.732051

(b) K=1

df.sort_values(['distance'])

X1 X2 X3 Y distance
5 -1 0 1 Green 1.414214
6 1 1 1 Red 1.732051
2 2 0 0 Red 2.000000
4 0 1 2 Green 2.236068
1 0 3 0 Red 3.000000
3 0 1 3 Red 3.162278

As we can see by sorting the data by distance to the origin, for K=1, our prediction is Green, since that's the value of the nearest neighbor (point 5 at distance 1.41).

(c) K=3

On the other hand, for K=3 our prediction is Red, because that's the mode of the 3 nearest neigbours: Green, Red and Red (points 5, 6 and 2, respectively).

(d) Highly non-linear Bayes decision boundary

A large value of K leads to a smoother decision boundary, as if the non-linearities where averaged out. This happens because KNN uses majority voting and this means less emphasis on individual points. For a large value of K, we will likely have a decision boundary which varies very little from point to point, since the result of this majority voting would have to change while for most points this will be a large majority. That is, one of its nearest neighbors changing from one class to the other would still leave the majority voting the same. By contrast, when K is very small, the decision boundary will be better able to capture local non-linearities, because it the majority of neighbors can vary significantly from point to point since the are so few neighbors. Accordingly, we would expect the best value of K to be small.

Imagine this simple example: a true linear boundary that splits the plane in two semi-planes (classes A and B), but with an additional enclave in one of the regions. That is, the true model has a small region of, say, class A inside the class B semi-plane. Would we more likely capture this region for small or large K? Say we have 3 neighboring data points in the class A enclave inside the large class B region with say 50 points. If K > 4, each of the 3 points in the enclave will be classified as B, since they always lose by majority voting (unless K is so large, say 100, that many of the points from A semi-plane region enter this voting). If, on the other hand, K < 4, each of the 3 class A points inside the enclave will be classified as class A, and we will capture this non-linearity of the regions.