Visualizing High Dimensional Decision Boundary into a 3D Space

I had been searching for so long on google for solutions to visualize a high dimensional decision boundaries, but it seems no so many people having done that. Actually It is neccesary especially for making sense of adversarial attacks.

The data preparation part is omitted, because it differs in various datasets. The data I used is AMDds for power data classification, which has a class 'inactive' and class 'active'. The two classes are imbalanced. You could just consider this dataset as a numpy variable X with 60 dimensions, Y is the label for X.

The targeted model is a well-trained keras deep neural network.

from tensorflow.keras.models import load_model
model=load_model('power_classification_model.h5')

Step1: Use PCA to fit the dataset, reducing the dimension from 60 to 2.

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
conponent = pca.fit_transform(X)
new_data = np.concatenate([conponent,Y.reshape([-1,1])],axis=1)
principalDf = pd.DataFrame(data = new_data
             , columns = ['principal component 1', 'principal component 2','target'])
principalDf.head()

Step2: Visualize the dataset on the 2d space to decide the range of the 2 components:

groups = principalDf.groupby('target')
fig = plt.figure() 
ax = fig.add_subplot()
colors = ['#606eca','#c13550']
i=0
for name, group in groups:
    ax.plot(group['principal component 1'], group['principal component 2'], 
              marker='o', linestyle='', ms=1, label=name,color=colors[i])
    i=i+1
ax.legend(['Inactive','Active'])
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.show()

 

Step3: Within the range of two components, create many points on grids. I choose a relatively low granularity to save some computational power, you can generate more points.

X1 = np.arange(-10000, 50000, 1000)
X2 = np.arange(-30000, 30000, 1000)
xx, yy = np.meshgrid(X1, X2)
Xfull = np.c_[xx.ravel(), yy.ravel()]

Step4: Inverse the PCA procedure to recover the grid points in the original space (from 2D to 60D), and the results for these points can be derived by querying the targeted model.

prob_mat = np.zeros(Xfull.shape[0])
for i in range (Xfull.shape[0]):  
  original_signal = pca.inverse_transform(Xfull[i,:])
  prob = model.predict_proba(original_signal.reshape([-1,60]))
  prob_mat[i] = prob[0,0]
  print(i)
prob_new = prob_mat.reshape([60,60])

Step5: Visualize the 3d surface.

fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(xx, yy, prob_new, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False,alpha=0.8)
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.show()