What Dense Layer is Capable of
Building a new model in Deep Learning is not easy. Figuring out why a model is not acting as expected is difficult. It is important to know how what each component is capable of. Dense layer is one of the most fundamental component of Machine Learning. Here, we explore what it is capable of through series of simple and easy-to-interpret experiments.
Problem
Let's consider a model that maps one-hot vectors into indices.
$$ \begin{bmatrix} [1 & 0 & 0] \\ [0 & 1 & 0] \\ [0 & 0 & 1] \end{bmatrix} \rightarrow \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix} $$This can be formulated as linear regression. It is easy to write this mapping in equation.
$$ \mathbf{x} \cdot \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \rightarrow y $$And there are different ways to express the same mapping using bias term.
$$ \mathbf{x} \cdot \begin{bmatrix} -1 \\ 0 \\ 1 \end{bmatrix} + \begin{bmatrix} 2 \\ 2 \\ 2 \end{bmatrix} \rightarrow y $$Experiment
This simple linear transformation is called Dense Layer or Fully-Connected Layer in Deep Learning, and in Tensorflow such model can be written as follow.
output_variable = tf.layers.dense(
input_variablbe, units=n_dim, activation=None, use_bias=use_bias)You can find the code for the experiment here.
Simple gradient descent is enough for finding the optima. The following figure shows training loss with different loss type and bias condition for 15 dimension case.
You can click legend to hide/show the corresponding plot.
The following figure illustrates the model output through the training.
Observation
There are couple of interesting this we can observe.- Bias learns faster than kernel.
- Existence of bias does not affect the speed kernel fits.
- Model cannot truely converge to optima when L1 loss is used.
- Shuffling the output axis in arbitrary order does not affect model training.
Bias terms converge to optima before kernel terms do.
You can disable plots other than Loss: L2, Bias: [0.] to see this.
Comparing the same loss type with/without bias term, the step model converges to the optima is same.
In fact the loss curve of Bias: [0,] starts at the same loss as Bias: None but quickly merges to Bias: [6.]. For L2 loss types, all the training cases converge around 1000 trainings.
With L1 loss, derivetive value is always constant and when that value is larger than the actual error, parameters keep crossing the optimal value and fructuates.
When label values are shuffled in arbibtrary order, model training is not affected.
Extention
Extending this to multi dimension, we get the same property on each axis independently.
$$ \begin{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix} \\ \begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} \\ \begin{bmatrix} 0 & 0 & 1 \\ 1 & 0 & 0 \end{bmatrix} \end{bmatrix} \rightarrow \begin{bmatrix} [1 & 2]\\ [1 & 3]\\ [3 & 1] \end{bmatrix} $$