What Dense Layer is Capable of Pt. 2
In the previous post, we built a simple linear regression model that maps one-hot vector to its label value. In this post, we build a model that reverse this operation.
Problem
Let's build a unit vector from scalar value.
$$ \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix} \rightarrow \begin{bmatrix} [1 & 0 & 0] \\ [0 & 1 & 0] \\ [0 & 0 & 1] \end{bmatrix} $$This is not linearly solvable. We need to introduce non-linearity that surpress output in parts of regions.
$$ \begin{align} \mathbf{h} & = \sigma ( x \cdot \mathbf{v} + \mathbf{b}_1 ) \\ \mathbf{y} & = \mathbf{h} W + \mathbf{b}_2 \end{align} $$Which can be implemented in Tensorflow as follow.
intermediate_variable = tf.layers.dense(
input_variable, units=n_intermediate_dim, use_bias=True,
activation=activation)
output_variable = tf.layers.dense(
intermediate_variable, units=n_output_dim, use_bias=True,
)You can find the code for the experiment here.
How does this model work? If we look at the each dimension of the output vector separately, we notice that they take non-zero value only when input scalar value is in certain region. This is not linear. To break the linear relationship, we pass this data point through an activation function. Then in the second dense layer activated values are combined to form the final input.
Experiment
As I run some experiments, it turned out that each activation function has its preferred input value range where it can perform the task better. So in this experiment, linear scaling is added before the first dense layer. Loss type is L2. With L1, the model does not converges to global optima. Optimizer is Adam as Gradient Descent is very slow.
Sigmoid and Tanh can converge well. ELU converges as well but was not very robust and scaling has to be performed in a specific way.
What is happening inside? Why are Relu activations fail to converge? Let's take a look into the intermediate layer output.
Each datapoint which is successfully converged, output has unique patterns. For the models which loss converged to zero, zero-crossing point of each intermediate feature's activation function can be found in the visualized range, but for the models which did not converge, some feature's zero-crossing point are outside of visualized range, meaning those features are not cotributing in the next layer.
In case of Tanh and Sigmoid activations, outputs from the intermediate layer take either 0 (-1 for Tanh) or 1. This means that what the model can output is bounded by the intermediate layer capacity.
So activation functions over N intermediate features combined, the output space is devided into N + 1 regions as follow.
