What Dense Layer is Capable of Pt. 2

2018-04-08T00:00:00+00:00

In the previous post, we built a simple linear regression model that maps one-hot vector to its label value. In this post, we build a model that reverse this operation.

Problem

Let's build a unit vector from scalar value.

$$ \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix} \rightarrow \begin{bmatrix} [1 & 0 & 0] \\ [0 & 1 & 0] \\ [0 & 0 & 1] \end{bmatrix} $$

This is not linearly solvable. We need to introduce non-linearity that surpress output in parts of regions.

$$ \begin{align} \mathbf{h} & = \sigma ( x \cdot \mathbf{v} + \mathbf{b}_1 ) \\ \mathbf{y} & = \mathbf{h} W + \mathbf{b}_2 \end{align} $$

Which can be implemented in Tensorflow as follow.

intermediate_variable = tf.layers.dense(
    input_variable, units=n_intermediate_dim, use_bias=True,
    activation=activation)

output_variable = tf.layers.dense(
    intermediate_variable, units=n_output_dim, use_bias=True,
)

You can find the code for the experiment here.

How does this model work? If we look at the each dimension of the output vector separately, we notice that they take non-zero value only when input scalar value is in certain region. This is not linear. To break the linear relationship, we pass this data point through an activation function. Then in the second dense layer activated values are combined to form the final input.

Experiment

As I run some experiments, it turned out that each activation function has its preferred input value range where it can perform the task better. So in this experiment, linear scaling is added before the first dense layer. Loss type is L2. With L1, the model does not converges to global optima. Optimizer is Adam as Gradient Descent is very slow.

Loss value over training. We see plateaus and sudden improves. Relu family (Relu, Relu6, Softplus, Leaky Relu) stays plateau and not sure if they can converge.
Sigmoid and Tanh can converge well. ELU converges as well but was not very robust and scaling has to be performed in a specific way.

Model output throught the training. In each image path, each row corresponds to one model output (one-hot encoded vector).

What is happening inside? Why are Relu activations fail to converge? Let's take a look into the intermediate layer output.

Output from the first dense layer (activation).

Each datapoint which is successfully converged, output has unique patterns. For the models which loss converged to zero, zero-crossing point of each intermediate feature's activation function can be found in the visualized range, but for the models which did not converge, some feature's zero-crossing point are outside of visualized range, meaning those features are not cotributing in the next layer.

In case of Tanh and Sigmoid activations, outputs from the intermediate layer take either 0 (-1 for Tanh) or 1. This means that what the model can output is bounded by the intermediate layer capacity.

Loss for different number of intermediate features. Loss reaches 0 at 15, which is one smaller than the output dimension.

So activation functions over N intermediate features combined, the output space is devided into N + 1 regions as follow.

Comments

What Dense Layer is Capable of

2018-04-01T00:00:00+00:00

Building a new model in Deep Learning is not easy. Figuring out why a model is not acting as expected is difficult. It is important to know how what each component is capable of. Dense layer is one of the most fundamental component of Machine Learning. Here, we explore what it is capable of through series of simple and easy-to-interpret experiments.

Problem

Let's consider a model that maps one-hot vectors into indices.

$$ \begin{bmatrix} [1 & 0 & 0] \\ [0 & 1 & 0] \\ [0 & 0 & 1] \end{bmatrix} \rightarrow \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix} $$

This can be formulated as linear regression. It is easy to write this mapping in equation.

$$ \mathbf{x} \cdot \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \rightarrow y $$

And there are different ways to express the same mapping using bias term.

$$ \mathbf{x} \cdot \begin{bmatrix} -1 \\ 0 \\ 1 \end{bmatrix} + \begin{bmatrix} 2 \\ 2 \\ 2 \end{bmatrix} \rightarrow y $$

Experiment

This simple linear transformation is called Dense Layer or Fully-Connected Layer in Deep Learning, and in Tensorflow such model can be written as follow.

output_variable = tf.layers.dense(
    input_variablbe, units=n_dim, activation=None, use_bias=use_bias)

You can find the code for the experiment here.

Simple gradient descent is enough for finding the optima. The following figure shows training loss with different loss type and bias condition for 15 dimension case.

Loss value over training. Loss indicates the type of loss function; L1 or L2. Bias indicates the initial bias value when used. Shuffle indicates that input-to-output mapping was shuffled in deterministic way the training data.
You can click legend to hide/show the corresponding plot.

The following figure illustrates the model output through the training.

How a simple linear regression learns over the time.

Observation

There are couple of interesting this we can observe.

Bias learns faster than kernel.

Bias terms converge to optima before kernel terms do.
You can disable plots other than Loss: L2, Bias: [0.] to see this.

Existence of bias does not affect the speed kernel fits.

Comparing the same loss type with/without bias term, the step model converges to the optima is same.
In fact the loss curve of Bias: [0,] starts at the same loss as Bias: None but quickly merges to Bias: [6.]. For L2 loss types, all the training cases converge around 1000 trainings.

Model cannot truely converge to optima when L1 loss is used.

With L1 loss, derivetive value is always constant and when that value is larger than the actual error, parameters keep crossing the optimal value and fructuates.

Shuffling the output axis in arbitrary order does not affect model training.

When label values are shuffled in arbibtrary order, model training is not affected.

Extention

Extending this to multi dimension, we get the same property on each axis independently.

$$ \begin{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix} \\ \begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} \\ \begin{bmatrix} 0 & 0 & 1 \\ 1 & 0 & 0 \end{bmatrix} \end{bmatrix} \rightarrow \begin{bmatrix} [1 & 2]\\ [1 & 3]\\ [3 & 1] \end{bmatrix} $$

In the next blog entry, we will explore the revese operation of linear regression, using Dense layer.

moto's blog

What Dense Layer is Capable of Pt. 2

Problem

Experiment

Comments

What Dense Layer is Capable of

Problem

Experiment

Observation

Extention

Next

Comments