API - Layers

TensorLayer provides rich layer implementations trailed for various benchmarks and domain-specific problems. In addition, we also support transparent access to native TensorFlow parameters. For example, we provide not only layers for local response normalization, but also layers that allow user to apply tf.nn.lrn on network.outputs. More functions can be found in TensorFlow API.

Name Scope and Sharing Parameters

These functions help you to reuse parameters for different inference (graph), and get a list of parameters by given name. About TensorFlow parameters sharing click here.

Get variables with name

tensorlayer.layers.get_variables_with_name(name=None, train_only=True, verbose=False)

Get a list of TensorFlow variables by a given name scope.

Parameters:

• name (str) – Get the variables that contain this name.
• train_only (boolean) – If Ture, only get the trainable variables.
• verbose (boolean) – If True, print the information of all variables.

Returns:

A list of TensorFlow variables

Return type:

list of Tensor

Examples

>>> import tensorlayer as tl
>>> dense_vars = tl.layers.get_variables_with_name('dense', True, True)

Get layers with name

tensorlayer.layers.get_layers_with_name(net, name='', verbose=False)

Get a list of layers’ output in a network by a given name scope.

Parameters:

• net (Layer) – The last layer of the network.
• name (str) – Get the layers’ output that contain this name.
• verbose (boolean) – If True, print information of all the layers’ output

Returns:

A list of layers’ output (TensorFlow tensor)

Return type:

list of Tensor

Examples

>>> import tensorlayer as tl
>>> layers = tl.layers.get_layers_with_name(net, "CNN", True)

Print variables

tensorlayer.layers.print_all_variables(train_only=False)

Print information of trainable or all variables, without tl.layers.initialize_global_variables(sess).

Parameters:train_only (boolean) –

Whether print trainable variables only.

• If True, print the trainable variables.
• If False, print all variables.

Initialize variables

tensorlayer.layers.initialize_global_variables(sess)

Initialize the global variables of TensorFlow.

Warning

THIS FUNCTION IS DEPRECATED: It will be removed after after 2018-09-30. Instructions for updating: This API is deprecated in favor of tf.global_variables_initializer.

Run sess.run(tf.global_variables_initializer()) for TF 0.12+ or sess.run(tf.initialize_all_variables()) for TF 0.11.

Parameters:sess (Session) – TensorFlow session.

Understanding the Basic Layer

All TensorLayer layers have a number of properties in common:

• layer.outputs : a Tensor, the outputs of current layer.
• layer.all_params : a list of Tensor, all network variables in order.
• layer.all_layers : a list of Tensor, all network outputs in order.
• layer.all_drop : a dictionary of {placeholder : float}, all keeping probabilities of noise layers.

All TensorLayer layers have a number of methods in common:

• layer.print_params() : print network variable information in order (after tl.layers.initialize_global_variables(sess)). alternatively, print all variables by tl.layers.print_all_variables().
• layer.print_layers() : print network layer information in order.
• layer.count_params() : print the number of parameters in the network.

A network starts with the input layer and is followed by layers stacked in order. A network is essentially a Layer class. The key properties of a network are network.all_params, network.all_layers and network.all_drop. The all_params is a list which store pointers to all network parameters in order. For example, the following script define a 3 layer network, then:

all_params = [W1, b1, W2, b2, W_out, b_out]

To get specified variable information, you can use network.all_params[2:3] or get_variables_with_name(). all_layers is a list which stores the pointers to the outputs of all layers, see the example as follow:

all_layers = [drop(?,784), relu(?,800), drop(?,800), relu(?,800), drop(?,800)], identity(?,10)]

where ? reflects a given batch size. You can print the layer and parameters information by using network.print_layers() and network.print_params(). To count the number of parameters in a network, run network.count_params().

sess = tf.InteractiveSession()

x = tf.placeholder(tf.float32, shape=[None, 784], name='x')
y_ = tf.placeholder(tf.int64, shape=[None, ], name='y_')

network = tl.layers.InputLayer(x, name='input_layer')
network = tl.layers.DropoutLayer(network, keep=0.8, name='drop1')
network = tl.layers.DenseLayer(network, n_units=800,
                                act=tf.nn.relu, name='relu1')
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop2')
network = tl.layers.DenseLayer(network, n_units=800,
                                act=tf.nn.relu, name='relu2')
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop3')
network = tl.layers.DenseLayer(network, n_units=10,
                                act=None, name='output')


y = network.outputs
y_op = tf.argmax(tf.nn.softmax(y), 1)

cost = tl.cost.cross_entropy(y, y_)

train_params = network.all_params

train_op = tf.train.AdamOptimizer(learning_rate, beta1=0.9, beta2=0.999,
                            epsilon=1e-08, use_locking=False).minimize(cost, var_list = train_params)

tl.layers.initialize_global_variables(sess)

network.print_params()
network.print_layers()

In addition, network.all_drop is a dictionary which stores the keeping probabilities of all noise layers. In the above network, they represent the keeping probabilities of dropout layers.

In case for training, you can enable all dropout layers as follow:

feed_dict = {x: X_train_a, y_: y_train_a}
feed_dict.update( network.all_drop )
loss, _ = sess.run([cost, train_op], feed_dict=feed_dict)
feed_dict.update( network.all_drop )

In case for evaluating and testing, you can disable all dropout layers as follow.

feed_dict = {x: X_val, y_: y_val}
feed_dict.update(dp_dict)
print("   val loss: %f" % sess.run(cost, feed_dict=feed_dict))
print("   val acc: %f" % np.mean(y_val ==
                        sess.run(y_op, feed_dict=feed_dict)))

For more details, please read the MNIST examples in the example folder.

Layer list

get_variables_with_name([name, train_only, …])	Get a list of TensorFlow variables by a given name scope.
get_layers_with_name(net[, name, verbose])	Get a list of layers’ output in a network by a given name scope.
set_name_reuse([enable])	DEPRECATED FUNCTION
print_all_variables([train_only])	Print information of trainable or all variables, without tl.layers.initialize_global_variables(sess).
initialize_global_variables(sess)	Initialize the global variables of TensorFlow.
Layer(prev_layer[, act, name])	The basic Layer class represents a single layer of a neural network.
InputLayer(inputs[, name])	The InputLayer class is the starting layer of a neural network.
OneHotInputLayer([inputs, depth, on_value, …])	The OneHotInputLayer class is the starting layer of a neural network, see tf.one_hot.
Word2vecEmbeddingInputlayer(inputs[, …])	The Word2vecEmbeddingInputlayer class is a fully connected layer.
EmbeddingInputlayer(inputs[, …])	The EmbeddingInputlayer class is a look-up table for word embedding.
AverageEmbeddingInputlayer(inputs, …[, …])	The AverageEmbeddingInputlayer averages over embeddings of inputs.
DenseLayer(prev_layer[, n_units, act, …])	The DenseLayer class is a fully connected layer.
ReconLayer(prev_layer[, x_recon, n_units, …])	A reconstruction layer for DenseLayer to implement AutoEncoder.
DropoutLayer(prev_layer[, keep, is_fix, …])	The DropoutLayer class is a noise layer which randomly set some activations to zero according to a keeping probability.
GaussianNoiseLayer(prev_layer[, mean, …])	The GaussianNoiseLayer class is noise layer that adding noise with gaussian distribution to the activation.
DropconnectDenseLayer(prev_layer[, keep, …])	The DropconnectDenseLayer class is DenseLayer with DropConnect behaviour which randomly removes connections between this layer and the previous layer according to a keeping probability.
Conv1dLayer(prev_layer[, act, shape, …])	The Conv1dLayer class is a 1D CNN layer, see tf.nn.convolution.
Conv2dLayer(prev_layer[, act, shape, …])	The Conv2dLayer class is a 2D CNN layer, see tf.nn.conv2d.
DeConv2dLayer(prev_layer[, act, shape, …])	A de-convolution 2D layer.
Conv3dLayer(prev_layer[, shape, strides, …])	The Conv3dLayer class is a 3D CNN layer, see tf.nn.conv3d.
DeConv3dLayer(prev_layer[, act, shape, …])	The DeConv3dLayer class is deconvolutional 3D layer, see tf.nn.conv3d_transpose.
UpSampling2dLayer(prev_layer, size[, …])	The UpSampling2dLayer class is a up-sampling 2D layer.
DownSampling2dLayer(prev_layer, size[, …])	The DownSampling2dLayer class is down-sampling 2D layer.
AtrousConv1dLayer(prev_layer[, n_filter, …])	Simplified version of AtrousConv1dLayer.
AtrousConv2dLayer(prev_layer[, n_filter, …])	The AtrousConv2dLayer class is 2D atrous convolution (a.k.a.
AtrousDeConv2dLayer(prev_layer[, shape, …])	The AtrousDeConv2dLayer class is 2D atrous convolution transpose, see tf.nn.atrous_conv2d_transpose.
Conv1d(prev_layer[, n_filter, filter_size, …])	Simplified version of Conv1dLayer.
Conv2d(prev_layer[, n_filter, filter_size, …])	Simplified version of Conv2dLayer.
DeConv2d(prev_layer[, n_filter, …])	Simplified version of DeConv2dLayer.
DeConv3d(prev_layer[, n_filter, …])	Simplified version of The DeConv3dLayer, see tf.contrib.layers.conv3d_transpose.
DepthwiseConv2d(prev_layer[, shape, …])	Separable/Depthwise Convolutional 2D layer, see tf.nn.depthwise_conv2d.
SeparableConv1d(prev_layer[, n_filter, …])	The SeparableConv1d class is a 1D depthwise separable convolutional layer, see tf.layers.separable_conv1d.
SeparableConv2d(prev_layer[, n_filter, …])	The SeparableConv2d class is a 2D depthwise separable convolutional layer, see tf.layers.separable_conv2d.
DeformableConv2d(prev_layer[, offset_layer, …])	The DeformableConv2d class is a 2D Deformable Convolutional Networks.
GroupConv2d(prev_layer[, n_filter, …])	The GroupConv2d class is 2D grouped convolution, see here.
PadLayer(prev_layer[, padding, mode, name])	The PadLayer class is a padding layer for any mode and dimension.
PoolLayer(prev_layer[, ksize, strides, …])	The PoolLayer class is a Pooling layer.
ZeroPad1d(prev_layer, padding[, name])	The ZeroPad1d class is a 1D padding layer for signal [batch, length, channel].
ZeroPad2d(prev_layer, padding[, name])	The ZeroPad2d class is a 2D padding layer for image [batch, height, width, channel].
ZeroPad3d(prev_layer, padding[, name])	The ZeroPad3d class is a 3D padding layer for volume [batch, depth, height, width, channel].
MaxPool1d(prev_layer[, filter_size, …])	Max pooling for 1D signal.
MeanPool1d(prev_layer[, filter_size, …])	Mean pooling for 1D signal.
MaxPool2d(prev_layer[, filter_size, …])	Max pooling for 2D image.
MeanPool2d(prev_layer[, filter_size, …])	Mean pooling for 2D image [batch, height, width, channel].
MaxPool3d(prev_layer[, filter_size, …])	Max pooling for 3D volume.
MeanPool3d(prev_layer[, filter_size, …])	Mean pooling for 3D volume.
GlobalMaxPool1d(prev_layer[, data_format, name])	The GlobalMaxPool1d class is a 1D Global Max Pooling layer.
GlobalMeanPool1d(prev_layer[, data_format, name])	The GlobalMeanPool1d class is a 1D Global Mean Pooling layer.
GlobalMaxPool2d(prev_layer[, data_format, name])	The GlobalMaxPool2d class is a 2D Global Max Pooling layer.
GlobalMeanPool2d(prev_layer[, data_format, name])	The GlobalMeanPool2d class is a 2D Global Mean Pooling layer.
GlobalMaxPool3d(prev_layer[, data_format, name])	The GlobalMaxPool3d class is a 3D Global Max Pooling layer.
GlobalMeanPool3d(prev_layer[, data_format, name])	The GlobalMeanPool3d class is a 3D Global Mean Pooling layer.
SubpixelConv1d(prev_layer[, scale, act, name])	It is a 1D sub-pixel up-sampling layer.
SubpixelConv2d(prev_layer[, scale, …])	It is a 2D sub-pixel up-sampling layer, usually be used for Super-Resolution applications, see SRGAN for example.
SpatialTransformer2dAffineLayer(prev_layer, …)	The SpatialTransformer2dAffineLayer class is a 2D Spatial Transformer Layer for 2D Affine Transformation.
transformer(U, theta, out_size[, name])	Spatial Transformer Layer for 2D Affine Transformation , see SpatialTransformer2dAffineLayer class.
batch_transformer(U, thetas, out_size[, name])	Batch Spatial Transformer function for 2D Affine Transformation.
BatchNormLayer(prev_layer[, decay, epsilon, …])	The BatchNormLayer is a batch normalization layer for both fully-connected and convolution outputs.
LocalResponseNormLayer(prev_layer[, …])	The LocalResponseNormLayer layer is for Local Response Normalization.
InstanceNormLayer(prev_layer[, act, …])	The InstanceNormLayer class is a for instance normalization.
LayerNormLayer(prev_layer[, center, scale, …])	The LayerNormLayer class is for layer normalization, see tf.contrib.layers.layer_norm.
SwitchNormLayer(prev_layer[, act, epsilon, …])	The SwitchNormLayer is a switchable normalization.
ROIPoolingLayer(prev_layer, rois[, …])	The region of interest pooling layer.
TimeDistributedLayer(prev_layer[, …])	The TimeDistributedLayer class that applies a function to every timestep of the input tensor.
RNNLayer(prev_layer, cell_fn[, …])	The RNNLayer class is a fixed length recurrent layer for implementing vanilla RNN, LSTM, GRU and etc.
BiRNNLayer(prev_layer, cell_fn[, …])	The BiRNNLayer class is a fixed length Bidirectional recurrent layer.
ConvRNNCell	Abstract object representing an Convolutional RNN Cell.
BasicConvLSTMCell(shape, filter_size, …[, …])	Basic Conv LSTM recurrent network cell.
ConvLSTMLayer(prev_layer[, cell_shape, …])	A fixed length Convolutional LSTM layer.
advanced_indexing_op(inputs, index)	Advanced Indexing for Sequences, returns the outputs by given sequence lengths.
retrieve_seq_length_op(data)	An op to compute the length of a sequence from input shape of [batch_size, n_step(max), n_features], it can be used when the features of padding (on right hand side) are all zeros.
retrieve_seq_length_op2(data)	An op to compute the length of a sequence, from input shape of [batch_size, n_step(max)], it can be used when the features of padding (on right hand side) are all zeros.
retrieve_seq_length_op3(data[, pad_val])	Return tensor for sequence length, if input is tf.string.
target_mask_op(data[, pad_val])	Return tensor for mask, if input is tf.string.
DynamicRNNLayer(prev_layer, cell_fn[, …])	The DynamicRNNLayer class is a dynamic recurrent layer, see tf.nn.dynamic_rnn.
BiDynamicRNNLayer(prev_layer, cell_fn[, …])	The BiDynamicRNNLayer class is a RNN layer, you can implement vanilla RNN, LSTM and GRU with it.
Seq2Seq(net_encode_in, net_decode_in, cell_fn)	The Seq2Seq class is a simple DynamicRNNLayer based Seq2seq layer without using tl.contrib.seq2seq.
FlattenLayer(prev_layer[, name])	A layer that reshapes high-dimension input into a vector.
ReshapeLayer(prev_layer, shape[, name])	A layer that reshapes a given tensor.
TransposeLayer(prev_layer, perm[, name])	A layer that transposes the dimension of a tensor.
LambdaLayer(prev_layer, fn[, fn_args, name])	A layer that takes a user-defined function using TensorFlow Lambda, for multiple inputs see ElementwiseLambdaLayer.
ConcatLayer(prev_layer[, concat_dim, name])	A layer that concats multiple tensors according to given axis.
ElementwiseLayer(prev_layer[, combine_fn, …])	A layer that combines multiple Layer that have the same output shapes according to an element-wise operation.
ElementwiseLambdaLayer(layers, fn[, …])	A layer that use a custom function to combine multiple Layer inputs.
ExpandDimsLayer(prev_layer, axis[, name])	The ExpandDimsLayer class inserts a dimension of 1 into a tensor’s shape, see tf.expand_dims() .
TileLayer(prev_layer[, multiples, name])	The TileLayer class constructs a tensor by tiling a given tensor, see tf.tile() .
StackLayer(layers[, axis, name])	The StackLayer class is a layer for stacking a list of rank-R tensors into one rank-(R+1) tensor, see tf.stack().
UnStackLayer(prev_layer[, num, axis, name])	The UnStackLayer class is a layer for unstacking the given dimension of a rank-R tensor into rank-(R-1) tensors., see tf.unstack().
SlimNetsLayer(prev_layer, slim_layer[, …])	A layer that merges TF-Slim models into TensorLayer.
SignLayer(prev_layer[, name])	The SignLayer class is for quantizing the layer outputs to -1 or 1 while inferencing.
ScaleLayer(prev_layer[, init_scale, name])	The AddScaleLayer class is for multipling a trainble scale value to the layer outputs.
BinaryDenseLayer(prev_layer[, n_units, act, …])	The BinaryDenseLayer class is a binary fully connected layer, which weights are either -1 or 1 while inferencing.
BinaryConv2d(prev_layer[, n_filter, …])	The BinaryConv2d class is a 2D binary CNN layer, which weights are either -1 or 1 while inference.
TernaryDenseLayer(prev_layer[, n_units, …])	The TernaryDenseLayer class is a ternary fully connected layer, which weights are either -1 or 1 or 0 while inference.
TernaryConv2d(prev_layer[, n_filter, …])	The TernaryConv2d class is a 2D binary CNN layer, which weights are either -1 or 1 or 0 while inference.
DorefaDenseLayer(prev_layer[, bitW, bitA, …])	The DorefaDenseLayer class is a binary fully connected layer, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.
DorefaConv2d(prev_layer[, bitW, bitA, …])	The DorefaConv2d class is a binary fully connected layer, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.
QuanDenseLayer(prev_layer[, n_units, act, …])	The QuanDenseLayer class is a quantized fully connected layer with BN, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.
QuanDenseLayerWithBN(prev_layer[, n_units, …])	The QuanDenseLayer class is a quantized fully connected layer with BN, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.
QuanConv2d(prev_layer[, n_filter, …])	The QuanConv2dWithBN class is a quantized convolutional layer with BN, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.
QuanConv2dWithBN(prev_layer[, n_filter, …])	The QuanConv2dWithBN class is a quantized convolutional layer with BN, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.
PReluLayer(prev_layer[, channel_shared, …])	The PReluLayer class is Parametric Rectified Linear layer.
PRelu6Layer(prev_layer[, channel_shared, …])	The PRelu6Layer class is Parametric Rectified Linear layer integrating ReLU6 behaviour.
PTRelu6Layer(prev_layer[, channel_shared, …])	The PTRelu6Layer class is Parametric Rectified Linear layer integrating ReLU6 behaviour.
MultiplexerLayer(layers[, name])	The MultiplexerLayer selects inputs to be forwarded to output.
flatten_reshape(variable[, name])	Reshapes a high-dimension vector input.
clear_layers_name()	DEPRECATED FUNCTION
initialize_rnn_state(state[, feed_dict])	Returns the initialized RNN state.
list_remove_repeat(x)	Remove the repeated items in a list, and return the processed list.
merge_networks([layers])	Merge all parameters, layers and dropout probabilities to a Layer.

Customizing Layers

A Simple Layer

To implement a custom layer in TensorLayer, you will have to write a Python class that subclasses Layer and implement the outputs expression.

The following is an example implementation of a layer that multiplies its input by 2:

class DoubleLayer(Layer):
    def __init__(
        self,
        layer = None,
        name ='double_layer',
    ):
        # manage layer (fixed)
        super(DoubleLayer, self).__init__(prev_layer=prev_layer, name=name)

        # the input of this layer is the output of previous layer (fixed)
        self.inputs = layer.outputs

        # operation (customized)
        self.outputs = self.inputs * 2

        # update layer (customized)
        self._add_layers(self.outputs)

Your Dense Layer

Before creating your own TensorLayer layer, let’s have a look at the Dense layer. It creates a weight matrix and a bias vector if not exists, and then implements the output expression. At the end, for a layer with parameters, we also append the parameters into all_params.

class MyDenseLayer(Layer):
  def __init__(
      self,
      layer = None,
      n_units = 100,
      act = tf.nn.relu,
      name ='simple_dense',
  ):
      # manage layer (fixed)
      super(MyDenseLayer, self).__init__(prev_layer=prev_layer, act=act, name=name)

      # the input of this layer is the output of previous layer (fixed)
      self.inputs = layer.outputs

      # print out info (customized)
      print("  MyDenseLayer %s: %d, %s" % (self.name, n_units, act))

      # operation (customized)
      n_in = int(self.inputs._shape[-1])
      with tf.variable_scope(name) as vs:
          # create new parameters
          W = tf.get_variable(name='W', shape=(n_in, n_units))
          b = tf.get_variable(name='b', shape=(n_units))
          # tensor operation
          self.outputs = self._apply_activation(tf.matmul(self.inputs, W) + b)

      # update layer (customized)
      self._add_layers(self.outputs)
      self._add_params([W, b])

Basic Layer

class tensorlayer.layers.Layer(prev_layer, act=None, name=None, *args, **kwargs)

The basic Layer class represents a single layer of a neural network.

It should be subclassed when implementing new types of layers. Because each layer can keep track of the layer(s) feeding into it, a network’s output Layer instance can double as a handle to the full network.

Parameters:

• prev_layer (Layer or None) – Previous layer (optional), for adding all properties of previous layer(s) to this layer.
• act (activation function (None by default)) – The activation function of this layer.
• name (str or None) – A unique layer name.

print_params(details=True, session=None)

Print all parameters of this network.

print_layers()

Print all outputs of all layers of this network.

count_params()

Return the number of parameters of this network.

get_all_params()

Return the parameters in a list of array.

Examples

• Define model

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder("float32", [None, 100])
>>> n = tl.layers.InputLayer(x, name='in')
>>> n = tl.layers.DenseLayer(n, 80, name='d1')
>>> n = tl.layers.DenseLayer(n, 80, name='d2')

• Get information

>>> print(n)
Last layer is: DenseLayer (d2) [None, 80]
>>> n.print_layers()
[TL]   layer   0: d1/Identity:0        (?, 80)            float32
[TL]   layer   1: d2/Identity:0        (?, 80)            float32
>>> n.print_params(False)
[TL]   param   0: d1/W:0               (100, 80)          float32_ref
[TL]   param   1: d1/b:0               (80,)              float32_ref
[TL]   param   2: d2/W:0               (80, 80)           float32_ref
[TL]   param   3: d2/b:0               (80,)              float32_ref
[TL]   num of params: 14560
>>> n.count_params()
14560

• Slicing the outputs

>>> n2 = n[:, :30]
>>> print(n2)
Last layer is: Layer (d2) [None, 30]

• Iterating the outputs

>>> for l in n:
>>>    print(l)
Tensor("d1/Identity:0", shape=(?, 80), dtype=float32)
Tensor("d2/Identity:0", shape=(?, 80), dtype=float32)

Input Layers

Input Layer

class tensorlayer.layers.InputLayer(inputs, name='input')

The InputLayer class is the starting layer of a neural network.

Parameters:

• inputs (placeholder or tensor) – The input of a network.
• name (str) – A unique layer name.

One-hot Input Layer

class tensorlayer.layers.OneHotInputLayer(inputs=None, depth=None, on_value=None, off_value=None, axis=None, dtype=None, name='input')

The OneHotInputLayer class is the starting layer of a neural network, see tf.one_hot.

Parameters:

• inputs (placeholder or tensor) – The input of a network.
• depth (None or int) – If the input indices is rank N, the output will have rank N+1. The new axis is created at dimension axis (default: the new axis is appended at the end).
• on_value (None or number) – The value to represnt ON. If None, it will default to the value 1.
• off_value (None or number) – The value to represnt OFF. If None, it will default to the value 0.
• axis (None or int) – The axis.
• dtype (None or TensorFlow dtype) – The data type, None means tf.float32.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.int32, shape=[None])
>>> net = tl.layers.OneHotInputLayer(x, depth=8, name='one_hot_encoding')
(?, 8)

Word2Vec Embedding Layer

class tensorlayer.layers.Word2vecEmbeddingInputlayer(inputs, train_labels=None, vocabulary_size=80000, embedding_size=200, num_sampled=64, nce_loss_args=None, E_init=<sphinx.ext.autodoc.importer._MockObject object>, E_init_args=None, nce_W_init=<sphinx.ext.autodoc.importer._MockObject object>, nce_W_init_args=None, nce_b_init=<sphinx.ext.autodoc.importer._MockObject object>, nce_b_init_args=None, name='word2vec')

The Word2vecEmbeddingInputlayer class is a fully connected layer. For Word Embedding, words are input as integer index. The output is the embedded word vector.

Parameters:

• inputs (placeholder or tensor) – The input of a network. For word inputs, please use integer index format, 2D tensor : [batch_size, num_steps(num_words)]
• train_labels (placeholder) – For word labels. integer index format
• vocabulary_size (int) – The size of vocabulary, number of words
• embedding_size (int) – The number of embedding dimensions
• num_sampled (int) – The mumber of negative examples for NCE loss
• nce_loss_args (dictionary) – The arguments for tf.nn.nce_loss()
• E_init (initializer) – The initializer for initializing the embedding matrix
• E_init_args (dictionary) – The arguments for embedding initializer
• nce_W_init (initializer) – The initializer for initializing the nce decoder weight matrix
• nce_W_init_args (dictionary) – The arguments for initializing the nce decoder weight matrix
• nce_b_init (initializer) – The initializer for initializing of the nce decoder bias vector
• nce_b_init_args (dictionary) – The arguments for initializing the nce decoder bias vector
• name (str) – A unique layer name

nce_cost

Tensor – The NCE loss.

outputs

Tensor – The embedding layer outputs.

normalized_embeddings

Tensor – Normalized embedding matrix.

Examples

With TensorLayer : see tensorlayer/example/tutorial_word2vec_basic.py

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> batch_size = 8
>>> train_inputs = tf.placeholder(tf.int32, shape=(batch_size))
>>> train_labels = tf.placeholder(tf.int32, shape=(batch_size, 1))
>>> net = tl.layers.Word2vecEmbeddingInputlayer(inputs=train_inputs,
...     train_labels=train_labels, vocabulary_size=1000, embedding_size=200,
...     num_sampled=64, name='word2vec')
(8, 200)
>>> cost = net.nce_cost
>>> train_params = net.all_params
>>> cost = net.nce_cost
>>> train_params = net.all_params
>>> train_op = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost, var_list=train_params)
>>> normalized_embeddings = net.normalized_embeddings

Without TensorLayer : see tensorflow/examples/tutorials/word2vec/word2vec_basic.py

>>> train_inputs = tf.placeholder(tf.int32, shape=(batch_size))
>>> train_labels = tf.placeholder(tf.int32, shape=(batch_size, 1))
>>> embeddings = tf.Variable(
...     tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))
>>> embed = tf.nn.embedding_lookup(embeddings, train_inputs)
>>> nce_weights = tf.Variable(
...     tf.truncated_normal([vocabulary_size, embedding_size],
...                    stddev=1.0 / math.sqrt(embedding_size)))
>>> nce_biases = tf.Variable(tf.zeros([vocabulary_size]))
>>> cost = tf.reduce_mean(
...    tf.nn.nce_loss(weights=nce_weights, biases=nce_biases,
...               inputs=embed, labels=train_labels,
...               num_sampled=num_sampled, num_classes=vocabulary_size,
...               num_true=1))

References

tensorflow/examples/tutorials/word2vec/word2vec_basic.py

Embedding Input Layer

class tensorlayer.layers.EmbeddingInputlayer(inputs, vocabulary_size=80000, embedding_size=200, E_init=<sphinx.ext.autodoc.importer._MockObject object>, E_init_args=None, name='embedding')

The EmbeddingInputlayer class is a look-up table for word embedding.

Word content are accessed using integer indexes, then the output is the embedded word vector. To train a word embedding matrix, you can used Word2vecEmbeddingInputlayer. If you have a pre-trained matrix, you can assign the parameters into it.

Parameters:

• inputs (placeholder) – The input of a network. For word inputs. Please use integer index format, 2D tensor : (batch_size, num_steps(num_words)).
• vocabulary_size (int) – The size of vocabulary, number of words.
• embedding_size (int) – The number of embedding dimensions.
• E_init (initializer) – The initializer for the embedding matrix.
• E_init_args (dictionary) – The arguments for embedding matrix initializer.
• name (str) – A unique layer name.

outputs

tensor – The embedding layer output is a 3D tensor in the shape: (batch_size, num_steps(num_words), embedding_size).

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> batch_size = 8
>>> x = tf.placeholder(tf.int32, shape=(batch_size, ))
>>> net = tl.layers.EmbeddingInputlayer(inputs=x, vocabulary_size=1000, embedding_size=50, name='embed')
(8, 50)

Average Embedding Input Layer

class tensorlayer.layers.AverageEmbeddingInputlayer(inputs, vocabulary_size, embedding_size, pad_value=0, embeddings_initializer=<sphinx.ext.autodoc.importer._MockObject object>, embeddings_kwargs=None, name='average_embedding')

The AverageEmbeddingInputlayer averages over embeddings of inputs. This is often used as the input layer for models like DAN[1] and FastText[2].

Parameters:

• inputs (placeholder or tensor) – The network input. For word inputs, please use integer index format, 2D tensor: (batch_size, num_steps(num_words)).
• vocabulary_size (int) – The size of vocabulary.
• embedding_size (int) – The dimension of the embedding vectors.
• pad_value (int) – The scalar padding value used in inputs, 0 as default.
• embeddings_initializer (initializer) – The initializer of the embedding matrix.
• embeddings_kwargs (None or dictionary) – The arguments to get embedding matrix variable.
• name (str) – A unique layer name.

References

• [1] Iyyer, M., Manjunatha, V., Boyd-Graber, J., & Daum’e III, H. (2015). Deep Unordered Composition Rivals Syntactic Methods for Text Classification. In Association for Computational Linguistics.
• [2] Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016). Bag of Tricks for Efficient Text Classification.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> batch_size = 8
>>> length = 5
>>> x = tf.placeholder(tf.int32, shape=(batch_size, length))
>>> net = tl.layers.AverageEmbeddingInputlayer(x, vocabulary_size=1000, embedding_size=50, name='avg')
(8, 50)

Activation Layers

PReLU Layer

class tensorlayer.layers.PReluLayer(prev_layer, channel_shared=False, a_init=<sphinx.ext.autodoc.importer._MockObject object>, a_init_args=None, name='PReluLayer')

The PReluLayer class is Parametric Rectified Linear layer.

Parameters:

• prev_layer (Layer) – Previous layer.
• channel_shared (boolean) – If True, single weight is shared by all channels.
• a_init (initializer) – The initializer for initializing the alpha(s).
• a_init_args (dictionary) – The arguments for initializing the alpha(s).
• name (str) – A unique layer name.

References

• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Convolutional Deep Belief Networks on CIFAR-10 [A. Krizhevsky, 2010]

PReLU6 Layer

class tensorlayer.layers.PRelu6Layer(prev_layer, channel_shared=False, a_init=<sphinx.ext.autodoc.importer._MockObject object>, a_init_args=None, name='PReLU6_layer')

The PRelu6Layer class is Parametric Rectified Linear layer integrating ReLU6 behaviour.

This Layer is a modified version of the PReluLayer.

This activation layer use a modified version tl.act.leaky_relu() introduced by the following paper: Rectifier Nonlinearities Improve Neural Network Acoustic Models [A. L. Maas et al., 2013]

This activation function also use a modified version of the activation function tf.nn.relu6() introduced by the following paper: Convolutional Deep Belief Networks on CIFAR-10 [A. Krizhevsky, 2010]

This activation layer push further the logic by adding leaky behaviour both below zero and above six.

The function return the following results:

• When x < 0: f(x) = alpha_low * x.
• When x in [0, 6]: f(x) = x.
• When x > 6: f(x) = 6.

Parameters:

• prev_layer (Layer) – Previous layer.
• channel_shared (boolean) – If True, single weight is shared by all channels.
• a_init (initializer) – The initializer for initializing the alpha(s).
• a_init_args (dictionary) – The arguments for initializing the alpha(s).
• name (str) – A unique layer name.

References

• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Rectifier Nonlinearities Improve Neural Network Acoustic Models [A. L. Maas et al., 2013]
• Convolutional Deep Belief Networks on CIFAR-10 [A. Krizhevsky, 2010]

PTReLU6 Layer

class tensorlayer.layers.PTRelu6Layer(prev_layer, channel_shared=False, a_init=<sphinx.ext.autodoc.importer._MockObject object>, a_init_args=None, name='PTReLU6_layer')

The PTRelu6Layer class is Parametric Rectified Linear layer integrating ReLU6 behaviour.

This Layer is a modified version of the PReluLayer.

This activation layer use a modified version tl.act.leaky_relu() introduced by the following paper: Rectifier Nonlinearities Improve Neural Network Acoustic Models [A. L. Maas et al., 2013]

This activation function also use a modified version of the activation function tf.nn.relu6() introduced by the following paper: Convolutional Deep Belief Networks on CIFAR-10 [A. Krizhevsky, 2010]

This activation layer push further the logic by adding leaky behaviour both below zero and above six.

The function return the following results:

• When x < 0: f(x) = alpha_low * x.
• When x in [0, 6]: f(x) = x.
• When x > 6: f(x) = 6 + (alpha_high * (x-6)).

This version goes one step beyond PRelu6Layer by introducing leaky behaviour on the positive side when x > 6.

Parameters:

• prev_layer (Layer) – Previous layer.
• channel_shared (boolean) – If True, single weight is shared by all channels.
• a_init (initializer) – The initializer for initializing the alpha(s).
• a_init_args (dictionary) – The arguments for initializing the alpha(s).
• name (str) – A unique layer name.

References

• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Convolutional Deep Belief Networks on CIFAR-10 [A. Krizhevsky, 2010]
• Rectifier Nonlinearities Improve Neural Network Acoustic Models [A. L. Maas et al., 2013]

Convolutional Layers

Simplified Convolutions

For users don’t familiar with TensorFlow, the following simplified functions may easier for you. We will provide more simplified functions later, but if you are good at TensorFlow, the professional APIs may better for you.

Conv1d

class tensorlayer.layers.Conv1d(prev_layer, n_filter=32, filter_size=5, stride=1, dilation_rate=1, act=None, padding='SAME', data_format='channels_last', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='conv1d')

Simplified version of Conv1dLayer.

Parameters:

• prev_layer (Layer) – Previous layer
• n_filter (int) – The number of filters
• filter_size (int) – The filter size
• stride (int) – The stride step
• dilation_rate (int) – Specifying the dilation rate to use for dilated convolution.
• act (activation function) – The function that is applied to the layer activations
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• data_format (str) – channels_last ‘NWC’ (default) or channels_first.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer (deprecated).
• b_init_args (dictionary) – The arguments for the bias vector initializer (deprecated).
• name (str) – A unique layer name

Examples

>>> x = tf.placeholder(tf.float32, (batch_size, width))
>>> y_ = tf.placeholder(tf.int64, shape=(batch_size,))
>>> n = InputLayer(x, name='in')
>>> n = ReshapeLayer(n, (-1, width, 1), name='rs')
>>> n = Conv1d(n, 64, 3, 1, act=tf.nn.relu, name='c1')
>>> n = MaxPool1d(n, 2, 2, padding='valid', name='m1')
>>> n = Conv1d(n, 128, 3, 1, act=tf.nn.relu, name='c2')
>>> n = MaxPool1d(n, 2, 2, padding='valid', name='m2')
>>> n = Conv1d(n, 128, 3, 1, act=tf.nn.relu, name='c3')
>>> n = MaxPool1d(n, 2, 2, padding='valid', name='m3')
>>> n = FlattenLayer(n, name='f')
>>> n = DenseLayer(n, 500, tf.nn.relu, name='d1')
>>> n = DenseLayer(n, 100, tf.nn.relu, name='d2')
>>> n = DenseLayer(n, 2, None, name='o')

Conv2d

class tensorlayer.layers.Conv2d(prev_layer, n_filter=32, filter_size=(3, 3), strides=(1, 1), act=None, padding='SAME', data_format='channels_last', dilation_rate=(1, 1), W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, use_cudnn_on_gpu=None, name='conv2d')

Simplified version of Conv2dLayer.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size (height, width).
• strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the shape parameter.
• act (activation function) – The activation function of this layer.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• data_format (str) – “channels_last” (NHWC, default) or “channels_first” (NCHW).
• W_init (initializer) – The initializer for the the weight matrix.
• b_init (initializer or None) – The initializer for the the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer (for TF < 1.5).
• b_init_args (dictionary) – The arguments for the bias vector initializer (for TF < 1.5).
• use_cudnn_on_gpu (bool) – Default is False (for TF < 1.5).
• name (str) – A unique layer name.

Returns:

A Conv2dLayer object.

Return type:

Layer

Examples

>>> x = tf.placeholder(tf.float32, shape=(None, 28, 28, 1))
>>> net = InputLayer(x, name='inputs')
>>> net = Conv2d(net, 64, (3, 3), act=tf.nn.relu, name='conv1_1')
>>> net = Conv2d(net, 64, (3, 3), act=tf.nn.relu, name='conv1_2')
>>> net = MaxPool2d(net, (2, 2), name='pool1')
>>> net = Conv2d(net, 128, (3, 3), act=tf.nn.relu, name='conv2_1')
>>> net = Conv2d(net, 128, (3, 3), act=tf.nn.relu, name='conv2_2')
>>> net = MaxPool2d(net, (2, 2), name='pool2')

Simplified Deconvolutions

For users don’t familiar with TensorFlow, the following simplified functions may easier for you. We will provide more simplified functions later, but if you are good at TensorFlow, the professional APIs may better for you.

DeConv2d

class tensorlayer.layers.DeConv2d(prev_layer, n_filter=32, filter_size=(3, 3), strides=(2, 2), padding='SAME', act=None, data_format='channels_last', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='decnn2d')

Simplified version of DeConv2dLayer.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size (height, width).
• out_size (tuple of int) – Require if TF version < 1.3, (height, width) of output.
• strides (tuple of int) – The stride step (height, width).
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• act (activation function) – The activation function of this layer.
• data_format (str) – “channels_last” (NHWC, default) or “channels_first” (NCHW).
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer (For TF < 1.3).
• b_init_args (dictionary) – The arguments for the bias vector initializer (For TF < 1.3).
• name (str) – A unique layer name.

DeConv3d

class tensorlayer.layers.DeConv3d(prev_layer, n_filter=32, filter_size=(3, 3, 3), strides=(2, 2, 2), padding='SAME', act=None, data_format='channels_last', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='decnn3d')

Simplified version of The DeConv3dLayer, see tf.contrib.layers.conv3d_transpose.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size (depth, height, width).
• stride (tuple of int) – The stride step (depth, height, width).
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• act (activation function) – The activation function of this layer.
• data_format (str) – “channels_last” (NDHWC, default) or “channels_first” (NCDHW).
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip bias.
• W_init_args (dictionary) – The arguments for the weight matrix initializer (For TF < 1.3).
• b_init_args (dictionary) – The arguments for the bias vector initializer (For TF < 1.3).
• name (str) – A unique layer name.

Expert Convolutions

Conv1dLayer

class tensorlayer.layers.Conv1dLayer(prev_layer, act=None, shape=(5, 1, 5), stride=1, dilation_rate=1, padding='SAME', data_format='NWC', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='cnn1d')

The Conv1dLayer class is a 1D CNN layer, see tf.nn.convolution.

Parameters:

• prev_layer (Layer) – Previous layer.
• act (activation function) – The activation function of this layer.
• shape (tuple of int) – The shape of the filters: (filter_length, in_channels, out_channels).
• stride (int) – The number of entries by which the filter is moved right at a step.
• dilation_rate (int) – Filter up-sampling/input down-sampling rate.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• data_format (str) – Default is ‘NWC’ as it is a 1D CNN.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (str) – A unique layer name

Conv2dLayer

class tensorlayer.layers.Conv2dLayer(prev_layer, act=None, shape=(5, 5, 1, 100), strides=(1, 1, 1, 1), padding='SAME', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, use_cudnn_on_gpu=None, data_format=None, name='cnn_layer')

The Conv2dLayer class is a 2D CNN layer, see tf.nn.conv2d.

Parameters:

• prev_layer (Layer) – Previous layer.
• act (activation function) – The activation function of this layer.
• shape (tuple of int) – The shape of the filters: (filter_height, filter_width, in_channels, out_channels).
• strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the shape parameter.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• use_cudnn_on_gpu (bool) – Default is False.
• data_format (str) – “NHWC” or “NCHW”, default is “NHWC”.
• name (str) – A unique layer name.

Notes

• shape = [h, w, the number of output channel of previous layer, the number of output channels]
• the number of output channel of a layer is its last dimension.

Examples

With TensorLayer

>>> x = tf.placeholder(tf.float32, shape=(None, 28, 28, 1))
>>> net = tl.layers.InputLayer(x, name='input_layer')
>>> net = tl.layers.Conv2dLayer(net,
...                   act = tf.nn.relu,
...                   shape = (5, 5, 1, 32),  # 32 features for each 5x5 patch
...                   strides = (1, 1, 1, 1),
...                   padding='SAME',
...                   W_init=tf.truncated_normal_initializer(stddev=5e-2),
...                   b_init = tf.constant_initializer(value=0.0),
...                   name ='cnn_layer1')     # output: (?, 28, 28, 32)
>>> net = tl.layers.PoolLayer(net,
...                   ksize=(1, 2, 2, 1),
...                   strides=(1, 2, 2, 1),
...                   padding='SAME',
...                   pool = tf.nn.max_pool,
...                   name ='pool_layer1',)   # output: (?, 14, 14, 32)

Without TensorLayer, you can implement 2D convolution as follow.

>>> W = tf.Variable(W_init(shape=[5, 5, 1, 32], ), name='W_conv')
>>> b = tf.Variable(b_init(shape=[32], ), name='b_conv')
>>> outputs = tf.nn.relu( tf.nn.conv2d(inputs, W,
...                       strides=[1, 1, 1, 1],
...                       padding='SAME') + b )

Conv3dLayer

class tensorlayer.layers.Conv3dLayer(prev_layer, shape=(2, 2, 2, 3, 32), strides=(1, 2, 2, 2, 1), padding='SAME', act=None, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='cnn3d_layer')

The Conv3dLayer class is a 3D CNN layer, see tf.nn.conv3d.

Parameters:

• prev_layer (Layer) – Previous layer.
• shape (tuple of int) – Shape of the filters: (filter_depth, filter_height, filter_width, in_channels, out_channels).
• strides (tuple of int) – The sliding window strides for corresponding input dimensions. Must be in the same order as the shape dimension.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• act (activation function) – The activation function of this layer.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (str) – A unique layer name.

Examples

>>> x = tf.placeholder(tf.float32, (None, 100, 100, 100, 3))
>>> n = tl.layers.InputLayer(x, name='in3')
>>> n = tl.layers.Conv3dLayer(n, shape=(2, 2, 2, 3, 32), strides=(1, 2, 2, 2, 1))
[None, 50, 50, 50, 32]

Expert Deconvolutions

DeConv2dLayer

class tensorlayer.layers.DeConv2dLayer(prev_layer, act=None, shape=(3, 3, 128, 256), output_shape=(1, 256, 256, 128), strides=(1, 2, 2, 1), padding='SAME', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='decnn2d_layer')

A de-convolution 2D layer.

See tf.nn.conv2d_transpose.

Parameters:

• prev_layer (Layer) – Previous layer.
• act (activation function) – The activation function of this layer.
• shape (tuple of int) – Shape of the filters: (height, width, output_channels, in_channels). The filter’s in_channels dimension must match that of value.
• output_shape (tuple of int) – Output shape of the deconvolution,
• strides (tuple of int) – The sliding window strides for corresponding input dimensions.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for initializing the weight matrix.
• b_init_args (dictionary) – The arguments for initializing the bias vector.
• name (str) – A unique layer name.

Notes

• We recommend to use DeConv2d with TensorFlow version higher than 1.3.
• shape = [h, w, the number of output channels of this layer, the number of output channel of the previous layer].
• output_shape = [batch_size, any, any, the number of output channels of this layer].
• the number of output channel of a layer is its last dimension.

Examples

A part of the generator in DCGAN example

>>> batch_size = 64
>>> inputs = tf.placeholder(tf.float32, [batch_size, 100], name='z_noise')
>>> net_in = tl.layers.InputLayer(inputs, name='g/in')
>>> net_h0 = tl.layers.DenseLayer(net_in, n_units = 8192,
...                            W_init = tf.random_normal_initializer(stddev=0.02),
...                            act = None, name='g/h0/lin')
>>> print(net_h0.outputs._shape)
(64, 8192)
>>> net_h0 = tl.layers.ReshapeLayer(net_h0, shape=(-1, 4, 4, 512), name='g/h0/reshape')
>>> net_h0 = tl.layers.BatchNormLayer(net_h0, act=tf.nn.relu, is_train=is_train, name='g/h0/batch_norm')
>>> print(net_h0.outputs._shape)
(64, 4, 4, 512)
>>> net_h1 = tl.layers.DeConv2dLayer(net_h0,
...                            shape=(5, 5, 256, 512),
...                            output_shape=(batch_size, 8, 8, 256),
...                            strides=(1, 2, 2, 1),
...                            act=None, name='g/h1/decon2d')
>>> net_h1 = tl.layers.BatchNormLayer(net_h1, act=tf.nn.relu, is_train=is_train, name='g/h1/batch_norm')
>>> print(net_h1.outputs._shape)
(64, 8, 8, 256)

U-Net

>>> ....
>>> conv10 = tl.layers.Conv2dLayer(conv9, act=tf.nn.relu,
...        shape=(3,3,1024,1024), strides=(1,1,1,1), padding='SAME',
...        W_init=w_init, b_init=b_init, name='conv10')
>>> print(conv10.outputs)
(batch_size, 32, 32, 1024)
>>> deconv1 = tl.layers.DeConv2dLayer(conv10, act=tf.nn.relu,
...         shape=(3,3,512,1024), strides=(1,2,2,1), output_shape=(batch_size,64,64,512),
...         padding='SAME', W_init=w_init, b_init=b_init, name='devcon1_1')

DeConv3dLayer

class tensorlayer.layers.DeConv3dLayer(prev_layer, act=None, shape=(2, 2, 2, 128, 256), output_shape=(1, 12, 32, 32, 128), strides=(1, 2, 2, 2, 1), padding='SAME', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='decnn3d_layer')

The DeConv3dLayer class is deconvolutional 3D layer, see tf.nn.conv3d_transpose.

Parameters:

• prev_layer (Layer) – Previous layer.
• act (activation function) – The activation function of this layer.
• shape (tuple of int) – The shape of the filters: (depth, height, width, output_channels, in_channels). The filter’s in_channels dimension must match that of value.
• output_shape (tuple of int) – The output shape of the deconvolution.
• strides (tuple of int) – The sliding window strides for corresponding input dimensions.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (str) – A unique layer name.

Atrous (De)Convolutions

AtrousConv1dLayer

tensorlayer.layers.AtrousConv1dLayer(prev_layer, n_filter=32, filter_size=2, stride=1, dilation=1, act=None, padding='SAME', data_format='NWC', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='atrous_1d')

Simplified version of AtrousConv1dLayer.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_filter (int) – The number of filters.
• filter_size (int) – The filter size.
• stride (tuple of int) – The strides: (height, width).
• dilation (int) – The filter dilation size.
• act (activation function) – The activation function of this layer.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• data_format (str) – Default is ‘NWC’ as it is a 1D CNN.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (str) – A unique layer name.

Returns:

A AtrousConv1dLayer object

Return type:

Layer

AtrousConv2dLayer

class tensorlayer.layers.AtrousConv2dLayer(prev_layer, n_filter=32, filter_size=(3, 3), rate=2, act=None, padding='SAME', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='atrous_2d')

The AtrousConv2dLayer class is 2D atrous convolution (a.k.a. convolution with holes or dilated convolution) 2D layer, see tf.nn.atrous_conv2d.

Parameters:

• prev_layer (Layer) – Previous layer with a 4D output tensor in the shape of (batch, height, width, channels).
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size: (height, width).
• rate (int) – The stride that we sample input values in the height and width dimensions. This equals the rate that we up-sample the filters by inserting zeros across the height and width dimensions. In the literature, this parameter is sometimes mentioned as input stride or dilation.
• act (activation function) – The activation function of this layer.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (str) – A unique layer name.

AtrousDeConv2dLayer

class tensorlayer.layers.AtrousDeConv2dLayer(prev_layer, shape=(3, 3, 128, 256), output_shape=(1, 64, 64, 128), rate=2, act=None, padding='SAME', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='atrous_2d_transpose')

The AtrousDeConv2dLayer class is 2D atrous convolution transpose, see tf.nn.atrous_conv2d_transpose.

Parameters:

• prev_layer (Layer) – Previous layer with a 4D output tensor in the shape of (batch, height, width, channels).
• shape (tuple of int) – The shape of the filters: (filter_height, filter_width, out_channels, in_channels).
• output_shape (tuple of int) – Output shape of the deconvolution.
• rate (int) – The stride that we sample input values in the height and width dimensions. This equals the rate that we up-sample the filters by inserting zeros across the height and width dimensions. In the literature, this parameter is sometimes mentioned as input stride or dilation.
• act (activation function) – The activation function of this layer.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (str) – A unique layer name.

Deformable Convolutions

DeformableConv2d

class tensorlayer.layers.DeformableConv2d(prev_layer, offset_layer=None, n_filter=32, filter_size=(3, 3), act=None, name='deformable_conv_2d', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None)

The DeformableConv2d class is a 2D Deformable Convolutional Networks.

Parameters:

• prev_layer (Layer) – Previous layer.
• offset_layer (Layer) – To predict the offset of convolution operations. The output shape is (batchsize, input height, input width, 2*(number of element in the convolution kernel)) e.g. if apply a 3*3 kernel, the number of the last dimension should be 18 (2*3*3)
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size (height, width).
• act (activation function) – The activation function of this layer.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (str) – A unique layer name.

Examples

>>> net = tl.layers.InputLayer(x, name='input_layer')
>>> offset1 = tl.layers.Conv2d(net, 18, (3, 3), (1, 1), act=act, padding='SAME', name='offset1')
>>> net = tl.layers.DeformableConv2d(net, offset1, 32, (3, 3), act=act, name='deformable1')
>>> offset2 = tl.layers.Conv2d(net, 18, (3, 3), (1, 1), act=act, padding='SAME', name='offset2')
>>> net = tl.layers.DeformableConv2d(net, offset2, 64, (3, 3), act=act, name='deformable2')

References

• The deformation operation was adapted from the implementation in here

Notes

• The padding is fixed to ‘SAME’.
• The current implementation is not optimized for memory usgae. Please use it carefully.

Depthwise Convolutions

DepthwiseConv2d

class tensorlayer.layers.DepthwiseConv2d(prev_layer, shape=(3, 3), strides=(1, 1), act=None, padding='SAME', dilation_rate=(1, 1), depth_multiplier=1, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='depthwise_conv2d')

Separable/Depthwise Convolutional 2D layer, see tf.nn.depthwise_conv2d.

Input:

4-D Tensor (batch, height, width, in_channels).

Output:

4-D Tensor (batch, new height, new width, in_channels * depth_multiplier).

Parameters:

• prev_layer (Layer) – Previous layer.
• filter_size (tuple of int) – The filter size (height, width).
• stride (tuple of int) – The stride step (height, width).
• act (activation function) – The activation function of this layer.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• dilation_rate (tuple of 2 int) – The dilation rate in which we sample input values across the height and width dimensions in atrous convolution. If it is greater than 1, then all values of strides must be 1.
• depth_multiplier (int) – The number of channels to expand to.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip bias.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (str) – A unique layer name.

Examples

>>> net = InputLayer(x, name='input')
>>> net = Conv2d(net, 32, (3, 3), (2, 2), b_init=None, name='cin')
>>> net = BatchNormLayer(net, act=tf.nn.relu, is_train=is_train, name='bnin')
...
>>> net = DepthwiseConv2d(net, (3, 3), (1, 1), b_init=None, name='cdw1')
>>> net = BatchNormLayer(net, act=tf.nn.relu, is_train=is_train, name='bn11')
>>> net = Conv2d(net, 64, (1, 1), (1, 1), b_init=None, name='c1')
>>> net = BatchNormLayer(net, act=tf.nn.relu, is_train=is_train, name='bn12')
...
>>> net = DepthwiseConv2d(net, (3, 3), (2, 2), b_init=None, name='cdw2')
>>> net = BatchNormLayer(net, act=tf.nn.relu, is_train=is_train, name='bn21')
>>> net = Conv2d(net, 128, (1, 1), (1, 1), b_init=None, name='c2')
>>> net = BatchNormLayer(net, act=tf.nn.relu, is_train=is_train, name='bn22')

References

• tflearn’s grouped_conv_2d
• keras’s separableconv2d

Group Convolutions

GroupConv2d

class tensorlayer.layers.GroupConv2d(prev_layer, n_filter=32, filter_size=(3, 3), strides=(2, 2), n_group=2, act=None, padding='SAME', W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='groupconv')

The GroupConv2d class is 2D grouped convolution, see here.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_filter (int) – The number of filters.
• filter_size (int) – The filter size.
• stride (int) – The stride step.
• n_group (int) – The number of groups.
• act (activation function) – The activation function of this layer.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (str) – A unique layer name.

Separable Convolutions

SeparableConv1d

class tensorlayer.layers.SeparableConv1d(prev_layer, n_filter=100, filter_size=3, strides=1, act=None, padding='valid', data_format='channels_last', dilation_rate=1, depth_multiplier=1, depthwise_init=None, pointwise_init=None, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='seperable1d')

The SeparableConv1d class is a 1D depthwise separable convolutional layer, see tf.layers.separable_conv1d.

This layer performs a depthwise convolution that acts separately on channels, followed by a pointwise convolution that mixes channels.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_filter (int) – The dimensionality of the output space (i.e. the number of filters in the convolution).
• filter_size (int) – Specifying the spatial dimensions of the filters. Can be a single integer to specify the same value for all spatial dimensions.
• strides (int) – Specifying the stride of the convolution. Can be a single integer to specify the same value for all spatial dimensions. Specifying any stride value != 1 is incompatible with specifying any dilation_rate value != 1.
• padding (str) – One of “valid” or “same” (case-insensitive).
• data_format (str) – One of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width).
• dilation_rate (int) – Specifying the dilation rate to use for dilated convolution. Can be a single integer to specify the same value for all spatial dimensions. Currently, specifying any dilation_rate value != 1 is incompatible with specifying any stride value != 1.
• depth_multiplier (int) – The number of depthwise convolution output channels for each input channel. The total number of depthwise convolution output channels will be equal to num_filters_in * depth_multiplier.
• depthwise_init (initializer) – for the depthwise convolution kernel.
• pointwise_init (initializer) – For the pointwise convolution kernel.
• b_init (initializer) – For the bias vector. If None, ignore bias in the pointwise part only.
• name (a str) – A unique layer name.

SeparableConv2d

class tensorlayer.layers.SeparableConv2d(prev_layer, n_filter=100, filter_size=(3, 3), strides=(1, 1), act=None, padding='valid', data_format='channels_last', dilation_rate=(1, 1), depth_multiplier=1, depthwise_init=None, pointwise_init=None, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='seperable')

The SeparableConv2d class is a 2D depthwise separable convolutional layer, see tf.layers.separable_conv2d.

This layer performs a depthwise convolution that acts separately on channels, followed by a pointwise convolution that mixes channels. While DepthwiseConv2d performs depthwise convolution only, which allow us to add batch normalization between depthwise and pointwise convolution.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_filter (int) – The dimensionality of the output space (i.e. the number of filters in the convolution).
• filter_size (tuple/list of 2 int) – Specifying the spatial dimensions of the filters. Can be a single integer to specify the same value for all spatial dimensions.
• strides (tuple/list of 2 int) – Specifying the strides of the convolution. Can be a single integer to specify the same value for all spatial dimensions. Specifying any stride value != 1 is incompatible with specifying any dilation_rate value != 1.
• padding (str) – One of “valid” or “same” (case-insensitive).
• data_format (str) – One of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width).
• dilation_rate (integer or tuple/list of 2 int) – Specifying the dilation rate to use for dilated convolution. Can be a single integer to specify the same value for all spatial dimensions. Currently, specifying any dilation_rate value != 1 is incompatible with specifying any stride value != 1.
• depth_multiplier (int) – The number of depthwise convolution output channels for each input channel. The total number of depthwise convolution output channels will be equal to num_filters_in * depth_multiplier.
• depthwise_init (initializer) – for the depthwise convolution kernel.
• pointwise_init (initializer) – For the pointwise convolution kernel.
• b_init (initializer) – For the bias vector. If None, ignore bias in the pointwise part only.
• name (a str) – A unique layer name.

SubPixel Convolutions

SubpixelConv1d

class tensorlayer.layers.SubpixelConv1d(prev_layer, scale=2, act=None, name='subpixel_conv1d')

It is a 1D sub-pixel up-sampling layer.

Calls a TensorFlow function that directly implements this functionality. We assume input has dim (batch, width, r)

Parameters:

• net (Layer) – Previous layer with output shape of (batch, width, r).
• scale (int) – The up-scaling ratio, a wrong setting will lead to Dimension size error.
• act (activation function) – The activation function of this layer.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> t_signal = tf.placeholder('float32', [10, 100, 4], name='x')
>>> n = tl.layers.InputLayer(t_signal, name='in')
>>> n = tl.layers.SubpixelConv1d(n, scale=2, name='s')
>>> print(n.outputs.shape)
(10, 200, 2)

References

Audio Super Resolution Implementation.

SubpixelConv2d

class tensorlayer.layers.SubpixelConv2d(prev_layer, scale=2, n_out_channel=None, act=None, name='subpixel_conv2d')

It is a 2D sub-pixel up-sampling layer, usually be used for Super-Resolution applications, see SRGAN for example.

Parameters:

• prev_layer (Layer) – Previous layer,
• scale (int) – The up-scaling ratio, a wrong setting will lead to dimension size error.
• n_out_channel (int or None) – The number of output channels. - If None, automatically set n_out_channel == the number of input channels / (scale x scale). - The number of input channels == (scale x scale) x The number of output channels.
• act (activation function) – The activation function of this layer.
• name (str) – A unique layer name.

Examples

>>> # examples here just want to tell you how to set the n_out_channel.
>>> import numpy as np
>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = np.random.rand(2, 16, 16, 4)
>>> X = tf.placeholder("float32", shape=(2, 16, 16, 4), name="X")
>>> net = tl.layers.InputLayer(X, name='input')
>>> net = tl.layers.SubpixelConv2d(net, scale=2, n_out_channel=1, name='subpixel_conv2d')
>>> sess = tf.Session()
>>> y = sess.run(net.outputs, feed_dict={X: x})
>>> print(x.shape, y.shape)
(2, 16, 16, 4) (2, 32, 32, 1)

>>> x = np.random.rand(2, 16, 16, 4*10)
>>> X = tf.placeholder("float32", shape=(2, 16, 16, 4*10), name="X")
>>> net = tl.layers.InputLayer(X, name='input2')
>>> net = tl.layers.SubpixelConv2d(net, scale=2, n_out_channel=10, name='subpixel_conv2d2')
>>> y = sess.run(net.outputs, feed_dict={X: x})
>>> print(x.shape, y.shape)
(2, 16, 16, 40) (2, 32, 32, 10)

>>> x = np.random.rand(2, 16, 16, 25*10)
>>> X = tf.placeholder("float32", shape=(2, 16, 16, 25*10), name="X")
>>> net = tl.layers.InputLayer(X, name='input3')
>>> net = tl.layers.SubpixelConv2d(net, scale=5, n_out_channel=None, name='subpixel_conv2d3')
>>> y = sess.run(net.outputs, feed_dict={X: x})
>>> print(x.shape, y.shape)
(2, 16, 16, 250) (2, 80, 80, 10)

References

• Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

Dense Layers

Dense Layer

class tensorlayer.layers.DenseLayer(prev_layer, n_units=100, act=None, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='dense')

The DenseLayer class is a fully connected layer.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_units (int) – The number of units of this layer.
• act (activation function) – The activation function of this layer.
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (a str) – A unique layer name.

Examples

With TensorLayer

>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.DenseLayer(net, 800, act=tf.nn.relu, name='relu')

Without native TensorLayer APIs, you can do as follow.

>>> W = tf.Variable(
...     tf.random_uniform([n_in, n_units], -1.0, 1.0), name='W')
>>> b = tf.Variable(tf.zeros(shape=[n_units]), name='b')
>>> y = tf.nn.relu(tf.matmul(inputs, W) + b)

Notes

If the layer input has more than two axes, it needs to be flatten by using FlattenLayer.

Drop Connect Dense Layer

class tensorlayer.layers.DropconnectDenseLayer(prev_layer, keep=0.5, n_units=100, act=None, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='dropconnect_layer')

The DropconnectDenseLayer class is DenseLayer with DropConnect behaviour which randomly removes connections between this layer and the previous layer according to a keeping probability.

Parameters:

• prev_layer (Layer) – Previous layer.
• keep (float) – The keeping probability. The lower the probability it is, the more activations are set to zero.
• n_units (int) – The number of units of this layer.
• act (activation function) – The activation function of this layer.
• W_init (weights initializer) – The initializer for the weight matrix.
• b_init (biases initializer) – The initializer for the bias vector.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (str) – A unique layer name.

Examples

>>> net = tl.layers.InputLayer(x, name='input_layer')
>>> net = tl.layers.DropconnectDenseLayer(net, keep=0.8,
...         n_units=800, act=tf.nn.relu, name='relu1')
>>> net = tl.layers.DropconnectDenseLayer(net, keep=0.5,
...         n_units=800, act=tf.nn.relu, name='relu2')
>>> net = tl.layers.DropconnectDenseLayer(net, keep=0.5,
...         n_units=10, name='output')

References

• Wan, L. (2013). Regularization of neural networks using dropconnect

Dropout Layers

class tensorlayer.layers.DropoutLayer(prev_layer, keep=0.5, is_fix=False, is_train=True, seed=None, name='dropout_layer')

The DropoutLayer class is a noise layer which randomly set some activations to zero according to a keeping probability.

Parameters:

• prev_layer (Layer) – Previous layer.
• keep (float) – The keeping probability. The lower the probability it is, the more activations are set to zero.
• is_fix (boolean) – Fixing probability or nor. Default is False. If True, the keeping probability is fixed and cannot be changed via feed_dict.
• is_train (boolean) – Trainable or not. If False, skip this layer. Default is True.
• seed (int or None) – The seed for random dropout.
• name (str) – A unique layer name.

Examples

Method 1: Using all_drop see tutorial_mlp_dropout1.py

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> net = tl.layers.InputLayer(x, name='input_layer')
>>> net = tl.layers.DropoutLayer(net, keep=0.8, name='drop1')
>>> net = tl.layers.DenseLayer(net, n_units=800, act=tf.nn.relu, name='relu1')
>>> ...
>>> # For training, enable dropout as follow.
>>> feed_dict = {x: X_train_a, y_: y_train_a}
>>> feed_dict.update( net.all_drop )     # enable noise layers
>>> sess.run(train_op, feed_dict=feed_dict)
>>> ...
>>> # For testing, disable dropout as follow.
>>> dp_dict = tl.utils.dict_to_one( net.all_drop ) # disable noise layers
>>> feed_dict = {x: X_val_a, y_: y_val_a}
>>> feed_dict.update(dp_dict)
>>> err, ac = sess.run([cost, acc], feed_dict=feed_dict)
>>> ...

Method 2: Without using all_drop see tutorial_mlp_dropout2.py

>>> def mlp(x, is_train=True, reuse=False):
>>>     with tf.variable_scope("MLP", reuse=reuse):
>>>     tl.layers.set_name_reuse(reuse)
>>>     net = tl.layers.InputLayer(x, name='input')
>>>     net = tl.layers.DropoutLayer(net, keep=0.8, is_fix=True,
>>>                         is_train=is_train, name='drop1')
>>>     ...
>>>     return net

>>> net_train = mlp(x, is_train=True, reuse=False)
>>> net_test = mlp(x, is_train=False, reuse=True)

Extend Layers

Expand Dims Layer

class tensorlayer.layers.ExpandDimsLayer(prev_layer, axis, name='expand_dims')

The ExpandDimsLayer class inserts a dimension of 1 into a tensor’s shape, see tf.expand_dims() .

Parameters:

• prev_layer (Layer) – The previous layer.
• axis (int) – The dimension index at which to expand the shape of input.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, (None, 100))
>>> n = tl.layers.InputLayer(x, name='in')
>>> n = tl.layers.ExpandDimsLayer(n, 2)
[None, 100, 1]

Tile layer

class tensorlayer.layers.TileLayer(prev_layer, multiples=None, name='tile')

The TileLayer class constructs a tensor by tiling a given tensor, see tf.tile() .

Parameters:

• prev_layer (Layer) – The previous layer.
• multiples (tensor) – Must be one of the following types: int32, int64. 1-D Length must be the same as the number of dimensions in input.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, (None, 100))
>>> n = tl.layers.InputLayer(x, name='in')
>>> n = tl.layers.ExpandDimsLayer(n, 2)
>>> n = tl.layers.TileLayer(n, [-1, 1, 3])
[None, 100, 3]

External Libraries Layers

TF-Slim Layer

TF-Slim models can be connected into TensorLayer. All Google’s Pre-trained model can be used easily , see Slim-model.

class tensorlayer.layers.SlimNetsLayer(prev_layer, slim_layer, slim_args=None, name='tfslim_layer')

A layer that merges TF-Slim models into TensorLayer.

Models can be found in slim-model, see Inception V3 example on Github.

Parameters:

• prev_layer (Layer) – Previous layer.
• slim_layer (a slim network function) – The network you want to stack onto, end with return net, end_points.
• slim_args (dictionary) – The arguments for the slim model.
• name (str) – A unique layer name.

Notes

• As TF-Slim stores the layers as dictionary, the all_layers in this network is not in order ! Fortunately, the all_params are in order.

Flow Control Layer

class tensorlayer.layers.MultiplexerLayer(layers, name='mux_layer')

The MultiplexerLayer selects inputs to be forwarded to output. see tutorial_mnist_multiplexer.py.

Parameters:

• layers (a list of Layer) – The input layers.
• name (str) – A unique layer name.

sel

placeholder – The placeholder takes an integer for selecting which layer to output.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, shape=(None, 784), name='x')
>>> # define the network
>>> net_in = tl.layers.InputLayer(x, name='input')
>>> net_in = tl.layers.DropoutLayer(net_in, keep=0.8, name='drop1')
>>> # net 0
>>> net_0 = tl.layers.DenseLayer(net_in, n_units=800, act=tf.nn.relu, name='net0/relu1')
>>> net_0 = tl.layers.DropoutLayer(net_0, keep=0.5, name='net0/drop2')
>>> net_0 = tl.layers.DenseLayer(net_0, n_units=800, act=tf.nn.relu, name='net0/relu2')
>>> # net 1
>>> net_1 = tl.layers.DenseLayer(net_in, n_units=800, act=tf.nn.relu, name='net1/relu1')
>>> net_1 = tl.layers.DropoutLayer(net_1, keep=0.8, name='net1/drop2')
>>> net_1 = tl.layers.DenseLayer(net_1, n_units=800, act=tf.nn.relu, name='net1/relu2')
>>> net_1 = tl.layers.DropoutLayer(net_1, keep=0.8, name='net1/drop3')
>>> net_1 = tl.layers.DenseLayer(net_1, n_units=800, act=tf.nn.relu, name='net1/relu3')
>>> # multiplexer
>>> net_mux = tl.layers.MultiplexerLayer(layers=[net_0, net_1], name='mux')
>>> network = tl.layers.ReshapeLayer(net_mux, shape=(-1, 800), name='reshape')
>>> network = tl.layers.DropoutLayer(network, keep=0.5, name='drop3')
>>> # output layer
>>> network = tl.layers.DenseLayer(network, n_units=10, act=None, name='output')

Image Resampling Layers

2D UpSampling

class tensorlayer.layers.UpSampling2dLayer(prev_layer, size, is_scale=True, method=0, align_corners=False, name='upsample2d_layer')

The UpSampling2dLayer class is a up-sampling 2D layer.

See tf.image.resize_images.

Parameters:

• prev_layer (Layer) – Previous layer with 4-D Tensor of the shape (batch, height, width, channels) or 3-D Tensor of the shape (height, width, channels).
• size (tuple of int/float) – (height, width) scale factor or new size of height and width.
• is_scale (boolean) – If True (default), the size is a scale factor; otherwise, the size is the numbers of pixels of height and width.
• method (int) –

  The resize method selected through the index. Defaults index is 0 which is ResizeMethod.BILINEAR.

  • Index 0 is ResizeMethod.BILINEAR, Bilinear interpolation.
  • Index 1 is ResizeMethod.NEAREST_NEIGHBOR, Nearest neighbor interpolation.
  • Index 2 is ResizeMethod.BICUBIC, Bicubic interpolation.
  • Index 3 ResizeMethod.AREA, Area interpolation.

• align_corners (boolean) – If True, align the corners of the input and output. Default is False.
• name (str) – A unique layer name.

2D DownSampling

class tensorlayer.layers.DownSampling2dLayer(prev_layer, size, is_scale=True, method=0, align_corners=False, name='downsample2d_layer')

The DownSampling2dLayer class is down-sampling 2D layer.

See tf.image.resize_images.

Parameters:

• prev_layer (Layer) – Previous layer with 4-D Tensor in the shape of (batch, height, width, channels) or 3-D Tensor in the shape of (height, width, channels).
• size (tuple of int/float) – (height, width) scale factor or new size of height and width.
• is_scale (boolean) – If True (default), the size is the scale factor; otherwise, the size are numbers of pixels of height and width.
• method (int) –

  The resize method selected through the index. Defaults index is 0 which is ResizeMethod.BILINEAR.

  • Index 0 is ResizeMethod.BILINEAR, Bilinear interpolation.
  • Index 1 is ResizeMethod.NEAREST_NEIGHBOR, Nearest neighbor interpolation.
  • Index 2 is ResizeMethod.BICUBIC, Bicubic interpolation.
  • Index 3 ResizeMethod.AREA, Area interpolation.

• align_corners (boolean) – If True, exactly align all 4 corners of the input and output. Default is False.
• name (str) – A unique layer name.

Lambda Layers

Lambda Layer

class tensorlayer.layers.LambdaLayer(prev_layer, fn, fn_args=None, name='lambda_layer')

A layer that takes a user-defined function using TensorFlow Lambda, for multiple inputs see ElementwiseLambdaLayer.

Parameters:

• prev_layer (Layer) – Previous layer.
• fn (function) – The function that applies to the outputs of previous layer.
• fn_args (dictionary or None) – The arguments for the function (option).
• name (str) – A unique layer name.

Examples

Non-parametric case

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, shape=[None, 1], name='x')
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.LambdaLayer(net, lambda x: 2*x, name='lambda')

Parametric case, merge other wrappers into TensorLayer

>>> from keras.layers import *
>>> from tensorlayer.layers import *
>>> def keras_block(x):
>>>     x = Dropout(0.8)(x)
>>>     x = Dense(800, activation='relu')(x)
>>>     x = Dropout(0.5)(x)
>>>     x = Dense(800, activation='relu')(x)
>>>     x = Dropout(0.5)(x)
>>>     logits = Dense(10, activation='linear')(x)
>>>     return logits
>>> net = InputLayer(x, name='input')
>>> net = LambdaLayer(net, fn=keras_block, name='keras')

ElementWise Lambda Layer

class tensorlayer.layers.ElementwiseLambdaLayer(layers, fn, fn_args=None, act=None, name='elementwiselambda_layer')

A layer that use a custom function to combine multiple Layer inputs.

Parameters:

• layers (list of Layer) – The list of layers to combine.
• fn (function) – The function that applies to the outputs of previous layer.
• fn_args (dictionary or None) – The arguments for the function (option).
• act (activation function) – The activation function of this layer.
• name (str) – A unique layer name.

Examples

z = mean + noise * tf.exp(std * 0.5)

>>> import tensorflow as tf
>>> import tensorlayer as tl

>>> def func(noise, mean, std):
>>>     return mean + noise * tf.exp(std * 0.5)

>>> x = tf.placeholder(tf.float32, [None, 200])
>>> noise_tensor = tf.random_normal(tf.stack([tf.shape(x)[0], 200]))
>>> noise = tl.layers.InputLayer(noise_tensor)
>>> net = tl.layers.InputLayer(x)
>>> net = tl.layers.DenseLayer(net, n_units=200, act=tf.nn.relu, name='dense1')
>>> mean = tl.layers.DenseLayer(net, n_units=200, name='mean')
>>> std = tl.layers.DenseLayer(net, n_units=200, name='std')
>>> z = tl.layers.ElementwiseLambdaLayer([noise, mean, std], fn=func, name='z')

Merge Layers

Concat Layer

class tensorlayer.layers.ConcatLayer(prev_layer, concat_dim=-1, name='concat_layer')

A layer that concats multiple tensors according to given axis.

Parameters:

• prev_layer (list of Layer) – List of layers to concatenate.
• concat_dim (int) – The dimension to concatenate.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> sess = tf.InteractiveSession()
>>> x = tf.placeholder(tf.float32, shape=[None, 784])
>>> inputs = tl.layers.InputLayer(x, name='input_layer')
[TL]   InputLayer input_layer (?, 784)
>>> net1 = tl.layers.DenseLayer(inputs, 800, act=tf.nn.relu, name='relu1_1')
[TL]   DenseLayer relu1_1: 800, relu
>>> net2 = tl.layers.DenseLayer(inputs, 300, act=tf.nn.relu, name='relu2_1')
[TL]   DenseLayer relu2_1: 300, relu
>>> net = tl.layers.ConcatLayer([net1, net2], 1, name ='concat_layer')
[TL]   ConcatLayer concat_layer, 1100
>>> tl.layers.initialize_global_variables(sess)
>>> net.print_params()
[TL]   param   0: relu1_1/W:0          (784, 800)         float32_ref
[TL]   param   1: relu1_1/b:0          (800,)             float32_ref
[TL]   param   2: relu2_1/W:0          (784, 300)         float32_ref
[TL]   param   3: relu2_1/b:0          (300,)             float32_ref
    num of params: 863500
>>> net.print_layers()
[TL]   layer   0: relu1_1/Relu:0       (?, 800)           float32
[TL]   layer   1: relu2_1/Relu:0       (?, 300)           float32
[TL]   layer   2: concat_layer:0       (?, 1100)          float32

ElementWise Layer

class tensorlayer.layers.ElementwiseLayer(prev_layer, combine_fn=<sphinx.ext.autodoc.importer._MockObject object>, act=None, name='elementwise_layer')

A layer that combines multiple Layer that have the same output shapes according to an element-wise operation.

Parameters:

• prev_layer (list of Layer) – The list of layers to combine.
• combine_fn (a TensorFlow element-wise combine function) – e.g. AND is tf.minimum ; OR is tf.maximum ; ADD is tf.add ; MUL is tf.multiply and so on. See TensorFlow Math API .
• act (activation function) – The activation function of this layer.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, shape=[None, 784])
>>> inputs = tl.layers.InputLayer(x, name='input_layer')
>>> net_0 = tl.layers.DenseLayer(inputs, n_units=500, act=tf.nn.relu, name='net_0')
>>> net_1 = tl.layers.DenseLayer(inputs, n_units=500, act=tf.nn.relu, name='net_1')
>>> net = tl.layers.ElementwiseLayer([net_0, net_1], combine_fn=tf.minimum, name='minimum')
>>> net.print_params(False)
[TL]   param   0: net_0/W:0            (784, 500)         float32_ref
[TL]   param   1: net_0/b:0            (500,)             float32_ref
[TL]   param   2: net_1/W:0            (784, 500)         float32_ref
[TL]   param   3: net_1/b:0            (500,)             float32_ref
>>> net.print_layers()
[TL]   layer   0: net_0/Relu:0         (?, 500)           float32
[TL]   layer   1: net_1/Relu:0         (?, 500)           float32
[TL]   layer   2: minimum:0            (?, 500)           float32

Noise Layer

class tensorlayer.layers.GaussianNoiseLayer(prev_layer, mean=0.0, stddev=1.0, is_train=True, seed=None, name='gaussian_noise_layer')

The GaussianNoiseLayer class is noise layer that adding noise with gaussian distribution to the activation.

Parameters:

• prev_layer (Layer) – Previous layer.
• mean (float) – The mean. Default is 0.
• stddev (float) – The standard deviation. Default is 1.
• is_train (boolean) – Is trainable layer. If False, skip this layer. default is True.
• seed (int or None) – The seed for random noise.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, shape=(100, 784))
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.DenseLayer(net, n_units=100, act=tf.nn.relu, name='dense3')
>>> net = tl.layers.GaussianNoiseLayer(net, name='gaussian')
(64, 100)

Normalization Layers

For local response normalization as it does not have any weights and arguments, you can also apply tf.nn.lrn on network.outputs.

Batch Normalization

class tensorlayer.layers.BatchNormLayer(prev_layer, decay=0.9, epsilon=1e-05, act=None, is_train=False, beta_init=<sphinx.ext.autodoc.importer._MockObject object>, gamma_init=<sphinx.ext.autodoc.importer._MockObject object>, moving_mean_init=<sphinx.ext.autodoc.importer._MockObject object>, name='batchnorm_layer')

The BatchNormLayer is a batch normalization layer for both fully-connected and convolution outputs. See tf.nn.batch_normalization and tf.nn.moments.

Parameters:

• prev_layer (Layer) – The previous layer.
• decay (float) – A decay factor for ExponentialMovingAverage. Suggest to use a large value for large dataset.
• epsilon (float) – Eplison.
• act (activation function) – The activation function of this layer.
• is_train (boolean) – Is being used for training or inference.
• beta_init (initializer or None) – The initializer for initializing beta, if None, skip beta. Usually you should not skip beta unless you know what happened.
• gamma_init (initializer or None) – The initializer for initializing gamma, if None, skip gamma. When the batch normalization layer is use instead of ‘biases’, or the next layer is linear, this can be disabled since the scaling can be done by the next layer. see Inception-ResNet-v2
• name (str) – A unique layer name.

References

• Source
• stackoverflow

Local Response Normalization

class tensorlayer.layers.LocalResponseNormLayer(prev_layer, depth_radius=None, bias=None, alpha=None, beta=None, name='lrn_layer')

The LocalResponseNormLayer layer is for Local Response Normalization. See tf.nn.local_response_normalization or tf.nn.lrn for new TF version. The 4-D input tensor is a 3-D array of 1-D vectors (along the last dimension), and each vector is normalized independently. Within a given vector, each component is divided by the weighted square-sum of inputs within depth_radius.

Parameters:

• prev_layer (Layer) – The previous layer with a 4D output shape.
• depth_radius (int) – Depth radius. 0-D. Half-width of the 1-D normalization window.
• bias (float) – An offset which is usually positive and shall avoid dividing by 0.
• alpha (float) – A scale factor which is usually positive.
• beta (float) – An exponent.
• name (str) – A unique layer name.

Instance Normalization

class tensorlayer.layers.InstanceNormLayer(prev_layer, act=None, epsilon=1e-05, name='instan_norm')

The InstanceNormLayer class is a for instance normalization.

Parameters:

• prev_layer (Layer) – The previous layer.
• act (activation function.) – The activation function of this layer.
• epsilon (float) – Eplison.
• name (str) – A unique layer name

Layer Normalization

class tensorlayer.layers.LayerNormLayer(prev_layer, center=True, scale=True, act=None, reuse=None, variables_collections=None, outputs_collections=None, trainable=True, begin_norm_axis=1, begin_params_axis=-1, name='layernorm')

The LayerNormLayer class is for layer normalization, see tf.contrib.layers.layer_norm.

Parameters:

• prev_layer (Layer) – The previous layer.
• act (activation function) – The activation function of this layer.
• others – tf.contrib.layers.layer_norm.

Switch Normalization

class tensorlayer.layers.SwitchNormLayer(prev_layer, act=None, epsilon=1e-05, beta_init=<sphinx.ext.autodoc.importer._MockObject object>, gamma_init=<sphinx.ext.autodoc.importer._MockObject object>, moving_mean_init=<sphinx.ext.autodoc.importer._MockObject object>, name='switchnorm_layer')

The SwitchNormLayer is a switchable normalization.

Parameters:

• prev_layer (Layer) – The previous layer.
• act (activation function) – The activation function of this layer.
• epsilon (float) – Eplison.
• beta_init (initializer or None) – The initializer for initializing beta, if None, skip beta. Usually you should not skip beta unless you know what happened.
• gamma_init (initializer or None) – The initializer for initializing gamma, if None, skip gamma. When the batch normalization layer is use instead of ‘biases’, or the next layer is linear, this can be disabled since the scaling can be done by the next layer. see Inception-ResNet-v2
• name (str) – A unique layer name.

References

• Differentiable Learning-to-Normalize via Switchable Normalization
• Zhihu (CN)

Object Detection Layer

class tensorlayer.layers.ROIPoolingLayer(prev_layer, rois, pool_height=2, pool_width=2, name='roipooling_layer')

The region of interest pooling layer.

Parameters:

• prev_layer (Layer) – The previous layer.
• rois (tuple of int) – Regions of interest in the format of (feature map index, upper left, bottom right).
• pool_width (int) – The size of the pooling sections.
• pool_width – The size of the pooling sections.
• name (str) – A unique layer name.

Notes

• This implementation is imported from Deepsense-AI .
• Please install it by the instruction HERE.

Padding Layers

Pad Layer (Expert API)

Padding layer for any modes.

class tensorlayer.layers.PadLayer(prev_layer, padding=None, mode='CONSTANT', name='pad_layer')

The PadLayer class is a padding layer for any mode and dimension. Please see tf.pad for usage.

Parameters:

• prev_layer (Layer) – The previous layer.
• padding (list of lists of 2 ints, or a Tensor of type int32.) – The int32 values to pad.
• mode (str) – “CONSTANT”, “REFLECT”, or “SYMMETRIC” (case-insensitive).
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> images = tf.placeholder(tf.float32, [None, 224, 224, 3])
>>> net = tl.layers.InputLayer(images, name='in')
>>> net = tl.layers.PadLayer(net, [[0, 0], [3, 3], [3, 3], [0, 0]], "REFLECT", name='inpad')

1D Zero padding

class tensorlayer.layers.ZeroPad1d(prev_layer, padding, name='zeropad1d')

The ZeroPad1d class is a 1D padding layer for signal [batch, length, channel].

Parameters:

• prev_layer (Layer) – The previous layer.
• padding (int, or tuple of 2 ints) –
  • If int, zeros to add at the beginning and end of the padding dimension (axis 1).
  • If tuple of 2 ints, zeros to add at the beginning and at the end of the padding dimension.
• name (str) – A unique layer name.

2D Zero padding

class tensorlayer.layers.ZeroPad2d(prev_layer, padding, name='zeropad2d')

The ZeroPad2d class is a 2D padding layer for image [batch, height, width, channel].

Parameters:

• prev_layer (Layer) – The previous layer.
• padding (int, or tuple of 2 ints, or tuple of 2 tuples of 2 ints.) –
  • If int, the same symmetric padding is applied to width and height.
  • If tuple of 2 ints, interpreted as two different symmetric padding values for height and width as (symmetric_height_pad, symmetric_width_pad).
  • If tuple of 2 tuples of 2 ints, interpreted as ((top_pad, bottom_pad), (left_pad, right_pad)).
• name (str) – A unique layer name.

3D Zero padding

class tensorlayer.layers.ZeroPad3d(prev_layer, padding, name='zeropad3d')

The ZeroPad3d class is a 3D padding layer for volume [batch, depth, height, width, channel].

Parameters:

• prev_layer (Layer) – The previous layer.
• padding (int, or tuple of 2 ints, or tuple of 2 tuples of 2 ints.) –
  • If int, the same symmetric padding is applied to width and height.
  • If tuple of 2 ints, interpreted as two different symmetric padding values for height and width as (symmetric_dim1_pad, symmetric_dim2_pad, symmetric_dim3_pad).
  • If tuple of 2 tuples of 2 ints, interpreted as ((left_dim1_pad, right_dim1_pad), (left_dim2_pad, right_dim2_pad), (left_dim3_pad, right_dim3_pad)).
• name (str) – A unique layer name.

Pooling Layers

Pool Layer (Expert API)

Pooling layer for any dimensions and any pooling functions.

class tensorlayer.layers.PoolLayer(prev_layer, ksize=(1, 2, 2, 1), strides=(1, 2, 2, 1), padding='SAME', pool=<sphinx.ext.autodoc.importer._MockObject object>, name='pool_layer')

The PoolLayer class is a Pooling layer. You can choose tf.nn.max_pool and tf.nn.avg_pool for 2D input or tf.nn.max_pool3d and tf.nn.avg_pool3d for 3D input.

Parameters:

• prev_layer (Layer) – The previous layer.
• ksize (tuple of int) – The size of the window for each dimension of the input tensor. Note that: len(ksize) >= 4.
• strides (tuple of int) – The stride of the sliding window for each dimension of the input tensor. Note that: len(strides) >= 4.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• pool (pooling function) – One of tf.nn.max_pool, tf.nn.avg_pool, tf.nn.max_pool3d and f.nn.avg_pool3d. See TensorFlow pooling APIs
• name (str) – A unique layer name.

Examples

• see Conv2dLayer.

1D Max pooling

class tensorlayer.layers.MaxPool1d(prev_layer, filter_size=3, strides=2, padding='valid', data_format='channels_last', name='maxpool1d')

Max pooling for 1D signal.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 3 [batch, length, channel].
• filter_size (tuple of int) – Pooling window size.
• strides (tuple of int) – Strides of the pooling operation.
• padding (str) – The padding method: ‘valid’ or ‘same’.
• data_format (str) – One of channels_last (default, [batch, length, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

1D Mean pooling

class tensorlayer.layers.MeanPool1d(prev_layer, filter_size=3, strides=2, padding='valid', data_format='channels_last', name='meanpool1d')

Mean pooling for 1D signal.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 3.
• filter_size (tuple of int) – Pooling window size.
• strides (tuple of int) – Strides of the pooling operation.
• padding (str) – The padding method: ‘valid’ or ‘same’.
• data_format (str) – One of channels_last (default, [batch, length, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

2D Max pooling

class tensorlayer.layers.MaxPool2d(prev_layer, filter_size=(3, 3), strides=(2, 2), padding='SAME', data_format='channels_last', name='maxpool2d')

Max pooling for 2D image.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 4.
• filter_size (tuple of int) – (height, width) for filter size.
• strides (tuple of int) – (height, width) for strides.
• padding (str) – The padding method: ‘valid’ or ‘same’.
• data_format (str) – One of channels_last (default, [batch, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

2D Mean pooling

class tensorlayer.layers.MeanPool2d(prev_layer, filter_size=(3, 3), strides=(2, 2), padding='SAME', data_format='channels_last', name='meanpool2d')

Mean pooling for 2D image [batch, height, width, channel].

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 4 [batch, height, width, channel].
• filter_size (tuple of int) – (height, width) for filter size.
• strides (tuple of int) – (height, width) for strides.
• padding (str) – The padding method: ‘valid’ or ‘same’.
• data_format (str) – One of channels_last (default, [batch, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

3D Max pooling

class tensorlayer.layers.MaxPool3d(prev_layer, filter_size=(3, 3, 3), strides=(2, 2, 2), padding='valid', data_format='channels_last', name='maxpool3d')

Max pooling for 3D volume.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 5.
• filter_size (tuple of int) – Pooling window size.
• strides (tuple of int) – Strides of the pooling operation.
• padding (str) – The padding method: ‘valid’ or ‘same’.
• data_format (str) – One of channels_last (default, [batch, depth, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

Returns:

A max pooling 3-D layer with a output rank as 5.

Return type:

Layer

3D Mean pooling

class tensorlayer.layers.MeanPool3d(prev_layer, filter_size=(3, 3, 3), strides=(2, 2, 2), padding='valid', data_format='channels_last', name='meanpool3d')

Mean pooling for 3D volume.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 5.
• filter_size (tuple of int) – Pooling window size.
• strides (tuple of int) – Strides of the pooling operation.
• padding (str) – The padding method: ‘valid’ or ‘same’.
• data_format (str) – One of channels_last (default, [batch, depth, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

Returns:

A mean pooling 3-D layer with a output rank as 5.

Return type:

Layer

1D Global Max pooling

class tensorlayer.layers.GlobalMaxPool1d(prev_layer, data_format='channels_last', name='globalmaxpool1d')

The GlobalMaxPool1d class is a 1D Global Max Pooling layer.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 3 [batch, length, channel].
• data_format (str) – One of channels_last (default, [batch, length, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

Examples

>>> x = tf.placeholder("float32", [None, 100, 30])
>>> n = InputLayer(x, name='in')
>>> n = GlobalMaxPool1d(n)
[None, 30]

1D Global Mean pooling

class tensorlayer.layers.GlobalMeanPool1d(prev_layer, data_format='channels_last', name='globalmeanpool1d')

The GlobalMeanPool1d class is a 1D Global Mean Pooling layer.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 3 [batch, length, channel].
• data_format (str) – One of channels_last (default, [batch, length, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder("float32", [None, 100, 30])
>>> n = tl.layers.InputLayer(x, name='in')
>>> n = tl.layers.GlobalMeanPool1d(n)
[None, 30]

2D Global Max pooling

class tensorlayer.layers.GlobalMaxPool2d(prev_layer, data_format='channels_last', name='globalmaxpool2d')

The GlobalMaxPool2d class is a 2D Global Max Pooling layer.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 4 [batch, height, width, channel].
• data_format (str) – One of channels_last (default, [batch, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder("float32", [None, 100, 100, 30])
>>> n = tl.layers.InputLayer(x, name='in2')
>>> n = tl.layers.GlobalMaxPool2d(n)
[None, 30]

2D Global Mean pooling

class tensorlayer.layers.GlobalMeanPool2d(prev_layer, data_format='channels_last', name='globalmeanpool2d')

The GlobalMeanPool2d class is a 2D Global Mean Pooling layer.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 4 [batch, height, width, channel].
• data_format (str) – One of channels_last (default, [batch, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder("float32", [None, 100, 100, 30])
>>> n = tl.layers.InputLayer(x, name='in2')
>>> n = tl.layers.GlobalMeanPool2d(n)
[None, 30]

3D Global Max pooling

class tensorlayer.layers.GlobalMaxPool3d(prev_layer, data_format='channels_last', name='globalmaxpool3d')

The GlobalMaxPool3d class is a 3D Global Max Pooling layer.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 5 [batch, depth, height, width, channel].
• data_format (str) – One of channels_last (default, [batch, depth, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder("float32", [None, 100, 100, 100, 30])
>>> n = tl.layers.InputLayer(x, name='in')
>>> n = tl.layers.GlobalMaxPool3d(n)
[None, 30]

3D Global Mean pooling

class tensorlayer.layers.GlobalMeanPool3d(prev_layer, data_format='channels_last', name='globalmeanpool3d')

The GlobalMeanPool3d class is a 3D Global Mean Pooling layer.

Parameters:

• prev_layer (Layer) – The previous layer with a output rank as 5 [batch, depth, height, width, channel].
• data_format (str) – One of channels_last (default, [batch, depth, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder("float32", [None, 100, 100, 100, 30])
>>> n = tl.layers.InputLayer(x, name='in')
>>> n = tl.layers.GlobalMeanPool2d(n)
[None, 30]

Quantized Nets

This is an experimental API package for building Quantized Neural Networks. We are using matrix multiplication rather than add-minus and bit-count operation at the moment. Therefore, these APIs would not speed up the inferencing, for production, you can train model via TensorLayer and deploy the model into other customized C/C++ implementation (We probably provide users an extra C/C++ binary net framework that can load model from TensorLayer).

Note that, these experimental APIs can be changed in the future

Sign

class tensorlayer.layers.SignLayer(prev_layer, name='sign')

The SignLayer class is for quantizing the layer outputs to -1 or 1 while inferencing.

Parameters:

• prev_layer (Layer) – Previous layer.
• name (a str) – A unique layer name.

Scale

class tensorlayer.layers.ScaleLayer(prev_layer, init_scale=0.05, name='scale')

The AddScaleLayer class is for multipling a trainble scale value to the layer outputs. Usually be used on the output of binary net.

Parameters:

• prev_layer (Layer) – Previous layer.
• init_scale (float) – The initial value for the scale factor.
• name (a str) – A unique layer name.

Binary Dense Layer

class tensorlayer.layers.BinaryDenseLayer(prev_layer, n_units=100, act=None, use_gemm=False, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='binary_dense')

The BinaryDenseLayer class is a binary fully connected layer, which weights are either -1 or 1 while inferencing.

Note that, the bias vector would not be binarized.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_units (int) – The number of units of this layer.
• act (activation function) – The activation function of this layer, usually set to tf.act.sign or apply SignLayer after BatchNormLayer.
• use_gemm (boolean) – If True, use gemm instead of tf.matmul for inference. (TODO).
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (a str) – A unique layer name.

Binary (De)Convolutions

BinaryConv2d

class tensorlayer.layers.BinaryConv2d(prev_layer, n_filter=32, filter_size=(3, 3), strides=(1, 1), act=None, padding='SAME', use_gemm=False, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, use_cudnn_on_gpu=None, data_format=None, name='binary_cnn2d')

The BinaryConv2d class is a 2D binary CNN layer, which weights are either -1 or 1 while inference.

Note that, the bias vector would not be binarized.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size (height, width).
• strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the shape parameter.
• act (activation function) – The activation function of this layer.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• use_gemm (boolean) – If True, use gemm instead of tf.matmul for inference. (TODO).
• W_init (initializer) – The initializer for the the weight matrix.
• b_init (initializer or None) – The initializer for the the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• use_cudnn_on_gpu (bool) – Default is False.
• data_format (str) – “NHWC” or “NCHW”, default is “NHWC”.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, [None, 256, 256, 3])
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.BinaryConv2d(net, 32, (5, 5), (1, 1), padding='SAME', name='bcnn1')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), padding='SAME', name='pool1')
>>> net = tl.layers.BatchNormLayer(net, act=tl.act.htanh, is_train=True, name='bn1')
...
>>> net = tl.layers.SignLayer(net)
>>> net = tl.layers.BinaryConv2d(net, 64, (5, 5), (1, 1), padding='SAME', name='bcnn2')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), padding='SAME', name='pool2')
>>> net = tl.layers.BatchNormLayer(net, act=tl.act.htanh, is_train=True, name='bn2')

Ternary Dense Layer

TernaryDenseLayer

class tensorlayer.layers.TernaryDenseLayer(prev_layer, n_units=100, act=None, use_gemm=False, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='ternary_dense')

The TernaryDenseLayer class is a ternary fully connected layer, which weights are either -1 or 1 or 0 while inference.

Note that, the bias vector would not be tenaried.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_units (int) – The number of units of this layer.
• act (activation function) – The activation function of this layer, usually set to tf.act.sign or apply SignLayer after BatchNormLayer.
• use_gemm (boolean) – If True, use gemm instead of tf.matmul for inference. (TODO).
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (a str) – A unique layer name.

Ternary Convolutions

TernaryConv2d

class tensorlayer.layers.TernaryConv2d(prev_layer, n_filter=32, filter_size=(3, 3), strides=(1, 1), act=None, padding='SAME', use_gemm=False, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, use_cudnn_on_gpu=None, data_format=None, name='ternary_cnn2d')

The TernaryConv2d class is a 2D binary CNN layer, which weights are either -1 or 1 or 0 while inference.

Note that, the bias vector would not be tenarized.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size (height, width).
• strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the shape parameter.
• act (activation function) – The activation function of this layer.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• use_gemm (boolean) – If True, use gemm instead of tf.matmul for inference. (TODO).
• W_init (initializer) – The initializer for the the weight matrix.
• b_init (initializer or None) – The initializer for the the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• use_cudnn_on_gpu (bool) – Default is False.
• data_format (str) – “NHWC” or “NCHW”, default is “NHWC”.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, [None, 256, 256, 3])
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.TernaryConv2d(net, 32, (5, 5), (1, 1), padding='SAME', name='bcnn1')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), padding='SAME', name='pool1')
>>> net = tl.layers.BatchNormLayer(net, act=tl.act.htanh, is_train=True, name='bn1')
...
>>> net = tl.layers.SignLayer(net)
>>> net = tl.layers.TernaryConv2d(net, 64, (5, 5), (1, 1), padding='SAME', name='bcnn2')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), padding='SAME', name='pool2')
>>> net = tl.layers.BatchNormLayer(net, act=tl.act.htanh, is_train=True, name='bn2')

DoReFa Convolutions

DorefaConv2d

class tensorlayer.layers.DorefaConv2d(prev_layer, bitW=1, bitA=3, n_filter=32, filter_size=(3, 3), strides=(1, 1), act=None, padding='SAME', use_gemm=False, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, use_cudnn_on_gpu=None, data_format=None, name='dorefa_cnn2d')

The DorefaConv2d class is a binary fully connected layer, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.

Note that, the bias vector would not be binarized.

Parameters:

• prev_layer (Layer) – Previous layer.
• bitW (int) – The bits of this layer’s parameter
• bitA (int) – The bits of the output of previous layer
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size (height, width).
• strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the shape parameter.
• act (activation function) – The activation function of this layer.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• use_gemm (boolean) – If True, use gemm instead of tf.matmul for inferencing. (TODO).
• W_init (initializer) – The initializer for the the weight matrix.
• b_init (initializer or None) – The initializer for the the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• use_cudnn_on_gpu (bool) – Default is False.
• data_format (str) – “NHWC” or “NCHW”, default is “NHWC”.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, [None, 256, 256, 3])
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.DorefaConv2d(net, 32, (5, 5), (1, 1), padding='SAME', name='bcnn1')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), padding='SAME', name='pool1')
>>> net = tl.layers.BatchNormLayer(net, act=tl.act.htanh, is_train=True, name='bn1')
...
>>> net = tl.layers.SignLayer(net)
>>> net = tl.layers.DorefaConv2d(net, 64, (5, 5), (1, 1), padding='SAME', name='bcnn2')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), padding='SAME', name='pool2')
>>> net = tl.layers.BatchNormLayer(net, act=tl.act.htanh, is_train=True, name='bn2')

DoReFa Convolutions

DorefaConv2d

class tensorlayer.layers.DorefaConv2d(prev_layer, bitW=1, bitA=3, n_filter=32, filter_size=(3, 3), strides=(1, 1), act=None, padding='SAME', use_gemm=False, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, use_cudnn_on_gpu=None, data_format=None, name='dorefa_cnn2d')

The DorefaConv2d class is a binary fully connected layer, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.

Note that, the bias vector would not be binarized.

Parameters:

• prev_layer (Layer) – Previous layer.
• bitW (int) – The bits of this layer’s parameter
• bitA (int) – The bits of the output of previous layer
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size (height, width).
• strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the shape parameter.
• act (activation function) – The activation function of this layer.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• use_gemm (boolean) – If True, use gemm instead of tf.matmul for inferencing. (TODO).
• W_init (initializer) – The initializer for the the weight matrix.
• b_init (initializer or None) – The initializer for the the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• use_cudnn_on_gpu (bool) – Default is False.
• data_format (str) – “NHWC” or “NCHW”, default is “NHWC”.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, [None, 256, 256, 3])
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.DorefaConv2d(net, 32, (5, 5), (1, 1), padding='SAME', name='bcnn1')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), padding='SAME', name='pool1')
>>> net = tl.layers.BatchNormLayer(net, act=tl.act.htanh, is_train=True, name='bn1')
...
>>> net = tl.layers.SignLayer(net)
>>> net = tl.layers.DorefaConv2d(net, 64, (5, 5), (1, 1), padding='SAME', name='bcnn2')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), padding='SAME', name='pool2')
>>> net = tl.layers.BatchNormLayer(net, act=tl.act.htanh, is_train=True, name='bn2')

Quantization Dense Layer

QuanDenseLayer

class tensorlayer.layers.QuanDenseLayer(prev_layer, n_units=100, act=None, bitW=8, bitA=8, use_gemm=False, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, name='quan_dense')

The QuanDenseLayer class is a quantized fully connected layer with BN, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_units (int) – The number of units of this layer.
• act (activation function) – The activation function of this layer.
• bitW (int) – The bits of this layer’s parameter
• bitA (int) – The bits of the output of previous layer
• use_gemm (boolean) – If True, use gemm instead of tf.matmul for inference. (TODO).
• W_init (initializer) – The initializer for the weight matrix.
• b_init (initializer or None) – The initializer for the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• name (a str) – A unique layer name.

QuanDenseLayerWithBN

class tensorlayer.layers.QuanDenseLayerWithBN(prev_layer, n_units=100, act=None, decay=0.9, epsilon=1e-05, is_train=False, bitW=8, bitA=8, gamma_init=<sphinx.ext.autodoc.importer._MockObject object>, beta_init=<sphinx.ext.autodoc.importer._MockObject object>, use_gemm=False, W_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, name='quan_dense_with_bn')

The QuanDenseLayer class is a quantized fully connected layer with BN, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_units (int) – The number of units of this layer.
• act (activation function) – The activation function of this layer.
• decay (float) – A decay factor for ExponentialMovingAverage. Suggest to use a large value for large dataset.
• epsilon (float) – Eplison.
• is_train (boolean) – Is being used for training or inference.
• beta_init (initializer or None) – The initializer for initializing beta, if None, skip beta. Usually you should not skip beta unless you know what happened.
• gamma_init (initializer or None) – The initializer for initializing gamma, if None, skip gamma.
• bitW (int) – The bits of this layer’s parameter
• bitA (int) – The bits of the output of previous layer
• decay – A decay factor for ExponentialMovingAverage. Suggest to use a large value for large dataset.
• epsilon – Eplison.
• is_train – Is being used for training or inference.
• beta_init – The initializer for initializing beta, if None, skip beta. Usually you should not skip beta unless you know what happened.
• gamma_init – The initializer for initializing gamma, if None, skip gamma.
• use_gemm (boolean) – If True, use gemm instead of tf.matmul for inferencing. (TODO).
• W_init (initializer) – The initializer for the the weight matrix.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• name (a str) – A unique layer name.

Quantization Convolutions

Quantization

class tensorlayer.layers.QuanConv2d(prev_layer, n_filter=32, filter_size=(3, 3), strides=(1, 1), padding='SAME', act=None, bitW=8, bitA=8, use_gemm=False, W_init=<sphinx.ext.autodoc.importer._MockObject object>, b_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, b_init_args=None, use_cudnn_on_gpu=None, data_format=None, name='quan_cnn2d')

The QuanConv2dWithBN class is a quantized convolutional layer with BN, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing. Note that, the bias vector would not be binarized.

Parameters:

• prev_layer (Layer) – Previous layer.
• bitW (int) – The bits of this layer’s parameter
• bitA (int) – The bits of the output of previous layer
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size (height, width).
• strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the shape parameter.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• act (activation function) – The activation function of this layer.
• bitW – The bits of this layer’s parameter
• bitA – The bits of the output of previous layer
• use_gemm (boolean) – If True, use gemm instead of tf.matmul for inferencing. (TODO).
• W_init (initializer) – The initializer for the the weight matrix.
• b_init (initializer or None) – The initializer for the the bias vector. If None, skip biases.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• b_init_args (dictionary) – The arguments for the bias vector initializer.
• use_cudnn_on_gpu (bool) – Default is False.
• data_format (str) – “NHWC” or “NCHW”, default is “NHWC”.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, [None, 256, 256, 3])
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.QuanConv2d(net, 32, (5, 5), (1, 1), padding='SAME', act=tf.nn.relu, name='qcnn1')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), padding='SAME', name='pool1')
>>> net = tl.layers.BatchNormLayer(net, act=tl.act.htanh, is_train=True, name='bn1')
...
>>> net = tl.layers.QuanConv2d(net, 64, (5, 5), (1, 1), padding='SAME', act=tf.nn.relu, name='qcnn2')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), padding='SAME', name='pool2')
>>> net = tl.layers.BatchNormLayer(net, act=tl.act.htanh, is_train=True, name='bn2')

QuanConv2dWithBN

class tensorlayer.layers.QuanConv2dWithBN(prev_layer, n_filter=32, filter_size=(3, 3), strides=(1, 1), padding='SAME', act=None, decay=0.9, epsilon=1e-05, is_train=False, gamma_init=<sphinx.ext.autodoc.importer._MockObject object>, beta_init=<sphinx.ext.autodoc.importer._MockObject object>, bitW=8, bitA=8, use_gemm=False, W_init=<sphinx.ext.autodoc.importer._MockObject object>, W_init_args=None, use_cudnn_on_gpu=None, data_format=None, name='quan_cnn2d_bn')

The QuanConv2dWithBN class is a quantized convolutional layer with BN, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.

Note that, the bias vector would keep the same.

Parameters:

• prev_layer (Layer) – Previous layer.
• n_filter (int) – The number of filters.
• filter_size (tuple of int) – The filter size (height, width).
• strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the shape parameter.
• padding (str) – The padding algorithm type: “SAME” or “VALID”.
• act (activation function) – The activation function of this layer.
• decay (float) – A decay factor for ExponentialMovingAverage. Suggest to use a large value for large dataset.
• epsilon (float) – Eplison.
• is_train (boolean) – Is being used for training or inference.
• beta_init (initializer or None) – The initializer for initializing beta, if None, skip beta. Usually you should not skip beta unless you know what happened.
• gamma_init (initializer or None) – The initializer for initializing gamma, if None, skip gamma.
• bitW (int) – The bits of this layer’s parameter
• bitA (int) – The bits of the output of previous layer
• decay – A decay factor for ExponentialMovingAverage. Suggest to use a large value for large dataset.
• epsilon – Eplison.
• is_train – Is being used for training or inference.
• beta_init – The initializer for initializing beta, if None, skip beta. Usually you should not skip beta unless you know what happened.
• gamma_init – The initializer for initializing gamma, if None, skip gamma.
• use_gemm (boolean) – If True, use gemm instead of tf.matmul for inferencing. (TODO).
• W_init (initializer) – The initializer for the the weight matrix.
• W_init_args (dictionary) – The arguments for the weight matrix initializer.
• use_cudnn_on_gpu (bool) – Default is False.
• data_format (str) – “NHWC” or “NCHW”, default is “NHWC”.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, [None, 256, 256, 3])
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.QuanConv2dWithBN(net, 64, (5, 5), (1, 1),  act=tf.nn.relu, padding='SAME', is_train=is_train, bitW=bitW, bitA=bitA, name='qcnnbn1')
>>> net = tl.layers.MaxPool2d(net, (3, 3), (2, 2), padding='SAME', name='pool1')
...
>>> net = tl.layers.QuanConv2dWithBN(net, 64, (5, 5), (1, 1), padding='SAME', act=tf.nn.relu, is_train=is_train,  bitW=bitW, bitA=bitA, name='qcnnbn2')
>>> net = tl.layers.MaxPool2d(net, (3, 3), (2, 2), padding='SAME', name='pool2')
...

Recurrent Layers

Fixed Length Recurrent layer

All recurrent layers can implement any type of RNN cell by feeding different cell function (LSTM, GRU etc).

RNN layer

class tensorlayer.layers.RNNLayer(prev_layer, cell_fn, cell_init_args=None, n_hidden=100, initializer=<sphinx.ext.autodoc.importer._MockObject object>, n_steps=5, initial_state=None, return_last=False, return_seq_2d=False, name='rnn')

The RNNLayer class is a fixed length recurrent layer for implementing vanilla RNN, LSTM, GRU and etc.

Parameters:

• prev_layer (Layer) – Previous layer.
• cell_fn (TensorFlow cell function) –

  A TensorFlow core RNN cell

  • See RNN Cells in TensorFlow
  • Note TF1.0+ and TF1.0- are different

• cell_init_args (dictionary) – The arguments for the cell function.
• n_hidden (int) – The number of hidden units in the layer.
• initializer (initializer) – The initializer for initializing the model parameters.
• n_steps (int) – The fixed sequence length.
• initial_state (None or RNN State) – If None, initial_state is zero state.
• return_last (boolean) –

  Whether return last output or all outputs in each step.

  • If True, return the last output, “Sequence input and single output”
  • If False, return all outputs, “Synced sequence input and output”
  • In other word, if you want to stack more RNNs on this layer, set to False.

• return_seq_2d (boolean) –

  Only consider this argument when return_last is False

  • If True, return 2D Tensor [n_example, n_hidden], for stacking DenseLayer after it.
  • If False, return 3D Tensor [n_example/n_steps, n_steps, n_hidden], for stacking multiple RNN after it.

• name (str) – A unique layer name.

outputs

Tensor – The output of this layer.

final_state

Tensor or StateTuple –

The finial state of this layer.

• When state_is_tuple is False, it is the final hidden and cell states, states.get_shape() = [?, 2 * n_hidden].
• When state_is_tuple is True, it stores two elements: (c, h).
• In practice, you can get the final state after each iteration during training, then feed it to the initial state of next iteration.

initial_state

Tensor or StateTuple –

The initial state of this layer.

• In practice, you can set your state at the begining of each epoch or iteration according to your training procedure.

batch_size

int or Tensor – It is an integer, if it is able to compute the batch_size; otherwise, tensor for dynamic batch size.

Examples

• For synced sequence input and output, see PTB example
• For encoding see below.

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> batch_size = 32
>>> num_steps = 5
>>> vocab_size = 3000
>>> hidden_size = 256
>>> keep_prob = 0.8
>>> is_train = True
>>> input_data = tf.placeholder(tf.int32, [batch_size, num_steps])
>>> net = tl.layers.EmbeddingInputlayer(inputs=input_data, vocabulary_size=vocab_size,
...     embedding_size=hidden_size, name='embed')
>>> net = tl.layers.DropoutLayer(net, keep=keep_prob, is_fix=True, is_train=is_train, name='drop1')
>>> net = tl.layers.RNNLayer(net, cell_fn=tf.contrib.rnn.BasicLSTMCell,
...     n_hidden=hidden_size, n_steps=num_steps, return_last=False, name='lstm1')
>>> net = tl.layers.DropoutLayer(net, keep=keep_prob, is_fix=True, is_train=is_train, name='drop2')
>>> net = tl.layers.RNNLayer(net, cell_fn=tf.contrib.rnn.BasicLSTMCell,
...     n_hidden=hidden_size, n_steps=num_steps, return_last=True, name='lstm2')
>>> net = tl.layers.DropoutLayer(net, keep=keep_prob, is_fix=True, is_train=is_train, name='drop3')
>>> net = tl.layers.DenseLayer(net, n_units=vocab_size, name='output')

• For CNN+LSTM

>>> image_size = 100
>>> batch_size = 10
>>> num_steps = 5
>>> x = tf.placeholder(tf.float32, shape=[batch_size, image_size, image_size, 1])
>>> net = tl.layers.InputLayer(x, name='in')
>>> net = tl.layers.Conv2d(net, 32, (5, 5), (2, 2), tf.nn.relu, name='cnn1')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), name='pool1')
>>> net = tl.layers.Conv2d(net, 10, (5, 5), (2, 2), tf.nn.relu, name='cnn2')
>>> net = tl.layers.MaxPool2d(net, (2, 2), (2, 2), name='pool2')
>>> net = tl.layers.FlattenLayer(net, name='flatten')
>>> net = tl.layers.ReshapeLayer(net, shape=[-1, num_steps, int(net.outputs._shape[-1])])
>>> rnn = tl.layers.RNNLayer(net, cell_fn=tf.contrib.rnn.BasicLSTMCell, n_hidden=200, n_steps=num_steps, return_last=False, return_seq_2d=True, name='rnn')
>>> net = tl.layers.DenseLayer(rnn, 3, name='out')

Notes

Input dimension should be rank 3 : [batch_size, n_steps, n_features], if no, please see ReshapeLayer.

References

• Neural Network RNN Cells in TensorFlow
• tensorflow/python/ops/rnn.py
• tensorflow/python/ops/rnn_cell.py
• see TensorFlow tutorial ptb_word_lm.py, TensorLayer tutorials tutorial_ptb_lstm*.py and tutorial_generate_text.py

Bidirectional layer

class tensorlayer.layers.BiRNNLayer(prev_layer, cell_fn, cell_init_args=None, n_hidden=100, initializer=<sphinx.ext.autodoc.importer._MockObject object>, n_steps=5, fw_initial_state=None, bw_initial_state=None, dropout=None, n_layer=1, return_last=False, return_seq_2d=False, name='birnn')

The BiRNNLayer class is a fixed length Bidirectional recurrent layer.

Parameters:

• prev_layer (Layer) – Previous layer.
• cell_fn (TensorFlow cell function) –

  A TensorFlow core RNN cell.

  • See RNN Cells in TensorFlow.
  • Note TF1.0+ and TF1.0- are different.

• cell_init_args (dictionary or None) – The arguments for the cell function.
• n_hidden (int) – The number of hidden units in the layer.
• initializer (initializer) – The initializer for initializing the model parameters.
• n_steps (int) – The fixed sequence length.
• fw_initial_state (None or forward RNN State) – If None, initial_state is zero state.
• bw_initial_state (None or backward RNN State) – If None, initial_state is zero state.
• dropout (tuple of float or int) – The input and output keep probability (input_keep_prob, output_keep_prob). If one int, input and output keep probability are the same.
• n_layer (int) – The number of RNN layers, default is 1.
• return_last (boolean) –

  Whether return last output or all outputs in each step.

  • If True, return the last output, “Sequence input and single output”
  • If False, return all outputs, “Synced sequence input and output”
  • In other word, if you want to stack more RNNs on this layer, set to False.

• return_seq_2d (boolean) –

  Only consider this argument when return_last is False

  • If True, return 2D Tensor [n_example, n_hidden], for stacking DenseLayer after it.
  • If False, return 3D Tensor [n_example/n_steps, n_steps, n_hidden], for stacking multiple RNN after it.

• name (str) – A unique layer name.

outputs

tensor – The output of this layer.

fw(bw)_final_state

tensor or StateTuple –

The finial state of this layer.

• When state_is_tuple is False, it is the final hidden and cell states, states.get_shape() = [?, 2 * n_hidden].
• When state_is_tuple is True, it stores two elements: (c, h).
• In practice, you can get the final state after each iteration during training, then feed it to the initial state of next iteration.

fw(bw)_initial_state

tensor or StateTuple –

The initial state of this layer.

• In practice, you can set your state at the begining of each epoch or iteration according to your training procedure.

batch_size

int or tensor – It is an integer, if it is able to compute the batch_size; otherwise, tensor for dynamic batch size.

Notes

Input dimension should be rank 3 : [batch_size, n_steps, n_features]. If not, please see ReshapeLayer. For predicting, the sequence length has to be the same with the sequence length of training, while, for normal RNN, we can use sequence length of 1 for predicting.

References

Source

Recurrent Convolution

Conv RNN Cell

class tensorlayer.layers.ConvRNNCell

Abstract object representing an Convolutional RNN Cell.

Basic Conv LSTM Cell

class tensorlayer.layers.BasicConvLSTMCell(shape, filter_size, num_features, forget_bias=1.0, input_size=None, state_is_tuple=False, act=<sphinx.ext.autodoc.importer._MockObject object>)

Basic Conv LSTM recurrent network cell.

Parameters:

• shape (tuple of int) – The height and width of the cell.
• filter_size (tuple of int) – The height and width of the filter
• num_features (int) – The hidden size of the cell
• forget_bias (float) – The bias added to forget gates (see above).
• input_size (int) – Deprecated and unused.
• state_is_tuple (boolen) – If True, accepted and returned states are 2-tuples of the c_state and m_state. If False, they are concatenated along the column axis. The latter behavior will soon be deprecated.
• act (activation function) – The activation function of this layer, tanh as default.

Conv LSTM layer

class tensorlayer.layers.ConvLSTMLayer(prev_layer, cell_shape=None, feature_map=1, filter_size=(3, 3), cell_fn=<class 'tensorlayer.layers.recurrent.BasicConvLSTMCell'>, initializer=<sphinx.ext.autodoc.importer._MockObject object>, n_steps=5, initial_state=None, return_last=False, return_seq_2d=False, name='convlstm')

A fixed length Convolutional LSTM layer.

See this paper .

Parameters:

• prev_layer (Layer) – Previous layer
• cell_shape (tuple of int) – The shape of each cell width * height
• filter_size (tuple of int) – The size of filter width * height
• cell_fn (a convolutional RNN cell) – Cell function like BasicConvLSTMCell
• feature_map (int) – The number of feature map in the layer.
• initializer (initializer) – The initializer for initializing the parameters.
• n_steps (int) – The sequence length.
• initial_state (None or ConvLSTM State) – If None, initial_state is zero state.
• return_last (boolean) –

  Whether return last output or all outputs in each step.

  • If True, return the last output, “Sequence input and single output”.
  • If False, return all outputs, “Synced sequence input and output”.
  • In other word, if you want to stack more RNNs on this layer, set to False.

• return_seq_2d (boolean) –

  Only consider this argument when return_last is False

  • If True, return 2D Tensor [n_example, n_hidden], for stacking DenseLayer after it.
  • If False, return 3D Tensor [n_example/n_steps, n_steps, n_hidden], for stacking multiple RNN after it.

• name (str) – A unique layer name.

outputs

tensor – The output of this RNN. return_last = False, outputs = all cell_output, which is the hidden state. cell_output.get_shape() = (?, h, w, c])

final_state

tensor or StateTuple –

The finial state of this layer.

• When state_is_tuple = False, it is the final hidden and cell states,
• When state_is_tuple = True, You can get the final state after each iteration during training, then feed it to the initial state of next iteration.

initial_state

tensor or StateTuple – It is the initial state of this ConvLSTM layer, you can use it to initialize your state at the beginning of each epoch or iteration according to your training procedure.

batch_size

int or tensor – Is int, if able to compute the batch_size, otherwise, tensor for ?.

Advanced Ops for Dynamic RNN

These operations usually be used inside Dynamic RNN layer, they can compute the sequence lengths for different situation and get the last RNN outputs by indexing.

Output indexing

tensorlayer.layers.advanced_indexing_op(inputs, index)

Advanced Indexing for Sequences, returns the outputs by given sequence lengths. When return the last output DynamicRNNLayer uses it to get the last outputs with the sequence lengths.

Parameters:

• inputs (tensor for data) – With shape of [batch_size, n_step(max), n_features]
• index (tensor for indexing) – Sequence length in Dynamic RNN. [batch_size]

Examples

>>> import numpy as np
>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> batch_size, max_length, n_features = 3, 5, 2
>>> z = np.random.uniform(low=-1, high=1, size=[batch_size, max_length, n_features]).astype(np.float32)
>>> b_z = tf.constant(z)
>>> sl = tf.placeholder(dtype=tf.int32, shape=[batch_size])
>>> o = advanced_indexing_op(b_z, sl)
>>>
>>> sess = tf.InteractiveSession()
>>> tl.layers.initialize_global_variables(sess)
>>>
>>> order = np.asarray([1,1,2])
>>> print("real",z[0][order[0]-1], z[1][order[1]-1], z[2][order[2]-1])
>>> y = sess.run([o], feed_dict={sl:order})
>>> print("given",order)
>>> print("out", y)
real [-0.93021595  0.53820813] [-0.92548317 -0.77135968] [ 0.89952248  0.19149846]
given [1 1 2]
out [array([[-0.93021595,  0.53820813],
            [-0.92548317, -0.77135968],
            [ 0.89952248,  0.19149846]], dtype=float32)]

References

• Modified from TFlearn (the original code is used for fixed length rnn), references.

Compute Sequence length 1

tensorlayer.layers.retrieve_seq_length_op(data)

An op to compute the length of a sequence from input shape of [batch_size, n_step(max), n_features], it can be used when the features of padding (on right hand side) are all zeros.

Parameters:data (tensor) – [batch_size, n_step(max), n_features] with zero padding on right hand side.

Examples

>>> data = [[[1],[2],[0],[0],[0]],
...         [[1],[2],[3],[0],[0]],
...         [[1],[2],[6],[1],[0]]]
>>> data = np.asarray(data)
>>> print(data.shape)
(3, 5, 1)
>>> data = tf.constant(data)
>>> sl = retrieve_seq_length_op(data)
>>> sess = tf.InteractiveSession()
>>> tl.layers.initialize_global_variables(sess)
>>> y = sl.eval()
[2 3 4]

Multiple features >>> data = [[[1,2],[2,2],[1,2],[1,2],[0,0]], … [[2,3],[2,4],[3,2],[0,0],[0,0]], … [[3,3],[2,2],[5,3],[1,2],[0,0]]] >>> print(sl) [4 3 4]

References

Borrow from TFlearn.

Compute Sequence length 2

tensorlayer.layers.retrieve_seq_length_op2(data)

An op to compute the length of a sequence, from input shape of [batch_size, n_step(max)], it can be used when the features of padding (on right hand side) are all zeros.

Parameters:data (tensor) – [batch_size, n_step(max)] with zero padding on right hand side.

Examples

>>> data = [[1,2,0,0,0],
...         [1,2,3,0,0],
...         [1,2,6,1,0]]
>>> o = retrieve_seq_length_op2(data)
>>> sess = tf.InteractiveSession()
>>> tl.layers.initialize_global_variables(sess)
>>> print(o.eval())
[2 3 4]

Compute Sequence length 3

tensorlayer.layers.retrieve_seq_length_op3(data, pad_val=0)

Return tensor for sequence length, if input is tf.string.

Get Mask

tensorlayer.layers.target_mask_op(data, pad_val=0)

Return tensor for mask, if input is tf.string.

Dynamic RNN Layer

RNN Layer

class tensorlayer.layers.DynamicRNNLayer(prev_layer, cell_fn, cell_init_args=None, n_hidden=256, initializer=<sphinx.ext.autodoc.importer._MockObject object>, sequence_length=None, initial_state=None, dropout=None, n_layer=1, return_last=None, return_seq_2d=False, dynamic_rnn_init_args=None, name='dyrnn')

The DynamicRNNLayer class is a dynamic recurrent layer, see tf.nn.dynamic_rnn.

Parameters:

• prev_layer (Layer) – Previous layer
• cell_fn (TensorFlow cell function) –

  A TensorFlow core RNN cell

  • See RNN Cells in TensorFlow
  • Note TF1.0+ and TF1.0- are different

• cell_init_args (dictionary or None) – The arguments for the cell function.
• n_hidden (int) – The number of hidden units in the layer.
• initializer (initializer) – The initializer for initializing the parameters.
• sequence_length (tensor, array or None) –

  The sequence length of each row of input data, see Advanced Ops for Dynamic RNN.

  • If None, it uses retrieve_seq_length_op to compute the sequence length, i.e. when the features of padding (on right hand side) are all zeros.
  • If using word embedding, you may need to compute the sequence length from the ID array (the integer features before word embedding) by using retrieve_seq_length_op2 or retrieve_seq_length_op.
  • You can also input an numpy array.
  • More details about TensorFlow dynamic RNN in Wild-ML Blog.

• initial_state (None or RNN State) – If None, initial_state is zero state.
• dropout (tuple of float or int) –

  The input and output keep probability (input_keep_prob, output_keep_prob).

  • If one int, input and output keep probability are the same.

• n_layer (int) – The number of RNN layers, default is 1.
• return_last (boolean or None) –

  Whether return last output or all outputs in each step.

  • If True, return the last output, “Sequence input and single output”
  • If False, return all outputs, “Synced sequence input and output”
  • In other word, if you want to stack more RNNs on this layer, set to False.

• return_seq_2d (boolean) –

  Only consider this argument when return_last is False

  • If True, return 2D Tensor [n_example, n_hidden], for stacking DenseLayer after it.
  • If False, return 3D Tensor [n_example/n_steps, n_steps, n_hidden], for stacking multiple RNN after it.

• dynamic_rnn_init_args (dictionary) – The arguments for tf.nn.dynamic_rnn.
• name (str) – A unique layer name.

outputs

tensor – The output of this layer.

final_state

tensor or StateTuple –

The finial state of this layer.

• When state_is_tuple is False, it is the final hidden and cell states, states.get_shape() = [?, 2 * n_hidden].
• When state_is_tuple is True, it stores two elements: (c, h).
• In practice, you can get the final state after each iteration during training, then feed it to the initial state of next iteration.

initial_state

tensor or StateTuple –

The initial state of this layer.

• In practice, you can set your state at the begining of each epoch or iteration according to your training procedure.

batch_size

int or tensor – It is an integer, if it is able to compute the batch_size; otherwise, tensor for dynamic batch size.

sequence_length

a tensor or array – The sequence lengths computed by Advanced Opt or the given sequence lengths, [batch_size]

Notes

Input dimension should be rank 3 : [batch_size, n_steps(max), n_features], if no, please see ReshapeLayer.

Examples

Synced sequence input and output, for loss function see tl.cost.cross_entropy_seq_with_mask.

>>> input_seqs = tf.placeholder(dtype=tf.int64, shape=[batch_size, None], name="input")
>>> net = tl.layers.EmbeddingInputlayer(
...             inputs=input_seqs,
...             vocabulary_size=vocab_size,
...             embedding_size=embedding_size,
...             name='embedding')
>>> net = tl.layers.DynamicRNNLayer(net,
...             cell_fn=tf.contrib.rnn.BasicLSTMCell, # for TF0.2 use tf.nn.rnn_cell.BasicLSTMCell,
...             n_hidden=embedding_size,
...             dropout=(0.7 if is_train else None),
...             sequence_length=tl.layers.retrieve_seq_length_op2(input_seqs),
...             return_last=False,                    # for encoder, set to True
...             return_seq_2d=True,                   # stack denselayer or compute cost after it
...             name='dynamicrnn')
>>> net = tl.layers.DenseLayer(net, n_units=vocab_size, name="output")

References

• Wild-ML Blog
• dynamic_rnn.ipynb
• tf.nn.dynamic_rnn
• tflearn rnn
• tutorial_dynamic_rnn.py

Bidirectional Layer

class tensorlayer.layers.BiDynamicRNNLayer(prev_layer, cell_fn, cell_init_args=None, n_hidden=256, initializer=<sphinx.ext.autodoc.importer._MockObject object>, sequence_length=None, fw_initial_state=None, bw_initial_state=None, dropout=None, n_layer=1, return_last=False, return_seq_2d=False, dynamic_rnn_init_args=None, name='bi_dyrnn_layer')

The BiDynamicRNNLayer class is a RNN layer, you can implement vanilla RNN, LSTM and GRU with it.

Parameters:

• prev_layer (Layer) – Previous layer.
• cell_fn (TensorFlow cell function) –

  A TensorFlow core RNN cell

  • See RNN Cells in TensorFlow.
  • Note TF1.0+ and TF1.0- are different.

• cell_init_args (dictionary) – The arguments for the cell initializer.
• n_hidden (int) – The number of hidden units in the layer.
• initializer (initializer) – The initializer for initializing the parameters.
• sequence_length (tensor, array or None) –

  The sequence length of each row of input data, see Advanced Ops for Dynamic RNN.

  • If None, it uses retrieve_seq_length_op to compute the sequence length, i.e. when the features of padding (on right hand side) are all zeros.
  • If using word embedding, you may need to compute the sequence length from the ID array (the integer features before word embedding) by using retrieve_seq_length_op2 or retrieve_seq_length_op.
  • You can also input an numpy array.
  • More details about TensorFlow dynamic RNN in Wild-ML Blog.

• fw_initial_state (None or forward RNN State) – If None, initial_state is zero state.
• bw_initial_state (None or backward RNN State) – If None, initial_state is zero state.
• dropout (tuple of float or int) –

  The input and output keep probability (input_keep_prob, output_keep_prob).

  • If one int, input and output keep probability are the same.

• n_layer (int) – The number of RNN layers, default is 1.
• return_last (boolean) –

  Whether return last output or all outputs in each step.

  • If True, return the last output, “Sequence input and single output”
  • If False, return all outputs, “Synced sequence input and output”
  • In other word, if you want to stack more RNNs on this layer, set to False.

• return_seq_2d (boolean) –

  Only consider this argument when return_last is False

  • If True, return 2D Tensor [n_example, 2 * n_hidden], for stacking DenseLayer after it.
  • If False, return 3D Tensor [n_example/n_steps, n_steps, 2 * n_hidden], for stacking multiple RNN after it.

• dynamic_rnn_init_args (dictionary) – The arguments for tf.nn.bidirectional_dynamic_rnn.
• name (str) – A unique layer name.

outputs

tensor – The output of this layer. (?, 2 * n_hidden)

fw(bw)_final_state

tensor or StateTuple –

The finial state of this layer.

• When state_is_tuple is False, it is the final hidden and cell states, states.get_shape() = [?, 2 * n_hidden].
• When state_is_tuple is True, it stores two elements: (c, h).
• In practice, you can get the final state after each iteration during training, then feed it to the initial state of next iteration.

fw(bw)_initial_state

tensor or StateTuple –

The initial state of this layer.

• In practice, you can set your state at the begining of each epoch or iteration according to your training procedure.

batch_size

int or tensor – It is an integer, if it is able to compute the batch_size; otherwise, tensor for dynamic batch size.

sequence_length

a tensor or array – The sequence lengths computed by Advanced Opt or the given sequence lengths, [batch_size].

Notes

Input dimension should be rank 3 : [batch_size, n_steps(max), n_features], if no, please see ReshapeLayer.

References

• Wild-ML Blog
• bidirectional_rnn.ipynb

Sequence to Sequence

Simple Seq2Seq

class tensorlayer.layers.Seq2Seq(net_encode_in, net_decode_in, cell_fn, cell_init_args=None, n_hidden=256, initializer=<sphinx.ext.autodoc.importer._MockObject object>, encode_sequence_length=None, decode_sequence_length=None, initial_state_encode=None, initial_state_decode=None, dropout=None, n_layer=1, return_seq_2d=False, name='seq2seq')

The Seq2Seq class is a simple DynamicRNNLayer based Seq2seq layer without using tl.contrib.seq2seq. See Model and Sequence to Sequence Learning with Neural Networks.

• Please check this example Chatbot in 200 lines of code.
• The Author recommends users to read the source code of DynamicRNNLayer and Seq2Seq.

Parameters:

• net_encode_in (Layer) – Encode sequences, [batch_size, None, n_features].
• net_decode_in (Layer) – Decode sequences, [batch_size, None, n_features].
• cell_fn (TensorFlow cell function) –

  A TensorFlow core RNN cell

  • see RNN Cells in TensorFlow
  • Note TF1.0+ and TF1.0- are different

• cell_init_args (dictionary or None) – The arguments for the cell initializer.
• n_hidden (int) – The number of hidden units in the layer.
• initializer (initializer) – The initializer for the parameters.
• encode_sequence_length (tensor) – For encoder sequence length, see DynamicRNNLayer .
• decode_sequence_length (tensor) – For decoder sequence length, see DynamicRNNLayer .
• initial_state_encode (None or RNN state) – If None, initial_state_encode is zero state, it can be set by placeholder or other RNN.
• initial_state_decode (None or RNN state) – If None, initial_state_decode is the final state of the RNN encoder, it can be set by placeholder or other RNN.
• dropout (tuple of float or int) –

  The input and output keep probability (input_keep_prob, output_keep_prob).

  • If one int, input and output keep probability are the same.

• n_layer (int) – The number of RNN layers, default is 1.
• return_seq_2d (boolean) –

  Only consider this argument when return_last is False

  • If True, return 2D Tensor [n_example, 2 * n_hidden], for stacking DenseLayer after it.
  • If False, return 3D Tensor [n_example/n_steps, n_steps, 2 * n_hidden], for stacking multiple RNN after it.

• name (str) – A unique layer name.

outputs

tensor – The output of RNN decoder.

initial_state_encode

tensor or StateTuple – Initial state of RNN encoder.

initial_state_decode

tensor or StateTuple – Initial state of RNN decoder.

final_state_encode

tensor or StateTuple – Final state of RNN encoder.

final_state_decode

tensor or StateTuple – Final state of RNN decoder.

Notes

• How to feed data: Sequence to Sequence Learning with Neural Networks
• input_seqs : ['how', 'are', 'you', '<PAD_ID>']
• decode_seqs : ['<START_ID>', 'I', 'am', 'fine', '<PAD_ID>']
• target_seqs : ['I', 'am', 'fine', '<END_ID>', '<PAD_ID>']
• target_mask : [1, 1, 1, 1, 0]
• related functions : tl.prepro <pad_sequences, precess_sequences, sequences_add_start_id, sequences_get_mask>

Examples

>>> from tensorlayer.layers import *
>>> batch_size = 32
>>> encode_seqs = tf.placeholder(dtype=tf.int64, shape=[batch_size, None], name="encode_seqs")
>>> decode_seqs = tf.placeholder(dtype=tf.int64, shape=[batch_size, None], name="decode_seqs")
>>> target_seqs = tf.placeholder(dtype=tf.int64, shape=[batch_size, None], name="target_seqs")
>>> target_mask = tf.placeholder(dtype=tf.int64, shape=[batch_size, None], name="target_mask") # tl.prepro.sequences_get_mask()
>>> with tf.variable_scope("model"):
>>>     # for chatbot, you can use the same embedding layer,
>>>     # for translation, you may want to use 2 seperated embedding layers
>>>     with tf.variable_scope("embedding") as vs:
>>>         net_encode = EmbeddingInputlayer(
...                 inputs = encode_seqs,
...                 vocabulary_size = 10000,
...                 embedding_size = 200,
...                 name = 'seq_embedding')
>>>         vs.reuse_variables()
>>>         tl.layers.set_name_reuse(True)
>>>         net_decode = EmbeddingInputlayer(
...                 inputs = decode_seqs,
...                 vocabulary_size = 10000,
...                 embedding_size = 200,
...                 name = 'seq_embedding')
>>>     net = Seq2Seq(net_encode, net_decode,
...             cell_fn = tf.contrib.rnn.BasicLSTMCell,
...             n_hidden = 200,
...             initializer = tf.random_uniform_initializer(-0.1, 0.1),
...             encode_sequence_length = retrieve_seq_length_op2(encode_seqs),
...             decode_sequence_length = retrieve_seq_length_op2(decode_seqs),
...             initial_state_encode = None,
...             dropout = None,
...             n_layer = 1,
...             return_seq_2d = True,
...             name = 'seq2seq')
>>> net_out = DenseLayer(net, n_units=10000, act=None, name='output')
>>> e_loss = tl.cost.cross_entropy_seq_with_mask(logits=net_out.outputs, target_seqs=target_seqs, input_mask=target_mask, return_details=False, name='cost')
>>> y = tf.nn.softmax(net_out.outputs)
>>> net_out.print_params(False)

Shape Layers

Flatten Layer

class tensorlayer.layers.FlattenLayer(prev_layer, name='flatten')

A layer that reshapes high-dimension input into a vector.

Then we often apply DenseLayer, RNNLayer, ConcatLayer and etc on the top of a flatten layer. [batch_size, mask_row, mask_col, n_mask] —> [batch_size, mask_row * mask_col * n_mask]

Parameters:

• prev_layer (Layer) – Previous layer.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, shape=[None, 28, 28, 1])
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.FlattenLayer(net, name='flatten')
[?, 784]

Reshape Layer

class tensorlayer.layers.ReshapeLayer(prev_layer, shape, name='reshape')

A layer that reshapes a given tensor.

Parameters:

• prev_layer (Layer) – Previous layer
• shape (tuple of int) – The output shape, see tf.reshape.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, shape=(None, 784))
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.ReshapeLayer(net, [-1, 28, 28, 1], name='reshape')
>>> print(net.outputs)
(?, 28, 28, 1)

Transpose Layer

class tensorlayer.layers.TransposeLayer(prev_layer, perm, name='transpose')

A layer that transposes the dimension of a tensor.

See tf.transpose() .

Parameters:

• prev_layer (Layer) – Previous layer
• perm (list of int) – The permutation of the dimensions, similar with numpy.transpose.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, shape=[None, 28, 28, 1])
>>> net = tl.layers.InputLayer(x, name='input')
>>> net = tl.layers.TransposeLayer(net, perm=[0, 1, 3, 2], name='trans')
[None, 28, 1, 28]

Spatial Transformer

2D Affine Transformation

class tensorlayer.layers.SpatialTransformer2dAffineLayer(prev_layer, theta_layer, out_size=None, name='spatial_trans_2d_affine')

The SpatialTransformer2dAffineLayer class is a 2D Spatial Transformer Layer for 2D Affine Transformation.

Parameters:

• prev_layer (Layer) – Previous layer.
• theta_layer (Layer) – The localisation network. - We will use a DenseLayer to make the theta size to [batch, 6], value range to [0, 1] (via tanh).
• out_size (tuple of int or None) – The size of the output of the network (height, width), the feature maps will be resized by this.
• name (str) – A unique layer name.

References

• Spatial Transformer Networks
• TensorFlow/Models

2D Affine Transformation function

tensorlayer.layers.transformer(U, theta, out_size, name='SpatialTransformer2dAffine')

Spatial Transformer Layer for 2D Affine Transformation , see SpatialTransformer2dAffineLayer class.

Parameters:

• U (list of float) – The output of a convolutional net should have the shape [num_batch, height, width, num_channels].
• theta (float) – The output of the localisation network should be [num_batch, 6], value range should be [0, 1] (via tanh).
• out_size (tuple of int) – The size of the output of the network (height, width)
• name (str) – Optional function name

Returns:

The transformed tensor.

Return type:

Tensor

References

• Spatial Transformer Networks
• TensorFlow/Models

Notes

To initialize the network to the identity transform init.

>>> import tensorflow as tf
>>> # ``theta`` to
>>> identity = np.array([[1., 0., 0.], [0., 1., 0.]])
>>> identity = identity.flatten()
>>> theta = tf.Variable(initial_value=identity)

Batch 2D Affine Transformation function

tensorlayer.layers.batch_transformer(U, thetas, out_size, name='BatchSpatialTransformer2dAffine')

Batch Spatial Transformer function for 2D Affine Transformation.

Parameters:

• U (list of float) – tensor of inputs [batch, height, width, num_channels]
• thetas (list of float) – a set of transformations for each input [batch, num_transforms, 6]
• out_size (list of int) – the size of the output [out_height, out_width]
• name (str) – optional function name

Returns:

Tensor of size [batch * num_transforms, out_height, out_width, num_channels]

Return type:

float

Stack Layer

Stack Layer

class tensorlayer.layers.StackLayer(layers, axis=1, name='stack')

The StackLayer class is a layer for stacking a list of rank-R tensors into one rank-(R+1) tensor, see tf.stack().

Parameters:

• layers (list of Layer) – Previous layers to stack.
• axis (int) – Dimension along which to concatenate.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, shape=[None, 30])
>>> net = tl.layers.InputLayer(x, name='input')
>>> net1 = tl.layers.DenseLayer(net, 10, name='dense1')
>>> net2 = tl.layers.DenseLayer(net, 10, name='dense2')
>>> net3 = tl.layers.DenseLayer(net, 10, name='dense3')
>>> net = tl.layers.StackLayer([net1, net2, net3], axis=1, name='stack')
(?, 3, 10)

Unstack Layer

class tensorlayer.layers.UnStackLayer(prev_layer, num=None, axis=0, name='unstack')

The UnStackLayer class is a layer for unstacking the given dimension of a rank-R tensor into rank-(R-1) tensors., see tf.unstack().

Parameters:

• prev_layer (Layer) – Previous layer
• num (int or None) – The length of the dimension axis. Automatically inferred if None (the default).
• axis (int) – Dimension along which axis to concatenate.
• name (str) – A unique layer name.

Returns:

The list of layer objects unstacked from the input.

Return type:

list of Layer

Time Distributed Layer

class tensorlayer.layers.TimeDistributedLayer(prev_layer, layer_class=None, layer_args=None, name='time_distributed')

The TimeDistributedLayer class that applies a function to every timestep of the input tensor. For example, if use DenseLayer as the layer_class, we input (batch_size, length, dim) and output (batch_size , length, new_dim).

Parameters:

• prev_layer (Layer) – Previous layer with output size of (batch_size, length, dim).
• layer_class (a Layer class) – The layer class name.
• args (dictionary) – The arguments for the layer_class.
• name (str) – A unique layer name.

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> batch_size = 32
>>> timestep = 20
>>> input_dim = 100
>>> x = tf.placeholder(dtype=tf.float32, shape=[batch_size, timestep, input_dim], name="encode_seqs")
>>> net = tl.layers.InputLayer(x, name='input')
[TL] InputLayer  input: (32, 20, 100)
>>> net = tl.layers.TimeDistributedLayer(net, layer_class=tl.layers.DenseLayer, args={'n_units':50, 'name':'dense'}, name='time_dense')
[TL] TimeDistributedLayer time_dense: layer_class:DenseLayer
>>> print(net.outputs._shape)
(32, 20, 50)
>>> net.print_params(False)
[TL] param   0: (100, 50)          time_dense/dense/W:0
[TL] param   1: (50,)              time_dense/dense/b:0
[TL]    num of params: 5050

Helper Functions

Flatten tensor

tensorlayer.layers.flatten_reshape(variable, name='flatten')

Reshapes a high-dimension vector input.

[batch_size, mask_row, mask_col, n_mask] —> [batch_size, mask_row x mask_col x n_mask]

Parameters:

• variable (TensorFlow variable or tensor) – The variable or tensor to be flatten.
• name (str) – A unique layer name.

Returns:

Flatten Tensor

Return type:

Tensor

Examples

>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> x = tf.placeholder(tf.float32, [None, 128, 128, 3])
>>> # Convolution Layer with 32 filters and a kernel size of 5
>>> network = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu)
>>> # Max Pooling (down-sampling) with strides of 2 and kernel size of 2
>>> network = tf.layers.max_pooling2d(network, 2, 2)
>>> print(network.get_shape()[:].as_list())
>>> [None, 62, 62, 32]
>>> network = tl.layers.flatten_reshape(network)
>>> print(network.get_shape()[:].as_list()[1:])
>>> [None, 123008]

Permanent clear existing layer names

tensorlayer.layers.clear_layers_name()

DEPRECATED FUNCTION

Warning

THIS FUNCTION IS DEPRECATED: It will be removed after after 2018-06-30. Instructions for updating: TensorLayer relies on TensorFlow to check naming.

Initialize RNN state

tensorlayer.layers.initialize_rnn_state(state, feed_dict=None)

Returns the initialized RNN state. The inputs are LSTMStateTuple or State of RNNCells, and an optional feed_dict.

Parameters:

• state (RNN state.) – The TensorFlow’s RNN state.
• feed_dict (dictionary) – Initial RNN state; if None, returns zero state.

Returns:

The TensorFlow’s RNN state.

Return type:

RNN state

Remove repeated items in a list

tensorlayer.layers.list_remove_repeat(x)

Remove the repeated items in a list, and return the processed list. You may need it to create merged layer like Concat, Elementwise and etc.

Parameters:x (list) – Input Returns:A list that after removing it’s repeated items Return type:list

Examples

>>> l = [2, 3, 4, 2, 3]
>>> l = list_remove_repeat(l)
[2, 3, 4]

Merge networks attributes

tensorlayer.layers.merge_networks(layers=None)

Merge all parameters, layers and dropout probabilities to a Layer. The output of return network is the first network in the list.

Parameters:layers (list of Layer) – Merge all parameters, layers and dropout probabilities to the first layer in the list. Returns:The network after merging all parameters, layers and dropout probabilities to the first network in the list. Return type:Layer

Examples

>>> import tensorlayer as tl
>>> n1 = ...
>>> n2 = ...
>>> n1 = tl.layers.merge_networks([n1, n2])