Skip to content

Commit fe9ebee

Browse files
zsdonghaozsdonghao
committed
layer, activation rtd
1 parent 132763a commit fe9ebee

18 files changed

+246
-32
lines changed

‎docs/modules/activation.rst‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,23 @@ API - Activations
44
To make TensorLayer simple, we minimize the number of activation functions as much as
55
we can. So we encourage you to use TensorFlow's function. TensorFlow provides
66
``tf.nn.relu``, ``tf.nn.relu6``, ``tf.nn.elu``, ``tf.nn.softplus``,
7-
``tf.nn.softsign`` and so on, see `TensorFlow API <https://www.tensorflow.org/versions/master/api_docs/index.html>`_.
7+
``tf.nn.softsign`` and so on. More TensorFlow official activation functions can be found
8+
`here <https://www.tensorflow.org/versions/master/api_docs/python/nn.html#activation-functions>`_.
9+
10+
11+
Creating custom layers
12+
------------------------
13+
14+
To implement a custom activation function in TensorLayer is very easy.
15+
16+
The following is an example implementation of an activation that multiplies its input by 2.
17+
For more complex activation, TensorFlow API will be required.
18+
19+
.. code-block:: python
20+
21+
def double_activation(x):
22+
return x * 2
23+
824
925
1026
.. automodule:: tensorlayer.activation
@@ -14,8 +30,6 @@ we can. So we encourage you to use TensorFlow's function. TensorFlow provides
1430
identity
1531
ramp
1632

17-
More TensorFlow official activation functions can be found
18-
`here <https://www.tensorflow.org/versions/master/api_docs/python/nn.html#activation-functions>`_.
1933

2034
Activation functions
2135
---------------------

‎docs/modules/files.rst‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,26 @@ API - Load, Save Model and Data
22
===================================
33

44
Load benchmark dataset, save and restore model, save and load variables.
5+
TensorFlow provides ``.ckpt`` file format to save and restore the models, while
6+
we suggest to use standard python file format ``.npz`` to save models for the
7+
sake of cross-platform.
8+
9+
10+
.. code-block:: python
11+
12+
# save model as .ckpt
13+
saver = tf.train.Saver()
14+
save_path = saver.save(sess, "model.ckpt")
15+
# restore model from .ckpt
16+
saver = tf.train.Saver()
17+
saver.restore(sess, "model.ckpt")
18+
19+
# save model as .npz
20+
tl.files.save_npz(network.all_params , name='model.npz')
21+
# restore model from .npz
22+
load_params = tl.files.load_npz(path='', name='model.npz')
23+
tl.files.assign_params(sess, load_params, network)
24+
525
626
727
.. automodule:: tensorlayer.files

‎docs/modules/layers.rst‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,191 @@ For example, we do not provide layer for local response normalization, we sugges
77
you to apply ``tf.nn.lrn`` on ``Layer.outputs``.
88
More functions can be found in `TensorFlow API <https://www.tensorflow.org/versions/master/api_docs/index.html>`_
99

10+
11+
Understand layer
12+
-----------------
13+
14+
All TensorLayer layers have a number of properties in common:
15+
16+
- ``layer.outputs`` : Tensor, the outputs of current layer.
17+
- ``layer.all_params`` : a list of Tensor, all network variables in order.
18+
- ``layer.all_layers`` : a list of Tensor, all network outputs in order.
19+
- ``layer.all_drop`` : a dictionary of {placeholder : float}, all keeping probabilities of noise layer.
20+
21+
All TensorLayer layers have a number of methods in common:
22+
23+
- ``layer.print_params()`` : print the network variables information in order (after ``sess.run(tf.initialize_all_variables())``). alternatively, print all variables by ``tl.layers.print_all_variables()``.
24+
- ``layer.print_layers()`` : print the network layers information in order.
25+
- ``layer.count_params()`` : print the number of parameters in the network.
26+
27+
28+
29+
The initialization of a network is done by input layer, then we can stacked layers
30+
as follow, then a network is a ``Layer`` class.
31+
The most important properties of a network are ``network.all_params``, ``network.all_layers`` and ``network.all_drop``.
32+
The ``all_params`` is a list which store all pointers of all network parameters in order,
33+
the following script define a 3 layer network, then ``all_params = [W1, b1, W2, b2, W_out, b_out]``.
34+
The ``all_layers`` is a list which store all pointers of the outputs of all layers,
35+
in the following network, ``all_layers = [dropout(?, 784), relu(?, 800), dropout(?, 800), relu(?, 800), dropout(?, 800)], identity(?, 10)]``
36+
where ``?`` reflects any batch size. You can print the layer information and parameters information by
37+
using ``network.print_layers()`` and ``network.print_params()``.
38+
To count the number of parameters in a network, run ``network.count_params()``.
39+
40+
41+
42+
.. code-block:: python
43+
44+
sess = tf.InteractiveSession()
45+
46+
x = tf.placeholder(tf.float32, shape=[None, 784], name='x')
47+
y_ = tf.placeholder(tf.int64, shape=[None, ], name='y_')
48+
49+
network = tl.layers.InputLayer(x, name='input_layer')
50+
network = tl.layers.DropoutLayer(network, keep=0.8, name='drop1')
51+
network = tl.layers.DenseLayer(network, n_units=800,
52+
act = tf.nn.relu, name='relu1')
53+
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop2')
54+
network = tl.layers.DenseLayer(network, n_units=800,
55+
act = tf.nn.relu, name='relu2')
56+
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop3')
57+
network = tl.layers.DenseLayer(network, n_units=10,
58+
act = tl.activation.identity,
59+
name='output_layer')
60+
61+
y = network.outputs
62+
y_op = tf.argmax(tf.nn.softmax(y), 1)
63+
64+
cost = tl.cost.cross_entropy(y, y_)
65+
66+
train_params = network.all_params
67+
68+
train_op = tf.train.AdamOptimizer(learning_rate, beta1=0.9, beta2=0.999,
69+
epsilon=1e-08, use_locking=False).minimize(cost, var_list = train_params)
70+
71+
sess.run(tf.initialize_all_variables())
72+
73+
network.print_params()
74+
network.print_layers()
75+
76+
In addition, ``network.all_drop`` is a dictionary which stores the keeping probabilities of all
77+
noise layer. In the above network, they are the keeping probabilities of dropout layers.
78+
79+
So for training, enable all dropout layers as follow.
80+
81+
.. code-block:: python
82+
83+
feed_dict = {x: X_train_a, y_: y_train_a}
84+
feed_dict.update( network.all_drop )
85+
loss, _ = sess.run([cost, train_op], feed_dict=feed_dict)
86+
feed_dict.update( network.all_drop )
87+
88+
For evaluating and testing, disable all dropout layers as follow.
89+
90+
.. code-block:: python
91+
92+
feed_dict = {x: X_val, y_: y_val}
93+
feed_dict.update(dp_dict)
94+
print(" val loss: %f" % sess.run(cost, feed_dict=feed_dict))
95+
print(" val acc: %f" % np.mean(y_val ==
96+
sess.run(y_op, feed_dict=feed_dict)))
97+
98+
For more details, please read the MNIST examples.
99+
100+
Creating custom layers
101+
------------------------
102+
103+
Understand Dense layer
104+
^^^^^^^^^^^^^^^^^^^^^^^^^
105+
106+
Before creating your own TensorLayer layer, let's have a look at Dense layer.
107+
It creates a weights matrix and biases vector if not exists, then implement
108+
the output expression.
109+
At the end, as a layer with parameter, we also need to append the parameters into ``all_params``.
110+
111+
112+
.. code-block:: python
113+
114+
class DenseLayer(Layer):
115+
"""
116+
The :class:`DenseLayer` class is a fully connected layer.
117+
118+
Parameters
119+
----------
120+
layer : a :class:`Layer` instance
121+
The `Layer` class feeding into this layer.
122+
n_units : int
123+
The number of units of the layer.
124+
act : activation function
125+
The function that is applied to the layer activations.
126+
W_init : weights initializer
127+
The initializer for initializing the weight matrix.
128+
b_init : biases initializer
129+
The initializer for initializing the bias vector.
130+
W_init_args : dictionary
131+
The arguments for the weights tf.get_variable.
132+
b_init_args : dictionary
133+
The arguments for the biases tf.get_variable.
134+
name : a string or None
135+
An optional name to attach to this layer.
136+
137+
def __init__(
138+
self,
139+
layer = None,
140+
n_units = 100,
141+
act = tf.nn.relu,
142+
W_init = tf.truncated_normal_initializer(stddev=0.1),
143+
b_init = tf.constant_initializer(value=0.0),
144+
W_init_args = {},
145+
b_init_args = {},
146+
name ='dense_layer',
147+
):
148+
Layer.__init__(self, name=name)
149+
self.inputs = layer.outputs
150+
if self.inputs.get_shape().ndims != 2:
151+
raise Exception("The input dimension must be rank 2")
152+
n_in = int(self.inputs._shape[-1])
153+
self.n_units = n_units
154+
print(" tensorlayer:Instantiate DenseLayer %s: %d, %s" % (self.name, self.n_units, act))
155+
with tf.variable_scope(name) as vs:
156+
W = tf.get_variable(name='W', shape=(n_in, n_units), initializer=W_init, **W_init_args )
157+
b = tf.get_variable(name='b', shape=(n_units), initializer=b_init, **b_init_args )
158+
self.outputs = act(tf.matmul(self.inputs, W) + b)
159+
160+
# Hint : list(), dict() is pass by value (shallow).
161+
self.all_layers = list(layer.all_layers)
162+
self.all_params = list(layer.all_params)
163+
self.all_drop = dict(layer.all_drop)
164+
self.all_layers.extend( [self.outputs] )
165+
self.all_params.extend( [W, b] )
166+
167+
168+
A simple layer
169+
^^^^^^^^^^^^^^^
170+
171+
To implement a custom layer in TensorLayer, you will have to write a Python class
172+
that subclasses Layer and implement the ``outputs`` expression.
173+
174+
The following is an example implementation of a layer that multiplies its input by 2:
175+
176+
.. code-block:: python
177+
178+
class DoubleLayer(Layer):
179+
def __init__(
180+
self,
181+
layer = None,
182+
name ='dense_layer',
183+
):
184+
Layer.__init__(self, name=name)
185+
self.inputs = layer.outputs
186+
self.outputs = self.inputs * 2
187+
188+
self.all_layers = list(layer.all_layers)
189+
self.all_params = list(layer.all_params)
190+
self.all_drop = dict(layer.all_drop)
191+
self.all_layers.extend( [self.outputs] )
192+
193+
194+
10195
.. automodule:: tensorlayer.layers
11196
12197
.. autosummary::

‎tensorlayer/__pycache__/__init__.cpython-34.pyc‎

898 Bytes
Binary file not shown.

‎tensorlayer/__pycache__/activation.cpython-34.pyc‎

1002 Bytes
Binary file not shown.

‎tensorlayer/__pycache__/cost.cpython-34.pyc‎

10.4 KB
Binary file not shown.

‎tensorlayer/__pycache__/files.cpython-34.pyc‎

25.2 KB
Binary file not shown.

‎tensorlayer/__pycache__/iterate.cpython-34.pyc‎

6.3 KB
Binary file not shown.

‎tensorlayer/__pycache__/layers.cpython-34.pyc‎

63.9 KB
Binary file not shown.

‎tensorlayer/__pycache__/nlp.cpython-34.pyc‎

23.3 KB
Binary file not shown.

0 commit comments

Comments
 (0)