原文链接: TensorFlow 使用 slim 模块搭建复杂网络

上一篇: scrapy 代理使用

下一篇: TensorFlow infogan 生成 mnist 数据集

参考

https://blog.csdn.net/guvcolie/article/details/77686555

https://www.cnblogs.com/zyly/p/9146787.html

引入模块

import tensorflow.contrib.slim as slim

简化模型搭建

input = ...
with tf.name_scope('conv1_1') as scope:kernel = tf.Variable(tf.truncated_normal([3, 3, 64, 128], dtype=tf.float32,stddev=1e-1), name='weights')conv = tf.nn.conv2d(input, kernel, [1, 1, 1, 1], padding='SAME')biases = tf.Variable(tf.constant(0.0, shape=[128], dtype=tf.float32),trainable=True, name='biases')bias = tf.nn.bias_add(conv, biases)conv1 = tf.nn.relu(bias, name=scope)net = slim.conv2d(input, 128, [3, 3], scope='conv1_1')

TF-Slim提供了标准接口用于组建神经网络，包括：

repeat

net = ...
net = slim.conv2d(net, 256, [3, 3], scope='conv3_1')
net = slim.conv2d(net, 256, [3, 3], scope='conv3_2')
net = slim.conv2d(net, 256, [3, 3], scope='conv3_3')
net = slim.max_pool2d(net, [2, 2], scope='pool2')net = slim.repeat(net, 3, slim.conv2d, 256, [3, 3], scope='conv3')
net = slim.max_pool2d(net, [2, 2], scope='pool2')

slim.repeat不但可以在一行中使用相同的参数，而且还能智能地展开scope，即每个后续的slim.conv2d调用所对应的scope都会追加下划线及迭代数字。更具体地讲，上面代码的scope分别为 'conv3/conv3_1', 'conv3/conv3_2' and 'conv3/conv3_3'.

stack

# Verbose way:
x = slim.fully_connected(x, 32, scope='fc/fc_1')
x = slim.fully_connected(x, 64, scope='fc/fc_2')
x = slim.fully_connected(x, 128, scope='fc/fc_3')# Equivalent, TF-Slim way using slim.stack:
slim.stack(x, slim.fully_connected, [32, 64, 128], scope='fc')

TF-Slim的 slim.stack操作符允许调用者用不同的参数重复使用相同的操作符是创建一个stack或网络层塔。slim.stack也会为每个创建的操作符生成一个新的scope。例如，下面是一个简单的方法去创建MLP：

在这个例子中，slim.stack调用 slim.fully_connected 三次，前一个层的输出是下一层的输入。而每个网络层的输出通道数从32变到64，再到128. 同样，我们可以用stack简化一个多卷积层塔：

# Verbose way:
x = slim.conv2d(x, 32, [3, 3], scope='core/core_1')
x = slim.conv2d(x, 32, [1, 1], scope='core/core_2')
x = slim.conv2d(x, 64, [3, 3], scope='core/core_3')
x = slim.conv2d(x, 64, [1, 1], scope='core/core_4')# Using stack:
slim.stack(x, slim.conv2d, [(32, [3, 3]), (32, [1, 1]), (64, [3, 3]), (64, [1, 1])], scope='core')

这种方式可以确保这三个卷积层共享相同的参数值，但是仍然没有减少代码规模。通过使用arg_scope，我们既能确保每层共享参数值，又能精简代码 ：

如例所示，arg_scope使代码更简洁且易于维护。注意，在arg_scope中被指定的参数值，也可以在局部位置进行覆盖。比如，padding参数设置为'SAME', 而第二个卷积层仍然可以通过把它设为'VALID'而覆盖掉arg_scope中的默认设置。

with slim.arg_scope([slim.conv2d], padding='SAME',weights_initializer=tf.truncated_normal_initializer(stddev=0.01)weights_regularizer=slim.l2_regularizer(0.0005)):net = slim.conv2d(inputs, 64, [11, 11], scope='conv1')net = slim.conv2d(net, 128, [11, 11], padding='VALID', scope='conv2')net = slim.conv2d(net, 256, [11, 11], scope='conv3')

vgg16 模型搭建以及各层shape

import tensorflow as tf
import tensorflow.contrib.slim as slim
def vgg_16(inputs,num_classes=1000,is_training=True,dropout_keep_prob=0.5,):# Collect outputs for conv2d, fully_connected and max_pool2d.with slim.arg_scope([slim.conv2d, slim.fully_connected, slim.max_pool2d]):print(inputs.shape)  # (16, 224, 224, 3)net = slim.repeat(inputs, 2, slim.conv2d, 64, [3, 3], scope='conv1')print(net.shape)  # (16, 224, 224, 64)net = slim.max_pool2d(net, [2, 2], scope='pool1')print(net.shape)  # (16, 112, 112, 64)net = slim.repeat(net, 2, slim.conv2d, 128, [3, 3], scope='conv2')print(net.shape)  # (16, 112, 112, 128)net = slim.max_pool2d(net, [2, 2], scope='pool2')print(net.shape)  # (16, 56, 56, 128)net = slim.repeat(net, 3, slim.conv2d, 256, [3, 3], scope='conv3')print(net.shape)  # (16, 56, 56, 256)net = slim.max_pool2d(net, [2, 2], scope='pool3')print(net.shape)  # (16, 28, 28, 256)net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv4')print(net.shape)  # (16, 28, 28, 512)net = slim.max_pool2d(net, [2, 2], scope='pool4')print(net.shape)  # (16, 14, 14, 512)net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv5')print(net.shape)  # (16, 14, 14, 512)net = slim.max_pool2d(net, [2, 2], scope='pool5')print(net.shape)  # (16, 7, 7, 512)# Use conv2d instead of fully_connected layers.net = slim.conv2d(net, 4096, [7, 7], padding="VALID", scope='fc6')print(net.shape)  # (16, 1, 1, 4096)net = slim.dropout(net, dropout_keep_prob,scope='dropout6')print(net.shape)  # (16, 1, 1, 4096)net = slim.conv2d(net, 4096, [1, 1], scope='fc7')print(net.shape)  # (16, 1, 1, 4096)# Convert end_points_collection into a end_point dict.net = slim.fully_connected(net, num_classes,activation_fn=slim.softmax, scope='fc8')print(net.shape)  # (16, 1, 1, 1000)print('---')return netin_x = tf.zeros([16, 224, 224, 3])res = vgg_16(in_x)

loss

import tensorflow as tf
vgg = tf.contrib.slim.nets.vgg# Load the images and labels.
images, labels = ...# Create the model.
predictions, _ = vgg.vgg_16(images)# Define the loss functions and get the total loss.
loss = slim.losses.softmax_cross_entropy(predictions, labels)

多输出的多任务模型：在这个例子中，我们有两个损失，分别是通过 slim.losses.softmax_cross_entropy和 slim.losses.sum_of_squares得到的。我们既可以通过相加得到 total_loss，也可以通过 slim.losses.get_total_loss()得到total_loss。这是怎么做到的呢？当你通过TF-Slim创建一个损失函数时，TF-Slim会把损失加入到一个特殊的Tensorflow的损失函数集合中。这样你既可以手动管理损失函数，也可以托管给TF-Slim。

# Load the images and labels.
images, scene_labels, depth_labels = ...# Create the model.
scene_predictions, depth_predictions = CreateMultiTaskModel(images)# Define the loss functions and get the total loss.
classification_loss = slim.losses.softmax_cross_entropy(scene_predictions, scene_labels)
sum_of_squares_loss = slim.losses.sum_of_squares(depth_predictions, depth_labels)# The following two lines have the same effect:
total_loss = classification_loss + sum_of_squares_loss
total_loss = slim.losses.get_total_loss(add_regularization_losses=False)

可以手动管理损失函数，也可以让TF-Slim知晓这个自定义损失函数，然后托管给TF-Slim。

# Load the images and labels.
images, scene_labels, depth_labels, pose_labels = ...# Create the model.
scene_predictions, depth_predictions, pose_predictions = CreateMultiTaskModel(images)# Define the loss functions and get the total loss.
classification_loss = slim.losses.softmax_cross_entropy(scene_predictions, scene_labels)
sum_of_squares_loss = slim.losses.sum_of_squares(depth_predictions, depth_labels)
pose_loss = MyCustomLossFunction(pose_predictions, pose_labels)
slim.losses.add_loss(pose_loss) # Letting TF-Slim know about the additional loss.# The following two ways to compute the total loss are equivalent:
regularization_loss = tf.add_n(slim.losses.get_regularization_losses())
total_loss1 = classification_loss + sum_of_squares_loss + pose_loss + regularization_loss# (Regularization Loss is included in the total loss by default).
total_loss2 = slim.losses.get_total_loss()

train

g = tf.Graph()# Create the model and specify the losses...
...total_loss = slim.losses.get_total_loss()
optimizer = tf.train.GradientDescentOptimizer(learning_rate)# create_train_op ensures that each time we ask for the loss, the update_ops
# are run and the gradients being computed are applied too.
train_op = slim.learning.create_train_op(total_loss, optimizer)
logdir = ... # Where checkpoints are stored.slim.learning.train(train_op,logdir,number_of_steps=1000,save_summaries_secs=300,save_interval_secs=600):

在该例中， slim.learning.train根据 train_op 计算损失、应用梯度step。logdir指定了checkpoints和event文件的存储路径。我们可以限制梯度step到任何数值。这里我们采用1000步。最后， save_summaries_secs=300 表示每5分钟计算一次summaries， save_interval_secs=600 表示每10分钟保存一次模型的checkpoint。

从检查恢复部分变量

# Create some variables.
v1 = tf.Variable(..., name="v1")
v2 = tf.Variable(..., name="v2")
...
# Add ops to restore all the variables.
restorer = tf.train.Saver()# Add ops to restore some variables.
restorer = tf.train.Saver([v1, v2])# Later, launch the model, use the saver to restore variables from disk, and
# do some work with the model.
with tf.Session() as sess:# Restore variables from disk.restorer.restore(sess, "/tmp/model.ckpt")print("Model restored.")# Do some work with the model...

# Create some variables.
v1 = slim.variable(name="v1", ...)
v2 = slim.variable(name="nested/v2", ...)
...# Get list of variables to restore (which contains only 'v2'). These are all
# equivalent methods:
#从检查点文件中恢复name='v2'的变量
variables_to_restore = slim.get_variables_by_name("v2")
# or 从检查点文件中恢复name带有2的所有变量
variables_to_restore = slim.get_variables_by_suffix("2")
# or 从检查点文件中恢复命名空间scope='nested'的所有变量
variables_to_restore = slim.get_variables(scope="nested")
# or 恢复命名空间scope='nested'的所有变量
variables_to_restore = slim.get_variables_to_restore(include=["nested"])
# or 除了命名空间scope='v1'的变量
variables_to_restore = slim.get_variables_to_restore(exclude=["v1"])      # Create the saver which will be used to restore the variables.
restorer = tf.train.Saver(variables_to_restore)with tf.Session() as sess:# Restore variables from disk.restorer.restore(sess, "/tmp/model.ckpt")print("Model restored.")# Do some work with the model...

当图的变量名与checkpoint中的变量名不同时，恢复模型参数

当从checkpoint文件中恢复变量时，Saver在checkpoint文件中定位到变量名，并且把它们映射到当前图中的变量中。之前的例子中，我们创建了Saver，并为其提供了变量列表作为参数。这时，在checkpoint文件中定位的变量名，是隐含地从每个作为参数给出的变量的var.op.name而获得的。这一方式在图与checkpoint文件中变量名字相同时，可以很好的工作。而当名字不同时，必须给Saver提供一个将checkpoint文件中的变量名映射到图中的每个变量的字典。

假设我们定义的网络变量是conv1/weights，而从VGG检查点文件加载的变量名为vgg16/conv1/weights，正常load肯定会报错（找不到变量名），但是可以这样：例子见下：

# Assuming that 'conv1/weights' should be restored from 'vgg16/conv1/weights'
def name_in_checkpoint(var):return 'vgg16/' + var.op.name# Assuming that 'conv1/weights' and 'conv1/bias' should be restored from 'conv1/params1' and 'conv1/params2'
def name_in_checkpoint(var):if "weights" in var.op.name:return var.op.name.replace("weights", "params1")if "bias" in var.op.name:return var.op.name.replace("bias", "params2")variables_to_restore = slim.get_model_variables()
variables_to_restore = {name_in_checkpoint(var):var for var in variables_to_restore}
restorer = tf.train.Saver(variables_to_restore)with tf.Session() as sess:# Restore variables from disk.restorer.restore(sess, "/tmp/model.ckpt")

微调训练

image, label = MyPascalVocDataLoader(...)
images, labels = tf.train.batch([image, label], batch_size=32)# Create the model,20类
predictions = vgg.vgg_16(images,num_classes=20)train_op = slim.learning.create_train_op(...)# Specify where the Model, trained on ImageNet, was saved.
model_path = '/path/to/pre_trained_on_imagenet.checkpoint'# Specify where the new model will live:
log_dir = '/path/to/my_pascal_model_dir/'# Restore only the convolutional layers: 从检查点载入除了fc6,fc7,fc8层之外的参数
variables_to_restore = slim.get_variables_to_restore(exclude=['fc6', 'fc7', 'fc8'])
init_fn = assign_from_checkpoint_fn(model_path, variables_to_restore)# Start training.
slim.learning.train(train_op, log_dir, init_fn=init_fn)