一、所需环境（安装附链接）

tensorflow和keras，具体版本安装看个人所需。
安装链接如下：
https://blog.csdn.net/qq_41760767/article/details/97441967?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.nonecase
安装好后查看keras版本

二、数据集准备

下载图像数据集train，在Home目录下新建子目录"data"，把下载的图像数据集train复制到"data"目录。具体如图所示：

import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
from IPython.display import Image
root_dir = os.getcwd()
data_path = os.path.join(root_dir,'data')
#根据目录路径
root_dir = os.getcwd()

#存放数据集的目录
data_path = os.path.join(root_dir,'data')
import os,shutil#原始的数据集目录
original_dataset_dir = os.path.join(data_path,'train')#存储小数据集的目录
base_dir = os.path.join(data_path,'cats_and_dogs_small')
if not os.path.exists(base_dir):os.mkdir(base_dir)#训练图像的目录
train_dir = os.path.join(base_dir,'train')
if not os.path.exists(train_dir):os.mkdir(train_dir)
#验证图像的目录
validation_dir = os.path.join(base_dir,'validation')
if not os.path.exists(validation_dir):os.mkdir(validation_dir)
#测试资料的目录
test_dir = os.path.join(base_dir,'test')
if not os.path.exists(test_dir):os.mkdir(test_dir)#猫的图片的训练资料的目录
train_cats_dir = os.path.join(train_dir,'cats')
ifnot os.path.exists(train_cats_dir):os.mkdir(train_cats_dir)
#狗的图片的训练资料的目录
train_dogs_dir = os.path.join(train_dir,'dogs')
if not os.path.exists(train_dogs_dir):os.mkdir(train_dogs_dir)#猫的图片的测试数据集目录
test_cats_dir = os.path.join(test_dir,'cats')
if not os.path.exists(test_cats_dir):os.mkdir(test_cats_dir)
#狗的图片的测试数据集目录
test_dogs_dir = os.path.join(test_dir,'dogs')
if not os.path.exists(test_dogs_dir):os.mkdir(test_dogs_dir)
#复制前600个猫的图片到train_cats_dir
fnames = ['cat.{}.jpg'.format(i) for i in range(600)]
for fname in fnames:src = os.path.join(original_dataset_dir,fname)dst = os.path.join(train_cats_dir,fname)if not os.path.exists(dst):shutil.copyfile(src,dst)
print("Copy next 600 cat images to train_cats_dir complete!")
#复制后面400个猫的图片到validation_cats_dir
fnames = ['cat.{}.jpg'.format(i) for i in range(1000,1400)]
for fname in fnames:src = os.path.join(original_dataset_dir,fname)dst = os.path.join(validation_cats_dir,fname)if not os.path.exists(dst):shutil.copyfile(src,dst)
print('Copy next 400 cat images to validation_cats_dir complete!')
#复制400张猫的图片到test_cats_dir
fnames = ['cat.{}.jpg'.format(i) for i in range(1500,1900)]
for fname in fnames:src = os.path.join(original_dataset_dir,fname)dst = os.path.join(test_cats_dir,fname)if not os.path.exists(dst):shutil.copyfile(src,dst)
print("Copy next 400 cat images to test_cats_dir complete!")#复制前600张狗的图片到train_dogs_dir
fnames = ['dog.{}.jpg'.format(i) for i in range(600)]
for fname in fnames:src = os.path.join(original_dataset_dir,fname)dst = os.path.join(train_dogs_dir,fname)if not os.path.exists(dst):shutil.copyfile(src,dst)
print("Copy first 600 dog images to train_dogs_dir complete!")
#复制后400个狗的图片到validation_dogs_dir
names = ['dog.{}.jpg'.format(i) for i in range(1000, 1400)]
for fname in fnames:src = os.path.join(original_dataset_dir, fname)dst = os.path.join(validation_dogs_dir, fname)if not os.path.exists(dst):shutil.copyfile(src, dst)
print('Copy next 400 dog images to validation_dogs_dir complete!')
#复制400张狗的图片到test_dogs_dir
fnames = ['dog.{}.jpg'.format(i) for i in range(1500, 1900)]
for fname in fnames:src = os.path.join(original_dataset_dir, fname)dst = os.path.join(test_dogs_dir, fname)if not os.path.exists(dst):shutil.copyfile(src, dst)
print('Copy next 400 dog images to test_dogs_dir complete!')

#进行一次检查，计算每个分组(训练/验证/测试)中各有多少张图片

print('total training cat images:', len(os.listdir(train_cats_dir)))
print('total training dog images:', len(os.listdir(train_dogs_dir)))
print('total validation cat images:', len(os.listdir(validation_cats_dir)))
print('total validation dog images:', len(os.listdir(validation_dogs_dir)))
print('total test cat images:', len(os.listdir(test_cats_dir)))
print('total test dog images:', len(os.listdir(test_dogs_dir)))

共1200个训练图像，600个测试图像，600个验证图像

三、网络模型

卷积网络(convnets)将是一组交替的Conv2D(具有relu激活)和MaxPooling2D层。从大小150x150（有点任意选择）的输入开始，我们最终得到了尺寸为７x7的Flatten层之前的特征图。

注意特征图的深度在网络中逐渐增加(从32到128)，而特征图的大小正在减少(从148x148到7x7)。这是一个你将在几乎所有的卷积网络(convnets)结构中会看到的模式。

由于我们正在处理二元分类问题，所以我们用一个神经元（一个大小为１的密集层(Dense))和一个sigmoid激活函数来结束网络。该神经元将会被用来查看图像归属于那一类或另一类的概率。

from keras import layers
from keras import models
from keras.utils import plot_modelmodel = models.Sequential()
model.add(layers.Conv2D(32,(3,3),activation='relu',input_shape=(150,150,3)))
model.add(layers.MaxPooling2D((2,2)))model.add(layers.Conv2D(64,(3,3),activation='relu'))
model.add(layers.MaxPooling2D((2,2)))model.add(layers.Conv2D(128,(3,3),activation='relu'))
model.add(layers.MaxPooling2D((2,2)))model.add(layers.Conv2D(128,(3,3),activation='relu'))
model.add(layers.MaxPooling2D((2,2)))model.add(layers.Flatten())
model.add(layers.Dense(512,activation='relu'))
model.add(layers.Dense(1,activation='sigmoid'))

Let’s take a look at how the dimensions of the feature maps change with every successive layer:

model.summary()

from keras import optimizersmodel.compile(loss='binary_crossentropy',optimizer=optimizers.RMSprop(lr=1e-4),metrics=['acc'])

四、Data preprocessing （数据预处理）

As you already know by now, data should be formatted into appropriately pre-processed floating point tensors before being fed into our network. Currently, our data sits on a drive as JPEG files, so the steps for getting it into our network are roughly:

Read the picture files.
Decode the JPEG content to RBG grids of pixels.
Convert these into floating point tensors.
Rescale the pixel values (between 0 and 255) to the [0, 1] interval (as you know, neural networks prefer to deal with small input values).

It may seem a bit daunting, but thankfully Keras has utilities to take care of these steps automatically. Keras has a module with image processing helper tools, located at keras.preprocessing.image. In particular, it contains the class ImageDataGenerator which allows to quickly set up Python generators that can automatically turn image files on disk into batches of pre-processed tensors. This is what we will use here.

from keras.preprocessing.image import ImageDataGenerator# All images will be rescaled by 1./255
train_datagen = ImageDataGenerator(rescale=1./255)
test_datagen = ImageDataGenerator(rescale=1./255)train_generator = train_datagen.flow_from_directory(# This is the target directorytrain_dir,# All images will be resized to 150x150target_size=(150, 150),batch_size=20,# Since we use binary_crossentropy loss, we need binary labelsclass_mode='binary')validation_generator = test_datagen.flow_from_directory(validation_dir,target_size=(150, 150),batch_size=20,class_mode='binary')

Let’s take a look at the output of one of these generators: it yields batches of 150x150 RGB images (shape (20, 150, 150, 3)) and binary labels (shape (20,)). 20 is the number of samples in each batch (the batch size). Note that the generator yields these batches indefinitely: it just loops endlessly over the images present in the target folder. For this reason, we need to break the iteration loop at some point

for data_batch, labels_batch in train_generator:print('data batch shape:', data_batch.shape)print('labels batch shape:', labels_batch.shape)break

五、训练

Let’s fit our model to the data using the generator. We do it using the fit_generator method, the equivalent of fit for data generators like ours. It expects as first argument a Python generator that will yield batches of inputs and targets indefinitely, like ours does. Because the data is being generated endlessly, the generator needs to know example how many samples to draw from the generator before declaring an epoch over. This is the role of the steps_per_epoch argument: after having drawn steps_per_epoch batches from the generator, i.e. after having run for steps_per_epoch gradient descent steps, the fitting process will go to the next epoch. In our case, batches are 20-sample large, so it will take 100 batches until we see our target of 2000 samples.

When using fit_generator, one may pass a validation_data argument, much like with the fit method. Importantly, this argument is allowed to be a data generator itself, but it could be a tuple of Numpy arrays as well. If you pass a generator as validation_data, then this generator is expected to yield batches of validation data endlessly, and thus you should also specify the validation_steps argument, which tells the process how many batches to draw from the validation generator for evaluation.

history = model.fit_generator(train_generator,steps_per_epoch=100,epochs=30,validation_data=validation_generator,validation_steps=50)

训练过程有点长，这里跑代码的时间是根据个人电脑显卡的优差。
然后训练完别忘了保存模型

model.save('cats_and_dogs_small_1.h5')

使用图表来秀出在训练过程中模型对训练和验证数据的损失(loss)和准确性(accuracy)数据

import matplotlib.pyplot as pltacc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']epochs = range(len(acc))plt.plot(epochs,acc,label='Training acc')
plt.plot(epochs,val_acc,label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()plt.figure()plt.plot(epochs,loss,label='Training loss')
plt.plot(epochs,val_loss,label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
plt.show()

These plots are characteristic of overfitting. Our training accuracy increases linearly over time, until it reaches nearly 100%, while our validation accuracy stalls at 70-72%. Our validation loss reaches its minimum after only five epochs then stalls, while the training loss keeps decreasing linearly until it reaches nearly 0.

Because we only have relatively few training samples (2000), overfitting is going to be our number one concern. You already know about a number of techniques that can help mitigate overfitting, such as dropout and weight decay (L2 regularization). We are now going to introduce a new one, specific to computer vision, and used almost universally when processing images with deep learning models: data augmentation.

六、使用数据填充

过度拟合是由于样本数量太少而导致的，导致我们无法训练能够推广到新数据的模型。

给定无限数据，我们的模型将暴露在手头数据分布的每个可能的方向:我们永远不会过度。数据增加采用从现有训练样本生成更多训练数据的方法，通过产生可信的图像的多个随机变换来"增加"样本。目标是在训练的时候，我们的模型永远不会再看到完全相同的画面两次。这有助于模型暴学习到数据的更多方面，并更好地推广。

在keras中，可以通过配置对我们的ImageDataGenerator实例读取的图像执行多个随机变换来完成。

datagen = ImageDataGenerator(rotation_range=40,width_shift_range=0.2,height_shift_range=0.2,shear_range=0.2,zoom_range=0.2,horizontal_flip=True,fill_mode='nearest')

这些只是列出一些可用的选项(更多选项，可以参考keras文档）。快速看一下这些参数:
●rotation_range是以度（0-180)为单位的值，它是随机旋转图片的范围。
●width_shift和height_shift是范围(占总宽度或高度的一小部分),用于纵向或横向随机转换图片。
●shear_range用于随机剪切变换。
●zoom_range用于随机放大图片内容。
●horizontal_flip用于在没有水平不对称假设（例如真实世界图片）的情况下水平地随机翻转一半图像。
看一下增强后的图像：

# This is module with image preprocessing utilities
from keras.preprocessing import imagefnames = [os.path.join(train_cats_dir, fname) for fname in os.listdir(train_cats_dir)]# We pick one image to "augment"
img_path = fnames[3]# Read the image and resize it
img = image.load_img(img_path, target_size=(150, 150))# Convert it to a Numpy array with shape (150, 150, 3)
x = image.img_to_array(img)# Reshape it to (1, 150, 150, 3)
x = x.reshape((1,) + x.shape)# The .flow() command below generates batches of randomly transformed images.
# It will loop indefinitely, so we need to `break` the loop at some point!
i = 0
for batch in datagen.flow(x, batch_size=1):plt.figure(i)imgplot = plt.imshow(image.array_to_img(batch[0]))i += 1if i % 4 == 0:breakplt.show()

如果我们使用这种数据增强配置来训练一个新的网络，我们的网络将永远不会看到相同重复的输入。然而，它看到的输入仍然是相互关联的，因为它们来自少量的原始图像 - 我们不能产生新的信息，我们只能重新混合现有的信息。因此，这可能不足以完全摆脱过度拟合(overfitting)。为了进一步克服过度拟合(overfitting)，我们还将在密集连接(densely-connected)的分类器之前添加一个Dropout层。

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu',input_shape=(150, 150, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))model.compile(loss='binary_crossentropy',optimizer=optimizers.RMSprop(lr=1e-4),metrics=['acc'])

使用数据填充(data augmentation)和dropout来训练我们的网络

train_datagen = ImageDataGenerator(rescale=1./255,rotation_range=40,width_shift_range=0.2,height_shift_range=0.2,shear_range=0.2,zoom_range=0.2,horizontal_flip=True,)# Note that the validation data should not be augmented!
test_datagen = ImageDataGenerator(rescale=1./255)train_generator = train_datagen.flow_from_directory(# This is the target directorytrain_dir,# All images will be resized to 150x150target_size=(150, 150),batch_size=32,# Since we use binary_crossentropy loss, we need binary labelsclass_mode='binary')validation_generator = test_datagen.flow_from_directory(validation_dir,target_size=(150, 150),batch_size=32,class_mode='binary')history = model.fit_generator(train_generator,steps_per_epoch=100,epochs=100,validation_data=validation_generator,validation_steps=50)

这个训练的时间就更长了，建议没事的时候再挂着去跑。

保存我们的模型，我们将在convnet可视化部分中使用它。

model.save('cats_and_dogs_small_2.h5')

再看一边结果：

acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']epochs = range(len(acc))plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()plt.figure()plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()plt.show()

由于数据的增加和丢失，我们不再过度拟合：训练曲线非常接近于验证曲线。我们现在能够达到82%的精度，比非正则模型相对提高了15%。
通过进一步利用正则化技术并通过调整网络参数（例如每个卷积层的滤波器数量或网络中的层数量），我们可能能够获得更好的精度，可能高达86-87%。

训练猫狗数据集（及图像增强后训练）相关推荐

kaggle（一）训练猫狗数据集
记录第一次使用kaggle训练猫狗数据集 import os import shutil os.listdir('../input/train/train') base_dir = './cat_do ...
【猫狗数据集】pytorch训练猫狗数据集之创建数据集
数据集下载地址: 链接:https://pan.baidu.com/s/1tJQIY0ob2EyQn3cDipPkow?pwd=7gch 提取码:7gch 猫狗数据集的分为训练集25000张,在训练 ...
神经网络学习小记录19——微调VGG分类模型训练自己的数据（猫狗数据集）
神经网络学习小记录19--微调VGG分类模型训练自己的数据(猫狗数据集) 注意事项学习前言什么是VGG16模型 VGG模型的复杂程度训练前准备 1.数据集处理 2.创建Keras的VGG模型 3 ...
神经网络学习小记录17——使用AlexNet分类模型训练自己的数据（猫狗数据集）
神经网络学习小记录17--使用AlexNet分类模型训练自己的数据(猫狗数据集) 学习前言什么是AlexNet模型训练前准备 1.数据集处理 2.创建Keras的AlexNet模型开始训练 1. ...
AlexNet训练分类猫狗数据集
AlexNet训练分类猫狗数据集视频教程来源视频程序以及猫狗数据集: 链接:https://pan.baidu.com/s/1Tqs5bFY2wVvtGeuFBWV1Yg 提取码:3zrd 一.数 ...
Tensorflow2.0实战练习之猫狗数据集(包含自定义训练和迁移学习)
最近在学习使用Tenforflow2.0,写下这篇文章,用来帮助和我一样的初学者,文章中如果存在某些问题,还希望各位指出. 目录数据集介绍数据处理及增强 VGG模型介绍模型搭建训练及结果展示 ...
Pytorch采用AlexNet实现猫狗数据集分类（训练与预测）
Pytorch采用AlexNet实现猫狗数据集分类(训练与预测) 介绍 AlexNet网络模型猫狗数据集 AlexNet网络训练训练全代码预测预测图片介绍 AlexNet模型是CNN网络中经 ...
keras笔记(3)-猫狗数据集上的训练以及单张图片多张图片的测试
之前也写过关于使用tensorflow在猫狗数据集上的训练,想要学习的可以看一下数据集下载猫狗数据集:https://pan.baidu.com/s/13hw4LK8ihR6-6-8mpjLKDA ...
tensorflow.js在nodejs训练猫狗分类模型在浏览器上使用
目录本人系统环境注意事项前言数据集准备处理数据集数据集初步处理将每一张图片数据转换成张量数据(tensor) 将图片转换成张量数组的代码和运行效果将图片的标注转换成张量数据(tenso ...

训练猫狗数据集（及图像增强后训练）

目录