执行python generate_tfrecord.py 出现 utf-8‘ codec can‘t decode

将 cvs转换为tfrecord ，终端输入指令：

python generate_tfrecord.py --csv_input=data/cup_train.csv --output_path=data/cup_train.record

其中python文件中的csv_input相当于data/cup_train.csv ，output_path相当于data/cup_train.record，也就是输出文件路径以及名称。

执行下面code：

# -*- coding: utf-8 -*-
"""
Created on Tue Jan 16 01:04:55 2018@author: Xiang Guo
""""""
Usage:# From tensorflow/models/# Create train data:python generate_tfrecord.py --csv_input=data/tv_vehicle_labels.csv  --output_path=train.record# Create test data:python generate_tfrecord.py --csv_input=data/test_labels.csv  --output_path=test.record
"""import os
import io
import pandas as pd
import tensorflow as tffrom PIL import Image
from object_detection.utils import dataset_util
from collections import namedtuple, OrderedDictos.chdir('D:\\tensorflow-model\\models\\research\\object_detection\\')flags = tf.app.flags
flags.DEFINE_string('csv_input', '', 'Path to the CSV input')
flags.DEFINE_string('output_path', '', 'Path to output TFRecord')
FLAGS = flags.FLAGS# TO-DO replace this with label map
def class_text_to_int(row_label):if row_label == 'tv':return 1elif row_label == 'vehicle':return 2else:Nonedef split(df, group):data = namedtuple('data', ['filename', 'object'])gb = df.groupby(group)return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]def create_tf_example(group, path):with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid:encoded_jpg = fid.read()encoded_jpg_io = io.BytesIO(encoded_jpg)image = Image.open(encoded_jpg_io)width, height = image.sizefilename = group.filename.encode('utf8')image_format = b'jpg'xmins = []xmaxs = []ymins = []ymaxs = []classes_text = []classes = []for index, row in group.object.iterrows():xmins.append(row['xmin'] / width)xmaxs.append(row['xmax'] / width)ymins.append(row['ymin'] / height)ymaxs.append(row['ymax'] / height)classes_text.append(row['class'].encode('utf8'))classes.append(class_text_to_int(row['class']))tf_example = tf.train.Example(features=tf.train.Features(feature={'image/height': dataset_util.int64_feature(height),'image/width': dataset_util.int64_feature(width),'image/filename': dataset_util.bytes_feature(filename),'image/source_id': dataset_util.bytes_feature(filename),'image/encoded': dataset_util.bytes_feature(encoded_jpg),'image/format': dataset_util.bytes_feature(image_format),'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),'image/object/class/text': dataset_util.bytes_list_feature(classes_text),'image/object/class/label': dataset_util.int64_list_feature(classes),}))return tf_exampledef main(_):writer = tf.python_io.TFRecordWriter(FLAGS.output_path)path = os.path.join(os.getcwd(), 'images')examples = pd.read_csv(FLAGS.csv_input)grouped = split(examples, 'filename')for group in grouped:tf_example = create_tf_example(group, path)writer.write(tf_example.SerializeToString())writer.close()output_path = os.path.join(os.getcwd(), FLAGS.output_path)print('Successfully created the TFRecords: {}'.format(output_path))if __name__ == '__main__':tf.app.run()

遇到以下几个问题：

1. 错误提示：AttributeError: module 'tensorflow' has no attribute 'app'

分析问题：由于tensorflow版本问题导致的

解决方式：将导入code

import tensorflow as tf

修改为：

import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

2.错误提示： File "C:\Program Files\python\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 84, in _preread_check
self._read_buf = _pywrap_file_io.BufferedInputStream(
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 112: invalid continuation byte

分析问题：通过跟踪打印，发现path 问题导致的

解决方式：将def main(_): 中 path = os.path.join(os.getcwd(), 'images') 这句中的image 修改为存放cup_train.csv 以及 jpg 和xml 的文件夹名称（ path = os.path.join(os.getcwd(), 'data')）。

3.错误提示：tf.python_io.TFRecordWriter UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 112: invalid continuation byte

分析问题：直接在pycharm 或者别的python 管理app中执行generate_tfrecord.py ，导致writer = tf.python_io.TFRecordWriter(FLAGS.output_path) 中的FLAGS.output_path 没有传递文件名

解决问题：可以直接在writer = tf.python_io.TFRecordWriter('D:\\models-master\\research\\object_detection\\data\\cup_train.record') 可以直接给出文件输出路径与名称；

修正后的code：

#-*- coding : utf-8 -*-
"""
Created on Tue Jan 16 01:04:55 2018@author: Xiang Guo
""""""
Usage:# From tensorflow/models/# Create train data:python generate_tfrecord.py --csv_input=data/tv_vehicle_labels.csv  --output_path=train.record# Create test data:python generate_tfrecord.py --csv_input=data/test_labels.csv  --output_path=test.record
"""import os
import io
import pandas as pd
#import tensorflow as tf
import tensorflow.compat.v1 as tffrom PIL import Image
from object_detection.utils import dataset_util
from collections import namedtuple, OrderedDictos.chdir('D:\\models-master\\research\\object_detection\\')flags = tf.app.flags
flags.DEFINE_string('csv_input', '', 'Path to the CSV input')
flags.DEFINE_string('output_path', '', 'Path to output TFRecord')
FLAGS = flags.FLAGS# TO-DO replace this with label map
def class_text_to_int(row_label):if row_label == 'cup':return 1else:Nonedef split(df, group):data = namedtuple('data', ['filename', 'object'])gb = df.groupby(group)return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]def create_tf_example(group, path):with tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as fid:encoded_jpg = fid.read()encoded_jpg_io = io.BytesIO(encoded_jpg)image = Image.open(encoded_jpg_io)width, height = image.sizefilename = group.filename.encode('utf8')image_format = b'jpg'xmins = []xmaxs = []ymins = []ymaxs = []classes_text = []classes = []for index, row in group.object.iterrows():xmins.append(row['xmin'] / width)xmaxs.append(row['xmax'] / width)ymins.append(row['ymin'] / height)ymaxs.append(row['ymax'] / height)classes_text.append(row['class'].encode('utf8'))classes.append(class_text_to_int(row['class']))tf_example = tf.train.Example(features=tf.train.Features(feature={'image/height': dataset_util.int64_feature(height),'image/width': dataset_util.int64_feature(width),'image/filename': dataset_util.bytes_feature(filename),'image/source_id': dataset_util.bytes_feature(filename),'image/encoded': dataset_util.bytes_feature(encoded_jpg),'image/format': dataset_util.bytes_feature(image_format),'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),'image/object/class/text': dataset_util.bytes_list_feature(classes_text),'image/object/class/label': dataset_util.int64_list_feature(classes),}))return tf_exampledef main(_):writer = tf.python_io.TFRecordWriter(FLAGS.output_path)path = os.path.join(os.getcwd(), 'data')examples = pd.read_csv(FLAGS.csv_input,encoding="unicode_escape")grouped = split(examples, 'filename')for group in grouped:tf_example = create_tf_example(group, path)writer.write(tf_example.SerializeToString())writer.close()output_path = os.path.join(os.getcwd(), FLAGS.output_path)print('Successfully created the TFRecords: {}'.format(output_path))if __name__ == '__main__':tf.app.run()

其中：path = os.path.join(os.getcwd(), 'data') 中的data就是终端执行时传递的文件名称

执行python generate_tfrecord.py 出现 utf-8‘ codec can‘t decode相关推荐

pycharm终端运行python文件_在PyCharm终端中执行python manage.py..._慕课问答
原来在DOS环境下需要先执行: conda activate命令,然后再执行 python manage.py runserver就可以了. 如下: D:\DjangoProject\django_ ...
使用Django在执行python manage.py startapp myApp创建应用
使用Django在执行python manage.py startapp myApp创建应用出现很长的报错信息并且最后两行提示: raise ImproperlyConfigured('mysqlc ...
python setup.py build,无法执行“python setup.py build”命令..！
我试图在BACKTRACK 5 R1虚拟机上安装pylorcon-1(Lorcon用于编写无线数据包的嗅探器的python包装)...我从[Pylorcon Official Page] [1]下载的 ...
执行python setup.py install时报错：error: can't create or remove files in install directory
文章目录报错 mac&linux Windows 报错在研究setuptools时,执行python setup.py install,遇到不能创建文件夹的权限报错,完整的错误日志如下: ...
Django配置MySQL执行python manage.py makemigrations 出现No changes detected报错
在给Django配置mysql的时候按照教程在Model.py文件中配置加入定义模型类的代码后执行数据迁移代码 python manage.py makemigrations 报错显示No cha ...
python编码错误：UnicodeDecodeError: 'utf8' codec can't decode
python编码错误:UnicodeDecodeError: 'utf8' codec can't decode (2012-11-01 17:16:23) 转载▼ 标签: 杂谈分类:python学 ...
python报错：UnicodeDecodeError: ‘gbk‘ codec can‘t decode byte 0xa3 in position 48
python报错: UnicodeDecodeError: 'gbk' codec can't decode byte 0xa3 in position 48: illegal multibyte s ...
Cpython源码分析03（*）_简要总结下Cpython是如何执行python test.py
当我们通过命令行传入参数的方式调用 python 解释器去运行一个模块的时候,比如: $ python test.py 图2.1中所示的过程将开始进行.(当然这只是其中一种运行 Python 程序的方 ...
学习Python的Django执行python manage.py startapp myApp创建应用出现的问题
最近在学习python的Django,跟着视频操作却也出现问题,创建应用的时候在终端输入python manage.py startapp myApp 出现了报错,报错如下: 报错信息有点多,但最后俩 ...

执行python generate_tfrecord.py 出现 utf-8‘ codec can‘t decode

执行python generate_tfrecord.py 出现 utf-8‘ codec can‘t decode相关推荐

最新文章

热门文章