关于Mask_RCNN的工程化应用cpu服务器部署日记(一)

1 技术可行性分析:

Mask_RCNN是目前表现最好的模型之一,和GAN系列的vidvid 和NLP的BERT在我的经验里都是最好用的模型之一,但是今日碰到了一个问题就是生产环境没有GPU,面临怎么去部署的问题.. 备注:在训练环境中已经训练成功了~~能做到用640张样本达到可接受的分割效果.但是线上使用阿里云服务,GPU太贵了.需要尝试一下cpu,如果能达到1000ms内,就帮公司省点钱呗.毕竟不是实时性要求很高的应用.

1.1创建生产环境的日志:

下载了conda, 在.bashrc 添加conda.sh

conda create -n py3cpu python=3.6.2

pip install numpy scipy Pillow cython matplotlib scikit-image keras==2.0.8 h5py IPython
pip install opencv-python imgaug

pip install tensorflow==1.4.0

conda和pip兼容性还是不错的.

进行model的推断模式.果然报错

Processing 1 images
image shape: (512, 512, 3) min: 0.00000 max: 255.00000 uint8
molded_images shape: (1, 512, 512, 3) min: -123.70000 max: 151.10000 float64
image_metas shape: (1, 15) min: 0.00000 max: 512.00000 int64
anchors shape: (1, 65280, 4) min: -0.17712 max: 1.11450 float32
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
return fn(*args)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
status, run_metadata)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[1] = 65343 is not in [0, 65280)
   [[Node: ROI/Gather_2 = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI/strided_slice_6, ROI/strided_slice_7)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/Documents/Mask_RCNN-master/samples/dish_food/test_model.py", line 118, in <module>
ma.test(start_id=0, stop=632)
File "/home/ubuntu/Documents/Mask_RCNN-master/samples/dish_food/test_model.py", line 90, in test
results = model.detect(patch_resized_images, verbose=1)
File "/home/ubuntu/Documents/Mask_RCNN-master/mrcnn/model.py", line 2479, in detect
self.keras_model.predict([molded_images, image_metas, anchors], verbose=0)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/keras/engine/training.py", line 1713, in predict
verbose=verbose, steps=steps)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/keras/engine/training.py", line 1269, in _predict_loop
batch_outs = f(ins_batch)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2273, in __call__
**self.session_kwargs)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[1] = 65343 is not in [0, 65280)
   [[Node: ROI/Gather_2 = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI/strided_slice_6, ROI/strided_slice_7)]]

Caused by op 'ROI/Gather_2', defined at:
File "/home/ubuntu/Documents/Mask_RCNN-master/samples/dish_food/test_model.py", line 29, in <module>
model_dir=MODEL_DIR)
File "/home/ubuntu/Documents/Mask_RCNN-master/mrcnn/model.py", line 1824, in __init__
self.keras_model = self.build(mode=mode, config=config)
File "/home/ubuntu/Documents/Mask_RCNN-master/mrcnn/model.py", line 1948, in build
config=config)([rpn_class, rpn_bbox, anchors])
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/keras/engine/topology.py", line 602, in __call__
output = self.call(inputs, **kwargs)
File "/home/ubuntu/Documents/Mask_RCNN-master/mrcnn/model.py", line 294, in call
names=["pre_nms_anchors"])
File "/home/ubuntu/Documents/Mask_RCNN-master/mrcnn/utils.py", line 826, in batch_slice
output_slice = graph_fn(*inputs_slice)
File "/home/ubuntu/Documents/Mask_RCNN-master/mrcnn/model.py", line 292, in <lambda>
pre_nms_anchors = utils.batch_slice([anchors, ix], lambda a, x: tf.gather(a, x),
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 2486, in gather
params, indices, validate_indices=validate_indices, name=name)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1834, in gather
validate_indices=validate_indices, name=name)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/home/ubuntu/anaconda3/envs/py3cpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): indices[1] = 65343 is not in [0, 65280)
   [[Node: ROI/Gather_2 = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](ROI/strided_slice_6, ROI/strided_slice_7)]]

报错的是names=["pre_nms_anchors"]的tensor, 查api,知道了tf.gather是按照indices 获取数组新集合的.但是scores的top 6000的indices应该和anchors的indices的范围应该是一致的才对.

需要进一步进行tensor的debug.

关于Mask_RCNN的工程化应用cpu服务器部署日记(一)相关推荐

【20210402期AI简报】TensorFlow-YOLOv3 从本地训练到服务器部署全过程
导读:本期为 AI 简报 20210402 期,将为您带来 8 条相关新闻,明日寒食节~ 记得有一期有小伙伴留言说需要手势识别相关,本期它来啦~ 本文一共 2200 字,通篇阅读结束需要 5~7 分钟 ...
MED-V服务器部署，MED-V服务器系列之一
MED-V服务器部署 [IT168 专稿]微软在2009年4月份发布的MDOP2009中终于提供了MED-V的正式版本.MED-V是 Microsoft Enterprise Desktop Vi ...
一步步实现SDDC--多角色服务器部署
实验摘要: 1>创建一台虚拟机 2>虚拟机安装VMware Tools(针对Windows操作系统) 正文: 通过之前的两篇文字,相信各位已经了解ESXI服务器的部署和基本配置. 在今天的 ...
HappyAA服务器部署笔记1(nginx+tomcat的安装与配置)
这是本人的服务器部署笔记.文章名称叫"部署笔记1"的原因是之后我对这个进行了改进之后,会有"部署笔记2","部署笔记3"...循序渐进,估计 ...
Linux服务器部署常用命令
Linux命令大全前言在日常开发中我们会做一些服务器部署的工作,对于开发来说,运维部署并不是我们的专项:有些命令不是经常使用,很快就会忘记:在这总结一下日常部署时常用的一些运维命令. 文件操作 t ...
国内首款 FPGA 云服务器，性能是通用 CPU 服务器 30 倍以上
2019独角兽企业重金招聘Python工程师标准>>> 版权声明:本文由薛梁原创文章,转载请注明出处: 文章原文链接:https://www.qcloud.com/communi ...
Linux 时间一致性环境NTP/Chrony服务器部署配置
一.背景生产环境中,一个业务系统往往涉及几十台甚至上百台主机或云主机构成,有些管理系统对整个环境时间一致性由强要求,即使没有,保证环境中时间的一致性,有利于业务交互时时间匹配,保证事务的一致性,甚至 ...
华为云鲲鹏服务器部署文档--java微服务
华为云鲲鹏服务器部署文档河南中电高科计算机技术有限公司 2020.5.9 适用于java微服务技术栈. CentOS 7.6 64bit ISO 适用于鲲鹏服务器arm架构的CentOS 7.6.1 ...
华为云鲲鹏服务器部署文档-修正版-CentOS+java微服务开发
华为云鲲鹏服务器部署文档河南中电高科计算机技术有限公司 http://www.ceht.com.cn 2020.5.12 适用于java微服务开发运行基础软件环境部署,实测通过. CentOS 7. ...

关于Mask_RCNN的工程化应用cpu服务器部署日记(一)

关于Mask_RCNN的工程化应用cpu服务器部署日记(一)相关推荐

最新文章

热门文章