FPN网络训练报错PaddingFIFOQueue '_1_get_batch/batch/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)

https://github.com/DetectionTeamUCAS/FPN_Tensorflow
按说明设置自己的训练集后，进入tool/运行train.py后报错：

PaddingFIFOQueue '_1_get_batch/batch/padding_fifo_queue' is closed and has insufficient elements (requested 1, current size 0)

查了一下，应该是xml文件中box坐标范围导致的。修改data/io/convert_data/io/convert_to_tfrecord.py中line68为

  gtbox_label = np.array(box_list, dtype=np.int32)  # [x1, y1. x2, y2, label]

  xmin, ymin, xmax, ymax, label = gtbox_label[:, 0], gtbox_label[:, 1], gtbox_label[:, 2],\
      gtbox_label[:, 3], gtbox_label[:, 4]

  xmin = np.where(xmin <= 0, 1, xmin)
  ymin = np.where(ymin <= 0, 1, ymin)
  xmax = np.where(xmax >= img_width, img_width-1, xmax)
  ymax = np.where(ymax >= img_height, img_height-1, ymax)

  # [ymin, xmin, ymax, xmax, label]
  gtbox_label = np.transpose(np.stack([ymin, xmin, ymax, xmax, label], axis=0))

即可。

TypeError: ...has type str, but expected one of: bytes

在python3环境中将数据转为TFRecord时遇到的问题
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
TypeError: 'rock' has type str, but expected one of: bytes

解决方式：

转byte feature时用tf.compat.as_bytes(value)将字节或 Unicode 转换为 bytes，使用 UTF-8 编码文本：

def _bytes_feature(value):

  value = tf.compat.as_bytes(value)#####

  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))

pip2与pip3共存

之前修改了导致pip2无法使用，导致一些开发工具无法使用。 可以先解除 python3的软连接, 然后再修改成 python2的. mac 自带的是 python2.7
1. 先获取2.7的路径:
which python2.7, 我的是/usr/bin/python2.7

2. 解除原有链接 unlink /usr/bin/python

3. 恢复默认链接: sudo ln -s python2 /usr/bin/python

行人重识别

项目地址：
https://github.com/huanghoujing/person-reid-triplet-loss-baseline#visualize-ranking-list
test命令：

python2 script/experiment/train.py -d '(0,)' --only_test true --dataset duke --last_conv_stride 1 --normalize_feature false --exp_dir exp --model_weight_file /home/zhoujie/person-reid-triplet-loss-baseline/model_weight/duke_stride1/model_weight.pth


项目地址：
https://github.com/layumi/Person_reID_baseline_pytorch/tree/master/tutorial
train命令：

python train.py --gpu_ids 0 --name ft_ResNet50 --train_all --batchsize 32  --data_dir /home/zhoujie/Person_reID_baseline_pytorch

coco数据集转换pascal VOC数据集

1）得到annotations/json格式：labelme2coco.py

输入： 
 # 把所有的jpg和json都放到了images目录下
    base_dir = '/Users/sisyphus/TIANCHI_XUELANG_AI_gangban/data/image'

输出：
 # 把训练集转化为COCO的json格式
    l2c_train = Lableme2CoCo()
    train_instance = l2c_train.to_coco(train_path)
    l2c_train.save_coco_json(train_instance, '/Users/sisyphus/TIANCHI_XUELANG_AI_gangban/data/annotations/instances_train2017.json')

    # 把验证集转化为COCO的json格式
    l2c_val = Lableme2CoCo()
    val_instance = l2c_val.to_coco(val_path)
    l2c_val.save_coco_json(val_instance, '/Users/sisyphus/TIANCHI_XUELANG_AI_gangban/data/annotations/instances_val2017.json')


2）得到根目录下单个json文件：getclassnum.py

输入：cocojson="/Users/sisyphus/TIANCHI_XUELANG_AI_gangban/data/annotations/instances_train2017.json"

输出：
COCO_train.json


3）得到VOC xml标注文件：createXML.py

输入：
_IMAGE_PATH = '/Users/sisyphus/TIANCHI_XUELANG_AI_gangban/data/imageonly/'

with open("COCO_train.json", "r") as f:
        ann_data = json.load(f)

输出：
_ANNOTATION_SAVE_PATH = '/Users/sisyphus/TIANCHI_XUELANG_AI_gangban/data/VOC2018/Annotations'

4）生成ImageSets/Main/text文件：darknet/traintxt.py

输入：
xmlfilepath = '/Users/sisyphus/tf-faster-rcnn/data/VOCdevkit2007/VOC2007/Annotations'

输出：
saveBasePath = "/Users/sisyphus/tf-faster-rcnn/data/VOCdevkit2007/VOC2007/"

ftrainval = open(os.path.join(saveBasePath,'ImageSets/Main/trainval.txt'), 'w') 
ftest = open(os.path.join(saveBasePath,'ImageSets/Main/test.txt'), 'w') 
ftrain = open(os.path.join(saveBasePath,'ImageSets/Main/train.txt'), 'w') 
fval = open(os.path.join(saveBasePath,'ImageSets/Main/val.txt'), 'w') 


PS：
数据增强：
/Users/sisyphus/TIANCHI_XUELANG_AI_gangban/src/DataAugForTrainAndTest_DET.py

重命名jpg或xml名称：
mAP/changeTextName.py

修改xml中对应图片名称：
py/xmlEdit2.py

tf faster rcnn推导测试集计算mAP报错

训练完毕后运行./experiments/scripts/test_faster_rcnn.sh 0 pascal_voc res101
测试mAP，报错：

and the problem:
VOC07 metric? Yes
000001
Traceback (most recent call last):
File "./tools/test_net.py", line 90, in
test_net(net, imdb, max_per_image=args.max_per_image, vis=args.vis)
File "/home/rs/myGithub/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 295, in test_net
imdb.evaluate_detections(all_boxes, output_dir)
File "/home/rs/myGithub/py-faster-rcnn/tools/../lib/datasets/pascal_voc.py", line 322, in evaluate_detections
self._do_python_eval(output_dir)
File "/home/rs/myGithub/py-faster-rcnn/tools/../lib/datasets/pascal_voc.py", line 285, in _do_python_eval
use_07_metric=use_07_metric)
File "/home/rs/myGithub/py-faster-rcnn/tools/../lib/datasets/voc_eval.py", line 127, in voc_eval
R = [obj for obj in recs[imagename] if obj['name'] == classname]
KeyError: '000001'

解决方法：

had the same problem with an old annotation cache. Just delete it
rm data/VOCdevkit2007/annotations_cache/annots.pkl and run the program again.
You should also delete the roidb cache:
rm data/cache/voc_2007_trainval_gt_roidb.pkl

PS：如要可视化测试结果，可在CPU电脑上运行tf-faster-rcnn/tools/demo_one.py程序。

tf faster rcnn 训练自己的数据集

1 按照https://github.com/endernewton/tf-faster-rcnn中介绍步骤安装项目。

2 用自己的数据集替换VOC2007数据集文件夹。

3 修改lib/datasets/pascal_voc.py中

self._classes = ('__background__',  # always index 0       "defect0","defect1","defect2","defect3","defect4","defect5","defect6","defect7","defect8","defect9")####################
 
4 lib/datasets/factory.py和experiments/scripts/train_faster_rcnn, experiments/scripts/test_faster_rcnn中，因为自己数据集名称和VOC2007一致，暂不用修改。

5 开始训练

./experiments/scripts/train_faster_rcnn.sh 0 pascal_voc res101
注意每次训练前删除output和data/cache文件夹。

训练时其它可能需要修改的参数：
1）修改迭代次数：experiments/scripts/train_faster_rcnn.sh文件中修改迭代次数

experiments/scripts/test_faster_rcnn.sh中也相应修改
2) 学习率设置：tf-faster-rcnn/lib/model/config.py文件中设置



6 tools/demo.py修改种类数
net.create_architecture("TEST", 11,#####################21


可能出现的错误：
1）InvalidArgumentError (see above for traceback): Nan in summary histogram for: 


it's because in the file 'pascal_voc.py', the function '_load_pascal_annotation' has an operation of make pixel indexes 0-based,the code is :
x1 = float(bbox.find('xmin').text) - 1
y1 = float(bbox.find('ymin').text) - 1
x2 = float(bbox.find('xmax').text) - 1
y2 = float(bbox.find('ymax').text) - 1
删除-1

2）assert(cfg.TRAIN.BATCH_SIZE % num_images == 0), 
ZeroDivisionError: integer division or modulo by zero

test.txt不能为空

xml和json无法显示中文

解决办法：在相关语句中添加“encoding='utf-8'”。
如：

xmldoc.write(xmlpath+xmllist[i], encoding='utf-8')

json.dump(instance, open(save_path, 'w', encoding='utf-8'), ensure_ascii=False, indent=1)

with open(path, "r", encoding='utf-8') as f:

ImportError: No module named Cython.Distutils

Ubuntu下使用tensorflow版faster-RCNN执行make命令报错

For python3 use

sudo apt-get install cython3

For python2 use

sudo apt-get install cython

安装OpenCV3.4.2

from: https://cloud.tencent.com/developer/article/1329166

安装OpenCV3.4.2

注意：想要在OpenCV中玩转YOLOv3，必须安装OpenCV3.4.2版本及以上。

Python版安装方法

1# 推荐安装外挂版
2pip install opencv-contrib-python
3
4# 或者安装正常版
5pip install opencv-python

Ubuntu使用darknet遇到的坑

修改makefile文件

GPU=1
CUDNN=1
NVCC=/usr/local/cuda-9.0/bin/nvcc #即改为本机CUDA安装地址

make后报错

include/darknet.h:16:23: fatal error: cudnn.h: 没有那个文件或目录
compilation terminated.
Makefile:89: recipe for target 'obj/gemm.o' failed
make: *** [obj/gemm.o] Error 1

是因为缺乏头文件cudnn.h

执行

cd /home/zhoujie/.conda/envs/tf/include

sudo cp cudnn.h /usr/include

再次make后报错

/usr/bin/ld: 找不到 -lcudnn
collect2: error: ld returned 1 exit status
Makefile:83: recipe for target 'libdarknet.so' failed
make: *** [libdarknet.so] Error 1

这是因为缺少库文件libcudnn.so

执行

cd /home/zhoujie/.conda/envs/tf/lib

sudo cp libcudnn.so /usr/local/cuda/lib64

make成功

执行推导图片命令后报错

libcudart.so.8.0: cannot open shared object file: No such file or directory

执行

sudo ldconfig /usr/local/cuda/lib64

就可以正常推导和训练