TensorRT学习总结

--- 如何安装
https://docs.nvidia.com/deeplearning/sdk/tensorrt-install-guide/index.html#installing

tensorrt安装后的so如下图所示.所以一台机器只能有一个版本的tensorrt.

TensorRT是什么

建议先看看这篇https://zhuanlan.zhihu.com/p/35657027
深度学习

• 训练
• 部署

平常自学深度学习的时候关注的更多是训练的部分,即得到一个模型.而实际工作很大一块的工作内容集中于如何将模型部署到具体的芯片上.你自己写的模型效果是很难优于成熟的知名的模型的.
以无人驾驶为例,拍摄到图片后,芯片上的加载的模型要能够识别出图片里是什么.对自动驾驶这种场景而言,对实时性地要求是非常高的.试想,从图片输入到模型,到模型识别出图片中前方有个人花了1分钟,你正以100km/h行驶,后果自然是灾难性的.
这就引出了推理引擎.model里的信息其实就是一些权重矩阵信息而已.输入图片数据后,进行一大堆的矩阵运算,得到最终图片的分类.推理引擎干的事情就是优化矩阵运算,缩短运算时间.

CUDA cuDNN

https://blog.csdn.net/u014380165/article/details/77340765

• CUDA是NVIDIA推出的用于自家GPU的并行计算框架，也就是说CUDA只能在NVIDIA的GPU上运行，而且只有当要解决的计算问题是可以大量并行计算的时候才能发挥CUDA的作用
• cuDNN（CUDA Deep Neural Network library）：是NVIDIA打造的针对深度神经网络的加速库，是一个用于深层神经网络的GPU加速库。如果你要用GPU训练模型，cuDNN不是必须的，但是一般会采用这个加速库

查看TensorRT版本

dpkg -l | grep TensorRT

convert_to_uff安装在哪里?

使用TensorRT推导SSD网络

以下是TensorRT中自带的一个例子

The sampleUffSSD example is based on the following paper, SSD: Single Shot MultiBox
Detector (https://arxiv.org/abs/1512.02325). The SSD network performs the
task of object detection and localization in a single forward pass of the network.
The tensorflow SSD network was trained on the InceptionV2 architecture using
the MSCOCO dataset.

The sample makes use of TensorRT plugins to run the SSD network. To use these
plugins the TensorFlow graph needs to be preprocessed.
 
Steps to generate UFF file:
    0. Make sure you have the UFF converter installed. For installation instructions, see:
        https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/#python and click on the 'TensorRT Python API' link.

    1. Get the pre-trained Tensorflow model (ssd_inception_v2_coco) from:
        http://download.tensorflow.org/models/object_detection/ssd_inception_v2_coco_2017_11_17.tar.gz

    2. Call the UFF converter with the preprocessing flag set (-p [config_file]).
        The config.py script specifies the preprocessing operations necessary for SSD TF graph.
        It must be copied to the working directory for the file to be imported properly.
        The plugin nodes and plugin parameters used in config.py should match the registered plugins
        in TensorRT. Please read the plugins documentation for more details.

        'convert-to-uff --input-file frozen_inference_graph.pb -O NMS -p config.py'

This script saves the converted .uff file in the same directory as the input with
the name frozen_inference_graph.pb.uff. Copy this converted .uff file to the
data directory as sample_ssd_relu6.uff <TensorRT Install>/data/ssd/sample_ssd_relu6.uff

The sample also requires a labels .txt file with a list of all labels used to
train the model. Current example for this network is <TensorRT Install>/data/ssd/ssd_coco_labels.txt

Steps to run the network:
    1. To run the network in FP32 mode, ./sample_uff_ssd
    2. To run the network in INT8 mode, ./sample_uff_ssd --int8

To run the network in INT8 mode, refer to BatchStreamPPM.h for details on how
calibration can be performed. Currently we require a file (list.txt) with
a list of all PPM images for calibration in the <TensorRT Install>/data/ssd/ folder.
The PPM images to be used for calibration can also reside in the same folder.

NOTE - There might be some precision loss when running the network in INT8
mode causing some objects to go undetected. Our general observation is that
>500 images is a good number for calibration purposes.

Python API

• Graph Surgeon API
• UFF API

Included within the Python API is the UFF API; a package that contains a set of utilities to convert trained models from various frameworks to a common format.
The UFF API is located in uff/uff.html and contains two conversion type tool classes called Tensorflow Modelstream to UFF and Tensorflow Frozen Protobuf Model to UFF.

TensorRT的python_api地址:https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/python_api/index.html

官方的readme是有问题的.

'convert-to-uff --input-file frozen_inference_graph.pb -O NMS -p config.py'

**开始排坑之旅! **
convert-to-uff并不是一个二进制文件,而是一个.py脚本. 位置在/usr/lib/python2.7/dist-packages/uff/bin/convert_to_uff.py,
/usr/lib/python3.6/dist-packages/uff/bin
具体参见这个https://devtalk.nvidia.com/default/topic/1025246/jetson-tx2/where-is-convert-to-uff/2

以及下面link中'3.2.4. Importing From TensorFlow Using Python'
https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#python_samples_section

运行脚本,提示

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/uff/bin/convert_to_uff.py", line 15, in <module>
    import uff
  File "/usr/lib/python2.7/dist-packages/uff/__init__.py", line 1, in <module>
    from uff import converters, model  # noqa
  File "/usr/lib/python2.7/dist-packages/uff/model/__init__.py", line 1, in <module>
    from . import uff_pb2 as uff_pb  # noqa
  File "/usr/lib/python2.7/dist-packages/uff/model/uff_pb2.py", line 6, in <module>
    from google.protobuf.internal import enum_type_wrapper
ImportError: No module named google.protobuf.internal

安装protobuf.ubuntu18.04默认是没安装pip的.要先装pip.

• sudo apt update
• sudo apt install python-pip
• pip install protobuf

再次执行脚本,报错

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/uff/bin/convert_to_uff.py", line 15, in <module>
    import uff
  File "/usr/lib/python2.7/dist-packages/uff/__init__.py", line 2, in <module>
    from uff.converters.tensorflow.conversion_helpers import from_tensorflow  # noqa
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/conversion_helpers.py", line 12, in <module>
    from .converter_functions import *  # noqa
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter_functions.py", line 12, in <module>
    from uff.converters.tensorflow.converter import TensorFlowToUFFConverter as tf2uff
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py", line 24, in <module>
    https://www.tensorflow.org/install/""".format(err))
ImportError: ERROR: Failed to import module (No module named tensorflow)
Please make sure you have TensorFlow installed.

安装tensorflow

pip install tensorflow-gpu
Collecting tensorflow-gpu
  Could not find a version that satisfies the requirement tensorflow-gpu (from versions: )
No matching distribution found for tensorflow-gpu

https://www.tensorflow.org/install/pip?lang=python2
尝试用以下方式安装
pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.12.0-cp27-none-linux_x86_64.whl
再次报错!

tensorflow_gpu-1.12.0-cp27-none-linux_x86_64.whl is not a supported wheel on this platform.

nvidia官方提供了官方说明,如何install TensorFlow for Jetson Platform

https://docs.nvidia.com/deeplearning/dgx/install-tf-xavier/index.html

• sudo apt-get install libhdf5-serial-dev hdf5-tools
• sudo apt-get install python3-pip
• pip3 install -U pip
• sudo apt-get install zlib1g-dev zip libjpeg8-dev libhdf5-dev
• sudo pip3 install -U numpy grpcio absl-py py-cpuinfo psutil portpicker grpcio six mock requests gast h5py astor termcolor
  又报错!!!

nano@nano-desktop:/usr/src/tensorrt/samples/sampleUffSSD$ sudo pip3 install -U numpy grpcio absl-py py-cpuinfo psutil portpicker grpcio six mock requests gast h5py astor termcolor
Traceback (most recent call last):
  File "/usr/bin/pip3", line 9, in <module>
    from pip import main
ImportError: cannot import name 'main'

搜索了一大圈,说啥的都有.应该是pip本身的bug.
https://stackoverflow.com/questions/54420766/python3-6-importerror-cannot-import-name-main-after-upgrading-pip-from-8-1-1
将pip3 install替换成python3 -m pip install.
或者按照有的文章里说的比如这篇https://blog.csdn.net/zong596568821xp/article/details/80410416去修改试一下/usr/bin/pip3.

• sudo python3 -m pip install -U numpy grpcio absl-py py-cpuinfo psutil portpicker grpcio six mock requests gast h5py astor termcolor
  这一步要耐心的多尝试几次,动不动就http连接失败,即便我已经用代理翻墙的情况下也是如此.

查看包安装到了什么位置:pip show命令

这里自己打开那个http地址看一下.官方文档里的版本1.12.0不存在.

• pip3 install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v42 tensorflow-gpu==1.13.1+nv19.3

等待一段时间.会自动卸载掉之前安装的版本过低的protobuf,重新安装符合要求的protobuf.

Successfully installed keras-applications-1.0.7 keras-preprocessing-1.0.9 markdown-3.1 protobuf-3.7.1 tensorboard-1.13.1 tensorflow-estimator-1.13.0 tensorflow-gpu-1.13.1+nv19.3 werkzeug-0.15.2

把.pb格式的模型转换为uff格式(TensorRT可以识别的格式)

python3 /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py --input-file ./ssd_inception_v2_coco_2017_11_17/frozen_inference_graph.pb -O NMS -p config.py

运行ssd模型

#拷贝uff模型文件到data/ssd下
cp ./ssd_inception_v2_coco_2017_11_17/frozen_inference_graph.uff ~/tensorrt/data/ssd/sample_ssd_relu6.uff
#去tensort/bin下执行sample_uff_ssd程序
~/tensorrt/bin$ ./sample_uff_ssd

坑又来啦!!!

nano@nano-desktop:~/tensorrt/bin$ ./sample_uff_ssd
../data/ssd/sample_ssd_relu6.uff
Begin parsing model...
End parsing model...
Begin building engine...
Killed

一番搜索,https://devtalk.nvidia.com/default/topic/1046206/tensorrt5-sample_uff_ssd-can-t-run/ 有说官方readme里的link下载的model不对.

https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
tensorflow官方提供的pretrained-model.
换个模型重新做一次转换.
这里尝试了2个不同的模型,均来自tensorflow官方github.一个在转uff格式时失败,一个转换后解析模型时失败 to do.

换回了最初的模型,尝试成功了,原因不明.........

--- update 20200306
https://s0docs0nvidia0com.icopy.site/deeplearning/sdk/tensorrt-install-guide/index.html