NVIDIA Jetson专用模型部署-容器模式
本文针对NVIDIA Jetson nano设备,介绍如何获取适配的端模型文件,并结合NVIDIA L4T Base镜像,将模型部署至BIE,完成AI图像识别。
1、jetson nano版本依赖说明
本实验的模型依赖于以下版本。
jetpack版本
查看jetpack版本,版本号为4.3。
nano@jetson-nano:~$ head -n 1 /etc/nv_tegra_release
# R32 (release), REVISION: 4.3, GCID: 21589087, BOARD: t210ref, EABI: aarch64, DATE: Fri Jun 26 04:38:25 UTC 2020
cuda-driver版本
查看cudu驱动版本,版本号为10.2。
nano@jetson-nano:~$ dpkg -l | grep cuda-driver
ii cuda-driver-dev-10-2 10.2.89-1 arm64 CUDA Driver native dev stub library
TensorRT版本
查看TensorRT版本,版本号7.1.3。
nano@jetson-nano:~$ dpkg -l | grep TensorRT
ii graphsurgeon-tf 7.1.3-1+cuda10.2 arm64 GraphSurgeon for TensorRT package
ii libnvinfer-bin 7.1.3-1+cuda10.2 arm64 TensorRT binaries
ii libnvinfer-dev 7.1.3-1+cuda10.2 arm64 TensorRT development libraries and headers
ii libnvinfer-doc 7.1.3-1+cuda10.2 all TensorRT documentation
ii libnvinfer-plugin-dev 7.1.3-1+cuda10.2 arm64 TensorRT plugin libraries
ii libnvinfer-plugin7 7.1.3-1+cuda10.2 arm64 TensorRT plugin libraries
ii libnvinfer-samples 7.1.3-1+cuda10.2 all TensorRT samples
ii libnvinfer7 7.1.3-1+cuda10.2 arm64 TensorRT runtime libraries
ii libnvonnxparsers-dev 7.1.3-1+cuda10.2 arm64 TensorRT ONNX libraries
ii libnvonnxparsers7 7.1.3-1+cuda10.2 arm64 TensorRT ONNX libraries
ii libnvparsers-dev 7.1.3-1+cuda10.2 arm64 TensorRT parsers libraries
ii libnvparsers7 7.1.3-1+cuda10.2 arm64 TensorRT parsers libraries
ii nvidia-container-csv-tensorrt 7.1.3.0-1+cuda10.2 arm64 Jetpack TensorRT CSV file
ii python-libnvinfer 7.1.3-1+cuda10.2 arm64 Python bindings for TensorRT
ii python-libnvinfer-dev 7.1.3-1+cuda10.2 arm64 Python development package for TensorRT
ii python3-libnvinfer 7.1.3-1+cuda10.2 arm64 Python 3 bindings for TensorRT
ii python3-libnvinfer-dev 7.1.3-1+cuda10.2 arm64 Python 3 development package for TensorRT
ii tensorrt 7.1.3.0-1+cuda10.2 arm64 Meta package of TensorRT
ii uff-converter-tf 7.1.3-1+cuda10.2 arm64 UFF converter for TensorRT package
2、上传模型文件包至对象存储
- 将模型文件包fruit-ai-model.zip上传至BOS。
-
模型文件包制作方式参考最后章节。
说明:这个模型包必须部署在能够连接公网的设备上才能生效,因为需要在线激活license,在无法联网的环境当中,无法使用。
3、云端配置
3.1. 创建边缘节点
如下图所示,创建边缘节点jetson-nano
3.2. 创建AI模型配置项
如下图所示,创建配置项:fruit-demo。引入上传至BOS对象存储的模型文件。
3.3. 创建AI推断服务
如下图所示,创建应用fruit-detection
在应用当中添加一个容器服务,如下图所示:
- 镜像地址:nvidia提供了一个面向NVIDIA Jetson系列的容器景象:NVIDIA L4T Base。nvidia官网获取方式:https://ngc.nvidia.com/catalog/containers/nvidia:l4t-base。 该容器镜像提供nvidia jetson的运行环境,能够实现在容器内跑支持jetson设备的AI模型服务。在支持容器化以后,可以通过BIE来管理边缘节点设备,以及提升模型服务部署的效率。
NVIDIA L4T Base官网镜像获取地址:nvcr.io/nvidia/l4t-base:r32.4.3。如果国内下载速度很慢,可以从百度cce上下载镜像:hub.baidubce.com/nvidia/l4t-base:r32.4.3-arm64
- 卷配置:类型为配置项;参数选择之前创建的文件配置项;容器目录:/sdk(与run.sh文件对应)。通过卷配置,实现AI推断服务与模型文件绑定。将推断服务下发至边缘节点的时候,会自动下载AI模型文件。
- 容器服务的端口映射:端口为24401;协议为TCP
- 容器服务的启动参数:bash;/sdk/run.sh
- 无需开启特权模式
3.4. 将应用部署至节点
在边缘应用当中,添加目标节点标签
4、边缘端配置
在jetson nano设备上安装节点
4.1 配置jetson上docker的default runtime为nvidia
执行docker info查看边缘节点容器环境,确保Default Runtime: nvidia
nano@jetson-nano:~/Downloads/fruit-ai-model$ docker info
Client:
Debug Mode: false
Server:
Containers: 38
Running: 21
Paused: 0
Stopped: 17
Images: 12
Server Version: 19.03.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: nvidia runc
** Default Runtime: nvidia**
Init Binary: docker-init
containerd version:
runc version:
init version:
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.140-tegra
Operating System: Ubuntu 18.04.5 LTS
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 3.871GiB
Name: jetson-nano
ID: O7GP:DDD5:5CIR:LEWJ:2BQ3:4WIW:VA4H:JDCP:5VGL:L2K3:PLZ7:KBHO
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
如果Default Runtime不是nvidia,修改/etc/docker/daemon.json文件,添加"default-runtime": "nvidia",修改完毕以后的/etc/docker/daemon.json文件
nano@jetson-nano:~$ cat /etc/docker/daemon.json
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}
修改完毕以后,重启docker
sudo systemctl restart docker
4.2 运行安装命令
- 在copy云端边缘节点安装命令
- 在边缘节点执行安装命令
5、验证边缘节点AI推断服务
通过浏览器打开在线推断服务:http://「ip」 :24401/,上传测试图片,推断结果如下,证明AI服务正常启动。
6、easyedge模型包制作方案
6.1. 利用easyedge获取适配jetson nano环境的SDK
上传原始模型
模型提交后,在我的模型界面生成端模型。选择对应的芯片和操作系统,点击发布,获取SDK文件。
6.2. SDK编译和验证
对SDK文件进行编译和认证,生成在jetson设备上可运行的模型。
- 注意:使用EasyDL的Jetson系列SDK需要安装指定版本的JetPack和相关组件。目前所支持的JetPack版本包括:JetPack4.4和JetPack4.2.2
模型资源文件默认已经打包在下载的SDK包中。Demo工程直接编译即可运行。
cd demo
mkdir build && cd build
cmake .. && make
sudo ./easyedge_demo {模型RES文件夹} {测试图片路径} {序列号}
编译过程参考:
`nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ls
CMakeCache.txt CMakeFiles cmake_install.cmake easyedge_batch_inference easyedge.log easyedge_multi_thread easyedge_serving install_manifest.txt Makefile
nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ./easyedge_batch_inference /home/nano/Downloads/easydl-sdk/RES /home/nano/Downloads/easydl-sdk/imgs/mix008.jpeg
2020-08-24 03:51:09,915 WARNING [EasyEdge] 548102246416 Only compile_level 2 is supported for this model on JetPack 4.4 DP version. Please read documentation for the details
2020-08-24 03:51:09,915 INFO [EasyEdge] 548102246416 Compiling model for fast inference, this may take a while (Acceleration)
2020-08-24 03:53:08,488 INFO [EasyEdge] 548102246416 Optimized model saved to: /home/nano/.baidu/easyedge/jetson/mcache/26119049355/m_cache, Don't remove it
Results of image /home/nano/Downloads/easydl-sdk/imgs/mix008.jpeg:
1, tomato, p:0.999717 loc: 0.672033, 0.405379, 0.80056, 0.569352
2, kiwi, p:0.999273 loc: 0.154121, 0.0717932, 0.393312, 0.399547
2, kiwi, p:0.999206 loc: 0.45734, 0.0788124, 0.735513, 0.390724
1, tomato, p:0.998933 loc: 0.385497, 0.0432633, 0.51477, 0.22167
1, tomato, p:0.998886 loc: 0.520549, 0.384235, 0.654002, 0.552556
1, tomato, p:0.998507 loc: 0.295452, 0.452287, 0.422471, 0.588263
1, tomato, p:0.998191 loc: 0.181686, 0.568888, 0.295577, 0.712147
1, tomato, p:0.996489 loc: 0.386017, 0.250961, 0.504955, 0.408554
2, kiwi, p:0.991238 loc: 0.350772, 0.568207, 0.61708, 0.87927
Done`
验证过程参考:
`nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ls
CMakeCache.txt CMakeFiles cmake_install.cmake easyedge_batch_inference easyedge.log easyedge_multi_thread easyedge_serving install_manifest.txt Makefile
nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ./easyedge_serving /home/nano/Downloads/easydl-sdk/RES "E60A-5124-5ACD-3C9B" 0.0.0.0 24401
2020-08-24 03:56:17,802 WARNING [EasyEdge] 548017537040 Only compile_level 2 is supported for this model on JetPack 4.4 DP version. Please read documentation for the details
2020-08-24 03:56:17,802 INFO [EasyEdge] 548017537040 Compiling model for fast inference, this may take a while (Acceleration)
2020-08-24 03:57:34,965 INFO [EasyEdge] 548017537040 Optimized model saved to: /home/nano/.baidu/easyedge/jetson/mcache/26119049355/m_cache, Don't remove it
2020-08-24 03:57:34,968 INFO [EasyEdge] 548017537040 HTTP is now serving at 0.0.0.0:24401, holding 1 instances
2020-08-24 03:57:55,502 INFO [EasyEdge] 548000241392 [access_log] "GET /" 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
2020-08-24 03:57:55,582 INFO [EasyEdge] 548000241392 [access_log] "GET /api/info" 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
2020-08-24 03:58:31,103 INFO [EasyEdge] 546816874224 [access_log] "POST /" 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
6.3. 生成模型包
- 创建run.sh,该文件为容器的启动文件
#SDK序列号,需要在线申请,此处为测试序列号
license_key=E60A-5124-5ACD-3C9B
#curr_dir为/sdk,后续在BIE推断服务配置当中用到,作为容器内的工作目录
curr_dir=/sdk
demo_dir=${curr_dir}/cpp/sdk/demo/build
lib_dir=${curr_dir}/cpp/sdk/lib
res_dir=${curr_dir}/RES
export LD_LIBRARY_PATH=${lib_dir}:${LD_LIBRARY_PATH}
#run
${demo_dir}/easyedge_serving ${res_dir} ${license_key}
2.将cpp目录、RES目录,以及run.sh文件打包成一个fruit-ai-model.zip文件,并上传至BOS对象存储。
- 注意:此处是将多个目录和文件压缩为一个zip文件,而不是直接压缩最外层的那个目录。如果压缩层级不对,将导致模型下载到边缘设备以后解压的目录不匹配。
- 编译好的模型包参考:fruit-ai-model.zip