kube模式部署NVIDIA Jetson专用模型
所有文档

          智能边缘 BIE

          kube模式部署NVIDIA Jetson专用模型

          本文针对NVIDIA Jetson nano设备,介绍如何获取适配的端模型文件,并结合NVIDIA L4T Base镜像,将模型部署至BIE,完成AI图像识别。

          Setp1 利用easyedge获取适配jetson nano环境的SDK

          上传原始模型

          image.png

          模型提交后,在我的模型界面生成端模型。选择对应的芯片和操作系统,点击发布,获取SDK文件。

          image.png image.png

          Setp2 SDK编译和验证

          对SDK文件进行编译和认证,生成在jetson设备上可运行的模型。

          • 注意:使用EasyDL的Jetson系列SDK需要安装指定版本的JetPack和相关组件。目前所支持的JetPack版本包括:JetPack4.4和JetPack4.2.2

          模型资源文件默认已经打包在下载的SDK包中。Demo工程直接编译即可运行。

          cd demo
          mkdir build && cd build
          cmake .. && make
          sudo ./easyedge_demo {模型RES文件夹}  {测试图片路径} {序列号}

          编译过程参考:

          `nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ls
          CMakeCache.txt  CMakeFiles  cmake_install.cmake  easyedge_batch_inference  easyedge.log  easyedge_multi_thread  easyedge_serving  install_manifest.txt  Makefile
          nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ./easyedge_batch_inference /home/nano/Downloads/easydl-sdk/RES /home/nano/Downloads/easydl-sdk/imgs/mix008.jpeg
          2020-08-24 03:51:09,915 WARNING [EasyEdge] 548102246416 Only compile_level 2 is supported for this model on JetPack 4.4 DP version. Please read documentation for the details
          2020-08-24 03:51:09,915 INFO [EasyEdge] 548102246416 Compiling model for fast inference, this may take a while (Acceleration)
          2020-08-24 03:53:08,488 INFO [EasyEdge] 548102246416 Optimized model saved to: /home/nano/.baidu/easyedge/jetson/mcache/26119049355/m_cache, Don't remove it
          Results of image /home/nano/Downloads/easydl-sdk/imgs/mix008.jpeg:
          1, tomato, p:0.999717 loc: 0.672033, 0.405379, 0.80056, 0.569352
          2, kiwi, p:0.999273 loc: 0.154121, 0.0717932, 0.393312, 0.399547
          2, kiwi, p:0.999206 loc: 0.45734, 0.0788124, 0.735513, 0.390724
          1, tomato, p:0.998933 loc: 0.385497, 0.0432633, 0.51477, 0.22167
          1, tomato, p:0.998886 loc: 0.520549, 0.384235, 0.654002, 0.552556
          1, tomato, p:0.998507 loc: 0.295452, 0.452287, 0.422471, 0.588263
          1, tomato, p:0.998191 loc: 0.181686, 0.568888, 0.295577, 0.712147
          1, tomato, p:0.996489 loc: 0.386017, 0.250961, 0.504955, 0.408554
          2, kiwi, p:0.991238 loc: 0.350772, 0.568207, 0.61708, 0.87927
          Done`

          验证过程参考:

          `nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ls
          CMakeCache.txt  CMakeFiles  cmake_install.cmake  easyedge_batch_inference  easyedge.log  easyedge_multi_thread  easyedge_serving  install_manifest.txt  Makefile
          nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ./easyedge_serving /home/nano/Downloads/easydl-sdk/RES "E60A-5124-5ACD-3C9B" 0.0.0.0 24401
          2020-08-24 03:56:17,802 WARNING [EasyEdge] 548017537040 Only compile_level 2 is supported for this model on JetPack 4.4 DP version. Please read documentation for the details
          2020-08-24 03:56:17,802 INFO [EasyEdge] 548017537040 Compiling model for fast inference, this may take a while (Acceleration)
          2020-08-24 03:57:34,965 INFO [EasyEdge] 548017537040 Optimized model saved to: /home/nano/.baidu/easyedge/jetson/mcache/26119049355/m_cache, Don't remove it
          2020-08-24 03:57:34,968 INFO [EasyEdge] 548017537040 HTTP is now serving at 0.0.0.0:24401, holding 1 instances
          2020-08-24 03:57:55,502 INFO [EasyEdge] 548000241392 [access_log] "GET /" 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
          2020-08-24 03:57:55,582 INFO [EasyEdge] 548000241392 [access_log] "GET /api/info" 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
          2020-08-24 03:58:31,103 INFO [EasyEdge] 546816874224 [access_log] "POST /" 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"

          Step3 生成模型包并上传至对象存储

          1. 创建run.sh。该文件为容器的启动文件`

          #SDK序列号,需要在线申请,此处为测试序列号
          license_key=E60A-5124-5ACD-3C9B
          #curr_dir为/sdk,后续在BIE推断服务配置当中用到,作为容器内的工作目录
          curr_dir=/sdk
          demo_dir=${curr_dir}/cpp/sdk/demo/build
          lib_dir=${curr_dir}/cpp/sdk/lib
          res_dir=${curr_dir}/RES
          export LD_LIBRARY_PATH=${lib_dir}:${LD_LIBRARY_PATH}
          #run
          ${demo_dir}/easyedge_serving ${res_dir} ${license_key}

          2.将cpp目录、RES目录,以及run.sh文件打包成一个easyedge-fruit-model.zip文件,并上传至BOS对象存储。

          • 注意:此处是将多个目录和文件压缩为一个zip文件,而不是直接压缩最外层的那个目录。如果压缩层级不对,将导致模型下载到边缘设备以后解压的目录不匹配。
          • 编译好的模型包参考:fruit-ai-model.zip

          Step4 将模型下发至边缘

          1. 创建边缘节点

          如下图所示,创建边缘节点jetson-nano

          image.png

          2. 创建AI模型配置项

          如下图所示,创建配置项:fruit-demo。引入上传至BOS对象存储的模型文件。

          image.png

          3. 创建AI推断服务

          如下图所示,创建应用fruit-detection

          image.png

          在应用当中添加一个容器服务,如下图所示:

          image.png

          image.png

          • 镜像地址:nvidia提供了一个面向NVIDIA Jetson系列的容器景象:NVIDIA L4T Base。nvidia官网获取方式:https://ngc.nvidia.com/catalog/containers/nvidia:l4t-base。 该容器镜像提供nvidia jetson的运行环境,能够实现在容器内跑支持jetson设备的AI模型服务。在支持容器化以后,可以通过BIE来管理边缘节点设备,以及提升模型服务部署的效率。
          官网镜像获取地址:nvcr.io/nvidia/l4t-base:r32.4.3
          若国内下载速度很慢,可以从百度cce上下载镜像:hub.baidubce.com/nvidia/l4t-base:r32.4.3-arm64
          • 卷配置:类型为配置项;参数选择之前创建的文件配置项;容器目录:/sdk(与run.sh文件对应)。通过卷配置,实现AI推断服务与模型文件绑定。将推断服务下发至边缘节点的时候,会自动下载AI模型文件。
          • 容器服务的端口映射:端口为24401;协议为TCP
          • 容器服务的启动参数:bash;/sdk/run.sh
          • 无需开启特权模式

          4. 将应用部署至节点

          在边缘应用当中,添加目标节点标签

          image.png

          5. 在jetson nano设备上安装节点

          5.1 执行docker info查看边缘节点容器环境,确保Default Runtime: nvidia

          nano@jetson-nano:~/Downloads/fruit-ai-model$ docker info
          Client:
           Debug Mode: false
           
          Server:
           Containers: 38
            Running: 21
            Paused: 0
            Stopped: 17
           Images: 12
           Server Version: 19.03.6
           Storage Driver: overlay2
            Backing Filesystem: extfs
            Supports d_type: true
            Native Overlay Diff: true
           Logging Driver: json-file
           Cgroup Driver: cgroupfs
           Plugins:
            Volume: local
            Network: bridge host ipvlan macvlan null overlay
            Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
           Swarm: inactive
           Runtimes: nvidia runc
          ** Default Runtime: nvidia**
           Init Binary: docker-init
           containerd version:
           runc version:
           init version:
           Security Options:
            seccomp
             Profile: default
           Kernel Version: 4.9.140-tegra
           Operating System: Ubuntu 18.04.5 LTS
           OSType: linux
           Architecture: aarch64
           CPUs: 4
           Total Memory: 3.871GiB
           Name: jetson-nano
           ID: O7GP:DDD5:5CIR:LEWJ:2BQ3:4WIW:VA4H:JDCP:5VGL:L2K3:PLZ7:KBHO
           Docker Root Dir: /var/lib/docker
           Debug Mode: false
           Registry: https://index.docker.io/v1/
           Labels:
           Experimental: false
           Insecure Registries:
            127.0.0.0/8
           Live Restore Enabled: false

          如果Default Runtime不是nvidia,修改/etc/docker/daemon.json文件,添加"default-runtime": "nvidia",修改完毕以后的/etc/docker/daemon.json文件

          nano@jetson-nano:~$ cat /etc/docker/daemon.json
          {
              "default-runtime": "nvidia",
              "runtimes": {
                  "nvidia": {
                      "path": "nvidia-container-runtime",
                      "runtimeArgs": []
                  }
              }
          }

          修改完毕以后,重启docker

          sudo systemctl restart docker

          5.2 运行安装命令

          image.png

          Step 5 验证边缘节点AI推断服务

          通过浏览器打开在线推断服务:http://「ip」 :24401/,上传测试图片,推断结果如下,证明AI服务正常启动。

          image.png

          上一篇
          部署通用CPU模型
          下一篇
          进程模式部署NVIDIA Jetson专用模型