推理说明

更新时间：2024-12-10

使用说明

准备工作

为了客户快速POC，AIAK提前缓存了模型权重、测试数据集，用户可以直接使用快速验证。

前置检查：使用命令查看CPU模式cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor，确保Mode策略使用performance进行测试，否则会影响性能！如果策略为其他类型，使用如下命令进行修改：

cpupower idle-set -d 2 > /dev/null
cpupower idle-set -d 3 > /dev/null
cpupower frequency-set -g performance > /dev/null

目前模型权重文件以及测试数据集缓存在对象存储BOS上，请选择百度云任一机器，执行以下命令安装BOS命令行工具：

# 下载bcecmd程序
wget https://doc.bce.baidu.com/bce-documentation/BOS/linux-bcecmd-0.4.5.zip
# 解压
unzip linux-bcecmd-0.4.5.zip
cd linux-bcecmd-0.4.5

镜像&权重&数据准备

拉取镜像

docker pull registry.baidubce.com/aihc-aiak/aiak-inference-llm:ubuntu22.04-cu12.3-torch2.4.0-py310_2.0.2.2

下载模型权重（非量化&量化）

./linux-bcecmd-0.4.5/bcecmd bos【模型链接】【本地路径】
# 例如下载30B模型
./linux-bcecmd-0.4.5/bcecmd bos sync bos://cce-ai-datasets/huggyllama/llama-30b /mnt/pfs/meta-llama/llama-30b/

模型	地址
llama2-7b-hf	bos://cce-ai-datasets/huggingface.co/meta-llama/Llama-2-7b-hf
llama2-7b-sq	bos://cce-ai-datasets/aiak_quanted_models/smooth_quant/llama2-7b/1-gpu/
llama-30b-hf	bos://cce-ai-datasets/huggyllama/llama-30b
llama-30b-sq(1-gpu)	bos://cce-ai-datasets/aiak_quanted_models/smooth_quant/llama-30b/1-gpu/
llama-30b-sq(4-gpu)	bos://cce-ai-datasets/aiak_quanted_models/smooth_quant/llama-30b/4-gpu/
llama-2-13b-hf	bos://cce-ai-datasets/huggingface.co/meta-llama/Llama-2-13b-hf
llama-2-13b-sq	bos://cce-ai-datasets/aiak_quanted_models/smooth_quant/llama2-13b/1-gpu/
llama-2-70b-hf	bos://cce-ai-datasets/huggingface.co/meta-llama/Llama-2-70b-hf
llama-2-70b-sq	bos://cce-ai-datasets/aiak_quanted_models/smooth_quant/llama2-70b/8-gpu/
llama-3-8b-hf	bos://cce-ai-datasets/huggingface.co/meta-llama/Meta-Llama-3-8B/
llama3-8b-sq	bos://cce-ai-datasets/aiak_quanted_models/smooth_quant/Meta-Llama-3-8B/1-gpu/
llama-3-70b-hf	bos://cce-ai-datasets/huggingface.co/meta-llama/Meta-Llama-3-70B/
llama3-70b-sq	bos://cce-ai-datasets/aiak_quanted_models/smooth_quant/Meta-Llama-3-70B
qwen2-72b-Instruct	bos:/cce-ai-models/huggingface.co/Qwen/Qwen2-72B-Instruct
qwen2-72b-sq	bos:/cce-ai-datasets/hac-aiacc/aiak_quanted_models/qwen2-72b-instruct/smooth_quant/8-gpu

下载测试数据集

./linux-bcecmd-0.4.5/bcecmd bos cp bos://cce-ai-datasets/ShareGPT_V3_unfiltered_cleaned_split.json  /mnt/pfs/data/ShareGPT_V3_unfiltered_cleaned_split.json

部署推理实例的服务端

需要规模使用时，请使用标准K8S模式发起部署

#启动docker
docker run --gpus all -itd --name aiak2.0.1_test  --shm-size=32768m --privileged --user=root --network host -v /PATH/TO/MODEL:/mnt/model registry.baidubce.com/aihc-aiak/aiak-inference-llm:ubuntu22.04-cu12.3-torch2.4.0-py310_2.0.2.2 /bin/bash
#进入docker
docker exec -it infer_test bash

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aiak-inference-test
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: aiak-inference-test
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: aiak-inference-test
    spec:
      containers:
        - command:
            - /bin/bash
            - '-c'
            - "trap 'kill $(jobs -p)' EXIT; sleep 3d"
          image: 'registry.baidubce.com/aihc-aiak/aiak-inference-llm:ubuntu22.04-cu12.3-torch2.4.0-py310_2.0.2.2'
          imagePullPolicy: Always
          name: vllm
          resources:
            limits:
              baidu.com/a800_80g_cgpu: "8"
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /usr/bin/docker
            name: docker
          - mountPath: /var/run/docker.sock
            name: docker-sock
          - mountPath: /mnt/pfs
            name: pfs
          - mountPath: /dev/shm
            name: cache-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: volcano
      securityContext: {}
      terminationGracePeriodSeconds: 60
      volumes:
      - hostPath:
          path: /usr/bin/docker
          type: ''
        name: docker

详细参数查阅《参数说明-服务端主要参数》章节

使用内置量化工具（可选）

进入docker服务的命令行进行操作，量化时间与模型大小和卡数有关，需要十几分钟到1小时左右

# 进入量化工具所在目录
cd /workspace/aiak-model-quant-tool/
# 示例
weight_only_int8量化
python3 model_quantization.py -i /input_path/ -o ./output_path/ -tp 1 -quant_type weight_only_int8 -t fp16
smooth-quant量化默认校准数据集
python3 model_quantization.py -i /input_path/ -o ./output_path/ -tp 1 -quant_type smooth_quant -t fp16 -sq 0.75
smooth-quant量化自定义校准数据集
python3 model_quantization.py -i /input_path/ -o ./output_path/ -tp 1 -quant_type smooth_quant -t fp16 -sq 0.75 --tokenids_path /XX/path_to/smooth_tokenids.txt

详细参数查阅《参数说明-内置量化工具参数》章节

量化输出示例

root@A800:/workspace/aiak-model-quant-tool# python3 model_quantization.py -i /input_path/ -o /output_path/  -quant_type weight_only_int8  -tp 2
=============== Argument ===============
out_dir: /Qwen-14B/
in_file: /Qwen-14B/
tensor_parallelism: 2
model_quantization_type: weight_only_int8
multi_query_mode: False
========================================
INFO 12-01 08:30:18 model_quantization.py:35] quantization an LLM engine model_config:QWenConfig {
INFO 12-01 08:30:18 model_quantization.py:35]   "architectures": [
INFO 12-01 08:30:18 model_quantization.py:35]     "QWenLMHeadModel"
INFO 12-01 08:30:18 model_quantization.py:35]   ],
INFO 12-01 08:30:18 model_quantization.py:35]   "attn_dropout_prob": 0.0,
INFO 12-01 08:30:18 model_quantization.py:35]   "auto_map": {
INFO 12-01 08:30:18 model_quantization.py:35]     "AutoConfig": "configuration_qwen.QWenConfig",
INFO 12-01 08:30:18 model_quantization.py:35]     "AutoModelForCausalLM": "modeling_qwen.QWenLMHeadModel"
INFO 12-01 08:30:18 model_quantization.py:35]   },
INFO 12-01 08:30:18 model_quantization.py:35]   "bf16": false,
INFO 12-01 08:30:18 model_quantization.py:35]   "emb_dropout_prob": 0.0,
INFO 12-01 08:30:18 model_quantization.py:35]   "fp16": false,
INFO 12-01 08:30:18 model_quantization.py:35]   "fp32": false,
INFO 12-01 08:30:18 model_quantization.py:35]   "hidden_size": 5120,
INFO 12-01 08:30:18 model_quantization.py:35]   "initializer_range": 0.02,
INFO 12-01 08:30:18 model_quantization.py:35]   "intermediate_size": 27392,
INFO 12-01 08:30:18 model_quantization.py:35]   "kv_channels": 128,
INFO 12-01 08:30:18 model_quantization.py:35]   "layer_norm_epsilon": 1e-06,
INFO 12-01 08:30:18 model_quantization.py:35]   "max_position_embeddings": 8192,
INFO 12-01 08:30:18 model_quantization.py:35]   "model_type": "qwen",
INFO 12-01 08:30:18 model_quantization.py:35]   "no_bias": true,
INFO 12-01 08:30:18 model_quantization.py:35]   "num_attention_heads": 40,
INFO 12-01 08:30:18 model_quantization.py:35]   "num_hidden_layers": 40,
INFO 12-01 08:30:18 model_quantization.py:35]   "onnx_safe": null,
INFO 12-01 08:30:18 model_quantization.py:35]   "rotary_emb_base": 10000,
INFO 12-01 08:30:18 model_quantization.py:35]   "rotary_pct": 1.0,
INFO 12-01 08:30:18 model_quantization.py:35]   "scale_attn_weights": true,
INFO 12-01 08:30:18 model_quantization.py:35]   "seq_length": 2048,
INFO 12-01 08:30:18 model_quantization.py:35]   "tie_word_embeddings": false,
INFO 12-01 08:30:18 model_quantization.py:35]   "tokenizer_class": "QWenTokenizer",
INFO 12-01 08:30:18 model_quantization.py:35]   "transformers_version": "4.34.0",
INFO 12-01 08:30:18 model_quantization.py:35]   "use_cache": true,
INFO 12-01 08:30:18 model_quantization.py:35]   "use_dynamic_ntk": true,
INFO 12-01 08:30:18 model_quantization.py:35]   "use_flash_attn": "auto",
INFO 12-01 08:30:18 model_quantization.py:35]   "use_logn_attn": true,
INFO 12-01 08:30:18 model_quantization.py:35]   "vocab_size": 152064
INFO 12-01 08:30:18 model_quantization.py:35] }
INFO 12-01 08:30:18 model_quantization.py:35]  quantization tp_size is: 2, 
INFO 12-01 08:33:29 model_quantization.py:35] ==quantization= state_dict===key: transformer.h.0.attn.c_proj.qscale ==value.shape: torch.Size([2, 5120])
INFO 12-01 08:33:29 model_quantization.py:35] ==quantization= state_dict===key: transformer.h.0.attn.c_proj.qweight ==value.shape: torch.Size([5120, 1280])
INFO 12-01 08:33:29 model_quantization.py:35] ==quantization= state_dict===key: transformer.h.0.ln_1.weight ==value.shape: torch.Size([5120])
INFO 12-01 08:33:29 model_quantization.py:35] ==quantization= state_dict===key: transformer.h.0.ln_2.weight ==value.shape: torch.Size([5120])
INFO 12-01 08:33:29 model_quantization.py:35] ==quantization= state_dict===key: transformer.wte.weight ==value.shape: torch.Size([152064, 5120])
('json_quant_config: ', {'w_bit': 8})
……
INFO 12-01 08:33:54 model_quantization.py:35] Quantization model save path: /output_path/2-gpu

启动推理服务的服务端

支持直接以入参形式来部署容器运行

python3 -m vllm.entrypoints.openai.api_server --model=<MODEL_PATH> --tensor-parallel-size 1 --dtype=float16 --port 8888 --enable-prefix-caching

详细参数查阅《参数说明-服务端主要参数》章节；

目前只支持huggingface的checkpoint类型，如果需要使用Megatron的checkpoint，需要先将其转换为HF格式。

测试推理服务的客户端

使用curl命令进行功能单测

流式请求

客户端发送请求时，stream设置为True。

确保启动服务端，如未启动服务端，可使用以下命令启动

python3 -m vllm.entrypoints.openai.api_server --model=/path/to/your/model/ --tensor-parallel-size=1 --dtype=float16 --gpu_memory_utilization=0.8

客户端请求示例

curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
    "model": "/path/to/your/model/",
    "prompt": ["中国的首都是"],"stream":"True"               
}'

注意：详细参数查阅《参数说明-客户端输入参数》章节；

得到返回

非流式请求

客户端发送请求时，stream设置为False，或者不设置（默认值为False）

确保启动服务端，如未启动服务端，可使用以下命令启动

python3 -m vllm.entrypoints.openai.api_server --model=/path/to/your/model/ --tensor-parallel-size=1 --dtype=float16 --gpu_memory_utilization=0.8

客户端请求示例

curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
    "model": "/path/to/your/model/",
    "prompt": ["中国的首都是"],"stream":"False"               
}'