视频手部重建（HaWoR）
更新时间：2026-07-29
简介
视频 3D 手部重建算子，对视频重建双手 3D 位姿，并直接产出下游 HandActionCompute 可消费的 hand_pose_seq，闭合「人类视频 → 动作」链路。离线 GPU 运行。
功能描述
• 从视频均匀抽取 frame_num 帧，用 MoGe-2 估计 FoV 换算 img_focal，用 YOLO 检测器做手部检测与跟踪。
• 用 HaWoR 做逐帧 3D 手部重建，模型前向得到关节点，产出所选手（默认右手，缺则左手）逐帧 6D 位姿。
• 输出 hand_pose_seq（position/rotation/gripper 逐帧），rotation 为 3x3 展平(9)，gripper 为拇指-食指指尖距离（原始尺度）。
算子参数
输入
| 输入 | 含义 |
|---|---|
| video | 视频（本地 / bos / http，经 run_on_local_path 落地），单个输入列 |
输出
| 输出 | 含义 |
|---|---|
| result | struct{hand_pose_seq: list<struct{position: list<float64>, rotation: list<float64>, gripper: float64}>, hand: string, num_frames: int32, fov_x: float64} |
参数
| 参数名称 | 类型 | 默认值 | 描述 |
|---|---|---|---|
| hawor_model_path | str | "/opt/aihc/models" | HaWoR 权重根目录，与 hawor_model_name 拼接为 checkpoint 文件路径 |
| hawor_model_name | str | "HaWoR-weights/hawor/checkpoints/hawor.ckpt" | HaWoR checkpoint 相对路径 |
| hawor_config_path | str | "/opt/aihc/models" | HaWoR 配置根目录，与 hawor_config_name 拼接 |
| hawor_config_name | str | "HaWoR-weights/hawor/model_config.yaml" | HaWoR 配置文件相对路径 |
| hawor_detector_path | str | "/opt/aihc/models" | YOLO 检测器根目录，与 hawor_detector_name 拼接 |
| hawor_detector_name | str | "HaWoR-weights/external/detector.pt" | YOLO 检测器权重相对路径 |
| moge_model_path | str | "/opt/aihc/models" | MoGe-2 权重根目录，与 moge_model_name/model.pt 拼接 |
| moge_model_name | str | "moge-2-vitl" | MoGe-2 模型目录名 |
| hawor_repo_dir | str | "/opt/aihc/models/HaWoR-repo" | HaWoR 仓库目录，不存在时运行时 git clone |
| frame_num | int | 16 | 均匀采样帧数，须 >= 1 |
| thresh | float | 0.2 | 手部检测置信度阈值 |
| prefer_hand | str | "right" | 优先选取的手，可选 "right"/"left" |
| fail_on_error | bool | False | 单行出错是否立即抛异常，默认 False 返回空结果 |
调用示例
Python
1from __future__ import annotations
2
3import os
4
5import daft
6from daft import col
7
8from daft.aihc.common.udf import aihc_udf
9from daft.aihc.functions.video.video_hand_reconstruction_hawor import (
10 VideoHandReconstructionHawor,
11)
12
13
14VIDEO = "bos://your-bucket/sample.mp4" # MOCK
15
16MODEL_PATH = "/opt/aihc/models" # MOCK
17
18# HaWoR checkpoint
19HAWOR_MODEL_PATH = MODEL_PATH
20HAWOR_MODEL_NAME = "HaWoR-weights/hawor/checkpoints/hawor.ckpt"
21
22# HaWoR 配置
23HAWOR_CONFIG_PATH = MODEL_PATH
24HAWOR_CONFIG_NAME = "HaWoR-weights/hawor/model_config.yaml"
25
26# YOLO 手部检测器
27HAWOR_DETECTOR_PATH = MODEL_PATH
28HAWOR_DETECTOR_NAME = "HaWoR-weights/external/detector.pt"
29
30# MoGe-2
31MOGE_MODEL_PATH = MODEL_PATH
32MOGE_MODEL_NAME = "moge-2-vitl"
33
34MANO_RIGHT_PATH = f"{MODEL_PATH}/xxx.pkl"
35
36# HaWoR 源码仓库
37HAWOR_REPO_DIR = f"{MODEL_PATH}/HaWoR-repo"
38
39
40os.environ.setdefault("BOS_ENDPOINT", "http://bj.bcebos.com")
41os.environ.setdefault("BOS_REGION", "bj")
42
43
44if __name__ == "__main__":
45 if os.getenv("DAFT_RUNNER", "native") == "ray":
46 import ray
47
48 ray.init(ignore_reinit_error=True)
49 daft.set_runner_ray()
50
51 daft.set_execution_config(
52 actor_udf_ready_timeout=6000,
53 min_cpu_per_task=0,
54 )
55
56 construct_args = {
57 "hawor_model_path": HAWOR_MODEL_PATH,
58 "hawor_model_name": HAWOR_MODEL_NAME,
59 "hawor_config_path": HAWOR_CONFIG_PATH,
60 "hawor_config_name": HAWOR_CONFIG_NAME,
61 "hawor_detector_path": HAWOR_DETECTOR_PATH,
62 "hawor_detector_name": HAWOR_DETECTOR_NAME,
63 "moge_model_path": MOGE_MODEL_PATH,
64 "moge_model_name": MOGE_MODEL_NAME,
65 "hawor_repo_dir": HAWOR_REPO_DIR,
66 "frame_num": 16,
67 "thresh": 0.2,
68 "prefer_hand": "right",
69 "fail_on_error": False,
70 }
71
72 samples = {
73 "video": [VIDEO],
74 }
75
76 ds = daft.from_pydict(samples)
77
78 ds = ds.with_column(
79 "result",
80 aihc_udf(
81 VideoHandReconstructionHawor,
82 construct_args=construct_args,
83 num_cpus=1,
84 num_gpus=1,
85 concurrency=1,
86 batch_size=1,
87 )(col("video")),
88 )
89
90 ds.show()评价此篇文章