How to Integrate Hand Tracking Data Into Your Hand Model

Overview

1. OpenXR Hand Tracking Plugin Setup

Supported Unity Engine version: 2020.2+

1.1 Enable OpenXR Plugins

Please enable OpenXR plugin in Edit > Project Settings > XR Plug-in Management:

Click Exclamation mark next to “OpenXR” then choose “Fix All”.

Add Interaction profiles for your device. (As following, taking Vive Controller as an example.)

1.2 Enable Hand Tracking extensions

2. Game objects settings

2.1 VR render camera settings

To display your hand in the correct position, add “TrackedPoseDriver” script to your VR render camera.

2.2 Add hand game objects

2.2.1 Skeleton hands

Create two empty game objects for each hand. We name it “Left” and “Right” here.

2.2.2 3D hands

Use the 3D model provided by our HandTracking sample as an example:

Assets > Samples > VIVE Wave OpenXR Plugin - Windows > {version} > HandTracking Example > Materials

3. Scripts for skeleton hands

Create new script and attach it to left or right hand game objects.

Add following namespaces to your script.

using VIVE.HandTracking;

Add following properties:

[Tooltip("Draw left hand if true, right hand otherwise")]
public bool isLeft = false;
[Tooltip("Use inferred or last-known posed when hand loses tracking if true.")]
public bool allowUntrackedPose = false;
private XrHandJointLocationEXT[] HandjointLocations = new XrHandJointLocationEXT[(int)XrHandJointEXT.XR_HAND_JOINT_MAX_ENUM_EXT];

3.1 Add scripts to Start function

Step 1: Start hand tracking detection:

HandManager.StartFrameWork(isLeft);

Step 2: For skeleton hand: Create game objects for 26 joints and links between joints.

3.2 Add scripts to Update function

Step 1: Get hand tracking detection result:

if(HandManager.GetJointLocation(isLeft, out HandjointLocations))
{
    UpdateJointLocation();//Refer to step2 below to update your hand model here.
}
else
{
    //Hide your hand model due to not detect the hand input 
}

Step 2: Update skeleton hand model:

if(allowUntrackedPose) //Use inferred or last-known pose when lost tracking 
{
    if ((HandjointLocations[i].locationFlags & (ulong)XrSpaceLocationFlags.XR_SPACE_LOCATION_POSITION_VALID_BIT) != 0)
    {
        position = HandjointLocations[i].pose.position;
        go.transform.localPosition = new Vector3(position.x, position.y, -position.z);
        go.SetActive(true);
    }
    else
    {
        go.SetActive(false);
    }
}

Note :

1. The skeleton model only takes the position information of each joint but no rotation information.
2. OpenXR uses a right-handed coordinate system, while Unity uses a left-handed coordinate system, so the z value of position needs to be flipped.

2. Update links in skeleton model.

Calculate link position and rotation based on points on both ends.

3.3 Add scripts to OnDestroy function

Stop hand tracking detection:

HandManager.StopFrameWork(isLeft);

4. Scripts for 3D hands

For Start function:

Please refer to step 1 in Section 3.1.

For Update function:

1. Please refer to step 1 in Section 3.2.
2. Update 3D hand model:

Step1: Update the position and orientation of wrist joint.

position = HandjointLocations[1].pose.position;
orientation = new Quaternion(
        -1 * (HandjointLocations[1].pose.orientation.x),
        1 * (HandjointLocations[1].pose.orientation.y),
        -1 * HandjointLocations[1].pose.orientation.z,
        1 * HandjointLocations[1].pose.orientation.w);
nodes[1].transform.localPosition = new Vector3(position.x, position.y, -position.z);
nodes[1].transform.localRotation = (orientation);
nodes[1].transform.Rotate(new Vector3(180.0f,0.0f,0.0f), Space.World);

Note :

1. 3D model uses wrist's position information to determine the position of the entire hand in space, and then uses the rotation information of each joint point to determine the shape of the hand in space.
2. OpenXR uses a right-handed coordinate system, while Unity uses a left-handed coordinate system, so the z value of position, x value of orientation and z value of orientation need to be flipped.
3. The 3D model needs to flip 180 degrees to adjust the initial position.

Step2: Update rotations of other points.

if (allowUntrackedPose) //Use inferred or last-known pose when lost tracking 
{
    if ((HandjointLocations[i].locationFlags & (ulong)XrSpaceLocationFlags.XR_SPACE_LOCATION_ORIENTATION_VALID_BIT) != 0)
    {
        orientation = new Quaternion(
            -1 * (HandjointLocations[i].pose.orientation.x),
            1 * (HandjointLocations[i].pose.orientation.y),
            -1 * HandjointLocations[i].pose.orientation.z,
            1 * HandjointLocations[i].pose.orientation.w);
        nodes[i].transform.rotation = orientation;
        nodes[i].transform.Rotate(new Vector3(180.0f, 0.0f, 0.0f), Space.World);
    }
}
else
{
    if ((HandjointLocations[i].locationFlags & (ulong)XrSpaceLocationFlags.XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT) != 0)
    {
        orientation = new Quaternion(
            -1 * (HandjointLocations[i].pose.orientation.x),
            1 * (HandjointLocations[i].pose.orientation.y),
            -1 * HandjointLocations[i].pose.orientation.z,
            1 * HandjointLocations[i].pose.orientation.w);
        nodes[i].transform.rotation = orientation;
        nodes[i].transform.Rotate(new Vector3(180.0f, 0.0f, 0.0f), Space.World);
    }
}

For OnDestroy function:

Please refer to Section 3.3.

5. Results