Please Select Your Location
Australia
Österreich
België
Canada
Canada - Français
中国
Česká republika
Denmark
Deutschland
France
HongKong
Iceland
Italia
日本
Korea
Latvija
Lietuva
Lëtzebuerg
Malta
المملكة العربية السعودية (Arabic)
Nederland
New Zealand
Norge
Polska
Portugal
Russia
Saudi Arabia
Southeast Asia
España
Suisse
Suomi
Sverige
台灣
Ukraine
United Kingdom
United States
Select your country to buy
België
Česká republika
Denmark
Iceland
Italia
Latvija
Lietuva
Lëtzebuerg
Malta
Nederland
Norge
Polska
Portugal
España
Suisse
Suomi
Sverige

How to Integrate Hand Tracking Data Into Your Hand Model

Overview

1. OpenXR Hand Tracking Plugin Setup.
1.1 Enable OpenXR Plugins.
1.2 Enable Hand Tracking extensions.
2. Game objects settings.
2.1 VR render camera settings.
2.2 Add hand game objects.
2.2.1 Skeleton hands.
2.2.2 3D hands.
3. Scripts for skeleton hands.
3.1 Add scripts to Start function.
3.2 Add scripts to Update function.
3.3 Add scripts to OnDestroy function.
4. Scripts for 3D hands.
5. Results.

1. OpenXR Hand Tracking Plugin Setup

Supported Unity Engine version: 2020.2+

1.1 Enable OpenXR Plugins

Please enable OpenXR plugin in Edit > Project Settings > XR Plug-in Management:

image1.png

Click Exclamation mark next to “OpenXR” then choose “Fix All”.

image2.png

image3.png

Add Interaction profiles for your device. (As following, taking Vive Controller as an example.)

image4.png


1.2 Enable Hand Tracking extensions

image5.png

2. Game objects settings

2.1 VR render camera settings

To display your hand in the correct position, add “TrackedPoseDriver” script to your VR render camera.

image6.png

2.2 Add hand game objects

2.2.1 Skeleton hands

Create two empty game objects for each hand. We name it “Left” and “Right” here.

image7.png

2.2.2 3D hands

Use the 3D model provided by our HandTracking sample as an example:

Assets > Samples > VIVE Wave OpenXR Plugin - Windows > {version} > HandTracking Example > Materials

image8.png

3. Scripts for skeleton hands

Create new script and attach it to left or right hand game objects.

Add following namespaces to your script.

using VIVE.HandTracking;

Add following properties:

[Tooltip("Draw left hand if true, right hand otherwise")]
public bool isLeft = false;
[Tooltip("Use inferred or last-known posed when hand loses tracking if true.")]
public bool allowUntrackedPose = false;
private XrHandJointLocationEXT[] HandjointLocations = new XrHandJointLocationEXT[(int)XrHandJointEXT.XR_HAND_JOINT_MAX_ENUM_EXT];

3.1 Add scripts to Start function

Step 1: Start hand tracking detection:

HandManager.StartFrameWork(isLeft);

Step 2: For skeleton hand: Create game objects for 26 joints and links between joints.

3.2 Add scripts to Update function

Step 1: Get hand tracking detection result:

if(HandManager.GetJointLocation(isLeft, out HandjointLocations))
{
    UpdateJointLocation();//Refer to step2 below to update your hand model here.
}
else
{
    //Hide your hand model due to not detect the hand input 
}

Step 2: Update skeleton hand model:

if(allowUntrackedPose) //Use inferred or last-known pose when lost tracking 
{
    if ((HandjointLocations[i].locationFlags & (ulong)XrSpaceLocationFlags.XR_SPACE_LOCATION_POSITION_VALID_BIT) != 0)
    {
        position = HandjointLocations[i].pose.position;
        go.transform.localPosition = new Vector3(position.x, position.y, -position.z);
        go.SetActive(true);
    }
    else
    {
        go.SetActive(false);
    }
}

Note :

  1. The skeleton model only takes the position information of each joint but no rotation information.
  2. OpenXR uses a right-handed coordinate system, while Unity uses a left-handed coordinate system, so the z value of position needs to be flipped.

  1. Update links in skeleton model.

Calculate link position and rotation based on points on both ends.

3.3 Add scripts to OnDestroy function

Stop hand tracking detection:

HandManager.StopFrameWork(isLeft);

4. Scripts for 3D hands

For Start function:

Please refer to step 1 in Section 3.1.

For Update function:

  1. Please refer to step 1 in Section 3.2.
  2. Update 3D hand model:

Step1: Update the position and orientation of wrist joint.

position = HandjointLocations[1].pose.position;
orientation = new Quaternion(
        -1 * (HandjointLocations[1].pose.orientation.x),
        1 * (HandjointLocations[1].pose.orientation.y),
        -1 * HandjointLocations[1].pose.orientation.z,
        1 * HandjointLocations[1].pose.orientation.w);
nodes[1].transform.localPosition = new Vector3(position.x, position.y, -position.z);
nodes[1].transform.localRotation = (orientation);
nodes[1].transform.Rotate(new Vector3(180.0f,0.0f,0.0f), Space.World);

Note :

  1. 3D model uses wrist's position information to determine the position of the entire hand in space, and then uses the rotation information of each joint point to determine the shape of the hand in space.
  2. OpenXR uses a right-handed coordinate system, while Unity uses a left-handed coordinate system, so the z value of position, x value of orientation and z value of orientation need to be flipped.
  3. The 3D model needs to flip 180 degrees to adjust the initial position.

Step2: Update rotations of other points.

if (allowUntrackedPose) //Use inferred or last-known pose when lost tracking 
{
    if ((HandjointLocations[i].locationFlags & (ulong)XrSpaceLocationFlags.XR_SPACE_LOCATION_ORIENTATION_VALID_BIT) != 0)
    {
        orientation = new Quaternion(
            -1 * (HandjointLocations[i].pose.orientation.x),
            1 * (HandjointLocations[i].pose.orientation.y),
            -1 * HandjointLocations[i].pose.orientation.z,
            1 * HandjointLocations[i].pose.orientation.w);
        nodes[i].transform.rotation = orientation;
        nodes[i].transform.Rotate(new Vector3(180.0f, 0.0f, 0.0f), Space.World);
    }
}
else
{
    if ((HandjointLocations[i].locationFlags & (ulong)XrSpaceLocationFlags.XR_SPACE_LOCATION_ORIENTATION_TRACKED_BIT) != 0)
    {
        orientation = new Quaternion(
            -1 * (HandjointLocations[i].pose.orientation.x),
            1 * (HandjointLocations[i].pose.orientation.y),
            -1 * HandjointLocations[i].pose.orientation.z,
            1 * HandjointLocations[i].pose.orientation.w);
        nodes[i].transform.rotation = orientation;
        nodes[i].transform.Rotate(new Vector3(180.0f, 0.0f, 0.0f), Space.World);
    }
}

For OnDestroy function:

Please refer to Section 3.3.

5. Results

Skeleton Model

3D Model

image9.png

image10.png