How to Integrate Hand Tracking Data Into Your Hand Model

OpenXR Hand Tracking Plugin Setup

Supported Unity Engine version: 2020.2+

Enable OpenXR Plugins:
- Please enable OpenXR plugin in Edit > Project Settings > XR Plug-in Management:

Click Exclamation mark next to “OpenXR” then choose “Fix All”.

Add Interaction profiles for your device.

As following, take Vive Controller as an example.

Enable Hand Tracking extensions

To draw skeleton or 3D hand model in your App.

- To display your hand in the correct position, add “TrackedPoseDriver” script to your VR render camera.

Add hand Gameobjects:
- For skeleton hand: Create two empty GameObjects for each hand. We name it “Left” and “Right” here.

For 3D hand(Use the 3D model provided by our HandTracking sample as an example):

Assets > Samples > VIVE Wave OpenXR Plugin - Windows > 1.0.4 > HandTracking Example > Materials

Create new script and attach it to left or right hand game objects.

Add this line to your script.

using UnityEngine.XR.OpenXR;

In Start function:

To Start hand tracking detection:

//left hand:true ,right hand:false
bool isLeft = true;
var feature = OpenXRSettings.Instance.GetFeature();
XrHandTrackerCreateInfoEXT createInfo = new XrHandTrackerCreateInfoEXT(
XrStructureType.XR_TYPE_HAND_TRACKER_CREATE_INFO_EXT,
        IntPtr.Zero,
        XrHandEXT.XR_HAND_LEFT_EXT,
        XrHandJointSetEXT.XR_HAND_JOINT_SET_DEFAULT_EXT);
int res;
if (isLeft)
{
	res = feature.xrCreateHandTrackerEXT(createInfo, out feature.m_leftHandle);
}
else
{
	createInfo.hand = XrHandEXT.XR_HAND_RIGHT_EXT;
	res = feature.xrCreateHandTrackerEXT(createInfo, out feature.m_rightHandle);
}
if (res != (int)XrResult.XR_SUCCESS)
{
	UnityEngine.Debug.LogError("Failed to create hand tracker with error code " + res);
}

For skeleton hand:
- Create game objects for 26 joints.
- Create game objects for links between joints.

In Update function:

Check if the hand tracking detection result is available.

var feature = OpenXRSettings.Instance.GetFeature();      
if (feature.m_leftHandle == ulong.MinValue && feature.m_rightHandle == ulong.MinValue)
	return;

Get hand tracking detection result:

//left hand:true ,right hand:false
bool isLeft = true; 
XrHandJointLocationEXT[] HandjointLocations = new XrHandJointLocationEXT[(int)XrHandJointEXT.XR_HAND_JOINT_MAX_ENUM_EXT];
var feature = OpenXRSettings.Instance.GetFeature();
XrHandJointsLocateInfoEXT locateInfo = new XrHandJointsLocateInfoEXT(
	XrStructureType.XR_TYPE_HAND_JOINTS_LOCATE_INFO_EXT,
	IntPtr.Zero,
	feature.m_space,
	(Int64)10 //An arbitrary number greater than 0
);
unsafe
{
  fixed (XrHandJointLocationEXT* ptr = HandjointLocations)
  {
  locations.type = XrStructureType.XR_TYPE_HAND_JOINT_LOCATIONS_EXT;
  locations.next = IntPtr.Zero;
     locations.isActive = 0;
  locations.jointCount = (int)XrHandJointEXT.XR_HAND_JOINT_MAX_ENUM_EXT;
      locations.jointLocations = (IntPtr)ptr;
  int res;
  if(isleftHand)
     {
      res = feature.xrLocateHandJointsEXT(feature.m_leftHandle, locateInfo, ref locations);
  	}
  else
     {
      res = feature.xrLocateHandJointsEXT(feature.m_rightHandle, locateInfo, ref locations);
      }
  	}
}
if(locations.isActive == 1)
{
	UpdateJointLocation();//Update your hand model here.
}
else
{
	//Hide your hand model due to not detect the hand input 
}

Update skeleton hand model:

The skeleton model only takes the position information of each joint but no rotation information.

Note :

OpenXR uses a right-handed coordinate system, while Unity uses a left-handed coordinate system, so the z value of position needs to be flipped.

Update links in skeleton model.
- Calculate link position and rotation based on points on both ends.
Update 3D hand model:

3D model uses wrist's position information to determine the position of the entire hand in space, and then uses the rotation information of each joint point to determine the shape of the hand in space.

Update the position and orientation of wrist joint.

Note :

OpenXR uses a right-handed coordinate system, while Unity uses a left-handed coordinate system, so the z value of position, x value of orientation and z value of orientation need to be flipped.
The 3D model needs to flip 180 degrees to adjust the initial position.

- Update rotations of other points.

In OnDestroy function:

To stop hand tracking detection:

//left hand:true ,right hand:false
bool isLeft = true; 
var feature = OpenXRSettings.Instance.GetFeature();
{
    int res = (int)XrResult.XR_SUCCESS;
    if (isLeft)
    {
        if (feature.m_leftHandle == ulong.MinValue) return;
        res = feature.xrDestroyHandTrackerEXT(feature.m_leftHandle);
    }
    else
    {
        if (feature.m_rightHandle == ulong.MinValue) return;
        res = feature.xrDestroyHandTrackerEXT(feature.m_rightHandle);
    }
    if (res != (int)XrResult.XR_SUCCESS)
    {
        UnityEngine.Debug.LogError("Failed to destroy hand tracker with error code " + res);
    }
}

Results:

Skeleton Model	3D Hand Model

Previous Next