Please Select Your Location
Australia
Österreich
België
Canada
Canada - Français
中国
Česká republika
Denmark
Deutschland
France
HongKong
Iceland
Italia
日本
Korea
Latvija
Lietuva
Lëtzebuerg
Malta
المملكة العربية السعودية (Arabic)
Nederland
New Zealand
Norge
Polska
Portugal
Russia
Saudi Arabia
Southeast Asia
España
Suisse
Suomi
Sverige
台灣
Ukraine
United Kingdom
United States

How to Integrate Hand Tracking Data Into Your Hand Model

OpenXR Hand Tracking Plugin Setup

Supported Unity Engine version: 2020.2+

  • Enable OpenXR Plugins:
    • Please enable OpenXR plugin in Edit > Project Settings > XR Plug-in Management:
image1.png

  • Click Exclamation mark next to “OpenXR” then choose “Fix All”.
image2.png
image3.png

  • Add Interaction profiles for your device.

As following, take Vive Controller as an example.

image4.png

  • Enable Hand Tracking extensions
image5.png

To draw skeleton or 3D hand model in your App.

    • To display your hand in the correct position, add “TrackedPoseDriver” script to your VR render camera.
image6.png

  • Add hand Gameobjects:
    • For skeleton hand: Create two empty GameObjects for each hand. We name it “Left” and “Right” here.
image7.png

  • For 3D hand(Use the 3D model provided by our HandTracking sample as an example):

Assets > Samples > VIVE Wave OpenXR Plugin - Windows > 1.0.4 > HandTracking Example > Materials

image8.png

  • Create new script and attach it to left or right hand game objects.

Add this line to your script.

using UnityEngine.XR.OpenXR;
  • In Start function:
    • To Start hand tracking detection:
      //left hand:true ,right hand:false
      bool isLeft = true;
      var feature = OpenXRSettings.Instance.GetFeature();
      XrHandTrackerCreateInfoEXT createInfo = new XrHandTrackerCreateInfoEXT(
      XrStructureType.XR_TYPE_HAND_TRACKER_CREATE_INFO_EXT,
              IntPtr.Zero,
              XrHandEXT.XR_HAND_LEFT_EXT,
              XrHandJointSetEXT.XR_HAND_JOINT_SET_DEFAULT_EXT);
      int res;
      if (isLeft)
      {
      	res = feature.xrCreateHandTrackerEXT(createInfo, out feature.m_leftHandle);
      }
      else
      {
      	createInfo.hand = XrHandEXT.XR_HAND_RIGHT_EXT;
      	res = feature.xrCreateHandTrackerEXT(createInfo, out feature.m_rightHandle);
      }
      if (res != (int)XrResult.XR_SUCCESS)
      {
      	UnityEngine.Debug.LogError("Failed to create hand tracker with error code " + res);
      }
  • For skeleton hand:
    • Create game objects for 26 joints.
    • Create game objects for links between joints.

  • In Update function:
    • Check if the hand tracking detection result is available.
      var feature = OpenXRSettings.Instance.GetFeature();      
      if (feature.m_leftHandle == ulong.MinValue && feature.m_rightHandle == ulong.MinValue)
      	return;
  • Get hand tracking detection result:
    //left hand:true ,right hand:false
    bool isLeft = true; 
    XrHandJointLocationEXT[] HandjointLocations = new XrHandJointLocationEXT[(int)XrHandJointEXT.XR_HAND_JOINT_MAX_ENUM_EXT];
    var feature = OpenXRSettings.Instance.GetFeature();
    XrHandJointsLocateInfoEXT locateInfo = new XrHandJointsLocateInfoEXT(
    	XrStructureType.XR_TYPE_HAND_JOINTS_LOCATE_INFO_EXT,
    	IntPtr.Zero,
    	feature.m_space,
    	(Int64)10 //An arbitrary number greater than 0
    );
    unsafe
    {
      fixed (XrHandJointLocationEXT* ptr = HandjointLocations)
      {
      locations.type = XrStructureType.XR_TYPE_HAND_JOINT_LOCATIONS_EXT;
      locations.next = IntPtr.Zero;
         locations.isActive = 0;
      locations.jointCount = (int)XrHandJointEXT.XR_HAND_JOINT_MAX_ENUM_EXT;
          locations.jointLocations = (IntPtr)ptr;
      int res;
      if(isleftHand)
         {
          res = feature.xrLocateHandJointsEXT(feature.m_leftHandle, locateInfo, ref locations);
      	}
      else
         {
          res = feature.xrLocateHandJointsEXT(feature.m_rightHandle, locateInfo, ref locations);
          }
      	}
    }
    if(locations.isActive == 1)
    {
    	UpdateJointLocation();//Update your hand model here.
    }
    else
    {
    	//Hide your hand model due to not detect the hand input 
    }
  • Update skeleton hand model:

The skeleton model only takes the position information of each joint but no rotation information.

image9.png

Note :

OpenXR uses a right-handed coordinate system, while Unity uses a left-handed coordinate system, so the z value of position needs to be flipped.

  • Update links in skeleton model.
    • Calculate link position and rotation based on points on both ends.
  • Update 3D hand model:

3D model uses wrist's position information to determine the position of the entire hand in space, and then uses the rotation information of each joint point to determine the shape of the hand in space.

  • Update the position and orientation of wrist joint.
image10.png

Note :

  1. OpenXR uses a right-handed coordinate system, while Unity uses a left-handed coordinate system, so the z value of position, x value of orientation and z value of orientation need to be flipped.
  2. The 3D model needs to flip 180 degrees to adjust the initial position.
    • Update rotations of other points.
image11.png

    • In OnDestroy function:
      • To stop hand tracking detection:
        //left hand:true ,right hand:false
        bool isLeft = true; 
        var feature = OpenXRSettings.Instance.GetFeature();
        {
            int res = (int)XrResult.XR_SUCCESS;
            if (isLeft)
            {
                if (feature.m_leftHandle == ulong.MinValue) return;
                res = feature.xrDestroyHandTrackerEXT(feature.m_leftHandle);
            }
            else
            {
                if (feature.m_rightHandle == ulong.MinValue) return;
                res = feature.xrDestroyHandTrackerEXT(feature.m_rightHandle);
            }
            if (res != (int)XrResult.XR_SUCCESS)
            {
                UnityEngine.Debug.LogError("Failed to destroy hand tracker with error code " + res);
            }
        }

Results:

Skeleton Model

3D Hand Model

image12.png

image13.png