Please Select Your Location
Australia
Österreich
België
Canada
Canada - Français
中国
Česká republika
Denmark
Deutschland
France
HongKong
Iceland
Italia
日本
Korea
Latvija
Lietuva
Lëtzebuerg
Malta
المملكة العربية السعودية (Arabic)
Nederland
New Zealand
Norge
Polska
Portugal
Russia
Saudi Arabia
Southeast Asia
España
Suisse
Suomi
Sverige
台灣
Ukraine
United Kingdom
United States
Select your country to buy
België
Česká republika
Denmark
Iceland
Italia
Latvija
Lietuva
Lëtzebuerg
Malta
Nederland
Norge
Polska
Portugal
España
Suisse
Suomi
Sverige

Integrate Hand Tracking Data Into Your Hand Model

OpenXR Hand Tracking Plugin Setup

Supported Unreal Engine version: 4.26 +

  • Enable Plugins:
    • Please enable “ OpenXR ” and “ OpenXRHandTracking ” plugins in Edit > Plugins > Virtual Reality.
image1.png
image2.png
  • Disable Plugins:
    • The " Steam VR " plugin must be disabled for OpenXR to work, please disable plugin in Edit > Plugins > Virtual Reality.
image3.png

Get OpenXR Hand Tracking Data

Getting Detection Result.

Detected left and right hand results are available from blueprint function “ GetMotionControllerData ”, and select left or right Hand.

image4.png

Render Hands using debug lines

Use XRMotionControllerData to draw the debug lines/spheres and form hands.

  • Enable Plugin:
    • Please enable “XRVisualization” plugin in Edit > Plugins > Mixed Reality.
image5.png
    • Create a Blueprint and choose Actor as the parent class:
      • In the Event Graph, we will call “Get Motion Controller Data” to get the hand tracking detection result.
      • After getting the hand tracking detection results, we will call the “ RenderMotionController ” node from the XRVisualization plugin to draw the debug line/sphere hands.
image6.png

Result

image7.png

Feed OpenXR HandTracking Data to Skeletal Mesh

Use XRMotionControllerData to update Skeletal Mesh hands transform.

Note: The hand skeleton models used in this Tutorial is the models in the ViveOpenXR sample.

Step1. Essential Setup
    • Import hand skeleton models.
    • Create a Blueprint for HandTracking (The following is called BP_RiggedHand ).
image8.png
image9.png

Step2. BP_RiggedHand: Implement transform update of Hand Skeletal Meshes by OpenXR Hand Tracking Data
    • In this tutorial we choose Actor as the parent class.
    • Add Poseable Mesh Component for the hand skeleton mesh in Components panel.
image10.png
      • Set Hand Skeletal Mesh in Details panel > Mesh > Skeletal Mesh.
image11.png
    • Add a new variable and name it “ Hand ” with type “ EControllerHand (enum) ” in My Blueprint We will use this variable to set which hand the current Actor is in the Level.
image12.png
image13.png


Go back to Level Editor default Interface, we will drag two “BP_RiggedHand” into Level (for both hands).

  • In Outliner panel > click BP_RiggedHand > Details panel > Hand Settings > Hand (Choose Right/Left hand).
image14.jpg image15.jpg


Remember to change the Hand Skeletal Mesh for each hand.

image16.jpg image16.jpg

  • Add a new variable and name it “ JointNames ” with type “ Name Array ” in My Blueprint Joint names are the bones name from the hand skeletal mesh. Later we will use this variable to update each hand bones transform.
image20.png

image19.png

  • Add a new variable and name it “ HandKeyPositions ” with type “ Vector Array ” in My Blueprint panel . We will use this array to hold the Hand Tracking Positions data in each Tick.

image21.png

image22.png

  • Add a new variable and name it “ HandKeyRotations ” with type “ Quat Array ” in My Blueprint. We will use this array to hold the Hand Tracking Rotations data in each Tick.
image23.png

image24.png

  • Event Tick : Get HandKeyPositions/ HandKeyRotations from Get Motion Controller Data” . We will use the “ Hand ” variable mentioned in the previous step to determine which hand data is the hand data to be obtained now. Set the hand tracking data obtained from this function to “ HandKeyPosition ” and “ HandKeyRotations ”.
image25.png

  • Event Tick : Get Is Valid from Get Motion Controller Data” . We can use one of the output “ Valid ” in XRMotionControllerData to determine whether to show/hide the hand model.
image26.png

  • Add a function and name it “ UpdateHandJointsTransfrom ” in My Blueprint
image25.png
    • Add some local variables in UpdateHandJointsTransform :
image28.png


EHandKeypoint
image29.png

  • Call “ ForEach EHandKeypoint ” and set the output ( Enum ) to “ TmpEnumValue ”, we will use it to check IsWrist and get the current hand bone location and rotation , after getting the hand position/ rotation and set them to “TmpJointPose” and “TmpJointRot” .
image30.png
  • If current bone is Wrist, we will use TmpJointPose to update this actor world location.

image31.png
  • Use TmpJointRot to update the current hand bone component space rotation , in this sample we need to add some offsets to correct the Axis of the hand skeleton mesh.
  • Note: Since the Hand Tracking Engine does not currently support metacarpal bones rotation, you need to ignore them when updating the rotation of each bone.

image32.png
  • Drag the UpdateHandJointsTransform function into the Event Graph and connect it after the Set Visiblity function, this action means that the game will call the UpdateHandJointsTransform function every tick (If the hand tracking data is valid.)
image33.png
  • Complete function flow in Event Graph
image34.png

Result

image35.png