Please Select Your Location
Australia
Österreich
België
Canada
Canada - Français
中国
Česká republika
Denmark
Deutschland
France
HongKong
Iceland
Italia
日本
Korea
Latvija
Lietuva
Lëtzebuerg
Malta
المملكة العربية السعودية (Arabic)
Nederland
New Zealand
Norge
Polska
Portugal
Russia
Saudi Arabia
Southeast Asia
España
Suisse
Suomi
Sverige
台灣
Ukraine
United Kingdom
United States

Integrate Hand Tracking Data Into Your Hand Model

OpenXR Hand Tracking Plugin Setup

Supported Unreal Engine version: 4.26 +

  • Enable Plugins:
    • Please enable plugin in Edit > Plugins > Virtual Reality:
      • OpenXR
      • OpenXRHandTracking
image1.png image2.png
  • Disable Plugins:
  • The "Steam VR" plugin must be disabled for OpenXR to work.
    • Please disable plugin in Edit > Plugins > Virtual Reality:
    • SteamVR
image3.png

Get OpenXR Hand Tracking Data

Getting Detection Result.

Detected left and right hand results are available from blueprint function “ GetMotionControllerData ”, and select left or right Hand.

image4.png

Render Hands using debug lines

Use XRMotionControllerData to draw the debug lines/spheres and form hands.

  • Enable Plugin:
    • Please enable plugin in Edit > Plugins > Mixed Reality:
      • XRVisualization
image5.png
    • Create a Blueprint and choose Actor as the parent class:
      • In the Event Graph, we will call “Get Motion Controller Data” to get the hand tracking detection result.
      • After getting the hand tracking detection results, we will call the “ RenderMotionController ” node from the XRVisualization plugin to draw the debug line/sphere hands.
image6.png

Result

image7.png

Feed OpenXR HandTracking Data to Skeletal Mesh

Use XRMotionControllerData to update Skeletal Mesh hands transform.

Note: The hand skeleton models used in this Tutorial is the models in the ViveOpenXR sample.

  • Essential Setup
    • Import hand skeleton models.
    • Create a Blueprint for HandTracking (The following is called BP_RiggedHand ).
image8.png
image9.png

  • BP_RiggedHand: Implement transform update of Hand Skeletal Meshes by OpenXR Hand Tracking Data
    • In this tutorial we choose Actor as the parent class.
    • In Components panel:
      • Add Poseable Mesh Component for the hand skeleton mesh.
image10.png
      • Set Hand Skeletal Mesh in Details panel > Mesh > Skeketal Mesh.
image11.png
    • Add Variables in My Blueprint panel
image12.png
      • Name: Hand
      • Type: EControllerHand (enum)
        • We will use this variable to set which hand the current Actor is in the Level.
image13.png


image14.png

Go back to Level Editor default Interface, we will drag two “BP_RigggedHand” into Level (for both hands).

  • In Outliner panel > click BP_RiggedHand > Details panel > Hand Settings > Hand (Choose Right/Left hand).
image15.png image16.png

Remember to change the Hand Skeletal Mesh for each hand.

image17.png image18.png

  • Name: JointName
  • Type: Name Array
    • Joint names are the bones name from the hand skeletal mesh.
    • Later we will use this variable to update each hand bones transform.
image20.png
image19.png

  • Name: HandKeyPositions
  • Type: Vector Array
    • We will use this array to hold the Hand Tracking Positions data in each Tick.

image21.png

image22.png

  • Name: HandKeyRotations
  • Type: Quat Array
    • We will use this array to hold the Hand Tracking Rotations data in each Tick.
image23.png

image24.png

  • Blueprint: Complete Functional Flow
    • Get Motion Controller Data: HandKeyPositions/ HandKeyRotations
      • We will use the “ Hand ” variable mentioned in the previous step to determine which hand data is the hand data to be obtained now.
      • Set the hand tracking data obtained from this function to “ HandKeyPosition ” and “ HandKeyRotations ”.
image25.png

  • Get Motion Controller Data: Is Valid
    • We can use one of the output “ Valid ” in XRMotionControllerData to determine whether to show/hide the hand model.
image26.png

  • Create Functions in My Blueprint panel
    • Name: UpdateHandJointsTransfrom
image27.png
    • Add some local variables in UpdateHandJointsTransform :
      • IsWrist (Type: Bool )
      • TmpEnumValue (Type: EHandKeypoint )
      • TmpJointPose (Type: Vector )
      • TmpJointRot (Type: Rotator )
      • TmpBoneName (Type: Name )
image28.png

EHandKeypoint

image29.png

  • Call “ ForEach EHandKeypoint ” and set the output ( Enum ) to “ TmpEnumValue ”, we will use it to check IsWrist and get the current hand bone location and rotation , after getting the hand position/ rotation and set them to “TmpJointPose” and “TmpJointRot” .
image30.png
  • If current bone is Wrist, we will use TmpJointPose to update this actor world location.
  • If current bone is not Wrist, we will use TmpJointPose to update the current hand bone world location.
image31.png
  • Use TmpJointRot to update the current hand bone component space rotation , in this sample we need to add some offsets to correct the Axis of the hand skeleton mesh.
image32.png
  • Drag the UpdateHandJointsTransform function into the Event Graph and connect it after the Set Visiblity function, this action means that the game will call the UpdateHandJointsTransform function every tick (If the hand tracking data is valid.)
image33.png
  • Complete function flow in Event Graph
image34.png

Result

image35.png