Integrate Hand Tracking Data Into Your Hand Model

OpenXR Hand Tracking Plugin Setup

Supported Unreal Engine version: 4.26 +

• Enable Plugins:
  • Please enable plugin in Edit > Plugins > Virtual Reality:
    • OpenXR
    • OpenXRHandTracking

• Disable Plugins:
• The "Steam VR" plugin must be disabled for OpenXR to work.

• • Please disable plugin in Edit > Plugins > Virtual Reality:
  • SteamVR

Get OpenXR Hand Tracking Data

Getting Detection Result.

Detected left and right hand results are available from blueprint function “ GetMotionControllerData ”, and select left or right Hand.

Render Hands using debug lines

Use XRMotionControllerData to draw the debug lines/spheres and form hands.

• Enable Plugin:
  • Please enable plugin in Edit > Plugins > Mixed Reality:
    • XRVisualization

• • Create a Blueprint and choose Actor as the parent class:
    • In the Event Graph, we will call “Get Motion Controller Data” to get the hand tracking detection result.
    • After getting the hand tracking detection results, we will call the “ RenderMotionController ” node from the XRVisualization plugin to draw the debug line/sphere hands.

Result

Feed OpenXR HandTracking Data to Skeletal Mesh

Use XRMotionControllerData to update Skeletal Mesh hands transform.

Note: The hand skeleton models used in this Tutorial is the models in the ViveOpenXR sample.

• Essential Setup
  • Import hand skeleton models.
  • Create a Blueprint for HandTracking (The following is called BP_RiggedHand ).

• BP_RiggedHand: Implement transform update of Hand Skeletal Meshes by OpenXR Hand Tracking Data
  • In this tutorial we choose Actor as the parent class.
  • In Components panel:
    • Add Poseable Mesh Component for the hand skeleton mesh.

• • • Set Hand Skeletal Mesh in Details panel > Mesh > Skeketal Mesh.

• • Add Variables in My Blueprint panel

• • • Name: Hand
    • Type: EControllerHand (enum)
      • We will use this variable to set which hand the current Actor is in the Level.

Go back to Level Editor default Interface, we will drag two “BP_RigggedHand” into Level (for both hands).

• In Outliner panel > click BP_RiggedHand > Details panel > Hand Settings > Hand (Choose Right/Left hand).

Remember to change the Hand Skeletal Mesh for each hand.

• Name: JointName
• Type: Name Array

• • Joint names are the bones name from the hand skeletal mesh.
  • Later we will use this variable to update each hand bones transform.

• Name: HandKeyPositions
• Type: Vector Array

• • We will use this array to hold the Hand Tracking Positions data in each Tick.

• Name: HandKeyRotations
• Type: Quat Array

• • We will use this array to hold the Hand Tracking Rotations data in each Tick.

• Blueprint: Complete Functional Flow
  • Get Motion Controller Data: HandKeyPositions/ HandKeyRotations
    • We will use the “ Hand ” variable mentioned in the previous step to determine which hand data is the hand data to be obtained now.
    • Set the hand tracking data obtained from this function to “ HandKeyPosition ” and “ HandKeyRotations ”.

• Get Motion Controller Data: Is Valid
  • We can use one of the output “ Valid ” in XRMotionControllerData to determine whether to show/hide the hand model.

• Create Functions in My Blueprint panel
  • Name: UpdateHandJointsTransfrom

• • Add some local variables in UpdateHandJointsTransform :
    • IsWrist (Type: Bool )
    • TmpEnumValue (Type: EHandKeypoint )
    • TmpJointPose (Type: Vector )
    • TmpJointRot (Type: Rotator )
    • TmpBoneName (Type: Name )

• Call “ ForEach EHandKeypoint ” and set the output ( Enum ) to “ TmpEnumValue ”, we will use it to check IsWrist and get the current hand bone location and rotation , after getting the hand position/ rotation and set them to “TmpJointPose” and “TmpJointRot” .

• If current bone is Wrist, we will use TmpJointPose to update this actor world location.

• If current bone is not Wrist, we will use TmpJointPose to update the current hand bone world location.

• Use TmpJointRot to update the current hand bone component space rotation , in this sample we need to add some offsets to correct the Axis of the hand skeleton mesh.

• Drag the UpdateHandJointsTransform function into the Event Graph and connect it after the Set Visiblity function, this action means that the game will call the UpdateHandJointsTransform function every tick (If the hand tracking data is valid.)

• Complete function flow in Event Graph

Result