Integrate Hand Tracking Data Into Your Hand Model
OpenXR Hand Tracking Plugin Setup
Supported Unreal Engine version: 4.26 +
- Please enable “ OpenXR ” and “ OpenXRHandTracking ” plugins in Edit > Plugins > Virtual Reality.
- The " Steam VR " plugin must be disabled for OpenXR to work, please disable plugin in Edit > Plugins > Virtual Reality.
Get OpenXR Hand Tracking Data
Getting Detection Result.
Detected left and right hand results are available from blueprint function “ GetMotionControllerData ”, and select left or right Hand.
Render Hands using debug lines
Use XRMotionControllerData to draw the debug lines/spheres and form hands.
- Please enable “XRVisualization” plugin in Edit > Plugins > Mixed Reality.
Create a Blueprint and choose
as the parent class:
- In the Event Graph, we will call “Get Motion Controller Data” to get the hand tracking detection result.
- After getting the hand tracking detection results, we will call the “ RenderMotionController ” node from the XRVisualization plugin to draw the debug line/sphere hands.
- Create a Blueprint and choose Actor as the parent class:
Feed OpenXR HandTracking Data to Skeletal Mesh
Use XRMotionControllerData to update Skeletal Mesh hands transform.
Note: The hand skeleton models used in this Tutorial is the models in the ViveOpenXR sample.Step1. Essential Setup
- Import hand skeleton models.
- Create a Blueprint for HandTracking (The following is called BP_RiggedHand ).
Step2. BP_RiggedHand: Implement transform update of Hand Skeletal Meshes by OpenXR Hand Tracking Data
- In this tutorial we choose Actor as the parent class.
- Add Poseable Mesh Component for the hand skeleton mesh in Components panel.
- Set Hand Skeletal Mesh in Details panel > Mesh > Skeletal Mesh.
- Add a new variable and name it “ Hand ” with type “ EControllerHand (enum) ” in My Blueprint We will use this variable to set which hand the current Actor is in the Level.
Go back to Level Editor default Interface, we will drag two “BP_RiggedHand” into Level (for both hands).
- In Outliner panel > click BP_RiggedHand > Details panel > Hand Settings > Hand (Choose Right/Left hand).
Remember to change the Hand Skeletal Mesh for each hand.
- Add a new variable and name it “ JointNames ” with type “ Name Array ” in My Blueprint Joint names are the bones name from the hand skeletal mesh. Later we will use this variable to update each hand bones transform.
- Add a new variable and name it “ HandKeyPositions ” with type “ Vector Array ” in My Blueprint panel . We will use this array to hold the Hand Tracking Positions data in each Tick.
- Add a new variable and name it “ HandKeyRotations ” with type “ Quat Array ” in My Blueprint. We will use this array to hold the Hand Tracking Rotations data in each Tick.
- Event Tick : Get HandKeyPositions/ HandKeyRotations from “ Get Motion Controller Data” . We will use the “ Hand ” variable mentioned in the previous step to determine which hand data is the hand data to be obtained now. Set the hand tracking data obtained from this function to “ HandKeyPosition ” and “ HandKeyRotations ”.
- Event Tick : Get Is Valid from “ Get Motion Controller Data” . We can use one of the output “ Valid ” in XRMotionControllerData to determine whether to show/hide the hand model.
- Add a function and name it “ UpdateHandJointsTransfrom ” in My Blueprint
- Add some local variables in UpdateHandJointsTransform :
- Call “ ForEach EHandKeypoint ” and set the output ( Enum ) to “ TmpEnumValue ”, we will use it to check IsWrist and get the current hand bone location and rotation , after getting the hand position/ rotation and set them to “TmpJointPose” and “TmpJointRot” .
- If current bone is Wrist, we will use TmpJointPose to update this actor world location.
- Use TmpJointRot to update the current hand bone component space rotation , in this sample we need to add some offsets to correct the Axis of the hand skeleton mesh.
Note: Since the Hand Tracking Engine does not currently support metacarpal bones rotation, you need to ignore them when updating the rotation of each bone.
- Drag the UpdateHandJointsTransform function into the Event Graph and connect it after the Set Visiblity function, this action means that the game will call the UpdateHandJointsTransform function every tick (If the hand tracking data is valid.)
- Complete function flow in Event Graph