Please Select Your Location
Australia
Österreich
België
Canada
Canada - Français
中国
Česká republika
Denmark
Deutschland
France
HongKong
Iceland
Italia
日本
Korea
Latvija
Lietuva
Lëtzebuerg
Malta
المملكة العربية السعودية (Arabic)
Nederland
New Zealand
Norge
Polska
Portugal
Russia
Saudi Arabia
Southeast Asia
España
Suisse
Suomi
Sverige
台灣
Ukraine
United Kingdom
United States
Select your country to buy
België
Česká republika
Denmark
Iceland
Italia
Latvija
Lietuva
Lëtzebuerg
Malta
Nederland
Norge
Polska
Portugal
España
Suisse
Suomi
Sverige

Integrate Facial Tracking Data With Your Avatar

OpenXR Facial Tracking Plugin Setup

Supported Unreal Engine version: 4.26 +

  • Enable Plugins:
    • Please enable plugin in Edit > Plugins > Virtual Reality:
image1.png
  • Disable Plugins:
    • The "Steam VR" plugin must be disabled for OpenXR to work.
    • Please disable plugin in Edit > Plugins > Virtual Reality:
image2.png
  • Project Settings:
    • Please make sure the “ OpenXR Facial Tracking extension ” is enabled, the setting is in Edit > Project Settings > Plugins > Vive OpenXR > Facial Tracking:
image3.jpg

Initial / Release Facial Tracker

If we want to use OpenXR Facial Tracking, we will need to do some steps to make sure the OpenXR Facial Tracking process is fine.

  • Initial Facial Tracker
    • Create Facial Tracker
      • We can pass in (or select) Eye or Lip in the input of this function to indicate the tracking type to be created.
      • This function is usually called at the start of the game.
image4.png
    • Release Facial Tracker
      • Destroy Facial Tracker: This function does not need to select which tracking type need to be released; instead, it will confirm by itself which tracking types need to be released. This function is usually called at the end of the game.

image5.png

Get Eye / Lip Facial Expressions Data

Getting Detection Result.

Detected Eye or Lip expressions results are available from blueprint function “ GetEyeFacialExpressions ” or “ GetLipFacialExpressions ”.

image6.png image7.png
image8.png image9.png

Feed OpenXR FacialTracking Eye & Lip Data to Avatar

Use data from “ GetEyeFacialExpressions ” or “ GetLipFacialExpressions ” to update Avatar’s eye shape or lip shape.

Note: The Avatar used in this tutorial is the model in the ViveOpenXR sample.

Note: This tutorial will be presented using C++.

Step1. Essential Setup
    • Import head and eyes skeleton models/textures/materials.
image10.png
image11.png image12.png

Step2. FT_Framework: Handle initial / release Facial Tracker
    • Create new C++ Class:
    • Content Browser > All > C++ Classes > click right mouse button > choose New C++ Class…
image16.jpg
  • After choose New C++ Class, it will pop out “ Add C++ Class ” window, in CHOOSE PARENT CLASS step:
    • Choose “ None ” and press “ Next ”.
image14.jpg
  • In NAME YOUR NEW CLASS step:
    • Class Type: Private
    • Name: FT_Framework
    • After the previous settings are completed, press Create Class .
image18.png

  • FT_ AvatarSample.h : Add the following #include statements at the top.
// Get facial experssion enums from ViveOpenXR plugin. 
#include "ViveFacialExpressionEnums.h" 
// These provide the functionality required for ViveOpenXR plugin - Facial Tracking. 
#include "ViveOpenXRFacialTrackingFunctionLibrary.h"  
  • FT_ AvatarSample.h : Add the following Properties under public:
public: 
    // Called every frame
    virtual void Tick(float DeltaTime) override;

    /** Avatar head mesh. */
    UPROPERTY(VisibleDefaultsOnly, Category = Mesh)
    USkeletalMeshComponent* HeadModel;
    /** Avatar left eye mesh. */
    UPROPERTY(VisibleDefaultsOnly, Category = Mesh)
    USkeletalMeshComponent* EyeModel_L;
    /** Avatar right eye mesh. */
    UPROPERTY(VisibleDefaultsOnly, Category = Mesh)
    USkeletalMeshComponent* EyeModel_R;
    /** Left eye's anchor. */
    UPROPERTY(VisibleDefaultsOnly, Category = Anchor)
    USceneComponent* EyeAnchor_L;
    /** Right eye's anchor. */
    UPROPERTY(VisibleDefaultsOnly, Category = Anchor)
    USceneComponent* EyeAnchor_R;

    /** If true, this Actor will update Eye expressions every frame. */
    UPROPERTY(EditAnywhere, Category = FacialTrackingSettings)
    bool EnableEye = true;
    /** If true, this Actor will update Lip expressions every frame. */
    UPROPERTY(EditAnywhere, Category = FacialTrackingSettings)
    bool EnableLip = true;

    bool isEyeActive = false;
    bool isLipActive = false;

  • FT_ AvatarSample.h : Add the following Properties under private .
private:
    /** ¬We will use this variable to set both eyes anchor. */
    TArray<USceneComponent*> EyeAnchors;
    /** ¬We will use this variable to set eye expressions weighting. */
    TMap<EEyeShape, float> EyeWeighting;
    /** ¬We will use this variable to set lip expressions weighting. */
    TMap<ELipShape, float> LipWeighting;
    
    /** ¬This TMap variable is used to store the corresponding result of Avatar's eye blend shapes(key) and OpenXRFacialTracking eye expressions(value). */
    TMap<FName, EEyeShape> EyeShapeTable;
    /** ¬This TMap variable is used to store the corresponding result of Avatar's lip blend shapes(key) and OpenXRFacialTracking lip expressions(value). */
    TMap<FName, ELipShape> LipShapeTable;

    /** This function will calculate the gaze direction to update the eye's anchors rotation to represent the direction of the eye's gaze. */
    void UpdateGazeRay();

    FORCEINLINE FVector ConvetToUnrealVector(FVector Vector, float Scale = 1.0f)
    {
        return FVector(Vector.Z * Scale, Vector.X * Scale, Vector.Y * Scale);
    }

    /** Render the result of face tracking to the avatar's blend shapes. */
    template<typename T>
    void RenderModelShape(USkeletalMeshComponent* model, TMap<FName, T> shapeTable, TMap<T, float> weighting);

  • FT_ cpp : Add the following #include statements at the top .
#include "FT_AvatarSample.h"
#include "FT_Framework.h"
#include "Engine/Classes/Kismet/KismetMathLibrary.h"
  • AFT_AvatarSample::AFT_AvatarSample() - Definition for the Skeletal Mesh that will serve as our visual representation.
// Sets default values
AFT_AvatarSample::AFT_AvatarSample()
{
PrimaryActorTick.bCanEverTick = true;
RootComponent = CreateDefaultSubobject<USceneComponent>(TEXT("RootSceneComponent"));
// Head
HeadModel = CreateDefaultSubobject<USkeletalMeshComponent>(TEXT("HeadModel"));
HeadModel->SetupAttachment(RootComponent);
// Eye_L
EyeAnchor_L = CreateDefaultSubobject<USceneComponent>(TEXT("EyeAnchor_L"));
EyeAnchor_L->SetupAttachment(RootComponent);
EyeAnchor_L->SetRelativeLocation(FVector(3.448040f, 7.892285f, 0.824235f));
EyeModel_L = CreateDefaultSubobject<USkeletalMeshComponent>(TEXT("EyeModel_L"));
EyeModel_L->SetupAttachment(EyeAnchor_L);
EyeModel_L->SetRelativeLocation(FVector(-3.448040f, -7.892285f, -0.824235f));
// Eye_R
EyeAnchor_R = CreateDefaultSubobject<USceneComponent>(TEXT("EyeAnchor_R"));
EyeAnchor_R->SetupAttachment(RootComponent);
EyeAnchor_R->SetRelativeLocation(FVector(-3.448040f, 7.892285f, 0.824235f));
EyeModel_R = CreateDefaultSubobject<USkeletalMeshComponent>(TEXT("EyeModel_R"));
EyeModel_R->SetupAttachment(EyeAnchor_R);
EyeModel_R->SetRelativeLocation(FVector(3.448040f, -7.892285f, -0.824235f));
EyeAnchors.AddUnique(EyeAnchor_L);
EyeAnchors.AddUnique(EyeAnchor_R);
}

  • AFT_AvatarSample::BeginPlay() - Create eye and lip facial tracker.
// Called when the game starts or when spawned
void AFT_AvatarSample::BeginPlay()
{
Super::BeginPlay();
this->SetOwner(GetWorld()->GetFirstPlayerController()->GetPawn());
// CreateFacialTracker_Eye
if (EnableEye)
{
FT_Framework::Instance()->StartEyeFramework();
}
// CreateFacialTracker_Lip
if (EnableLip)
{
FT_Framework::Instance()->StartLipFramework();
}

  • AFT_AvatarSample::BeginPlay() - Store the result corresponding to the eye blend shapes(key) of the avatar and OpenXRFacialTracking eye expressions(value) in the EyeShapeTable .
EyeShapeTable = {
{"Eye_Left_Blink", EEyeShape::Eye_Left_Blink},
{"Eye_Left_Wide", EEyeShape::Eye_Left_Wide},
{"Eye_Left_Right",EEyeShape::Eye_Left_Right},
{"Eye_Left_Left",EEyeShape::Eye_Left_Left},
{"Eye_Left_Up", EEyeShape::Eye_Left_Up},
{"Eye_Left_Down", EEyeShape::Eye_Left_Down},
{"Eye_Right_Blink", EEyeShape::Eye_Right_Blink},
{"Eye_Right_Wide", EEyeShape::Eye_Right_Wide},
{"Eye_Right_Right",EEyeShape::Eye_Right_Right},
{"Eye_Right_Left", EEyeShape::Eye_Right_Left},
{"Eye_Right_Up", EEyeShape::Eye_Right_Up},
{"Eye_Right_Down", EEyeShape::Eye_Right_Down},
//{"Eye_Frown", EEyeShape::Eye_Frown},
{"Eye_Left_squeeze", EEyeShape::Eye_Left_Squeeze},
{"Eye_Right_squeeze", EEyeShape::Eye_Right_Squeeze}
};

  • AFT_AvatarSample::BeginPlay() - Store the result corresponding to the lip blend shapes(key) of the avatar and OpenXRFacialTracking lip expressions(value) in the LipShapeTable .
LipShapeTable = {
        // Jaw
        {"Jaw_Right", ELipShape::Jaw_Right},
        {"Jaw_Left", ELipShape::Jaw_Left},
        {"Jaw_Forward", ELipShape::Jaw_Forward},
        {"Jaw_Open", ELipShape::Jaw_Open},
        // Mouth
        {"Mouth_Ape_Shape", ELipShape::Mouth_Ape_Shape},
        {"Mouth_Upper_Right", ELipShape::Mouth_Upper_Right},
        {"Mouth_Upper_Left", ELipShape::Mouth_Upper_Left},
        {"Mouth_Lower_Right", ELipShape::Mouth_Lower_Right},
        {"Mouth_Lower_Left", ELipShape::Mouth_Lower_Left},
        {"Mouth_Upper_Overturn", ELipShape::Mouth_Upper_Overturn},
        {"Mouth_Lower_Overturn", ELipShape::Mouth_Lower_Overturn},
        {"Mouth_Pout", ELipShape::Mouth_Pout},
        {"Mouth_Smile_Right", ELipShape::Mouth_Smile_Right},
        {"Mouth_Smile_Left", ELipShape::Mouth_Smile_Left},
        {"Mouth_Sad_Right", ELipShape::Mouth_Sad_Right},
        {"Mouth_Sad_Left", ELipShape::Mouth_Sad_Left},
        {"Cheek_Puff_Right", ELipShape::Cheek_Puff_Right},
        {"Cheek_Puff_Left", ELipShape::Cheek_Puff_Left},
        {"Cheek_Suck", ELipShape::Cheek_Suck},
        {"Mouth_Upper_UpRight", ELipShape::Mouth_Upper_UpRight},
        {"Mouth_Upper_UpLeft", ELipShape::Mouth_Upper_UpLeft},
        {"Mouth_Lower_DownRight", ELipShape::Mouth_Lower_DownRight},
        {"Mouth_Lower_DownLeft", ELipShape::Mouth_Lower_DownLeft},
        {"Mouth_Upper_Inside", ELipShape::Mouth_Upper_Inside},
        {"Mouth_Lower_Inside", ELipShape::Mouth_Lower_Inside},
        {"Mouth_Lower_Overlay", ELipShape::Mouth_Lower_Overlay},
        // Tongue
        {"Tongue_LongStep1", ELipShape::Tongue_LongStep1},
        {"Tongue_Left", ELipShape::Tongue_Left},
        {"Tongue_Right", ELipShape::Tongue_Right},
        {"Tongue_Up", ELipShape::Tongue_Up},
        {"Tongue_Down", ELipShape::Tongue_Down},
(承接上一頁)
        {"Tongue_Roll", ELipShape::Tongue_Roll},
        {"Tongue_LongStep2", ELipShape::Tongue_LongStep2},
        {"Tongue_UpRight_Morph", ELipShape::Tongue_UpRight_Morph},
        {"Tongue_UpLeft_Morph", ELipShape::Tongue_UpLeft_Morph},
        {"Tongue_DownRight_Morph", ELipShape::Tongue_DownRight_Morph},
        {"Tongue_DownLeft_Morph", ELipShape::Tongue_DownLeft_Morph}
    };

We can find blend shapes in Avatar skeleton mesh asset.

  • AFT_AvatarSample::RenderModelShape() - Render the result of face tracking to the avatar's blend shapes.
template<typename T>
void AFT_AvatarSample::RenderModelShape(USkeletalMeshComponent* model, TMap<FName, T> shapeTable, TMap<T, float> weighting)
{
    if (shapeTable.Num() <= 0) return;
    if (weighting.Num() <= 0) return;
    for (auto &table : shapeTable)
    {
        if ((int)table.Value != (int)EEyeShape::None || (int)table.Value != (int)ELipShape::None)
        {
            model->SetMorphTarget(table.Key, weighting[table.Value]);
        }
    }
}

  • AFT_AvatarSample::UpdateGazeRay() - This function will calculate the gaze direction to update the eye's anchors rotation to represent the direction of the eye's gaze.
void AFT_AvatarSample::UpdateGazeRay()
{
    // Caculate Left eye gaze direction
    FVector gazeDirectionCombinedLocal_L;
    FVector modelGazeOrigin_L, modelGazeTarget_L;
    FRotator lookAtRotation_L, eyeRotator_L;
    if (EyeWeighting.Num() <= 0) return;
    if (EyeWeighting[EEyeShape::Eye_Left_Right] > EyeWeighting[EEyeShape::Eye_Left_Left])
    {
        gazeDirectionCombinedLocal_L.X = EyeWeighting[EEyeShape::Eye_Left_Right];
    }
    else
    {
        gazeDirectionCombinedLocal_L.X = -EyeWeighting[EEyeShape::Eye_Left_Left];
    }
    if (EyeWeighting[EEyeShape::Eye_Left_Up] > EyeWeighting[EEyeShape::Eye_Left_Down])
    {
        gazeDirectionCombinedLocal_L.Y = EyeWeighting[EEyeShape::Eye_Left_Up];
    }
    else
    {
        gazeDirectionCombinedLocal_L.Y = -EyeWeighting[EEyeShape::Eye_Left_Down];
    }
    gazeDirectionCombinedLocal_L.Z = 1.0f;
    modelGazeOrigin_L = EyeAnchors[0]->GetRelativeLocation();
    modelGazeTarget_L = EyeAnchors[0]->GetRelativeLocation() + gazeDirectionCombinedLocal_L;
 
(承接上一頁)
    lookAtRotation_L = UKismetMathLibrary::FindLookAtRotation(ConvetToUnrealVector(modelGazeOrigin_L), ConvetToUnrealVector(modelGazeTarget_L));
    eyeRotator_L = FRotator(lookAtRotation_L.Roll, lookAtRotation_L.Yaw, -lookAtRotation_L.Pitch);
    EyeAnchors[0]->SetRelativeRotation(eyeRotator_L);

    // Caculate Right eye gaze direction
    FVector gazeDirectionCombinedLocal_R;
    FVector modelGazeOrigin_R, modelGazeTarget_R;
    FRotator lookAtRotation_R, eyeRotator_R;
    if (EyeWeighting[EEyeShape::Eye_Right_Left] > EyeWeighting[EEyeShape::Eye_Right_Right])
    {
        gazeDirectionCombinedLocal_R.X = -EyeWeighting[EEyeShape::Eye_Right_Left];
    }
    else
    {
        gazeDirectionCombinedLocal_R.X = EyeWeighting[EEyeShape::Eye_Right_Right];
    }
    if (EyeWeighting[EEyeShape::Eye_Right_Up] > EyeWeighting[EEyeShape::Eye_Right_Down])
    {
        gazeDirectionCombinedLocal_R.Y = EyeWeighting[EEyeShape::Eye_Right_Up];
    }
    else
    {
        gazeDirectionCombinedLocal_R.Y = -EyeWeighting[EEyeShape::Eye_Right_Down];
    }
    gazeDirectionCombinedLocal_R.Z = 1.0f;
    modelGazeOrigin_R = EyeAnchors[0]->GetRelativeLocation();
    modelGazeTarget_R = EyeAnchors[0]->GetRelativeLocation() + gazeDirectionCombinedLocal_R;
(承接上一頁)
    lookAtRotation_R = UKismetMathLibrary::FindLookAtRotation(ConvetToUnrealVector(modelGazeOrigin_R), ConvetToUnrealVector(modelGazeTarget_R));
    eyeRotator_R = FRotator(lookAtRotation_R.Roll, lookAtRotation_R.Yaw, -lookAtRotation_R.Pitch);
    EyeAnchors[1]->SetRelativeRotation(eyeRotator_R);

  • AFT_AvatarSample::Tick() - Update avatar’s Eye and Lip shapes every frame and update eyes gaze direction.
// Called every frame
void AFT_AvatarSample::Tick(float DeltaTime)
{
    Super::Tick(DeltaTime);

    //Update Eye Shapes
    if (EnableEye)
    {
        UViveOpenXRFacialTrackingFunctionLibrary::GetEyeFacialExpressions(isEyeActive, EyeWeighting);
        RenderModelShape(HeadModel, EyeShapeTable, EyeWeighting);
        UpdateGazeRay();
    }
    
    //Update Lip Shapes
    if (EnableLip)
    {
        UViveOpenXRFacialTrackingFunctionLibrary::GetLipFacialExpressions(isLipActive, LipWeighting);
        RenderModelShape(HeadModel, LipShapeTable, LipWeighting);
    }
}

  • AFT_AvatarSample::EndPlay() - Release Eye and Lip Facial Tracker.
void AFT_AvatarSample::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
    Super::EndPlay(EndPlayReason);
    // DestroyFacialTracker_Eye
    FT_Framework::Instance()->StopAllFramework();
    FT_Framework::Instance()->DestroyFramework();
    EyeShapeTable.Empty();
    LipShapeTable.Empty();
}

  • Create Blueprint class based on FT_AvatarSample : Content Browser > All > C++ Classes > Public > click right button on FT_AvatarSample > choose Create Blueprint class base on FT_AvatarSample
  • image20.jpg
  • In NAME YOUR NEW FT AVATAR SAMPLE step:
    • Name: BP_FT_AvatarSample
    • Path: All > Content > Blueprints
    • After the previous settings are completed, press Create Blueprint Class .
    • image21.jpg
  • After Blueprint class is created: Open the blueprint in All > Content > Blueprints , we will find the BP_FT_AvatarSample .
  • image22.jpg
  • After opening it, confirm that Parent class is FT_AvatarSample . You can see that there are components from the C++ parent in the Component panel
  • image23.jpg

  • Set Hand Skeletal Mesh in HeadModel > Details panel > Mesh > Skeletal Mesh.
image24.jpg image25.jpg
  • Set Left Eye Skeletal Mesh in EyeModel_L > Details panel > Mesh > Skeletal Mesh.
image26.jpg image27.jpg

  • Set Right Eye Skeletal Mesh in EyeModel_R > Details panel > Mesh > Skeletal Mesh.

image28.jpg image29.jpg

  • Go back to Level Editor Default Interface, we will drag the “ BP_FT_AvatarSample ” into Level. You can decide if you need to enable or disable Eye and Lip in: Outliner panel > click BP_FT_AvatarSample > Details panel > Facial Tracking Settings > Enable Eye / Enable Lip .

image30.jpg

Result

image49.png