Interact with the Real World: OpenXR Scene Understanding
OpenXR Scene Understanding Plugin Setup
Supported Unreal Engine version: 4.27 +
-
Enable Plugins:
-
Please enable plugin in Edit > Plugins > Virtual Reality:
- OpenXR
-
Please enable plugin in Edit > Plugins > Virtual Reality:

-
Disable Plugins:
- The "Steam VR" plugin must be disabled for OpenXR to work.
-
-
Please disable plugin in Edit > Plugins > Virtual Reality:
- SteamVR
-
Please disable plugin in Edit > Plugins > Virtual Reality:

-
-
Project Settings:
-
Please make sure the “
OpenXR Scene Understanding extension
” is enabled, the setting is in Edit > Project Settings > Plugins > Vive OpenXR > Scene Understanding:
- Enable Scene Understanding
-
Please make sure the “
OpenXR Scene Understanding extension
” is enabled, the setting is in Edit > Project Settings > Plugins > Vive OpenXR > Scene Understanding:
-
Project Settings:

Introduction to Blueprint Nodes for OpenXR Scene Understanding
-
Used to limit the range that will be scanned
-
These bounding volumes are used to determine which scene components to include in the resulting scene.
- Set Scene Compute Oriented Box Bound : Set the bounds of the box shape.
- Set Scene Compute Sphere Bound : Set the bounds of the sphere shape.
- Set Scene Compute Frustum Bound : Set the bounds of the frustum shape.
- Clear Scene Compute Bounds: Before the game ends, you need to clear all bounds by calling Clear Scene Compute Bounds .
-
These bounding volumes are used to determine which scene components to include in the resulting scene.

-
Controls the update speed of the scene and will affect the quality of the scene result
- This function will be trading off speed against the quality of the resulting scene.
-
- Set Scene Compute Consistency

-
Identify the level of detail of meshes
- This function will identify the level of detail of visual mesh compute.
-
- Set Mesh Compute Lod

-
Essential Setup
-
Creating an AR Session:
- 0The AR-related content needs to be set here because the OpenXR Scene Understanding uses OpenXRARTrackedGeometry to make the scanned meshes appear in the level.
- Unreal Setting up AR project Tutorial
- Create Data Asset: Content Browser > All > Content > click right mouse button > choose Miscellaneous > Data Asset
-
Creating an AR Session:
-
- After choose Data Asset , it will pop out “ Pick Class For Data Asset Instance ” window:
-
-
- Choose “ ARSessionConfig ” and press “ Select ”.
-


-
-
Open
ARSessionConfig
and enable some settings under Details Panel > AR Settiongs > World Mapping
- Enable Generate Mesh Data from Tracked Geometry
- Enable Generate Collision for Mesh Data
-
Open
ARSessionConfig
and enable some settings under Details Panel > AR Settiongs > World Mapping

-
Open Level Blueprint and add
Start AR Session
and
Stop AR Session
- Remember to set the “ARSessionConfig” as input to Start AR Session.

- Create a Blueprint for Scene Understanding and drag into the level.


-
BP_SceneUnderstanding: Display scanned mesh in the level
- In this tutorial we choose Actor as the parent class.
-
-
In Components panel:
- Add a Cube Static Mesh Component for Set Scene Compute Oriented Box Bounds .
- Add ARTrackableNotify
-
In Components panel:



-
-
Add Variable in
My Blueprint
panel
- Name: spatialMeshes
- Type: ARTrackedGeometry Array
-
Add Variable in
My Blueprint
panel


-
Add Functions in
My Blueprint
panel
-
StartSpatialMapping
- In this function, we need to call “ Toggle ARCapture ” function and set input “ On Off ” bool to true and “ Capture Type” to Spatial Mapping .
-
StartSpatialMapping


-
-
StopSpatialMapping
- In this function, we need to call “ Toggle ARCapture ” function and set input “ On Off ” bool to false and “ Capture Type” to Spatial Mapping .
-
StopSpatialMapping


-
-
In
Event Graph
- Event BeginPlay:
-
We will add some functions that affect the
quality
and
area
of the scanned mesh.
- Set Scene Compute Oriented Box Bound
- Set Scene Compute Consistency
- Set Mesh Compute Lod
- Start Spatial Mapping (This is the function done in the previous step.)
-
In
Event Graph

-
-
On Add/Remove Tracked Geometry (ARTrackableNotify):
- These two Events will control the number of mesh currently scanned.
-
On Add/Remove Tracked Geometry (ARTrackableNotify):
- Event Tick:
- You can get the current Spatial Meshes and Set Material on them.

- Event BeginPlay:
-
We will need to call
Stop Spatial Mapping
function and clear the bound you use at the beginning.
- Stop Spatial Mapping (This is the function done in the previous step.)
- Clear Scene Compute Bounds
- Clear the Spatial Meshes array

Result
Note:
- White Area: meshes representing the real-environment objects.
- Black Area: no objects in the space.
Spawn balls and interact with generated meshes
This sample is currently only supported Unreal Engine version: 5.0+ .
Because the physics system used by the meshes generated is Chaos Physics .
-
Create a Blueprint
- Create a Blueprint for the “Spawn ball” and drag it into the level.
- You can place as many as you want in the level.


-
BP_SpawnBall: Make a sphere static mesh to simulate a physical fall.
-
In
Components
panel:
- Add a Sphere Static Mesh Component to pretend the ball.
-
In
Components
panel:

-
In
Details
panel:
-
Enable Physics:
- Physics > Enable Simulate Physics
- Make sure the Collision Presets is “PhysicsActor”
-
Enable Physics:

-
-
In
My Blueprint
panel:
- Add a variable to hold the origin location of the ball.
-
In
My Blueprint
panel:

-
-
In
Event Graph
panel:
- Set the origin location of the ball when the game starts.
-
In
Event Graph
panel:

-
-
- Track the height of the ball in each frame, and reset its position if it is below -600.
-

Result

Note:
- Red Balls: Virtual Objects.
- White Area: It means that there are scanned meshes .
- Black Area: It means that there are no objects in the space.