Please Select Your Location
Australia
Österreich
België
Canada
Canada - Français
中国
Česká republika
Denmark
Deutschland
France
HongKong
Iceland
Italia
日本
Korea
Latvija
Lietuva
Lëtzebuerg
Malta
المملكة العربية السعودية (Arabic)
Nederland
New Zealand
Norge
Polska
Portugal
Russia
Saudi Arabia
Southeast Asia
España
Suisse
Suomi
Sverige
台灣
Ukraine
United Kingdom
United States

Interact with the Real World: OpenXR Scene Understanding

OpenXR Scene Understanding Plugin Setup

Supported Unity Engine version: 2020.2+

  • Enable OpenXR Plugins:
    • Please enable OpenXR plugin in Edit > Project Settings > XR Plug-in Management:
      image1.png

  • Click Exclamation mark next to “OpenXR” then choose “Fix All”.
    image2.png
    image3.png
  • Add Interaction profiles for your device.
    Following, take Vive Controller as an example.
    image4.png

  • Enable Scene Understanding extensions
    image5.png

Custom XR Meshing Subsystem

The custom Meshing subsystem is native code snippet to supply mesh with OpenXR Sceneunderstanding related functions. Within OpenXR this functionality can be exposed by using `OpenXRFeature` to manage the subsystem. You can use the dll provided by our SceneUnderstanding sample or refer to the appendix to build your custom subsystem.

Assets > MeshingFeaturePlugin > windows > x64 > MeshingFeaturePlugin.dll

Create an OpenXR Feature which sets up the custom Meshing Subsystem created before

  • Defining a feature which override the “SceneUnderstanding_OpenXR_API” class and provide attribute as below when running in the editor.
    image6.png
  • Override functions as below.
    • To manage the lifecycle of Unity subsystems.
image7.png
  • Intercept create session function
image8.png

  • Create class which contains function from custom XR Meshing Subsystem.
image9.png

  • Add function to convert right-handed transform.
image10.png
  • Enabled the created OpenXR feature.
image11.png

Get Scene understanding results and draw mesh in your app

  • Camera setting
    • To display mesh in the correct position, add “TrackedPoseDriver” script to your VR render camera.
image12.png
  • Adjust background
image13.png
  • Create empty game object named MeshingSample here.
image14.png

  • Create cube for setting scene understanding computation box bound and set options in inspector.
image15.png image16.png

  • Settings for mesh
    • Create empty game object named Meshes here.
image17.png

  • Create prefab for mesh named emptyMesh here.
    • Add Mesh Renderer and setting materials.
image18.png

  • Add Mesh Collider
image19.png

  • Add Mesh filter.
image20.png

  • Create new script to draw mesh with scene understanding results.
    • Add following namespaces to your script.
image21.png

  • Add the following properties:
image22.png

  • In Start function:
    • Get and check XRMeshSubsystem
image23.png

  • In Update function:
    • Get the states of all tracked meshes. When state is “Updated”, generate mesh if the existence of the mesh that Unity doesn’t know. When state is “Removed”, destroy mesh if Unity knows the existence of the mesh.
image24.png

  • Set scene computation box bound.
image25.png

  • Attach it to MeshingSample object and set options in inspector.
image26.png image27.png

Results

image28.png

The blue part is the background, the white part is the created meshes, the upper horizontal plane is the ceiling, and the front vertical plane is the wall.

Interact with the Real World

  • Let other virtual objects interact with real-environment meshes
    • Create empty prefab named sphere here.
      • Add Mesh Renderer and setting materials.
image29.png

  • Add Sphere Collider
image30.png

  • Add Rigidbody.
image31.png

  • Add Mesh filter.
image32.png

  • Add DestrySelf script
image33.png

  • Create new script to spawn sphere from main camera and attach the script to camera.
    • Add properties and set options in inspector.
image34.png image35.png

  • Add below codes to spawn targets.
image36.png

  • Results
image37.png

The balls that shoot forward will collide with the vertical plane then fall and stay on the horizontal plane.

Appendix

Custom XR Meshing Subsystem

  • Add following namespaces.
    image38.png
  • Add following function pointers.

image39.png

  • Define class to make an XRSceneMSFT can be managed by shared pointers
image40.png

  • Define class to store mesh data belonging to a UnityXRMeshId.
image41.png

  • Add following variables.
image42.png

  • Add following codes to handle main Unity events, which must export UnityPluginLoad and UnityPluginUnload functions. IUnityInterfaces enables the plug-in to access these functions.
image43.png image43.png

  • Implement subsystem specific lifecycle events in UnityPluginLoad function as follows
    • Define Initialize function for the subsystem. (Reset scene computation bounds and define mesh provider here.)
image44.png

  • Define GetMeshInfos function for the mesh provider

    (refer to “OpenXRSceneUnderstanding\Samples~\MeshingFeaturePlugin\Native~\meshing_provider.cpp” line 129)

    image45.png
    image46.png
    image47.png
    image48.png
    image49.png

  • Define AcquireMesh function for the mesh provider
    (refer to “OpenXRSceneUnderstanding\Samples~\MeshingFeaturePlugin\Native~\meshing_provider.cpp” line 300)
    image50.png
    image51.png
    image52.png

  • Define ReleaseMesh function for the mesh provider
    image53.png

  • Define SetMeshDensity function for the mesh provider
    image54.png

  • Define SetBoundingVolume function for the mesh provider
    image55.png

  • Define Start function for the subsystem.
    • Create a scene observer
      image56.png

  • Define Stop function for the subsystem.
    • Clear mesh data and destroy the scene observer.
      image57.png

  • Define Shutdown function for the subsystem.
    • Destroy the reference space.
      image58.png

  • Define function to get OpenXR function pointers and create reference space.
    image59.png
    image60.png

  • Define function to set a scene compute oriented box bound in a right-handed world space.
    image61.png