Aardvark and Augmented Virtual Reality

Author: Hurst Digital | | Categories: Augmented reality , Virtual reality

Applications vs. Tools

Often when a user wants to do something in Virtual Reality, they want to be immersed in an experience. The thing could be bashing monsters in Skyrim, it could be visiting digital worlds with people around the world in VR Chat, or it could be creating artwork in Tiltbrush. This experiences forms the backbone of every user’s time in VR. They are the base-level reality that users are opting-into when they put on their headset. They expect to spend somewhere between tens of minutes and several hours in these experiences before they take the headset off and pop back out to the real world.

But there’s another kind of interaction that many users want that is poorly served by building it into a full immersive base-level world. These are the kinds of things that you would do for seconds or minutes at a time, rather than hours. They’re also tools or applications that run in the background while you’re doing other things. They are things you would do while waiting at a load screen, or while taking a breather before diving into the next quest.

In many ways, the tools I’m talking about here are the same things you would use your smartphone for in the real world: checking email, yelling at somebody on Twitter, streaming music from Spotify, getting a notification that you’re about to be late for a meeting from your calendar, or pulling up a note-taking app to write down something you want to remember later. Almost none of these applications exist in VR.


A few early steps have been made in this direction. In 2015, SteamVR added an overlay interface that allows applications to draw rectangles of arbitrary content on top of whatever main app the user is running. That was later extended to support 3D panorama. This has resulted in a few applications: OVRDrop and OVR Toolkit let you interact with desktop windows from inside VR, PlutoVR lets you chat with your friends, and Turn Signal tells you when you should turn left instead of right to avoid wrapping yourself up in your tether. Oculus later built similar desktop window functionality into their system direction with Dash 2.0. Building these applications has turned out to be challenging for developers, however, and they have not arrived in the quantities I expected when the overlay interface was first coming online.

The biggest reason for that is that the existing overlay interfaces expose the wrong set of primitives. They allow applications to produce 2D textures and accept “laser mouse” input from the system. Exactly how to wrangle a UI system into producing off-screen textures is left as the developer’s responsibility. If the developer wants to draw something move complex like a 3D model the difficulty increases significantly. Laser mouse interfaces work for 2D desktop windows, but most other applications need some kind of 3D spatial interaction if they really want to be VR-native. Existing overlay interfaces don’t provide anything like that.

Image for post


Enter Aardvark. Aardvark is an Augmented Reality layer that incorporates multiple applications, or “gadgets”, on top of existing VR worlds. These gadgets can come from any number of different authors and all draw and work together to provide the user with a set of standard tools they can bring from one VR experience to another. No gadget takes up your entire view; they are all expected to work in concert, not just with each other, but also without interfering with the base-level world that the user is currently in.

An Aardvark gadget is inherently spatial. Instead of handing a 2D quad to be composited, a gadget provides a scene graph that can include quads, models, transforms, and any number of interactive components. This graph is then rendered together with all the other active gadgets’ graphs to produce a single layer on top of the base-level VR application. This eliminates strange depth behavior between gadgets. It also allows gadgets to interact with each other much more directly than is possible with overlay applications.

Aardvark also brings the ease of deployment and development from HTML, CSS, and JavaScript into VR. Each gadget is essentially an off-screen browser “tab”. Gadgets can be bookmarked and reopened later and shared via URLs. They also be run on-demand when the user wants to use a gadget without the need to install anything. Deploying a new build is as simple as uploading to a web server. The gadgets themselves can be entirely client-side, or can contact back-end services using any network or streaming capabilities they could use in a desktop browser.

Aardvark is an open-source browser that runs on Windows and can run over any SteamVR application on any SteamVR-compatible headset. It is now available as an alpha build you can download from GitHub. It is in good shape for developers to start building gadgets to fill in this empty middle ground in VR. If that sounds like you, go grab a buildread the documentationrun through the tutorial, and get started. If you run into trouble, hit us up in the slack and we’d be happy to help out.