Running Gamescope as a SteamVR overlay

Posted Monday, 31st of July 2023 at 22:30

SteamVR's VR view showing glxgears running inside an overlay.

Recently, Valve’s Gamescope microcompositor has been getting support for running as a VR overlay. If you’re not familiar with Gamescope, it essentially provides an implementation of what’s required to show interactive applications on a screen but with low latency and features like FSR upscaling. It’s used on the Steam Deck to run the gaming mode of SteamOS, and it can be used “nested” to run games inside a window while making them think they’re running on a full screen of any resolution you choose.

Crucially, applications run inside Gamescope are separated from the main desktop environment of the system. This means that applications running in the VR overlay mode will appear only in the VR overlay, and will run at the native refresh rate of the headset with very little latency which massively improves the experience.

As a VR overlay, Gamescope has plenty of uses. You can run games, desktop apps, even a full desktop environment like KDE Plasma. It’s a huge improvement over SteamVR’s default desktop mode for using applications in VR, as long as you don’t need them to actually show on your screen.

However, Gamescope’s VR support is still in relatively early stages. It hasn’t shipped yet, is likely very unfinished and will require building from source and doing some fiddling to get it working on current SteamVR. Fortunately, that’s what this post is here for!

Building Gamescope

Firstly, you’ll need to clone my fork of Gamescope. Along with a script and an icon to launch Gamescope with, it only contains one code change - switching the output format for VR sesions from A2B10G10R10 (10 bits per RGB channel) to B8G8R8A8 (8 bits for RGB channel). Trying to use the hardcoded 10bpc format will just give you a garbled mess.

Next you’ll need to make sure you have Meson and Ninja installed to build Gamescope. From there you’ll need to run the commands listed in the README file, which I’ll copy below:

git submodule update --init
meson build/
ninja -C build/

This will generate an executable at build/src/gamescope. You can test this quickly by running build/src/gamescope -- glxgears. If everything worked, you should see the classic glxgears on your screen.

Running Gamescope in VR

In its current state, Gamescope requires a number of command line arguments to be passed in order to appear as a VR overlay. I’ve provided a script, launch_vr.sh in my fork to make things slightly easier. You can use it by running ./launch_vr.sh -- <application name>. You can also set the screen and height with -w and -h, and change the apparent physical width of the screen in metres with -p.

If you don’t want to use the script or want to play with more options, here’s what you need to pass to Gamescope for it to work in VR:

--openvr enables the VR overlay mode.
--vr-overlay-icon <absolute icon path> specifies an icon, which is required otherwise Gamescope crashes.
--vr-overlay-key gs forces it to use the overlay key “gs”. It doesn’t have to be that specifically, but changing the default seems to be required for it to appear in SteamVR.

These options (and more) are nicely listed at the top of main.cpp in the Gamescope source.