In games with complex UI, creating a library that supports that UI and tools that allow designers quickly iterate changes could be a daunting and a time-consuming task. A task that you’d like to solve once and forever, and not write a new solution for each new project, or even for each new company you work at. Life, as it can be said, is too short to roll your own UI libraries!

So, in desperation, you begin to search for a third-party universal UI library. Once, this was the domain of CEGui and the like, but the current generation of game-specific UI frameworks is dominated by Scaleform and Coherent UI. Although, if HTML-based UI is what you want, you may simply choose Awesomium.

Unfortunately, this trio has some problems with performance, especially apparent on mobile devices. Just a few years ago, I’ve seen a nearly empty screen rendered with Scaleform take up 50% of frame time on iPhone4.

So, I always wondered why does no one use Qt for game UI – a library that is well known for being one of the less-fucked-up UI toolkits for desktop applications. This is not actually entirely true – a list on Qt Project Wiki has some games that do use it, but it’s mostly open source, small-time projects or ports of old games.

Of course, it’s obvious why you wouldn’t use Qt Desktop Widgets in a game. They are not at all suited for hardware-accelerated rendering, and while you may try to work around this problem, it’s far more troubles than its worth. However, for a long time, Qt had a different library, the one that allows drawing hardware-accelerated widgets: QtQuick.

Not only it is said to be specifically targeted at mobile devices, it also has a very nice text format for describing UI screens, which is well-suited for quick iteration.

Still, I have never yet heard of a professional game developer using QtQuick. As I could find no posts or articles that could give me the reason for this, I just had to check it out for myself.

Why NOT use Qt:

The first problem and the farthest one from the technical mind is the question of licensing. Qt is dual-licensed under LGPL3 and commercial license. This means that if you’re interested in platforms that do not allow dynamical linking (like iOS), you WILL need a commercial license unless you’re prepared to make object code of your game available for possible re-linking with a different version of Qt, and even then it is impossible on iOS.

It costs 79$ per a developer who uses Qt per month. As far as I could understand, “use” includes just “building code that uses Qt”, so you’ll have to pay for each programmer on your team. Still, it’s not that much money, compared to alternatives.

Also, if you choose to use Qt in your commercial game, you should apply for a commercial license from the very beginning. If you fail to do so, you will later need to negotiate special conditions with Qt sales department. The reason for this is obvious: if you could develop the whole game using the free version of Qt, and then just buy a single license for the final build, you would do it.

Now, onward to more technical arguments. The main drawback of Qt is its size. A nearly empty desktop application that uses QML will have a size about 40Mb (with dynamic linking). On Android, the size will be around 22Mb (in unpacked form; APL will be slightly smaller), which is just way too much for a mobile application! Statical linking can reduce the size dramatically, but see above about licensing.

On Android, Qt offers a kind of a crutch that allows multiple applications share a single installation of Qt libraries called (Ministro), but for obvious reasons, it’s not a good solution and one that cannot be made to work on iOS or Windows Phone.

On the other hand, before you dismiss Qt as just another bloated dinosaur, you should consider that all its direct competitors like Scaleform and Coherent are all of the similar sizes. Unity is slightly smaller (an empty APK with Unity will be around 10Mb). So, Qt only loses in this category if you compare it with simpler, custom-made solutions (i.e. an UI library you wrote yourself).

Another potential drawback of Qt is that it’s not ready to be used in Web applications (via Emscripten). A work is in progress to make it work with the Web, but it’s not quite there yet, as far as I know. It’s not something a lot of developers would care right now, but at my current job, we are targeting Web platform sometimes.

Why use Qt:

The main pro-Qt argument is QML. QML is easy to write and extend, and Qt has a visual designer for it. Also, it has a good set of widgets available.

I must point out, however, that Qt Designer is not Windows Forms constructor. Even for the base widgets, it does not allow one to fully change every property of a widget without switching to the textual representation. For example, you will not be able to add normal/pressed images for a button from Designer.

It’s not a fatal flaw, but it means you can’t simply fob off UI design on someone who only knows how to edit UI visually, i.e. an artist.

I haven’t yet tested QtQuick performance on mobile platforms, but it is my opinion that it should be satisfactory. Also, the recent Qt 5.7 release promises to speed it up even more with the new, special QtQuick Controls 2.0, designed for mobile.

Integration & Usage

Let’s move to more interesting things. In this section, I will try to describe programming and architectural challenges you will face if you choose QtQuick/QML as your UI library.

The main loop

The first thing you’re going to notice is that Qt likes to own the main loop of your application. Many game engines also want to do it. Someone has to give. In my case, Nya engine, which is a 3D engine we use at work, has no problems with letting Qt have the main loop to itself, and after a minimal initialization is able to use Qt’s OpenGL context to render the 3D scene.

Still, if your engine just can’t let go of the main loop, it’s not the end of the world. Qt might be made to relinquish the possession of the main loop, as long as you sometimes call processEvents method of Qt’s application class. A post on StackOverflow has an example of how to do this, along with critique.

You can also draw render your 3D scene into a texture, and then use this texture as a node in QML’s scene graph, as described in Mixing Scene Graph and OpenGL, however, this can have performance cost on mobile devices. Also, you can use QQuickRenderControl to better control QML’s rendering, and even use different (shared) OpenGL contexts for UI and the game, eliminating the need to tiptoe around OpenGL state.

Now, if you let Qt have the main loop, a question arises: when do your engine get to render the scene? For that, QQuickView object, which is used to load and render QML-based UI has two signals called beforeRendering and afterRendering, which you can subscribe to. The first is emitted before UI is rendered, and this is the place to draw your 3D scene. The second is emitted after UI is drawn, and you might want to draw some overlaid particles here, or maybe some parts of the scene that should exist over UI (for example, a character’s 3D doll in Equipment screen of an RPG).

An important note: use Qt::ConnectionType::DirectConnection type of connection, when subscribing to these signals, or you will get errors about access to OpenGL context from a different thread.

Also, do not forget to forbid Qt from clearing buffer before drawing UI by calling setClearBeforeRendering( false ).

Qt is a very economy-minded library. It does not redraw UI if nothing changed in it. If it does not redraw UI, before/afterRendering do not get called, and we do not render 3D scene. So, in every call to afterRendering, it is useful to call update() method of QQuickView, which will force it to be redrawn during the next frame, which is what we want.

More about rendering

You should remember that your engine now shares OpenGL context with Qt. Which means you should be careful with it. Firstly, Qt will change it during its rendering calls. Which means you’ll have to make it current before drawing anything, and restore YOUR OpenGL state. In Nya engine, it’s as simple as a call to apply_state(true), but in your case, it might be more difficult.

Secondly, after you’re done with drawing your stuff, you should restore context’s state to what Qt expects by calling m_qt_wnd->resetOpenGLState().

Also, since the context will be created by Qt, and not by your engine, you must make sure your engine does not try to do anything OpenGL-related before the context is ready. To do so, subscribe to openglContextCreated. Or just to all your initialization in the first beforeRendering call.

Interactions between C++ and QML

So, our code now draws a 3D scene, and then Qt draws its UI. But they can’t do anything with each other. That won’t do.

If you’re using QtCreator, or some other IDE that has Qt’s code generation step (MOC) integrated, you’ll have far fewer problems here. Just hook up signals and slots of QML objects to C++ objects, and enjoy the result.

If you don’t use QtCreator, you can generate a project for your favorite IDE using Qt’s qmake feature like this:

To generate a Visual Studio project from a .PRO file:

qmake -tp vc path/to/project/file.pro

To generate an XCode project from a .PRO file:

qmake -spec macx-xcode path/to/project/file.pro

However, even if you do not have MOC, you still can get it all to work without too much sweat.

There (QML -> C++)

Qt presently has two ways to connect signals and slots. One, the old one, is to specify their names, and the new one is to specify pointers to member functions, or even lambda function (as a slot). Unfortunately, the two does not mix, and QML only supports name-based connections, which, on C++ side, require MOC.

Which just means we can’t connect QML signals to our own objects. But we CAN connect them to Qt’s own objects. Here’s a nifty technique, that you’ll probably end up using anyway, even if you have MOC:

There is a class called QSignalMapper. It has a single slot called map(), to which any number of signals can be connected from different objects. For each object that is connected to the mapper, you can set mapping, from a pointer to that object, to some identifying information (pointer to QObject, a string, or integer). If you do, then, when map() slot gets triggered by a signal, it will, in turn, emit mapped() signal, with argument corresponding to the object that sent the first signal. Here’s how we’re going to use it.

Let’s create a QSignalMapper for each type of signal that can be emitted by QML (“clicked()” for buttons, “accepted()” for text fields, etc). When we need to subscribe to a signal from an object in QML, we first connect that signal to map() slot of QSignalMapper and then connect QSignalMapper’s mapped() signal to our own handler (maybe even lambda function!). It works, since QSignalMapper is a C++ class, and therefore can be used with the second form of slot/signal connection, which do not require MOC.

Here’s an example:

QObject *b1 = m_qt_wnd->rootObject()->findChild<QObject*>( "b1" );
QObject::connect( b1, SIGNAL( clicked() ), &m_clickMapper, SLOT( map() ) );
QObject *b2 = m_qt_wnd->rootObject()->findChild<QObject*>( "b2" );
QObject::connect( b2, SIGNAL( clicked() ), &m_clickMapper, SLOT( map() ) );

m_clickMapper.setMapping( b1, "b1" );
m_clickMapper.setMapping( b2, "b2" );

QObject::connect( &m_clickMapper, static_cast<void(QSignalMapper::*)(const QString&)>(&QSignalMapper::mapped), [this, t1]( const QString &sender ) {
    if ( sender == "b1" )
        m_speed *= 2.0f;
    else if ( sender == "b2" )
        m_speed /= 2.0f;
} );

Have a look at the test project’s code at the end of the article to see an example of a wrapper around that mechanism, that makes it easier to use it.

If you have MOC, you can also pass a C++ object into QML context, and call its functions, slots and signals directly from QML:

SignalsHub signal;
engine.rootContext()->setContextProperty(QLatin1String("signal"), &signal);

And back again (C++ -> QML)

Here, things would not go smoothly. We simply can’t connect a C++ signal with a QML slot without MOC. On the other hand, why even bother?

We have at least two (or maybe one-and-a-half) ways to manipulate QML objects from C++ without using signals. First, we can set properties of QML objects directly, via calls to setProperty( “propName”, value ) method. This is slightly limited, but the main problem about this is that you can’t call setProperty from rendering thread (meaning, you can’t call it from before/afterRendering).

To work around this limitation, you can place calls to setProperty from the main thread, somehow. For example, by having a QTimer call your UI-updating function periodically. This approach will lead to the need for synchronization, but it’s workable.

However, there is another way. We can’t send signals to QML, and we can’t easily set object’s properties. But we can invoke object’s methods! So, if you need to interact with an UI object, you can define a method inside it (for example, setNewText), and then call it from C++ via invokeMethod:

QVariant a1 = "NEW TEXT";
m_fps_label->metaObject()->invokeMethod( m_fps_label, "setText", Q_ARG(QVariant, a1) );

Actually, you can also use invokeMethod to call QML object’s slots and signals. Just don’t forget the Qt::QueuedConnection parameter, to make sure the call happens on the main thread.

Important: in this case, invokeMethod may only accept QVariant as arguments. And if you need the method to return something, you should use Q_RETURN_ARG( QVariant, referenceToReturnVariable ) macros.

Resources

Now, we have a basic integration between QML and our game working. But there is another aspect we should consider: where does QML get its resources?

If you have simply specified an image name in your QML file, then it is read from disk. But what if we want QML to get its images from our game’s resource system like our engine does? Resources can be compressed, encrypted, preloaded into memory – there is no end of reasons to use a resource provider instead of accessing them directly from disk.

What we need, is a Virtual File System. Unfortunately, Qt’s QAbstractFileEngine got deprecated in 5.x release for “performance reasons” (a bit of discussion). I wonder what kind of code in their implementation lead to this, because all our games work with various resource providers hidden behind a kind of VFS, and have no problems with performance because of this…

There is no replacement for QAbstractFileEngine, but if you like to live dangerously, you still can use it because it was only made private, not removed completely (yet). Just link private Qt library and include private headers to make it work.

If you want to stay within the current library’s limitations, you don’t have many options, but there are some.

QMLEngine, the main class for loading and displaying QML files, allows you to register QQuickImageProvider class, which would allow you to load images (and only images) from your resource system.

To make QMLEngine use your QQuickImageProvider instead of direct file access, you need to specify the path to the image as “image:/my_provider/images/button_up.png” instead of simple “images/button_up.png” (where “my_provider” is the name of the provider which you used when registering it with QMLEngine). This will break compatibility of your QML files with Qt Designer because it doesn’t know anything about your custom image provider. It’s probably can be fixed by writing a plugin for the editor, but there is another way.

Along with QQuickImageProvider, QMLEngine allows registering of another useful class, QQmlAbstractUrlInterceptor. This Interceptor intercepts all URLs used by QML file before they are used, and can modify them in any way. Which is just what we need! Whe we see URL type UrlString, and maybe “.png” in path, we change it to use our image provide:

QUrl result = path;
QString short_path = result.path().right( result.path().length() - m_base_url.length() );
result.setScheme( "image" );
result.setHost( "my_provider" );
result.setPath( short_path );
return result;

setScheme – makes QML search for an ImageProvider
setHost – name of our provider
srcPath – unfortunately, Interceptor receives URLs after they were appended to QMLEngine’s baseUrl property, which is usually equals to QDir::currentPath(). So, to get the original path back, we’ll have to cut that part of path out, to replace “file:///C:/Work/Test/images/button_up.png” with “image:/my_provider/images/button_up.png”.

Resources 2 – resource harder (and fail)

Here’s a short tale about how I (nearly) managed to get QML to load ALL resources from my resource system without using QAbstractFileEngine.

QMLEngine accepts yet another useful helper class, called NetworkAccessManagerFactory. Basically, this allows us to control how http requests are handled. What if, thought I, we could replace all requests to QML resources with network requests, and then instead of network load resources from our resource system by using a custom NetworkAccessManagerFactory (or, to be more precise, a custom NetworkAccessManager and NetworkReply)?

It nearly worked, too! I managed to get my fake “http” requests work by replacing “scheme” part of URLs in Interceptor with “http” and then by handling them in my custom NetworkAccessManager. However, a single type of file called qmldir has a nasty assert in code, that disallows loading of said qmldir from anywhere, but “file:///” paths. I was unable to work around it, and without the ability to load qmldir, we are unable to import QML modules from our resource system, which renders the whole thing useless.

Resources Redux

Actually, Qt has its own resource system! It allows you to compile a bunch of resources into an rcc file, and then use them, by supplying “qrc:/” prefix.

There are two ways to register a resource with Qt. Both are overloaded calls to a static function QResource::registerResource. The first one takes a file name. This one is simple: point it to an rcc file on disk, use loaded resources. Nice, but not what we want right now. The second one accepts a pointer to “rccData”. The documentation helpfully notices that it “Registers the resource with the given rccData at the location in the resource tree specified by mapRoot, and returns true if the file is successfully opened; otherwise returns false.”. Er, what? What file, what are you talking about, documentation? Go home, you’re drunk!

In reality, this entry is a victim of a bad copy-and-paste. This version of call actually accepts a pointer to data loaded from existing rcc file. For example, from an rcc file we just pulled out of our own resource system. Bingo? Well, no, not quite.

You should note that to pull this through, we need to have the whole rcc file in memory. And since it contains images, it might be a BIG file. On platforms where memory is precious, this is not the way to go.

Still, it can be made to work, if, for example, you carefully separate images for different screens into different qrc files, and load/unload them as required.

Also, please note that this version of registerResource accepts a pointer to data, but not the data size. That’s because Qt likes to live dangerously. Inside, it treats the pointer as guaranteed to have at least a valid header (4 magic bytes + some information about data size and other properties). If you pass it a pointer to junk, well, access violation, here we come!

This method can be used in combination with one described above (UrlInterceptor+ImageProvider) to achieve a workable solution for loading QML resources from the resource system if you put only QML files and modules into rcc file.

Deployment

If you think that the game is ready to be deployed after you wrote all the code and packed all resources, well, you’re wrong. Qt is a big library, with lots of DLLs and QML modules that need to be shipped with your application.

QtCreator handles this step automatically, but if you’re not using it, you’re still, fortunately, not down to hunting all necessary files manually.

Qt helpfully provides utilities called windeployqt, androiddeployqt etc. to help with deployment. They all have different command line syntax, but they work. For example, windeployqt only needs a path to your executable and to your QML folder to prepare all necessary files and put them into specified folder.

Conclusions

So, CAN we use QtQuick/QML to create game UI? Yes, we can. The main question is, should we? And the answer to THAT one depends a lot on your circumstances.

For example, if you’re ready to use QtCreator, a lot of small, but annoying problems will just disappear, and the development will go much smoother. However, if you wish to remain with your beloved Visual Studio, XCode or vi, prepare to experience small pains along the way.

If you’re developing a game with hundreds of megabytes of resource, than 25-40Mb of Qt’s DLLs will not matter all that much. However, if you’re working on a mobile platform, especially in a market with small recommended package size (like China or Iran), then the size of Qt could be unacceptable.

If you’d like to avoid rolling out your own UI library and editor, QtQuick/QML can have better performance than its competition. However, if you’d like to use a familiar technology, like Flash or HTML, you’d better stick with Scaleform or Coherent.

Integration of Qt into a new project should not be too hard, but moving an already-established project to use QtQuick for UI might prove troublesome.

In the end, the decision is yours 🙂

An example for integration between Qt and Nya engine can be found at
GitHub MaxSavenkov/nya_qt.