Making games is hard. It’s even harder when you work alone or in a team consisting of only a few people, especially when making your own tech. That’s why every little victory counts. In ‘Little Victories in Game Development’ we take a look at things that went really right; things that payed back their development cost many times over, and made our work much easier or more fun!

In the last part of Little Victories in Game Development we talked about an issue with quickly finding and selecting certain objects in the level editor. We briefly mentioned the fact that game levels can have many many objects, sometimes tens of thousands. We solved the issue with selecting certain specific objects very elegantly using object selection groups. However, sometimes we don’t know which objects we want to select. What if we want to group all “small rocks” together, or we want to know how many modular building pieces using a specific material are in a given corner of the map? That’s what level data queries are all about.

Big data

Lately I have been adding lots of foliage and grass to one of the demo levels for Project Slide. You can paint foliage and grass in the level editor by clicking and dragging with the mouse. The foliage paint tool looks like this:

The blue-ish circle is the tool radius. New foliage objects are painted inside the circle, and the editor will rotate and scale the objects according to the rules set up for each foliage object type. You can later use the foliage scale tool to “paint a new scale” onto foliage, or you can select individual objects and scale/rotate/move them around as you see fit. Foliage objects are true game objects in the Basis level editor. This may not seem like a big deal (game objects are relatively lightweight in the engine) but it means that each patch of grass gets its own name and it will show up in the level objects window.

For the actual game, foliage objects are packaged into a much smaller and lighter data structure.

As you can imagine, this contributes greatly to the number of objects in the level. Selecting the foliage objects we just painted looks like this:

That’s 30-40 objects just for this small area, and these are relatively large patches of grass. There are grass patch types with only a few blades per object. After playing around with the foliage painting tool for a while the Level Objects window quickly starts looking like the image on the left. Every foliage object gets its own entry in the list.

Now, you might consider this bad editor design. Why would the editor create full game objects for each patch of grass, and you would have a point there. However, this approach was chosen because it is very flexible. The foliage painting tool is just a specialized version of the object painting tool. You can make any object a “paintable” object, everything from small rocks to space stations. Also, all the normal editor commands work with the foliage objects as well (selection, transformation, duplication etc.).

Okay, so we have a lot of objects in the editor. What does that mean? First of all, it means that finding specific objects can be tricky. We can select objects by clicking on them in the scene, we can filter objects by name and we can organize them into layers so that the full list of names does not show up in the window, but it still means working with a large set of data. Futhermore, it means that the level files get larger and heavier with time.

Obviously, this can be a problem with any object type, not just paintable objects such as foliage. Game levels typically contain large amounts of props, eg. rocks, trees etc.

Clearing out the weeds

After working with the level for a few days and watching the object count grow, I started wondering: “Is the scale of the grass patches correct?”, “Do some of the objects get so small that they are very hard to see?”, “Could I just remove some of them?”. When adding foliage using the paint tools, you can sometimes accidentally scale down objects too much, making them quite pointless.

I started going over the areas with lots of grass, trying to spot objects that were too small to contribute to the visuals, and removing them. This was a tedious thing to do, as you can imagine. It is manual labor, and the positive effects are small compared to the time spent doing it.

At this point it should be noted that Basis level editor documents are really just JSON files. The format looks something like this:

{
	/* Level-wide data fields omitted... */
    "layers": [
        {
            "name": "Base",
            "autoLoad": true,
            "locked": false,
            "gameObjects": [
                {
                    "name": "Base:Terrain1",
                    "type": "Terrain/Terrain",
                    "pos": [ 0.0, 0.0, 0.0 ],
                    "rot": [1.0, 0.0, 0.0, 0.0 ],
                    "exposedPropertyValues": {
                    	/* Component properties here... */
                	}
                }
            ],
            /* More objects here... */
        },
        /* More layers here... */
    ] 
}

I have omitted certain fields which are not important for this discussion. The file contains an object with an array of layers. Each layer contains an array of game objects. Each game object contains its name and type, its transform (if any) and a key-value store of properties for its components. It’s all human-readable information. Or perhaps more importantly, it is information readable by any program that can process JSON data.

My first idea was to write a few regular expressions that would give me all “grass patches that are small enough”. I would open the file in an external text editor, run my cleanup and reopen the file in the level editor. That didn’t strike me as very user friendly. Luckily I never got started with this. I realized that everyone uses json, and that there probably would be tools for this kind of thing.

Enter jq

Through a bit of googling I found jq. The author describes jq as a “lightweight and flexible command-line JSON processor”, but it is a turing-complete functional programing language as well. I am not going to go too deep into how jq works. There is an excellent tutorial and a YouTube video about it. However, you need a little bit of jq knowledge to follow along.

jq can process structured data, transform it and output it in another format. For example, you can give it an input file, an output file and a jq program to use. A jq program consists of one or more “filters”. Each filter accepts one piece of input data and it can output 0, 1 or many pieces of output data. Filters can be combined into pipelines where each filter is run, one after the other. The data is passed from one filter to the next, allowing you to set up very complex data transformations (although I find that you seldom have to do anything terribly complex).

Without having to do any kind of pre-processing of my level data, here is a jq program which returns the number of layers in the level file:

.layers | length

The “.” is the identity filter and it simply forwards the data as-is. The “.layers” means ‘get the member called “layers” of the input data’. We then use the pipe ("|") to forward the layers array to the built-in “length” filter, and the return value is the length of the layers array. What about the number of game objects? It gets a little trickier, but not much:

[ .layers[].gameObjects[] ] | length

We start by getting the layers again, with “.layers[]”. The empty [] after the filter means that we return each element in the layers array as a separate piece of output, rather than the whole array as a single piece of output. Then, for each layer output by the filter, we do the same for the gameObjects. Now we have “.layers[].gameObjects[]” which means ‘get all game objects of all layers and return them as separate pieces of output’. That’s not what we want to do. We want the number of them, so we wrap the whole filter in square brackets to create an array where each game object is an element, and finally we forward that array to the built-in length filter.

It took me about a day to grok the jq language, after which I had no problems writing queries. It bears repeating that I did not have to make any changes to the level file structure. Even if the files are tens of megabytes in size, jq has no problem running my queries in under a second.

About those weeds…

Let’s get back to my original problem. How do I find all grass patches that are too small to be visible, and thus can be deleted? I started by placing a few grass patches of a given type in the level, trying to determine how large they should be. For the object type named “Foliage/TerrainColoredGrassPatch1”, I decided that anything with a scale below 0.7 is too small and should be removed. I then came up with the following jq program. It returns those objects, as an array of names:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Helper filter which returns all game objects of all layers, one after another.
def getAllGameObjects: .layers[].gameObjects[];

# Helper filter which returns the scale for an object.
def getObjectScale:
    if .exposedPropertyValues.transform.mScale.value[0] == null then
        1.0 # The scale is not set, ie. it is implicitly 1.0
    else
        .exposedPropertyValues.transform.mScale.value[0]
    end;

# All the filters are wrapped in [] which means that the results will be an array.

[
    getAllGameObjects |

    # Select only the objects of the given type.
    select(.type == "Foliage/TerrainColoredGrassPatch1") |

    # Select the objects with a scale smaller than 0.5
    select(getObjectScale < 0.70) |

    # For each object, return the name.
    .name
]

You can define you own filters in jq using the “def” keyword, and here you see two of mine. The first one is really just a nicer way to “get all game objects”. The second one, getObjectScale, is a bit more complex. An object which has not been explicitly scaled won’t have any scale in the json data, which means we need to consider that case as a scale of 1.0.

At the moment the editor forces uniform scaling, which means we can just read the scale from the X component of the mScale vector, at index 0.

You can see me digging down into the properties of the components of the object. We get the transform component and read its mScale member. The program gets all game objects, selects those that are of the type “Foliage/TerrainColoredGrassPatch1”, then selects those that have a scale below 0.7, and returns the names of each object as an array. The select() function is a jq built-in function. It filters out any input which does not pass the condition given to it.

I decided jq is kickass and integrated it into the level editor in about a day. You can now run jq programs as queries on the data of the current level by using the integrated jq editor. It is really just a code editor which runs the jq exe as a separate process and looks at what comes back. The jq programs won’t write to the level data at all. Instead I have a drop-down menu where you can select how you want to use the output of the jq program. The currently available modes are “Print” and “Select Objects”. The Print mode simply outputs the data to the editor log, which is handy for writing and debugging jq programs. It also allows you to ask questions like ‘How many objects of this type are in this area?’ and you’ll get the answer in the log. The Select Objects mode expects the output of the program to be an array of object names and will select those objects in the editor. More modes can be added later if needed.

Running the jq program which selects all objects of type “Foliage/TerrainColoredGrassPatch1” with a scale below 0.7 results in this selection:

1172 objects were selected. Those would have been a pain in the ass to find by hand. Of course, you don’t have to remove them. Just being able to select stuff based on a query is incredibly powerful. Those objects can be scaled bigger, moved to a layer which is unloaded for later use, or whatever you want. The sky is the limit! Also, I can quickly run the query multiple times with different scales or add another filter to it, to identify the correct set of objects.

Let’s do one more. Here’s is a program which selects all objects that have the word “Tree” in its name and sits at a Y coordinate of 25 or above in the game world:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

[
    getAllGameObjects |

    # Select the objects the names of which contain the given substring.
    select(.name | contains("Tree")) |

    # Select the objects whose position Y coordinate is at 25 or larger.
    select(.pos[1] >= 25) |

    # For each object, return the name.
    .name
]

The contains() function is another one of jq’s built-in functions. As you can see, by making small changes to the script we can query completely different things. Of course, since jq is a full programming language this is barely scratching the surface of what it can accomplish.

Conclusion

Adding data queries to the level editor turned out to be fairly little work with big benefits. If you have felt the same frustrations working with large data sets, you might want to consider doing something similar. For me it was especially easy since I could simply run my level data through jq. I am sure there are other similar programs out there to help you, depending on your data format, engine etc. Even if you are using a premade engine such as Unity or Unreal you should be able to add querying capabilities by dumping the data to a query-friendly format and running queries on that.