Under the Hood: New Lighting System

The changes that we are introducing with the 1.40 update for both American Truck Simulator and Euro Truck Simulator 2 have been visible to the public for some time now, in the currently still ongoing Open Beta builds. By far the most visible change is the new lighting system, which has turned out to be the largest graphical rework for us in years. In today's blog post we would like to reveal more about what we have been through to make this change happen; let's take a look under the hood once again.

Let there be light!

Our 3D engine's approach to representing light and color in scenes - and in the rendering buffers that end up being displayed on-screen - has been established over a decade ago. It was an era when the state of the art was to represent all RGB values in the 0-255 numerical range, to fit inside a byte of information. That's how all values were defined, whether in a texel of a texture map (the "skin" of a 3D object, a tree, vehicle, or skybox), or in the encoding of a light source's intensity (from a vehicle's tail lights to street lamps, all the way to the sun intensity), and combining all these in the rendering stage into a pixel; it was again the representation in the display buffer to be shown on the screen, with only 256 shades of each base color.

In recent years, the explosion of available memory in 3D accelerators, the progress in computational power, and flexibility in shading languages for programming 3D cards' operations on fragments have opened the way to store and compute color and light values at much higher precision and range - with floating-point numbers. Instead of just layering color filter operations, which was, essentially, all the tricks that we had available in the 8-bits per color component era, the switch to using real numbers is quite a revolution in how a game engine can truly represent light sources in realistic ranges and ratios. In the real world, the difference between a light bulb's intensity and the sunlight is many orders of magnitude. If we are able to store the intensity values of light sources properly, and more importantly to perform all the light vs. surface operations at their proper ranges of values, without a major loss of precision, we can do true high dynamic range (HDR) light in the graphics pipeline. When we started experimenting with the new technique some 18 months ago and saw the first results, there was no way back for us, we knew we wanted this in the engine. 

What's new?

We had to start with the light sources (and for a while, we were hopeful that this will suffice). Each and every light source is now defined with a realistic value gleaned from real-world measurements (we have done a ton of sampling with a light meter over various daylight and weather settings) and reference tables. In reality as well as in the game world now, tens of thousands of lux units now brighten day scenes, while at night the ambient brightness approaches just a few lux units. Basing all light source values on reality gives us way more confidence as opposed to the old days when a new light would just fit into the system. It's no longer a question of should it be a 150 or 170 with no actual meaning to the values. Now we know and have tables to pick for a street lamp or a neon light at a gas station. In light computation formulas, we can now go with proper quadratic drop attenuation, to have lights behave in a physically-correct manner. With the higher precision, we can even go as far as introducing a small random variation in light intensity to otherwise identical light sources, to simulate a street's night feeling better, without having unnaturally-identical lamps.

When all the internal light computations are done, we have to adjust the resulting values for the screen by emulating camera exposure. Much like a modern camera in your mobile phone, we need to take all the rich values and variety of incoming light - sun or shade, early morning, high noon, dusk, or deep night - and "squish" or "expand" the values into the range appropriate for the screen. Our exposure emulation system has a critical impact on the final look and feel of the game - it's partly physics, but also partly about a subjective artistic feeling too. It has to be adjusted dynamically not just based on incoming light from the game world, but it also has to take into account the difference between the light inside and outside of a vehicle's cabin, to allow dashboard instruments to be legible. The human eye is an incredible creation, able to adjust and adapt to a vast range of light intensities as focus switches between objects. We had to use a lot of smart math when analyzing the light histogram for each frame and between frames to fit all these values into the screen's inevitably limited dynamic range and color gamut, in order to make everything that's portrayed both believable and discernible. Some improvements were a cherry on top, like the fact that with the internal range of floating-point buffers of cube maps, we can have much shinier reflections off of glossy surfaces, such as water bodies, glass, or car paint. We have followed this development path in a small team happily until the realization that while the intermedia results were promising, we won't be able to pull this off without re-balancing ALL the existing materials/textures on all game objects. When a ray of light is cast on the surface of any object, what is reflected or refracted depends not just on the properties of the incoming light, but largely on the properties of the object's surface. Over the many years of development, we have accumulated tens of thousands of textures, some from photographic sources at various settings, some painted by hand, some through the process of procedural generation in various smart tools. As the years went by, the authoring of these sources has been subject to various schools of thought, or different creation pipelines, but the "old light" system was held together by the proverbial duct tape and was in any case imprecise enough to reveal certain imbalances that were present. The new exposure emulation system is way more sensitive and simply requires physically correct brightness albedo values. Our concrete surface can no longer be some sort of grey, and a tree cannot just be nice and green. And that was when a vast majority of our art team had to be involved for a major overhaul.

What had to be changed?

Once our small pioneering cross-discipline team implemented all of the necessary code and GPU shader changes, which took many months of experimentation, we had to involve a lot more people. First, we had to readjust all the light sources in the world and any vehicle lights to the new logic. In some cases, it was a relatively trivial numeric conversion somewhere in a table. In others, it may have required a map designer to revisit a particular location to make the manual adjustment. In many cases, it meant that a game object like a whole factory or delivery yard with pre-defined light sources had to be re-opened, adjusted, and re-exported again in a 3D tool like Maya. We are talking thousands and thousands of such retouches, in some cases involving not just a few dozen "numbers" defining lights, but also major changes to lightmaps of the objects, where fake shadows and light effects were "baked in" for a better look in the old light system. Finally, for some weeks, the majority of all our teams had to stop working on anything else related to our future projects, and go over all the game objects, vegetation, vehicles, and effects; re-balancing things, again and again, to sync them on a shared agreed-upon albedo reference table. We had to revisit the fundamentals of many sub-systems, very notably the vehicle lights and bulbs taking many iterations and fine-tuning to make them look good at various daylight settings. We have a new set of skyboxes at a higher resolution, and a slightly richer variety of weather, too. Ultimately, no stone was left unturned, as we had to evaluate and adjust every single element of the games.

What does this all mean?

The new light system and the start of our new physical light model is a critical step towards a complete and better physical model of the whole world in our games. We are not quite there yet with true physically based rendering, but the 1.40 update represents a major step in the right direction. Cleaning up the old art and basing all light intensities and operations on fundamental physics make future systemic work in this area finally possible. As with many things in long-term development, while this step is a minor revolution, it is at the same time only a milestone towards a bigger vision. 

From the graphics artists' point of view (but this applies to mod authors, too!), we are introducing a new concept of creating and keeping all game art "in check" and in balance with the rest of the art. Materials based on photos, frequently used as sources of in-game art, now have to go through a calibration process before they are applicable, otherwise one may end up with a surface that is either too dark or too bright, affecting the exposure and the perception of the general scene. Often, it's worth exploring alternate material creation methods, like the high-resolution procedural generation in tools like Substance Designer. More trust is placed into the engine's rendering system now; rather than faking and baking subtle shadows and light effects into models. Between new light and SSAO, our shader code now plays a greater part in creating the final look; this results in a bit less responsibility, control, and possibly work from the artist. This may be both good and bad depending on what one is trying to achieve in each particular case.

As with each major change, there were a ton of teething problems, a lot of dead-ends to reverse from, and the amount of effort we have been through far exceeded the expectations we had a year ago. We still have a bit of Open Beta ahead of us to iron out the last few wrinkles. But even now, we are very confident that the changes are worth all the hard work. Just wait until you drive your truck in the early morning hours along with the coast of Italy, or on a nice late afternoon in Sweden, in a drizzle in Washington state, during scorching noon in Spain, or pulling into a well-lit truck stop deep into the night. Through all the process of changes, we tried our best not to cause any major increase in the load on the CPU or GPU. A few of the new effects have a slightly higher performance cost, but overall we believe the changes are very well balanced, or allow for sensible trade-offs in game options, to keep the performance acceptable. We definitely do not want to lose players and fans that play our games without the newest hardware by pushing too hard on graphical effects. We are looking forward to bringing you further improvements and new features, so hold on to your steering wheels!

More of a fan of changelogs? Let's look at:

The major internal changes

• The scene is lit using real-life intensities (illumination: tens of thousands of lux for the daytime,  tens of lux at night) 
• To get light ratios into play - the rising sun overpowering street lamp lights and truck headlights, etc. This means we had to adjust all light sources and also readjust every emissive material in the game.

• We have a new display mapping function for better artistic control (exposure, contrast, toe, shoulder) 
• We needed to get those real-life intensities onto the computer screen in a similar way as a camera's exposure works. 

• Improved and more sensitive histogram for exposure selection
• This is due to the actual exposure being selected differently in every scene, and this varies also with differing light levels and the current weather type. 

• Light source (street lamps, vehicle lights) changes. 
• We have switched light sources to a more physical attenuation ('inverse square falloff'). 
• We are in the process of switching to a physical-unit-based configuration to reduce guesswork. 

But also many smaller visual tweaks 

• Relightable reflection 
• Reflections in shop windows and smooth surfaces better matches the time of day and weather 

• Improved reflection behavior 
• On truck paint, shop windows, and glass in general 

• Sun horizon shadow
• Visible when the sun rises or sets 

And a zillion other small tweaks, fixes, and changes.

As you can imagine by now - all this has been a tremendous amount of work for our team. The initial bright spark (pun intended) that made this all possible came from a small team of programmers and a visual effects artist (pictured below). They have kept hacking, iterating, and laying out plans until the whole team had it "easy", and all that was left to do was just to throw in many years of additional man-hours to adjust everything to the new logic. 

The new light system, just as the implementation of FMOD support for the SFX part of our games, is still under further development, polishing, and tuning. And there is still a lot of work ahead of us. At this juncture, we feel that it's already more or less in a presentable shape, and you may experience it too if you decide to give it a try in an open beta build right now on Steam.

So please, if you'd like to help us - put on your favorite sunglasses and help us catch the perfect sunrise! We're looking forward to reading your thoughts and opinions on the new light system. Thank you very much!