Intel researchers used the Cityscapes Dataset, a collection of images of a central German city captured with a car camera, to make footage from the popular game more realistic.
Thanks to the new approach, in addition to processing the image generated by the Grand Theft Auto V game engine, the neural network also uses other processed data that the game engine has access to, such as the depth of objects in the scene and information about how lighting is processed and rendered. The result is a photorealistic video.
Intel researcher Stephen Richter stated that the “cornerstone” of their method is the convolutional network, a deep learning architecture commonly used for image processing. The team trained their convolutional networks on real images to transform GTA V graphics into a reality model.
For games, simulations, and movies, a huge amount of work needs to be done to model objects, materials, and so on to make them look realistic. But with the right convolutional networks set up, they can automatically generate these things directly from real photo collections.
Stephen Richter, Intel Researcher
The result is simply amazing: reflections appeared in the windows of the cars, the roads were paved with smooth asphalt, and the vegetation acquired a brighter texture. However, Richter notes that the updated color palette was related to recording equipment. The cityscapes were captured with a car camera with a distinctive green tint. Therefore, images that have been enhanced to look like cityscapes also take on this green tint. The views were recorded using a diverse range of cameras, including, for example, smartphone cameras.
In addition, Richter said the result was influenced by unrealistic aspects of GTA V. There is much less traffic in GTA than in the real world, whether in cities in California or Germany. The way Intel’s method is set up shouldn’t change that and won’t. It will not add cars or pedestrians if they are not in the original frame of the game. It’s the same with garbage on the street or on the sidewalk. Therefore, clean GTA scenes with less dirt and traffic than the real world may appear less realistic than real photographs.