11. Building Simulations with Generative Artificial Intelligence

While the scenario just presented is currently largely in the realm of speculative fiction, the technology to produce such a simulation has made rapid advances over the last few years, suggesting that it may shift from speculation to practical realisation in the next decade. Further, in addition to being able to provide virtual reality (VR) simulations of familiar environments for training purposes, it is potentially possible that such experiences could even be simulated with generative AI using augmented reality (AR) in situ. Currently, however, building realistic simulations for immersive technologies—such as VR and AR—is a complex and time-consuming process.

Three-dimensional (3D) simulations usually begin with modelling the geometry and textures of every object that will appear in the simulation. Despite advances in 3D modelling techniques and the wide availability of existing models online, this is a highly specialised and time-consuming task. Beyond the modelling of physical form, a simulation also needs to model behaviour. For the simulation to be realistic, that behaviour must be accurate to reality or at least be plausible to reality. Lastly, for the simulation to be immersive, it must provoke a strong sense of presence, convincing any participants that what is happening is “real”. This typically involves high-fidelity images, sound, haptics, kinaesthetics and beyond.

Our speculative scenario made use of sensory experience beyond mere visual simulation, incorporating sound, proprioception, kinaesthetic, haptic and even olfactory simulation (e.g. the smell of smoke). Not all of these sensory modalities are currently well synthesised by AI, and a heavy commercial focus on visual and audio synthesis currently dominates the well-known foundational AI models (Bommasani et al., 2021).

The idea that through new generative AI technologies we can construct high-fidelity simulations of real-world events presents a step change in developing simulation systems. Rather than labouring over detailed 3D models, building complex simulations by hand or using digital media such as cinema or photomedia to construct a rich simulation experience, generative AI potentially presents the opportunity to deliver high-fidelity simulations simply by describing them in language.

Current text-to-image (TTI) systems rely on diffusion models. These models are trained by adding noise to a training set, forcing the model to learn how to convincingly reconstruct image representations. This approach has significant advantages in image generation quality over previous methods such as generative adversarial networks (GANs) (Goodfellow et al., 2020). The fundamental innovation of TTI systems lies in the integration of two different approaches—a language transformer model that accepts image descriptions as text and an image generator that synthesises the image. The transformer is usually based on CLIP (Contrastive Language–Image Building Simulations with Generative AI 5 Pre-training) models, a neural network that learns visual concepts from natural language supervision (Radford et al., 2021). This is a significant improvement over previous models such as convolution neural networks (CNNs), which excelled at basic classification of objects in an image but could not recognise more salient concepts such as style, context or semantics. The image generator uses a multi-step process that operates in the image latent space, using a UNet neural network and scheduler. The output of this “diffusion” process is an image tensor that is decoded into an image by an autoencoder.

To illustrate the potential of these systems for simulation, we used an open-source version of Stable Diffusion. Figures 11.1 and 11.2 show two sample AI-generated images created using Stable Diffusion. The prompts used were “national geographic photo of an Australian bushfire, landscape, trees” (left) and “national geographic photo of fire-fighters with a hose fighting a large bushfire” (right). As can be seen, the prompts generate quite “realistic” images that would typically be associated with Australian bushfires. Using phrases such as “national geographic photo” pushes the system into producing high-quality, documentary-like images, as would typically be associated with National Geographic (we could have specified “old black and white daguerreotype” or “Banana Fish anime” to completely change the aesthetic style of the image).

This simple example highlights some of the issues with creating prompts: that one needs to be quite specific in the prompt about details such as surface aesthetics, style, context, etc. Such a requirement leads to much of the prompt language containing references to the visual aesthetics of the image: including style, lighting, level of detail, even descriptions of camera lens focal lengths, angle or position of the shot and other various cinematic conventions. The necessity of providing such detail on surface aesthetics, composition, etc. mirrors CLIP’s ability to capture these image qualities as general image features irrespective of the objects depicted in them.

Figure 11.3 shows another example of generative AI simulation of flooding events. To create these images, the following prompts were used: “national geographic photo of people piling sandbags in an Australian town after flooding” (left) and “national geographic photo of an Australian town after mild flooding” (right). In this example, the prompts are interpreted correctly, but only to a point. The way the sandbags are being piled does not really make practical sense (they would be unlikely to mitigate the effects of rising water), the “people” depicted do not have natural poses or body parts, the layout of the town is impossible, and so on. An obvious solution to these issues would be to provide more specific directions.

However, providing too much detail on the objects and their relations in the scene does not necessarily translate into the generated image. Using Fig. 11.1 as an example, if we were to modify the prompt used in the second image to be more specific about the exact number of firefighters, specifics of their individual poses, details of their uniforms or their specific location in the image, the results are unlikely to directly match the prompt. For example, if we specified “five firefighters”, we may get any number between two and ten or more. This is due to the way CLIP converts the input prompt into a latent embedding.

Another potential approach is to use an image-to-image (ITI) method. These are largely similar to TTI systems, and most popular diffusion-based TTI platforms offer an ITI mode. Using this approach, imagery can be adapted using a text prompt to make modifications and transformations while preserving important characteristics of the scene. There are a few potential methods for generating realistic scenes more easily using ITI. For example, simple 3D environments can be created using existing 3D models and then modified using generative AI to render more realistic environments for simulation. This is demonstrated in Fig. 11.3 using Stable Diffusion. A basic 3D environment is transformed into a more complex scene through the use of a text prompt while preserving the overall structure and characteristics of the scene. There are a few potential problems with this approach. For instance, it can be a challenge to control the content that might appear in the generated scene due to the diversity of images used to train the model. In Fig. 11.3, the generated image features a boat in place of a shadow, which appears in the input image. This can be mitigated by adjusting the strength of the transformation. Another approach is to use fine-tuning methods, including the creation of custom low-rank adaptors (Hu et al., 2021) to guide content generation more explicitly with examples and custom embeddings. Despite offering considerably more control, problems with content consistency still cannot be entirely avoided with approaches such as these (Figs. 11.5 and 11.6).

Aside from adjusting and controlling content with prompting, there are potentially other, more direct content control methods that may be more practical. One could use specifically curated models, such as fire and/or water generators. These models could be specifically trained to add photo-real fire and water effects on input images using ITI methods, similar to the approach taken in Memo Akten’s Learning to See: Gloomy Sunday (Akten et al., 2019), as shown in Fig. 11.7. Here, models were created from datasets of open water, fire, flowers and other categories of images, and these were then used for generation.

An A I-generated image and a photo. On the left is a landscape featuring bushes and trees with large fires. On the right is a photo of a firefighter who extinguishes a large bushfire with an extinguisher and a hose. 2 other firefighters stand at a distance.

Using this approach, models need not understand a wide range of different kinds of imagery, have knowledge of context, nor draw on complex text prompts in order to guide generative image models. Instead, input images could be adapted by models with highly limited yet detailed and carefully engineered outputs. These kinds of models are far simpler than large, monolithic text-guided diffusion systems and as a result can be run in real time in high definition on modern hardware. Furthermore, as these models could be specifically tailored to the problem of disaster simulation, they could also contain information on behaviour, for example, through the application of fluid dynamics models within the content generation pipeline.

This approach could be applied to the generation of content for simulations in AR environments. Focused, bespoke generative AI models such as those based on ITI approaches described above could quite easily be deployed in order to transform real-time stereoscopic image feeds from AR headset cameras, with the transformed output rendered directly to the headset in real time. This creates the opportunity for generating disaster simulations in real-world environments, where participants experience photo-real 3D generative AI simulations superimposed on the world as it exists. This requires less computing power as only the transformed elements need rendering. In addition, it could provide the opportunity to conduct disaster readiness training in situ with greater fidelity, allowing firefighters and other emergency services personnel to model scenarios in specific locations where there are known challenges, such as in public buildings and city centres and with potentially vulnerable communities.

Another advantage of specifically engineering models for simulation is that they can be more transparently usable and explainable than currently popular contemporary text-guided diffusion systems. For example, it can be challenging for users of contemporary generative TTI systems to understand precisely which aspects of their prompt may be having the greatest impact.

As illustrated in Table 11.1, human prompt writers often tend to over-equate the complexity, poetics and quality of the prompt with that of the resultant generated image. The table shows a relatively long human-authored prompt (left) and the resultant image generated by Midjourney (middle). We ran this image through a state-of-the-art BLIP (Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation) image captioning model (Li et al., 2022), which can generate a descriptive caption for any image. The captioning model gives an overall description of the image, much like the way a human would, as the model recognises not only the objects in a scene but also the relationships and basic surface aesthetic properties. Table 11.1 (right) shows the results of this “machine eye” perception of the image. As can be observed, this is a far more direct and literal description of the image than that of the original prompt.

2 A I-generated images of flooding events. On the left are 4 people who pile sandbags on a heap of soil in a flooded area. Trees and houses are partially submerged in the background. On the right is a flooded area with partially submerged trees and houses.

A 3 D rendering and a photo. A 3 D rendering of a boat in place of a shadow under the bridge. A photo of an anchored boat in a flooded lane under a bridge.

A photo and an A I-generative image. On the left is a photo of a data cable with an adaptor partially hidden in a fabric held by a person's hands. On the right is an A I-generative image of a slab of rock with crashing waves.

As we noted elsewhere, “one of the key factors that contributes to the capability of TTI models is their access to [very large] datasets used for training and validation. Achieving the visual quality and diversity that they are capable of reproducing requires a [vast] corpus of human-created imagery, which is typically scraped from the internet, in a practice that has been dubbed ‘data laundering’. Scraped datasets—which [may] include copyrighted media—rely on special exemptions for ‘academic use’ to avoid any legal barriers preventing their use, or for copyright owners to claim against (Baio, 2022). For [example], Stability AI (the creators of Stable Diffusion) funded the Machine Vision & Learning research group at the Ludwig Maximilian University of Munich to [undertake] the model training and a small [not-for-profit] organisation, LAION, to create the training dataset of approximately 5.85 billion images, many of which are [copyrighted], and in general appropriated for this purpose without the image [creators’] direct permission” (McCormack et al., 2023).

We further noted that artists have “raised [concerns] about the ethical and moral implications of their work being used in such systems. These concerns include the appropriation of an [individual] artist’s ‘style’, mimicry, and even the replacement of a [specialist] human artist or illustrator. Furthermore, there is [currently] no easy way to be excluded or removed from such datasets” (McCormack et al., 2023), and any mechanisms are generally “opt-out”, meaning that unless you take action to prevent your own data from being excluded, it is considered fair game for scraping. The use of copyrighted material in AI training data is currently being tested legally in several different countries. Governments may need to draft new legislation to deal with these issues, as has already happened in the European Union.

The use of copyrighted images in datasets highlights the question of whether training models on copyrighted data should be considered plagiarism or a form of copyright infringement. As we pointed out elsewhere, “being able to easily generate an image in [a specific] artist’s or [house] style [(e.g. ‘National Geographic’)] without paying for that artist to create it (or paying any royalties or licensing fees), allows users of such technology to bypass the traditional economic, legal and moral frameworks that have supported artists and businesses traditionally. Generating copyright-free images immediately for commercial use without the cost or time involved in securing copyright from a [human] artist may become an attractive proposition, raising the interesting legal [question of who would be the defendant in any copyright infringement case brought about by this scenario” (McCormack et al., 2023).

Beyond ethical questions involving the sources of data and representational bias are the mechanisms by which many large companies try to ensure that generative AIs are “safe”. Many models are augmented with what is known as “Reinforcement Learning from Human Feedback” (RLHF), where outsourced workers in developing economies are paid to “sit for several hours every day watching videos of harmful content and analyzing textual descriptions of hate speech, sexual violence, bestiality, and violence” (Ngila, 2023). This human tagging or classifying of unsafe content is used to train additional AIs that filter results to prevent the underlying generative system from showing harmful content.

Some people have already developed psychological dependencies or been prompted to take real-world action following advice from generative AI systems, with both positive and negative results, including suicide, divorce or self-harm. As models become even more sophisticated, we are likely to see new forms of human–AI relationships with potentially dangerous results. In the context of simulation, there are a number of important considerations for the simulation to be credible. Generative AI suffers from what is euphemistically referred to as “hallucinations”—factually incorrect or erroneous results. The implications for a generative AI “hallucinating” in a simulation context can range from benign to catastrophic, depending on context and situation. For example, the “Australian town” depicted in Fig. 11.4 does not exist, and no real town would be structured in the way it is depicted. Simulations may be speculative, allowing us to ask, “what if…”, but if the answer is based on factual inaccuracies, the value of the simulation may be worthless.

In this chapter, our main focus has been on exploring the use of AI to generate realistic imagery for visual simulation. As discussed, many contemporary systems use text prompts to generate output—e.g. text to image, text to video and text to 3D model—and in these ways are not too distinct from a text-based search. We have also explored how more bespoke generative AI systems can potentially play a significant role in the future. However, an obvious current limitation of this analysis is that the interaction with such generative AI systems is uni-modal.

However, as our simple scenario in Sect. 1 demonstrates, an immersive simulation is a multimodal experience, encompassing multiple senses and ways of interacting. Multimodal interaction has been well studied from a human–computer interaction perspective (see, e.g. McCormack et al., 2018 for an overview). Recently, multimodal generative AI systems have been gaining traction. These systems consider multiple modes of input and output (e.g. text, image, video) allowing cross-modalities to be considered. For example, Google DeepMind recently announced a new AI platform, which they call “Gemini” that allows “reasoning seamlessly across text, images, video, audio, and code” (Google DeepMind, 2023). While still in development, multimodal AI systems have the potential to analyse scenes or environments and to then ask questions that would require expertise (“how safe would this exit be in a fire?”, “where is the safest place to go if this area is under imminent threat of flooding?”). It may be that these multimodal systems are better able to generalise scenarios as a result of constructing representations from a greater number of dimensions, for example, combinations of sound, image and text. The capability and power of such systems is potentially enormous, and more research is needed in order to understand how they might one day be deployed for the purposes of simulation.