Automated Resonance

This post builds on “What makes a great Space Center flight”. Here, we will explore the limits of automation in achieving aesthetic resonance and catharsis in a crew.

What is it that makes a great home cooked meal? The people, the familiar faces and flavors, the cutlery that only grandma seems to have, all of these little things add up to the indelible fingerprint of how thanksgiving dinner tastes with family versus how it might taste in a restaurant. Of course, if your family goes to the same restaurant every year for the holidays, you may find that meal has the same hallmarks of nostalgia. For myself, Thanksgiving reminds me of my grandmother’s apple pie, and Christmas reminds me of catered dinners at the same restaurant every year. Both holidays carry similar resonance. If you sit me down, blindfolded, in front of either meal, a different set of memories will come flooding back.

This is true, even if the more mass-produced restaurant experience lacks the fine craftsmanship of grandma’s pie crust.

The goal of this essay is to explore how to achieve narrative excellence in spite of automation.

What can we automate?

The gist is that anything in the software can be automated. For a bridge simulator, because a flight director and a crew are entirely disintermediated by technology (lighting, sound, and of course station controls), there are opportunities to automate virtually all of the experience. The only piece I can think of that’s lacking is Flight Director’s creative cognition, where they are taking everything in and making decisions about what the next bit of storytelling ought to be.

If a computer can emulate that bit of cognition, then I suspect it can automate the entire experience - catharsis-inducing experiences included.

Building a computer at that level is beyond the scope of these essays. Maybe generative AI could slot into that seat at some point. For now, I think it’s more appropriate to dial back our ambitions a bit by introducing some constraints: Even if we had such an artificial intelligence, it would still be constrained by the ability to voice act its way through a flight. Doing so convincingly may be the true final frontier of automated flight directing, and we’re definitely not there yet.

So, this leaves us with a few things that we can automate:

1. The obvious stuff: We can shake lights, play sounds, add and remove entities and components from the physics simulation.
2. Non-linear narrative structure: We can employ some sort of non-linear narrative structure to allow choices to branch and gather. This contrasts with historical timelines which are essentially linear, but has strong prior art in things like interactive fiction and choose-your-own-adventure books. Most notably, the Ink scripting language, Inform7, and Twine.
   • It’s no small secret that I’m a fan of Ink, however I don’t think it is well-suited to a bridge simulator since it’s more of a markup language that can be extended for scripting. Bridge simulators are generally light on markup and heavy on commands. Even so, I believe Ink could pull off a bridge simulator and it still could be desirable for its authorability (more on that later).
   • Ink is not designed for extending or remixing narrative scenarios, e.g. having a universal collection of damage control steps that we can mix with some bespoke damage steps that only apply to a specific mission. This is a big strike against it, since we want something that can be fractally extended and remixed over time.
3. State-based reactions: We can watch for changes in game state and react in specific ways. If conditions change such that we shouldn’t react, we can cancel the watch.
4. Flight Director Prompts: We can present options to a flight director as part of a dialog sequence, allowing for full voice acting while remaining on the rails of a given scenario.
5. Manual Intervention: We can anti-automate as a sort of escape hatch, if the automation isn’t working as expected. This allows for manual intervention and control when necessary. If we record such escapes, we can use them to improve story elements to avoid such manual control in the future.
   • Having a path to learn from such “mistakes” in fully automated flight directing is a key insight that I think any bridge sim trying to achieve automated flight directing must consider.

Nonlinearity in storytelling

Books and movies are profoundly linear experiences. Although they may sequence events in a non-linear fashion, you read or watch them from start to finish.

Sporting events are also quite linear, although the future outcome is unclear. The rules of any sport generally evolve to produce entertaining outcomes.1

Video games are interesting, since they are inherently nonlinear, but historically have not benefitted from supporting profoundly wide branches of narrative space. Inkle studios work directly in this space, trying to craft “storytelling machines” (what they informally call their games) that feel like you can try anything, despite having finite options. See below for some of the tricks they use to achieve this.

Table top RPGs (TTRPGs) like Dungeons & Dragons are inherently nonlinear, since they are designed to be played with a group of players, each with their own goals and motivations. This allows for a wide range of outcomes and experiences, as players can choose to pursue different paths and make different decisions. Games are reined in by rulebooks and game masters, who are responsible for maintaining the balance of the game and ensuring that it remains entertaining. A GM is functionally the same as a flight director in that way. As such, TTRPGs may provide significant prior art when designing semi-automated and “gm-less” bridge simulator games.

How D&D achieves nonlinear storytelling

D&D benefits from having a way to translate player input into structured attempts at action (that way generally being a human being who understands the rules and the language spoken around the game table). Such structured attempts can be evaluated against the rules of the game, with dice bringing in a bit of entropy. The narrative meaning of a proposed action and its eventual outcome are tracked only in the heads of the players, with some degree of state maintained in changes to character state, such as health, mana, and experience points. One could say that the “on paper” game state could be modeled and maintained automatically, whereas the narrative game state is more fluid and maintained in the heads of the players and the game master.

Bridge simulators have a strong advantage here: any proposed action is going to happen through interacting with either the software controls or NPCs voiced by a flight director. For software interactions, those can be observed and reacted to by the software itself. NPC interactions may still involve some amount of improvising, but could be constrained to options presented to the voice actor so that they stay “on script”.

For D&D, the more important part is the generative next steps of the narrative experience. It’s one thing to have a group of people around a table with character sheets, taking turns saying that they are going to go into a dungeon. It’s another thing to have someone then tell you what happens next when you get in the dungeon, or knowing what happens when you go left or right. A game master may have a map of the dungeon prepared in advance, with various encounters and an overarching structure or story to tell along the way.

As an example, “Overboard!” is a murder mystery game with simulated NPCs written using Ink. The game tracks what each NPC knows, and based on that knowledge the game can end up in all sorts of states. “Expelled!” builds on the concept, with NPCs gossipping and sharing knowledge with each other, creating a complex web of interactions that leads to unexpected outcomes. As Jon Ingold puts it in a GDC talk, narrative games have a spectrum of freedom that ranges from, “What am I supposed to do now?” to “What can I do now?” to “Yikes! What have I done?”

Another unique contribution of Ink-driven stories is the LIST data structure, which is a composite data structure of an enum, an array, a set, and perhaps a dictionary. LISTs are worth studying because they allow for inferring an explosion of narrative state combinations from a handful of variables - very valuable when trying to tell a complex story. This GDC talk from Jon Ingold explores how LIST can bring order to a non-linear “go kill the wolf so you can get a chicken” quest that has many potential outcomes, ranging from “I’m the town hero and they gave me a chicken as a reward” to “Oh no, the wolf killed the peasant, at least I can now take his chicken since he’s not here anymore”.

How Inkle achieves nonlinear storytelling

First off, they wrote their own scripting language to make it easier to branch and gather dialogue trees. Whereas most dialogue systems function in somewhat linear ways, Ink is designed for arbitrary flow through a graph of options. Content is gated behind preconditions that must be met before a chunk of content or a choice becomes visible.

By designing their games in this way, Ink’s lead writer Jon Ingold claims he can contribute little nuggets of story here and there without much regard for how it weaves into the overall corpus of content that already exists. In a word, the narrative design process becomes fractal when you can incrementally improve any part of the system without breaking other bits. More on the technical details can be found in another essay, “All you need is a teleprompter” and “Ink for show control”.

Achieving resonance in non-linear storytelling

Bringing it all together, resonance is going to come through proper sequencing and timing of events. With the right tools at our disposal, we can create bits of content that know how long it’s been since an emotionally charged moment, perhaps suggesting that now could be the right time for a good joke. We might know if we’re in a mode of trying to repair the ship and gather our bearings, if we need to wait for certain crew tasks to be completed before continuing the main story, or if we need to bring the tension on as we race against the clock to save the day. In each of these moments, we need to identify the following:

1. When should we introduce another unit of content?
2. What type of content should we be looking for?

If we can answer those two questions reliably, with respect to the current state of the mechanical simulation and the narrative simulation, and if we have enough content to work with, I believe it is possible to automate resonant storytelling that is compelling, unique, and that brings surprises along the way.

I haven’t written much here about surprise, but one of the best bits of resonance are good surprises - things that subvert our expectations and lead us to think differently about the world. A good litmus test for a narrative engine is whether or not it can deliver a good surprise at the right time.