Microsoft says it has developed a new generative AI model that can produce “complex gameplay sequences.”
The Xbox maker’s Research Game Intelligence team debuted the World and Human Action Model (WHAM), which it has nicknamed Muse, earlier today and described it as a “generative AI model of a video game that can generate game visuals, controller actions, or both.”
Okay. At this stage, we need to tap the ‘generative AI isn’t actually building video games’ sign.
Microsoft stated that Muse, which was trained on data provided by Hellblade developer Ninja Theory, is currently capable of generating visuals at a resolution of 300 x 180 pixels. It also required 1 million training updates before producing a stable output featuring correct model behavior and mechanics.
Microsoft explained current instances of Muse have been trained on more than 1 billion images and controller actions, corresponding to over seven years of continuous human gameplay. It said that data was “ethically” sourced from Bleeding Edge, the four-versus-four multiplayer title developed by Ninja Theory.
The company had previously used player data from the title to drive human-like navigation experiments, but in 2022 apparently started wondering what might happen if that information was funneled into a machine learning model foie gras style. Thus, Muse was born.
“As the team got to work, some of the key challenges included scaling up the model training. We initially used a V100 cluster, where we were able to prove out how to scale up to training on up to 100 GPUs; that eventually paved the way to training at scale on H100s,” said Microsoft, explaining how it birthed the technology.
“Key design decisions we made early focused on how to best leverage insights from the large language model (LLM) community and included choices such as how to effectively represent controller actions and especially images.”
Microsoft heralds generative AI ‘milestone’ but Xbox studio head claims tech won’t be used for content creation
Microsoft described the development and unveiling of Muse as a “milestone” that perhaps demonstrates how generative AI models can “effectively support human creatives.”
For instance, in one demo Muse was prompted with a modified visual featuring a completely new character. It then generated a gameplay sequence that showed how that character might be “adapted” into the experience.
Discussing the advent of Muse in a video blog, Xbox boss Phil Spencer said developing the model was about leveraging “the intersection of art of science” to enable creators to “do things they’ve never done before.” It remains to be seen what those nebulous “things” might be.
Spencer also pondered how AI models could be used to bolster game preservation by imagining a scenario that would arguably concern some preservationists.
“You can imagine a world where, from gameplay data and video, a model could learn old games and really make them portable to any platform where these models can run,” he said. “These models and their ability to learn how a game plays—without the necessity of the original engine running on the original hardware—I think opens up a ton of opportunity.”
Ninja Theory studio head Dom Matthews explained he specifically wants to learn how AI can help teams “push their creative ambitions.”
“Technology like this for me, and for our studio here at Ninja Theory, is not about using AI to generate content, but is actually about creating workflows and approaches that allow our team of 100 creative experts to do more, to go further, to iterate quicker, to ideate quicker, to bring the ideas in their head to life in a tangible form,” he added.
“Although this is technology we haven’t used in the creation of our games—and we don’t intend to use this technology for the creation of content—I think the interesting aspect for us is how can we use technology like this to make the process of making games quicker and easier for our talented team so they can focus on the thing that is really special about games, which is that human creativity.”
Of course, there is substantially less human creativity on display at Microsoft these days after the studio laid off around 2,550 game developers in 2024.
Those interested can learn more about Muse (and view example outputs) on the Microsoft Research website.