3CG: Collaborative and Creative Content Generation in Game Design 
M Charity


     ZEESHAN: So, the first speaker, I think one of the great things about Papers We Love, this longevity, we have been doing this for how many years, Darren? Conference, 5, but Papers We Love is like 2014, something like that. Many years ago I tried to get Mike Cooke, which is a really cool person working in graphics and video gaming and all this stuff. I asked him about Papers We Love. He loves Papers We Love. He was like, you got to get this person. M. They are awesome. They're gonna love it, they're gonna do this talk, it's gonna be really great. And that's how we have M today speaking for us. M is at NYU, the Game Innovation Lab, doing work with AI, procedural generation and creative coding. I'm super-excited about this talk. One thing to do this, Chas Emerick, one of our organizers, really pushed on us this year to have creative, awesome talks.
     So me, M, that's why we want to start with them, to start a off and kick off this different kind of conference this year. We're really excited about that. Without further ado, I will introduce M now to kick some ass here about gaming and collaborative, creative content. So, thank you, M.
     [ Applause ]
     M: Hello, can everybody hear me? All good in the back? Awesome. Zeeshan, thank you for that amazing intro. Oh, my gosh, I've never had such a good intro. Oof... too loud? Good? Okay. I'll talk kind of quiet and not get too excited. Hi, everybody. My name is M. I'm a Ph.D. student at NYU Tandon. I'm fourth year, my pronouns are they/them. And I'll be talking about 3CG, which is collaborative and creative content generation in game design. So, I had a big enough introduction already. Let's jump right into it.
     For an outline, this will be what I'm gonna talk about. I'm gonna give a little bit of background about what is AI, content generation with AI, mixed-initiative interactions, PCG in games, and some systems that talk about these topics, my Baba is y'all, Wizard Sokoban, Aesthetic Bot, and future with games and content generation in general using AI. And I will have a bunch of references and link -- I will have a bunch of references and links at the bottom. Like right like this. It will be kind of tiny. But at the very end of the talk, I will have a master list of links. If you need to take a screenshot or get your camera out, by all means, go ahead. That's at the end of the presentation.
     What is AI? Big question to get us started off with this talk. For me, when I was a kid, I watched a lot of cartoons and movies that had really cute movies that I really, really liked. We have BMO on the top left, and Goddard from Jimmy Neutron and beta max and BB-8 from Star Wars and Disney. They're cute, helpful robots that aren't the main character. They help the main character with whatever problems that they're going through and just act a nice little buddy for the protagonist.
     But I also love in like sci-fi movies like the AI that may not have been all sweet and cutesy. So, for example, we have Joshua from War Games. Right here. And we also have Hall from 2001: A Space Odyssey. These are AI systems that had good intentions, but honestly just ended up being antagonists to the story. And they were the driving force of conflict for their protagonists. And another one of my favorite robots from the movie Chappie, and Jay and Roy from Blade Runner 2049. And these are put in morally ambiguous situations and have good intentions, but end up wavering between the two moral conflicts. These make for really interesting AI robots that we may not see in our lifetime. But it's nice to consider there are so many different sides and aspects that can come from AI.
     So, what really is AI? Outside of science fiction. Like in our world today? So, as my adviser kind of puts it, AI  is about making computers able to do things which currently only humans can do. And also, just AI  is just "Search." It really is. It's just search. So, it's mostly searching for a solution. So, how to answer a question? Which words to autocorrect in your text? What information do you want to extract from your email in order to put into your calendar if you want to get a flight ticket or put a birthday, that you can get from an email and it's very, very helpful. Searching for the best movie based on movies that you watched or what your friends are watching.
     What you're getting recommendations for on Netflix or your Twitter feed. Whatever. Also, what kind of story to make that matches your prompt. That you gave the AI system. And making sure that it's aligned with like what you intended.
     And so much more! Most of it is just search.
     So, here's some AI in action. All of these like videos and streams were actually taken from my phone last night. That's how relevant and how incorporated AI  is. This first one over here is just autocorrect. Like a GIF of me just typing in autocorrect. But it's nice how the AI  is working in the background to fix your text based on the context and the intentions that you have. The second one is the Google Assistant on my phone. I was in Germany recently for the first time ever and I didn't know a lot of German. I was looking up how you say whatever in German. It's nice that the Google system picked up on that, and gave me a suggestion, hey, how do you say thanks in German? I know now. It's danke, or danke schon. And the second is Google Maps. Maybe I wanted to visit friends in New York. And I wanted to go to my favorite bar, and see how to get there. Google Maps would tell me different paths I could take, how long it will take, how it shows up. It gives nice options without asking for it. It's nice that it can do it. And the last one is the TikToks favorite feeds. And the AI  is using a recommendation system to keep your attention for as long as possible. That's why it took an hour to make it. It was doing what it does best, holding my attention. TikTok in particular, it learns from where your attention is spent and how much you interact with certain videos to give you more videos like that. That's why it's really, really popular social media.
     So, AI with content generation. What is this now? We have AI, previously, they were just kind of finding a solution that was just a one-time fix. Do this for me. I got it. What about AI generating stuff? Machines making stuff? So, this is in -- using the previous definition, searching for new and quality samples. It can be art, text, movies, games, music, so much more. And usually this is based on some kind of rules or grammars or models or whatever kind of systems that the AI might need in order to make this new stuff. And I really like this image over here. This is a AI-generated image itself. It comes from this new model called Midjourney. I gave it the prompt of a robot silting down at a typewriter and painting simultaneously. And after a couple of iterations and about 5 minutes of just going through running the model itself, I got that image. And it looks pretty nice. AI made that for me, thank you, AI. For there's a lot of AI content generation going on today. These came from social media over the summer. This first one, again, similar to the last image with the AI-generated image. This is a text image generation that uses I think stable diffusion. I'm not sure if it's Dali 2 or not. But this Twitter user, Hard Mario, is a big fan of testing out the system and trying prompts. This is McDonald's in Japan. The AI generated it in the Ego period Japan style of art.
     And this is a new movie submitted to the Cannes short film festival, it was AI-generated. It used style transfer to make the movie. It's in the style of a painted, interpretive dance movie about a crow. I'll show a clip of this later. Super-cool. Definitely check it out when you get a chance.
     And also, we haves transformers and writing tools. This is a tool that was developed recently by a startup company that I worked for over the summer. Their intention is to combine AI writing tool with a regular human writer to make a brand new story. In this case, we train an AI like an author and it is like the author. You give it text, and it writes the text. It's really, really nice and writes in the same style as the original author's intent, even though you have a different context going on.
     What are mixed initiative systems coming off that? Mixed initiative systems are content generation that really involves the human in the design process. Previously, we had just humans making art or music or text or whatever. And we have AI making music or art or text or whatever. Now if we combine those two together, it's a collaborative process. And either party can have varying disagrees of the interaction. Maybe the human starts and the AI comes in and edits is, or the AI starts off with the sample and the human edits it. This is a growing interest in AI. There's not many samples currently out yet. Yet.
     But it's basically AI collaborative content generation. This picture is an example of AI content generation. This is PicBreeder, it will create image samples and pick out which ones they like best. And from that, the AI will develop more samples that look like this, a little bit deviating from them, and it's just a back and forth process of like I generate something. You tell me which one looks better. Let's see.
     So, just list off some mixed initiative systems in action. There's game-based, of course. There's LodeEncoder, which is this image over here. This is designing LodeRunner levels along with AI. The AI gives suggestions of levels and you can use the suggestions to paint your own level. Mix and match the levels together to make a brand now, original-looking level. There's a system, Ropossum, cut the rope levels. Really old Android game back in the day. Tanagra, designing platforming game levels. We have EDD, designing dungeon levels. And Pitako, designing games in general also with AI suggestions for sprites. And then there's other systems not really involved with games. As I mentioned before, we have PicBreeder, you can evolve generated images through selection that the AI gives group and then image-to-image, this is this image over here, you can generate samples from line drawings. In this case, whatever you draw in there, the AI will try to turn that image into a cat. That's a nice-looking cat over there. But you can also draw really, really bad and it will make it into a cat, a terrifying-looking cat.
     Even in the samples I showed previously, these are mixed initiative systems. The first with the Edo-period Japan, these use three different human-made sources. McDonald's, and Edo period Japan, and the McDonald's images on Google. And takes examples of Edo-period Japanese art and combines them together to give new samples that have never before been seen or created do get really nice-looking images over here. The crow video is based off a human dancer video and used style transfer on top of it to make a brand new movie which was a little bit more artistic. And much more interesting, honestly. But still just as cool as the original source image. But it's cool that it can transfer that painting style on top of something already made.
     And lastly with the writing Transformer tool. It does need like are a source text in order to get this kind of brain and writing style in order to further progress this. In this case, it used the thought -- [Audio stopped for captioner] --  it can include maps and trains. Just making it really nice and flowy for a player to navigate on and making it distinct every teem you play through. NPCs and encounters, maybe different dialogue and different NPCs in the play through. That's all procedurally generated. And there's many methods you can go about to make this kind of content. There's search-based, tree search, A star, grammar
     Based. In this case, the tree and the spaceship on the right are generated using a grammar structure building off the previous tree parts generated, the previous spaceship structures that were generated. Noise and fractals. Used with train generation to make it really smooth and make it not so choppy in between. And sometimes uses Perlin noise, or a math equation. And machine learning. Sometimes you need a model. This is from a variational autoencoder to make this brand new map. It's trained on previously-made user maps in order to get to this final structure over here.
     So, PCG in games, it's old, but it's become very, very popular very recently. Especially with Indie games in the last 20 years. My favorite game, Nethack, from the 1980s, this is a dungeon crawler game. You have to navigate through the dungeon, go to the bottom and back up. The cool thing is the rooms are randomly generated. You never have the same play through every time. And never encounter the same items or the same characters and enemies. Everything is completely randomly generated. It's super-cool. Played it for 8 years, never beaten it, it's very hard. I have some memorable encounters that I can tell my friends about when I played this game. And I will never experience them again. But it's really nice to look back on.
     And there, this is one of my adviser's favorite games, Civilization 4. I'm not as familiar, but from what he's told me, the encounters himself and the map generations and the civilizations are random encounters. That's PCBG. And another favorite game. Spelunky. This is the rogue-like genre that came up. This is a platforming game, each part of the level itself is generated through like a grammar structure and also just through a little bit of sprinkling of the terrain in there. Again, the encounters and the enemies and the items and the map design are all completely generated every single time a player goes through a new run. And we have Faster than Light. Again, another game where the encounters are randomly generated, the weapons, items, everything is randomly generated. Different play through every single time. This fell into the trend of the rogue-like genre coming in the 2010s. And another one of my favorite games, Hades. The focus is on the set narrative, it depends on the repetitive play throughs of the player in order to progress the story. It really kind of takes advantage of, all right, the player is going to keep dying. That's the point. Every time they try to make their way back to the overworld, the encounters and the ability upgrades they get and the items they can pick up are completely different every single time. Very, very fun. We have a lot of examples of PCG in games. This is PCG in action. And you can see, this is an RL agent that is exploring a Nethack play through.
     And, of course, the map themselves are completely randomly generated. The items that they pick -- sorry -- the items that they pick up are randomly generated and the enemies they encounter are randomly generated. It's hard to play and hard to teach an AI agent. That's from the Nethack challenge hosted by Meta every year. If you're interested, definitely try to apply. And you have Hades over here. And the boons from Aphrodite. And they have 10 or 12, they have different effects.
     some are good, some are epic, some okay. Every time you encounter her, different set. And it's an interesting strategy on the player's part.
     And lastly, this is a diagram of how the Spelunky diagrams are generated. Starts off with a 4x4 grid, and make sure there's a start and end and it can be reached by navigating through the grid. And fills in the other sections with random stuff, treasure, maybe some shops. But it's not necessary to pass through in order to get to the bottom of the level. And inside these cells, you have maybe a template level structure that allows -- that guarantees that a player can move from point A to point B. Fill it in with enemies or extra terrain jumping out of the side just to make it a little bit more interesting and not look like it was made from a template. And then yeah. Add some more enemies and make sure it's playable. And using this on all 16 of these cells. You get really interesting level design. Really interesting just play through.
     So, what can it do to games? It can reduce the design time, it's automatically generated. Unlike other games where it's made by level or character designers, whatever. If you have the AI taking care of it, you don't have to worry about it. It's nice. And you can create unique experiences for each player on every play through. You will never have the same experience every time, which is awesome and addicting. It adds diversity to the assets. Instead of one set item every time, maybe you have a bunch of variations on the same item, which makes it interesting in terms of game play and aesthetics for the game. It can allow adaptation to the player outside of the original design constraints. Most of the time you have a set player in mind, what about the players that play it in a different way? PCG it take care of this and design different levels for different plays. Makes it interesting. And help formalize the design process. If you're relying on AI, you can understand what systems are at play and how a players interact with it. Making it as title as possible. And what goes into designing a level or an NPC encounter. That kind of stuff.
     But, of course, PCG has limitations. It's not a one size fix all, or fix everything for game design. There's a game, No Man's Sky that came out. It was a space I space exploration. It was cool, thousands of planets and didn't encounters. Like a normal PCG game might go. But the problem is, most of the planets kind of look the same. That's been a problem with PCG. It's called the 10,000 bowls of oatmeal problem. Which is when you have, yeah, you got a lot of stuff generated. But most of the time, it just looks the same with very little variation and doesn't make a difference. There's also the problem with like broken generation. This is a screenshot from Minecraft. You have this nice, procedurally-generated mountain hill area. It looks very nice. Has a river and a hill and everything. But there's a weird desert structure in the middle. That's a temple only found in the desert region of Minecraft. It's hard if a player built a temple right there or it naturally generated from the seed. Either way, it's not supposed to be there. It's very plausible that the terrain generator just put that there on accident just because of a broken generation.
     And also, there's the problem of aesthetics. On the left-right here, we have the human-made Mario level from 1985. And on the right, we have a procedurally-generated Mario level. Both levels are playable, but arguably, this one just looks a lot better. This one definitely looks like an AI probably made it. Maybe a human made it. I don't know. But like it -- this one still looks a lot better.
     So, how do we avoid -- just to start off with these -- to address these problems, how do we avoid the 10,000 bowls of oatmeal? This was a term coined by Kate Compton, another researcher, PCG generation, particularly for AI simples. On the GIF over here, you can see this is some samples of plants being generated and it looks really pretty. It starts off with a lot of diversity at first. But as they evolve and as players choose certain ones they like, the involver comes to a stasis mode, with very little variation. Different heights, one or two tree branches extra, but they look the same. How it get rid of that problem?
     To start off, this is a game we paced the system on, Baba is You. It's a deterministic, small-scale, dynamic and simple Sokoban-like puzzle game where the player has to move to the goals. And it pushes grids around, the solve puzzle thing. But the trick is, the boxes themselves don't need to go into specific placements. Baba got stuck behind a wall, couldn't get the flag needed to win. It needed to break the rule, wall is stop, in order to break the wall that's stopping it. It makes really interesting rule combinations and interesting puzzles from the rule management of the game. This was from the jam version of the game which was made in like 2 weeks or so. And it was released as a full steam game by Hempuli 2 years later. And it's way more tools and it's a fun puzzle game. It's a fun game for AI to generate the levels or even try to solve the game.
     That's what we did. We made a mixed initiative collaborative level editor for the game Baba is You. It's like Super Mario 2, where they can build it, but in this case, Baba is You. And like they can edit the levels. There's a central goal, how to make more levels of the rule combination to expand the diversity of what kind of levels are in the system itself. It includes a way you can play the game from the main website. Test it out through an AI solver or you can play it yourself. And over here, users can just like edit their levels within the system and test it out later. So, to get AI involved with this mixed initiative process, we have to break down the levels into a much simpler way that it can understand. So, each map is represented as a 2D array, as an ASCII map representation, the words themselves are capital letters, and the sprites are lower case levels. You can see the direct from ASCII to GUI. You can replace the 2D tiles within the array, replace the sprites itself. And the rules that are available at the start and the end of the level. At the start, the initial rules are placed, what rules are in play. And at the end, what rules were needed in order to get to the solution of the levels. Might be very different. Might need to break or make a rule, make the same rules and rearrange them. Doesn't matter. But it makes it easy for the AI to interpret it when the game is broken down into these structures themselves.
     , so the AI interaction goes like this. It wants to make a new level that involves towards a certain fitness that has these criteria. It tries to minimize the words that are not used. If it's not part of the potential solution or part of an active rule, tries to get rid of it so it doesn't clutter the map. It tries to reduce the excess of sprite objects. Tries to get rid of sprites that look gross and don't add to the final solution. Make sure there's not much empty space. Reduce the ratio of size of the map to the empty space. You want a little bit of empty spice to look a little bit nicer. You want to have the same patterns as another level submitted. Use user-submitted levels or game-based levels, and place them inside the map so it kind of looks user-made. But it also shortens the process of trying to individually replace each tile in order to get a brand new role. And you have to have a WIN and a YOU condition to make it playable. If those aren't present, it's not possible to play or win the map. And this ends you were in a fitness value that the AI can understand, it's the variables that it keeps in mind. And in order it get diversity, it tries to rank the population of levels by novelty of rule combinations. If it finds a map that has a different rule combination that previously wasn't in the database, it's like, ooo... I like that a lot better than what was already made. The generator might get maps like this. This one looks awful because it's got way too much sprites going on. And this one, too much white space, this is better, but not anything new. There's no rules in there, oh, X is stop or Y is kill. It's a basic get to the flag, you win. This one is better. But it has, again, some extraneous sprites everywhere. It's gonna rank that a little bit lower. This one's probably the best. It has new rules over here that's different from the -- just the starting rules. And it's got nice structure. Every sprite is used. So, it's probably gonna use that as like the best map to show the user.
     And, you have this collaborative loop. So, the human will probably try to design levels itself. Or the AI might give like a starter level like in the previous map. And from there, the human can select either to mutate the level so that it uses that previous fitness function to make a better level, or the human itself can manually edit what the AI had already and together get a new level. It's nice, when you submit to the website, you have a database of levels with different authorship. In the first column, we have levels that are only made by the users. So, no AI interaction whatsoever. Purely edited by the human itself. And then we also have levels that are PCG-only. So, only-edited by the AI, no human interaction other than pushing a button. That doesn't really count. It's got that procedurally-generated touch. And we have the last column, which is a mix of both human and AI working together to make a brand new level. A nice touch of AI creativity and human creativity mixed into the loop.
     So, of course, this system has some drawbacks, of course. It's not perfect. So, as you can see over here, most people still prefer to just make levels themselves. Not really wanting to let the AI take over. Just like 20%. It's not bad. Could be better. And like an even smaller minority of levels that are just pure PCG. So, humans still like being in control. Which is fine. Part of the reason is because the AI  is possibly over-writing the intentional creativity designs of whatever they were designing originally with the level. So, it can get really frustrating where you have a nice level and you try to let the AI take over to potentially make it better and it makes it worse. There's nothing really you can do about that. And part of the reason that it's probably that bad is because there's lack of transparency from the AI decisions. It's going and generating something without telling you why it's doing something or really understanding why this is a bad decision or a bad edit in the first place.
     Also, just the AI might not be as creative to make these kinds of levels. Human levels look like human levels and they look a lot better in general, they have more intention behind them. But the AI, it saw the ASCII, it doesn't understand what goes where, why it should go where. Lack of creative design. Yeah. Some levels look bad from the AI design. Kind of hard to teach that. But I'll get into that in a different system later. Also, just not much encouragement for diversity for the levels. The humans have some kind of intended creative design going through making a level. Not really caring about what's in the database itself and trying to make these brand new levels that have maybe different rule combinations. And have also, it's just designer miscommunication. That's something we can work on with AI-human interaction. So, how do we avoid the broken generations now?
     We have that Minecraft problem, we have a temple in a mountain, which is a problem. But there's also, people used to take out the ladders in their Sims game. If you were the Sim, that would suck. And if the AI did that to you, you would be very, very upset. How do we prevent the AI from taking out the ladder even though we do it to them? So, Wizard Sokoban, going off the framework, we have these solvable levels, perfectly-designed, nothing wrong with it. Easy. Can do. But then maybe we get a level where it's just completely unsolvable and we can't do anything about it. The player is boxed in, can't get to the goals to push them around to get them where they need to go. What about the levels that kind of look fine, but maybe aren't solvable? What do you do about that? How do you know that the generation is broken?
     For this, the AI interaction works like this. We ask the PCG system to create a map with these specific rules. Has to have walls on edges of the map, equal number of goals and crates, otherwise you can't win. Exactly one player, maybe design more than one, but that's harder. Keep it at one. Each axis must stay within a range of 6 to 10, keeping it on the screen so it doesn't get crazy with a thousand by a thousand maps. And place more wall tile to add challenge. And we get an ASCII-based translation that the AI translates into a renderable picture that the human can understand. But what do you do if there are dead-ends or future dead-ends? Or if you have the player boxed in? And what do you do, all of these are fine, these are fulfilled, but what if the level is completely unsolvable? How do you address that?
     We can have the competitive loop. A generated map. An A star agent that tries to solve the level before the player looks at it. Maybe we have the agent try to solve it and iterate through 10,000 times and search for a solution. And then what? Maybe can't find the solution. And we have a broken map that we gave the player. And they will get pissed off. Or will they? What if we made it a game. Say the AI can't solve the problem and challenge the player to solve it. And we don't know whether or not it is solvable. That's what Wizard Sokoban is. It's finding the -- the process of the procedurally generated level and trying to challenge the solving capabilities of an artificial agent. We have a collaborative loop now.  The human can actually ask the -- or try to show the AI, like, no, this is actually a solvable level. You just need to try harder or something like that. You have all these potentially -- or theoretically unsolvable levels that the AI couldn't solve.
     That last one, if you give it to a human, yeah, they could solve it. It's really nice. We can have a system where a human can help an AI figure something out. Either by imitating its solvable capabilities or giving more iterations and chances to solve it. So, of course, the system also has drawbacks. It's not perfect.
     The AI just may not be smart enough to solve a level. Even if we teach it how to solve like a human, it might not be able to do it in the first place. And also, repeated evaluation can get tiring. This is a game. A player is willingly wanting to solve the levels, but in a beta testing or a game where they test PCG levels or systems, they're going to get tired. No one wants to test a level a thousand times. We make AI do it all the time, they don't care. Supposedly. But this is user with fatigue. It's a drawback with the human-AI interactions that we're also trying to solve. Also, just waiting for an AI to solve these kind of levels can take a really long time. So, the problem with this game, even though we can give it X many iterations and keep adding, it's going to take a while and definitely breaks that game loop of a user waiting for the AI to figure out, is this solvable or not? I'm not sure, give me more time. Try to solve it. Even if it might not be enough steps, we don't know. How many is enough steps? Could be completely you been solvable. Why waste time adding more steps and iterations?
     Also, how to apply to harder problems, it's a Sokoban, it's easy for AI to figure out if it's solvable or not. How about something like a 3D open world game. Where it's multiple players going on. Possibly multiple PCG systems at work. It might not be able to apply to that. So, lastly, how do we avoid the artificial-looking generation? Going back to the Mario example. It was like, yeah, it's playable. Nothing really wrong with it, technically. But it just does not look good. Nobody wants to play something -- people might wanted to play it, I don't know. But they probably would rather play in level as well. And the previous systems have this, this is also a playable level, but it looks like crap. Too much sprites going on and too chaotic looking. And Wizard Sokoban, too boring. Not enough going on. One goal, one goal -- one goal, one crate. And two blocks out of the way. Doesn't have an effect on the end goal puzzle.
     How do we get better-looking levels? So, we have an AI system. We have it generating these kind of maps. Like grid world maps. And maybe we have some kind of sets of tiles. These tiles in particular have no effect on the game play. It just looks nice. Maybe they're like decorational tiles like flowers or trees or dirt. Whatever else you need in a tile-based map game. So, how do we figure out what maps look good? And also, how do we get human feedback that's consistent enough and quickly enough that they won't get tired from just repeatedly evaluating the same maps over and over again? What's a system or a interface where people are on it constantly and also seeing a lot of content all at once?
     What about Twitter? A lot of people use Twitter. I use Twitter. I used it this morning. I went through god knows how many posts. And saw so many things. It would be cool if you could use that with AI and with AI-generated maps to get feedback very, very quickly. Now we have an AI that can learn what looks good from a lot of people over a period of time. That's what Aesthetic Bot is. It's seamless based on aesthetic design rather than just the playability.
     The AI interaction for Aesthetic Bot is this. 16 tiles, very easy to work with. And you have a nice-looking Pokemon-style map over there. The interesting thing is, we have multiple tile sets. Apply them to the among us tile set, it looks like crap. You don't want that. In this case, we have to teach five different bots five different aesthetic styles. Not every one is the same. We have a random map and maybe set tile patterns that the AI can use. And just quickly replaces parts of the map one at a time using these tile patterns. And you get a bunch of different evaluations of maps that are only aesthetically-based. You don't have to worry about whether it's playable or not. You don't have the previous problem with Sokoban of testing it out. It just looks like whatever it looks like when it comes out. You have a nice, constant feedback loop, this is technically the bot. Gets in the map, breaks it down into a value of I think it's going to be good or bad. Uses that on the entire population and it tries to evolve and mutate a set of maps and just goes over and over again until it reach ascertain point where it takes the best map from the population. Uses that, puts it on Twitter, gets feedback on whether or not it's a good-looking map or not and just goes back over again. It's that simple.
     There's this competitive loop with the system itself now. You have actually user-made labs. Which comes from a level editor, a map editor, I should say. And the users can make whatever map they want. Use the tileset, make it whatever size. And the same is the Baba is You all system. And we have the AI-made maps, we have a set, picks the one that it thinks looks the best. And pairs them up together and puts it as an image on Twitter. Asks which one do you like better? Not which is user made, not which is a game, which do you like better? It posts a poll, A or B. And the nice thing about Twitter, it doesn't show you what other people are voting on. It's based on your own preferences. There's no right or wrong answer. It's just A or B. And at the end, you can see which one most people think. And at the same time, you probably can't see it here, 25 votes from this map here.
     You have a bunch of people comparing which one do you like better? No rhyme or reason. People like A better than B. Don't know why, but that's what the people voted for. You have a cool collaborative loop that the AI learns from the constant evaluation and constant feedback of like users put -- inputting their own maps that they designed. So, over here is some examples of such. So, this is a user-designed map. It has a nice, interesting-looking building right here and then in future generations with the AI  is like, this did really well in the polling. Let me put something similar and see if I get good votes. And same with other maps, repeated Among Us characters. May not look as well, but at least it tried. And the dungeon designs, you have an interesting maze-looking map. Traps over here, treasure chests put in corners and it tries to emulate that and do the same in its maps. It's interesting it learns aesthetic patterns from the highly-rated user levels. Without learning why does it look good. People liked it from the poll, let me try it myself. Like an artist.
     It has drawbacks, I need to mention it. Originally when we brought out this. Intending to think, users make game-looking maps. This looks like it's from a video game, Pokemon and Zelda and Pac-Man. That looks nice. As the system got more traction, more people started following it and putting maps, we saw these interesting-looking levels that also look really good, but are also really complex. We have these multi-layer houses and interlocking trees like a forest. Then we have this like Celtic knot of Pac-Man levels. Stuff we didn't think people could do with the sprite set. But it's awesome. And with artistic, like a duck. Or somebody saying hi with the holes in Zelda. Or a face, nothing to do with Pac-Man at all. But they all look nice. And people are most likely going to vote for them. That's what we learned and also forgot when designing this, people have different types of aesthetic. And, of course, the AI learns them all at once and kind of try to apply them all at once.
     And at the same time, it can learn some bad habits. I showed some really nice-looking levels. But there also exists very bad user levels, which are very chaotic looking. I would say they look bad. But maybe some people would find them appealing. But a very small minority of people. At a time, AI can learn these bad habits. It takes the patterns from the user set. It also applies those bad patterns into its own dataset. Also, it just has a really hard time competing with exceptionally good level design. These levels, I can't make those. Those were really, really good. In fact, made by actual game designers. There's a Twitter user that's really interacting where -- [Audio stopped] -- and it doesn't really stay consistent. Again, there's different kinds of aesthetics. People have different art styles or maybe there's like different level design styles that come and go.
     So, the AI just kind of goes with whatever is most happening more recently and it's constantly changing. This can be a good or a bad thing. Honestly. It doesn't learn a set aesthetic. It learns what's more popular at the moment. And also, how do we apply this to actual content generation? This was the intent of visual design. Didn't have any playability whatsoever. How to take it to the next, the game is there, but don't overwrite important key game elements? Aesthetic is a hard problem to define and to solve. It's interesting, because with this project, we were able to get a system that allowed people to interact with it very consistently and quickly without tiring anything out. And then the bot also actually did learn some interesting things.
     So, the future of PCG systems in games. This is probably the last slide, honestly. So, I have a wishlist for what I want more games to use. Not just like Indie games, but maybe A AA games in the future. Recommendation systems. I don't see a lot of that. I see that in AI, streaming services. I don't see that a lot in games. It would be nice to have a system, have you thought about using this weapon? I saw you like using me lee weapons instead of range weapons. That would be nice to have in a game, learning from your preferences. At the same time, having it adapt to your difficulty level. It notices you're having a hard time, maybe gives a brand new level to try that's a lot easier. At the same time if you're going through levels super-quickly, no challenge whatsoever, having the game adapt. Here's a harder level. Brand new, never-before-seen, give it a shot. See how you do.
     Companion agents. I love them. But sometimes in video games, they can be bad, Navi is a infamous companion agent that's intrusive, not helpful. But it would be nice to have a good companion agent that you want around while playing. Playing the game or designing the levels, something that's flies and helpful. And it would be flies to have levels that kind of changed the generation patterns or changed their own structures based on the feedback from the players or community of players and happens over time and not just immediately upon release. A set of levels, players like this level style. Generate new levels that follow in the style and see what happens. What else experimental I can -- it can develop over time?
     And then lastly, just some PCG that focuses on aesthetics. So, a lot of PCG in games is all right, how do I make a level that is functional and playable without worrying about what it looks like? It's hard. As we saw with the last system, it's very hard to define aesthetics and figure out what is a good-looking sample that the users will actually like.
     And again, content generation is just not for games. These experiments can be applied to text, art, movies, music. Anything that involves machines or involves humans creating things, like we said before. And, of course, there's new systems that come out that push the boundaries of AI. As we saw in the very, very beginning with the three systems, those came out in a couple months. And who knows what's going to come out in a few months, a week after this event happens. We should be very aware of the ethical concerns even with the previous systems, there was a lot of pushback. People got upset with the film. This isn't a human-made film. This is kind of against going artists, this isn't -- AI  is not an artist. Even with the text image generation. It would have biases towards certain demographics or only producing this kind of demographic. We need to be more aware of what's happening with AI. It's becoming very, very real. Maybe don't give it nuclear warheads or military weapons. But even on a minor scale, how is this going to affect the people they interact with?
     So end up, short quotes from two of my favorite researchers. We can only see a short distance ahead, but we can see plenty there that needs to be done. That's from Alan Turing, computing machinery and intelligence. Considered the father of AI. One of the biggest papers, he was talking about the future of AI and how AI  is gonna interact with the world. And then with Margaret Boden, computers can come up with new ideas, and help people to do so, both their failures and their successes help us to think more clearly about our own creative powers. In order for humans and AI to progress, we need to learn from each other in order to get further.
     So, as promised, here is the giant link list to any background readings you might want to look at or research papers I mentioned in this talk. It's all there, I'll give it a second. And by all means, if you have any questions, talk to me, or come up afterwards and ask questions. Some of the newer systems, I'm getting familiar with myself with these systems and I'm pretty sure everybody else in the field is testing it out. I might not have as much information about the diffusion or the style transfer. But I would love to hear your thoughts and talk about it.
     And finally, I'm just gonna plug myself and my other awesome lab mates. That's my email, find me basically anywhere on the Internet,@MasterMilkX. That's my icon, I'm on Twitter, Itch.io, GitHub, Discord. Check out my adviser, Julian Togelius, and my lab mates. That's my lab website and my adviser's website.
     Contact them, read the papers. It's awesome. And then if you really liked this talk, please be sure to follow others in the PCG field. There's the PCG at FDG every year. This workshop is focused on PCG generation and especially in games. So, it's most of the works in there are just like only PCG. It's really awesome. Really get some groundbreaking work with games and AI generation. And then also IEEE conference and games, AIIDE, happening in California next month. And GECCO, not really about games per se, but focused on evolutionary content generation, evolutionary algorithms in general. And a lot of the content I talked about use evolutionary algorithms it in and the LAIKA Team, about the writing and the transformers, that's in beta. They're an awesome team. And you can use a writing tool if you need it and train another author to write whatever story that you're using. Yeah. Thank you for -- thanks for listening.
     [ Applause ]
     >> Thank you, M. Thank you so much. We're going to take the break for 10 minutes. You can find M here to ask questions. And then we will have Deirdre Connolly up in 10 minutes.