There will be a few months between now and the next update.
I am starting a PhD+Masters program in statistics in the last week of July.
There are at least five textbooks I want to read between now and then, two of which are directly relevant for the future of this project. The others establish important background I need to know.
Once there and I'm ready to develop the project a bit, there is also a CS department. Plenty of people there will be way better than me at fixing the browser problem, so we can simulate thousands of games per day instead of just 100. I may be able to seek some level of university-sponsorship, and if the project becomes successful we can publish.
Future of the Project
This thread featured versions 1.0 and 1.1, using temporal difference learning.
Basically, a bunch of information on the game's state was compiled every turn, and fed to a Neural Network which then estimates the value of each possible move.
The thing is, I was using around 1000 inputs because there is a HUGE number of variables to consider in pokemon. What is the probability each of your Venasaur's attacks cause a burn? Paralysis? Sleep? Confusion? Leech seed? That is 20 inputs right there.
Well, a lot of them are zero so we don't actually think about them.
The neural network has a lot on its plate: it has to learn all the patterns, the meanings of all the data.
Here is a great article on optimizing keyboard layouts for swype (swipe typing, where you don't lift your finger off the touchscreen keyboard), which also provides a couple of images that are great for visualizing neural networks.
The word "phish" was entered twice, but one of the swipes was given a bit of humanized randomness (we don't swype with perfectly precise straight lines):
Here the words "airtight" and "allright" were swyped. If you look at your standard keyboard, you can see that the swype pattern is extremely similar for both.
But the neural network was able to figure that out, and responded with "not match" (black output circle).
Blue/red lines mean passing a positive/negative signal. Brighter the line, the stronger the signal.
They fed a variety of inputs, such as translations, second derivatives, etc, and from this information that we have no idea how to make much sense of (other than different is different!"), the network was able to pick up patterns and determine precisely how important every input is, and develop internal firing structure to make sure it gets the right answer.
If you've ever used voice recognition on your smart phone, those are handled by neural networks too. Years ago people always joked about how bad voice recognition is, but these days neural networks are actually doing a marvelous job.
Facebook used a neural network for their deepface project, which is able to correctly see if two photos are of the same person 97.25% of the time -- compared to a human's 97.53%; almost exactly the same!
Facebook hasn't implemented that network on their actual website, if you're wondering why they don't seem that good at tagging people in photos.
The thing is, in all these examples, neural networks were fed a bunch of data that seems almost impossible to make sense of (how the heck are you supposed to go from pixel information or a pattern of sound waves to words???), and make sense of it through learning the structure and patterns within.
Learn what connections are real, which connections matter, and how much, when interpreting the data (and in which direction they should go). Learning this structure is an insanely complicated feat; think of it this way: the possibility it came upon that works is 1 out of 3+ possibilities (fire positive, fire negative, don't fire)^(10 nodes in the next layer)^(11 nodes in this layer) = 3 * 10^52 that mostly don't work or are total nonsense, for just the first layer in one case (eg, "match")!!! That is impressive stuff.
Of course, the network works with weights that get multiplied by the inputs to determine what gets passed on (if you've taken linear algebra: dot product of the weight and message vectors; then apply some function such as sigmoid squashing function, and that is the message that gets passed; in the graphs it was message * weight that determined color & intensity). These occur on a continuous scale, so this discrete representation isn't accurate. But the idea to illustrate complexity still stands.
Does Pokemon have this same problem? Was I dealing with a bunch of inputs you have no idea how to interpret?
What could it possibly mean if attack A will do 51.85 - 61% damage, and the opponent has 49% of their health left??? Uh...
Pokemon is a human-created game with structure that is mostly obvious how to interpret. There are lots of real questions that seem fuzzy; eg is it better to switch, use rocks, or leech seed? Especially because you don't know what your opponent is doing.
Why go through the pain of learning the structure of 1000 inputs - most of them at "no" most of the time, and the patterns of it all, when we already know most of the answers as logical fact?!
It isn't some mystery how they interplay, and we can pin down fairly well what precisely our uncertain questions are.
This is why, instead of using a neural network, my intention is for Version 2.0 to use a Bayesian Belief Network. A BBN has a bunch of nodes representing states of belief, that can pass messages to those for which we're relevant. We can thus explicitly represent all the facts we do know, and the structure of how things are related in the ways they really are. There is no sense in investing a great deal of effort into learning the reality we already know for certain.
The nodes can represent concrete observable things, as well as useful concepts (using concepts vs working on just inputs has major computational advantages;
here is a great article using simple non-academic langauge).
This also means that, rather than having code on one level that tries to learn the state of the game and feeds the state to the decision-maker, it is all just one network:
The network will have nodes representing the pokemon and their moves. Observations will update elements of the network itself. Projected damage will also be a node in the network.
Combining what was once two separate things greatly simplifies the whole ordeal, and focuses learning on the things we actually want to learn. The fact that each element corresponds in an obvious way with reality (we know what concept each node and connection refers to), we can also initialize it at realistic guess values, so that we don't have to grown at a mentally-deficient-SirSwitchalot.
Another example: consider
WhiteStag98's comment. We could create and prune connections between nodes as we desired. Meaning we can add a connection between the moves of each opposing pokemon, so that the probabilities get updated from the base rate as soon as know the team, and then further as we learn about moves. How these connections work in reality are going to have to be learned. Also, the more connections, the longer it will take for the network to calculate all of its levels of belief and make decisions -- so there are balances to consider (if there were literally none, we'd just connect everything arbitrarily and call it a day :P).
There is a
lot I have to learn before implementing this. Thankfully there already seems to be several research papers on using BBNs for reinforcement learning, so that will be something I can look at after I finish my textbooks. Although it's likely that there are a lot more textbooks in my future, too.
This is a long way off, but I will provide updates and check here periodically.
