Pokémon battles are turn-based games in which two players compete with teams of 6 Pokémon. The large number of different Pokémon and different game rulesets allows for enormous variability in gameplay and strategy. One popular game format on the battle emulator site Pokémon Showdown (“Showdown”) is the “random battle”, where players compete with teams of six Pokémon randomly selected at battle start. Ideally, these randomly generated teams should be evenly balanced, so that, going into a battle, both players have nearly equal chances of winning, and thus the winner is indeed “random”.
From the Showdown developers’ perspective, there are a number of factors and methods to consider in designing this balanced gameplay. Some primary features to consider are Pokémon's stats (e.g., HP, Attack, Defense) and types (e.g., Fire, Psychic, Ground), and players’ Elo ratings (within Showdown’s system). We aim to investigate whether or not the team generation procedure is balanced; in particular, if Showdown’s team construction algorithm is balanced, then we expect that comparison of said features should not strongly indicate a player's chance of winning.
Showdown stores a large amount of information about each match played on the site, including turn-by-turn records of player moves, allowing for a full, turn-by-turn reconstruction of each match from its publicly available "replay" (or "log") JSON file. We saved about 16,000 Showdown match replays, and then employed a custom-made parser (<class Battle> in tools/battle.py) to extract and organize match information from these JSONs.
Having two teams of 6 Pokémon, each with their own stats and characteristics, gave us many potential features to consider, for example:
While Team Advantage was our most impactful feature, it has known drawbacks, such as failing to consider the specific moves or abilities for each Pokémon. In an attempt to address this we introduced the Stat-Booster Differential, which records the difference in the number of Pokémon on each team that know a stat-boosting move, or have a stat-boosting ability.
For a “baseline” model, it was sensible to use a single-feature predictor based on players’ Elo rating differences, as the Elo system maps this difference to estimated probabilities of each player winning. Beyond the baseline model, we considered Logistic Regression models both with and without intercept terms (the "biased" and "unbiased" models, respectively), the standard Decision Tree and Random Forest models decision-tree based models, and the boosting models HGBoost and XGBoost.
We trained these models on the features which optimized their cross-validation accuracy: for the unbiased logistic regression, those features were Elo differential, total advantage, differential in the number of move boosters, and differential in the number of ability boosters. The remaining models trained on all of those features plus type diversity differential and total stat differential.
Accuracy was chosen over other metrics for its ease of interpretation and because our application does not require special precautions against “false positives and negatives”, such as applications in, e.g., medical diagnosis. Additionally, we wanted to select a model that was well-calibrated : since Player 1 lost in 52% of the matches, we wanted a model which would predict that Player 1 would lose approximately 52% of the time.
On our training data, the baseline- and unbiased Logistic Regression models were 52.1% and 53.1% accurate, and both were well-calibrated. The remaining models had accuracies all close to the former two, but were poorly calibrated, leading us to select the unbiased logistic regression as our final model.