Skip to content

AI as a testing tool

Estimated time to read: 10 minutes

There are several ways to use AI as a testing tool.

  • Analytics
  • Predicting behavior;
  • A/B Testing;
  • Game Environment Automated Testing via AI agents;
  • Test Case Generation;
  • Anti-cheat systems;

Analytics

Analytics is the most common way to use AI as a testing tool. You can use AI to track the user behavior and use the data to improve the game. But with that you can only analyze the past.

You might want to track all user interactions, and use AI to analyze the data and give you insights on how to improve the game. This will be the core of many other AI testing tools.

The common ways to track the user interactions are:

  • Send events to a server;
  • Use a third-party service to track the user interactions;
  • Progression funnels;
  • Heatmaps;
  • User paths;
  • Map all deaths / kills / wins / losses;
  • Store the replay of the user interactions;

Predicting behavior

You can train an AI model to predict the behavior of the user to abandon the game, and intervene before it happens.

In order to achieve this, you can track the user interactions and the consequences of those interactions. You can use a supervised learning algorithm to predict the behavior of the user. Once you discover the pattern, you can intervene and try to change the user behavior.

Example: If the player is loosing too much, you can give him a boost to keep him playing. Or automatically change the difficulty of the game. Another good example is when you predict the user is going to abandon the game, you can give him a reward to keep him playing, or allow him to ask for more lives on social media friends.

Forcing the user to take a break

Sometimes you want to avoid the user to get burned out and force him to take a break. This is a common practice in mobile games.

If your game gives rewards for plaing every day, or every session. You can use AI to predict when the user is going to play again and send him a notification to play again.

This can be a bit shady, but, another use case is to force the game to get harder if the user is playing too much. And when it loses, add a timer to unlock the game again. You can even use this moment to show ads, or ask for money to unlock the game again. Can you think in a game like this?

A/B Testing

A/B testing is a way to compare two versions of a configuration setting or a feature to determine which one is better. It relies on remote configuration and the statistical analysis to determine which one is better.

The process is simple:

  • A developer create 2 scenarios to test. Ex.: the color of a button to buy a product (Red or Blue);
  • The system will randomly select one of the scenarios to show to the user;
  • The system will collect data from the user interaction;
  • The system will compare the data from the two scenarios and determine which one is better;
  • The system will select the best scenario to be the default one;

Game Environment Automated Testing via AI agents

This can be really hard to implement, but in summary is to create AIs that can play as humans and test the game. This can be used to test the game balance, the game difficulty, the game mechanics and the game performance.

If you are just trying to test game rules, or economy, you might wanna try to use a genetic algorithm to evolve the best strategy for a given game.

If you are looking for creating a bot to find hardlocks where the player might fall and not recover, or detect bugs, you might try to use a reinforcement learning algorithm.

This field is so vast that is hard to cover in a single section. I will use this in class just to see how it works.

Test Case Generation

You can use AI to generate test cases for your game. There are plenty of LLMs online that can can read your code and generate test cases for you.

Anti-cheat systems

You can detect cheaters using AI. But you will have to be careful to not ban innocent players. You can use AI to detect patterns of cheating and intervene before it happens.

Possible patterns to detect: Speed hacks; Aim bots; Wall hacks, ESP hacks, Macros; Auto-clickers; Memory hacks. and much more.

Shadow banning

It is a common technique to ban cheaters. You can shadow-ban a cheater by making him play with other cheaters only. This way, the cheater will not know he is banned, but he will only play with other cheaters. This can be done using AI to detect the cheaters and put them in the same match.

Serious Sam 3: BFE (Serious Digital Edition):

The protagonist encounters an invincible, extremely fast and screaming scorpion-like enemy, making the game nearly impossible to progress.

Game Dev Tycoon:

In the pirated versions, players find themselves struggling to make a profit as their virtual game studio is plagued by piracy.

Batman: Arkham Asylum:

Batman's cape doesn't work properly, leading to a rather comical and dysfunctional experience.

Mirror's Edge:

Faith is unable to progress past a certain point due to an inability to grab a ledge, hindering the player's ability to complete the level.

Earthbound (Mother 2):

In pirated copies, the game triggers a constant stream of inescapable enemy encounters.

Can you think in other examples?