Game testing, as opposed to other forms of testing, has been highly dependent on manual black box testing. Even though there have been large advances in automation or AI related technology, gaming has remained manual since building an automation system will be just as big as the game itself. Even then, the results may not be accurate.
While manual testing has been evolving over the decades, some of the core elements are still practiced to this date. While it’s a well-known model, it helps to list down some of the best practices that all companies must focus while testing a game:
Define the process: Establishing a process before test start. Use the process as a guideline for the QA team. This can further be modified / updated over time to fit the QA team’s need. Break down the whole testing cycle into phases (including but not limited to) – Alpha, Beta, Regression, Merge, and Release phases.
Tailoring the testing: No testing process will be a ‘one-size fits all’. Each game should be treated as an individual entity and the testing process should focus on the game elements. This should help to set the right amount of attention needed to test all areas of game.
Dev testing: While most companies think they need to involve QA only after the game is feature complete. However, this approach is inadequate as the QA team may find issues with the core functionality which will take more time debugging. Having a lean dev team (1 to 2 members) right from a working prototype will help to iron out the core elements of the game. This saves time and costs since the development team can just focus on adding features and finishing sprints. They would not have to revisit the core elements.
Verified checklists: Test cases are a great way of verifying all areas of the game. These are vital as it eliminates the risk of assumption or overlooking certain features. However, as games go through various stages of development, certain test cases may become obsolete or irrelevant. These need to be verified and changed according to the feature change in the game. Routine cleanup of checklists needs to become a mandatory activity.
Ad-hoc testing: This should be given a high priority and focus while testing the game. While executing checklists will give a good coverage, it will only help to have a linear approach while testing the game. To test the game exception handling system, the testers must try to break it in multiple ways. Straying from the non-linear path and trying to produce as many non-favorable outcomes is the strongest way to perform ad-hoc tests. No approach is deemed crazy and the wilder the tests are, the chances that they will make a good game into a great one.
Communication: Just finding issues is half the work performed. The ability to communicate issues in a neat and timely manner will make a world of difference. While bug writing is very simple, it does take a good amount of experience to write it effectively. The best approach is to keep the sentences concise without losing any content. It also helps when the game stakeholders interact with the QA team on a common chat-based groups. This helps in clearing doubts, refining the QA process and basically set day-to-day communication. For defect uploading, using bug databases is the most fool-proof method. Usage of Excel sheets, checklists, etc. are not recommended as it may lead to accidental loss of data. While some databases come at premium costs, there are many which are available for free on the internet. They are simple to setup and use.
QA greenlight: The gaming industry is a dynamic and robust place where unexpected changes and unrealistic deadlines unfortunately happen to be the norm. However, there should not be any shortcuts when it comes to performing QA. Every feature needs to be properly vetted by QA and only when it has been declared bug free, the build should be moved to production. Unverified builds can lead to multiple complications and could have a negative effect in the community.
Simulators VS actual devices: This should not even be a debate. While simulators can help to test a game with varying effects, there is no guarantee that the game will perform the same on a real device. Testing it on physical devices will give the most accurate data related to how the game behaves / performs.
Independent QA partner: While it’s great to have QA performed in-house, there can be a few issues that may be brushed off as feature. There may also be some degree of compromise due to employees’ personal relationship. Hence, its better to have an independent team to provide QA solutions. This will help in sorting all the issues as the deliverables will be unbiased and will focus on finding most issues.
Focus group: It may be great to have a game that’s bug free. But it boils down to the most important question, is the game fun to play? Having focus groups will help to understand what makes the game tick. These groups focus on multiple factors like immersiveness, difficulty, rewarding system, progression, what they loved / hated about the game, and UX elements. This will help the production team to make necessary changes and make the game more interesting prior to release.
Emerging trends
While manual testing has been and will be most favorable approach into game testing for the foreseeable future, there has been good progression in other forms of QA. Automation solutions have become more robust and are providing solutions for testing back-end and the engineering areas of development. Performance testing has had a higher amount of dependency on both server and client side. Many companies have been working with performance benchmarks to optimize their games. As the number of games grow, so does the security aspect around it. Many firms work with ethical hackers who try to break the game and verify if it is secure enough. Penetration testing is vital to ensure that the game isn’t exploited leading to millions of revenue loss. AI based technology has advanced a great deal and have competed and won against the top e-sport teams in the planet. AI based QA can become a reality in the next decade.
Author: Mr. Sudarshan Ranganathan, Vice President – Gaming Practice (iXie Gaming), Indium Software
