What QA Automation does in game development: tasks, challenges, competencies
According to Shukov, QA Automation is a young direction that came from software development. The industry emerged from the symbiosis of QA engineers and developers who write code – this discipline was born at the junction of these two specialties.
The need for test automation came from the ever-growing size of the games themselves. The amount of work is so big that no testing team can handle it.
When ten releases are released a year, you have nine of them checked perfectly, and the tenth - some bugs are allowed. It seems that people are the same, but over time, the human factor accumulates and shoots at the most inopportune moment. In parallel with this, testers grew up who wanted to connect technologies - so that the program would check the program. The essence of our work is to create such a software package that will take the build of the game, run a set of tests on it and give the result: are you all right?
Alexander Shukov
QA Automation Team Lead, Wargaming.netShukov said that one of the main problems that automation solves is the lack of resources, since the number of functional regression tests is constantly growing. For example, in “Tanks” there are training rooms that are available since the release. The team unloads seven to eight major updates per year, so the code is constantly changing and supplemented.
That being said, every time the developers have to check if this training room is working. There were situations when some other feature changed, but due to the coherence of the code, the room fell off, although no one touched it. This leads to the fact that each new release requires health tests. And the cost of this regression becomes more and more expensive every year, because more and more features come out. At some point, it is no longer enough to hire another tester. It helps up to a point, but then the costs outweigh the benefits.
Romanov added that autotests allow you to test the most stable things as many times as you like – the machine can test at any time, so you can put the process overnight and see the result in the morning. At the same time, tests scale very well: you don’t have to buy devices for testing, you can virtually run the test on multiple platforms and get the result right away.
As Shukov noted, the task of automation is not to replace testers. It allows you to free up the QA resource that is busy in the regression run. Usually, the need to spend resources on the old leads to the fact that the new is not tested, and comes out in worse quality than it could. As a result, the players are unhappy.
According to Shukov, automation is a young industry. There were programmers already in the 60s, testers – too (then there was the position of “engineer”, whose task was to ensure the work of something that someone had already done before). Automation appeared in the 2000s, and in games six to seven years ago. Even now, there are not so many companies that have an entire automation department.
This is because the game development model itself has been greatly transformed. When the task was to release a new version of Doom every three years, and each next game was very different from the previous one (up to the engine), it was not profitable to invest in automation. It’s easier to connect 1000 people from outsourcing closer to the release, test it, fix it and send it to release. And it worked.
Companies that make small single-coil projects still live fine without test automation. And when you are working on the development of a multi-year product, it cannot be adequately implemented without automation.
To one degree or another, automation has been around for a long time. But there were no full-fledged frameworks for testing game logic (in fact, autotest engines). Such examples have appeared relatively recently.
About the gaming test framework
Shukov said that the problem with game devs is that there is no standardized software yet. Of course, in recent years, work has been going on to unify the engines, but this area has yet to develop and develop. When five to seven engines remain, people will start investing in building test frameworks.
According to Shukov, one should not start with the development of a framework – it is long, expensive, and it is not a fact that it will be justified. The framework is needed to work with complex regression, when there is a complex scenario of behavior, and the initial amount of resources that need to be invested in development is calculated in man-years.
It is much easier and more efficient to start with external checks: content checks or some specific narrow game mechanics. Half the difficulty of automated testing is asking the right question of what needs to be tested. For example, it is very easy and cheap to check that all textures in the Floor folder have the same resolution – such a script is easy to write.