Global Navigation
Navigation
Lineage
Games PC Functionality and Compatibility Testing Case Study
KETNETKICK 2
“We never distribute new game functions to our players without a check-up by Testronic Labs”
(Maarten Geens, VRT)
“Reliable company, very flexible to work with. The good co-operation and fast feedback (day to day) quickly made us feel comfortable about the quality of the product.”
(Bjorn Quirynen, VRT)
1. The Client
VRT is the national Belgian television and radio broadcasting station, offering two main channels on the public television network, and 6 channels on the radio network (FM). The television channels are "EEN" and "KETNET/CANVAS". While "EEN" is the main station for the bigger audience, "KETNET/CANVAS" is a split channel that broadcasts television programs for children on KETNET and special interest programs on CANVAS. While KETNET runs most of the time during the day, CANVAS broadcasts during the evening and at night.
VRT values the quality of their multimedia products and therefore contacted Testronic Labs Belgium for testing purposes. The testing was to focus on PC Compatibility and PC Functionality.
Testronic Labs drew up a PC Functionality test plan designed to comprehensively cover all elements of the game. For PC Compatitibility, TL experts defined a list of configurations to cover an agreed range of current market permutations – the wide range of processors, graphics cards, sound cards and internet connections available in the well-appointed Belgian lab made this a straightforward process for the client.
VRT were already familiar with the Testronic Labs service having successfully collaborated on a similar title. This project, "KETNETKICK 1", was a major success amongst Belgian children so returning to TL for the QA of its successor game was the only option.
2. The Background
KetnetKick 1 and KetnetKick 2 are being developed by Larian Studios, known for their role-playing games "Divine Divinity", "Beyond Divinity", "The L.E.D. Wars" and their promising future title "Divinity 2 - Ego Draconis".
KetnetKick 2 is a single player game with built-in multiplayer functionality. The game is based on adventures (these adventures are scripted and can be dynamically inserted into the game by the development team), riddles, minigames and special so-called studios where children can compose music, make drawings, make stop-motion movies with a webcam, make comic books etc.Multiplayer functionality allows children to play as a team against AI players. It also features a simple chat function and a "maintain your friends" section. Player data is online-stored on remote servers.
The game is an example of true interactivity, beyond mere single and multiplayer functionality as the creations that kids make in the in-game “studios” are published on the broadcasting station's website, and displayed during dedicated television shows! The media crossover of game/internet/broadcast television has proved a hit amongst teenagers.
3. The Project
The project consisted of several parts in several runs. Here we distinguish the following test types that were carried out:
PC Compatibility
- to make sure that the game displayed the graphics correctly on a wide range of graphics cards
- to make sure that the game performed adequately when running on a broad scope of processors and memory sizes
- to make sure that the different operating systems supported by the game did not influence the basic functionality
- to make sure that the performance of the game was adequate when multiplayer sessions were initiated by minimun specification systems
- to make sure that the game played adequately when using different types of internet connections (cable, fibre, adsl, dsl)
PC Functionality
- to make sure that all the functionality offered by the game was functioning properly, both single player and multiplayer
- to make sure that multiplayer sessions were error-free
When looking at the number of test rounds we can distinguish the following:
- A first round of testing occurred at the beginning of 2008. But the game dramatically changed content and improved a lot during the middle of 2008.
- A second round of testing occurred in October 2008, where the first run consisted out of both PC Compatibility and PC Functionality testing.
- A third round was used for regression testing
- A fourth round was used to test the first patch that fixed several issues discovered during regression testing. During this run, the patching mechanism was tested as well.
4. The Test
For PC Compatibility a tailored system list was created to suit VRT's needs regarding minimum and recommended system specifications. Approximately 60 systems were used during the test, and they all featured different hardware in order to make sure that the game would run on market-available hardware.
For PC Functionality a large test plan was created that covered all the aspects of the game. The test plan and test script covered almost everything from install, gameplay, storyline and multiplayer sessions to uninstall. The script was closely followed by the test team, which allowed them to cast an expert eye over every minor game detail.
A dedicated “exploratory testing” team were in place to ensure that items which the test plan did not cover were checked as well. During testing, the test team received new instructions from the VRT development team on a regular basis. This helped the test team to focus on specific parts of the game that displayed weak spots.
For all of the tests, reporting was done in an online database system called MANTIS. This allowed Testronic Labs to enter bugs quickly, including screenshots, movie clips and log files to illustrate where appropriate.
Since the database was online, both the test team and the development team had access to the bugs. A useful feedback system in the Mantis database ensured that communications were fluent and that issues could be resolved without delay and with minimum inconvenience on the client side.
5. The Feedback / Results
The biggest objective for VRT was to get their game on the market in time, and at a quality level that was high enough to overcome customer dissatisfaction.
During testing, a high number of bugs were located. For PC Compatibility testing a total number of 91 bugs were reported, while the PC Functionality tests allowed the test team to find 192 bugs.
Due to the good planning at VRT and the flexibility of Testronic Labs in running the project in the desired timeframe; VRT were able to release the game at the scheduled date despite the proliferation of bugs.
6. Client Satisfactory Quotes
“We never distribute new game functions to our players without a check-up by Testronic Labs” (Maarten Geens, VRT)
“Reliable company, very flexible to work with. The good co-operation and fast feedback (day to day) quickly made us feel comfortable about the quality of the product.” (Bjorn Quirynen, VRT)
