|
||||||||||
Torchlight Usability Report Publish date: April 21st, 2011 Software Version: 1.15
Table of Contents A usability test was conducted on site at DigiPen Institute of Technology. The test was using the demo version of the software running on an HP laptop, where the goal was to determine if the current level of tuning difficulty is inline with the broad casual target market. There were similar tests done pre-launch which were met with good results, however a post launch test was ordered as we’ve been receiving review and forum feedback that the game is indeed too difficult. The test was conducted on April 20, 2011 where 3 participants were gathered by our testing committee who randomly selected participants who fell within our target demographic pool. There were 2 separate tests conducted that day and both were held post a dry run test with internally selected participants. All testing was completed in a single day and results were compiled and delivered the same day – complete with all test answers, demographic profiles and recommendations & findings.
Why This Test was Conducted and Background This test was conducted to test the usability of the Torchlight Demo as a part of the final project for DigiPen's GAM 252 course. Torchlight seemed to be a good post game that would easily be accessible to all students for the purpose of the assignment. Pre-production on Torchlight began in August 2008, shortly after the dissolution of Flagship Studios. Full production on the game started around November 2008, giving the entire project a development period of approximately 11 months with 25 team members.
The impact this testing will have on the game will be minimal as not only will these tests probably never reach the developers, but the game has already been shipped. Any current projects that Runic Games are working on will be unaffected as well.
TYPE OF TEST The type of test we performed was a summative test wherein the game had already shipped and we attempted to substantiate the intended goals and targets.
CONDITIONS Location: Digipen Institute of Technology Hardware: HP Pavilion dv9535nr Notebook PC Software Version: 1.15 demo Screen size: 17” Proximity to other testers: No proximity. The tester was not near any other testers.
USER PROFILES / PERSONAS Age Range: 26-30 Sex: Male (100%) Female (0%) Hardware owned: XBOX 360 (33%) PS3 (33%) Wii (66%) Owned following RPG games: Oblivion, Diablo series, Dragon Quest IX Location: North America Purchase Behavior: 3-10 games a year Play behavior: 2-10 hours a week
SESSION FORMAT How the tests were done were we would ask if a nearby user would be willing to test an RPG game for about 15 minutes, and we had the main menu up and running ready for the new player. We monitored play up until the Orden Mines: Level 2. We monitored how long it took them to get through the menus and how long it took them to get to the first sublevel of the Orden Mines. After they completed testing, we had a questionnaire ready for them to answer. Upon completing the questionnaire, the user was instructed that they were no longer needed and said thanks!
EFFICIENT ? Definition: This test was to see if users were able to complete our section of the game in a timely fashion.
Main Menu navigation: Pass Town navigation: Pass Dungeon traversal: Pass
Outcome: Success. Ranging from 8 to 18 minutes, efficiency was determined to be a successful one, as users were able to navigate the menus, traverse through the town, and complete the first level in the Orden Mines grotto in a timely fashion.
ENGAGING ? Definition: This test was to see if users were wrapped up in the game and seeing if they were getting into it.
Story engagement: Fail Interesting NPCs: Fail UI familiarity: Fail
Outcome: Fail. Users did not seem to be very engaged while playing. They complained often about too much text or unhelpful tips. See the appendix questionnaire for some of these problems. Being used to other RPG type games, a majority of the users felt the experience fell flat.
EASY TO LEARN ? Definition: This test was to see if users were able to pick up and play the game with little to no frustration.
Menu navigation: Pass NPC/Enemy interaction: Fail Terrain traversal: Pass
Outcome: Success. Users were all able to quickly choose the player avatar and animal companion with ease. Navigating the town at first took a few minutes to fully get the hang of it, but it did not impede them from continuing the test. Users complained that once a tooltip was gone, you could not re-access it, but overall battle and movement was easy to pick up and use effectively.
Was the Games music appropriate for all situations? Scenario #1: Menu / Character choosing Result: 100% of users stated they felt the music was appropriate and that the music never felt repetitive.
Scenario #2: Torchlight Town / Ambient Result: 66% of the people playing the game found the town music appealing and fitting for the setting of the game.
Scenario #3: Dungeon / Battle Result: 33% of the users who played the game did not turn off their speakers to enjoy the ambience of the dungeon’s battle music with enemies.
Was the menu interface easy to navigate and understand? Scenario #1: Creating a new character Result: 100: of users were able to discern what character they wanted to choose.
Scenario #2: Choosing difficulty Result: 66% of users felt the difficulty was easy and intuitive to choose. 33% Felt the descriptions were lying to them and chose an easier mode.
Scenario #3: In-game menu Result: 66% of users felt the in-game menu was intuitive, while the remaining percentage was confused by the fact that there were no keyboard options.
Were the in-game controls comfortable and appropriate? Scenario #1: Accessing map Result: 33% of users found it easy and to access the map.
Scenario #2: Left click to move, interact, and attack. Result: 33% success rate among testers. Most were confused about how exactly the click system worked.
Scenario #3: Were hotkeys intuitive Result: 66% thought they were intuitive. The controls reminded them of Diablo, with some tweaks.
Heuristic #1: User Control and Freedom Test Case #1: The user should understand that where they should go in the beginning of the game is down into the mines. Findings: During the introductory Torchlight town, the user is once given a block of text telling them where to go, but after they close it, wander around and talk to people and eventually forget what to do. Playtesters were not able to be trained properly because of this. Result: Fail Recommendations: Add a map blinker or add something to the quest inventory to let the user know where to go at all times if they forget.
Test Case #2: Non-Linear traversal options for the player Findings: Players took various routes throughout the town and each dungeon had multiple ways to go and randomized. Result: Pass Recommendations: None. Torchlight handles nonlinearity very well aside from the stairwell chokepoints.
Heuristic #2: Consistency and standards Test Case #1: The keyboard layout of hotkeys mirrors that of similar RPG titles Findings: Users naturally pressed ‘I’ to bring up items menus and similar hotkeyed submenus as they worked similarly in Diablo and other dungeon crawls. Result: Pass Recommendations: A few choice hotkeys like pressing A for map instead of TAB Why only change a couple of controls?
Test Case #2: Clicking on NPCs produce the same action each time Findings: Players consistently clicked and misclicked friendly and enemy NPCs. Accidentally talking to friendly NPCS while trying to attack enemy NPCs, and running toward an enemy instead of attacking them. Result: Fail. Recommendations: Left click should not do everything in the game. Moving, attacking, talking, selecting are all too many options for one button. Move navigation to another button or open up WASD for movement.
Test Subject #1: Age: 26 Sex: Male Games owned of the RPG nature: Arena and Fallout Tactics Types of games you play: FPS’, puzzlers, action Location: Redmond, WA Number of game purchases in a year: 10 How many hours of games do you play a week: 3
The following questions range on a scale from 1 to 10. 1 being bad, and 10 being awesome.
The following questions were yes/no or short answer questions.
The control feature of the player because when in combat, clickin on an enemy often ended in me charging the enemy.
I liked that when I die, I get a choice between spawning where I died at the cost of experience and gold, or at the beginning of the dungeon with only lost gold, or in the most nearby town with no loss of XP or gold.
No, I felt each deeper dungeon looked identical to the previous (higher level) mines. (usability edit: see appendix comparison image to see what the tester meant)
No. I played as the female Vanquisher and I felt my arrows were weak.
Test Subject #2: Age: 30 Sex: Male Games owned of the RPG nature: None Types of games you play: RPG, FPS, Sim, Race Location: Redmond, WA Number of game purchases in a year: 3 How many hours of games do you play a week: 2
The following questions range on a scale from 1 to 10. 1 being bad, and 10 being awesome.
The following questions were yes/no or short answer questions.
The left mouse button being used for both movement AND attack.
None.
Don’t know.
No. I did not feel “powerful”. I felt the ATK+DEF ratio was balanced.
Test Subject #3: Age: 26 Sex: M Games owned of the RPG nature: Diablo, Fallout 2, StoneKeep Types of games you play: RPG, action, Strategy Location: Redmond, WA Number of game purchases in a year: 3-10 How many hours of games do you play a week: 8
The following questions range on a scale from 1 to 10. 1 being bad, and 10 being awesome.
The following questions were yes/no or short answer questions.
I did not like how the story was told. The scrolling text and voiceovers were boring and uninspired which I felt added nothing to the game and ruined the RPG experience.
Combat was quick and fun and the spider animations looked great!
They were repetitive with little variation. I only saw a couple of dungeons though.
Well, I played on “normal” difficulty and most enemies would go down really fast.
Number of participants 3
Quote typifying findings “This just makes me want to play Diablo II.”
Comparison picture A - Orden Mine Floor 1
Comparison picture B - Orden Mine Floor 2
As shown above, the deeper levels show little to no visual progression of moving deeper into the mines.
♞ TOP ♞
|