Executive Summary

A usability test was conducted on site at DigiPen Institute of Technology. The test was using the demo version of the software running on an HP laptop, where the goal was to determine if the current level of tuning difficulty is inline with the broad casual target market. There were similar tests done pre-launch which were met with good results, however a post launch test was ordered as we’ve been receiving review and forum feedback that the game is indeed too difficult.

The test was conducted on April 20, 2011 where 3 participants were gathered by our testing committee who randomly selected participants who fell within our target demographic pool. There were 2 separate tests conducted that day and both were held post a dry run test with internally selected participants.

All testing was completed in a single day and results were compiled and delivered the same day – complete with all test answers, demographic profiles and recommendations & findings.

Summary Top Findings

• Positives

  • 80% of players were able to complete all sections of the game under 20 minutes.

  • All players stayed on track 100% of the total play time.

• Negatives

  • 100% of players had problems with controls.

  • Over 66% of players disliked the storytelling.

• Merchants

  • 33% of the players never talked to the same merch again, thinking they wouldn’t have new wares, as they never stated they would..

• Difficulty Level

  • Players assumed the mode that stated it was for RPG veterans would be too hard, even if they had previously played RPG games.

• Visual Cues

  • Less than 50% of players felt that the Torchlight town efficiently led them to the Orden mines.

Why This Test was Conducted and Background

This test was conducted to test the usability of the Torchlight Demo as a part of the final project for DigiPen's GAM 252 course. Torchlight seemed to be a good post game that would easily be accessible to all students for the purpose of the assignment.

Pre-production on Torchlight began in August 2008, shortly after the dissolution of Flagship Studios. Full production on the game started around November 2008, giving the entire project a development period of approximately 11 months with 25 team members.

Dev Cycle Impact

The impact this testing will have on the game will be minimal as not only will these tests probably never reach the developers, but the game has already been shipped. Any current projects that Runic Games are working on will be unaffected as well.

Methodology

TYPE OF TEST

The type of test we performed was a summative test wherein the game had already shipped and we attempted to substantiate the intended goals and targets.

CONDITIONS

Location: Digipen Institute of Technology

Hardware: HP Pavilion dv9535nr Notebook PC

Software Version: 1.15 demo

Screen size: 17”

Proximity to other testers: No proximity. The tester was not near any other testers.

USER PROFILES / PERSONAS

Age Range: 26-30

Sex: Male (100%)

Female (0%)

Hardware owned:

XBOX 360 (33%)

PS3 (33%)

Wii (66%)

Owned following RPG games:

Oblivion, Diablo series, Dragon Quest IX

Location: North America

Purchase Behavior:

3-10 games a year

Play behavior: 2-10 hours a week

SESSION FORMAT

How the tests were done were we would ask if a nearby user would be willing to test an RPG game for about 15 minutes, and we had the main menu up and running ready for the new player. We monitored play up until the Orden Mines: Level 2. We monitored how long it took them to get through the menus and how long it took them to get to the first sublevel of the Orden Mines.

After they completed testing, we had a questionnaire ready for them to answer. Upon completing the questionnaire, the user was instructed that they were no longer needed and said thanks!

Test Goals and Objectives

EFFICIENT ?

Definition: This test was to see if users were able to complete our section of the game in a timely fashion.

Main Menu navigation: Pass

Town navigation: Pass

Dungeon traversal: Pass

Outcome: Success. Ranging from 8 to 18 minutes, efficiency was determined to be a successful one, as users were able to navigate the menus, traverse through the town, and complete the first level in the Orden Mines grotto in a timely fashion.

ENGAGING ?

Definition: This test was to see if users were wrapped up in the game and seeing if they were getting into it.

Story engagement: Fail

Interesting NPCs: Fail

UI familiarity: Fail

Outcome: Fail. Users did not seem to be very engaged while playing. They complained often about too much text or unhelpful tips. See the appendix questionnaire for some of these problems. Being used to other RPG type games, a majority of the users felt the experience fell flat.

EASY TO LEARN ?

Definition: This test was to see if users were able to pick up and play the game with little to no frustration.

Menu navigation: Pass

NPC/Enemy interaction: Fail

Terrain traversal: Pass

Outcome: Success. Users were all able to quickly choose the player avatar and animal companion with ease. Navigating the town at first took a few minutes to fully get the hang of it, but it did not impede them from continuing the test. Users complained that once a tooltip was gone, you could not re-access it, but overall battle and movement was easy to pick up and use effectively.

Test Cases

Was the Games music appropriate for all situations?

Scenario #1: Menu / Character choosing

Result: 100% of users stated they felt the music was appropriate and that the music never felt repetitive.

Scenario #2: Torchlight Town / Ambient

Result: 66% of the people playing the game found the town music appealing and fitting for the setting of the game.

Scenario #3: Dungeon / Battle

Result: 33% of the users who played the game did not turn off their speakers to enjoy the ambience of the dungeon’s battle music with enemies.

Was the menu interface easy to navigate and understand?

Scenario #1: Creating a new character

Result: 100: of users were able to discern what character they wanted to choose.

Scenario #2: Choosing difficulty

Result: 66% of users felt the difficulty was easy and intuitive to choose. 33% Felt the descriptions were lying to them and chose an easier mode.

Scenario #3: In-game menu

Result: 66% of users felt the in-game menu was intuitive, while the remaining percentage was confused by the fact that there were no keyboard options.

Were the in-game controls comfortable and appropriate?

Scenario #1: Accessing map

Result: 33% of users found it easy and to access the map.

Scenario #2: Left click to move, interact, and attack.

Result: 33% success rate among testers. Most were confused about how exactly the click system worked.

Scenario #3: Were hotkeys intuitive

Result: 66% thought they were intuitive. The controls reminded them of Diablo, with some tweaks.

Heuristic #1: User Control and Freedom

Test Case #1: The user should understand that where they should go in the beginning of the game is down into the mines.

Findings: During the introductory Torchlight town, the user is once given a block of text telling them where to go, but after they close it, wander around and talk to people and eventually forget what to do. Playtesters were not able to be trained properly because of this.

Result: Fail

Recommendations: Add a map blinker or add something to the quest inventory to let the user know where to go at all times if they forget.

Test Case #2: Non-Linear traversal options for the player

Findings: Players took various routes throughout the town and each dungeon had multiple ways to go and randomized.

Result: Pass

Recommendations: None. Torchlight handles nonlinearity very well aside from the stairwell chokepoints.

Heuristic #2: Consistency and standards

Test Case #1: The keyboard layout of hotkeys mirrors that of similar RPG titles

Recommendations: Left click should not do everything in the game. Moving, attacking, talking, selecting are all too many options for one button. Move navigation to another button or open up WASD for movement.

Questionnaire Results

Test Subject #1:

Age: 26

Sex: Male

Games owned of the RPG nature: Arena and Fallout Tactics

Types of games you play: FPS’, puzzlers, action

Location: Redmond, WA

Number of game purchases in a year: 10

How many hours of games do you play a week: 3

The following questions range on a scale from 1 to 10. 1 being bad, and 10 being awesome.

1. How appropriate was the games’ music? 9

2. The game’s menu interface was easy to understand and navigate. 10

3. The in-game controls were comfortable / appropriate. 4

4. The tutorial adequately prepared me to play the game. 5

5. It was easy to understand what was happening in combat. 5

6. The game’s pacing was appropriate. 6

The following questions were yes/no or short answer questions.

7. What did you dislike most, and why.

The control feature of the player because when in combat, clickin on an enemy often ended in me charging the enemy.

8. What did you like most, and why.

I liked that when I die, I get a choice between spawning where I died at the cost of experience and gold, or at the beginning of the dungeon with only lost gold, or in the most nearby town with no loss of XP or gold.

9. Did you feel progression as you went deeper into the mines?

No, I felt each deeper dungeon looked identical to the previous (higher level) mines.

(usability edit: see appendix comparison image to see what the tester meant)

10. Did your character feel powerful? Explain.

No. I played as the female Vanquisher and I felt my arrows were weak.

Test Subject #2:

Age: 30

Sex: Male

Games owned of the RPG nature: None

Types of games you play: RPG, FPS, Sim, Race

Location: Redmond, WA

Number of game purchases in a year: 3

How many hours of games do you play a week: 2

The following questions range on a scale from 1 to 10. 1 being bad, and 10 being awesome.

1. How appropriate was the games’ music? 4

2. The game’s menu interface was easy to understand and navigate. 8

3. The in-game controls were comfortable / appropriate. 4

4. The tutorial adequately prepared me to play the game. 5

5. It was easy to understand what was happening in combat. 6

6. The game’s pacing was appropriate. 5

The following questions were yes/no or short answer questions.

7. What did you dislike most, and why.

The left mouse button being used for both movement AND attack.

8. What did you like most, and why.

None.

9. Did you feel progression as you went deeper into the mines?

Don’t know.

10. Did your character feel powerful? Explain.

No. I did not feel “powerful”. I felt the ATK+DEF ratio was balanced.

Test Subject #3:

Age: 26

Sex: M

Games owned of the RPG nature: Diablo, Fallout 2, StoneKeep

Types of games you play: RPG, action, Strategy

Location: Redmond, WA

Number of game purchases in a year: 3-10

How many hours of games do you play a week: 8

The following questions range on a scale from 1 to 10. 1 being bad, and 10 being awesome.

1. How appropriate was the games’ music? 8

2. The game’s menu interface was easy to understand and navigate. 9

3. The in-game controls were comfortable / appropriate. 9

4. The tutorial adequately prepared me to play the game. 6

5. It was easy to understand what was happening in combat. 8

6. The game’s pacing was appropriate. 9

The following questions were yes/no or short answer questions.

7. What did you dislike most, and why.

I did not like how the story was told. The scrolling text and voiceovers were boring and uninspired which I felt added nothing to the game and ruined the RPG experience.

8. What did you like most, and why.

Combat was quick and fun and the spider animations looked great!

9. Did you feel progression as you went deeper into the mines?

They were repetitive with little variation. I only saw a couple of dungeons though.

10. Did your character feel powerful? Explain.

Well, I played on “normal” difficulty and most enemies would go down really fast.

Findings

Number of participants

3

Quote typifying findings

“This just makes me want to play Diablo II.”

Appendix

Comparison picture A - Orden Mine Floor 1

Comparison picture B - Orden Mine Floor 2

As shown above, the deeper levels show little to no

visual progression of moving deeper into the mines.