[chisigmail] Usability Questionnaires - SUMI, MUMMS etc, has anybody got any direct experience or comments?

[Toby Biddle]

Hi Tref,
 
I agree with Shane, and I do have a bit of testing/statistical knowledge
having worked for various traditional market research companies in a
previous life.
 
We evaluated both SUMI and MUMMs for use in our business for our client
projects and decided not to proceed with either of these, or in fact any
survey-based usability questionnaires.  These questionnaires are essentially
'customer satisfaction surveys' that focus on design and technical issues
relating to software and user interfaces.  They don't really go far enough
to be able to identify specific usability problems with an application.
Also, usability is more about observation than it is about numbers and
statistics.
 
The only way to identify usability problems is to conduct user research,
preferably one-on-one user testing.  And even better is to couple this with
an analysis of usage statistics.  So, if your application is web-based
you'll be able to collect a range of data that can tell you what your users
are doing and whether any changes you make are having a positive or negative
impact, or indeed no impact at all.
 
Toby Biddle
Research Director
 
t: (03) 9684 3470
m: 0402 113 104
f: (03) 9684 3434
e:  <mailto:toby at UsabilityOne.com> toby at UsabilityOne.com
 
119 Ferrars Street, South Melbourne, Victoria, 3205 

____________________________________________________________________________
________________________________________________________
The information contained in this e-mail and any files transmitted with it
is private and confidential. It is intended for the named addressee only. If
you are not the intended addressee you are prohibited from storing, copying
or using the information in any way. If you received this e-mail due to a
transmission error please notify the sender immediately. No liability is
accepted by UsabilityOne for any losses caused by viruses contracted during
transit over the Internet or present in any receiving system. This e-mail is
not intended to create legally binding commitments on behalf of
UsabilityOne, nor do its contents reflect the corporate views or policies of
UsabilityOne. 
  _____  

From: chisigmail-bounces at chisig.org [mailto:chisigmail-bounces at chisig.org]
On Behalf Of Shane Morris
Sent: Wednesday, December 13, 2006 11:33 AM
To: Tref Gare; chisigmail at chisig.org
Subject: Re: [chisigmail] Usability Questionnaires - SUMI, MUMMS etc, has
anybody got any direct experience or comments?
 
Hi Tref, I don't have specific experience, but I will offer this at the risk
of getting shot down by someone who actually knows what they're talking
about. The questionnaires tend to focus on generic measures like
satisfaction, confidence etc. If your site is at all transaction based (and
you do call it an 'application'), then identifying a few representative
tasks, defining them well (including success criteria), then running period
usability evaluations with metrics (completion rate, completion time, #
errors on the way...) will give you more specific data than tracking generic
(and subjective) measures alone.
 
And if you do you'll be among a exclusive group of organisations who have
actual before+after data, which will make you very popular on the speaking
circuit - I guarantee it.
 
(Note that when doing actual measurements (with a stop watch) then you may
need more subjects than a 'traditional' usability test in order to get
significant data. (mind you, the same applies to the surveys). Can someone
with greater testing/statistical powers should chime in at this point?)
 
Shane
 
Shane Morris  |  User Experience Evangelist  |  Microsoft Australia  |  +61
438 818 888  |   <mailto:shanemo at microsoft.com> shanemo at microsoft.com  |
<http://blogs.msdn.com/shanemo> blogs.msdn.com/shanemo
 
 
 
From: chisigmail-bounces at chisig.org [mailto:chisigmail-bounces at chisig.org]
On Behalf Of Tref Gare
Sent: Wednesday, 13 December 2006 11:23 AM
To: chisigmail at chisig.org
Subject: [chisigmail] Usability Questionnaires - SUMI, MUMMS etc, has
anybody got any direct experience or comments?
 
I'm trying to define some benchmarks against which to compare ongoing
progress in the usability of a specific web application.  Ie: quantifiable
metrics which I can point at and (hopefully) say, look, this stuff works.
 
To that end I've been looking into conducting user group surveys at regular
intervals and was wondering if anyone has any experience with or advice
about existing survey/questionnaires like SUS, SUMI, WAMMI and MUMMS.
 
I'd be curious to hear anybody else's experience or opinions.  Has anybody
used them, do you think they are worthwhile for the stated purpose, are they
as outdated as their websites seem to suggest?
 
Thanks in advance
 
Tref
 
 
  
Please note our new address and phone numbers
 

Tref Gare
Usability Engineer
Aconex
The easy way to save time and money on your project

Level 1, 696 Bourke Street
Melbourne, VIC, 3000, Australia
Tel: +61 3 9240 0200 Fax: +61 3 9240 0299
Email: tgare at aconex.com <mailto:rphillpotgare at aconex.com>   www.aconex.com
<http://www.aconex.com/> 

 

This email and any attachments are intended solely for the addressee. The
contents may be privileged, confidential and/or subject to copyright or
other applicable law. No confidentiality or privilege is lost by an
erroneous transmission. If you have received this e-mail in error, please
let us know by reply e-mail and delete or destroy this mail and all copies.
If you are not the intended recipient of this message you must not
disseminate, copy or take any action in reliance on it. The sender takes no
responsibility for the effect of this message upon the recipient's computer
system.
 
 
 
 
 
 
 
 
 
 
 
 
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://seven.pairlist.net/pipermail/chisigmail/attachments/20061213/70d275f2/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3868 bytes
Desc: not available
Url : http://seven.pairlist.net/pipermail/chisigmail/attachments/20061213/70d275f2/attachment-0001.jpe