[chisigmail] Usability Questionnaires

[Chris Khalil]

Tref,

Well I'd echo most of the points already made by Shane, Rod and Ash.  I guess the important thing, from your point of view, is to present a lightweight, approximate measure (i.e. score) of usability that can be used to demonstrate to managers that UCD practices are having an impact on the product.  Managers tend to like graphs and scores of these things.
First, a few references you might find useful:
 
Our old friend Jakob has an interesting view on comparative testing 
A decent book on the subject is  ‘A Practical  Guide to Usability Testing’ by Dumas and Redish
I’d also echo Ash’s comments around the CIF, if you’re interested you might want to check out this HTML template  and some general info around the subject
 
As we all know the key metrics around which you could obtain usability benchmarks are efficiency, effectiveness, ease of learning and satisfaction. There are plenty of other very useful and insightful metrics, but these are your vanilla benchmarks.  The choice of metrics clearly has an effect on the tool you use.  Each measure has its own weight in the overall picture, depending on the goals of your project.  For example, for single use UI’s ease of learning is going to be important. For interfaces which are used heavily and repeatedly (I suspect this is the case with you guys at Aconex), efficiency can be critical. For leisure/pleasure interfaces satisfaction may be of prime concern, and so on. It's up to you) to decide which goals you aim for and therefore which tool you utilise.
 
As for tools you can actually use to do these assessments you may want to consider a modified Heuristic Evaluation  (HE) approach (to reflect the choice of metrics appropriate for the task) augmented with a SUS (to capture the user’s satisfaction with the product).  I’ve used this in the past and it’s worked reasonably well.  To make it clear, this is not User Testing, rather Summative Usability Evaluation.
 
The modified HE approach does requires some back explanation to the user but gives an overall score (which is what you are mainly seeking) and lists the perceived weaknesses in the UI.  Some examples HE’s links etc that might be worth checking out are: 
 
http://www.acm.org/perlman/question.cgi?form=NHE
http://www.id-book.com/catherb/
http://www.stcsig.org/usability/topics/articles/he-checklist.html
http://www.d.umn.edu/is/support/Training/Online/webdesign/testing.html
 
The key to using the HE is to define a mainline scenario/use case (choose one which is representative and repeatable) and then re-test against this at every iteration with new users each time and then follow-up up with a SUS or QUIS so the user can tell you exactly how she feels.
 
I recognise there is still some debate about value of discount methods like HE 
i.e. the 2000 ACM survey, org strategies & usability methods
“[HE is] the most commonly used method…, even though its effectiveness to strategic usability is ranked far below usability testing, field studies, usage scenarios, task analysis, and participatory design (Rosenbaum).”
but I’ve found that when a project has time & resource constraints, when access to users or other stakeholders make it less practical to utilise more formal, full-blown empirical usability lab testing then a modified HE approach works quite well. It has, with minimal input from myself or other HCI or usability specialists provided useful data in the past, and shown objective measures to management which (hopefully) reflect an ever increasing usability and satisfaction score.
 
 
If you’re interested in the details of the HE approach I’ve used etc, please just drop me a line.
 
Hope my ramblings make some kind of sense.  Oh, and say hi to Jo Rice for me!
 
Chris


---
Dr Christopher Khalil
Senior Interaction Designer
Acumentum Pty.
T. +61 (0)3 9863 0309
M. +61 (0)4 06 075167


----- Original Message ----
From: "chisigmail-request at chisig.org" <chisigmail-request at chisig.org>
To: chisigmail at chisig.org
Sent: Wednesday, 13 December, 2006 4:21:09 PM
Subject: chisigmail Digest, Vol 21, Issue 6


Send chisigmail mailing list submissions to
    chisigmail at chisig.org

To subscribe or unsubscribe via the World Wide Web, visit
    http://seven.pairlist.net/mailman/listinfo/chisigmail
or, via email, send a message with subject or body 'help' to
    chisigmail-request at chisig.org

You can reach the person managing the list at
    chisigmail-owner at chisig.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of chisigmail digest..."


Today's Topics:

   1. Re: Usability Questionnaires (Farmer, Rod)
   2. Usability Job Advertisement - Singapore (BTB)
   3. Re: Usability Questionnaires (Tref Gare)
   4. Re: Usability Questionnaires - SUMI, MUMMS etc,    has anybody
      got any direct experience or comments? (Ash Donaldson)


----------------------------------------------------------------------

Message: 1
Date: Wed, 13 Dec 2006 12:03:54 +1100
From: "Farmer, Rod" <farmerra at dis.unimelb.edu.au>
Subject: Re: [chisigmail] Usability Questionnaires
To: chisigmail at chisig.org
Message-ID:
    <0143EE5AAF4F52498315968138D232740150E352 at dismail.dis.unimelb.edu.au>
Content-Type: text/plain; charset="iso-8859-1"

Tref,

Adding to Shane's comments, a lot of the factors included in the questionnaires are "rule of thumb" and typically not supported by a lot of clearly linked theory ( seeing as your view was on quantitative measure, I challenge you to define and assess "experience" on something more robust than a Likert scale), and typically provide little support for tailoring analyses to your current problem.

As for the quality or usefulness of these questionnaires, they're only as good as their purpose, and these ones are typically used for gaining data rather than gaining insight. Shane's comments are valid, and you'll find lots of reasons why not to rely upon questionnaires for assessing usability in most usability/interaction design/HCI books.

As for statistics, you should consider Power and Effect size relative to your objectives before jumping into testing. If Shane is referring to Nielsen's "5 uses is enough" number for usability assessment, then you should forget about just about every form of quantitative statistical analysis other than descriptive statistic (means, medium, ratios, etc.). Stick to variability and forget about correlations.

You could start to use non-parametric measures if you get above 10 users, but again, underlying your data is still the fact that you have, well, 10 users. Also, if you do get enough users, you might want to run a linear regression or PCA over those surveys to assert item independence.

My personal view on the use of these questionnaires is that they can be useful for developing preliminary profiles of your users. Being more qualitative than quantitative myself, I then use those profiles to develop more specific questions to address emerging issues. Obviously, this is not always possible given resource constraints.

Hope this helps...


Rod
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://seven.pairlist.net/pipermail/chisigmail/attachments/20061213/5eafce81/attachment-0001.html 

------------------------------

Message: 2
Date: Wed, 13 Dec 2006 13:51:21 +1100
From: BTB <tbentley at gmail.com>
Subject: [chisigmail] Usability Job Advertisement - Singapore
To: chisigmail at chisig.org
Cc: ciler at pinnacle-search.com
Message-ID:
    <a82127740612121851u3af9734cx1ffa75b092bb7e4c at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Note: if you are interested in this job opportunity please use the contact
details at the end of this e-mail.  Do not reply to me, as I am sharing this
on behalf of Ms. Ciler Gazi.
Todd
**********

Our Clients, a major US based computer manufacturer, are looking for the
following candidates to be based in their Design Centre in Singapore:

*HUMAN FACTORS ENGINEERING MANAGERS
*

   - Responsible for product usability during all phases of development
   across a variety of hardware and software platforms
   - Work closely with design teams to develop solutions for improved
   usability and user satisfaction through user-centred design and testing by
   evolving UI standards and guidelines, developing prototypes, translating
   field or lab customer data into usable design solutions, conducting
   usability tests, analyzing user behaviour and requirements, and leveraging
   results to spark focused design efforts.
   - Ensure the final design meets the marketing requirements, quality
   and cost goals, product specifications, and achieves world-class standards
   of product usability


*Requirements:
*

   - Advanced degree in HCI, Human Factors, Behaviourial Science,
   Computer Science, Industrial Engineering, or a related discipline, with at
   least 6 years of Usability/HF experience, preferably in the PC or consumer
   electronics sector
   - Experience in developing and testing both hardware and software, and
   conducting formal usability testing
   - Experience in working with Industrial Designers on hardware products

   - Proven track record in implementing user-centred designs
   - Able to work in tight date-driven schedules and manage multiple
   programs
   - Experience with multi-disciplinary teams
   - Knowledge of HTML, Java or other programming languages is a plus


If you are keen to explore these opportunities please contact Ms Ciler Gazi
on (65) 6453-5668 or ciler at pinnacle-search.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://seven.pairlist.net/pipermail/chisigmail/attachments/20061213/37aafe42/attachment.html 

------------------------------

Message: 3
Date: Wed, 13 Dec 2006 15:42:38 +1100
From: "Tref Gare" <tgare at aconex.com>
Subject: Re: [chisigmail] Usability Questionnaires
To: "'Farmer, Rod'" <farmerra at dis.unimelb.edu.au>,
    <chisigmail at chisig.org>
Message-ID: <005a01c71e71$1e2dbc40$0300a8c0 at TGARELAPTOP>
Content-Type: text/plain; charset="us-ascii"

Thanks all,



Many valid points raised which shall be added to the brew.



I should qualify my original question a tadd to say we're not looking to
identify usability issues via the questionnaires but to get a comparable
"state of the nation" measurement at regular intervals.  It would be by
necessity a high level view and hopefully indicate whether the perceived
usability of the system is improving or not over time.  There is a
significant internal marketing angle at play here as well. Ie: for our
purpose "gaining data rather than gaining insight" might not be so bad as
long as I can use that data to lobby for further resources and sway.  



The question is whether or not the data itself is that useful in the context
of a complex web based application.  I know SUMI and WAMMI compare the
results against a range of other systems and give comparative "Scores".  I
wonder if those scores are comparable over time and relevant to our
particular environment/system. I don't get the impression that these types
of survey are widely used in the industry and was curious as to why.



Ash Donaldson forwarded me the email of one of the original creators of
WAMMI and SUMI so I'll forward those questions on to him and see what comes
back.



Many thanks again for the responses.



Cheers



Tref









  _____  

From: chisigmail-bounces at chisig.org [mailto:chisigmail-bounces at chisig.org]
On Behalf Of Farmer, Rod
Sent: Wednesday, 13 December 2006 12:04 PM
To: chisigmail at chisig.org
Subject: Re: [chisigmail] Usability Questionnaires



Tref,

Adding to Shane's comments, a lot of the factors included in the
questionnaires are "rule of thumb" and typically not supported by a lot of
clearly linked theory ( seeing as your view was on quantitative measure, I
challenge you to define and assess "experience" on something more robust
than a Likert scale), and typically provide little support for tailoring
analyses to your current problem.

As for the quality or usefulness of these questionnaires, they're only as
good as their purpose, and these ones are typically used for gaining data
rather than gaining insight. Shane's comments are valid, and you'll find
lots of reasons why not to rely upon questionnaires for assessing usability
in most usability/interaction design/HCI books.

As for statistics, you should consider Power and Effect size relative to
your objectives before jumping into testing. If Shane is referring to
Nielsen's "5 uses is enough" number for usability assessment, then you
should forget about just about every form of quantitative statistical
analysis other than descriptive statistic (means, medium, ratios, etc.).
Stick to variability and forget about correlations.

You could start to use non-parametric measures if you get above 10 users,
but again, underlying your data is still the fact that you have, well, 10
users. Also, if you do get enough users, you might want to run a linear
regression or PCA over those surveys to assert item independence.

My personal view on the use of these questionnaires is that they can be
useful for developing preliminary profiles of your users. Being more
qualitative than quantitative myself, I then use those profiles to develop
more specific questions to address emerging issues. Obviously, this is not
always possible given resource constraints.

Hope this helps...


Rod

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://seven.pairlist.net/pipermail/chisigmail/attachments/20061213/c9f3613c/attachment.htm 

------------------------------

Message: 4
Date: Wed, 13 Dec 2006 16:20:54 +1100
From: Ash Donaldson <ash at produxi.com>
Subject: Re: [chisigmail] Usability Questionnaires - SUMI, MUMMS etc,
    has anybody got any direct experience or comments?
To: chisigmail at chisig.org
Message-ID: <50728B2A-F2DA-42CD-9F3F-10208A4C60C4 at produxi.com>
Content-Type: text/plain; charset="windows-1252"

Tref,

Sorry to add confusion to the pot, but I'm going to add my two cents  
which includes 3 recommendations (depending on what level of detail  
you want / need).

1. The WAMMI is the most widely validated psychometric questionnaire  
for usability.  It serves as an excellent method for determining a  
general measure of usability against the largest reference database  
of responses of its kind; and / or setting a benchmark against which  
to measure future design efforts.

Check the site for reviews and papers:
http://www.wammi.com/

Dr. Jurek Kirakowski is well known in this field as being the  
authority on HCI stats and psychometrics - and one of the creators of  
the WAMMI and SUMI.  I'm sure he'd be more than happy to answer any  
questions you have.  Drop him a line at:
jurek.kirakowski at wammi.com

2. Now, since I worked on making it an International Standard, I have  
to also plug an alternative method which is excellent for more  
specific measures of the facets of a product:
ISO/IEC 25062:2006 Common Industry Format (CIF) for usability test  
reports
http://www.iso.org/iso/en/CatalogueDetailPage.CatalogueDetail? 
CSNUMBER=43046

This standard provides a template and guidance for conducting a  
rigorous summative usability evaluation (a replicable experiment,  
noting independent, dependent and possible confounding variables).   
It requires that a benchmark evaluation first be done, then any  
further evaluations can demonstrate any changes in usability derived  
from design changes.

3. Finally, an interesting, cheap and simple method of reliably  
measuring overall customer satisfaction (not just usability) is using  
"The Ultimate question" technique, popularised by Fred Reichheld a  
few years back (he's just published a book on it this year).  It's  
the simplest and most telling survey with just one question:
"Would you recommend this product / service / business to a friend?"
This is a simple barometer of how the product, service or business is  
doing at any point in time.

Some also like to add the conditional question:
"Why / Why not?" - to collect some qualitative data.  Of course, the  
reliability of this data would be much lower.

Reichheldt (2006) The Ultimate Question: Driving Good Profits and  
True Growth
http://www.amazon.com/Ultimate-Question-Driving-Profits-Growth/dp/ 
1591397839/sr=8-17/qid=1165980096/ref=sr_1_17/102-7032928-7036151? 
ie=UTF8&s=books

I hope this helps.

Cheers,

Ash Donaldson
Principal
Produxi Pty Ltd
Designing better user experiences

+61 (0)414 55 9996
ash at produxi.com

http://www.linkedin.com/in/ashdonaldson



On 13/12/2006, at 11:22 AM, Tref Gare wrote:


> I?m trying to define some benchmarks against which to compare  

> ongoing progress in the usability of a specific web application.   

> Ie: quantifiable metrics which I can point at and (hopefully) say,  

> look, this stuff works.

>

>

>

> To that end I?ve been looking into conducting user group surveys at  

> regular intervals and was wondering if anyone has any experience  

> with or advice about existing survey/questionnaires like SUS, SUMI,  

> WAMMI and MUMMS.

>

>

>

> I?d be curious to hear anybody else?s experience or opinions.  Has  

> anybody used them, do you think they are worthwhile for the stated  

> purpose, are they as outdated as their websites seem to suggest?

>

>

>

> Thanks in advance

>

>

>

> Tref

>

>

>

>

>

>

>

> Please note our new address and phone numbers

>

>

>

> Tref Gare

> Usability Engineer

>

> Aconex

> The easy way to save time and money on your project

>

> Level 1, 696 Bourke Street

>

> Melbourne, VIC, 3000, Australia

> Tel: +61 3 9240 0200 Fax: +61 3 9240 0299

> Email: tgare at aconex.com  www.aconex.com

>

>

>

> This email and any attachments are intended solely for the  

> addressee. The contents may be  privileged, confidential and/or  

> subject to copyright or other applicable law. No confidentiality or  

> privilege is lost by an erroneous transmission. If you have  

> received this e-mail in error, please let us know by reply e-mail  

> and delete or destroy this mail and all copies. If you are not the  

> intended recipient of this message you must not disseminate, copy  

> or take any action in reliance on it. The sender takes no  

> responsibility for the effect of this message upon the recipient's  

> computer system.

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

>

> _______________________________________________

> chisigmail mailing list

> chisigmail at chisig.org

> http://seven.pairlist.net/mailman/listinfo/chisigmail


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://seven.pairlist.net/pipermail/chisigmail/attachments/20061213/663e9274/attachment.html 

------------------------------

_______________________________________________
chisigmail mailing list
chisigmail at chisig.org
http://seven.pairlist.net/mailman/listinfo/chisigmail


End of chisigmail Digest, Vol 21, Issue 6
*****************************************

Send instant messages to your online friends http://uk.messenger.yahoo.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://seven.pairlist.net/pipermail/chisigmail/attachments/20061214/dda9c1cf/attachment-0001.html