[games_access] RE: game access for learning disabled

Wed May 10 05:19:48 EDT 2006

Hi,

I got about 7 years of experience with earcons due to my MA in audio design and my PhD research on game audio design (in short "functionality of game audio") . 

I would like to point out that it is easy to confuse "earcons" with "auditory icons". I personally do not like using both these terms, because they are confusing to many people. To get a better understanding of their difference, it is best to think of these sounds as "auditory references" - both auditory icons and earcons try to refer to something (for instance, an action or event in the game). However, their operations are totally different.

Earcons is basically "an auditory sign language" (note the word sign, as I will refer to this word later on). Earcons use a system of tones, with which motives can be created, to communicate almost "musically". A motive can consist of one to five tones. By changing all variables except one a new earcon is created. These related earcons can form families. Blattner and Brewster did a lot of research on this subject and wrote down several guidelines for the creation and use of earcons. Earcons can put used in a string to form a sentence like: [earcon1=create] + [earcon6=new file] = [earcon1, earcon6] which means "create new file". Blattner came up with a system which would allow more complex communication using a sentence of multiple earcons (words), in which each word would be either based on rhythm, melody or timbre (or instrument). An example of 2 simple earcons is the sound in Windows XP when you connect and disconnect something through USB. The major downpoint of earcons is that you need to learn the (often not intuitive) earcon language in order to understand what is meant.
Auditory Icons are 'real world' or abstract sounds that often have an intuitive tie with what they represent (Gaver 1986, 1993b). As with visual icons, the keyword to auditory icons is association. Examples of auditory icons are the sound of clearing the trashcan in Windows (you hear a 'real world' sound referring to something being trashed) and receiving an new email in Outlook Express (abstract sound referring to new mail!).

However, if you are using these term, the discussion can easily arise whether or not a certain sound is an abstract auditory icon (email sound) or an earcon (USB connect sound). Therefore I would like to share another vocabulary with you that I think is (much) better. It's basically nothing new. Many years ago an antropologist named Turner introduced a set of terms to describe types/forms of references he found in his field of study (antropology). It turns out that these terms can be applied to basically every field that deals with references. The vocabulary is Signals, Signs, Symbols and Cultural Symbol:

A signal is something that users associate with whatever it refers to without learning or conventional agreement. It is basically the most direct reference. So the sound of an explosing in a game refers to an explosion. The sound of an approaching car refers to an approaching car. Signals only refer to one thing.

A sign is something (thing, event, colour, sound, etc.) that people have come to learn to associate with whatever it refers to through learning, conditioning and/or conventional agreement. There are two types of signs: iconic signs and non-iconic signs. An iconic sign is modelled to whatever it refers to - its appearance has a relationship with whatever it refers to. An non-iconic sign does not have a relationship between the appearance of the sign and whatever it refers to. The example of the sound you hear when you empty your trash in Windows is an iconic sign. The sound of an new email in Outlook is a non-iconic sign. Most traffic signs are a combination of the two. For instance, The "No Cyclists regulatory sign" (http://www.transport.gov.za/library/legislation/roadtraffic/signs/R219.jpg) consists of a iconic sign referring to a bicycle and a non-iconic sign (red circle with line) referring to "prohibited". Combined this is a sign referring to "cycling prohibited". It is often easier to define the meaning of an icon sign than that of a non-iconic sign because of the appearance the icon sign has.
As you can see, there are several resemblences between auditory icons/earcons and signs. Basically, an auditory icon is an iconic sign, and an earcon is a non-iconic sign (language), all in the audio domain. Spoken and written language (words) are also non-iconic signs, as is sign language for the deaf. Onomatopoeia, words that imitate the sounds associated with the objects or actions they refer to, could be considered to be iconic signs. Musical leitmotivs or jingles could be considered to be musical signs. Signs only refer to one thing. 

A symbol is basically a sign that refers to more than one thing - a symbol is multivocal. The meanings of a symbol is contextually and culturally dependend and requires (personal) interpretation by the user. A user does not only understand the meaning of a symbol through learning, but also by creativity and action on his part. For instance, the sound of car horns in a given context such as an animation could be a symbol for the implications (good or bad) of city-life. Some words (or sentences) are symbols as well. A national anthem could be considered to be a musical symbol. 

A cultural symbol (or "ritual symbol") does not only communicate meanings but also theme(s). A theme is more than just a meaning - it often an expression of the nature, structure or directions of the given culture.

All of these (signal, sign, symbols, themes) should be considered in their context, without it you cannot define their meaning. There have been other researchers that have used this terminology for their own modifications (and who came up with many types if signs, for instance). However, in my opinion *as a designer*, these cloud the functionality of such a vocabulary and aside from that, I found some to be incorrect in my opinion.

So back to disabilities and games:

The most appropiate sound design for people with an auditory processing problem would, in theory, be either a signal or an iconic sign - since these are the most easily understood and leave the least room for interpretation (basically what Reid did with the Closed Captions in his Doom 3 Mod is use very well known non-iconic signs (written language) to refer to game audio). However, I would like to adress another issue that is important for deciding which accessible design solution you are going to use:

I am of the opinion that games consist of a diegetic part and a non-diegetic part. The diegetic part being the "world in/of the game" (for instance the dungeonworld in Dungeon Keeper 2, the city in SimCity, the racing track/city with cars etc. in Need for Speed) and the non-diegetic part being the part that does not exist in this world, mostly seen visually as the Head Up Display (HUD). This can be passive (score, health bars) or interactive (button interface such as in the Sims, where the HUD becomes a controller for the diegetic part of the game). 
Audio is coming from both these part of the game. Depending of the game of course, the diegetic part of the game usually has auditory signals (explosions, footsteps, city sounds, etc.) while the non-diegetic part usually has more signs (sounds for when the score is raised, alarm for when the health is too low, clicks for when an interactive part of the HUD is activated) and symbols (music not coming from the world of the game, but from 'the game' itsself). 

Now, in the case of the sounds in Mario Bros, we could start a big (pointless) discussion whether or not these are to be considered sounds coming from the world where Mario is in (making them more a signal - in the world of mario hitting a stone makes that sound) or sounds not coming from the world of Mario, but from the game (referring to the score being raised). And the music, would Mario hear this as well or is this meant for the gamer only? We'll never know, because there's too much room for interpretation. 
However, in more current games, the distinction is becoming more clear and important too. Sound belonging to the diegetic part of the game usually reacts dynamically to a users perspective on the world (panning, volume changes, acoustics (reverb, etc.) and are also real-world sound sources. Sound belonging to the non-diegetic part of the game usually doesn't have these features and usually consists of abstract, ICT-like sound design (of course, due to the huge variety of games, exeptions exist). Music is usually always outside of the game world, although some games play with this (the filter in the vehicle settings in GTA:San Andreas is a good example).

What I am getting at is this: 

If you have a game in which music is referring to the game becoming more dangerous and someone with an auditory processing problem would not understand it, than you could choose for sound to fix the problem (speech sign: "danger is coming", musical sign: a leitmotiv/jingle - think of the Jaws theme, but then only a few bars, sound sign: alarm sound). However, you could choose to do this visually as well (visual radar, alarm icon, etc.). But you have to take care that your solution is logical. You could use an auditory sign language (earcon) to communicate that the aliens are invading the west part of the castle, but if you have a game which does not really feature a HUD (such as the Syberia adventures) and every sound is a signal coming from the diegetic part of the game, a player could assume that when an earcon is played, that this is supposed to be a sound within the game world (because you, as a designer, suddenly introduce a new, non-diegetic dimension to the game). 

This is even more difficult when you want to communicate something in the game world that doesn't make a sound *with* sound. For instance, many designers of blind-accessible have faced the problem with making walls accessible for blind gamers (using radars to avoid walls, giving walls a sound, letting the player bump into a wall and using the collision sound to communicate where the wall is in relation to the avatar, etc.). My personal solution is "no walls if they their function could be replaced by something else" ;) however, this decision has a lot of impact on your diegetic game design.
In audio games/blind-accessible games designers often make the BEEP-mistake (remember my post to Robert recently?). Many audio games resemble a collection of non-iconic signs (beeps, clicks, eeps, etc.) referring to aliens, vehicles, menu options, enemies, directions, etc. Not only is it not fun to listen to just beepbeepbeeperdebeepshootshootbeepbieppepebeeppow, it is very hard to learn the different meanings of the sound. In one of my own designs, Drive, I tried to be as simple as possible with the sound and still many people have trouble understanding the boosters (a non-iconic sign) whereas everybody understands the engine (a signal) when they first play. I think Michelle can testify to that ;)

So when dealing with sound referrals to make a game accessible, think about what part of the sound you are trying to make accessible, what the function of that sound is in the game, where the sound originates from (the game world or outside of the game world), and base your accessible solution on that.

So if the music tries to communicate that a giant stone ball is about the come rumbling down the stairs in TombRaider but this is still out of vision for a couple of seconds, you could choose to add something to the game world - either visually, like an earthquake, or auditory, like the rumbling of the ball (both signals). Or  you could choose to have the non-diegetic part of the game communicate this: a visual or auditory sign communicating "danger!". What I think is important to acknowledge is that games are not simply "human-computer interfaces" - an ATM machine with an auditory interface requires a very different approach than a computer game with audio, the biggest difference between the two being "experience". Much fun and experience comes from what happens in the game world and not so much in the HUD. That a stone ball is about to come rumbling down the stairs and squash your avatar, is exciting for a player, and it is that excitement that I would like to make accessible too. So not just give a deaf gamer or a gamer with a CAPD an equal chance to play the game, but also give them an equal chance to experience the game.

Greets,

Richard (the info above mostly comes from my PhD thesis which will hopefully be ready in November)

http://www.audiogames.net 

----- Original Message ----- 
From: "Reid Kimball" <rkimball at gmail.com>
To: <lynnvm at alltel.net>; "IGDA Games Accessibility SIG Mailing List" <games_access at igda.org>
Sent: Wednesday, May 10, 2006 6:12 AM
Subject: Re: [games_access] RE: game access for learning disabled

http://reid.rbkdesign.com/?p=41 - I wrote a little bit about the
concept of Earcons in games.

-Reid

On 5/9/06, Reid Kimball <rkimball at gmail.com> wrote:
> Interesting info, thanks for sharing. I think the concept of Earcons
> is actually used extensively in games. A classic example is Mario,
> jumping has a distinct "boing" sound and collecting coins has another
> specific sound. The sounds when distinct and tied to only one action
> clue the user that they have accomplished something. In more complex
> 3D games, a player may not see they are collecting objects, but
> because of the "earcon" they know they must have done the action that
> the earcon represents.
>
> -Reid
>
> On 5/9/06, Lynn Marentette <lynnvm at alltel.net> wrote:
> >
> >
> >
> >
> >
> > Hi-
> >
> >
> >
> > I haven't had much time to interact here lately, between work and taking
> > classes.
> >
> >
> >
> >  I've done some thinking about game access for people with learning
> > disabilities and attention deficits. I am a school psychologist, so I have
> > worked with many students who have milder disabilities over the years. Most
> > of the students I know really like to play computer or video games, but some
> > get frustrated with certain genres.
> >
> >
> >
> > A few months ago I wrote about the concept of "Universal Design for Gaming",
> > based on the principles of Universal Design for Learning developed by David
> > Rose and Anne Mayer at CAST (Center for Applied Special Technology -
> > http://www.cast.org.In a nutshell, in an ideal world, all games (and
> > instruction), would be designed from the very beginning with Universal
> > Design principles in mind.
> >
> >
> >
> > Many of the students I work with have auditory processing problems,
> > short-term auditory memory deficits, or problems with working memory.  Even
> > thought they might have an average or higher IQ, this can be a problem when
> > they play games, as it is in life.
> >
> >
> >
> > Icons would make many games more accessible for people with a wide range of
> > disabilities.  For example, for those who have memory problems, icons could
> > be embedded in the game (with the option of turning them off or on), to give
> > the player hints throughout the game.
> >
> >
> >
> >
> >
> > Earcons might have some use in making games more accessible for people who
> > have auditory processing problems. I've noticed that in many games,
> > background sounds, even background music, provide players with hints about
> > what is about to happen next.  Gamers who have auditory processing problems
> > may not pick up on this, even though they hear the sounds. An earcon could
> > serve the same purpose.  The earcon option could be turned on or off.
> >
> >
> >
> > Here is someone's webpage about earcon research:
> >
> > http://www.dcs.gla.ac.uk/~stephen/research.shtml#earcons
> >
> >
> >
> >
> >
> > There are plenty of people who have visual-spatial difficulties - they don't
> > play games where they are likely to get lost and frustrated. Hints- through
> > earcons, icons, text, or a clear map system (in-game GIS?) might be helpful.
> >
> >
> >
> >
> >
> > This is off the subject a bit: I noticed that there was a link to Priority
> > Woods school, in the UK.  Is that the school that was linked to the old
> > Peepo.com?  Some of the students I work with have severe disabilities, and I
> > used to take them to Peepo.com sometimes.
> >
> >
> >
> > Lynn Marentette
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> >  From: games_access-bounces at igda.org [mailto:games_access-bounces at igda.org]
> > On Behalf Of games_access-request at igda.org
> >  Sent: Friday, May 05, 2006 12:00 PM
> >  To: games_access at igda.org
> >  Subject: games_access Digest, Vol 22, Issue 7
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > games_access mailing list
> > games_access at igda.org
> > http://seven.pairlist.net/mailman/listinfo/games_access
> >
> >
> >
>
_______________________________________________
games_access mailing list
games_access at igda.org
http://seven.pairlist.net/mailman/listinfo/games_access
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist7.pair.net/pipermail/games_access/attachments/20060510/575380a6/attachment.htm>