The ELF conference starts today in Istanbul, Turkey, at the gorgeous Bogazici University.
After a smooth registration and a few welcoming opening speeches, Professor Tim McNamara delivers his opening plenary on Assessment and ELF.
Here is a summary.
Previously, much has been written about ELF and testing.
Jenkins (2006) challenged Cambridge in resisting implications of ELF.
Taylor (2006) wrote about the difficulties and challenges with applying ELF to testing.
Leung & Jenkins have recently stressed again the importance of recognizing ELF in language testing (in press).
Critique on how criteria of language testing has acted as a roadblock has also been articulated by Seidlhofer.
e.g.
Here are some surprising descriptions in the CEF descriptions
B2 Conversation – Can sustain relationship with NS without unintentionally amusing or irritating them or requiting them to behave other than they would with a NS.
B2 Informal Discussion with Friends – Can keep up with an animated discussion between NS.
Whole section on ‘Understanding conversation between NS’ in CEFR, with no description for those of A1 level (as if to say forget about it).
Assumptions are that
– The interlocutors are assumed to be NS
– The responsibility for successful communication is held to lie with the NNS
– English treated only as a foreign language, like other foreign languages (Seidlhofer, 2011)
Instead
What would ELF test look like (Harding, 2011)
– Ability to tolerate and comprehend diff varieties of English
– Abiltiy ot negotiate meaning
– Ability to use Phonological features crucial for intelligibility
– Awareness of appropriate pragmatics
– Ability to accommodate
These are reflected in ICAO language proficiency requirements
Because international aviation is an ELF setting,
And air traffic controller communication with pilot, either of whom may be NNS.
Simultaneous communication going on between single air traffic controller and several pilots.
Recognition of ELF character of communication : compulsory requirements
1. Standard radiotelephony phraseology: Standardized set of words and phrases for use in all routine communication (restricted language)
2. Plain language:
- The spontaneous creative and non-coded use of a given natural language used only when standardized phraseology cannot serve an intended transmission.
- User with high prof must accommodate their uses of English
- Use of a lot of repetition verbatim e.g. readback and hearback
ICAO’s analysis of language as a factor in fatal avaiation accidents
– incorrect use of standardized phraseology
– lack of plain language prof
– the use of more than one language in the same airspace
Thus ICAO prof test policy
Criteria : Pronunciation, structure, vocab, fluency, comprehension and interactions.
If NS, then immediately highest level and not need to test
Lack of faith in validity of tests and policy
Doing the ICAO tests in
Korea:
– Test content in multiple versions published online
– Repeated attempts allowed until version prepared for appeared
– All personnel now compliant
Japan
– Professionally made test for Level 4 rejected
– 80% of personnel would lose jobs
– Easier test used
– All personnel now complaint
Study into miscommunication (fatal!) in Korea
– Miscommunication due to failure of NS to adhere to ICAO policies
– Use of fixed phrases vs spontaneous speech
– Accent, word choice, speed of NS pilots.
– Preference of Korean pilots for communicating with Japanese ATCs, *(because the Japanese adhere to ICAO convention with meticulous precision) cf problems in US, e.g.LAX
– Miscommunication often due to NS waffling, or lack of professional competence (he didn’t know about the adjacent airways).
– An experienced controller is able to know what is happening with just one word.
Lack of validity with ICAO prof tests (and designers are NSs)
Strong performance criteria
– Judging performance against real-world criteria
– Incorporating ability for use (Hymes 1972)
– Testing all participants (NS and NNS)
Weak performance criteria:
– Focus on lang prof alone, narrowly conceived
– Judging against lang criteria only
– e.g. using ELF stimulus material in listening
– cf Korean pilots pre for new destinations by listening to Vietnamese voices.
Tests need to:
– Define difficulty/ability measurement continuum
– The more challenging the task that a person can manage, the higher their ability.
– Ability and difficulty are measured on a single scale
– Cf high jump – ability expressed in terms of the height of the bar.
Test takers need to :
– Negotiate
– Deal with variation
– Accommodate
– Repair
– But traditional criteria still used
– Issue of pairing –cf diving – build in ‘degree of difficulty’?
– Issue of distinguishing contributions of individuals for score reporting purposes.
Assessing NS Perf
– Research on NS performance on communicative tests (Most NS can’t get 9 on IELTS)
– Problem of requiring NS to be tested
– Problem of motivation – hospital example – assessment of moral qualities.
Conclusion:
Thinking about testing and ELF raises broader issue in language testing: performance tests.
Cost and complexity of performance tests have seen return of indirect measures. E.g. in Pearson Test of automated assessment of speech – NS norms central
There are constructive directions in language testing research which can inform ELF testing
But change won’t happen without a struggle – we may be in for a long wait
.
At the end of his talk, Ana Mauranen says the issue of testing NSs is a valid one, so as to ensure equal starting point.
Tim McNamara answers: NS have a strong political & social advantage so do not expect them to give it up without a fight.
Another audience member asks how he seems to be talking about specific purpose testing. But what about general English testing?
His answer: We can apply specific purpose context to general context. e.g. emphasis on communicative competence, ability to accommodate with our language use and accents, etc.
And with this opening plenary, ELF5 is now in full swing…
chiasuanchong 