Devil’s Advocate vs Phil Wade on Exams and Testing

This series is inspired by a conversation between Mike Hogan and myself about examining the controversies in ELT. We wanted to consider the different positions taken by different members of the industry. However, to do so, we’d need a debate, a disagreement of sorts. And it became apparent that we either tend to agree with members of our PLN (flying creatures of the same feathers and all that), or would keep an open mind and be fairly polite and supportive of one another (that is why we tweet and blog). Seeing that, the only way to get a real debate going was to actively play Devil’s Advocate (DA).

The following debate took place as an Instant-Messaging Chat on Skype. The statements of here are of the DA and in no way represent my beliefs about teaching. This is merely a tool to spark a dialogue between you, the reader, and all those involved in this project. You can find previous instalments of DA here.

The third victim on DA is Phil Wade.

Phil has taught in various schools and universities and is now teaching in-house managers and MA students. He is passionate about exams, and has set up and run courses for IELTS, CAE, TOEIC, and most other test. This has led him to become an examiner, work on pre-tests, and create a range of exam preparation material, tests, and online course. Phil can be found on Twitter and sharing ideas on blogs such as his own. 

Chia:  Phil, are you ready to be hot-seated?

Phil:  Famous last words. I’ve booked a hospital bed for tonight.

Chia: Hahaha…actually, from previous discussions, I’m of the understanding that you are the one with a gripe to air today…

Phil:  Oh, I’m northern so just like moaning. My gran once made a waiter go to a shop to buy tomatoes because the salad didn’t have any.

“You can’t have a bloody salad without tomatoes”.

Chia:  Hahaha…and your rant today is about exams? What issues could you have with them?

Phil:  Hmm. I have countless grudges against exams and the result they are having on EFL and students.

Chia:  I guess you are not talking about the positive effect that exams have in motivating students then?

Phil:  Motivating? Well, if putting pressure on students to pass some exam which often is not actually useful, then perhaps they are, but how about all the students who fail or aren’t allowed to do exam prep classes because they are too low. Quite de-motivating perhaps?

Chia: Being motivating does not mean letting everyone pass and giving everyone a pat on the back…

Being down on exams just because some students fail is like saying sports day is demotivating and scrapping it completely because some students would not get a gold medal.

In real life, there are goals, and there are targets to aim for. Exams provide that benchmark of achievement so that students have clear goals to strive for. And when these goals are reached, the student can feel a great sense of self-confidence and fulfillment.

Phil: Many school/university exams are made so easy that everyone gets 100%. There’s not much motivation or self-fulfillment there if everyone gets the same mark.

Marking is also very subjective and the examiner is the one who decides if answers are correct. To avoid this problem though, some places just use MCQ and GF questions.

Chia:  But let’s consider this: a teacher/school might give an exam simply to remind students to study and review their work, and might make it slightly easier so that the students’ confidence is boosted. Is that such a bad thing?

Phil:  Stress. That’s what a lot of my students say when I announce a test will take place or if you surprise them with one they may say it is unfair. Why not just set up an activity that requires them to use and demonstrate their knowledge? After all, that’s what we want, not just a lower order answering of questions or ticking of boxes…

Chia: Sure, students are going to tell you it’s stressful…but a lot of students do excel when given the competitive element and attempt to outdo themselves and measure their progress…

Of course, testing our students in the classroom does not just mean discrete item tests, but should definitely include activities that allow us to measure their communicative competence as well…But those are tests too, aren’t they?

Phil:  Of course, including the huge numbers in counselling or who commit suicide due to exam stress in Asia where depression amongst students is the norm, due partly to continual tests.  Testing communicative competency is a tough one, breaking it down into categories is very difficult and certainly open to interpretation.

Chia: It’s not the fault of exams as such but the fault of society that drive people to suicide. And you know that in Asia, the issue isn’t with just EFL exams but university entrance exams…. But since our topic today is EFL exams…

Phil:  OK. I will restrict my defence to EFL exams then as you wish. Wouldn’t you say parents camping out outside exam centres and jumping on candidates to find out what they were asked just to help their son/daughter get a better mark is a bit OTT? Or 18-year-old kids getting sent to the UK and not being allowed back until they get band 6 or TOEIC 750? Yet, this has led many students to be quite happy about failing as they get to stay abroad….

Chia: But many of these EFL exams such as IELTS and the Cambridge main suite exams aim to provide an accurate description of the student’s level and a benchmark of achievement for the students, don’t you think?

Phil: Benchmark of achievement? Achieving what though? Passing by any means necessary which includes memorization, buying papers and answers from the net, cramming the night before or planning how to meet every single band category? It’s this kind of exam industry that’s killing EFL .

As for IELTS, really? There are plenty of students walking round with 6 on speaking who can’t really speak and a whole lot more who did crash courses to get 5.5 who can neither write a real essay or debate. They learn IELTS English and cram in every trick, strategy and memorize phrases to get the band. What’s even worse is all the students doing FCE, IELTS, CAE general courses who just learn exam English every day…….

Chia: Really? I’ve often found that by talking to students for a bit, you can more or less judge relatively accurately what sort of IELTS bandscore they might be. e.g. I was speaking to this student the other day and I thought, ‘I bet you she’s an IELTS 6’ and true enough, she was. Doesn’t such regular occurrences testify to the fact that the exam is reliable?

Phil:  So, why doesn’t every 6 get a 6? Some get 5 or even 7? Test day nerves, being prepared, getting the right questions, time management, etc, all play a part. If your lovely student was asked to talk about her garden for 2 minutes but doesn’t have one, then what kind of mark would she get? And then how would she handle a 4-minute debate about open green spaces if she didn’t know about it or even understand it. There are cultural clash problems, examiner bias and training issues, not to mention the fact the many examiners can’t agree exactly on the scores, which is why there is some official difference allowed….

Chia:  Phil, you don’t have a Woolly Mammoth, but could you describe one for two minutes?

The student doesn’t need to have a garden to describe one…

I’m sure the student has heard of a garden or seen one on TV (thanks to the influence of Western TV and media) to be able to describe one.

If not, creativity on the part of the student could allow her to talk about not knowing what a garden is for two minutes, or ask to talk about something related to gardens that are more relevant to her.

Phil:  No and even less than no if I was from a country that didn’t have that on the curriculum. OK, I’ll be more specific about my garden example, ‘Do you have a garden?’ ‘What do people do in their gardens in your country?’. If you lived in a flat all your life in a big city and had intermediate level English without strategies to deal with that question, then you would get stuck. Many of these Cambridge exams, if not all, are made in England by seasoned examiners or people with many years experience, and every new batch has to be different. Yet the big markets for IELTS remain in India and China. This clash means that questions are often very culturally-bound and thus don’t work very well. While some are just daft and many people refuse to ask them…

Chia:  It seems to me that the issue you have is with the universal structure and format of the IELTS exam. Would you not be happier, as you seem to have suggested, with culturally and regionally sensitive versions of IELTS?

Phil: Regionally sensitive? Sounds good but how can you compare Xiao Lin in Shanghai with Sara in Paris then? That’s the problem. BULATS in a way is helping as it adapts to the answers students give, or so I am told.

Chia: But regardless of the cultural or regional biasness of the exams, couldn’t the candidates employ the strategies and tricks you were referring to earlier to cope with these issues?

Phil: Exam strategies and task item knowledge are useful, but some schools lecture 1000 candidates for 2 days and promise a score of 5.5. Having marked these students on their writing and speaking, you can see that they all just memorise answers and paste them together, but it’s often enough to get 5/5.5. They also ‘crack the codes’ allegedly by analysing so many reading/listening papers that they know what works and even the style of the papers. A more shocking example is that there used to be a website that listed all the speaking examiners and what type of questions they preferred and how to answer them.

Chia:  Using such illegal and underhanded means to try and get the scores needed is clearly unacceptable practice, but tests and exams do not always need to be the be all and end all. It can be a positive reinforcement to their learning.

Our students often come to our classes with an objective. And the objective often can be narrowed down to specific goals in communicative ability. Shouldn’t they want to know what kind of return on their investment (time and money) they are getting? Continual classroom testing can be seen as benchmarking and demonstrating progress to the student, as well as identifying areas on which they still have to work. Why therefore would your learners not want to jump at such an opportunity?

Phil: Continual classroom testing? Perhaps frequent informal pair/group tests for revision can maintain motivation and give students a good measure of how they are doing but nowadays aren’t we more about individual goals and progression? A bog standard test from a book cannot do so.

Chia:  We are talking about continual testing, but no one said anything about using a bog standard test from a book to carry this out. Of course testing should be tailored to the needs and goals of the students as well as what they have been learning. Good testing procedures can examine communicative competency and aren’t usually about discrete item tests. You can’t deny me that…!

Phil:  Continual assessment sounds like a DELTA discussion where everyone says how great it is but can’t agree on how to do it. I’ve had the argument many times and finally got to run some courses just based on continual testing. It really shocked students and they wanted to know how they would be assessed and thus we had to create VERY detailed marking schemes that cause a lot of time to get eaten up and long meetings about how to use them. Testing communicative competence is rather difficult. You could just give a general ‘participation’ mark but that is less reliable and open to scrutiny if someone complains, but then again so is the previous example….

Chia:  How about giving students a mark based on your own professional opinion and stop worrying about detailed marking schemes and criteria?

Phil:  Own opinion?

Chia:  You (the generic you referring to all teachers) are a professional, you should be able to do that….And if you aren’t able to, than you should be questioning what you are doing in the first place?

Phil: But isn’t that biased and based on if I like them or if they reached my standard? Consciously perhaps no but unconsciously there will be some bias. Why not swap classes and judge students you don’t know? I have had years of this problem where I set up mark schemes to test speaking and establish quotas for how many people can get A, B… yet teachers very rarely stick to them and just say “they were all so good” or “they are the nicest students I have”. Don’t forget, students are sometimes very good at playing teachers.

Chia:  Are you saying that teachers are any less professional than sports referees? Being unbiased is a fundamental part of who referees are. Are you saying teachers are only semi-professional at what they do? Unbiased objectivity is a part of their professionalism. Just like every sales person who leaves their conscience in bed when they go to work every morning, a teacher involved in testing should also leave their personal opinions in bed when they go to work.

Phil:  Oh. You’re trying to get me in trouble here. I can see the lynchmob waiing at the door. All I’m saying is that we are human and if a student is always late or was once rude to you, then that will naturally affect how you mark him. If we are talking about examiners, then I think there is research to show how they can be affected by body language, the time of day, if the student is the first/last candidate, if they are polite, eye contact etc. The same for the student in that if the examiner is tired and not very polite when they enter they may lose confidence. I even had 1 who just headbutted the desk as he couldn’t cope with it while many just stand up and leave. A completely professional person who cannot be influenced by anything would be wonderful but probably a robot, sounds like TOEFL IBT  (The online version of TOEFL with a computer speaking test so I hear)

Chia: It’s not testing as such that you are against but the influence external factors can have on examiners that you have your gripe with. If science fiction is anything to believe, then terminators, transformers and other robots should be engaged to do the testing…

Phil:  That is where we are headed as Cambridge are working on online versions of their main exams, and why not? I know that stuff like IELTS, FCE etc are very difficult to set up and aren’t run all the time while TOEIC can be run in your school quite easily but BULATS can be done online and you get the results straightaway, it’s cheapish too. We can’t stop technology and it will eventually be good enough to assess students but not now…..

Chia:  Are you really admitting that using technology and testing students online is the way to go?

Are you really suggesting that the practicality of running the test is more important than its reliability?

In an era where we are moving towards using English as a lingua franca and more concerned with one’s ability to communicate and to negotiate meaning… in an era when CEF has progressed to assessing one’s level according to ‘can do’ statements and not discrete item grammar tests, are you suggesting that online tests would be better than the professional opinion of a language teacher at evaluating the learner’s communicative competence?

Phil:  Well, it’s happening more and more. For instance, in translation, machine versions are becoming the norm in many places and are then just tidied up. We’ll get to a point where programmes can assess writing and speaking but I’m sure Nik Peachey can add a lot more to this.

It depends on who you are asking. It took me 7 months to set up and run and deal with an inhouse IELTS test, and it caused a lot of problems. I would also add that it was FAR less reliable than one in an exam centre. Why? Because students were relaxed, too relaxed, and it was sandwiched between other classes. If they’d gone to a centre they could’ve got ‘in the mode’ so to speak better. Yet, as demand rises we are seeing more and more inhouse tests for 200, 500 students or in Asia for 1000+…

Chia: Phil, you seem to be contradicting yourself here. Earlier, you said that students find tests stressful and that the nerves ensured that they weren’t able to perform at their best and show their level. Now, you are saying students are too relaxed and so couldn’t speak well???

Phil:  Contradiction? Perhaps, but not really. Overstressed is one thing, like in China where students break down and cry in tests or the police come and rescue them from traffic jams so they don’t miss tests or when they pay body doubles to replace them in IELTS speaking tests. By relaxed, I mean that developing an ‘exam mentality’ or being mentally prepared for the exam is useful. The trip to the centre, having breaks and doing the speaking after the other papers all in a professional ‘exam centre’ seems logical. Far better than leaving your maths class for 15 minutes to go to your usual classroom and do a speaking test for 15 minutes.

I am also against testing because of the backwash/washback effect it can have. In my last department students wouldn’t do anything unless it was for TOEIC so the first year, they were tested with TOEIC and had TOEIC-ish classes, then they did a TOEIC prep class and did the test and all failed then did another prep class and so on. Over 20% left with no diploma as they failed 4 times and missed out on lots of other courses. If we’d tested them on what they had learned, then I could have passed them. But companies demand the TOEIC, so we do it even though most students admit they will never need English and that they actually don’t need to speak/write, but what the heck, why rock the boat? This ‘failing culture’ makes some places look ‘superior’ and makes the exam companies and publishers lots of money. Teachers rake it in for extra classes and students re-sit classes, so it is actually in the interests of such schools to fail students. I do NOT agree with this at all!

Chia:  It sounds like what is needed is some kind of standardization of these exams and for teachers to recommend students to embrace exams for the positive backwash it can offer, rather than the negative ones. Perhaps it is the responsibility of the teachers involved and the examiners to provide appropriate feedback to the exam boards so that the exams can be more reliable, and not just be concerned about face validity.

Don’t you think that the professional teacher should be choosing the appropriate tests to suit their students’ needs, and guiding students by ensuring that the exam provides some structure to their learning process but doesn’t become the only goal in their learners’ journey?

Phil: Yes, feedback to exam boards is needed. IELTS do lots of research on the effects and use of their test, which is brilliant, but it still remains ‘an exam entrance test’ for many, while FCE, CAE are ‘exams for life’ which is silly as when you are 90 you probably won’t be as good….

You try telling students what test to take, good luck! They all have a goal and they choose TOELF, TOEIC or IELTS for university. The traditional way was TOEFL for the US and Koreans and French still love TOEIC. I have encouraged students to do CAE/CPE/BEC but if they aren’t motivated enough they wouldn’t put enough work in. I had 1 girl who had done FCE, CAE, CPE, IELTS and TOEIC. Why? Because her teachers needed measurable goals….

We got lots of low levels demanding high scores. I know that some schools are proud of their ‘100% pass rates’ and do this by only prepping students who already could pass…..

Chia:  Wait a minute. You were talking about negative backwash earlier and how students became obsessed with preparing for the exams. Shouldn’t exams be about testing the students’ existing levels? Why then the need of exam preparation classes? Don’t exam preparation classes do all the things that you disagreed with earlier? – You said that some schools just teach students exam strategies and as a result…’They all just memorize answers and paste them together but it’s often enough to get 5/5.5. They also ‘crack the codes’ allegedly by analyzing so many reading/listening papers that they know what works and even the style of the papers.’

Exam preparation classes shouldn’t exist. Students can do the practice tests on their own so as to familiarize with the test format. But ultimately, for an exam to be reliable in testing the students’ level, there shouldn’t be any preparation classes to help students fool the examiners…

Phil: I’ve had CPE+ students who decided to do the test for fun and passed but they got help. Teacher advised books, gave them tips and did mock speaking exercises with them….

There is an increasing amount of self-prep stuff out there from books to online stuff. There are even online courses with video F2F speaking tests and written analyzed essay marking. This blended approach is very good and is developing.

I wouldn’t say the prep class are there to fool the examiners but some are. The reason is demand. Students will get what they want if they have money. They are so obsessed due to parental pressure that they buy these ‘crash courses’. Some are also so busy that they have no option. I’m sure you have prepared for a test and not just revised everything you did. It’s like a driving lesson where you practice on the test route…

If, and I do emphasize IF, you insist on subjecting your poor emotionally distraught students to taking a test then prepare them with info about the format, the question types, how to answer them, time management tips, common topics and give them some practice. I really hate the ‘test by testing’ approach, which is based on the belief that students will get better by just doing tests while teachers go get a coffee.

Chia:  But there’s a thin line between giving students tips and exam strategies, and giving them templates and ‘codes to crack’, isn’t there?

Phil:  But give them the choice and which would they choose?

1. Here are lots of tips which may or may not be useful or

2. Here is an essay template you can use to answer any FOR/AG essay and get 5.

Chia:  They would choose to pass the exam and get good grades of course.

And whichever is the path of least resistance…

Phil:  Dada. Human nature.

Chia:  Therefore, because it is so difficult to draw that distinction between giving helpful tips and help with exam preparation and feeding the student with the way to getting the grade they want, shouldn’t exam preparation classes just be banned?

Phil:  Oh, Sneaky! I doubt most teachers (me included) know these tricks so it probably isn’t a problem but I really do think an exam teacher should get to know the exam they are teaching very well. They should know how it’s structured, how texts are designed and thus tested, common questions/topics and then help their students prepare at their own level.

There are some excellent BC webinars by Sam McCarter who just says he uses a text to prepare his students and get them to look at what could be tested and how. This is very useful and better than a 2-hour lesson of endless reading exercises. The same goes for a speaking/discussion class that brings in language, grammar and finally transfers it to an exam style. We can teach exams in a useful way and not just do exam style stuff that students soon get tired of. Last year, some colleagues and I did this and I was surprised that almost half of the classes were made up of students who did not want to take the test at all but enjoyed the classes and learned a lot. That was their motivation.

Chia:  Ah ha! So you are admitting that exams can be a source of motivation for students and can serve as a cherry on top of the cake…the cake being the enjoyable language learning that they are involved with that is designed not just solely to help them get better exam scores but also to improve their overall proficiency of English, thus helping them to achieve their personal goals and provide a return on their investment too?

Phil: I knew you’d say that. Sorry to burst your balloon but these students liked the class because they didn’t want to do the alternative, which was a TOEIC test test test prep class. They came to IELTS and just relaxed and still passed the TOEIC with high scores. These students were also happy, very happy to learn GE as they had never had it before, so in comparison to learning about plugs, electrical engineering and doing Scientific writing it was the most fun they’d ever had. In a language school students should be doing these topics already….

Chia: I suppose the type of test and what it is testing, and how the exam prep class is conducted, can make a big difference. TOEIC, although popular with the companies in Japan and Korea, simply isn’t reliable and students who take it are merely exposed to the negative backwash of retaking the test until they get the score they desire. On the other hand, a format like IELTS, although less practical and therefore more expensive to administer than TOEIC, seems much more reliable and IELTS exam preparation classes allow for flexibility and opportunities for students to actually make progress in their general level of English.

Phil:  Yes, I knew a kid who had done TOEIC 8 times. He was burnt out. I would agree that IELTS is probably the best and most regular test at the moment and the inclusion of speaking and writing can make a good class but the average coursebook doesn’t focus enough on those productive elements which most students get lowest marks in.

Chia:  So you are not entirely against tests or testing then, Phil?

Phil:  I am against exams taking over courses and schools and the negative effects they can have. I used to have students who had tests every week and they just went from cramming 1 to cramming another and learnt nothing. But, we can’t escape. They are still needed and useful and despite all the drawbacks will continue to be useful for quite a while.

Chia:  Indeed. Sorry for setting up some naughty little traps for you, Phil. I was after all playing DA…

Phil:  No, it was very funny!

Epilogue: Phil and Chia are still PLN-buddies who regularly banter on Twitter. Phil still works with exams and exam preparation courses, and any opinions expressed are Phil’s own and do not represent any organization he is associated with. Chia was only playing DA, as usual.

Dogme in Exam Preparation Classes

It’s often widely argued that Dogme cannot be applied in Exam preparation classes as they often follow a syllabus and have strict guidelines as to where students are headed and the language they need to know in order to pass the exam.

However, many exams these days no longer utilize discrete item tests like gaps fills. Instead, most exams seem to be looking at what students can do with the language they know, and through the use of topics as prompts, assess the range and accuracy of language students use to communicate and organise their ideas. For instance, students answer essay questions such as ‘Sports do not bring people together. They tear people apart. Do you agree?’

In fact, the IELTS exam itself is a bit like a Dogme lesson – Here’s a topic, let’s see what emerges.

So why do we feel that in order to effectively prepare students for these exams, we need to systematically take them through the exercises of a coursebook and strictly follow a syllabus?

I took an IELTS preparation class recently and was prescribed a neat little coursebook which I embarked on trying out. However, in the classroom, instincts seemed to take over and using the topics of the coursebook as  a departure point, I started to do the following:

For lexis

  • Get students to brainstorm words related to the topic while I mindmap on the board.
  • Do lots of revision and recall sessions using back-to-board or board rush games.

For speaking and discussions

  • Get students to discuss certain issues related to the topic in pairs, mindmap on their mini-white boards, and feedback to the class.
  • Divide the class into two with one group agreeing and the other disagreeing and conduct a class debate after some prep time.
  • Get students to close their eyes and visualize a scene they have to describe as the teacher raises awareness of all their senses, taking them through the sights and smells of the scene. Students then open their eyes and take turns describing to their partner (great for IELTS Speaking Part 2).

For writing

  • Give each pair a different essay question to draft the main points of the essay for 3 minutes. After 3 minutes, the question is passed on to the next pair. After the brainstorming ideas for all those different questions, the class writes the introductory paragraph to the first question. The teacher writes one too. They show each other the paragraph and the class is guided into noticing certain features of writing the teacher has used. The class attempts to emulate what the teacher has done with the introduction paragraph of the next topic.

The above procedure can be done not just with introductions, but with any paragraph or short piece of writing. Repeat as many times as you like because each time, different language issues emerge, and it allows you to take students through everything from linking words to thematic structure (theme and rheme) to how to write overview statements.

For listening

  • Get students to pick a TV programme to watch as homework. You can specify the genre e.g. documentaries, or the film e.g. An Inconvenient Truth, Supersize Me. Students have to make notes and summarise the film for fellow classmates.
  • Do intensive listening exercises e.g. using the BBC News Headlines in class.

For grammar

  • So much grammar emerges from the discussions and the writing tasks that it is really a matter of the teacher being principled in the eclectic way they improvise in the classroom.
  • I find corrections for written homework best done with the whole class as a delayed correction slot so that students can learn from each other’s mistakes and think about how they can reformulate sentences to make them better.

So my one-month IELTS class went the Dogme way and feedback from my students was overwhelmingly positive. A couple said that they had never learnt so much in a month before.

And the irony is – everything they learnt came from them.