Inclusive and Supportive Education Congress International Special Education Conference Inclusion: Celebrating Diversity? 1st - 4th August 2005. Glasgow, Scotland

home about the conference programme registration accommodation contact

Results of a Pilot Trial of Scottish Qualifications Authority Examinations in Accessible Electronic format for Students with Disabilities

Paul D. Nisbet, Dr. Stuart Aitken, Neel Shearer
Communication Aids for Language and Learning (CALL) Centre,
Moray House School of Education
 University of Edinburgh, Edinburgh, Scotland
Paul.Nisbet@ed.ac.uk

Abstract

The implementation of inclusive policies in Scottish schools has resulted in an increase in the number of students with disabilities applying to use ‘special’ or ‘alternative arrangements’ in national Scottish Qualification Authority (SQA) examinations. The type of arrangement varies according to a combination of the student’s particular disability or difficulty with print and the subject and nature of the exam, and may involve: extra time; transcription of the paper; use of Information and Communication technology (ICT); reader and/or scribe; enlargement of the paper; transcription of the paper into Braille; use of signing. In 2004, 9,904 students applied to SQA to use alternative arrangements in 45,680 examinations. Administration of arrangements is challenging and expensive: for example a student using scribe and/or reader, in addition to human support, requires private accommodation and separate invigilation for the examination. Discussions with schools and SQA suggested that the use of ICT in examinations could be considerably enhanced if papers were made available in an electronic format, to allow students to type answers directly on screen, and use text-to-speech software to address visual or reading difficulties. Six schools participated in a pilot project to evaluate SQA Standard Grade examinations in ‘Accessible PDF’ format. A primary aim of the project was to compare the effectiveness and independence of the electronic format with the use of human scribes and readers. Students found the electronic examinations easy to use; almost all reported they felt greater independence using them; and most thought that SQA examinations should be made available in the electronic format for students requiring alternative arrangements. The most common suggestion for improvement concerned the quality of the electronic speech, which was regarded as less clear and pleasant than a human reader. Responses from school staff were extremely positive because of the independence offered by the electronic format and because of the potential to reduce demands on staff, accommodation and cost compared with human scribe or reader.

Background

The Scottish Qualification Authority is the national awarding body responsible for student assessment in Scotland. Most students, at around age 15, sit Standard Grade examinations in fourth year of secondary school; Higher examinations are taken in fifth year; and Advanced Higher courses in the sixth year. Intermediate 1 and 2 examinations also have recently been introduced.

Physically, the majority of these formal examinations take the form of booklets. Students with additional support needs who have difficulty accessing the paper can employ one or more Special or Alternative Assessment Arrangements (SQA 2004). For example, a student with physical difficulties may require assistance from a helper, use of a scribe, or transcription of the paper by a teacher; candidates with specific reading or writing difficulties may be permitted extra time, use of a reader and/or scribe; visually-impaired students can utilise enlarged papers, or papers printed in Braille; and pupils with a hearing impairment may access the assessment in sign language.

The precise Alternative Assessment Arrangement (AAA) used is chosen according to the student’s difficulty, and the published assessment arrangements. The AAA is intended to ensure that no candidate is disadvantaged, should ‘not compensate for a candidate’s inability to meet set standards’, ‘should be tailored to meet the individual needs of candidates’, and ‘should reflect, as far as possible, the candidate’s normal way of learning and producing work’. (SQA, 2004, pp.5-6).

In recent years the number of students requesting AAA has increased: from 3,094 in 1995, to 9,904 in 2004. Most students will request AAA for several assessments, with the total number of requests in 2004 rising to 45,680: an increase of 17% from 2003 (SQA 2005). The increase has resource and cost implications for both SQA and schools. For example, a student using a scribe and/or reader requires a private room, and separate invigilation, in addition to a member of staff to act as the scribe or reader.

Aim

The use of ICT is permitted as a type of Alternative Assessment Arrangement, most commonly for subjects that require reasonable quantities of writing: word processing, spellcheckers, word prediction and speech recognition have all been used to sit examinations. Consultation with schools (Nisbet, 2003a) regarding the use of ICT in examinations established a demand for papers to be made available in electronic format, to allow candidates to type answers directly on screen. The aim of this project was to create and trial Scottish Qualification Authority examinations in electronic format.

Method

The format chosen for the electronic examinations was Adobe Portable Document Format (PDF) because: SQA could generate PDF files quickly and easily; the PDF version matched the paper version exactly; and PDF, as a format, is reasonably accessible to students with disabilities (Nisbet, 2003b). However, a basic ‘read-only’ PDF paper would not be suitable: previous research had shown it was essential that students would be able to type answers directly on screen, and that a text-to-speech capability could benefit students with reading or visual difficulties (Disseldorp & Chambers, 2002; Elkind, Cohen & Murray, 1993; Higgins & Raskind, 1997).

SQA provided eight 2003 Standard Grade Examination Papers in PDF: English Reading Text; English Reading Questions; English Writing; French Reading; Biology; History; Maths 1; Maths 2. Adobe Acrobat Professional 6 was used to add ‘Form fields’ to the question papers so that students would be able to type their answers into the paper. The process of converting a question paper into question and answer format was relatively quick: for example it took less than 10 minutes to add 34 fields to the English question paper. The size of the form fields corresponded to the writing space available in the question paper; spell checking was enabled; a red border gave contrast with different background colours; and navy blue text was used to contrast with the black text of the paper itself. In the majority of cases, text form fields were inserted, and ‘Radio Button’ fields were also employed for multiple-choice questions where a ‘tick’ was required.

Adobe Acrobat Reader 6 does have a built-in text-to-speech facility, but it has limited facilities, and so PDFAloud, from TextHelp Systems (TextHelp, 2005) was used to provide a more sophisticated text reading facility. PDFAloud is well integrated into Acrobat, easy to use, and highlights text as it reads. TextHelp Systems provided an evaluation copy of the authoring program which enabled the project team to ‘speech enable’ the SQA examinations papers. Once an examination paper was speech-enabled, it could be read using a free version of PDFAloud, downloadable from the TextHelp web site.

Six schools across Scotland were approached to participate in the project. Four were mainstream secondary schools, one was a school for students with physical difficulties, and one a unit for students with visual impairment attached to a mainstream secondary school. The project team visited each school to install Adobe Acrobat Standard 6 and PDFAloud software and the PDF examinations, and train staff. It was necessary to use Acrobat Standard, rather than the free Acrobat Reader, because while answers could be typed into the form fields using Reader, they could not be saved or printed out. Schools were provided with backup copies of software and examinations on CD, and a User's Guide written by the project team.

In discussion with schools, it was agreed to trial the PDF examinations with students who had either just sat their Standard Grade preliminary examinations (practice assessments which are taken prior to the main diet of examinations), or who had sat Standard Grade the year before, in 2003. Staff were asked to conduct the trials in ‘exam like’ conditions, although given timetable restrictions, it was recognised that students might complete the exam in more than one session. Staff were responsible for identifying students to participate in the trials, which took place from January to March 2004.

An evaluation questionnaire was designed to investigate students’ perceptions of the usability of the electronic examinations, compared with a human scribe and/or reader, and also to research other factors, such as reliability, confidence, familiarity and independence.

Results

Ease of use

Evaluation questionnaires were returned for 31 pupils in five schools, in respect of 94 examinations. Students were asked to compare ease-of-use (on a scale from 1 to 10) of the text-to-speech in the electronic examination with a human reader, if they had used one in the preliminary or previous examination, and ease-of-use of typing their answer in the examination, with a scribe. Overall, the students regarded the electronic examinations as being slightly less easy to use than a reader or scribe.

The mean ease of use for text-to-speech was lower than that of a human reader for all examination subjects; subjects that involved the most reading and writing (English, French and History) were easier to use than maths and biology papers.

Table 1: Ease of use of text-to-speech, compared to Readers

Students commented that they liked the text-to-speech because it was more independent and less embarrassing and they were more comfortable reading a piece of text several times if necessary. The most common negative comment about the text-to-speech concerned the quality of the synthetic voice. All of the schools used Windows computers for the trial and the PDFAloud software was installed with standard Microsoft voices, and free Lernaut and Hauspie TTS3000 British English and French voices. It should be noted that more recent computer text-to-speech products have improved greatly upon those available at the time the study was carried out. ,

The ease-of-use scores for typing into the PDF compared with a scribe show a slightly different pattern. Students who used electronic English, French and History papers all found them easier to use than a scribe: the mean score for these subjects was 8.93 compared with 8.00 for scribes. Students again liked the independence offered by the PDF exams, the opportunity to edit and change typed answers, the option of the spellchecker, and felt the size of the form field gave an indication of the length of answer expected. The electronic biology and maths papers were regarded as being more difficult to use, with a mean score of 4.71.

Table 2: Ease of use of typing into the PDF, compared to Scribes

There was also a difference in scores between schools, and therefore, possibly, in how well the tools addressed pupils’ different support needs. The mean score for the communication subjects given by students from the mainstream secondary schools was 9.48, compared with 9.00 for the maths and biology papers. The majority of these students were described as having specific learning difficulties. In contrast, students with physical or visual impairment found the maths and biology papers to be much more difficult to use (mean score of 3.5 and 3.78) than the communication papers (mean scores of 8.38 and 8.00). Comments from students suggested that the reason for this is that the electronic maths and biology papers, which contain diagrams and charts, are more difficult to manipulate than papers that are mainly composed of text. Students with a visual impairment, who required the paper magnified on screen, had difficulty viewing the text and the relevant diagram at the same time, while many of the questions involved drawing and worked examples using scientific and mathematical symbols, which is challenging using the Acrobat Standard drawing tools.

Table 3 and Figure 1: Comparison of PDF ease of use across subject and school

Other factors

A second part of the questionnaire asked students to score, using a 5-point scale, their familiarity, ease of learning, reliability, independence, and confidence of using readers, scribes and PDF examinations. Students were also asked if they felt that examinations should be made available in the electronic PDF format.

As we might expect, students felt less familiar with the PDF examinations than with scribe or reader. Human readers were easier to learn to use than text-to-speech, but typing into the exam was easier to learn than a human scribe. Reliability and confidence scores were very similar, and show that readers were felt to be more reliable than text-to-speech, but scribes were less reliable than typing into the examination. The mean scores given for independence were striking and significant: pupils rated scribes and readers at only 2 and 2.06 out of 5, whilst the scores for text-to-speech and keyboarding were 4.79 and 4.68 respectively. 27 out of 28 students said that SQA should provide examinations in the accessible PDF format.

Conclusions

The project investigated the use of electronic examinations papers in Acrobat PDF, for use by students with additional support needs. Students compared the ease of use of the electronic papers with the use of readers and scribes. Human readers have two main advantages over text-to-speech: they are easier to use, and the quality and intelligibility of the voice is (usually) better. Students were less familiar with the electronic papers than with readers, and the synthetic voices supplied to read the papers were of poorer quality than some of the new speech engines available such as ScanSoft’s Realspeak or AT&T’s Natural Voices. With practice, and better speech engines, we might expect the scores for the text-to-speech to improve with respect to a reader. Further research to investigate intelligibility of higher quality synthetic voices should be carried out.

Typing into the electronic exams scored more highly than use of scribes. Despite being less familiar, they were easier to learn, more reliable, students felt more comfortable using them and they provided a much higher level of independence than scribes.

The PDF examinations were generally reliable. Staff in one school reported that some of the completed electronic papers were ‘lost’, possibly as a result of issues with the school network. One form field in a Maths paper was found to be incorrectly prepared, and it was also necessary to alter colours of the form fields for one particular student who required a particular high contrast colour scheme.

Staff in schools compared the completed electronic examinations with the papers completed by the students in their preliminary or previous assessments, and a sample of the papers were also marked by SQA markers, who commented that ‘The range of answers proved the same as for the main cohort and there was nothing usual or unexpected’ and ‘I think the standard of grammar is better than equivalent handwritten scripts’ (Nisbet, Aitken, Shearer, 2004, p. 26). PDF examinations should therefore provide an accurate method of measuring attainment, although further trials would be needed to confirm this.

In addition to providing greater independence, staff in schools were enthusiastic about the potential of PDF examinations to improve management and administration of AAA, and reduce costs. One Principal Teacher of Support for Learning wrote: ‘This year we have 19 needing readers/scribes for Standard Grade History and 10 using computers – that means 19 extra rooms, 19 members of staff and 19 extra invigilators while the computer users just need one room and one invigilator – and technical support standing by in case of computer problems.’ (Nisbet, Aitken, Shearer, 2004, p. 27).

Acknowledgements

The authors are grateful to the Scottish Qualifications Authority and the Scottish Executive Education Department for providing funding for this work. We also thank the staff and students of Ashcraig, Broughton High, Perth High, Earlston High, Kirkcaldy High and Uddingston Grammar schools the schools who participated in the project. Lastly, we would like to thank staff at TextHelp Systems, for providing the evaluation copy of the PDFAloud software.

References

Disseldorp, B. and Chambers, D. (2002). Independent access: Which students might benefit from a talking computer? In S. McNamara and E. Stacey (Eds), Untangling the Web: Establishing Learning Links. Proceedings ASET Conference 2002. Melbourne, 7-10 July. http://www.aset.org.au/confs/2002/disseldorp.html

Elkind, J., Cohen, K., & Murray, C (1993). Using computer-based readers to improve reading comprehension of students with dyslexia. Annals of Dyslexia, 42, 238-259.

Higgins, E., & Raskind, M. (1997). The compensatory effectiveness of optical character recognition/speech synthesis on reading comprehension of post-secondary students learning disabilities. Learning Disabilities: A Multidisciplinary Journal, 8, 75-87.

Nisbet, P.D. (2003a) An investigation into the Accessibility of SQA Assessments in Electronic Format. Project report to Scottish Qualifications Authority. CALL Centre, University of Edinburgh.

Nisbet, P.D. (2003b) Accessibility of SQA assessments in Portable Document Format. Project report to Scottish Qualifications Authority. CALL Centre, University of Edinburgh.

Nisbet, P.D., Aitken, S., Shearer, N. (2004) Trial of External Papers in Accessible PDF for Candidates with Additional Support Needs. Project report to Scottish Qualifications Authority. CALL Centre, University of Edinburgh.

SQA (2004) Guidance on Assessment Arrangements for Candidates with Disabilities and/or Additional Support Needs. Scottish Qualifications Authority. ISBN 1 85969 532 9.

SQA (2005) Special Assessment Arrangements, Diets 2002-2004. Personal correspondence.

TextHelp Systems (2005) PDFAloud. (software) www.pdfaloud.com

home . about the conference . programme . registration . accommodation . contact