ALCE Linking to CEFR
Linking the ALCE™ to the CEFR
Hellenic American University and Hellenic American Union have implemented a Linking Project to link the Advanced Level Certificate in English (ALCE™) to the Common European Framework of Reference for Languages (CEFR), in accordance with Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching and Assessment (CEFR) Manual (Council of Europe, 2003; 2009).
The result of the Project is a Claim by Specification and Standard Setting, compiled in accordance with the Manual, which shows that the ALCE™ examination is targeted at the C1 level of the CEFR. This document has been submitted to the Council of Europe. As a result, articles concerning the ALCE™ Linking Project have been published and presentations at international conferences have been given.
Purpose and Context of the Linking Project
The Advanced Level Certificate in English (ALCE™) is a high-stakes examination designed for candidates who require certification of their competency in English as a second or other language at an advanced proficiency level. The candidates are tested on their communicative competency in all four language skills: reading, writing, listening and speaking, as well as in their vocabulary and grammar resources. The examination is administered four times annually in Europe, with monthly administrations also available in specific countries.
The goal of the Linking Project was to assess the level of the ALCE™ examination in terms of the levels defined by the CEFR. The ALCE™ examination was designed to bridge the gap between examinations claimed to be at B2 and C2 level and it was therefore necessary to investigate whether the ALCE™ examination could be linked to the C1 level.
The procedures to link tests to the Common European Framework of Reference for Languages (CEFR) are divided into four phases, as proposed in the Preliminary Pilot Version of the Manual for Relating Language Examinations to the CEFR:
- Phase 1: Familiarization
- Phase 2: Specification
- Phase 3: Standardization
- Phase 4: Empirical validation
Phase 1: Familiarization
An in-depth familiarization process with the content and levels of the CEFR was carried out. This involved studying the Manual and the Preliminary Pilot Version of the Manual, a discussion of the aims, objectives and content of the CEFR, familiarization with the global scales defining the six levels and the descriptors, and familiarization with the descriptors from the DIALANG scales.
Phase 2: Specification
The specification phase involved Project members mapping the coverage of the ALCE™ in relation to the categories and levels of the CEFR. For this a full description of the format, content and rationale of all sections of the test was needed. Using information gained during this process, a content analysis of the ALCE™ was then carried out in order to complete Forms 1 – 23 of the Preliminary Pilot Manual. These completed forms were included in the Report to support the Claim by Specifications.
Phase 3: Standardization
The procedures for the standard setting of the ALCE™ examination followed those set out in the CEFR literature. The procedures included the selection of the most appropriate and effective method of standard setting for each section of the test, the selection of a large number of judges, based on their qualifications and experience of language teaching and testing, the training of the judges on the appropriate criteria, as defined in the CEFR Manual, the establishment by the judges of cut-off scores for each section and data analysis of the judges’ cut-off scores. In order to justify a Claim by Specification, full details of each of these procedures, including detailed information on the data analysis, was included in the Report.
The Advanced Level Certificate in English (ALCE™) examination is divided into four parts: a Listening Section of 40 items; a Grammar, Vocabulary and Reading Section, comprising 40 grammar items, 40 vocabulary items and 20 reading items; a Writing Section with one task; and a Speaking Section.
A separate total score and cut-off score is provided for each of the four Sections, and these are then recalibrated to give equal weighting in the final scoring. The overall pass/fail grade is awarded through aggregate scoring, which allows candidates a narrow fail in one Section, provided their overall score is above the combined overall cut-off score.
The procedures for the standard setting of the ALCE™ examination follow those set out in the Common European Framework of Reference for Languages, the Preliminary Pilot Version of the Manual on Relating Language Examinations to the Common European Framework of Reference for Languages, and the Reference Supplement to the Preliminary Pilot Version of the Manual for Relating Language Examinations to the Common European Framework of Reference for Languages.
The procedures used were as follows:
- the selection of the most appropriate and effective method of standard setting for each Section of the ALCE™ examination
- the selection of a large number of judges, based on their qualifications and experience of language teaching and testing
- the training of the judges on the appropriate criteria, as defined in the CEFR and the Manual for the Advanced Level Certificate in English
- the establishment by the judges of cut-off scores for each Section
- data analysis of the judges’ cut-off scores
Since the Sections of the ALCE™ examination which test receptive skills (i.e. the Listening Section and Grammar, Vocabulary, Reading Section) are machine scored, the standard setting for these two Sections followed precisely the same procedures. However, as the Sections which test productive skills (i.e. the Speaking Section and the Writing Section) are scored by individual raters, the standard setting for each was performed separately.
The standard setting was carried out on the items used in the January 2006 ALCE™ examination, and occurred in successive meetings during January and February 2006. Standard setting was again carried out on the Writing Section in June 2006.
Phase 4: Empirical Validation
Empirical validation of the ALCE™ will be published in forthcoming reports.