In contrast with the multitude of cognitive diagnostic assessment (CDA) studies on reading tests, CDA studies on listening tests are relatively scarce. In particular, little research has been conducted on EFL listening tests that are built for diagnostic purposes, although CDA analyses on such tests may provide more diagnostic information than tests for general proficiency purposes. This study is an initial CDA attempt to validate a general EFL listening proficiency test that is structurally compatible with diagnostic uses, comprising distinct clusters of items to represent multiple dimensions.
The data used in the study were 1,357 candidates’ item-level response data of the listening subtest of Aptis for Teens Advanced. The test is built on a model of cognitive processing and limited to three kinds of target information with increasing cognitive demand, from factual information to interpretive meaning at the utterance level to meaning at discourse level. To facilitate diagnosis, the Q-matrix is specified in accordance with Aptis developers’ item-attribute coding in the test specifications, with its three attributes corresponding to the three kinds of target information. CDA analyses were completed with the GDINA R package, version 2.7.8.
The results showed that:
- the cognitive attributes measured by the test could be distinguished from each other
- factual information was the easiest attribute as expected, but interpretive meaning at the utterance was more difficult than meaning at discourse level for candidates, against the test developers’ assumptions
- the candidate profiles at different CEFR levels partially matched the test developers’ assumptions
- the classification reliability at the test, attribute and pattern levels was generally satisfactory.
These findings support the use of Aptis for Teens Advanced listening tasks as a diagnostic tool to inform remedial learning and instruction, apart from more general uses for admissions, placement, and progress evaluation.