Tineke Brunfaut and Gareth McCray. This research examined the cognitive processing of 25 test-takers while completing Aptis reading tasks. The study investigated test-takersʼ task processing in general, and according to a number of task and test-taker characteristics.
More specifically, sub-analyses were conducted to explore potential differences in cognitive processing between tasks targeting different CEFR levels, and between test-takers of different L2 reading proficiency and overall L2 proficiency. To this end, a combination of eye-tracking and retrospective interviews with eye-tracking traces as stimuli was used. Test-takersʼ L2 (reading) proficiency was measured by means of the full Aptis test.
It was found that test-takers engaged in a wide range of cognitive processes while completing the Aptis reading tasks, including the lower- and higher-level processes defined in Khalifa and Weir (2009) (with the exception of intertextual representation). Although successful item completion was most often associated with a careful global and/or local reading approach, expeditious reading was conducted by some test-takers on some tasks. Only a few potential threats to the testʼs construct validity were identified, and these risks were associated with specific individual items, not with the tasks or test as a whole (so these may be solved at the item writing level).
Different patterns were observed in the main forms of processing used to complete the different CEFR-linked tasks, which seem to be largely related to task type (more so than CEFR target level). Although these patterns did not constitute threats to the overall Aptis componentʼs construct validity, the B1 gap-fill tasks may at least partly elicit different cognitive reading processes than those set out to be tested with this specific task type. Some trends were also noticed in the processing conducted by test-takers of different levels of L2 (reading) proficiency, although these were weaker (potentially due to the sample of participants).
Overall, the data indicate that the Aptis reading component as a whole samples widely from the construct of reading. These findings provide key information for Aptis validation purposes.
Methodologically, the combined use of eye-tracking and stimulated recalls proved achievable and, moreover, fruitful. The two methods allowed balancing the strengths and weaknesses of each individual method, generating a richer and wider-reaching set of data than each alone, and allowing triangulation of the findings of each method.