testing the validity and reliability of research instruments

If the research informing policies is founded on inconsistent measurements, they are likely to derail proper policy making with grave consequences for language teachers, learners, and the wider society. Sound measurements are also crucial for education policy makers who rely on research findings to choose, adapt, and implement language education policies. Additionally, in action research, the effectiveness of educational interventions can only be examined through sound measurements of key variables. For the practitioners who rely on research findings to improve their language teaching practices, it is imperative that such research is based on sound measurements of constructs. Such studies are warranted on the grounds that they carry immediate implications for practitioners, policy makers, and researchers. That said, whereas in educational measurement and in language testing, much attention has been paid to investigating reliability and validity of tests used for selection and achievement purposes, the quality of measuring instruments used for research purposes in Applied Linguistics and language teaching remains underexplored. For an instrument to be valid, it has to be consistent (though the term consistency is more precise compared to reliability, in this paper, we used them interchangeably). In educational measurement literature and in language testing, confidence in measurements depends on their consistency and validity. Findings are discussed in light of discursive and sociomaterial considerations and a few implications are suggested. Finally, reliability was more frequently reported in the domestic corpus than in the international corpus. In most cases, primary reliability for the actual data was reported. Further, the most frequently examined facet of reliability was internal consistency estimated via Cronbach’s alpha. However, for 93 instruments, no reliability index was reported and this held true for the measurement of both language and non-language constructs. It was found that information about instrument origin was provided in the majority of cases. The following qualities in measuring instruments were used to code the data: the instrument origin, instrument reliability, reliability facets examined, reliability computation procedures utilized, and the source of reliability reported (i.e., primary or cited). This study examined the reported reliability of 211 quantitative instruments used in two samples of domestic and international theses in Applied Linguistics. Despite this, research into data collection instruments used in Applied Linguistics and particularly in the thesis genre remains limited. The credibility of conclusions arrived at in quantitative research depends, to a large extent, on the quality of data collection instruments used to quantify language and non-language constructs.

Testing the validity and reliability of research instruments