Non-statistical techniques for assessing the validity and reliability of measurement instruments in quantitative research
Keywords:
validity, reliability, non-statistical techniques, expert judgment, measurement instruments, quantitative research, logical evidenceAbstract
This article analyzes non-statistical and non-mathematical techniques used to assess the validity The article critically analyzes non-statistical and non-mathematical techniques for assessing the validity and reliability of measurement instruments in quantitative research. It argues that the quality of these instruments is key to scientific credibility, but warns that the dominant use of statistical coefficients has reduced the assessment to a limited approach, neglecting logical, rational, and procedural evidence. Based on an exhaustive literature review, it identifies more than thirty techniques grouped into three types of validity—content, construct, and criterion—and three dimensions of reliability—internal consistency, stability, and equivalence. Notable among these are expert judgment, the Delphi method, cognitive interviews, document analysis, the nomological network, logical deduction of hypotheses, logical parallel forms, and standardization of procedures. The study contextualizes each technique from a historical and epistemological perspective, explaining its foundations, applications, and contributions. It concludes that these techniques are not inferior to statistics, but rather complementary and indispensable, and recommends a methodological triangulation that combines quantitative and qualitative evidence in accordance with international standards.
Downloads
References
AERA, APA, NCME. (2014). Standards for educational and psychological testing. American Educational Research Association.
Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). American Council on Education.
Anastasi, A. (1981). Psychological testing (5th ed.). Macmillan.
Bloom, B. S. (1956). Taxonomy of educational objectives: The classification of educational goals. Longmans, Green.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071.
Bowen, G. A. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27-40.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105.
Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Sage Publications.
Cattell, R. B. (1973). Personality and mood by questionnaire. Jossey-Bass.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Rand McNally.
Cronbach, L. J. (1947). Test "reliability": Its meaning and determination. Psychometrika, 12(1), 1-16.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334.
Cronbach, L. J., & Gleser, G. C. (1965). Psychological tests and personnel decisions (2nd ed.). University of Illinois Press.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302.
Dalkey, N., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458-467.
DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). Sage Publications.
Escobar-Pérez, J., & Cuervo-Martínez, A. (2008). Validez de contenido y juicio de expertos: Una aproximación a su utilización. Avances en Medición, 6(1), 27-36.
Grant, J. S., & Davis, L. L. (1997). Selection and use of content experts for instrument development. Research in Nursing & Health, 20(3), 255-263.
Guilford, J. P. (1936). Psychometric methods. McGraw-Hill.
Guilford, J. P. (1954). Psychometric methods (2nd ed.). McGraw-Hill.
Guion, R. M. (1976). Recruiting, selection, and job placement. Brooks/Cole.
Gulliksen, H. (1950). Theory of mental tests. John Wiley & Sons.
Guttman, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10(4), 255-282.
Hambleton, R. K. (1984). Validating the test scores. In R. A. Berk (Ed.), A guide to criterion-referenced test construction (pp. 199-230). Johns Hopkins University Press.
Harden, R. M. (2001). AMEE Guide No. 21: Curriculum mapping: A tool for transparent and authentic teaching and learning. Medical Teacher, 23(2), 123-137.
Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7(3), 238-247.
Hernández-Sampieri, R., Fernández-Collado, C., & Baptista, L. P. (2014). Metodología de la investigación (6th ed.). McGraw-Hill.
Jacobs, H. H. (1997). Mapping the big picture: Integrating curriculum and assessment K-12. ASCD.
Kane, M. T. (2001). Current concerns in validity theory. Jornal of Educational Measurement, 38(4), 319-342.
Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Praeger.
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1-73.
Kerlinger, F. N. (1973). Foundations of behavioral research (2nd ed.). Holt, Rinehart & Winston.
Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Sage Publications.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Sage Publications.
Linstone, H. A., & Turoff, M. (Eds.). (1975). The Delphi method: Techniques and applications. Addison-Wesley.
Loevinger, J. (1957). Objective tests as instruments of psychological theory. Psychological Reports, 3(3), 635-694.
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382-385.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563-575.
Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30(10), 955-966.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Macmillan.
Muñiz, J. (1998). La medición de lo psicológico. Psicothema, 10(1), 1-21.
Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489-497.
Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.
Rulon, P. J. (1939). A simplified procedure for determining the reliability of a test by split-halves. Harvard Educational Review, 9(1), 99-103.
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124(2), 262-274. https://psycnet.apa.org/fulltext/1998-10661-006.html
Spearman, C. (1904). "General intelligence," objectively determined and measured. American Journal of Psychology, 15(2), 201-292.
Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3(3), 271-295.
Thorndike, E. L. (1904). An introduction to the theory of mental and social measurements. Teachers College Press.
Thurstone, L. L. (1931). The reliability and validity of tests. Edwards Brothers.
Tourangeau, R. (1984). Cognitive science and survey methods. In T. B. Jabine, M. L. Straf, J. M. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey methodology (pp. 73-100). National Academy Press.
Tourangeau, R., Rips, L., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press.
Underwood, B. J. (1957). Interference and forgetting. Psychological Review, 64(1), 49-60.
Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage Publications.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Jose Humberto Puente (Autor/a)
This work is licensed under a Creative Commons Attribution 4.0 International License.
- Los autores/as conservan los derechos de autor y ceden a la revista el derecho de la primera publicación, con el trabajo registrado con la licencia de atribución de Creative Commons 4.0, que permite a terceros utilizar lo publicado siempre que mencionen la autoría del trabajo y a la primera publicación en esta revista.
- Los autores/as pueden realizar otros acuerdos contractuales independientes y adicionales para la distribución no exclusiva de la versión del artículo publicado en esta revista (p. ej., incluirlo en un repositorio institucional o publicarlo en un libro) siempre que indiquen claramente que el trabajo se publicó por primera vez en esta revista.
- Se permite y recomienda a los autores/as a compartir su trabajo en línea (por ejemplo: en repositorios institucionales o páginas web personales) antes y durante el proceso de envío del manuscrito, ya que puede conducir a intercambios productivos, a una mayor y más rápida citación del trabajo publicado.
