MANY-FACET RASCH MEASUREMENT PARTIAL CREDIT MODEL AND OPTIMIZING RATING SCALE CATEGORY EFFECTIVENESS FOR RUBRICS: CONCEPT AND ANALYSIS EXAMPLE
Article Sidebar
Main Article Content
Abstract
To develop rubrics through the analysis of The Many-Facet Rasch Measurement Partial Credit Model (MFRM-PCM) and optimizing rating scale category effectiveness benefits in helping determine an appropriate number of score levels in rubrics development. Rubrics development through this model allows for the investigation of validity and reliability by analyzing more than two facets related to assessment. This is suitable for performance assessments where raters are involved in assessment results. This academic article is written to provide knowledge and understanding to teachers and educators about the concept of MFRM-PCM and optimizing rating scale category effectiveness analysis, offering examples of analysis and interpretation. The aim is to help readers apply this in rubrics development in a school or institution context. Developing rubrics prepares teachers for competency-based learning, where learners demonstrate learning outcomes through performance or work that reflects their knowledge, attitudes, and skills. It also assists university instructors in aligning their teaching with learning assessment standards in higher education, which emphasizes developing learners to achieve expected learning goals through assessment and feedback.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
The owner of the article does not copy or violate any of its copyright. If any copyright infringement occurs or prosecution, in any case, the Editorial Board is not involved in all the rights to the owner of the article to be performed.
References
Adams, R. J., Wu, M. L., & Wilson, M. (2012). The Rasch rating model and the disordered threshold controversy. Educational and Psychological Measurement, 72(4), 547-573. DOI: 10.1177/0013164411432166
ASEAN University Network. (2020). ASEAN University network quality assurance: Guide to AUN-QA assessment at programme level version 4.0. ASEAN University Network.
Bond, T. G., Yan, Z., & Heene, M. (2021). Applying the Rasch model: Fundamental measurement in the human sciences (4th ed.). Routledge.
Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences. Springer.
Brookhart, S. M., & Chen, F. (2015). The quality and effectiveness of descriptive rubrics. Educational Review, 67(3), 43–368. DOI:10.1080/00131911.2014.929565
Brookhart, S. M. (2018). Appropriate criteria: Key to effective rubrics. Frontiers in Education, 3, Article 22. DOI: 10.3389/feduc.2018.00022
Chapman, M., Chuang, P., Bitterman, T., & Elliott, H. (2024). Development of a new WIDA writing scoring rubric for grades 1-12 (Report No. TR-2024-2). Wisconsin Center for Education Research.
Clarkson, B., & Luca, J. (2001). Improving assessment: Rubrics in a tertiary multimedia course. Association for the Advancement of Computing in Education.
Edwards, K. E., Edwards, A. S., & Wesolowski, B. C. (2019). The psychometric evaluation of a wind band performance rubric using the multifaceted Rasch partial credit measurement model. Research Studies in Music Education, 41(3), 343-367. DOI: 10.1177/1321103X18773103
Eckes, T. (2009). Introduction to many-facet Rasch measurement. Peter Lang Edition.
Eckes, T. (2019). Many-facet Rasch measurement: Implications for rater mediated language assessment. In V. Aryadoust & M. Raquel (Eds.), Quantitative data analysis for language assessment volume I: Fundamental techniques (pp. 153-176). Routledge.
Furtak, E. M. (2022). Formative assessment. Routledge. DOI: 10.4324/9781138609877-REE62-1
Gervis, J. (2016). The operational definition of competency-based education. The Journal of Competency-Based Education, 1(2), 98-106. DOI: 10.1002/cbe2.1011
George, D., & Mallery, P. (2019). IBM SPSS Statistics 26 Step by Step (16th ed.). Routledge.
Griffith, W. I., & Lim, H. (2012). Performance-based assessment: Rubrics, web 2.0 tools and language competencies. MEXTESOL Journal, 36(1), 1-12.
Harris, L. R., & Brown, G. T. L. (2022). Self-assessment. Routledge. DOI: 10.4324/9781138609877-REE1-1
Hodges, T. S., Wright, K. L., Wind, S. A., Matthews, S. D., Zimmer, W. K., & McTigue, E. (2019). Developing and examining validity evidence for the Writing Rubric to Inform Teacher Educators (WRITE). Assessing Writing, 40, 1-13. DOI: 10.1016/j.asw.2019.03.001
Hughes, S., McMullen, C., Jackson, L., McNamara, G. Matthews, L., Camaradou, J., & Walker, A. (2022). Development and validation of the symptom burden questionnaire for long covid (SBQ-LC): Rasch analysis. BMJ, 377, Article e070230. DOI: 10.1136/bmj-2022-070230
Janssen, G., Meier, V., & Trace, J. (2015). Building a better rubric: Mixed methods rubric revision. Assessing Writing, 26, 51-66. DOI: 10.1016/j.asw.2015.07.002
Johnson, R. L., Penny, J. A., & Gordon, B. (2009). Assessing performance: Designing, scoring and validating performance tasks. The Guilford Press.
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144. DOI: 10.1016/j.edurev.2007.05.002
Jonsson, A., & Panadero, E. (2017). The use and design of rubrics to support assessment for learning. In D. Carless, S. M. Bridges, C. K. Y. Chan & R. Glofcheski (Eds.), Scaling up assessment for learning in higher education (pp. 99–111). Springer.
Jönsson, A. (2022). Scoring rubrics. Routledge DOI: 10.4324/9781138609877-REE2-1
Klimovski, D. (2013). Does a well-defined scoring rubric lead to an improvement in student results? Proceedings of the 2013 AAEE Conference, Australia.
Lallmamode, S. P., Daud, N. M., & Kassim, N. L. A. (2016). Development and initial argument-based validation of a scoring rubric used in the assessment of L2 writing electronic portfolios. Assessing Writing, 30, 44-62. DOI: 10.1016/j.asw.2016.06.001
Linacre, J. M. (1994a). Many-faceted Rasch measurement. MESA Press.
Linacre, J. M. (1994b). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328.
Linacre, J. M. (2002). Understanding Rasch measurement: Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85-106.
Linacre, J. M. (2003). Rasch power analysis: Size vs. significance: Standardized Chi-Square fit statistic. Rasch Measurement Transactions, 17(1), 918–919.
Linacre, J. M. (2023). A User Guide to FACETS (Version 3.87.0) [Computer software]. https://www.winsteps.com/a/Facets-Manual.pdf
Linacre, J. M. (2024). A User Guide to WINSTEPS (Version 5.8.1) [Computer software]. https://www.winsteps.com/a/Winsteps-Manual.pdf
Maydeu-Olivares, A., & Montaño, R. (2013). How should we assess the fit of Rasch-type models? Approximating the power of goodness-of-fit statistics in categorical data analysis. Psychometrika, 78(1), 116-133. DOI: 10.1007/S11336-012-9293-1
Morke, A. M., Dornan, T., & Eika, B. (2012). Outcome (competency) based education: An exploration of its origins, theoretical basis, and empirical evidence. Advances in Health Sciences Education, 18, 851-863. DOI: 10.1007/s10459-012-9405-9
Moskal, B. M. (2000). Scoring rubrics: What, when and how? Practical Assessment, Research, and Evaluation, 7(7), 1–5. DOI: 10.7275/a5vq-7q66
Myford, C. M., & Wolfe, E. W. (2004). Detecting and measuring rater effects using Many-facet Rasch measurement: Part II. Journal of Applied Measurement, 5(2), 189-227.
Office of the Education Council. (2020). Active competency-based learning. 21 Century Print.
Panadero, E., & Jonsson, A. (2013). The use of scoring rubrics for formative assessment purposes revisited: A review. Educational Research Review, 9, 129-144. DOI: 10.1016/j.edurev.2013.01.002
Pancorbo, G., Primi, R., John, O. P., Santos, D., & De Fruyt, F. (2021). Formative assessment of social-emotional skills using rubrics: A review of knowns and unknowns. Frontiers in Education, 6, Article 687661. DOI: 10.3389/feduc.2021.687661
Popham, W. J. (1997). What's wrong - and what's right - with rubrics. Educational Leadership, 55(2), 72-75.
Raykov, T., & DiStefano, C. (2021). Evaluating restrictive models in educational and behavioral research: Local misfit overrides model tenability. Educational and Psychological Measurement, 81(5), 980-995. DOI: 10.1177/0013164420944566.
Reddy, M. Y. (2011). Design and development of rubrics to improve assessment outcomes: A pilot study in a Master's level business program in India. Quality Assurance in Education, 19(1), 84-104. DOI: 10.1108/09684881111107771
Stevens, S. D., & Levi, A. J. (2012). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning (2nd ed.). Routledge.
Thao, N. H. (2023). End-of-term speaking assessment rubric for grade 6 English teachers teaching new English textbooks in Vietnam. International Journal of Education and Pedagogy, 5(1), 127-144.
Thepsathit, P., & Tangdhanakanond, K. (2024). Many-facet Rasch measurement model: Concepts and necessary statistic for performance assessment. Journal of Education and Innovation, 26(1), 435-448.
Van der Linden, W. J. (2017, 31 January). IRT-van-der-Linden-2. https://www.rasch.org/audio/IRT-van-der-Linden-2.mp3
Wilson, M. (2013). Seeking a balance between the statistical and scientific elements in psychometrics. Psychometrika, 78(2), 211-236. DOI: 10.1007/S11336-013-9327-3
Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370-371.
Yuen, B. P. L., & Sawatdeenarunat, S. (2020). Applying a rubric development cycle for assessment in higher education: An evidence-based case study of a science communication module. Asian Journal of the Scholarship of Teaching and Learning, 10(1), 53-68.
