Assessing Students' Problem-Solving Skills in Heat Learning: Preliminary Validation of a Five-Stage Physics Assessment Instrument
Abstract
Conventional physics assessments often emphasize final numerical answers and therefore provide limited information about where students' reasoning breaks down during problem solving. This study developed and preliminarily validated a process-oriented assessment instrument for heat learning. A quantitative preliminary instrument-validation design was used. Four contextual essay problems were decomposed into five stages focusing on the problem, describing the physics, planning the solution, executing the plan, and evaluating the answer yielding 20 scored sub-items using a 0–4 analytic rubric. Five experts evaluated content validity using I-CVI, S-CVI/Ave, and S-CVI/UA, followed by an empirical trial with 30 Grade 12 students. Empirical analyses included corrected item-total correlations, Cronbach's alpha, item difficulty, and item discrimination. The instrument obtained S-CVI/Ave = 0.960 and S-CVI/UA = 0.800, with I-CVI values ranging from 0.80 to 1.00. Corrected item-total correlations ranged from 0.605 to 0.909, Cronbach's alpha was 0.975, difficulty indices ranged from 0.583 to 0.750, and discrimination indices ranged from 0.406 to 0.750. These findings provide preliminary evidence that the instrument can support stage-specific diagnosis of students' problem-solving performance in heat learning. Larger-sample validation and inter-rater reliability studies are still required.
References
Burkholder, E. W., Miles, J. K., Layden, T. J., Wang, K. D., Fritz, A. V., & Wieman, C. E. (2020). Template for teaching and assessment of problem solving in introductory physics. Physical Review Physics Education Research, 16(1), 010123. https://doi.org/10.1103/PhysRevPhysEducRes.16.010123
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Holt, Rinehart, and Winston.
Docktor, J. L., Dornfeld, J., Frodermann, E., Heller, K., Hsu, L., Jackson, K. A., Mason, A., Ryan, Q. X., & Yang, J. (2016). Assessing student written problem solutions: A problem-solving rubric with application to introductory physics. Physical Review Physics Education Research, 12(1), 010130. https://doi.org/10.1103/PhysRevPhysEducRes.12.010130
Gurcay, D., & Gulbas, E. (2015). Development of three-tier heat, temperature, and internal energy diagnostic test. Research in Science & Technological Education, 33(2), 197–217. https://doi.org/10.1080/02635143.2015.1018154
Heller, J. I., & Reif, F. (1984). Prescribing effective human problem-solving processes: Problem description in physics. Cognition and Instruction, 1(2), 177–216. https://doi.org/10.1207/s1532690xci0102_2
Heller, P., Keith, R., & Anderson, S. (1992). Teaching problem solving through cooperative grouping. Part 1: Group versus individual problem solving. American Journal of Physics, 60(7), 627–636. https://doi.org/10.1119/1.17117
Kelley, T. L. (1939). The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30(1), 17–24. https://doi.org/10.1037/h0057123
Leinonen, R., Asikainen, M. A., & Hirvonen, P. E. (2013). Overcoming students' misconceptions concerning thermal physics with the aid of hints and peer interaction during a lecture course. Physical Review Special Topics - Physics Education Research, 9(2), 020112. https://doi.org/10.1103/PhysRevSTPER.9.020112
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. https://doi.org/10.1097/00006199-198611000-00017
Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497. https://doi.org/10.1002/nur.20147
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. https://doi.org/10.1002/nur.20199
Taber, K. S. (2018). The use of Cronbach's alpha when developing and reporting research instruments in science education. Research in Science Education, 48, 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach's alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd
Copyright (c) 2026 Ardian Fachreza, Andhy Setiawan, Muslim Muslim
This work is licensed under a Creative Commons Attribution 4.0 International License.
The BIP is under the Creative Commons Attribution 4.0 International License, according to which:
1) Authors retain copyright with the work simultaneously licensed under the Creative Commons Attribution that allows the sharing of articles published with the acknowledgment of authorship and the initial publication in this journal.
2) The authors are authorized to make additional contracts separately for distribution of the version of the work published in this journal (for example, publication in an institutional repository or as a chapter of the book), as long as there is recognition of authorship and initial publication in this journal.
3) Authors are authorized and encouraged to publish and distribute their work online (for example, in institutional repositories or on their personal pages) at any time before or during the editorial process, as it increases the impact and reference of the published work.
