Development of a diagnostic assessment test to evaluate science misconceptions in terms of school grades: A rasch measurement approach

Soeharto Soeharto

doi:10.36681/

Authors

Soeharto Soeharto University of Szeged https://orcid.org/0000-0003-4332-7401

DOI:

https://doi.org/10.36681/

Keywords:

Rasch measurement, diagnostic assessment, student misconceptions

Abstract

This study aims to evaluate the psychometric properties of the developed diagnostic assessment test and to identify student misconceptions in science in terms of school grades. 153 students were gathered by using random sample from 10th to 12th grade in senior high schools. The 32 items of the two-tier multiple-choice diagnostic test were administered to assess student misconceptions in science using the online system (eDia) and paper-based test. The results confirmed the validity and reliability of the developed test based on Rasch measurement. Student misconceptions in science were found statistically significant among school grades, [F(2, 152) = 10.93, p <.01]. The 12th-grade students have higher misconceptions than the students at 10th- and 11th-grade. No statistically signiﬁcant diﬀerence was found between boys and girls for all grade level (p> 0.05). The Stepwise multiple regression confirmed that the grades are the predictor of student misconception in science, [F(152) = 10.208, p <0.01], explaining 25,2% variances of student misconception in science. This study gave preliminary evidence that the developed test well measured student misconceptions and evaluated students’ misconception in science concepts.

Downloads

Download data is not yet available.

References

AAAS. (2019). Project 2061 | American Association for the Advancement of Science. https://www.aaas.org/programs/project-2061

Adadan, E., Savasci, F., & Martin, R. E. (2012). An analysis of 16–17-year-old students’ understanding of solution chemistry concepts using a two-tier diagnostic instrument. International Journal of Science Education, 34(4), 513–544.

Adams, D., Joo, M. T. H., Sumintono, B., & Oh, S. P. (2020). Blended Learning Engagement in Higher Education Institutions: A Differential Item Functioning Analysis of Students’ Backgrounds. Malaysian Journal of Learning and Instruction, 17(1), 133–158.

Allen, M. (2014). Misconceptions in primary science. McGraw-hill education.

Andrich, D. (2018). Advances in social measurement: A Rasch measurement theory. Perceived Health and Adaptation in Chronic Disease, 66–91.

Arslan, H. O., Cigdemoglu, C., & Moseley, C. (2012). A Three-Tier Diagnostic Test to Assess Pre-Service Teachers’ Misconceptions about Global Warming, Greenhouse Effect, Ozone Layer Depletion, and Acid Rain. International Journal of Science Education, 34(11), 1667–1686. https://doi.org/10.1080/09500693.2012.680618

Barbic, S. P., & Cano, S. J. (2016). The application of Rasch measurement theory to psychiatric clinical outcomes research: Commentary on… Screening for depression in primary care. BJPsych Bulletin, 40(5), 243–244.

Bond, T. G., & Fox, C. M. (2007). Rasch modeling applied: rating scale design. Applying the Rasch Model: Fundamental Measurement in the Human Sciences (2nd Ed., Pp. 219–233). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.

Boone, W. J., Staver, J. R., & Yale, M. S. (2013). Rasch analysis in the human sciences. Springer.

Boone, W. J., Townsend, J. S., & Staver, J. R. (2016). Utilizing multifaceted Rasch measurement through FACETS to evaluate science education data sets composed of judges, respondents, and rating scale items: An exemplar utilizing the elementary science teaching analysis matrix instrument. Science Education, 100(2), 221–238.

Bradley, K. D., Peabody, M. R., Akers, K. S., & Knutson, N. (2015). Rating Scales in Survey Research: Using the Rasch model to illustrate the middle category measurement flaw. Survey Practice, 8(2).

Butler, J., Mooney Simmie, G., & O’Grady, A. (2015). An investigation into the prevalence of ecological misconceptions in upper secondary students and implications for pre-service teacher education. European Journal of Teacher Education, 38(3), 300–319. https://doi.org/10.1080/02619768.2014.943394

Chou, Y.-T., & Wang, W.-C. (2010). Checking dimensionality in item response models with principal component analysis on standardized residuals. Educational and Psychological Measurement, 70(5), 717–731.

Christensen, K. B., Makransky, G., & Horton, M. (2017). Critical values for Yen’s Q 3: Identification of local dependence in the Rasch model using residual correlations. Applied Psychological Measurement, 41(3), 178–194.

Csapó, B. (1998). Iskolai tudas. Osiris Kiadó.

Csapó, Beno, & Molnár, G. (2019). Online diagnostic assessment in support of personalized teaching and learning: The eDia system. Frontiers in Psychology, 10(JULY). https://doi.org/10.3389/fpsyg.2019.01522

Driver, R., & Easley, J. (1978). Pupils and Paradigms: a Review of Literature Related to Concept Development in Adolescent Science Students. Studies in Science Education, 5(1), 61–84. https://doi.org/10.1080/03057267808559857

Engelhard Jr, G. (2013). Invariant measurement: Using Rasch models in the social, behavioral, and health sciences. Routledge.

Erman, E. (2017). Factors contributing to students’ misconceptions in learning covalent bonds. Journal of Research in Science Teaching, 54(4), 520–537. https://doi.org/10.1002/tea.21375

Eshach, H., Lin, T., & Tsai, C. (2018). Misconception of sound and conceptual change: A cross‐sectional study on students’ materialistic thinking of sound. Journal of Research in Science Teaching, 55(5), 664–684.

Fajarini, F., Utari, S., & Prima, E. C. (2018). Identification of students’ misconception against global warming concept. International Conference on Mathematics and Science Education of Universitas Pendidikan Indonesia, 3, 199–204.

Fariyani, Q., Rusilowati, A., & Sugianto, S. (2017). Four-tier diagnostic test to identify misconceptions in geometrical optics. Unnes Science Education Journal, 6(3).

Fisher, W. P. J. (2007). Rating Scale Instrument Quality Criteria. Rasch Measurement Transactions, 21(1), 1095. http://www.rasch.org/rmt/rmt211m.htm

Galvin, E., & Mooney, S. G. (2015). Identification of Misconceptions in the Teaching of Biology: A Pedagogical Cycle of Recognition, Reduction and Removal. Higher Education of Social Science, 8(2), 1–8. https://doi.org/10.3968/6519

Greiff, S., Molnár, G., Martin, R., Zimmermann, J., & Csapó, B. (2018). Students’ exploration strategies in computer-simulated complex problem environments: A latent class approach. Computers & Education, 126, 248–263.

Griﬃn, P. (2010). Item response modelling: An introduction to the Rasch model. Assessment Research Centre Faculty of Education, The University of Melbourne.

Gurel, D. K., Eryilmaz, A., & McDermott, L. C. (2015). A review and comparison of diagnostic instruments to identify students’ misconceptions in science. Eurasia Journal of Mathematics, Science and Technology Education, 11(5), 989–1008. https://doi.org/10.12973/eurasia.2015.1369a

Hagell, P. (2014). Testing rating scale unidimensionality using the principal component analysis (PCA)/t-test protocol with the Rasch model: the primacy of theory over statistics. Open Journal of Statistics, 4(6), 456–465.

HAKİM, A., KADAROHMAN, A., & SYAH, Y. M. (2016). Effects of the natural product mini project laboratory on the students conceptual understanding. Journal of Turkish Science Education, 13(2), 27–36.

IBM SPSS. (2017). IBM SPSS Statistics for Windows, version 25. In Armonk, NY: IBM SPSS Corp.[Google Scholar].

Keeley, P. (2012). Misunderstanding misconceptions. Science Scope, 35(8), 12. https://www.questia.com/library/journal/1G1-294903479/misunderstanding-misconceptions

Khine, M. S. (2020). Rasch Measurement. In Rasch Measurement. https://doi.org/10.1007/978-981-15-1800-3

Kiray, S. A., Aktan, F., Kaynar, H., Kilinc, S., & Gorkemli, T. (2015). A descriptive study of pre-service science teachers’ misconceptions about sinking-floating. Asia-Pacific Forum on Science Learning & Teaching, 16(2).

Kirbulut, Z. D., & Geban, O. (2014). Using three-tier diagnostic test to assess students’ misconceptions of states of matter. Eurasia Journal of Mathematics, Science and Technology Education, 10(5), 509–521. https://doi.org/10.12973/eurasia.2014.1128a

Korkmaz, S. D., Bahadir, A., Aybek, E. C., & Suat, P. A. T. (2018). Evaluating the gifted students’ understanding related to plasma state using plasma experimental system and two-tier diagnostic test. Journal of Education in Science Environment and Health, 4(1), 46–53.

Köse, S. (2004). Effectiveness of conceptual change texts accompanied with concept mapping instructions on overcoming prospective science teachers’ misconceptions of photosynthesis and respiration in plants. Published Ph. D., Karadeniz Technical University, Institute of Natural and Applied Sciences, Trabzon.

Laliyo, L. A. R., Botutihe, D. N., & Panigoro, C. (2019). The development of two-tier instrument based on distractor to assess conceptual understanding level and student misconceptions in explaining redox reactions. International Journal of Learning, Teaching and Educational Research, 18(9), 216–237.

Leaper, C., Farkas, T., & Brown, C. S. (2012). Adolescent girls’ experiences and gender-related beliefs in relation to their motivation in math/science and English. Journal of Youth and Adolescence, 41(3), 268–282.

Linacre, J. M. (1998). Detecting multidimensionality: which residual data-type works best? Journal of Outcome Measurement, 2, 266–283.

Linacre, J. M. (2020). Winsteps® (Version 4.7.0) [Computer Software]. (4.7.0). Winsteps.com. https://www.winsteps.com/

Liu, X. (2007). Elementary to high school students’ growth over an academic year in understanding concepts of matter. Journal of Chemical Education, 84(11), 1853.

Martin, R. E. (2005). Teaching science for all children: Inquiry methods for constructing understanding. Allyn & Bacon.

Morais, M. de F. (2013). Creativity: Challenges to a key concept for the XXI century.

Morrison, G. R., Ross, S. J., Morrison, J. R., & Kalman, H. K. (2019). Designing effective instruction. John Wiley & Sons.

Murdoch, J. (2018). Our preconceived notions of play need to challenging. Early Years Educator, 19(9), 22–24. https://repositorium.sdum.uminho.pt/handle/1822/26465

OECD. (2016). PISA 2015 results (Volume I): Excellence and equity in education. https://doi.org/10.1787/9789264266490-en

OECD. (2020). Science performance (PISA) (indicator). https://doi.org/doi: 10.1787/91952204-en

Park, M., & Liu, X. (2019). An Investigation of Item Difficulties in Energy Aspects Across Biology, Chemistry, Environmental Science, and Physics. Research in Science Education. https://doi.org/10.1007/s11165-019-9819-y

Peşman, H., & Eryılmaz, A. (2010). Development of a three-tier test to assess misconceptions about simple electric circuits. The Journal of Educational Research, 103(3), 208–222.

Planinic, M., Boone, W. J., Susac, A., & Ivanjek, L. (2019). Rasch analysis in physics education research: Why measurement matters. Physical Review Physics Education Research, 15(2), 1–14. https://doi.org/10.1103/PhysRevPhysEducRes.15.020111

Ratnasari, D., & Suparmi, S. (2017). Effect of problem type toward students’ conceptual understanding level on heat and temperature. Journal of Physics: Conference Series, 909(1), 12054.

Samsudin, A., Afif, N. F., Nugraha, M. G., Suhandi, A., Fratiwi, N. J., Aminudin, A. H., Adimayuda, R., Linuwih, S., & Costu, B. (2021). Reconstructing Students’ Misconceptions on Work and Energy through the PDEODE* E Tasks with Think-Pair-Share. Journal of Turkish Science Education, 18(1), 118–144.

Soeharto, Csapó, B., Sarimanah, E., Dewi, F. I., & Sabri, T. (2019). A review of students’ common misconceptions in science and their diagnostic assessment tools. Jurnal Pendidikan IPA Indonesia, 8(2), 247–266. https://doi.org/10.15294/jpii.v8i2.18649

Soeharto, S. (2016). Implementation of Text Transformation in Physics Education to Reduce Students’ Misconception. JETL (Journal Of Education, Teaching and Learning), 1(2), 56. https://doi.org/10.26737/jetl.v1i2.38

Stefanidou, C. G., Tsalapati, K. D., Ferentinou, A. M., & Skordoulis, C. D. (2019). Conceptual Difficulties Pre-Service Primary Teachers Have with Static Electricity. Journal of Baltic Science Education, 18(2), 300.

Sumintono, B., & Widhiarso, W. (2014). Aplikasi model Rasch untuk penelitian ilmu-ilmu sosial (edisi revisi). Trim Komunikata Publishing House.

Taber, K. S. (2018). The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science Education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2

Taslidere, E. (2016). Development and use of a three-tier diagnostic test to assess high school students’ misconceptions about the photoelectric effect. Research in Science & Technological Education, 34(2), 164–186.

TOPALSAN, A. K., & BAYRAM, H. (2019). Identifying Prospective Primary School Teachers’ Ontologically Categorized Misconceptions on the Topic of" Force and Motion". Journal of Turkish Science Education, 16(1), 85–109.

Treagust, D. F. (1988). Development and use of diagnostic tests to evaluate students’ misconceptions in science. International Journal of Science Education, 10(2), 159–169.

Tsui, C., & Treagust, D. (2010). Evaluating secondary students’ scientific reasoning in genetics using a two‐tier diagnostic instrument. International Journal of Science Education, 32(8), 1073–1098.

Wandersee, J. H., Mintzes, J. J., & Novak, J. D. (1994). Research on alternative conceptions in science. Handbook of Research on Science Teaching and Learning, 177, 210.

Zlatkin-Troitschanskaia, O., Shavelson, R. J., & Kuhn, C. (2015). The international state of research on measurement of competency in higher education. Studies in Higher Education, 40(3), 393–411.

Zwick, R., Thayer, D. T., & Lewis, C. (1999). An empirical Bayes approach to Mantel‐Haenszel DIF analysis. Journal of Educational Measurement, 36(1), 1–28.