Rasch Validation of Instrument Measuring Gen-Z Science, Technology, Engineering, and Mathematics (STEM) Application in Teaching during the Pandemic

Hilman Qudratuddarsi, Riyan Hidayat, Raja Lailatul Zuraida binti Raja Maamor Shah, Nurihan Nasir, Muh Khairul Wajedi Imami, Rusdi bin Mat Nor

Abstract


The impact of the Covid-19 pandemic has had a far-reaching effect on higher education institutions, and individual student assessments have garnered much attention during the pandemic. This study aimed to validate Science, Technology, Engineering, and Mathematics (STEM) application instruments using the Rasch analysis employing Winsteps version 3.73. A survey was conducted with 201 respondents from two provinces in Indonesia. The students were selected by convenience sampling and answered the adopted STEM application instrument. The STEM application instruments were adapted, and these were divided into seven sub-constructs derived from STEM disciplines. Rasch Modelling was employed for data analysis using Winsteps version 3.7.3 to analyse reliability, separation, item fit statistics, unidimensionality, and rating scale calibration. Each sub-construct fulfilled a minimum of 0.65 for Cronbach alpha, item, and person reliability, and most of them had more than 1.5 person and item separation. In general, each item had a good score of the mean square, Z-tolerated standard, and point measure correlation, indicating fulfilment of the Rasch measurement model. The analysis also showed unidimensionality assumption and an excellent rating scale. This study contributed to the body of STEM knowledge by using Rasch Modelling to test the validity and reliability of STEM application instruments. 

https://doi.org/10.26803/ijlter.21.6.7


Keywords


COVID-19 pandemic; Gen-Z; STEM education; higher education; Rasch model

Full Text:

PDF

References


Abbitt, J.T., & Boone, W.J. (2021). Gaining insight from survey data: An analysis of the community of inquiry survey using Rasch measurement techniques. Journal of Computing in Higher Education, 33, 367–397. https://doi.org/10.1007/s12528-020-09268-6

Adams, D., Chuah, K. M., Sumintono, B., & Mohamed, A. (2021). Students’ readiness for e-learning during the COVID-19 pandemic in a South-East Asian university: A Rasch analysis pandemic. Asian Education and Development Studies, 11(2), 324–339 https://doi.org/10.1108/AEDS-05-2020-0100

Andrich, D. (1982). An index of person separation in latent trait theory, the traditional KR. 20 index, and the Guttman scale response pattern. Education Research and Perspectives, 9(1), 95–104. https://search.informit.org/doi/abs/10.3316/aeipt.13636

Alkhadim, G.S., Cimetta, A.D., Marx, R.W., Cutshaw, C.A., & Yaden, D.B. (2021). Studies in educational evaluation validating the research-based early math assessment (REMA) among rural children in Southwest United States. Studies in Educational Evaluation, 68. https://doi.org/10.1016/j.stueduc.2020.100944

Alnahdi, A.H. (2018). Rasch validation of the Arabic version of the lower extremity functional scale. Disability & Rehabilitation. 40, 353–359. https://doi.org/10.1080/09638288.2016.1254285

Anderson, E.W., & Mittal, V. (2000). Strengthening the satisfaction-profit chain. Journal of Service Research, 3(2), 107–120. https://doi.org/10.1177/109467050032001

Aryadoust, V., Ng, L.Y., & Sayama, H. (2020). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing, 38(1), 6–40. https://doi.org/10.1177/0265532220927487

Aykan, A., & Y?ld?r?m, B. (2022). The integration of a lesson study model into distance STEM education during the covid-19 pandemic: Teachers’ views and practice. Technology, Knowledge and Learning, 27(2), 609–637. https://doi.org/10.1007/s10758-021-09564-9

Aziz, F., Anom, M., Rashid, A., Noor, M., Othman, A., & Ismail, I.R. (2021). Factor influencing Gen-Z preferred working environment in Malaysia. Turkish Journal of Computer and Mathematics Education, 12(7), 2727–2733. https://doi.org/10.1186/s12913-020-05114-8

Badmus, O., & Omosewo, E.O. (2018). Evolution of STEM, STEAM and STREAM education in Africa: The implication of the knowledge gap. International Journal of Research in STEM Education (IJRSE), 2 (2), 99–106. https://doi.org/10.31098/ijrse.v2i2.227

Bakker, A., Cai, J., & Zenger, L. (2021). Future themes of mathematics education research: An international survey before and during the pandemic. Educational Studies in Mathematics, 107(1), 1–24. https://doi.org/10.1007/s10649-021-10049-w

Bond, T.G., & Fox, C.M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge.

Boone, W., Staver, J., & Yale, M.S. (2014). Rasch Analysis in the Human Sciences. Springer.

Boone, W.J., & Noltemeyer, A. (2017). Rasch analysis: A primer for school psychology researchers and practitioners. Cogent Education, 4(1), 1416898. https://doi.org/10.1080/2331186X.2017.1416898

Bradley, K.D., Peabody, M.R., Akers, K.S., & Knutson, N. (2015). Rating scales in survey research: Using the Rasch model to illustrate the middle category measurement flaw. Survey Practice, 8(2). https://doi.org/10.29115/SP-2015-0001.

Bravini, E., Giordano, A., Sartorio, F., Ferriero, G., & Vercelli, S. (2016). Rasch analysis of the Italian lower extremity functional scale: Insights on dimensionality and suggestions for an improved 15-item version. Clinical Rehabilitation, 31(4), 532–543. https://doi.org/10.1177/0269215516647180

Carlisle, D.L., & Weaver, G.C. (2018). STEM education centers: Catalyzing the improvement of undergraduate STEM education. International Journal of STEM Education, 5(1), 1–21. https://doi.org/10.1186/s40594-018-0143-2

Chang, C.C., Su, J.A., & Lin, C.Y. (2016). Using the Affiliate Stigma Scale with caregivers of people with dementia: Psychometric evaluation. Alzheimer's Research & Therapy, 8(1), 1–8. https://doi.org/10.1186/s13195-016-0213-y

Chen, W.H., Lenderking, W., Jin, Y., Wyrwich, K.W., Gelhorn, H., & Revicki, D.A. (2014). Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data. Quality of Life Research, 23(2), 485–493. https://doi.org/10.1007/s11136-013-0487-5

Chicca, J., & Shellenbarger, T. (2018). Connecting with generation Z: Approaches in nursing education. Teaching and Learning in Nursing, 13, 180–184. https://doi.org/10.1016/j.teln.2018.03.008.

Chua, Y.P. (2020). Mastering Research Methods (3rd Ed.). McGraw-Hill.

Cilliers, E. J. (2017). The Challenges of teaching generation Z. International Journal of Social Science, 3(1), 188-198. DOI-https://dx.doi.org/10.20319/pijss.2017.31.188198.

Clarke, D.J. (2013). Contingent conceptions of accomplished practice: The cultural specificity of discourse in and about the mathematics classroom. ZDM Mathematics Education, 45, 21–33. https://doi.org/10.1007/s11858-012-0452-8

Cole, D., Kitchen, J.A., & Kezar, A. (2019). Examining a comprehensive college transition program: An account of iterative mixed methods longitudinal survey design. Research in Higher Education, 60(3), 392–413. https://doi.org/10.1007/s11162-018-9515-1

Creswell, J.W. (2012). Educational Research: Planning, conducting, and evaluating quantitative and qualitative research. Educational Research (Vol. 4). Pearson.

Creswell, J.W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE Publications, Inc.

Csobanka, Z.E. (2016). The Z generation. Acta Technologica Dubnicae. 6(2), 63–76. https://doi.org/10.1515/atd-2016-0012.

DiStefano, C., Greer, F. W., & Dowdy, E. (2019). Examining the BASC-3 BESS parent form–preschool using rasch methodology. Assessment, 26(6), 1162–1175. https://doi.org/10.1177/1073191117723112

English, L.D. (2016). STEM education K-12: Perspectives on integration. International Journal of STEM Education, 3(1), 1–8. https://doi.org/10.1186/s40594-016-0036-1

Garry, F., Hatzigianni, M., Bower, M., Forbes, A., & Stevenson, M. (2020). Understanding K-12 STEM education: A framework for developing STEM literacy. Journal of Science Education and Technology, 29(3), 369–385. https://doi.org/10.1007/s10956-020-09823-x

Geng, J., Jong, M. S. Y., & Chai, C. S. (2019). Hong Kong teachers’ self-efficacy and concerns about STEM education. Asia-Pacific Education Researcher, 28(1), 35–45. https://doi.org/10.1007/s40299-018-0414-1

Gocen, A., & Sen, S. (2021). A validation of servant leadership scale on multinational sample. Psychological Reports, 124(2), 752–770. https://doi.org/10.1177/0033294120957246

Gomez, A., & Albrecht, B. (2013). True STEM education. Technology and Engineering Teacher. https://www.thefreelibrary.com/True+STEM+education.-a0353994952

Hidayat, R., Habibi, A., Mohd Saad, M.R., Mukminin, A., & Wan Idris, W.I.B. (2018). Exploratory and confirmatory factor analysis of PERMA for Indonesian students in mathematics education programmes. Pedagogika, 132(4), 147–165. https://doi.org/10.15823/p.2018.132.9

Hidayat, R., Idris, W.I.W., Qudratuddarsi, H., & Rahman, M.N.A. (2021). Validation of the Mathematical modeling attitude scale for Malaysian mathematics teachers. Eurasia Journal of Mathematics, Science and Technology Education, 17(12), 1-–6. https://doi.org/10.29333/ejmste/11375

Hidayat, R., Qudratuddarsi, H., Mazlan, N.H., & Zeki, M.Z.M. (2021). Evaluation of a test measuring mathematical modelling competency for indonesian college students. Journal of Nusantara Studies (JONUS), 6(2), 133–155. https://doi.org/10.24200/jonus.vol6iss2pp133-155

Hsiao, P.W., & Su, C.H. (2021). A study on the impact of STEAM education for sustainable development courses and its effects on student motivation and learning. Sustainability 2021, 13(7), 3772. https://doi.org/10.3390/su13073772

Iseppi, L., Rizzo, M., Gori, E., Nassivera, F., Bassi, I., & Scuderi, A. (2021). Rasch model for assessing propensity to entomophagy. Sustainability, 13(8). https://doi.org/10.3390/su13084346

Ishak, A.H., Osman, M.R., Manan, S.K.A., & Saidon, R. (2016). Rasch model scale calibration analysis for Islamic value. International Review of Management and Marketing, 6(7), 11–16. https://www.econjournals.com/index.php/irmm/article/view/3164

Ishak, A.H., Osman, M.R., Mahaiyadin, M.H., Tumiran, M.A., & Anas, N. (2018). Examining unidimensionality of psychometric properties via rasch model. International Journal of Civil Engineering and Technology, 9(9), 1462–1467. http://iaeme.com/Home/issue/IJCIET?Volume=9&Issue=9

Jin, Y., Rodriguez, C.A., Shah, L., & Rushton, G.T. (2020). Examining the psychometric properties of the redox concept inventory: A Rasch approach. Journal of Chemical Education, 97(12), 4235–4244. https://doi.org/10.1021/acs.jchemed.0c00479

Kang, N.H. (2019). A review of the effect of integrated STEM or STEAM (Science, Technology, Engineering, Arts, And Mathematics) education in South Korea. Asia-Pacific Science Education, 5(1), 1–22. https://doi.org/10.1186/s41029-019-0034-y

Kelley, T.R., & Knowles, J.G. (2016). A conceptual framework for integrated STEM education. International Journal of STEM Education, 3(1), 1–11. https://doi.org/10.1186/s40594-016-0046-z

Kim, B.H., & Kim, J. (2016). Development and validation of evaluation indicators for teaching competency in STEAM education in Korea. Eurasia Journal of Mathematics, Science and Technology Education, 12(7), 1909–1924. https://doi.org/ 10.12973/eurasia.2016.1537a

Li, Y., Wang, K., Xiao, Y., & Froyd, J.E. (2020). Research and trends in STEM education: a systematic review of journal publications. International Journal of STEM Education, 7(1), 1–16. https://doi.org/10.1186/s40594-020-00207-6

Li, Y., Wang, K., Xiao, Y., Froyd, J.E., & Nite, S.B. (2020). Research and trends in STEM education: A systematic analysis of publicly funded projects. International Journal of STEM Education, 7(1), 1–17. https://doi.org/10.1186/s40594-020-00213-8#Sec20

Linacre, J.M. (1999). Investigating rating scale category utility. Journal of Outcome Measurement, 3(2), 103–122. https://pubmed.ncbi.nlm.nih.gov/10204322/

Linacre J.M. (2017). Winsteps® Rasch measurement computer program. Winsteps.com, Beaverton. https://www.winsteps.com/index.htm

Linacre, J.M. (2018). Winsteps® Rasch measurement computer program User’s Guide. Winsteps.com, Beaverton. https://www.winsteps.com/index.htm

Linacre, J.M. (2020). Winsteps® (version 4.5.2) [Computer Software]. Winsteps.com. Retrieved April 28, 2020. https://www.winsteps.com/

Maarouf, S.A. (2019). Supporting academic growth of English language learners: Integrating reading into STEM curriculum. World Journal of Education, 9(4), 83–96. https://doi.org/10.5430/wje.v9n4p83

Meeth., L.R. (1978). Interdisciplinary studies: A matter of definition. Change, 7(10), 10. https://doi.org/10.1080/00091383.1978.10569474

Mohd Shahali, E.H., Halim, L., Rasul, M.S., Osman, K., & Mohamad Arsad, N. (2019). Students’ interest towards STEM: A longitudinal study. Research in Science & Technological Education, 37(1), 71–89. https://doi.org/10.1080/02635143.2018.1489789

Nuangchalerm, P., Prachagool, V., Prommaboon, T., Juhji, J., Imroatun, I., & Khaeroni, K. (2020). Views of primary Thai teachers toward STREAM education. International Journal of Evaluation and Research in Education, 9(4), 987–992. https://doi.org/10.11591/ijere.v9i4.20595

Ozkan, G., & Umdu Topsakal, U. (2020). Investigating the effectiveness of STEAM education on students’ conceptual understanding of force and energy topics. Research in Science & Technological Education, 1–20. https://doi.org/10.1080/02635143.2020.1769586

Padhmasari, T. (2016). Pembelajaran PAI dalam kurikulum semesta: Studi kasus di SMA Trensains Tebuireng Jombang [The Islamic education of universalism curriculum: Case study in Trensains senior high school of Tebuireng Jombang]. [Master’s Thesis, Maulana Malik Ibrahim State Islamic University]. http://etheses.uin-malang.ac.id/id/eprint/6229

Parmin, P., Saregar, A., Deta, U.A., & El Islami, R.A.Z. (2020). Indonesian science teachers’ views on attitude, knowledge, and application of STEM. Journal for the Education of Gifted Young Scientists, 8(1), 17–31. https://doi.org/10.17478/jegys.647070

Perignat, E., & Katz-Buonincontro, J. (2019). STEAM in practice and research: an integrative literature review. Thinking Skills and Creativity, 31, 31–43. https://doi.org/10.1016/j.tsc.2018.10.002

Peters-Burton, E.E., Lynch, S.J., Behrend, T.S., & Means, B.B. (2014). Inclusive STEM high school design: 10 critical components. Theory Into Practice, 53(1), 64–71. https://doi.org/10.1080/00405841.2014.862125.

Poláková, P., & Klímová, B. (2019). Mobile technology and generation Z in the English language classroom–A preliminary study. Education Sciences, 9(3), 1–11. https://doi.org/10.3390/educsci9030203

Purnami, W., Sumintono, B., & Wahyu, Y. (2021). Investigation of person ability and item fit instruments of eco-critical thinking skills in basic science concept materials for elementary pre-service teachers. Jurnal Pendidikan IPA Indonesia, 10(1), 127–137. https://doi.org/10.15294/jpii.v10i1.25239

Rahayu, W., Putra, M.D.K., Iriyadi, D., Rahmawati, Y., & Koul, R.B. (2020). A Rasch and factor analysis of an Indonesian version of the student perception of opportunity competence development (SPOCD) questionnaire. Cogent Education, 7(1). https://doi.org/10.1080/2331186X.2020.1721633

Rahayu, W., Putra, M.D.K., Rahmawati, Y., Hayat, B., & Koul, R.B. (2021). Validating an Indonesian version of the what is happening in this class? (which) questionnaire using a multidimensional rasch model. International Journal of Instruction, 14(2), 919–934. https://doi.org/10.29333/iji.2021.14252a

Rasch, G. (1960a). Studies in Mathematical Psychology: I. Probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.

Rasch. G. (1960b). Probabilistic Models for Some Intelligence and Achievement Tests. Danish Institute for Educational Research.

Salzman, H., & Benderly, B.L. (2019). STEM performance and supply: Assessing the evidence for education policy. Journal of Science Education and Technology, 28(1), 9–25. https://doi.org/10.1007/s10956-018-9758-9

Sanders, M. (2009). STEM, STEM education, STEMmania. The Technology Teacher, 68(4), 20–26. https://www.researchgate.net/publication/237748408_STEM_STEM_education_STEMmania

Scoulas, J.M., Aksu Dunya, B., & De Groote, S.L. (2021). Validating students’ library experience survey using rasch model. Library and Information Science Research, 43(1), 101071. https://doi.org/10.1016/j.lisr.2021.101071

Sen, S., & Gocen, A. (2021). A psychometric evaluation of the ethical leadership scale using Rasch analysis and confirmatory factor analysis. Journal of General Psychology, 148(1), 84–104. https://doi.org/10.1080/00221309.2020.1834346

Shanahan, M-C., Burke, L.E., & Francis, K. (2016). Using a boundary object perspective to reconsider the meaning of STEM in a Canadian context. Canadian Journal of Science, Mathematics and Technology Education, 6(2), 129–139. https://doi.org/10.1080/14926156.2016.1166296

Sing, A.P., & Dangmei, J. (2016). Understanding the generation Z: The future workforce. South- Asian journal of multidisciplinary studies, 3(3), 1–5. https://www.researchgate.net/publication/305280948_UNDERSTANDING_THE_GENERATION_Z_THE_FUTURE_WORKFORCE

STEM Task Force Report. (2014). Innovate: A blueprint for science, technology, engineering, and mathematics in California public education. Californians Dedicated to Education Foundation. https://www.cde.ca.gov/pd/ca/sc/documents/innovate.pdf

Stohlmann, M., Moore, T.J., & Roehrig, G.H. (2012). Considerations for teaching integrated STEM education. Journal of Pre-College Engineering Education Research, 2(1), 28–34. https://doi.org/10.5703/1288284314653.

Suryadi, B., Hayat, B., Dwirifqi, M., & Putra, K. (2021). The Indonesian version of the Life Orientation Test-Revised (LOT-R): Psychometric properties based on the Rasch model. Cogent Psychology, 8(1). https://doi.org/10.1080/23311908.2020.1869375

Szymkowiak, A., Melovi?, B., Dabi?, M., Jeganathan, K., & Kundi, G.S. (2021). Information technology and Gen-Z: The role of teachers, the internet, and technology in the education of young people. Technology in Society, 65. https://doi.org/10.1016/j.techsoc.2021.101565

Talmon, G.A. (2019). Generation Z: What’s next? Medical Science Educator. 29, 9–11. https://doi.org/10.1007/s40670-019-00796-0

Tennant, A., & Conaghan, P.G. (2007). The Rasch measurement model in rheumatology: What is it and why use it ? When should it be applied , and what should one look for in a Rasch paper ? Arthritis & Rheumatism, 57(8), 1358–1362. https://doi.org/10.1002/art.23108

Timms, M.J., Moyle, K., Weldon, P.R., & Mitchell, P. (2018). Challenges in STEM Learning in Australian Schools: Literature and policy review. Australian Council for Educational Research

Togou, M.A., Lorenzo, C., Cornetta, G., & Muntean, G.M. (2020). Assessing the effectiveness of using fab lab-based learning in schools on K–12 students’ attitude toward STEAM. IEEE Transactions on Education, 63(1), 56–62. https://doi.org/10.1109/TE.2019.2957711

Suwono, H., Fachrunnisa, R., Yuenyong, C., & Hapsari, L. (2019). Indonesian students’ attitude and interest in STEM: An outlook on the gender stereotypes in the STEM field. Journal of Physics: Conference Series, 1340(1), 1-7. https://doi.org/10.1088/1742–6596/1340/1/012079

Van Zile-tamsen, C. (2019). Using Rasch analysis to inform rating scale development. Research in Higher Education, 58(8), 922–933.

Wahono, B., & Chang, C.Y. (2019a). Assessing teacher’s attitude, knowledge, and application (AKA) on STEM: An effort to foster the sustainable development of stem education. Sustainability, 11(4). https://doi.org/10.3390/su11040950

Wahono, B., & Chang, C.Y. (2019b). Development and validation of a survey instrument (AKA) towards attitude, knowledge and application of STEM. Journal of Baltic Science Education, 18(1), 63–76. https://doi.org/10.33225/jbse/19.18.63

Wandari, G.A., Wijaya, A.F.C., & Agustin, R.R. (2018). The effect of STEAM-based learning on students' concept mastery and creativity in learning light and optics. Journal of Science Learning, 2(1), 26–32. https://doi.org/10.17509/jsl.v2i1.12878

Widhiarso, W., & Sumintono, B. (2016). Examining response aberrance as a cause of outliers in statistical analysis. Personality and Individual Differences, 98, 11–15. https://doi.org/10.1016/j.paid.2016.03.099

Yakman, G., & Lee, H. (2012). Exploring the exemplary STEAM education in the US as a practical educational framework for Korea. Journal of the Korean Association for Science Education, 32(6), 1072–1086. https://doi.org/10.14697/jkase.2012.32.6.1072

Zouda, M. (2018). Issues of power and control in STEM education: A reading through the postmodern condition. Cultural Studies of Science Education, 13(4), 1109–1128. https://doi.org/10.1007/s11422-017-9820-6


Refbacks

  • There are currently no refbacks.


e-ISSN: 1694-2116

p-ISSN: 1694-2493