For the power point discussion comparing quantitative and mixed methods research, click here: RSmith – july 10th presentation
This is a comparison of the articles “Assessing the quality of mixed methods research: toward a comprehensive framework” written by Alicia O’Cathain (2015) and “Online self-paced high-school class size and student achievement” written by Chin-His Lin, Jemma Bae Kwon and Yining Zhang (2018).
O’Cathain, A. (2015). Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework. SAGE Handbook of Mixed Methods in Social & Behavioral Research, 531-556. doi:10.4135/9781506335193.n21
Research is an important part of education, as it allows educators and policymakers to make informed decisions about their practices. However, there is much debate as to which methodology yields the most relevant results. Researchers have sets of criteria to evaluate the quality of their research. These criteria are specific to the methodology that they have chosen, for example, quantitative research’s criteria are “validity, reliability, replicability, and generalizability.” (O’Cathain, pg. 4, 2015) Whereas the criteria for qualitative research are “credibility, confirmability, transferability and dependability.” (O’Cathain, pg. 5, 2015) Unfortunately, there has been little around how to assess the quality of mixed methods research.
Through a literary review, Alicia O’Cathain compares previous studies in order to develop a comprehensive framework for assessing the quality of mixed methods research. O’Cathain further discusses how “it is important to assess the quality of mixed method research” in order to “offer guidance to researcher, to establish a common language and provide direction for future development.” (pg. 2, 2015) O’Cathain discusses 8 domains of criteria to help assess the quality of mixed methods research in the “first attempt at a comprehensive framework.” (pg. 24, 2015)
Domains of Assessing Quality of Mixed Method Researcher (found pages 14-22)
1) Planning Quality
2) Design Quality
3) Data Quality
4) Interpretive Rigor
5) Inference Transferability
6) Reporting Quality
7) Synthesizability
8) Utility
This comprehensive framework can be used to “address the needs of the variety of stakeholders who want to assess the quality of mixed methods research.” (O’Cathain, pg. 3, 2015) Furthermore, each stakeholder, including teachers, researchers, participants and policymakers, have “different needs and [are] likely to be interested in different aspects of the quality of mixed methods research.” (O’Cathain, pg. 3, 2015)
It is a valuable skill for teachers and students to be able to evaluate the qualities of different methodological research. Whether it be qualitative, quantitative or mixed methods, it is important that readers assess research to ensure it effectively answers the research question.
Lin, C., Kwon, J. B., & Zhang, Y. (2018). Online self-paced high-school class size and student achievement. Educational Technology Research and Development,67(2), 317-336. doi:10.1007/s11423-018-9614-x
This article is a little more personal for me, as I am currently employed at a distributed learning school, where we teach both fully online courses and blended courses where we see students face-to-face and deliver content online. I found it very interesting that with all the virtual high schools available to students there has been very little research done at the K-12 level.
Dr. Chin-Hsi Lin examines student success rate in relation to class sized in self-paced online courses. Before this article most academic research for online class sizes were conducted at a post-secondary level. Dr. Chin-Hsi Lin discusses the “relationship between class size and student learning outcomes” (Lin, Kwon, and Zhang, pg. 317, 2018) understanding that “small classes are generally perceived as desirable” (Lin et al., pg. 317, 2018) for both teachers and students.
“Online self-paced high-school class size and student achievement” uses quantitative research methodology to assess the relationship between online class size and student success rate. Data was collected from “an accredited state-wide virtual school in the Midwestern U.S.” (Lin et al., pg. 321, 2018), the courses were self-paced, communication with teachers and peers was asynchronous and teachers were provided with “fully designed online courses.” (Lin et al., pg. 321, 2018) Comparing class size with student achievement included the following data:
- Sample size
- 20,540 records
- 12,032 students
- 233 courses
- 6 subjects
- Enrollment Data for students
- Name of the course
- Semester enrolled in
- Grades
- Instructor identifying numbers
It is important to note that class size “was calculated as the sum of the students who had completed [the course], regardless of whether they had passed or failed.” (Lin et al., pg. 323, 2018)
Lin et al. found that class size did have an impact on student achievement, but not in the way that he had expected. Optimal class size was related to subject area and was much higher than previous assumptions indicated. Class size had no significant impact on English, or foreign languages, but did effect math, social studies, science and the arts up to a specific number of students. Lin et al. found that students’ “final grades increased as class size increased up to a maximum [number] of students but decreased if class size rose beyond that point.” (pg. 329, 2018) The average maximum number of students in a class before seeing a decrease in achievement was 35-40.
“Online self-paced high-school class size and student achievement” included some optional survey questions. Students were asked what grade level they were in, and why they had chosen to take an online course, however, only 30% of participants responded. If this study would have been conducted as a qualitative or mixed methods study, it would have allowed researchers insight into the online school. Including students’ perception of online courses, their experience, and how effective they believed the online course was at teaching the content. It also would have allowed researchers to examine how teachers felt about online courses, how they managed the marking and most importantly how frequently they were able to connect with individual students.
Qualitative research may have also addressed how both students and teachers defined success in an online self-pace course. In this study, “learning outcomes were the course grades reported by the virtual school,” and class size was “calculated as the sum of the students who had completed [the course], regardless of whether they had passed or failed.” (Lin et al., pg. 323, 2018) Some would argue that ‘success’ is not accurately represented by a final grade, but rather should include evidence of students’ understanding of content and their ability to communicate their knowledge using various media. One challenge of using the quantitative research methodology for this study was that researchers were not able to accurately assess student characteristics “such as motivation and previously identified abilities … interventions on the part of instructors … and parents.” (Lin et al., pg. 332, 2018) If this study would have included components of qualitative research, they may have been able to address these blind spots.
A shortfall of this study is that the quality of pre-designed courses is unknown to the reader. As all online teachers were provided with a “fully designed online course” (Lin et al., pg. 321, 2018), it is difficult to assess the quality of education student participants were offered. The “lack of responsibility for curriculum development on the part of its teacher … could have reduced their preparation time … enabled them to effectively teach larger classes than might be normally possible.” (Lin et al., pg. 331, 2018) The success of students may have been influence by the quality of course, more so than the class size. Another limitation of this research is the sample size, all data was collected from a single self-paced online school. It would be interesting to complete a cross-examination of various virtual schools to see if the results were consistent. Previous research indicates that “small classes are generally perceived as desirable” (Lin et al., pg. 317, 2018) so it is interesting that this study would have very different results. Lin et al. recommend that “policymakers should be more cognizant of the balance … between maximizing educational access and having small classes.” (pg. 331, 2018)
Online schools and online course availability are becoming more prominent in the education system. “Schools are continuing to expand their online course offerings, both to overcome school-level challenges and to meet student needs.” (Lin et al., pg. 317, 2018) Online courses afford the learners flexibility in and most importantly access to education. Lin et al. recommend that future studies “should aim to disentangle the effects of class size by considering additional factors” including “student satisfaction, and the amount of student-teacher interaction.” (Lin et al., pg. 333, 2018)
References
Lin, C., Kwon, J. B., & Zhang, Y. (2018). Online self-paced high-school class size and student achievement. Educational Technology Research and Development,67(2), 317-336. doi:10.1007/s11423-018-9614-x
O’Cathain, A. (2015). Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework. SAGE Handbook of Mixed Methods in Social & Behavioral Research, 531-556. doi:10.4135/9781506335193.n21
Recent Comments