MTSS Model Evaluation

A mixed methods research design structured the evaluation process using qualitative and quantitative procedures to measure MTSS model implementation and impact. This evaluation design focused on using regression techniques to determine significance of thirteen independent variables across learning behaviors, student attributes, and teaching behaviors on passing Basic and Intermediate Algebra.

Quantitative data were collected daily during each semester of the two year study, including:

  • class attendance records;
  • instructor records of time spent using traditional or explicit instruction methods for each class every day;
  • daily academic progress records from computerized homework assignments and quizzes for each learning outcome required to pass the course; and,
  • math mentor records reporting student time spent in Math Assistance Center each day as well as their classroom observations of instructional methods used when assigned as undergraduate learning assistants in Tier I classes two times a week or in Tier III classes five times a week.

In addition, student attribute measures and mathematics fluency pre-tests were administered at the beginning of each semester while final grades were collected at the end of each term.

Qualitative data were gathered from student and instructor perception surveys administered at the end of each semester. Likewise, instructor interviews were completed at the end of the final semester of the two year project. And lastly, a log was kept to record research and development team meeting discussions and professional development activities throughout the two year project.

Participatory action research is a qualitative research method used to help educators improve teaching and learning practices while contributing to the educational research knowledge base (Coghlan & Brannick, 2009; Holly, Arhar, & Kasten, 2005; McNiff & Whitehead, 2011; Smith & Fernie, 2010; Svensson & Doumas, 2013). While traditional educational research methods focus on explanations and experimentation for universal generalizations, participatory action research drives the professional development process to support student learning (Greenwood, Whyte, & Harkavy, 1993; Hoedebeck, 2011; Kemmis, McTaggart, & Nixon, 2014; McIntyre, 2008; McTaggart, 1991; Reason & Bradbury, 2001). Given that participatory action research is presented as a social research method and process, it is an emergent progression that works effectively to merge individual participation, collegial interaction, and knowledge generation to improve academic achievement (Flood, 2010; Somekh, 2005; Stringer, 2013).

As importantly, the university will continue to monitor progress to degree of the 805 students enrolled in Basic and/or Intermediate Algebra during the two year project for longitudinal evaluation.

Student Demographics

Ethnicity 805 Participants Percent of Total
White/NonHispanic 628 78%
American Indian/Alaskan Native 71 9%
Hispanic/Latino 46 6%
Black or African American 13 2%
Native Hawaiian or other Pacific Islander 2 0%
Two or more races 7 0%
Race/Ethnicity Unknown 6 0%
Unreported 32 4%
Female 487 60%
Male 318 40%
Traditional (<25) 676 84%
Non-traditional (>24) 129 16%

Preliminary results include:

  • Student attribute measures were revised and validated.
  • MTSS model improved student success in remedial mathematics.
  • Student and instructor perceptions confirmed the MTSS model influenced passing Basic and Intermediate Algebra.
  • Fall semester pass rates increased.

Improved Pass Rates

Course Prior 6 Yr Avg. Fall 2009 Fall 2010 Fall 2011 Fall 2012 Fall 2013
Basic Algebra 45% 71% 74% 83% 77% 83%
Intermediate Algebra 50% 69% 67% 72% 83% 78%

Note: Course structural changes started in Fall 2009; MTSS Model implemented in Fall 2010.

Since the beginning of the two year project, dissemination has included press releases and presentations. When a university press release announced the award, the research and development team were invited to present the project to BHSU administrators and the Executive Director of the South Dakota Board of Regents. In addition, the team presented the project during an overview of grants awarded to the university at a campus-wide information session prior to beginning the fall, 2012 term.

During the second year of the project, peer-reviewed presentations at regional, national, and international conferences in mathematics and education shared early model success. Subsequently, research and development team members were invited to discuss possible MTSS model replication at private and state universities in Iowa and Colorado.