A mixed methods research design was utilized to evaluate effectiveness of this post-secondary MTSS model. This evaluation framework followed Boylan and Bonham’s (2012) industry standards for effective remedial programs and the Education Commission of the States (2011) guidelines for Accountability and Continuous Improvement in Remedial Education.
Descriptive and inferential statistics were used to account for variance in achieving high stakes outcomes (i.e., passing the developmental course, passing college algebra, passing the Collegiate Assessment of Academic Proficiency, and matriculating to degree). In addition, qualitative research methods were incorporated throughout the two year project to measure on-going student, instructor and math mentor response to development, implementation, and evaluation of the post-secondary MTSS model.
Boylan and Bonham proposed using descriptive statistics to direct the quantitative evaluation process in measuring effectiveness of developmental education by asking the following questions:
- How many students participated in the program/courses?
- How many hours of tutoring were offered?
- How many sections of developmental courses were offered?
- What % of the students who entered the course stayed for the entire term?
- What % of those who stayed the entire term earned a C or better?
- What were the g-scores for those taking the course or receiving tutoring?
- How many of those who participated in the course/program remained for one semester?
- What % of those who passed the lowest level developmental course took and passed the next level developmental course?
- What % of those who passed the highest level developmental course took and passed the next level curriculum course in that subject?
- What % of those who took one or more developmental courses were retained from fall to fall?
- What % of those who took one or more developmental courses graduated within 2,3,4,5,6 years?
In addition, they recommended the following four questions guide the qualitative evaluation process:
- To what extent are student users satisfied with the program?
- What are faculty/staff perceptions of the program?
- What are faculty/staff perceptions of the program's students?
- What is the impact of the program on the campus as a whole?
To answer such questions, our mixed methods evaluation design used multiple regression techniques to determine the relative contribution of the independent variables in accounting for variance in achieving high stakes outcomes for students in this developmental math study (Cohen, 1988; Field, 2010; Fox, 2008; Warner, 2008). Likewise, qualitative methods were used to gather perception data from students, instructors, and math mentors four times during the two year project. Their initial input and on-going feedback shaped a continuous improvement process for professional development training with the instructors and math mentors to implement and evaluate the post-secondary MTSS model. This particular qualitative method is referred to as Participatory Action Research (Greenwood, Whyte, & Harkavy, 1993; Hoedebeck, 2011; Kemmis, McTaggart, & Nixon, 2014; McIntyre, 2008; McTaggart, 1991; Reason & Bradbury, 2001). And lastly, longitudinal data is collected to track student success throughout the developmental course sequence, success in college algebra, passing the CAAP, and eventually, matriculating to degree.