College Quarterly Logo

Volume 22 • Issue 2 (2019)

Proactive advising for at-risk students: Comparing a predictive model to faculty identification

David Waddington

Abstract

This study investigates the alignment of a predictive model created to categorize first semester students by risk level of not completing their studies with the faculty-identification of students displaying risk behaviours of the same cohort at Mohawk College. Data created by Finnie et al. (2017), is compared to a sample of first semester students that were identified as being “at-risk” by faculty members that had no exposure to the results of the predictive model. Results indicate that there is alignment between the predictive ability of the model and the faculty-perceived risk of student success in a variety of variables including overall identification of “at-risk” behaviour, semester GPA, and next semester re-registration. Relevant contributions to advising practices in a college setting are provided through the alignment of the results generated from a predictive model and the practices of both advisors and faculty members.


Identification of at-risk students: Comparing a predictive model to faculty identification

Early alert interventions are central to many conversations on supporting student retention and success (Finnie, Fricker, Bozkurt, Poirier, & Pavlic, 2017; Dial, 2015; Beck & Davidson, 2015; Faulconer, Geissler, Majewski, & Trifilo, 2014; Tampke, 2013). In particular, Finnie et al. (2017), in partnership with Mohawk College, investigated the participation rates of first-semester students in various advising programs at the college in Hamilton, Ontario. What was unique about this study was that Finnie et al. (2017) used a predictive model to categorize first-semester students into three levels of risk classifications prior to investigating the use of advising services on campus.

Mohawk College has been supporting students in many proactive ways. Specific to this paper, the practice of faculty members providing Student Success Advisors (SSAs) with lists of students that they feel are demonstrating behaviours that may place them at-risk of successfully completing their program allows SSAs to target their proactive outreach strategy to students that appear to need the most support. While the report by Finnie et al. (2017) summarized the accuracy of the predictive model and highlighted the advising participation rates among students in each at-risk category, this paper recognizes an opportunity to better understand the efficacy of the college’s faculty-advisor partnered approach to proactive advising.

Predictive models are mathematical functions that create a prediction for an outcome based on the values of predictor variables. Once the model has been tested for predictive accuracy by comparing to a testing sample, a probability value ranging from 0 to 1 is produced and is representative of (in this case) the amount of risk that a student will leave college early (Finnie et al. 2017). Variables that were included in this model include age, regional status, high school grade point average, program area, credential sought, career clarity, confidence in abilities, hours of work per week, commitment to education, assessment score, and missed assessment. When considering the use of a predictive model in an early alert, or proactive advising application, one key advantage of this tool is that once validated, it can prove to be an accurate tool from which to base proactive advising practices geared to help support those that are most at risk of leaving college early (Finnie et al., 2017). This accuracy, as noted by Finnie et al. (2017) is a result of complex series of steps during the development and application phases.

In addition to supporting student success at the college level, predictive models have been used to uncover variables responsible for the development of early math skills. Aragon, Navarro, Aguilar, Cerda, and Garcia-Sedeno (2016) were able to provide evidence that literacy is a key predictor of math skill development in children between the ages of 5 & 6. In this study, similar to the one carried out by Finnie et al. (2017) this understanding allows for practitioners to better target their support strategies based on the results of the predictive model.

Essa and Ayad (2012) point to two limitations when using predictive models for supporting student success strategies. The first concern identified is the ability to generalize across courses and institutions. The second limitation they identified is the lack of insight that the models provide to support the design of supportive interventions. For example, a proactive advising strategy utilized at Mohawk College involves collaboration between SSAs and faculty members to identify and reach out to students displaying behaviours not conducive to academic success. These behaviours could include poor attendance, failed assignments or tests, or inactivity with the online learning management system among others. This exercise, carried out each semester, roughly between weeks 3-10 is somewhat casual with no definitive expectations around what constitutes at-risk behaviours. Faculty members provide advisors with lists of students that they deem to be at risk of achieving success and from there, the advisors reach out to these students in a proactive manner. Where necessary, faculty members and advisors may have additional discussions about students with specific needs or concerns as part of this outreach process. This basic process is an example of the way that advisors and faculty members can work together to help better understand the needs of each student that requires additional support to achieve academic success.

Smith, Therry, and Whale (2012), also investigated the use of a predictive model to support the identification of at-risk students. Specifically, they were looking to learn whether student characteristics and background information in their student database would inform a reliable prediction model to assist in the identification of students that were at risk of failure in their first year accounting program. While the predictive model proved to be useful in the identification of at-risk students, it is important to note that without support from a variety of support services and academic leads, improved student performance would not have been possible. Findings of Smith, Therry, and Whale (2012) suggest that providing generalizable results is a challenge when developing an early alert mechanism and attribute at least a portion of that challenge to the complexity of developing a predictive model. With generalizability, low insights into support strategies, and complexity acknowledged as barriers to relying on predictive models, this paper seeks to investigate a long standing student support strategy by applying predictive at-risk data to data collected through Mohawk College’s advisor-faculty support strategy.

Partnerships with faculty members to support student success have had a positive impact on student transition (Llamas, 2010) and retention (Williamson, Goosen, & Gonzalez, 2014). There is also room for caution when celebrating the impact of faculty partnerships as uncovered by Fletcher (2012). At Mohawk College, SSAs and faculty members collaborate to inform and execute a proactive advising strategy involving the identification of and outreach to at-risk students. While this proactive strategy is not the same as found in Finnie et al. (2017), it does serve as the beginning of a formalized proactive outreach strategy to students identified as potentially at-risk in each term of study. The intersection of this manual proactive advising strategy carried out by the SSAs and the faculty members, and the presence of the predictive model developed by Finnie et al. (2017) created an environment that allowed for an investigation of consistencies shared by these techniques.

This led to four research questions. 1. What is the relationship between the predictive model developed by Finnie et al. (2017) and the faculty identification of at-risk, first semester students at Mohawk College in the fall 2015 semester? 2. Are students more or less inclined to respond to proactive outreach based on their assigned risk level? 3. Do students identified as at-risk by faculty members perform differently academically (GPA) based on the risk level associated with the scores from the predictive model created by Finnie et al. (2017)? 4. Is there a difference in semester-to-semester retention within the faculty identified at-risk students between each of the three identified risk levels outlined by the predictive model developed by Finnie et al. (2017)?

Methods

During the Fall 2015 semester, SSAs received the names of 771 first semester students that had been identified as ‘at-risk’ by faculty members from across each of the schools within the college. One member of the SSA team would compile one master list of faculty-identified at-risk students, logging variables such as Student ID, Name, Program, Course, Response to Outreach, and End of Semester Promotion Status. This master list of students was used in the comparison of the predictive model and the longstanding proactive outreach strategy to support faculty-identified at-risk students. Students are identified as ‘at-risk’ by satisfying one of a variety of criteria. Such criteria include, but are not limited to, poor attendance, lack of engagement with the online learning management system, failed tests, incomplete assignments etc. In support of the already established advising practice, the names of the identified students were shared with the SSAs.

The ‘at-risk’ lists from faculty members were received by the SSAs between weeks 4-12 of the academic semester, which runs for 16 weeks. Upon receipt of an ‘at-risk’ list, the SSA would reach out to each of the students. SSAs reached out in a variety of ways. Methods of outreach included mass email, individual email, or phone call. The mass email would have included a broad message indicating the specific behaviour that the student was displaying along with references to a variety of resources and an encouragement to make an appointment to meet with the SSA. The individual emails were fashioned similarly, however, would include a personal salutation and where relevant, may include a more personalized message depending on the previous interactions that the SSA may have had with the student. Telephone outreach involved the SSA calling the student at the number associated with their student account. If there was no answer, the SSA would leave a voicemail. There were no scripts provided for these outreach methods. Additionally, it is also important to note that neither the SSAs nor the faculty members knew of the results of the predictive model developed by Finnie et al. (2017).

If a student responded to SSA outreach, the SSA would track that data on their outreach list. One of the SSAs then collated the completed lists. The data from this collated list and data collected from the Finnie et al. (2017) study were merged. Once merged, data from the outreach dataset were reviewed for alignment with the level of risk as determined by the predictive modelling dataset. Using Minitab 18 software, additional analysis in the form of two sample Z-Tests to analyze the difference in variables embedded within the research questions between individuals in each of the High, Medium, and Low risk categories were carried out. Descriptive statistics of the sample of students in each of the at-risk categories as dictated by Finnie et al. (2017) are also presented.

Results

Of the 771 students identified as at-risk by faculty members, 296 identified as female while 475 identified as male. The average age for the female students identified by faculty as at-risk was 21.7 years, while the average age for males was 20.2 years. Female students in this group ranged in age from 18 to 60, and males 18 to 40. Counts of faculty identified at-risk students revealed that 365 of the 771 assessed into the High Risk group, 241 of 771 assessed into the Medium Risk group, and 165 of 771 assessed into  the Low Risk Group. The overall semester GPA mean was 42.96%. The GPA mean for the High Risk group was 38.64%; the Medium Risk group 45.38% and; the Low Risk group, 48.93%. See Table 1.

Table 1 Summary of Descriptive Statistics
Variable Value
Female 296
Mean Age (female) 21.7 years
Age Range 18 - 60
Male 475
Mean Age (male) 20.2 years
Age Range (male) 18 – 40
Low Risk Group 165
Medium Risk Group 241
High Risk Group 365

There was a significant difference in GPA between students in the Low Risk group (M=48.93, SD=28.3) and the High Risk Group (M=38.64, SD=25.4), conditions t(527)=4.17, p = 0.000. There was a significant difference in GPA between students in the Medium Risk group (M=45.38, SD=27.4), and the High Risk Group (M=38.64, SD=25.4), conditions t(603)=3.10, p = 0.002. There was not a significant difference between the Low Risk group (M=48.93, SD=28.3) and the Medium Risk group (M=45.38, SD=27.4), conditions t(404)=1.27, p = 0.206. See table 2.

Table 2 Statistical Summary of GPA and Risk Group (*indicates significance at 0.05 level)
Groups t – Score p Value
Low/High 4.17 0.000*
Medium/High 3.10 0..002*
Low/Medium 1.27 0.206

503 email outreach messages were sent to the faculty-identified at-risk students by SSAs. The breakdown of predictive model-identified risk level indicates that 213 emails were sent to students in the High Risk category, 176 emails were sent to students in the Medium Risk Category, and 114 emails were sent to students in the Low Risk category. In total, 60 of the 503 students that received an outreach message responded to their SSA and connected with them. 24/213 (11.27%) from the High Risk group, 19/176 (10.80%) from the Medium Risk group, and 17/114 (14.91%) from the Low Risk group. Two-sample Z-Tests revealed no significant difference in the rate of response to outreach between each group. See Table 3.

Table 3 Statistical Summary of Outreach Response Rates
Groups Z – Score p Value
Low/High -0.95 0.34
Medium/High -0.15 0.88
Low/Medium -1.04 0.30

Institutional re-registration data was available for 697 of the students identified by their faculty as at-risk. Registration data showed that faculty identified students in the Low Risk group re-registered for semester 2 at a rate of 73.61% (106/144); Medium Risk group re-registered at a rate of 68.33% (151/221); and the High Risk group re-registered at a rate of 63.75% (211/331). Two-Sample Z-Test for Proportions were conducted and revealed that students in the Low Risk group registered for semester two significantly more than did students from the High Risk group z = -2.10, p = 0.04.  Two-sample Z-Tests conducted on the relationship between Medium Risk and High Risk (Z = 1.11, p = 0.27) as well as between Medium Risk and Low risk (Z = -1.08, p = 0.28) did not reveal a significant difference. See Table 4.

Table 4 Re-registration summary by Risk Group (* indicates significance at 0.05 level)
Groups Z – Score p Value
Low/High -2.10 0.04*
Medium/High 1.11 0.27
Low/Medium -1.08 0.28

Discussion

Responses to the research questions.

The first research question asked “What is the relationship between the predictive model developed by Finnie et al. (2017) and the faculty identification of at-risk first semester students at Mohawk College in the fall semester 2015?” The distribution of students identified as at-risk by faculty members appears to align well with what one may expect given the level of risk as determined by the predictive model. What is seen in this case is that just under half (47.3%) of students that were identified as at-risk by faculty members fell within the high risk group whereas 21.4% of students identified by faculty as at-risk came from the low-risk group as identified by the predictive model. The remaining 31.2% of faculty identified at-risk students were identified as medium risk by the predictive model. Put another way, 78.6% of students identified as at-risk by faculty members fell into the medium and high risk categories established by the predictive model developed by Finnie et al. (2017). These data suggest that a faculty member’s ability to identify at-risk students aligns well with what the predictive model suggests. The key difference here is that the faculty identification of at-risk behaviours comes after students begin their classes. Beck and Davidson (2015) provide similar support for the alignment of predictive models and student behaviour by demonstrating that identifying factors contributing to student risk factors earlier in the student lifecycle allow support initiatives to be implemented earlier. With that in mind, they collected data between weeks 6 and 10 to verify the accuracy of the predictive tool in use. A strategy similar to the one used in this study.

The second research question asked, “Are students more or less inclined to respond to proactive outreach based on their assigned risk level?” The data revealed no significant differences in the rate at which students respond to proactive outreach across all groups of risk level. Insights on the motivations of students are beyond the scope of this project. Understanding why students choose to or not to respond to outreach would add a much-needed qualitative lens to further research. Despite the lack of significant findings, it is worth noting here that the 3.64% difference in response rate between the Low-risk and High-risk groups may have a very practical impact on campus. Across a thousand students, we should expect to see approximately 36 more students engaging with their advisor. In an institution that supports over 15 000 students, this may have implications on service delivery regarding engagement targets and workflow expectations. Although there is literature that supports the notion that student attrition will occur despite an institutions best attempts (Fletcher, 2012), research also shows that students who seek advising are more likely to persist (Beck & Davidson, 2015; Faulconer et al., 2014; Tampke, 2013).

The third research question asked “Do students identified as at-risk by faculty members perform differently academically (GPA) based on the risk level associated with the scores from the predictive model created by Finnie et al. (2017)?” As expected, GPA did decrease with an increase in risk level. No statistically significant difference was discovered in GPA between the low (48.93%) and medium (45.38%) risk groups nor was there a significant difference between the medium (45.38%) and high-risk (38.64%) groups. There was a significant difference in GPA discovered between the high (38.64%) and low risk (48.93%) groups. The importance of this finding is echoed in Zhang, Fei, Quddus, and Davis (2014) as this group noted that post-secondary institutions may be negatively impacted by academically at-risk students from a funding perspective. Institutions that rely on funding for graduates could miss financial resources to improve their support programs if they do not support academically at-risk students early on in their transition to college. There is further support for the need of early identification of at-risk students in Finnie et al. (2017) through the acknowledgement that students coming to college with a GPA of a D or below are more likely to not persist. The minimum grade required for a pass in most classes at Mohawk is 50%. The average GPA achieved by all of the faculty – identified at-risk students was below that mark.

The final research question sought to examine whether or not there was a difference in semester-to-semester retention within the faculty identified at-risk students between each of the three identified risk levels outlined by the predictive model developed by Finnie et al. (2017). This study revealed a significant difference in the re-registration rates between students in the Low-Risk category to their High-Risk counterparts. Aligning with the results reported by Finnie et al. (2017), students in the Low-Risk category did in fact re-register at a higher rate than other students. While strong relationships with advisors help to promote student retention and success (Laskey & Hetzel, 2011), this study suggests that there is not a difference in the engagement rate with advisors, while findings from Finnie et al. (2017) suggest that it is in fact the most successful students that meet with their advisors the least.

Considerations for practice

The collection of faculty-identified at-risk student lists was a practice that began in the fall of 2014. At that time, there were no specific directions provided with respect to when at-risk students should be identified, what constitutes “at-risk” behaviour, the messaging associated with proactive outreach, nor was the expectation around the outreach process standard across all SSAs.
When considering the best approach to advising students proactively, conventional wisdom suggests that the earlier that proactive advising can take place, the better (Finnie et al., 2017; Beck & Davidson 2015; Dial, 2015; Faulconer et al., 2014; Tampke, 2013; Fletcher 2012; Smith, Therry, & Whale, 2012). Assuming that collecting and analyzing data to create a predictive model for each intake of college students is not feasible to initiate proactive advising techniques, a different solution is required. Based on the consistency shown between the predictive model and faculty-identification of at-risk students, collaboration between faculty members and (in this case) SSAs may be a reliable solution.

While participating in proactive advising is not a prescribed duty for faculty members, those that have participated in the outreach initiative appear comfortable with the concept of equifinality as described by Gresov and Drazin (1997). The concept of equifinality is an important one when considering a partnered proactive advising approach in many post-secondary institutions. Gresov and Drazin (1997) suggest that the structures of an organization should not be a deterrent for carrying out the functions of an organization. In this situation, that translates to suggest that despite the formal duties associated with positions within the college, (structures) the goal of the college is to promote student success (function). It is not feasible for faculty members to only teach, nor is it possible for student service areas to support student success and retention without input from faculty members. Llamas (2010) was able to demonstrate that faculty members and school counsellors were able to work together to create a course designed to help first semester students transition to post-secondary life. Of importance to that study were the three factors identified as keys to collaboration: (i) horizontal relationships; (ii) a spirit of mutual help and confidence; and (iii) autonomy. Those involved in the collaboration identified these factors as necessary for a successful collaboration because it allowed individuals to contribute despite their area of expertise and provided an environment where sharing and creativity were valued.

Another benefit to a faculty and SSA partnership to support student success and retention is that a traditional view of faculty advising is that students will, at times see faculty advising as a form of discipline instead of support (Williamson, Goosen, & Gonzalez Jr., 2014). Like the advising programs described by Dial (2015), much of what the SSAs at Mohawk College do is rooted in appreciative advising. This is a supportive, co-constructed approach to developmental advising allowing advisors to build rapport with students. Feedback from the SSAs suggest that once they have had an opportunity to work with a student, they quickly become a place of familiarity or comfort for that student. As the results on student response to outreach above indicate however, there is still a lot of work to do to get students to connect with their advisor in a timely fashion.

Limitations

This study resulted from post hoc observations and an element of serendipity that saw Finnie et al. (2017) report on data collected from student activity in the fall of 2015. This was the first semester that faculty identified at-risk outreach data was tracked.  As a result, definitions pertaining to the identification of “at-risk” behaviours according to faculty were not clearly articulated or controlled. This affects the generalizability of this study. Faculty members, like all individuals, undoubtedly have varying thresholds for what they deem to be “at-risk” behaviour. This loosely defined term may have created false positives and/or false negatives regarding faculty-identified at-risk students.

Also important to consider in future versions of such an investigation are the timelines associated with the identification of, and outreach to, at-risk students. During the collection of the faculty-identified at-risk student data, SSAs received such information anywhere from weeks 3-12 during the semester. Intuitively, providing effective advising support becomes increasingly difficult as time in the semester runs out. For future studies, a clear timeline put in place to collect faculty-identified at-risk student information would allow for a consistent proactive advising strategy to be put in place to support those students.

The proactive advising techniques used by advisors varied in rigour. Primarily due to other demands and a lack of formal direction as it pertains to this project, there are differences in the number of times that advisors sought to reach out to students and the mode in which they did so. Note that this is not a criticism of the work done by the advisors. Advisors working in a decentralized model such as this support their academic areas in subtly different ways guided by principles of developmental and proactive advising.

It may also be beneficial to future research projects to account for the student, advisor, and faculty experiences in a qualitative manner. This study is unable to shed light on questions that would uncover why students choose to or not to respond to SSA outreach. Nor does is uncover any of the actions that students took after receiving outreach from an SSA.

Considering the advisor and faculty experiences, a qualitative account of the rationale behind the wording and approach to outreach practices and identification of at-risk students would help to inform future practices for supporting at-risk students.
Despite these limitations, this paper represents the real operations of an advising team. Such descriptions of the intimate operational details of advising practices are missing in the literature and this paper represents a useful contribution. It also demonstrates the complexity of this work and the difficulty in identifying and supporting at-risk students, even when a powerful predictive model is in use. Conversely, the value of faculty identified at-risk students should not be understated. This is an important practice that has meaning to faculty, advisors, and students on a variety of levels.

Acknowledgement

A sincere thank you to the faculty members that contributed to the proactive advising carried out by the SSAs. Their openness to their role as partners in supporting student success at Mohawk was an enhancement to the work led and carried out by the SSAs. This project is a concrete example of a conscious effort to make student success a priority for everyone working at the institution.

The Student Success Advisors deserve a moment of recognition for their effort to evolve by making a shift to record their proactive advising practices. The tracking, analysis, and application of data has the potential to elevate the field of advising by providing more quantifiable results from which to inform decision making. Such successes may also lend themselves to strengthened recruitment efforts in addition to the support they provide for student persistence and retention.

Finally, a moment of recognition to Dr. Ross Finnie and the team at the Education Policy Research Initiative (EPRI) for their collaboration with Mohawk College. As a result of the previous research completed between these parties, the opportunity to further investigate the advising practices at Mohawk College was made possible.

References

Aragon, E., Navarro, J. I., Aguilar, M., Cerda, G., and Garcia-Sedeno, M. (2016). Predictive model for early math skills based on structural equations. Cognition and Neurosciences. 57, 489-494.

Beck, H. P. and Davidson, W. B. (2015). Improving the retention of first-year college students: A temporal model of assessment and intervention. Journal of the First-Year Experience & Students in Transition, 27(2), 83-99.

Dial, M. T. (2015). Scuccess Connect: An appreciative approach to early alert at a large flagship university. Journal of Appreciative Education. 2(2), 24-33.

Essa, A. and Ayad, H. (2012). Improving student success using predictive models and data visualisations. Research in Learning Technology. 20, 58-70.

Faulconer, J., Geissler, J., Majewski, D. and Trifilo, J. (2014). Adoption of an early-alert system to support university student success. Educational Technology. 45 – 48.

Finnie, R., Fricker, T., Bozkurt, E., Poirier, W., Pavlic, D. (2017). Using predictive modelling to inform early alert and intrusive advising interventions and improve retention. Toronto: Higher Education Quality Council of Ontario.

Fletcher, D.M.K. (2012). A national study of student early alert programs at two-year institutions of higher education. (Doctoral dissertation) Arkansas State University. Jonesboro, AR.

Gresov, C. and Drazin, R. (1997). Equifinality: Functional equivalence in organization design. Academy of Management Review. 22(2), 403-428.

Laskey, M. L. and Hetzek, C. L. (2011). Investigating factors related to retention of at-risk college students. Learning Assistance Review. 16(1), 31-43.

Llamas, J. M. C. (2010). Collaboration between faculty members and school counselors: An experience from a case-based course. Innovative Higher Education. 36, 177-187.

Rutschow, E. Z., and Mayer, A. K. (2018). Early findings from a national survey of developmental education practices. Centre for the Analysis of Postsecondary Readiness. Retrieved from https://www.mdrc.org/sites/default/files/2018_CAPR_Descriptive_Study.pdf

Smith, M., Therry, L. and Whale, J. (2012). Developing a model for identifying students at risk of failure in a first year accounting unit. Higher Education Studies. 2(4), 91-102.

Tampke, D. R. (2013). Developing, implementing, and assessing and early alert system. Journal of College Student Retention. 14(4), 523-532.

Williamson, L.V., Goosen, R. A., Gonzalez Jr., G. F. (2014). Faculty advising to support student learning. Journal of Developmental Education. 38(1), 20-24.

Zhang, Y., Fei, Q., Quddus, M., and Davis, C. (2014). An examination of the impact of early intervention on learning outcomes of at-risk students. Research in Higher Education Journal. 26, 1-12.


David Waddington

David Waddington holds an administrative position at Mohawk College supporting a number of Career and Student Success related initiatives. Working in post-secondary education in a variety of capacities since 2007, David’s focus is on the role that education plays in preparing individuals for their lives as productive, contributing members of society. David holds an Honours BA in Psychology, with a minor in Philosophy, from Brock University and recently completed his M. Ed. – Postsecondary Studies with Memorial University.