Summer 1996 - Volume 3 Number 4
Qualitative Measures In Assessing Student Outcomes From A College Education
Assessment of student outcomes from a college education is an increasingly important measure of an institution's (or a college system's) relevance, effectiveness, and accountability. Today's planned assessments by college leaders and institutional researchers need to go beyond the traditional indicators of outcome measures (e.g. graduation rates, placement statistics) and move into more qualitative assessments derived from a student's college experience. Student outcomes data can offer important comparisons in successive years of analysis by which one can address stakeholder concerns about the quality of education, and can identify relevant student measures of "success" from a college education for different career program areas.
On a national level in Canada, assessing student outcomes related to a college education has never been an easy task for college leaders and institutional researchers. As Dennison and Levin noted, the Canadian college systems are quite diverse in comparison across provinces, and moreover "there are identifiable differences among the purposes of colleges in rural and urban centres" in many provinces (Dennison and Levin, 1988). Although this diversity has strengthened the relevance and accountability of Canadian post-secondary systems, it becomes a more difficult and challenging research task to generalize student outcomes across the country.
Colleges bring together a variety of resources to provide opportunities to students who wish to develop intellectually, vocationally and emotionally. College leaders have often defined their institutions' mandates through statements of objectives in a number of educational areas. Such learning or outcome statements can include: (a) general knowledge and generic skills, (b) vocational skills, and (c) social or psychological outcomes. Generally, however, college educators have relied more heavily on traditional quantitative measures such as placement statistics (perhaps including employment rates and beginning job salaries) as primitive indicators of success.
These measures, however, do not necessarily assess outcomes in terms of the students' achievement of overall educational goals or the necessary skill sets required for successful employment. Therefore, if outcome measures are to attempt to qualify numerical outputs in such a way that effectiveness and quality are addressed, educators need more meaningful data based on clear educational goal statements.
By setting out desired student outcomes and then assessing the extent to which they have been accomplished, stakeholders can provide an overall improvement plan for the educational process itself. Astin documents a longitudinal study of student experiences at over 200 colleges, focusing on academic development, personal development, and satisfaction (Astin, 1993). Astin (et al, 1993) and others identify the importance of attention to outcome as well as the experiences that lead to these outcomes.
Accountability in institutions of higher learning is chronicled by Aper (1993) who emphasizes the place of student outcomes assessment, whereas Lingrell (1992) provides literature related to outcomes assessment and points to college senior level surveys as essential elements in addressing issues of effectiveness.
The interest in using outcomes assessment for formal academic programs is growing. As concern over the quality of graduates increases and college education outcomes (especially in the areas of critical thinking, reasoning and communications skills) gain relevance, educators may be required to make outcome evaluation a higher institutional priority. For example, Bok (1986) believes that an educational institution must study the impact of the learning process on students and assess the effects of its academic programs; without such critical self-evaluation, he contends, educational institutions will not progress.
College graduates can provide institutional researchers, faculty and administrators with crucial information on what "their judgments about the college experience" (Wilson). Pace reports that graduates can share a great deal about what they have learned in college including their critical thinking abilities, knowledge and skills acquired in a specialized occupational field, as well as their personal and social development (Pace, 1979).
Ewell (1983) notes that research on student outcome measures can benefit academic program review and evaluation, institutional planning, and ongoing public relations efforts that include marketing the institution to the greater college community. Pragmatic applications of outcome research can be seen to benefit academic accreditation studies and contribute to the development of quality program standards. They are also helpful in establishing baseline standards data to successfully promote education and training efforts to business and industry.
Researchers have organized the learning outcomes of a college education into several topical areas such as intellectual knowledge, general education and communications skills (Creamer et al, 1991). Pascarella and Terenzini (1991) synthesized 25 years of educational research on how a college education (four-year liberal arts and science programs) affects students. Their work explores relevant studies on the impact higher education can have on students, including their moral development, quality of life after college, and psycho-social changes.
Cousineau and Landon (1989) assessed student outcomes from several academic program areas and identified many key factors related to those outcome measures. They confirmed that academic skills and satisfaction are affected positively by increased involvement in college life. Evers and Gilbert (1991) described student outcome measures resulting from formal post-secondary instruction. Their findings show that formal instruction is "a major source of development for thinking and reasoning skills, problem solving skills, planning and organizational skills, time management skills, ability to conceptualize, learning skills, and quantitative, mathematical and technical skills." And the relative value and importance of assessing students' general education learning outcomes in higher education also have been raised by Clagett (1991).
Increasingly, governments have put pressure on colleges to define quality educational outcomes more adequately. One provincial government set up a College Standards and Accreditation Council (see Johnston and Shapiro, 1992). The establishment of this Council sent a strong message to provincial colleges, as a whole, that greater accountability of public funds must be forthcoming. The establishment of accreditation 'machinery' and clear standards will help satisfy stakeholders' demands for better educational accountability, and will encourage the entire college system to continue more formalized outcome assessment practices and procedures.
Shaver (1990) points out that the 'push' for better outcome assessments originates from three main sources, namely educational institutions themselves, consumers of the educational process, and government initiatives. Generally, colleges have demonstrated their mandate in terms of employment outcomes or placement rates expressed as a percentage of graduates. "To get a good job, get a college education," is the traditional adage of Canadian college systems. Nevertheless, a poor economic climate, unemployment, and chronic underfunding by successive provincial and federal governments have prompted many educators to value college education outcomes not just in quantitative terms, but in qualitative terms as well.
Consumers of education (students, parents and employers) are "registering increasing concerns over the quality of educational preparation" (Shaver, 1990: 9). In addition, community committees that advise colleges on program and course matters, and who also represent the other stakeholders in the educational process, are increasingly expressing greater concern over educational quality.
As employment opportunities for qualified graduates fluctuate, the consumers of a college education will wish to emphasize more of the value-added qualitative outcomes of their post-secondary experiences. Clarkson and Gordon (1989) noted that a general trend in North America will be increased public attention on the productivity or outcomes of our educational system. If college leaders and institutional researchers focus on measures that show a significant decline of outcomes in the future, perhaps a more convincing case may be made for better funding provisions from the public purse. It is clear that additional efforts to provide hard evidence of quality outcomes from education and training providers in the college systems will be required.
The identification of desirable student outcomes can be realized from several sources, including both the relevant literature on academic research into outcomes of a college education, and primary source material in institutional or government documents. Business lobbies and think-tanks have also put forward what they believe are relevant outcomes for college graduates. The Conference Board of Canada, for example, has produced a comprehensive list of critical skills (1992) required of a successful workforce.
Planned outcome assessment research allows Canadian colleges to more confidently discuss what can be described as the "great self-evident" outcomes of higher education, that is, some often-overlooked measures (e.g., cognitive, affective, generic and career) and other important factors (such as self-understanding, human relationships, career advancement and general education) which relate to the goals of a college system. In addition, student outcomes assessment can be applied to the academic planning and accountability process. By identifying student performance outcomes for different career program areas, the data from longitudinal outcomes research can be used as effective accountability measures.
Much work needs to be done in establishing standards for student outcome measures. This area of student outcome assessment promises to be an important activity as colleges are confronted with the challenges of accreditation, stakeholder concerns, and increased competition for education and training.References Dennison, J.D. and J.S. Levin (1988). "Goals of Community Colleges in Canada: A 1987 Perspective." Canadian Journal of Higher Education. 21: 2.
Astin, A.W. (1993). "What Matters in College," Liberal Education. 79: 4 (Fall).
Astin, A.W. et al (1993). "Principles of Good Practice for Assessing Student Learning." Leadership Abstracts. 6: 4 (April).
Aper, J.P. (1993). "Higher Education and the State: Accountability and the Roots of Student Outcome Assessment." Higher Education Management 5: 3 (November).
Lingrell, S.A. (1992) "Student Outcomes Assessment: The Senior Survey." ERIC Document ED 351897 .
Bok, D. (1986) "Toward Higher Learning: The Importance of Assessing Outcomes," Change. 18: 6 (November/December).
Wilson, P.A (no date). "A Study of Selected Outcomes for Ohio Graduates of 1980, 1976 and 1971." unpublished doctoral dissertation, Ohio University.
Pace, C.R. (1979). Measuring the Outcomes of College. San Francisco: Jossey-Bass.
P. Ewell, P. (1983). "Implementing Assessment: Some Organizational Issues," in T. W. Banta, ed., Implementing Outcomes Assessment: Promise and Perils. New Directions for Institutional Research No. 59. San Francisco: Jossey-Bass.
Creamer, D.G. et al (1990) College Student Development: Theory and Practice for the 1990s. Washington: American College Personnel Association.
Pascarella E.T., and P.T. Terenzini (1991). How College Affects Students. San Francisco: Jossey-Bass.
Cousineau J., and B. Landon (1989). "Measuring Academic Outcomes and Identifying What Influences Them." Paper presented at the A.I.R. Forum, Baltimore.
Evers, F.T. and S.N. Gilbert (1991). "Outcome Assessment: How Much Value Does University Education Add?" Canadian Journal of Higher Education. 21: 2.
Clagett, C.A. (1991). "Meeting Student Outcomes Accountability Mandates: The Institutional Research Approach." Community/Junior College. 15.
Johnston R., and B. Shapiro (1992). The College Standards and Accreditation Council: The Report of the CSAC Establishment Board to the Minister of Colleges and Universities. Toronto: Council of Regents of the Ontario Colleges of Applied Arts and Technology.
Shaver, K. (1990). "Evaluating Educational Outcomes in Higher Education: An Overview of Recent Trends," in Glen A. Jones, ed., Higher Education Group. Toronto: Ontario Institute for Studies in Education.
Clarkson and Gordon (1989). Tomorrow's Customers. Toronto: Clarkson and Gordon.
Conference Board of Canada (1992). Employability Skills Profile: The Critical Skills Required of the Canadian Workforce. Toronto: Conference Board of Canada.
Wm. Berry Calder is President and CEO of the College of the Rockies, in Cranbrook, British Columbia, Canada. Diane C. Melanson is a Counselor at Algonquin College in Nepean, Ontario, Canada.
• The views expressed by the authors are those of the authors and do not necessarily reflect those of The College Quarterly or of Seneca College.
Copyright © 1996 - The College Quarterly, Seneca College of Applied Arts and Technology