It is generally accepted that Canadian postsecondary education enjoys an international reputation for quality. However, Canada does not have an accepted, national framework to understand, measure, or clearly define the actual quality of the postsecondary education sector. In this, Canada is seen as unique when compared with other developed countries. Canada has thirteen educational jurisdictions (provinces and territories), each with different institutions, and institutional mandates and missions. One of the results of this structure is that agreement on how to conceptualize and assess quality across the Canadian postsecondary education sector becomes a complex challenge. A single approach may not be acceptable, or appropriate, if applied on a national level. As the Canadian Council on Learning said in its report, Up to Par: The Challenges of Demonstrating Quality in Canadian Post-secondary Education,“Developing a pan-Canadian framework for understanding quality in PSE may be necessary to promote and improve Canada’s PSE sector, while ensuring also that students can make decisions about how best to meet their educational aspirations” (p. 5). Oldford (2006) contextualizes the issue as relating to: “…the ‘patchwork’ nature of Canada’s various and uncoordinated quality assurance mechanisms…In short, Canada lacks a consistent and comprehensive approach to quality assurance… (p.vii).” She further says that while, “[T]he Canadian quality assurance system is rich in methods of program assessment…one aspect that the Canadian systems lack…is the meta-evaluation of these program assessment methods at the institutional level (p.ix).”
As education is a provincial responsibility with federal “transfer funding”, consideration here will be mainly on the Ontario situation. The system for post-secondary education in Ontario consists of three main types of institutions: universities, publicly assisted colleges, and private career colleges. In Ontario, the provincial government has articulated its priorities for postsecondary education as being: Accountability, Accessibility and Quality.
However, the requirements and responsibilities for the various sectors within the postsecondary system are different in all of these so-called priority areas. Certainly all institutions are “accountable” to the provincial government; however the manner and complexity of this accountability varies. Similarly with respect to accessibility, there are variances that impact the institutions and their mandates. Colleges have an “open access” mandate while the other types of institutions operate on a “selective access” mandate. In terms of quality the challenges and issues are complex. The fact that the concept of quality is hard to define, and therefore hard to measure, simply adds to the complexity of the challenges.
The purpose of demonstrating quality in postsecondary education institutions is generally conceived as,
…one of either improvement or accountability….Quality assurance for the sake of improvement is often a process internal to an institution. Quality assurance…for the purpose of accountability…examines ‘what one is doing in relation to goals that have been set’…and is usually linked to public information. (Canadian Council on Learning, 2009, p.8)
To address this dichotomy of purpose Ontario’s Colleges have developed and implemented a quality assurance process that is one of quality assurance and continuous improvement. Before delving into this model, it is important to see the provincial context of agencies that are responsible for quality in postsecondary education in the province.
The college system is often represented by Colleges Ontario (CO), the voluntary advocacy and lobbying association of the 24 publicly assisted colleges in Ontario. Residing with Colleges Ontario, and responsible to an arms-length Management Board, is the Ontario College Quality Assurance Service (OCQAS). OCQAS provides quality assurance at the program-level through the Credentials Validation Service (CVS) and institutional-level quality assurance and continuous improvement through the Program Quality Assurance Process Audit (PQAPA) process.
The university system is often represented by Council of Ontario Universities (COU) which, similar to CO, is the association of the public universities in Ontario. COU has recently (2010) developed the Ontario University Council for Quality Assurance (OUCQA) to replace the former Undergraduate Program Review and Assessment Committee (UPRAC) and the Ontario Council of Graduate Studies (OCGS). This quality assurance service for the universities in Ontario provides a similar and parallel service to that of the OCQAS for the colleges. One of the main differences, however, is that the OUCQA does not provide any institutional-level quality monitoring or assurance. While they consider quality processes within schools, departments, and faculties within a university, there is no overall institutional-level rolling up of the processes as there is with the OCQAS.
In addition to these two ‘sector-based’ quality assurances services, Ontario also has two agencies that operate as arms-length agents for the provincial government: the Higher Education Quality Council of Ontario (HEQCO) and the Postsecondary Education Quality Assessment Board (PEQAB). Each has different roles and responsibilities with respect to quality assurance in the postsecondary sector in Ontario. At the risk of not doing full justice to the mandate and operation of either of these agencies, there will only be a brief description of each offered here. HEQCO has a mandate to provide evaluative data for the post-secondary education sector, and research-based advice to the government, while PEQAB provides recommendations to the Minister (of the day) with respect to any postsecondary institution (college or out-of-province University) who wishes to offer degree-level programming. For a full description of these agencies please see their respective websites at www.heqco.ca and www.peqab.gov.on.ca.
As can be seen then there are challenges in the higher education sector that require addressing. The purpose of this article is to describe how the public college sector is moving forward to address the quality assurance issue within its institutions.
As noted above, the college sector has developed the OCQAS as its quality assurance mechanism. The Service is unique in many aspects and is unlike other similar agencies involved in quality assurance. Some of the unique factors that are particular to the OCQAS are:
- While it was mandated into existence by the provincial government(The Colleges of Applied Arts and Technology Act, 2002) the actual Service is ‘owned and operated’ by the colleges and is not a government agency;
- The Service is funded by, and responsible to, the 24 member colleges, and not to the government;
- It is arms-length from any individual institution, works with all 24 member colleges in Ontario, and physically resides in the offices of the association of colleges (CO); and,
- Its activities are overseen and directed by a Management Board whose membership consists of those internal to the college system and those who are external to the college system, thus creating a level of independence for the decisions made and directions taken by the OCQAS.
The Credentials Validation Service was the first of the quality assurance services to be developed and began operation in February 2005. The mandate from the provincial government for CVS (Minister’s Binding Policy Directive, 2003) is to:
- provide reasonable assurance that all postsecondary programs of instruction, regardless of funding source, conform to the Credentials Framework (Minister’s Binding Policy Directive, 2003, Appendix A) and system-wide titling principles; and,
- maintain the integrity of College credentials province wide.
The need for such a body as the CVS became evident as the impact of this legislation (Minister’s Binding Policy Directive, 2003) affected college programs in a number of significant ways. First, the responsibility for the approval of programs was moved from the Minister to each college’s local Board of Governors, thus creating 24 separate approval bodies across the province, where there had previously been one central approval body (government).
Second, the legislation and supporting policy framework changed the basis of credentialing from a time-based approach to an outcomes-based approach. This change represented a huge change for the way colleges thought about, planned for, and developed their programs, and was accompanied by a third change: that being a renaming of the credentials being offered.
The former time-based approach to credentialing that colleges had been using since their inception in 1967 meant that a college would grant a certificate to the graduate of any program that was less than four semesters in length and a diploma to any graduate of a program that was four or more semesters in length. Under the outcomes-based approach to credentialing, colleges needed to articulate a level of outcomes that showed the academic rigour and the breadth and depth of the program first, and then this set of outcomes would essentially determine the credential to be conferred.
The issues related to credentialing, the outcomes-based definition of the credentials and the name of the credentials, were supported by the introduction of the Credentials Framework (Minister’s Binding Policy Directive, 2003). The policy stipulates that colleges must grant credentials consistent with the Credentials Framework. Thus, the CVS became the overseer of this new process, consistent with the expectations and mandate spelled out for it in the policy directive.
The process undertaken by CVS to meet this mandate is one that ensures all programs of instruction meet or exceed the requirements of the government’s expectations. By validating all new and revised programs, assurances are given that despite the fact there are now 24 different and separate “approval” bodies across the province (through the local Board of Governors at each college), all similar programs will have similar titles, will meet similar outcomes , and result in the conferring of the same credential for graduates.
Since its inception, the CVS has processed, on average, 200 proposals a year, about 80% of which are for new programs. The remaining 20% of the activity is for programs that, as a result of a program review at the college, have resulted in the need for a change of title or credential. In addition to the actual validation of these programs, the CVS provides training and consultation to the staff at the colleges in the areas of writing program learning outcomes and program development.
At the program development and program quality level CVS is similar to other jurisdictions and operations which “approve” programs offered by higher education institutions (colleges and universities). In Canada the majority of these approval bodies and operations are government agencies or quasi-government/arms-length agencies such as the Maritime Provinces Higher Education Council (MPHEC), Campus Alberta, or the British Columbia Council on Admissions and Transfers (BCCAT). As mentioned earlier, the Ontario experience is even more diverse in that there are different bodies with different mandates for colleges (OCQAS) and for universities (CUCQA) as well as the provincial bodies such as the Higher Education Quality Council of Ontario (HEQCO) and the Postsecondary Education Quality Assessment Board (PEQAB). It is interesting to note that while CVS has been operating since 2005 the quality council (i.e., OUCQA) for the university sector, with its parallel structures and functions, began operations only in July 2010 when it essentially replaced the responsibilities of the Undergraduate Program Review and Assessment Committee (UPRAC) and the Ontario Council of Graduate Studies (OCGS).
While these similarities at the program quality level are noted, it is at the institutional quality level that the differences really emerge between what is happening with the colleges in Ontario and the other higher education institutions in the province and across the country.
At the institutional level, the OCQAS has developed and operates the Program Quality Assurance Process Audit (PQAPA). This, the second aspect of the quality assurance service developed and implemented is a critical component of the colleges’ self-regulatory mechanism. It began as, and continues to operate as, a mechanism to ensure the existence and operation of institutional-level quality assurance policies, practices, and procedures in every public college in Ontario. The process, which is an academic audit process, made a deliberate choice to focus on quality assurance and not on quality assessment as these two terms have different meanings and focus in the literature on quality in higher education. Quality assessment starts from the premise that quality or quality systems need to be found, or that institutions need to be assessed to determine if and where quality exists. Quality assurance, on the other hand, starts with the belief that quality and quality systems exist in institutions, and the main focus of the quality assurance process is to find the evidence to support this assumption.
Staff of the OCQAS, under the direction of the Management Board and a group of vice-presidents academic (VPAs) from the colleges looked at other jurisdictions in the world that were doing institutional-level quality assurance in an attempt to find a suitable or comparable model that could be used with the colleges in Ontario. This research was carried out against the backdrop of a set of guiding principles (OCQAS PQAPA Orientation Manual) that had been developed by the Committee of Presidents and given to the Management Board. As a result there was no other exact model that would meet the guidelines and meet the needs of the colleges. However, there were some consistent themes around what high quality institutions looked like and how they operated that began to emerge as various quality assurance systems and models were considered. As a result, the PQAPA model was built on a set of themes (or as were called quality characteristics or criteria) that were gleaned from this research (OCQAS, PQAPA Orientation Manual).
In 2006 a model for the academic audits was approved for implementation as a pilot project involving 5 of the 24 colleges. While it began as a pilot, it was understood that full implementation would be done on a five year cycle with 5 colleges undergoing audits each year. In this first round of audits OCQAS asked for volunteer colleges to be part of the pilot. Once identified, the colleges were provided with some orientation and training to familiarize them with the quality criteria and the process and they were then asked to rate themselves as having Met, Partially Met, or Not Met the five quality criteria that had been agreed to. Once they had completed this self-assessment we asked them to submit the report and the evidence they had that would substantiate their ratings. Once this documentation was received it was provided to external audit teams that had been trained and selected by OCQAS, and the audit teams reviewed the documentation submitted by each college and then conducted a site visit. There were five different audit teams – one per college – and they too had received orientation and training prior to their selection and deployment. The focus of the audit was for these audit teams to audit the institutions’ self-study and evidence to establish and confirm that the colleges were doing what they said they were doing in the area of quality assurance.
The results of these five audits are depicted in Table 1:
|# of Colleges||# of criteria Met||# of criteria Partially Met||# of Criteria Not Met|
Following this round of audits, the Management Board engaged Dr. William Massy, an external expert to evaluate, “…the model and its implementation through the pilot round…” (OCQAS website). As a result of this review and evaluation some changes were made to the process to increase the rigour and clarity for the colleges and for the audit teams. With these changes in place a full implementation of the PQAPA was undertaken and colleges had the opportunity to ‘volunteer’ for when they would be ‘ready’ to complete the self-study and the audit process.
With a schedule in place that was primarily determined by the remaining 19 colleges themselves, and on the understanding that we would do five audits per year, the PQAPA process was implemented between September 2007 and May 2010. The results of these audits are presented in Table 2:
Table 2: Results of PQAPA Audit of 19 Ontario Colleges
|# of Colleges||# of criteria Met||# of criteria Partially Met||# of Criteria Not Met|
These results were not, in the opinion of the Management Board, surprising in that they showed the colleges to be at various stages of “readiness” in terms of their quality assurance systems and, as this model was developed on the premise of it being a continuous improvement model, each college now had documented evidence to show areas where they were doing well and areas where there needed to be positive changes made if they were to improve their quality systems.
At the end of the last round of audits (which meant that all colleges had been through one audit), the Management Board again commissioned an external review and evaluation of their overall operations (including CVS and the PQAPA). The review was conducted by a three person review committee chaired by Dr. John Randall. It was anticipated that after one complete set of audits and five years of operation, some changes would need to be made to the quality assurance service. The challenge faced by the Management Board was how to continuously improve a continuous improvement model without losing momentum and without disadvantaging any college in the process. As there had been improvements and refinements made to the audit process from the pilot round to the final round, it was decided that rather than have a year when no audits would be done (anticipating that as changes were to be made to the process or the model, it would take a year to implement the changes and orient the colleges to the changes) the five colleges that were involved in the pilot project round would be audited again, using the improved audit process. Therefore in 2010-2011, the five colleges that had been part of the pilot project implementation round of audits in 2006 were audited for the second time. The results of this round of audits are presented in Table 3:
Table 3: PQAPA Audit results of 2011audit of the 5 colleges involved in the initial pilot phase
|# of Colleges||# of criteria Met||# of criteria Partially Met||# of Criteria Not Met|
One of the interesting findings from this round of audits was the fact that while there were again 2 colleges deemed to have met all five criteria, they were not the same two colleges that had met all five criteria in the pilot round. This provided clear evidence that while two of the colleges did in fact show continuous improvement, three of the colleges did not based on the revised audit criteria and process.
What does this really mean in terms of quality processes in Ontario’s public colleges, and what have we learned from this? The results are consistent with expectations and are useful to show, on the basis of validated evidence, there is a commitment on the part of the colleges to quality and to continuous improvement. When we consider the results from all the audits done (29) between 2006 and 2011 we see the following picture (Table 4).
Table 4: Results of 29* college audits (2005-2011)
|# of Colleges||# of criteria Met||# of criteria Partially Met||# of Criteria Not Met|
*This includes 2 audit rounds of the initial 5 colleges in the pilot phase, and one audit of each of the remaining 19 colleges
The biggest surprise in this is that it was in the area of program quality review (Criterion 5) where colleges had the most difficulty meeting the expectations and requirements. Further investigation into this finding revealed that it really came down to the fact that colleges were, in this case, unable to find any credible evidence to support the notion, or the fact, that quality work was happening. As well, it underscored the importance of being able to produce evidence that supports the ratings assigned to the Criteria in the PQAPA self-assessment process.
Following the review and evaluation of the OCQAS by Dr. John Randall and his team in June 2010, a report containing 20 recommendations was submitted to the Management Board. These recommendations were considered by the Management Board over that summer and, at their meeting in September 2010 decisions were taken on the recommendations and plans were developed for the next phase of the quality assurance service.
On the CVS-front, there was little change recommended. One of the issues raised was that the CVS had two separate “images” among various groups of stakeholders. One was that of a gate-keeper with a compliance function; the other was that of consultant and facilitator. As Randall pointed out in his report to the Management Board, “[S]triking a balance between gate-keeping and advising is a challenge which faces all organizations and individuals with compliance, accreditation or regulatory functions….In our judgment…CVS strikes a balance which enables it to uphold the wider public interest whilst being appropriately helpful…” (Randall et al, 2012). While it was initially a surprise to the staff of CVS to know these two expectations were held by various stakeholder groups, there was some satisfaction in knowing they were doing it right and were encouraged to continue their functions.
In relation to the PQAPA, there were some specific recommendations that the Management Board embraced and implemented. The Board believed that, as Randall had underscored, the OCQAS had most of the component pieces in place to become a stand-alone, fully-recognized accrediting agency. To pursue this idea the Board would have to expand its membership and include members who were external to the college system.
This was accomplished by the initial addition of two membership places on the Board: one representing a provincial student association (College Student Alliance); and, one place to be filled by a business/industry person with experience in quality assurance. The expanded Board began with the meeting in September of 2010. Further discussions and plans continue at the Management Board to move Board membership to include more external members and achieve a balance of members between those who are “internal” to the college system and those who are “external”.
A second area of change was to the actual quality criteria used in the quality audits. This change was to add one more quality criterion that would deal specifically with the existence of program-level learning outcomes in place for every program offered by the colleges. In the initial criteria and requirements, there were references to the existence of learning outcomes spread across many of the criteria, without there being a major focus specifically on this; this needed to change. As a result there are now 6 quality criteria, one that specifically focuses on, and requires the colleges to have program-level learning outcomes for every program of instruction offered.
There were some other minor revisions made to some of the existing requirements found in the other criteria; revisions that are designed to provide more clarity of expectation for colleges and audit teams and to improve the overall audit process.
A third recommendation that was acted on by the Management Board was to add an additional staff position to the Service. From its inception in 2005, the staffing complement had been one full time position and one part-time bilingual position to provide language specific service (French) to our two French-language colleges. It was recommended that in order for the Service to move forward and to continue to improve and develop into a fully recognized quality agency, additional staff would be required. Additional staffing was added in November of 2011.
A fourth area of recommendations that the Management Board acted upon was a commitment to move towards establishing a model of accreditation for the colleges. This commitment by the Management Board was ground-breaking for a number of reasons:
- Canada has no national system for accrediting institutions of higher education, a point that was stressed by Oldford in her work, “Exploring Options for Institutional Accreditation in Canadian Post-Secondary Education” (2006)
- While there is a great deal of program-level accreditation happening in both colleges and universities, any institution that is seeking accreditation at this time must apply to an accrediting body outside of Canada.
- For colleges the challenge was on the international and global level where “accreditation” is a given and is understood as a mark of quality/excellence. By providing the opportunity for colleges to become accredited by a recognized Canadian (local or national) accrediting agency, would allow colleges to function on a level playing field in terms of international student recruitment and program development.
The first step in this process was for the OCQAS to apply to the International Network of Quality Assurance Agencies in Higher Education (INQAAHE) to be recognized as meeting the Guidelines of Good Practice published by INQAAHE. To be recognized by this international agency as meeting their standards would provide the recognition needed by OCQAS to become an accrediting agency. This recognition by INQAAHE was granted to OCQAS in December 2011. At that time, OCQAS was the seventh member agency, worldwide, to receive this standard of recognition, and the first in Canada and the second in North America.
Following this recognition a proposal was developed by the Management Board for the colleges’ Committee of Presidents to obtain their permission to move the PQAPA from an academic audit model to an accreditation model. In May 2012 the Committee of Presidents approved this plan with an implementation date of September 2015.
The intervening time (between 2012 and 2015) will allow the OCQAS to complete a second round of audits with all the colleges, thus providing a solid basis for colleges to prepare for the accreditation process. It will also allow the OCQAS time to provide orientation and training to the colleges, as well as to fully develop a strong pool of accreditors.
Clearly this move from audit to accreditation is a big step and one that will be of benefit to the colleges and continue to position them at the forefront of quality work in higher education in Ontario as well as within Canada and on the global stage.
Canadian Council on Learning (November 24, 2009). Challenges in Canadian Post-secondary Education: Up to Par: The Challenge of Demonstrating Quality in Post-secondary Education.
Higher Education Quality Council of Ontario. (HEQCO) www.heqco.ca
International Network of Quality Assurance Agencies in Higher Education http://www.inqaahe.org/
Minister’s Binding Policy Directive Framework for Programs of Instruction, Government of Ontario, 2003.
Oldford, S. A., (August 2006). Exploring Options for Institutional Accreditation in Canadian Post-Secondary Education, University of Victoria
Ontario Colleges of Applied Arts and Technology Act, 2002 http://www.e-laws.gov.on.ca/html/statutes/english/elaws_statutes_02o08f_e.htm
Postsecondary Education Quality Assessment Board (PEQB) www.peqab.ca/
Ontario College Quality Assessment Service, www.ocqas.org
Ontario College Quality Assessment Service. Program Quality Assurance Process Audit Orientation Manual, www.ocqas.org
Randall, J. et al, Review of Ontario College Quality Assurance Service: Report of the Independent External Panel, June 2012. The entire report can be found on the website www.ocqas.org
Tim Klassen, Director, Ontario College Quality Assurance Service; he may be contacted at email@example.com