Applied Research (AR) in Canadian community colleges is driven by a mandate, via the collective voice of Colleges and Institutes Canada – a national voluntary membership association of publicly supported colleges and related institutions – to address issues of interest to industry, government, and/or community. AR is supported through significant federal and provincial level funding mechanisms as well as funding from the private sector. Performance measurement tools have largely been developed by government agencies such as the Natural Sciences and Engineering Research Council (NSERC) and the National Research Council of Canada (NRC), which are external to the colleges that engage in AR. This paper presents an overview of AR in Canadian community colleges and institutes and provides recommendations for the development of sustainable performance measurement frameworks for AR in Canadian community colleges and institutes.

Canada is home to a network of over 150 colleges and institutes that originally developed as a result of federal legislation and financial support in the 1960s in order to foster Canada’s transition to an industrial economy. This aforementioned group of institutions is represented by Colleges and Institutes Canada (formerly known as Association of Canadian Community Colleges), a voluntary membership association of colleges and related institutions. This shift occurred primarily through certificate and diploma level trades and other skills-based training with a direct connection to industry competency requirements (Madder, 2006; Fisher, 2009).

In recent years, Canada’s community colleges and institutes have positioned themselves to respond to the emerging knowledge economy through various means, including active advocacy and support for an Applied Research agenda (ACCC, 2010). Applied Research (AR) involves the generation of new knowledge, like basic research, but is client-driven in that it seeks to address matters of direct practical relevance to industry, community, or government (Roll-Hansen, 2009; OECD, 2002).

Colleges and Institutes Canada (CIC), mentioned above, has been promoting Applied Research since 2002 (Madder, 2006). AR in the Canadian college system has developed substantially from its humble beginnings at the turn of the 21st century with little activity and virtually no formal support. By 2012, nearly 100 institutions had dedicated AR divisions with almost 5,000 industry and community research partnerships, over 1,500 faculty and staff working on AR projects, roughly 24,000 students participating in AR activities, and $179,000,000 in external funding (ACCC, 2012).

ACCC (2011) constructed an applied research college typology based on the state of development of their applied research activities, including

• institutions with neither applied research activities nor governance mechanism;
• novice innovation institutions;
• established innovation institutions; and
• integrated innovation institutions.

Federal support, through the College and Community Innovation (CCI) Program, has played a significant role in the enhancement of AR activities in the Canadian community college system. The CCI program was piloted in 2002 and made permanent in 2008. This competitive granting program is administered by the Natural Sciences and Engineering Research Council (NSERC) on behalf of NSERC, the Social Sciences and Humanities Research Council (SSHRC), and the Canadian Institutes of Health Research (CIHR). CCI’s mandate is to support colleges to leverage their existing relationships with SMEs (Small and Medium Sized Enterprises) in order to address innovation related challenges faced by those SMEs, including AR, commercialization, technology transfer and new technology adaptation/adoption (ACCC, 2013). The CCI budget has increased from approximately $15,000,000 in its inception year of 2008 to just over $30,000,000 in 2013 (Deyirmendjian et al., 2013). Other federal sources of funding for college level innovation include the Canadian Foundation for Innovation and Canada’s National Research Council. In addition, various provincial and regional sources of funds support applied research in community colleges.

Public sector performance measurement is a universal expectation across jurisdictions and at different levels of government in both Canada and the US. Scholars trace the beginnings of Canadian performance measurement back to initiatives designed to improve productivity in American cities at the turn of the 20th century (McDavid & Hawthorn, 2006). Planning, programming, and budgeting systems (PPBS) were incorporated into public sector governance in the 1960s and 1970s. This emphasis on accountability declined in popularity through the 1980s, however, a resurgence in interest in performance measurement occurred during the development of New Public Management (NPM) in the 1990’s, with an increased use of sophisticated private sector approaches to governance. Since then, performance measurement has been used as a means to facilitate effective public sector management in a number of western countries (McDavid & Hawthorn, 2006).

College research that is funded by the CCI programme requires reporting on a number of NSERC mandated performance indicators. NSERC has 6 grant types within their CCI programme and of those, only 3 require performance reporting: the Industrial Research Chairs for Colleges programme (12 indicators), the Innovation Enhancement grant (24 indicators), and the Technology Access Centre grant (9 indicators) (St Pierre, 2013). This is the primary, if not only, performance reporting requirement common to applied research at Canadian colleges and its purpose is primarily summative. An evaluation of NSERC’s CCI program by Goss Gilroy (2012) revealed that performance reporting requirements were onerous and lacking in guidance regarding level of detail required. Luke and Van den Berg (2013), in a recent presentation at the last annual ACCC Applied Research Symposium in Quebec City, indicated that Canadian colleges and NSERC are exploring ways to improve and enhance performance reporting requirements for applied research activities.

Perhaps the perception of NSERC CCI performance reporting as onerous and confusing is, in part, due to the fact that the measures have been developed by an external agency (NSERC) for summative (e.g., accountability) purposes.

Dual-purpose (formative and summative) performance measurement systems, although uncommon, offer the promise of a compromise between utility for managers and the accountability requirements of funders. The city of Lethbridge, Alberta, has balanced the uses of their performance measurement system for both formative purposes, such as internal management of process, and for summative purposes such as external reporting to council (Hildebrand & McDavid, 2011). The authors also delimit a set of conditions that facilitate more participatory approaches to performance measurement in local government, some of which are of relevance to AR performance measurement systems in Canadian colleges and institutes (Hildebrand & McDavid, 2011), including:

• Institutional size – the medium- to larger-sized governments (and in this case institutions) have more person power to implement performance measurement systems with dual purpose. Careful choice of indicators, data collection techniques, and reporting will be necessary to ensure that the process is feasible for small, medium, and large size Canadian colleges.
• Polarization – the more polarized a given political culture, the more difficult it will be to implement a dual purpose performance measurement system since performance results may – or may be perceived to – be used for punitive purposes. From personal experience, the author has noticed that faculty stakeholders tend to conflate institutional or program level performance measurement with individual performance appraisal which can, depending on the organizational culture of the given college, create reluctance to participate in an applied research performance measurement initiative.  Faculty trepidation may be minimized if managers clearly differentiate AR performance measurement activities from individual performance and if the activity is explicitly non-punitive. 
• Bottom-up approach – dual purposed systems are more likely when program managers and other stakeholders (key users of the performance measurement system) are involved in designing and implementing the system.

Another study by Pollitt et al. (2010), on performance regimes in health care systems, cautions against several negative trends that occur over time within a given performance measurement system. These include:

• Increase in the number of performance indicators – performance measurement systems tend to start with relatively few indicators that expand in number to broaden the system’s scope of coverage, the results of which are often merged into complicated aggregate measures that tend to erode public confidence in the system. An increase in the number of indicators may make the system impractical to implement, however additional indicators may need to be added as new research clusters are incorporated into a given college’s applied research activities.
• Shift from a formative approach to a summative approach – over time, performance indicators shifts from formative uses for internal improvement to externally or publically available summative uses (e.g., ranking systems or league tables). Summative uses tend to engender gaming the measures to avoid unwanted consequences of failure in high-stakes contexts. Stakeholders involved in AR in the college system will have to be on guard for performance measurement system usage pattern shifts - particularly as the programme increases in profile and public recognition - and be prepared to advocate for the continuance of a dual purpose performance measurement system.

Effective performance measurement of applied research activities in the college system have the potential to contribute greatly to both the efficiency and effectiveness of AR programs which would ultimately provide direct benefit to community college’s key stakeholders: students and regional employers. The following recommendations could help to improve the utility and functionality of community college AR performance measurement systems:

• Borrow performance evaluation frameworks and specific measures from university research, private sector innovation, and non-profit fields.  The performance of university research programs has typically been measured via peer review and bibliometric approaches (Geuna & Martin, 2003). Performance measurement of private sector innovation activity is more wholistic in that it typically involves measurement at one of more of the recognized stages of the innovation cycle including idea generation, concept development, concept evaluation, development, and implementation (du Preez & Louw, 2008). Non-profit performance measurement frameworks are typically as diverse as the non-profits that use them (Eckerd & Moulton, 2011), however many focus on both the process and the impacts of the work on the beneficiary communities. To date, most community college research has emphasized technological innovation (ACCC, 2013). Measures from the private sector innovation field are most appropriate for this type of community college applied research. As more community colleges delve into social innovation research, the frameworks and measures used by nonprofits will become increasingly relevant.  University research performance measures have the least relevance for community college applied research programs however peer review and bibliometric measures could have ancillary value as indicators of contributions to the broader applied research academic community.
• Use a mix of formative and summative approaches. Community colleges can use AR performance evaluation as both an organizational learning tool and a means to report on effectiveness and impact. ACCC (2011) described a typology of AR institutions along a continuum ranging from community colleges with no formal innovation governance structures to those with an innovation agenda that is integrated into multiple levels of the organization and the local economy. An adaptive approach to performance evaluation of community college AR would include a greater emphasis on formative measures and approaches at the early innovation stages as a means to foster institutional capacity building and therefore movement along the continuum.
• Pair institutional research tools and expertise with the needs of community college applied research offices. Most community colleges have dedicated institutional research staff. Ensuring that applied research performance measurement systems are compatible with institutional research frameworks could facilitate leveraging the aforementioned expertise for performance data collection and analysis and greater integration of AR into broader institutional planning initiatives.
• Establish a national community college applied research performance evaluation working group. Currently, performance measures in use for community college AR are determined by the primary funder, the Natural Sciences and Engineering Research Council (NSERC).  Externally mandated and generated performance measures are not uncommon, but have led to documented cases of ‘gaming’ measures (Pollitt et al., 2010) in order to avoid possible negative repercussions from donors.  A participatory approach to performance tool development, implementation, and refinement would result in greater ownership on the part of the colleges and other stakeholders and would allow for the generation and dissemination of best practices around AR performance measurement and management. This would consequently result in a more active and authentic use of the performance measurement system. Such a performance evaluation working group should consist of an array of stakeholders including college institutional and applied research administrators, research faculty, students, industry/employer partners, and representatives from funding agencies.

A sustainable performance measurement framework for AR in Canadian community colleges and institutes will borrow proven performance measurement elements from the public, private, and non-profit sectors. Coordination with internal institutional research divisions will ensure that AR is consistent with a given college’s strategic direction. A context-dependent mix of formative and summative approaches and direct involvement of college AR staff in the development of the performance measurement framework will also contribute to the long-term utility and sustainability of a given framework. Finally, the establishment of a national community college AR performance measurement working group will help to both engender ownership of these performance tools, and encourage a national community of practice.

ACCC (2013). The College and community innovation program: smart investments for business. Ottawa, Association for Canadian Community Colleges. Last accessed on November 29th, 2014 from: https://www.fanshawec.ca/sites/default/files/roles/research_publisher/
ACCC-Smart_Investments-March2013.pdf

ACCC (2012). Fact sheet: applied research at colleges and institutes 2011-2012. Last Accessed on November 29th, 2014 from: http://www.collegesinstitutes.ca/wp-content/uploads/2014/05/201302_AR-FactSheet2011-12.pdf

ACCC (2011). Productivity through innovation: applied research at Canada’s Institutes and Colleges. Last accessed on July 5th, 2014, from: http://www.accc.ca/wp-content/uploads/archive/pubs/studies
/2011_innovation_eng.pdf

ACCC (2010). Canada’s demographics and advanced skills crisis: people without jobs, jobs without people.  Submission to the House of Commons Standing Committee on Finance, pre-budget consultations 2011. Last accessed on November 29th, 2014 from: http://www.accc.ca/ftp/briefs-memoires/201008_FinanceBrief.pdf

Deyirmendjian, J., Thibault, M., & St.-Pierre, P. (2013). CCI Program Update. Presented at the ACCC Annual Applied Research Symposium, April 15th, 2013, Quebec City. Last accessed on November 29th, 2014 from: http://www.collegesinstitutes.ca/wp-content/uploads/2014/05
/ACCC2013_Applied-Research_Symp.pdf

du Preez, N.D., & Louw, L. (2008). A Framework for Managing the Innovation Process. Proceedings of the Portland International Conference on Management of Engineering and Technology, 27-31 July, 2008, p.546-558.

Eckerd, A., & Moulton, S. (2011). Heterogeneous roles and heterogeneous practices: understanding the adoption and uses of non-profit performance evaluations. American Journal of Evaluation. 32(1): 98-117. Last accessed on November 29th, 2014 from: http://aje.sagepub.com.ezproxy.library.uvic.ca/content/32/1/98.full.pdf+html

Fisher, R. (2009). A framework for research at Canadian colleges. College Quarterly. 12(4): last accessed on July 5th, 2014 from http://www.collegequarterly.ca/2009-vol12-num04-fall/fisher.html

Geuna, A. & Martin, B.R. (2003). University research evaluations and funding: an international comparison. Minerva. 41: 277-304. Last accessed on July 6th, 2014 from: http://link.springer.com.ezproxy.library.uvic.ca/article
/10.1023%2FB%3AMINE.0000005155.70870.bd

Goss Gilroy Inc. (2012). Evaluation of the College and Community Innovation Fund (CCI) presentation to the committee on research partnerships. Delivered at the ACCC Annual Applied Research Symposium, Quebec City Canada, April 15, 2013. Last accessed on July 6th, 2014 from: http://www.accc.ca/xp/index.php/en/2012/doc_download
/274-accc-2013-applied-research-symposium-presentations

Hildebrand, R., & McDavid, J.C. (2011). Joining public accountability and performance management: a case study of Lethbridge, Alberta. Canadian Public Administration. 54(1): 41-72.

Luke, R., & Van den Berge, B. (2013). Developments for performance measurement in applied research and technology development. Delivered at the ACCC Annual Applied Research Symposium, Quebec City Canada, April 15, 2013. Last accessed on July 6th, 2014 from: http://www.accc.ca
/xp /index.php/en/2012/doc_download/274-accc-2013-applied-research-symposium-presentations

Madder, J. (2006). Innovation at Canadian colleges and institutes. Ottawa: Association of Canadian Community Colleges. Last accessed on July 5th, 2014 from http://www.accc.ca/ftp/pubs/brochures/2006_a-research.pdf

McDavid, J. & Hawthorn, L.R.L. (2006). Program Evaluation and Performance Measurement: An Introduction to Practice. Thousand Oaks, CA: Sage Publications.

OECD. (2002). Frascati manual: proposed standard practice for surveys on research and experimental development. Paris: OECD. 254pp. Last accessed on July 5th, 2014 from: http://www.uis.unesco.org
/Library/Documents/OECDFrascatiManual02_en.pdf

Pollitt, C., Harrison, S., Dowswell, G., Jerak-Zuiderent, S., & Bal, R. (2010). Performance regimes in health care: institutions, critical junctures and the logic of escalation in England and the Netherlands. Evaluation. 16(1) 13-29.

Roll-Hansen, N. (2009). Why the distinction between basic (theoretical) and applied (practical) research is important in the politics of science. Centre for the Philosophy of Natural and Social Science Contingency and Dissent in Science Technical Report 04/09. 31pp. Last accessed on July 5th, 2014 from: http://www.lse.ac.uk/CPNSS/projects
/CoreResearchProjects /ContingencyDissentInScience/DP/DPRoll-HansenOnline0409.pdf

St. Pierre, P. (2013). Personal communication (NSERC-CCI program officer).

Keith Williams is Director of Adult Learning, Assiniboine Community College Adult Collegiate in Brandon, MB. He can be reached at: williamsk@assiniboine.net.