Spring 2005 - Volume 8 Number 2
An Experimental Design to Study the Effectiveness of PBL in Higher Education, in First Year Science Students at a University in Peru, South America M.A. Lic. Abstract
An experimental study was designed to study the effectiveness of Problem Based Learning (PBL) in the context of higher education in an urban-city university in Lima, Peru. In the fall semester of 2004, eleven sections of Chemistry 1 were offered to first year students in the College of Science at this University. In six of these eleven sections students were exposed to PBL; and in the other five sections, to instruction based on lectures mixed with small group activities. There were near 60 students in each section; approximately 660 students participated in the study. The basic measure of the study was a pre and post-test that incorporated questions for each of the levels of Bloom's taxonomy. Students in the PBL sections scored statistically significantly higher in the post-test for the higher order of thinking skill items (analysis-synthesis and evaluation), than students in the non-PBL sections. The latter did better in the lower skill items (knowledge and application). The hypothesis that PBL develops higher order of thinking skills among first year college students, more effectively than other non- PBL approaches, was strongly supported by the data.
Theoretical Section Literature Review
A Different Way of Learning
Boud and Feletti (1997) suggest that the discussion about instruction in the context of higher education should be addressed from the perspective of the future role of the students in society. If students are expected to contribute to society, they will need to acquire more than storage of factual subject matter knowledge related to their professions. Students will also have to deal with the world of industry and business in a context of diversity and change. Besides the subject matter knowledge of their profession, they will require other types of knowledge and skills: critical reasoning, logical and analytical approach to problems, reasoned decision making, individual and team communication skills, and skills of self-evaluation.
In June 1994, the Wingspread Conference brought together state and policy makers, leaders of higher education, and accreditation communities to discuss the issue of quality in undergraduate education. The discussion was based on the assertion that "substantial improvement in American undergraduate education was needed in order to prepare students to function successfully in current business and industrial environments" (Duch, Groh and Allen, p.4). The conference developed the following list of characteristics of quality performance in college of university graduates: high skill levels in communication and use of technology; the ability to arrive at informed judgments and function in a global community through flexibility and adaptability; technical competence in a given field; finally, the ability to deploy all of the previous characteristics to address specific problems in complex, real world settings (Duch, Groh and Alleen p.5).
Problem Based Learning and Cooperative Learning at Higher Education
Besides the subject matter knowledge of their profession, students in college need to acquire skills to communicate their ideas more effectively: informed and critical reasoning, decision-making, self-evaluation and so on. At its most fundamental level, Problem Based Learning (PBL) is characterized by the use of "real world" problems as a context for students to learn critical thinking, problem solving skills, and to acquire knowledge of the essential concepts of the course (Duch, 2002, p.7).
Through PBL students learn critical thinking and problem solving skills while they acquire knowledge of the essential concepts of the course. Through Cooperative Learning (CL) instructors promote individual and group accountability and ensure the teaching of social skills. Through a careful class planning, instructors make sure that each student perceives that he or she is linked with others and that students take the time to engage in group processing (group work evaluation), and reflection on their learning. "While (CL) is never easy to implement, when all the critical elements are in place, it is very powerful" (Johnson & Johnson, p.28). Because of their characteristics, we argue that PBL and CL are two methods of instruction that can help develop the necessary skills for the work place and the professional life in undergraduates.
Bloom's Taxonomy and the Classification of Learning Objectives.
One way to look at how PBL and CL develop higher order of thinking skills among students is to focus on the performance of higher order of thinking items of students in the context of a test. Benjamin Bloom (1956) and his colleagues developed a set of hierarchical learning outcomes in which different levels of thinking were organized from top to bottom. This taxonomy has proved to be helpful to instructors in the process of teaching and evaluation of students' learning. It has let them realize, for example, at what level (e.g. comprehension or analysis) are students assimilating the course contents. In Bloom's taxonomy objectives are divided into three domains: cognitive, affective, and psychomotor. The best known and more disseminated in the context of upper curricular levels, is the cognitive domain.
In this study we have focused on the cognitive domain; and within this cognitive domain, we have looked at how students perform differently on the lower-order skills -knowledge, comprehension and application-; and on the higher order skills -analysis synthesis, and evaluation. We are defining knowledge as the ability to remember previously known material; comprehension, as the ability to grasp the meaning of material; and application, as the ability to use learned material in new concrete situations; i.e. the applications of rules, methods, concepts, laws or theories. Knowledge, comprehension and application are considered lower levels of thinking skills in Bloom's Taxonomy. Analysis refers to the ability of breaking down material into its components, and synthesis to the ability of putting parts together to form a new whole. The instructors participating in this study -who have at the same time designed the test-have considered these two abilities at the same level of difficulty (like two different sides of the same coin). Evaluation is concerned with the ability to judge upon defined criteria, within a given purpose and in a given context. Analysis-synthesis and evaluation are considered higher levels of thinking skills in Bloom's taxonomy.Significance of the Study and Research Question
It has been argued at this University in Perú, (from a previous survey done with professors of the Colleges of Arts and Sciences) that students in the college of science are only learning at the levels of knowledge and comprehension. Contrary to that, students in the College of Arts and Humanities seem to be developing through instruction higher order of thinking skills, such as analysis and evaluation. From this situation the main question of this study arises: How can the teaching of introductory science courses in the college of science at this University be improved, so that higher order of thinking skills can be promoted through instruction? Our research hypothesis is that PBL develops higher order of thinking skills among first year college students, more effectively than other non- PBL approaches. To make our claim more credible we have designed an instrument in which the items correspond to each of the different levels in Bloom´s taxonomy. We believed that this instrument would help us look at the differences on the performance of students focusing on the performance of students in the higher orders of thinking skills.Research Method Research Design
During the fall semester of 2004, the professors at the Department of Chemistry in the College of Science at PUCP, offered 11 sections of Chemistry 1 to first year college students who had just finished high school. In six of these eleven sections the content of the course was taught using PBL (characterized by the use of complex real world problems and team work). In the other five the same content was taught using a different type instruction (characterized by the delivery of lectures mixed with small group activities). This is the context in which an experimental study has been designed, with a control group (the five non-PBL sections) and an intervention group (the six PBL sections). The basic measure was a pre and post test where subjects have been randomly assigned to both conditions. This test has been designed by the professors of both methods of instruction including questions for each of the levels of Bloom's taxonomy. Also a survey has been designed to check if the two methods of instruction were implemented as described by the instructors in both conditions. Descriptive statistics and t-tests have been performed in order to see if the differences between the means in both conditions were due to the treatment or not.Sample
The population that the sample represents is first year college students majoring in science who have just left high school, and enrolled in the course of Chemistry 1. There are approximately 60 students in each of these 11 sections and they have been randomly assigned to each section (see table 2). Students have also been equally distributed among classes depending on their scores in the admission exam to the university. This university has four different admission systems (la prueba del talento1, la prueba de la primera opción2, CEPRE3, and Bachillerato4). Students have been ranked according to their qualifications in each of these different admission systems from top to bottom. The university admissions office has assigned students to each of the eleven sections of Chemistry 1, looking at their scores and making sure that there is a normal distribution in each class (see Appendix 1). There are approximately 660 students participating in this study.
The study has been conducted within the standards for ethical research at this University. For this purpose, we held several meetings with the central administration, the Dean of the College of Science, and the people from the Dirección Académica de Investigación, which is the instance at this University that supervises all the research projects that take place in campus. Students also signed a consent form in which they were told what their participation in the study demanded from them, along with the purpose and objectives of the study.Measures
The basic measure in this study was a test designed by the instructors in both conditions and a survey to measure the fidelity of the implementation of the two methods of instruction.
Instructors in the two conditions met in the month of February and designed the test. To design the test the instructors in both conditions considered two important aspects: (1) Bloom's taxonomy - they have elaborated questions for each of the levels of Bloom's taxonomy. (2) The inclusion of two basic concepts of the course: "unit mole" and "stoichiometry". There is one question for each of these two topics at the levels of knowledge, comprehension, and application. There is one question that integrates both concepts (unit mole and stoichiometry) for the levels of analysis-synthesis, and evaluation. It is also important to note that, due to the fact that the instructors considered syntheses and analysis at the same level of difficulty; there is only one question that implies both mental processes (synthesis and analysis).
There are in total eight questions, six belong to the lower order of thinking skills (knowledge, comprehension, and application) and two questions correspond to the higher order of thinking skills (synthesis and analysis- considered both in one item- and evaluation). The measure of reliability of the post test estimated by Cronbach's alpha turned out to be .66.
The following are a couple of examples of a question corresponding to the lower order of thinking skills, and a higher order item:
Question # 3 - comprehension level:
If the equation for the combustion of propane (C3H8) is the following C3H8 + 5 O2 > 3 CO2 + 4 H2O
Then we can state that:
Question # 8 - evaluation level:
Thanks to the findings and discoveries of science, a trip of a whole crew to Mars becomes more plausible. Suppose that in order to explore a broad surface of the red planet it has been decided to send a vehicle propelled by an internal combustion engine with a capacity for 10 moles of fuel. Recall that Mars's atmosphere does not have oxygen, so it will have to be sent from planet Earth along with the fuel. Decide which fuel would be the most appropriate between gasoline (C8H18) and ethanol (C2H6O). Justify your answer.
The last two questions in the test were open ended questions as opposed to the first six questions that were multiple choice items (see Appendix 2). Finally, it is also important to mention that the instructors developed a rubric to evaluate the two last open ended items of the test. The teacher assistants in each section using this rubric corrected the tests (pre and post); the professors themselves have not participated directly in scoring the tests. The scale for each question ranged from 0 to 10 points. The questions in the lower levels of Bloom taxonomy are multiple choice questions; therefore, the only possible outcome was 0 or 10. In the other open ended questions students have received the values of 0, 5 or 10 according to the criteria of evaluation set on the rubric.
The second measure was a survey to measure the fidelity of the implementation of the two methods of instruction. The survey had six items describing the main features for each methodology as stated by the faculty. Students, according to the instruction they received, filled either survey (PBL or non PBL instruction) ranking each item from 0 (I do not agree) to 4 (I strongly agree). The survey for each method of instruction is also presented in the appendix section (see Appendix 3). The measure of reliability for the survey estimated by Cronbach's alpha turned out to be .81, which is a satisfactory index. The results of these surveys are presented in the findings section of this paper.Description of Treatment and Control Groups
The following is a description of both methods of instruction. It is important to notice that the same content in terms of learning units has been covered throughout the semester in both methods of instruction (see Appendix 4).
There were two sessions each week. The professors lectured when the class first met, basically using overheads and leaving time for questions (approximately 10 minutes). The lecture lasted the whole period of the two-hour class. During the second session, students formed small groups as required to solve exercises and problems. Within the group work students first solved these exercise or problems in pairs, and then the whole group would reconvene to solve a final task. All of this took place under the supervision of the instructor and the teacher assistants in the class.
At the beginning of the learning unit, students were presented with a problem (scenario, case) in their groups. The problem was complex, and it was also a real world problem expected to be solved by the whole group and not merely by an individual. A problem like this has been given to students for each unit of learning in Chemistry 1. Students, guided by the instructors, have done some readings, solved a series of activities, and held some discussions that would lead to the solution of the problem. Instructors informed students where to find the resources and other materials needed to solve the activities that were implied in the problem. Students had to work during the class sessions, individually or in teams with the orientation of the instructor and the teacher assistants around activities designed by the instructors in both sessions.Findings Overview of Statistical Procedures
We have considered two basic measures: the test and the surveys for the verification of the two different types of instructions. For the test (used as a pre and post- test) we have performed the t-tests at each of the levels of Bloom's taxonomy, to make our hypothesis that PBL develops higher order of thinking skills among first year college students, more effectively than other non- PBL approaches, more credible. We are also showing tables with descriptive statistics: means and standard deviations.
For the surveys we have also used descriptive statistics to see how students in both conditions ranked each of the six items in the survey.
The Pre and Post-Test
The pre text was given at the beginning of the semester ,on the first day of class. The post-test, on the last session when students met with the instructor in the semester. There were approximately sixty students in each section, 6 sections were randomly assigned to the intervention group and five, to the control group. There were 364 students in the PBL group and 304 in the non-PBL group (see tables 1 and 2). Table number three, presents the descriptive statistics for each question and the results of the independent sample t-tests.
Significance level was set up at ≤ 0.05. As the table shows, all p-values for the higher order of thinking skills in the post-test (items seven and eight) turned out to be statistically significant. The p-value for item seven (analysis-synthesis) is 0.00 (p. < 0.05) and the p-value for item eight (evaluation) is 0.002 (p. < 0.05). In these two items the means for students in the PBL condition was higher (see table 3).
A serendipitous finding in this study has been that the p-values for items one (knowledge) and five (application) turned out to be statistically significant. The p-value for item one was 0.001 (p.< 0.05) and the p-value for item five was 0.003 (p.<0.05) . In these two items the mean of the students in the non-PBL condition was higher than the mean of the students in the PBL condition, (see table 3 for means and mean differences).
We have finally combined all the levels of Bloom's taxonomy (see table 4) into "created variables": lower skills (knowledge, comprehension and application) and higher skills (analysis-synthesis and evaluation). We have created these variables to look for similarities and differences between the groups in the pre and post-tests. The following table (table number four) shows the results of the descriptive statistics and t-test for the "created variables" in the pre-test and post-test.
When we combined the levels of Bloom's taxonomy into these "created variables" we found that the p-values for the pre-test turned out not to be statistically significant: 0.19 for the lower skills, and 0.52 for the higher order skills. No difference was found in the pre test between the conditions (PBL and no-PBL). The only statistical significance between the groups was found in the post test. All p-values in the post test turned out to be statistically significant: 0.04 (p< 0.05) for the lower skills and 0.00 (p. <0.05) for the higher order of thinking skills. Therefore we can conclude, that the observed difference in the post-test between the means of the two conditions in both ( the lower and higher order of thinking skills) were due to the treatment and not to chance alone.
A total number of 172 students answered the survey, 128 students in the PBL group and 44 students in the non-PBL group. Unfortunately the surveys were distributed to students at the very end of the semester, and not all of the students were available to answer them. The data in the table below (table number 6) demonstrate how students in both conditions ranked each of the six items in the survey. In both conditions the (rounded) mean score for each item was 3 ("I agree").These findings support the assumption that the treatments were applied as designed.
Our hypothesis that PBL develops higher order of thinking skills among first year college students more effectively, than other non- PBL approaches , was strongly supported by the data. A serendipitous finding in this study has been the fact that other non-PBL approaches to instruction at higher education (in the context of science teaching on first year college students) contribute to develop lower order of thinking skills among students such as memory and application.
Even though our hypothesis was strongly supported by the data we can also consider the possibility of a rival explanation to our findings. A possible rival explanation could be related to a problem of instrumentation. There could be a concern with the issue of "double blind" correction. Although the instructors did not score the tests, and these were scored by the teacher assistants using a rubric; some of the teacher assistants knew the hypothesis of our study and were also aware of which group had the experimental treatment. Therefore a blind correction with more than one process of scoring -the teacher assistants and maybe other instructors that did not know which group had the experimental treatment- would have contributed to make our findings even more credible.Limitations
In the test, we have counted on six items to measure the lower level skills; and only on two items to measure the higher order skills. Therefore, future research is still encouraged as well as the replication of this study with an instrument that includes the same number of questions for each of the levels of Bloom's taxonomy. The reliability for these two higher order items of the test was estimated by Cronbach's alpha, and turned out to be 0.53.Conclusions
The findings in this study suggest two different things. On the one hand, students in the non-PBL condition scored statistically significantly higher 0.04 (p. <0.05) in the post-test in the low level items; these questions required from them the skills of memory, and the ability to apply concepts into different settings. On the other hand, students in the PBL condition scored statistically significantly higher 0.00 (p. <0.05) in the higher order items. These questions required from them, the ability to do analysis-synthesis and evaluation. This is particularly interesting in the context of higher education since the latter, are the skills that we want our students to develop in college.Recommendations
In order to better contribute to the literature and the research that claims that PBL promotes higher order of thinking skills among science college students, a replication of this study with a more reliable instrument is recommended as well as more research in this mater. Now, if future research confirms our findings, then more dissemination of PBL is encouraged among first year college students in science.Acknowledgements:
This research was funded partially by the office of the Vice Academic Rectory at Pontificia Universidad Católica del Perú (PUCP) and partially by the Institute for Transforming Undergraduate Education (ITUE) at the University of Delaware (UD) in a collaborative project funded by ALO/USAID. We also want to thank the chemistry faculty members at PUCP: Juan Carlos Cedrón, Miguel Chong, Javier Nakamatzu, Juana Robles and Vilma Ulco, for their collaboration in this research. We are especially thankful for their effort in the process of designing the test as well as the support provided during the semester in which the study took place. We also want to thank very especially Dr. Chris Clark for his feedback in the process of the study design; Dr. James Raths, for the orientation provided in the process of analyzing the data, writing the research report, and for reviewing this paper on several occasions; and Dr. Radna Nandakumadar for her valuable comments on the statistics. Finally we want to thank Dr. George Watson for his continuous efforts on disseminating innovative approaches of teaching at university level throughout Latin America, as well as promoting the implementation of measurement techniques and research together with this effort of faculty professional development.References: Mc. Keachie's,W.,(2002).Teaching Tips. Strategies, Research, and Theory for College and University Teachers. Houghton Mifflin Company. Boston. New York.
Duch, B.,Grog,S.,Allen,D.,(2001). The Power of Problem Based Learning. A practical How to for teaching Undergraduate Course in any Discipline. Sterling, VA: Stylus Publishing.
Boud,D., and Feletti, G., (1997). The Challenge of Problem-Based Learning, New York: St. Martin's Press.
Johnson,D., Johnson,R., Smith,K., (1998).Cooperative Learning Returns to College. What evidence is there that it works. In Change July/August 1998.
Bloom B., (Ed.). (1956). Taxonomy of Educational objectives, Handbook I: Cognitive Domain. New York:Longmans, Green.
Anderson, L.W.,&Krathwoohl, D.R. (Eds.).(2001). A taxonomy for learning , teaching and assessment: A revision of Bloom's taxonomy of educational objectives. New York: Longman.
Woolfolk, A., and Kolter, W., (2003). Instructional Leadership.A learning-centerd guide.Allyn and Bacon, Boston.
Pre and Post Test-English Version
Pre and Post Test-Spanish Version
3.1. PBL Survey:
Evaluate each of the following statements considering the type of instruction you have received this semester:
3.2. Survey Non-PBL
Evaluate each of the following statements considering the type of instruction you have received this semester:
Endnotes 1. The Prueba del Talento is taken by students that have already finished high school. It is like an SAT, a standardized test that measures skills; if students make a specific score they enter this University.
2. The Prueba de la primera opción is taken by students that are currently in high school (senior year) it consists on the same type of standardized testing that measures skills as the previous one, but students are also asked to participate in an interview. Students who do poorly in the interview will not qualify for campus and will have to attend a parallel program specially designed to help students acquire the necessary skills to succeed in the first years of college. After successfully completing the program of Ciclo Inicial students can go for their freshmen or sophomore years to campus leaving this parallel program-ciclo inicial- behind.
3. CEPRE is a training program designed to help students that are currently in High school, or that have finished high school, or that have failed in their attempt be admitted at this University. It prepares them for the admission exam. Some students that do well in this training program can also enter this University without taking the admission exam.
4. Bachillerato, consists of a group of International programs for High School diploma like Maturita, Baccalauréat, International Baccalaureate, Habitur, and others that are currently being implemented at some private High Schools in Lima. These programs are funded with federal money from the governments of Italy, France, Germany, The United Kingdom, USA and others. If a student has successfully completed high school in any of the schools that have these programs, then they have free access to PUCP and they do not need to take any sort of admission exam.
Maria Teresa Moreno Alcázar is a research assistant at ITUE (Institute for Transforming Undergraduate Education) at the University of Delaware. She is currently a Ph.D. candidate at the School of Education in UD as well. Maria is involved in the process of training and dissemination of Problem Based Learning in South America. Her research interests are in the areas of curriculum, cognition and instruction. She can be reached at email@example.com
Victoria Landa Fitzgerald is a Chemistry faculty member at Pontificia Universidad Católica, del Peru PUCP in the College of Science at this university. She is also a member of MAGIS PUCP which is an institute at PUCP that delivers workshops for professional development of faculty members , and also promotes the reflection and research about Higher Education. She is currently conducting workshops on PBL and Cooperative Learning for the teaching of science at university level and can be reached at firstname.lastname@example.org
• The views expressed by the authors are those of the authors and do not necessarily reflect those of The College Quarterly or of Seneca College.
Copyright © 2005 - The College Quarterly, Seneca College of Applied Arts and Technology