Committee Members

Hadley Wood (chair) – Vice Provost for Faculty Development and Dean of General Education, Professor of French and Literature, Becky Havens (liaison) – Vice Provost for Educational Effectiveness and Dean of the College of Social Sciences and Professional Studies; Professor of Economics, Lori Carter – Associate Professor of Mathematics and Computer Science, Karen Davis – Assistant Professor of Education; Director, PLNU, Arcadia, Kerry Fulcher – Professor of Biology; Chair, Department of Biology, Karl Martin – Professor of Literature; Chair, Department of Literature, Journalism, and Modern Languages and Barbara Taylor – Associate Professor of Nursing

Because the work on Educational Effectiveness Indicators was strongly tied to the shared curriculum, that is, to general education, this WASC committee worked very closely with the faculty as a whole and with the General Education Task Force in particular. In addition, since EEI’s need to be tested, the committee also worked in conjunction with the Assessment Committee. The involvement in this work of the faculty in the initial stages and of both these committees subsequently played an important role in the general institutional engagement in this work (Description and Summary of Faculty Participation).

Committee Charge

The EEI WASC committee was charged to develop answers to the following questions :

• What specific outcomes indicate that PLNU has successfully educated its students?
• What particular items would predict student success in these crucial areas?
• How can PLNU adequately measure those particular items?
• What are the results of those measures?

Selection of Critical Student Outcomes

During an active learning exercise and subsequent general discussion in faculty workshop, Point Loma Nazarene University faculty made a thorough list of the learning outcomes that all students receiving a PLNU degree should demonstrate. After this list was thematically organized (Summary of Faculty Work), it was presented to faculty for critique during a faculty meeting and faculty input was duly noted.

For assessment purposes, it was decided to make a clear distinction between goals (what faculty can reasonable control) and desires (what is beyond faculty control) (General Learning Outcomes). In addition, faculty noted that many of the goals selected (skill in writing, skill in communication, critical reading and thinking, for example) are skills that one spends a life pursuing. Faculty urged the committees involved in this work to think carefully about how far along the developmental road students should be by the end of their degree at Point Loma Nazarene University.

After examining a set of comparator schools selected in part for similarity of size and mission and in part to define a range of potential options, it was determined that the outcomes selected by PLNU faculty were very similar to those adhered to by the vast majority of institutions. In turn, these outcomes (thinking skills, general knowledge and attitudes conducive to successful social living) were central to the thinking found in most of the current literature consulted about general university-level education (Research Tools).

Discussion of Educational Effectiveness Indicators

During the first meetings, the committee worked to establish an understanding of the basic purpose of our task. We understood effectiveness indicators to be specific, measurable facts whose presence would be a solid predictor of success in a particular area that might have many more aspects than just the effectiveness indicator itself. It is in this sense that standardized tests such as the SAT or the ACT are effectiveness indicators of success in college. College involves far more than the skills measured in these tests, but they are good predictors.

We thus avoided focusing on coverage of all the detailed areas of general education and instead worked to identify specific, measurable items that would be excellent predictors of success in the knowledge, competencies and attitudes that the faculty had identified as expectations of all Point Loma Nazarene University graduates.

This approach aligned our work well with the suggestions present in the AAC&U document, Our Students’ Best Work : A Framework for Accountability Worthy of Our Mission (2004)(see bibliography, Research Tools). We wanted to avoid the easy solution of the currently available common tests that do not test our students’ highest skill level, particularly the critical thinking, evidence-based reasoning, integrative thinking, and problem-solving skills that interested us.

Finally, we also wanted to examine what knowledge, skills and attitudes the students had appropriated as their own. We recognize that there is a vast difference between what one knows immediately after a course and what one has actually absorbed and integrated into one’s own set of skills, one’s compilation of knowledge and one’s personal attitudes. It is this latter that we have tried to assess.

• EEI 1: 75% of students will be able to recognize theologically or biblically informed rhetorical elements appearing in non-theological contexts.

The ability to use and manipulate knowledge, to connect it to new contexts, to integrate it with other knowledge—this represents the kind of thinking we want our students to be able to do with what they have learned. Thus, rather than test a series of biblical or theological facts, we chose to focus on the student’s ability to recognize theological or biblical concepts when these concepts were present in a very different context than the one in which the student originally learned them. We believe that students can only engage in this level of thought with concepts that they have indeed appropriated as theirs. We also felt that this measured higher intellectual skills than just item recognition.

• EEI 2: 75% of students will be able to articulate a personal spiritual position and place it within a spectrum of Christian traditions.

Given the religious nature of Point Loma Nazarene University, we also felt it was important that students graduating from the university be able to at least articulate their own personal belief system and that they understand how that position related to a spectrum of religious beliefs, especially within Christianity. Although we would like all our students to leave Point Loma Nazarene University with a vibrant, personal belief in Jesus Christ, we recognize the existence and the importance of personal free will and choice. Thus, we are concerned that students understand what their personal spiritual choices are and that they have reasons for making those choices. 

• EEI 3:  Majors from 75% of the academic departments will score at or above the summative competence benchmark set by the department.

We will rely on the discipline-specific summative assessment measures developed by major departments to determine whether students have indeed developed competence in a major field. The task of measuring students’ quantitative skills has been substantially facilitated by the nature of the General Education mathematics requirement. Our General Education mathematics course is less concerned with arithmetical manipulations than with the ability to apply quantitative methods to solve real-life problems. We have thus used one of the complex problems typical of that class as a question in one of the summative General Education exams that we have constructed.

• EEI 4: 75% of students will be able to interrogate an issue from at least three different disciplinary perspectives.
• EEI 5: 75% of students will  be able to organize ideas to support a position (either in writing or orally) on summative GE exam
• EEI 6: 75% of students will be able to present objections to their own position as logical as shown on a summative GE exam.
• EEI 7: 75% of students will be able to apply quantitative methods to solving real-life problems.

We felt that there were three good predictors of critical thinking. First would be the writing skills needed to construct a well-reasoned argument—the ability to formulate sub-points in support of a position, the ability to perceive logical objections to one’s own position. Second would be the ability to apply quantitative thinking to the solution of real-life problems. Third is the ability to interrogate an issue from different disciplinary perspectives. We also felt that this measure was a good indicator of the kind of general knowledge of different disciplines that we want our students to have. We are less interested in students recognizing a set of unrelated facts than we are in their ability to understand the ways in which social scientists, artists, and scientists formulate truth. We also recognize that at age 21, with little life experience behind them, our students will not yet be the mature adult thinkers we want them to eventually become. If, however, they already understand that people from different disciplines use different kinds of evidence, ask different kinds of questions and approach the world from different perspectives, we feel confident that they are on the way to appropriating their liberal arts knowledge critically.

• EEI 8: 75% of students will be able to interrogate an issue from a different racial, class or gender position as shown on a summative GE exam.
• EEI 9: 75% of students will have had active participation in a program involving substantial contact with another culture or sub-culture
• EEI 10: 75% of those students (EEI 9) will report some sense of their own cultural otherness
• EEI 11: 75% of students will have participated in a service activity at PLNU.
• EEI 12: 75% of students will have had 10 or more opportunities to work in a team in or outside of class.

The major attitudes that we wish to examine, what we consider a kind of basic social literacy requirement, are a willingness to work with others as part of a team, an attitude of service to others, and a basic openness to diversity. Although we understand that these items are particularly difficult to measure with any accuracy, we also believe that these goals are important enough that it is important the try to assess our success in achieving them. In the case of an attitude of service and the ability to be part of a team, we felt that repeated exposure to opportunities to exercise these skills would be a good indicator of a positive attitude. We are aware, of course, that exposure can yield negative results. But we also believe that a bad experience is more likely to have a negative effect if there are few other experiences with which to balance it. In other words, the more numerous the opportunities, the more likely the student is to have developed a positive attitude.

In the case of diversity, we felt that good predictors of an openness to others who are radically different from us would be the ability to see things from the other’s perspective and the comprehension that the culture that we take for granted can be very foreign to others. We feel that both these abilities will be present in an individual able to see him or herself as culturally constructed and thus limited.

• EEI 13: 75% of students will demonstrate the ability and willingness to balance the varied aspects of their lives as shown on a senior-year assessment exam.

Finally, as a university that values the education of whole individuals, we want to determine whether our students are capable of perceiving themselves as whole persons and committed to working to lead balanced lives. We recognize that for a large number of individuals in our culture, such balance is not easily gained and needs to be consciously and purposely chosen.

Discussion of Assessment Means for Education Effectiveness Indicators

After the EEI’s were constructed, the committee turned their work over to the General Education and the Assessment committees. These two committees habitually work in concert since the Dean of General Education is a permanent member of the Assessment committee. The current plan that has been approved by both these committees is to test general education via three different home-constructed exams, two of which have already been written and given to graduating seniors, the first in April 2004 and the second in April 2005.

The first exam is a holistic writing exam in which the student needs to defend a position in a well-reasoned, evidence-based argument. The exam also asks the student to indicate what might be a Christian response to the particular issue. This aspect was included to determine how individualized or how “borrowed” a student’s belief system might be (see Appendix 8). This 60-minute exam was taken by a small number of senior volunteers (43) in April 2004 and the exams were graded by a team of faculty members in early June, using a grading rubric and the 2-person consensus method that is a standard practice among composition faculty (see Appendix 9). The results of that exam were disappointing, though not unforeseen. Although our students can indeed formulate a clear thesis about a significant topic, they are much weaker in the tasks of identifying basic sub-points needed to prove an argument, of bringing pertinent evidence to bear, and of dealing with a logical objection to their argument. The General Education committee is not concerned with the quality of the Freshman Writing class, but with the quality and quantity of writing students are asked to do during the rest of their education at Point Loma Nazarene University. The committee intends to include, in its General Education reshaping, measures that will ensure that formative writing continues past the Freshman year.

The second exam that the General Education Committee constructed is a multi-subject exam that was administered in April 2005. This exam consisted of 3 questions. One of these examined students’ comprehension of theological concepts, the second looked at their quantitative reasoning skills, and the third assessed their critical thinking and flexibility of perspective (see Appendix 10). The mathematics department has already graded the second question and we were very pleased to see that 76% of students demonstrated an ability to use appropriate quantitative methods for solving a real-life problem. Grading rubrics have been constructed for the first and third questions. When answering the question on religion, although 84% of our students could appropriately identify passages of political rhetoric in which theological concepts were embedded, and 72% could explain what theological concept was being referenced, only 47% could successfully complete the further task of indicating how the theological concept was functioning as a vehicle for separate political purposes (see Appendix 11). Although we believe that this lower percentage is due in part to the further sophistication of this third level of analysis, we also recognize that students would have fared substantially better had the question’s directions been better written and our grading rubric more appropriately constructed.

The last exam that we hope to construct and to administer this coming April would involve four questions: a question aimed at students’ understanding of diversity, a question to determine their reading skill, a question to explore an aspect of critical thinking and some measure of their comprehension of the necessity of considering many aspects of their life when making major personal assessments (see Appendix 12).

Recommendations

1. The General Education revision needs to provide more ongoing formative writing experiences for our students.
2. These experiences need to purposely build on the work done in the Freshman Writing Course.
3. The faculty need to be better informed about what happens in the Freshman Writing Course and they need to know what kinds of progress they ought to expect of students.
4. PLNU needs to provide writing support for all classes that engage in formative writing with students. This can be done through the Department of Literature, Journalism and Modern Languages’ Writer’s Studio, through the Center for Teaching and Learning or through the two working together.
5. Increasing efforts should be made in general education courses to make more explicit the meta-cognitive levels of their disciplinary work.
6. Increasing efforts should be made to let faculty know what is being covered in general education courses (overarching themes, common goals, meta-cognitive skills), so that faculty teaching subsequent courses can interact with that material by reinforcing, responding to, or qualifying that previous work). This work can be coordinated by the Dean of General Education who can also create opportunities of the dissemination of appropriate information.
7. The general education revision should include a minimal core of 8 courses (2 per year for each of a “regular student’s” 4 years). A core will permit programmatic work on development of skills and on continued conversation around larger issues.
8. Courses comprising the core should work at identifying shared themes and concerns as well as discussing ways to anticipate the other course (in the fall) or reference the other course (in the spring).
9. A one-unit post-international-experience course should be developed. Students need to take this course on re-entry into the U.S. after a semester abroad or after a LoveWorks experience. The course should focus on helping students to imagine an impact of the international experience on their future life beyond just being a page in their picture album.
10. The General Education Task Force should seriously look at ensuring that statistical literacy is being included in a required (rather than optional) general education offering.
11. The General Education Task Force should consider ways to include work on reading skills in either new or existing courses.
12. The General Education Committee should encourage the departments that offer multi-section General Education courses to identify a portion of the course (theme, method, text, experience) that would be shared across all sections. General Education courses should be seen as being jointly owned between the individual professor, the department and the faculty.
13. The General Education Committee needs to create a mechanism for the ongoing assessment and revision of general education. Part of this work should consist in a re-conceptualization of General Education as being principally about the needs of students and not about the needs of departments; thus General Education should be seen as belonging to all faculty members and not just to individual departments. This mechanism should include ongoing conversation with students about general education.
14. The Dean of General Education needs to work actively at developing some institutional vision of the nature, purpose, and developmental impact of general education. This informational effort needs to be accomplished through short, but regular communications.
15. The Dean of General Education needs to encourage faculty to do this same kind of ongoing communication about general education and its purposes with students.
16. Continued integration of the curricular and co-curricular should be pursued through the General Education Committee, the First-Year Experience Committee, the Assessment Committee, and through shared programs such as service learning, learning/living communities, and the Preface Reading Program. PLNU community members need to stay alert to other collaborative ventures that can be developed.