WASC Home > EER Reports
Back
WASC EER Group Three: Program Review
Reflective Essay


Committee Members

Karen Sangren (chair) – Professor of Art, Keith Bell (liaison) – Vice Provost of Academic Administration, Carol Blessing – Professor of Literature, Leon Kugler – Director of Athletic Training; Professor of Kinesiology and Kim Schaeffer – Professor of Psychology, Brad Whitaker – Associate Professor of Mathematics.

Committee Charge

A key focus of the new WASC approach to accreditation is the concept of Quality Assurance. This means, among other things, the ability and determination of an institution to hold itself accountable to procedures, policies, and standards. Quality assurance processes often are tested rigorously during the Program Review cycle. In the context of Educational Effectiveness, the institution must be able to demonstrate vital quality assurance processes. Since the University is experiencing robust health at the undergraduate level and rapid expansion at the graduate level, it is appropriate that Group Three is asked to analyze current protocols for Program Review. Specifically, Group Three will “determine answers to the following questions: To what extent are these quality assurance processes actually improving program quality? and, What is the evidence that student learning outcomes are improved as a result” (Institutional Proposal, page 8). To conclude its work, Group Three will make both immediate and long-term recommendations to the President.

Development of PLNU Departmental/Program Review

History

Program Review formally began at Point Loma Nazarene University in 1990 when David Strawn, Professor of Math and Computer Science, was named to a newly created position at PLNU, Dean of the Liberal Arts. Following his appointment, Dr. Strawn visited Calvin College, Grand Rapids, MI, and Bethel University, St. Paul, MN, to examine their institutional program review practices.

Dr. Strawn conducted a cost analysis of program review at these institutions. He noted that each institution did significant reviews of their academic programs on a regular basis. The reviews aided in determining program effectiveness and controlling unwarranted growth of academic curricula.

In 1992-1993, following field research, Dr. Strawn wrote the first formal Program Review document for Point Loma Nazarene College (now University) (see Program Review: Cycle 1). It outlined directives for departmental program reviews. The document was brought to the department chairs for feedback. In 1994, the academic deans and chairpersons approved a formal plan for program review. In addition, an ad hoc committee was formed to develop assessment strategies. Program reviews became an integral part of assessment at PLNU.

PLNU Departmental/Program Review Cycle I and Cycle 2 Models

A PLNU Program Review schedule was developed with the expectation that each academic department would complete a full program review every ten years (Cycle 1), with a modified review conducted five years later (Cycle 2). Four departments at PLNU were the first to conduct Cycle 1 Program Reviews beginning in 1995-1996: Biology, Business, Chemistry, and History/Political Science. The final department completed its Cycle 1 review in 2001-2002. No departmental curricular or program changes could be made until Cycle 1 Program Reviews were completed.

The Cycle 1 PLNU Program Review model provided specific content and directives for the academic departments (see Program Review: Cycle 1). However, essential questions invigorated the reviews:

  • How well is our department/program functioning?
  • What standards of excellence should be used to evaluate our program? (i.e. state/national standards, exemplary programs, external specialists)
  • What is the status of our department’s learning outcomes?
  • What is being done? vs. Where does the program “need to go”?
  • “Here is our program…is it the right program?”
  • Do the programs have a sense of “mission fit” with PLNU?
  • “Is our assessment doing anything?”

The PLNU Departmental/Program Review Cycle 2 model was introduced in 2001. This was a modified departmental program review and quality assurance assessment conducted by departments five years after Cycle 1. In some cases, Cycle 2 program reviews were used to “fine tune” changes made after Cycle 1. In other programs, significant changes were made following Cycle 2 reviews (see 2006 Program Review Survey Respondent Comments). In the interim, departments/schools had also engaged in more informal assessment strategies (see 2006 Program Review Survey Respondent Comments). By 2002 these strategies were formalized using the Jim Nichols Assessment Model (see Institutional Proposal Appendix B and Assessment Committee Report to the Faculty, November, 30, 2005).

WASC Group Three: Program Review and Quality Assurance Processes


The WASC Program Review Committee assembled data on PLNU program review and quality assurance processes from three key sources:
  1. Interviews with PLNU administrators involved with program review development and oversight
  2. PLNU Departmental/Program Reviews Self-Study Documents
  3. 2006 WASC PLNU Program Review Survey: Part I and Part II (see also the Instructional Letter)

WASC PLNU Program Review Committee Findings


The WASC PLNU Program Review Committee found Point Loma Nazarene University had developed a robust climate of program review and assessment since the last WASC visit (see Program Review Committee Study, Summative Findings, Program Review Chart and 2006 Program Review Survey Respondent Comments). Each administrator was committed to quality assurance processes and had been actively involved in fostering program review efforts and assessment at PLNU.

The Cycle 1 Program Review (ten-year) departmental studies, which began in 1995, inaugurated a culture of review and assessment at PLNU. Cycle 2 Program Review (five-year), asked departments/schools to conduct modified reviews. Each review periods were followed in interim years by assessment strategies to measure program effectiveness and student learning outcomes (see Institutional Proposal Appendix B and Assessment Committee Report to the Faculty, November, 30, 2005). These assessment efforts were directed through the PLNU Office of Educational Effectiveness, which collected data to address the Institutional Proposal question, “What is the evidence that student learning outcomes are improved as a result?”

The most significant findings on the effectiveness and quality assurances of program review at Point Loma Nazarene University were collected through the 2006 WASC PLNU Program Review Survey. These responses are itemized and examined in Program Review Committee Study, Summative Findings, Program Review Chart and 2006 Program Review Survey Respondent Comments. Survey questions and resulting data addressed the WASC charge, “To what extent are PLNU quality assurance processes actually improving program quality?” ( Institutional Proposal, page 8 ).

Broader observations from the Program Review Committee’s study included the following:

  1. A lack of clarity between the definitions of “assessment” vs. “program review” at PLNU dogged the Committee throughout its work (see Program Review Committee Study). Quality assurance terms were often intermingled. It was obvious that extensive and productive work had taken place in program review and assessment at PLNU during the last ten years. However, was program review a subset of assessment, or was it a conjoined twin? How much of program review is essentially assessment? (see Program Review and Assessment Diagram). These questions made the Vice Provosts’ quality assurance leadership domains also unclear. Despite this, program review was notably the launching pad for effective departmental assessment efforts at PLNU since the last WASC visit.
  2. Academic departments utilizing external accreditation standards for analysis or guidance appeared to produce Department/Schools Self-Study PLNU Program Review Documents with more deliberate form and sequence than departments who compared their curricula to programs in respected institutions nation-wide. Despite this, all PLNU academic departments reached specific conclusions from their self-study efforts and cited program strengths and weaknesses. Each document demonstrated serious departmental self-study activities and ended with recommendations for future development and specific curricular changes.
  3. Where state or national standards were not used, departments compared themselves to exemplary programs in American higher education (i.e. Department of Literature, Journalism, and Modern Languages). In doing so, they may have indirectly used national standards, if these comparable higher education programs had employed discipline-based standards to build their own programs (see 2006 Program Review Survey Respondent Comments).
  4. Discussions with Vice Provosts suggested that Cycle 2 Program Reviews were not as effective as the original ten-year reviews, because departments/schools were not required to complete their five-year reviews before curricular changes could be made. However, the WASC PLNU Program Review Survey data (see 2006 Program Review Survey Respondent Comments) indicated that fifty percent of departments/schools felt their Cycle 2 Program Reviews were of equal value to their Cycle 1 reviews. Some indicated that the Cycle 2 model continued the culture of program review in their departments/schools. Other departments/schools indicated their Cycle 2 reviews were still pending. In the interim years, some departments had also conducted extensive self-studies for external accreditation of certain program units, prior to making curricular changes (i.e. Department of Music, School of Business, Department of Consumer and Family Sciences).
  5. Problems of timing and differentiation between types of reviews and assessment became apparent during the Committee’s study. The Jim Nichols Assessment Model was introduced in 2002. As new assessment expectations filtered back to the departments, faculty members often confused these quality assurance/educational effectiveness efforts with simply another version of “program review” (see 2006 Program Review Survey Respondent Comments). In time, because PLNU departments were scheduled to conduct program reviews in groups of four to five departments per year over a five-year period, some departments had not completed their Cycle 1 reviews while others had already moved on to other assessment efforts, followed by Cycle 2 reviews. Program review and assessment burnout was becoming a challenge on campus. This was particularly true for departments who had also written self-study documents for external professional accrediting agencies within the same five-year time periods.
  6. The full complement of full-time faculty members in each department or school appeared to be more fully engaged in the Cycle 1 and Cycle 2 program reviews than in the more formalized assessment activities (i.e. Jim Nichols Model).
  7. While the Committee interviews with administrators and examination of documents provided valuable findings on PLNU program review, the WASC PLNU Program Review Survey: Part I and Part II yielded the most direct data from key participants: the departments/school/and centers (see Summative Findings, Program Review Chart and 2006 Program Review Survey Respondent Comments).

General observations included:

  • Survey data indicated that program review at PLNU did work, that the majority of department/schools/centers were involved, understood what was expected, conducted reviews, and saw concrete outcomes (i.e. 82% updated curricula and restructured majors; 76% added new courses and 65% eliminated old courses). However, there were some areas, particularly graduate programs and academic centers, which needed to participate in PLNU reviews. In some cases, this could be an adjunct to external accreditation efforts.
  • The WASC Program Review Survey data suggested that there were a variety of ways that quality assurance practices were being done on campus outside the Cycle 1 and Cycle 2 PLNU Program Review or Jim Nichols Assessment models. Departments had used other forms of quality inventory, review, and control for review and assessment purposes, but had not necessarily attributed them to aspects of “program review.” (see Summative Findings, Program Review Chart and 2006 Program Review Survey Respondent Comments).
  • PLNU Program Reviews (Cycle 1 or 2) had not been conducted in most campus Academic Centers. However, three had conducted modified reviews (i.e. Wesleyan Center, Institute of Politics and Public Service, Early Childhood Center) (see Survey Participants Information).
  • Not all departmental/school program units were subject to PLNU Program Reviews. In such cases, department or schools used their external accreditation self-studies in place of PLNU Program Reviews. However, where external accreditation agencies or standards drove departmental/school program reviews, certain expectations of PLNU Cycle 1 and Cycle 2 reviews could go unaddressed.
  • To some respondents, having two Program Review models (Cycle 1 and Cycle 2) was confusing. This indicated an update and redesign of the PLNU Program Review Cycle 1 (ten-year) and Cycle 2 (five-year) models are needed. The timing of program reviews also appeared to require better coordination with accreditation and/or other departments who shared courses.
  • Finally, several respondents indicated that program review needed to be an ongoing process, not one that starts and stops every five or six years. It was integral to maintaining program quality. This may have indirectly indicated that assessment procedures in the interim years between program reviews were not sufficiently evident in their departments/schools/or centers.

Sample Program Review Documents:

Department of History and Political Science
Department of Mathematical, Information and Computer Sciences
School of Business (Undergraduate Program)
School of Business (MBA Program)


Recommenations to the President

  1. The distinctions between quality assurance, assessment, and program review at PLNU should be more clearly defined in written form. A document should be created to include information about the interface between these three elements and to identify the University officials responsible for each. The document should include protocols for the execution of program reviews and assessment, standardized terminology, and a glossary of terms.
  2. Current University Cycle 1 and Cycle 2 Program Review formats should be updated to provide a selection of program review models that accommodate differing profiles of departments, schools (including interdepartmental majors), and academic centers and can be conducted in conjunction with external accreditations (see Model 1 and Model 2 for suggested models).
  3. All program units in every department, school, or academic center should be subject to PLNU Program Review Cycle 1 (complete/ten-year) and Cycle 2 (modified/five-year follow-up) reviews. The timing of such activities should be flexible enough for departments/schools/centers to coordinate with external accreditation years.
  4. All PLNU faculty in departments, schools, and centers need to become active participants in the culture of quality assurance on campus. Program review and assessment are essential aspects of measuring educational quality and effectiveness at PLNU.