WASC Home > EER Reports
Back

WASC PLNU Program Review Committee Study

The PLNU WASC Program Review Committee met weekly during Spring Semester, 2006.  Primary efforts of the committee focused on 1) the General Charge for Group Three: Program Review WASC Committee, 2) clarification of distinctions between program review and assessment at PLNU, and 3) the development and distribution of a WASC Program Review Survey to all chairs or directors of departments, schools, and university academic centers.

During initial meetings, WASC Program Review Committee members examined the scope of their responsibilities and determined the parameters of the study.  Early inquiries focused on the distinctions between “assessment” and “program review” at PLNU.

Working definitions of program review and assessment available to the WASC Program Review Committee from University documents were as follows:

Program Review:

Department program review is a systematic, periodic, collection, analysis, and interpretation of data and the writing of reports related to the health and quality of a college department and or program.  It is the attempt to arrive at an objective statement as to how the department or program fits into the college mission, how it fits similar programs in the academic community, its strengths and weaknesses, and its cost effectiveness. (Strawn, 1994)

Assessment:

Assessment at Point Loma Nazarene University is the gathering, synthesis, and utilization of information in order to facilitate on-going improvement in the institution’s effectiveness to achieve its mission and to achieve learning objectives in major academic programs and general education. (Havens, 2002)

Early in the discussions, the WASC Group Three: Program Review Committee invited the Interim Provost and two Vice Provosts to meet with it individually and to answer questions about program review at PLNU since the last WASC visit.  They were also asked to address the Committee Charge:

Charge to the WASC Program Review Committee:

Specifically, Group Three will “determine answers to the following questions:  To what extent are these quality assurance processes actually improving program quality? What is the evidence that student learning outcomes are improved as a result?” (Institutional Proposal, page 8).

The Interim Provost and WASC Liaison discussed the Committee’s responsibilities and possible strategies to compile the information needed to write the WASC Program Review Report.  He also brainstormed with the Committee concerning useful information or data that could be collected in a survey instrument.  The Committee would need to design a survey and distribute it to each of the University academic departments, schools, and centers.  Data collected from the survey would be analyzed and submitted in the WASC Program Review Committee’s final report.

The Vice Provost for Educational Effectiveness and Dean of the College of Social Sciences and Professional Studies was asked to discuss the work of the WASC Assessment Committee and clarify what she saw as the distinction between institutional assessment (i.e. effectiveness and learning outcomes) and the departments/schools/ centers’ program reviews (i.e. discovery/self-study processes).  The Vice Provost stressed that the primary importance of program reviews to WASC was they should ultimately lead to higher student learning outcomes.  Strategies to evaluate these results became an aspect of “assessment”.

The Vice Provost also made reference to the assessment study that was begun under her leadership in 2002.  In this study, academic departments and schools were expected to develop plans which used the Jim Nichols Assessment Model: 

  • Program Mission
  • Intended Educational Outcomes
  • Means of Assessment and Criteria for Success
  • Summary of Data Collected
  • Use of Results

In order to apply the Nichols Model at PLNU, the Vice Provost and the Assessment Committee provided workshops to better inform department chairpersons and faculty members about the use of this new quality assurance model.  The workshops were also intended to help faculty members become acquainted with the WASC culture of academic assessment. 

The Vice Provost for International Studies and Dean of the College of Arts and Sciences, later met with the Program Review Committee.  The Vice Provost discussed the history of program review at PLNU over the past ten years.  He also discussed institutional expectations of departmental program reviews conducted for the 1995-2000 PLNU Program Review “Cycle 1,” during the administrative leadership of the former Provost. That is, in order for program/curriculum changes to be made in any department, a complete program review had to be completed first.

In any one of these years, 4-5 academic departments at a time conducted complete PLNU program reviews. In addition to “mission fit” issues in the PLNU program review directives, the departments often utilized their discipline’s state and /or national standards to assist in examining aspects of their programs (i.e. Department of Art and Design).  These standards addressed curriculum content, instructional strategies, leadership, advising, follow-up of graduates, and operations (i.e. technology, facilities, and equipment), among other program features.  

When state or national standards were not used, departments compared themselves to exemplary programs in American higher education (i.e. Department of Literature, Journalism, and Modern Languages). In doing so, they may have indirectly used national standards, if these comparable higher education programs had employed discipline-based standards to build their own programs.

The modified “Cycle 2” PLNU Program Review was required for departments or schools by PLNU five years after their Cycle 1 reviews. In six departments, WASC Program Review Survey respondents indicated that their Cycle 2 PLNU Program Reviews were of equal value as Cycle 1.  They had made significant changes to their programs after their Cycle 2 reviews. Other departments used their Cycle 2 Program Reviews simply to fine-tune departmental curricula (see Survey Summative Findings, Survey Results and Respondant Comments).

WASC Program Review Committee followed its interviews with administrators by examining PLNU Program Review Self-Study documents generated since 1995. This work provided the background necessary to develop a WASC PLNU Program Review Survey instrument.  The Committee’s charge was essentially to conduct a “program review of PLNU program reviews.”  This appeared to be the ultimate example of academic bureaucracy, but none-the-less, it was an essential effort to examine the nature and effectiveness of PLNU departmental reviews over the past 10 years.  The Program Review Committee needed to ask,  “To what extent are these quality assurance processes actually improving program quality?”

The WASC Program Review Committee first examined PLNU Departmental/Program Review volumes for content, standards used, and means by which departments met institutional expectations through extensive self-study.  Academic departments utilizing external accreditation standards for analysis or guidance appeared to produce documents with more deliberate form and sequence than departments who compared their curricula to programs in respected institutions nation-wide.  Despite this, all PLNU academic departments reached specific conclusions from their self-study efforts and cited their programs’ strengths and weaknesses. Each document ended with departmental recommendations for future development and proposed specific curricular changes.

WASC PLNU Program Review Survey

The WASC Program Review Committee used its interviews and documents research to design survey questions intended to gather information about the effectiveness of Cycle 1 and Cycle 2 PLNU Program Reviews. All Program Review Committee members were engaged in creating questions for Part II of the survey instrument.  The questions were designed to solicit feedback from the broadest number of participants.

Certain department program review data was better collected in a chart format. WASC PLNU Program Review Survey: Part I included:

  • Type of reviews done by the department/school/center (including exernal reviews, etc.)
  • Years completed
  • External accrediting agencies for the majors
  • Standards used (including external reviewers)
  • Lead author of the reviews
  • Other contributing authors of the reviews
  • Other departments consulted while doing the review of the majors
  • Instruction, facilities, or both reviewed
  • Survey of graduates done with program reviews

Questions included in Part II of the WASC Program Review Survey solicited more reflective information from the departments/schools/centers.  Questions included such issues as:  How satisfied were the participants with their PLNU Program Reviews?  Were the goals and procedures clear to the departments?  Did the faculty members recognize the value of an on-going culture of program review at PLNU?  How valuable were the external reviewers?  Were significant changes made in the programs after the reviews were completed? Additional questions addressed the differences between Cycle 1 and Cycle 2 reviews and their comparative value to the departments/schools/centers.

The WASC Program Review Survey (Part I and Part II) was distributed electronically in May, 2006.  It was preceded by a cover letter from the Interim Provost’s Office to inform department chairs and directors of schools/centers about the WASC Program Review Survey and encouraged them to complete the forms expeditiously.

As of September 2006, the WASC PLNU Program Review Committee had heard from 100% of the thirty WASC Program Review Survey participants. Where PLNU Cycle 1 or 2 program reviews had not been conducted (i.e. several academic centers and professional schools), departments/schools/centers were asked to submit an e-mail statement to the fact, so the Committee’s final report would reflect full participation from each of the campus programs.  Some directors of professional programs answered the survey questions based on external accreditation self-studies conducted in their programs.

The final stage in the  Program Review Committee’s  work  was  to tabulate the results of data  collected  in  the WASC Program Review Surveys, submitted  by  the  departments/schools/ centers between May and September, 2006.  An analysis of the survey responses is presented in in Survey Summative Findings, Survey Results and Respondant Comments.