Venia ~ Veritas ~ Sanctitas - University Motto

Preface

The Educational Effectiveness Review, like the Capacity and Preparatory Review before it, has been a significant educational vehicle for the Point Loma Nazarene University community. This report represents a significant step along the outcomes road. It reports on the results of assessment across the academic programs, explores the impacts of new program development and ongoing program review on program quality and student learning, and evaluates the ways the University’s goals of To Teach, To Shape, To Send actually have an impact on students’ lives. While there clearly is more remaining to be done for this outcomes approach to be fully institutionalized, we acknowledge that we are now asking ourselves the right questions.

Development of the Educational Effectiveness Review Report. The Educational Effectiveness Review (EER) report for Point Loma Nazarene University (PLNU) is the result many months of work by a substantial cross-section of senior administrators, professional staff, and faculty. This EER report for the Western Association of Schools and Colleges (WASC) provides evidence for PLNU’s assertion that it meets the accrediting agency’s Core Commitment to Educational Effectiveness. That commitment is summarized in the following sentences, taken from the WASC 2001 Handbook of Accreditation:

The institution evidences clear and appropriate educational objectives and design at the institutional and program level. The institution employs processes of review, including the collection and use of data, that assure delivery of programs and learner accomplishments at a level of performance appropriate for the degree or certificate awarded.

Under the direction of a Steering Committee, we carefully followed the outline of tasks as found in the Institutional Proposal approved by WASC in 2004. We took the research and commentary that six assigned sub-groups completed and reworked materials focusing on outcomes, so as to stay with the spirit of the Institutional Proposal. The following pages contain the various stipulated elements, including updates on the recommendations put forward in the Capacity and Preparatory Review (CPR) report (October 4-6, 2006), our response to the CPR visiting team’s suggestions and the Commission’s letter of February 27, 2007, a description of PLNU’s Educational Effectiveness approach, several analytical essays following the Special Themes Model (as described in the WASC 2001 Handbook of Accreditation), and a concluding integrative component. In addition, we have provided additional evidence for the analysis of Educational Effectiveness in updates to the Institutional Portfolio and new data exhibits.

The Institutional Proposal outlines the context of our EER report and prescribes five principle elements. Having opted for the themes model, we begin the report by providing updates on institutional actions and documented results since the CPR visit. Second, since our Educational Effectiveness Indicators (EEI’s) are embedded in the University’s assessment program, we revisit the EEI inventory as part of a comprehensive review. Third, understanding that vital quality assurance processes are critical in sustaining educational effectiveness, we identify assessment, new program approval, and program review processes as the focus of three essays. We understand that these are particularly important factors in developing new graduate programs while sustaining quality in a more traditional undergraduate setting. Fourth, in order to assist us in significant engagement with the thematic statements that have evolved as helpful shorthand for the institution’s Mission Statement: To Teach, To Shape, To Send , we devote reflective essays to each of these three major themes for the University. This aspect of the report is in direct response to one of our intended outcomes from the reaffirmation of accreditation process, namely, to “ensure that the new mission statement is embraced, shared, and embodied throughout the institution” (Institutional Proposal, page 3). Finally, we summarize our experience with the entire accreditation review process, pointing to fresh understandings, improvements and implemented recommendations, identifying shortcomings, and establishing an appropriate agenda for future action.

Capacity and Preparatory Review Results. The CPR report produced an initial array of 83 recommendations to the President. Subsequent to these findings, the institution established a small group of professional staff and faculty members to function as a Recommendations Sub-committee for the purpose of coming to terms with these findings and recommendations, in cooperation with the WASC Steering Committee and the President. The recommendations were codified by the Office of Institutional Research according to the “responsible persons or group” classifications indicated in the CPR report. Subsequently, individuals named to the group were assigned to on-going conversations with contact persons with the goal of determining progress made. The Sub-committee also made appropriate decisions relative to strategy and, in cooperation with the Steering Committee and the President, established a list of priority recommendations.

First, we determined that all recommendations labeled as “on-going” were fundamental commitments that the University would retain beyond the accreditation review cycle. Second, we noted that a small group of recommendations had been accomplished already, either during the days and weeks between the filing of the CPR report and the actual visit of the CPR team or immediately thereafter. Third, the Sub-committee recognized early in its review the need to prioritize the recommendations to the President for more efficient benefit to the institution, a point also clearly made in the CPR visiting team report.

The findings of the Recommendations Sub-committee and the Steering Committee, of which the President is a member, identified the following issues as appropriate priorities emerging from the CPR cycle:

• Incorporation of the recommendations gleaned throughout the entire review process into University planning;
• Continued development of a sound infrastructure and organizational model for both undergraduate and graduate studies that includes effective assessment protocols, together with specialty accreditations where appropriate;
• Attention to a deepened and extended approach to the assessment of all aspects of learning outcomes at the University, including general education, major programs at both undergraduate and graduate levels, and co-curricular programs;
• Continuation of efforts by the University to increase retention and graduation rates for all programs, especially analyzing disaggregated data by various categories; and
• Clarification of the University’s policy on academic freedom with the opportunity for extended dialog on its implications for teaching and scholarship in the context of our mission and vision.

The President has referred these priorities to the Strategic Planning Committee for their attention, as well as to the Administrative Cabinet who also serve as the University’s budget committee. This will help to ensure the possibility of developing specific strategies for these priority issues as well as appropriate funding in the annual budget cycle. Beyond this, we have created a spreadsheet that summarizes the current status of all recommendations made to the President in the CPR report.  (See Responses to the CPR Visiting Team and the Commission.)

Context of the Educational Effectiveness Review Report. Moving from the CPR process to the EER process has proven to be an important source of learning for Point Loma Nazarene University. As we moved from thinking about what is in place to evaluating what difference does this make in student learning , we found that we needed a new set of questions and a new mindset. We know that this shift in mindset will serve the institution well for years to come. In addressing the requirements for the EER report, we recognize the central importance of focusing on inviting sustained engagement by the institution on the extent to which PLNU fulfills its educational objectives. In the context of this report, we attempt to review our efforts to assess educational effectiveness, examine practices for evaluating student learning in order to improve the process of teaching and learning, test the alignment of resources with educational activities, and maintain a sustained engagement that will be on-going and not merely an effort to respond to subsequent reaffirmation of accreditation processes.

Reflecting on the three quality assurance processes of assessment, new program development procedures, and program review has been particularly beneficial to the University. We know that vital quality assurance processes are critical in sustaining educational effectiveness. We identified these three areas as we continue to focus more on student learning, develop new graduate programs and teaching locations while also sustaining quality at the undergraduate level in an enrollment-capped environment, and utilize an improved process of program review.

The Themes Model (as described in the WASC 2001 Handbook of Accreditation) has helped to provide evidence that we have a system of quality assurance for student learning and organizational effectiveness that demonstrates “an institutional and leadership focus; appropriate educational objectives; learning outcomes that are widely shared and reflected in academic programs and policies, faculty who take collective responsibility for demonstrating and reviewing attainment of those expectations; a growing culture of evidence that is well established and used regularly for improvement; and on-going, regular collection and use of evidence to assure program delivery and learner accomplishments” (WASC presentation, January 8, 2004).

Approach to Educational Effectiveness

Markers of Educational Effectiveness. In contemplating our approach to an analysis of educational effectiveness, we recognized the appropriateness of mentioning both historical and current issues in this context. For example, it is clear that there already exist several markers of effectiveness embedded in the University’s culture, since its founding in 1902 in the Los Angeles area and especially within the last 34 years in the San Diego community. As pointed out in the Institutional Proposal, the vision of our founder, Dr. Phineas F. Bresee, was for a substantial university of 3,000 students with various professional schools and an undergraduate residential liberal arts college. The institution that has emerged since the turn of the twentieth century has, in fact, become the realization of Dr. Bresee’s grand expectations.

One hundred and five years later, PLNU is a prominent Christian university that has Mission and Vision statements and stated Core Values that are congruent with our historical roots. We can point to increasingly selective admissions patterns along with dramatic improvements in persistence and graduation rates, strength in leadership development, robust connections to its sponsoring denomination (the Church of the Nazarene), and a commitment to the liberal arts and professional programs. In addition, the University can claim financial health and fiscal responsibility; a respected pre-medical program; a successful School of Nursing prominent in the greater San Diego area; a long-standing undergraduate research program in the sciences; an excellent teaching faculty committed to professional improvement, scholarship, research, and service; a series of specialized accreditations (allied heath education, business, dietetics, education, music, nursing); and a functioning strategic planning design. Moreover, team members from the recent CPR visit recognized and applauded many of these characteristics. Aside from these markers, however, PLNU understands that educational effectiveness cannot be fully measured in traditional ways, and we are determined to cultivate and sustain a “culture of evidence” with increased emphasis on demonstrating student learning and enhanced educational vitality. Consequently, PLNU has seen a dramatic shift from the use of descriptive portrayals of educational effectiveness to more analytical and evidentiary representations.

This important transformation is producing results that began during the early timeframe for creation of the CPR report, that continue in the current EER cycle, and that will endure in the years to come. With the establishment of an institutional Data Portfolio , the monitoring of trends has become easier and more efficient, and staffing in the Office of Institutional Research has increased in addition to the establishment of a routine for collection protocols (CIRP, YFCY, CSS, NSSE, alumni surveys). The assessment of general education learning outcomes continues as first outlined in the CPR report, and academic departments and schools are adopting new approaches to monitoring educational effectiveness including standardized tests, senior capstone courses, and multiple measures. Not only have the inventory of Educational Effectiveness Indicators tables been updated in connection with the EER cycle, but the President, Provost and Strategic Planning Committee are designing updated structures (including staffing) to support PLNU's growing culture of evidence. Plans are also underway to make more systematic and effective the utilization of alumni surveys, in an attempt to capture data on the acquisition of graduate education, professional growth, and other value-centered measures of effectiveness after students are well into post-baccalaureate experiences. With the naming of two new Vice Presidents in the co-curricular units of Spiritual Development and Student Development, assessment efforts will continue and grow in sophistication. All of these markers resonate with the two focal points of our EER report.

Two Focal Points. Our approach to educational effectiveness, as approved by WASC in our Institutional Proposal, includes two broad areas of attention: 1) three quality assurance processes and 2) three institutional themes. To accomplish this, we established six study groups of professional staff, faculty, and administrators and began collecting data, analyzing our findings, and drawing conclusions that resulted in recommendations to the President.

Quality Assurance Processes

Introduction. In preparation for opening the focus on quality assurance addressing assessment, new program development, and program review, we charged each study group with specific target questions. In general, we wanted to determine and verify the impact of assessment in enhancing institutional quality and student learning and establish to what extent quality assurance processes actually are improving program quality and student learning. We present these reflective essays in the following section, each with an opening statement regarding the institutional culture, evidence and analysis, a summary of our conclusions, and final recommendations to the President for action.

Assessment

Perhaps one of the most important aspects of verifying quality at an institution is the establishment of efficient and effective assessment protocols that provide data to be interpreted in meaningful ways that can point to necessary improvements. The following reflective essay looks at a variety of issues, including the University’s EEI’s, an evaluation of the PLNU Assessment Program, improvements in student learning, institutional engagement, evidentiary support for conclusions, and finally assessments in academic majors, general education, and the co-curriculum. This reflective essay addresses CFR’s 1.2, 2.4, 2.6, 4.4, 4.5, 4.6, 4.7, and 4.8.

The Institutional Culture

At Point Loma Nazarene University there is a strong commitment to improvement. It is a culture and a practice steeped in who we are and is even rooted in our theology. John Wesley, forefather of our Christian faith tradition, believed that our gifts and talents belong to God. As a community of faith in God that believes our work is an act of worship, we are compelled to strive for excellence. We genuinely care about our students and want the best experience for them.

The University, therefore, has a core philosophy of improvement. This is evidenced by the work spear-headed by the University’s standing Assessment Committee that assists units in reviewing programs and finding ways to improve them. The formal assessment program using a Nichols Model ( Institutional Proposal p. 48) began in 2001, although periodic academic program review was systematized in 1994. In the latter model, academic units used information from assessing student learning outcomes as part of the second round of program review. Units then instituted program changes following the review and the assessment of student learning outcomes.

While a cornerstone of our identity, until recently the practice of improvement work has happened to a large extent by relying on the goodwill of individuals more than on structures and common processes. The WASC review process has helped us add structure and form to our commitment to improvement. In the last seven years, we have made progress in identifying learning outcomes befitting our Mission, measuring them, reflecting on the meaning of data, and making improvements based on the evidence. To this point, evidence shows that while our educational programs are generally effective, there are improvements to be made. Having a system that forces us to reflect about what improvements could be made, based on assessing identified learning outcomes that we care about, has made us better.

The philosophy of the University Assessment Committee is to build structures that are flexible enough to allow academic departments and schools to set their own learning outcome goals, but that also provide guidelines requiring those goals to be set within the context of the University’s mission.  Our philosophy mandates that they are measured using meaningful tools with external validity and that the data are used for program improvement. In addition, the Committee’s purpose is to build breadth and depth across the University and to gain 100 percent compliance by using a carrot rather than a stick—by providing positive support to do the work of assessment, and by helping academic and co-curricular units see the value of assessment. The Committee encourages units to set a few goals at a time, measure them creatively and reliably, set a target that they truly aspire to reach, and learn from the process. Setting targets that are realistic yet ambitious allows room for program improvement.

At PLNU we expect academic and co-curricular assessment leaders to spend time doing the creative planning of assessment goals and measurements, and to make decisions about needed program changes. Ideally, the Committee would like to provide the staff support to assist leaders in data collection, data analysis, and report generation.

Evidence and Analysis

The Educational Effectiveness Inventory . The Educational Effectiveness Inventory, currently contains an assessment plan for general education (12 Educational Effectiveness Indicators, called EEI’s), 64 plans for academic programs (one EEI, 57 undergraduate majors and/or concentrations, eight graduate programs, and five teaching credentials in 17 academic departments/schools), and plans for co-curricular programs (four in Student Development and five in Spiritual Development). Plans are under construction but still need to be posted for two undergraduate academic majors in Exercise Science (Department of Kinesiology), a new major in International Studies (Department of History and Political Science), and a new graduate program (master’s degree in General Biology).

We have continued to work throughout the WASC Institutional Proposal stage, Capacity and Preparatory Review report, and the Educational Effectiveness Review to update these plans and to close the loop by adding findings data and resulting program changes. To date, data exists for all of the general education outcomes, 73 percent of the academic program outcomes, all of the Student Development program objectives, and 80 percent of the Spiritual Development program objectives (see Data on Co-Curricular Outcomes). Assessment data has been used to begin discussions about improvements in the general education curriculum and to make program changes in all of the academic units for which data has been collected. Findings from assessment activities have had an impact on the co-curricular programs in Student Development and Spiritual Development. (For details see program-specific data tables.)

Evaluating the University Assessment Program. The University completed an extensive evaluation of the assessment program in the CPR Group Six: Assessment reflective essay, focusing specifically on capacity. Findings at that time included recommendations for increased staff support to do the work of assessment, and better support mechanisms to help those responsible for assessment activities at the unit level to develop in their expertise and do their work. We continue to consider these recommendations in the budget process at the Administrative Cabinet level, and the Assessment Committee is instituting some improved support mechanisms based on these recommendations (web page, mentorship, development opportunities). A culture shift continues as movement toward full compliance occurs and assessment activities become routine.

Analyzing the assessment plans currently posted to draw conclusions about whether the University is an effective educational institution also requires another broad look at the effectiveness of the assessment program itself. Further recommendations for the University assessment program become evident as we analyze the findings from academic plans. Gaps in the assessment program also become evident as academic plans are analyzed as a whole, as we consider co-curricular program assessment broadly, and as we review linkages between the assessment program and other University processes. Recommendations will emerge at the systems level as we analyze the effectiveness of the entire assessment program.

Improvements in Student Learning. Implementation of a systematic assessment program at PLNU has given focus and intentionality to our stated philosophy of improvement. It has forced us to think about and identify student learning outcomes for programs, measure them, reflect on the data gathered, and consider ways to change our programs as a result. In some cases, the data served to inform leaders that the measurement methods were inadequate, and they altered the tools. In others, the data informed leaders that students were performing well. In yet others, student performance did not meet targets, and the assessment activities helped identify where the deficiencies occurred so that changes could be made. All of these circumstances represent an important learning process and reflect the culture of the institution as a learning organization.

Overall, assessment activities have informed the University that students are learning (meeting knowledge targets), growing (being shaped professionally and ethically), and that they are adequately prepared to enter the world of professions and/or continue formal studies at an advanced-degree level. These results align with the University Mission summarized in the phrases: To Teach, To Shape, To Send (see reflective essays below).

University-wide Engagement and Improvement . Analysis of the general education program outcomes discovered that students are skilled at lower levels of Bloom’s Taxonomy, but student performance in higher level skills (critical thinking) are less impressive (see EER Task Force I: To Teach report as well as the reflective essay on To Teach). The General Education Task Force is currently deliberating changes in the general education program in order to address these findings.

An assessment of major programs finds students have attained an appropriate level of knowledge, skills, research, recognition/appreciation, maturity/growth, faith integration, moral/ethical behavior, preparation for future study, influence (in field) and life-long learning (more than half of the outcomes cited met targets). Performance on production/performance, career preparation and engagement in the field needs attention (less than half of the outcomes cited met the targets).

Co-curricular results for the Office of Spiritual Development indicate generally that PLNU is moving from input-focused to output-driven data collection. Results indicate that PLNU student participation in volunteer service exceeds that of Comparator Institutions, and student growth in Christian disciplines between freshman and senior years meets targets, though there are outcome areas for which data is not yet available (see Spiritual Development Learning Outcomes).

The Office of Student Development collects and reviews extensive data annually on retention; engagement; social, emotional, and physical development; and leadership. The last five years reveal an increase in student retention, a steady growth in student participation in co-curricular activities, acceptable growth targets in wellness measures, and an expansion in leadership opportunities (see Student Development Learning Outcomes). The data identified diversity as an area needing improvement. We have instituted changes, as reflected in the CPR report (see CPR Group Seven: Diversity), based on the findings, including the creation of new opportunities for student engagement, wellness program modifications, and additional leadership opportunities.

Centers serve a variety of purposes to enrich students’ educational experiences and have an indirect impact on teaching and learning by supporting faculty performance. All Centers have specific goals (see Centers by Unit and Mission and the home pages of the individual  Centers). There is anecdotal evidence that the Centers are making an impact on student learning, but we need to develop a systematic way to measure learning outcomes. Some have collected data  and made changes to programs based on the findings, and generally we have found Centers to have an outcome-driven focus and conduct periodic review, even if a uniform system is not used across all Centers (see Centers by Output and Program Impact).

Evidentiary Support for Conclusions. While the documentation of assessment activities that exist is an on-going challenge, the Assessment Committee recently updated assessment plans and reviewed them in their current form by developing analytic tools showing common outcomes, instruments, findings, and use of results across the 64 academic assessment plans. Further, the Committee developed a spreadsheet to identify 22 common outcomes across 64 programs, which of the 22 outcomes was selected by each academic program, whether data was collected for each of the cited outcomes, and whether the targets were met, not met, or unclear. A second spreadsheet tracks the common tools used by academic programs to measure outcomes, and a third summarizes the common types of changes that resulted from assessment activities.

Assessment of Academic Majors. We have  Assessment plans for 64 out of 67 (96 percent) academic programs on the PLNU Assessment website, revealing measurement of specific student learning outcomes (100 percent), collection and reflection on the meaning of the data (73 percent), outcomes met (67 percent of outcomes for which data exists), and changes made to programs as a result of data collection (100 percent). (For details see Data on Common Learning Outcomes by Department and Program Changes by Department.) In assessment of academic majors it is helpful to analyze more in depth the commonalities from the standpoint of outcomes, methods of measurement, results, and the program changes made as a result of assessment.

Relative to outcomes, we can see that common themes identified in assessment plans mirror the general student outcomes of the University’s mission: learning (teaching), growing (shaping), and serving (sending). There are 22 distinct learning outcome categories cited by at least one academic program. These are listed by mission, category, and frequency in Common Learning Outcomes by Program. Knowledge is the most often cited student learning outcome (cited by 80 percent of programs) and career preparation is the second most often cited outcome (45 percent). Clinical skills in practice (39 percent) and technical skills (33 percent), are the third and fourth ranked outcomes respectively.

There are six common outcome categories cited that pertain to the teaching mission of the University: knowledge (80 percent), technical skills (ranked second), production of work or performance, writing skills, critical thinking skills, and dispositions or attitudes. There are eight common outcome categories cited by programs that relate to the shaping mission of the University, involving student growth or maturation processes: clinical or practical skills (39 percent), faith integration (20 percent), analysis or research skills, recognition or appreciation, maturity or growth, writing or reflection, communication, and lifestyle commitments. In the sending mission group, eight common outcome categories emerge: career preparation (45 percent), engagement (27 percent), further academic study (22 percent), moral or ethical behavior, analysis or evaluation in practice, influence in the field, life-long learning, and service. Outcomes that might be considered distinctives of a Christian university—faith integration, moral or ethical behavior, and service—were cited by 31 percent of the programs. Faith integration was identified by 20 percent; moral or ethical behavior by 9 percent; and service by 2 percent. These seem relatively low, and yet since academic programs were asked to identify three to five learning outcomes, it is not surprising that most programs identified professional or discipline-specific outcomes rather than more generic mission or faith-based outcomes.

An analysis of subgroups by mission and the number of outcome citations that occur in each mission area, referenced in Common Learning Outcomes by Citation, reveals that 43 percent of the outcomes cited are in the teach subgroup, 28 percent are in the shape subgroup, and 30 percent are in the send subgroup. These results reflect a fairly balanced set of outcome citations across programs, but a higher proportion of outcomes appear in the teach area, again possibly due to the fact that learning outcomes may be easier to quantify.

We also provide a breakdown of common outcomes by graduate and undergraduate programs in Common Learning Outcomes for Graduate and Undergraduate Programs. The highest ranked outcomes for graduate programs are knowledge (88 percent), practical skills (75 percent), and career preparation (38 percent). This pattern mirrors undergraduate programs in ranking, although for graduate programs practical skills are cited more often (75 percent for graduate and 34 percent for undergraduate), and career preparation is cited less often (38 percent for graduate and 46 percent for undergraduate), results that may reflect the professional nature of most of the graduate programs, taken by working adults, and with a greater focus on practical skills and less emphasis on career preparation.

In connection with common methods of measurement, each program cites a variety of methods to measure student learning outcomes (see  Methods Used by Academic Programs to Measure Outcomes). The frequency represents the number of times a particular method is cited, or how many of the 64 programs use each of the methods. The method most often used for assessment of academic programs is a senior exam (61 percent), and the second most popular is an alumni survey (55 percent).

It is important to know something about the reliability of the evidence collected to assess student learning outcomes. Each of the methods used by academic programs appears as either direct or indirect (self-reported) evidence, and each was identified as externally evaluated, internally evaluated through jury review, or internally evaluated. The validity of the evidence is relatively higher if direct evidence is used and if the data is externally evaluated. Having evidence that is internally evaluated by a jury provides the next best validity. Validity of Common Instruments identifies how many of the 252 instruments cited by academic programs are direct evidence (74 percent) or self-reported evidence (26 percent), and how many have external validity (68 percent), internal validity with juried review (28 percent), or internal validity (4 percent). The greatest reliability in data collection is for instruments having direct evidence with external validity (42 percent). However, a significant proportion of the outcomes have direct evidence with either external validity or internal validity with juried review (70 percent), and nearly all outcomes are measured with either external validity or internal validity with juried review (96 percent). In sum, the instruments used to assess Point Loma’s academic programs are largely direct evidence with a high degree of external validity.

In terms of common results, we have collected data on 73 percent of the 254 learning outcomes cited by academic programs. The percentage of outcomes for which data has been collected by outcome and mission category appear in Data on Common Learning Outcomes. Outcomes that might be harder to measure seem to be ones where data collected is lower, such as faith integration (38 percent) or shaping goals in general (69 percent for the subgroup). Outcomes cited by relatively few programs have a great degree of variability with respect to percentages of data collected, due to the low number of outcomes cited.

Data on Common Learning Outcomes for Graduate and Undergraduate Programs relays data gathered on outcomes by graduate and undergraduate programs. Ninety-two percent of the outcomes cited for graduate programs have data findings, compared to 71 percent for undergraduate programs. This shows comparatively greater compliance but for significantly fewer outcomes (25 in graduate programs compared to 229 in undergraduate programs).

The existence of data on outcomes by department or school appears in Data on Common Learning Outcomes by Department, which reveals the unevenness of the number of outcomes and the progress on measuring them across units. Ten out of the 19 listed have collected data on all of their outcomes. Four of the 19 have collected data on over half of their outcomes, three have data on nearly half, and two have collected data on approximately 30 to 40 percent of their program outcomes. There remain three programs without plans, not included here. The Department of Music has the largest number of programs and by far the most outcomes to measure. The School of Theology and Christian Ministry and the Department of Literature, Journalism, and Modern Languages have the next most outcomes to measure. While a majority of outcomes cited have been measured, there remain 27 percent of the outcomes without data. However, 100 percent of departments and schools have measured at least some of their outcomes. These results underscore the need for additional support to assist academic units in measuring their outcomes and a need for more guidance in streamlining outcomes and measurement.

Findings for the outcomes, for which data exists, appear in Analysis of Common Findings by Outcome. These findings represent data on 186 outcomes, or 73 percent of the outcomes, and they show whether the targets for outcomes have been met, not met, or if it is unclear whether the target has been met. Of the 186 learning outcomes that have reported findings, 124 (67 percent) met their targets, 12 (6 percent) did not, and 50 (27 percent) either reported findings that were too ambiguous or data that was inadequate to determine whether targets were met.

Findings by department/school and by graduate-undergraduate subgroups for outcomes for which data exists appear in Analysis of Common Findings by Department. For eight academic units and two joint programs, or 10 out of 19 (just over half), all of the outcome targets were achieved. Only two programs did not meet any of the outcome targets for which they report findings. If considered by individual outcomes, 67 percent of the outcomes for which data exists met their targets. Forty-eight percent of graduate program outcome targets were met, and 69 percent of undergraduate program targets were achieved. Only 6 percent of all outcomes were not met, 7 percent of undergraduate outcomes, and none of the graduate outcomes fell short of targets. Only four of the 19 academic units targeted outcomes that fell short of their desired levels, ranging between 6 to 50 percent of the outcomes they targeted.

With respect to program changes, in a few cases program changes led to documented improvement in student learning. Most programs have either used results to change curriculum and/or operational elements of the program, or revised data collection efforts. While all of these efforts toward change show that Point Loma is becoming a learning organization, they also show that the primary work ahead centers around more effective data collection efforts.

Common threads across academic programs with respect to the use of results from assessment activities appear in Program Changes by Department, which also shows the national standards by area and program review cycles. These results also appear according to the frequency of changes by category in Program Changes by Category.

Curriculum and data collection are the most often cited changes resulting from assessment activities in undergraduate units (76 percent). Operations and curriculum are the most often cited changes in graduate units (75 percent). There are six undergraduate academic units (35 percent) and two graduate programs (50 percent) that met targets and did not find a reason to make changes in their programs but continue collecting data and monitoring the findings (see Program Changes by Department and Program Changes by Category).

Assessment of General Education. An analysis of general education program outcomes discovered that students are skilled at lower levels of Bloom’s taxonomy, but are weak in critical thinking skills and diversity (see the 2007 EEI GE Summary, the  EER Task Force I: To Teach, and the reflective essay on To Teach). The General Education Task Force is currently working on using assessment results to improve the program (see the CPR Group Two: Educational Effectiveness Indicators reflective essay).

Assessment of Co-Curricular Programs. Assessment results for Office of Spiritual Development indicate that PLNU is strong in service outcomes relative to Comparator Institutions but that there is a need for improved data collection (see Spiritual Development Learning Outcomes) . The Office of Student Development assessment reveals good results in growth, wellness, and leadership, but also the need for improvement in diversity (see Student Development Learning Outcomes). Changes have been instituted as a result of the findings. A review of assessment activities in co-curricular areas reveals the need for improvement in data collection methods.

There are ten Centers at the University that enrich the education of students through a variety of academic and co-curricular activities, listed in Centers by Unit and Mission. As mentioned previously, Centers serve a variety of purposes to enrich students’ educational experiences and to have an impact on teaching and learning indirectly by supporting faculty performance. All have specific goals and ways to measure effectiveness (see Centers by Unit and Mission). Some have collected data and made changes to programs based on the findings (see Centers by Outputs and Program Impact). Generally, centers have an outcome-driven focus and conduct periodic review, even if a uniform system is not used across all. The pertinent question is: what work has been done by centers to set goals, review their programs, and discern their impact on students’ lives?

While there is no systematic, uniform review that is required for PLNU Centers, evidence reveals that the centers set appropriate goals, regularly evaluate their activities, and adjust them to meet objectives. A general review of Centers’ methods of measuring effectiveness and findings from data collection appears in Centers by Outputs and Program Impact and reveals that centers are to a degree output-focused and that a few have collected data on outcomes. However, we believe there is significant enrichment activity going on in the Centers and that a rich opportunity to collect data relative to their impact on students is being missed.

An initial assessment of the study abroad program was conducted in 2004, revealing a significant impact on student learning. Students reported experiencing increased growth, enhanced learning about cultural and cultural differences, and gaining a new perspective on the United States. Students who studied abroad also experienced greater college satisfaction and higher retention rates. An updated assessment of the study abroad program is currently underway (2007).

Summary of Conclusions

In this reflective essay, we have attempted to show evidence of a link between program improvement and the systematic assessment process. We have demonstrated common outcomes in keeping with the University mission, appropriate tools of measurement, and resulting program changes. The analysis of 64 academic assessment plans shows that data has been collected on 73 percent of the outcomes selected by academic programs, and that 67 percent of the outcomes for which data exist have met their criteria for success. We have made program changes due to assessment activities; while we can only identify one case of double loop closure, through significant dialog with departments and schools and the huge efforts to document assessment work at the University, we believe that our challenge is one of documentation.  We remain committed to a culture of improvement and rigorous assessment activity.

Point Loma’s strengths are that we care about student learning; we have a good structure, good compliance, common outcomes that fit our mission, and a good start on data collection. We have findings on most outcomes. We have a culture of improvement, and we are making good progress on becoming a learning organization. Systematic measurement of general education learning outcomes has occurred, and work is underway to analyze the findings and use the results to improve the general education program. Assessment of the co-curricular is underway, particularly in Spiritual Development, Student Development, and Centers. Finding ways to document effectively the assessment activities in the co-curricular arena and to deepen these activities across the University is one of the current challenges.

Our weakness is that we are missing data on some outcomes, and we have some unclear findings on outcomes for which we have data. As pointed out in the CPR Group Six: Assessment reflective essay, we are missing one central common structure to collect, analyze, and report data so it can be used effectively and efficiently by the community, and we lack sufficient staff support to do the work of assessment.

Directions for Improvement. The University has identified several issues that provide direction for improvements in efficiency, effectiveness, interpretation, and analysis. In terms of efficiency and effectiveness, these include the creation of a broad-based documentation system, the identification of ways to measure faith-based (shaping) outcomes, the commitment of staff support, and the streamlining of outcomes and measurement methods. In addition, we need to bridge the processes already in place (program review, assessment, and professional accreditations), establish mechanisms for assessment of the co-curriculum, and organize an alumni survey that is centralized in the University with data results disaggregated by identified units. Finally, we need to discover ways to document program improvements to determine whether assessment processes have an impact on learning outcomes, and capture the impact of Centers on the enrichment of student learning.

Relative to improving the interpretation and analysis of data, PLNU needs to share common outcomes among academic program leaders for the purpose of refining outcomes, and discuss the relationship between assessment results in general education and academic majors. Likewise, we need to stratify outcomes and findings by area to see if insights emerge, and provide developmental activities for leaders of assessment efforts to build expertise and skills, such as workshops on methods of measurement, writing a good survey instrument, and thematic analysis.

An effective assessment system of the future will intertwine micro-level and macro-level assessments with strategic planning. Micro-level assessment needs to intertwine program review and annual assessment, where program review (five-year cycle) answers the question, “Are we doing the right thing?” Annual assessment activities seek to answer the question, “Are we doing what we claim relative to student learning outcomes?” In the latter, unit directors and chairs/deans design outcome goals for programs and ultimately use findings to make improvements; there is institutional support for the design of instruments, implementation and deployment of instruments, and the collection and reporting of findings in a coherent, usable fashion. Macro-level assessment entails the design of broad outcome goals at the Administrative Cabinet and Strategic Planning Committee level, using findings to make broad University improvements.

As a result of our work in preparing this reflective essay, we have developed the following recommendations to the President. Upon receipt of these recommendations, the President will develop cooperatively a prioritized list of all EER recommendations, assign each to a responsible person or group, and establish a timeline for action. As appropriate, these recommendations will be the focus of review by the Strategic Planning Committee for funding consideration, as needed.

See EER Group One: Assessment.

Recommendations

(See Recommendations to the President)

1.1 Improve data collection systems, both in efficiency and effectiveness.

1.2 Improve data interpretation and analysis.

New Program Development

A second aspect of verifying quality at an institution is the establishment of appropriate mechanisms for addressing new program approvals. This is especially important as the University attends to program quality improvements at the undergraduate level (while limited in a capped environment on the main campus) and improves guarantees of quality to new programs at the graduate level, with the challenges of infrastructure, assessment of learning, and quality controls. The following reflective essay looks at new program protocols and then examines two new program approval cases-in-point: the undergraduate major in International Development Studies, and the graduate program in General Biology. This reflective essay  directly addresses CFR’s 2.2, 3.8, 4.2, and 4.4. In addition, it indirectly speaks to issues in CFR's 2.3, 2.4, 2.7, 2.13, 3.1, 3.3, 3.5, 3.7, and 3.11 .

The Institutional Culture

As any educational institution evaluates its opportunities, there is occasion to add new programs to the curriculum. This process must be done with care to ensure alignment with institutional mission, appropriate financial and staff support, and with the intent of enhancing the quality and depth of student learning in the institution. How does a new program move from an idea to an institutionally adopted curriculum? And how is the institution guaranteeing the integrity, appropriateness, and mission-fit of new programs as they enter the stream of curriculum offerings at the institution? Clearly, the answer to these questions resides in a number of procedures and policies both formal and informal.

Point Loma Nazarene University has been in a season of robust health at the undergraduate level since reaching the City of San Diego-imposed undergraduate enrollment cap at the turn of the Twenty-First Century. In addition, the past several years have seen rapid expansion at the graduate level both on the main campus in San Diego and in the Regional Centers as far north as Bakersfield. As proposed in the Institutional Proposal, in such a robust environment, it is appropriate to examine and analyze current protocols for the development of new programs. While the beginning ideas and plans for a new program are usually generated as a part of the University’s program review activities (see EER reflective essay on Program Review), and ongoing quality assurance is a part of the PLNU’s assessment efforts (see CPR and EER reflective essays on Assessment), a key element in evaluating the effectiveness of the process for developing new programs centers on the adoption phase of these programs. This reflective essay proposes to determine answers to the following questions: What are the protocols for the development of new programs? Are quality assurance processes built into the process? and, How are the programs assessed and modified based on those evaluative findings?

New Program Protocols. In the Point Loma governance structure, proposals for new programs are evaluated through the Academic Policies Committee (APC) for undergraduate programs or the Graduate Studies Committee (GSC) for graduate programs. In each case, there are two different processes involved. First, each program must be evaluated against a set of established criteria to ensure that quality issues have been thoroughly addressed. Second, a separate set of protocols evaluate the necessary steps for adopting the new program into planning and organization at the University.

An initial review of these two important processes revealed a divergence between the undergraduate and graduate structures. The undergraduate process had a clearly defined adoption process but no stated criteria for proposing a new program. Ironically, the graduate process had clearly defined criteria through which a new program would be added but lacked a clearly stated adoption process. This governance issue was resolved in the fall of 2006 by the careful work on both the APC and the GSC to adopt parallel processes on both fronts (see Criteria for New Undergraduate Programs, Adoption Procedure for New Undergraduate Programs, Criteria for New Graduate Programs, and Adoption Procedure for New Graduate Programs). The faculty approved these changes in its October meeting (see Faculty Meeting Minutes 10/18/06).

In spite of the fact that these two important components of new program addition did not exist as formal documents, review of recent additions, at both the undergraduate and graduate level, demonstrate that there was an intuitive understanding of both quality criteria and adoption into institutional planning initiatives. To conclude this reflection, these new programs are then evaluated in terms of quality assurance and positive impact on student learning within the context of the institution’s mission and values.

Evidence and Analysis

To assess how the two processes described above contribute to quality assurance and student learning, we conducted a careful review of two recent additions to the curriculum: the undergraduate major in International Development Studies in the Fermanian School of Business and a master’s degree in General Biology.

Baccalaureate Degree in International Development Studies (IDS). The faculty of the Fermanian School of Business began in 2000 to revise the Economics major, a program that had been characterized by very low enrollments. First, the major was converted into an unsuccessful Economic Development major, and finally into a successful International Development Studies major and minor (see Enrollment Patterns in Three Business Majors). In the late 1990’s, the declining enrollment of students in the Economics major required that the School of Business either eliminate the major, offer under-prescribed courses at a loss to the University, or redesign the major in a way that retained the basic core of the Economics major and yet attracted a wider range of the undergraduate student population.

The initial attempt to redesign the major was to take the core components of the Business major and add a small set of Economic Development courses. This approach left the School with a degree program characterized by an oversized number of required courses that did not attract students desiring to major in Business or Economics.

Following this unsuccessful revision, the faculty set out to identify a market niche that would be a strong mission fit as well as complement the Business major. The faculty conducted extensive research into a variety of programs across the nation, considered options with a faith and service components, visited other campuses (including Chalmers Institute, Covenant College and School of International Leadership and Development, and Eastern University), and gathered informal perceptions from then-enrolled business students. The conclusion was that a strong International Development major and minor would require adding Economics courses and eliminating the series of Business courses that did not directly contribute to desired student learning and service outcomes in International Development. The School also believed that the new IDS minor would complement other majors including, Sociology, Political Science, Education, and Nursing.

The International Development Studies major and minor show a potential for success and have led to a new dimension within the Fermanian School of Business that enhances its mission fit and service goals. While the major is only beginning to generate graduates, some early anecdotal indications suggest that it has potential for significant impact on student learning. For example, in the spring of 2007, Economics professor Senyo Abjibolosoo spent a sabbatical in Ghana, working with two students majoring in International Development. The impact of this field learning is long-lasting (see the Ghana Newsletters). Similarly, the work of the Armenian Center for International Development has put students in learning situations that shape their activities beyond Point Loma (see the 2005-2006 PLNU Annual Report featuring the IDS major).

Master’s Degree in General Biology. The master’s degree in General Biology (M.A. and M.S.) was suggested by the Department of Biology as part of their ongoing commitment to educational outreach. In 1997, Point Loma launched the University Now program, designed to introduce college-level biology to high school students in underprivileged schools in local San Diego public schools. The program includes an invitation to students to spend time during the summer on Point Loma’s main campus working in the laboratories of the Department of Biology. This program was initially supported by a grant from the Howard Hughes Medical Institute. Between 1997 and 2003, 81 students participated in University Now. The program was evaluated by tracking the number of participants who went on to attend college. Summary data for University Now demonstrates that a remarkable 80 of 81 students went on to post-secondary education, supporting claims that student learning and future aspirations were both impacted by the program.

As the University Now program got underway, it became clear that support of science teachers in the public schools would be the next logical step. In 1998, the University launched the Perspectives on Science program (POS), a series of lectures that connect San Diego science teachers with high-level researchers. Feedback from the POS participants pointed out that there were no appropriate master’s-level programs in the field of biology in the San Diego area. Existing evening programs were only in Education (not Biology), and Biology programs were only offered in daytime formats. A survey of POS participants in 2003 showed interest in a master’s degree in Biology for public school science educators offered by Point Loma.

Faculty created a steering committee that included experts within the University’s Department of Biology and the School of Education, together with representatives from the public school community, to design a conceptual framework for a master’s degree. In 2003 the faculty then presented this initial structure to the Provost and Chief Academic Officer. Following a dinner with a POS focus group (see focus group notes), faculty reshaped the conceptual structure and presented it to key individuals within the science education community in San Diego. This resulted in a model that provided more flexible entry points in place of a more rigid cohort model.

In order to test this initial model in a market research study, faculty leaders enlisted the assistance of a team headed by a faculty member of the Fermanian School of Business who specializes in marketing. The test included the mailing of surveys to members of the San Diego Science Alliance. Given general support from the surveys and the critical mass of PLNU science faculty with background in science education, the faculty developed a formal proposal and submitted it through regular channels to the GSC. This was followed by a series of informational meetings between the GSC and Biology faculty, addressing general concerns about the launch of a new program. The faculty gave specific attention to the criteria for adding graduate programs discussed above. At the same time, faculty and the Provost presented the general concept to the Administrative Cabinet. Discussions with the Provost proceeded periodically with the assumption that full plans were being shared with the Cabinet and that any concerns they had were being incorporated into planning.

The GSC approved the proposal for a master’s degree in General Biology on November 19, 2004 (see GSC minutes). This action went to the full faculty for approval on December 8, 2004. At this point, it became clear that while the Cabinet had general knowledge of the proposal, a full review of financial requirements and enrollment models had not taken place. Working closely with the Vice President for Finance and Administrative Services, we determined that what was needed included a two-year model of enrollment that yielded sufficient contributions to overhead costs, following a start-up period.

The program began in the summer of 2006 with an enrollment of 13 students. In the initial modeling of the program, faculty leadership assumed that most students would opt for a non-thesis option. As students entered the first year of the program and looked toward completion in the second year, however, it became clear that this assumption was in error; nearly three of four students opted for a thesis. This had a positive impact on the revenue picture for the program but also demanded more faculty oversight. This oversight called for the hiring of an additional faculty member in Biology with a background in science education to assist with this demand. Revenue projections resulting from increased enrollment and the more popular thesis option allowed this additional hire without placing undue burden on the financial strength of the new program. As the program looks to its second recruiting class and moves toward long-term sustainability, outcomes assessment plans will allow for further improvements of this program, improvements designed to strengthen public school biology teachers in the San Diego market and beyond (see the process of approval narrative for further details).

Impact on Program Quality and Enhancement of Student Learning. The PLNU Institutional Proposal (page 10) attempted to make a connection between new program addition and program review, quality assurance, and improved student learning. It is for this purpose that we conducted a careful examination of new programs.

The specific review of the International Development Studies major and the master’s degree in General Biology demonstrates that both of these new programs were in some embryonic form at the time the Institutional Proposal was written. When these programs were in development, not the new program documents, nor the conversations at the APC or GSC levels, nor any Administrative Cabinet level conversations required a new program proposal to make explicit assumptions about enhanced quality or enhanced student learning.

For example, in the new International Development Studies major, it was far more likely that potential impact was stated in terms of strengthening a weak major. We note that building enrollment does not immediately correlate with quality concerns. In terms of ensuring program currency in light of institutional mission and core values, the revision in a new major does infer improved student learning, but the focus of the curriculum revision was one built out of best practices as known by the faculty.

By contrast, the master’s degree in General Biology can trace its origins to deep concerns about student learning. Beginning with underprivileged high school science students in local public schools, the concerns moved on to the learning outcomes of their classroom teachers. It does not appear that these intended outcomes were ever quantified and made part of the ongoing assessment processes of the University, in large measure because the program has not yet had any students complete the program.

Because the faculty members overseeing both the International Development Studies major and the master’s program in General Biology are deeply committed to the mission, values, and ethos of Point Loma Nazarene University, there is a great deal of confidence that awareness of program quality issues and student learning outcomes runs deep. As the University continues on its path toward embracing continuous quality improvement, these issues should become more explicit.

Summary of Conclusions

In our review of new program development, we have learned several lessons in our analysis of the protocols for the development of new programs at the University. We have identified four examples of institutional learning in this analysis. These include: the necessity of aligning parallel process for the undergraduate and graduate levels, the value of establishing specific measures to be addressed in new program development, the need to articulate procedures particularly for programs housed in Regional Centers, and the challenges of addressing new program development in the context of a fully built undergraduate environment.

Parallel Processes. The first lesson learned in the preparation of this reflective essay is that PLNU needs to ensure that the parallel processes of new program development for undergraduate and graduate programs are sufficiently aligned with one another. While the addition of a graduate program in a Regional Center may face very different dynamics than the restructuring of a program on the main campus, we must make certain that we ask the same critical questions and sufficiently answer them before a program is added.

Outcome Specificity. A second lesson is that we can only evaluate what we specifically ask new programs to address in their planning. If we focus on improved quality and enhanced student learning for new programs in the future, we must find ways of making these expectations more explicit at the earliest stages of program planning. In many ways, the language of intended outcomes is a new vocabulary to many faculty members, and we will need to provide training in how to ask and answer the critical questions. In a related fashion, we need to make sure we build evaluative processes along the way around the measures of program quality and student learning.

Review and Approval Process. A third lesson is that we need to be able to articulate the review and approval process for our Regional Centers in ways that are within the spirit of off-campus programming. Both of the programs added recently were housed on the main campus in Point Loma. The practice for the Regional Centers may follow a similar course but reflect some uniqueness. For example, we have recently learned that adding an existing program offering (e.g., a reading certificate) to a Regional Center when it has been modeled at another location may not constitute a new program but simply a new implementation. On the other hand, when a new program is added at a Regional Center, it must fit within the long-range planning for that location and in no way detract from its overall commitment to program quality.

Change in a Capped Environment. A fourth lesson speaks specifically to the challenges of new program development on the main campus in Point Loma. Because the University operates under a City of San Diego-imposed enrollment cap of 2,000 FTE students, we can no longer suggest new programs with the promise of enrollment growth. It may be more helpful for the process to involve explicit assumptions about program modification that speak to program quality and student learning. In the world of limited resources that is higher education in the 21^st century, it may be necessary to think about ideal program balance that involves moving resources to maximize quality and student learning across the curriculum.

As a result of our work in preparing this reflective essay, we have developed the following recommendations to the President. Upon receipt of these recommendations, the President will develop cooperatively a prioritized list of all recommendations, assign each to a responsible person or group, and establish a timeline for action. As appropriate, these recommendations will be the focus of review by the Strategic Planning Committee for funding consideration, as needed.

See EER Group Two: New Programs.

Recommendations

(See Recommendations to the President)

2.1 Develop explicit statements concerning expectations of enhanced quality and improved student learning as part of new program proposal processes.

2.2 Review new program development and adoption processes to articulate the relationship to program review protocols.

2.3 Evaluate the impact of the main-campus enrollment cap on the nature of new program development.

Program Review

A third factor in Point Loma’s efforts with regard to quality assurance is a system of academic program review, a valuable and productive exercise that rigorously tests our commitments to procedures, policies, and standards. The purpose of this reflective essay, as stated in the Institutional Proposal , is to rehearse briefly the historical development of program review efforts at the University, demonstrate the value of these protocols, and research the assertion that program review assists the units to improve program quality and student learning. This reflective essay directly or indirectly addresses CFR's 1.2, 1.3, 2.2, 2.4, 2.6, 2.7, 4.2, 4.4, 4.5, 4.6, 4.7, and 4.8.

The Institutional Culture

Establishing a Formal System. Plans for a formal system of academic program review began at PLNU in 1990 when the University established the position of Dean of Liberal Arts, now called Dean of Arts and Sciences. The appointment of an experienced and respected faculty member to this position prompted subsequent visits by the newly appointed dean to Calvin College and Bethel University (MN) for an examination of program review practices at these institutions. From those visits, the dean determined that the concept of program review offered the possibility of a significant and formalized quality assurance process characterized by a systematized cyclical schedule, helpful indicators of program effectiveness, and information that aided in monitoring growth by academic unit.

Following this field research during the 1992-1993 academic year, the dean drafted a program review document for consideration at Point Loma Nazarene University. The plan, submitted to unit chairs for comment, outlined directives for PLNU’s proposed program review processes. Finally, a formal proposal for a system of review program received approval in 1994 from the academic leadership team, made up of deans and department chairs. In addition, the University established an ad hoc committee to develop review strategies. Following that, program review cycles became an integral part of our efforts to evaluate academic programs at the University.

Subsequent Developments in Program Review . It was the expectation at the inception of the program review design that each academic unit would complete a program review every ten years (a first round) with a modified review five years later (second round). Four academic areas at PLNU were the first to conduct the initial program reviews beginning in 1995-1996: Biology, Business, Chemistry, and History/Political Science. The final department completed its first round review in 2001-2002. According to established policy, no curricular or program changes could be made until first-round program reviews had been completed, providing a significant motivation to all concerned.

The first round in the University’s program review model provides specific content and directives for the academic units (see Program Review: Cycle 1). Additionally, essential questions invigorate the reviews such as the health of unit functions, identification of standards of excellence, status of learning outcomes, future directions, mission fit, and the results of assessment.

The University introduced the second round of review (Cycle 2) in 2001. This second round, slightly modified from the first round, functions as a quality assurance assessment conducted by academic units five years after the first round. In some cases, second-round reviews have become the means of fine-tuning changes made after the first round. In other programs, units have made significant changes following the second reviews (see 2006 Program Review Survey Respondent Comments). In the interim, departments and schools began to engage in more informal assessment strategies (see 2006 Program Review Survey Respondent Comments). By 2002, the University took steps to formalize these strategies using the Nichols Assessment Model (see Institutional Proposal Appendix B and Assessment Committee Report to the Faculty, November, 30, 2005).

In preparation for this reflective essay, we assembled data on PLNU program review and quality assurance processes from three key sources: interviews with University administrators involved with program review development and oversight, program reviews and self-study documents, and a 2006 Program Review Survey: Part I and Part II (see also the Instructional Letter).

Evidence and Analysis

Though informal approaches to addressing program quality certainly have existed at the University for decades, our reflection during the Educational Effectiveness Review on processes utilized across the years affirms a thoroughly tested yet evolving approach to program review, especially in more recent years. Since the initial action of academic leadership 12 years ago in designing protocols for evaluating academic units and their effectiveness, program review has matured and developed into an efficient and profitable discipline. To discover more literally the patterns of use, benefits, and concerns relative to these processes, we recently surveyed all academic units at the University.

Improvements in Program Quality. Our survey data indicates that program review at PLNU functions reasonably well, involving a majority of academic units. Indeed, most understand the expectations of program review processes, recognize the value of systematic reviews, and can claim concrete outcomes as a result. For example, 82 percent of academic units report updated curricula and restructured majors, while 76 percent added new courses with 65 percent eliminating obsolete courses. Eighty-six percent of respondents agree that program reviews are a necessary aspect of maintaining quality; 78 percent of respondents recognize the inherent value of program review “as a means to promote academic quality at PLNU” (2006 Program Review Survey Respondent Comments). Most review experiences involved at least one and sometimes two external evaluators from respected institutions, and they resulted in significant changes to programs including, at times, important new hires in specific areas. In sum, program review as a quality assurance process has produced improvements in personnel, in curricular offerings, in increased attention to state and national standards, in the expansion of community partnerships, in the job opportunities for students, and in improved acceptance rates at the graduate school level.

Two Examples of Program Review . In an effort to analyze more closely the process of program review at Point Loma, we looked at recent studies conducted by two academic units: the Department of Family and Consumer Sciences (College of Social Sciences and Professional Studies) and the Department of Art and Design (College of Arts and Sciences).

Program review in the Department of Family and Consumer Sciences (FCS) led to student learning improvement through better knowledge and better preparation for careers. The first program review in 1999 generated extensive curriculum overhaul, producing a curriculum with greater depth and breadth. Subsequent outcomes assessments of student knowledge revealed a general upward trend in scores.

The first program review also began a trend of on-going review. In 2003, the Department also examined the Early Childhood Learning Center academic program, resulting in significant changes. In that same year, FCS wrote a proposal for accreditation by the American Dietetic Association (ADA), and submitted an application for eligibility in 2004. Subsequent to the self-study required for candidacy, Point Loma was approved for a Didactic Program in Dietetics (DPD) in 2005.

A combination of the program review process, on-going review, and student outcomes assessment contributed to changes leading to improvements in student learning. The FCS senior seminar contains an outcomes assessment component to solicit feedback on the program by graduating seniors. FCS has used this student feedback to improve program experiences which will better prepare students for professional practice, including more field trips and guest speakers. These focus groups also revealed the need to introduce the philosophy of the discipline of FCS earlier in the students’ program. The Department will be submitting a proposal to the Academic Policies Committee (APC) to make this change in the 2007-2008 academic year.

Program review in Department of Art and Design has led to better quality student artwork and an improvement in students’ ability to articulate intellectual thought, or reflection on their work. The first program review in 2001 laid out a 5-year plan with recommendations to expand variety in the curriculum and in the diversity of coursework. The Department proposed restructuring in the major, the addition of upper-division coursework, and the embedding in the curriculum of an increased focus on the conceptual content of artwork. As an example of the impact on student learning, a course in 20^th century art history was added to the curriculum and within a year following the change, students were able to reflect intelligently on their own artwork within the deeper context of recognizable 20^th century artists.

Outcomes assessment data showed a significant quality improvement in student artwork production. As all students’ senior exhibits are evaluated by a panel of professional external reviewers, there was a noticeable improvement in the comments of reviewers over the time since the Department instituted changes due to program review, including recognition that, in many cases, Point Loma senior art exhibit work is as good as graduate students’ artwork elsewhere.

It should also be mentioned that the data suggested there are a variety of ways that analyses of quality are accomplished at the University outside normal program review or the Nichols assessment model. Some units utilize other forms of quality inventory, review, and control for review and assessment purposes. Some academic units (e.g., the School of Nursing) use external accreditation self-studies in place of a traditional PLNU program review. Some centers, whose programs are markedly distinct from traditional aspects of the academic department or school, have approached program review in the context of rigorous assessment efforts, without adapting the measures typical of reviews at the present time (see EER reflective essay on Assessment).

Concerns Related to Program Review. While we can boast of many things concerning the utilization of program review in assuring ourselves of quality, some concerns have arisen during our writing of this reflective essay. These concerns include the definitions, participation levels, and coordination of efforts with respect to external accreditations and the University’s assessment program.

Some respondents to our survey indicated that the meanings and applications of some terminologies have blurred with the development of a robust assessment program at the University. The definition of program review may need clarification as to the academic units to which the cycles apply, such as University centers. Furthermore, to some survey respondents a better definition of the distinctions between first and second rounds may be needed. This may call for a general updating or design revision of the PLNU program review cycles.

Relative to participation concerns, the survey of academic units revealed some gaps in both reporting and participating in standard program review activities. While the majority of units did participate in significant ways, there were some areas in graduate studies that apparently did not. In some cases, this could be due to their use of external standards such as those required by the California Commission Teacher on Credentialing (CCTC). Likewise, some University academic centers conducted modified reviews, such as the Wesleyan Center, the Institute of Politics and Public Service, and the Early Childhood Learning Center.

There may also be some work to be done in understanding the coordination of program reviews and our continuing annual efforts in assessment. As new assessment expectations filtered back to academic departments and schools, faculty members sometimes confused these quality assurance/educational effectiveness efforts with simply another version of “program review” (see 2006 Program Review Survey Respondent Comments). In time, because academic units were scheduled to conduct program reviews in groups of four to five per year over a five-year period, some had not completed reviews while others had already moved on to specific assessment efforts, followed by the second round of program reviews. We faced the challenge of coordinating our efforts, particularly for units that had also written self-studies for external professional accrediting agencies within the same five-year time periods. The timing of program reviews could be better coordinated with the reaffirmation of program accreditations and/or the complications of synchronizing with other units that shared courses. Also, since some academic units avail themselves exclusively of external accreditation procedures, it is possible that certain expectations of PLNU program reviews even could go unaddressed.

While the College deans suggested that the second round of program reviews seem not to be as effective as first round ten-year reviews, due perhaps to the fact that units are not required to complete their five-year reviews before curricular changes can be made, 50 percent of academic units felt second rounds were of equal value to their first reviews. Some indicated that the second round continued the culture of evidence in their respective units. In the interim years, some units have conducted extensive reviews for external accreditations, prior to making curricular changes (Department of Music, the Fermanian School of Business, Department of Family and Consumer Sciences).

Pending Issues. We gathered findings on the effectiveness and quality assurances of program review through the 2006 WASC PLNU Program Review Survey. These responses are itemized and examined in Program Review Committee Study, Summative Findings, Program Review Chart and 2006 Program Review Survey Respondent Comments. Survey questions and resulting data addressed the charge, “To what extent are PLNU quality assurance processes actually improving program quality?” (Institutional Proposal, page 8). The weakness of this particular survey instrument is that it focused on the faculty’s informal conclusions and assumptions concerning the perceived effects of program review. As such, their assessment is very limited and provides only an input in this regard. We know that, while there are positive programmatic comparisons with respected institutions, we must engage multiple measures of assessing student learning that not only complement the capacity concerns of program review but also point to the educational effectiveness toward which all review benefits must point. Each round of program review was followed in interim years by an increasing number of assessment strategies to measure program effectiveness and student learning outcomes (see Institutional Proposal Appendix B and Assessment Committee Report to the Faculty, November, 30, 2005). These assessment efforts were directed through the PLNU Office of Educational Effectiveness, which collected data to address the Institutional Proposal question, “What is the evidence that student learning outcomes are improved as a result?”

Summary of Conclusions

The University’s system for academic program review has been in place since 1994, providing a valuable quality assurance function. We have demonstrated that this system (ten-year reviews followed by a five-year review) is firmly institutionalized in academic departments and schools and is, in fact, an expectation in on-going efforts for improvement. This process has brought qualitative changes to many programs since its beginning.

We can assert that Point Loma Nazarene University has developed a robust climate of program review and assessment since the last WASC visit in 1997. Administrators and academic leaders are committed to quality assurance processes and are actively involved in fostering these efforts at PLNU. Whether they utilize external accreditation standards or choose to compare themselves with respected institutions nationwide, all PLNU academic departments and schools are able to reach specific conclusions from their program review efforts, citing program strengths and weaknesses. Each program review document demonstrates serious self-study activities ending with recommendations for future development, including specific program changes (see Mathematical, Information and Computer Sciences Program Review for a sample program review and further documents in the Team Room evidence display).

As a result of our work in preparing this reflective essay, we have developed the following recommendations to the President. Upon receipt of these recommendations, the President will develop cooperatively a prioritized list of all EER recommendations, assign each to a responsible person or group, and establish a timeline for action. As appropriate, these recommendations will be the focus of review by the Strategic Planning Committee for funding consideration, as needed.

See EER Group Three: Program Review.

Recommendations

(See Recommendations to the President)

3.1 Study the current program review plan, addressing primarily the clarification of definitions, participation, and coordination with other quality assurance efforts.

3.2 Redesign program review processes to create a focus on the improvement of student learning, including external review mechanisms and robust assessment efforts.

3.3 Consider establishing exemplary program review formats in order to provide a selection of models accommodating differing profiles of departments, schools (including interdepartmental majors), academic centers, and reviews that flow out of external accreditations.

Institutional Themes

Introduction. After reviewing the four recommended approaches to the Educational Effectiveness Review, we chose the Special Themes Approach (as described in the WASC 2001 Handbook of Accreditation) believing that it held the most promise to assist us in accomplishing our intended institutional outcomes for this review. At the outset of our new strategic planning process several years ago, we took more than an entire year to develop a new mission statement. It is as follows:

Point Loma Nazarene University exists to provide higher education in a vital Christian community where minds are engaged and challenged, character is modeled and formed, and service becomes an expression of faith. Being of Wesleyan heritage, we aspire to be a learning community where grace is foundational, truth is pursued, and holiness is a way of life.

A short version of this statement emerged over time in the phrases “To Teach, To Shape, To Send.”   Since one of our intended outcomes from the review process is to “ensure that the new mission statement is embraced, shared, and embodied throughout the institution,” we decided to use these categories as themes to frame a large portion of the Educational Effectiveness Review.  We were careful to ensure that the analytical essays for each theme went beyond a description of activities to include analysis and reflection on the evidence that the institution has collected about teaching students (learning), shaping students (growth), and sending (service)—both while enrolled as actively engaged men and women and as alumni.

Since these themes encompass such a broad spectrum of activities and outcomes, the Steering Committee drafted several guiding questions to focus the investigation and subsequent discussions (see Institutional Proposal). These guiding questions identified specific areas of investigation, questions for engagement, and initial sources of evidence to guide the writing of the reflective essays. While these questions for engagement provided some general guidance, as we moved to the actual construction of the report we made two important discoveries. First, a review of the questions shows a much clearer statement of intended outcomes in the To Teach arena than in the others. Second, our interest in focusing on outcomes rather than process, or simple capacity, highlighted the reality that we have not operationalized in a systematic way all the questions in ways that would allow a compendium of data sources to inform our evaluation. Addressing each of these discoveries will be an important component of the post-EER work at Point Loma.

Theme One: To Teach

The first institutional theme that emerges from our Mission is TO TEACH. This aspect of our institutional purposes establishes for us the fundamental focus of teaching excellence and scholarship among the faculty but also the mirror of student learning and intellectual growth. In this reflective essay, we consider the institutional context, evidence and analysis, a summary of conclusions, and recommendations for action and improvement. This reflective essay addresses CFR’s 1.2, 1.6, 2.3, 2.4, and 2.6.

The Institutional Culture

Point Loma Nazarene University seeks to be a teaching/learning community " where minds are engaged and challenged " and " truth is pursued ." These key phrases from the institution’s Mission Statement  plainly express our commitment to teaching as central to the mission, the structures, and the daily life of the University. This essay, in keeping with our purposes as stated in the Institutional Proposal, seeks to elaborate on the University's teaching mission and the student learning that results from our efforts.

We can cite several descriptive facts concerning the value of teaching at the University. Teaching is the first-mentioned of PLNU’s eight Core Values. In addition, teaching is embedded into the evaluation and reward structure of the University. Student evaluation of teaching is required of all faculty members yearly and peer evaluation of teaching is a part of the full evaluation profile that faculty regularly submit. Moreover, an important factor in the granting of sabbaticals is the applicability of the project to the candidate’s teaching. Effective teaching is also required of faculty members for  promotion and tenure.

The University supports teaching through a variety of internal centers, grants, and programs. New faculty members attend an intensive orientation to PLNU through a day-long New Faculty Orientation at the beginning of the academic year and through the New Faculty Seminar, a faculty course that is offered in a 2½ hour-a-week format to faculty in the fall. The seminar covers a variety of topics in its 15 weekly meetings, but fully one-third of these deal with teaching issues. Additionally, an Adjunct Faculty Orientation also attempts to provide new part-time faculty with a portion of what is experienced in the New Faculty Seminar. The University also provides internal grants from the Wesleyan Center  for Twenty-First Century Studies, the Alumni Office and the Center for Teaching and Learning (CTL) for faculty members engaged in certain kinds of course development.

The CTL has sponsored ongoing programs to improve faculty teaching since the 2001-2002 academic year. These programs are accessible to all faculty members, full-time or adjunct, new or experienced. The CTL provides teaching tips, offers support for attendance at pedagogical conferences, directs regular reading and discussion groups, arranges for sessions on the pedagogical use of technology, and supports initiatives such as service learning, writing in the disciplines, and active learning.

Although support systems and structures are critical to the encouragement of excellent teaching, the actual effectiveness of teaching cannot be demonstrated by the existence of these systems. There are, however, some measures separate from support systems and structures that can be utilized to determine the general effectiveness of teaching at PLNU. The IDEA evaluation service, that provides the student evaluation form for the majority of PLNU faculty, recently provided student evaluations for all PLNU courses during each of five consecutive academic years, from 2000-2001 to 2005-2006. Looking particularly at objectives, progress toward objectives, excellence of teacher and excellence of the course, we can see in this Combined IDEA Data a general trend of increased effectiveness of teaching.

This composite data indicates that faculty are making application of course material or analysis and critical evaluation an increasingly important factor among their course goals (faculty using the IDEA form are asked to select three to five of their top objectives).  We cannot assume that the absence of particular objectives means that they are of no importance to the professor since the instructor selects from a range of possible objectives. More meaningful, however, is the strong student self-report of significant gains toward intellectual objectives. PLNU students indicate a stronger sense of intellectual progress than do the students in the IDEA system at large, while only in the area of development of creative capacities do PLNU students score below the average. Most important, PLNU students have shown increasing gains in higher intellectual skills such as application, analysis, and critical evaluation of knowledge.

The only solid measure of teaching effectiveness, however, remains the student learning that results. We therefore re-conceptualized the question about the effectiveness of faculty teaching as a question about the adequacy of student learning, especially student learning in central areas (general education and majors), in areas about whose value there is consensus among faculty (writing,