Committee Members

Becky Havens (chair) – Vice Provost for Educational Effectiveness; Dean of the College of Social Sciences and Professional Studies; Professor of Economics, Hadley Wood (liaison) – Vice Provost for Faculty Development and Dean of General Education; Professor of French and Literature, Keith Bell – Interim Provost and Chief Academic Officer, Rebecca Flietstra – Associate Professor of Biology, Cheryl Gaughan – Director of Records and Institutional Research, Jim Johnson – Professor of Education, Ron Kirkemo – Professor of Political Science, Becky Modesto – Director, Community Ministries, Keith Pedersen – Associate Professor of Music, Anne-Elizabeth Powell –Librarian, Assistant Professor, Barbara Rutledge – Director, Academic Computing, Anita Smith – Professor of Nursing and Caye Smith – Associate Vice President for Student Development 

Committee Charge

The purpose of this report is to evaluate the University’s assessment plan, whose quality the WASC Review Team identified as a concern in their 1996 report. They specified the need to assess student learning outcomes and all support services, to carefully analyze and interpret data, and to integrate data into a coherent, overall plan that informs decision making.

This report will consider the following questions as three charges, given to Group Six, the University Assessment Committee, by the WASC Steering Committee:

Executive Summary

This report reveals a multi-year process whereby the assessment of student learning has become a common project of all members of Point Loma Nazarene University.

The attempt to institute quality improvement processes in student learning at Point Loma Nazarene University dates back to the early 1990s, with the establishment of program review and an ad hoc Assessment Committee, before the previous WASC self-study in 1994. However, it is after the WASC visit in 1994, which challenged PLNU to improve its use of data to drive decision-making and deepen the culture of assessment, that a renewed effort to create structures incorporating common models across all units of the University ensued.

This report identifies multiple efforts taken by University leaders to build formal structures for assessment activities, processes, oversight and support. It reveals a dynamic process whereby a culture shift has occurred in academic and co-curricular units, whose members have become involved in a continual learning process whose purpose is to improve unity, coherence, and outputs within a democratized governance structure. Rather than being forced on a recalcitrant faculty unable to understand the purpose and benefits of assessment, the process has embraced faculty and co-curricular units in a community wide effort for a common on-going learning process. While that process has been slow, it has been steady, and the process itself has been necessary to achieve deep cultural change. 

An evaluation of the current University assessment program reveals good understanding and significant compliance across academic programs and co-curricular units, with defined learning objectives and measurement tools, and data being collected and used in thoughtful ways to improve program quality and impact student learning. In addition, major advances have been made in the assessment of broad educational outcomes in general education and the crafting of educational effectiveness indicators which provide a link between institutional core values, broad educational outcomes and measurement tools allowing evidence-based decision making at an institutional level.

While a solid assessment structure is now in place and significant progress has been achieved in developing a learning organization, there are key improvements necessary to sustain the current momentum. Better structures of support for data collection, staff support resources, support for learning and development, and improved systems to document assessment activities is necessary to provide the proper support systems to sustain these significant developments over the last ten years. One of the most significant conclusions drawn from this review is that for Point Loma Nazarene University is to fully develop an evidence-driven culture, additional personnel resources with expertise in data management and data mining will be needed to support and encourage the assessment work of faculty and staff in academic and co-curricular areas.

Capacity for Assessment: Major Steps Taken Since the Last WASC Review

Initial Assessment Structures: 1985-1999

Until recently, academic programs at PLNU, as at most American universities, basically functioned though an organization of individual academic departments free to design and structure their courses as they saw best.  Departments were run as isolated units, contributing courses to academic major programs and General Education. In this system, the focus was on input (teaching), not output (actual learning), with few systematic mechanisms for rethinking programs like general education, for linking courses to a common conception of general education, or for making outcomes beyond individual course grades a priority concern.

The rise of the assessment movement in higher education has called into question that single focus on input.  The assessment program at Point Loma has been designed to accomplish three purposes:  1) carry out an assessment program that will improve student learning, 2) help faculty become more aware of this common project and of the value of assessment, and 3) motivate faculty to align their individual classes to the University’s mission.

The University’s initial efforts towards reaching these goals involved the establishment of an Office of Institutional Research (OIR), the creation of an ongoing cycle of departmental program review, and the formation of an ad hoc assessment committee.

The Office of Institutional Research (OIR) has completed external surveys and administrative (1985), and has conducted freshman surveys (1988) and senior surveys (1993) based on the Cooperative Institutional Research Program (CIRP) of the UCLA Higher Education Research Institute (HERI). Your First College Year (YFCY), a survey given at the close of the freshmen year, was instituted in 2003. Some of this information is now provided for the assessment of some general education goals, of some Key Performance Indicators (KPIs) and of some student development goals.

In 1994, Dr. David Strawn, Dean of Liberal Arts, developed a 5-year program review system after substantial research of post-tenure review processes and program improvement systems at other schools in the Council of Christian Colleges and Universities (CCCU). Subsequently, no academic department could submit curricular changes without completing a well-researched, evidence-based program review that grounded their curricular recommendations. All PLNU academic departments have now completed at least one cycle of program review and most have completed two cycles. For many departments, program review resulted in substantial program improvement or development. The Department of Business, for example, sought and achieved professional program accreditation from the Association of Collegiate Business Schools and Programs (ACBSP) in 2000, based on its 1995-1996 program review. Several other similar professional accreditations were later sought and achieved through the program review process, including Athletic Training (2003), Music (2003) and Dietetics (2005).

An ad hoc Assessment Committee was created in 1994 and Dr. David Strawn was named chair. The committee agreed upon a definition of assessment, started an inventory and joined the CCCU/FIPSI project. In preparation for the 1996 WASC visit, the ad hoc Assessment Committee prepared a report entitled, “Assessment At Point Loma Nazarene College,” which contained an assessment plan, expected uses of assessment data, an assessment inventory and resulting changes, assessment instruments in use, and future directions.

The 1996 WASC reviewers’ report commended the college for these efforts but found assessment to be at an “embryonic stage” and said the process would require “monitoring and energy to move it forward.”  While the University had made a start, it still needed a systematic analysis of data, an integration of data into the overall assessment of the institution, and the use of this data to inform decision-making and strategic planning. In addition, data needed to focus on student learning outcomes rather than exclusively on quality of input measures.

Creation of Permanent Assessment Structures: 2000-2005

In order to accomplish these further goals, the Provost appointed Dr. Becky Havens as Vice Provost for Educational Effectiveness with responsibility for institution-wide assessment; Dr. Havens established a standing Assessment Committee and an assessment model for University-wide educational effectiveness; assessment support structures were put into place; and Dr. Hadley Wood was named Dean of General Education with responsibility for general education oversight and assessment.

In March 2000, during a 2-day workshop, Dr. Jim Nichols initiated PLNU academic and co-curricular leaders into a flexible and workable assessment process, the Nichols model. The university adopted this five-column model of assessment planning and leaders were trained in the selection of student outcomes relevant both to specific programs and university mission, the designation of means of assessment, the establishment of criteria for success, and the connection of subsequent results to program change and development. At this same point, the University adopted a timeline that would lead to the University being prepared to submit its assessment program to WASC in 2004 and the President issued a strong statement of support for assessment (see President Brower’s Letter of Commitment to a Comprehensive Model of Institutional Effectiveness). An ad hoc Assessment Committee was reinstituted in the fall of 2000 by the Provost, who named Dr. Becky Havens chair and gave her the responsibility to build an institution-wide assessment program. The Provost named her Associate Provost in 2001 with specific responsibility for assessment, and the faculty approved a standing University Assessment Committee in the spring of 2002 (see Proposed University Assessment Committee Structure). This permanent University Assessment Committee was given responsibility to oversee the work of assessment and move it forward (see Current University Assessment Committee Structure for the current committee structure).

The feasibility of the Nichols plan and its ability to meet WASC standards was investigated in the fall of 2002 when Dr. Hal Whelply came and met with the Office of Institutional Research, the General Education Committee and the Assessment Committee. He reported that the Nichols Model was a credible plan and should meet WASC’s requirements for outcomes information. He also stressed the importance of solid evidence as central to any successful assessment effort.

Adoption of the Nichols model helped define and focus the responsibility of the Assessment Committee as one of support to department chairs, school deans, educational support units and the Office of Institutional Research in the assessment process, in program reviews, and in linking assessment to programmatic change. At the same time, the Assessment Committee became the central location for summary documentation and reports on university assessment and started working to ensure the link between assessment and the University’s strategic and academic planning processes.

Recognizing the need for more generalized faculty learning about assessment, the Assessment Committee organized a series of five workshops for faculty in 2002-2003 to explain the Nichols Model, and to experiment with rubrics as a means of quantifying and measuring educational outcomes (see Assessment Workshops, 2002-2003). In addition, various members of the assessment committee met with specific departments and co-curricular units to provide focused tutorials on assessment as part of the committee’s ongoing work of support and instruction.

At the same time, other elements of the institution were developing programs that would complement and become part of the overall institutional cultural change. The Strategic Planning Committee developed Key Performance Indicators (KPIs) for the institution such as retention rates, class sizes, and the percentage of full-time faculty. The Office of Institutional Research, organized earlier to collect institutional and survey data, could now link that data to the institutional learning process. 

As crucial structures have been established and their roles focused, the institution has been able to move forward through several important steps.

1) The Assessment Committee created a draft planning and goals guide for the fall of 2002 that clearly indicated that assessment was to become a continuing process, not simply a pre-WASC self-study “snapshot” of the institution. The University was to become a “learning community” in which assessment is a continual process so that people at all organizational levels, individually and collectively, continually increase their capacity to produce intended results (see PLNU Assessment Structure: The Nichols Model in Appendix B of the Institutional Proposal). All academic departments were required to file an annual assessment report (see Annual Assessment Report Filing Guidelines).

2) During the 2003-2004 academic year, 85% of departments had submitted assessment plans, and the committee worked on developing a summative assessment tool and rubric for general education (GE), which it was able to administer to graduating seniors in May 2004.

3) In the spring of 2004 the Committee completed an Assessment Inventory for the WASC Proposal, submitted on May 1, 2004. This inventory (Institutional Proposal Appendix B) provided a broad overview of institutional assessment and helped identify areas needing further development. Equally important, gathering data for this inventory required numerous meetings and contacts between assessment committee personnel and department leaders responsible for assessment planning and thus advanced the developmental role of the committee.

4) During the opening faculty convocation in the fall of 2004, the faculty as a whole worked on developing broad educational outcomes for the University. Dr. Hadley Wood, newly appointed Dean of General Education, grouped the outcomes by categories, reported these in the first faculty meeting, and, after broad faculty discussion and input, referred these outcomes to a WASC Committee (Group Two in the Capacity and Preparatory Review). This WASC committee developed Educational Effectiveness Indicators (EEIs) and referred those back to the Assessment Committee to assess whether those general educational goals were being met.

5) The General Education Committee, now taking a programmatic rather than an individual course view of GE, developed first principles and common objectives for general education. The GE Committee, in conjunction with WASC Group Two and the Assessment Committee, designed two more summative general education exams that would complete a comprehensive assessment plan for all the newly designed EEIs. 

6) Finally, the Assessment Committee set the further goal of “closing the loop” by clarifying through written guidelines what should be submitted annually in the department assessment report to monitor program changes taking place from the assessment process. Assessment Committee progress, goals, and expectations for departments were articulated to the entire faculty during faculty meetings (see faculty meeting reports in Assessment Committee Report to Faculty 11/17/2004 and Assessment Committee Report to Faculty 11/30/2005).

Assessment was now the common project of all members of the organization. Academic and co-curricular units were involved in a continual learning process whose purpose is to improve unity, coherence, and outputs within a democratized governance structure. Rather than being forced on a poorly informed or recalcitrant faculty, the process has embraced faculty and co-curricular units in a community-wide effort for a common on-going learning process. While that process has been slow, it has been steady, and the process itself is necessary to achieve deep cultural change. 

Findings: Current State of Assessment and Its Effectiveness

Current Assessment Activities: March 2006

Through sustained efforts over the last decade, Point Loma Nazarene University has made major strides in the establishment of an effective and efficient assessment system. The current assessment system includes the features listed below. Some of these have already been discussed in detail; others will be discussed further in this section. The assessment mechanisms now include the following:

• a uniform, yet flexible structure for university-wide assessment (Nichols model)
• an Assessment Committee, chaired by the Vice Provost for Educational Effectiveness, with responsibility for and oversight of a clearly designated structure of assessment planning for academic programs, co-curricular programs, and general education
• an identified assessment cycle that can connect to meaningful programmatic change, academic planning and strategic planning
• a workable plan in full implementation for general education assessment, administered by the Dean of General Education, with input from and supported by the Assessment Committee
• ongoing general education assessment aligned with clearly expressed GE principles and goals, with educational effectiveness indicators, and with a GE philosophy statement, itself aligned to the academic plan and school mission
• almost complete compliance with the assessment effort from academic units
• growing compliance with the assessment effort from co-curricular units
• established mechanisms for reporting of assessment efforts and results
• a university-wide assessment inventory tool that allows the Assessment Committee a broad view of PLNU assessment and the ability to quickly spot areas needing development
• a start to developing mechanisms for data storage
• a start to developing mechanisms to allow for data availability to appropriate units
• a beginning of reporting mechanisms to faculty at large
• ongoing efforts at training and teaching faculty and other assessment leaders
• a cultural shift of the University toward becoming a learning organization

The current assessment structure that is in place for all academic majors, general education, and many co-curricular programs at Point Loma Nazarene University follows the Nichols model (referenced in the Institutional Proposal Appendix B). Units specify, for each of their programs, three to five student learning outcomes that are clearly linked to University mission. For each outcome, units identify measurement tools; collect, analyze and interpret data; and use the results to inform program improvement.

All academic departments have now provided verbal assessment update reports to the Assessment Committee in the spring of 2005, revealing that there is indeed wide compliance with the University assessment program, and significant assessment activities are being conducted by academic units (see summary in Summary of Reports to Assessment Committee by Department Chairs and School Deans). An Educational Effectiveness Inventory is given in Educational Effectiveness Indicators (updated from an earlier version in the Institutional Proposal, Institutional Proposal). The current inventory reveals that approximately 90-95% of academic units are on target with an assessment plan, measurement tools and data gathering activities, and that approximately 75% of the units have been collecting and using assessment data for between three to fourteen years. In addition, a significant number of academic units have achieved external professional accreditation, requiring rigorous review against national standards. A chart listing these programs, their professional accreditation, and current status can be found in PLNU Departments with National Accreditation.

A variety of assessment tools are used across academic units, including but not limited to nationally-normed exams, exit interviews, and portfolio assessment. A brief summary of selected assessment tools, and the academic units utilizing those tools, is provided in Overview of Assessment Tools Currently in Use. Reports on the status of assessment were communicated to the faculty on November 17, 2004 and November 30, 2005 (deferred to January 18, 2006), the later outlining current assessment activities (see Assessment Committee Report to Faculty 11/17/2004 and Assessment Committee Report to Faculty 11/30/2005).

What continues to be a weakness is the lack of a central location and a standardized reporting system to document learning outcomes, measurement instruments, findings of data analysis, and use of the data for program improvement. While a culture of assessment activities has grown steadily richer and deeper, the survey of University assessment leaders confirms the need to mentor and guide academic leaders in the documentation of assessment activities and to develop better support structures for this work.

An Evaluation of the Assessment Program

In preparation for this report a dialogue with chairs about their needs occurred during the annual report to the Assessment Committee in spring 2005, and a survey of the assessment process was given to the leaders in academic and co-curricular units responsible for assessment activities. A summary of the comments during dialogue with chairs appears in Summary of Reports to Assessment Committee by Department Chairs and School Deans. The survey instrument and distribution list appear in Assessment Structure Evaluation: Survey Instrument and an executive summary of the results appears in Assessment Structure Evaluation: Executive Summary of Survey Results.

A chart summarizing the strengths and weaknesses in the current assessment program that are identified by the survey appears in Assessment Structure Evaluation: Summary of Strengths and Weaknesses. Strengths of the assessment program include a general understanding by members of the University, an understanding of the function of the Assessment Committee, a recognition that assessment efforts are going well in individual units, the fact that departments are on target with objectives, goals, and data collection, and a positive feeling about the assessment system’s flexibility and the Assessment Committee’s guidance.

While negative attitudes in the areas of support and usefulness of assessment to promote change are in the minority, there is a large neutral group. Nearly half of assessment leaders reported either negative or neutral attitudes in the areas of support for assessment activities or usefulness of assessment to promote program improvements. In other words, a large minority cannot report a positive or satisfactory experience. These data reveal the need for further investigation and improvement in the assessment program structure and support mechanisms. In addition, dialogue with department chairs and school deans during their annual reports to the Assessment Committee revealed the need for better structures to track students through their educational program and beyond. In particular, these academic leaders believe there is a need to track alumni and students who change their majors (see Summary of Reports to Assessment Committee by Department Chairs and School Deans).

Recommendations

These recommendations result from direct feedback from assessment leaders (see Assessment Structure Evaluation: Survey Instrument for a list) and from committee dialogue with department chairs and school deans in spring 2005 (see Summary of Reports to Assessment Committee by Department Chairs and School Deans). Deficiencies identified generally relate to structures of data collection, closing the loop, support for assessment activities, and documentation of work.

The following recommendations are provided for consideration by the University:

1. Improve structures of data collection, especially the following:
• Improve tracking system for alumni
• Improve tracking system for students who change majors

2. Improve support for assessment leaders through additional personnel, guidance and mentoring
• Formalize system to distribute assessment funds for testing, evaluator stipends, workshop attendance and development
• Assess adequacy of the current budget to support assessment activities
• Create a new position devoted to supporting assessment activities
  Suggestion: recruit a staff member with interpersonal and assessment skills, and who has a significant portion of his/her position description allocated for University assessment. This new staff position could be housed in the Office of Institutional Research.
• Improve communication with assessment leaders, provide regular updates, post a calendar of events, deadlines and assessment resources
• Work on recognizing the Assessment Committee as a valuable resource to facilitate outcome-based assessment efforts

3. Improve system to document assessment activities
• Improve clarity of reporting guidelines
• Standardize documentation system
• Provide support for units with external accreditation to seamlessly comply with standardized documentation and formatting guidelines

4. Improve and deepen culture of assessment and assessment activities
• Provide broad enough scope for both curricular and co-curricular units to understand and relate to the value of assessment
• Discuss, evaluate and support leader motivation for conducting assessment activities
• Help assessment leaders and faculty/staff discover how to use the results of assessment activities for program improvement
• Reach full compliance (100%) of assessment activities (plan, measurement, data analysis and use of results) for all academic degrees and major programs
• Link assessment objectives more explicitly to the University mission, “To teach, to shape, to send”
• Continue progress toward assessment of co-curricular programs, with particular attention to areas that can provide insight into student learning and formation
• Continue progress toward assessment of the general education program
• Improve measurement of educational effectiveness indicators
• Increase visible supportive role of administrative leadership toward the importance of assessment
• Work on strategies to improve attitudes toward assessment
  Suggestion: bring in peers from other CCCU institutions that are doing exemplary and effective assessment work, to provide motivation and illustrate its value. Set up meetings with the Assessment Committee and match these external peers to specific PLNU disciplines/departments or co-curricular areas.

One of the most significant conclusions drawn from this review is that for Point Loma Nazarene University to fully develop an evidence-driven culture, increased personnel resources devoted to data management and data mining are needed. This will require staff with expertise in data mining, statistical analysis and forecasting, and who also have the interpersonal skills necessary to engender trust and encourage the assessment work of the faculty and staff in academic and co-curricular areas.