Assessment Committee Members

Becky Havens (chair) – Vice Provost for Educational Effectiveness; Dean of the College of Social Sciences and Professional Studies; Professor of Economics, Hadley Wood (liaison) – Vice Provost for Faculty Development and Dean of General Education; Professor of French and Literature, Cheryl Gaughan – Director of Records and Institutional Research, John Hawthorne – Provost and Chief Academic Officer, Jesus Jimenez – Professor of Mathematics, Jim Johnson – Professor of Education, Ron Kirkemo – Professor of Political Science, Jeanne Maiden – Associate Professor of Nursing, Becky Modesto – Director, Community Ministries, Keith Pedersen – Professor of Music, Anne-Elizabeth Powell – Librarian, Assistant Professor, Barbara Rutledge – Director, Academic Computing, and Caye Smith – Associate Vice President for Student Development

Committee Charge

Group One will revisit the prescribed Inventory on Educational Effectiveness, since these indicators are embedded in the University’s assessment program. It will outline the philosophy, scope, design, operation, and results of the review and then evaluate the assessment program (Institutional Proposal, page 8). This reflective essay will focus on two issues: 1) to what extent there have been broad, University-wide engagement and improvement actions concerning student learning outcomes as a result of the program, and 2) what evidence can be provided to support the conclusions (Institutional Proposal, page 8). Group One will make recommendations for future actions relative to the link between evaluation and assessment and institutional actions.

Introduction

To what extent do quality assurance processes actually improve student learning at Point Loma Nazarene University?

Point Loma Nazarene University (PLNU) has a core philosophy of improvement. This is evidenced by work, both systematic and irregular, to review programs and find ways to improve them. The formal assessment program using a Nichols model started in 2001, although periodic program review was systematized in 1994. Academic units used information from assessing student learning outcomes, now a part of the systematic assessment program, in the second round of program review. Program changes were instituted as a part of program review and the assessment of student learning outcomes.

This report will look for evidence of a direct link between program improvement and the systematic assessment process. Common outcomes in keeping with the University mission, appropriate tools of measurement, and resulting program changes will be demonstrated. The analysis of 64 academic assessment plans will show that data has been collected on 73% of the outcomes selected by academic programs, and findings will reveal that 67% of the outcomes for which data exist have met their criteria for success. Program changes have resulted from assessment activities, and in one case actual improvements in student scores based on the results of changes caused by assessment results are identified. While it may not seem impressive that we can only identify one case of double loop closure, through significant dialog with departments and the huge efforts to document massive assessment work at the University, the committee believes that this represents a criticism of our documentation, not a criticism of our culture of improvement or our actual assessment activity.

Systematic measurement of general education learning outcomes has occurred, and work is underway to analyze the findings and use the results to improve the general education program. Assessment of the co-curricular is underway, particularly in Spiritual Development, Student Development, and centers. Finding ways to effectively document the assessment activities that have occurred in the co-curricular arena and to deepen these activities across the University is one of the current challenges.

Overall, Group One will conclude that there is evidence to indicate that at least some improvements in student learning have been a direct result of the assessment program.

Institutional Culture

What do we believe about assessment at PLNU?

At Point Loma Nazarene University there is a deep commitment to improvement. It is a culture and a practice steeped in who we are, and that has its roots in our theology. As a community of faith in God who believes that our work is an act of worship, we are compelled to excellence. We genuinely care about our students, and want the best experience for them. This is a deeply held value in a community of people who share a common faith and mission.

John Wesley, forefather of our Christian faith tradition, believed that our gifts and talents belong to God. In our stewardship of resources he believed in making all you can, and giving all you can.

While a cornerstone of our identity, until recently the practice of improvement work has happened to a large extent by relying on the goodwill of individuals more than structures and common processes. The WASC review process has helped us add structure and form to our firm belief in improvement.

In the last seven years significant progress has been made to identify learning outcomes befitting our mission, measure them, reflect on the meaning of data, and make improvements based on the evidence. In addition, evidence shows that while our educational programs are generally effective, there are certainly improvements to be made. Having a system that forces us to reflect about what improvements could be made based on assessing identified learning outcomes that we care about, has made us better.

What is our philosophy of program improvement?

The philosophy of the Assessment Committee is to build structures that are flexible enough to allow academic departments to set their own learning outcome goals, but that provide guidelines requiring those goals are set within the context of the University mission, that they are measured using meaningful tools with external validity, and that the data are used for program improvement. In addition, the Committee’s purpose is to build breadth and depth across the University, and to gain 100% compliance by using a carrot rather than a stick—by providing positive support to do the work of assessment, and by helping academic and co-curricular units see the value of assessment.

The committee encourages academic units to set a few goals at a time, measure them creatively and reliably, set a target that they truly aspire to become, and learn from the process. The point is not to hit every target, but to learn from the systematic process. Setting targets that are realistic yet ambitious allows room for program improvement.

The committee expects academic and co-curricular assessment leaders to spend time doing the creative planning of assessment goals and measurements, and to make decisions about needed program changes, but ideally the committee would like to provide the staff support needed to assist leaders in data collection, data analysis, and report generation.

Analysis

Revisit the Educational Effectiveness Inventory

The Educational Effectiveness Inventory, currently contains an assessment plan for general education (12 Educational Effectiveness Indicators, called EEI’s), 64 plans for academic programs (one EEI, 57 undergraduate majors and/or concentrations, 8 graduate programs, and 5 teaching credentials in 17 academic departments), and plans for co-curricular programs (4 in Student Development and 5 in Spiritual Development). There is some overlap in assessment plan postings for Masters in Education plans and teaching credentials, and some overlap for academic programs allowing students the option of a Bachelor of Arts degree or a Bachelor of Science degree. There are 64 academic plans posted covering 70 distinct degree or credential programs.

Plans are under construction but still need to be posted for an undergraduate academic major in exercise science (Department of Kinesiology, Physical Education and Athletic Training), a new major in international studies (Department of History and Political Science), and a new graduate program in biology (Masters in Biology).

Work has continued throughout the WASC proposal stage, capacity review, and the effectiveness review, to update these plans and to “close the loop” by adding findings data and program changes as a result. To date, findings data exists for all of the general education outcomes, 73% of the academic program outcomes, all of the Student Development program objectives and 80% of the Spiritual Development program objectives (see Data on Co-Curricular Outcomes). Assessment data has been used to begin discussions about improvements in the general education curriculum, and to make program changes in all of the academic programs for which data has been collected. Findings from assessment activities have impacted the co-curricular programs in Student Development and Spiritual Development.

Evaluate the University Assessment Program

An extensive evaluation of the assessment program was completed in the CPR Group Six: Assessment report, specifically its capacity. Findings at that time included recommendations for increased staff support to do the work of assessment, and better support mechanisms to help those responsible for assessment activities at the unit level, to develop in their expertise and do their work. These recommendations are being carefully considered in the current budget process by the Cabinet, and the Assessment Committee is instituting some improved support mechanisms based on these recommendations (web page, mentorship, development opportunities, etc.). A culture shift continues as movement toward full compliance occurs and assessment activities become routine.

Analyzing the assessment plans currently posted to draw conclusions about whether Point Loma Nazarene University is an effective educational institution also requires another broad look at the effectiveness of the assessment program itself. Further recommendations for the University assessment program become evident as the findings from academic plans are analyzed. Holes in the assessment program also become evident as academic plans are analyzed as a whole, as co-curricular program assessment is considered broadly, and as linkages between the assessment program and other University processes are reviewed. Recommendations will emerge at the systems level as the effectiveness of the assessment program is analyzed.

How has systematic assessment led to improved student learning?

Implementation of a systematic assessment program at Point Loma Nazarene University gave focus and intentionality to our purported philosophy of improvement. It forced us to think about and identify student learning outcomes for programs, measure them, reflect on the data gathered, and consider ways to change our programs as a result.

In some cases the data served to inform leaders that the measurement methods were inadequate, and they altered the tools. In others the data informed leaders that students were performing well. In yet others, student performance did not meet targets and the assessment activities helped identify where the deficiencies occurred so changes could be made. All of these circumstances represent an important learning process, and reflects the culture of Point Loma Nazarene University as a learning organization.

Overall, assessment activities have informed the University that students are learning (meeting knowledge targets), growing (being shaped professionally and ethically), and that they are adequately prepared to enter the world of professions and/or continue to further study. These results align with the University mission—to teach, to shape, and to send.

To what extent have there been broad, University-wide engagement and improvement actions concerning student learning outcomes as a result of the assessment program?

The general education program outcomes analysis discovered that students are skilled at lower levels of Bloom’s taxonomy, but student performance in higher level skills (critical thinking) are less impressive (see the EER Task Force I: To Teach report as well as the reflective essay on To Teach). The General Education Task Force is currently deliberating changes in the general education program in order to address these findings.

The result of assessment of major programs generally finds students have attained an appropriate level of knowledge, skills, research, recognition/appreciation, maturity/growth, faith integration, moral/ethical behavior, preparation for future study, influence (in field) and life-long learning (more than half of the outcomes cited met targets), but performance on production/performance, career preparation and engagement (in field) needs attention (less than half of the outcomes cited met targets).

Co-curricular results for Spiritual Development indicate that PLNU is moving from input-focused to output-driven data collection. Results indicate that Point Loma student service exceeds that of Comparator Institutions, and student growth in Christian disciplines between freshman and senior years meets targets, though there are outcome areas for which data is not yet available (see Spiritual Development Learning Outcomes).

Student Development collects and reviews extensive data annually on retention, engagement, social, emotional and physical development, and leadership. The last five years reveals an increase in student retention, a steady growth in student participation in co-curricular activities, acceptable growth targets in wellness measures, and expansion in leadership opportunities (see Student Development Learning Outcomes). Diversity was identified as an area needing improvement. Changes have been instituted based on the findings, including the creation of new opportunities for student engagement, wellness program modifications, and increasing leadership opportunities. Data is not yet available from an alumni survey which is being developed.

Centers serve a variety of purposes to enrich students’ educational experiences and impact teaching and learning indirectly by supporting faculty performance. All centers have specific goals (see Centers by Unit and Mission and the home pages of the individual Centers). There is anecdotal evidence that the Centers are making an impact on student learning but a systemic way to measure learning outcomes needs to be developed.  Some have collected data and made changes to programs based on the findings. Centers are found generally to have an outcome-driven focus and conduct periodic review, even if a uniform system is not used across all centers (see Centers by Output and Program Impact).

What evidence can be provided to support the conclusions?

While documenting the assessment activities that exist is an on-going challenge, the Assessment Committee updated assessment plans and reviewed them in their current form by developing analytic tools showing common outcomes, instruments, findings, and use of results across the 64 academic assessment plans.

A spreadsheet was developed identifying 22 common outcomes across 64 programs, which of the 22 outcomes was selected by each academic program, whether data was collected for each of the cited outcomes, and whether the targets were met, not met, or unclear. A second spreadsheet was created to track the common tools used by academic programs to measure outcomes, and a third summarizes the common types of changes that resulted from assessment activities.

Assessment of Academic Majors

Assessment plans for 64 out of 67 (96%) academic programs are posted on the PLNU Assessment web site, revealing measurement of specific student learning outcomes (100%), collection and reflection on the meaning of the data (73%), outcomes met (67% of outcomes for which data exists), and changes made to programs as a result of data collection (100%).  For details see Data on Common Learning Outcomes by Department and Program Changes by Department. 

Common Outcomes

Determination of what constitutes a common outcome category, its definition, and whether individual programs cited that outcome, is subject to human interpretation, but the Assessment Committee made an attempt to assess and categorize all outcomes listed by academic programs through finding common threads and categorizing them accordingly. Once common outcomes were identified, a natural subgroup emerged for each of the mission areas: to teach, to shape, and to send. These areas emerged in part by what academic programs reported as their own teach, shape, and send learning outcomes, and in part through the analysis made by the Assessment Committee of common threads that emerged across all programs.

The common themes identified in assessment plans mirror the general student outcomes of learning (teaching), growing (shaping), and serving (sending). There are 22 distinct learning outcome categories cited by at least one academic program. These are listed by mission, category and frequency in Common Learning Outcomes by Program. Knowledge is the most often cited student learning outcome (cited by 80% of programs) and career preparation is the second most often cited outcome (45%). Clinical skills in practice (39%) and technical skills (33%), respectively, are the third and fourth ranked outcomes.

There are six common outcome categories cited that pertain to the teaching mission of the University: knowledge, technical skills, production of work or performance, writing skills, critical thinking skills, and dispositions or attitudes. Of these, knowledge is the most often cited outcome, cited by 80% of the 64 academic programs. Technical skill is the second highest ranked outcome in the teach group, and the fourth ranked outcome overall, cited by 33% of the programs.

There are eight common outcome categories cited that relate to the shaping mission of the University, that involve student growth or maturation processes: clinical or practical skills, faith integration, analysis or research skills, recognition or appreciation, maturity or growth, writing or reflection, communication, and lifestyle commitments. Of these, clinical skill, selected by 39% of the programs, is the highest ranked outcome in the shaping mission group, and the third highest ranked overall. The second highest ranked outcome in the shaping mission group is faith integration (20%).

In the sending mission group, eight common outcome categories were identified: career preparation, engagement, further study, moral or ethical behavior, analysis or evaluation in practice, influence in the field, life-long learning, and service. Career preparation, cited by 45% of academic programs, is the highest ranked outcome in this group, and the second highest ranked overall. The second highest ranked outcome in the sending group is engagement (27%) and the third is further study (22%).

Outcomes that might be considered distinct to a Christian university—faith integration, moral or ethical behavior, and service—were cited by 28% of the programs (18 distinct programs cited one or more of the three outcomes). Faith integration was identified by 20%; moral or ethical behavior by 9%; and service by 2%. These seem relatively low, and yet since academic programs were asked to identify three to five learning outcomes, it is not surprising that most programs identified professional or discipline-specific outcomes rather than more generic mission or faith-based outcomes. It makes sense that there would be a general expectation that the faith mission is an objective of the general education program, and would be assessed by it.  However, another explanation might be that faith-based outcomes are more difficult to quantify and harder to test which suggested that providing support for measurement of mission-specific faith-based outcomes would be beneficial.

Considering subgroups by mission and the number of outcome citations that occur in each mission area, referenced in Common Learning Outcomes by Citation, reveals that 43% of the outcomes cited are in the teach subgroup, 28% are in the shape subgroup, and 30% are in the send subgroup.

These results reflect a fairly balanced set of outcome citations across programs, but a higher proportion of outcomes are cited in the teach area. One explanation for these results might be that teach (knowledge or skill-based) outcomes are easier to quantify, and thus receive greater attention by academic programs. However, it would be interesting to reflect on whether the results found by general education assessment (see the EER Task Force I: To Teach report as well as the reflective essay on To Teach) regarding greater performance by Point Loma students in lower order Bloom’s taxonomy items are supported or reinforced by the greater focus among academic majors in teach (knowledge and skill-based) outcomes. This may prove to be a highly fruitful discussion with long-term implications for program change, both for general education and majors.

A breakdown of common outcomes by graduate and undergraduate programs is displayed in Common Learning Outcomes for Graduate and Undergraduate Programs. The highest ranked outcomes for graduate programs are knowledge (88%), practical skills (75%), and career preparation (38%), which mirrors undergraduate programs in ranking, although for graduate programs practical skills are cited more often (75% for graduate and 34% for undergraduate) and career preparation is cited less often (38% for graduate and 46% for undergraduate). These results could be reflecting the professional nature of most of the graduate programs, which are enrolled by working adults, and reveal a greater focus on practical skills and less emphasis on career preparation.

The break-out by mission subgroups reveals a balance among teach, shape, and send outcomes for graduate programs, just as it does for undergraduate, but for graduate programs there seems to be a greater emphasis on shaping outcomes (36% for graduate compared to 27% for undergraduate) and a smaller emphasis on teaching outcomes (40% for graduate and 43% for undergraduate) and sending outcomes (24% for graduate compared to 31% for undergraduate). Again, the relatively greater emphasis on shaping, and less on teaching and sending for graduate programs compared to undergraduate programs, seems appropriate, as they are relatively more focused on professional growth and maturation (shaping) processes.

Common Methods

Each program cites a variety of methods to measure their student learning outcomes. These methods are summarized in Methods Used by Academic Programs to Measure Outcomes. The frequency represents the number of times a particular method is cited, or how many of the 64 programs use each of the methods. The most often used method for assessment of academic programs is a senior exam (61%) and the second most often used method is an alumni survey (55%).

It is important to know something about the reliability of the evidence collected to assess student learning outcomes. Each of the methods used by academic programs was categorized as either direct or indirect (self-reported) evidence, and each was identified as externally evaluated, internally evaluated through jury review, or internally evaluated. The validity of the evidence is relatively higher if direct evidence is used, and if the data is externally evaluated. Having evidence that is internally evaluated by a jury provides the next best validity.

Validity of Common Instruments identifies how many of the 252 instruments cited by academic programs are direct evidence (74%) or self-reported evidence (26%), and how many have external validity (68%), internal validity with jury review (28%) or internal validity (4%). The greatest reliability in data collection is for instruments having direct evidence with external validity (42%). However, a significant proportion of the outcomes have direct evidence with either external validity or internal validity with jury review (70%), and nearly all outcomes are measured with either external validity or internal validity with jury review (96%).

Thus the instruments used to assess Point Loma’s academic programs are largely direct evidence with a high degree of reliability.

Common Results

Data has been collected on 73% of the 254 learning outcomes cited by academic programs. The percent of outcomes for which data has been collected by outcome and mission category is shown in Data on Common Learning Outcomes.

The teach mission subgroup has 108 cited outcomes, followed by the send subgroup with 76 and the shape with 70. With respect to data collection efforts, the send outcomes subgroup has data on 82% of its outcomes; the teach subgroup has 70%; and shape has 69%.

In the mission subgroups, outcomes that are most often cited by programs have the greatest degree of data collected: 69% for knowledge in the teach subgroup, 80% for clinical skills in the shape subgroup, and 79% for career preparation in the send subgroup.

Outcomes that might be harder to measure seem to be ones where data collected is lower, such as faith integration (38%) or shaping goals in general (69% for the subgroup). Outcomes that are cited by relatively few programs have a great degree of variability with respect to percentages of data collected, which is expected due to the low number of outcomes cited.

Data on Common Learning Outcomes for Graduate and Undergraduate Programs relays data that has been gathered on outcomes by graduate and undergraduate programs. Ninety-two percent of the outcomes cited for graduate programs have collected data, compared to 71% for undergraduate programs. This shows comparatively greater compliance but for significantly fewer outcomes (25 in graduate programs compared to 229 in undergraduate programs).

The existence of data on outcomes by department is shown in Data on Common Learning Outcomes by Department, which reveals the unevenness of the number of outcomes and the progress on measuring them across departments. Ten out of the 19 departments listed have collected data on all of their outcomes. Four of the 19 have collected data on over half of their outcomes, three have data on nearly half, and two have collected data on around 30-40% of their program outcomes. There are still three programs without plans, not included here. Music has the largest number of programs, and by far the most outcomes to measure. Theology and Literature have the next most outcomes to measure. While a majority of outcomes cited have been measured, there are still 27% of the outcomes without data. However 100% of departments have measured some of their outcomes. These results indicate the need for additional support to assist departments in measuring their outcomes, and a need for more guidance in streamlining outcomes and measurement.

Findings are presented for the outcomes for which data has been collected in Analysis of Common Findings by Outcome. These findings represent data on 186 outcomes, or 73% of the outcomes, and they show whether the targets for outcomes have been met, not met, or if it is unclear whether the target has been met. Of the 186 learning outcomes that have reported findings, 124 (67%) met their targets, 12 (6%) did not, and 50 (27%) either reported findings that were too ambiguous or data that was inadequate to determine whether targets were met.

Findings by department and by graduate-undergraduate subgroups for outcomes for which data exists appear in Analysis of Common Findings by Department. For eight departments and two joint programs, or 10 out of 19 academic units (just over half), all of the outcome targets were achieved. Only two programs did not meet any of the outcome targets for which they report findings. If considered by individual outcomes, 67% of the outcomes for which data exists met their targets. Forty-eight percent of graduate program outcome targets were met, and 69% of undergraduate program targets were achieved.

Only 6% of all outcomes were not met, 7% of undergraduate outcomes and none of the graduate outcomes fell short of targets. Only four of the 19 academic units targeted outcomes that fell short of their desired levels, ranging between 6-50% of the outcomes they targeted.

For two programs, music and education, all of their outcomes had unclear findings, that is, findings for which it was impossible to determine if the targeted outcome was met. Six other academic units had unclear findings for between 18 to 80% of the outcomes. Fifty-two percent of graduate program outcome findings and 23% of undergraduate program outcome findings were unclear. Fifty out of 186 (27%) outcomes for which data was reported had findings that were unclear. These findings suggest that there is a need for additional guidance and better support for programs to do the work of assessment, and the need for developmental activities for assessment leaders.

Another possible interpretation of these results is that unclear results reveal the difficulty in measurement for some outcomes, and there is a need for expertise and support for assessment leaders in crafting workable tools of measurement.

Resulting Program Changes

In a few cases, program changes led to documented improvement in student learning. Most programs have either used results to change curriculum, operational elements of the program, or revised data collection efforts. All of these efforts toward change show that Point Loma has become a learning organization. They also show that the primary work ahead centers around more effective data collection efforts.

Common threads across academic programs with respect to the use of results from assessment activities are shown in Program Changes by Department, which also shows the national standards by department, achievement of professional accreditation, and program review cycles. These results are summarized further according to the frequency of changes by category in Program Changes by Category.

All programs for which there are national standards use those in on-going review of curriculum and assessment instruments. Although this report focuses on assessment of student learning outcomes, it is worthwhile to see a chronology and the backdrop of program review cycles and its link to assessment. Program review began in 1994, and Point Loma academic programs have undergone two cycles. During the second program review cycle, student learning outcomes and assessment data was included in the review. Thus a fairly recent marriage between program review and assessment has occurred. There needs to be further work to bridge these two processes with intentionality, and link both to professional accreditation for programs that have achieved it.

Curriculum and data collection are the most often cited changes resulting from assessment activities in undergraduate departments (76%). Operations and curriculum are the most often cited changes in graduate departments (75%). There are six undergraduate departments (35%) and two graduate departments (50%) that met targets and did not find a reason to make changes in their programs, but will continue collecting data and monitoring the findings (see Program Changes by Department and Program Changes by Category).

There is evidence from one program that assessment activities identified a content area weakness, and motivated curricular change. Following the curricular change the department retested the students to see if the curricular changes were effective in improving students’ knowledge. They found that the students’ scores had improved, providing evidence of an improvement in student learning as a direct result of the assessment program.

Assessment of General Education

An analysis of general education program outcomes discovered that students are skilled at lower levels of Bloom’s taxonomy, but weak in critical thinking skills and diversity (see the EER Task Force I: To Teach report as well as the 2007 EEI GE Summary). The General Education Task Force is currently working on using assessment results to improve the program.

Assessment of Co-Curricular Programs

Assessment results for Spiritual Development indicate that PLNU is strong in service outcomes relative to comparators, but that there is a need for improved data collection (see Spiritual Development Learning Outcomes).  Student Development assessment reveals good results in growth, wellness and leadership, but the need for improvement in diversity (see Spiritual Development Learning Outcomes). Changes have been instituted as a result of the findings. A review of assessment activities in co-curricular areas reveals the need for improvement in data collection methods.

There are ten centers at Point Loma Nazarene University that enrich the education of college students through a variety of academic and co-curricular activities, listed in Centers by Unit and Mission. Centers serve a variety of purposes to enrich students’ educational experiences and impact teaching and learning indirectly by supporting faculty performance. All centers have specific goals and ways to measure effectiveness (see Centers by Unit and Mission).  Some have collected data and made changes to programs based on the findings (see Centers by Outputs and Program Impact). Generally centers are found to have an outcome-driven focus and conduct periodic review, even if a uniform system is not used across all centers.  The pertinent question for this report is: what work has been done by centers to set goals, review their programs, and discern their impact on students’ lives?

While there is no systematic uniform review that is required for PLNU centers, evidence collected reveals that the centers set appropriate goals, regularly evaluate their activities and adjust them to meet objectives. A general review of centers’ methods of measuring effectiveness and findings from data collection is summarized in Centers by Outputs and Program Impact and reveals that centers are to a degree output focused and a few have collected data on outcomes. However, it seems that there is significant enrichment activity going on in the centers and that a rich opportunity to collect data on student impact is being missed.

Conclusions

What conclusions can be drawn from the evidence?

Point Loma’s strengths are that we care about student learning; we have a good structure, good compliance, common outcomes that fit us, fit our mission, and a good start on data collection. We have findings on most outcomes. We have a culture of improvement, and we have become a learning organization.

Our weakness is that we are missing data on some outcomes, and we have some unclear findings on outcomes for which we have data. We are missing one central common structure to collect, analyze, and report data so it can be used effectively and efficiently by the community, and we are missing staff support to do the work of assessment.

The analysis of evidence reveals that at least some improvements in student learning have been a direct result of the assessment program. In a few cases, program changes led to documented improvement in student learning. Most programs have used the findings of data collected from the assessment program to change curriculum, alter operational elements of the program, or revise data collection efforts.

The assessment program has helped Point Loma become a learning organization. The primary work ahead should focus on more efficient and effective data collection efforts, and improvements in the analysis and interpretation of data.

What does the next generation of assessment look like at Point Loma?

An effective assessment system of the future will intertwine micro level and macro level assessments with strategic planning.

Micro level assessment needs to intertwine program review and annual assessment:

• Program Review (5-year cycle) – are we doing the right thing?
• Assessment (annual) – are we doing what we claim (student learning outcomes)?
• Chair/co-curricular leaders: design outcome goals for programs
• Director of Assessment: design instruments, implement/deploy instruments, collect data, and report findings in a coherent, usable fashion
• Chair/co-curricular leaders: use findings to make program changes (improvements)

Macro level assessment:

• Cabinet/administrators/Strategic Planning Committee: design broad outcome goals for University
• Director of Assessment/Institutional Research: design instruments, deploy instruments, collect data, and generate reports
• Cabinet/administrators/Strategic Planning Committee: use findings to make broad University changes (improvements)

Directions for Improvement

How can data collection systems be improved in efficiency and effectiveness?

Suggestions for improving data collection systems may include:

• Create an effective broad-based documentation system for assessment
• Work on identifying and creatively measuring faith-based (shaping) mission outcomes
• Devote staffing resources to the work of assessment to support instrument design, data collection, and report generation
• Guide assessment leaders in streamlining outcomes and measurement methods
• Bridge the processes of program review, assessment, and professional accreditation with more intentionality
• Find more effective ways to measure outcomes and demonstrate assessment of the co-curricular
• Centrally organize an alumni survey that is disaggregated by academic units
• Find a way to document assessment program changes over time, in order to determine whether program changes made as a result of the assessment program actually have an impact on learning outcomes
• Capture the impact of centers on the enrichment of student learning

Suggestions for improving the interpretation and analysis of data may include:

• Share common outcomes among academic program leaders to open a dialogue about refining outcomes
• Discuss the relationship between assessment results in general education and academic majors
• Stratify outcomes and findings by area: sciences, social sciences, humanities and professional studies to see if insights emerge
• Provide developmental activities for leaders of assessment efforts to build expertise and skills, such as workshops on methods of measurement, writing a good survey, thematic analysis, etc.

Recommendations

What recommendations emerge from the analysis of evidence?

1. Improve data collection systems, both in efficiency and effectiveness.

2. Improve data interpretation and analysis.