Assessment Journey: Warning Signs, Change Indicators, New Opportunities
10:35 AM - 10:55 AM
Objective: The library organization embarked on the journey to diagnose the strength of what we do, identify the warning signs of roles and services becoming extinct, discover change indicators, and map out new opportunities. The main objective is to rethink the library core business, align it with the user needs, and clarify it for major library stakeholders and clientele.
Methods: The library reinvention journey started by creating a customer segmentation matrix to map user groups and uncover customer preferences and insights. The findings prompted librarians to clarify how they envision the potential utilization of the library services, as well as the information and skills acquired through these transactions. As a result, librarians identified and defined data elements to collect, developed data collection tools available at all service points, and came to an agreement on data collection procedures. One form captured information on one-to-one transactions, while the other captured data describing complex interactions, instructional sessions, and special projects. Besides demographic data, the forms captured unique elements, such as intended use of information received. After concluding a yearlong pilot data collection, the entire process was revised to improve data collection consistency and focus the level of detail for data collected.
Results: For all transactions during 2009/10 year, the 3 top intended uses of information received were to complete a course assignment (1,166), incorporate into patient care (2,210), and increase personal knowledgebase (1,374). The next 3 most common uses were to prepare to give instruction (226) and to write a manuscript or prepare a grant application (411). About 7,000 individuals attended library sessions and were provided with advanced advisory or consulting services. Of those, 553 were faculty, 361 were academic and clinical staff, 1,292 were doctoral students, 526 were medical residents and fellows, 906 were graduate students, and 735 were undergraduate students.
Conclusions: Correlation of data on users and services helped us develop new services to reach more users, even with fewer resources available, and gave us an opportunity to better describe the services’ impact to stakeholders.
Authors: Barbara Abu-Zeid, Reference Librarian; Amy E. Allison, AHIP, Clinical Informationist; Sandra G. Franklin, AHIP, Director; Anna Getselman, Associate Director; Woodruff Health Sciences Center Library; Emory University, Atlanta, GA
Pervasive Assessment: Integrating Assessment into the Organization
10:55 AM - 11:15 PM
Objective: Rethink assessment! More than an activity required by funders, assessment must be pervasive throughout library operations and a formal responsibility of every professional librarian. With fewer people coming into the library and more people doing their own searches, traditional metrics are decreasing. We must rethink how we collect and interpret evidence of our continued relevance and value.
Methods: When entirely new organization whose goal is to improve access to health information was created, the need to demonstrate success was a given. There is no traffic to count, and there are no ticks to report on the reference statistics. What tools do we need, what kinds of data do we collect, and what do we do with it? The National Network of Libraries of Medicine, MidContinental Region, has experience using surveys, focus groups, interviews, funders’ site visits, and staff reporting. Information comes from different groups with different needs and priorities, allowing us to determine where needs overlap and where they are unique. As we repeatedly collect and analyze data, we learn what is important and how to use it. We determine the resources and services having the greatest impact and identify and prioritize our efforts, continually demonstrating our value.
Results: Decisions are made in every project area based on evidence. The organization is more agile and able to both discontinue ineffective practices and incorporate new ones in response to the evidence. By incorporating assessment into all aspects of the region's programs, liaisons are able to offer appropriate, effective services to meet all network members’ information needs.
Conclusions: Assessment is not easy: It takes effort and commitment. Assessment should be present from the beginning. Plan for data collection; think about kinds of data and their source. Work continuously to ensure all parties are knowledgeable and comfortable with the process and the rationale. Make it part of the culture and “owned” by everyone. Do not just collect data and stick it in a drawer (or save it in a file). It has to be reviewed, analyzed, and summarized so you can take action, improve services, and show value.
Authors: Betsy Kelly, Assessment and Evaluation Liaison, National Network of Libraries of Medicine, MidContinental Region, Becker Medical Library, School of Medicine, Washington University, St. Louis, MO; Claire Hamasu, Associate Director, National Network of Libraries of Medicine, MidContinental Region, Spencer S. Eccles Health Sciences Library, University of Utah, Salt Lake City, UT
Using Information Management Competencies Drives Objectives and Evaluation
11:15 AM - 11:35 AM
Objective: The New York Medical College Health Sciences Library has long sought meaningful integration of pertinent information management competencies into the curriculum of the three schools it serves. Recently a list of information management competencies was drafted, and efforts are underway to apply these competencies into the curriculum and evaluate the success of students and residents in meeting them. The competencies provide a framework for setting objectives and evaluation. This paper describes the process, draft competencies, testing, and our relevant instruction and evaluation efforts. Questions addressed include the following: What do our students and residents need to be able to do? Who decides? How to set objectives to meet them? How to teach them? And how to assess outcomes?
Methods: Formative evaluation. Draft competencies were presented to select faculty, faculty library committee, and curriculum committee chairs, with plans to pilot in three academic programs. The curricula of the programs were analyzed for appropriate application of competencies. The digital curriculum for all three schools were analyzed, and information management competencies were added as units or “courses” in Moodle and the Digital Curriculum database.
Results: All librarians involved in the library’s instructional program participated in the drafting of the competencies document and related educational objectives, exercises, and evaluation methods. Instruction and evaluation are variable, depending on the program needs and requirements: self-assessment, pretest, assignment, posttest, and librarian assessment of an individual’s competency attainment. Internal accountability measures were also developed.
Conclusions: The development of a set of institutional core information management competencies has allowed the library to begin new conversations about new roles and integrate instruction and evaluation of these skills into the curriculum of the college’s academic programs. Liaisons to the various academic departments view this document as a tool that can be used to promote instruction ideas and to structure their own exercises and evaluations related to those activities. Authors: Marie T. Ascher, AHIP, Associate Director, User Support, Education, and Research; Diana J. Cunningham, AHIP, Associate Dean and Director; Health Sciences Library; New York Medical College, Valhalla, NY
Assessing Reference Services Using the Reference Effort Assessment Data (READ) Scale
11:35 AM - 11:55 AM
Objective: To assess the reference services at the Biomedical Library, University of California-San Diego, by using the Reference Effort Assessment Data (READ) Scale. Gathering statistics with this new six-point scale will give us more complete data and will help us to shape the use of our staff and resources in more effective ways.
Methods: The READ Scale was implemented to record the statistics kept at our three service points and off-desk. The six READ categories allow recording the effort of each reference encounter, by taking into consideration the time spent and the expertise needed, rather than just the type of question (i.e., directional, informational, or search). A task force was created to become familiar with the scale and to train staff in order to normalize its use across our three service points. Revised forms were created to track the statistics, the standard READ cheat sheets were modified with more descriptions and examples specific to our library, and a process was developed to document the actual questions being assigned at level 4 or higher. These statistics will be analyzed, with the goal of gaining more insight into how best to restructure and utilize our shrinking staff and resources.
Results: The presentation will report on the data collected at our service desks
