Real-Time Reference in a MOO: Promise and Problems
This paper describes an experimental real-time reference service provided in a Multi User Object Oriented environment accessible on the Internet. Conducted by staff and volunteers of the Internet Public Library during a five week period in the Fall of 1995, the experiment focused both on the use of the MOO as a viable technology for providing a real-time text based reference service and the potential of providing real-time reference. Out of this experience the participating librarians identified potential uses, pitfalls and promise for providing reference on the Internet. This paper outlines some of these preliminary observations, some observations of those coordinating the project and some suggestions for and questions about the next generation of online reference at the IPL.
In March, 1995, students and faculty at the University of Michigan School of Information and Library Studies announced and "opened" a new Internet Web Site called the Internet Public Library. As a component of this site, the students had built a MOO - a Multi-User Object Oriented environment. MOOs were originally developed as a real-time interactive gaming environment on the Internet. The students who developed the IPL MOO hoped that this environment would allow experiments in real-time interaction for librarians all over the world - providing an opportunity to exchange ideas, hold meetings, provide reference services and even develop a social environment where librarians could talk to colleagues who they would otherwise never meet. By early Fall 1995 there were several regular active participants on the MOO. As a part of an ongoing seminar at the University of Michigan focusing on developing and expanding the potential of the Internet Public Library, students, in cooperation with the existing MOO community and the Internet Public Library staff agreed to run a five week pilot reference service in November/December 1995.
The MOO Environment
The IPL MOO is a text based environment that can be accessed through telnet. When users first arrive they "enter" a lobby with a description of that environment. They can "move" through the environment into different rooms by typing commands. When the user "moves" to a new room there is a new description of the room. There are "objects" in the MOO that can be manipulated through commands. The user can read signs, pick up objects, drop them, hand them to others. The user can also "talk" to other users by typing in whatever they would like to say. When they hit the return key it appears to all the other users in the same "room." Although the set of commands that a user needs to learn in order to communicate with others is relatively small, it does require any new user several minutes to learn how to negotiate the environment.
The designers of the IPL MOO created rooms and an environment that mimicked a physical library. There is a teen room, the stacks and a reference room, as well as some more unusual rooms such as a cafe. The text-based MOO allows the user to create a mental image of the environment. For more information about the Internet Public Library MOO see the IPL MOO Introduction.
MOOs have been used successfully for classes and other real-time interaction by a variety of users at other institutions. In general, these uses have been in a context where participants have an ongoing commitment to learning how to navigate the MOO. The reference project presented a new challenge for MOOs - potential single time users interested in seeking information rather than in ongoing use of the MOO.
Several considerations went into the design of the pilot project. We wanted to get significant feedback from a number of librarians about their experience providing reference in the MOO environment. In order to do that we decided that the librarians needed to spend several hours providing reference. On the other hand, we were completely dependent on their voluntary participation. We also recognized that while some librarians would have official support from their employers others would not have access to print and other resources. We settled on a design that asked each librarian to serve a total of 10 hours over a five week period in two hour shifts with at least two other librarians on duty at the same time.
The original research project was structured to accommodate 18 volunteers who would work in teams of three for two hour shifts once a week over five weeks. This would have allowed us to offer 12 hours of reference per week. We recruited volunteers from the Internet Public Library's existing email reference volunteers, three library related listservs and two statewide ALA conferences in New York and Wisconsin. Within a week we had over 80 offers to assist us. Of those we received over 50 completed background questionnaires. Based on information provided by their answers to the questionnaire about time availability, access to resources and experience in providing reference we chose 44 volunteers. The remaining volunteers were not chosen largely because we could not accommodate their schedules. We also continued to receive inquiries through the course of the project. We placed at least 4 members on each of 11 teams and expanded our coverage to 22 hours per week.
Who Were the Volunteers
The forty four librarian volunteers were from Canada(5), the Netherlands(1), New Zealand(1),Guam(1) and the United States. Our volunteer from Guam and one other volunteer, although unable to serve offered back up support services by answering extended research questions if needed. A few of our other volunteers never participated. Of the remaining forty volunteers, twenty five had professional library degrees, six were para-professionals with extensive reference experience (two of which were currently enrolled as students in a professional program), six were students enrolled in a professional library program and one had a degree in a closely related field.
Although the statistics on the number of years served are incomplete, of those who reported on years of service eight had over 15 years experience with reference services as a part of their job responsibilities. An additional eleven had served between 3 and 15 years. Thirteen of the participants reported less than 3 years of reference experience.
The librarians served in academic settings, school libraries, public libraries and professional organizations. Reported reference expertise included social science, engineering, music, the physical sciences, humanities, fine arts, education, and medical reference.
The volunteer librarians were directed to hold reference interviews in any of three rooms in the MOO - the reference room, the interview room or the reading room. The activity in each of these rooms was recorded during the reference shifts. The librarians provided notification to the interviewees either by showing them a welcome notice that explained the project or explaining the project themselves to each interviewee at the start of the reference interview. After the interview each patron was asked to respond to a series of evaluative questions while still on the MOO and record their responses privately. In addition, the librarians were asked to complete a open ended questionnaire at the end of the pilot. They were also encouraged to attend a series of informal debriefing sessions held on the MOO after the completion of the pilot. These recorded debriefing sessions provided an opportunity to compare experience with librarians from other teams.
Although there was an effort to capture all of the activity on the MOO, we lost a significant amount of data because both librarians and patrons would start and conduct entire reference interviews in other "rooms" of the MOO, forgetting that they were not in the reference area. In many cases the librarians did not get patron evaluations before the patron disappeared. In addition, several librarians who dropped out over the course of the pilot did not return the final questionnaires.
Despite these gaps in our data collection, we still gathered significant information and insight into the potential of real-time reference in a text-based environment. Indeed the problems that we encountered with data collection are as informative of the challenges of working in a distributed environment as the data we did collect.
Announcing the Reference Service
Notices about the pilot reference services were sent out to relevant library related listservs, a mailing list of friends of the Internet Public Library and other groups. Volunteer librarians also advertised it wherever they could. An announcement about the availability of mealtime reference was placed prominently on the front page of the Internet Public Library and was listed as one of the reference options on the reference pages of the library. Several signs and notices in the MOO itself announced and explained the nature of the pilot. Over the course of the pilot we discussed other ways to "get the word out" but were concerned about advertising so heavily for a program that was only to last five weeks. In addition to the announcements, an article about the service appeared in the online version of the Chronicle for Higher Education and an article in Library Journal about the Internet Public Library also appeared.
Implementation of Pilot
Librarians were placed on teams based on their levels of experience in providing traditional reference services, the times they were able to serve, whether they had access to print resources, their familiarity with the Internet and its resources and their level of experience in MOO environments. Every team had at least one experienced reference librarian, one person who had access to print resources and one who had spent time looking at resources on the Internet. An attempt was made to ensure that at least one person per team had experience in a MOO environment. Teams were assigned to a weekly time slot and were expected to work together to assign roles, negotiate absences and organize their own way of managing their 2 hour shift.
The MOO environment requires an understanding of several commands in order to move around and talk with other users. In the week before the pilot, several training sessions were held for newcomers to MOO technology. These training sessions lasted about an hour , led by a volunteer familiar with the MOO. It was difficult to schedule sufficient training sessions and difficult to target the training to a large group who had differing levels of experience on the MOO. Thus our volunteers started with varying levels of comfort in the MOO. In addition, the librarians who had been participating in the MOO since its opening stopped by the initial sessions and offered additional assistance to new librarian users. It was up to the librarians to then train reference patrons on the basic MOO commands that would enable them to seek reference help.
A MOO reference project home page was developed. This page included announcements about the project, biographies of the participants, a list of reading materials about conducting reference interviews, pointers to instructions about how to use the MOO and the schedule of reference teams and guidelines for doing reference on the MOO. Although a significant amount of information was available on this page, we were to later find out that it was frequently not referenced by the volunteers - even when they had questions.
A email group list was created for each team and they were encouraged to correspond through it with one another. A master mailing list allowed coordinators to send out information relevant to the entire group. Several groups also posted updates about their experiences to this mailing list.
Reference Hours and Process
Reference assistance was available to patrons for two hours on Sundays, Tuesdays, Thursdays and Saturdays, four hours on Mondays and eight hours on Wednesdays. The hours were distributed from 9:00 am to 11:00 pm Eastern Standard Time. This was done to accommodate volunteer schedules but also to provide some insight into when we were likely to get the heaviest use.
Each librarian was assigned a regular "character" on the MOO. They used the same name each time they logged on. They were able to append their character name with were a label that designated them as an on-duty librarian. This allowed all users to figure out who were the on-duty librarians.
Although each team organized itself, they generally followed a similar pattern. When a new user logged on, a team member would go and greet the user and see if they were interested in reference services. If the user was interested, they would make sure that the user could navigate through the MOO and communicate with others. If the user could not they would teach the user basic MOO commands. They would then direct the user to the reference area. Members of the team would explain the reference process and would conduct a reference interview. One or more members of the team would then look for the appropriate reference materials - either using online sources or traditional print sources. They would return to the patron with the source of the information. The patron would then go out and look at the sources. If the librarians and patron were discussing a Web site, patrons who had browsers could look at the material and decide whether it was appropriate. The patron was then asked to fill out an evaluation form in another room.
Although the data collection methods were designed with the idea that we would evaluate both how the reference interview compared with traditional reference interviews and whether the MOO was an effective medium for the conduct of real-time reference interviews, it quickly became apparent that evaluating the medium - the MOO - was going to take precedence. Both librarians and patrons encountered a variety of hurdles and positive aspects related to the technology that had an impact on the reference process. In addition, while we did have patrons we collected data on such a small sample that we chose not to evaluate the sessions themselves for how the on-line interaction affected the interview process. It seemed more useful to focus on understanding the technical issues so that later iterations of on-line reference service could learn from our struggles and triumphs. In this section, I will look at some of the evaluative comments by both patrons and librarians that shed light on the effectiveness of real-time reference on the Internet. It is beyond the scope of this paper to actually evaluate the recorded interactions.
Over the course of the 100+ hours providing reference service on the MOO we encountered 50 to 60 patrons. This participation level was rather disappointing to most of the volunteers and staff. Although we have no means of asking those who did not come in for reference assistance why they did not come there are several plausible explanations to consider:
We can not be sure how much each of these factors played a role in limiting participation. Given that each of these factors could dramatically limit participation, further investigation is warranted. However there is still much to be learned from the observations of patrons, participating librarians and coordinators of the project.
Whenever the librarian teams helped a patron they were to ask the patron to fill out the MOO Patron Evaluation.The patron would be taken to one of the empty rooms that had recorders and was asked to read and answer seven questions about the service they received. For a variety of reasons (including librarians forgetting to ask patrons and patrons decision to not respond) only 19 evaluations were completed. However the comments from those evaluations are helpful in assessing the benefits and problems of the MOO environment. All of the respondents were satisfied with the assistance received from the librarians. Several respondents remarked positively about the helpfulness and friendliness of the librarians.( See Appendix A). Consistently patrons identified interaction with the librarians as the most positive aspect of their MOO reference experience. This is not surprising since frequently studies have shown that patron satisfaction with the reference interview has more to do with the nature of the interaction with the librarian than with whether the information gained was correct or useful. (See Durrance, JC. "Factors that Influence Reference Success: What Makes Questioners Willing to Return" The Reference Librarian Haworth Press: No. 49/50, 1995), pp. 243-265.). A variety of negative aspects surfaced. These most often included concerns about the confusion when there were multiple participants in a conversation. The response to a question or comment might not come until after several other participants had commented. This led to disjointed conversation. In addition, patrons commented that the librarian could point them to resources but not necessarily evaluate them with the patron to see if they really met the patron's needs. Despite these limitations, those patrons who did fill out evaluations were generally pleased with the service they received, the information they received and the idea that they could get real-time online assistance.
Of the original 44 volunteers, 2 dropped before starting the project, two offered services as email back up but did not serve and 4 either dropped out during the course of the project or never showed. The remaining 36 offered feedback through an extensive questionnaire; and through a debriefing session held on the MOO. Depending on their shift time, the composition of their teams, the technical problems they encountered and the general activity on the MOO during their shift, their impressions of the MOO and its utility as a means of providing on-line reference varied greatly. As a group, the librarians faced a "crazy in the stacks" - a difficult patron who was disruptive to the process, patrons trying to do a month's worth of research in a night, patrons that couldn't articulate their needs, angry people, friendly people, grateful people. In other words, some of the librarians were occasionally bemused by how much the MOO world mimicked their own "real world" libraries.
The comments collected below do not represent all of the comments made by the volunteer librarians but seem most useful to informing a discussion of the next phase of on-line reference at the IPL and elsewhere.
Positive Aspects of the MOO Experience
When asked what they liked most about the MOO experience they consistently replied that the opportunity to meet other librarians, share ideas and learn from other reference styles was the best part. Most liked their experience of working on a team, in part because they had "backup" and because it often presented them with a different perspective on how to do reference. Many reported that they wished the teams had been more structured while other teams created their own structure. Others liked the fact that they could work at home - in their pajamas if they liked!
Negative Aspects of the MOO Experience
When asked what they liked least, the answers were considerably more diverse. They included comments such as:
In addition, we had a couple of teams that never figured out how to work together, either because attendance was inconsistent or roles were confused. These librarian consistently reported the most negative reactions to the pilot.
Was the Experience Worthwhile?
Most of the librarians reported that the experience was worthwhile because they were participating in something novel, they had made new friends or they had learned from the other librarians. Most of them were disappointed by the lack of patrons and the fact that working the MOO tied them down to their desks for two hours without allowing them to work on other things.
MOO Reference Compared with Traditional Reference Environments
When asked how the experience compared with doing reference face to face or on the phone, the most frequent observation was that the librarian lost a considerable amount of information because there was no voice inflection or body language to help them gauge when the patron was satisfied. In addition, many of the librarians reported that they felt like they set patrons adrift when they went to search for information. Most found that they did less bibliographic instruction because it was less complicated to get the information and present it to the patron than it was to try to teach the patron how to do it. The librarians would describe what was found rather than how they found it. Many commented that it took longer to answer simple questions because the answer had to be typed in. Others felt that "chatter" in the room confused both librarian and patron in the interview process. On the other hand, many of the librarians liked having the question written down right in front of them. Several librarians speculated that the patrons asked more precise questions because they had to type in the question. It is also possible that several of our patrons were librarians who are experienced at framing questions. Several commented that on-line reference would not substitute for face to face interviews but that on evenings, or weekends or in remote areas it provided a good alternative.
Almost half of the librarians reported that they had experienced technical problems unrelated to the MOO itself. These problems included severe lag time, down time and problems with getting a telnet connection. Concerns about the MOO and its interface included:
Ideas for a Better Service
Ideas for a better interface (short of video conferencing) included:
Other Comments and Observations
In a debriefing session on the MOO shortly after the project was over several of the librarians got into a discussion of how the MOO is an anonymous place - as much of the online world can be. They pondered the questions of what "authority" they have in a place like a MOO to provide information. How is their authority perceived and is it good for the information provider to be anonymous? What are the ethics of being anonymous?
Another concern was frequently raised by the librarians. Who should we serve in an online environment? Some raised the question by suggesting that we focus on a particular population and their needs (teens for example). Some suggested providing reference services by specialty rather than a general reference service. Others raise the question by suggesting that the reference service should focus solely on the Internet. Others have raised the question of whether the online community (a relatively well educated population) really needs a reference service.
As coordinator of the project, I had a unique view of the pilot. In a distributed environment few of the librarians met librarians other those those on their own teams. A few managed to drop by other sessions or develop relationships with the existing MOO community. These librarians acted as important bridges between the teams with which they interacted. But most of the communication between teams flowed through me and the email list we had created. This severely limited the flow of ideas. In the course of reviewing the final evaluations I realized that despite the email communications and the MOO home page, several of the librarians were not aware or were confused by several aspects of the MOO. This information never reached me. In another work setting, this might be less likely to occur - both because the concern could be quickly communicated orally and because it would be communicated in "real-time." Unless I was on the MOO when the question arose, the librarian would have to remember to email me after the session was over and then wait for the answer. The asynchronous nature of communication was often frustrating to all parties. In addition, although the information was posted on the home page it frequently did not get communicated. Indeed, those who participated at the University of Michigan were more likely to catch me in "real life" and ask the question. Those distributed around the world did not have that opportunity.
In comparison with other volunteer efforts I have coordinated, I found this to be very difficult. Communication took longer because everything had to be typed. Communication tended to flow one way - from me to others - not giving me the level of feedback to which I was accustomed in other settings. And it was not possible to get everyone in a room for a conversation. Like the librarians who didn't get feedback from patrons, I often was unsure whether the librarians understood everything. The challenges of coordination in a distributed environment should be considered in any plans for a real-time reference service if the providers of service are distributed across the country or world.
Directions for the Future
The comments provided by the librarians on the final evaluations, in debriefing sessions and in informal conversations on the MOO do not present a cohesive picture of what the next generation of real-time reference should be. Indeed the pilot itself raises several questions. These questions include:
If we can find satisfactory answers to these questions then our next question is "What tool can we use to deliver a more effective service?" No immediate clear direction emerges. The Internet Public Library has consistently tried to provide service to the greatest number of the on-line community. Recognizing that many users do not have high levels of connectivity, the library has provided service which conserves bandwidth and has chosen to mark-up its documents with html that is accessible to most browsers. The MOO environment, which only requires a telnet, connection fits with this philosophy. Although there are increasing numbers of "high tech" interactive tools that might provide considerably more interactivity for a real time reference service they frequently require that the user down-load a client before using the tool. For the casual "walk-in" reference patron this is would get them to the answer of their reference questions no more quickly than learning the MOO. In addition these tools often require considerable bandwidth.
However, several clear messages emerge from the comments of the librarians and the patrons. Any tool used to provide on-line reference must:
If we exclude from the discussion those interfaces that are highly graphical in nature, require the down-loading of a client or use significant amounts of bandwidth, we are left with a smaller set of options. These include: