IN REVISION 1/83
Use of Online Bibliographic Retrieval Systems:
Recent Research and Future Directions
With Discourse Analysis of Titles and Search Protocols
LS 663 Online Information Service Graduate School of Library Studies
Dr. Gerald Lundeen University of Hawaii
SUMMARY of ONLINE USE STUDIES∑∑∑∑∑∑∑∑∑∑∑..2
APPLIED PSYCHOLINGUISTICS in LIBRARY SCIENCE∑∑∑..3
The human activity which we term information-seeking or searching is accomplished through language use. Searchers use language or, more technically, discourse in order to obtain information. The actual information is received in discourse form. Information is őmade of‚ discourse and so thinking is necessarily in the form of discourse. We need not be aware of the discourse in our thinking because we automatically process language unselfconsciously. Thus, discourse is used at all levels in the search process. Searchers use discourse in their thinking in order to proceed from one decision point to the next during a search. They retrieve discourse in subject heading, descriptor, and title form, processing the information in their discourse thinking (James & Gordon, 1978).
Discourse thinking, a technical term used by Jakobovits & Gordon, may be more colloquially referred to as talking to oneself, having a mental conversation, thinking it over, etc. These everyday activities are related to searching behavior in thinking.
Researchers In several fields have used similar concepts when studying human mental functioning with the aim to understand and enhance human potential.
CONCEPT RESEARCHER FIELD OF STUDY
DISCOURSE THINKING JAMES SOCLAL PSYCHOLOGY
INNER SPEECH VYGOTSKY PSYCHOLOGY (RUSSIA)
VERBAL BEHAVIOR B F SKINNER PSYCHOLOGY
SELF-VERBALIZATION MEICHENBAUM PSYCHOLOGY
INTERIOR DIALOG JAMES SOCIAL PSYCHOLOGY
INTERNAL DIALOG CASTANEDA ANTHROPOLOGY
What goes on during these mental processes? This question is of interest to researchers in other related fields, such as,
Decision Science Information Science Communication
Management Science Cognitive Anthropology Cognitive Psychology
Since librarians are by definition purveyors of relevant ,discourse, i.e., providing the right title or source to the user; it behooves us to investigate the nature of titles, as well as the nature of human information processing of titles which lies in the discourse thinking of searchers. This is particularly true in the business of online bibliographic retrieval services where titles are the main commodity being traded. In striving to satisfy the informational needs of users librarians learn new skills, adapt, and change to improve accessibility of information to users. This striving issue has led as well to abundant research on how various online bibliographic retrieval systems affect their users, resulting in much continuous improvement of man-system compatibility (Bellardo, 1981).
The literature I read in this area and the literature summarized by Bellardo (1981) and Kiewitt (1979)~ though dealing with specific systems in local situations and varying in the measures obtained and the method of analysis employed to obtain those measures, held the following characteristic in common: All research efforts in examining online retrieval use analyzed discourse produced by the searcher-subjects and recorded in writing as notes, check marks on questionnaires, evaluative comments in interviews, or as print-outs of their search moves and resulting title references. Any analysis of information processing will necessarily involve analysis of discourse produced during the search process.
SUMMARY OF ONLINE USE STUDIES
The sample of studies read yielded the following measures of „useš:
Recall, Relevance, Precision
Unit cost per relevant citation
Costing and cost analysis
User assessment of value of search
Effectiveness of Training methods
Manual versus Online Contrasts
Interactive versus Solitary
Ratings and Statistical
Ratings and Statistics
Ratings and Experimental
Printųout, Protocol analysis,
Search Log statistics,
Computer Monitoring of transactions
The overall results of these studies is that users find the online bibliographic search worthwhile primarily because of the speed resulting in perceived time savings for the user. As well, the studies reported that users were satisfied with the abundance of references whether relevant or not be–cause they found them interesting or useful for various reasons. Finally, users consistently reported in these studies that they have the desire to do their own searches and find searches where they are present with a search intermediary very useful and more thorough than either their own searches or searches done for them by intermediaries alone.
There are several fields in the social sciences, as mentioned earlier which use methods of őlanguage use analysis,‚ termed discourse analysis. Since literature searching involves the use of discourse in mental decision-making i.e., thinking, it will be useful and illuminating to apply techniques of discourse analysis to the discourse generated in the process of searching.
Some recent online use studies report using what I would call techniques of discourse analysis on the print-out histories of searches. Penniman (1982) refers to his techniques as „protocol analysis of user transcripts.š Chapman refers to her method of search print-out analysis as „state transition analysisš where a state is defined as a different phase in the search as searching proceeds from command to command to the end results. Thus she developed a way of looking at patterns of commands used in searching to get to the goal of retrieving relevant titles. Fenichel (1981) in her attempt to
„discover those behaviors associated with the process of online
bibliographic searching that are correlated with success.š p.23
analyzed „search transcriptsš for five different groups of users with varying degrees of experience in online searching. She identified twelve „search process or search effort variablesš which she used to analyze the print-out histories. Some examples of these variables are, frequency of command types, search modification, errors, total number of descriptors, connect time, etc. These physical readings combined with questionnaire responses for relevance judgments by the searchers led her to conclude:
„The only clear cut differences that could be
attributed to experience were that the Novices
searched more slowly and made more errors than
the Experienced subjects. However, there is some evidence that
the searchers with the greatest overall experience who also had ERIC
database experience had higher values on a group of measures
called search effort variables than other Experienced subjects.š p.29
Her data showed that the Novices performed as well as Experienced searchers on the precision measure. She noted great variation in search patterns but similarity in retrieving relevant results.
In similar recent research in this area Sewell & Teitelbaum (1982) performed analyses of „traffic logsš of searches and their corresponding print-outs. Marcus (1982) reported on a technique termed „search effectiveness analysisš involving „unobtrusive monitoringš by the researcher of user activity, printųout analysis of the search, and notes provided by the searcher made during or directly after the search, and an interview with the searcher regarding their rationale for the search strategy used, problems encountered, usefulness of the search session, etc. However, Marcus did not present in this report detailed results of these analyses, except for the print-out data in similar fashion to the aforementioned studies.
Tessier et. al (1977) reports a very different yet related technique of discourse analysis using videotapes of searches. The user and intermediary searcher later separately viewed their tape. An „analystš was present at the viewing who helped them to be an audience to their own interpersonal search behavior patterns. The researchers reported that the videotape analysis allowed them to improve and change their search habits.
Williams & Curtis (1977) contrasted print-out analysis with questionnaire responses from users on their satisfaction ratings for two searches on the same topic. The two searches were performed in two conditions, either by the user alone and with an intermediary searcher, or by the intermediary searcher alone and with the user. Their main conclusion in this study is that the searches rated most satisfactory were the collaborative searches. The beneficial effects of collaboration were experienced in our lab search sessions where we worked in pairs on search exercises. Students consistently commented on the helpfulness of their partners to aid the search through appropriate interventions. These interventions involved pointing out errors of all kinds, suggesting alternative strategies, trying various pathways to see what results, reaffirming effective search logic, suggesting modifications or refinements, and others. In all this feedback among the student searchers in our lab, which may be regarded as discourse analysis for a special purpose, learning was enhanced by the partner giving relevant information.
Several questionnaire surveys have been used to obtain user‚s ratings on various aspects of the search process and the resulting title references. In a questionnaire survey by Hitchingham et.al (1982) on use of online bibliographic retrieval services for ready-reference questions, researchers intend to evidence widespread and growing use of this service for the ready-reference function in libraries. The results of their survey were to be reported at the ASIS 1982 Meeting at Columbus, Ohio. Their survey included acquiring a sample of ready-reference questions asked.
Schwerzel et.al (1982) required users to rate their own perceptions of their skill in searching. In an interview after performing searches users rated themselves on their ability to select appropriate databases, develop search strategies, and carry out the mechanics of searching. The rating scales used were confidence/difficulty in learning. They note that the end-users they studied rated themselves as willing to learn and confident in searching.
Warden (1981) administered an in-house questionnaire at GE headquarters asking new and regular users to rate the effectiveness of the service on several normative parameters, including
„the importance of inter-active feedback between
the user and the search intermediary in improving
search precision.š p.113.
Bellardo‚s (1981) review article of the online bibliographic retrieval use studies from 1972-1981 summarized over fifty different research projects. She points out that
„most of this research has been problem-oriented or concerned with examining existing systems, rather than with theoretical or philosophical issues (generally a characteristic of library and information science).š p.188
Concluding her review she calls for a fresh approach:
„New evaluation measures... sensitive to the
immediate, direct man-machine interaction of
online retrieval are needed.š p.209
Bellardo indicates a need to relate to the nature of the search process and how the user actually relates to the title references provided.
Similarly, Kiewitt (1979) in a review of online use studies concludes that the quality and amount of information available to searchers, users, or decision-makers influences the outcome of their decisions. In other words, awareness of alternatives in an information system facilitates freedom of choice of information in that system. Kiewitt refers to this as a šreceiver-controlledš information system.
The consensus among the reviewers I read is that further and future study of online use must take into account the users own context of experience and cognitive processes. The more recent studies take this issue into consideration in their design but due to a lack of theoretical framework for analysis results are inferential. I see the next phase of research focusing more on the searcher, whether end-user or intermediary, who will be trained to provide transcripts of decision-making steps they use in a given search. As well, the interactive effects of joint searching need to be examined in this way. The purpose of performing this type of descriptive research on searchers should be to study how searchers improve, adapt, and change their search habits as a direct result of the intervention of a partner to the search. In the field of psychology this is termed the „effects of modelingš in behavior change. Many researchers noted that searchers change habits readily when they perceive a more masterful method of obtaining their goal. This type of basic applied psycholinguistic research in library science is needed if we are to understand the nature of discourse use in information-seeking.
APPLIED PSYCHOLINGUISTICS in LIBRARY SCIENCE
In a study six years ago (Nahl, 1976) I extracted 100 titles from the vita of a psychology professor in the UH Psychology Department. I clustered these titles into five categories to which I assigned headings that were recognizable subject headings of the research specialty he works in. I gave these same titles to 20 graduate students in psychology, some in that specialty, and asked them to cluster the titles using the five categories I used. I found a 95% agreement among the clustering‚s of the graduate students and my own categorizations. This result proved to me that among people familiar with a topic domain, specialty, or field there exists a common mapping of the topic. Their ability to group the titles in a standard way prompted much thinking over the following years. Of particular interest to me were the ideas about the nature of discourse which I received from courses and study in social psychology, and within that psycholinguistics (James and Gordon, 1978).
I can best summarize what I learned about őwhat‚s in a title?‚ by referring to three distinct processing stages that I or others use reg–ularly when looking at a title. I shall describe the three stages by using illustrations. Let us consider the following title:
Scientific Research in Online retrieval: A Critical Review by Trudi Bellardo, 1981
The first stage of processing this title may be called „Expansionš in that the searcher attempts to expand the title‚s brevity into a more extended proposition or sentence. Thus: There is scientific research going on dealing with the activity of online retrieval and this article systematically samples that research and judges and prioritizes them on the basis of explicit standards. An expansion such as this one is always possible with titles. This is because a title is an encapsulated message from the author to the fellow researcher. Of course searchers may not at all be aware or conscious of engaging in this expansion process. In fact psycholinguists often emphasize that native speakers of a language though quite fluent are almost never capable of specifying grammatical relations or rules which however, are easily visible to a trained teacher or linguist. In the same way a searcher automatically and unselfconsciously expands the title of an article in order to derive its meaning or message.
Here is another example:
The Use of On-Line Information Retrieval Services
by P.W. Williams and J.M. Curtis, ASLIB Convention Program, 1977
This title needs to be considered within the context of its publication, after which it may be expanded as follows:
This article presents research on reactions of users to services provided for them as searchers of an information retrieval system.
Note that there is possibility of some error in the expansion in that mis-attributions occur as the searcher takes in and evaluates the significance of particular lexical items or other attendant information such as the place of publication, reputation of the author(s), and other norms followed by writers in that area. Some searchers will be more accurate than others in their expansion, and one would expect that experience and training would reduce important mis-attributions. Note also that expanded titles are similar to sentences one can find in the Abstract of articles. This is because they share similar information presentation functions. As well, when reading the abstract, the searcher is likely to find out the accuracy of the expansion.
Following the expansion stage, the second stage of processing a title may be called „Evaluation.š Here the searcher considers for relevance the information in the expansion. For example, in my search for articles to read for this term paper the title at the top of this page was retrieved. As soon as I interpreted it, that is made my expansion, I said to myself, „Oh goodie. Here is a user study!š) which indicates that I considered it to be relevant. A common method of recognizing the relevance of an expanded title is to use the language of classified indexes ūsince they contain a standardized lexicon that the searcher learns with experience and familiarity with the field. Thus, through knowledge of indexing language searchers can consider the appropriateness or relevance of the information in the title or in the expansion of the title.
Following expansion of the title, and following the evaluation of relevance, the third and final stage of perusing a title by searchers is „Judgment.š Here searchers select) valuate, and prioritize on the basis of explicit standards which promote their end purposes or uses, their final and ultimate perusal of the information. For example, at this point searchers decide to request for an abstract or even the entire article. This follows the searcher‚s judgment of anticipated validity, desirability, applicability, and usability of the information in the article or source. Investigating the searching process from this three-fold psycholinguistic perspective involves examining the thinking processes of searchers. In order to discover what implicit expansions are made by searchers, a self-report methodology must be employed. Searchers themselves must report on their on-going decision-making processes. Formats for reporting need to be evolved for this type of study. Students of L.A. James Social Psychology 222 course have used this method successfully to report their attributions in various social situations they experience. Currently, some of these students are reporting on tape recorders their on-going mental library search behavior. Jakobovits calls this training to thoughts out loud on a task „witnessing methodologyš since the
searcher must be a witness to their own mental functioning and make it explicit in oral or written transcripts (James & Gordon, 1978). I will be examining these student reports in order to evolve a method for studying the search process which other researchers may use.
What has been presented may not be sufficient to enable librarians to use discourse analysis techniques to study search behavior. Never the less a new direction is indicated in the marriage between the field of applied psycholinguistics and library studies. This discourse analysis methodology is a potentially important alternative to the merely statistical, thus empowering librarians with both tools. This direction may be summarized as a focus on being an audience to the self while searching and producing self-witnessing accounts. I can even see the possible usefulness of doing this activity self consciously on a regular basis. I predict that it will suggest new theories of search behavior.
For example, I noticed the following types of behavior among the students in our lab in pointing out errors a strategy to partners:
ųtypographical errors, syntax errors, logical errors, proximity or
ųfaster strategies to use for retrieval, adding or deleting descriptors
ųtrying out various syntax features of the system
ųpointing out significant proof in retrieved references which led to modifying and refining retrievals
ųgiving moral support when the mind goes blank or the machine goes off
ųrecalling particulars on databases which aids selecting the appropriate ones, aid in reading blue sheets and
printųout formats, asking pertinent questions.
Auster, Ethel „Organizational Behavior and Information Seeking: Lessons for Librarians.š Special Libraries 73(3), July 1982, 173-82.
Bellardo, Trudi „Scientific Research in Online Retrieval: A Critical Review.š
Library Research 3, 1981, 187-214.
Chapman, Janet L. „A State Transition Analysis of Online Information-Seeking
Behavior.š Journal of the American Society for Information Science
32(5), Sept. 1981, 325-33.
Fenichel, Carol H. „Online Searching: Measures that Discriminate among Users with Different Types of Experiences.š Journal of the American Society for Information Science 31(1), Jan. 1981, 23-32.
Hitchingham, Eileen and Titus, Elizabeth, and Pettengill, Richard „Online Services at the Reference Desk.š ASIS Proceedings 19, 1982, 133-34.
James, Leon A. & Gordon, B.Y. Workbook for the Study of Social Psychology, 2nd Edition University of Hawaii Psychology Depart–ment, 1978.
Kiewitt, Eva L. „Evaluating Information Retrieval Systems.š Chapter 3 in: Information Retrieval Systems: The PROBE Program, by Kiewitt, E.L., Westport, Conn.: Greenwood Press, 1979, 38-87.
Marcus, Richard S. „Computers Versus Humans as Search Intermediaries.š ASIS Proceedings 19, 1982, 182-85.
Nahl, Diane „An Empirical Method for the Study of Topic Domains in Psychologyš University of Hawaii Psychology Department, 1976.
Penniman, W. David „Modeling and Evaluation of On-Line User Behavior.š ASIS Proceedings 19, 1982, 231-35.
Schwerzel, Sharon W. and Emerson, Susan V. and Johnson, David L. „Self-Evaluation of Competencies in Online Searching by End-Users After Basic Training.š ASIS Proceedings 19, 1982, 272-75.
Sewell, Winifred and Teitelbaum, Sandra „Preliminary Observations of Non-Mediated Search Behavior of Pathologists and Pharmacists.š ASIS Proceedings 19, 1982, 276-78.
Tessier, Judith A. and Crouch, Wayne W. and Atherton, Pauline „New Measures of User Satisfaction with Computer-Based Literature Searches.š Special Libraries 68, Nov. 1977, 383-89.
Warden, Carolyn L. „User Evaluation of a Corporate Library Online Search Service.š Special Libraries 72(2), April 1981, 113-17.
Williams, P.W. and Curtis, J.M. „The Use of On-Line Information Retrieval Services.š ASLIB Program 11(1), Jan. 1977, 1-9.