(DRAFT)

IN REVISION 1/83

Use of Online Bibliographic Retrieval Systems:

Recent Research and Future Directions

With Discourse Analysis of Titles and Search Protocols

By

Dian Nahl-James

LS 663 Online Information ServiceÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ Graduate School of Library Studies

Dr. Gerald LundeenÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ University of Hawaii

December 6,1982

CONTENTS

INTRODUCTION····················..1

SUMMARY of ONLINE USE STUDIES···········..2

APPLIED PSYCHOLINGUISTICS in LIBRARY SCIENCE···..3

BIBLIOGRAPHY····················10

INTRODUCTION

The human activity which we term information-seeking or searching is accomplished through language use. Searchers use language or, more technically, discourse in order to obtain information. The actual information is received in discourse form. Information is Îmade ofâ discourse and so thinking is necessarily in the form of discourse. We need not be aware of the discourse in our thinking because we automatically process language unselfconsciously. Thus, discourse is used at all levels in the searchÊ process. Searchers use discourse in their thinking in order to proceed from one decision point to the next during a search. They retrieve discourse in subject heading, descriptor, and title form, processing the information in their discourse thinking (James & Gordon, 1978).

ÊÊÊÊÊÊÊÊÊÊÊ

ÊÊÊÊÊÊÊÊÊÊÊ Discourse thinking, a technical term used by Jakobovits & Gordon, may be more colloquially referred to as talking to oneself, having a mental conversation, thinking it over, etc. These everyday activities are related to searching behavior in thinking.

Researchers In several fields have used similar concepts when studying human mental functioning with the aim to understand and enhance human potential.

ÊÊÊ CONCEPTÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ RESEARCHERÊÊÊÊÊÊÊÊÊ FIELD OF STUDY

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ DISCOURSE THINKINGÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ JAMESÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ SOCLAL PSYCHOLOGY

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ PSYCHOLINGUISTICS

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ INNER SPEECHÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ VYGOTSKYÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ PSYCHOLOGY (RUSSIA)

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ VERBAL BEHAVIORÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ B F SKINNERÊÊÊÊÊÊÊÊÊÊÊÊÊ PSYCHOLOGY

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ SELF-VERBALIZATIONÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ MEICHENBAUMÊÊÊÊÊÊÊÊÊÊ PSYCHOLOGY

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ INTERIOR DIALOGÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ JAMESÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ SOCIAL PSYCHOLOGY

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ INTERNAL DIALOGÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ CASTANEDAÊÊÊÊÊÊÊÊÊÊÊÊÊÊ ANTHROPOLOGY

What goes on during these mental processes? This question is of interest to researchers in other related fields, such as,

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ Decision ScienceÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ Information ScienceÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ Communication

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ Management ScienceÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ Cognitive AnthropologyÊÊÊÊÊÊÊÊÊÊÊÊ Cognitive Psychology

Since librarians are by definition purveyors of relevant ,discourse, i.e., providing the right title or source to the user; it behooves us to investigate the nature of titles, as well as the nature of human information processing of titles which lies in the discourse thinking of searchers. This is particularly true in the business of online bibliographic retrieval services where titles are the main commodity being traded. In striving to satisfy the informational needs of users librarians learn new skills, adapt, and change to improve accessibility of information to users. This striving issue has led as well to abundant research on how various online bibliographic retrieval systems affect their users, resulting in much continuous improvement of man-system compatibility (Bellardo, 1981).

The literature I read in this area and the literature summarized by Bellardo (1981) and Kiewitt (1979)~ though dealing with specific systems in local situations and varying in the measures obtained and the method of analysis employed to obtain those measures, held the following characteristic in common: All research efforts in examining online retrieval use analyzed discourse produced by the searcher-subjects and recorded in writing as notes, check marks on questionnaires, evaluative comments in interviews, or as print-outs of their search moves and resulting title references. Any analysis of information processing will necessarily involve analysis of discourse produced during the search process.

SUMMARY OF ONLINE USE STUDIES

The sample of studies read yielded the following measures of ãuseä:

MEASURES Recall, Relevance, Precision Unit cost per relevant citation Costing and cost analysis Time savings User effort User assessment of value of search Effectiveness of Training methods Manual versus Online Contrasts Interactive versus Solitary Connect time Terminal time Search Phases

METHODS Ratings and Statistical Statistical Formula Statistical Formula Contrastive ratings Ratings Ratings Experimental set÷up Ratings and Statistics Ratings and Experimental Statistics Statistics Print÷out, Protocol analysis, Search Log statistics, Computer Monitoring of transactions

The overall results of these studies is that users find the online bibliographic search worthwhile primarily because of the speed resulting in perceived time savings for the user. As well, the studies reported that users were satisfied with the abundance of references whether relevant or not beÐcause they found them interesting or useful for various reasons. Finally, users consistently reported in these studies that they have the desire to do their own searches and find searches where they are present with a search intermediary very useful and more thorough than either their own searches or searches done for them by intermediaries alone.

There are several fields in the social sciences, as mentioned earlier which use methods of Îlanguage use analysis,â termed discourse analysis. Since literature searching involves the use of discourse in mental decision-making i.e., thinking, it will be useful and illuminating to apply techniques of discourse analysis to the discourse generated in the process of searching.

Some recent online use studies report using what I would call techniques of discourse analysis on the print-out histories of searches. Penniman (1982) refers to his techniques as ãprotocol analysis of user transcripts.ä Chapman refers to her method of search print-out analysis as ãstate transition analysisä where a state is defined as a different phase in the search as searching proceeds from command to command to the end results. Thus she developed a way of looking at patterns of commands used in searching to get to the goal of retrieving relevant titles. Fenichel (1981) in her attempt to

ãdiscover those behaviors associated with theÊ process of online

bibliographic searching that are correlated with success.ä p.23

analyzed ãsearch transcriptsä for five different groups of users with varying degrees of experience in online searching. She identified twelve ãsearch process or search effort variablesä which she used to analyze the print-out histories. Some examples of these variables are, frequency of command types, search modification, errors, total number of descriptors, connect time, etc. These physical readings combined with questionnaire responses for relevance judgments by the searchers led her to conclude:

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ ãThe only clear cut differences that could be

attributed to experience were that the Novices

searched more slowly and made more errors than

the Experienced subjects. However, there is some evidence that

the searchers with the greatest overall experience who also had ERIC

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ database experience had higher values on a group of measures

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ called search effort variables than other Experienced subjects.ä p.29

Her data showed that the Novices performed as well as Experienced searchers on the precision measure. She noted great variation in search patterns but similarity in retrieving relevant results.

ÊÊÊÊÊÊÊÊÊ

ÊÊÊÊÊÊÊÊÊ In similar recent research in this area Sewell & Teitelbaum (1982) performed analyses of ãtraffic logsä of searches and their corresponding print-outs. Marcus (1982) reported on a technique termed ãsearch effectiveness analysisä involving ãunobtrusive monitoringä by the researcher of user activity, print÷out analysis of the search, and notes provided by the searcher made during or directly after the search, and an interview with the searcher regarding their rationale for the search strategy used, problems encountered, usefulness of the search session, etc. However, Marcus did not present in this report detailed results of these analyses, except for the print-out data in similar fashion to the aforementioned studies.

ÊÊÊÊÊÊÊÊÊ

ÊÊÊÊÊÊÊÊÊ Tessier et. al (1977) reports a very different yet related technique of discourse analysis using videotapes of searches. The user and intermediary searcher later separately viewed their tape. An ãanalystä was present at the viewing who helped them to be an audience to their own interpersonal search behavior patterns. The researchers reported that the videotape analysis allowed them to improve and change their search habits.

ÊÊÊÊÊÊÊÊ

ÊÊÊÊÊÊÊÊ Williams & Curtis (1977) contrasted print-out analysis with questionnaire responses from users on their satisfaction ratings for two searches on the same topic. The two searches were performed in two conditions, either by the user alone and with an intermediary searcher, or by the intermediary searcher alone and with the user. Their main conclusion in this study is that the searches rated most satisfactory were the collaborative searches.Ê The beneficial effects of collaboration were experienced in our lab search sessions where we worked in pairs on search exercises. Students consistently commented on the helpfulness of their partners to aid the search through appropriate interventions. These interventions involved pointing out errors of all kinds, suggesting alternative strategies, trying various pathways to see what results, reaffirming effective search logic, suggesting modifications or refinements, and others. In all this feedback among the student searchers in our lab, which may be regarded as discourse analysis for a special purpose, learning was enhanced by the partner giving relevant information.

ÊÊÊÊÊÊÊÊ

ÊÊÊÊÊÊÊÊ Several questionnaire surveys have been used to obtain userâs ratings on various aspects of the search process and the resulting title references. In a questionnaire survey by Hitchingham et.al (1982) on use of online bibliographic retrieval services for ready-reference questions, researchers intend to evidence widespread and growing use of this service for the ready-reference function in libraries. The results of their survey were to be reported at the ASIS 1982 Meeting at Columbus, Ohio. Their survey included acquiring a sample of ready-reference questions asked.

Schwerzel et.al (1982) required users to rate their own perceptions of their skill in searching. In an interview after performing searches users rated themselves on their ability to select appropriate databases, develop search strategies, and carry out the mechanics of searching. The rating scales used were confidence/difficulty in learning. They note that the end-users they studied rated themselves as willing to learn and confident in searching.

Warden (1981) administered an in-house questionnaire at GE headquarters asking new and regular users to rate the effectiveness of the service on several normative parameters, including

ãthe importance of inter-active feedback between

the user and the search intermediary in improving

ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ search precision.ä p.113.ÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊ

Bellardoâs (1981) review article of the online bibliographic retrieval use studies from 1972-1981 summarized over fifty different research projects. She points out that

ãmost of this research has been problem-oriented or concerned with examining existing systems, rather than with theoretical or philosophical issues (generally a characteristic of library and information science).ä p.188

Concluding her review she calls for a fresh approach:

ãNew evaluation measures... sensitive to the

immediate, direct man-machine interaction of

online retrieval are needed.ä p.209

Bellardo indicates a need to relate to the nature of the search process and how the user actually relates to the title references provided.

Similarly, Kiewitt (1979) in a review of online use studies concludes that the quality and amount of information available to searchers, users, or decision-makers influences the outcome of their decisions. In other words, awareness of alternatives in an information system facilitates freedom of choice of information in that system. Kiewitt refers to this as a äreceiver-controlledä information system.

The consensus among the reviewers I read is that further and future study of online use must take into account the users own context of experience and cognitive processes. The more recent studies take this issue into consideration in their design but due to a lack of theoretical framework for analysis results are inferential. I see the next phase of research focusing more on the searcher, whether end-user or intermediary, who will be trained to provide transcripts of decision-making steps they use in a given search.Ê As well, the interactive effects of joint searching need to be examined in this way. The purpose of performing this type of descriptive research on searchers should be to study how searchers improve, adapt, and change their search habits as a direct result of the intervention of a partner to the search. In the field of psychology this is termed the ãeffects of modelingä in behavior change. Many researchers noted that searchers change habits readily when they perceive a more masterful method of obtaining their goal. This type of basic applied psycholinguistic research in library science is needed if we are to understand the nature of discourse use in information-seeking.

APPLIED PSYCHOLINGUISTICS in LIBRARY SCIENCE

In a study six years ago (Nahl, 1976) I extracted 100 titles from the vita of a psychology professor in the UH Psychology Department. I clustered these titles into five categories to which I assigned headings that were recognizable subject headings of the research specialty he works in. I gave these same titles to 20 graduate students in psychology, some in that specialty, and asked them to cluster the titles using the five categories I used. I found a 95% agreement among the clusteringâs of the graduate students and my own categorizations. This result proved to me that among people familiar with a topic domain, specialty, or field there exists a common mapping of the topic. Their ability to group the titles in a standard way prompted much thinking over the following years. Of particular interest to me were the ideas about the nature of discourse which I received from courses and study in social psychology, and within that psycholinguistics (James and Gordon, 1978).

I can best summarize what I learned about Îwhatâs in a title?â by referring to three distinct processing stages that I or others use regÐularly when looking at a title. I shall describe the three stages by using illustrations. Let us consider the following title:

Scientific Research in Online retrieval:Ê A Critical Review by Trudi Bellardo, 1981

The first stage of processing this title may be called ãExpansionä in that the searcher attempts to expand the titleâs brevity into a more extended proposition or sentence. Thus:Ê There is scientific research going on dealing with the activity of online retrieval and this article systematically samples that research and judges and prioritizes them on the basis of explicit standards.Ê An expansion such as this one is always possible with titles. This is because a title is an encapsulated message from the author to the fellow researcher. Of course searchers may not at all be aware or conscious of engaging in this expansion process. In fact psycholinguists often emphasize that native speakers of a language though quite fluent are almost never capable of specifying grammatical relations or rules which however, are easily visible to a trained teacher or linguist. In the same way a searcher automatically and unselfconsciously expands the title of an article in order to derive its meaning or message.

Here is another example:

The Use of On-Line Information Retrieval Services

by P.W. Williams and J.M. Curtis, ASLIB Convention Program, 1977

This title needs to be considered within the context of its publication, after which it may be expanded as follows:

This article presents research on reactions of users to services provided for them as searchers of an information retrieval system.

Note that there is possibility of some error in the expansion in that mis-attributions occur as the searcher takes in and evaluates the significance of particular lexical items or other attendant information such as the place of publication, reputation of the author(s), and other norms followed by writers in that area. Some searchers will be more accurate than others in their expansion, and one would expect that experience and training would reduce important mis-attributions. Note also that expanded titles are similar to sentences one can find in the Abstract of articles. This is because they share similar information presentation functions. As well, when reading the abstract, the searcher is likely to find out the accuracy of the expansion.

Following the expansion stage, the second stage of processing a title may be called ãEvaluation.ä Here the searcher considers for relevance the information in the expansion. For example, in my search for articles to read for this term paper the title at the top of this page was retrieved. As soon as I interpreted it, that is made my expansion, I said to myself, ãOh goodie. Here is a user study!ä) which indicates that I considered it to be relevant. A common method of recognizing the relevance of an expanded title is to use the language of classified indexes ðsince they contain a standardized lexicon that the searcher learns with experience and familiarity with the field. Thus, through knowledge of indexing language searchers can consider the appropriateness or relevance of the information in the title or in the expansion of the title.

Following expansion of the title, and following the evaluation of relevance, the third and final stage of perusing a title by searchers is ãJudgment.ä Here searchers select) valuate, and prioritize on the basis of explicit standards which promote their end purposes or uses, their final and ultimate perusal of the information. For example, at this point searchers decide to request for an abstract or even the entire article. This follows the searcherâs judgment of anticipated validity, desirability, applicability, and usability of the information in the article or source.Ê Investigating the searching process from this three-fold psycholinguistic perspective involves examining the thinking processes of searchers. In order to discover what implicit expansions are made by searchers, a self-report methodology must be employed. Searchers themselves must report on their on-going decision-making processes. Formats for reporting need to be evolved for this type of study. Students of L.A. James Social Psychology 222 course have used this method successfully to report their attributions in various social situations they experience. Currently, some of these students are reporting on tape recorders their on-going mental library search behavior. Jakobovits calls this training to thoughts out loud on a task ãwitnessing methodologyä since the

searcher must be a witness to their own mental functioning and make it explicit in oral or written transcripts (James & Gordon, 1978). I will be examining these student reports in order to evolve a method for studying the search process which other researchers may use.

ÊÊÊÊÊÊÊÊÊ

ÊÊÊÊÊÊÊÊÊ What has been presented may not be sufficient to enable librarians to use discourse analysis techniques to study search behavior. Never the less a new direction is indicated in the marriage between the field of applied psycholinguistics and library studies. This discourse analysis methodology is a potentially important alternative to the merely statistical, thus empowering librarians with both tools. This direction may be summarized as a focus on being an audience to the self while searching and producing self-witnessing accounts. I can even see the possible usefulness of doing this activity self consciously on a regular basis. I predict that it will suggest new theories of search behavior.

FOOTNOTE

For example, I noticed the following types of behavior among the students in our lab in pointing out errors a strategy to partners:

÷typographical errors, syntax errors, logical errors, proximity or

ÊÊÊ free÷text errors

÷faster strategies to use for retrieval, adding or deleting descriptors

ÊÊÊ (inclusiveness/redundancy)

÷trying out various syntax features of the system

÷pointing out significant proof in retrieved references which led to modifying and refining retrievals

÷giving moral support when the mind goes blank or the machine goes off

÷recalling particulars on databases which aids selecting the appropriate ones, aid in reading blue sheets and

ÊÊÊ print÷out formats, asking pertinent questions.

BIBLIOGRAPHY

Auster, Ethel ãOrganizational Behavior and Information Seeking: Lessons for Librarians.ä Special Libraries 73(3), July 1982, 173-82.

Bellardo, Trudi ãScientific Research in Online Retrieval: A Critical Review.ä

Library Research 3, 1981, 187-214.

Chapman, Janet L. ãA State Transition Analysis of Online Information-Seeking

Behavior.ä Journal of the American Society for Information Science

32(5), Sept. 1981, 325-33.

Fenichel, Carol H. ãOnline Searching: Measures that Discriminate among Users with Different Types of Experiences.ä Journal of the American Society for Information Science 31(1), Jan. 1981, 23-32.

Hitchingham, Eileen and Titus, Elizabeth, and Pettengill, Richard ãOnline Services at the Reference Desk.ä ASIS Proceedings 19, 1982, 133-34.

James, Leon A. & Gordon, B.Y. Workbook for the Study of Social Psychology, 2nd Edition University of Hawaii Psychology DepartÐment, 1978.

Kiewitt, Eva L. ãEvaluating Information Retrieval Systems.ä Chapter 3 in: Information Retrieval Systems: The PROBE Program, by Kiewitt, E.L., Westport, Conn.: Greenwood Press, 1979, 38-87.

Marcus, Richard S. ãComputers Versus Humans as Search Intermediaries.ä ASIS Proceedings 19, 1982, 182-85.

Nahl, Diane ãAn Empirical Method for the Study of Topic Domains in Psychologyä University of Hawaii Psychology Department, 1976.

Penniman, W. David ãModeling and Evaluation of On-Line User Behavior.ä ASIS Proceedings 19, 1982, 231-35.

Schwerzel, Sharon W. and Emerson, Susan V. and Johnson, David L. ãSelf-Evaluation of Competencies in Online Searching by End-Users After Basic Training.ä ASIS Proceedings 19, 1982, 272-75.

Sewell, Winifred and Teitelbaum, Sandra ãPreliminary Observations of Non-Mediated Search Behavior of Pathologists and Pharmacists.ä ASIS Proceedings 19, 1982, 276-78.

Tessier, Judith A. and Crouch, Wayne W. and Atherton, Pauline ãNew Measures of User Satisfaction with Computer-Based Literature Searches.ä Special Libraries 68, Nov. 1977, 383-89.

Warden, Carolyn L. ãUser Evaluation of a Corporate Library Online Search Service.ä Special Libraries 72(2), April 1981, 113-17.

Williams,ÊÊÊÊ P.W. and Curtis, J.M. ãThe Use of On-Line Information Retrieval Services.ä ASLIB Program 11(1), Jan. 1977, 1-9.

BACK TO INDEX