TAXONOMY OF LIBRARY SKILLS AND ERRORS
FROM DISCOURSE ANALYSIS OF SEARCH PROTOCOLS
Diane Nahl-Jakobovits James
April 28, 1983
LS 695 Research Methods in Librarianship Dr. Therese Bissen Bard
TABLE OF CONTENTS
1. RESEARCH PROBLEM
The Implicit Librarian………………………………………………………….1
Library Searching as a Problem-Solving Activity……………………………2
Discourse Analysis of Search Protocols…………………………………….3
Table 1: Theoretical Taxonomy of Library Skills…………………………….4
Questions & Hypotheses……………………………………………………..5
2. PROCEDURES & ANALYSIS
Research Design & Methodology……………………………………………..7
Figure 1: Design of the Study………………………………………………….8
Data Gathering Activity…………………………………………………………9
Analysis of Results…………………………………………………………….10
Statistical Analysis of Data…………………………………………………….11
1. RESEARCH PROBLEM
The field of library science needs a comprehensive theory of library use in order to systematically investigate and explain all types of library phenomena. Theory generates research hypotheses which may be tested, in turn yielding practical solutions to library use issues. A taxonomy is the beginning of a theory if the scheme is made explicit by identifying the presuppositions of the taxonomy (Krathwohl & Bloom et al, 1964). In this proposed field study the objective is the construction of an empirical taxonomy of library skills and errors as evidenced in the process of searching for information in an academic library. This research approach assumes that library skills are dependent on stratified problem-solving and information evaluation sub-skills. Further, library skills are dependent on motivational and attitudinal factors.
Such a taxonomy would allow researchers and practitioners in the field:
i. to determine the kind of library instruction that improves student use of the library.
ii. to test library skills
iii. to determine what problem-solving steps are involved in library searching.
iv. to identify discrete items of library behavior that fit into known domains and zones of behavior
THE IMPLICIT LIBRARIAN
Wegner & Vallacher (1977) in their book Implicit Psychology present a model of how the “implicit psychologist” within us uses social experience to evolve personal psychological theories to explain and rationalize daily round events. They cite studies which illustrate how people construct social reality in their mental (cognitive) processes. The authors argue that this process of rationally explaining to ourselves the implications of events in our daily life is critical to social existence.
Similarly, we can view each person as having an “implicit librarian” with a personal cataloging system, complete with filing rules in memory for storage and retrieval of information. Simon (1981) equates human cognition with information processing. Since the library is actually modeled upon library users’ requirements for storage and retrieval of external information (“explicit librarian”), therefore the library is an image of the “implicit librarian.” Hence, if we identify the taxonomy of library skills and errors of the “implicit librarian” in users we will have identified the theory of library use that would lead to valid library practices and better use.
LIBRARY SEARCHING AS A PROBLEM-SOLVING ACTIVITY
Researchers in the fields of librarianship, education, psychology, and information and decision science (Rogers, 1980; Bloom & Border, 1950; Bloom & Krathwohl, 1956 & 1964; Simon, 1981) over the past thirty years have stressed the necessity for systematic investigations of the decision-making steps, thought sequences, and actions people carry out when searching for a solution to a particular problem. In the case of librarianship a search for specific information in the library may be regarded as a problem-solving activity. Problem-solving involves a sequence of behaviors performed by the searcher either visibly (actions and movements) or invisibly (feelings and thoughts). In order to make visible thought processes researchers developed the idea of “protocol analysis” of “thinking aloud problem-solving records” (Bloom & Broder, 1950) or “recording verbal protocols” (Simon, 1981) where the subject thinks aloud into a tape recorder while solving a problem, thus making explicit the usually implicit decisions and strategies.
This method of generating data on higher mental processes is fruitful because we can explicitly determine how these data relate to actions taken by problem solvers, what skills are involved in searching for solutions, what types of errors are made and at which point, how errors could have been avoided, or what alternatives are seen and used. Hence, if we employ a method of self-verbalizing the inner speech produced during a library search for information we can obtain on-going verbal reports which empirically document the steps involved in the search process. We can thus come to an understanding of what the library user must go through step-by-step in seeking and finding information in the library.
Rogers (1980), in her review of the recent undergraduate bibliographic instruction literature, determined that
There is no clearly defined concept of research strategies--or search strategies…Generally it is used to refer to some sort of systematic approach to information. (p. 69)
As well, Ford (1979) emphasizes the increasing need and pressure on students to ‘learn how to learn’-—skills which include finding and making effective use of information sources. (p. 252)
These, as well as other, researchers in librarianship 1 realize the importance of documenting user search behavior (Hardesty, 1979; Nahl-James, 1982). My own review of recent user studies in libraries confirmed the view that librarians need methods and data that allow them to see ‘library use’ from the user perspective in order to provide appropriate service.2
In order to generate verbal reports on search sequences, subjects must be trained in a self-observation method that allows them to identify their own behavioral units, thought processes, and to verbally describe physical actions give microdescriptions of sequences of behavior, and catalog the behavioral units according to a theory of self.3
For the purpose of this study the Self-Witnessing Method and the Three Fold Self Ennead Matrix will form the methodological and theoretical basis for generating, documenting, and analyzing the on-going verbal reports of the steps and aspects involved in searching for information in the library (Jakobovits 1983). James’ theory is independently congruent with Bloom et al (1956) and Krathwohl et al (1964): the Taxonomy of Educational Objectives which sets forth a developmental taxonomy of skills in education for curriculum development and testing of students.
DISCOURSE ANALYSIS OF SEARCH PROTOCOLS
We can tell intuitively as librarians that it makes sense to ask—-what are user skills? Also intuitively we can expect that library skills are organized from simple to complicated. Finally, we can intuitively know that user skills would lie in different domains: some skills having to do with the user’s purpose, motivation, or intent; some skills with the user’s plans or means; and some with the user’s actual executions.
Using the system proposed by James and the work of Bloom and Krathwohl I constructed a Theoretical Taxonomy of Library Skills through a content analysis of the discourse produced in the verbal protocols generated in a pilot study of this approach. This is presented in Table 1 p. 4.
Inspection of the Table will reveal that search skills have a hierarchical arrangement as indicated by Levels. Furthermore, the Table shows how search skills are interconnected and influence each other across the Domains (Affective Skills, Cognitive Skills, Sensorimotor Skills). This Table constitutes a theory of library search behavior. The search protocol data to be collected will be organized by reference to this Table and will show whether my alignment of skills is supported by the facts. If the Table is found to be valid it can serve as the beginning of an integrated and comprehensive theory of library which further research will discover.
QUESTIONS & HYPOTHESES
This proposed research concerns the following questions;
i. Are library skills and errors stratified?
ii. What is the degree of correlation between library system (explicit librarian) and the user’s own organization of information (implicit librarian)?
iii. Do library skills and errors correlate with general information-seeking habits and strategies or are they specialized?
iv. Are errors situationally caused or caused by the personality of the searcher?
v. Are errors interrelated or dependent?
vi. Is it possible to deepen the relationship between librarian and user so that the “implicit librarian” of each could influence one another which is a far superior relationship than survey feedback affords.
Social Condition: two social conditions Dyadic and Solitary will test the interpersonal effect of searching such that some Subjects will search alone and some with partners.
Task Level: each Level will be defined according to a Comprehensive Discourse Analysis scheme developed by James (1983). This system requires some training for its use. Table 1 lists specific examples I constructed using definitions given in James (1983). Many more examples in each box can be constructed. Comparison of this Table with Bloom and Krathwohls’ skill continuum shows a close agreement on the nature of the skills. (See Appendix _) Domain of Behavior: each Domain will be defined according to James and Bloom and Krathwohl: Affective Domain including goals, intentions, purposes, interests, attitudes, values, appreciation, emotion, acceptance/rejection, biases, etc. Cognitive Domain including thinking, reasoning, reflecting, understanding, recognizing, problem-solving, insight, mental acts, remembering, etc. Sensorimotor Domain including overt actions, eye movements, hand-eye coordination, physiological reactions, etc.
There will be two: Identified Skill Types and Identified Error Types in the protocols generated.
Hypothesis 1 There will be more skill and error types identified from the data reports generated by Subjects in the Dyadic Social Condition.
Hypothesis 2 The number of Skill and Error Types in each Level (1,11,111) generated from the data reports will correspond in numerical size to these three levels, such that Task Level 1 will generate more Level I entries, etc.
Hypothesis 3 The frequency of Skill and Error Types for the three Domains of behavior will be equally rich.
i. The Three-Fold Self Ennead Matrix consisting of the Levels of human behavior (James, 1983),
ii. The order of the Do ins and Levels,
iii. Transformation of a developmental continuum into three discrete Levels (Bloom et al, 1956; Krathwohl et al, 1964; James, 1983),
iv. Variation in ability of the Subjects is assumed to be normal; results will allow better pre-test measures to determine ability or experience of Subjects,
v. The validity of Analysis Method of the Self-Witnessing Method and the Comprehensive Discourse analysis of verbal protocols.
2. PROCEDURES & ANALYSIS
The Subjects in the proposed field study will be trained undergraduate students enrolled in an introductory social psychology class at the University of Hawaii.4 The training they receive for the data gathering activity is an integral part of their course work. As well, they will be given lectures in class instructing them in how to use the Self Witnessing Method to investigate their own habits and strategies in doing library research for a course assignment.
RESEARCH DESIGN & METHODOLOGY
The subjects will be assigned to one of two Social Conditions: Dyadic or Solitary. The research design is indicated in Figure 1. Subjects randomly assigned to the Dyadic Condition will pair up in class by random selection by the professor or according to some criteria to be decided. The Dyads will then be randomly assigned to complete one of three library research tasks as part of a course assignment. The subject matter of the three tasks is necessarily course related, thereby involving the student’s striving or motivation in accomplishing the assigned task.
Groups 1, 2, & 3 share the Dyadic Social Condition but differ on Task Level, with Group 1 receiving the simplest task, Group 2 a more complex task, and Group 3 the highest level task. Groups 4, 5, & 6 share the Solitary Social Condition, differing on Task Level with Group 4 receiving the simplest task, Group 5 more complex, & Group 6 the highest level task.
Twenty Subjects will be assigned to each Group yielding 6 x 20 = 120 total Subjects, 40 Subjects per task (Dyadic + Solitary), and 60 Subjects per Social Condition (Tasks I, II, III).
The three library research tasks (I, II, III) correspond to the three developmental levels on the Ennead Matrix for the Theoretical Taxonomy (see: Table 1, p. 4), such that Task I is “Find a Book”, consisting in locating the correct catalog card entry from prechosen titles and then locating the book on the shelf; Task II is “Trace a Topic”, consisting in locating varied sources throughout history and presenting summaries of varied viewpoints from writers on a prechosen course topic; Task III is “Contrastive Textbook Critique”, consisting in locating varied social psychology textbook views on the topics in one prechosen chapter of the course textbook and providing a synthesis of the views.5
Data Gathering Activity
The dyads will go to Hamilton Library to work together on the same task, each partner performing one of two Social Roles: Searcher or Huddle-buddy. The Searcher will carry out the library research task instructions while the Huddle-buddy will tape record the Searcher’s thinking-out—loud sequences throughout the search process until the task is completed. The Huddle-buddy will encourage the searcher to verbalize ongoing decisions for the record and will take notes of their own observations of the Searcher’s behavior during the task. The reporting format for the observations will be the Ennead Matrix, as the tasks are also an application for the students of their training in the use of the Matrix.6 The Huddle-buddy will be instructed to give the Searcher feedback at times when the Searcher is blocked to alternatives so that the search may continue, however, the major role of the Huddle-buddy as partner is to facilitate the verbalizing and thinking—out-loud of the ongoing decision-making process of the Searcher.
Upon completion of the search task the partners in the Dyad will listen to the tape, meet to discuss what occurred, and jointly categorize their observations of behaviors occurring during searching, including categorizing particular sequences from the tape according to the particular Domain of the particular behavior (see: Table 1, p.9) (A=Affective, B= Cognitive, C=Sensorimotor). They will then each write a separate report including the categorization matrix on their findings to complete the assignment; they will submit both reports with their tape.
Subjects in the Solitary Condition will go to Hamilton Library alone after receiving the prechosen titles or topics from the professor in class. They will use a tape recorder to record their own thinking out-loud sequences as they perform the search task. They will later listen to their tape and categorize their behavioral sequences according to the Ennead Matrix in the three Domains of behavior. They will write a report of their findings gleaned from their Matrix analysis and submit it along with their tape.5
ANALYSIS of RESULTS
Since the data (behavioral descriptions) are reported by trained Subjects the scoring problem is greatly reduced and entails only identifying the Subject’s self-description entry as a skill inventory. For example, the following brief inventory of skills and errors is obtained from applying the scoring technique in Table 1, p. ‘f-, to a segment of a self-witnessing report done under a “solitary social condition” obtained in a pilot study in 1982. A plus (-i-) is for an identified skill item, minus (-) for errors.
-IIA +IIB +IIA +IIB -IIA -IIIC
7:36 PM “Gosh, there is not even one book on the other topics.
Well, I’ll just look up Self-Actualization.
I hope that it is o.k.
7:38 Now I’ll look up Library Searching.
7:39 Only one book on that subject too.
I feel so depressed.”
-IIA Not trusting the System
+IIA Continuing Desire to Develop the Ability to Search for & Locate Information Effectively
+IIB Operational Sub-routines for Searching Used
-IIIC Satisfaction in Accomplishing a Successful Search
This Subject’s search performance profile score may have been different under a Dyadic Social Condition. Identified Skill Items and Error Items will be hierarchically presented from simple to complex in the three Domains of behavior (Affective, Cognitive, Sensorimotor) as seen in Table 1, p. 4. Frequencies for each box in Table 1 will be Types of Skills and Types of Errors identified (see: Figure 1, p. 8).
Operational Definitions of Variables-Independent Variables & Dependent Measures
There will be two dependent measures: Types of Skills identified & Types of Errors identified. The independent variables are three: Social Condition (Dyadic vs. Solitary), Task Level (I, II, III), and Domain (A, B, C). Thus the results will show various types of skills and errors as a function of Social Condition, Task Level, and Domain. An example may be seen below in the Analysis section.
Social Condition: (see: p. 5) Subjects will work together or alone on assigned library research tasks in social
Task Level: (see: p. 5) Three library research tasks on course topics are assigned to Subjects in both Social
Conditions, each task at a different level of complexity and difficulty according to the Theory of
Library, Table 1, p. 4.
Domain: Domain: (see: p. 6) Three Domains of Behavior are specified in Table 1.
Inventory Items: Skills & Errors: Skills will be identified as those entries or sequences which show a positive
direction in accomplishing the search; Errors will be identified by those items showing a
negative direction or blocking or impeding completion of the search task. See example p. 10.
Statistical Analysis of the Data
Since Subjects will be randomly assigned to the two independent conditions (Dyadic vs. Solitary & Task Level I vs. II vs. III), we would not expect a systematic difference among the groups in library familiarity, experience, and knowledge, or in intelligence or problem-solving capacity. This is especially true in this design since it uses college students within the context of a psychology course, which works for homogeneity of variance across the groups in ability.
The effect of Social Condition will be tested by a Chi Square Contingency Matrix comparing the overall frequency for Groups 1,2,3 as against the overall frequency for Groups 4,5,6. (See: Figure 1, p. 8). Hypothesis 1
says more inventory items will be generated from the data reports under the Dyadic Condition.
The effect of Task Level will be tested by comparing the overall frequency for Groups 1,4 as against Groups 2,5 as against Groups 3,6. A separate Chi Square Contingency Matrix is to be calculated for each of the three theoretically defined levels (see: Table 1. p. 4). This is indicated in Figure 1 by broken lines. Hypothesis 2 is that the number of inventory items generated from the data reports will correspond in numerical size to these three levels. Thus, “Find a Book”, task (Level I, Groups 1,4) will generate more Level I items as defined by Table I than Level II and Level III items; but “Trace a Topic” task (Level II, Groups 2, 5) will generate more Level II items than Level I or Level III items. Finally, “Contrastive Textbook Critique” task (Level III, Groups 3, 6) will generate more Level III items than Level I and II items.
The effect of Domain will be tested by a Chi Square Contingency Matrix comparing the overall frequency of inventory items in the six “A” boxes 1,6,9,12,15,18 as against the six “B” boxes 2,5,8,11,14,17 as against the six “C” boxes 3,4,7,10,13,16. Hypothesis 3 says that the frequency of inventory items for the three Domains will be equally rich, i.e., the three frequencies will not be significantly different from each other.
Interaction effects will also be examined though at this stage it is difficult to make predictions.
Results will be presented as contingency tables and interaction effects will be graphed for better presentation.
LIMITATIONS OF THIS STUDY
A more adequate sampling of tasks would provide a more comprehensive database of skill types and error types for the taxonomy. Future research is planned which will sample several search tasks in each of the three Levels.
The special training that is required for Subjects to gather and report the data within the framework of a particular college course is an obvious limitation. The self-witnessing method is exportable to researchers in other locations and fields with adequate instruction and practice.8
Similarly, there is specialized skill involved in using the Ennead Matrix system for categorizing human affairs. This system is exportable and will be further explicated in forthcoming publications by Jakobovits. The validity of the system has not been tested empirically heretofore.
1. Researchers in the field of online bibliographic searching use various types of sequence analysis methods to study
the “print-out histories” of particular searches conducted under both interpersonal search conditions and solitary search conditions. These methods of analyzing the discourse produced by the computer through the searcher’s
command sequences which interact with particular databases, promises to aid in identifying effective online
search protocols that will save time, effort, and cost. See my LS 663 paper listed in the References for a review of hat recent literature.
2. See the references in the Bibliography of Course Readings.
3. See samples of self-witnessing reports by the students of Social Psychology 222, “Community—Classroom,” Generational Curriculum, Daily Round Archives, Psychology Dept., University of Hawaii, 1975-1983.
4. Psychology 222, Introduction to Social Psychology, as taught by L.A. Jakobovits, emphasizes acquisition of a social psychological method of analysis (SWM) used by students to study and analyze their own steps in learning the content of social psychology.
5. See Appendix — for a sample set of written instructions given to students in a pilot study if this research in 1982.
6. See James reference for a definition of the category system.
7. Source: Self-Witnessing Data Report, J.C. Fall, 1982, p.10. Daily Round Archives, University of Hawaii, Psychology Dept.
8. See James reference in the Bibliography of Course Readings for an explanation of the training procedure.
Bloom, Benjamin S. & Lois J. Broder. The Problem Solving Processes of College Students: An Exploratory Investigation. Chicago, Ill.: University of Chicago Press, 1950.
Bloom, Benjamin S. Ed. Taxonomy of Educational Objectives: The Classification of Educational Goals: Handbook I. The Cognitive Domain. New York:
David McKay Co., 1956.
Ford, Nigel. “Towards a Model of ‘Library Learning' in Educational Systems.” Journal of Librarianship 11(4), Oct 1979:247—260.
Hardesty, Larry & Nicholas Lovrich Jr. & James Mannon. “Evaluating Library-Use Instruction.” College and Research Libraries 40, July 1979:309-317.
James, L.A. “Towards a Comprehensive Approach to Discourse Analysis and Its Applications.” Hawaii: Faculty Symposium Talk, English as a Second Language Dept., University of Hawaii, March 1983, (mimeo).
Krathwohl, David R. & Benjamin S. Bloom & Bertram B. Masia. Taxonomy of Educational Objectives: The Classification of Educational Goals: Handbook II. The Affective Domain. New York: David McKay CÔ., 1964.
Nahl-James, Diane. “Use of Online Bibliographic Retrieval Systems:
Recent Research and Future Directions with Discourse Analysis of
Titles and Search Protocols.” LS 663 course paper, Graduate
School of Library Studies, University of Hawaii, Dec 1982.
Rogers, Sharon. “Research Strategies: Bibliographic Instruction for Undergraduates.” Library Trends Summer 1980:69-81.
Simon, Herbert A. “Information-Processing Models of Cognition." Journal for the American Society of Information Science 32, 1981:364-377.
Wegner, Daniel & Robin Vallacher. Implicit Psychology: An Introduction to Social Cognition. new york: Oxford University Press, 1977.