PROBLEMS IN
ASSESSING LANGUAGE PROFICIENCY
Dr. Leon James
(c)1999
Chapter 4 of a book now out of print: Foreign
Language Learning (Rowley, Mass.: Newbury House, 1970).
PROBLEMS IN ASSESSING LANGUAGE PROFICIENCY
A criticism of current practices in FL education that recurs throughout the earlier chapters is that the traditional methods used for assessing language proficiency attained by students are inadequate. Grade-point averages obtained in FL courses and standardized discreet-point tests (both written and oral) are not sufficient indicators of the ability to use the FL for communicative purposes. One could even argue that these assessment procedures are not only insufficient, bur irrelevant. Furthermore, one important reason why research on the comparative effectiveness of different teaching methods has never yielded definitive results is that the methods and procedures employed for assessing what the student has acquired have been inadequate. There is a critical need in the language testing field for tests of communicative competence. Some possible approaches to this problem were briefly outlined in the last section of Chapter 3 under the heading of "Criterion evaluation." (See p. 106). This chapter considers the problem of assessment in greater detail. The first section provides the rationale for a "functional" approach and gives a few illustrations on how one would go about developing tests of language skills. The last two sections have a more "molecular" orientation and are concerned chiefly with the assessment of transfer effects between the individual's two languages. This part of the book is more "experimentally" oriented than the rest, and here the FL teacher can gain a
glimpse of the academic researcher in action. It is involved, esoteric, and sometimes abstruse, and may be of only peripheral interest to the practitioner. Nevertheless, I encourage the reader to plough..: through these sections, not only because he will gain a better under-., standing of the psycholinguist's mind (hopefully!), but also because they raise interesting and important issues on the nature of bilingualism: Are there different "types" of bilinguals? To what extent does the native language affect the acquisition of the second language? Under what conditions may we expect positive and negative transfer? And many others.
A FUNCTIONAL APPROACH TO THE ASSESSMENT OF LANGUAGE SKILLS
The problem of assessing language skills is a practical concern. Developments in this area have proceeded under the impetus of necessity rather than under the guidance of a coherent theoretical under- standing of the nature of language and communication. There is nothing opprobrious in such a strategy, given the various social needs which the field of language testing has had to meet, and no doubt the activities associated with this endeavor will-and should-continue in the near future. On the other hand there is no wisdom or advantage in ignoring the potential contribution of theoretical developments in the fields concerned with the study of language and in this section I shall attempt to outline what I consider to be the implications of recent theoretical developments for the practical problems which are of concern to the language testing field.
4.1 COMMUNICATIVE COMPETENCE
Because of their practical orientation, the contributors to the language testing field are more aware than their theoretically oriented colleagues that language is primarily a communicative skill. Or so it would seem. And yet a cursory examination of the "state of the art" in language testing indicates that the progress of the work to date has been chiefly oriented towards the assessment of linguistic com-
PROBLEMS IN ASSESSING PROFICIENCY 151
petence rather than communicative competence. Since much of the rationale of the proposal to be outlined relates to the distinction to be made between these two types of competencies, it is necessary to examine it in some detail. The distinction is recognized intuitively at certain embarrassing moments in our social interactions when we don't know what to say or how to say something, when we fail to communicate "a point," when we "put our foot in our mouth," when we misunderstand "the words" of someone, or, at less awkward moments, when we infer intent and other psychological states on the basis of someone's utterances, when we identify a person's geographic origin and his social and educational background on the basis of his speech, and so on. The study of linguistic competence has traditionally excluded these wider considerations involved in the use of language as indicated by the emergence of "paralinguistic," "exolinguistic," "sociolinguistic," and "psycholinguistic" hybrid fields whose purpose was to cover various aspects of the communicative act. Linguists have acknowledged the fact that the study of language use must necessarily encompass these wider competencies, sometimes referring to the difference as linguistic competence versus linguistic performance, but they have generally maintained that the study of linguistic competence per se need not concern itself with social-psychological factors. The development of language tests as measures of linguistic skills derives its justification from this traditional position taken by linguists. However, the adequacy of such a position must now be challenged as a result of the evidence that has accumulated in these hybrid disciplines as well as from recent work in linguistics itself, especially, but not exclusively, that aspect of the field that has concerned itself with the semantic component of language. Let us consider some specific issues involved in this argument. It has been established (see for example the review by ENin-Tripp, 1967) that speakers of a language have a command of various "codes" that can be defined as a set of restriction rules that deter- mine the choice of phonological, syntactic, and lexical items in sentences. These restriction rules form a code by virtue of the fact that they are related to the same set of variables such that, given the presence of a particular variable, it will then require a specifiable set of rules determining the choice of these items in speech. For ex-
ample, the choice of address form in English (using the title "Mr." followed by last name), the choice of a particular phonological variant (the "-ing" ending of the progressive form of verbs), and the choice of certain lexical equivalents ("Good Morning" vs. "Hiq are all determined by the same social variable which relates to the status relation between the speaker and listener. (Contrast the utterance "Good morning Mr. Gibbs! How is your work coming along" with "Hi, John! How's your work comin' along?"-both of which are possible for the same speaker, but he would not use both under the same situation: they belong to different codes.) The choice of "Tu" and "Vous" in French (and in German along analogous lines) is determined by specifiable social factors between two interlocutors. In Thai (and several other languages) it is impossible to carry out a conversation without explicitly encoding certain relations that hold between speaker and listener (status, age, degree of friendship, family ties, etc.). These selection rules, and others of this type, are as necessary a part of the linguistic competence of a speaker as those with which we are more familiar in syntax such as accord in gender, number, and tense, and it would seem to be entirely arbitrary to exclude them from a description that deals with linguistic competence. There have been a number of recent proposals (e.g., Katz and Fodor, 1963; Osgood, 1968) that attempt to describe the meaning relations among words in terms of a set of semantic features (e.g., "animate," "male," "human," etc.). Certain linguistic facts can be explained by assigning particular semantic features to words. For example, if one assumes that MAN is coded for an "animate" and a "human" feature (among others), DOG for "animate" and "non- human," TABLE for "nonanimate" and "nonhuman," SPEAK for "human" and RUN for "animate," then one can explain why the following sentences are permissible or nonpermissible, as the case may be:
The man did not wish to speak. "The dog did not wish to speak. "The table did not wish to speak. The man ran away. The dog ran away. "The table ran away.
PROBLEMS IN ASSESSING PROFICIENCY 153
The nature and identity of the semantic features that must be postulated is still a matter of dispute. One practical criterion for the choice of a feature is whether it helps explain a linguistic fact. Thus to account for the fact that the expression "to beg proudly" is anomalous (as opposed to "to beg humbly" which is apposite), Osgood (1968) postulates a "supraordinate-subordinate" feature for interpersonal verbs (to beg" is coded for "subordinate" and "proudly" for "supraordinate"-hence the anomaly of the combination). I have pointed out elsewhere (Jakobovits, 1968c) that OLD MAN is coded for "passivity" and "weakness" which accounts for the fact that the following two utterances are paraphrases of each other:
(a) Don't worry about him. He can't interfere. He's just an old man .
(b) Don't worry about him. He can't interfere. He's just a
passive old man.
weak old man.
The addition of "passive" or "weak" in (b) is redundant, showing that OLD MAN is coded for a "passivity" and a "weakness" feature. Note further that (c) is anomalous:
(c) "Don't worry about him. He can't interfere. He's just
an active old man.
a strong old man.
It is possible under certain circumstances to ask native speakers directly whether a particular word is coded one way or another for any particular feature. The Cross-cultural Atlas of Affective Meanings represents such an attempt in which a group of native speakers in some 20 language/culture communities were asked to code 600 words in terms of three generalized affective features: Evaluation, Potency, and Activity (see Osgood, 1964; Jakobovits, 1966; 1969). The relevance of this Atlas for the present discussion lies in the fact that the affective features of words that represent translation equivalents are sometimes universal (e.g., MOTHER is code as "+Evaluation," DEATH is coded as "-Evaluation," AIRPLANE is coded as "+Activity," ROCK is coded as "+Potency," etc.) and sometimes
cross-culturally variable (e.g., PRAYER is coded "+Evaluation" in the United States but "-Evaluation" in Yugoslavia). The "+Evaluation" and "-Activity" coding of SUNDAY in American English accounts for the interpretability of the sentence "On this island, my friend, every day is Sunday." In Delhi Hindi, SUNDAY (meaning: the first day of the week, the Day of the Lord) is coded as "O Evaluation" and "O Activity" (i.e., "Zero") and the sentence in question would not be interpretable. Note, however, that in order to account for these linguistic facts it is necessary to take into account specific cultural differences between Delhi Hindi and American English cultures. Again it would be a matter of pure arbitrariness to declare that these cultural factors are to be excluded from an account of linguistic competence. Finally I would like to mention one other specific issue which argues against a restricted view of linguistic competence. One would certainly insist that any serious account of linguistic competence would include the derivation of the meaning of a (well formed) sentence. Central to such an account is the fact that speakers of a language are capable of recognizing whether two sentences have the same meaning, that is, that they are paraphrases of each other. That the following two sentences are paraphrases of each other, namely
(a) Kiki chewed up the bathroom rug.
(b) The rug in the bathroom was chewed up by Kiki.
can be adequately demonstrated by showing the transformations which relate the two sentences to each other. But what would it take to demonstrate that the next two sentences are paraphrases of each other?
(c) You're on, Rex.
(d) You're next, Rex.
The relation between these two sentences is in terms of functional equivalence (of intent) within the context in which either of them may be uttered. It would do no good to argue that the relation involved is arbitrary, nonlinguistic as might be the case between two equivalent secret passwords, since it can be shown that functional equivalence in meaning can be rendered by novel paraphrases (e.g.,
PROBLEMS IN ASSESSING PROFICIENCY 155
the telephone is out of order, the telephone isn't functioning, the telephone doesn't work, the telephone is dead, etc.). In order to account for a speaker's ability to recognize the equivalence of these sentences, one will have to consider factors other than those usually included in a restricted sense of linguistic competence (for example, intention of the speaker, see below).
4.2 THREE LEVELS OF MEANING
In order to be able to account for the minimum range of linguistic phenomena involved in communicative competence, it will be necessary to incorporate in the analysis of an utterance three levels of meaning. I shall refer to these as linguistic, implicit, and implicative.
4.2.1 Linguistic Meaning
I shall retain this term to refer to the traditional concerns of linguists as outlined in such recent treatises as Chomsky (1965) and Katz (1966). Briefly, this includes a dictionary of lexical meanings and their projection rules, syntactic relations as defined by a derivative transformational theory, and honological actualization rules.
4.2.2 Implicit Meaning
I shall use this term to refer to the elliptically derived conceptual event which an utterance represents. Let me use the following ex- ample to clarify what I mean. Consider the meaning of "agreed to" in each of the following sentences:
(a) The University agreed to pay his moving expenses.
(b) After the President invoked the Taft-Hartley Act, the strikers agreed
to go back to work.
(c) Joan agreed to marry him.
(d) They agreed to settle the matter out of court.
Note that in each of the four instances "agreed to" has some different particular implications which one can infer on the basis of the situation to which the sentence as a whole refers. These particular implications are a function of the situational context that the sentence in which the verb is used refers to and therefore are potentially infinite. The dictionary meaning of the verb applies certain definite restrictions on the kind of particular implications it may have in a sentence and thus, although these implications are potentially in- finite, they are not randomly variable. There are many other in- stances in which this type of nonrandom but infinite variability principle operates. The concept of "triangularity" can be particularized as a visual form (among others) with infinite, but nonrandom, variability: each particular triangle will have its own unique properties different from any other particular triangle. What I am arguing is that the effective information that is being transmitted in communication via sentences-that which the speaker intends the listener to understand-is the particular meaning, not the general, for this is what represents the conceptual event ("idea") that he attempts to communicate to the listener. In order to recover the particular meaning of a word intended by the speaker, the listener must engage in an inferential process which makes use of his knowledge of the general (dictionary) meaning of words as well as his knowledge of the overall situation to which the sentence as a whole refers. This inferential process yields what I have called "the elliptically derived conceptual event" which an utterance represents and which I have used as the definition for implicit meaning. Note that the effective carrier of the message is the implicit meaning of the sentence, not the linguistic one. Hence, we see once more that linguistic competence must include factors outside a narrow view of linguistic manipulations and that the assessment of such competence by means of language tests must therefore also include these larger issues.
4.2.3 Implicative Meaning
We have seen that words have a relation to a generalized abstract meaning (linguistic meaning) as well as to a particularized meaning
PROBLEMS IN ASSESSING PROFICIENCY 157
that relates to a specific conceptual event (implicit meaning). Utterances as a whole have, in addition, a relation to several aspects of the speaker: his intention, his psychological state, his definition of the interaction and certain "claims" (both intentional and unintentional) he makes about the status of his utterance, etc. In some cases these implications are necessary to recover the intended meaning of the utterance (e.g., "Do you have a match?" is not a question to be answered verbally but a request for fire to light a cigarette). In other cases, the implications of an utterance are by-products not essentially related to conveying intended meaning (e.g., phonological clues that reveal geographic origin). I shall refer to both types of implications as the implicative meaning of an utterance.
4.3 A FUNCTIONAL APPROACH
Within the framework that I have outlined the problem of assessing language skills becomes the problem of describing the specific manner in which an individual functions at the three levels of meaning identified above. The relations that hold between the three levels, both for a speaker and a listener, is a matter to be determined empirically for each individual, although it is to be expected that such empirical studies will show that some of these relations will be universal for all language users. The development of specific tests for assessing these relations should be guided by an overall model of the functional elements in communicative competence. Because of the practical concerns involved in language testing, one need not post- pone the task of developing language tests until such time as adequate theoretical models become available. In fact, it might even be the case that such empirical evidence that the language tester accumulates would constitute useful information for building such theoretical models. I would argue, however, that the development of language tests on a piecemeal basis and without considerations de- rived from a model that encompasses the full range of phenomena involved in communicative competence, can only yield artificial laboratory exercises that have little significant relationship to the use of language in real life situations. I believe it is presently possible to outline the major elements to be included in a model of language use
even if it may be insufficient in details, and I have described elsewhere (Jakobovits, 1968c) a classification scheme based on an analysis of the three levels of meaning discussed above. I will not discuss the full classification scheme here but present below an over- view of it:
1. A functional analysis of meaning
1.1 linguistic meaning
1.2 implicit meaning
1.2.1 aspectual
1.2.1.1 extended duration
1.2.1.2 limited duration
1.2.1.3 inception
1.2.1.4 iteration
1.2.1.5 gradualness
1.2.1.6 chronicity
1.2.1.7 termination
1,2.1.8 collectiveness
1.2.1.9 inaction
1.2.1.10 directionality
1.2.1.11 intensity
1.2.1.12 nonchalance
1.2.2 affective
1.2.3 aesthetic
1.2.4 contextual
1.2.5 situational
1.3 implicative meaning
1.3.1 sociolinguistic 1.3.1.1 geographic origin
1.3.1.2 education and SES
1.3.1.3 professional
1.3.2 emotional
1.3.3 intention
1.3.3.1 mands
1.3.3.1.1 request
1.3.3.1.2. command
1.3.3.1.3. prayer
1.3.3.1.4. question
PROBLEMS IN ASSESSING PROFICIENCY 159
1.3.3.1.5. advice
1.3.3.1.6. warning
1.3.3.1.7. permission
1.3.3.1.8. call
1.3.3.2 pseudo mands
1.3.3.2.1. superstitious
1.3.3.2.2. magical
1.3.3.3 intraverbals
1.3.3.3.1. grammaticalness
1.3.3.3.2. word associations
1.3.3.3.3. conventions
1.3.3.3.4. expressions
1.3.3.3.5. recitations
1.3.3.4 private tacts
1.3.3.4.1. feelings
1.3.3.4.2. opinions
1.3.3.4.3. rationalizations
1.3.3.5 public tacts
1.3.3.5.1. instructing
1.3.3.5.2. describing
1.3.3.5.3. entertaining
1.3.3.6 autistic utterances
1.3.3.6.1. verbal fantasy
1.3.3.6.2. self-tacting
1.3.3.6.3. self-manding
1.3.3.7 autoclitics
1.3.3.7.1. importance
1.3.3.7.1.1. stress pattern
1.3.3.7.1.2. repetition
1.3.3.7.1.3 adverbial intensification
1.3.3.7.1.4. syntactic inversion
1.3.3.7.2. existential status
1.3.3.7.2.1 assertion
1.3.3.7.2.2. negation
1.3.3.7.2.3. obligation
1.3.3.7.2.4. possibility
1.3.3.7.2.5. probability
1.3.3.7.2.6. certainty
1.3.3.7,2.7. hear-say
1.3.3.7.2.8. permission
1.3.3.7.2.9. imperative
1.3.3.7.2.10. irony
1.3.3.7.3. relational status
1.3.3.7.3.1. explicit hinges
1.3.3.7.3.2. implicit hinges
1.3.3.7.3.3. zero hinges
2. Elements of style
2.1 internal consistency
2.2 appropriateness
2.3 effectiveness
2.4 aesthetic value
2.5 affective value
3. Selection factors in the act of composing
3.1 intention
3.2 audience
3.3 situation
3.4 style
3.4.1 intraverbal influences
3.4.2 formal constraints
3.4.3 blendings
4. The structure of the message
4.1 syntactic
4.2 conceptual
4.4 A PROGRAM OF TEST DEVELOPMENT
It is not possible within the context of this section to specify in detail a program of test development that will take into account the full range of linguistic phenomena covered by the classification scheme sketched out above. Furthermore, the development of such a testing
PROBLEMS IN ASSESSING PROFICIENCY 161
program would require the collaboration of a team of workers with specialized skills which I do not possess. Nevertheless, I can perhaps offer some tentative suggestions with respect to some methodological approaches that may be used in connection with many of the linguistic phenomena involved in the classification scheme.
4.4.1 Judgments of Acceptability
This technique consists of asking a subject to make an evaluation of the acceptability or appropriateness of an expression, or alternately, presenting him with two similar expressions and asking him to choose the more appropriate of the two. The technique is well suited for assessing sensitivity to aspectual qualities of words. Aspectual quality (1.2. 1) is part of the implicit meaning of a word and is based on inferences one can draw about characteristics of the specific conceptual event implied by a word. These characteristics are not specified directly by the dictionary meaning of the lexical items involved but are instead implied by them. Some examples of aspectual qualities in English are the following: extended duration (as exemplified by the following contrasts: "to stare" versus "to notice," "fearful" versus "frightened," "state of normalcy" versus "state of crisis"), limited duration ("to strike" versus "to beat," "to gulp" versus "to sip"), inception ("to doze off" versus "to doze," "obsolescent" versus "obsolete," "adolescence" versus "manhood"), and iteration ("to nag" versus "to criticize," "to sob" versus "to yelp," "wave" versus "line"). It is possible to construct phrases that contain modifying elements which are either consistent or inconsistent with the aspectual quality of the modified element. In each of the following examples the second phrase contains an inconsistent modifier; given a choice, the subject should choose the first alternative if he is sensitive to the implicit meaning of the expression:
He was frightened momentarily/He was fearful momentarily
He took a long sip/He took a long gulp
He had been dozing for some time when the phone rang/He had
been dozing off for some time when the phone rang.
He criticized me only once/He nagged me only once
4.4.2 Semantic Differential Technique The usual method requires a subject to rate a word on a set of seven-point bipolar adjectival scales (see Osgood, Suci, and Tannenbaum, 1957). The selection of scalar opposites varies with the
purpose of the test. Scales for the assessment of three dominant and universal affective features have been identified empirically for English and some 20 other languages (see Osgood, 1964;Jakobovits,1966). These consist of Evaluation (for American English: nice- awful, good-bad, sweet-sour), Potency (big-little, powerful-powerless, strong-weak), and Activity (fast-slow, alive-dead, noisy-quiet). For other affective features, different scales would be used. The technique is general enough to allow for changes in the stimuli that are rated (e.g., they may consist of phrases and sentences) as well as the nature of the scales used for the ratings (e.g., instead of the seven-step scale one may use an "either-or" two-step scale). Sensitivity to the aspectual quality of intensity (1.2.1.113, for example, can be revealed by asking a subject to say which of two choices is more or less intense: "icy" versus "cold," "filthy" versus "dirty," "ravenous" versus "hungry," "to shatter" versus "to break," etc.
4.4.3 The Questionnaire Method
This approach is convenient for assessing a subject's sensitivity to implicative aspects of the meaning of utterances: socio-linguistic (1.3.1,), emotional (1.3.2.), and intentional(l .3.3.), Since nonverbal factors (e.g., gestural, visual, situational) provide particularly important clues to implicative meaning, the problem of assessing an individual's skill in this area is likely to be quite involved. Without implying that these nonverbal factors should be left out of the testing program let me suggest that utterances, even in the absence of such clues, do allow for the derivation of quite a bit of information about the speaker-and the skill involved in this can be assessed more readily within the confines of the usual testing situation. Phonological variables present in tape recordings offer clues to socio-linguistic characteristics of the speaker and his emotional state. Syntactic and contextual variables offer clues to intentions of the speaker. Here are
PROBLEMS IN ASSESSING PROFICIENCY 163
some examples in connection with intentions (1.3.3.). "Pass the salt," "Hands up," "Take your coat off. It's hot in here," "Now listen to this," are utterances which reveal the speaker's interest in inducing some action in the listener. These utterances are classified as mands (1.3.3.1.) and the four given here are instances of a request, a command, an advice, and a call, respectively. The speaker continually uses devices in utterances that have the function of commenting on certain aspects of the message. These are called autoclitics (1.3.3.7.). For example, stress pattern and repetition indicate that the speaker wants to lay emphasis on his utterance ("You should not infer from this that..." "I like it very, very much"). Comments on the existential status (1.3.3.7.2.) of assertions include, among others, obligation ("I had to give in"), possibility ("Couldn't it happen?"), probability (He is likely to forget"), hear- say ("The two accident victims are reported to have died"), irony ("Aren't we having fun!"), etc.
4.4.4 Acting out Situations
The questionnaire approach just described assesses decoding (passive skills). Reversing the task by asking a subject how he would say something under specific conditions becomes an encoding task that assesses corresponding active skills. He now assumes the role of the speaker by acting out the communicative act called for by the instructions. These may be simple or complex. An example of a simple situation would be to ask the person to transform an utterance in such a way as to express certain intentions. For instance, the sentence "He will be going to the meeting," which is an assertion, is to be transformed to indicate negation (He will not be going to the meeting), obligation (He has to go to the meeting), permission (He is allowed to go to the meeting), certainty (He most definitely will be going to the meeting), irony (Wouldn't he just love to go to the meetind), and so on. More complex instructions might require the subject to construct public tacts (1.3.3.5.) appropriate to various situations (reporting a past event, describing an ongoing activity, telling a joke, etc.). Encoding and decoding skills are not necessarily related. Thus, one might successfully identify the geographic origin
of a speaker without being able to imitate him; or, one may re- cognize that a speaker is angry without being able to talk like an angry man upon demand. (The latter case exemplifies a further distinction to be made between volitional control of encoding skills and the ability to carry out the same skills under natural conditions but without volitional control.)
4.4.5 Paraphrasing
"Rewording" is a traditional test for assessing understanding. But rewording often changes the tone or style of utterances, which means in fact that the alternative construction is not semantically equivalent: it may have different implicit and implicative meanings. I use the term semantic modulation to refer to the possibility of alternative phonological strings that are related in linguistic meaning but are different in their implicit and implicative aspects. Consider some examples in semantic modulation:
He spoke loudly/He spoke with a thunderous voice
The light was too bright/It was a blinding light
It was uncomfortably hot and humid/The weather was stifling
He worked extremely hard/He worked his fingers to the bone
The second sentences of each pair in these examples are recognized as paraphrases of their respective first sentences (they have equivalent linguistic meaning) but they are marked in their implicit meaning by the aspectual quality of intensification (1.2.1.1 i.). The ability to manipulate the implicit meaning of an utterance without changing its linguistic meaning is an important skill in "creative writing." Similarly, the ability to manipulate implicative meaning is a mark of the effective writer and speaker (contrast "That's not the way to do it" with "If I were you, I would do it differently").
4.4.6 Intraverbal Skills
Intraverbals (1.3.3.3.) refer to aspects of verbal behavior that are not marked for intent. The most obvious instance of intraverbal in-
PROBLEMS IN ASSESSING PROFICIENCY 165
fluences is grammaticalness. Another instance is the associative relation that holds between words as exhibited in the traditional word association test. Conventional phrases (greetings, acknowledgement of thanks, condolences, etc.), expressions (proverbs, idioms, platitudes), and recitations (lines of poetry, prayers, etc.) all involve intraverbal skills that should be included in any testing program that attempts to assess communicative competence.
4.5 SUMMARY
I have pointed out that the traditional approach to the assessment of language skills derives its rationale from a restricted definition of linguistic competence. I have argued that such a restricted view is totally inadequate to account for the most basic phenomena in language use and, hence, language tests based on this rationale are nothing but artificial laboratory exercises that have little in common with the skills involved in the everyday use of language. To study adequately the latter process, one must recognize the basic functions of communication via language and the total information available in the communicative act. I have described in an outline form a classification scheme for identifying the functional elements to be taken into account in the study of communicative competence and have suggested some methodological approaches that one might adopt in the development of a testing program designed to assess skills involved in the use of language.
4.6 COMPOUND-COORDINATE BILINGUALISM
Since Ervin and Osgood's (1954) discussion of the problem, compound-coordinate bilingualism has received the attention of several investigators (see, for example, Lambert, Havelka, and Crosby, 1958; Jakobovits and Lambert, 1961;Preston, 1965; Kolers, 1963; and others: see the review by Macnamara, 1967). The distinction between the two types of bilinguals offered by Ervin and Osgood (1954) was a unidimensional one, in terms of the semantic relation between the bilingual's two languages. The compound bilingual was supposed to have a single meaning system hooked up to two
different input (decoding) and output (encoding) channels corresponding to the bilingual's two languages. This type of bilingualism was assumed to come about as a result of learning the second language in the same environmental setting as the first, and using the first language as the indirect channel of acquisition. In this kind of a psycholinguistic system every word in the second language was a mere replica of a word in the first language with a one-to-one correspondence in meaning between the two translation equivalents. This is why the "compound" bilingual system is sometimes referred to as a "fused" system (Weinreich, 1953).
On the other side of this semantic continuum is located the "separate" system of the coordinate bilingual who is assumed to possess two independent meaning systems corresponding to his two languages. This situation is achieved, according to this view, as a result of direct language acquisition in a linguistic-cultural community different from one's own. There is no one-to-one relation between the two meaning systems and, in fact, translation equivalents are mere approximations, their closeness depending on the similarities in the two cultures involved. A direct test of the semantic distance between a few translation equivalent words for French- English bilinguals was carried out by Lambert, Havelka, and Crosby (1958). These authors divided their sample of bilinguals into three groups. One was composed of uni-cultural compounds who learned French in school by the indirect method through the medium of English. The second group was composed of uni-cultural coordinates who learned French by the direct method and often used it in a setting different from that where they habitually used English; the two settings, although different, were nevertheless both North American. The third group was composed of bicultural coordinates; these were often native speakers of French (in Quebec) who lived in the bicultural setting of Canada. Having selected a set of common words in English (House, Drink, Poor, Me) and their French translation equivalents, the investigators then obtained semantic differential ratings for each word using the appropriate scale in English and French. The results showed that the semantic distances between translation equivalent terms were largest for the bicultural coordinates and smallest for the uni-cultural compounds (a statistically significant difference), with the uni-cultural coordinates falling near
PROBLEMS IN ASSESSING PROFICIENCY 167
the uni-cultural compounds and not significantly different from them. These results, which represent partial confirmation of the predictions of Ervin and Osgood (1954), are further strengthened by a subsequent study (Jakobovits and Lambert, 1961) which demonstrated that reducing the meaning of a word (through semantic satiation brought about by verbal repetition, see Lambert and Jakobovits, 1960) in one language also brings about a corresponding reduction in meaning of its translated equivalent in the second language for compound bilinguals but not for coordinate bilinguals-as indeed the theory would predict.
The theory of compound-coordinate bilingualism as described above may be quite useful in understanding certain aspects of the semantic functioning of bilinguals. However, it may be advantageous to extend the compound-coordinate distinction to other areas of psycholinguistic functioning. The remainder of this section will explore some possible ways by which such an extension may be achieved.
It would be useful to recognize at the outset that bilingualism involves two different aspects of the question of interrelatedness of two distinct language/culture systems. One such aspect (R3 deals with the relation between the bilingual's two language/culture systems, and may be represented as follows:
RA=f(XL1?XLz)
where X stands for the speaker and L for the language spoken. In this diagram the first language is represented by a rectangle marked by
horizontal lines (on the right), while the second language is represented by a rectangle marked by vertical lines (on the left). The overlapping area of intersection is marked by both horizontal and vertical lines and represents interference. In this example L, is larger than L, to indicate dominance. The second aspect of the question of interrelatedness deals with the comparison between each of the two languages of a bilingual to that of an idealized monolingual member of the two respective language/culture communities (RB & Rc). This relationship may be represented as follows:
bilingualism, where X stands for an idealized monolingual. It is possible to devise tests which would measure the extent of interrelatedness that exists for each of these two aspects within various areas of psycholinguistic functioning (e.g., semantics, grammar, decoding, etc.) and several of such tests will be described below. At this point, however, it would be useful to restate the definition of compound-coordinate bilingualism keeping in mind the extended perspective presented above.
The ideal coordinate bilingual represents a state of complete functional independence of two language/culture systems in all their aspects: semantic, lexical, syntactic, phonological, morphemic, cultural, attitudinal, etc. In his case, RA (see diagram) has a minimum value while R, and Re have maximum values. In its extreme form, coordinate bilingualism represents linguistic schizophrenia: one individual with two separate and theoretically unrelated cultural-linguistic systems co-occur within the same body. In compound the two linguistic systems are so fused and inter-dependent that the second language represents a mere alternative
PROBLEMS IN ASSESSING PROFICIENCY 169
channel for the overt manifestation of the same underlying system represented by the first language. In this state R, and R, have maxi- mum values, while the other, R,, has a minimum value. In its extreme form compound bilingualism represents a translation machine: for every coded bit in L, there must be a corresponding bit in L, with the relationship between the two exactly specified. Let us now examine some implications which follow from these considerations.
4.6.1 Degree of Bilingualism
Under this topic we should include both the question of bilingual interference as well as bilingual proficiency. Bilingual interference can be defined, following Mackey (1965), as the use of elements in one language while speaking another. The consequence of such intrusions is an increase in the value of R, and a decrease in the value of R,: the bilingual's speech in L, (XLZ) Will differ from the idealized version (XLZ ) in direct proportion to the extent of interference from L, (RA); furthermore, the bilingual's first language may show uncharacteristic departures from the standard (XL3-a problem he did not encounter as a monolingual. One would expect that the most common form of bilingual interference would be in the direction of L,: that is, high values in RA would lead to low values in Rg leaving Re relatively unimpaired. However, there is evidence (see Mackey, 1965) that in certain areas of psycholinguistic functioning, particularly in semantics, the value of Re may decrease as a result of backlash interference (see below).
Bilingual proficiency has sometimes been defined (e.g., Macnamara, 1967) in terms of the relative competence of the bilingual in his two languages: where the two competencies are equal, bilingual proficiency is said to be maximal (sometimes referred to as a state of "balanceness," see Lambert, Havelka, and Gardner, 1959). Where competence in one language surpasses that in the other, a state of bilingual "dominance" is inferred. I have pointed out elsewhere (Jakobovits, 1967) the difficulty with such a definition. When bilingual proficiency is estimated, as has often been the case (see Macnamara, 1967), by means of indirect measures which may be
contaminated by nonlinguistic factors, the relative or difference score would not be acceptable without prior determination of interaction effects. When bilingual proficiency is established by means of direct competence tests, then the difference score obscures the distinction between RB and RC, two elements which are theoretically independent. For these reasons it would be preferable in discussing the problem of degree of bilingualism, to keep separate these two r_elations expressed as XLZ-f gL2 (RB) on the one hand and XL1® XL1 (RC) On the other. The first of these must be measured by proficiency tests in the second language standardized on monolingual speakers. The second relation is of interest to the problem of what might be called "backlash interference"-the intrusions into and modifications of one's mother tongue due to the learning of a foreign language.
One may raise the question of the relation between the two aspects of bilingualism discussed above, namely degree and compound-coordinatedness. Let us recall the elements in our system:
Compound-Coordinate
Degree of Bilingualism (D) Bilingualism (CP-CR)
(i) Proficiency in L2 (PL2) (i) RB=f(XL2® XL2)
(ii) Proficiency in L, (PL1) (ii) RC=f(XL1® XL1)
(iii) RA=f(XL1® XL2)
These relations have already been explicated above. Now let us consider the following two hypothetical cases:
Case Y:
PL2 is high; PL1 is unimpaired
RB is maximal; RC is maximal; RA is minimal
Case Z:
PL2 is low; PL1 is unimpaired
RB is minimal; RC is maximal; RA is high
As can be seen, case Y represents a bicultural balanced coordinate bilingual: proficiency in the second language is high while functioning in the mother tongue is unimpaired; furthermore, this bilingual functions as a native in both languages (RB and RC) without showing cross-linguistic interference (RA). This case then is the linguistic schizophrenia referred to earlier. Case Z is weak in the second language (PL2) while he retains the characteristics of his language/culture
PROBLEMS IN ASSESSING PROFICIENCY 171
group (PLl and RC) he shows considerable interference in the second language (RA) which renders his speech in that language uncharacteristic (RB). In other words, case Z represents a uni-cultural dominant compound bilingual. The reader may explore for himself other possible combinations using this schema. By manipulating the various elements in a systematic fashion, one at a time, some interesting cases arise, but these will not be taken up here. Instead, let us consider the extension of the model by the addition of two other dimensions. One dimension relates to the various areas of psycholinguistic functioning as might be indexed by such tests as semantic distance, word association structure, adequacy of translation, language switching, reading comprehension, pronunciation accuracy, and some additional ones which will be described below. The other dimension to be considered is that of development over time. Figure 1 is a schematic representation of a four dimensional model of bilingual description using the elements discussed above. The first dimension, that of de- gree of bilingualism (PL), is given by the relative size of the two rectangles: the left hand rectangle with the vertical lines represents the second language (L2) and the right hand rectangle with the horizontal lines represents the mother tongue (L1). Proficiency in L2 and L1 is indicated by the size of their respective rectangles. The second dimension, that of compound coordinate bilingualism, is indicated as follows:RA (i.e" XLI® XL2 or interference)is given by the amount of overlap between the two rectangles; RB (i.e., XL2® XL2)and RC (XL1® XLl)is given by the density of the vertical and X horizontal lines within the rectangles: high density represents a high degree of relationship. The third dimension, that of psycholinguistic area of functioning is represented on the vertical axis (a. vs. b. vs. c3, and development in time, the fourth dimension, lies along the horizontal axis (1 vs. 2 vs. 3). Let a stand for grammatical intrusions, let b stand for pronunciation accuracy, and c for a measure of word association structure (see below for details). Let I stand for the beginning stages of second language acquisition, let 2 be several months later spent in a bicultural setting, and let 3be at the completion of the language acquisition process. Looking at part a.l of the figure, we notice a dominant uni-cultural compound (L, on the right is dense and larger than L2 on the left, which it envelops) whose
FIGURE 1. A FOUR-DIMENSIONAL MODEL OF BILINGUAL DESCRIPTION.
grammatical performance in L2 is poor (note its small size). His pronunciation in L2 (b.l) is practically unintelligible as the phonemic system of L1 completely dominates it. A test of word association structure (c.l) shows relatively better performance (note larger size of L2). A few months later, and with the opportunity of bicultural experience such as that offered by extensive travel in the foreign country, our language learner has considerably improved in gram-
PROBLEMS IN ASSESSING PROFICIENCY 173
matical performance (note larger size of L2 at a.2), although his pronunciation still shows heavy contamination (b.2). His greatest progress is exhibited on the word association test where his performance increasingly resembles the cultural norm (note increase in size of L2 at c.2 and greater density of vertical lines). At this point an interesting phenomenon occurs, "backlash interference." As a result of becoming more bilingual the individual experiences a typical departure from his mother tongue as compared to idealized monolingual speakers of that language. At the end of the successful acquisition process, the grammatical performance of this bilingual (a.3) shows the coordinate pattern (large, dense rectangles with no overlap). His pronunciation (b.3) is good but not native, and his word associa- tion structure (c.3) shows a curious pattern: his performance is excel- lent (large rectangles) and native-like (dense lines), but the two linguistic systems interact (note overlap): the bilingual has developed what might be called a "cross-cultural semantic system"-a kind of "coloring" or "flavoring" of semantic factors which comes about through cross-cultural contact (see Jakobovits, 1966).
Once again the reader may want to explore further the various combinations that are possible in this kind of a schema. The ad- vantage of what may appear to be a gratuitously complex representation is that it facilitates systematic elaboration of possible stages of bilingualism and may thereby increase our understanding of the processes involved.
4.6.2 Translation and Language Switching
In their analysis of compound coordinate bilingualism, Ervin and Osgood (1954) are led to the hypothesis that extensive translation on the part of a coordinate bilingual will transform him into a compound system. They argue that continued interaction between the distinct meaning systems of the two languages washes out the differences between them. However, the opposite effect is also logically possible: continued interaction can act as discrimination training through contrast and forced differentiation and, in fact, experienced translators seem to be attuned to stylistic and connotative differences between translation equivalents to a degree not apparent
in less experienced bilinguals. The Ervin and Osgood model furthermore neglects the problem of a language switching mechanism which must be postulated to account for the fact that bilinguals, and especially compounds, are able to function in one language without extensive interference from the other language. Figure 2 represents an elaboration of the Ervin and Osgood model by including hypothetical language switches on the decoding and encoding sides as well as filter mechanisms (separators) whose function it is to channel signs into their respective languages at the decoding and encoding stages. Consider first the compound bilingual. If the input switch is "on," incoming signs will be shunted into either LA or LB, depending on the "language set." In other words the switch has directional properties. The signs are then fed into the common meaning system where they are decoded. Now, if the output switch is also "on," the
FIGURE 2, A MODEL OF COMPOUND (TOP) AND COORDINATE (BOTTOM) BILINGUALISM.
PROBLEMS IN ASSESSING PROFICIENCY 175
encoding response will "exit" through LA or LB, again depending on the direction of the switch. One must also consider the case where the bilingual (compound or coordinate) is set to decode from either language (e.g., in a bilingual environment). In that case the switch is turned "off" and the incoming signals are filtered through the "sign separator" mechanism and fed into the meaning system through LA or LB. For the coordinate bilingual, there is no need for a "response separator" mechanism since the meaning systems in the two languages are distinct and feed into separate output systems. For the compound, however, such a filter on the encoding side must be postulated to allow for the possibility that the output switch be "off" (e.g., where the bilingual is set to encode in either language). On the other hand, it is necessary to postulate a "translation switch" for the coordinate to allow for his capacity to attend to LA in the decoding stage and to render its equivalent in the encoding stage. Now let us see how the model depicted in Figure 2 can be used to make predictions about the behavior of bilinguals. The following are some outcomes of the assumptions that have been made:
1. For both compounds and coordinates decoding will be less efficient when the input switch is "off" (e.g., when he has to attend to signs in both languages at the same time-see Measures 17 and 18 below). The reason for this expectation is that the route through the sign separator is less direct and the filtering process should take measurable time.
2. The requirements to encode in either language (see Measures 16 and 19 below) should be more disruptive for the compound than the coordinate, again because of the time requirements of the filter separator.
3. Instructions to translate into a specified language should be a more efficient task for the compound than for the coordinate bilingual. The greater efficiency of this task for the compound can be seen in Figure 2: If the output switch is "on" and directed, say, toward La, the meaning system feeds directly into the output sys- tem. For the coordinate bilingual, there must first be an "equivalence search" between the two meaning systems through the translation switch, and this should take measurable time. It should be noted that if the translation time is very short, such as might be the case for word-to-word translation of common lexical items, the coordinate's
disadvantage relative to the compound would be difficult to measure. This might account for the failure of some investigators (see Macnamara's 1967 review) to find latency differences under these conditions. Where the translation task is less automatic (unfamiliar words, phrases, sentences) the present prediction should be verifiable. The reader can explore further for himself several other predictions that Figure 2 suggests. Macnamara (1967) has discussed the kind of experimental data that would be required to verify the validity of the switch model-the number of switches it is necessary to postulate and whether switches take measurable time for operating. Some translation and switching tests are described below.
4.6.3 Degree of Interrelatedness
Under this topic we shall discuss some of the factors that lead to compound-coordinate bilingualism. We should consider three types of factors: language acquisition context and usage, attitudinal and motivational variables, and cross-cultural distinctiveness. Within each of these areas, it is necessary to distinguish among the various aspects of psycholinguistic functioning. In the semantic area the evidence on connotative distance between translation equivalent words resulting from separate acquisition contexts and usage has already been discussed above. The effects of some attitudinal variables on semantic change has been discussed by Lambert (1967) and his coworkers (see, for example, Lambert, Gardner, Barik, and Turnstall, 1963). In the latter report it was shown that semantic Interdependence is positively related to higher achievement in an intensive language acquisition program. The measure involved was the semantic distance (as indexed by the semantic differential) between the meaning of some words in English (the mother tongue) at the beginning and at the completion of the course, as compared to the meaning of the translation equivalent words in French. Thus all three aspects of interrelatedness were presumably involved, although, unfortunately, they were not all directly measured. Let us examine how this type of change would affect the three aspects of interrelatedness discussed earlier in this paper. The value of RC (i.e, XL1® XL1) may be presumed to have decreased from its previous maximal value: there is
PROBLEMS IN ASSESSING PROFICIENCY 177
a decrease in coordinateness and the meaning of words in the mother tongue of this bilingual is no longer representative of monolinguals of his culture. RA (i.e, XL1® XL2) in the semantic area decidedly increased: as Lambert and his associates put it, the students "permit the two languages to interact, and this is to their advantage in the learning of the second language" (p. 368). This would lead to greater compoundness. However, change in the value of RB (i.e" XL2 ® XL2,) is in doubt: the students' semantic systems may have progressed X towards the native pattern of L2-in which case the value of RB increased promoting coordinateness, or, also possible, their semantic system may have moved away towards a compromise position which is not characteristic of the culture and is colored by the mother tongue, as Lambert et al. put it: "...these students develop generalized super linguistic concepts which incorporate the unique semantic features of the concepts in each of their two languages" (p. 368). Thus it would appear that contrary to what one might expect from an emphasis on "direct" teaching methods, an increase of compound- ness in the semantic area of connotation during certain stages of the acquisition process may facilitate progress in a FL.
Figure 3 represents another hypothesized relation between attitudinal factors and compound-coordinateness. The two psychological variables considered here are ethnocentrism and language learning orientation. The former trait is measured by a questionnaire intended to ascertain the extent of the student's rigidity of belief in the superiority of his own culture, while the latter trait is ascertained by his answers to questions about his purposes for learning a foreign language: instrumental reasons (e.g., usefulness in work or travel, meeting the Ph.D. language requirements) versus integrative reasons (e.g., getting to know the people better; see Lambert et al., 1963, for greater detail). Figure 3 says that high ethnocentrism coupled with an instrumental orientation are psychological factors which promote compound bilingualism. The reason for this is presumably that a highly ethnocentric individual would not tolerate a decrease in the value of Re (i.e, XL1® XLl) as this would estrange him from his own culture in whose superiority he so firmly believes. The over all implications of the relationship represented in Figure 3 are rather intriguing, since it indicates that the opposite number of the highly ethnocentric, instrumentally oriented student, namely the self-
FIGURE 3. HYPOTHESIZED RELATION BETWEEN COMPOUND-COORDINATE BILINGUALISM AND ATTITUDINAL STRUCTURE.
deprecating, integratively oriented "culture grabber" characteristic of young, enthusiastic language learners eager for "foreign culture," would also promote compoundness: this comes about, presumably, because this kind of an individual freely allows a decrease in RC (i.e, XL1® XL1)as he eagerly "leaves behind" the norms of his culture. X ~X Figure 3 also shows that coordinateness will be promoted by moderate levels of ethnocentrism and bi-directional tendencies in orientation: in other words the psychologically conflictual individual. Such a person would tend to maintain RC (i.e, XL1® XLI) at a high level, while moving at the same time towards high values for RB (i.e, XL2® XL2). This type of an individual would be expected to experience the "threes of anomie" as his linguistic schizophrenia develops into the full-fledged coordinate pattern.
PROBLEMS IN ASSESSING PROFICIENCY 179
The third type of factor to be discussed in this section, as mentioned above, is cross-cultural distinctiveness. A general hypothesis may be formulated as follows: cultural distinctiveness promotes coordinate bilingualism while cultural similarities inhibit it. It would be difficult for an Acadian in Halifax, Nova Scotia, or a Cajun from Louisiana, to be a coordinate bilingual. Similarly, one cannot expect a Japanese immigrant to San Francisco to be a compound bilingual, even after twenty years and half-a-dozen American monolingual grandchildren. Although it is true that certain semantic characteristics related to affective structure tend to be universal (see Osgood, 1962; Jakobovits, 1966), other characteristics tend to be uniquely peculiar to individual cultures, and these must be considered in evaluating the factors that determine compound-coordinate bilingualism.
4.6.4 Some Measures of Compound-Coordinate Bilingualism and Their Interrelatedness
In this final section we will consider a number of tests which may be used to index the various aspects of compound-coordinate bilingualism as discussed here. The following is a brief description of some 27 measures of compound-coordinate bilingualism. The first nine are in the semantic area; the tests numbered 10 to 14 are in the lexical area; those numbered 15 to 19 are translation and language switching tests; 20, 21, and 22 are grammatical tests that require
transformation and paraphrasing; tests 23-25 deal with a judge's subjective ratings of various bilingual skills; and the last two (26 and 27) are proficiency tests in the two languages (discrete-point type).
4.6.4.1 Semantic Distance Test
The geometric distance formula D is used with the semantic differ- ential to index connotative differences between words (described in Osgood, Suci, and Tannenbaum, 1957). When translation equivalent concepts are used for the same individual, as in Lambert, Havelka, and Crosby (1958),it becomes one index of RA, (i.e" XLI® XL2) where a large D would indicate coordinateness. When a bilingual's ratings are compared with a monolingual (or with a norm profile for
a monolingual group), it becomes an index of RB or RC, depending on whether one deals with L2 or L1 (large D values indicate compoundness).
4.6.4.2 Semantic Cross-satiation Test
This is an indirect test of semantic distance and is complicated by the unavoidable occurrence of language switching due to the nature of the test. The bilingual repeats a word in L, (or LP) and a test is then made for the semantic satiation which is presumed to transfer to L2 (or L1). Jakobovits and Lambert (1961) have used this test with clearcut results: cross-linguistic satiation occurs with compound systems but not with coordinates.
4.6.4.3 Associative Interference Task
This test incorporates a transfer of learning design in which bilinguals first learn to memorize a list of commonly occurring words in one language, then receive further training on an interpolated list of words which is made up of the translation equivalent words in the other language of the previously learned list. The theoretical predictions based on transfer of learning theory are clear: because of the identity in meaning of the two lists for compounds, positive transfer is expected; for the coordinates, either less positive transfer or actual negative transfer should be exhibited. This test was successfully used by Lambert, Havelka, and Crosby (1958).
4.6.4.4 Semantic Congruity Test
The procedures for this test were developed by Osgood (1966)in connection with the investigation of semantic feature analysis. The subject is given a set of verb-adverb combinations and asked to make a judgment as to their acceptability. Take for example the following items involving the combination of the verb to attack and three adverbs.
Apposite Permissible Anomalous
To attack violently X
To attack stupidly X
To attack meekly X
PROBLEMS IN ASSESSING PROFICIENCY 181
The acceptability judgments appear on the right. Osgood assumes that the acceptability rating is determined by the interaction between the semantic features for which the verb and adverb are characteristically coded. This test, when used with bilinguals, would be a measure of both RB and RC. By careful selection of the items it should be possible to test quite subtle differences in meaning. Because of this, the test is likely to be more sensitive and efficient in detecting small differences than the previous ones described.
4.6.4.5 Metaphor Sensitivity Test
The principle involved here is similar to that in the previous test but it is considered separately here because the nature of the competence that underlies the judgment process is believed to develop differently in the two situations. More specifically, metaphor sensitivity is believed to originate "deeper" in the semantic structure of a language and represents a type of competence that cannot be learned adequately from dictionaries. This type of "Sprachgefuhl" would be particularly sensitive to coordinate achievement in the second language (RB).
4.6.4.6 Semantic Range Test
Mackey and Savard (1967) have described an index of "coverage" which assigns a numerical value to words on the basis of a combination of characteristics which include power of definition, inclusion, extension, and combination-all measures relating to some aspect of the semantic range of words. It is proposed that these or equivalent indices be adapted for the use of individual speakers in order to permit comparisons of individual scores to the population norm. These would then become indices of RB.
4.6.4.7 Word Association Test
Words in both languages are given as stimuli and the subject is free to respond in either language by giving as many words as he can think of within a certain period of time, usually 60 seconds (see Nbble, 1952). The following indices can be derived from this procedure: (i) number of responses to the stimuli in L1; (ii) number of responses to the stimuli in L2;(iii) the difference between (i) and (ii); (iv) number
of responses in the language of L1 for stimuli in L1; (v) number of responses in the language of L2 for stimuli in L2; (vi) number of responses in L2 for stimuli of L1; (vii) number of responses in L1 for stimuli of L2; (viii) number of translation responses for stimuli in L2 number of translation responses for stimuli in L2;(ix) proportion of responses in L2 for Stimuli in L1 [i.e., [(vi)/(I)];(x) proportion of responses in L1 for stimuli in L2 [i.e., [(vii)/(ii)];(xi) number of "bilingual" responses for L1, where applicable (i.e., cognates or homographs for related languages) (xii) number of "bilingual" responses for L2, again where applicable.
4.6.4.8 Paired Associate Task
This test is a measure of learning ability with meaningful materials. The bilingual's comparative performance in the two languages may be estimated, and the general level of his performance compared to the norm within the population. Various procedures for paired associate learning are possible. A convenient procedure which does not require a memory drum is that used in the Modern Language Aptitude Test (Carroll and Sapon, 1958).
4.6.4.9 Semantic Structure of Affective Meaning
This rather complex procedure consists of the following steps: (a) semantic differential ratings of concepts in both languages are obtained from the bilingual;(b) these ratings are then analyzed by means of Tucker's three mode factor analysis (see Levin, 1965) and compared to group factor analyses using various factorial similarity indices (see for example Kuusinen, 1967). These measures should reveal significant departures of semantic structure from the cultural norm (RB and RC).
4.6.4.10 Word Association Structure
There are several methods which have been used to arrive at indices of similarity in word association structure (see Deese, 1965). Kolers (1963), Rosenzweig(l959), Lambert and Moore (1966), Jenkins and Russell (1960), and Moran (1966) have made use of some of these indices in comparisons of word association structure across languages.
PROBLEMS IN ASSESSING PROFICIENCY 183
4.6.4.11 Semantic Cluster Analysis
This measure is an index of vocabulary structure based on factor analysis and cluster analysis techniques described in Osgood, Suci, and Tannenbaum (1957) under the heading "concept factor analysis." Visual representation of vocabulary structure is possible and facilitates comparisons (see for example, Osgood and Luria, 1954 and Jakobovits and Osgood, 1967).
4.6.4.12 Spew Hierarchy
Relying on certain complex arguments presented by Underwood and Schulz(1960) and summarized in their "Spew Hypothesis," Osgood, Archer, and Miron(1962) developed elicitation procedures which yield association data that when treated by means of the information statistic H, reveal a response hierarchy that is stable and haracteristic of a language/culture group. Comparisons of such "spew hierarchies" (to which word frequency distributions are related in a crude way) should reveal the degree of the bilingual's coordinateness.
4.6.4.13 Word Categorization Task
There is considerable evidence (Mandler, 1966; Jenkins and Russell, 1952; Cofer, 1965) that the lexical items in the vocabulary of an individual are structured in his memory along patterns that are organized in clusters which can be revealed under conditions of free recall and spontaneous emissions. There would be considerable interest in determining the stages of development of such vocabulary clusters in learners of a second language and the extent to which compound-coordinateness affects the establishment of such clusters.
4.6.4.14 Word Stereotypy
This measure is related to the H index used in the determination of the spew hierarchy described above (4.6.4. 12). It gives an indication of the "richness" of a concept and its stereotypy, that is the extent to which subjects under conditions of a restricted word association task will tend to give the same high frequency responses. Again, the interest here, as in the last few measures just described, lies in the
determination of the degree of correspondence between the bilingual's vocabulary structure and that of the population norm.
4.6.4.15 Translation Test
Facility in translation as indexed by latency and accuracy measures may be related to compound-coordinate bilingualism in so far as it involves language switching and semantic correspondence, as discussed earlier in this section. It would appear that a translation test involving sentences or meaningful discourse should be more revealing than the translation of single words since the latter is likely to be either too simple and automatic-when the word in isolation has a single meaning, or too difficult and conflictual-when the word has multiple meanings. One should also keep in mind the possibility that translation may be a separate special skill only partially related to the usual skills of decoding and encoding required in everyday discourse. If this were the case, translation should not be used as a measure of language proficiency.
4.6.4.16 Language Alternation Test: Encoding
The purpose of this test is to force the bilingual to encode in both languages in rapid succession. The task has two levels of difficulty: one consists of naming objects shown to the bilingual in each language, alternating with each new stimulus; the other level consists of persuading him (a difficult task with some!) to tell a story while alternating languages for "chunks" of discourse as he goes along. The size of these chunks (i.e, words, phrases, sentences) at reasonable fluency is likely to be the significant variable.
4.6.4.17 Language Alternation Test: Decoding A
Here the bilingual's ability to understand written and oral material is being tested. The text is made up by alternating languages using phrase chunks. Here is an example of a sentence in such a text using English and French as the two languages: "The government of Henry the Seventh, de son fils, et de ses petits enfants was, on the whole, plus arbitraire, than that of the Plantagenets."
PROBLEMS IN ASSESSING PROFICIENCY 185
4.6.4.18 Language Alternation Test: Decoding B
This differs from the previous test in that the alternating chunks are smaller. Here is an English-French example: "A complex appareil used in deux ou plus parts de la world suggests un rapport between eux in very measure to leur complexity." Understanding of both kinds of texts can be evaluated by requiring the subject to answer questions based on the text. Latency measures may also prove to be useful.
4.6.4.19 Pronunciation Switch Task
This test is added for the special case where the two languages possess homographs either because of linguistic closeness (as in English and French) or because of linguistic contact (e.g., English origin words in the linguistically unrelated languages of India that use a common script). A list of homographs is prepared and the bilingual is instructed to read it as fast as he can by alternating languages with each word. Both speed and accuracy are recorded. Here is an ex- ample of such a list in English and French: FORMER, OR, CHAIR, SEIZE, LAME, POUR, TAPER, COMMENT, HABIT, MAIN, SON, COIN, FOUR, CASE, ON, CENT, RIDE, CHAT, DRESSER, BUT, PIN, SORT, DOT, LOIN, BOND.
4.6.4.20 Capsule Sentences
The peculiar (but nevertheless, regular) transformations that characterize newspaper headlines result in sentences whose meaning often eludes non-native speakers of a language whose proficiency otherwise is quite adequate. This test is designed to measure the bilingual's knowledge of these peculiar transformations by requiring him to recover the original sentence from the "capsule" sentence. For example: given the headline "Tells Parties for Army Aids by U.S. Firm," the subject may recover "He reveals that parties are given by a U.S. firm on behalf of army personnel." The recovered sentence should contain not only the factual information but also the sub rosa implications ("reveals" for "tells").
4.6.4.21 Sentence Transformation Test
The purpose of this test is to measure the bilingual's capacity to transform sentences from one mode to another (e.g., from active to passive; from negative question to active declarative, etc.; see Miller,1962). Error and speed are the two relevant measures.
4.6.4.22 Cloze Procedure
In this test a text is mutilated by deleting every fifth word and the incomplete text is then given to the subject for completion. The choice of words in the bilingual's attempt to restore the text is compared to the original or to the performance of native monolinguals. Accuracy can be scored in terms of exact type or in terms of semantic correspondence to the original. The Cloze procedure was first described by Taylor (1953).
4.6.4.23 Subjective Evaluation of Competence
A competent bilingual judge, in a face-to-face interview with the subject, evaluates the bilingual's competence in five areas: phenol- ogy, syntax, vocabulary, oral comprehension, and fluency. The judge has available a rating sheet which he fills out while he is conversing with the subject. Here is an example of such a rating sheet:
Accent
foreign:-----:-----native
Grammar
inaccurate:-----:----accurate
Vocabulary
inadequate:------:---adequate
Fluency
uneven -- : --- even
Oral Comprehension
incomplete : ------: --- complete
PROBLEMS IN ASSESSING PROFICIENCY 187
The six-point scale is defined, from left to right, as follows: extremely, quite, more left than right, more right than left, quite, extremely. A pretest of the reliability of this scale using two sophisticated judges rating French-English bilinguals yielded interjudge correlations in the high 80's and low 90's for the five scales.
4.6.4.24 Subjective Evaluation of Compound-Coordinateness.
During the interview the judge engages the bilingual in conversation about his language acquisition history and usage, asking sufficiently specific questions to enable him to rate the individual on four measures of compound-coordinate bilingualism: compound acquisition, coordinate acquisition, compound usage, and coordinate usage. In each case a rating is given which may vary from 0 to 100. In a pretest using French-English bilinguals, interjudge reliability was found to be in the high 60's and low 70's for the four measures. Compound acquisition and usage correlated positively with each other (in the 70's) and negatively (in the 60's) with coordinate acquisition and usage (the latter also correlating positively with each other, in the 70's).
4.6.4.25 Subjective Evaluation of Attitude
Two measures are included under this heading: an orientation rating (discussed earlier in this Chapter) and a rating of the attitude of the subject toward bilinguality: how advantageous or important it was to be bilingual, whether he'd require his children to achieve a high degree of bilingualism, and so on. Both of these ratings may vary from 0 to 100.
4.6.4.26 and 27 Proficiency Tests in Both Languages
In many languages special proficiency tests have been developed and are widely known and used. Where these are not available, achievement scores in language courses or some other substitute tests would have to be used. In all cases, the question of validity will have to be carefully considered. These problems cannot be dealt with here and the reader is referred to the discussions in Lade (1961)and Mackey (1965), and the discussion in Chapter 3 and at the beginning of this Chapter.
4.7 TRANSFER EFFECTS FROM ONE LANGUAGE TO THE OTHER
Transfer is perhaps the single most important concept in the theory and practice of education. In its most general form the principle of transfer refers to the hypothesis that the learning of task A will affect the subsequent learning of task B and it is this expectation that justifies educational training in schools as a form of preparation for the subsequent demands that society will impose upon the individual. In fact, some educators (e.g., Ferguson, 1956) consider transfer as the general case and learning as the specific case where task B is highly similar to task A and learning of task B thus constitutes further practice on task A.
The general paradigm that applies to all transfer studies can be summarized as follows:
Experimental Group: Learn A Learn B
Control Group 1: Learn B
Control Group 2: Learn X Learn B
In the experimental group, the transfer effects that are presumed to operate are of two types: specific and nonspecific. The former are attributed to the habits acquired in learning task A which are utilized in the learning of task B. The latter are attributed to general factors involved in the learning of any task such as warm-up effects and the development of learning sets. By comparing the performance of the experimental group to that of Control Group l, the combined effects of specific and nonspecific transfer can be assessed, whereas by com- paring their performance to that of Control Group 2, the specific transfer effects can be assessed without the contamination of the nonspecific factors. The interest in transfer in second language learning has been most explicit in attempts at specifying interference effects through contrastive analyses between the second language (L2) and the first (L1). The most extensive analysis of the potential mechanisms of interference between two languages is that of the classic work of Weinreich (1953). Today contrastive analysis is a widely accepted technique among second language teachers and is commonly used to guide classroom and laboratory activities in terms of the type of materials to be introduced at specific times and the nature of the
PROBLEMS IN ASSESSING PROFICIENCY 189
practice exercises to which the learner is to be exposed. In addition the operation of transfer is implicitly assumed both within the language learning situation and in the applicability of the course instruction to skills the individual will be called upon in using the language outside the school. The expectation of transfer within the language instruction situation relates to the notion of cumulative gains to be obtained so that language acquisition is conceived of as a step-by-step building process of successively more complex stages and whatever is learned in the early stages is expected to be retained and transferred to later stages. There is the further expectation that knowledge of the common elements which constitute the structure of the language will transfer to the various skills; for instance, a grammatical relation learned orally is supposed to facilitate the acquisition of the same relation in reading or writing.
The external applicability of the classroom training similarly rests on assumptions of transfer effects so that a student drilled in pattern practice is expected to be able to transfer these skills to the real life situation of an ordinary conversation with a native speaker. Reading of literature in the second language and indirect exposure to the foreign culture through the content of sentences, pictures, and films are similarly justified in that such knowledge is expected to facilitate later understanding of "field-like" situations. Obviously then, a major portion of what constitutes second language teaching today rests its justification on the explicit or implicit recognition of the operation of transfer effects.
4.7.1 The Extrapolation of Knowledge
The literature on transfer (when the term is considered in its broadest sense) is possibly more extensive than that on any other topic in psychology and education. It can be subdivided into two categories: on the one hand, one finds a mass of experimental reports on highly specific aspects of transfer effects under laboratory learning conditions; on the other hand, one finds a substantial body of writings which deals with the operation of transfer in real life settings such as the school and industrial training programs. In a pattern now familiar in the social sciences whereby scientific exactitude is in-
versely proportional to realistic concerns, we find that whatever specific knowledge there is on transfer concerns tasks and settings that are largely irrelevant to crucially important educational concerns, and the amount of specific and reliable knowledge on the latter is practically nonexistent. Careful reviews of the vast literature pertaining to transfer are invariably pessimistic. Concerning the work on paired-associate learning Betts (1966) concludes that "the data presented here and the equations derived from these data are much too tentative to be offered to educators as practical solutions to their problems" (p. 135). Concerning the work on interference theory in rote learning one of the most authoritative experts on the subject recently con- cluded that "thus far, there is little empirical evidence in support of [the] assumption [that] forgetting outside the laboratory is a function of the same variables and represents the same processes as are observed in formal studies of interserial interference" (Postman,1961, p. 166). In an extensive, formal, and detailed report on restricted aspects of learning Russian under controlled laboratory conditions, Crothers and Suppes (1967) conclude that "...in these second language learning experiments the proportion of correct responses usually seemed to depend more on the detailed structure of the item than on the presentation-order variables that were examined" (p. 311). This pessimistic conclusion is particularly note-worthy because the work of Crothers and of Suppes is often cited as examples par excellence by proponents of the formalized experimental approach to problems in learning and teaching.
I am citing these conclusions, which are representative and describe accurately the "state of the art," because I want to draw attention to a tendency that has become quite widespread in the social sciences generally and in education, including second language teaching, specifically. This tendency consists of an attempt to justify teaching practices of a particular kind by appealing to experimental findings in the laboratory that deal with materials of an entirely different order of complexity than the learning task in question. An example of this, which by now ought to be notorious, is the description of language learning and language behavior that is based on conditioning principles developed through laboratory studies with rats and rote learning studies with human subjects. A more recent
PROBLEMS IN ASSESSING PROFICIENCY 191
instance, one which is very much topical today, concerns the justification of pattern drill practice in second language teaching by appealing to laboratory experiments on motor skills and to principles of generalization and transfer derived from stimulus conditioning and verbal rote learning studies. While I think it is both justifiable and desirable to apply knowledge which one might gain from laboratory experimentation to educational and social concerns, I do not think that this extrapolation carries greater scientific merit than the extrapolation that comes from knowledge acquired outside the laboratory through observation, experience, and insight. The extrapolation process in both instances is not a rigorous matter and cannot be scientifically justified. Chomsky's evaluation of this tendency has been quite harsh: "...there has been a natural but unfortunate tendency to "extrapolate,' from the thimbleful of knowledge that has been attained in careful experimental work and rigorous data processing, to issues of much wider significance and of great social concern. This is a serious matter. The experts have the responsibility of making clear the actual limits of their understanding and of the results they have so far achieved, and a careful analysis of these limits will demonstrate, I believe, that in virtually every domain of the social and behavioral sciences the results achieved to date will not support such 'extrapolation' "(1968, Preface).
I suspect that these remarks will be welcomed by those practitioners of the teaching arts who have always preferred to follow their flair and who have maintained a distaste for the analytic and molecular formulations of the experimentalists. However, I would like to dispel immediately any sense of security that a smug subjectivity may provide. It would be a mistake to assume that because extrapolation from the laboratory is not a scientific process, therefore we must give up all attempts at developing systematic evaluation criteria whereby teaching methods can be made more effective. It is possible to be systematic without at the same time being rigid about the manner in which we make additions to our knowledge. The analysis of transfer effects in second language learning which I sketch in this section is molecular and has, I believe, a systematic character which could potentially organize the knowledge that language teachers already have from their experience in the classroom in such a way that they could make more effective use of such knowledge by
rendering it more explicit. Hopefully, it might have the additional quality of enabling them to increase the usefulness and signficance of the knowledge they may gain through additional and continuing experience in teaching.
4.7.2 Some General Principles of Transfer
In this attempt to summarize some principles of transfer I will rely on the experimental work that is extant without, however, the pretense that the statements are scientifically rigorous. One should more properly consider them as fairly specific descriptions of current knowledge about transfer effects, knowledge which was arrived at by various procedures, which include interpretation of experimental data, observational data gathered through the experience of educators, and selections from what is common knowledge in the culture (sometimes called "common sense").
A general formulation of the transfer problem must deal with five basic elements: task A, training or practice on task A, training or practice on task B, and the relation between task A and task B. Since our specific interest here lies in language learning, let us refer to the five elements as follows: PL1 (proficiency on task A), PL2 (proficiency on task B), TL1, (training in L2), TL2 (training in L2) and RL1-L2 (the relation between L, and L2). The transfer effects to be R expected in second language learning can then be expressed by the following formula:
PL2 = f(PL1, tL1, tL2, RLI-L2)
This formula says that attained proficiency in L2, will be some joint function of attained proficiency in L1, training in L1, training in L2, and the relationship between L1, and L2. Consider now some of the possible relations that may hold in this formulation (see Ferguson, 1956) and how they are reflected by this formula. In the case where tL2 is constant and RL1-L2 is fixed we have:
PL2 = f(PL1)
PROBLEMS IN ASSESSING PROFICIENCY 193
where PL1= f(tL1). This says that given fixed amounts of training, proficiency attained in L2 is a function of native language proficiency, which no doubt reflects a general language ability. It is known that verbal ability and verbal intelligence are significant factors in second language learning although the shared common variance is fairly small. Furthermore, certain special abilities whose development may very well depend on TL1 factors in early life, such as memory for coding phonetic materials and sensitivity to grammatical relations (see Carroll, 1966), seem to share substantial common variance with PL2
In situations where tL1 is very large, has attained a stable value(e.g., an adult learning a second language), the general formula reduces to:
PL2=f(tL2)
where RL1-L2 is assumed fixed. This expresses the usually recognized function in second language learning where achievement is said to be a function of training in that language.
Where tL1 is very large, PL1 stable, and tL2 fixed, the general formula reduces to:
PL2 = f(RL1-L2)
which can be recognized as expressing the common knowledge that "exotic" languages are more difficult to learn than the more familiar ones.
An interesting feature which emerges by rearranging the elements in the general formula is expressed by the equation:
PL1=f(PL2, RL1-L2)
This equation says that when RLI-L2 has a particular value (e.g., with closely related languages) the learning of a second language may come to have interference effects on the mother tongue. This refers to what I have called elsewhere "backlash interference" as distinct from the more familiar linguistic interference which is expressed by the equation:
PL2 = f(PL1,RLI-L2)
One of the factors which is known to be important in the operation of transfer concerns the common elements between the two tasks; in the present discussion this refers to RLI-LZ. I have pointed out earlier that this relation has three different aspects, as follows:
1. RA = f(XL1 - XL2)
2. RB = f(XL2 - XL2)
3. RC = f(XL1 - XL1)
where X refers to a particular speaker (either of L1 or of L2) and X refers to an idealized monolingual of L, or L2. Equation 1 (RA) refers to the relation between the bilingual's two languages; equation 2 (RB) refers to the relation between the bilingual's second language and that of an idealized monolingual speaker of L2, equation 3 (RC) of an idealized monolingual speaker of L1. These three relations are theoretically independent and their simultaneous value within the various linguistic domains defines the bilingual's position on the compound-coordinate continuum.
4.7.3 Transfer Effects in Second Language Learning
In attempting to assess the transfer effects in second language learning it will be necessary to consider the various relations separately since the same situation may have differential effects on each of them. Consider for instance the values to be expected for each of these relations on the basis of Osgood's three laws of interlist similarity (as reviewed by Jung, 1968). The three principles can be stated as follows:
1. Two tasks in which the stimuli are the same but the responses different (A-B, A-D) will yield negative transfer which increases the more the responses get dissimilar1 .
1Following established practice in the literature on paired-associate learning, the first letter in expressions like A-B or A-D refers to antecedent stimulus factors while the second letter
PROBLEMS IN ASSESSING PROFICIENCY 195
2. If the responses are the same, but the stimuli are different (A-B, C-B), positive transfer is expected which increases the more the stimuli get similar.
3. When both stimuli and responses are different in the two tasks (A-B, C-D), negative transfer is expected which increases the more the stimuli get similar (see Figure 4).
FIGURE 4. TRANSFER EXPECTATIONS IN SECOND LANGUAGE ACQUISITION BASED ON OSGOOD'S TRANSFER SURFACE. (OSGOOD. 1953. FIGURE 172. p. 532.)
refers to consequent response factors. The comma in expressions like "A-B, A-D" separates the first task from the second. Identity of letters is intended to indicate that the same factors are believed to hold in the two tasks. Thus, the expression "A-B, A-D" refers to two tasks in which the stimulus conditions are the same but the response conditions different. Please note that "stimulus" and "response" as used here do not carry the usual rigorous denotations and connotations which they have in the literature on "S-R theory."
In second language learning "stimulus similarity" may be taken to refer to the environmental conditions that are antecedents to linguistic utterances: these include the physical, external environment (to be referred to as EL1 Or EL2) as well as the mental, internal environment (to be referred to as EL1, or EL2). "Reponse similarity" may be taken to refer to the structural relations between the linguistic systems of the two languages (to be referred simply as L1 and L2).
We are now ready to express the transfer expectations in second language learning which Osgood's three laws would suggest (see Figure 1):
1. In a compound setting with the indirect method of teaching (where EL1 and EL2 are highly similar, and so are EL1 and EL2), we would expect negative transfer which increases the more L1 and L2 get dissimilar.
2. With closely related languages (where L1 and L, are similar) we would expect positive transfer which increases the more EL1 and EL2 (and EL1 and EL2) get similar, i.e, in a compound setting with the indirect method.
3. When EL1 and EL2 (and EL1 and EL2) are relatively different (as with the direct method in a coordinate setting) and L1 and L2 are different (as with unrelated languages) negative transfer is expected which increases as EL1 and EL2 get similar.
These principles can be restated from the point of view of the interest in maximizing positive transfer and minimizing negative transfer.
Hypothesis 1. With unrelated languages a coordinate setting (symbolized as CR) will yield less negative transfer than a compound setting (symbolized as CP).
Hypothesis 2. With related languages a CP setting will yield more positive transfer than a CR setting.
Note that the second hypothesis here derived seems to be in conflict with the widespread inclination among most proponents of the audiolingual method to maintain as much as possible a direct learning method and to avoid as much as possible any use of L1 in the learning situation. There is some evidence, however (Lambert,
PROBLEMS IN ASSESSING PROFICIENCY 197
Gardner, Barik, and Turnstall, 1963), which supports the expectation that individuals studying a second language that is closely related to their first language (in this case French being studied by Americans) do better in the course if they show evidence of allowing a CP interaction in their meaning systems. In our terms, EL1 and EL2, were not kept separate even though there was an effort on the part of the instructors to maintain overt environmental separation (EL1 and EL2)by offering to the American students enrolled in the intensive summer course the distinctive French-Canadian atmosphere in bilingual Montreal and not permitting, under threat of expulsion, any use of English. Despite these efforts on the part of the instructors the students who acted internally as CP bilinguals achieved higher grades. To be sure, this evidence is only suggestive with respect to the transfer expectation expressed as Hypothesis 2 and, because it seems to conflict with our common sense expectations, we should be wary of it. Nevertheless, now that we are alerted to its possibility, we should watch out for possible confirming evidence whenever possible. Jung (1968) describes an elaboration contributed by Martin of Osgood's three laws of similarity. Consider the three components of specific transfer in two successively learned paired-associate tasks. They are response learning, the establishment of forward associations, and the establishment of backward associations. The findings are apparently consistent with the following statements:
1. With the A-B, C-B paradigm (different stimuli, same response) there is positive transfer from the first task (A-B) to the second task (C-B) for response learning.
2. With the A-B, A-D paradigm (same stimuli, different responses), there is negative transfer for the establishment of forward associations (A-B connections in the first task conflict with the A-D associations in the second task).