Putting a test of multiword expressions to a test

This plenary described the rationale for and development and validation of a new test of multiword expressions, called the Phrasal Vocabulary Size Test, intended to complement existing tests of vocabulary that only assess receptive knowledge of individual word forms.

Five vocabulary myths

Is reading the best way to learn vocabulary? Should you pre-teach vocabulary? Should students be taught to guess the meaning of unknown words from context? What is the best way to remember new words? Many teachers - and even students - believe they know the answers to those questions, but some widely held beliefs are not actually supported by empirical research evidence. In this plenary, we will explore some of the more common notions regarding vocabulary teaching and acquisition, and will discuss these beliefs in the light of what the latest research actually shows.

How good are children and adults at identifying collocations?

It is well established that collocations are pervasive in authentic language. However, how reliably do learners notice them? The researchers will present new evidence showing that both children and adults often misread collocations, thus negatively affecting comprehension. Pedagogical implications will be highlighted, including practical ways to address the issue.

The Phrasal Expressions List (PHRASE List) and what it can do for you

Studies in corpus linguistics have long shown that multiword expressions are extremely common in English. While this fact alone is interesting, language educators also want to know the answers to other questions:

- What types of multiword expressions are most likely to present problems to L2 learners?
- Which are the most useful multiword expressions for language teaching/learning?
- Which are the most common multiword expressions in English?

The answers to those questions have proved elusive. However, there have been recent attempts to redress this gap (Simpson-Vlach & Ellis, 2010; Shin & Nation, 2008), and now, introduced in this session, there is the PHRASal Expressions List (PHRASE List).

This talk will be broken down into four segments:

1. Wordlists and the need for a phrase list
2. The development and rationale for the PHRASE List
3. Pedagogical (teaching) applications of the PHRASE List
4. Questions and discussion

Lists of words for language teaching have existed for many years (e.g. West, 1954), and such lists have been used to inform the writing of graded readers, for example, and vocabulary tests. What all such lists have in common is that they do not incorporate phrasal vocabulary (e.g. 'might', 'well' versus 'might as well'), and therefore are not fully representative of the English lexicon.

The PHRASE List was carefully developed by the researchers over several years, implementing a meticulous quantitative (corpus extraction) and qualitative (manual selection of certain types of phrases) methodology which has now been used for the development of a new vocabulary test, for example, and to create text analyzer software that will allow teachers automatically check texts for key phrases useful for learners.

Concrete classroom applications of the list will be emphasized.

The PHRASE List: the development and applications of a list of multiword expressions

For some time research has shown that vocabulary is not comprised of single orthographic words alone, but also combinations of words that together form distinct meanings (e.g. 'might' versus 'might as well'). Moreover, such multiword expressions are often non-compositional in meaning, requiring holistic processing – much like individual lexical items. Furthermore, multiword expressions are prevalent in naturally-occurring speech and writing, and some occur as frequently as what are considered high-frequency individual words. Because frequency is the most common criterion to select and rank lexis for vocabulary assessment purposes, a strong argument can be made to include these highly-frequent multiword expressions in measures of vocabulary size and depth, as well as pedagogy in general. However, this is seldom done at the moment, because 1) there is still a bias towards thinking of vocabulary in terms of single words, and 2) there is no established list of the highest frequency and most useful multiword expressions.

The presenter will describe a study involving the careful compilation of a list of multiword expressions, frequency-matched with word families from the British National Corpus, that could be used to more systematically include phrasal expression in language teaching and testing. The rationale and methodology for the research will be presented, as well as the criteria that were used for the selection (and exclusion) of the items on the list. Practical applications will be discussed, including how the list can be used to devise more valid vocabulary tests, reading tests and teaching syllabuses.

Putting a test of multiword expressions to a test

There already exist many tests that aim to provide an approximate measure of how many words a person ‘knows’, but it is accepted now that vocabulary knowledge consists of more than just individual orthographic words. Multiword expressions, such as 'by and large' and 'in order to' are prevalent in both reading and writing, and have been shown to negatively affect comprehension when not known. Despite their importance, few attempts have been made to develop a test that assesses knowledge of them the same way individual vocabulary words are tested. In this talk, such a test will be discussed, including its development and what resulted when it was piloted and field tested.

Assessing L2 knowledge of multiword expressions

Current tests of vocabulary knowledge only assess individual words, yet it is now well established that multiword expressions are also a very important part of the lexicon. A new test that aims to integrate the assessment of multiword expressions will be presented in this talk, as well as a discussion about the challenges that arose during its development, and how those challenges were met.

Vocabulary testing: beyond individual words

Validating the English Profile Wordlists: How do learners use phrases?

This talk will draw on ongoing research carried out in joint collaboration with Cambridge ESOL and English Profile to both validate the multiword expressions ('phrases') from A1 to C1 levels, and through the learner and native-speaker data analyze similarities and differences in their production, potentially drawing inferences regarding acquisition. Practical implications for pedagogy will be highlighted.

Development of a list of multiword expressions for language teaching and testing

Evidence of lack of processing of multiword lexical items in reading tests

To what extent do students notice multiword formulaic sequences in reading tests, and to what degree do such items affect comprehension? The preceding questions formed the basis of a research undertaking that aimed to better understand what effect – if any – the presence of idiomaticity in text has on L2 reading comprehension. Using a within-groups, repeated measures design, a two-part test of reading comprehension revealed not only a significant difference between levels of comprehension when texts contained multiword expressions (even when the texts themselves in each test part were written using the exact same pool of high-frequency words), but also that the candidates consistently had an inflated view of how good their comprehension was of the texts that contained idiomaticity, via a self-reported measure of comprehension.

In brief, on the basis of the results of the study, it can be asserted that even learners at higher proficiency levels tend to not notice many multiword expressions in text, and when they do are generally not good at guessing what they mean in context. Moreover, there is a tendency for candidates to overestimate how much they understand as a function of those unnoticed and/or misunderstood expressions.

Therefore, drawing on the results of this study, the presenter will posit that

1) multiword expressions are prevalent in naturally-occurring texts, and those expressions are mainly comprised of the most common words in English – though the meaning of those individual words does not always match the meaning of the words in combination – and therefore can be ‘deceptively transparent’ (Laufer, 1989);
2) as the results of this study provide evidence that multiword expressions can negatively affect comprehension, a principled methodology for their inclusion in L2 reading tests can positively contribute to the validity of those tests;
3) a reliance on the frequency information of individual words alone to inform the design of a reading test and/or determine the ability of a text to discriminate between thresholds of proficiency can result in a test of limited construct validity;
4) particularly in determining the thresholds of higher levels of reading proficiency, a demonstrated recognition and/or understanding of the most common non-compositional multiword expressions can be one way in which we decide when someone is (or is not) at a level.

Limitations of the study in question will be presented, as well as possible directions for future research and applications.

The effect of frequency and idiomaticity on L2 reading comprehension

To what extent do L2 learners notice idiomatic formulaic sequences reading, and to what degree do such items affect comprehension? Using a within-groups, repeated measures design, a two-part test of reading comprehension revealed not only a significant difference between levels of comprehension when texts contained multiword expressions (even when the texts themselves in each test part were written using the exact same pool of high-frequency words), but also that participants consistently had an inflated view of how good their comprehension was of the texts that contained idiomaticity, via a self-reported measure of comprehension.

In short, on the basis of the results of the study, it can be asserted that even learners at higher proficiency levels tend to not notice many multiword expressions in text, and when they do are generally not good at guessing what they mean in context. Moreover, there is a tendency for candidates to overestimate how much they understand as a function of those unnoticed and/or misunderstood expressions.

A Phrasal Expressions List for Language Testing

For some time research has shown that vocabulary is not comprised of single orthographic words alone, but also combinations of words that together form distinct meanings. Moreover, such multiword expressions are often non-compositional in meaning, requiring holistic processing – much like individual lexical items.  Furthermore, multiword expressions are prevalent in naturally-occurring speech and writing, and some occur as frequently as what are considered high-frequency individual words.  Because frequency is the most common criterion to select and rank lexis for vocabulary assessment purposes, a strong argument can be made to include these highly-frequent multiword expressions in measures of vocabulary size and depth.  However, this is seldom done at the moment, because 1) there is still a bias towards thinking of vocabulary in terms of single words, and 2) there is no established list of the highest frequency and most useful multiword expressions.

A study is therefore presented involving the careful compilation of a list of multiword expressions, frequency-matched with word families from the British National Corpus, that could be used to methodically include multiword lexis on tests that purport to assess vocabulary ‘thresholds’ (e.g. the Vocabulary Levels Test - VLT).  The rationale and methodology for the research is presented, as well as the criteria that were used for the selection (and exclusion) of the items on the list. Implications are discussed, especially as they pertain to the extent to which multiword vocabulary items may constitute one key criterion that distinguishes one proficiency level from another.  Finally, the poster also shows how the list is being used to enhance an existing test (VLT) by adding multiword expressions to the current frequency levels, creating a test that combines individual words and multiword expressions in a way that mirrors their distribution in the real world.     

The English Profile Wordlists: validation and future developments

Co-presented with Annette Capel; Responsible for 'Phrasal Expressions' component

Towards a more systematic inclusion of multiword expressions in SLA research and pedagogy

Although it is now widely accepted that the mental lexicon comprises not only individual words, but also multiword expressions of various types, SLA research and pedagogy still do not generally account for such expressions when investigating reading comprehension, for example, or estimating vocabulary size. One possible reason for this gap is that little is known about the effect multiword expressions have on comprehension when learners are not aware of their presence in text, and it has thus far been largely assumed that number of commonly-occurring expressions that could negatively affect comprehension is relatively small (Nation, 2006).

This presentation will report on two recent studies. The aim of the first study was to explore the extent to which L2 readers may overestimate how much they understand in text as a function of multiword expressions (e.g. by and large, every so often) that they either misunderstand or misidentify as separate single orthographic words. Through comprehension questions and participant self-reported comprehension ratings, the results indicate that when learners encounter multiword expressions in text, they often believe they have understood the expression even though they have not. A question left unanswered in the aforementioned study, however, was how frequently semantically opaque expressions naturally occur in text. To address this question, the presenter conducted a corpus-based study which aimed to extract a list of expressions that, when reading, may cause difficulty for L2 learners. Unlike some earlier estimates, the exercise produced a list of hundreds of commonly-occurring expressions that, due to certain semantic properties, may pose comprehension difficulties for learners. The implications and applications of the list of expressions that resulted from this research will be discussed.

The presenter will conclude by demonstrating how SLA research can benefit from taking multiword expressions into account when studying such areas as reading comprehension and vocabulary acquisition.

The PHRASE List: the development and applications of a list of multiword expressions

Islands of reliability

Upon investigating the spoken English of German speakers, Dechert (1980) observed that they often used the same combinations of words over and over as a way to maintain fluency, and he called these recurring expressions "islands of reliability." Research since then has shown that these "islands" are a ubiquitous and in fact necessary part of communication, particularly oral communication. Examples of this phenomenon will be shown in this talk, and related empirical research will be presented. Most importantly, the talk will highlight implications for teachers and students of such "islands."

Why multiword expressions make a difference

Most studies on reading and listening comprehension suggest that knowledge of at least 5,000 word families is enough for a basic comprehension of most communication in English. However, the speaker will present more recent evidence that shows that a focus on 'words' alone can mean that students are missing over 10 percent of the true meaning of a text. Although an awareness that multiword expressions (e.g. 'by and large', 'might as well') are an important part of the lexicon, L2 pedagogy practitioners have struggled to both prioritize them and systematically incorporate them in formal instruction. Possible solutions will be offered, and practical, hands-on activities will be present throughout the workshop.

Moving from 'correct' to 'it depends'

Despite what our students might often expect, often there is not one "right" answer to the question "Is this correct?" In fact, it might be best to get both our students and teachers used to the idea that sometimes the best answer to that question is "it depends." Language is of course highly context-sensitive, and even the slightest change in the way we formulate a message can affect the way it is interpreted. Rather than an obstacle or difficulty, in this talk we will discuss how "it depends" can be a powerful and empowering pedagogical mechanism.

Fluency: the big role of little words

As the saying goes, "sometimes it's the little things." Research has shown that for effective comprehension to occur in English, one must know at least 8,000 of the most common words (Nation, 2006). But which of those words are really the most important? The highest frequency words in English (i.e. the first 1,000) are usually 'little' words - little words that do a lot of work! In this talk, the speaker will show that often times it's the little words (e.g. so, well, just) and combinations of little words (e.g. sort of, a bit, in terms of) that can make all the difference when it comes to fluency.