Introduction
An examination of the major models of
language proposed since Oller’s (1979)
unitary factor of language proficiency
shows that the field of language teaching
has moved towards a more componentially
diverse view of language proficiency.
Scholars such as Canale and Swain (1980),
Canale (1983), Oller (1983), Bachman
(1990), Celce-Murcia, Dörnyei, and
Thurrell (1995) Bachman and Palmer
(1996) and more recently Purpura (2004)
have built on the each other’s previous
works and tried to define and redefine
these components. Perhaps the most
comprehensive treatment of grammar in
the testing context is the model proposed
in Purpura (2004).
After describing his model, Purpura
suggests that assessment of grammatical
knowledge is still in need of more research
with regard to how the construct can be
defined and measured.
The current study is based on the general
model of grammar proposed by Purpura
(2004) and attempts to investigate how
construct of grammatical knowledge can
be measured in light of more recent
proposals about L2 learners’ grammatical
development. According to Rimmer
(2006) grammatical development can be
studied by measuring dimensions of range,
that is, the type and frequency of the forms
used and their complexity and accuracy
(Rimmer, 2006). The current study
suggests that grammatical development
can also be studied by measuring the
dimension of strength. Since it is generally
accepted that as language learners
progress, their grammar becomes more
sophisticated (Ortega, 2003), this paper
investigates the sophistication by
examining amount and type of change
along the dimensions complexity, range,
and strength. A multidimensional
conceptualization of the construct of
grammatical knowledge can be valuable
for developing the specifications of
grammar tests in general and diagnostic
tests in particularly. It allows a more
detailed feedback to teachers, test-takers,
and other stake-holders about learners’
development of grammatical knowledge.
The construct of grammatical knowledge
As regards the models of language, there
are two general perspectives to describe
linguistic phenomena (Purpura, 2004): a)
syntactocentric perspective, where syntax
is the central feature to be observed and
analyzed, such as traditional grammar,
structural linguistics, and transformational-generative grammar b) communication
perspective of language, where the
observational and analytic emphasis is on
meaningful language use, such as corpus
linguistics, and systemic-functional
linguistics. The basic difference between
the two perspectives is that
communication-based perspectives of
language emphasize that language is more
than form and linguistic forms do not have
a fixed meaning in their use. Therefore,
grammaticality becomes synonymous with
appropriacy, naturalness, and
acceptability.
Purpura’s (2004) conceptualization of
grammatical knowledge seems roughly to
conform to Hymes’s knowledge of the
possible, which in turn can be linked to
locutionary meaning in Autsin’s (1962)
speech act theory. Compared to Hymes’
(2001) formulation, syntactocenteric
perspectives focus on the knowledge of the
possible form, while the communication
perspectives focus on the semantic
feasibility, pragmatic appropriacy, and
attested naturalness of exploited forms
during language use.
Proposing a framework to test
grammatical knowledge, Purpura (2004)
distinguishes between the grammatical
knowledge, grammatical ability, and
grammatical performance.
Grammatical knowledge refers to a set of
informational structures related to
grammatical form and meaning available
for use in long-term memory (Purpura,
2004). According to this model of
grammatical knowledge, knowledge of
words and structures involve two
dimensions: form and meaning. In this
respect, the two terms grammatical
knowledge and lexico-grammatical
knowledge are interchangeable. This view
resonates well with that of Bardovi-Harlig
(1995, 2001) who notes that in the process
of SLA learners make connections among
forms with meanings and use and they
need to be able to “distinguish among
semantically neighboring forms”
(Bardovi-Harlig, 2001, p. 218).
Therefore, in grammar assessment “the
primary assessment goal is to determine
whether learners are able to use forms to
get their basic point across accurately and
meaningfully” (Purpura, 2004, p. 274). It
should be noted that as far as grammatical
knowledge is concerned, Purpura (2004)
defines grammatical meaning as instances
of language in which what is said is what
is meant and intended. For example,
sometimes, people are able to produce
accurate grammatical forms but these
forms do not reflect the meaning they
really intend to communicate (e.g., I am
interesting instead of I am interested).
Concerning assessment of grammatical
knowledge, Purpura argues that “the
primary assessment goal is to determine
whether learners are able to use forms to
get their basic point across accurately and
meaningfully” (p. 274). Along the same
lines, Widdowson (2003) asserts that the
most important and practical to test in EFL
situations is testing the capability of
learners to capitalize the knowledge of the
possible. Thus, assessing both grammatical
form and meaning, provides a more
comprehensive assessment of the test-takers’ grammatical ability than just
providing information on form or on
meaning alone.
However, there exits some concerns vis-à-vis measuring the construct of
grammatical knowledge. As an instance,
whether grammar should be tested
implicitly or explicitly needs to be
addressed. According to Alderson (2005)
implicit testing of grammar is justified
because grammar is at the heart of the
language and is implicated in all of the
skills. Thus, if one does not know the
syntactic forms they will not be able to
read, write, listen, and speak. By testing
the skills one can implicitly test grammar.
Explicit testing of grammar is so deeply
rooted in language testing that despite the
arguments for more implicit testing of
grammar they are still popular. Explicit
testing of grammar brings with itself a
promising corollary: such tests can be
easily bent for diagnostic purposes. Also,
if two test-takers know a grammatical
structure, their knowledge may not be
identical. Knowledge of grammar is not an
all or nothing phenomenon but can vary
along a number of dimensions.
Identification and measurement of those
dimensions has the potential to help with
portraying a more complete picture of
learner’s profile of strengths and
weaknesses.
A framework for investigating complexity,
range, and strength of grammatical
knowledge
Complexity is perhaps the most
systematically studied dimension in the
literature whereas range has always been
traditionally used to prepare grammar
tests. These dimensions can facilitate
studying different aspect of the knowledge
in language learners. However, dimensions
of grammatical knowledge are not limited
to these two.
Measuring complexity
It is generally understood that a simple
clause has only a subject, verb, and object
or complement. Also, by definition a
simple phrase (e.g., a noun phrase) has a
determiner and a head noun, or a
prepositional phrase has minimally a
preposition as its head followed by a noun,
a pronoun, or a gerund. To make complex
grammar, these simple patterns should be
modified or something should be added
together.
However, it seems that the mainstream
view of complexity in not constant and is
evolving. Biber and Gray (2010) argue
that the notion of complexity has changed
in the past couple of centuries: carrying
out a historical corpus analysis, they
concluded that whereas the 19th century
prose made frequent use of casuals
embedding, contemporary academic
writing uses more phrasal modifiers
embedded in noun phrases as tools to
elaborate grammatically simple patterns.
Furthermore, the spoken and written
language seem to be complex in different
ways. Biber, Gary, and Poonpon (2011)
show that clausal subordinations are more
common in conversation than academic
writing. In academic writing complex
noun phrase constituents rather than clause
constituents and complex phrases rather
than clauses are more common.
According to Rimmer (2006), the
complexity component is multifaceted and
includes syntactic, psycholinguistic,
markedness, and at times can be related to
the frequency component. However, as he
suggests, the notion of complexity, which
is usually based on tradition and intuition,
may not be very reliable. A related notion
to complexity is grammatical difficulty.
For DeKeyser (2005) it appears that some
factors account for the difficulty:
complexity of form, complexity of
meaning, and complexity of the form-meaning relationship, frequency, and also
complexity of processing.
Norris and Ortega (2009) propose a
multidimensional framework for
measuring syntactic complexity. After
reviewing several measures of syntactic
complexity, they argue certain measures
are more revealing for specific proficiency
groups; for beginning levels coordination
index, for intermediate levels mean
number of clauses per t-unit, and for
advanced levels mean number of words
per clause are recommended.
The results of the study by Biber, Gary,
and Poonpon (2011) suggest that after
learning the simple patterns in grammar,
L1 learners go through progressive stages
of grammatical complexity:
from finite dependent clauses
functioning as constituents in
other clauses, through
intermediate stages of nonfinite
dependent clauses and phrases
functioning as constituents in
other clauses, and finally to the
last stage requiring dense use of
phrasal (nonclausal) dependent
structures that function as
constituents in noun phrases. (p.
29-30)
As it appears, complexity is not a single
unified construct, therefore, a single
measure may not adequately represent it
(see also Ortega 2003; Rimmer, 2006,
2008; Ellis and Yuan, 2005; Robbinson,
2007; Ravid and Berman, 2010). Yet,
obviously it would not be feasible to
address grammatical complexity at length
in one study. This study focuses on one
area of complexity. Wolfe-Quintero,
Inagaki, and Kim (1998) provide an
extensive survey of research on L2 writing
development and conclude that clauses per
t-unit (C/T) and dependent clauses per
independent clause (DC/C) are the best
complexity measures of the late 90s.
Biber, Gary, and Poonpon (2011) maintain
that many linguists from different
theoretical backgrounds consider
dependent clauses as one of the most
important types of grammatical
complexity. Thus, this study has limited
itself to studying relative clauses and
conditional clauses as hallmarks of
syntactic complexity. Complexity is
operationalized by tallying the scores on
the complex items on the test regardless of
their format or spec.
Measuring range (variety)
For practicality purposes the study focuses
on range (a component of variety),
keeping frequency of structures aside from
the equation since enough corpus data is
not available at present. Range can be
operationalized by using a list of
grammatical categories similar to the one
used for DIALANG project (Alderson,
2005) as it is both comprehensive and
practical for test design purposes. Range,
then, is defined as the number and type of
categories for which test-takers show a
degree of knowledge of grammar as
measured by a correct answer to an item
on the test. For example, one of the items
on the list is concerned with verb
inflection. According to Bardovi-Halig
there are a few reasons the subsystem of
tense and aspect is of interest for SAL
studies: study of time and aspect is central
to most ESL and EFL curricula, many
language programs require mastery of
certain tense and aspects for advancement
from one level to another, and tense and
aspect play a central role in grammatically
focused teaching materials. Also, many
important English tests in Iran such as
those administered by the National
Organization or Educational Measurement
such as TOLIMO, and Ph.D. entrance
exams such as UTEPT have items testing
tense and aspect.
In the present study, range of grammatical
knowledge is defined as the number and
type of grammatical categories for which
test-takers show a minimum degree of
knowledge by a correct response to at least
one of the three items having the same
spec but different item formats.
Measuring strength
Although there is some research on the
construct of strength of vocabulary
knowledge (Laufer & Goldestein, 2004;
Laufer, Elder, Hill, & Congdon, 2004), it
seems that such a construct has not been
explicitly proposed for grammatical
knowledge. Measuring the strength
dimension can complement description of
the profile of the grammatical knowledge
of language learners. As an instance, two
learners’ (Learner A and B) range of
grammatical knowledge can include a
number of tense and aspects, they could
have also learned how to make relative
clauses. However, this description does
not provide information about these
learners’ degree of the control over those
structures. Learner A might be able to
recognize the correct choice in a multiple-choice (MC) question, but fail to find the
mistake on an editing item. On the other
hand, Learner B may be able to perform
well on both item formats. Therefore, it
could be concluded that learner B’s
knowledge of that structure may be
stronger. For the purposes of the current
research, strength of grammatical
knowledge is defined as the extent to
which a test-taker can answer correctly a
variety of items requiring different types
of cognitive processing, all measuring the
knowledge of the same grammatical
structure. Strength in this study is
operationalized as a correct answer to all
the three item formats of MC, editing, and
translation, measuring the same structure
but in different formats.
The present study
Although many testing researchers have
attempted to measure various aspects of
language ability, measurement of
grammatical knowledge has largely been
under-theorized (Purpura, 2004). Purpura
(2004) reminds that there is a glaring
paucity of information on assessing
grammar and research on the validity of
inferences made upon them; more
specifically he deplores lack of consensus
on:
(1) what constitutes
grammatical knowledge, (2)
what type of assessment tasks
might best allow teachers and
testers to infer that grammatical
knowledge has been acquired
and (3) how to design tasks that
elicit grammatical knowledge
from students for some specific
assessment purpose, while at the
same time providing reliable
and valid measures of
performance. (p. 4)
While lack of research may be due to a
change of trend towards more integrative,
performance based assessment, lack of
adequate research about different
dimensions of grammatical knowledge,
especially in diagnostic and placement
language tests, could lead to
underrepresentation of the construct and
threaten the validity of the inferences
made based on those tests. As a result,
grammatical assessment studies that have
imitated the specifications of grammar
section of pre-2005 TOEFL with only MC
type item formats may have risked
construct underrepresentation, in case they
have made claims relating to a test-taker's
profile of weaknesses and strengths with
regard to knowledge of grammar.
Purpura (2004) discusses how his
framework could be the basis for
designing assessment tasks ranging from
selected response to extended production.
With regards to diagnostic tests, he
observes that learning-oriented assessment
of grammar might include cloze, selected-
response, limited-production and all sorts
of extended-production tasks. However,
the potential of different item formats to
provide useful information and what may
constitute useful information for whom
and why is not dealt with extensively in
his book. In addition he has not discussed
how development of grammar of learners
at different levels of proficiency is
different.
A more comprehensive, theoretical model
of grammatical knowledge, such as the
one used in this study, which includes
components of range, complexity, and
strength of grammatical knowledge, has
not been represented in the underlying
constructs of tests measuring grammatical
knowledge. This study attempts to explore
the relationship among different
dimensions of grammatical knowledge and
whether and how grammatical knowledge
develops along those dimensions, hence
the research questions:
1) Does the test of complexity, range,
and strength of grammatical
knowledge produce reliable
scores?
2) Which of the predictors of
complexity, range, and strength of
grammatical knowledge can best
predict the knowledge of EFL
students with lower overall
grammatical knowledge?
3) Which of the predictors of
complexity, range, and strength of
grammatical knowledge can best
predict the knowledge of EFL
students with higher overall
grammatical knowledge?
Method
Participants
The participants of the main study were
250 male and female non-English major
EFL students studying English at various
English institutes and/or universities in
Iran. Judging by the class levels and the
estimation of their teachers, their
proficiency level ranged between
elementary to upper intermediate. After a
preliminary screening and scoring of the
test papers, some participants were
excluded from the final analysis for partial
completion of the test. Thus, data from 92
participants was discarded and data of 158
participants remained for the main
analysis.
Instruments
Test of grammatical knowledge
The test included a number of grammatical
categories from a list similar to DIALANG
project (Alderson, 2005). The
specifications for the test were prepared
following the model suggested in
Davidson and Lynch (2002). They
included detailed information about how
three types of items (Multiple choice,
editing, and translation) should be written.
Further, they provided several sample
items and indicated how the test was
supposed to be assembled and
administered. Three item types were
included on the test on the grounds that
although multiple-choice questions are
commonly employed for testing language
knowledge, in the recently proposed
systematic approach to item writing (Shin,
2012) and elsewhere in the literature (e.g.,
Brown & Hudson, 1998) using various
item formats are more desirable. The
reason, as Buck (2001) argues, is that “ all
items have their particular strengths and
weaknesses and tend to engage different
skills. By using a variety of different task
types, the test is far more likely to provide
a balanced assessment” (153).
The following are examples of items on
the test:
Translation from L1 to L2
ماشینی که ما خریدیم سفید است.
/The car that we bought is white./
Editing the sentence by changing a word
or phrase.
The letter it Jack received was from the
company.
MC
This coat, _______ that man sold me, is
too big.
a. whom b. who c. which d. whose
In order to review the test specifications
and evaluate the quality of the items, two
Ph.D. holders with expertise in language
testing and five native and non-native
speakers of English who were also TESOL
students and had the experience of
teaching grammar were recruited. The
feedback and comments from the
reviewers were voice recorded,
transcribed, analyzed. Afterwards, the
specifications and the test items were
revised and the grammar test was
prepared. The test, which comprised of 89
items, was piloted twice on two groups of
participants (N = 40, N = 33), who were
similar to the ones in the main study.
To analyze the item statistics, classical test
theory (CTT) was employed. Based on the
results, the faulty items were revised and
piloted once again. Consequently, the
main test was prepared and administered
to 158 participants. Rasch Model was used
to analyze and calibrate the items on the
main test.
Data Collection and scoring
The data was collected over five months in
several administrations of the test. To
ensure consistency between different test
administrations, a set of guidelines were
developed and the proctors were instructed
to follow uniform procedures. Each
administration of the test took about one
hour and a half and the participants who
finished the test before the end of the exam
time were allowed to leave the session.
In the present study, range of grammatical
knowledge is calculated when a test-taker
provides a correct response to at least one
of the three items having the same spec but
different item formats. Thus, tallying the
number of grammatical categories a test-taker knows yields the range score.
Strength of grammatical knowledge is
calculated by tallying the number of
grammatical structures when the test-taker
provides correct answers to all the three
items, having the same spec but different
formats. In other words, a test-taker who
has answered all the three formats
pertaining to a particular spec correctly is
deemed to have a strong command of that
grammatical structure. Complexity of
grammatical knowledge is calculated by
tallying the number of correct answers on
the complex items on the test, regardless
of their format or spec.
Results and discussion
Reliability of the test. As CTT statistics
were used to analyze two pilot studies, it
was expected that the items on the test
already met the standards of CTT. The
mean test score was 55.16 and the standard
deviation was 16.32. Further, the measure
enjoys a high reliability index (Cronbach's
Alpha = 0.95). In order to enrich the
validity argument, CTT and Rasch Model
statistical procedures were employed for
ascertaining quality of items that were
included in the test. Data was analyzed
both by Winsteps version 3.70.0, a Rasch
model based software, developed by
Linacre (2010). As regards the sample
size, DeMars (2010) suggests that studies
with sample sizes as small as 100 or 200
can use Rasch Model. According to
Linacre (1994) a sample size of 150 would
yield item calibrations that are stable
within logits in 99% confidence interval in
Rasch model.
As regards the assumption of
unidimensionality, as Table 1 indicates,
Rasch dimension explains 37.6% of the
variance in the data from the performance
of the participants on the test and the
largest secondary dimension explains only
4.1% of the variance. As the variance
explained by the second dimension is
negligible in comparison to the variance
explained by the first dimension, the
measure could be considered
unidimensional.
statistics at the approximate range of .75 to
1.3 are acceptable, according to Lincare
(2010) they should ideally be in the range
of .7 to 1.3. However, Linacre (2002)
suggests that items with fit statistics as low
as .5 or as high as 1.5 are still reasonably
productive for practical measurement
purposes. Analysis for fit statistics
indicated that the majority of the items
were within the range of .75 to 1.3. No
item had a misfit (i.e., an infit index above
1.3).
According to McNammara (1996), items
with an infit above of 1.3 are either poorly
written items or do not measure the same
construct as the rest of items. Therefore,
all the items on the test can be considered
to be well written and all seem to test the
same construct. There were two items,
with infit indexes of .68, and two with infit
of .70 and .74. These items’ infit is only
slightly below .75, that means the
information provided by these items could
be gained by data from other items; in
other words, overfiting items are
redundant (McNamara, 1996). Therefore,
the more redundant the items the greater
their distance from .75. However, the few
overfiting items on the test seem to be only
marginally below McNammara’s (1996)
criterion and almost at the range proposed
by Linacer (2010) and certainly reasonably
productive for measurement purposes as
suggested by Linacre (2002).
As regards outfit statistics, there were
seven marginally overfiting items and one
underfitting item. Outfit statistics show the
sensitivity of the items to the ability of the
test-taker, that is, the greater the difference
between the ability level and the difficulty
of the item the greater the deviation of
outfit from one. However, according to
Linacre (2002), misfitting items with
regard to their outfit statistics are less of a
threat to measurement: “This is more
sensitive to responses to items with
difficulty far from a person, and vice-versa. For example, outfit reports overfit
for imputed responses, underfit for lucky
guesses and careless mistakes” (Linacre,
2002 p.878). Since all of these items
showed particularly good infit statistics
and the outfits were well within 0.5 to 1.5
range of Linacre (2002) they were
considered productive for measurement
purposes.
Regarding the assumption of local
independence, 5 pairs of items were
identified as candidates for dependency
(i.e., one of the items could be redundant).
Further scrutiny showed that the
correlation between two pairs on the list
was moderate (r = .44) and the rest of
correlations were weak (i.e., less than r =
0.35). Therefore, it was safe to assume that
the whole test, to a very large extent, met
the criteria of local independence of items.
Data analysis showed that the items
covered a wide range of difficulty from -4.3 to +3.7 logits. However, test
information curve indicated that the test
was more reliably informative for the
ability levels approximately between -2 to
+2 logits, that is, the ability estimates for
test-takers at the extreme levels of ability
had a larger margin of error due to the fact
that there were not as many very difficult
or very easy items on the test. Since the
majority of the test-takers self-identified
themselves as being lower intermediate,
the test reported mostly weakness for a
beginner EFL test-taker and mostly
strength for an upper-intermediate, while it
reported a balanced profile of weaknesses
and strengths for lower-intermediate
students. This is entirely congruent with
the expectations based on which the test
was designed.
The discussion above addresses a validity
concern reflected in first question of this
research about the reliability of the scores.
It appears that the measure can be
considered reasonably reliable and the
majority of items on the test meet the
statistical criteria specified in the literature
on CTT and Rasch model.
Investigating the consistency of
relationship
To examine the relationship among
complexity, range, and strength and how
the relationship among these dimensions
changes as learner’s grammatical
knowledge develops, two sets of analyses
were conducted after dividing the
participants into a higher (HG) and a
lower group (LG). The total scores for the
participants on each subset of translation,
editing, and MC was calculated in
standardized z scores and aggregated to
create a composite total score for the test.
The students with at least half a standard
deviation above the mean were labeled as
High and those with more than half a
standard deviation below the mean were
considered Low. Table 2 shows the
descriptive statistics for the dimensions of
Grammatical Knowledge (GK) for the
Higher and the Lower group of EFL
learners. In order to discover the extent to
which the three predictors of complexity,
range, and strength were related to the
criterion (i.e., GK) and to each other, a
correlation analysis was followed by a
regression analysis. Theoretically, all of
the predictors were expected to be
correlated with the criterion, because
regression analysis is based on the
correlations among variables. Therefore,
first, the relationship among all the
predictors of GK and the actual GK of the
participants was studied separately for the
lower and higher group using Pearson
product-moment correlation.
As the results presented in Table 3
suggest, the reason for calculating
correlations was to examine the possibility
of the differential effect of developmental
stage of grammatical knowledge on the
strength of the relationships. It was found
that all correlations were significant at p <
0.05 for the LG.For the HG all the
correlations were significant except for
two: between GK and range (r = .26, p >
0.05) and between range and strength (r =
0.12, p > 0.05). The correlation values
between the other variables revealed
significant and rather strong relationships
across the higher and lower groups. These
results suggest that for lower proficiency
learners all three dimensions of knowledge
show moderate to strong relationship
amongst themselves and with GK.
However, at higher levels, the importance
of range seems to have decreased and
other dimensions could possibly better
describe grammatical knowledge of more
advanced students.
In order to provide more evidence for this
argument, a stepwise multiple regression
model was developed to identify the most
economical model to describe the state of
grammatical knowledge at two high and
low proficiency levels.
In other words, the results of the
correlation analysis (Table 3) indicated
that most of variables were highly
correlated. This was the motivation to
consider the possibility of identifying a
smaller set of variables that would be as
efficient as the total set of factors. To find
the extent to which complexity, range, and
strength dimensions of grammatical
knowledge can explain the greatest
amount of variance in EFL students’
knowledge of grammar, stepwise multiple
linear regression analysis, as one of the
strongest statistical analyses used for
predictive purposes (Brace, Kemp &
Sneglar, 2000), was conducted separately
for each group to investigate whether the
model of GK in two groups is different.
This model was employed to examine the
relationship among the criterion (i.e.,
participants’ actual GK measured by their
responses to the instrument) and the three
predictors and to identify a smaller set of
predictors of grammatical knowledge that
can predict the same amount of variation
among language users of lower and higher
proficiency.
The result of the analysis for the lower
group
Results of stepwise regression for the LG
are presented in Table 4. Examination of
the results indicated that out of seven
theoretically possible models, three
models were more plausible. Notably,
among three models, the first one included
only dimension of range.
More specifically range with the adjusted
R2 = 0.721 was the first suggested model,
that is, this predictor alone explained
almost 72 % of the variance in the LG’s
knowledge of grammar (Table 4). Adding
a second predictor (i.e., strength) only
added 8.1% to the explanation of variance
(R2change = 0.081). Adding a third
predictor (i.e., complexity, in the third
model, improved it by 3.7 % [R
2
change =
0.037]). These findings seems to endorse
the argument that for lower level learners
the role of range of grammatical
knowledge is more prominent, in
comparison to the HG (discussed below),
as their interlanguage has not become
complex and they still may have partial
control on their knowledge and cannot
correctly employ it in different contextual
and cognitive settings (i.e., they cannot
demonstrate the similar amount of control
in attempting different item formats
measuring the same structure).
Although range alone explained 72% of
variation in predicting grammatical
knowledge and the contribution of the
other variables was rather small, these
Models were significant at P = 0.000 level
(Table 5). By calculating the coefficients
of stepwise regression, it was found that
the assumption of lack of collinearity
among predictors is met. When some of
the independent variables are entirely
predicted by the other variables,
collinearity exists. For this reason,
independent variables are examined for
tolerance value, a statistic that indicates
collinearity among predictors. This value
ranges from 0 to 1; the closer the value is
to 0, the stronger the relationship is
between the predictor in question and the
rest of the predictors. In fact, existence of
the predictors whose tolerance value is
below 0.001 is a cause for concern. If the
tolerance value of a predictor is below
0.001, it should be removed from
the analysis. Moreover, the threat of
collinearity among the predictors was not
alarming, that is, tolerance values were all
above 0.001 which means that the
relationships are not collinear, hence the
dependability of the regression.
The result of the analysis for the higher
group
The stepwise regression preformed for the
HG point to the inclusion of two plausible
models out of seven conceivable ones.
Unlike the findings for the lower group,
range was excluded from either of the
suggested models. Table 6 provides the
information regarding the explanatory
power of each of the two models: the first
model had a large value of adjusted R2 =
0.77, meaning the predictor of strength
alone could explain 77 % of variability in
the criterion. When complexity was added
to the second model, it improved the
previous one by 7.9% (R2= 0.079).
As it was the case for the results obtained
for the LG, both models proposed for the
HG were also significant at p < 0.000
(Table 7). Moreover, there was no
collinearity among the predictors since all
the values for the collinearity tolerance
were above the critical point of 0.001.
One-predictor models
By comparing the results of stepwise
regression for the higher and the lower
group, we can decide on the set of
predictors that are more informative for
assessment purposes. This comparison
reveals that the one-predictor models may
not be the most informative ones for
assessing grammatical knowledge of high
and low ability EFL learners.
Nevertheless, data suggests that the best
predictor of grammatical knowledge of for
LG is range and for the HG is Strength.
Two-predictor models
A closer look at the two-predictor models
shows that while for the LG range and
strength have been included in the model,
for the HG strength and complexity have
been selected. It can be inferred that
strength can be the common dimension for
both of the two-predictor models. Further,
a two-predictor model can best explain the
variability in HG learner’s knowledge; this
model explains 7.9% more of variance in
GK of high participants respectively.
According to this finding, after strength,
the dimension of complexity seems to be
the second important predictor of
grammatical knowledge for the HG. This
might be due to the fact that complex
structures are usually the ones that are
more difficult; hence they are acquired in
later stages. In other words, the reason for
the identification of variables for high and
low might be related to proficiency level
of the participants; it can be argued that
proficiency and exposure to language may
play a determining role in the development
of a strong and complex grammar.
Three-predictor models
The only three-predictor model was
proposed for the LG and included range,
strength, and complexity. The addition of
complexity to the two-parameter model
improved it by 3.7% and provided the
model with the most explanatory power
among the three. This raises some
questions about the role of complexity in
the grammatical knowledge of the EFL
learners with lower proficiency: whether it
is economical to develop a set of items
measuring complexity for lower level
learners? However, such a decision is
related to the purpose of the test. For a
diagnostic test of grammar, inclusion of all
the three dimensions seems necessary as it
allows comparison between stages of
development in learner’s knowledge.
For the LG the role of the range dimension
seems to be more prominent; they also
show degrees of ability with regard to
other dimensions, which helps with
portraying a more comprehensive picture
of their weaknesses and strengths.
Likewise, for learners of higher ability,
dimension of range may not be the most
informative. Nevertheless, it depicts that
the learner has made a great deal of
progress along that dimension; this piece
of information can be quite motivating if
provided to the learners in an assessment
for learning context.
Conclusions
This paper analyzed a test of grammatical
knowledge, the specifications of which
included three item formats, to measure
three the dimensions of grammatical
knowledge. It found that all the three
dimensions tended to correlate with each
other and could be used to describe the
state of EFL learners’ grammatical
knowledge. However, for lower levels the
role of the range dimension seemed more
conspicuous and for learners with higher
proficiency the role of strength was more
evident.
Nevertheless, all the dimensions could be
used to lend meaning to the scores by
describing the state of development of
learners. Further, since the dimension of
range is operationalized as correctly
answering one item in a set of item
formats measuring the same structure, and
strength is operationalized as measuring
the same structure via different item
formats, it may be justified to use different
item formats for measuring the same
structure because it can increase chances
of detection of existence of knowledge by
at least answering one of three items
correctly.
It can also guarantee that a learner has a
stronger knowledge of a certain structure
and that their answers are not based on
chance. The findings imply that a
multidimensional model of grammar can
help with inferences about test-takers'
grammatical knowledge for a variety of
assessment purposes. Specification of
grammar tests can consider the potential of
incorporating a wider variety of item
formats to enable a more comprehensive
assessment of the grammatical knowledge
of EFL students.
The results of such an assessment can be
used to provide more detailed feedback to
students, which is a requirement in the
context of assessment for learning
(Assessment Reform group 2002, Black,
Harrison, Lee, Marshall, & Wiliam 2004;
Wiliam, 2011) and advocated by numerous
language testers (e.g., Spolsky 1990;
Shohamy 1992; Huhta 2008; Jang 2009).
Further research is suggested to employ
the framework for grammatical knowledge
proposed in this study as it can offer a
means of measuring the increase in
complexity, range, and strength of second
language grammar as learners’ language
proficiency develops. Arguably, it has the
potential to help researchers to
systematically study development of
grammar along different dimensions in
various sociolinguistic contexts such as
EFL or ESL. Moreover, learning gains can
be measured where the focus of instruction
is not grammar— for example, grammar of
the learners can be measured after a
reading course to find any improvements
in the complexity, range, and strength of
grammatical knowledge.
