Introduction
Differential item functioning (DIF) occurs
when equally knowledgeable individuals
from different subgroups are of different
likelihood of correctly answering (or
endorsing) an item (Shepard, Camilli, &
Averill, 1981). DIF is a necessary but not
sufficient condition for bias (Zumbo,
1999). In other words, bias only exists
when the groups illegitimately differ in
their performance on an item. Because
items indicating DIF can function as a
threat to the validity of a test, DIF analysis
has become an essential step in the
validation of a test (Camilli & Shepard,
1994; Zumbo, 2007; Zumbo, & Rupp,
2004) especially in high-stakes testing
situations (Pae & Park, 2006).
The present study was an attempt to focus
on several hypothetical factors that may
cause DIF on a reading comprehension
test. As pinpointed by McNamara and
Roever, (2006), the literature lacks
research on social or contextual sources of
DIF. As such, several hypothetical social
sources of DIF were also brought into
focus in this study.
Background
Many studies have been conducted on
DIF. The majority of these studies have
focused on gender or ethnicity and only
some have shed light on other sources of
DIF. Among the studies conducted,
mention could be made of those focusing
on gender (e.g., Li & Suen, 2012; Pae,
2012; Ryan & Bachman, 1992; Takala &
Kaftandjieva, 2000), ethnicity (e.g.,
Hammer, Pennock-Roman, Rzasa, &
Tomblin, 2002; Stoneberg, 2004),
language background (e.g. Harding, 2011;
Li & Suen, 2012), topic familiarity or
content knowledge (Pae, 2004), and field
of study (Barati, Ketabi, & Ahmadi, 2006;
Näsström (2004)).
Such studies have indicated that for
example east Asian test takers outperform
test takers from European-language
backgrounds on Michigan English
Language Assessment Battery (Li & Suen,
2012); that females tend to outperform
males in tests of verbal and written
abilities, especially if constructed response
items are included (Willingham & Cole,
1997); they also outperform males on
aesthetics, human affairs, and
contextualized reading items (Carlton &
Harris, 1992) on the mood, impression, or
tone items of a reading passage (Pae,
2012), on items related to human relations,
human rights, aesthetic and on items
referring to stereotypical female activities
(O'Neill & McPeek, 1993; Sadker &
Sadker, 1994), whereas males outperform
females on antonyms, and analogies
(Carlton & Harris, 1992), on reading items
of inferencing type (Carlton & Harris,
1992; Lawrence & Curley, 1989), on
items involving visualization and items
calling upon knowledge and experiences
acquired outside of school (Hamilton &
Snow, 1998), and on items related to
science and on items referring to
stereotypical male activities (O'Neill &
McPeek, 1993; Sadker & Sadker, 1994).
The results of studies on DIF have
sometimes been contradictory and far from
being conclusive. Furthermore, the studies
have been limited to few sources of DIF
and lack of research is felt on
social/contextual sources of DIF
(McNamara & Roever, 2006). Very few
studies have focused on how social factors
may create DIF on different tests. The
present study tried to fill this research gap
by focusing on a number of social factors
that may explain DIF on a reading
comprehension test. It also tried to shed
more light on a number of non-social
factors mentioned as sources of DIF in the
literature. As such, the following questions
were put forward in the current study:
1. Can text familiarity, gender,
interest, guessing, and contextual
variables (location, income, and
educational status) function as
sources of DIF on EFL reading
comprehension items?
2. To what extent do the results of
DIF detection methods (IRT and
LR) converge?
3. Can item-level differential
performance manifest itself at the
scale-level analysis?
Method
Design of the study
Unlike the DIF studies in the literature
which select an exploratory approach
toward the analysis of DIF sources, the
present study benefited from a mixed
exploratory-confirmatory approach in this
regard; that is, in dealing with social
sources of DIF, an exploratory approach
was adopted in which the test was
administered to the test takers who were
divided into groups based on the social
variables (location, income, & educational
level) and then their performance on the
test was analyzed for instances of DIF.
This approach is the one adopted in DIF
studies in the literature. The present study,
however, adopted a confirmatory approach
in studying sources of DIF mentioned in
the literature; that is, instead of running
the test and then searching for the probable
sources of DIF, the study, following
McNamara and Roever's (2006)
suggestion, selected the most cited sources
of DIF in the literature and intentionally
included them in the reading test to see
whether they really manifest themselves in
the DIF study. The assumption was that if
such claimed variables are really sources
of DIF, then an item which embraces such
variables should necessarily indicate DIF.
A counterargument for this will hold that
such sources may be context and test
specific and hence do not necessarily
manifest themselves in DIF analysis of
any test in any context.
Participants
The participants of this study included 203
English as a Foreign Language (EFL)
learners (93 males and 110 females)
selected from among around 300
intermediate-level students at Shiraz
University Language Center (SULC) in
the spring semester in 2011. Only those
participants who based on their final
scores in previous semesters were roughly
identified as intermediate were selected for
the study. Due to the administrative
limitations, only the participants’ scores
on the regular tests administered by the
institute at the end of each term were taken
into account and no specific proficiency
test was employed. However, it should be
mentioned that this was quite enough for
the purpose of the present study because
the DIF software (BILOG MG)
automatically estimates the participants’
proficiency based on the test used for DIF
analysis and does not need a separate test
to be used for proficiency. The reason why
we tried to roughly homogenize the
participants based on their proficiency was
only to make sure that the test used for
DIF analysis was not too far from their
level of proficiency and hence more
realistic answers would be given by the
participants; otherwise, DIF studies rely
on a single test.
Instrumentation
Reading comprehension test
A reading comprehension test was
developed for the purpose of the study.
The passages of the reading test were
originated from Anderson’s (2007, 2008)
books entitled “ACTIVE Skills for
Reading: Book 2” and “ACTIVE Skills for
Reading: Book 3”. The books were
selected from among a series of four books
to fit the intermediate proficiency level.
Furthermore, as these books are not
usually taught in the Iranian language
institutes or universities, there was a very
low chance for the students to have seen
the texts before. Attempt was made to
include the hypothetical sources of DIF.
To this end, first of all, on the basis of
gender-based familiarity with text topic, 6
short passages were chosen. Following the
literature (Bugel & Buunk, 1996;
Newman, Groom, Handelman, &
Pennebaker, 2008; O’Neill & McPeek,
1993) some topics were hypothesized to
be gender-stereotypical; that is, such
topics as practical affairs, money,
mechanical tools, soccer, occupation were
considered male favorable, whereas topics
related to humanities, social sciences,
aesthetics and human relations, wedding,
and household chores were considered
female favorable. Therefore, two passages
were taken as male-friendly (What does a
Million Dollars Buy? & Meet Freddy Adu,
Soccer Sensation), two passages as
female-friendly (Wedding Customs & The
Right Job for Your Personality), and two
others as gender-neutral (Are Human
Beings Getting Smarter? & A Different
Kind of Spring Break).
To develop the test, each of the 6 passages
was followed by five commonly used
questions in reading comprehension tests
(Broukal, 2007). They were items asking
for details (facts), reference to a word or
phrase, vocabulary knowledge, main idea
(theme), and inference (logical
conclusion). Therefore, the final test was a
30-item reading comprehension test. Two
faculty members who were experts in L2
language testing and had worked on DIF
checked the reading texts and questions in
line with the suggestions made in the
literature to make sure that the final test
served the purpose of the study. The
reliability of the test was also estimated
through KR21which turned out to be .52.
The moderate reliability observed was not
far from logic as due to the confirmatory
nature of the study the sources of error
variance (DIF sources) were intentionally
included in the test which could noticeably
decrease the reliable variance proportion
in the measurement equation.
DIF questionnaire
The second instrument was a very short
questionnaire the first part of which was a
set of bio-data questions asking the test
takers to provide information about their
gender, residency neighborhood, families’
monthly income, and parents’ educational
level. The second part, however, was
attached to the end of each passage and
comprised these questions:
a) How familiar was the text topic or
content to you?
b) How much were you interested in
the reading topic?
c) Which item did you answer by
guessing?
The purpose of this section was to collect
information on the participants’ topic
interest, topic familiarity and guessing.
They were expected to indicate their level
of familiarity with and interest in the text
by selecting one of the five options very
much, much, to some extent, little, and
very little/ none.
Data collection procedure
The factors included in DIF analysis were
literature-based variables hypothesized to
contribute to test takers’ differential
performance on EFL reading tests. They
were gender, text familiarity, text interest,
guessing, location in two levels
(down=living in less expensive
neighborhoods with a housing price of
$700-$1500 per squared meter; up=living
in the expensive neighborhoods with a
housing price of $2200-$2800 per squared
meter), income in two levels (low=less
than $1000 per month, high= more than
$1000 per month), and educational level in
two levels (academically educated,
academically uneducated).
Data analysis
It is recommended that more than one
method of DIF analysis be employed in
DIF studies to come to more dependable
results (Camilli, 2006; Camilli & Shepard,
1994; Pae, 2012, Uiterwijk & Vallen,
2005). In line with this suggestion, the
present study employed two methods: (a) a
classical method: logistic regression (LR),
(b) 1-p item response theory (IRT) model.
This could add to the dependability of the
results and made it possible to compare the
degree of correspondence between the
results of the two methods.
Results and discussion
DIF analysis based on Logistic Regression
Overall, about 47% (14 items) of the
whole test displayed DIF through the use
of LR. Only three of these items indicated
large DIF and the majority indicated
moderate DIF based on the criteria
recommended by Hidalgo and Lopez-Pina
(2004); that is, negligible DIF: ΔR
2
< 0.13,
moderate DIF: 0.13 ΔR
2
0.26), and
large DIF: ΔR
2
> 0.26. In what follows the
results of DIF analysis based on LR are
presented in detail for each of the
hypothetical sources.
DIF based on gender
As previously mentioned, the reading texts
were selected with an eye toward the
gender differences in topic familiarity of a
reading test (e.g., Brantmeier, 2003; Pae,
2012). Overall, five items (17% of the
whole test) were flagged for DIF based on
gender. The results of LR indicated that in
the female-friendly passages 13.33% of
the items; that is, four items (two inference
items, one vocabulary item, & one
reference item, ) favored females. In the
male-friendly passages only a single item
(3.33% of the items) which was a
vocabulary item favored males over
females; and in the neutral passages no
item indicated DIF in favor of a gender.
The results of DIF overall, functioned in
line with the literature on gender-based
DIF that there exists DIF or differential
performance on gendered texts (e.g.,
Carlton & Harris, 1992; Lawrence &
Curley, 1989). In other words, the females
got advantage of the female-friendly texts
and the males were favored by male-friendly texts. Neither group was reported
to gain benefit from the gender-neutral
texts. More specifically, the results
partially echoed those of the study by
Newman et al. (2008) that most of the
differences between men and women are
related to the application of function
words (e.g., reference-type item in the
present study) and lexical words (e.g.,
vocabulary-type items in the present
study).
DIF based on the familiarity with text
topic or content
One part of familiarity was discussed
under the rubric of gender-based DIF.
However, mention was made that that
particular analysis did not find any DIF for
gender-neutral texts. Thus, familiarity was
also considered separately with the
intention of its effect on the items
dispensed with the gender influence. The
idea was that although two passages were
claimed to be female-friendly, two male-friendly and two neutral based on the
literature, still individual differences were
possible in terms of their familiarity with
the texts regardless of their gender. For
example, saying that a text is more female-friendly based on the topic does not
eliminate the chance that some males
could be familiar with such texts. As such,
the study focused on familiarity with the
text topic as a factor for differential
performance regardless of the gender.
Therefore, the participants were divided
into three groups based on their familiarity
with each text (highly-familiar,
moderately-familiar, and slightly-familiar). This was done based on the
participants’ answers to the questionnaire
items indicating their level of familiarity
with each text. Therefore, those selecting
very much and much options were
considered as the highly-familiar group,
those selecting to some extent as the
moderately-familiar and those selecting
little and very little as the slightly-familiar
group. Then DIF analysis was performed.
Unlike the results of DIF based on gender,
the results of DIF based on familiarity
found some traces of DIF in the gender-neutral texts. Overall, topic familiarity was
found to be the source of DIF for three
items, two of which were related to the
gender-neutral texts. Therefore, the results
were not in line with the results of gender-based DIF. It is therefore manifest that,
with three items (10% of the whole test)
indicating DIF, the familiarity was not
supported to have a leading role in DIF
occurrence since only one inference-type
item (3.33% of the whole test) favored the
moderately-familiar group and two items
of details- and vocabulary-type (6.66% of
the whole test) benefited the slightly-familiar group.
This finding is against the literature (e.g.,
Sadker & Sadker, 1994) and does not
support the assumption that higher
familiarity may lead to higher chances of
endorsing an item correctly. Therefore,
taken with the results of the gender-based
DIF together, it seems that gender
(focusing on gender-friendly or gender-stereotypical texts) will provide us with a
better explanation of DIF. In other words,
familiarity is better to be considered
together with the gender effect to explain
the DIF on a test.
DIF based on the interest in text topic
To avoid the possible bias, test developers
may want to select unfamiliar text topics,
but by doing so the texts may not be so
interesting and relevant to the test taker
(Bachman, 1990). The challenging task of
a test writer is to avoid either extreme, i.e.
to develop topics that are very general and
innocuous and at the same time interesting
and relevant. Familiarity and interest pose
difficulty in the design, development, and
analysis of reading tests. Thus, we can
hypothesize that the familiar texts would
be interesting to a particular group often to
the detriment of the other. To study the
effect of interest as a source of DIF, like
what we did for the familiarity, at first we
divided the participants into three groups
based on their interest in each text; that is,
highly-interested, moderately-interested,
and slightly-interested. This was done for
each text separately and then DIF was
checked for each item. Contrary to our
expectations, the notion of interest in text
topic did not turn out to be an influential
source of DIF because only one item
(3.33% of the whole items) which was a
vocabulary item favored the highly-interested group over the others.
DIF based on guessing
It goes without saying that guessing is a
hypothetical source of DIF in an MC test.
In this study, the test takers were asked to
inform the researcher whether for each
passage with five items, they answered a
particular item by chance. That is, they
were asked to say if they guessed at a
particular item or not based on which they
were divided into three groups of low,
mid, and high guessers. The results of DIF
analysis flagged five items (16.66% of the
whole test) based on guessing which
favored low-guessers. In other words, the
results indicated that being a member of
the non-guesser (or low-guesser) group
increased the probability of endorsing
these items correctly. It seems that those
who had guessed in this study were mostly
wild guessers and thus their guessing did
not help them. Guessing, therefore, was
not supported to play a significant role in
DIF results as low/non-guessers were
more successful.
DIF based on location
Only one item was found to support the
idea that the test takers' neighborhood
(location) can function as a source of DIF.
This item which was a vocabulary item
functioned to the favor of those living in
rich neighborhoods. Thus, living in
different neighborhoods (and by
generalization in different cultural
settings) was corroborated by a single item
(3.33 % of the whole test) to function
significantly in the test takers’ differential
performance on the vocabulary-type
reading item. This finding, though based
on a single item, supports Zumbo &
Gelin's (2005) idea that by ignoring socio-geographic differences one may lose the
whole picture of DIF.
DIF based on income
Some social groups, as a result of high
income, may have access to more high-quality educational opportunities which, in
turn, can lead to their gradually better
performance on language tests. Income, in
this study, emerged as a source of DIF in
two items. One of the items was an
inference-type item and the other one a
main-idea item. Both items indicated
much higher probability of endorsing the
items correctly for those from higher-income families. Thus, income was
supported only by two items (6.66% of the
whole test) to distinguish between test
takers in performing on the main idea- and
inference-type reading items.
DIF based on the educational level
Unlike the location and income variables
that reverberate the community-level
contextual factors, the family (or parental)
educational level is less susceptible to the
community at large and compares test
takers at the individual level (Zumbo &
Gelin, 2005). To take note of this
contextual factor, the test takers were
asked to report whether their families or
parents were academically educated or
uneducated. The results of DIF analysis in
this regard indicated that only one item
(inference-type item) presented this
contextual factor as a significant predictor
of DIF in favor of the test takers coming
from educated families. Thus, a small
portion of the test (3.33%) supported the
advantage of the educated group.
IRT analysis
Overall, 33.33% (10 items) of the whole
test displayed DIF through the use of IRT.
The results of IRT analysis for gender and
familiarity variables did not show any
meaningful and statistically significant
DIF. As such, in what follows the
remaining DIF sources (interest, guessing,
location, income, & educational level) are
taken into consideration. In each section
the difficulty differences between the
contrasting groups, called group threshold
differences, and the standard error of
measurement are provided for each item.
Items for which the threshold difference is
roughly twice (1.96) or more the size of
the standard error display DIF at the p =
.05 level (Thissen, Steinberg, & Wainer,
1993). The only parameter to be attended
to in this program was the difficulty value
(b) and therefore the lower threshold value
for a particular group means that the item
was easier for them. That is, the negative
or positive direction of the threshold
differences indicates which particular
subgroup was favored.
DIF based on the interest in text topic
The IRT analysis revealed the threshold
differences output for the interest in texts.
As Table 1 indicates (see Appendix), items
1, 3, 5, 20, and 27 displayed DIF. The
table indicates that in both mid-low and
high-low comparisons, Item 1 functioned
as a DIF item (Thissen et al., 1993). The
negative threshold values in both cases
indicate that the item (details type) was
more difficult for the low group than the
mid and high groups. Thus, those with
high and moderate interest in Text 1 were
favored by item 1. For item 3 the results
demonstrate that in both cases (high-low,
high-mid) the item was more difficult for
the high group. Thus, it is concluded that
the low and mid groups were favored by
Item 3. By the same token, the results
indicate that for Item 5, the low and mid
groups benefited from the item. In item 20,
the negative threshold difference (-0.792)
indicates that for the low group the item
was more difficult; and hence the high-interest group was favored by this
inference-type item. Finally, for item 27 it
is clear that the high group was favored
because their threshold was lower than
that of the mid group. The detection of
16.66 % of the whole items (five out of
30) as including DIF demonstrates a
greater proportion of interest effect in the
IRT analysis in comparison with the other
factors.
DIF based on guessing
The negative threshold differences for
items 11 and 14 and the positive difference
for item 15 indicate that all the high
guessers had more likelihood to get the
details- (11), main idea- (14), and
inference-type (15) items right. Thus,
guessing as a speculative source of DIF
indeed made the MC items function in
favor of the high guessers.
DIF based on location
Regarding the residential neighborhoods
of the respondents (up vs. down) only item
4 (3.33% of the whole test) revealed
significant differential performance
between the groups (1.417). Thus, the up
group was reported to take advantage of
this main idea-type item.
DIF based on income
Out of the 30 items, only one item (3.33%
of the whole test) displayed DIF and
supported the hypothetical function of
income in the IRT analysis. Only item 19
was flagged for DIF. The negative
threshold difference for this main idea-type item makes it clear that the low-income group found the item easier to
endorse. In fact, those test takers coming
from low-income families were favored by
this particular item. As you will see in the
DTF section, income was one of the
sources transferred from the item level to
the test level and had a proportional effect
on the test bias.
DIF based on educational level
The comparison between the difficulty
parameters of the educated- and
uneducated-family test takers indicated
that only item 5 (inference type) displayed
significant DIF. The positive threshold
difference reveals that upon answering
item 5, the educated group was favored
because for them the item was easier to
endorse. Therefore, only 3.33% of the
whole items supported the speculative
effect of the parental educational level on
differential performance of the test takers
on reading comprehension.
Comparing the LR and IRT results
The overall results of DIF analyses are
summarized in Table 2. The first column
represents the hypothetical DIF sources;
the second and the third illustrate DIF
identified through LR and IRT
respectively. The last column shows the
items identified by both methods and
therefore indicates the agreement between
the two methods.
As the above table indicates, only items 5
and 14 were identified as DIF by both DIF
detection methods. This indicates a low
level of correspondence between the
results of the two methods. The IRT
framework found no DIF related to the
gender and familiarity variables. With
respect to the interest variable both
techniques could flag some DIF items.
Through a small portion of
correspondence between the models,
interest as a source of DIF, which acts in a
balanced correspondence with familiarity
in reading comprehension (Bachman,
1990), was reported to function in a mixed
direction, i.e. leading some items to favor
interested test takers and some others to
benefit the uninterested ones. Guessing, as
a source of DIF in MC reading items was
identified by both methods but with
different functions. That is, the LR method
indicated that low guessers were more
successful, but the IRT model reported
conversely that the DIF items were in
favor of those test takers who guessed
highly at the answers, hence a total
divergence between the two methods. The
results of both methods for location, in a
complete correspondence, indicated that
those living in the up neighborhoods were
favored. In a similar vein, both
frameworks identified a single item (item
5), in a complete convergence, as
displaying education-oriented DIF in favor
of those coming from educated families.
However, for income-based DIF, the
methods stood in a sharp contrast. In fact,
unlike LR, IRT found that those coming
from the low-income families were
favored.
Let us interpret what we have come up
with so far in two parts. First, as the above
table depicts the two methods overlap in
the preemption of only two DIF items.
Why is there so much variance between
them? Why are their results not supportive
of each other? This finding could be
explained in terms of the sample size. It is
mentioned that for binary items at least
200 people per group is required for
LR to function well and smaller sample
sizes could deteriorate the results
(Zumbo, 1999). Further explanation of this
finding, may come from the significance
level used for DIF analysis. Usually IRT is
more accurate and does much better with a
smaller sample size than LR in flagging
items for DIF. So a Bonferroni correction
test could be useful to apply to the LR
analysis results for more careful analysis
(Alavi & Karami, 2010; Runnels, 2013;
Thompson, 2006). This can lead to a lower
number of items to be identified as DIF.
However, this procedure was not of much
help in the present study to resolve the
differences in the results of IRT and LR
DIF analyses because the differences did
not lie in the number of items rather in the
type of items.
However, overall the results of the two
methods should be considered together, as
there would be no method which is
foolproof with all samples and for all
contexts. Both methods, likewise,
indicated DIF related to the contextual
factors despite the low reputation of the
factors in the literature. Furthermore, the
presence of more DIF related to guessing
rendered it as a very important source of
DIF.
Second, of all the sources in the study
guessing caused the most DIF followed by
interest, gender, familiarity, and contextual
factors. The study centered around reading
comprehension tests and that is why the
emergence of guessing DIF for MC items,
interest in topics, and gender DIF for
gendered passages was not unconceivable.
Those sources that could be
manipulatively included in the research
yielded the results much more in keeping
with the literature than those factors over
which, in an ex post facto manner, the
researchers had no control, i.e. the
contextual (social) factors.
Analyzing differential test functioning
(DTF)
The presence of DIF items is considered to
be a threat to the test validity to the extent
that it manifests itself in the scale-level
differential performance (Pae & Park,
2006; Zumbo, 2003). The results of a
multiple-regression analysis indicated that
although the effect of the variables on the
total score was significant, only three
variables (gender, income, and interest)
were transferred from DIF to DTF level
and added to the complexity of the DTF
analysis (Pae and Park, 2006; Takala &
Kaftandjieva, 2000). The remaining DIF
sources, like the results of Zumbo’s (2003)
study, did not emerge in the scale-level
results.
Conclusion
The primary purpose of this study was to
move beyond the hypothetical
interpretation of DIF by following
McNamara and Roever's (2006)
suggestion, conducting an empirical
confirmatory study. Its outlook was
toward taking advantage of the
falsification philosophy in a way that rival
hypothetical DIF sources were proposed
and none was presumed to predict the why
of DIF occurrence. Two DIF sources were
included in the reading test and the others
were checked through a questionnaire
attached to the end of each subtest
(passage). The study could not support the
notion of familiarity with text topic as a
major source of DIF. For gender only the
LR model could detect some male- and
female-friendly DIF items. With the small
number of DIF items for the interest,
guessing, and income-related DIF, the
results of the two DIF detection methods
stood in a noticeable contrast to each
other. However, the methods were found
to be in a complete convergence in regard
to the location and family educational
status. Two other findings were related to
the item type and bias in the test. The
results indicated that gender could have an
impact on reference- and vocabulary-type
reading items. Furthermore, the DTF result
made it clear that of all the sources, only
three factors (gender, income, & interests)
were transferred to the test level.
The findings of the study overall may
warn language teachers and test
developers about failing to pay due
attention to DIF on reading comprehension
tests. This would be double problematic in
criterion-referenced tests where the
number and size of DIF items can greatly
affect the mastery and nonmastery
decisions made based on a single cutoff
score. Even more problematic would be
ignoring DIF on high stakes tests which
are usually used for gate-keeping
purposes. The stigma of failing in such
tests may distort the educational life (or
even the entire personal life) of the test
takers.
