UIUC Quant Brownbag

Several statistics have been proposed to detect Differential Item Functioning (DIF) and quantify its magnitude. However, not all DIF statistics quantify the same population parameters and, therefore, cannot be interpreted interchangeably. We focus on deriving principled guidelines for the interpretation of the non-compensatory differential item functioning (NCDIF) parameter as a DIF effect size measure. This parameter quantifies the expected (average) effect of the possible differences between item parameters on the (squared differences of the) item score for the focal population of examinees. We first investigate in which situations the Delta Mantel-Haenszel (ΔMH) is comparable to NCDIF, so that the ETS cutoff points can serve meaningfully as a benchmark. Then, we examine the parameter’s behavior under various conditions of uniform and nonuniform DIF, as well as the effect that the distribution of the focal group exerts on its magnitude. Lastly, using one of the estimators of the NCDIF parameter, we evaluate the accuracy of the derived classification rules for NCDIF and identify an approximate bias correction for this estimator. Overall, these results provide useful guidelines for interpreting the magnitude of NCDIF that are consistent with its specific nature and improve the alignment of DIF classifications with the magnitude of the NCDIF parameter.