A Simple Lesson in Item Validity: Qualitative Data Informing Quantitative Findings

Noreen Stuckless, University of Toronto, Toronto, Canada

Over the past several decades, there has been a substantial increase in the number, complexity and utilization of sophisticated statistical analyses. Even relative newcomers to research can produce findings with one of the many user-friendly statistical software programs.

Findings typically are judged on the basis of statistical significance, effect size, confidence intervals and other similar indices. However, in some cases, interpretations may be erroneous if they are based solely on quantitative evidence -- particularly if a quantitative-type item can have different meanings for diverse groups of participants. The following is an example of qualitative input from research participants, allowing researchers to have a better understanding of the participants' responses. It represents a simple lesson in item validity.

The example involves findings from a three-year longitudinal study on mediation and lawyer-based divorce conflict resolution, a project my colleague Desmond Ellis and I conducted. Participants, both wives and husbands, were asked to respond to the question, "Do you fear your ex-partner?" This question was asked at two times: in an interview conducted very shortly after the separation (Time I) and in an interview conducted one year after separation (Time II). In addition, participants were given the opportunity to expand qualitatively if they replied "yes" at either time.

The results indicated that at Time I, contrary to our expectations, an approximately equal number of wives and husbands reported "yes," when asked if they feared their ex-partner. At Time II the number of women who reported fearing their ex-husbands increased, while the number of men who reported fearing their ex-wives decreased sharply, to zero.

These quantitative results led our research team to engage in a variety of interesting speculations to explain both the unexpected symmetry in the initial responses at Time I and the disparity at Time II. However, an examination of the qualitative comments clarified the issue and made our postulated explanations less plausible.

The reasons given by the women and men for answering "yes" were very different. In the interviews conducted at Time I, shortly after separation, wives who answered "yes" to the dichotomous variable stated that they feared being beaten, locked away, run over and verbally abused on the telephone and at work. By contrast, with one exception (a husband who feared his wife's new boyfriend), husbands had a very different understanding of the concept of fearing their ex-partner. Men reported that they feared their ex-partners might humiliate them, take their children away and cause them to have a less favorable life style because of child-support payments lowering disposable income.

At Time II, in one-year post-separation interviews, only ex-wives answered "yes" when asked if they feared their ex-partner. Their qualitative comments included statements such as "He wasn't violent before, but now he threatens me," and "Increasingly I'm afraid to meet with him by myself."

This additional information explained both the original symmetry at separation and the disparate changes over time. For the men, fears were resolved by Time II interviews; whereas, for the women, new fears materialized. The ex-husbands' fears of humiliation, losing their children and having a less favorable lifestyle most probably would have been addressed by the second time they responded to the "fear" item, with no new fears surfacing. By contrast, the increased reports of ex-wives' fear of their former partners were consistent with the literature about increased danger after separation, particularly for women. Without the qualitative information, erroneous conclusions could have been made based solely on the quantitative data as represented by the responses to the items.

On reflection, however, the real problem lay in a question that held alternative meanings and therefore differential validity for husband and wife: "Do you fear your spouse?" Physical threats vs. loss of something are two very different interpretations of the word "fear," depending on gender. In psychometric literature, this is called differential item functioning.

Short qualitative responses can be quickly hand-coded or reported anecdotally and, as shown here, may have important implications for the interpretation of findings. On a related note, there are good software packages such as NUD*IST or ETHNOGRAPHY that may be used for larger qualitative files. These packages allow the researcher to make decisions on the coding of individual passages while also enabling links with quantitative packages such as SPSS.

ISTSS Research Methodology Special Interest Group sponsored this report. For more information, contact chairs Daniel and Lynda King, National Center for PTSD, VA Boston Healthcare System, Boston, Mass. E-mail: [email protected], or [email protected].