The Method in Our Madness: Data Collection and Analysis for Our Study of Higher Education, Part III

by Wendy Fischman and Howard Gardner

When hearing about our ambitious national study of higher education, colleagues often ask us how we went about carrying out the study and how we will analyze the various kinds of data and draw conclusions. At first blush, the decision to carry out approximately 2000 semi-structured hour-long interviews across ten deliberately disparate campuses, to record and transcribe them, and then to analyze the resulting “data” seems overwhelming—and not just to others! Moreover, when asked for the “hypotheses” being tested, we always reply that we did not have specific hypotheses—at most, we knew what general issues we wanted to probe (e.g. academic, campus life, and general perspectives on the means and goals of higher education). Additionally, we wanted to discover approaches and programs that seemed promising and to probe them sufficiently so that we could write about them convincingly and—with luck—evocatively.

An Earlier Model

We did not undertake this study entirely free of expectations. Our colleague Richard Light, now a senior adviser to the project, spent decades studying higher education in the United States; he provided much valuable background information, ideas about promising avenues to investigate, and some intriguing questions to include in our interview. Both of us (Wendy and Howard) had devoted over a decade to an empirical study of “good work” across the professions. In that research, planned and carried out with psychologists Mihaly Csikszentmihalyi and William Damon and their colleagues, we had interviewed well over 1200 workers drawn from nine distinct professions. The methods of interviewing—and the lack of guiding hypotheses—were quite similar. Because we were frequently asked about our methodological approach, we prepared a free-standing paper on the “empirical basis” of good work. In addition, reports on our findings yielded ten books and close to 100 articles; moreover this project led to several other lines of research—see TheGoodProject.org. Our prior work on “good work” served as a reasonable model as we undertook an equally ambitious study of higher education.

In this blog and the two previous in this series, we seek to convey the “method” to our undertaking.

Part III. Key Additional Analyses in our Study

As in any comprehensive study, there are innumerable kinds of analyses that we could do. Indeed, in the aforementioned study of good work, we produced dozens of research papers on specific questions. We published a book of over 300 pages on responses to a single question in our interview questionnaire: “In your own work, to whom or what do you feel responsible?”

In a rough-and-ready way, for our current study of higher education, we distinguish among the following kinds of analyses:

a.) Low-hanging fruit. These are questions that we can simply score numerically, such as the two rank order questions about the purposes of college and problems on campus. Also relatively easy to quantify are questions that require a definition, including “What does the term liberal arts mean to you?” and “If you could describe the students on this campus with a single adjective, which descriptor leaps to mind?” (Preview: “diverse” and “quirky” happen to be two of the most prevalent responses on some campuses.)

b.) Questions requiring more analysis. We ask respondents, “If you could give one book to a graduating senior, what book would it be?” It’s easy to score whether participants respond at all. But a classification of the kinds of books mentioned, and the reasons for the selection, takes time, as does a classification of what happens on the occasions—very frequent—when students can’t think of a book. (Preview: Dr Seuss’s books for children, Malcolm Gladwell’s books for general readers, and Dale Carnegie’s How to Win Friends and Influence People are mentioned often.) Also, the reasons invoked when participants reject the question can be sophisticated or banal (e.g. “I don’t know because I never read” vs. “It all depends on the kind of student, her interests, the breadth of her prior reading, my relationship with her, etc.”).

c.) Use of “Big Data.” The advent of methods for analyzing large amounts of text expeditiously is transforming the analysis of qualitative data. We ourselves did not have experience with “big data.” But we were fortunate to secure the services of an expert research methodologist, John Hansen, who has done these kinds of analyses and indeed has experience working with unstructured data, such as text. With his help (and that of Reid Higginson, doctoral student at HGSE), we have been able to look at the frequency of particular words and phrases across participants (and campuses and constituencies), as well as where these words are used. For example, we can look to see the frequency of “stress” and if this word is used before or only after we ourselves introduce the concept (as part of the rank order question about problems on campus).

So far, initial passes through the language used by student participants has uncovered a fascinating phenomenon. Whether a first year student or a graduating student, whether at highly selective or less selective schools, the frequency of individual words and phrases across students at the various schools are strikingly similar. There are no major outliers; for example, the word “stress” does not appear more frequently with students at one particular school. This surprising result was also a relief; it suggests that differences in coding of responses are not just based on the actual words the students use, but rather on the context, thinking, and messages behind their words.

We suspect, however, that when larger segments of text are used (e.g. patterns of words, not single words), we may find that certain words or combinations of words distinguish between more or less sophisticated responses—for example, students who exhibit higher LASCAP vs. lower LASCAP (see the previous blog post for an explanation of this concept). Should this be the case, it might mean that in the future, interview questionnaires—and perhaps even interview recordings—could be scored automatically. In addition, more sophisticated analyses could turn up more subtle patterns in word usage—we have only scratched the surface of the endless number of analyses that one can imagine, given the amount of data collected.

d.) Emotional analysis. It is now possible to monitor recorded interviews and to measure emotions—for example, degree of stress in general, or stress with respect to certain topics. Given the importance of “mental health” and “belonging” issues, these would be well worth investigating by means of emotional indicators in the recordings. But, like dozens of other issues that we can think of, such analyses will need to be carried out by other investigators, or by the current investigators at another time.

A Survey!

Few investigators in the future are likely to undertake a study of this magnitude: 2000 interviews (as well as associated studies of related topics) on 10 disparate campuses over a six-year period. Thanks to the support of The Spencer Foundation, we have developed a survey which can be taken online, and which correlates reasonably well with the findings in our hand-administered and hand-scored measures. (Of course, not every topic can be probed in a survey—one of many reasons that we have carried out face-to-face interviews!) Once our own results have been published, we will make this survey available to interested, competent parties.

One might ask why we did not simply develop the survey initially—and save a lot of time and trouble for two thousand human beings and many dedicated researchers. In truth, we could never have come up with an effective survey unless we had done a considerable amount of the exploratory work entailed in our own study. Nor would we have been able to anticipate the kinds of mental models that emerged. In addition, we take it as an article of faith that one can learn far more by speaking directly to a human being, in his or her own surroundings, than by examining the results of a multiple choice, machine scored instruments. (As just one example, skilled interviewers can confront individuals with contradictions in their responses—even as the interview participants themselves may note contradictions, a sign of LASCAP.)

However, as a result of our study and the analysis of data in which we are now deeply involved, it should be possible to come up with more focused hypotheses. And these, in turn, can and should be investigated by appropriate measures—which will certainly include surveys.

Conclusion: A Few Words about Our Own Motivation

As is probably evident, we would never have undertaken this endeavor were we not very concerned about the current state of higher education in the United States, and particularly that strand that is not committedly vocational. Moreover, our concerns have only multiplied in recent years, in view of societal trends and disturbing results in polling about public attitudes toward higher education, not even to mention ignorance or antipathy toward “liberal arts.” We hope that our findings and recommendations will be read, listened to, discussed, and debated. If we are fortunate, these conversations will lead to improvements, and perhaps reorientations, in how higher education is carried out in the United States and perhaps elsewhere.

By agreement, we will share our findings on a confidential basis with the schools that generously permitted us to work with their constituencies. In publications, we will not discuss school-specific findings. We believe, however, that the questions, concepts, and methods of data analysis that we have developed should be useful to the entire sector; and we would be pleased to assist in this process.

Over the coming months, we will continue our series of blogs in which we present preliminary findings and recommendations emanating from our study. Stay tuned!

We thank our colleagues Dick Light, as well as John Hansen and Reid Higginson from our research team, for their helpful comments on earlier drafts of the three blogs in this series.

© 2018 Wendy Fischman and Howard Gardner