First Steps Toward Going Public: Our Study at the Start of 2019
By Howard Gardner and Wendy Fischman
So here we are, right at the edge of going public. It’s been nearly seven years since, with our esteemed colleague Richard Light, we conceived of a major—arguably the major—qualitative study of higher education in the United States in our time. From 2012-2018, with the support of a remarkably gifted and hard working staff, we carried out semi-structured hour-long interviews of over 2000 individuals spread over 10 deliberately disparate campuses. We sampled from 8 different constituencies: on each campus 50 freshmen, 50 graduating students, and a smaller but still substantial number of faculty, senior administrators, trustees, parents, young alums, and job recruiters (though the latter were not seen on the 10 individual campuses). All of these interviews were recorded; we have each read through all of the transcripts; and between us, we probably did 500 interviews in person, some solo, some as a pair.
Importantly, most of the interviews we carried out, with the exception of students, parents, and young alums (which were typically done by Skype), were typically done on campus, usually by a pair of trained interviewers. On those campus visits, the 3, 4, or 5 members of the team who had made the trip would meet twice a day, typically over a drink or a meal, and talk about general impressions, and what we had learned, what we remained curious or uncertain about. These conversations were important because we often learned information about the campus which the rest of the team did not know, and sometimes these tidbits resulted in additional interviews, observations, photos, even an additional trip. Necessarily, we also drew comparisons across campuses: "Campus X seems much more troubled than we had anticipated"; or "This is the first campus where we have seen deliberately diverse groups 'hanging' together or studying together."
Within the research group, all of the members of which have taken an oath of silence, so to speak, we developed more general ideas about the several campuses, and, indeed, we also generated pretty strong expectations about the overall study. For example, mental health emerged early as a much larger issue across campuses than we had anticipated. Also, when it became clear that thoughts about the arts, or thoughts about ethical dilemmas, rarely came up spontaneously, we added suitable open-ended questions to the latter part of the questionnaire.
We had begun the study with a clear interest in the “liberal arts and sciences” and, for short, dubbed our study “LAS21”—for "Liberal Arts and Sciences in the 21st Century." But as we quickly learned through our interviews, too many individuals either do not know the phrase “liberal arts” altogether; or they misinterpret it as meaning “anything goes” or “courses that tilt toward left wing instructors and topics.” Accordingly, going forward, we will use a more neutral label for the study—such as “Higher Education in the 21st Century”; and for our concept of “LAS Capital” (a phrase we love for its Marxist economic and literary connotations), we have now substituted the innocuous (though still whimsical) HED (Higher Ed) Capital.
In our work, we have sought to follow the researchers’ code: we have been prudent about keeping silent about both our impressions of individual campuses—which we will never speak publicly or write about—and our suspected findings. And so we have only mentioned publicly, or to close colleagues, conclusions that are quite clear (e.g. the aforementioned prevalence of mental health issues or the ignorance about the meaning of the phrase “liberal arts”).
Finally, in the summer of 2018, our data collection was complete. We heaved a huge sigh of relief—not least because, while there were numerous small snafus, the study itself was never compromised. No campus ever withdrew or even complained! We owe enormous thanks to our entire staff, who have been steadfast in their dedication and their silence. They set a very high bar for the field!
During the latter years of the study (starting in year 4), we began to analyze our data. Some questions and observations were easily scored, even by those without any statistical training. These questions were either direct items in the interview (e.g. “What book would you recommend to a graduating student?”) or the two rank-order questions we asked (about problems prevalent on campus and about the principal purpose of college). But we also wanted as well to carry out statistical tests en route, and to see whether and to what extent various concepts, responses, and words and phrases correlate (or not). John Hansen and Reid Higginson, excellent statisticians, helped us launch the study, and now Shelby Clark, a postdoctoral fellow at Project Zero, is taking the statistical lead. We will be poring over the qualitative and quantitative data in the coming months (and years). Shortly, readers will begin to see our findings emerge in writing (for example, further blogs in this series), while audience members will encounter them in formal and informal talks.
Hitherto, neither of us had personally worked with so-called “big data.” We were fortunate that, under John Hansen’s leadership, we dipped our toe in this area, looking at the occurrence and frequency of over 100 words across all of the student transcripts (at the time, well over 500 freshmen and seniors). As we studied reports each week, we wondered whether we were using time wisely because we were not seeing any remarkable differences across campuses or constituencies. But then we realized that this very “non-finding” was actually a fascinating finding. And so, under the guidance of Shelby Clark, we approached “big data” in another way. No longer did we use a “top-down” method in which we explored the usage of specific words that we identified (as interesting or important). Instead we used a “bottom-up” approach, looking for the words that are most common for students across schools. Of the top 100 words, we find that still, for all intents and purposes, students across class status and across campuses use the same words. Put more concretely, whether you are a senior at a selective school or a less selective school, we cannot find major systematic differences in the individual words that you use. It appears that age (or stage in life) trumps the kind of campus students find themselves on.
Usually social scientists are disappointed by a null finding. But in this case, we were very pleased. For this finding, which seems robust so far, means that differences in higher level coding (e.g. level of sophistication of the reasoning) is not a reflection of the level of sophistication of vocabulary. It may well be that differences do exist if one looks at broader swathes of text—that’s an assignment that we will undertake as time and resources allow.
In the coming months, two important milestones in our project will be reached:
For the first time, we are giving public accounts of what we did and what we think it means. In early January 2019, we spoke to more than 300 college presidents at the annual meeting of the Council of Independent Colleges (CIC) in Scottsdale. In addition, Howard gave three lectures at the Harvard Graduate School of Education. Importantly, those and other imminent lectures will NOT be streamed or published—because we are still not certain which findings are definitive and which ones will have to be modified or even scuttled. However, in the previous blog in this series, we reported the seven "takeaways" that we presented to the college presidents of the CIC.
Once we are certain about specific findings, we will report them in this blog. We will also begin to outline one or more books in which we report our findings. In truth, we could easily outline a dozen books and several dozen articles, either technical or for the general public. For example, we could just write one book alone about the uses of the single word “diversity” across constituencies and campuses. We will face the choice of what to emphasize, for particular audiences, and which results and recommendations to highlight, or to save for another occasion.
Stay tuned!
© 2019 Wendy Fischman and Howard Gardner