Mortality
and the Choice
Problem
in Research Methods
Where have I been?
When I
last posted an entry to this blog about a year ago, I was feeling poorly, and I
got worse over the coming months. In the
spring semester, I staggered through the closing weeks of the course I was
teaching and barely managed to finish the page proofs of my new book.[1] After that, I was no longer capable of
effective work. In May 2014, following remarkable
incompetence by those reading my X-rays, who twice missed seeing a large tumor,
I was diagnosed with late-stage lung cancer.
Now, after many weeks of treatment and almost as many weeks recovering
from treatment, my strength has returned to the point that I can resume writing
in this blog—with perhaps a better understanding of certain issues and topics.
What does my
glimpse of death have to do with research methods?
Nothing
focuses the attention on the choice problem like confronting your own mortality;
you gain an enhanced appreciation for judging alternatives. You often have to make decisions, with varying
degrees of complexity, about crucial matters, potentially matters of life and
death. There is no assurance that you’ve
made the right decision. Even after you have
made the decision and your doctors have acted on it and evaluated the
consequences, you still cannot know
whether another option might have been better. You can speculate, of course,
but you can’t ever be sure. The choices
I had to make, and the principles for making them, are remarkably parallel to issues
of methodological decision making.
Choice is
unavoidable & uncertainty is certain
The
first principle of methodological choice is unavoidable uncertainty. The same is true of medical treatment
choices. Patients usually want certainty
from their medical advisors. And those
who seek the advice of methodologists usually do too. Certainty, being able to give unambiguous
advice, is often taken as a sign of competence.
But claiming more certainty than is merited can be dishonest and professionally
unethical.
Uncertainty
does not equal ignorance or weakness.
Just the opposite is often true.
Unwarranted certainty on the part of advisors may cause you to breathe a
sign of relief, but your confidence is based more on faith than knowledge. Hunch-based medicine hardly seems like a good
idea.
The same
ideas—necessary choices made in the face of uncertainty—have been the focus of
my research and writing for many years, but making decisions about my own health
put the theme of decision making while struggling with unavoidable uncertainty in
a clearer, though harsher, light.
Uncertainty in
planning research
Consider
the kinds of questions you might ask yourself when approaching a research
project. Should you pursue your research
question with surveys or interviews or some combination of the two? If you combine them, should you use the
interviews to help construct the survey questions, to aid in interpreting the
answers to the survey questions, or both?
There is no way to know in advance.
And, even after the study is completed, there is not even any way to know in retrospect. You might be able to say, “OK, this research
turned out pretty well,” or that “this study would have been more persuasive
had I taken a different approach.” But
you can’t really know; “do-overs” are rarely possible. What you can do when following a research
agenda is to think of your work as cyclical.
You make a choice when selecting a research strategy. You take action using that strategy. Then you evaluate the results of that action
and alter your future choices accordingly, as illustrated below.
Choice
Action
Evaluation
Atul Gawande’s take
on these issues
In the
abstract, such a graphic makes it all seem pretty simple. But the complications that arise when
applying the choice-action-evaluation-choice . . . feedback loop are
challenging. An excellent source for examining the
relations in this cycle—in both medical and social research contexts—are the
writings of Atul Gawande, most recently his book about end-of-life choices, Being Mortal (2014). By stressing the social and human contexts of
decision making in health care, Gawande humanizes it and highlights the natural
links between it and methodological decision making.
The
first point is that there is no invariably correct decision. The right decision depends on your
goals. Do you want to give up some
privacy and autonomy by entering a nursing home in order to live longer than
you would in a less restrictive environment?
Or do you prefer to maintain your autonomy at all costs, being able, for
example, to sleep, eat, and lock your door when you want? Is that autonomy worth increased risks to
your safety and longevity? Assisted
living is an intermediate option, but choosing the right assisted-living facility
for you is no simple matter. And when
the end gets nearer, as it inevitably must, do you want hospital or
hospice? Everything depends on what you
value, and, therefore, on deciding
what you actually value. Different
values will lead to different “correct” choices. And if you have, in Gawande’s terms, “priorities
beyond merely being safe and living longer,” such as privacy and autonomy, you
may find yourself in conflict with your loved ones. Of course, you might like to be safe, to live
longer, and also to retain your privacy and autonomy. But there are usually unavoidable tradeoffs
among these goals. There is no way to
maximize them all.
What to do—and how
to do it.
Dying
isn’t curable, but many diseases are.
And it is comparatively easier to prolong life with good interventions
than it ever has been. Medicine can be
highly effective; but practitioners applying the same techniques often have
very different levels of success.
Gawande is well known for his earlier works uncovering the reasons for
differences between excellent and mediocre practice. Again, there are strong parallels between
medical and research practice.
Gawande’s
Checklist Manifesto (2011) is a good
example. His argument is that in any
field where the work is complex the quality of work is improved if you use
checklists. There are two basic reasons to
use checklists: to be sure (1) that you
don’t forget something important; and (2) that you consider all the options
available to you. Gawande emphasizes the
first, but the second is arguably the more important. In medicine, for example, do you choose
chemo, radiation, surgery, or some combination?
Only then, after you have chosen, can you focus on how to implement your
choices most effectively. In social
research, do you use interviews, experiments, ethnographic observation, or some
combination—and if a combination, how much of each and to what end?
I first
encountered Gawande’s writings in an article in the New Yorker magazine[2] in
which he examined the widely varying success of clinical treatments for cystic
fibrosis. All the clinics he studied
followed the same procedures. Indeed
they had to do so to maintain their status as approved clinics. But their
success rates in dealing with this debilitating disease differed greatly. To
find out why some clinics were much more effective than others Gawande
conducted intensive case studies of the most successful clinics. What did they do that set them apart? Essentially, the people working in them were relentless
in implementing the protocol, the same protocol that all other clinics
followed, but not as rigorously.
Excellence
in implementation
After
you decide what to do, how do you do it, how do you implement your treatment
plan—or your research design? What makes
one physician or researcher superb and another mediocre? The answer may be remarkably simple: relentlessness
in applying the plan, laser-like focus, and fastidious attention to detail. The same is true, I think, in the practice of
social research. For example, in my work
with students conducting ethnographic observations for their doctoral research,
one distinguishing characteristic between the students who make major
contributions to their fields and those who barely get through is unyielding
attention to detail and inexhaustible effort—first at observation, then at
recording observations, then at coding and re-coding, and re-re-coding those
observations. When you take these steps
you are making invisible choices, what you do when no one is looking. It can be hard to realize that recording,
initially coding, and then recoding your fieldnotes may require much more
time—perhaps three to five times as much—as the time spent doing the observations
on which the subsequent work is based.
And, unlike with more quantitative forms of analysis, there are few
routine methods, recipes, algorithms, or step-by-step guidelines to fall back
on.
The same
kind of variety occurs in experimental research. Some experiments are good enough to get
published, but fairly soon pass unnoticed into oblivion. Others set the standard for their field. There are many reasons for this, of
course. But even experimental research
by investigators studying the same phenomena—and using methods comparable
enough that they can be summarized in a meta-analysis—vary markedly in
outcomes, as measured by effect sizes.
One difference has been called “super-realization.” Small experiments, especially those coming
early in the research on a topic, usually obtain higher effect sizes; this has
been widely noticed in both medical and educational research. There are several explanations for the
differences between small experiments and large ones. One is that small-scale, proof-of-concept
experiments are often conducted by enthusiastic pioneers. Others who follow the paths set by pioneers
might be less relentless with the attention to detail or less rigorous in
applying the treatment or independent variable.
Deciding
on a method or set of methods is important, but so is deciding how much effort
you will exert when you implement them. Not
only do you have to make good decisions trying to implement the original plan
well. Sometimes you need to alter the
plan, maybe even going back to the beginning on the basis of what you’ve
learned along the way. It’s easier, of
course, to gloss over problems and forge ahead so as to meet a deadline.
Responsibility for
making choices—and for implementing them
Do you
make your own choices, or do you follow tradition or, do you let others (who
are often following tradition) make them for you? And once you’ve decided what to do, how
energetically do you implement your decisions?
Decision
making is hard because there is never any guarantee that you will make or you
have made the right choices. Uncertainty
is inevitable, and if you care about the results, such uncertainty can be very
stressful. But that is the nature of
things. There is no one best method, nor
is there any one best method for choosing among methods.
The
“decision problem” pervades even pure mathematics, the most abstract of
scholarly disciplines where you could expect human values and foibles to have
limited play. About 100 years ago, one question
was whether there was a definite method that could be used to correctly decide
whether an assertion was true. Alan
Turing showed that the answer, in a word, was No. At about the same time Kurt Gödel drew
similar conclusions about the unprovability of mathematical assumptions. If in the most abstract of fields, pure
mathematics, there is uncertainty about the truth of any assertion, it is surely
true in more messy fields like medical and social research, where human goals,
foibles, and limitations will always intrude.
Still, we cannot stop.
Mathematics did not end with Gödel and Turing. Nor will social and medical research cease as
we confront the fact that there are no unerring ways to choose among the best
options. Decisions are inevitable and
inevitably uncertain.
[1] Vogt, Vogt, Gardner,
& Haeffele (2014). Selecting the Right Analyses for Your Data
(New York: Guilford Press).
[2] New Yorker, Dec. 6, 2004; on line at http://www.newyorker.com/magazine/2004/12/06/the-bell-curve.
