CONGRATULATIONS! IT'S A PARADOX

A Correlation Paradox

Paradoxes pop up pretty often in quantitative research—more than most of us realize.  They can be wonderful things—surprising, puzzling, even funny.  And they are often instructive.  I discovered the correlation paradox when studying trends in state support for public higher education.  My colleagues and I were discussing the best way to demonstrate trends in our data, and someone said, how about a correlation between state income and higher education spending?   I said “sure, let me call up the data and see what we get.”  A minute later, when the answer appeared on the screen, one colleague (who didn’t like me) beamed maliciously; he said “that’s impossible; as we all know blah, blah, blah.”  The smartest among us said nothing, but she did utter a barely audible “harrumph.”  The project director said, playfully I think, “So Paul, remind me again, how much are we paying you?”  The statistical result I projected onto the screen, the correlation I calculated, seemed impossible.  Everybody thought I must have made an error.  And I did too.  But it wasn’t an error.  It was a paradox.  To describe it, I’ll sketch in some background information.

A correlation is a co-relation.  It is a measure of the association between two things (called variables). The correlation is a number (called a coefficient) between zero and 1.  When there is no relation between the two things, the number is zero.  When the relation is perfect the coefficient is 1.  An example of a strong correlation might be the ages and heights of the children in a pediatrician’s practice.  The older children would usually be taller; the younger would usually be shorter.  There might be some exceptions, but a strong general pattern would probably hold. The correlation might be around 0.75, not perfect (which would be 1.0), but pretty strong.  That’s obviously because when age goes up so too does height.  When two variables, such as age and height, move in the same direction, this is called a direct relation or a positive correlation.

Sometimes correlations are “inverse.”  When one thing goes up the other goes down.  For example, when the price of gasoline goes up the sales of gas guzzlers goes down.  The numbers used to describe the relation between cost of gas and sales of gas hogs are the same—they range from 0 to 1.  The only difference is that with an inverse relation the number is negative.  It might be around −0.75.  Since the number used to describe them is negative, such inverse relations are often called “negative correlations.”  That is not because there is anything undesirable about them.  The negative number is just a way to describe an inverse relation.  For example, the more people exercise the less likely they are to have heart attacks.  Because the amount of exercise and the number of heart attacks go in opposite directions, they are negatively correlated, but more exercise and fewer deaths are “positive” things.

These are very well known statistical facts.  My students almost always get them right on exams.  So where does the paradox described above come in?  It occurs with trend data.  When studying trend data, variables that are moving in opposite directions over time do not always, as they “should,” have a negative correlation.  This ought to be impossible—by definition:  the correlation coefficient describing the association between two variables moving in opposite directions must be negative. However, when looking at the trend data for two variables that are clearly moving in opposite directions, sometimes the correlation is positive, not negative.  That is the paradox.  A paradox is something that is very surprising, because it is impossible by definition or by logic, but actually seems to be true. 

I was shocked (and embarrassed) the first time I encountered a positive correlation between variables that I knew were moving in opposite directions.  My surprised colleagues and I were working on a project studying correlations between trends in states’ incomes—Gross State Product or GSP—with state appropriations for public higher education.  In many states, the trend over the years in state income was clearly upward, but the trend in state spending on higher education was just as clearly downward over the decades we studied.  But the correlation between the two variables really was positive, not negative—despite the fact that they were moving in opposite directions. 

My colleagues were very surprised. They thought I had made some stupid mistake.  And I even thought I must have messed up.  I recalculated the correlations.  I made sure I had entered the data correctly.  I double checked to make sure I had pointed at and clicked the right options in the software.  I even calculated the correlations by hand—for the first time in many years.  But the correlation would not change.  The correlation between two variables moving in opposite directions remained positive. 

So what happened?  What caused the paradoxical results?  Paradoxes like this are not unknown. The two most familiar among statisticians are Simpson’s Paradox and the related Lord’s Paradox.  Naturally I named mine “Vogt’s Paradox” (we do so love to name things after ourselves) and I described it briefly in my Dictionary of Statistics and Methodology.[1] The paradox is not much of a mystery once you know the source of the anomaly.  Like the Simpson and Lord paradoxes, the correlation paradox has a fairly simple explanation.  It can occur when change scores rather than absolute values are used to calculate the correlations.  In this case, what was happening is that when state income went up, so too did their expenditure on higher education, but not by as much.  When states’ income went down, spending on higher education went down too, but by more.   In most individual years the two variables (income and expenditure) moved in the same direction; that accounts for the positive correlation.  But the magnitudes of movements were such that over time the two got further and further apart.  Each year that the two increased, income increased more.  And each year that the two decreased, expenditures decreased more.  That accounted for the overall inverse trend.

Like other paradoxes, explaining it in words often doesn’t help.  It can be hard to see what is happening until you look at some worked out examples.  For Simpson’s paradox, the Wikipedia entry has some good ones.  If you want to see a small set of example data illustrating Vogt’s paradox, send me an e-mail (wpaulvogt@gmail.com) and I’ll reply with a short document containing an illustration. 

---

[1] 4^th edition, Sage Publications, 2011.

Ten Evaluation Aphorisms

Program Evaluation:  Ten Aphorisms

Some months ago, I was asked to give a brief overview of the most important principles of program evaluation. The audience was directors of projects that had received federal funding to improve teacher quality through professional development programs.  The talk was extemporaneous, so no verbatim text exists, but in response to requests for copies of the talk, an outline is posted here.

The basic principles of program evaluation are simple. They are mostly a matter of common sense.  That’s why they can be summarized in aphorisms—pithy statements that contain a lot of truth.  Some might see the following as a list of platitudes or as an insult to their intelligence.  Still, these simple principles are overlooked remarkably often.  In my experience when a project evaluation goes wrong the problem can be traced to ignoring one or more of principles sketched in these 10 aphorisms.

  

1.     If a program does not have a theory of change, it cannot be fully evaluated. 

• Your project proposals say that certain planned actions will lead to specific outcomes.  Your theory of change explains how the actions will lead to the outcomes.  Without that explanation of how actions lead to outcomes, your proposals will have limited value as a guide to practice now and in the future.

2.     If you don’t know where you started, you can’t tell how far you’ve come.

• Without baseline data you can’t gauge your progress.  It is easy to overlook the obvious importance of baseline data.  Think of how often people forget to check the odometer before beginning a trip.

3.     If you want to measure change, don’t change the measure.

• Multiple measurements are good, advantageous even, but to compare before with after use the same technique.  Don’t use an attitude survey for before and a test of knowledge for after--or vice versa. 

4.     To collect useful evidence, it helps to know what you are going to collect.

• It’s always good to be ready for surprises, but a data collection plan is essential.  If you want to be surprised, you have to have expectations.

5.     If you don’t know how to analyze the data you plan to collect, don’t collect it.  Or change your plan.

• Your time is too valuable to collect data hoping that it might someday be useful to somebody for something.

6.     To use the data you’ve collected, you’ve got to be able to find it.

• A data management plan is as important as a data collection plan and a data analysis plan.

7.     Everyone loves a good story, but nobody trusts stories very much as evidence.

• Narratives are essential in any good project evaluation, in part because they help describe causal processes (the how of number 1 above), but narratives are not very effective for establishing that outcomes occurred or for estimating their size. 

8.     Statistically significant findings may not be important—and vice versa. 

• Statistical significance and p-values can sometimes provide evidence of project success, but they are not usually the best evidence—even for experiments.  Effect sizes and confidence intervals are better for quantitative outcomes.

9.     Nobody cheats on an eye exam. 

• Why not?  Because all parties have a vested interest in getting good data and drawing accurate conclusions.  Project evaluation should be like that.  It should not be an adversarial relationship between the project managers and evaluators. 

10.     This year’s evaluation is the heart of next year’s application.

• A rigorous evaluation of your current project is the best preparation for success in the next round of grant competitions.  Thinking about that fact might give you an incentive to persist when evaluation work is getting tiresome.  Remember that you own the evaluation.  It is not only for external compliance and to describe what you’ve done in the past.  It can help you plan for the future.

The Paradox of Inquiry: Searching for Methodological Innovations

Fisher Fuller has posted another comment.  He is certainly persistent in by demanding that I specify the criteria we used to define and select methodological innovations.  That’s challenging but fair.  It’s challenging because, as I suspect he already guessed, we did not know at the outset exactly what we meant by an “innovation in social research methods.”  I copy and paste his comment below.

Vogt did not reveal any criteria for deciding what is and is not an innovation in research methods.  He only told us that he talked to some people and informally polled others. Consensus among friends and vote counting do not add up to principles of decision making on this important issue.  I agree that Williams and Vogt discovered what seem to be some important innovations, but I disagree that they have a reliable method other researchers could use in the future to discover more.

Our method for studying research methods was more ethnographic and exploratory rather than confirmatory.  We were studying, so to speak, the culture of researchers, specifically what they believed about developing and using new research methods.  If one of our informants suggested that a particular method was an important innovation, we recorded it, reflected on it, collected it, and juxtaposed it with the beliefs other researchers shared.  Then we thought about it quite a bit about our collection of beliefs and made our choices.  That was the nub of our method.

I doubt that Fuller will be satisfied with this approach.  We asked people who we thought would know, and we paid careful attention to what they told us.  Fuller is right that we don’t really explain how we knew whom to ask and what to ask them and what kinds of answers would be satisfactory.  Obviously, Malcolm Williams and I didn’t know exactly what kinds of innovation we were going to find, but we did have some ideas—some sensitizing concepts in Herbert Blumer’s terms.

Fuller’s questions returns us to Plato’s paradox of inquiry.   Meno asks Socrates, “How will you inquire into that which you do not know?  What will you put forth as the subject of your inquiry?  And if you find what you want, how will you ever know that this is the thing which you did not know?”  While this view is sometimes described as one of Plato’s principles, Plato/Socrates clearly rejected it.  Socrates replies to Meno that we ought not to listen to this sophistical argument about the impossibility of inquiry because it will keep us from actively seeking knowledge.  Still, while Socrates dismisses the sophistical argument, he spends the rest of the dialogue trying to answer it. 

And exploratory researchers are still working on how to deal with the paradox of inquiry.  It is a perennial question, one that is always worth asking and trying to answer.  But any answer will be incomplete and temporary.  Rejecting the paradox of inquiry, as Plato points out, is necessary if we are to be active seekers after knowledge.  Plato’s argument, I think, can be read as ultimately being pragmatic.  Rejecting the paradox and forging ahead—all the while remembering the kernel of truth it contains—makes critical inquiry and the growth of knowledge possible.  Simply accepting the paradox chokes off further inquiry.  The paradox’s threat to the vigorous pursuit of knowledge is especially great in empirical and therefore partly inductive fields of inquiry such as social research. 

What is an innovation in research methods?

Fisher Fuller raised some challenging questions in a comment, copied here:

I'm suspicious of the whole project. What is a methodological innovation anyway? How do you presume to be able to identify them? Are you trying to create another set of restrictive canonical guidelines? If you can describe an innovation in a summary chapter, is it really an innovation any longer? 

These are serious questions.  Fuller is surely right that the idea of an innovative canon in research methods is almost a contradiction in terms.  If it’s canonical it chokes off innovation.

The short answer to the question of how we identified innovations in the Handbook is that we asked our colleagues, both informally in chats and more formally in some fairly extensive e-mail polling.  We had discussions with anyone who would talk with us, starting with Chris Rojek, our editor at Sage Publications who originally suggested the idea for the Handbook. 

We tried to be inclusive rather than restrictive in our understanding of what constitutes a methodological innovation in social research and we uncovered several broad classes of innovation.  Here are some examples:

• Some innovations arise because new developments in social life and the need to develop novel ways of studying them.  An example is Mike Thelwall’s studies (in chapter 9) of human communication on the Web.  Another is Elizabeth Griffiths’ discussion of geographic information systems (in chapter 21).
• A fertile area of innovatory growth occurs in fields that are not brand new but that are undergoing rapid development, often by generating links between separately developed methods.  One such pair of methods, developing separately and together, is multilevel modeling and structural equation modeling.  For discussions of their joint development see chapters 26 by Rex Kline and 27 by Keenan Pituch & Laura Stapleton. 
• Combining methods is a common source of new ideas about old methods.  An illustration is chapter 13 by John Hitchcock & Bonnie Nastasi on using mixed methods for construct validation.
• A last example is applying a new method to an area of research where it had not been used before.  For instance, Qualitative Narrative Analysis (see Ragin & Schneider’s chapter 8) is applied to program evaluation (see Vogt et al., chapter 15).

 We won’t be surprised to hear from readers who wonder how we could have missed this or that innovation or how we could have mistaken a short-lived fad for a new approach.  We welcome such discussions, questions, and criticisms because they are a key source of further innovation.

New Book on Innovation in Research Methods

The Sage Handbook of Innovation in Social Research Methods is now available (Sage Publications, 2011).  Edited by Malcolm Williams and Paul Vogt, it contains 27 chapters exploring innovatory methodological approaches from conceptualizing research problems through data analysis.  Further details are available at www.sagepublications.com.

My chapter* is on "Innovations in Program Evaluation: Comparative Case Studies as an Alternative to RCTs."  We argue that randomized controlled trials (RCTs) are seldom feasible when evaluating large-scale social and educational interventions.  Further, "gold-standard thinking," which assumes there is one best method for all research problems, is a mistake because it discourages thought and restricts methodological innovation. Building on our research evaluating educational programs, we show how a comparative case study approach can be an effective alternative to RCTs.  We illustrate by applying Charles Ragin's Qualitative Comparative Analysis (QCA) to the evaluation of projects in a large educational program.
      *Written with colleagues Dianne Gardener, Lynne Haeffele, and Paul Baker.

Overview

Posts and discussions of important topics in research methods in the social and behavioral sciences

Helpful for users of Paul Vogt’s research methods texts, reference books, and other publications

Contains discussions of and supplements to those works, including

• Dictionary of Statistics and Methodology (Sage Publications; 1st edition, 1993; 4th edition [with Burke Johnson], 2011]).
• Quantitative Research Methods for Professionals (Allyn & Bacon, 2007).
• When to Use What: Guidelines for Selecting Research Methods in the Social Sciences, vol 1 (Guilford Press, 2011, forthcoming).

Source for insights and new information about research methods