As part of our jobs we make decisions that impact the lives of coworkers, users and customers, not to mention the bottom line. Decisions are made in various states of uncertainty, which is why we seek data. Often, however, we want data even when it will not produce better decisions. We want it for psychological reasons.
As one example, we tend to prefer data that are highly correlated. The more we get the more confident we feel, even though we’re the ones doing the sampling. Or consider the “disjunction effect.” If we know we’ll do a thing given some information, call it “X,” and that we’ll do this thing without X, then X shouldn’t matter. (This is Savage’s “Sure Thing Principle.”) We often feel, however, that we still need to know X before acting.
Though “normal,” such behavior can cost organizations a lot of money. How often do we desire data simply as a form of hand holding? How much time and money are spent gathering, processing, packaging, and rolling data uphill to executives that don’t change or improve the decisions made? In short, how much money is spent on “data” that don’t serve as information?
BS Data
Unfortunately, a lot of the data that ARE used to make decisions shouldn’t be. Survey results, for instance, are a notorious source of BS data. People often misrepresent in surveys, or, if that’s too strong, simply can’t answer what’s being asked. So, they rationalize, guesstimate, or just put something down to get the damn thing done.
Survey questions also tend to be biased, further confounding the results. As Erika Hall (2015) notes, this makes surveys dangerous when coupled with their ease of “blasting out.” Add to this the illusion of objectivity quantitative survey results provide and we have a potent mix for being misled.
Even when surveys do inform, they are often not very helpful. Say a survey shows people are largely dissatisfied with something. This might be sound data, but often only confirms what was already known. The real question is, what to DO about it? Pulsing satisfaction has not been shown to be predictive of behavior, and really doesn’t help improve the situation.
Another common culprit is correlations—someone sits down with data not collected to test an a priori hypothesis and gets correlation-happy. He finds some very interesting results, which may even receive a lot of hype. And yet it’s the old Texas Sharpshooter Fallacy, finding a cluster of bullet holes and painting a bullseye around them.
Sadly, in most orgs there is a bias for certain forms of learning. Oddly, qualitative research tends to be dismissed out of hand, while less useful practices are unquestioningly “blessed” as near-monopolistic. This double standard exposes a troubling absence of the one research tool that really counts: genuine curiosity and the desire to learn our way past current assumptions.
The illusion of objectivity of quantitative results ignores that they are ultimately produced by a series of QUALITATIVE decisions. Whatever stat being touted is the result of a long chain of such decisions, from how variables were selected, to how they were operationally defined, to the research protocol and/or experimental design, to how data were collected, cleaned, sampled, as well as the choice of analysis used.
Quantitative data are only ever as good as the decisions made to produce it, and this is true regardless of the cost. Spotlighting this, like much else, may require culture change. If still rewarding people for creating more output, more work, and more busyness, then we’re essentially incentivizing the generation of waste, both in the form of largely unneeded features and solutions as well as all the BS data gathered to paint such efforts as “wins.”
BS Research Requests
We live in the age of “big data,” and yet while executives run around shouting that everyone should “learn to code,” apparently no one should be learning how to interpret data. This ignorance generates waste. After all, if a certain proportion of data is waste, then so is the research that produced it.
One way to cut down on this is to wise up and start saying no to BS requests for research. Frankly, research requests are often a game (see Berne, 1964). This means there is a con (the bait), an ulterior motive (the real reason for the interaction), and a cross-up (when the truth comes out). (You can read more about games here.) The setup is the invitation to use research expertise supposedly to resolve an open question.
Common ulterior motives include: 1. Wanting to increase confidence in a decision already made; 2. Wanting to procure data to persuade others concerning a decision already made; and/or 3. Wanting to put on a show that executives take research and experimentation seriously.
The cross-up is when the stakeholder pivots to attacking the researcher for failing to proffer the results actually being shopped for. Learning to sniff out this trap and turn down the game up front is another way to reduce waste.
We can frame this in terms of Savage’s Sure Thing Principle, mentioned above. Again, if A is going to happen if X occurs and A is going to happen if X does not occur (~X), then X is informationally irrelevant to A. Let’s say A is a decision and X is research. If the research is irrelevant to the decision-making process, then it’s waste. This is often the case in orgs with HiPPO prioritization, where the “research” done is largely theater. (HiPPO stands for “highest paid person’s opinion.”)
Common Tells
OK, so let’s say you’re the researcher and I’m the stakeholder. For your own toolkit, here are some common “tells” to look for, some common signs that something is up. Call them “smells.” They tip that either I have an ulterior motive, am basically setting you up to fail, or, perhaps more likely, I just don’t know what I’m talking about.
Let’s say I…
…judge the usefulness of data based on its alignment with a particular view. (Am I a lawyer?)
…tout another result over yours, with no interest in how either were generated. (Again, am I more interested in learning my way forward or in arguing a case?)
…express dissatisfaction with the findings, without discussing methodology, and want more research done to “fix” it. (You’re not here to inform my decisions—you’re here to provide me with marketing material! Get with it! The research will continue until the findings improve!)
…have some impressive stats on a PowerPoint slide but dodge questions on where they came from. (I’m acting like a politician.)
…keep sampling stats to fit a desired narrative, at odds with the general conclusion of the overall research effort. (I’m cherry picking.)
…want to know whether your results are “statistically significant” but can’t carry a coherent conversation about confidence intervals, effect sizes, and practical significance. (People who don’t understand statistical significance shouldn’t be basing any decisions on it.)
…make a big deal of the significance of some result, ignoring effect size and practical significance. (Say I significantly increased satisfaction, which isn’t predictive of behavior anyway, and let’s never mind I only increased it from 60% to 63%.)
…keep claiming something is predictive of something else, based on a correlation of r = .2 or .3 or some other common result. (If the correlation is r = .25, an almost medium effect size, the degree of overlap between the variables is r2 = 6%, hardly justifying any claim of “predictiveness.”)
…think that significance is the odds the results are due to chance. (Though even many researchers think this, it’s not true.)
…think that if a result is significant, this means the same finding will be significant again if the study is repeated. (It means no such thing.)
…think, for whatever reason, that a single significant quantitative result is more noteworthy than qualitative data from research done right. (Is my bias showing?)
…criticize your qualitative research for not having a “representative sample size.” (Blatant statistical ignorance is often no obstacle to feigned expertise.)
…tout a very interesting significant result that contradicts the qualitative research done to date. (Dig further. There are good odds the touted result is false. See Twyman’s Law.)
…want you to ask users/customers/stakeholders what to build and call it “research.” (Sure, we do user research—we have this guy who is expert at the system and he tells us what the users need. He’s a “super user.”)
…want you to ask people to predict their own future behavior and base decisions on it. (Yeah, they told us which design they like and what they’ll do with it. We’ve covered our asses, right?)
…want research artifacts created (personas, experience maps, etc.) but then do not use them to drive decisions. (But they do look so nice on our cube walls, don’t they?)
…push a particular way of aggregating the data, when other ways paint a different picture. (Say what you will. When I break the data’s back and force it through this inappropriate analysis, it supports the decisions we made behind closed doors six months ago.)
…discount alternative analyses and interpretations of the same data without good reason. (What do you mean interpreting data is always a qualitative undertaking? Quit blathering on about logic and whatnot. Besides, I like discounts.)
…ignore relevant variables when including them changes the results. (Well just look at our velocity! Stop talking about wait waste and cost of delay!)
…ignore glaring confounds. (Confound schmonfound. Who cares if all the participants who don’t drink can’t for medical reasons?)
…seem to think making scientific conclusions requires inferential statistics.
…think data can replace critical thinking and logic. (Hold it right there. This is the age of big data, remember? It’s certainly not the age of logic and critical thinking!)
Hall’s tweet above is spot on. Often the very people insisting on “more data” aren’t qualified to interpret the data being asked for in the first place. (I personally wouldn’t call that out though 😊.)
The main takeaway here is that those doing the research need to get better at vetting their stakeholders. We have a right to interview prospective clients and if it doesn’t look like the research will be value-adding, why do it? We’d only be setting ourselves up to fail.
Avoid research theater.
Research theater produces data waste.
References
Berne, E. (1964). Games people play: The psychology of human relationships. NY: Grove Press, Inc.
Hall, E. (2015). On surveys. Medium. Retrieved on April 30, 2019 from: https://medium.com/mule-design/on-surveys-5a73dda5e9a0.
Savage, L. J. (1954). The foundations of statistics. USA: John Wiley & Sons, Inc.
Charles Lambdin 
One thought on “When Your Data and Research Are BS”