One thing I’ve noticed about pop-academic books is the sneaky way they use studies.
I don’t think it’s intentionally sneaky, as you’ve got to use what’s available and if your only conclusion was “It’s unclear”, you wouldn’t have a book. Still, it’s easy to over-value the conclusions of books that lean on lots of research because of the sheer volume of studies cited.
Often each individual study is fraught with problems, and the authors sometimes admit as much. You might read something like this,
“One study, which didn’t directly examine this topic, but was following a sort of similar thing 20 years ago, although controversial and some say non-reproducible, might not have had the proper data collecting methods and didn’t control for X,Y, and Z, but seemed to show a relationship like this, and that kind of seems to imply this other relationship I’m talking about. Of course it’s not definitive so take it with a grain of salt, but even if just roughly, directionally true, it at least doesn’t disprove my theory.”
OK, so that one instance is fine. The author admits it’s not a great study or definitive proof, but throws it out as sort of some tiny sliver of possibly admissible evidence. You take it with the recommended grain of salt and move on. Then they cite another study with a similar disclaimer about it’s flaws and limitations. Then another. By the end of the chapter, 20 individually shitty studies subconsciously pile up in your mind as one big near-definitive proof.
But 20 studies with a low probability of proof don’t add up together to equal one with a high probability of proof.
The disclaimers are responsible, and it’s not out of bounds to cite as weak supporting evidence an imperfect study. Still, if you stack up enough of them, it creates the impression for some kind of powerful “Every smart scientific person who has ever done a study has found this conclusion or something like it and you can’t escape it” that is unwarranted.
I don’t know that the writer can or should do much about this problem, except maybe build the bones of the argument with logic alone, then cite studies only as examples of possible manifestations of the theory, instead of beginning with the studies. But it behooves us as readers to beware. Adding two or ten bad studies together doesn’t equal one good study.