4.25 years in the sausage factory / what’s next for Dr. Olek PhD
As many know, I just defended my dissertation.
I am not quite out of the woods yet, as I’m staying on a 6 month post-doc contract, which was done to let me finish up the work that I’m most excited about. Defending when I did was mostly a practical choice as tying up all those loose ends is a significant time sink. Now, I can just do the science, write this paper, get paid a bit more, start scheming out the next moves, and patch up the holes in my condo to get ready to sell on my way out. The short-term post-doc also sets a time limit on myself, so I don’t let it drag on. After six months max, I will be out.
Even though I’ve only got a couple toes out the door at the moment, I think the letters now in front and back of my name give me enough credibility to write about the incentive issues in academia, and how they manifest in poor outcomes for science, biology especially since that is my domain.
A saying from some random British economist has really stuck with me: “the purpose of a system is what it does.” Further, there is “no point in claiming that the purpose of a system is to do what it constantly fails to do.”
The purpose of academic science is to produce scientific headlines which titillate a chimeric scientisto-cultural expert without too much regard as to whether they are true. Sometimes they’re true, sometimes they’re not. Increasingly it is also to support a political consensus and position some perspectives as fact based, and some as anti-science.
How does it achieve its purpose?
Most importantly, the current research publishing system creates counter-incentives to motivate rigorous work. Publish or perish is the refrain. To count at all, the work you do has to be published in an academic journal. It can be years of poorly compensated work to get a publication at all. The only relatively proportionate reward you can get is a publication in one of the top journals, like Cell, Nature, Science. These are so vaunted that the set has an informal acronym: CNS. Publication in those journals earns you a lot of clout, maybe enough to become a professor. I am confident the last paper I’m working on has a good chance to be in one of those journals, so I may pull some punches here, but, this system creates problems.
It takes a specific type of work to even be considered from review in those journals, as the first step is editorial review. Basically you send in the paper, and the arbiter of cool decides if it’s splashy enough. If it’s not splashy enough, you get rejected here. Work sounding cool to many scientists is valued more highly than the quality of the evidence provided for claims. This can be good in preventing useless research, but pushes scientists to compete on the same few sexy topics with sketchy data. It also motivates arriving at certain results prior to actually doing the experiment, which I’ll get back to.
Here's a few linked examples from a big name in the field of evolutionary biology; Andreas Wagner. All three of these papers make big claims on the nature of evolution but rely on experimental conclusions that interpret data in the most “generous” way. In case anyone wants to get in the weeds, the first one in Science is more explainable by the “neutral drift” being weakly selective and increasing the frequency of a specific mutation prior to the evolution. I’d say there’s enough in the paper to conclude this is the case with 80% certainty. Another RNA based paper in Nature by him has the same problem. The last in Science just chooses a bottom threshold which allows through some noise and uses that to make conclusions about the signal. That one I’m sure of and would undermine the main claim of the paper about the ruggedness of landscapes.
While I react like this when I see yet another paper by Wagner in CNS with this type of problem:
This is not unique to Wagner, and uncovering more outright fraud in high profile research has become common. The absolute biggest names in science have this issue (though once you’re running a massive institute it’s hard to imagine the big names are involved at all in reviewing figures). People may have heard of the recently unraveled Alzheimer’s research fraud, committed over 25 years by a professor and top NIH official overseeing $2.6 billion in funding. This research has resulted in big drug trial failures, companies founded on fraudulent science. People got antibodies injected into their blood based on this fraudulent research. There are other, similar scale frauds in Alzheimer’s research being unraveled at the moment.
Theranos was an example that seems kooky and camp in hindsight, but at the time was supported by huge names in science like Eric Topol who runs a very well-funded institute at Scripps and was constantly vaunted as an expert during COVID.
People imagine this as a few bad apples, but the issue is built into the incentives. If you pick a sexy topic where you can come to the “right” conclusions and design clever experiments, you have a lot more “flexibility” in analyzing your data. This produces a weird game, where its optimal to be the exact right amount of rigorous to get the paper through, but not so rigorous as to accidentally make obvious the more trivial and correct explanation for your data. If the reviewers already think your paper is factually correct based on the title and abstract, they’re going to be less critical and pushy on rigor than if they don’t like what you’re saying. It’s human nature.
This game is obvious if you think about the design stage of projects. You always have to consider to whom the work would be interesting, which ends up being judged based on prior papers in the field. Then you consider what you’d data you’d have to generate to evidence the result that would be splashiest and most guaranteed to get published. There’s nothing wrong with that in theory, but in practice you end up chasing a specific outcome before you know if it’s based on reality.
This system of judging the value of science counter-incentivizes doing rigorous work, especially in the case of small papers because they are hardly worth it at all. Disproving errors in the field will just not happen until the house of cards gets too high, and investors start to lose money.
Even the most august journals have problems like this because to sort out technical issues in those papers took me hours of poring through the supplementary data. Very few people do that, not reviewers apparently. Now that I have this information, I can tell my lab about it. For the most part, they are not that interested. They’re busy, and it’s “much easier to fool people than it is to convince them they’ve been fooled.” People don’t like finding out the paper they enjoyed was a con, it’s unsettling.
Do I really want to write to Science and butt heads with a very senior academic? Do I want to use my hands to do extra work to try to rigorously prove my concerns are the better interpretation? When the best-case scenario is that I get clean data with which I can’t get a big publication, of course I don’t.
There is a nice explication of what lots of crappy data means from this paper on “the paradoxical nature of easily improvable evidence.” It uses Bigfoot as an analogy. We have a lot of really terrible grainy pictures of bigfoot, which have not improved alongside improving camera technology and ubiquity of camera phones. It should be easy to prove bigfoot exists with a good picture. The fact that the evidence hasn’t improved means that the thing it’s meant to prove is unlikely to be true. When the evidence for a big scientific claim is rife with suspicious artifacts, it’s evidence that the claims are the opposite of the truth. Bigfoot isn’t real.
So with pools of flawed data all you can do is update your own mental model. That is useful. If the experiments are well designed and the data just came out differently than they claim, then you can invert the claims of the paper and accept that as likely to be true. It’s the opposite of what was claimed for the purpose of getting into CNS. Neutral drift does not facilitate evolution in the narrow setting of a few mutations around a sequence in evolving to a specific function. Landscapes are smooth, and evolution of a single protein / single function converges to the same place. Other incorrect science like the phenomena of biased signaling in GPCRs, macrophage polarization, persist. Understanding this does give you an edge among scientists though, and you can see questions other people can’t.
Because the claims are likely false, but apparently popular among reviewers, you should just try to avoid working directly on these areas. You will need to do very rigorous work to get it published to the same level as the incorrect work that inspired it. No nit will be too small to pick because the reviewers already think your work is wrong and they don’t want it published. Reviewers are generally people who published related papers in the past, in this system which rewards fitting in. So sometimes it’s personal, because your paper would be undermining the past work of the reviewers directly.
To the degree that people understand this problem, it should be considered as an existential issue for academic science. It makes it very challenging to figure out what is supported by data. To accurately gauge the issue, you must be apprised of the various popular positions of the field, and whether something is intuitive or unpopular among experts. That can only tell you where there might be bias in publication, not whether the bias is justified.
To know with more certainty, you must be familiar with the techniques used and have many hours to spare. If a topic got politicized at some point you don’t have a prayer of getting to the bottom of it without expertise or time to develop some expertise. If you arrive at the conclusion that an unpopular position of a politicized topic is more likely to be correct; you can expect to be labeled as fringe, schizo, denier, etc.
There’s not much worse that you can do for your research career than to arrive at an unpopular position. This phenomenon is most evident in climate science for people like formerly decorated atmospheric physicist Richard Lindzen at MIT. He is pretty dramatically out of step with the consensus on CO2 driven climate change but had a successful academic career before then, publishing 200 papers, Harvard undergrad, National Academy of Sciences, high h-index, etc (awards below). Aft ect, he describes loss of research funding, and issues publishing papers. According to him, on two separate incidences, editors who let his papers through to publication were fired immediately after publishing as a result of external pressure. It’s easy to see why that might motivate editors to not let through his papers and would dissuade other researchers who might agree with Lindzen from voicing their opinions.
Another example is Judith Curry, a professor who formerly ran the Atmospheric Science department at Georgia Tech after advising the NOAA Climate Working Group, and National Academies Space Studies board on climate. She became the target of climate consensus after criticizing the science behind the famous “hockey stick” graph showing recent unprecedented warming. In reality, the most famous version of the graph, used by Al Gore in “inconvenient truth” is the subject of a lot of controversy because of shoddy statistical methods which mean you could generate the hockey stick graph from random data, and borderline data manipulation by picking and choosing datasets to exclude / include in generating that chart.
After raising and publicizing these issues on her blog, Judith Curry was maligned as a denier and was bullied out of her position at Georgia Tech.
Her views are largely consistent with the International Panel on Climate Change (IPCC). The IPCC is the body regarded as the international authority on climate change, and is pessimistic but hardly apocalyptic in it’s predictions. She was mostly targeted for her raising these issues to a layperson audience on her blog which other climate scientists said gives credence to deniers.
The arguments on climate issues veer from scientific to emotional pretty quickly, and I expect many reading this article may feel the instinct to start slapping the “fringe” label. I’d urge you to observe and resist that instinct and know that these credentialed people understand the upcoming career impact of publicizing their unpopular position.
It will make it really hard to do the thing you trained your whole life to do, and turn you into derision piñata. The podcast appearances of Lindzen and Curry (even if they’re on cringe media) make them seem quite sane, at the very least. I recommend both pretty highly as a counter-balance and example of how scientific disagreement in the current climate plays out. My response to these meta-dramas for which I don’t have relevant expertise is to dramatically reduce my certainty and make my views into a bunch of maybes.
COVID was also rife with this consensus enforcement, and I talk about it in other posts. It’s worth noting that Stanford Prof John Ionnidis, the most cited researcher of all time (who also called Theranos as fraudulent very early), was maligned as a industry funded crank during this period for pointing out uncertainty in the early data, and then accurately estimating the IFR of COVID.
For this reason, I express very little confidence about even the most “settled” of scientific consensus questions except where I have specific expertise. The strength of the consensus is self-reinforcing. Just saying “I’m not sure” about some of these scientific questions evokes emotional reactions.
Everyone has seen this picture of the bullet locations in returning WWII planes. The canonical story is that the dumb engineers thought this was because planes were getting shot here more for some reason. The smart engineers realized the other places were getting shot, but those planes weren’t coming back. This is relevant to science of course, since the papers with the wrong findings just aren’t getting published.
This means bad ideas will persist for long times, and PI’s who accepted the popular position won’t realize they may be sending their PhD students on a wild goose chase. Sometimes at year 4.5, the PhD student might have a dataset that can be “generously” interpreted to fit the mold of the other papers which made a splash. Even if they’re uneasy about it, they’ll do it. The alternative for the student is martyring yourself, which few people are impressed by. It’s just part of the game of science.
Beyond incentivizing arriving at the “correct” results, prizing certain journals so far above others counter-incentivizes development on technology issues. For one, papers which say “we improved the function of a thing that exists” have a lower upper ceiling for publication, aka it won’t get into CNS. Fewer people care, but that work is generally more useful in my opinion.
Also, if you fix a technology issue and characterize it well, the publication itself can be a bit of a black eye for the PI who made the tech, because it can be an admission that there’s a previously unknown or undisclosed flaw in their technology.
This flaw might have already cost other lab’s wasted time, which is embarrassing. It’s a short-term sting, for what you hope will be a much bigger longer-term boon since it makes the technology work better. There’s no way through it than to eat the sting. But you aren’t in grad school that long, so you won’t see many of the benefits of improving technology, you won’t get a big publication, and it will likely not make your PI happy. You have to be a really specific type of person to be so bothered by a technical issue that it overcomes those incentives. Speaking from personal experience.
Honestly, these incentive issues would be simple to fix by being laser focused on logic, rigor, and consistency. If little old Dr. Olek PhD can isolate crucial problems in the papers, the “experts” can do so too. Maybe a new AI model could be deployed to efficiently highlight potential issues like this to decrease the time sink of finding them. Maybe better incentivizing and reducing barriers to debunking bad science after publication would mean scientists and amateurs take the time to influence the record.
Better catching of data issues would disincentivize the current gaming of the system to produce sexy shadows. On a second order level the reduction in clout chasing means more resources / man hours could be spent on improving technology, so science can genuinely progress. The science would appear stuck and less sexy for a while, but it would slowly fix the system.
Unfortunately, those participating in the current system of journals, professors, reviewers, and grant writing agencies quite like the state of affairs. They made their way in it. Why would they risk their jobs and prestige to admit reform is needed and biology research has been filled with nonsense? It’s worth noting that there’s an incumbency bias in research funding. Overhauling the bad science would mean the luminaries of a field lose funding to new scientists. If a field is well funded and seems important, revealing that it’s it based on an artifact would be a big change in who can get paid to do research, as the funding for bad ideas (sometimes) dries up after they are revealed to be bad. The luminaries have a lot of weight to throw around in funding and peer review, and understandably don’t like that possibility.
Of course, in academia the practical regular workplace incentives are also missing. Normal working life is governed by a consent relationship. You’re hired/fired at the will of the employer and choose to work/quit at your own will. Quitting a PhD program, especially in biology, has a big sunk cost. From what I saw in industry, people with PhD’s were trusted with more resources and more important projects than people without them. Aka, you make more money and can build bigger stuff. (Unless you get away from the science side of biotech companies, of course. Science doesn’t pay that well anyway.) It’s unclear to me how much of it is self-selection, irrational bias, or if it’s a basic reality of hiring in a field where research is expensive, and mistakes are costly. I have my intuitions, and they are not that you should need a PhD to run big projects.
Most jobs don’t have a timed structure where you’re expected to be there at least 4 years but less than 7 years. At year 3 if you’re on track to finish on time, quitting falls out the window as an option. At that point, you just grit your teeth and do the best you can to get something out of it because you’ll have access to more options on the other side. Otherwise you wasted years of prime working age.
Now, if you are incentivized against quitting, and you know it, and your PI knows it, than your PI doesn’t have to treat you particularly well. Frankly, I’m a confrontational person and I get a lot of useful work done, so I have options to address situations. Still, not being likely to quit I tolerated more than I would have in a regular workplace, because you can’t fight every battle. Other people have much worse stories about PI’s than I do. Being worked insane hours, shafted in authorship of work, forced to do menial doomed projects, kept on for extra years of grad school, etc.
The other side of regular workplace incentives is rewards, like bonuses, promotions. You can’t really get either of those in academia. If you can just glom on to other people’s papers, then you can derive a lot of the benefit of the work with very little energy expended. At a good company, doing the unsexy work of development in addition to the sexy work of building, would be rewarded financially. If you don’t get your flowers at the company you’re at, you still develop a reputation as a valuable (if not difficult) person, who will be able to earn accordingly if you can navigate finding a company who needs someone like you. In academia, being friendly with the right people, not shaking the boat, expressing the correct scientific and political beliefs, are 45% of what matters. 50% of what matters is being good at convincing the government to give you US taxpayer dollars to fund the research and feed university coffers. 5% is producing true information that advances the human knowledge base.
So, briefly, back to me.
I have a couple papers pre-printed and submitted to peer review. Both are tech development, one was purely good, one was faced with the sticky issues I mentioned. I have a couple projects that made it 70% of the way and then just fell by the wayside and will likely never be submitted to a journal, but I will post on my blog because they did produce useful information.
I have one last paper coming which I am excited about and proud of. It sidesteps a lot of the contentious issues in the field, supporting some weird positions and some accepted positions. It has produced new, interesting, and true information. The system is useful and lets me do some stuff that no one in history has done before, dozens of times at once. It was my idea, resulting from my flagellum rabbit hole, and I built it (on top of a lot of great work done by Gordon Rix, thanks G dawg! And along with support from Alireza Tanoori, Yutong (Lexi) Yu, and Rory Williams). I think the technology might be useful for real commercial applications, but I’m not sure. It will be of interest to researchers for sure and opens new doors of research to humanity.
I am just starting to poke my head around and get a feel for what kind of opportunities are open to me. I have a refined skill set in building high throughput biosystems to extract useful information about proteins and cell behavior and am confident in my abilities measured against anyone worldwide in this domain. I’m looking for early to super early-stage startups which need to build out new biosystems, or for someone who might be interested in adapting the system I’m publishing soon for commercial ends (if you attended my defense, it’s the ORACLE work). Am interested in AI mediated protein / system design and new capacities there, since I think my system will have natural applications, but I haven’t yet put in the time to be sure how much of the AI wave I think is hype and how much is real.
Regardless, I am excited for the transition out of the sausage factory into the real world. More soon.