Rage Against the (epistemological) Machine

This is gonna seem incongruent with my last post, but it’s not. Well, mostly not.

Most “scientific knowledge” is not true.

“What?!” I can hear you yell. Let me explain:

I’ve run into this problem a few times in a range of scientific literature, but this is the most salient example I’ve run into.

In 2019 I was tasked with helping to start a drug development program for a cancer drug, with the goal of helping the immune system fight cancer. In short, there is an immune cell type called “macrophages” which have an important role in regulating the immune system. Normally, these cells eat bacteria that get into the body, eat dead and infected cells, and make sure another cell type “T-Cells” don’t go too crazy in clearing up infections. When you catch the flu for example, much of the symptoms, fever, fatigue, etc. are a result of your T-Cells doing too much. Macrophages sometimes help in directing the T-Cells to the right place, but sometimes they help by turning off the T-Cells. In cancer, macrophages are thought to do too much turning off of the T-Cells, which hurts your body’s ability to fight solid tumors.

The idea for this drug was that we’re going to modulate macrophages in the tumor. There are lots of things that make tumors a hard place for T-Cells, but the fact that tumors are acidic is part of the story. Macrophages change their gene regulation in response to an acidic environment, and there’s some evidence that acid tends to make macrophages more immune suppressing. There is a protein that allows macrophages to detect that they’re in an acidic environment, and the idea was: we’re going to block that.

My early experiments looked good, some of the classic markers that are used to tell whether macrophages were immune suppressing went down in response to modulation of my target, and some immune activating ones went up. So the project progressed. The next stage of drug development is ramping up what’s called a compound screen, where you try to find some starting material from which to come up with a medicine. This is very expensive and time consuming, and in the mean time I was given the task of characterizing the role of our target in macrophage behavior. Enter: The M1 and M2 macrophage.

The M1 and M2 macrophage is what’s taught in intro cell-biology classes and graduate level immunology classes all the same. Textbooks exist on M2 macrophages. Only problem, it’s complete dogshit.

The idea is that we start with one macrophage population, M0, and then, depending on the signals the macrophage encounters it either becomes immune stimulating (M1) or immune suppressing (M2). This idea is ubiquitous in the discipline. Papers are published constantly on whether a certain signal or disease makes a macrophage more M1 or M2 like. Companies are founded off of M1 and M2 macrophages.

The concept came from a 1970’s paper where some scientists took monocytes (which give rise to macrophages) out of a mouse, grew them in dishes, and treated them with an immune activating or an immune suppressing signal (cytokines, immunologists will know them as IfnG and IL-4). They observed that different genes get turned on in response to these signals. One gene in particular, arginine, seemed particularly important. If they modulated arginine in a live mouse, they showed that they could make the mouse better at fighting tumors.

The field exploded with these types of studies. Lets throw other immune stimulating signals (LPS) and suppressing (IL10, TGF-b, IL-13) signals on them and see what genes get upregulated and what get downregulated. The field lacked sufficient humility to realize that there is not a single gene that all of the immune suppressing signals activate and in fact, many of the genes that are turned on in response to these immune suppressing signals are also turned on in response to immune activating signals, and vice versa.

And the arginine metabolism? Human macrophages don’t even express arginine, in other words, the core finding that started the field is completely irrelevant to humans.

Eventually the false framework seemingly became too wrong to do any science in. So the field fixed it, with M1, M2a, M2b, M2c macrophages. They realized that the dichotomy was not useful, so they just added another couple buckets. “It’s a spectrum” they say. “No it’s not” I respond.

Actual, good, scientists have tried to figure out what’s going with macrophages upon stimulation with one or combinations of these “M1/M2” stimuli. And it does not fit within the framework. My favorite paper on the topic treated macrophages with a bunch of these stimuli and sequenced every gene being expressed by macrophages in response to them. They realized the macrophage could be better modeled as interactions between a bunch of gene regulatory systems. Instead of moving in 2D M1-M2 space, the macrophage exists in 9D space.

A later paper, from an incredible lab, Dana Pe’er, did single cell RNA-seq on immune cells in a tumor. This means they can see every gene being expressed by a single cell. This is an extremely powerful method, that starts to allow us to actually figure out what’s going on in a cell (although I’m sure will seem primitive in 20 years). Their findings on macrophages:

Every recent quantitative study finds the same thing; M1 and M2 macrophages do not exist, and the concept doesn’t describe macrophage behavior. Dogshit.

So who cares?

The problem is not limited to our understanding of macrophages, but with the system of science. I find these kinds of issues nearly every time I drill down into a specific area of biology. While my expertise does not extend past biology, it is my understanding that this is going on in all fields of science. This Roger Penrose interview on quantum physics, and string theory is illustrative of the broader issue.

What tends to happen is: a phenomena is complicated and difficult to explain, and perhaps unexplainable given current technology. Someone finds something that offers a partial answer to the question, or at least operates by the rules of an imaginary system that allows further description of other phenomena. The new idea generally doesn’t require revisiting too much previous knowledge, and is very convenient to apply. Other scientists adopt the idea, and mold it to fit their needs. People become “experts” on an imaginary system. The “experts” publish papers, which are accepted by the peer review of other “experts”. The new “experts” become professors, and thus the new generation of instructors and peer reviewers, and we get very stuck.

Once these things get put in place, its very hard to dismantle them. Papers that go against the grain or do not pay sufficient dues to the work of the previous experts will not get published, unless they are so technically proficient that they are undeniable. There is a significant cost to this, as people who point out that the wrong thing is bullshit too early in their career are going to have a hard time getting funding and staying a scientist. Sloppy work that is not too disruptive will still get published, as long as it fits whatever the peer reviewers already thought was true. Publications lead to more research grants granted to the lab that publishes, and the establishment of new, funded grants to study the imaginary phenomena more deeply. Professors, seeking accolades and achievement continue to push and find things that fit within the imaginary system. These professors know a lot of words to describe their actually non-existent phenomena, and anyone who thinks it’s wrong simply doesn’t understand it as well as them.

This is why most biological studies are not replicable. This Slate piece does a nice job of communicating the scale of the issue. Biologists are working with complex systems in which we cannot see 99.9% of the things going on. If you run enough experiments, with enough conditions, you will find something that fits within the imaginary system. But for a system built on teetering, fragile, and hyper specific knowledge, good luck trying to make it happen again. The problem (mostly) isn’t dishonesty, or taking bad notes, its that our understanding of how things work in biology is fundamentally lacking.

BUT: most of our knowledge being wrong doesn’t mean that we cannot generate true knowledge, or that all knowledge is wrong. Well controlled experiments will generate knowledge with predictive power. The fact that some studies do repeat is evidence of that. With a careful eye and a skeptical mind it is possible to read a field of literature, and find which assumptions are likely to be true, and design experiments to test the other assumptions. The imaginary system hits a wall, as its predictive power is limited by how true it actually is. The real system that underlies it does not hit a wall until it has perfectly described the physical phenomena.

Read papers with the presumption that it’s not true, and only allow yourself to be convinced by the data, never the text. The supplementary figures are where the real data is.

Eventually, these false systems fall apart. This is Thomas Kuhn’s concept of a paradigm shift. The knowledge we have from published papers is a smattering of cause and effect, but the structures constructed around them do not describe the true underlying behavior of the system. But, eventually, the tools to investigate phenomena get better, and the old guard dies; clutching a pipette and a rejection rubber stamp. The arc of science is long but it bends towards truth.