Product Management is Pre-Science
We have to be humble and honest about our field. It’s not science.
I am actually curious how many will find this to be controversial or obvious in our field — there is definitely a mix. But, whether we are talking about our markets and products themselves, our product teams’ operations, or our product development practices — it is not scientific.
“Not scientific” maybe doesn’t even mean much on its own; and, yet, importantly, there are opportunities to provide more value when we declare this clearly. For example, staying humble when considering big decisions or when we choose where to spend our energy.
Product management and development are in a pre-science phase. Similar to pre-science medical theories of the past like bloodletting, humans giving birth to rabbits, and miasma theory. This should give us pause, those obsolete ideas are now known to be wrong and outlandish, far from accurate… and that’s our comparable for current “best practices” and “data driven”.
So let’s be humble, curious, and questioning with our ideas today — because we really don’t know. To illustrate our not knowing, let’s compare to some theories and people from the past, as well as, to simply brewing good espresso today.
On the one hand, our current product data, theories, and practices might be close to science; and, on the other hand, we might be very far off. We need to avoid the traps and temptations of certitude (usually feigned certitude in practice). This often goes awry for product folks when confronted with another business unit or competition who claims they are “certain”. We then put the pressure on ourselves to say something is certain when it isn’t. When it reality it’s okay that it’s uncertain. In place of certitude, enjoy the journey, the discovery, and the exploration. This honesty and clarity will provide greater value today to your markets, teams, and stakeholders. Moreover, we can all do much more towards actually improving scientific understanding in product management/development when we admit we just aren’t there yet.
Moving forward efficiently relies on clear shared understanding of what we know and what we don’t. Historically, even the greats got some of it wrong. It’s not something to be afraid of. Failure is okay. We can move the whole industry forward and still get a bunch of it wrong along the way. The point is we need to acknowledge we aren’t there yet and then increase experimentation until we do get there.
Going to examples from the pre-science past like Pythagoras the great mathematician (see mystical Pythagoreans’ beliefs for where he went wrong), Plato founder of higher education (see all matter made of earth, fire, air, and water), Hippocrates founder of modern medicine (see four humours), and Newton (see alchemy), all of them proved to be way off in some of their ideas; and, even when following more of a scientific process Einstein (see quantum physics) and Newton (see gravity) were still slightly off with some of their physics. Even these great thinkers had some degree of ignorance, but showed we can contribute to our craft today and move closer to scientific knowledge. Acknowledging our ignorance and scientific adolescence means we don’t look bad by not knowing. So we don’t have to feel embarrassed, less-than, or apologetic. Nor do we have to blindly follow something or someone that presents itself as accurate today. There’s little benefit in following blindly when we don’t know. We move forward more efficiently when we acknowledge what we don’t know.
Bringing this into product terms. Data driven and data informed are a couple of frequently used terms used in today’s data/scientific product context. Let’s put these out there as a summary to set the current stage in relation to scientific knowledge.
- Data driven (a “hot” term these days) simply leads us to bad decision making. It’s almost never data derived from scientific method. At best, most data we collect leads us to averages (usually the mean) of where markets have been, not where markets are going. And, if you build for largest average, there isn’t actually anyone who fits “average”, so who’s it for and is this really the pinnacle of our “science”? Furthermore, because of bias or science/data immaturity, we often can make data say what we want (consciously, but often unconsciously)— which helps us “know inaccurately” instead of merely “wrongly assume”. And when we “know” we tend to make larger (off-target) bets due to perceived increase confidence and reduced risk — and we simply lose more.
- Data informed is much better than being data driven, because it includes an aspect of art. Here we admit it’s not science, which is great. But, then, why aren’t we working towards more scientific practices? Also, most folks I talk to are not comfortable with it being art. But let’s get comfy with the art aspect for now, because it’s important to acknowledge this is where we are today; and, if you’re not, then at least work to get here. This is actually where people in product today can make a huge impact by understanding markets with deep empathy for their problems and then creating a shared vision — a vision derived from empathy and tempered with expertise, good taste, and principles. So cool!
How did we get here? Why is product management/development pre-science instead of scientific?
- Seems like an interesting area to explore one day, but not a focus of our time today.
- Maybe it’s too early? Maybe it’s too many variables? Maybe a correlation between top-down companies and top-down deduction, as opposed to, a bottom-up induction through scientific method? (please comment if you have ideas)
Although we aren’t going into how we got here for the product discipline, a good use of our time is a modern analogy. I think there is an interesting parallel in espresso and the origin story of the Decent Espresso machine. They set out with a pretty common goal for the field, to make a great espresso machine. They wanted to pair the taste of espresso from Italy with the technology of the coffee industry in Seattle. In sum, they realized the problem is they didn’t have the necessary data. Not just them. Everyone! There was simply too much information and too many variables to “know scientifically” how to do it; thus, it was mostly an “art”. That is, we don’t know scientifically how do we get good tasting coffee across all the different beans — and then control it during brewing. Why? Because it’s damn hard. The solution for Decent Espresso, invest (a lot) in making it scientific. To my understanding, this is revolutionizing the espresso field. Both collecting data (from experts) and feeding those data back to all users.
In most product management/development, we face the same issue of not being scientific. Our industry, like many espresso machines in the market, work in a certain pseudo-science. We pull some levers, shoot some steam, and wave our hands around — it looks good. But it’s not getting us closer to repeatable process.
So how hard would it be to create a repeatable process for product management/development? To continue the espresso analogy, if you’re a curious amateur at-home-barista like me and you tried making espresso, it’s kind of confusing why this process isn’t clearer, simpler, and repeatable. I get different results despite the fact that I have the industry recommended tools: top recommended equipment/tools/instructions, beans measured to tenth of grams, a consistent grind, brewing at a controlled temperature, and outputting the same weight of coffee. And, yet, all the variables aren’t perfectly controlled and there are more variables we aren’t controlling for. It is damn hard to do, but we lose clarity and opportunity for progress when we look past or accept these shortcomings. Of course, product management/development has many more variables, almost none of which we can control, so it’s no wonder we aren’t even close to a repeatable scientific process.
Although Decent Espresso is gathering data to get much closer to scientific answers with espresso (I am looking forward to it), those sorts of data are still a long way off for Product.
Since we in Product aren’t there yet, depending on your measures of success, trying to be more scientific with tools we have today often doesn’t make us better. It potentially even leads to making worse decisions at a higher energy cost. To illustrate a gap from scientific knowledge to what is typical for product teams, maybe try to imagine scientific brewing (link to Descent Espresso if it helps) compared to a product status-quo equivalent (I’m confident a lot of people can relate to this at one time or another) of not knowing what beans your market wants, not knowing what beans you put in the espresso machine, you know the average taste of the espresso you made over the last quarter, you know often people have your espresso, and your best (most scientific?) indicator of success is if someone hypothetically would recommend the espresso to another. I hope it’s clear these aren’t very close to a scientific approach. We can do much better. Spending more time on this type of information, or using the equivalent of one or even all of these, should be proportionally limited to the information’s usefulness.
Moreover, where this can get deeply worse is when we pretend these types of data are scientific knowledge. We are rarely, if ever scientific, so it’s relatively safe to assume you are not being scientific. To give a bit more colour to how we don’t follow scientific methods: our hypothesis are rarely isolate enough to be truly measurable, often the testing and measurements are pseudo-random recruitment/assignment, we rarely replicate our studies with different contexts, samples, and experimenters. Even when we do “more scientific” we often have massive sample sizes and only look at p-value (the difference between groups is not due to chance) and we (often conveniently) ignore effect size (the amount of difference attributed to the treatment). So, again, an improvement that a lot of product teams can make is simply remembering to be honest with ourselves, it’s not science. And, since we are often so far from science, we can improve by asking, how do we best use our time? More science or less?
Side note: Other areas related to scientific practice like hypothesis clarity and research questions are not the most relevant to get into details for this topic; but, worth mentioning because they are related and hugely valuable. So, two areas to spend more time are: even when we have a hypothesis, that hypothesis is rarely clear to anyone/everyone involved; and, even if we did follow scientific process well, we are rarely validating that we are asking the most valuable research question to begin with. So a partial answer, one option I recommend is more time here.
I see such great opportunity here for product management by merely creating a clear shared understanding of where we are today with your products, teams, and stakeholders. We can free up a lot of energy in many of our companies by understanding we are pre-scientific. Also, this will result in better decisions being made and done so more efficiently. Win-win-win, that’s the dream!
The other opportunity here is the gap analysis leaves loads of room to improve our system with this realization (or remembering) we are in a pre-science state. With this position, we can continue to evolve and invest in more scientific ways to measure where markets are going, how product management teams operate, and how products are developed. If the state of the art is data-informed, let’s not rest on that.
Maybe, just maybe, we can all work towards spending less energy crunching pseudo-scientific data that tells us things like where market averages where last quarter and instead repurpose that energy to take our industry into a scientific future. What an exciting time and place to be!
