👋 Hello, I’m Tom — I’m the tech lead for Webflow’s Applied AI team. Our team’s mandate is to accelerate the development of AI-based features in the Webflow product, and recently we shipped our biggest AI feature yet: the AI Assistant.

AI Assistant will eventually have all sorts of “skills,” but today its main skill is creating new sections in an existing Webflow site. It lives in the right side of the Designer and looks like this:

I’m writing this post after our public beta launch at Webflow Conf 2024. It felt like a good time to reflect on the ups and downs of developing an AI feature at Webflow.

The AI Assistant launch was great!

This post spends a lot of time exploring the challenges we faced while developing AI Assistant, but – spoiler alert – I want to say up front that this story has a happy ending. The AI Assistant launch went off without a hitch and our post-launch product-market fit score (41%) surpassed all expectations.

Pulling this off required herculean efforts from the newly-formed Applied AI team. I'm forced to use the unspeakably cringey “flying the plane while we’re building it” metaphor — this team had to figure out so many new things about AI product development while at the same time tackling an extremely ambitious AI product.

As a result of this team’s efforts we’ve launched the AI Assistant and its first “skill,” section generation. In addition, we’ve accumulated the AI-specific infrastructure, processes, and institutional knowledge that will accelerate the addition of many more AI Assistant skills in the near future. A smashing success all around!

But we got sucked into many “AI traps”

Although it had a happy ending, the AI Assistant story also had its dramatic setbacks. And while all software projects have setbacks, I couldn’t shake the feeling that this project had unique setbacks — things that seemed specific to AI projects. The typical advice around software development, especially around agile software development, seemed so much harder to apply to this project.

I kept coming back to four “AI traps” that seemed to constantly nudge this project away from quick, iterative, experimental development toward large, risky, waterfall-style development. And while I think there were other factors that were outside our control (e.g., we simply couldn’t have built today’s AI Assistant using 2023’s state-of-the-art models) — these traps explain a good chunk of our AI Assistant delays.

Trap 1: Overestimating current AI abilities

It’s very tempting to assume that we’re about five minutes away from an AI apocalypse. Every morning I wake up to a deluge of Slacks/emails/Tweets announcing new state-of-the-art models, trailblazing AI tools, and magical demos. Even the AI detractors are usually in the “AI is too powerful” camp.

While the detractors might be right in the long term, right now AI is often very stupid. For example, GPT-4o still thinks that white text is readable on white backgrounds:

“The .paragraph class provides a white color for the paragraph text within the cards, ensuring readability against the white background of the cards.”

— Real GPT-4o response

This tendency to overestimate the current technology slips into Webflow’s roadmap planning. At many points during AI Assistant’s development we said stuff like:

• “…the quality scorecard needs to be 70% green before the April private beta launch…”

• “…let’s just train a model on [insert hideously complex domain here]…”

• “…it’ll take us around 5 engineer-weeks to teach the AI Assistant to apply existing styles…”

All of these claims had some supporting evidence, like demos, proof of concepts (PoCs), or maybe just saved ChatGPT conversations. But in every case we assumed that if we caught a whiff of a successful output from a large language model (LLM), we could eventually coax that same LLM to produce perfect outputs for every edge case.

It turned out we were extremely wrong about all of the assumptions above. It was way more challenging than we thought to improve the scorecard. Training models is slow, expensive, and difficult to do for broad use cases. And we’re still trying to improve how the AI Assistant applies site styles.

This tendency to overestimate AI abilities meant that our agile-looking roadmap with bite-sized goals — e.g. ”we’ll ship AI-generated gradients next week to see if users like it” — turned into a months-long slog to produce features that we weren’t sure our users even wanted.

Trap 2: Demo enthusiasm

This is the trap I fell into the most often, because I love demos.

And — to be clear — I think demos are a crucial part of developing AI features. But it’s far too easy to see a demo of “working code” and think, “This feature is basically launched! We just need to add some University articles, update some copy, and bing bang boom let’s SHIP IT!”

But here are a few things that make demos not shippable:

• They only demo the “happy path.” I spend more time talking to the AI Assistant than I spend talking to my wife, so I know exactly the right phrases to whisper in its ear to get it to work. This makes my demos look pretty good! But real users are very likely to use language that the AI Assistant doesn’t like, and will therefore have less success.

• They only demo the successes. This is the whole point of Loom’s “restart” button: if the demo gods abandon me in the middle of the demo I can just hit “restart”, and you don’t get to see the 50 failed attempts before the one that worked.

• They only work in special cases. For a while, the AI Assistant only worked on blank sites. Then it worked on a single template. Then it worked for a few more templates. But if you just stumbled upon a random demo you might think, reasonably, that the AI Assistant worked everywhere!

• They rely on a technology we can’t actually use. I did a whole bunch of demos that relied on TailwindCSS to create sections. These demos looked great, because GPT is a world-class expert at using TailwindCSS (since there are loads of great Tailwind examples in its training data). But the AI Assistant can’t just add weird Tailwind utility classes to everyone’s Webflow site, so these demos didn’t really mean much for the actual AI Assistant.

All of these “details” mean that the cool AI demo you just watched is a great starting point for more investigation but not an invitation to begin roadmap planning. All too often we jumped from “cool demo” to “okay the marketing launch is planned for April 4.”

Trap 3: The marketing moment

Speaking of marketing… what’s that huge gray thing looming in the corner over there?

Elephant in the room:
‍
‍Shipping an AI product isn’t only about solving customer problems, it’s also about marketing.

In the last couple years there has been enormous pressure on tech companies to ship AI features. No company wants to be “left behind” — still making carriages in a post-Model T world — so they scramble to ship something AI-related.

Amid this pressure, many tech companies quickly launched half-baked AI features that were universally mocked. This is a family blog so I won’t name names, but I’m sure you’re familiar with some of the botched AI launches of the past couple years.

I don’t know if any single person at Webflow ever said this explicitly, but our emergent behavior can be summed up as a reaction to these failed AI launches. It’s as if we said:

“Dang, we really need to launch an AI product to show the world we’re an AI company. But all these recent AI launches are pretty embarrassing, right? We should build an AI feature, but we’d better be sure it’s perfect and awesome before we launch it.”

— Webflow’s collective consciousness

And I think this was one of our biggest mistakes. This sort of thinking was crystallized into these product requirements:

• It has to use AI

• It has to be press-worthy

• It has to be nearly perfect before showing it to anyone

This leads to very waterfall-style development. We can’t cobble together a quick MVP and release it to a bunch of users, because then they’d make fun of us! And we can’t ship something small, because then we can’t have our splashy marketing moment.

And this flawed reasoning dovetails perfectly with the sunk cost fallacy, which says:

“Holy smokes, we’ve spent a long time on this project—it’d better be really awesome when we launch it!”

— Webflow’s collective consciousness, again

All of this led to a long period of pre-launch development on AI Assistant. And although I think that development actually went pretty well, it would’ve been much easier to make product decisions if we had launched something smaller, sooner.

Trap 4: Moat-driven development

Everyone knows that software is all about moats. You can buy an Airbnb clone for under $2k, but Airbnb itself is worth $86 billion — and most of that delta is because of its moat, in the form of its very-difficult-to-replicate two-sided market.

So if we want Webflow to become an industry leader — a la Airbnb — we need to build a moat, right? And an AI-powered moat would be even better!

I think this is reasonable logic — we should always be on the lookout for possible moats we can build to put distance between us and our competitors. But I’m wary of “moat-driven development,” where we start with the moat and then try to back ourselves into a beloved product.

Ideally, we’d build products at the center of this Venn diagram:

But this is hard! It’s pretty much never clear up front where a particular product will fall in this diagram. So the danger of “moat-driven development” is that you start with something you know is in the blue (“moat”) circle, and then you sort of flail around trying to find adjacent parts of the solution space that overlap the pink (“user”) and yellow (“AI”) circles. But if this doesn’t work out, you’re left with a totally useless moat.

Some examples of moat-driven development:

• “Let’s train a model from scratch to do [something really cool] — then we can figure out what product to build on top of it.”

• “Let’s use fine-tuned models instead of off-the-shelf models to do [something really cool] — then we can figure out what product to build on top of it.”

• “Let’s build our own [insert expensive infra here] — then we can figure out what product to build on top of it.”

Obviously nobody says the second part out loud, but it’s implied when the first part is suggested without reference to any particular user problem, or without a discussion of other (cheaper, better) solutions to a user problem.

And the thing about moats is that — pretty much by definition — they’re very expensive to build. If it was easy to build a moat, then everyone would do it, and then… it wouldn’t be a moat! So starting with the moat means investing a lot of time and money into a project that might never turn into a useful feature.

To me, it seems way safer to start by investing in features you know are in the pink (“user”) circle, and then try to see which of those things, if any, you might be able to nudge in the direction of the yellow or blue circles. Because even if you end up investing in something solely in the pink circle, you’ve still built a useful feature.

But we should still be “moat aware.” As we stumble through the solution space of features to build, we should always be looking for features that users want and might have moats.

This sort of “natural” moat-finding has already happened a couple times with the AI Assistant:

• Building natively. As mentioned above, building an AI demo that spits out code in a popular framework (e.g. Tailwind) is relatively easy. But these demos are constrained by (1) the limits of that framework and (2) the lack of access to site context (like classes, variables, and components). But by building a native AI Assistant within Webflow, we’ve created a tool that’s much more useful than the myriad framework-based UI generators.

• The Webflow Library. LLMs are bad at UI design. That’s why all successful “generate UI with AI” features have relied on pre-built components or UI libraries. But these products are only as good as their libraries, and building proprietary design libraries is expensive and slow. But we’ve already done it! Copying the AI Assistant would require copying our design library, which is protected IP.

In short: chasing moats can lead to long waterfall-style development of infra that we don’t know users want.

Avoiding the traps

Okay enough doom and gloom — let’s talk about the right way to build AI features!

How to avoid AI traps

1. Spend more time on product and engineering experimentation.
   Engineers want to build cool prototypes using new tech, and product managers want to solve meaty customer problems. But neither has all of the info needed to discover where that Venn diagram overlaps. The solution? We should allocate ample experimentation time for engineers to build quick PoCs that they can show to product managers, who can suggest refinements to those PoCs based on customer needs, etc. And we should spend more time in the experimentation phase making sure that the PoCs can eventually be turned into production-quality features.

2. Treat demos as support for more investigation, not roadmap planning.
   Demos are invaluable for communicating possible features to all stakeholders. But a demo is not proof that a feature is shippable. We should only exit the “experimentation phase” when we have other evidence that a proposed feature will meet our quality bar.

3. Ship MVPs — don’t chase a marketing moment.
   AI features tend to have far more degrees of freedom than traditional software features. A user can type anything into that chat box, and you’re kidding yourself if you think you can predict what they’ll type before you launch. That’s why launching early is crucial for AI features — there’s no way to guess what users will do. Luckily, the era of “AI marketing moments” seems to be fading — good riddance!

4. Start with the user problem — the moats will come later.
   We should all be “moat aware” — i.e., we should be on the lookout for possible moats adjacent to the current feature — but we should be very wary when someone suggests building a moat without referencing a particular user problem and the other ways of solving that problem.

Easier said than done!

Just to reiterate: I fell for all of these traps, often multiple times. I’m writing them down here not to point any fingers, but to help other product teams avoid making the same mistakes as they build the next wave of AI-based features.

I am incredibly proud of the team that got us here — and the product we ultimately shipped — and I can’t wait to use some of these lessons to accelerate Webflow’s future AI product development!

Know someone else who could use this advice? Please share the original Webflow blog post (not this Substack post). 🙏 Thank you!

The first revolution in physician references

The year is 1986, and Harvard Medical School professor Burton Rose just read a study he couldn’t believe.

Published in the Annals of Internal Medicine, this study of practicing physicians had a bold conclusion: “Only 30% of physicians' information needs were met during the patient visit, usually by another physician or other health professional.” In other words, 70% of the questions doctors had during their workday went unanswered—and doctors virtually never consulted a paper or textbook to find an answer.

This result shocked many—but it particularly bugged Dr. Rose, who had just written the canonical textbook on kidney disease. What was the point of publishing new textbooks if practicing doctors wouldn’t consult them?

Distraught, Dr. Rose was determined to find a way to supply doctors with the answers they needed to do their jobs. So he turned to the new hot new tech of the 80s: personal computers. Dr. Rose had played around with some early PCs and noticed two things: (1) they could store much more information than a textbook, and (2) they could be updated more frequently.

So he pitched his textbook publisher on the idea of a computer-based textbook. And…the publisher instantly shot him down. “Nobody is making money in computers,” they said.

So Dr. Rose took matters into his own hands and launched a computer-based textbook from his basement. The first version was a nephrology textbook, written by him, and distributed through the mail on floppy disks. Since he planned on sending out updated floppies every few months, he called his new company UpToDate.

More than thirty years later, UpToDate serves over 2 million clinicians and generates $800M in revenue for its parent company, Wolters Kluwer. It is far and away the most popular clinical decision support tool—the odds are very good that any doctor you meet in the street will have used it.

It’s a compelling story: a foresighted doctor glimpses the ways in which a new technology (personal computers) can revolutionize the practice of medicine; he seizes that opportunity; and he builds a software juggernaut that vastly improves the practice of medicine. My respect for Dr. Burt Rose runs deep.

But UpToDate’s revolution happened in the 90s. Thirty years later, Dr. Burt Rose would find today's web-based version of UpToDate eerily familiar. Sure, updating articles doesn’t involve FedExing floppy disks, but the “search and skim” workflow has barely changed at all. UpToDate is still just a textbook you can read in your browser.

And yet—technology has changed dramatically in the last 30 years! Shouldn’t there be faster, easier ways to get clinicians the information they need?

I’m so glad you asked.

Ways I would improve UpToDate

Improvement #1: Make it more up to date

When UpToDate launched in 1992, updated articles were shipped (via floppy disk) to users three times a year—and even today, most articles I’ve looked at were updated 1-2 months ago. This was blistering fast when UpToDate launched—back then, medical information doubled every 19 years. But today, medical information doubles every 73 days! Imagine searching for information about COVID in May 2020—would you trust results that were two months out of date?

A better version of UpToDate would integrate answers from new studies and guidelines as soon as they were published. This wasn’t even remotely possible in 1992—when most resources were still paper-based—but today this would be much easier to accomplish.

Improvement #2: Generate one-sentence answers

Most of the time, doctors are looking for specific answers, not full-length articles. A doctor who searches “amoxicillin dose for 6yo with otitis media” is not trying to read a 10-page article on ear infections—she’s looking for a one-line answer like “the recommended amoxicillin dose is 40-45 mg/kg every 12 hours.”

Again, this wasn’t possible in 1992—but improvements in search and natural language processing (NLP) have made it possible to give instant answers to specific questions (just try Googling “what is the molar mass of aluminum sulfate” for an example).

Improvement #3: Allow users to add their own references

Often, doctors are part of hospitals or medical groups that have their own guidelines. This means they have a two-step process for answering any question: (1) check the giant pile of PDFs in your hospital’s "guideline" Dropbox, and then (2) check UpToDate for general guidelines.

But it’d be way easier if doctors could just upload their guidelines to UpToDate, and then let UpToDate return answers from those guidelines. Putting all relevant references in one place makes it way less likely that a doctor will accidentally miss an important guideline.

It also opens up the possibility that institutions could share their guidelines with other institutions (shouldn’t medical guidelines be pretty universal, after all?).

The next phase of UpToDate: a prototype

Of course—and I think you saw this coming—I couldn’t resist trying to build the 2023 edition of UpToDate. So I teamed up with pediatric hospitalist Kevin Grassi to build a physician reference based on GlacierMD (a medical knowledge engine I've written about before), tailored specifically for pediatricians. And in burst of creativity I’ve decided to call it: GlacierMD - Pediatrics.

I'm happy to report that it has rudimentary versions of all the features mentioned above. Here’s Kevin walking through some of those features:

It's an early prototype—but at the very least, it shows that UpToDate (and its competitors) have plenty of room for improvement. Feedback so far has been positive, and we’re working quickly to add more features that UpToDate lacks.

If you’re a pediatrician and are interested in giving GlacierMD a spin (or know someone who might be), you can join the beta here:

Take me to GlacierMD

Thanks for reading! If you’re interested in following the GlacierMD journey (or reading about the intersection of software/medical research/product development), you can subscribe to my Substack here:

Subscribe now

Have you heard? Artificial intelligence is here at last—even the New York Times is writing about it!

There are lots of AI models floating around, but the one that really broke the internet is called ChatGPT, created by OpenAI. What is ChatGPT, exactly?

Let's ask!

🙏 Praise be to ChatGPT!

But what if it was actually useful

I'm very grateful that OpenAI spent millions on R&D to make Twitter more entertaining. But what if we could use AI to solve actual problems?

For example: I know some people who have COVID-related hair loss. These people desperately want to know when their hair will stop falling out. Seems like something this Gift of the Lord—trained on the sum of human knowledge1—would know.

Well that's annoying!

Undeterred, I submitted exactly the same question four seconds later, and—surprisingly!—got a totally different answer:

Well, that's a lot of words that basically boil down to "ask your doctor." But even if ChatGPT did give a concrete answer on the second go-round, how could I trust it? It just told me two totally different things!

AI has some flaws

So already I've encountered some shortcomings of our AI2 overlords:

• Their knowledge is out of date

• Their knowledge is inconsistent

• They give vague, unhelpful answers

• They hallucinate (i.e., they give answers to questions that they don't have knowledge about)

But what if they didn't do that

There's probably a really fancy way to address these problems using abstruse mathematics, a pile of arXiv preprints, and LaTeX.

But what if instead I just wrote a bunch of bad Javascript? Here's what I'm thinking:

1. Pull recent/relevant studies from Semantic Scholar

2. Rank them

3. Pull out relevant snippets

4. Summarize

I'd still use the fancy AI (GPT-3.5, a cousin of ChatGPT) for steps 2-4, but I'd only ask it to perform very specific tasks on a predefined set of studies. And I'd tell it very sternly not to hallucinate!

Loading your answer…

Okay so obviously the only reason I’m writing this post is because I did those things. I’ll skip over the boring bits (the Javascript would make your eyes bleed) and jump right to the exciting demo-GIF:

(Look I might not know what a tensor is but damn that’s a nice loading screen. 🙌)

(Yes I’m aware that ChatGPT will soon be designing better ones. 😭)

Bingo!

Wow! That went way better than expected.

But can I trust it?

Look, I don't really trust these polite AI chatbots, and I certainly don't trust polite AI chatbots with a thick layer of untested Javascript plastered on top.

But the cool thing is that I don't really need to trust them. Unlike the other "text in, text out" AIs, this one actually has an audit log. You can see exactly what excerpts were used to generate the answer and what studies those excerpts came from.

But…Google?

Yeah okay sure—you could just Google this question. But

1. You wouldn’t get to tell your friends you used ✨ArTiFiCiAl InTeLiGeNcE✨

2. You (once again) wouldn’t know where the answers came from. Most of my Google searches turn up (heavily SEO-optimized) editorial-style articles with zero sources listed. But my Javascript-AI Frankenstein thing gives answers that can always be traced back to primary research.

Can I try it?

Sure, do your worst! (Until my OpenAI credits run out.)

Ask the Almighty AI

Who are you? What is this site?

I'm Tom Cleveland, and I'm interested in giving people access to better health information. I've been trying to do this in various ways over the last five years, and GlacierMD.com is where the prototypes go.

Sound like something you're interested in? Subscribe to this newsletter and receive…monthly?…updates about this project.

Subscribe now

Thanks for reading!

So I was pretty shocked when—three days ago—somebody showed me an amazing health website. It was user-friendly, evidence-based, and gave clear, actionable advice. Instead of assaulting you with ads and pop-ups, It answered your questions and helped you make important health-related decisions. It evoked trust and clarity and healthfulness. It was called DynaMed Decisions.

The first thing that caught my eye was their breast cancer screening guide. Unlike most health websites, DynaMed understands that you're here to make a decision, and the whole page is framed around a single decision in particular: Should I get a mammogram?

After answering a few questions, I'm immediately presented with an awesome visualization of the probability of different outcomes for people who do and don't get mammograms:

Understanding the probabilities involved in something like cancer screening is really hard, even for doctors—yet this simple diagram makes it super obvious. And if that wasn't clear enough, all my important questions are answered with regard to every possible choice:

If you aren't blown away by these screenshots then you haven't spent much time trawling WebMD, Healthline, or even the more reputable (but equally unhelpful) government health websites. Clear, concrete, evidence-based advice is really hard to come by on the internet.

Hungry for more, I click on "Flu Vaccine: Should I get it?" because hey it's almost flu season and I've been meaning—

The paywall slaps me in the face. I sit there for a second, stunned.

But slowly I recover my senses. You know what? Maybe it is worth it. My health is really important, and if this resource helps me live a healthier life then honestly I'd be willing to shell out—

I spit-take my coffee all over my screen. Request a demo?

Ugh. We all know that "request a demo" is landing-page-speak for "you're a schmuck and you can't afford this enterprise software."

What's happening here? This whole time the tool was talking to me, the patient—e.g. "...your risk of breast cancer…"—not to my physician. So why is this awesome patient-focused website making it impossible for patients to use it?

Patients are terrible customers

The answer, of course, is money. DynaMed can get a lot more money from your doctor than they can get from you. So they designed DynaMed decisions to be accessed by your doctor, on their iPad, while sitting next to a patient.

It turns out that hospitals and medical groups have several big incentives to pay for good health information

• Their health care staff need to satisfy training and continuing education requirements (CME, MOC, etc.)

• They're rated on certain patient outcomes (e.g. HAC scores) that need to stay above a certain level

• It makes their staff more efficient, which brings down costs

So most hospitals decide it's worth it to spend $399/year per doctor for access to DynaMed. And that insane price tag lets EBSCO (DynaMed's parent company) spend a lot of money on delivering high-quality health information with great UX.

But consumers? Most of us aren't willing to shell out a fraction of that for health information. So instead, we're left with the dregs: a mishmash of vague, ass-covering statements from WebMD, scant information from public health officials, and a cacophony of outrageous claims from people trying to build their #wellness audiences.

And DynaMed is the cheap option. Their larger competitor, UpToDate, charges a whopping $696/year ($58/month) per doctor. But it's really good. Just compare their (free) article on depression management to WebMD's page on depression treatments. While WebMD does mention some of the same treatments, it's missing the practical guidelines, the supporting evidence, and the detailed discussion of tradeoffs. You can't really make any sort of health decision using the WebMD article—its only purpose is to rank highly on Google and not ruffle any feathers. What's great about UpToDate and DynaMed is that they're decision-making tools—and decision-making is why people Google their health problems in the first place!

The existence of DynaMed and UpToDate prove one thing: consumer health information could be so much better.

Good health information is just out of reach

So here's the reality we find ourselves in:

• There's a treasure trove of high-quality, evidence-based, practical health information available on the internet

• This health information could dramatically improve your life

• The only way to see it is to borrow your doctor's iPad

It boils my blood. Is there nothing to be done?

3 trends are making health information more profitable

Despite the depressing reality of health information economics, I think building a consumer-focused health reference is getting more and more feasible every day. Three secular trends are providing some serious tailwinds: new publishing models, more open access research, and major advances in AI.

New publishing models

It's tough to remember, but ten years ago nobody paid for information on the internet. If you pulled out your credit card in front of your computer it was because you were buying something physical. But these days we're all used to paying for good information—it's common to have subscriptions to the New York Times, Consumer Reports, podcasts, and any number of niche Substacks.

Despite this trend, today's popular medical sites are still stuck in the past: WebMD, Healthline, Everyday Health, and the rest are all ad-supported. This works when you've reached the scale that these companies operate at, but an ad-based model is really hard to bootstrap—which might explain the dearth of new WebMD competitors!

But the subscription model gives me hope. Platforms like Substack have shown us it's possible to make quite a bit of money focusing on very niche topics—which would enable an upstart to start with a specific topic and slowly expand to eventually cover all health topics.

More open access research

The open access research movement is gaining serious momentum. Since 2015, over half of newly-published medical research is open access. "Open access" means that the costs of publication are borne by other parties (research institutions, nonprofits, governments, etc.) and not by the reader. This is already a good trend for consumers!

But it's also good for anyone trying to distribute high quality health information. The raw inputs to a health reference site are peer-reviewed medical studies—and if these inputs are free then the cost of producing these references is significantly reduced.

Rapid improvements in AI

As mentioned above, the typical publishing process for a reference like UpToDate involves paying a highly-credentialed expert to spend hours diving into primary research and writing a long-form article about a topic. This is a very expensive way to turn primary research into useful health information.

I believe there will always be experts involved in the generation of medical references. But the rise of large language models and other AI techniques have made it possible to automate the most tedious parts of these reviews, like searching for all relevant studies, screening out irrelevant studies, extracting key information, and synthesizing that information into summary statistics.

AI's recent progress is staggering. Five years ago there was no robust way to extract key information from free-text studies. Today, you can literally ask GPT-3 to "Extract p-value, standard error, outcomes measures, and sample size from the following study…" and it'll do it in a couple seconds (see Elicit.org for a great example of using AI for this purpose).

This kind of automation can vastly reduce the cost of producing high-quality health information.

Making it all work

Previously, it wasn't profitable to deliver physician-quality health information to patients. But the trends above make me hopeful that we've reached a tipping point where it's just become possible to do it. Open access journals and AI make it cheaper to produce this information, and subscription-based models make it easier to bootstrap an information-based business.

But do we really need better health information?

We deserve better

I think there's a sort of learned helplessness when it comes to online health information. Everyone knows that the big health websites are terrible. But I think we have this idea that health is complicated, and providing useful health information at scale is impossible.

But the existence of DynaMed and UpToDate make it clear that it is possible. It literally already exists. We just need to get it to the people that need it the most.

I'm serious about solving this problem, so last Friday I quit my job so I could try and figure this all out. Building a better WebMD definitely won't be easy—but hopefully I've convinced you that it's possible. I'm doing my damndest to take advantage of the trends mentioned above, and I'll keep this Substack updated with the progress.

You can check out the (humble!) beginnings here: https://www.longcovidstudies.org

Ideas? Questions? Feedback? Try me on Twitter: @tomjcleveland

And I'm not just some crank with an ax to grind—satisfaction with online health information is incredibly low, around 38%. What's more, this satisfaction has been very stable over time—one study found that users in 2008 were just as unhappy with health information as they were in 2017!

This is pretty odd. In those nine years two things happened: (1) software got vastly more powerful and (2) researchers published eight million new health citations—a 47% increase in all published health knowledge. So why hasn't any of this innovation made online health information even slightly better?

It's baffling. The sum of human health knowledge is large and growing rapidly—but the meager, watered-down portion allotted to consumers hasn't grown with it.

This has always been a problem, but COVID has brought this knowledge gap into sharp relief: consumers have been flocking to health websites in droves, searching in vain for answers to their COVID-related questions. And—thanks to the blistering pace of COVID-related research—those answers are changing frequently, making it almost impossible for a typical person to find up-to-date COVID recommendations.

In other words, online health information has always been bad, but COVID made the situation dire. So why is no one talking about it?

I think we've collectively gotten so used to terrible health information that we can't even imagine a better version of WebMD. So let's toss around some ideas—what would good health information look like?

For starters, we'd change the format. Most online health content is buried in long, wordy articles that manage to say almost nothing in 2,000 words (and 8-10 pharmaceutical ads). This isn't by accident—the goal of these articles isn't to improve your health, it's to rank higher on Google! And every SEO expert knows that long articles rank higher than short ones, so now "treating covid headache" takes you to a thousand-word article when really I just want short summaries of the top three treatments.

We don't tolerate this "editorial style" of information on other websites. Imagine if each Zillow listing was a college-length essay, and instead of saying "1,700 square feet, 3bd/2ba, currently listed for $800k" it said "this house is medium-sized, although some visitors report feeling that this house is smaller or larger than that. It probably has some number of bedrooms, and is listed for a certain number of US dollars." We'd never use Zillow again! And yet for some reason we accept this as the norm with health information.

Instead, it'd be great to have a bit more structure in our health data. What if I want a list of antidepressants, ranked by efficacy? What if I want to filter out the ones with nausea as a side effect? Or maybe I just want to see the ones with the best evidence. And why—dear god why—can't I see the percent of people who experience a particular side effect? I'm obviously not going to take Lexapro if there's a 90% chance I'll experience "brief feelings similar to electric shock"—but if it's a 0.01% chance then I'd consider it. I know that data exists—WebMD just doesn't want to give it to me.

So better health information would be quantitative, sortable, filterable, etc.—in short, it would have all of the features we'd expect from Zillow, Amazon, Expedia, Airbnb, Uber, etc.—all of whom are very good at conveying lots of complicated information to consumers through a 3x5'' screen that they glance at for three seconds.

Next, we'd make sure this data was updated frequently, ideally every day. New research often changes the established medical orthodoxy (that's the point of new research!) but the current editorial-based health references usually update their articles once every few months, and big overhauls happen more like annually, if ever. Even well-studied diseases (e.g. diabetes) have new research every few weeks—so a health reference that's a year old can easily have outdated information.

Finally, we'd put the evidence front and center. Most consumer health websites either (1) bury their evidence in the footnotes or (2) dispense with evidence altogether. You might be saying, "well, I wasn't going to read the evidence anyway, I'm sure WebMD has someone vet their articles." But I promise: the evidence does matter, even for the "non-science" folks.

For example: the WebMD articles on essential oils and aspirin are written in almost the same tone—even though the evidence for aspirin is an order of magnitude stronger than the evidence for essential oils. But because both pages lack a good summary of the evidence, a typical user might think that both treatments are equally likely to be effective. Gah!

Instead, a good health reference would give you a rough idea of the confidence that researchers have about the topic. It might list all the studies for a claim, along with structured info about the study: size, design, affiliation, etc.—as well as a clear summary of what this evidence means.

So to summarize: a good version of WebMD would have

• structured, quantitative information

• real-time updates

• summaries of supporting evidence

So why hasn't anyone done this yet?

The short answer: cost.

As I've already mentioned, there's a lot of medical research out there, with more and more piling up every day. And it costs a lot of money to hire a subject-matter expert to comb through mountains of data and write a 2,000-word treatise on zinc supplements. Imagine how much more it would cost to pay this person to organize this data in a structured way, make daily updates, and provide exhaustive evidence to back up their claims.

And these days, ad-based publishers are seeing their margins decline—most of the big online publishers are moving toward paid models instead. Ad-based publishers like WebMD certainly don't have piles of cash sitting around waiting to be dumped into quality research and product development.

But—but!—there are two trends that might just solve this problem:

1. Automated evidence synthesis is improving rapidly

2. More consumers are paying for premium content (Substack, podcasts, online news, etc.)

Each of these trends approaches the cost problem from a different direction—(1) means publishers can now provide higher-quality health info at lower cost, and (2) means publishers can make more money from that same health info, which they can plow back into improving the information. Let's look at each of these trends in turn.

Automated evidence synthesis is a fancy way of saying "turning health studies into useful information, automatically." And it's 2022 so obviously "automatically" is a euphemism for "using a computer." We're a pretty long way from total automation, but several parts of the synthesis pipeline have gotten much easier recently, thanks to ✨technology✨.

And the other trend—a move away from ad-based models toward subscription-based models—also seems like a boon for online health information. Not only does this mean that publishers can spend more money on their data—it also aligns the incentives between the publisher and the user. WebMD's primary goal is to increase ad revenue—which means writing lots of SEO fluff and packing it with garish ads for the latest hair-loss pill. "Providing useful, evidence-based health information" is way down the list of WebMD's priorities. But subscriber-supported websites need to be constantly worried about user happiness—one bad experience and a user will immediately cancel their subscription.

Taken together, these two trends make me optimistic about the future of online health information.

So here's the tl;dr, which I've helpfully buried at the bottom of this article:

• Online health information is terrible

• We should make it structured, timely, and evidence-based

• This might actually be possible, thanks to better automation and subscription-based models

And—wouldn't you know it—I was so convinced by my own argument that I made a prototype of "good WebMD," demonstrating that this isn't a pipe dream. It's called GlacierMD and yes I've tried this before.

And it's live! You can check it out at glaciermd.com or you can click this fancy button:

Show me a better WebMD

So far it only has data on long COVID, but hopefully it gives you a flavor of what online health information could be. Thanks for reading! If you have any thoughts or ideas on this topic—or some brutal feedback on my prototype—I'd love to hear about it at tom@tjcx.me. Thanks for reading!

"I really can't recommend it enough."

My dental hygienist had just finished fifteen minutes of brain-rattling tooth-scraping and was now trying to pitch me on a procedure involving lasers.

"Lasers?" I said lamely. "What are, uh…, the benefits of lasers?"

The DH swiveled and looked at me above her protective glasses—a look that indicated she pitied my profound gingival ignorance. "It zaps away all of the bacteria from the roots of your teeth. Amazing technology. I'd recommend it to anyone with your level of periodontal disease."

"I have periodontal disease?" That sounded serious.

"Not yet," she said, her voice filled with deadly caution. "But you have pre-periodontal disease. If it gets worse you'll have full-blown periodontitis." She mic-dropped her latex gloves into the trash.

"How much does it cost?" I asked, a little sheepishly. It seemed gauche to talk about money when my gum health was hanging on by a thread.

"Well unfortunately your insurance won't cover it," she said. "So it'll be about four-hundred dollars."

"Four…hundred…dollars?" My eyebrows hit the ceiling.

"Yup!" said the DH cheerfully. "Trust me, it's worth every penny."

“…”

The rest of our conversation was an awkward blur. Somehow I managed to fend off the DH and her expensive laser, but as I was leaving she mentioned—in an unmistakably threatening tone—that she'd made a note in my file and that I should strongly consider laser therapy at my next visit. I left the office feeling like I had postponed the inevitable.

But as I walked home my guilt turned to indignation. How was I supposed to make this decision? On the one hand I go to the dentist because they're the experts—but on the other hand I know that dental offices are businesses like any other, driven by the cold logic of capitalism to foist as many treatments onto their patients as possible. So how can I make sure that this laser thing is worth the cost?

Laser bacterial reduction (LBR) is just one example of the many high-stakes medical decisions that you and I and everyone else are forced to make constantly.

In the pre-Szaszian era, medicine was something that happened to you. You would put your health in your doctor's hands and they would unilaterally decide your course of treatment. But in today's patient-centered world, doctors are more like consultants: they provide suggestions, you Google those suggestions and provide counter-suggestions, your health professional Googles your counter-suggestions, etc. Medicine has become a three-way negotiation between you, your doctor, and The Algorithm.

In addition, most of today’s health decisions are made outside of MD-earshot. You're vegan and wondering if you should take a B12 supplement. You have a headache and aren't sure what painkiller to take. You want to avoid COVID and aren't sure what kind of mask to buy. These are all health decisions! And most of them are made by individuals, not by medical professionals.

Which brings me back to my dental hygienist and her expensive laser. I have six months until my next cleaning—that's six months to figure out if this treatment is worth it. But where do I start?

Hahaha just kidding. You're reading this in a year that starts with "2" so the only possible starting point is Google. Let's see what Google has to say about "laser bacterial reduction" (LBR):

The first thing I notice is the Google Scholar widget at the top. Nice! I'm all about evidence-based medicine, and the fact that Google's top results are three peer-reviewed articles is A+.

Or so I thought. The first study (Moritz 1997) is a small pilot study with only 50 subjects. It found that lased subjects had fewer bacteria around their gums—but there's no direct evidence that laser therapy fights periodontal disease.

The second study (Moritz 1998) is actually just a 6-month follow-up with the subjects from the first study. This study has some promising info: pocket depth (a direct measure of periodontal disease) was 1.9 mm shallower in lased patients than non-lased patients. But again, this was a 1998 pilot study with 50 subjects. Not exactly definitive.

The third study (Dewsnup 2009) is an in vitro study on 55 extracted teeth, comparing an erbium, chromium:yttrium-scandium-gallium-garnet laser (what?) with a sodium hypochlorite wash. I'm interested in a diode laser, on still-very-firmly-attached teeth, versus no treatment, so this isn't remotely relevant to my problem.

So far this is pretty weak evidence. But my gums are on the line here, so let's keep scrolling.

The next section is the "People also ask" section. It has four questions:

• Is laser bacterial reduction worth it? Answer: "The benefits of Laser Bacterial Reduction make the treatment a worthwhile investment." Source: A dentist's office named, in all seriousness, "Fidler on the Tooth." There is zero evidence cited by Dr. Fidler's page, which exists to advertise the laser bacterial reduction offered by her office.

• How long does laser bacterial reduction last? Answer: "Laser bacterial reduction therapy is a quick, painless procedure that can be added to your routine teeth cleaning." Source: Rachor Dental Care, another dentist's office. This Google Answer doesn't answer its own question, nor does the linked article, which is more (evidence-free) dental office marketing fluff.

• Is laser treatment good for gum disease? Answer: "It is an excellent alternative to conventional gum surgery because it permits treatment without stitching or cutting and with less discomfort to the patient." This doesn't really answer my question, since I'm not comparing LBR with surgery. This answer is brought to you by a dental office out of Murfreesboro, TN, and the only citation is to a DentistryIQ page, which itself cites zero evidence to support its claims.

• Is laser gum cleaning worth it? Answer: "Laser dental cleaning has proven to be effective at restoring the health of gums, allowing patients to keep their teeth as they grow older." Source: Active Dental, a Texas dentist's office. This answer is pretty relevant! But unfortunately the linked article doesn't cite a lick of evidence—you just have to take their definitely-very-biased word for it.

Okay—so far we have a single pilot study and four marketing blogs without even a suggestion of evidence. But what about the regular search results?

There's a version of this post where I go through each of the remaining ten search results and take tasteful-but-hilarious jabs at their jaw-dropping lack of evidence. What fun we would've had! But several close family members have gently suggested that I just summarize this bit instead.

So here's the bottom line: All ten search results are dental office marketing blogs and there isn't even a whiff of peer-reviewed evidence. Here's that sentence in helpful graphic form:

I.e., 14/17 results (82%) are literally trying to sell you the thing you're searching for, 2/17 are irrelevant research, and 1/17 is relevant but low-quality (and there are better articles to surface here, as we'll see below).

I found this horrifying. I come to Google because they're supposed to be the best. So here I am with a pretty basic question: "Does this laser bacterial reduction thing work?" and almost all of the results are marketing garbage.

And this kind of thing is far too common. How often has Google responded to your medical question with putrid, SEO-optimized landfill? Google has found 29,100,000 results for my query, but it's like an all-you-can-eat breakfast buffet that serves nothing but reconstituted eggs and Yoplait. You know there’s a gourmet, peer-reviewed omelet station around here somewhere, but you won't find it by wandering around the main rotunda of Google's sad smorgasbord.

And this is where we find ourselves. We've shrugged off the yoke of medical paternalism only to find ourselves wandering aimlessly in a jungle of unsubstantiated medical claims. 

This is probably where most people would've stopped their research. But most people go outside! So I revved up all my favorite health databases and started a casual literature review.

I was hoping to find some systematic reviews of LBR studies. Systematic reviews form the apex of the Evidence Pyramid, and every 90s kid knows that the apex is the tastiest part of any good pyramid.

And I'm in luck! I found several reviews on the use of diode lasers to treat gum disease:

• Sgolastra 2012. A systematic review of five RCTs, no significant effect.

• Slot 2014. A systematic review of nine RCTs, no significant effect.

• Roncati 2014. A systematic review of six RCTs (four of which used a diode laser), statistically significant but not clinically significant (i.e., a very small effect size).

• Cobb 2017. A narrative review of 21 RCTs, no significant effect. “[T]he quality of the evidence is less than auspicious.”

• Pawelczyk-Madalińska 2021. A systematic review of 15 RCTs, too much bias/heterogeneity to make conclusions.

These reviews tell quite a different story from the Google search results. One review found a tiny (“clinically insignificant”) effect, one review concluded that the evidence was too terrible to make any conclusions, and the three remaining reviews declared that diode lasers had no significant effect whatsoever.

This evidence might not be conclusive—but it is actionable! I'm trying to figure out if I should pay my DH $400 to sear my mouth with a sci-fi weapon—and these published reviews make it pretty clear that the only sane answer is "no."

Allow me to interrupt this gripping narrative to point out the absurdity of this situation. My DH clearly pushes this LBR thing on all her patients, and other dental office blogs I've read (see last section) make it pretty clear that any dentist's office with a diode laser thinks everyone would benefit from a regular laser touch-up. So how do most patients make this decision? Here are some guesses:

1. They say, "sure," and get LBR

2. They say, "I'll think about it," observe that the entire first page of Google results is rabidly pro-LBR, and get LBR

3. They say, "I'll think about it," spend four hours screening and synthesizing results from Cochrane, PubMed, Google Scholar, etc., and end up politely declining LBR

The problem is that strategy (3) is the only one that arrives at the "right" answer, but virtually nobody has the time or inclination to execute it. Since you've waded this far into a treatise on my gums, I suspect that you, dear reader, might be a (3)-type-person yourself—but I think we can agree that most people presented with my choice are going to get LBR.

So we've arrived at the punchline for this rambling periodontal memoir—the syllogism at the center of my obsession with consumer health information:

1. Consumers make most of their own health decisions

2. Consumer health information is atrocious

3. Ergo, consumers make bad health decisions

Premise (1) is true whether you like it or not. The rise of unregulated supplements, the increased emphasis on nutrition, the internet, and direct-to-consumer advertising have all conspired to put consumers at the center of their own healthcare. As mentioned earlier, most health decisions are made without doctorial input, and those decisions that do involve a doctor are often made with lots of patient input (e.g. "I desperately need that unpronounceable drug I saw in a vaguely cheerful TV spot once").

We can't put the patient-centered toothpaste back in the tube, so the only way out of this cruel logic is to attack premise (2): consumer health information is atrocious—so let's make it better. Accessible, high-quality health information is good for consumers and it's better for healthcare professionals, who can spend more time discussing reasonable treatments with well-informed patients and less time battling misleading DTC marketing campaigns.

This was my motivation for starting GlacierMD: I wanted to give consumers the information they needed to make evidence-based health decisions. And although I lost that battle (spectacularly) I'm still fighting the war! If this is a topic that interests you, I'd urge you to

• Subscribe to this newsletter,

• Sign up for the (v2) GlacierMD beta, and

• Email me any random thoughts you might have on the topic

Online medical information is awful—but it doesn’t have to be!

But we were talking about my gums.

My next teeth cleaning had arrived, and I checked in for my appointment with more than the usual amount of dread. The front desk lady showed me to my room and closed the door ominously behind her.

I sat in that vinyl recliner for what seemed like hours, rehearsing exactly how I would reject LBR and trying to anticipate possible rebuttals from the DH. I considered the wisdom of moving to a new zip code and starting a new life with a new dentist. But just as I was about to bolt the door opened and in waltzed the DH.

But she wasn't my DH.

"Hi there…"—checks clipboard—"...Thomas." She looked up at me. "I'm Denise. Your regular hygienist had to pick up her kid from school, so I'll be subbing in today."

"Hi, uh…" I took a few moments to recover. "Nice to meet you."

I knew I was being abrupt and weird but I just couldn't contain myself. "My regular hygienist mentioned something about a laser thing," I say, pretending that my browser history wasn't bursting with LBR studies. "Is that something I should get?"

"Laser bacterial reduction?" This DH looks amused. "I don't usually recommend that for my patients—I find it's not very effective. Probably not worth the cost."

I laugh a little too loud.

I’ve said it before: online health information is bad. Here’s me in 2019 complaining about the terrible search results for over-the-counter painkillers:

If you’re among the 77% of Americans that Google their health problems, insipid answers like this won’t surprise you. But we should be surprised, because researchers carry out tens of thousands of clinical trials every year. And hundreds of clinical trials have examined the effectiveness of painkillers. So why can’t I easily Google those results?

Then COVID-19 happened. As much as the painkiller information annoyed me, it was leagues better than the information we started getting about COVID prevention and treatment.

The pandemic created conditions where (1) everyone was searching the internet for COVID-related health information and (2) the science on COVID prevention and treatment changed quickly. This created a COVID information ecosystem that was so balkanized and fluid that even my homogenous group of friends had drastically different views on mask wearing, social distancing, vaccines, treatments, and risks.

“I believe in science,” one of my friends said, smugly.

That’s wonderful, I thought, but which science?

Of course what most people mean when they say “I believe in science” is “I believe in scientists,”—or, more often, in journalists or politicians who claim to be “following the science.”

And this is why science has become so politicized. The Enlightenment ideal of science was that it was so pure and objective that it couldn’t be politicized—there was no arguing with empirical observation! But in reality 99.999% of people aren’t running their own randomized controlled trials on COVID vaccines and observing the results. In fact, 99.999% of people aren’t even reading the published results from vaccine trials. Instead, most people read headlines or tweets authored by someone who read a summary of an official statement based on a summary of a review of a published result. And that’s where politics can creep in.

You read a headline: “Pfizer vaccine is 96% effective at preventing COVID-19.” You immediately book your vaccine appointment and start celebrating. But a couple days later a friend looks horrified when you tell her you’re getting a vaccine.

“The Pfizer vaccine? Didn’t you read about the myocarditis stuff?”

So you Google “covid vaccine myocarditis” and suddenly you’re waist deep in a rabbit hole. You read peer-reviewed papers about increased heart-swelling in some vaccine recipients. You scan scary-looking myocarditis data from the Vaccine Adverse Events Reporting System (VAERS), which is published by a government body. You watch countless YouTube videos from licensed physicians urging people to avoid COVID vaccines. If “believing in science” means “believing in credentialed scientists,” then you have a problem—they don’t agree!

And this is where politics comes in. If you’re someone who trusts the government, mainstream news, and academia, then you’ll probably side with the pro-vaccine people. If you’re someone who distrusts these groups, then you’ll say, “Well, I’m hearing different things, but the people I trust say the vaccine is dangerous, so I might as well hold off and see how this shakes out.”

And that’s pretty reasonable! For the record, I absolutely think that the benefits of COVID vaccines outweigh the risks—but my point is that I very much empathize with the people who see lots of apparently contradictory information about novel injections and get a little skittish.

The only reason I'm so confident about vaccines is that I have the time and statistical literacy to read a bunch of the primary literature. I know about the limitations of VAERS data; I know the relative risks of myocarditis vs. COVID complications; I can quantify vaccine efficacy over time. In short: I know how to put all of these seemingly-conflicting medical claims in a context that resolves the conflict and points to a fairly clear conclusion.

But most people don’t have the time or background knowledge to do this. And they shouldn’t have to! The problem is that medical communication today is really bad.

To see what I mean, let’s start by talking about good medical communication. Good medical communication, I want to argue, has three attributes:

• It’s quantitative

• It’s exhaustive

• And it’s accessible

Let’s take each of these in turn.

Good medical communication is quantitative, because qualitative data is useless for making real-world decisions. For example, the Mayo Clinic has an article titled How well do face masks protect against COVID-19? in which they answer the titular question with this zinger:

Face masks combined with other preventive measures, such as getting vaccinated, frequent hand-washing and physical distancing, can help slow the spread of the virus that causes COVID-19.

There is precisely zero information in that sentence. You could replace the phrase “face masks” with “shrimp gumbo” and that sentence would still be 100% accurate. If all you can tell me about masks is that, combined with other measures, they can slow the spread of COVID-19—that's not very compelling! Give me a number! Quantify the effectiveness of mask wearing by itself! If you tell me masks reduce my odds of getting COVID by 50%, then of course I'll wear a mask. But if it's only 0.05% then maybe I won’t bother. Numbers matter.

But quantitative data alone isn't enough—good medical communication is also exhaustive. By "exhaustive" I mean that it includes all research on a given topic. Exhaustive communication bolsters your argument ("look at all this data!") and helps undermine counterarguments ("we know about the conflicting data, but here's why it doesn't matter").

At first glance, this BBC article appears to thoroughly debunk the use of Ivermectin for treating COVID-19. It says that of the 26 "major" Ivermectin trials, one third have statistical errors or indications of fraud, and the other two thirds don't support the use of Ivermectin.

But then someone on Twitter links you to ivmmeta.com and, boy, these people do not mince words:

The [BBC] article reports that 26 studies were examined, however there are 79 studies, authors have not reported their results for all 26, and authors have not provided their data after repeated requests. Currently they have not even provided a list of the 26 studies.

And suddenly the BBC article has some explaining to do! And Ivmmeta.com does list all 79 studies, with lots of citations and numbers to boot. A counter-counter-argument against ivmmeta.com requires an even more exhaustive catalog of evidence (which, by the way, plenty of people have provided).

Often it seems like scientific communicators have a low opinion of the average consumer. "Don't overwhelm people with data," I can hear some comms consultant saying. "Use short words and repeat yourself." Gah! Sure, summarize the data, but also provide the data—and make sure you include all of it.

Exhaustive medical communication also needs to be updated frequently. The latest recommendation about Ivermectin use from the WHO was published in March 2021—we've had two more variants and a bunch more Ivermectin trials since then. So if I'm weighing the WHO’s stale recommendation against ivmmeta.com's recommendation—which is updated weekly—it seems like I should put a lot more emphasis on the latter.

And finally: quantitative, exhaustive medical information is no good unless it's accessible. Anyone with an internet connection should be able to discover and understand this information.

Peer-reviewed meta-analyses, for example, fail the accessibility test on two counts. First, they are often literally inaccessible behind publisher paywalls. Academic publishing is an unholy scam, and some researchers are trying to do something about it, but there's still a lot of critical information that's too expensive for the average person to read.

And even if you can find an open-access meta-analysis, it's inaccessible in a second way: it's incredibly boring! Your typical systematic review is a wall of text punctuated by dense tables, flow charts, and forest plots—which nerdier readers might find scintillating, but trust me: the average consumer feels unendurable boredom when faced with a funnel plot.

So good communication is consumer friendly. It doesn't omit technical details (that would violate the exhaustive requirement)—but it uses progressive disclosure to present users with exactly as much information as they need at that moment.

Medical communication can also be inaccessible if it’s condescending. Here’s a prime example from the official FDA Twitter account:

You read that right: the FDA thinks you’re acting like a cow.

Let's imagine for a moment that the FDA article linked in this tweet is a fantastic piece of medical communication (it's not). Who the $&@! is going to click that link? People who think Ivermectin doesn't treat COVID are going to feel smugly superior and keep scrolling (why read an article that tells them what they already know?) and people who do think Ivermectin could treat COVID feel extremely insulted, block the FDA Twitter account, and also keep scrolling. This tweet makes the Ivermectin discourse worse in every possible way. Ugh.

Okay so to summarize: good medical communication should be quantitative, exhaustive, and accessible. Unfortunately, the vast majority of medical communication fails to live up to these standards, and as a result we end up in a situation where lots of people believe things that are contradicted by current evidence.

There’s lots of "old man yells at cloud" energy in this post so far.

But I took a few deep breaths and decided that complaining wasn’t really helping, and in a fit of hubris I decided to create a tiny example of the kind of medical communication I envision.

Since using a COVID-related test subject would be way too fraught, I needed to find another medical subject to use as my guinea pig. A good test subject would be:

1. Interesting

2. Controversial

3. Low stakes

So I chose zinc lozenges! If you've ever been to a drugstore you've seen Cold-eeze, Zicam, or store-brand zinc lozenges that purport to shorten the duration of the common cold. Ever wonder if those actually work? It's an interesting subject because everyone gets colds and everyone wishes they were shorter—so if cheap lozenges worked they'd be amazing. It's controversial because the evidence for zinc lozenges is mixed. And it's low stakes because zinc lozenges are relatively harmless and cheap.

So I spent a few hours compiling the existing evidence on zinc lozenges and stitched it together with a bunch of really terrible Javascript.

Here's what I've got so far:

You can play around with it here:

Check out the demo

Here's why I like it:

• It’s quantitative. There are numbers! Most importantly, the top-line result contains a number: "...zinc lozenges reduce cold durations by X%" (where X depends on your study selection). Knowing how to quantify the effect size of zinc helps you make real world decisions about zinc—like whether it's worth the cost, hassle, and risk of side effects.

• It’s exhaustive. I ran the last literature search on February 2, 2022, and I have weekly reminders set to run the search again and update this list with new zinc trials (see Search strategy section of the site for more details). This means I have trials that were published after the last meta-analysis (e.g. Hemilä 2020). And I allow users to play around with all relevant clinical trials, even if some researchers have excluded them from a particular meta-analysis.

• It’s accessible. It’s free as in beer! And while I'm certainly not a design expert, it's free of a lot of clutter, ads, and dense prose that a lot of other medical sources have. And (hopefully) it's simple enough to be readable by non-experts, but informative enough to not come off as paternalistic or condescending.

But it definitely has a way to go! Here are some things I'd like to add eventually:

• Adverse event data. You can't really make an informed medical decision unless you know the risks of a certain treatment—but right now my zinc page only tells you about possible benefits. Having a section that lists reported adverse events, along with their incidence, would help people understand the tradeoffs involved.

• More trial data. I want to add more info about each trial, like

  • GRADE rating

  • Preregistration status

  • Funding source

• Better UI. It’s super ugly right now!

It’s a small start, but I think this page is pretty useful for synthesizing the current research on zinc lozenges. What do you think? Helpful? Misleading? Confusing? Simplistic? Send me some feedback via email (tom@tjcx.me) or Twitter (@tomjcleveland—but please, no condescending cow tweets).

What’s a front page, old-timer?

Look I know it sounds crazy but they used to print the NYT on trees. A small child would bike past your house every morning and literally throw this tree-bundle at your front door and you’d flip through it over breakfast.

But it’s 2021 so I’m talking about the digital front page: everything that shows up when you first load www.nytimes.com. It looks like this:

If you scroll to the bottom of this page you might notice that more stories are loaded as you scroll. I’m not including these, because that content (usually under the heading “Other Stories”) is clearly Page 2 material.

Methodology

In a similar fashion to my A/B testing methodology, I wrote a web scraper that

• Scrapes the front page of the New York Times every five minutes

• Pulls out all articles

• Associates them with canonical data from the NYT API

The end result is a time series of article data, answering the question: What articles were on the front page at time X?

Article turnover

The first question I had was: How often does the front page change?

If you’re old enough to remember tree-based newspapers, you might expect that articles stay on the front page for about a day. And you’d be sorta right. The vast majority of front-page articles stay on the front page for less than 24 hours, with the median article lasting about 9.5 hours.

But—get this—10% of front-page articles stay there for less than an hour:

In a post-paper world this makes sense. The NYT front page is precious real estate—if an article is underperforming they should quickly replace it with something that attracts more eyeballs. But it also means that we (readers) are missing out on the 10% of articles that weren’t engaging enough to survive.

So what articles do survive? In other words, what articles spend the most time on the front page?

The most promoted articles

The chart above is truncated at 48 hours—it omits a small percentage of articles that stay on the front page for several days. See, for example, this hard-hitting piece of reporting on scrambled eggs, which sat on the front page for six days:

In fact, most of the long-tenured articles I looked at were brazen puff pieces. Some even crossed the line into “this smells like an ad” territory. For example, it’s hard to look at this article as anything other than an ad for Norwegian Wool:

And it was on the front page for four days! Let’s not forget: a military junta is shooting protestors in Myanmar; 30,000 U.S. organizations have been hacked by the Chinese; there are serious wars happening in Afghanistan, Ethiopia, and Yemen—but the New York Times is saving their front-page real estate for the real news: a jacket that will make your shoulders pop at a conference in the Alps.

Taking a closer look at fluff

Maybe I’m being unfair—maybe “real news” changes quickly, and therefore has high turnover, even if overall the front page is mostly “real news.”

Here’s a better methodology: add up all the front-page hours for all articles and then see how these hours are split between “real news” and “fluff.”

Unfortunately there’s no obvious “fluff” field in the NYT API, so I had to get a bit creative.

Front page time, by tag

I knew that the NYT assigned tags to all its articles in an effort to organize them by topic. Since I’m trying to figure out which articles are “fluff,” I figured that these tags might provide a clue.

So I started by divvying up front-page time by tag:

The chart above (and the rest of the charts to follow) show the distribution of front-page hours, answering the question, “How much total front-page time is allotted to articles of type X?”

Anyway, this chart paints a fluff-less picture of the NYT front page. The top five tags are either about

1. US politics, or

2. COVID

And like, yeah, those are pretty serious topics in 2021.

But while these tags sound serious, it’s pretty easy to find fluffy articles with serious tags. The “Coronavirus (2019-nCoV)” tag certainly includes serious news, but it also applies to this wedding idea listicle, a review of wellness apps, and this weird admonition to have your kids do more chores.

So looking at tags is interesting but it doesn’t map cleanly to the fluff/news categories that I’m really after.

Front page time, by section

I know from TV that old-timey newspapers used to be organized into sections—each one physically separated from the rest—so that you and the other ad executives could trade them back and forth on the train while drinking scotch for breakfast.

These days newspaper sections are a bit more subtle (you can find them in the URL), but for our purposes sections have one great quality: each article can only have one. This means that an article’s section is a pretty good description of what the article is about (whereas an article can have an unlimited number of tags).

So once again I took a look at total front page time, this time by section:

Did you guess that the opinion section was the most-promoted section of the NYT? I was a bit surprised—I think of the front page as being full of breaking news—but the rest of these rankings make sense to me.

Sadly, I’m still running into the same problem as before: these sections don’t map very cleanly to my fluff/news categories. So while the serious-sounding “U.S.” section certainly contains lots of real news, it also has articles about poignant college-admission essays, bowling drones, and snowplows with punny names.

Ugh—back to the drawing board.

Catching a break

At this point I’m getting desperate. I Google “machine learning fluff articles” and “GPT-3 for puff pieces” to no avail. I start thinking about how long it would take me to go through the ~3,500 articles in my database and manually sort them into fluff/news categories.

But then I get lucky. I am, uh, casually browsing the New York Times’ undocumented GraphQL API (we all have our COVID hobbies) and come across a field called “tone” with these possible values:

This looked promising! It’s not something that the NYT displays on its website, but clearly “tone” is important enough for them to store in their article database. So I added this new field to my own database an discovered that roughly 55% of articles were NEWS, 35% were FEATURE and 10% were OPINION:

What do these “tones” mean? NEWS seems pretty straightforward, as does OPINION, and NO_TONE_SET is rare enough to ignore. But what kind of tone is FEATURE? I took a random sample of FEATURE-toned headlines to find out:

Bingo! I couldn’t have made up fluffier headlines. Clearly “feature” is some sort of industry term for fluff, and the New York Times already tracks it!

I did a little digging into what “feature” means and it turns out I’m a complete philistine: “feature” is an extremely well-known and widely-used category in the news biz—as Wikipedia tells me:

Unlike straight news, the feature story serves the purpose of entertaining the readers, in addition to informing them. Although truthful and based on good facts, they are less objective than straight news. Unlike straight news, the subject of a feature story is usually not time sensitive. It generally features good news. [Emphasis mine.]

And here I thought I had discovered fluff!

Front page time, by tone

Okay now that I’ve added tone to my database I can finally answer the question: How much of the NYT front page is fluff?

So here it is, the big reveal:

Answer: about 31.1% of the front page is fluff.

Or, put another way: less than half of the front page is news.

What about editorials?

The opinion section defies my news/fluff ontology—on the one hand editorials have been a central part of newspapers since forever, and they exert a powerful influence over public discourse. At their best, editorials help us understand the unceasing torrent of news and frame it within a broader context—so calling them “fluff” seems uncharitable.

But I’m still not convinced that editorials are news, either. There’s certainly no news happening when David Brooks holds forth on topics like How to love America, The case for Mike Bloomberg, or Nine non-obvious ways to have deeper conversations. These are more like sermons, where the point is moral or cultural transmission, not news.

Conclusion

We won’t solve the “are editorials news?” debate here, but it raises a more interesting question: What are we doing when we read the news?

I think some people view “reading the news” as an educational activity—reading the news means being informed, it means being a good citizen, it means being educated. In this view, scrolling through the NYT is good, while scrolling through Instagram is bad.

But I don’t think it’s that simple. The above analysis shows that less than half of what you read when scrolling through the NYT is news. The other half is

1. Other people’s opinions about that news

2. Stuff about scrambled eggs, trendy wool coats, Peloton, etc.

If you like reading the non-news stuff, that’s fine! But I often feel like there’s a bait-and-switch happening when I read the NYT. It’s too easy for me to slip from “I’m opening the NYT to stay updated on breaking news” to “I’ve been reading about holiday hams for half an hour.”

And this is probably by design! The NYT would prefer that you not shut your laptop when you’ve finished scanning today’s 10-15 news headlines—so they supplement it with white-collar clickbait like Shopping for Daybeds.

So recently I’ve been trying to read just news. I even created this page that only displays the NEWS articles from the NYT front page so that I can more easily ignore the other stuff.

It’s a little less fun than the full NYT—but you won't waste time reading articles like When One Fridge Is Not Enough.

Next time

This post was all about what types of articles the NYT promotes—but another interesting question is: what types of articles do people read?

So next time I’ll be diving into article engagement: which articles are most likely to be clicked, emailed, or shared on social media? If you aren’t already a subscriber, smash the button below to stay tuned.

Subscribe now

And yet it rarely attracts the kind of close scrutiny of, say, a Fox News. And that’s totally reasonable! Fox News is an absurd clown show and deserves every criticism it gets.

But I’m still curious about the NYT. It paints a certain picture of the world—and (in my circles) this ends up being the default picture, whether or not you agree with it.

I wanted to know more about this picture. So for the next few weeks I’ll be publishing a series of posts on the New York Times, drawing on data scraped from their front page and pulled from their official API.

This first post is all about A/B testing: how the NYT tests different headlines and how they change over time.

A/B Testing at the New York Times

Okay look it’s 2021—it would be shocking if the New York Times wasn’t A/B testing headlines.

And the NYT is pretty open about it:

The Times also makes a practice of running what are called A/B tests on the digital headlines that appear on its homepage: Half of readers will see one headline, and the other half will see an alternative headline, for about half an hour. At the end of the test, The Times will use the headline that attracted more readers.

But I still had questions:

• How many articles are A/B tested?

• How many headlines are tested for each article?

• Does it help?

• How different are the headlines?

Methodology

I wrote a script that does the following:

1. Scrapes the NYT homepage

2. Pulls out all the headlines

3. Associates them with article metadata from the official NYT API

4. Shoves them into a database

The script runs every five minutes. I started doing this on February 13, 2021, so what follows is based on three weeks of data.

Results

As promised, the NYT does in fact A/B test their headlines. Roughly 29% of articles have multiple headlines, and the most headlines observed for a single article (so far) is eight.

A lot of these headline changes are pretty minor—it’s common for the NYT to fix a capitalization or punctuation error after an article has been published.

And sometimes it seems like the copy editors can’t quite make up their minds: should “in” be capitalized?

Other times, the NYT changes headlines as the story evolves. Here’s a great story told in headlines:

But most headline swaps are clearly A/B tests looking for more clicks. Here’s an article about Biden’s governing style, with a pretty dramatic headline switch:

The only reason to make this kind of change is if you’re trying to boost engagement. And it worked! This article broke into the “most viewed” list a few hours after the headline swap (which supports my theory that liberals love reading about Trump).

Note: The above chart (and others in this post) make it look like a headline switches completely from one to another, without any actual A/B testing. This is just an artifact of how I’ve grouped the columns—each bar represents half an hour, but within that half my scraper sees the headline change back and forth many times, despite the colors being grouped together.

But not all A/B tests have such success. Here’s an A/B test that definitely failed (you might have to squint to see the tiny blue smidge on a smaller screen):

I hope this failure didn’t discourage the kooky NYT editor behind “Jumping Jehoshaphat!” The NYT could definitely use more Bugs Bunny-isms.

But overall there’s a pattern to these A/B tests: headlines tend to get more dramatic as time passes. Take this article about Cuomo’s sex scandal:

In the first swap, Cuomo goes from being attacked to being under siege, and in the second swap he’s no longer revising his plan, he’s apologizing.

And it worked: as the headline changes we can see this article climbing the “most viewed” chart.

The headlines get even spicier in this article about Trump’s CPAC address:

Trump starts off addressing conservatives and claiming leadership of G.O.P. but in the final headline Trump has a hit list and is firing a warning shot. And sure enough, the bombastic rhetoric propels this article onto the “most viewed” list.

Okay last example: this wildly popular article about Oprah’s interview with Meghan Markle:

I actually watched this interview—all two hours of it—and I can tell you that the first two headlines are a much better summary of what went down. Yes, Meghan does reveal that she contemplated suicide, but it’s a five-minute interlude in an interview that has a lot more going on. For example, none of these headlines mention the role that racism played in Meghan’s distress—a subject that takes up way more screen time than her suicidal thoughts.

Does it work?

The articles above got way more popular after some A/B testing—but is this true in general?

I crunched the numbers, and it turns out that A/B-tested NYT articles are 80% more likely to rank on a “most popular” list. And—not surprisingly—more headline testing correlates with more engagement:

Caveat: headline count and engagement are correlated but who knows which way the causation runs. It makes sense to me that the more you A/B test a headline, the more likely it is for your article to get shared/liked/clicked. But it’s also possible that the NYT spends more time tweaking articles that are already popular. (Although, anecdotally, many of the articles I’ve looked at have lots of headline swaps before the article lands on a most-X chart.)

So what did I learn?

The NYT tests headlines to increase engagement

Duh.

Overall, this testing is pretty restrained

I’m pretty surprised how few headlines get tested by the NYT. Most articles aren’t A/B tested, and most of the ones that are A/B tested have just two headlines.

I’d sorta assumed that the NYT editors told their staff to submit all drafts with, like, six possible headlines, and that some automated system would test all six in the first hour. But clearly this isn’t true—despite the fact that the data indicates more A/B testing would boost engagement.

One possible explanation: 62% of the NYT’s revenue comes from subscriptions, and only 27% comes from advertising (and ad revenue is falling year over year). This means that views aren’t as important as subscriptions—and a front page full of clickbait would likely scare off potential subscribers.

But it still leads to emotionally-charged headlines

The NYT might be more restrained than BuzzFeed, but we should keep in mind that it’s not a neutral observer. As the examples above show, A/B tested headlines paint a picture that’s a lot more dramatic than the reality. Frequent NYT readers will end up thinking the world is scarier and more shocking than it really is.

Next time

In the next post I’ll look at the front page of the NYT, specifically:

• How long articles stay on the front page

• Which articles spend the most (and least) time on the front page

• What kind of content is most likely (and least likely) to appear on the front page

• How front-page time correlates with overall engagement

• And other fun tidbits!

The data

If my free-tier EC2 instance hasn’t fallen over you can browse the live headline data yourself:

Live headline data

Keep in mind:

• All data is from February 13, 2021 onward

• The site is cached for 30 minutes, so it might be slightly stale

Let me know if you find something interesting!

In the chart above, the dotted blue line represents the payoff for a stock. It’s linear because a $1 increase in share price means a $1 increase for the stockholder, and a $1 decrease in share price means a $1 decrease for the stockholder.

But the red line—representing the payoff function for a call option on this stock—is not linear. Above $100 (the strike price) the payoff function mirrors that of the underlying stock, but below $100 there’s no more loss.

In other words: you get all the upside but none of the downside.

Unfortunately the market for options is pretty efficient, which means this positive convexity is priced into the option premium. But real-life actions also have payoff curves, and ideally these would also have positive, asymmetric payoffs.

Real-life actions tend to be less-efficiently priced, because

• Nobody knows your circumstances better than you do

• Humans have all sorts of quirks and biases

• Personal decisions are rarely zero sum, but financial markets are

I’ve compiled a non-exhaustive list of actions with (positive) convex payoffs below. Let me know if I missed any!

The List

Giving gifts

Twice in Shoe Dog, Phil Knight sends unsolicited gifts to people’s homes. First, he sends legendary track coach Bill Bowerman a pair of Tiger shoes, which ultimately induces Bowerman to join (what will become) Nike, providing Knight with a powerful business partner.

Then, after visiting his supplier in Japan, he sends a money order to one of the supplier’s employees “for a new bike” to replace the one he lost in WWII. Later, this employee provides Knight with valuable inside information about his employer.

The point is that giving a gift has a small fixed cost (like an option) but the possibility of providing very large returns.

Asking for big things

Asking for stuff has a limited cost (getting rejected) but unlimited gain (getting anything you ask for). The bigger the ask, the better your expected value! Try asking for

Promotions

The downside of asking for promotions is that you annoy your boss. The possible upsides are

1. You get it immediately

2. You plant the idea in your boss’s head

3. You make it seem reasonable that you should be promoted soon

4. You learn what needs to happen in order to get promoted

So just ask.

Raises

See above. As with investing/interest, small percentage increases in your salary will compound over time, so ask for raises often. (Or switch companies to get one.)

Investments (in your business)

There’s currently a dearth of good startups and a deluge of investor money, so you’d probably be surprised how many people would be willing to invest in your new venture.

But even in less favorable conditions it can never hurt to ask. People with lots of money are always looking for

1. Good returns

2. Something interesting to do

3. Insider status

All of which make them eager to invest. And receiving investment can have huge returns, allowing you to scale your business.

More business (from customers)

Okay sure, asking a prospect “Hey there want to buy my product?” isn’t exactly a nuanced approach to sales. But “asking” in this context could also mean running ads, telling friends, cold emailing people, etc. The upsides here are

1. You get more customers

2. You learn why nobody wants to buy your thing

Both of which are immensely valuable. Whereas the “asking” part has virtually zero downside.

Sabbaticals

Obviously it depends on where you work, but in some industries and role the “sabbatical” (i.e. long vacation) is relatively common. It’s expensive to hire your replacement, so if you’re burnt out or considering switching companies, tell your employer you need a break. The worst that happens is they say “no”! And the best case scenario is you get to take a massive paid vacation.

Partnerships (with other people/businesses)

Alright I know I used this example from Shoe Dog already, but Phil Knight partnering with Bill Bowerman was the decision that created Nike. And it happened because Bowerman made the (zero-risk) decision to simply ask Phil Knight if he could join Blue Ribbon. He ended up owning 49% of the company that would become Nike, and (if Shoe Dog is to be believed) got to just hang out at his house and mess around with waffle irons.

Meeting people

Meeting new people can be uncomfortable, but that’s pretty much the only downside. The upside, however, is virtually limitless: you could meet your future life partner, your future business partner, your best friend, your key investor, your biggest client, etc. Sow lots of seeds, and some of them will become sequoias.

Go to meetups

It’s easier to talk to strangers when you all have something to talk about—and what are meetups but groups of people getting together to talk about one thing? Go to a variety of meetups, both to maximize your chances of meeting people but also to learn more stuff!

Join a sports team

Sports seem to breed camaraderie, so joining a sports team can be the fastest way to pick up some new friends. Plus it counts as exercise experimentation, so that’s two birds with one stone.

Volunteer

Putting a bunch of people in a room and saying “okay go be friends now” generally doesn’t work—you need some kind of activity to bond over. And volunteering is a great bonding activity. You’ve more-or-less filtered out all the assholes (who generally don’t volunteer) and volunteers are generally in an open, friend-making mood. Plus you end up learning about something (political campaigns, homeless programs, environmental cleanup) that you probably had no prior experience with (see learning).

Organize something

Benjamin Franklin organized the first American fire department, the first American library, and the first American philosophical society. He was an organizing machine, and it propelled him to great heights. If you find yourself saying “wouldn’t if be nice if there was…” just organize it!

Creative

Writing a blog post

Yeah okay it’s a bit on the nose but blog post-writing has some very asymmetric payoffs. The cost of writing one is fixed and relatively small, but the impact can sometimes be huge. My post on losing \$40K still gets thousands of hits a day, and it took a few hours to write. Sure, many of your posts will be read by zero people, but occasionally thousands of people will read one, which could lead to meeting new people, new business, or exposure to new ideas.

Tweeting

This has the same benefits of writing a blog post, but with far less overhead (and arguably the same payoff profile). I’m just not very good at limiting myself to 280 characters.

Creating a product

Creating a product can also have outsize returns. It’s not always worthwhile (the initial investment could be quite high) but launching the right product or business could change your life.

Learning

The only downside to learning new things is the time you spend doing it—but the upside is enormous.

Reading (the right) books

We can read faster than we can listen (to podcasts) or watch (videos)—so if learning is a convex activity, then reading books will provide the best bang for your buck. And within the book category there are two subcategories that will get you even better payoffs.

Very old books

Old books are good because

1. They’re Lindy: a book that has survived centuries must have some good (or at least thought-provoking) ideas

2. They’re unpopular: old books are rarely on bestseller lists, which means the information they provide will be more likely to give you a novel idea (pun intended).

Very specialized books

Specialized books are good because

1. Deep knowledge in a certain area tends to produce innovations

2. Again, they’re unpopular: you’ll likely stumble across something that few others have

Experimenting

Experimentation usually involves ratcheting: we can ignore bad results but build on good results. This means that the downside is low (almost zero) but the upside (if you discover something immensely important) is huge.

Below are some examples of things we can all experiment with, but keep an eye out for other opportunities to run experiments.

Diet

Having a good diet makes you more confident, more energetic, and generally better at taking on the world. There is a lot of conflicting diet advice out there, but the costs of self-experimentation are pretty low—so just try some stuff!

In the worst case you’ll get a stomach ache, but in the best case you’ll dramatically improve your health!

Exercise

This is pretty similar to the above section: experimenting with different forms of exercise isn’t hard—but the benefits of finding a type of exercise that works for your body, your schedule, and your preferences can considerably improve your life.

Routine

We are our habits. Life isn’t about one or two momentous decisions, it’s about the little things we do consistently, every day. A corollary is that modifying these habits can lead to massive long-term changes in our lives.

You could argue that the payoff function for habits is in fact symmetric: developing bad habits is bad in the same way that developing good habits is good. But—like a lot of things in this list—careful experimentation can limit the downside (you can stop if things start looking bad) but keep the massive upside. So beware of developing bad habits!

Switching jobs

I don’t think I’ve ever talked to somone who regretted switching jobs. Randall Munroe agrees:

Switching roles

Switching roles (i.e. careers) is usually easier if you do it at your current company, and the downsides are often smaller than people think. Generally speaking, companies will let you transition back to your old role, or transition again to another role.

Transitioning roles give you upside in the form of

1. Professional knowledge about a totally new domain

2. A better personality/role fit

both of which can have massive positive long-term effects.

Switching companies

This is similar to switching roles, but for people who like their current career path. Switching companies gives you

1. Insight into a different work culture / business model

2. A new set of coworkers to meet (see meeting people)

3. (Usually) A higher salary

Investments

(Disclaimer: none of this is financial advice.)

Not all investments have asymmetric payoffs, and many that do have already been priced accordingly (e.g. options) so investing at random is a bad idea.

Mis-priced investments with large upside and capped downside tend to be ambitious startups, but could also be collectibles (e.g. trading cards) or (dare I say) cryptocurrencies. The downside is capped at zero, but the upside is (virtually) limitless. That’s true for all investments of course, but some investments are more likely to have explosive growth than others.

Things you know about

The market rewards specific knowledge: so if you’re an expert in something, invest in it! Obsessed with Pokemon? Maybe you know something about trends in the Pokemon world that would let you make money investing in Pokemon cards. Obsessed with sailboats? Maybe you know something about the latest sailboat technology that lets you value sailboat companies more accurately.

People you know about

The market cannot possibly price every individual correctly—humans are complicated creatures, and there are a lot of us. This means that some individuals (usually those early in their careers) will be undervalued. Take a look at the people you spend time with and decide if any of them has a brighter future than others might suspect. Then invest in them by…

Investing in them

Is this person starting a business? Great! Invest in it. You’ll get a screaming deal because nobody else knows what a rockstar they are.

Joining them

Is this person starting a business? Great! Join it. You’ll get responsibility and upside that would be impossible once your talented friend is better known.

Hiring them

Are you in a position to hire someone? Then hire your talented friend! They are literally undervalued, so you’ll be getting a Wagyu steak for the price of chuck.

Companies you know about

Large, publicly traded companies are researched to death, but small startups are largely ignored. Among the small businesses started by your friends or family, are any worth more than they’re currently valued? If so, invest in them by

Working for them

Working for a successful company is good for several reasons:

1. (Sometimes) you get equity which will increase in value

2. Growing companies promote faster

3. Growing companies can pay higher salaries

4. Growing companies are a good learning experience (i.e. they must be doing something right)

Buying equity in them

If you’re an accredited investor you can invest in any company—so just ask! If you’re not, you’ll have to focus on public companies, which you’re less likely to have specific (tradable) knowledge of.

• Quiet: The Power of Introverts in a World That Can’t Stop Talking

• The Alter Ego Effect: The Power of Secret Identities to Transform Your Life

• Super Human: The Bulletproof Plan to Age Backward and Maybe Even Live Forever

• Indistractable: How to Control Your Attention and Choose Your Life

• Loonshots: How to Nurture the Crazy Ideas That Win Wars, Cure Diseases, and Transform Industries

• The Wisdom of Failure: How to Learn the Tough Leadership Lessons Without Paying the Price

I usually glance at this self-help altar out of the corner of my eye while feigning interest in gallon-sized packages of M&Ms. I’m not one of those people, constantly searching for redemption in the acid-free pages of an expensive hardcover. But invariably I see a title I just can’t resist, and with flushed face I start reading the jacket of some book that promises me success in business, health, or love.

Almost all the books on the table are in this category—self-help—and its very presence intrigues me. Popular opinion on self-help ranges from ridicule to accusations of outright fraud, yet the bestseller lists practically burst with books like Gottlieb’s Maybe you should talk to someone (self-help through therapy), Epstein’s Range (how generalists can outperform specialists), and Levitin’s Successful Aging (no explanation needed).

We’re embarrassed by self-help, but we’re also attracted to it. We like reading it, but we’re skeptical that it works. We suspect self-help isn’t useful, but everyseriouslist of business books turns out to be comprised entirely of self-help books.

And, perhaps most infuriatingly, rejecting self-help turns out to be hard. Any attempt to articulate a theory against self-help ends up sounding eerily like self-help itself. Much has been made about the rise of self-care: an alleged rejection of the insecure striving encouraged by self-help. But somehow these new “anti-gurus”—Marianne Williamson, Mark Manson, Sarah Knight—sermonize about acceptance and tranquility in exactly the same tenor as the self-help gurus before them.

So which is it? Is our obsession with self-help embarrassing or admirable? Is self-help snake oil or salvation?

I’m going to argue that it’s both. Some self-help is terrible, individualistic hucksterism that the US has exported around the world. But good self-help also exists, and it provides a high-leverage way to lead a better, more fulfilling life.

Let’s talk about wisdom. There are lots of definitions of wisdom, but I like Paul Graham’s: “a wise person knows what to do in most situations, while a [knowledgeable]1 person knows what to do in situations where few others could.” In other words, wise people are moderately successful in many domains, while knowledgeable people are very successful in a few. Here’s a beautiful graph I made:

So a neurosurgeon is smart because she can solve a narrow set of problems that few can, but she may not be wise because, on balance, she can’t solve as many of life’s problems as someone else. (She could also be wise of course, but it’s not related to her smartness.)

How did the doctor become so smart? Well, she read books of course! And the stuff in those books was highly-specific, arcane knowledge about anatomy and neurons and…brain stuff. This brain stuff is devilishly difficult to learn, but once our doctor has completed her (lengthy) training she can solve problems few can.

But what if you wanted wisdom, not knowledge? Are there books that contain wisdom? In other words, are there books that give you general-purpose, one-size-fits-all advice for navigating life?

Of course there is! It’s called self-help.

The stuff in, say, Angela Duckworth’s Grit is wisdom. You can tell because

1. It’s incredibly easy to understand—e.g. “sticking with long-term goals improves life outcomes”

2. But very difficult to apply to real life

WisdomKnowledgeEasily understoodDifficult to learnWidely applicableNarrowly usefulHard to implementEasy to implementSelf-helpTextbooks

Viewing self-help through this wisdom/knowledge lens clarifies the above paradoxes. We’re embarrassed by self-help because (at its best) it’s full of banal platitudes—but these are platitudes because they’re so general. Specific rules like “if your boss likes golf and you want a raise, ask for it while taking her golfing” are too specific to be wisdom.

And we like reading self-help because it makes sense. When Tim Ferriss tells us to ruthlessly cut meetings out of our lives, it seems obvious! We nod our heads vigorously in agreement. But a few weeks later we find our calendar unchanged. We look back on The 4-Hour Work Week and think, what a bunch of nonsense, this whole self-help thing is bogus.

But this is a hallmark of wisdom: it’s trivial to read but nearly impossible to put into practice. We feel divinely inspired while reading Minimalism, but when it’s time to actually cull our wardrobes, it turns out we have good reasons for keeping everything! For wisdom, the devil is in the details, and the details are exactly what nobody else can help you with.

The wisdom/knowledge distinction also explains why there’s such a large overlap between business books and self-help: “business” has so much conceptual real estate that solving “business” problems requires tools that are closer to wisdom than to knowledge—no business book can predict what sorts of situations (businesses, market conditions, etc.) the reader will encounter, so instead it offers general, obvious-sounding rules.

Okay you get it: self-help is wisdom. So what?

The wisdom/knowledge distinction is more than a curiosity; it can help us get more from the self-help genre. Once we understand the purpose of self-help (acquiring wisdom), we’re in a better position to find good self-help and to extract the most from it. Here are a few tips and tricks.

Read self-help that makes sense

A good rule of thumb is that arcane wisdom is rarely right. By arcane I mean anything that is not immediately obvious, like Gwyneth Paltrow’s exhortation to put rocks in your vagina. So advice can either be arcane or obvious, general or specific.

And we want the general and obvious advice. If you hear something dubious like “drink lemon juice, cayenne pepper, and maple syrup for 40 days straight” because it will “eliminate toxins…cleanse the kidneys…[and] purify the glands” you should first ask yourself: is this an obvious universal truth? Does this make sense? Can I apply this to a variety of situations?

If the answer is “no,” then it probably isn’t the kind of sage wisdom we’re looking for. That isn’t to say that specific knowledge can’t dramatically improve your life—certainly reading about new lifesaving medical treatments is worthwhile—but a good rule of thumb for filtering out bad self-help is to ask yourself: is this obvious?

It turns out that almost all the criticism of self-help involves stuff in the lower-right quadrant. When the New Yorkerpokes fun at at Tim Ferriss they dwell on his human growth factor injections, his resveratrol overdose, his habit of weighing his feces—but they completely ignore Ferriss’ more banal platitudes about defining your goals, shutting out distractions, and living intentionally. But it’s these latter bits of wisdom that have made Ferriss famous, and when we read Four Hour Work Week it’s this wisdom that enthralls us, not his breezy tutorial for doubling your reading speed.

A good corollary here is that old self-help tends to be better, because

1. Wisdom tends to be stable over time

2. Pseudoknowledge is eventually exposed

Examples, examples, examples

Anecdotally, it seems that general rules require more examples to be understood. Carnegie’s “be hearty in your approbation and lavish in your praise” sounds great. But I need to hear about Charles Dickens being spurred to greatness by a little praise from an editor, how a man’s horrid children became angels after some kind words from their parents, how a business owner raised productivity by complimenting the craftsmanship of his workers. When we hear enough examples we can begin to see the places in our own lives where this wisdom fits.

Be patient (with self-help and yourself)

It’s a common complaint that self-help books only give high-level advice and never explain specifically how to implement these changes.

But the whole thing with wisdom is that you can’t prescribe a one-size-fits-all method of application. Good wisdom applies to so many different situations, in so many ways, that going through every possibility would take millennia.

So this is the final lesson: self-help is hard. We shouldn’t beat ourselves up if one reading of Letters from a Stoic doesn’t transform us overnight. Reading wisdom is the easiest part of becoming wise.

A large part of the self-help’s bad reputation is due to the fact that lots of people read self-help, but those same people seem to keep reading self-help.

Inc. Magazine has a particularly infuriating piece with the alarmist title 11 Billion Reasons The Self Help Industry Doesn’t Want You To Know The Truth About Happiness. The whole argument is pretty much right there in the title, actually. Since the self-help industry is worth $11 billion—and, you know, some people out there still need help—then obviously the whole thing is this cynical scam where publishers trick the American public into buying stuff they know is bullshit.

But I think we read a lot of self-help because we need to. As I’ve already mentioned, we need lots of examples to drive this wisdom home. We should be more forgiving of self-help (the genre) and more forgiving of ourselves. Putting wisdom into practice takes requires reading, reflection, and practice—but it’s worth it.

Don’t be embarrassed

We (the cognoscenti) are terrified of sounding trite, of repeating obvious truths, of saying things that have been said for millennia—so we laugh nervously at phrases like “accept yourself,” or we utter them with a certain ironic distance—but the reason these sayings sound trite is that understanding them s trivial, it is easy and available to anyone, regardless of social circle or education. But practicing these things is incredibly hard.

”…in the day to day trenches of adult existence, banal platitudes can have a life or death importance.”

- David Foster Wallace, This is water

In a discussion about the perils of self-help culture, British journalist Will Storr told the New Yorker that “He [Storr] is quick to say that he isn’t encouraging anything quite as clichéd as self-acceptance. At the same time, he reports that he has, in fact, come to accept himself.”

I think that says it all.

1. Graham compares wise with smart, but (a) smart doesn’t lend itself to a great noun (smartness?) and (b) I care more about the material, not the subject. [return]

“If we send a single push notification, 7% of people open the app,” he said. “But if we send two push notifications, five minutes apart, 12% of those people open the app.”

Intuitively we can sense that sending two, or ten, push notifications in five minutes is—long term—probably a bad idea. This is how we got follow-up emails, clickbait articles, and Satan’s own exit pop-ups.

But this overoptimization is part of a larger, more pernicious trend: the rise of ubiquitous, high-fidelity feedback.

Instagram gives you instant feedback on your life; Twitter gives you instant feedback on your thoughts; exit polls give politicians instant feedback on their positions; focus groups give Hollywood instant feedback on their movies.

And now we have Tay the racist Twitter bot, the Marvel cinematic universe, and Donald Trump.

But clearly we need some feedback. I’m not yearning for the days of waterfall software development. So what’s the right amount of feedback?

Iam by no means a machine learning expert. But there’s a metaphor in machine learning that I really like, and I’m going to abuse it a bit to make a point.

Imagine you’re at the top of a mountain. You’re trying to hike to the bottom of the mountain, but unfortunately you’re surrounded by a dense fog.

Luckily you have a special instrument—let’s call it a slope-finder—that tells you which direction is most downhill from your current position. So you hike a little bit, check the instrument, change course, and hike a bit more.

This is the famous analogy for gradient descent, a machine learning technique for (among other things) updating the weights of a neural network.

But it’s also a good analogy for problem solving in general: we move in the most promising direction, periodically updating that direction based on external feedback.

So our original question—How much feedback should we absorb?—becomes: How often should we check our slope-finder?

Picture Janet Yellen descending this foggy mountain. JY is pretty cautious, so she checks her slope-finder after every single step. It’s slow, it’s tedious, but she knows she’ll never overshoot her destination.

But now imagine Bear Grylls trying to get to the bottom of our foggy mountain. BG checks his slope-finder once and then sprints sixteen miles in that direction. When he finally checks his slope-finder again it points him 90-degrees left. He turns and sprints for another 16 miles.

BG, you can imagine, will die of exhaustion before he stops in the bottom of that valley. He will overshoot the bottom constantly, criss-crossing it like an idiot.

But JY also has a problem: she’s very likely to get stuck in the nearest hole. You can imagine her taking tiny little steps downhill, constantly following the slope-finder, right into the bottom of some dried-up lake.

And so it is with feedback. If you ignore all feedback (like BG) and charge forward, immune to all criticism and advice, then you’ll probably never reach your goals.

If on the other hand you’re a slave to feedback (like our imaginary JY) you’ll move slowly and likely settle for a mediocre solution, because the best solution sometimes requires walking uphill.

So what are we to do?

The machine learning people have some ideas. What we’re calling feedback is inversely correlated with machine learning’s step size.1 The greater the step size, the less often you’re checking in for feedback.

Unfortunately there’s no right answer to choosing a step size: sometimes science is more art than science. It depends a lot on what your loss function—err, mountain range—looks like.

But what’s interesting is that often changing the step is the best strategy. From Wikipedia:

In order to achieve faster convergence, prevent oscillations and getting stuck in undesirable local minima the [step size] is often varied…[emphasis mine]

Again—I am not an expert—but here are a few ML strategies for varying the step size (aka feedback).

1. Gradually decrease it (annealing)

The idea here is to start with a relatively large step size and gradually decrease it over time. You take large steps at the beginning (less feedback), when you’re far away from your goal, but smaller and smaller steps (more feedback) as you zero-in on the optimal solution.

2. Cycle it between values (SGDR)

Stochastic gradient descent with restarts is a fancy way of saying “gradually decrease the step size over time, but occasionally jump back to a large step size.” The “restart” part helps you jump out of local minimums, and makes it more likely that you’ll finally settle on a globally optimal solution.

3. Give it momentum

This one is intuitively appealing. The learning rate increases when the error cost gradient heads in the same direction for a long time, allowing the “ball” to skip over local minima and to quickly traverse long downhills.

So, tying this all together, here’s what I’m taking away from this tortured metaphor:

1. Too little feedback is bad (you’ll never find an optimal solution)

2. Too much feedback is bad (you’ll move slowly and get stuck in local minima)

3. It’s best to vary the amount of feedback (according to the above strategies)

There is a time for lots of feedback (when you’re close to your goal) but there’s also a time for charging ahead, Twitterverse be damned.

I think this is difficult for humans. It seems we’re either very sensitive to feedback (e.g. Sarah doesn’t like my haircut? Let me shave my whole head) or totally hardened to it (e.g. stubborn and arrogant). But gradient descent tells us that we need to occasionally take lots of feedback and occasionally ignore it.

“It is easy in the world to live after the world’s opinion; it is easy in solitude to live after our own; but the great man is he who in the midst of the crowd keeps with perfect sweetness the independence of solitude.”

- Ralph Waldo Emerson, Self-reliance

I worry that the deluge of instant, high-fidelity feedback is driving us toward local minima. Usually feedback encourages us to move down the path of least resistance, to make the easiest gains—but often the best gains are made by walking uphill for bit.

Imagine you’re one of these people who tweets a lot. You tweet about a lot of things—your pet turtle, your midnight snack, your controversial opinion about fiddle-leaf fig trees—but sometimes you tweet about Donald J. Trump.

And, lo and behold, you notice that your tweets about Trump get more likes than your other tweets. So—consciously or unconsciously—you start tweeting more and more about Trump, saying more and more outlandish things, until you wake up one day and discover you are a Trump shock jockey.

“Trump shock jockey” is, at best, a locally optimal solution. You could have many more followers, and a much better Twitter account, if instead of taking tiny steps at the behest of your current followers, you took big swings and found a more worthwhile subject matter.

“The young character, which cannot hold fast to righteousness, must be rescued from the mob; it is too easy to side with the majority.”

- Seneca, Letters from a Stoic

1. More commonly known as learning rate, but step size is more consistent with the mountaineering analogy. [return]

You have a mind-shattering headache. You’re standing in the aisle of your local CVS, massaging your temples while scanning the shelves for something—anything—to make make the pain stop.

What do you reach for? Tylenol? Advil? Aleve?

Most people, I imagine, grab whatever’s cheapest, or closest, or whatever they always use. But if you’re scrupulous enough to ask Google for the best painkiller, here’s how your friendly neighborhood tech behemoth would answer:

Oh thanks Google that’s just all of them.

If you’re among the 77% of Americans that Google their health problems, insipid answers like this won’t surprise you. But we should be surprised, because researchers carry out tens of thousands of clinical trials every year. And hundreds of clinical trials have examined the effectiveness of painkillers. So why can’t I Google those results?

And so in the year of our lord 2017 I had a Brilliant Startup Idea: use a structured database of clinical trials to provide simple, practical answers to common medical questions.

As a proof-of-concept I tried this by hand: I made a spreadsheet with every OTC painkiller trial I could find and used R to run a network meta-analysis, the gold standard of evidence-based medicine.

The results were pretty interesting, and exactly the kind of thing I was looking for back in the sad sterile aisles of CVS:

A wave of exhilaration washed over me. Here was a problem that

1. Was interesting

2. Could help people

3. I knew how to solve

A perfect bullseye. After a few hours searching domains I came up with a name for my project: GlacierMD.

Over the next nine months I would quit my job, write over 200,000 lines of code, hire five contractors, create a Delaware C-Corp, add four doctors to my advisory board, and demo GlacierMD for twelve Bay Area medical practices. I would spend $40K of my own savings buying clinical trials and paying contractors to enter said trials into the GlacierMD database.

On July 2, 2018, GlacierMD powered the world’s largest depression meta-analysis, using data from 846 trials, beating Cipriani’s previous record of 522.

Choirs of angels sang in my ears. Here I was, living the Silicon Valley dream: making the world a better place through technology.

Two weeks later GlacierMD was dead.

“That’s an awesome idea,” said Carl. “It sounds like something worth working on.”


Carl was my boss. We worked at a startup that leveraged autonomous blockchains to transfer money from naïve investors to slightly less naïve twenty-somethings. There are worse gigs.

And here was Carl telling me that my startup idea would bring such benefit to humanity that I simply had to quit, his roadmap be damned. I nodded knowingly, feeling the weight of this responsibility resting on my proud shoulders.

“Thanks Carl,” I said. “I’ll try to mention you when I accept my Nobel.”

I quit two weeks later and started coding at a blistering pace. I drew all sorts of inscrutable diagrams with dry-erase pens on my parents’ windows. I hired a motley crew of Egyptian contractors to start entering clinical trials into my database. I commissioned a logo, registered my domain, and started obsessing over color schemes.

When I finally finished the MVP I showed it to the head of product at the company I’d just left. I watched him as he watched my demo, waiting for his eyes to melt with the glory of it all. Instead he just sorta shrugged.

“Lots of people make medical claims on the internet,” he said. “Why should I trust yours?”

I started babbling about network meta-analyses, statistical power, and p-values, but he cut me off.

“Yeah okay that’s great but nobody cares about this math crap. You need doctors.”

Goddamnit he was right. If nobody could be bothered with the math, then I was no better than Gwyneth Paltrow hawking vagina eggs. To build trust I needed to get endorsements from trustworthy people.

So I called up some friends, some buddies, some friends-of-friends. “Would you like to be an advisor for my cutting-edge health-tech startup?” I’d ask, while eating Dominos in my parents’ laundry room. I’d give them 1% of this extremely valuable, high-growth startup and in exchange I could plaster their faces all over my website.

Four of these doctors agreed. This is called making deals ladies and gentlemen and I was like the lovechild of Warren Buffet and Dr. Oz.

Things are going great. My friends and family all tell me they love the site. Even some strangers on the internet love it. “I know right,” I tell them. “So how much would you pay for this?”

“Hahahahahahah,” they say in unison. “Good one!”

I forgot that the first law of consumer tech is nobody pays for consumer tech. But no problemo, I say to myself. This is why Eric Schmidt invented ads. I’ll just plaster a few banners on GlacierMD and bing bang boom I’ll be seasteading with Peter Thiel before Burning Man.

But then I look at WebMD’s 10-Qs and start to spiral. Turns out the world’s biggest health website makes about $0.50/year per user. That is…not enough money to bootstrap GlacierMD. I’m pouring money into my rent, into my Egyptian contractors, into AWS—I need some cash soon.

What I need are people willing to pay for this thing. What about doctors? Doctors have money, right? Maybe doctors, or practices, or whatever—someone in the medical industry—maybe they would shell out some cash for my on-demand meta-analyses.

So I listened to a few podcasts and became a sales expert. I started cold calling people using scripts from the internet and tried to convince them to sit through a GlacierMD demo.

In the meantime I receive some worrying messages from my Egyptian contractors.

“I think it’s time to talk about a raise,” one of them says.

“I feel that I have become exceptional at my job,” says another. “Please consider a raise or I will stop working.”

“Please increase my pay,” says the third, including helpful screenshots demonstrating how to give said raise through the Upwork website.

Are my contractors unionizing? I wonder. I glance obliquely at my shrinking bank account statement, grit my teeth, and approve the raises. At this rate I’ll hit zero in a matter of weeks.

But my sales calls start paying off. Miraculously I find some doctors that are willing to talk to me. So I borrow my parents’ car and drive out to the burbs to meet a doctor I’ll call Susan.

Susan has a small practice in downtown Redwood City, a Silicon Valley town that looks 3-D printed from the Google Image results for main street.

Susan is a bit chatty (she’s a psychiatrist) but eventually I demo GlacierMD. I show her how you can filter studies based on the demographic data of the patient, how you can get treatment recommendations based on a preferred side effect profile, how you can generate a dose-response curve. She oohs and aahs at all the right points. By the end of the interview she’s practically drooling.

Hook, line, and sinker I think to myself. I’m already contemplating what color Away bags would look best in the back of my Cybertruck when Susan interrupts my train of thought.

“What a fun project!” she says enthusiastically.

Something in her tone makes me pause. “Uh, yeah,” I say. “So what would you imagine a product like this—one that could change the very practice of medicine—how much would you pay for such a service?”

“Oh, uh—hmmmm,” she said. “I don’t know if we can spare the budget here, to be honest. It’s very fun…but I’m not sure if our practice can justify this cost.”

If you read enough sales books most of them tell you that when people say your product is too expensive what they really mean is your product isn’t valuable enough. Susan acted like I was offering her Nirvana as a Service so the conversation has taken quite a wild turn.

“So you don’t think this product is useful?”

“Oh sure! I mean, I think in many cases I’ll just prescribe what I normally do, since I’m comfortable with it. But you know it’s possible that sometimes I’ll prescribe something different, based on your metastudies.”

“And that isn’t worth something? Prescribing better treatments?”

“Hmmmm,” she said, picking at her fingernails. “Not directly. Of course I always have the best interests of my patients in mind, but, you know, it’s not like they’ll pay more if I prescribe Lexapro instead of Zoloft. They won’t come back more often or refer more friends. So I’d sorta just be, like, donating this money if I paid you for this thing, right?”

I had literally nothing to say to that. It had been a bit of a working assumption of mine over the past few weeks that if you could improve the health of the patients then, you know, the doctors or the hospitals or whatever would pay for that. There was this giant thing called healthcare right, and its main purpose is improving health—trillions of dollars are spent trying to do this. So if I built a thing that improves health someone should pay me, right?

I said goodbye to Susan and tried to cheer myself up. I had ten more meetings with doctors all over the Bay Area—surely not all of them were ruthless capitalists like Susan. Maybe they would see the towering genius of GlacierMD and shell out some cash.

But in fact everyone gave me some version of Susan’s answer. “We just can’t justify the cost,” a pediatrician told me. “I’m not sure it’s in the budget,” said a primary care physician. “It’s awesome,” said a hospitalist. “You should try to sell this!” Ugh.

So in July 2018, nine months and $40K after starting GlacierMD, I shut it down. I fired my contractors, archived the database, and shut down the servers. GlacierMD was dead.

Make something people want. It’s Y-Combinator’s motto and a maxim of aspiring internet entrepreneurs. The idea is that if you build something truly awesome, you’ll figure out a way to make some money off of it.

So I built something people wanted. Consumers wanted it, doctors wanted it, I wanted it. Where did I go wrong?

Occasionally I like to disconnect from the IV drip of internet pseudoknowledge and learn stuff from books. I know, it’s weird—maybe even a bit hipster. But recently I read Wharton’s introductory marketing textbook, Strategic Marketing Management. The very first chapter has this to say:

“To succeed, an offering must create value for all entities involved in the exchange—target customers, the company, and its collaborators.”

- Strategic Marketing Management

All stakeholders. You can’t just create value for the user: that’s a charity. You also can’t just create value for your company: that’s a scam. Your goal is to set up some kind of positive-sum exchange, where everyone benefits, including you. A business plan, according to this textbook, starts with this simple question: how will you create value for yourself and the company?

I winced audibly when I read this. How much time I could’ve saved! If I’d articulated at the beginning how I expected to extract value from GlacierMD, maybe I would’ve researched the economics of an ad-based model, or I would’ve validated that doctors were willing to pay, or hospitals, or insurance companies.

A few months after shuttering GlacierMD and returning to corporate life my buddy pitched me a new startup idea.

“It’s called Doppelganger,” he said. “It’s super simple—you upload a selfie to the database, and then it uses AI or whatever to instantly find everyone in the database who—”

“Looks like you,” I finished for him.

“Exactly,” he said, grinning ear to ear. “How awesome would that be? You should build it!”

I mean, I dunno, it sounds like something fun to do at parties. In a narrow sense, it’s something I want, but there’s no way in hell I’m going to devote any time to this. Doppelganger has created value for the customer but not for the company.

“Call me when you have a business plan,” I said, lacing up my Allbirds and riding my Lime scooter into the sunset.

Epilogue: I’m taking another stab at this whole GlacierMD thing! Follow along at glaciermd.com.

This lasts for about thirty seconds before I get bored. The guy next to me chuckles at something on Kanye’s Twitter feed. Wait Kanye? I thought he was off Twitter! In seconds my own phone is out, and I’m furiously scrolling through Reddit, looking for fresh memes.

And so goes the common trope: we all hate our phones, social media, and Netlfix—but we also love them.

Eventually tiring of plain-vanilla bus-ride superiority, I brought a book with me on the bus. Not a Kindle, or an iPad, but a bona fide bound book.

After every sentence or two I would gaze incredulously around the bus with a sneaky grin. These dumb cows, I thought, look at them melt their brains with their stupid social feeds. They can’t focus like I can. I was about to turn back to Sapiens when I noticed a woman in the back of the bus who wasn’t staring at her phone. Nor was she reading a book. She wasn’t even listening to a monotonous NPR podcast.

This psychopath was sketching. She had a charcoal pencil and was sketching a human figure. She didn’t look happy exactly. But she looked engaged. She looked intent. And all of a sudden my smugness fled. I realized that reading a book was really just like reading Reddit—both were consumptive activitives.

Were previous generations really better off because they merely watched TV, or listened to radio, or read books? All of these activities are passive. All of these activities involve letting external thoughts temporarily replace your own. Today’s smartphones differ from medieval books only in degree—all media is created to be consumed.

I had to start creating.

Corín Tellado was a Spanish author who wrote over 5,000 books in her 81-year lifespan. That’s over sixty books a year, assuming she came out of the womb fully literate. She claimed she could write a book in two days, and I’m inclined to believe her.

I can’t write a tweet in two days. Nevertheless, I was inspired by Tellado to stop consuming and start creating. If we categorize all our waking moments as either consumptive or creative, I was probably at a 10:1 ratio, and that’s only if I label “copy/pasting stock images into Keynote presentations” as “creative.” Tellado was probably closer to 1:1. How could I get closer to that?

On my next bus ride I pulled out my phone and opened the iOS notes app. I created a new note titled Essay.

I stared at that off-white screen for several long minutes without typing anything. I don’t have anything to write about, I thought. Also, I’m not even a good writer…maybe I should look online for some tips…

But that was a bad idea and I knew it. So I swallowed my pride and just started writing:

Why I think creating stuff is a good thing.

An essay by Tom C.

This was going to be rough.

I think people consume too much, and don’t create enough. This is bad I think.

Like, really rough.

But I kept typing; pouring inane, kindergarten-level drivel into my phone.

The average person can type about 20 words per minute on a phone. My commute is about 30 minutes each way, so that’s a maximum of 1,200 words a day. Realistically I’m going to have to spend some time thinking about what to write, so let’s be conservative and shoot for 500 words a day.

I read somewhere that a good blog post is 1,600 words…so if I work fast I could pump one out every three days, right?

The first day I wrote 100 horrible words. Really super bad. No one will ever see these words.

Day two I wrote 80 words. I think I psyched myself out over how bad the previous day went.

The third day I only kept writing out of stubbornness. I knew this project was doomed, but I felt like really dooming it.

By then end of the first week I had 500 words in the bank. I didn’t feel good about it, but it felt slightly better than scrolling through 47 #AndThatsWhyYouAreMyEx tweets.

My essay was garbage. But it was my garbage.

So I kept at it, day after day. I once again started feeling smugly superior to my fellow bus riders. Look at me creating, I thought. Look at me contributing to the world, while these reptiles just distract themselves with their phones until they die.

This arrogance lasts for a few seconds until I re-read the stream-of-consciousness dogshit I’m typing into my phone.

I’ve never done sales, but I know a little bit about the “sales funnel.” You hire college grads to make 500 cold calls in a week. Of those 500 calls, maybe 5% agree to watch a demo of the product. Then some slightly-more-senior sales people do this demo, and perhaps 20% of those demos end up in contract negotiation. And 60% of those end up as actual sales. That means you need 500 cold calls to make three sales.

I think that’s what creating is like. You need to write 500 words to get three good ones. Or 500 sketches, business ideas, or recipes. If you’re really really good, you can increase your overall conversion rate from 0.6% to 1%—but the most reliable way to get better results is to just produce more crap.

So I kept at it. week after week. Sometime in the middle of the fourth week I had 3,000 words in my Notes app. That’s when I started editing.

Editing is hard because you realize how bad you are. But editing is easy because we’re all better at criticizing than we are at creating.

The good thing about forcing yourself to produce a bunch of garbage is that you don’t feel bad deleting it. You’re not married to any of your work, because you wrote it half-awake on the 1BX while some crazy dude negotiated his fare with the driver. I gleefully deleted hundreds of words, trying not to think about the neanderthal who must’ve written them.

It was exactly a month after I started this creative project that I finished this first essay. I read it over for the bazillionth time and thought this only a small tire fire.

In it’s final form, the essay was almost exactly 1,600 words. I called it Consume less, create more.

You are, in fact, reading that essay right now. It may not look like it, but this is the product of an entire month of bus rides. If you live in San Francisco and saw someone on the 1BX smashing the face of his phone like he was angry at it, you probably saw me writing this essay.

Perhaps three people will read this essay, including my parents. Despite that, I feel an immense sense of accomplishment. I’ve been sitting on buses for years, but I have more to show for my last month of bus rides than the rest of that time combined.

Smartphones, I’ve decided, are not evil. This entire essay was composed on an iPhone. What’s evil is passive consumption, in all its forms.

Twitter, Facebook, Instagram—we can all agree that these are serious timewasters. But what about The Economist or War and Peace? How much can you really remember from all of those New York Times op-eds you’ve read? Could you summarize the major themes of Grapes of Wrath?

Most knowledge worth having comes from practice. It comes from doing. It comes from creating. Reading about the trade war with China doesn’t make you smarter—it gives you something to say at dinner parties. It gives you the illusion that you have the vaguest idea what is happening in our enormously complex world.

A lot of ink has been spilled about the perils of modern technology. How it distracts us, how it promotes unhealthy comparisons with others, how it makes us fat, how it limits social interaction, how it spies on us. And all of these things are probably true, to some extent.

But the real tragedy of modern technology is that it’s turned us into consumers. Our voracious consumption of media parallels our consumption of fossil fuels, corn syrup, and plastic straws. And although we’re starting to worry about our consumption of those physical goods, we seem less concerned about our consumption of information.

We treat information as necessarily good, and comfort ourselves with the feeling that whatever article or newsletter we waste our time with is actually good for us. We equate reading with self improvement, even though we forget most of what we’ve read, and what we remember isn’t useful.

So stop reading and start creating. Paint, draw, compose, code, or plan. It will be hard. It will be slow. It will be frustrating. But I promise it will be worth it.