Lee
Basnight

AI Product Management

Some random Tuesday morning thoughts about AI Product Management...

Most AI products fail in the same place, which is that gap between what the model can do and what the user actually experiences. Closing that gap is a large part of AI product management and it looks nothing like the product management most people are used to.

Traditional product management assumes deterministic software. You define a feature, engineering builds it, then the same input reliably produces the same output every time. AI breaks that assumption at the root. You are now shipping a PROBABILISTIC system that can give three different answers to the identical question, so your job becomes managing behavior across a whole range of possible outputs while keeping that range inside acceptable bounds. *That part

That shift changes almost everything about the role. Your spec turns into a set of evaluations, because the only honest way to know whether an AI feature works is to test it against hundreds of real cases and measure how often it lands. Your success metric centers on acceptance rate, task completion and how often a user trusts the output enough to act on it. Your roadmap has to account for model behavior you cannot fully control, which means designing fallback paths, guardrails and human review for the moments the model gets it wrong.

Good AI product management is spending most of your time making boundary decisions. It's knowing which parts of the product should be handled by a human and which parts belong to plain deterministic code. Because just using AI for everything, is how you end up with an inefficent and expensive product that hallucinates in places where a simple rule would have been perfect. Prompts and instructions should be treated as a real product surface that gets versioned and tested. They design the feedback loop that turns everyday usage into the data that makes the next version better.

The model is just the tool, while the ACTUAL product is everything you build around that tool, to make it reliable, trustworthy and genuinely useful to a human being who's trying to complete a task. Anyone can do anything with AI now, so the real work is knowing how to build around it. ✌🏾

You Can BUILD Netflix at Home!

Hey people,

I mean that title literally. With today's AI-assisted development tools, I could build something close to Netflix from my own house, which means almost anyone reading this could do the same. The ability to build software has been genuinely democratized, which is one of the most exciting shifts of this whole era.

Being able to build the thing is usually where the trouble starts, because the sheer ability to build quietly convinces people to skip everything that makes the build worth doing. This is the part I keep coming back to with founders: you have to lay the plumbing before you write the poetry. The code is the poetry. The plumbing is the business system that carries your product to an actual human being who will pay for it.

So before the first line of code gets written, a few questions deserve honest answers. Is there real demand for the product you want to build, or does that demand mostly live inside your own head? Do you have an outreach and marketing plan with the people to actually run it, or is the plan quietly just "build it and they will come"? What does the roadmap look like at five months, ten months and fifteen months, once the novelty has worn off and the daily work becomes maintenance and iteration? Those answers decide whether you are building a business or building an expensive hobby.

And to be clear, the expensive hobby is completely valid. If you are building to explore, to learn, or purely for the joy of making something, that is genuinely awesome and none of this applies, so build freely and enjoy the process. The plumbing conversation is for the people who intend to turn the build into revenue, because those are the builds that quietly fail for reasons that have nothing to do with the quality of the code.

If you are sitting on an idea and you want to pressure-test the business system underneath it before you pour months into building, that is exactly the kind of work I do. Reach out at leethepolymath@gmail.com

What AI is Quietly Doing to Modern Work

Everyone's been arguing for two years about whether AI is going to take our jobs and I think we've all been looking at the wrong thing.

This is what's actually going on. AI is quietly showing everybody who in a company is creating value and who was just standing between the work and the people doing it. Nobody designed it to do this, but it's happening. It's a side effect of how good these tools have gotten at the middle-layer stuff that used to be an entire human role. When an LLM can knock out a decent first draft of a brief in twelve seconds, the question pivots from "who writes the brief?" and starts being "who can sculpt that draft into something a client can actually understand." That's a very different question and a lot of orgs are realizing they don't have a great answer.

So what do you do about it? Here's a few things that have been working for me and the friends and companies I've consulted for.

Audit yourself before the system does it for you. Pick some things you did at work last week. Ask honestly whether a decent LLM with the right prompt could've done 70% of that work. If yes, ask yourself what the other 30% is and whether your week is actually built around that part.

Stop simply using systems. Start building them. The person running the spreadsheet is replaceable. The person who built the spreadsheet, defined the rules, knows where the data comes from and decided what to automate is way harder to replace. Make yourself invaluable.

Use the tools. I run into senior people every week who have strong opinions about AI and have spent maybe twenty minutes actually using it. That's a losing pov. The bar is low here. Sit with the tools for an afternoon. Get a feel for them. Form your opinions from real experience instead of a Substack you glanced over.

Make your contribution legible. A lot of people did good work in environments where they never had a chance to explain their contribution. Everyone just gives credit to AI and blames the human contributor. If you can't write down what you contributed in a way someone else would understand, that's your homework. Write it down. Make it plain.

Be a force multiplier, not a gatekeeper. Gatekeepers slow things down in exchange for control. AI eats gatekeepers for breakfast. Force multipliers raise the ceiling on what their team can do and that's a leveraged human skill that AI can't really replace. Be the second one.

The whole thought process is really about honesty. Honesty about what you actually do, what it creates and whether your seat is built to last in a system that just got a bunch of new tools for the parts you used to own.

The good news is that this is one of the most fluid moments the workplace has ever had. The people who come out of this stronger are the ones who get real about their own contribution first and make the adjustments before somebody else is forced to do it for them.

Process is one of the most important tenets of proper AI systems

If you ask me what made AI implementation work at the places I've led it for, I'll tell you it was the process. The decision rules, the SOPs, the version control for prompts, the eval protocols. The boring operational layer that nobody puts on a slide.

I think most AI failures look the same up close. A team picks a tool. The pilot looks impressive. Leadership signs off. A quarter goes by, then another. The tool is technically still in the stack, but most of the team has stopped using it. They've gone back to doing things the way they did them before the AI initiative, sometimes with the AI cost still on the budget. Ask anyone on that team why, and you'll hear answers like "it's faster to just do it myself" or "I never know if I can trust the output for this kind of customer." Both of those are process problems. The model works. The team is missing a shared rule for when the model is the right call.

This is what process work actually looks like in practice. It's a document, often a few pages, that sits in a shared drive nobody opens for fun. It says things like "for outbound sales emails to existing customers, AI drafts go to manager review before send. For internal newsletters, AI drafts go straight to send. For anything in legal or compliance categories, AI stays out of the workflow." It's mundane. Writing it takes a week. Nobody wants to be the one writing it. Once it exists, every member of the team makes the same decision the same way, and a manager can stay out of the loop for individual calls. That's where the efficiency gain comes from. At Trane I'd put 40 to 50 percent of the gain on documents like this. The model was a smaller piece of the story than most people assume.

There's also a cultural reason this layer gets skipped. The language is unsexy. Governance, decision criteria, eval protocols, prompt registries, content policies. It sounds like compliance. It sounds like the part of the company that slows things down. So when AI strategy gets discussed, the conversation tends to drift toward tools and capabilities, because those are the parts you can demo. Process is harder to demo. It's just a doc.

If I were starting an AI program tomorrow, I'd budget more time for process design than felt reasonable. I'd write the SOP first, before I picked a tool. I'd be willing to look slow for a quarter to be fast for the next two years. Most teams skip this work, which is why most AI initiatives stall out.

Real vs. Fake Online

Some morning thoughts. Twelve months from now, even the most trained eye won't be able to tell what's real and what's fabricated online. That's not a prediction meant to spark fear, it's simply where we're heading.

The deeper shift, though, is this: the consumer or client won't care.

We've been here before. People were skeptical of the internet. They doubted streaming. They dismissed blockchain. In every case, once the technology matured, the debate shifted from "Is this real?" to "Is this useful? Does it work for me?"

The same is happening now with AI-driven creative work. In a world where digital output can be perfectly simulated, the only real currency left will be trust and quality. That means businesses and creators will need to let go of their philosophical debates about "purity" or "authenticity" in creative work.

Clients won't ask, "Was this made by a human or a machine?" They'll ask, "Does this solve my problem? Does this move me? Do I trust this brand to deliver consistently?"

This is the inevitability of progress: technology compresses time. What took decades to normalize before is now happening in months. The winners won't be the loudest moral objectors. They'll be the ones who embrace the shift early, without losing sight of quality and trust.

When AI Misses the Mark

As someone in Design Operations, my job is all about making creative work smoother and more effective. For the past four years, I've taken a deep dive into AI implementation for creative pipelines, seeing firsthand how it can boost efficiency and open new doors.

But there's a flip side to this exciting "AI gold rush." Many companies are scrambling to inject AI everywhere, seeing it as a quick fix for saving money. They're doing this without proper data to train the AI, and without enough usability testing to see if it actually helps people. This quick-fire approach often creates more problems than it solves, especially when it comes to customer satisfaction.

When AI misses the mark: my bank experience

This morning, I had a frustrating experience with my bank that perfectly illustrates this problem. My issue was simple, but I was sent to an AI agent first. This bot, clearly not well-equipped, immediately, incorrectly and hurriedly guessed at my problem, offering useless solutions. It was a loop of frustration, wasting my time.

When I finally managed to explain what I needed to the machine and asked to speak to a human, the system did something that really got to me: it sent me to another AI agent. Whether intentional or not, it felt like they were trying to deceive me into thinking I was talking to a person, or just hoping I'd give up. It left a really sour taste. And the thing is, my problem still wasn't solved.

This wasn't just an inefficiency; it was a clear sign of how poor AI implementation can damage trust and leave customers feeling unheard.

AI is here to stay, but so is humanity

The truth is, AI isn't going anywhere. It's already woven into many professional processes and will only grow. But my experience shows why human involvement, or at least the option to connect with real people, will remain crucial. AI should be a powerful tool that helps humans, not a barrier that prevents genuine connection and problem-solving.

For businesses, this means understanding that true efficiency and customer loyalty come from thoughtful, human-centered AI design. It's about empowering your teams and delighting your customers, not just cutting costs.