The Power of 95% Confidence
Keeping You and Your LLM Accountable
The more I work with LLMs, the more bizarre the experience seems to me. By default, I (and many others) communicate with LLMs as if they are human beings who are living in our computers.
LLMs aren't sentient (at least not yet!), but the back-and-forth nature of our conversations with them makes it seem like they are. I think it's one reason why people naturally say "please" and "thank you" in their chats. We actively know that we are communicating with a machine, but our subconscious actions tell a more complex story.
What tends to break that veil of sentience is when the LLM doesn't understand your vision or requirements. Whether it's through hallucinations or by going off track, this is a widespread problem and one that many intelligent people are working on.
Frankly, I think that it's the thing that turns "non-technical" people off of LLMs. They have a mission-critical task to be done, try delegating it to the LLM, see plenty of errors and falsehoods (some obvious and some not-so-obvious) and then do all of the work themselves.
These problems aren't going away anytime soon. With that said, I've been borrowing another strategy to address these problems. It's simple, yet it has helped me avoid traps that could lead to further bugs or problems in my Cursor projects.
Like I mentioned two weeks ago, in all of my Cursor projects, I create a task manager. At its core, the task manager helps me stay on track and keeps the LLM accountable. I go step by step and ask Cursor to review the tasks from earlier so it better understands what we've done to that point.
But whenever I'm starting a new chat, I conclude my prompt with the following:
"If you aren't at least 95% sure of any of the instructions or tasks that you are about to take on, please stop and ask me clarifying questions."
This has been surprisingly effective. There are two benefits here: first, you ensure that the LLM knows what you want. That's the most obvious benefit.
But it's the second benefit that is arguably more important.
Essentially, the LLM's clarifying questions can spark plenty of ideas. They can be everything from future feature ideas to ideas on how to best structure your project. There have been plenty of times where the LLM highlighted something that I had missed (or hadn't even thought to consider) when building out projects.
Even if you aren't coding, this "prompt coda" can be really helpful. The LLM can basically raise hidden assumptions that you used when outlining your task.
It doesn't entirely solve the hallucination problem. Even if you use the 95% certainty language, the LLM can fail in executing your task. Or it can go off on a tangent that you didn't anticipate. That being said, speaking from experience, this prompt coda has helped me better direct the LLM and better execute on what I want. Plus, it has helped me unearth some hidden assumptions in the directions that I was providing.
As always, I encourage you to try this out for yourself. The best way to do any of this is to get first-hand experience. If you try the 95% confidence language and see some success, let me know! I'd be happy to hear about it.
Something I Found Interesting This Week
Context7: MCP servers have been all the rage in recent months. While I haven't used them as much as other developers, I have found two MCP servers to be helpful as I code with Cursor. The first is Supbase's MCP, which lets me more easily view and edit the database in my projects. But beyond that, I've found Context7 to be really helpful.
Essentially, Context 7 is an MCP server that helps you access the most recent documentation when you are coding. LLMs are trained on old data and programming languages frequently change. Context7 solves this problem by accessing the most recent documentation and injecting that documentation into your prompt. It reduces coding hallucinations and makes the development process much easier. It's free and I recommend trying it out.
Prompt of the Week
Here's a different prompt that I've found interesting:
"From what you know about me: what parts of my life could I better organize? How would you develop and execute a plan? Lay out your plan step by step."