· Weekly Bites · 3 min read
Weekly Bites #1: LLM Hallucinations, Predicting the Future, and Early Hires
Managing LLM hallucination trade-offs, why Zillow lost $550M on predictions, and finding early startup hires.
This week’s bites:
- LLM Hallucinations: Trade-offs between confidence and accuracy in AI responses
- Predicting the Future: Why Zillow’s $550M loss shows AI prediction limits
- Early Hires: Finding founding employees through your network
LLM halucination
People often complain about hallucination in LLMs, i.e., they make up things and present them as fact. Since the underlying tech predicts the next most probable tokens, they do not know if they are wrong. It is a trade off between false positives and false negatives. By false positives, I mean confident but wrong answers. By false negatives, I mean refusing or staying vague when the model could answer.
Same with people, we do not always know we are wrong. Some always say what is on their mind, right or wrong. Some only speak when they are sure, and end up saying nothing. Both have costs.
With techniques like grounding search and extended thinking, we can reduce hallucination a lot. But we likely cannot fully eliminate it, there is always a small chance. So there is a trade off, and we have to find a balance. One simple balance is to lower temperature and require sources for factual claims. That reduces false positives, but may increase false negatives.
Predicting the future is hard
Eight years ago Zillow ran a $1.2M Kaggle contest (biggest prize to that date) with a simple twist I loved: no hidden labels. You predict house prices for the next 3 months, then wait for the real-world results to show up. That looked fair and clean. No leak possible. It also made the whole thing a little bit suspenseful. Only time does the judging.
A few years later, Zillow tried to use price forecasts in the real world with Zillow Offers. They bought homes and resold them. For a while the Zestimate even helped decide the starting offer price. Then in 2021 they shut it down, reporting a $550M loss and saying near-term price forecasts were too unpredictable. I see a connection, but I am not saying one caused the other.
My takeaway: predicting the future is still hard in practice. Models can be sophisticated, but there are too many variables out there. LLMs push AI to a new level with lots of new capabilities. But predicting the future is still elusive, probably for a long time.
Startup early hires
In the beginning, cofounders can often cover the key work, so you may not need to hire yet. But if you are solo or very lean, you might need a few “founding employees” (not cofounders).
The main question is: how do you find them?
Try your own network first, ideally even before you launch. Why?
- Early hires can make or break the company. You want people who are qualified, ambitious, and a good fit.
- Joining this early is risky for them. Knowing you and your track record helps them commit without worrying the company might fold in a few months.
That should be sufficient for 3 to 6 months. During that time, the founder should expand the network to prepare for the next hires, so both sides have time to get to know each other and check for good fit.
Final notes:
- How to retain your early hires? One of the best ways is to let them work at the technical frontier where they can build something meaningful.
- Be careful hiring close friends. If the business goes south, your relationship might too.
