This is an original summary and career interpretation by H1B Sponsorship Jobs, informed by the linked sources. It is not authored, endorsed or sponsored by those publishers. Source findings and our suggested preparation steps serve different purposes.
What the authors argue
In their September 2026 Lenny’s Newsletter essay, Hamel Husain and Shreya Shankar put error discovery before automated measurement. Reviewing complete user-session traces helps a team discover its own quality criteria; an agent can identify obvious patterns but may miss failures that depend on product judgment. Their argument is about choosing what deserves evaluation, rather than accumulating generic metrics. This is a practitioner methodology, not a controlled claim that one evaluation process improves every product. The published examples should be read in their original settings, including the customer expectations and workflows that made a particular response unacceptable.
Our career interpretation: make the judgment visible
For an aspiring AI PM, the useful portfolio artifact is a decision record. Choose a small, consented or synthetic task set and explain what the user was trying to accomplish. Separate a wrong fact from a failed handoff, an unusable format and an unauthorized action. Those errors can demand different product fixes even when the final text sounds fluent. Show two borderline cases and explain how reviewing them changed your definition of success. This is our suggested exercise, not a hiring requirement attributed to Lenny’s Newsletter or to any company. It demonstrates a reasoning process that a reviewer can inspect and challenge.
Build an exercise you can defend
Write a compact annotation guide, label a sample yourself, and ask another person to judge a few cases without seeing your answers. Discuss disagreement before designing an automated check. Then change one part of the system and rerun both ordinary and difficult examples. Report the test-set size, sampling method and unresolved errors beside any score. A small, transparent experiment is more useful than an impressive percentage with no denominator. In interviews, distinguish what you observed from what you expect would happen at production scale, and explain which additional evidence would change your release decision.
Put it into practice
Your next research steps.
- Create a failure taxonomy from a small trace sample.
- Show one product decision that changed after error review.
- Keep private user information out of a public portfolio.
Read the originals
Sources & verification dates.
- Advanced evals: How to find (and fix) hidden AI failures in your product ↗
Lenny’s Newsletter — Hamel Husain and Shreya Shankar · Published 2026-09-22 · Checked 2026-10-22