Why Measurement Beats Optimization (And Why I Keep Having to Remind Myself)
About six months into building COS, I spent a week rewriting the output format for one of the scoring dimensions. The reports looked cleaner. The language was tighter. I felt good about the work.
Then I realized I had no data on whether users were actually reading that section of the output. I had been optimizing something I hadn't measured. I had no idea if cleaner output on that dimension moved anything — engagement, conversions, trial-to-paid rate, anything. The section might have been the most-read thing on the page. It might have been skipped entirely. I didn't know, and I had just spent a week on it.
That is the builder's trap I keep falling into. It feels productive to optimize. The work has a clear before and after. You can see the improvement. What you can't see — until you've built the measurement — is whether it was the right thing to improve.
There's a specific tension here that I find hard to talk about without sounding like I'm lecturing, so I'll just describe it as accurately as I can.
Optimization feels like progress. You have something that works, you identify a way it could work better, you make the change. The loop is tight and satisfying. Measurement feels preliminary — like the thing you're supposed to do before the real work starts. In practice, most builders I know (myself included) skip or underinvest in the measurement loop and go straight to optimizing, because optimizing is more obviously productive.
The problem is that optimization without measurement is just faster iteration toward an outcome you haven't validated matters.
You can optimize your email subject lines for three months and get a 10% lift in open rate, and still have no idea whether email is the channel where your actual buyers are. You can clean up your onboarding flow and never measure whether the people dropping off at step three are dropping off because the UX was confusing or because they were wrong-fit to begin with. The optimization is real; the leverage on the outcome may not be.
This is the thing I kept coming back to when building COS.
The obvious thing to build was a copy generator. People ask for it constantly — "can it write the email for me?" "can it suggest a rewrite?" It would have been easier to build than what we actually built, and it would have been easier to explain. Copy generation is a legible product.
But I kept asking: what's the measurement gap? Where are people flying blind in a way that costs them real money? And the answer was consistently: they don't know if their copy fits the personality type of their actual buyer. They have a hunch. They optimize the hunch. They A/B test slightly different versions of the same framing, neither of which is calibrated to the reader's actual cognitive style.
The measurement problem is that nobody has a reliable way to know whether their B2B copy is reaching the right psychological profile — not just the right job title, but the right underlying person. So they optimize tone and format and CTA language without knowing if the fundamental frame is wrong.
That's the gap I wanted to close. Not "here is better copy" but "here is a score that tells you whether your copy fits its reader — and specifically what's mismatched." The measurement first. The optimization can follow once you know what you're optimizing toward.
The question I've started using to catch myself is: "do I know this matters, or do I think this matters?"
It's a small distinction and it sounds obvious when you write it out. But in practice, most optimization decisions are based on what I think matters — which is usually a combination of personal taste, pattern-matching from prior work, and what other tools in the space seem to prioritize. That's not nothing. But it's not the same as data.
The places where I've been wrong about what matters share a common structure: I had strong intuition, I acted on it, and I never built the measurement that would have told me whether the intuition was right. The week I spent on output formatting is one example. There are others. In each case, measurement would have redirected the effort — or validated that the effort was worth it.
What I'm actually advocating for here is slower. Measurement loops take time to build and time to produce signal. Optimization is faster. But faster iteration in the wrong direction isn't speed — it's a more efficient way to miss.
Build the measurement first. Optimize what it tells you to optimize.
If you're building something where you're trying to understand whether your messaging fits the people it's supposed to reach, that's exactly the gap COS was designed to close. You can see how it works at semalytics.com/cos.