Sentiment scores wording, not subject

I run a thing called Headlines. It reads about thirty-five RSS feeds every hour, clusters the headlines into stories by shared vocabulary, and ranks them by how many independent newsrooms ran each one. No summarising, no generated text — every line is a real headline, linked to its source.

Each headline also gets a sentiment score, from a small hand-written word list. The obvious feature to build on top of that is “show me the good news.” I built it. Then I looked at what it returned.

The most positive headlines were a war and a fraud

Two of the highest-scoring headlines on the day I checked:

• “Trump says Hamas agreed to next phase of Gaza peace deal” — +0.65
• “AI scammers outperform humans when it comes to building trust” — +0.69

Nothing is broken. Agreed, peace, deal, outperform and trust are positive words, and the scorer is doing exactly what it says on the tin. The problem is what I assumed it meant.

Sentiment scores wording. It does not score subject. A war reported in the language of diplomacy scores like diplomacy.

Anyone filtering on that gets positively-phrased hard news, which is not what “show me the good news” promises. So I cut the feature rather than ship a promise the number could not keep.

Most headlines score nothing at all

The second thing I found was quieter, and it changes how you read any average built on this:

Two thirds of headlines contain no word the list recognises. They score zero — not “neutral” in any considered sense, just unmatched. Every daily mean is therefore diluted toward zero by construction, and a mean of -0.09 is not “a mildly bad day”, it’s “the third of headlines that scored at all leaned negative.”

That is fine as long as it’s labelled. It stops being fine the moment somebody calls the chart “mood”.

The mistake I nearly published

Here is the one I’m most glad I checked. The interesting question is not “which outlet is negative” but “which outlet words the same story more warmly than its peers.” Those sound similar. They are not.

Average every outlet’s raw sentiment and you don’t measure framing, you measure beat. Al Jazeera covers conflict, so it scores negative. TechCrunch covers product launches, so it scores positive. Nobody is spinning anything.

Controlling for the story means scoring each outlet against the mean of every outlet that covered that same event:

def deltas(stories, min_outlets=3):
    """{outlet: (sum_of_deltas, count)} for one run's stories."""
    acc = defaultdict(lambda: [0.0, 0])
    for s in stories:
        cov = s.get("coverage") or []
        if len(cov) < min_outlets:
            continue
        mean = sum(c["sentiment"] for c in cov) / len(cov)
        for c in cov:
            a = acc[c["publisher"]]
            a[0] += c["sentiment"] - mean
            a[1] += 1
    return acc

Measured on one day, the naive average called Al Jazeera the most negative outlet at −0.244. Controlled for the story, it was −0.066. Most of that number was the subject, not the wording.

The min_outlets=3 is not arbitrary either. With two outlets, whatever one says the other is defined as its exact opposite — that’s arithmetic, not evidence. Three is the smallest number where the mean isn’t just the other outlet’s opinion.

What I did instead

I stopped publishing the number and started recording it.

One run is nowhere near enough data. Within-story headline spread runs about 0.18, so resolving a 0.05 difference between two outlets needs roughly 47 observations each. A single run gives an outlet 5 to 16, and every confidence interval I computed on one run spanned zero — not one outlet was distinguishable from “no effect at all.”

Published on one run, that table would be noise with real newsrooms’ names on it, reordering itself every hour. Accumulated, an outlet appearing in ten shared stories a day clears 47 inside a week.

So it writes a row a day and says nothing until it has earned the right to. The summary function returns a confidence interval and a ranked flag rather than a bare average, specifically so nothing downstream can quietly present a number that hasn’t.

The actual lesson

I’ve been a data analyst for five years and I still nearly shipped this. The scorer worked. The code was correct. The tests, had I written them, would have passed. Everything was green and the conclusion was wrong, because I never checked what the number meant — only that it was computed.

The check that caught it took ten minutes: sort by the metric, read the top five, ask whether they look like what you promised.