Content archetypes on an 18,752-page inventory

A content team with 18,000 pages reviews about twenty a week. One pass through the whole inventory takes eighteen years. So nobody makes 18,000 decisions. They set a few policies and apply them.

That means the useful question isn’t “which page is worst.” It’s “what kinds of page are there and what does each kind need?”

The tool a team has for this today is a hand-written rule with thresholds somebody picked. I built one first so I’d have something honest to beat. It works. It also drops 8,130 pages, 43% of the inventory, into a single bucket and tells the team to monitor all of them. That bucket is the problem I went after.

The lane was recommended to me. The reasoning became mine the moment I saw that 43% number and understood what it was hiding.

1. 01

   I built the rule first and froze it before fitting anything. If you write the baseline after the model, you write a baseline the model beats.

2. 02

   No product flags as inputs. A health score is a decision someone already made. Feed it in and you’ve trained a model to reproduce the rule you’re supposed to be testing against.

3. 03

   K-Means at k = 6, chosen on silhouette and stability across seeds, not on which number told a better story.

4. 04

   Three tests, decided before I saw the results. Does the grouping come back across random seeds? Does each group survive removing the feature that defines it? Does it hold on a client it never saw?

The last two were the ones I didn’t want to run.

Six archetypes came out. They reproduce across seeds at ARI 0.998. Five survived the ablation test. One didn’t.

• High-Yield Core 4,924, kept
• Long-Form Untouched 4,598, kept
• Recently Published, Establishing 4,113, kept
• Long-Tenured Plateau 3,595, kept
• Intermittent Exposure 985, kept
• Measured-Engagement Minority 537, withdrawn

Counts across the full 18,752-page inventory, as reported in the paper’s archetype table.

The one that failed was 537 pages I’d called a distinct engagement group. Drop the engagement column and they scatter; the largest destination takes 34% of them. It wasn’t a behaviour. It was one column wearing a name I’d given it. I withdrew it from the paper and wrote down why.

What the surviving five bought: the rule’s 43% catch-all splits into five measurably different populations. The one worth the money is 735 pages the rule called “steady low” that carry a median 10,620 impressions at 0.42% CTR, the highest-exposure pages in the bucket the rule told teams to ignore.

Leave-one-client-out across 17 clients gives mean ARI 0.909, but the worst client is 0.465. For a new client this is a starting point that needs checking, not a finished answer. So the deliverable is a review queue with reason codes and a 13.1% low-confidence bucket routed to “not sure” instead of to an action. Nothing is automated. No page gets deleted or merged on this evidence, because nothing here measured whether a page should exist.

Next thing I’d do: hold back a slice of the queue, touch nothing on it, and see whether the treated group actually diverges. Without that, none of this is proven to work, and I’d rather say so than ship a number I can’t defend.