A content inventory prevents it: a single document listing every published piece with its title, URL, and a one-paragraph summary. An AI tool reads that document before writing, rather than scanning the whole site each time. Checking a few hundred summaries is fast and cheap, while re-reading a thousand full pages before every draft is neither.

What Is a Content Inventory?

A content inventory is a single structured list of everything an organisation has published, holding a short summary of each item rather than its full text. It differs from a sitemap in carrying meaning rather than addresses, and from an archive in being deliberately lossy: the point is that it is small enough to read in one pass.

For anyone running content through an AI tool, the inventory is what makes duplicate checking practical. An agent such as Dubb Agent can hold a few hundred summaries in context and compare a proposed topic against all of them at once, which is not possible when the alternative is crawling every page on the site before writing a word.

The Problem With a Thousand Posts

Every organisation that has published consistently for a few years arrives at the same place. There is more content than any one person can hold in their head, and nobody currently working there wrote most of it.

Our own blog is past a thousand posts. Nobody remembers all of them. So when someone proposes a topic, the honest answer to "have we covered this?" is usually "probably, somewhere, in some form."

That was survivable when publishing was slow. It stops being survivable the moment you add AI to the process, because the bottleneck moves. Drafting used to be the slow step and it acted as a natural brake. Now drafting takes minutes, and the only thing standing between you and republishing your own back catalogue is whether anyone checked.

The instinct is to hand that checking to the agent. Scan the blog, make sure this is not a duplicate. It is the right instinct and, done the obvious way, it is the wrong implementation.

What Google Actually Does With Duplicates

Worth settling this before the workflow, because the reason most people give for caring about duplication is not quite right, and the real reason is more useful.

There is no duplicate content penalty. This is the most durable myth in SEO and Google's own documentation on consolidating duplicate URLs does not describe one. What it describes is canonicalization: when Google finds pages that substantially overlap, it picks one to represent the group and consolidates the ranking signals into that one. Google's guidance is that most sites do fine without specifying a canonical preference at all.

So nobody is being punished. What happens is quieter and, in practice, more annoying.

Google chooses, and it may not choose what you wanted. Write a better article on a topic you already covered, and Google may keep showing the old one. The new piece is not penalised, it is simply not the version that gets shown.

Your own pages compete for the same query. Several near-identical posts split links and relevance signals between them instead of concentrating on one strong page. This is the cost that actually shows up in traffic, and it is not a penalty either. It is arithmetic.

The wrong page gets the attention. An outdated post outranking its own replacement is a common and entirely avoidable situation.

Deliberate, deceptive duplication, such as scraping other sites or spinning the same page across hundreds of URLs, is a different matter and is covered by spam policies. Publishing two similar articles on your own blog is not that. It is a self-inflicted efficiency problem, and the fix is editorial rather than technical.

Why Scanning Everything Every Time Fails

Asking an agent to scan the whole site before each piece sounds thorough. Three things go wrong.

It is expensive. Every check re-reads material that has not changed since the last check. On a large site, run daily, the cost is significant and entirely repeated.

It is slow. Enough that people stop doing it, which is the failure that matters. A check that adds twenty minutes to every article gets skipped the first time someone is in a hurry, and then permanently.

It is unreliable. This is the part people underestimate. Searching for near-duplicates only works if you guess the right search terms, and near-duplicates are precisely the case where the wording differs. A post about framing yourself on a screen recording and a post about webcam overlay settings are the same article; no keyword search reliably connects them.

I can speak to that last one directly. Working this way, I have started writing a piece before discovering that five existing posts already covered the topic, and I have proposed an article that turned out to be a near-copy of one published weeks earlier. Both would have been caught in seconds against a list of summaries. Neither was caught by searching, because searching only finds what you think to look for.

What Goes In the Inventory

The design goal is the smallest record that still lets you recognise overlap. Excerpts, not articles.

What deliberately stays out is the body text. A thousand full posts is the thing you were trying to avoid loading. A thousand four-sentence summaries is a document an agent can hold comfortably.

Building It Once

Building the content inventory is a one-time exercise, and it is the only part that takes real effort.

Start from your sitemap, which already lists every published URL. That gives you the spine. Then, for each item, produce the summary. An agent can do this in batches: fetch the page, return the title, the primary topic, and three or four sentences on what it actually argues.

Two instructions worth giving it. First, summarise what the piece claims, not just what it is about. "Covers screen recording" is useless for deduplication. "Argues that the webcam bubble should be small and matched to the page, and covers shape, sizing, placement and the double-click to full screen" is what lets you spot the overlap. Second, have it flag pieces it finds that are already near-duplicates of each other, because you will have some and you may as well learn that now.

Batch it. A few dozen at a time, spot-checking the output, beats one enormous run you cannot verify.

Where to Keep It

A spreadsheet or a document both work, and the choice matters less than the rule that there is exactly one of them.

A spreadsheet gives you sorting and filtering, which is useful once you are past a few hundred rows and want to group by keyword. A document is easier for an agent to read straight through and easier to skim yourself.

What actually matters is that it lives where the agent can reach it without being handed a file each time, and that nobody makes a second copy. A content inventory that exists in three slightly different versions is worse than none, because now you have to decide which one is true.

How the Agent Actually Uses It

The workflow is short. Before writing anything, the agent reads the content inventory and compares the proposed topic against it, then reports one of four outcomes.

Nothing close. Write it.

Adjacent but distinct. Write it, link to the neighbours, and say explicitly how it differs so a future editor does not collapse them.

Substantial overlap. Stop and report. Name the existing piece, its URL, and what overlaps. Then decide deliberately: re-scope to the gap, or leave it alone.

Near-identical. Do not write a new post. Update the existing one instead, which keeps the URL, its age, and whatever links it has accumulated.

The reporting matters as much as the checking. An agent that silently re-scopes is making an editorial decision on your behalf without telling you. What you want is the collision surfaced, with the URLs, so a person makes the call.

When the Inventory Says Stop

The hardest case is the third one, and it is worth thinking through before you are in it.

You have a good idea, drafted or nearly drafted, and the inventory says three posts already sit on that query. The instinct is to publish anyway, because the new piece is better. That instinct is usually wrong, and not because of any penalty. It is that a fourth post on a query three of your own pages already split does not create a fourth share of attention. It creates a four-way split.

Three better options. Re-scope to the part none of the existing posts cover, and link to them from the new piece. Refresh the strongest existing post with your new material, keeping the URL and its accumulated authority. Or consolidate, merging the weak overlapping posts into one strong page and redirecting the rest.

Consolidation is the option people avoid, because deleting published work feels like loss. Four thin posts splitting a query are worth less than one good page, and the inventory is what makes that visible.

Keeping It Current

A content inventory decays the moment you publish something. Keeping it current is trivial if you attach it to publishing, and impossible if you leave it as a separate chore.

The rule is one line: nothing counts as published until its entry exists. Add the row as the last step of publishing, the same way you would add a redirect or check a canonical tag. It takes under a minute because you wrote the summary anyway.

Then reconcile against the sitemap quarterly to catch what slipped through, and flag anything untouched for eighteen months as a refresh candidate. That second pass turns the inventory from a defensive tool into a planning one.

What Else It Gives You

Deduplication justifies the build. Three other uses come free once a content inventory exists.

Internal linking that is actually possible. The reason most internal linking is bad is that nobody can remember what exists to link to. A list of summaries makes relevant links findable in seconds, and a verified URL beats a guessed one.

Messaging consistency. When every summary states what the piece argues, contradictions become visible. Two posts giving opposite advice is worse than a duplicate and much harder to notice without a list.

Gap analysis. Sorting by primary keyword shows both the queries you have covered four times and the obvious ones you have never covered at all. The second list is usually the more interesting.

Ways to Hold the Inventory Compared

Several reasonable options, differing mostly in how much setup they need and how easily an agent reads them.

The last row is the default everyone starts with, and it works until roughly the point where you can no longer remember what you have published. After that it quietly stops working, and the way you find out is by publishing something twice.

Common Mistakes and Troubleshooting

Putting full articles in it. That recreates the problem. Summaries only.

Summarising the topic instead of the argument. "About email deliverability" will not catch an overlap. What the piece claims will.

Leaving out non-blog content. Landing pages, social posts, and video descriptions collide with blog posts constantly. If it is published, it goes in.

Maintaining it separately from publishing. A weekly update chore lasts about three weeks. Attach it to the publish step.

Keeping more than one copy. Two inventories is zero inventories.

Letting the agent re-scope silently. You want the collision reported with URLs, not quietly worked around.

Treating a collision as a stop sign. Often the right answer is refreshing the existing post, which is better than either writing or abandoning.

Believing there is a penalty. There is not. The cost is your own pages competing, which is reason enough.

Best Practices I Actually Follow

Check before drafting, not after. The check is worth nothing once the article exists, because by then nobody wants to hear it.

Record the check. Note what was compared and what was found. It is how the next person knows the decision was deliberate.

Write the summary at publish time. You know the piece best on the day it goes out.

Sort by keyword occasionally. Clusters show up immediately and they are always worse than anyone expected.

Prefer a refresh to a new post when the topic is covered. An updated page keeps its URL, its age, and its links.

Include the things nobody thinks of as content. Help articles and long social posts rank, and they collide.

Proof: Why This Actually Works

The mechanism behind a content inventory is unglamorous. Comparing a proposed topic against a few hundred summaries is a small, fast operation. Comparing it against a thousand full pages is a large, slow one that mostly re-reads unchanged material.

Two patterns hold consistently. The first concerns when checks get skipped: a duplicate check that adds meaningful time to every article stops happening within weeks, and the teams that keep checking are the ones for whom it costs almost nothing. Reliability comes from cheapness, not from discipline.

The second concerns what searching misses. Near-duplicates are precisely the case where two pieces use different vocabulary for the same idea, which is the case keyword search handles worst. Reading summaries catches those; searching for terms does not, and the failure is silent, which is what makes it expensive.

What I take from it is that the inventory is not really an SEO artefact. It is the memory of an organisation that publishes faster than anyone can remember, and AI is what made that gap wide enough to need fixing.

Frequently Asked Questions

Does Google penalise duplicate content?

No. Google's documentation on consolidating duplicate URLs describes canonicalization rather than a penalty: when pages substantially overlap, Google picks one to represent the group and consolidates the ranking signals into it.

The real cost is that Google chooses, and it may not choose the page you wanted, and that several similar pages of your own split signals between them rather than concentrating on one strong page. Deliberate deceptive duplication, such as scraping or spinning one page across many URLs, falls under spam policies and is a separate matter.

What is a content inventory?

A single document listing everything you have published, with a short summary of each item rather than its full text. Each entry carries the title, URL, primary keyword, a three or four sentence summary of what the piece argues, the published and updated dates, and which channel it lives on.

It differs from a sitemap by carrying meaning rather than just addresses, and it is deliberately lossy. The point is that it stays small enough to read in one pass.

Why not just ask AI to scan my whole site?

Because it is expensive, slow, and less reliable than it sounds. Every scan re-reads material that has not changed, and a check that adds twenty minutes to each article gets skipped the first time someone is in a hurry.

The reliability problem matters more. Finding near-duplicates by search only works if you guess the right terms, and near-duplicates are exactly the case where two pieces use different words for the same idea. Reading summaries catches those; searching often does not.

What should each inventory entry contain?

Title as published, full URL, one primary keyword, a three or four sentence summary, the published and updated dates, and the channel. Keep the body text out, since loading full articles recreates the problem you were solving.

Summarise what the piece argues rather than what it is about. "Covers screen recording" will not catch an overlap. A sentence naming the specific claim and what is covered will.

How do I keep a content inventory up to date?

Attach it to publishing rather than treating it as a separate task. Nothing counts as published until its entry exists, and adding the row takes under a minute because you already know what the piece says.

Reconcile against your sitemap quarterly to catch anything that slipped through, and flag items untouched for eighteen months as refresh candidates.

What should I do when the inventory finds an overlap?

One of three things, and publishing anyway is rarely the right one. Re-scope the new piece to whatever the existing posts do not cover and link to them. Refresh the strongest existing post with your new material, which keeps its URL, age and links. Or consolidate several thin overlapping posts into one strong page and redirect the rest.

Consolidation is the option people avoid because removing published work feels like loss, but several thin posts splitting one query are worth less than a single good page.

Know What You Have

The exercise is a single afternoon. Pull every URL from your sitemap, have an agent write three or four sentences on each one describing what it argues, put them in one document, and add a row to it every time you publish from then on.

What you get back is the ability to answer "have we covered this?" in seconds rather than guessing. That question was answerable from memory when you had forty posts. At a thousand it is not, and adding AI to the writing side without fixing the knowing side is how organisations end up competing with themselves.

Dubb Agent can both build the inventory and read it before drafting, which keeps the check cheap enough that it actually keeps happening. That is the whole design goal: not a thorough check, a check nobody is tempted to skip.

If you change one thing after reading this, build the list before you write the next piece. The check is only useful while the article is still an idea.