The Google March 2024 core update made scaled content abuse a death sentence for programmatic SEO sites, so I built a pipeline that treats every generated page as a first-class citizen. This Next.js, MongoDB, and TypeScript system uses a topic taxonomy to enforce per-page uniqueness, forward-dates all content, and serves it through a dynamic sitemap. Here's exactly how it works, what broke, and what I'd change.
Architecture Overview
The pipeline has four distinct layers: the taxonomy engine, the content generator, the rollout scheduler, and the delivery layer. The taxonomy engine defines what pages exist and why they're unique. The generator produces content from templates. The scheduler controls when pages go live. The delivery layer serves them through Next.js with a dynamic sitemap.ts.
graph TD
A[Topic Taxonomy in MongoDB] --> B[Content Generator]
B --> C[Rollout Scheduler]
C --> D[(MongoDB Pages Collection)]
D --> E[Next.js API Routes]
E --> F[Dynamic sitemap.ts]
F --> G[Googlebot]
D --> H[Server-Side Rendering]
The key insight is that MongoDB is the single source of truth. Every page has a status field — draft, scheduled, or live — and the sitemap only exposes pages with status: 'live'. This means Google never sees a half-baked page or a URL that returns a 404.
Key Technical Decisions
1. Taxonomy-Driven Uniqueness
Instead of generating pages from a keyword list, I built a topic taxonomy where each node has a clear parent-child relationship. This guarantees semantic distance between pages. Two pages about "best running shoes" and "best trail running shoes" are different because the taxonomy forces distinct intents and content structures.
// taxonomy.ts
interface TopicNode {
slug: string;
parentSlug: string | null;
intent: 'informational' | 'commercial' | 'transactional';
requiredSections: string[];
uniquenessScore: number; // computed from parent/child distance
}
The uniquenessScore is calculated at generation time. If two sibling nodes score below a threshold, the pipeline refuses to generate one of them. This prevents the classic "thin content" problem where pages differ only by a city name.
2. Forward-Dated Rollout
Instead of publishing 5,000 pages on day one, the scheduler limits output to 20 pages per day. This mimics natural editorial growth and gives Google time to crawl and index each page properly.
// scheduler.ts
export async function getPagesForToday() {
const today = new Date().toISOString().split('T')[0];
return await Page.find({
status: 'scheduled',
publishDate: { $lte: today },
}).limit(20);
}
The rollout rate is configurable per site. For a new domain, I start at 5 pages per day. For an established domain with existing crawl budget, I'll push it to 50.
3. Dynamic Sitemap with Last Modified
The sitemap doesn't just list URLs — it tracks lastModified for every page. Google uses this to prioritize crawls, and it signals that content is actively maintained rather than dumped and abandoned.
// app/sitemap.ts
import { getLivePages } from '@/lib/db';
export default async function sitemap() {
const pages = await getLivePages();
return pages.map((page) => ({
url: `https://suhailroushan.com/${page.slug}`,
lastModified: page.updatedAt,
changeFrequency: 'weekly',
priority: page.isPillar ? 0.9 : 0.7,
}));
}
What Broke and How I Fixed It
Problem 1: Keyword Cannibalization From Template Drift
After generating about 300 pages, I noticed two pages ranking for the same query. The templates had drifted — one page had an extra section that made it more comprehensive, but the other was still indexed. Google was confused about which one to rank.
The fix was a canonicalization rule in the taxonomy. Every page gets a canonicalSlug field. If two pages fall below the uniqueness threshold, the older one gets a 301 redirect to the newer, more comprehensive version. This consolidated ranking signals instead of splitting them.
Problem 2: Sitemap Size Blowing Past Limits
At around 40,000 pages, the sitemap response exceeded the 50,000 URL limit and Google started ignoring it entirely. The fix was paginating the sitemap with an index file.
// app/sitemap.ts
export default async function sitemap() {
const totalPages = await countLivePages();
const pageCount = Math.ceil(totalPages / 50000);
if (pageCount === 1) {
return generateSitemapPage(1);
}
return {
sitemaps: Array.from({ length: pageCount }, (_, i) =>
`/sitemap/${i + 1}.xml`
),
};
}
How to Build Something Similar
Start with the taxonomy, not the content. Define your topic hierarchy in a spreadsheet first. Every node needs a parent, a unique intent, and at least three required sections that differentiate it from siblings. If you can't write those three sections, the page shouldn't exist.
Then build the MongoDB schema with status and publishDate fields from day one. Don't add them later — retrofitting a rollout schedule onto an existing dump of pages is painful. The scheduler is the simplest part; the taxonomy is where all the thinking happens.
Would I Build It the Same Way Again?
What would you do differently?
I'd use a proper headless CMS instead of raw MongoDB for the content generation layer. Managing templates and content blocks in code is fine for a solo project, but it doesn't scale if you need non-developers to review or edit content before it goes live.
Is 20 pages per day the right number?
It depends on your domain authority. A brand-new domain with zero backlinks should start at 5 pages per day. An established site with consistent crawl activity can handle 50. The key is watching Search Console's crawl stats and adjusting based on whether Google is actually fetching your sitemap.
Does this actually prevent the scaled content abuse penalty?
It reduces the risk significantly, but it's not a guarantee. Google's algorithm is looking for pages that provide unique value. If your taxonomy forces genuine differentiation and your content is written by someone who understands the topic, you're in good shape. If you're just swapping city names in a template, no pipeline will save you.
The one thing you should know before building this: the taxonomy is 80% of the work. The code is easy — deciding what pages deserve to exist, and proving they're different from each other, is the actual challenge. Get that right first, and the rest is just plumbing.