Personal website 404 traffic surged; I realized the issue wasn't in the code, but in the external link paths.

Over the past few days, I’ve been watching the XBSTACK backend, and what’s most glaring isn’t low traffic—it’s the rising number of 404 hits.

At first, I thought I’d messed up the Astro routing or that a recent deployment had dropped some pages. But after digging into Cloudflare logs, I realized many of these requests weren’t coming from internal site clicks. Instead, they were arriving from Zhihu, Juejin, search engine caches, and historical external links, all carrying broken paths.

404 is insidious.

It doesn’t crash as obviously as 500, nor does it make the homepage inaccessible so you can spot it immediately. It acts more like a leak: users arrive, hit a wall, and leave; search engines arrive, hit a wall, and lose trust; I only see overall traffic numbers but have no idea where the traffic is leaking out.

So this time, instead of writing another AI Agent tutorial, I stopped to investigate the site itself.

404 isn’t a minor technical glitch; it’s a traffic funnel problem

A few 404 hits are normal. Any public website gets hammered by crawlers, scanners, old links, and typos pointing to non-existent paths.

But if 404 keeps growing, you can’t just dismiss it as “normal internet noise.” Especially for personal sites with limited traffic, every valid visit is expensive. A reader clicking through from Zhihu might have finished an article and then casually navigated to the AI Development Hub or used the site search. If their first impression is an 404, they’ll likely close the tab immediately.

Search engines are affected too.

They won’t penalize your entire site over a random junk path, but if certain old paths are consistently pointed to by sitemaps, external links, and internal links yet keep returning 404, it indicates a dirty discovery path for the site. For a content-heavy site like XBSTACK with over a hundred articles and multiple sections, 404 isn’t a single-point failure; it’s a crack in the content asset system.

I previously wrote a post titled XBSTACK Content Quality Audit Build Log, which mainly covered cover images, internal links, duplicate topics, Pagefind, sitemaps, and robots.txt. That audit proved there were no obvious high-risk issues with the internal structure. However, the increase in online 404 reminded me: a clean internal structure doesn’t guarantee clean external paths.

I ruled out a “total site routing collapse” first

When troubleshooting 404, the first step wasn’t to immediately write redirects or blame the framework.

I started by looking at a few basic facts:

首页：正常
AI 总入口：正常
LangGraph / MCP / Workflow 专题页：正常
文章详情页：正常
sitemap-index.xml：可访问
sitemap-0.xml：可访问
Pagefind 搜索索引：构建正常

This indicates that the issue is not a site-wide build failure, nor is it a complete breakdown of Astro dynamic routing.

If the entire routing system had collapsed, the symptoms would be far more obvious: numerous normal articles would fail to load, sitemap paths would go offline in bulk, and search pages might become inaccessible. But that is not what happened here. The problem is isolated to specific paths, which often contain old slugs, typos, truncated links, or traces of historical external links.

In other words, this isn’t a case of “the site is down,” but rather “some entry points are pointing to the wrong place.”

The sources of 404 are often not just one

I categorize these personal websites, 404, into five types.

The first type is outdated internal links. For example, a previous article might have referenced a specific path, but the slug was later changed without updating the body text. When readers click through from that older post, they encounter a 404 error.

The second type is broken links from external platforms. When articles are reposted on Zhihu, Juejin, CSDN, Xiaohongshu, or WeChat Official Accounts, copying the URL with even a single extra character or a missing slash can permanently create 404. Unlike internal links, external platforms do not automatically update when I fix the link on my own site.

The third scenario is search engine caching of old paths. If an old page was previously crawled and you later migrated categories or changed slugs, search engines will continue to access the old addresses for a short period.

The fourth scenario is historical remnants in the sitemap. Theoretically, sitemaps should only contain valid URLs, but if the generation script or deployment cache isn’t refreshed promptly, it may continue to instruct search engines to crawl outdated paths.

The fifth scenario is spam scanning. For example, someone might scan /wp-admin/, /phpmyadmin/, /.env, /admin/login. This kind of 404 doesn’t require fixing; retaining 404 is actually the correct outcome.

The key point is: Do not treat these five scenarios with a single approach.

If you redirect all spam scans 301 to the homepage, it may appear that 404 has decreased, but you are actually polluting your logs. Later, when reviewing the backend, you won’t be able to distinguish between genuine users hitting a wall and scripts scanning randomly.

What to look for in Cloudflare logs

The greatest value of edge-layer logs from providers like Cloudflare isn’t just telling you “there’s an 404,” but rather showing you “who hit which path from where.”

I review them using four fields.

path       请求路径
referer    来源页面
user-agent 请求客户端
count      相同路径出现次数

If a path’s Referer originates from Zhihu or Juejin and appears repeatedly, prioritize handling it. This indicates genuine external links driving traffic rather than spam scanning.

If the User-Agent is Googlebot, Bingbot, or Baiduspider, and the path resembles an old article slug, check the sitemap and legacy backlinks. Search engines are not the enemy; they simply followed an old trail to get there.

If the path matches /.env, /wp-login.php, or /phpmyadmin, there is no need to worry. Retain 404 for these requests directly, applying rate limiting only at the Cloudflare WAF level if necessary.

The scenario I fear most is another type: paths that appear to be normal articles but actually do not exist.

/ai/langgraph-observability/
/ai/n8n-webhook-hardening/
/ai/xbstack-content-audit/
/tools/compound-calculator/

This path isn’t discovered by hackers; it’s replicated by humans or platforms. It’s only a hair’s breadth away from the correct path, yet that tiny gap is enough to make users hit a wall.

The Most Common Path Issues in Astro Sites

Astro static sites are inherently stable, but path governance relies on human diligence.

I currently focus on these categories of issues:

First, inconsistent trailing slashes.

Some links end with /ai/foo, while others use /ai/foo/. If the site lacks a unified configuration standard, external platforms may apply their own truncation or encoding, resulting in divergent paths.

Second, missing redirects after slug changes.

The most common scenario for content sites is this: early titles were written hastily, and later they were revised for SEO purposes, including changes to the title and slug. However, old slugs were not redirected via 301. The site looks clean internally, but all external legacy links are broken.

Third, truncation of Chinese titles when converted to slugs.

When some platforms copy links, they automatically process Chinese characters, spaces, punctuation, or trailing slashes. If an article’s slug is already too long, or if special characters have slipped into the path, the probability of errors increases significantly.

Fourth, category migrations.

XBSTACK had some early content migrated between archive, lens, ai, and horizon. The migration itself was fine, but old paths must be preserved as fallbacks afterward. Otherwise, search engines and legacy external links will continue to follow the old routes.

Fifth, the sitemap is out of sync with actual pages.

If the sitemap still contains old URLs, it’s essentially inviting search engines to crawl non-existent pages. This issue is more critical than broken external links because it represents an active error signal emitted by the site itself.

Which 404 Issues Must Be Fixed, and Which Can Be Ignored

I’m no longer aiming to reduce 404 to zero. That’s unrealistic and unnecessary.

I only address three categories.

The first category involves 404 with genuine external link sources. For example, when the Referer comes from Zhihu, Juejin, CSDN, or WeChat Official Account redirect pages, and the path clearly corresponds to an existing article on the site. These should either be fixed in the original external source or handled internally via 301.

The second category consists of old paths showing signs of search engine access. Specifically, paths frequently crawled by Googlebot, Bingbot, or Baiduspider should be redirected if they correspond to a currently existing article.

The third category is 404 caused by outdated internal links. This must be fixed in the content, not just handled with redirects as a fallback. Internal links are an asset I have full control over; they cannot be allowed to remain broken for long.

The rest can be ignored.

/wp-admin/
/wp-login.php
/phpmyadmin/
/.env
/server-status
/random-test-path

These aren’t traffic; they’re noise. Trying to fix noise into an entry point is the real mistake.

My 404 Repair Checklist

I’ll handle this in the following order.

1. 从 Cloudflare 导出最近 7 天 404 路径
2. 按 count 排序，找出高频路径
3. 按 referer 分组，识别知乎、掘金、搜索引擎来源
4. 用站内搜索确认这些路径是否对应现有文章
5. 有对应文章：建立 301 或修正外链
6. 无对应文章但有真实需求：考虑补一个专题页或兜底页
7. 垃圾扫描路径：保留 404，不做跳转
8. 重新生成 sitemap 并检查旧路径是否残留
9. 检查文章正文和专题页的内部链接
10. 让 404 页面提供搜索、AI 总入口、Tools 入口，而不是只有一句 Not Found

Item 10 is particularly important.

The 404 page should not be a tombstone; it should serve as a rescue entry point. When users hit a non-existent path, they should at least be able to continue searching through All Articles, Site Search, AI Engineering, or Tools.

If the 404 page merely displays the cold, unhelpful message Not Found, it effectively drives users away.

Don’t Overdo Redirect Rules

Many people fix 404 by swinging to the other extreme: redirecting any non-existent path straight to the homepage.

That’s incorrect.

Redirects should be as precise as possible.

/old-ai-agent-url/  -> /ai/ai-agent-architecture/
/old-n8n-webhook/   -> /ai/n8n-webhook-production-hardening/
/old-content-audit/ -> /ai/xbstack-content-quality-audit-builder-log/

If you can’t find a clear corresponding page, don’t force a redirect to the homepage. Users are looking for answers to specific questions, not the homepage. Search engines also detect these irrelevant redirects; over time, this weakens your page signals.

My rule is simple: if there’s a one-to-one match, use 301; if not, let the 404 page handle navigation; and if it’s clearly junk, continue with 404.

What This Taught Me

Once a personal site reaches a certain stage, the real work isn’t just “what am I publishing today.”

It also involves periodically reviewing which articles have been cited externally, which paths have changed, which old entry points are still being accessed by search engines, and which pages exist but leave users with no next step once they arrive.

This is why I increasingly emphasize content inventory management.

XBSTACK already has topic hubs like LangGraph, MCP, and Workflow, as well as tool pages like Compound Interest Calculator, AI Earnings Report Assistant, and Lunest. As the number of articles grows, paths become part of the asset itself.

Titles can be updated, structures optimized, and pages refactored, but old paths shouldn’t be casually discarded.

Because search engines remember them, external platforms remember them, and reader bookmarks remember them.

Pre-Publish Checklist

From now on, every time I publish or change a slug, I’ll take a few extra steps.

- 新文章 slug 是否短、稳定、无特殊字符
- sitemap 是否包含新 URL
- 旧文章是否需要补内链
- 是否存在相近旧题导致重复
- 外链分发时是否复制了最终 URL
- 改标题是否不改 slug，除非必须
- 改 slug 时是否同步 301
- Cloudflare 是否能按路径查到异常访问
- 404 页面是否有搜索和核心入口

These actions aren’t complex, but they determine whether a personal website is just a pile of files or a true asset.

Continue Reading

• Web Builder Log: Personal Website, Astro, Cloudflare, and Search Console Review
• Personal Website to 160 Posts: Why I Finally Realized My Traffic Was Zero
• Search Console Shows Impressions But No Clicks: The Problem Isn’t Indexing, It’s Titles and Entry Points
• SEO Automation in Practice: Building an Auditable Index and Growth Pipeline
• XBSTACK V3 Refactoring Log
• Cloudflare Serverless in Practice: Building a Zero-Cost, Globally Accelerated AI Full-Stack Architecture
• AI Development Hub: Systematic Index of Agents, LangGraph, MCP, and Workflows

404 The scariest part isn’t that pages don’t exist.

It’s thinking no one visits your site when they actually do—they just hit a wall and leave.

For a personal website, troubleshooting 404 isn’t about ops perfectionism; it’s about systematically recovering every bit of lost traffic.