Add X-Robots-Tag: noindex for .mintlify.app subdomains when custom domain is configured

[Pietrus-914]

Problem
When a custom domain is configured (e.g., docs.example.com), the default .mintlify.app subdomain remains publicly accessible and crawlable by search engines. This creates a duplicate content issue that can negatively impact SEO.
Both URLs serve identical content and both get indexed by Google.

Why canonical tags are not enough
Mintlify correctly sets the canonical URL to the custom domain, but search engines treat canonical tags as hints, not directives. Google may still:

Index the .mintlify.app subdomain
Split link equity between both URLs
Show the wrong URL in search results
Waste crawl budget on duplicate pages

Proposed solutions
Option 1: X-Robots-Tag HTTP header (recommended)
Automatically add the following HTTP header for .mintlify.app subdomains when a custom domain is configured:
X-Robots-Tag: noindex, nofollow
This is the industry standard approach. Vercel and Netlify already do this automatically for preview deployments and non-production URLs.

Option 2: 301 redirect
Implement automatic 301 redirects from *.mintlify.app to the configured custom domain. This would:

Pass full link equity to the custom domain
Ensure users always land on the correct URL
Completely eliminate duplicate content

Option 3: Configuration option
Add a setting in docs.json to control subdomain behavior:

"seo": {
"disableMintlifySubdomain": true
}
Or in the dashboard under Domain Settings.

Impact
This affects all Mintlify users who configure custom domains. Without this fix, users must either:

Accept potential SEO penalties from duplicate content
Manually request URL removal through Google Search Console (temporary fix)

Thank you for considering this feature. Happy to provide additional details if needed.