feat: integrating_robots_through_multi_channel_configuration

[tdonthu]

PR Type

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no API changes)
[ ] Build-related changes
[ ] CI-related changes
[ ] Documentation content changes
[ ] Application / infrastructure changes
[ ] Other:

What Is the Current Behavior?

Issue Number: Closes #

What Is the New Behavior?

Does this PR Introduce a Breaking Change?

[ ] Yes
[x] No

Other Information

AB#92194

[dhhyi]

If this is the way forward, the robots.txt handling in server.ts as well as the used library should be removed:

intershop-pwa/server.ts

Lines 207 to 231 in 0897afd

	// seo robots.txt
	const pathToRobotsTxt = join(DIST_FOLDER, 'robots.txt');
	if (fs.existsSync(pathToRobotsTxt)) {
	server.use(robots(pathToRobotsTxt));
	} else {
	server.use(
	robots({
	UserAgent: '*',
	Disallow: [
	'/error',
	'/maintenance',
	'/account',
	'/compare',
	'/recently',
	'/basket',
	'/checkout',
	'/register',
	'/login',
	'/logout',
	'/forgotPassword',
	'/gdpr-requests',
	'/contact',
	],
	})
	);