# --- DEFAULT RULES (Allow All Standard Search Engines) --- User-agent: * Allow: / # --- SITEMAP REFERENCE --- # This is the single, non-gzipped index file you must submit/reference. # CHANGE 'http://localhost:8080' to your actual domain (e.g., https://linux-vserver.org) for a live site. Sitemap: https://linux-vserver.org/sitemap/sitemap-index-my_wiki_db.xml # --- BLOCK SPECIFIC BOTS --- # Block Ahrefs User-agent: AhrefsBot Disallow: / # Block Semrush User-agent: SemrushBot Disallow: / # Block other Semrush bots just in case User-agent: SemrushBot-SA Disallow: / User-agent: SemrushBot-crawler Disallow: / # Block Majestic User-agent: MJ12bot Disallow: / # Block Wayback Machine (Internet Archive) User-agent: ia_archiver Disallow: / # Block Common Crawl User-agent: CCBot Disallow: / # Block archive.is User-agent: archive.is Disallow: /