# ============================================ # YallaMotor Robots.txt # ============================================ # Block known crawlers that do not add SEO value or consume excessive resources User-agent: grapeshot Disallow: / User-agent: SemrushBot Disallow: / User-agent: PetalBot Disallow: / # ============================================ # ============================================ # Main directives for all other bots User-agent: * # Disallow access to internal/private areas Disallow: /user/ Disallow: /conversations/ Disallow: /sales/ # Explicitly allow public-facing content Allow: / # All public pages are allowed: # - Main Landing Page, Offers, New Cars (brand, model, version, and search) # - Used Cars (landing, search, specific brand, ad details) # - News, Videos, Expert Car Ratings, etc. # - Arabic versions (via /ar/ in the URL) # ============================================ # Sitemaps – adjust the URLs if necessary per subdomain. Sitemap: https://www.yallamotor.com/sitemap.xml