Class: RobotsController

Inherits:
ActionController::Base
  • Object
show all
Defined in:
app/controllers/robots_controller.rb

Overview

Serves a per-host robots.txt.

The public marketing site (www / apex) gets the full crawl policy; internal
surfaces (crm, api, scan, mcp) disallow all crawling. This replaces the nginx
map $host $robots_file mapping that was lost in the Kamal/Thruster migration
— Thruster serves a single static public/ file for every host, so per-host
robots has to be decided in the app.

Inherits from ActionController::Base (not ApplicationController/CrmController)
so no authentication/locale filters run — otherwise crm.* would 302 the
crawler to a login page instead of returning the disallow-all policy.

Constant Summary collapse

NOCRAWL_SUBDOMAINS =

Subdomains whose entire surface is non-public — never crawl or index them.
scan is intentionally absent: it's handled entirely at the Cloudflare edge
(the scan-* rules in lib/tasks/redirects.rake 301-redirect it to www), so it
never reaches this app and needs no entry here.

%w[crm api mcp].freeze
NOCRAWL_BODY = 
"User-agent: *\nDisallow: /\n"

Instance Method Summary collapse

Instance Method Details

#show ⇒ Object 



22
23
24
25
26
27
28
29
30
 
# File 'app/controllers/robots_controller.rb', line 22

def show
  expires_in 6.hours, public: true

  if nocrawl_host?
    render plain: NOCRAWL_BODY, content_type: 'text/plain'
  else
    render template: 'robots/crawl', formats: [:text], layout: false, content_type: 'text/plain'
  end
end