Module: Heatwave::Crawler::Guards

Defined in:
app/services/heatwave/crawler/guards.rb

Overview

Reopens Heatwave::Crawler (defined in crawler.rb) to house the Guards
module of shared fetch-safety and bot-wall heuristics.
Shared fetch safety + bot-wall heuristics, promoted from the Sunny
fetch_url tool so every crawler consumer gets the same protection:

  • Guards.validate_public_url! SSRF guard — http(s) only, no embedded
    credentials, DNS-resolved IPs must not be loopback/private/link-local.
    Caveat (documented in the supercrawler plan): the guard resolves once
    and the fetcher re-resolves, so DNS-rebinding is theoretically possible;
    accepted risk today.
  • Guards.bot_challenge? Cloudflare/Amazon/etc. interstitial markers.
  • Guards.blocked_content? post-extraction captcha/block phrases.
  • Guards.garbled_content? binary/non-text body detection.

Constant Summary collapse

BOT_CHALLENGE_STATUSES =

HTTP statuses that are bot-wall responses even without body markers.

[403, 503].freeze
BOT_CHALLENGE_MARKERS =

Raw-HTML markers of a bot interstitial (Cloudflare, Amazon, generic).

[
  'Just a moment',
  'cf-browser-verification',
  '__cf_chl_',
  'Enable JavaScript and cookies to continue',
  'Checking your browser',
  'Click the button below to continue shopping',
  'Enter the characters you see below',
  'Sorry, we just need to make sure',
  'api-services-support@amazon.com',
  'To discuss automated access to Amazon',
  'Please verify you are a human',
  'Access Denied</title>',
  'Robot or human?'
].freeze
BLOCKED_CONTENT_MARKERS =

Downcased extracted-text markers of a block/captcha page.

[
  'click the button below to continue shopping',
  'enter the characters you see below',
  'sorry, we just need to make sure',
  'please verify you are a human',
  'automated access',
  'unusual traffic from your computer',
  'robot or human?'
].freeze

Class Method Summary collapse

Class Method Details

.blocked_content?(text) ⇒ Boolean

Returns true when the extracted content is a block page.

Parameters:

  • text (String) —

    extracted plain text

Returns:

  • (Boolean) —

    true when the extracted content is a block page



95
96
97
98
99
100
 
# File 'app/services/heatwave/crawler/guards.rb', line 95

def blocked_content?(text)
  downcased = text.to_s.downcase
  # rubocop:disable Style/ArrayIntersect -- substring matching, not element equality
  BLOCKED_CONTENT_MARKERS.any? { |marker| downcased.include?(marker) }
  # rubocop:enable Style/ArrayIntersect
end

.bot_challenge?(body, status_code) ⇒ Boolean

Returns true when the response is a bot interstitial.

Parameters:

  • body (String) —

    response body

  • status_code (Integer) —

    HTTP status

Returns:

  • (Boolean) —

    true when the response is a bot interstitial



81
82
83
84
85
86
87
88
89
90
91
 
# File 'app/services/heatwave/crawler/guards.rb', line 81

def bot_challenge?(body, status_code)
  return true if BOT_CHALLENGE_STATUSES.include?(status_code)

  body = body.to_s
  # A short page whose only payload is the challenge platform script.
  return true if body.length < 15_000 && body.include?('challenge-platform')

  # rubocop:disable Style/ArrayIntersect -- substring matching, not element equality
  BOT_CHALLENGE_MARKERS.any? { |marker| body.include?(marker) }
  # rubocop:enable Style/ArrayIntersect
end

.garbled_content?(body) ⇒ Boolean

Returns true when the body looks binary/non-text.

Parameters:

  • body (String) —

    raw response body

Returns:

  • (Boolean) —

    true when the body looks binary/non-text



104
105
106
107
108
109
110
 
# File 'app/services/heatwave/crawler/guards.rb', line 104

def garbled_content?(body)
  sample = body.to_s.byteslice(0, 2048) || ''
  return false if sample.empty?

  non_text = sample.each_byte.count { |byte| byte < 9 || (byte > 13 && byte < 32) || byte == 127 }
  non_text.to_f / sample.bytesize > 0.15
end

.reject_private_targets!(uri) ⇒ void

This method returns an undefined value.

Parameters:

  • uri (URI::HTTP)

Raises:

  • (ArgumentError) —

    when any resolved IP is loopback/private/link-local,
    or when the host does not resolve at all (keeps fetch's contract of
    returning a guarded Result rather than leaking a raw SocketError)



126
127
128
129
130
131
132
133
 
# File 'app/services/heatwave/crawler/guards.rb', line 126

def reject_private_targets!(uri)
  Resolv.each_address(uri.host) do |addr|
    ip = IPAddr.new(addr)
    raise ArgumentError, 'Private or local network targets are not allowed.' if ip.loopback? || ip.private? || ip.link_local?
  end
rescue SocketError
  raise ArgumentError, "Host could not be resolved: #{uri.host}"
end

.sanitize_url_for_log(url) ⇒ String

Returns host+path, safe for log lines (no query/credentials).

Parameters:

  • url (String)

Returns:

  • (String) —

    host+path, safe for log lines (no query/credentials)



114
115
116
117
118
119
 
# File 'app/services/heatwave/crawler/guards.rb', line 114

def sanitize_url_for_log(url)
  uri = URI.parse(url.to_s.strip)
  "#{uri.host}#{uri.path}"
rescue URI::InvalidURIError
  '[invalid-url]'
end

.validate_public_url!(raw_url, allow_private_hosts: false) ⇒ URI::HTTP

Returns the parsed URI.

Parameters:

  • raw_url (String) —

    the URL to validate

  • allow_private_hosts (Boolean) (defaults to: false) —

    skip the private-IP rejection
    (own-site callers such as the internal link validator)

Returns:

  • (URI::HTTP) —

    the parsed URI

Raises:

  • (ArgumentError) —

    when the URL is not fetchable



59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'app/services/heatwave/crawler/guards.rb', line 59

def validate_public_url!(raw_url, allow_private_hosts: false)
  uri = begin
    URI.parse(raw_url.to_s.strip)
  rescue URI::InvalidURIError => e
    # URI.parse raises URI::InvalidURIError (a URI::Error, NOT an
    # ArgumentError) on malformed input — e.g. an unrendered template link
    # like ".../{{ locale }}/...". Honor this method's documented
    # ArgumentError contract so Crawler#fetch's `rescue ArgumentError` turns
    # it into a guarded failure Result instead of crashing the caller
    # (AppSignal #6332: ArticleLinkCheckerWorker).
    raise ArgumentError, "Malformed URL: #{e.message}"
  end
  raise ArgumentError, 'Only http:// and https:// URLs are supported.' unless uri.is_a?(URI::HTTP) && uri.host.present?
  raise ArgumentError, 'URLs with embedded credentials are not supported.' if uri.userinfo.present?

  reject_private_targets!(uri) unless allow_private_hosts
  uri
end