Class: CatalogItemUrlWorker

Inherits:
Object
  • Object
show all
Includes:
Sidekiq::Job
Defined in:
app/workers/catalog_item_url_worker.rb

Overview

Sidekiq worker that validates a CatalogItem's external retailer URL.

Catalogs in the Oxylabs price-check rotation defer to the authoritative
retailer probe (see #perform); everything else gets a cheap Googlebot-shaped
HTTP GET and records +url_valid+ from the status.

See Also:

Constant Summary collapse

HEADERS =

The following headers were extracted from a chrome inspector mimic as curl request

{
  'User-Agent': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
}.freeze
URL_PROBE_POLICY =

Crawler policy for the naive status probe: Googlebot-shaped UA, no
redirect following, 10s timeouts — the same shape the legacy Faraday
client had. The crawler's direct tier also applies Web Bot Auth signing
and the SSRF guard.

Heatwave::Crawler::Policy.new(
  user_agent: HEADERS[:'User-Agent'],
  follow_redirects: false,
  timeout_seconds: 10
).freeze
FUNNEL_ENQUEUE_PAUSED_CATALOG_IDS =

Oxylabs-rotation catalogs whose probe is currently unreliable: we still
defer to the probe (skip the naive GET) but do NOT actively enqueue a new
one — it would just burn Oxylabs budget on failures. Remove a catalog here
once its probe is healthy again.
rona.ca re-enabled: it now probes via the universal source + JS render
against sitemap-discovered product URLs (Retailer::RonaUrlDiscovery), and a
stale URL self-heals via the on-failure rediscovery path
(Retailer::UrlRediscovery) before ProbeAutoSkipper would give up on it.

See Also:

  • Oxylabs follow-up (Basecamp todolist 498301955)
[
  CatalogConstants::LOWES_CANADA
].freeze

Instance Method Summary collapse

Instance Method Details

#http_get(url) ⇒ Heatwave::Crawler::Result

Probes +url+ through the crawler's direct tier (Googlebot-shaped UA,
no redirect following, 10s timeouts) and returns the probe Result.
Transport failures are re-raised so #perform keeps its established
rescue/telemetry path.

Parameters:

  • url (String) —

    the absolute URL to probe

Returns:



52
53
54
55
56
57
 
# File 'app/workers/catalog_item_url_worker.rb', line 52

def http_get(url)
  result = Heatwave::Crawler.fetch(url, mode: :probe, policy: URL_PROBE_POLICY)
  return result if result.success?

  raise(result.details[:exception] || HTTP::ConnectionError.new(result.error.to_s))
end

#perform(catalog_item_id) ⇒ Boolean?

Validates the catalog item's URL. Oxylabs-rotation catalogs are funnelled
through the authoritative retailer probe (the naive GET is skipped); other
catalogs get the naive GET, still deferring to any recent probe. Blank URLs
and skip_url_checks items are no-ops.

Parameters:

  • catalog_item_id (Integer) —

    the CatalogItem to probe

Returns:

  • (Boolean, nil) —

    +url_valid+ (true/false) after a completed naive GET;
    nil when funnelled to Oxylabs, skipped, or left inconclusive by a transport error



67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
 
# File 'app/workers/catalog_item_url_worker.rb', line 67

def perform(catalog_item_id)
  catalog_item = CatalogItem.where.not(skip_url_checks: true).find(catalog_item_id)
  return if catalog_item.url.blank?

  # Catalogs in the Oxylabs price-check rotation: the retailer probe (JS
  # render + rotating geo-IP) is authoritative for `url_valid`, and a naive
  # Googlebot-spoofed GET from our single server IP is doomed against their
  # anti-bot — it only produces Faraday transport-error noise on AppSignal
  # (#5565 / #5386). Skip the GET and funnel through Oxylabs instead.
  return funnel_through_oxylabs(catalog_item) if catalog_item.catalog&.external_price_check_enabled?

  # Other catalogs keep the cheap naive GET, but still defer to a recent
  # probe when one happens to exist (same authoritative-result reasoning:
  # if Oxylabs with JS + geo-IP couldn't reach the page, our single-IP
  # Googlebot GET won't either, so re-probing only adds AppSignal noise).
  if recent_probe?(catalog_item)
    logger.info "Skipping URL check for #{catalog_item.id} — recent probe exists (authoritative)"
    return
  end

  res = nil
  begin
    res = http_get(catalog_item.url)
    logger.info "#{catalog_item.url} result was #{res.status}"
  rescue Faraday::ConnectionFailed, Faraday::TimeoutError, Faraday::SSLError,
         HTTP::Error, SocketError => e
    # Network-level failure — couldn't determine whether the page exists.
    # Don't flip url_valid to false; the URL might be fine and the next
    # probe will tell us. Log and bail without touching the row.
    # Intentionally don't re-raise — Sidekiq retries would just hit the
    # same flaky upstream without new info.
    #
    # Trend the volume as a *counter metric*, not an exception report.
    # This worker probes ~thousands of third-party retailer URLs daily,
    # many of which block our Googlebot-spoofed User-Agent or briefly fail
    # DNS; a baseline of transport failures is expected operational noise,
    # not actionable per-occurrence. Reporting it via
    # `ErrorReporting.informational` created AppSignal *exception incident*
    # #5565 — and even though background_info is "Never Notify", every
    # occurrence re-opened the incident (closed 7× May–Jun 2026, always
    # reopened; #5012/#5013 before it closed 8× each). A counter keeps the
    # volume trended and per-host filterable (the per-retailer backoff
    # signal the triage kept asking for) without the zombie-incident churn.
    logger.warn "Transport error retrieving #{catalog_item.url}: #{e.class} #{e.message}"
    track_transport_error(e, catalog_item)
    return nil
  end

  valid = res&.status == 200
  catalog_item.update_columns(url_valid: valid, url_last_checked: Time.current)
  valid
end