Class: Retailer::WebhookResultProcessor

Inherits:
Object
  • Object
show all
Defined in:
app/services/retailer/webhook_result_processor.rb

Overview

Processes results received via Oxylabs webhook callback.
Uses factory pattern to delegate extraction to retailer-specific extractors.

See Also:

Constant Summary collapse

RENDER_RETRY_LIMIT =

A not_found on a JS-rendered retailer is almost always a degraded Oxylabs
render — a truncated page with no price markup at all — rather than a
genuinely missing product; a product_mismatch can be the same truncated
render minus the element carrying our identifiers. A fresh fetch recovers
~75% of the time (proven on Wayfair, where ~1 in 4 fetches comes back as a
partial ~1.2MB page vs the full ~2.4MB). Re-fetch up to this many times,
per item/day, before accepting the verdict.

2

Instance Method Summary collapse

Constructor Details

#initialize(api: nil, unblocker_api: nil) ⇒ WebhookResultProcessor

Returns a new instance of WebhookResultProcessor.

Parameters:

  • api (Retailer::OxylabsApi) (defaults to: nil) —

    DI seam for tests — defaults to a
    real client with default settings

  • unblocker_api (Retailer::WebUnblockerApi) (defaults to: nil) —

    DI seam for tests —
    defaults to a real client with default settings



22
23
24
25
 
# File 'app/services/retailer/webhook_result_processor.rb', line 22

def initialize(api: nil, unblocker_api: nil)
  @api = api || Retailer::OxylabsApi.new
  @unblocker_api = unblocker_api
end

Instance Method Details

#probe_via_unblocker(catalog_item, url) ⇒ CatalogItemRetailerProbe?

Retry a page that yielded no content at the Scraper API through the
Oxylabs Web Unblocker — opt-in per retailer via
Extractors::Base::WEB_UNBLOCKER_FALLBACK (Wayfair: chronic 613s). The
fetched HTML feeds the normal result path, so extraction, identity
validation and catalog-item bookkeeping are identical to a
webhook-delivered page.

Parameters:

  • catalog_item (CatalogItem)
  • url (String) —

    the URL that faulted at the Scraper API

Returns:

  • (CatalogItemRetailerProbe, nil) —

    nil when the fallback is
    unavailable (not opted in / not configured / fetch failed)



77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
 
# File 'app/services/retailer/webhook_result_processor.rb', line 77

def probe_via_unblocker(catalog_item, url)
  extractor_class = Retailer::Extractors::Factory.for(catalog_item.catalog).class
  return nil unless extractor_class.const_get(:WEB_UNBLOCKER_FALLBACK)

  api = @unblocker_api || Retailer::WebUnblockerApi.new
  return nil unless api.configured?

  # Keep the catalog's storefront geo (Wayfair-CA must not scrape via US
  # egress — wrong currency/pricing); extractors without a storefront_geo
  # default to the client's US state rotation.
  geo = extractor_class.respond_to?(:storefront_geo) ? extractor_class.storefront_geo(catalog_item.catalog) : nil

  result = api.fetch(url, geo_location: geo)
  return nil unless result.success?

  process_result(catalog_item, { 'content' => result.html, 'url' => url },
                 { 'url' => url }, allow_unblocker_fallback: false)
end

#process(payload) ⇒ Object

Note:

Oxylabs callback only notifies that the job is done - we must fetch results separately

Process a single webhook payload from Oxylabs

Parameters:

  • payload (Hash) —

    The webhook payload containing job notification



31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
 
# File 'app/services/retailer/webhook_result_processor.rb', line 31

def process(payload)
  job_id = payload['id']
  status = payload['status']

  Rails.logger.info "[Oxylabs Webhook] Processing job #{job_id} with status: #{status}"

  unless status == 'done'
    Rails.logger.warn "[Oxylabs Webhook] Job #{job_id} not done (status: #{status})"
    return
  end

  # Find the catalog item from URL params or context
  catalog_item = find_catalog_item(payload)
  return unless catalog_item

  # Fetch the actual results from Oxylabs API
  result = @api.job_results(job_id)
  unless result.success?
    Rails.logger.error "[Oxylabs Webhook] Failed to fetch results for job #{job_id}: #{result.error}"
    return
  end

  results = result.data
  if results.blank?
    Rails.logger.warn "[Oxylabs Webhook] Job #{job_id} returned empty results"
    return
  end

  content_size = results.first&.dig('content').to_s.size
  Rails.logger.info "[Oxylabs Webhook] Fetched results for job #{job_id}: #{content_size} bytes"

  # Process the result
  process_result(catalog_item, results.first, payload)
end