Class: Retailer::BatchPriceChecker

Inherits:
Object
  • Object
show all
Defined in:
app/services/retailer/batch_price_checker.rb

Overview

Batch price checker using Oxylabs Push-Pull API with webhook callbacks.
Submits jobs asynchronously with per-item callback URLs. Results are
processed by WebhookProcessors::OxylabsProcessor when webhooks arrive.

Creates pending WebhookLog entries for each submitted job to enable:

  • Audit trail of all submitted jobs
  • Detection of missed webhook callbacks
  • Retry logic via StaleTranscriptionRecoveryWorker

In development, use the dev tunnel (script/dev_tunnel_api.sh) to receive callbacks.

Examples:

Check all retailers

checker = Retailer::BatchPriceChecker.new
checker.check_all_retailers

Check a single catalog

checker = Retailer::BatchPriceChecker.new
checker.check_catalog(Catalog.find(18))

Constant Summary collapse

BATCH_SIZE =

Submit jobs in batches of 50

50
COSTCO_REQUEST_SPACING =

Seconds to wait between Costco API probes. Costco's price API
rate-limits request bursts; ~8s spacing keeps a full catalog run
within limits.

8

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ BatchPriceChecker

Returns a new instance of BatchPriceChecker.

Parameters:

  • options (Hash) (defaults to: {}) —

    configurable options

Options Hash (options):

  • api (Retailer::OxylabsApi) —

    API client to use (defaults to a new instance with a 120s timeout)

  • url_constructor (Retailer::UrlConstructor) —

    URL constructor to use (defaults to a new instance)

  • logger (Logger) —

    logger to use (defaults to Rails.logger)



35
36
37
38
39
40
 
# File 'app/services/retailer/batch_price_checker.rb', line 35

def initialize(options = {})
  @api = options[:api] || Retailer::OxylabsApi.new(timeout: 120)
  @url_constructor = options[:url_constructor] || Retailer::UrlConstructor.new
  @logger = options[:logger] || Rails.logger
  @price_checker = Retailer::PriceChecker.new(api: @api, url_constructor: @url_constructor, logger: @logger)
end

Instance Attribute Details

#api ⇒ Object (readonly)

Returns the value of attribute api.



29
30
31
 
# File 'app/services/retailer/batch_price_checker.rb', line 29

def api
  @api
end

#logger ⇒ Object (readonly)

Returns the value of attribute logger.



29
30
31
 
# File 'app/services/retailer/batch_price_checker.rb', line 29

def logger
  @logger
end

#url_constructor ⇒ Object (readonly)

Returns the value of attribute url_constructor.



29
30
31
 
# File 'app/services/retailer/batch_price_checker.rb', line 29

def url_constructor
  @url_constructor
end

Instance Method Details

#callback_url_for(catalog_item_id) ⇒ String

Generate a callback URL with a time-limited authentication token (24 hours)
Each job gets its own callback URL with the catalog_item_id embedded

Parameters:

  • catalog_item_id (Integer) —

    The catalog item ID to embed in the callback URL

Returns:

  • (String) —

    The callback URL based on environment:

    • Production: api.warmlyyours.com
    • Staging: api.warmlyyours.ws
    • Development: api-{hostname}.warmlyyours.dev (via Cloudflare tunnel)


49
50
51
52
53
54
55
56
 
# File 'app/services/retailer/batch_price_checker.rb', line 49

def callback_url_for(catalog_item_id)
  if Rails.env.development?
    Retailer::CallbackTokenService.dev_callback_url(catalog_item_id: catalog_item_id)
  else
    # Production and staging use API_HOSTNAME_WITHOUT_PORT constant
    Retailer::CallbackTokenService.callback_url(catalog_item_id: catalog_item_id)
  end
end

#check_all_retailers ⇒ Hash

Check all catalogs that have external_price_check_enabled
Jobs are submitted asynchronously - results arrive via webhook

Returns:

  • (Hash) —

    Summary of submitted jobs by catalog



61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'app/services/retailer/batch_price_checker.rb', line 61

def check_all_retailers
  catalogs = Catalog.where(external_price_check_enabled: true)
  results = { total_submitted: 0, catalogs: {} }

  catalogs.each do |catalog|
    submitted = check_catalog(catalog)

    results[:total_submitted] += submitted
    results[:catalogs][catalog.name] = submitted

    @logger.info "[BatchPriceChecker] #{catalog.name}: #{submitted} jobs submitted"
  end

  @logger.info "[BatchPriceChecker] Total: #{results[:total_submitted]} jobs submitted, awaiting callbacks"
  results
end

#check_catalog(catalog) ⇒ Integer

Check all items in a catalog using batch processing

Parameters:

Returns:

  • (Integer) —

    Number of jobs submitted



81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
 
# File 'app/services/retailer/batch_price_checker.rb', line 81

def check_catalog(catalog)
  items = catalog.catalog_items
                 .where(state: 'active')
                 .probeable
                 .includes(:catalog)
                 .to_a

  return 0 if items.empty?

  # Per-catalog cadence: skip items whose last probe is newer than
  # `probe_cadence_days` ago. Default cadence is 1 day (no filter), but
  # slow-moving catalogs (Costco, Lowes Canada) can be set to 3
  # or 7 to cut request volume proportionally. Items that have never
  # been probed (`url_last_checked: nil`) are always included.
  cadence_days = catalog.probe_cadence_days.to_i
  if cadence_days > 1
    original_count = items.size
    cutoff = cadence_days.days.ago
    items = items.reject { |item| item.url_last_checked.present? && item.url_last_checked > cutoff }
    skipped_count = original_count - items.size
    @logger.info "[BatchPriceChecker] Skipped #{skipped_count} items checked within last #{cadence_days}d (cadence=#{cadence_days}d) for #{catalog.name}" if skipped_count > 0
  end

  return 0 if items.empty?

  @logger.info "[BatchPriceChecker] Processing #{items.size} items for #{catalog.name}"

  # Costco and Walmart Seller use synchronous retailer APIs (no Oxylabs job /
  # webhook callback). Probe the whole catalog in one pass so Costco's
  # inter-request spacing holds continuously, instead of resetting at every
  # 50-item slice boundary. Walmart is our own Marketplace account — same as
  # Amazon Seller — so /v3/items is the system of record; scraping walmart.com
  # is what filled the dashboard with product_mismatch "scrape failures".
  api_count = probe_via_own_api(catalog, items)
  return api_count if api_count

  # Process in batches and count total submitted
  items.each_slice(BATCH_SIZE).sum do |batch|
    process_batch(batch, catalog)
  end
end

#check_single_item(catalog_item) ⇒ Integer

Check a single catalog item
Only processes active items that aren't set to skip URL checks

Parameters:

Returns:

  • (Integer) —

    1 if submitted, 0 if skipped



127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
 
# File 'app/services/retailer/batch_price_checker.rb', line 127

def check_single_item(catalog_item)
  # Only check active items (skip pending_onboarding, phased_out, etc.)
  unless catalog_item.state == 'active'
    @logger.info "[BatchPriceChecker] Skipping item #{catalog_item.id} - state is '#{catalog_item.state}' (not active)"
    return 0
  end

  # Two gates: the human's permanent opt-out, and the auto-backoff a failing
  # item earns for a week (Retailer::ProbeAutoSkipper).
  unless catalog_item.probeable?
    @logger.info "[BatchPriceChecker] Skipping item #{catalog_item.id} - #{catalog_item.unprobeable_reason}"
    return 0
  end

  @logger.info "[BatchPriceChecker] Processing single item #{catalog_item.id}"
  catalog = catalog_item.catalog
  api_count = probe_via_own_api(catalog, [catalog_item])
  return api_count if api_count

  process_batch([catalog_item], catalog)
end