Class: Retailer::SearchUrlDiscovery

Inherits:
Object
  • Object
show all
Includes:
CatalogConstants
Defined in:
app/services/retailer/search_url_discovery.rb

Overview

Finds a canonical product URL by running the retailer's OWN site search for
our manufacturer SKU, then proving the page it lands on is actually ours.

This is the last-resort strategy, for retailers that neither expose a
deterministic PDP keyed by an id they gave us (UrlConstructor) nor
hand us the mapping through an API (WalmartItemIdDiscovery,
BestBuyUrlDiscovery) nor left a resolved URL on a past probe row
(ProbeHistoryUrlDiscovery). Today: Wayfair and Lowe's Canada.

== Why this is not the search fallback PR #1480 deleted

That fallback STORED the search URL itself as the thing to probe, and "checked"
identity by looking for the identifier it had just put in the query — it always
matched, so any wrong landing was accepted and its price fed MAP reporting.

Here the search URL is never stored and never probed. It is a lookup step whose
only output is a candidate PDP, and a candidate is accepted only when BOTH:

  1. its URL is a direct product page (UrlPatterns), and
  2. the PDP's own content carries our SKU.

Condition 2 is asserted exclusively against pages that satisfy condition 1 —
never against the search-results page, which of course contains the SKU we
just searched for. That ordering is the whole safety property; inverting it
reintroduces the 2026-07-09 bug.

Examples:

Retailer::SearchUrlDiscovery.new.discover_url(catalog_item)

Constant Summary collapse

SEARCH_URL_BUILDERS =

Retailer site-search endpoints, by catalog.

Lowe's Canada was removed: the chain was absorbed into RONA and lowes.ca
no longer resolves at all (NXDOMAIN), so searching it could only ever burn
requests. Its catalog is retired and its one still-sold product is covered
by the Rona catalog.

{
  WAYFAIR_CANADA => ->(sku) { "https://www.wayfair.ca/keyword.php?keyword=#{CGI.escape(sku)}" },
  WAYFAIR_USA => ->(sku) { "https://www.wayfair.com/keyword.php?keyword=#{CGI.escape(sku)}" }
}.freeze
MAX_FETCH_ATTEMPTS =

Oxylabs returns an empty body rather than an error when a target challenges
the request, and these storefronts do it intermittently — a bare retry
usually lands. Bounded so a hard block can't spin: at ~40s per rendered
fetch, a third attempt costs more than it recovers.

2
MAX_CANDIDATES =

Pages to open when indexing one parent. Wayfair advertises a handful of piid
variants per parent (4 for the WSHC-240 cables), and the index is built once
per parent and answers every item in that family, so this only needs to be
large enough for the biggest family — not for a whole results page.

12
RENDER_WAIT_SECONDS = 
8

Constants included from CatalogConstants

CatalogConstants::ALL_MAIN_CATALOG_IDS, CatalogConstants::AMAZON_CATALOG_IDS, CatalogConstants::AMAZON_CA_CATALOG_IDS, CatalogConstants::AMAZON_EU_CATALOG_IDS, CatalogConstants::AMAZON_NA_SELLER_IDS, CatalogConstants::AMAZON_SC_BE_CATALOG_ID, CatalogConstants::AMAZON_SC_CATALOG_IDS, CatalogConstants::AMAZON_SC_CA_CATALOG_ID, CatalogConstants::AMAZON_SC_DE_CATALOG_ID, CatalogConstants::AMAZON_SC_ES_CATALOG_ID, CatalogConstants::AMAZON_SC_FR_CATALOG_ID, CatalogConstants::AMAZON_SC_IT_CATALOG_ID, CatalogConstants::AMAZON_SC_NL_CATALOG_ID, CatalogConstants::AMAZON_SC_PL_CATALOG_ID, CatalogConstants::AMAZON_SC_SE_CATALOG_ID, CatalogConstants::AMAZON_SC_UK_CATALOG_ID, CatalogConstants::AMAZON_SC_US_CATALOG_ID, CatalogConstants::AMAZON_SELLER_IDS, CatalogConstants::AMAZON_US_CATALOG_IDS, CatalogConstants::AMAZON_VC_CATALOG_IDS, CatalogConstants::AMAZON_VC_CA_CATALOG_ID, CatalogConstants::AMAZON_VC_CA_CATALOG_IDS, CatalogConstants::AMAZON_VC_DIRECT_FULFILLMENT_CATALOG_IDS, CatalogConstants::AMAZON_VC_US_CATALOG_IDS, CatalogConstants::AMAZON_VC_US_WASN4_CATALOG_ID, CatalogConstants::AMAZON_VC_US_WAX7V_CATALOG_ID, CatalogConstants::AMAZON_VC_WAT0F_CA_CATALOG_ID, CatalogConstants::AMAZON_VC_WAT4D_CA_CATALOG_ID, CatalogConstants::AMAZON_VENDOR_CODE_TO_CATALOG_ID, CatalogConstants::BESTBUY_CANADA, CatalogConstants::BUILD_COM, CatalogConstants::CANADIAN_TIRE, CatalogConstants::CA_CATALOG_ID, CatalogConstants::COSTCO_CANADA, CatalogConstants::COSTCO_CATALOGS, CatalogConstants::COSTCO_USA, CatalogConstants::EU_CATALOG_ID, CatalogConstants::HOME_DEPOT_CANADA, CatalogConstants::HOME_DEPOT_CATALOGS, CatalogConstants::HOME_DEPOT_USA, CatalogConstants::HOUZZ, CatalogConstants::LOCALE_TO_CATALOG, CatalogConstants::LOWES_CANADA, CatalogConstants::LOWES_USA, CatalogConstants::RONA_CANADA, CatalogConstants::US_CATALOG_ID, CatalogConstants::WALMART_CATALOGS, CatalogConstants::WALMART_SELLER_CANADA, CatalogConstants::WALMART_SELLER_USA, CatalogConstants::WAYFAIR_CANADA, CatalogConstants::WAYFAIR_CATALOGS, CatalogConstants::WAYFAIR_GERMANY, CatalogConstants::WAYFAIR_USA

Instance Method Summary collapse

Methods included from CatalogConstants

amazon_catalog?, amazon_seller_catalog?, costco_catalog?, home_depot_catalog?, walmart_catalog?, wayfair_catalog?

Constructor Details

#initialize(api: nil, logger: Rails.logger) ⇒ SearchUrlDiscovery

Returns a new instance of SearchUrlDiscovery.

Parameters:



61
62
63
64
 
# File 'app/services/retailer/search_url_discovery.rb', line 61

def initialize(api: nil, logger: Rails.logger)
  @api = api || Retailer::OxylabsApi.new(timeout: 120)
  @logger = logger
end

Instance Method Details

#discover_url(catalog_item) ⇒ String?

Returns a PDP URL proven to be this item's, or nil.

Parameters:

Returns:

  • (String, nil) —

    a PDP URL proven to be this item's, or nil



68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
 
# File 'app/services/retailer/search_url_discovery.rb', line 68

def discover_url(catalog_item)
  builder = SEARCH_URL_BUILDERS[catalog_item.catalog_id]
  sku = catalog_item.sku.to_s
  return nil if builder.nil? || sku.blank?

  page = fetch(builder.call(sku))
  return nil if page.nil?

  # A variant-specific URL always wins over the page we happened to land on.
  #
  # An exact-SKU search often redirects straight to a product page, and that
  # page may even declare our part number — but if its URL carries no variant
  # discriminator it is the PARENT, and probing it reads the parent's default
  # variant. WSHM-240-03040 landed exactly that way, on the 120V parent of a
  # 240V product. Consulting the index first costs nothing after the first item
  # in a family, since it is memoized per parent.
  landed = page[:url]
  indexed = variant_index_for(page)[normalize(sku)]
  return indexed if indexed.present?

  return landed if product_page?(landed) && variant_specific?(landed) && identifies?(page[:content], sku)

  nil
end