Class: Heatwave::Crawler::Tiers::Playwright::Limiter

Inherits:
Object
  • Object
show all
Defined in:
app/services/heatwave/crawler/tiers/playwright/limiter.rb

Overview

Concurrency lease for the single-node Playwright accessory. Renders
are the most expensive thing the crawler does and the accessory is
one container on one host, so concurrent renders are capped with
SET-NX-with-TTL slot keys: atomic, and crash-safe (an abandoned slot
frees itself when the lease TTL lapses).

When no slot frees within WAIT_BUDGET, Limiter.with_slot returns nil
instead of yielding — the tier turns that into an error outcome so
the crawler escalates rather than queueing browsers.

Constant Summary collapse

KEY_PREFIX = 
'crawler:playwright:slot'
SLOTS = 
2
LEASE_TTL = 
120.seconds
WAIT_BUDGET = 
10.seconds
WAIT_INTERVAL = 
0.5

Class Method Summary collapse

Class Method Details

.with_slot(wait_budget: WAIT_BUDGET) { ... } ⇒ Object?

Returns the block's value, or nil when no slot
became available within +wait_budget+.

Parameters:

  • wait_budget (ActiveSupport::Duration) (defaults to: WAIT_BUDGET) —

    how long to wait for
    a free slot before giving up (injectable for tests)

Yields:

  • runs the render while holding a slot

Returns:

  • (Object, nil) —

    the block's value, or nil when no slot
    became available within +wait_budget+



28
29
30
31
32
33
34
35
36
37
 
# File 'app/services/heatwave/crawler/tiers/playwright/limiter.rb', line 28

def with_slot(wait_budget: WAIT_BUDGET)
  slot = acquire_slot(wait_budget)
  return nil unless slot

  begin
    yield
  ensure
    store.delete(slot)
  end
end