Class: AdSpendBackfillWorker

Inherits:
Object
  • Object
show all
Includes:
Sidekiq::Job
Defined in:
app/workers/ad_spend_backfill_worker.rb

Overview

Walks AdSpendSyncWorker back over a date range, one day per job.

Deliberately not a loop inside a single job. Ad-spend history can span
hours because Amazon reports complete asynchronously, and
kamal deploy cycles every Sidekiq worker: a long-running job would be
interrupted mid-range and lose its place. Instead each job does one day and
enqueues the next, so the remaining work lives in Redis rather than in a
process — a deploy requeues the interrupted day and the chain carries on.

Resumable and re-runnable by design:

  • the cursor advances monotonically, so a day that legitimately has no
    spend can't wedge the chain;
  • a day is skipped only once it has completed with every provider
    succeeding, recorded as a marker. Data presence alone is not a safe
    signal: a partially-completed day still wrote rows for the providers
    that succeeded, and a day with genuinely zero spend writes none at all —
    either would be mistaken for "done";
  • SourceDataPoint upserts, so a day processed twice overwrites rather
    than doubling.

A day whose provider fails is logged and left unmarked, so the chain still
covers the whole range and a re-run retries exactly the days that failed.

AdSpendBackfillWorker.perform_async('2026-06-28', '2026-07-27')

Past Amazon's 60-day retention, scope the range to the providers that keep
history that far back — otherwise every day fires an Amazon request certain
to be rejected, and each rejection opens the cohort cooldown:

AdSpendBackfillWorker.perform_async(
'2025-08-01', '2026-06-04', AdSpendSyncWorker::DEEP_HISTORY_PROVIDERS
)

Serialization is the point: firing a month of days in parallel would mean
hundreds of concurrent Amazon report requests. The advisory lock enforces
that even if two chains get started by mistake.

Constant Summary collapse

LOCK_RETRY_DELAY =

Retry delay when another chain holds the lock. Short — we only need to
outlast whatever day it is working — but non-zero so we don't spin.

2.minutes

Instance Method Summary collapse

Instance Method Details

#perform(cursor_string, end_date_string, providers = nil) ⇒ void

This method returns an undefined value.

Parameters:

  • cursor_string (String) —

    ISO date to process on this pass

  • end_date_string (String) —

    ISO date, inclusive, to stop after

  • providers (Array<String>, nil) (defaults to: nil) —

    restrict the run to these providers;
    nil runs every one. Scope a historical range to
    AdSpendSyncWorker::DEEP_HISTORY_PROVIDERS — past Amazon's 60-day
    retention every day would otherwise fire a doomed Amazon request whose
    rejection opens the cohort cooldown and stalls the chain.



56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
 
# File 'app/workers/ad_spend_backfill_worker.rb', line 56

def perform(cursor_string, end_date_string, providers = nil)
  cursor = Date.parse(cursor_string)
  end_date = Date.parse(end_date_string)
  scope = AdSpendSyncWorker.scope_for(providers)
  return Rails.logger.info("[AdSpendBackfillWorker] finished through #{end_date}") if cursor > end_date

  # Cohort-wide gate, checked before any provider work: whoever tripped the
  # throttle pauses everyone, not just themselves. Only a run that includes
  # Amazon is in that cohort — it is the sole provider that opens the gate.
  remaining = AdSpendSyncWorker.throttleable?(providers) ? Marketing::AdSpend::RunState.cooldown_remaining : 0
  if remaining.positive?
    Rails.logger.info("[AdSpendBackfillWorker] #{cursor} held off — #{remaining}s left on the throttle cooldown")
    return self.class.perform_in(remaining.seconds, cursor_string, end_date_string, providers)
  end

  # Hand the day to a batch and stop. The providers run in parallel and
  # AdSpendDayFinalizer advances the chain when they all land — so this job
  # does not hold a thread while Amazon's reports generate, and a retry is
  # scoped to the provider that actually failed.
  if Marketing::AdSpend::RunState.completed?(cursor, scope:)
    log_skip(cursor)
    return self.class.perform_async((cursor + 1).to_s, end_date_string, providers)
  end

  if Marketing::AdSpend::RunState.in_flight?(cursor, scope:)
    Rails.logger.info("[AdSpendBackfillWorker] #{cursor} already has a batch in flight; deferring")
    return self.class.perform_in(LOCK_RETRY_DELAY, cursor_string, end_date_string, providers)
  end

  locked = Marketing::AdSpend::RunState.with_day_lock do
    # Re-check inside the lock: another chain may have finished this day, or
    # started its batch, while we waited for it.
    if Marketing::AdSpend::RunState.completed?(cursor, scope:) ||
       Marketing::AdSpend::RunState.in_flight?(cursor, scope:)
      :skip
    else
      # AdSpendSyncWorker claims the day itself — every dispatcher routes
      # through it, so the claim belongs there rather than in each caller.
      AdSpendSyncWorker.new.perform(cursor_string, end_date_string, providers)
    end
  end

  return if locked && locked != :skip # the finalizer owns the chain from here

  if locked == :skip
    log_skip(cursor)
    self.class.perform_async((cursor + 1).to_s, end_date_string, providers)
  else
    # Another chain is mid-day. Advancing would let this one race ahead of a
    # cohort that may be about to throttle.
    Rails.logger.info("[AdSpendBackfillWorker] #{cursor} deferred — another backfill holds the lock")
    self.class.perform_in(LOCK_RETRY_DELAY, cursor_string, end_date_string, providers)
  end
end