Class: ImageEmbeddingPopulationWorker

Inherits:
Object
  • Object
show all
Includes:
Sidekiq::IterableJob, Sidekiq::Job
Defined in:
app/workers/image_embedding_population_worker.rb

Overview

Nightly worker to backfill images missing Gemini Embedding 2 unified embeddings,
replacing all legacy Jina v4 embeddings in the process.

Uses Sidekiq::IterableJob so progress is checkpointed after each image — a
mid-run deploy or worker restart resumes from the last successful record rather
than restarting from scratch.

Each iteration queues an ImageFullAnalysisWorker job which runs the full
pipeline: pHash → Gemini Embedding 2 → Gemini Flash vision.

Prioritizes product primary images first (most critical for search), then all
remaining active images, both groups ordered by id for stable cursor resumption.

Scheduled: Nightly at 2:30 AM CT

Examples:

Manual trigger (default: 5000 images/run)

ImageEmbeddingPopulationWorker.perform_async

Custom limit

ImageEmbeddingPopulationWorker.perform_async('limit' => 1000)

Product primary images only

ImageEmbeddingPopulationWorker.perform_async('product_images_only' => true)

Constant Summary collapse

DEFAULT_LIMIT =

Default limit.

5000
TARGET_MODEL =

Target model (GA), sourced from the canonical registry. Changing the
registry id from the pre-GA preview arms the throttled re-embed: images
still tagged with the old id fall back into the candidate scope and are
re-embedded to the GA model over nightly runs.

AiModelConstants.id(:unified_embedding)

Instance Method Summary collapse

Instance Method Details

#build_enumerator(options = nil, cursor:) ⇒ Object

Parameters:

  • options (Hash) (defaults to: nil) —

    job options (string or symbol keys)

  • cursor (Object) —

    iteration cursor managed by sidekiq-iteration

Options Hash (options):

  • limit (Integer) —

    maximum images to queue this run (default DEFAULT_LIMIT)

  • product_images_only (Boolean) —

    restrict to product primary images



45
46
47
48
49
50
51
52
53
54
55
56
57
58
 
# File 'app/workers/image_embedding_population_worker.rb', line 45

def build_enumerator(options = nil, cursor:)
  options ||= {} # sidekiq-iteration omits the arg entirely when no options were enqueued
  @limit = (options[:limit] || DEFAULT_LIMIT).to_i
  @product_images_only = options[:product_images_only].to_b
  @queued_count = 0

  scope = build_scope
  candidate_count = scope.count
  log_info "Starting: #{candidate_count} images need Gemini embedding (limit: #{@limit}, product_only: #{@product_images_only})"

  return nil if candidate_count.zero?

  active_record_records_enumerator(scope.limit(@limit), cursor: cursor)
end

#each_iteration(image, *_args) ⇒ Object

Each iteration.

Parameters:

  • image (Image) —

    the image to each

  • _args (Array) —

    the args

Returns:

  • (Object) —

    the result



65
66
67
68
69
70
 
# File 'app/workers/image_embedding_population_worker.rb', line 65

def each_iteration(image, *_args)
  ImageFullAnalysisWorker.perform_async(image.id, { 'force' => false })
  @queued_count += 1

  log_info "Progress: #{@queued_count} queued" if (@queued_count % 500).zero?
end

#on_complete ⇒ Object

On complete.

Returns:

  • (Object) —

    the result



75
76
77
78
79
80
81
 
# File 'app/workers/image_embedding_population_worker.rb', line 75

def on_complete
  if @queued_count.zero?
    log_info 'All images already have Gemini embeddings — nothing to queue'
  else
    log_info "Complete: queued #{@queued_count} images for AI analysis"
  end
end