Class: Embedding::TextUnifier

Inherits:
Object
  • Object
show all
Defined in:
app/services/embedding/text_unifier.rb

Overview

Backfills Gemini Embedding 2 vectors into content_embeddings.unified_embedding
for text content, so text and images share one multimodal vector space.

It reads each source content_type = 'primary' text row, re-embeds its
content with Gemini (batched), and upserts a sibling content_type = 'unified' row tagged gemini-embedding-2. Chunked records are deliberately
excluded: only the normal EmbeddingWorker can preserve their complete
unified_chunk_<n> storage shape.

HISTORICAL — the migration this exists for is DONE. primary is the
pre-unification content_type; nothing writes it any more (the live path in
Models::Embeddable#generate_embedding! always sets unified /
unified_chunk_N), the OpenAI embedding column was dropped in PR #1055
(2026-06-07), and the read path cut over to unified_search long ago. The
legacy primary rows that remain are a fixed, non-growing set. This class is
kept only so the conversion is reproducible if a stray primary row ever
reappears. Always build input through TextUnifier.candidate_scope; it recognizes both
valid unified storage shapes and correlates them by locale.

INERT by default: nothing invokes this from a model callback or the live
search path. Run it explicitly via rake embeddings:backfill_unified_text
(count-first, gated) per the runbook.

See Also:

  • doc/tasks/202606051030_TEXT_EMBEDDING_UNIFICATIONdoc/tasks/202606051030_TEXT_EMBEDDING_UNIFICATION.md

Constant Summary collapse

TEXT_TYPES =

Embeddable TEXT types eligible for unification — every embeddable type
except Image (images are embedded multimodally by the image pipeline).
Includes the sensitive internal types (CallRecord/Activity/Communication).

%w[
  Post Article Showcase Video Item ProductLine SiteMap ReviewsIo
  CallRecord Activity Communication AssistantBrainEntry
].freeze
MODEL =

GA multimodal model written into unified_embedding.

ContentEmbedding::UNIFIED_MODEL
DIMENSIONS =

MRL output width (HNSW-compatible; matches image unified embeddings).

1536
BATCH_SIZE =

Items per Gemini batchEmbedContents request.

Embedding::Gemini::MAX_BATCH_SIZE
MAX_CONTENT_LENGTH =

Truncate to stay within the model's ~8k-token text window.

Models::Embeddable::MAX_CONTENT_LENGTH

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(dimensions: DIMENSIONS) ⇒ TextUnifier

Returns a new instance of TextUnifier.



91
92
93
 
# File 'app/services/embedding/text_unifier.rb', line 91

def initialize(dimensions: DIMENSIONS)
  @dimensions = dimensions
end

Class Method Details

.backfill(primary_rows, dimensions: DIMENSIONS) ⇒ Hash

Backfill a set of source primary rows.

Parameters:

  • primary_rows (ActiveRecord::Relation, Array<ContentEmbedding>) —

    content_type='primary' text rows to mirror into the unified space

  • dimensions (Integer) (defaults to: DIMENSIONS) —

    output vector width

Returns:

  • (Hash) —

    counts — :processed, :skipped, :failed



52
53
54
 
# File 'app/services/embedding/text_unifier.rb', line 52

def self.backfill(primary_rows, dimensions: DIMENSIONS)
  new(dimensions: dimensions).backfill(primary_rows)
end

.candidate_scope(types: TEXT_TYPES) ⇒ ActiveRecord::Relation<ContentEmbedding>

Find legacy primary rows that can safely be converted to one unified row.

A current single row completes the migration slot. Any chunk row also
excludes the source, regardless of model metadata: this legacy converter
must never add a single row beside a chunked shape. The normal embedding
refresh path owns validation and repair of chunk rows.

Parameters:

  • types (Array<String>) (defaults to: TEXT_TYPES) —

    embeddable types to inspect

Returns:



65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
 
# File 'app/services/embedding/text_unifier.rb', line 65

def self.candidate_scope(types: TEXT_TYPES)
  ContentEmbedding
    .where(embeddable_type: types, content_type: 'primary')
    .where(
      <<~SQL.squish,
        NOT EXISTS (
          SELECT 1
          FROM content_embeddings unified
          WHERE unified.embeddable_type = content_embeddings.embeddable_type
            AND unified.embeddable_id = content_embeddings.embeddable_id
            AND unified.locale IS NOT DISTINCT FROM content_embeddings.locale
            AND (
              (
                unified.content_type = 'unified'
                AND unified.embedding_model = ?
                AND unified.unified_embedding IS NOT NULL
              )
              OR unified.content_type LIKE 'unified\\_chunk\\_%' ESCAPE '\\'
            )
        )
      SQL
      MODEL
    )
    .preload(:embeddable)
end

Instance Method Details

#backfill(primary_rows) ⇒ Object 



95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
 
# File 'app/services/embedding/text_unifier.rb', line 95

def backfill(primary_rows)
  counts = { processed: 0, skipped: 0, failed: 0 }

  each_batch(primary_rows) do |rows|
    prepared = rows.filter_map do |row|
      content = content_for(row)
      if content.blank?
        counts[:skipped] += 1
        nil
      else
        { row: row, content: content, content_hash: content_hash_for(row) }
      end
    end
    next if prepared.empty?

    vectors = Embedding::Gemini.embed_texts(prepared.pluck(:content), dimensions: @dimensions)
    write_batch(prepared, vectors, counts)
  end

  counts
end