Module: Models::Embeddable

Extended by:
ActiveSupport::Concern
Included in:
Activity, Article, AssistantBrainEntry, CallRecord, Communication, Image, Item, ProductLine, ReviewsIo, Showcase, SiteMap, Video
Defined in:
app/concerns/models/embeddable.rb

Overview

Concern for models that support vector embeddings for semantic search.
Include this in any model that should be searchable via AI-powered
semantic search.

Examples:

Basic usage

class Showcase < ApplicationRecord
  include Models::Embeddable

  def self.embeddable_content_types
    [:primary, :visual]
  end

  def content_for_embedding(content_type = :primary)
    case content_type.to_sym
    when :primary
      [name, description, tags&.join(', ')].compact.join("\n\n")
    when :visual
      main_image&.meta_description
    end
  end
end

Finding similar content

showcase = Showcase.find(123)
showcase.find_similar(limit: 5)

Manual embedding generation

showcase.generate_embedding!(:primary, force: true)

Constant Summary collapse

MAX_CONTENT_LENGTH =

Maximum content length for embedding (roughly 30k chars, within the
Gemini text window).

30_000

Has many collapse

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.embeddable_content_types ⇒ Array<Symbol>

Override in model to define what content types are embeddable.
Common types: :primary, :visual, :transcript, :specifications.

Examples:

def self.embeddable_content_types
  [:primary, :transcript]
end

Returns:

  • (Array<Symbol>) —

    list of content types to embed.



70
71
72
 
# File 'app/concerns/models/embeddable.rb', line 70

def embeddable_content_types
  [:primary]
end

.embedding_partition_class ⇒ Class?

Returns the partition embedding class for this model. Maps model
names to their ContentEmbedding partition subclasses by convention.

Examples:

Post.embedding_partition_class # => ContentEmbedding::PostEmbedding
Image.embedding_partition_class # => ContentEmbedding::ImageEmbedding

Returns:

  • (Class, nil) —

    partition class, or nil when no
    ContentEmbedding::<Model>Embedding constant is defined.



121
122
123
124
 
# File 'app/concerns/models/embeddable.rb', line 121

def embedding_partition_class
  partition_class_name = "ContentEmbedding::#{name}Embedding"
  partition_class_name.safe_constantize
end

.regenerate_all_embeddings(batch_size: 100, scope: nil) ⇒ Integer

Batch regenerate embeddings for all records by enqueueing
EmbeddingWorker for each record in scope.

Examples:

Regenerate all

Post.regenerate_all_embeddings

Regenerate published only

Post.regenerate_all_embeddings(scope: Post.published)

Parameters:

  • batch_size (Integer) (defaults to: 100) —

    number of records to process per batch.

  • scope (ActiveRecord::Relation, nil) (defaults to: nil) —

    optional scope to filter
    records; defaults to all.

Returns:

  • (Integer) —

    count of records queued for embedding generation.



85
86
87
88
89
90
91
92
93
94
95
96
 
# File 'app/concerns/models/embeddable.rb', line 85

def regenerate_all_embeddings(batch_size: 100, scope: nil)
  records = scope || all
  count = 0

  records.find_each(batch_size: batch_size) do |record|
    EmbeddingWorker.perform_async(record.class.name, record.id)
    count += 1
  end

  Rails.logger.info "Queued #{count} #{name} records for embedding generation"
  count
end

.semantic_search(query, limit: 10) ⇒ Array<ApplicationRecord>

Semantic search within this model type, over the unified Gemini space.

Extra keyword args (e.g. +:locale+, +:exclude_sensitive+) are forwarded to
ContentEmbedding.unified_hybrid_search.

Examples:

Post.semantic_search("spa wellness tips")
Image.semantic_search("bathroom with heated floors")

Parameters:

  • query (String) —

    natural-language search query.

  • limit (Integer) (defaults to: 10) —

    maximum results to return.

Returns:



108
109
110
111
 
# File 'app/concerns/models/embeddable.rb', line 108

def semantic_search(query, limit: 10, **)
  ContentEmbedding.unified_hybrid_search(query, limit: limit, types: [name], **)
                  .map(&:embeddable)
end

.with_shape_lock(embeddable_type:, embeddable_id:, content_type:, locale:) ⇒ Object

Serialize one embedding storage-shape swap across the live and legacy
writers. Gemini calls stay outside this lock; only the final recheck and
database write are protected.

Parameters:

  • embeddable_type (String) —

    polymorphic source type

  • embeddable_id (Integer) —

    source record id

  • content_type (String) —

    canonical unified content-type prefix

  • locale (String) —

    exact embedding locale

Returns:

  • (Object) —

    value returned by the protected block



47
48
49
50
 
# File 'app/concerns/models/embeddable.rb', line 47

def self.with_shape_lock(embeddable_type:, embeddable_id:, content_type:, locale:, &)
  key = ['embedding_swap', embeddable_type, embeddable_id, content_type, locale].join('/')
  ContentEmbedding.with_advisory_lock!(key, timeout_seconds: 30, &)
end

Instance Method Details

#content_embeddings ⇒ ActiveRecord::Associations::CollectionProxy<ContentEmbedding>

Returns embedding rows for this record.

Returns:

  • (ActiveRecord::Associations::CollectionProxy<ContentEmbedding>) —

    embedding rows for this record



55
 
# File 'app/concerns/models/embeddable.rb', line 55

has_many :content_embeddings, as: :embeddable, dependent: :destroy

#content_for_embedding(_content_type = :primary) ⇒ String

Override in model to provide content for embedding. The returned
string is what gets converted to a vector embedding.

Examples:

def content_for_embedding(content_type = :primary)
  [title, description, body].compact.join("\n\n")
end

Parameters:

  • _content_type (Symbol) (defaults to: :primary) —

    type of content to embed.

Returns:

  • (String) —

    text content to embed.

Raises:

  • (NotImplementedError) 


136
137
138
 
# File 'app/concerns/models/embeddable.rb', line 136

def content_for_embedding(_content_type = :primary)
  raise NotImplementedError, "#{self.class} must implement #content_for_embedding"
end

#embeddable_locales ⇒ Array<String>

Override in model to specify all locales that should have embeddings.
Return an array if content exists in multiple languages. Defaults to
only the primary locale.

Examples:

Model with multiple translations

def embeddable_locales
  publication_locales.presence || ['en']
end

Returns:

  • (Array<String>) —

    list of locale codes.



166
167
168
 
# File 'app/concerns/models/embeddable.rb', line 166

def embeddable_locales
  [locale_for_embedding]
end

#embedding_content_hash(content_type = :primary, locale: nil) ⇒ String

Generate content hash for change detection.

When a model's content_for_embedding accepts a locale: keyword
(e.g. Post, where Liquid rendering varies per locale) the hash is
computed for that locale so stale-detection is also per-locale.
Models that do not declare a locale: keyword receive the same hash
regardless of locale, preserving their existing behaviour.

Parameters:

  • content_type (Symbol) (defaults to: :primary) —

    type of content.

  • locale (String, nil) (defaults to: nil) —

    locale to hash for; defaults to the
    model's locale_for_embedding.

Returns:

  • (String) —

    first 32 chars of SHA256 hash of the content.



182
183
184
185
 
# File 'app/concerns/models/embeddable.rb', line 182

def embedding_content_hash(content_type = :primary, locale: nil)
  content = locale_aware_content_for_embedding(content_type, locale || locale_for_embedding).to_s
  Digest::SHA256.hexdigest(content)[0..31]
end

#embedding_current?(embedding, content_type = :primary, locale: nil) ⇒ Boolean

Verify that an already-loaded single embedding row matches the record's
current source content without querying for that row again. A single row
can never prove a chunked embedding set is complete, so chunked records
deliberately return false and must use #embedding_stale?.

Parameters:

  • embedding (ContentEmbedding, nil) —

    already-loaded embedding row

  • content_type (Symbol) (defaults to: :primary) —

    logical content type

  • locale (String, nil) (defaults to: nil) —

    content locale

Returns:

  • (Boolean) —

    true only when the supplied row is current and complete



213
214
215
216
217
218
219
220
221
 
# File 'app/concerns/models/embeddable.rb', line 213

def embedding_current?(embedding, content_type = :primary, locale: nil)
  return false if embeddable_chunked?

  locale ||= locale_for_embedding
  !embedding_row_stale?(
    embedding,
    expected_hash: embedding_content_hash(content_type, locale:)
  )
end

#embedding_eligible? ⇒ Boolean

Whether this record is currently allowed to retain embeddings. Models with
lifecycle or corpus rules override this; unrestricted embeddables remain
eligible by default.

MUST stay public. It is called with an explicit receiver on a different
object — with_embedding_write_lock re-reads the row under a lock and asks
locked_record.embedding_eligible? — which Ruby forbids for a private
method even between instances of the same class. Living below private made
that raise NoMethodError: private method 'embedding_eligible?' called for
every embeddable except Article, which happened to declare its own public
override (AppSignal #3972).

Returns:

  • (Boolean) 


476
477
478
 
# File 'app/concerns/models/embeddable.rb', line 476

def embedding_eligible?
  true
end

#embedding_stale?(content_type = :primary, locale: nil) ⇒ Boolean

Check whether the embedding for content_type/locale needs
regeneration.

Parameters:

  • content_type (Symbol) (defaults to: :primary) —

    type of content.

  • locale (String, nil) (defaults to: nil) —

    locale to check; defaults to the model's
    locale_for_embedding.

Returns:

  • (Boolean) —

    true when the embedding is stale or missing.



194
195
196
197
198
199
200
201
202
 
# File 'app/concerns/models/embeddable.rb', line 194

def embedding_stale?(content_type = :primary, locale: nil)
  locale ||= locale_for_embedding
  return chunked_embedding_stale?(content_type, locale:) if embeddable_chunked?

  embedding_row_stale?(
    find_content_embedding(content_type, locale:),
    expected_hash: embedding_content_hash(content_type, locale:)
  )
end

#embedding_type_name ⇒ String

Returns the type name to use for content_embeddings. Uses the actual
class name instead of the base class for STI models, so semantic
searches can filter by type.

Returns:

  • (String) —

    type name for the polymorphic association.



308
309
310
 
# File 'app/concerns/models/embeddable.rb', line 308

def embedding_type_name
  self.class.name
end

#embedding_vector ⇒ Array<Float>?

Returns the primary embedding vector for this record.

Returns:

  • (Array<Float>, nil) —

    embedding vector, or nil when no primary
    embedding exists.



459
460
461
 
# File 'app/concerns/models/embeddable.rb', line 459

def embedding_vector
  find_content_embedding(:primary)&.unified_embedding
end

#find_content_embedding(content_type = :primary, locale: nil) ⇒ ContentEmbedding?

Find a content embedding using the correct type name for STI models.
The logical content_type is mapped to its canonical Gemini "unified"
row (see #unified_content_type), since that is where vectors now live.

Parameters:

  • content_type (Symbol) (defaults to: :primary) —

    type of content.

  • locale (String, nil) (defaults to: nil) —

    locale to find; defaults to the model's
    locale_for_embedding.

Returns:



244
245
246
247
248
249
250
251
252
253
 
# File 'app/concerns/models/embeddable.rb', line 244

def find_content_embedding(content_type = :primary, locale: nil)
  locale ||= locale_for_embedding

  ContentEmbedding.find_by(
    embeddable_type: embedding_type_name,
    embeddable_id: id,
    content_type: unified_content_type(content_type),
    locale: locale.to_s
  )
end

#find_similar(limit: 5, same_type_only: true) ⇒ Array<ApplicationRecord>

Find content similar to this record via the shared ContentEmbedding
similarity index.

Examples:

showcase.find_similar(limit: 5)
post.find_similar(same_type_only: false) # Cross-type search

Parameters:

  • limit (Integer) (defaults to: 5) —

    maximum number of results to return.

  • same_type_only (Boolean) (defaults to: true) —

    when true, restrict results to the
    same model type.

Returns:



450
451
452
453
 
# File 'app/concerns/models/embeddable.rb', line 450

def find_similar(limit: 5, same_type_only: true)
  ContentEmbedding.find_similar(self, limit: limit, same_type_only: same_type_only)
                  .map(&:embeddable)
end

#generate_all_embeddings!(force: false) ⇒ Array<ContentEmbedding>

Generate embeddings for all content types declared by
embeddable_content_types.

Parameters:

  • force (Boolean) (defaults to: false) —

    regenerate even if existing embeddings are not
    stale.

Returns:

  • (Array<ContentEmbedding>) —

    created or updated embeddings (one
    per content type, blank entries skipped).



433
434
435
436
437
 
# File 'app/concerns/models/embeddable.rb', line 433

def generate_all_embeddings!(force: false)
  self.class.embeddable_content_types.flat_map do |content_type|
    Array.wrap(generate_embedding!(content_type, force: force))
  end
end

#generate_chunked_embeddings!(content_type = :primary, force: false, locale: nil) ⇒ Array<ContentEmbedding>

Generate chunked embeddings for long content. Splits content into
overlapping chunks and creates an embedding for each. Use this for
documents that exceed the token limit (e.g. long articles, PDFs).

Examples:

Embed a long document in chunks

article.generate_chunked_embeddings!(:primary)

Parameters:

  • content_type (Symbol) (defaults to: :primary) —

    type of content to embed.

  • force (Boolean) (defaults to: false) —

    regenerate even if the existing embedding is
    not stale.

  • locale (String, nil) (defaults to: nil) —

    locale for the content; defaults to the
    model's locale_for_embedding.

Returns:



324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
 
# File 'app/concerns/models/embeddable.rb', line 324

def generate_chunked_embeddings!(content_type = :primary, force: false, locale: nil)
  unless embedding_eligible?
    content_embeddings.delete_all
    return []
  end

  locale ||= locale_for_embedding
  locale_str = locale.to_s

  return [] unless force || embedding_stale?(content_type, locale: locale)

  content = locale_aware_content_for_embedding(content_type, locale_str)
  if content.blank?
    delete_embedding_shape_if_still_blank(content_type, locale: locale_str)
    return []
  end

  chunker      = Embedding::ContentChunker.new(content)
  chunk_prefix = unified_content_type(content_type)

  # A record must be represented by EITHER a single `primary` row OR
  # `primary_chunk_*` rows — never both, or retrieval would carry a stale
  # duplicate vector for it. In BOTH branches the stale shape is purged only
  # AFTER the replacement embeddings succeed, so a transient Gemini failure
  # can never delete a record's vectors and leave it unsearchable.
  unless chunker.needs_chunking?
    vector = generate_embedding_vector!(content, content_type:)

    return with_embedding_write_lock(content_type, locale: locale_str) do
      # A concurrent worker may have repaired this slot while Gemini ran.
      next [] unless force || embedding_stale?(content_type, locale:)

      row = persist_single_embedding!(
        content_type,
        locale: locale_str,
        vector:,
        content_hash: Digest::SHA256.hexdigest(content.to_s)[0..31]
      )
      # Only once the single row lands do we drop any prior chunk rows from a
      # longer previous version.
      if row
        ContentEmbedding.where(
          embeddable_type: embedding_type_name, embeddable_id: id, locale: locale_str
        ).where('content_type LIKE ?', "#{chunk_prefix}_chunk_%").delete_all
      end
      [row].compact
    end || []
  end

  # Long content: embed FIRST (batched Gemini call). If it fails, leave the
  # existing vectors untouched and let EmbeddingWorker's rescue retry.
  chunks  = chunker.chunks
  vectors = Embedding::Gemini.embed_texts(chunks, dimensions: ContentEmbedding::UNIFIED_DIMENSIONS)
  fresh   = chunks.each_index.select { |i| vectors[i].present? }
  if fresh.empty?
    Rails.logger.warn "Chunk embedding produced no vectors for #{self.class}##{id}; leaving existing rows intact"
    return []
  end

  # Atomic swap: drop the stale shape (plain primary + prior chunk rows) and
  # write the new chunk rows in one transaction. The shared advisory lock
  # serializes live and legacy writers for the same slot; without it two jobs
  # can interleave delete+insert and the loser hits the unique index
  # (AppSignal #3591). Timeout raises so Sidekiq retries the whole job.
  embeddings = with_embedding_write_lock(content_type, locale: locale_str) do
    # A concurrent worker may have repaired this slot while Gemini ran.
    next [] unless force || embedding_stale?(content_type, locale:)

    ContentEmbedding.transaction do
      ContentEmbedding.where(
        embeddable_type: embedding_type_name, embeddable_id: id, locale: locale_str
      ).where('content_type = ? OR content_type LIKE ?', chunk_prefix, "#{chunk_prefix}_chunk_%").delete_all

      fresh.map do |index|
        ContentEmbedding.create!(
          embeddable_type: embedding_type_name,
          embeddable_id: id,
          content_type: "#{chunk_prefix}_chunk_#{index}",
          locale: locale_str,
          unified_embedding: vectors[index],
          embedding_model: ContentEmbedding::UNIFIED_MODEL,
          embedding_dimensions: ContentEmbedding::UNIFIED_DIMENSIONS,
          content_hash: Digest::SHA256.hexdigest(chunks[index])[0..31]
        )
      end
    end
  end || []

  Rails.logger.info "Generated #{embeddings.size} chunk embeddings for #{self.class}##{id} (locale: #{locale_str})"
  embeddings
end

#generate_embedding!(content_type = :primary, force: false, locale: nil) ⇒ ContentEmbedding, ...

Generate or update an embedding for this record.

Chunk-enabled records are automatically routed through
#generate_chunked_embeddings! so this entry point cannot create a
conflicting single row.

Examples:

post.generate_embedding!(:primary)
video.generate_embedding!(:transcript, force: true)
site_map.generate_embedding!(:primary, locale: 'fr')

Parameters:

  • content_type (Symbol) (defaults to: :primary) —

    type of content to embed.

  • force (Boolean) (defaults to: false) —

    regenerate even if the existing embedding is
    not stale.

  • locale (String, nil) (defaults to: nil) —

    locale for the content; defaults to the
    model's locale_for_embedding.

Returns:



273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
 
# File 'app/concerns/models/embeddable.rb', line 273

def generate_embedding!(content_type = :primary, force: false, locale: nil)
  unless embedding_eligible?
    content_embeddings.delete_all
    return
  end

  return generate_chunked_embeddings!(content_type, force:, locale:) if embeddable_chunked?

  locale ||= locale_for_embedding
  locale_str = locale.to_s

  return unless force || embedding_stale?(content_type, locale: locale)

  content = locale_aware_content_for_embedding(content_type, locale_str)
  if content.blank?
    delete_embedding_shape_if_still_blank(content_type, locale: locale_str)
    return
  end

  vector = generate_embedding_vector!(content, content_type:)
  with_embedding_write_lock(content_type, locale: locale_str) do
    persist_single_embedding!(
      content_type,
      locale: locale_str,
      vector:,
      content_hash: Digest::SHA256.hexdigest(content.to_s)[0..31]
    )
  end
end

#has_embedding?(content_type = :primary, locale: nil) ⇒ Boolean

Check whether this record has an embedding for the given content type
and locale.

Parameters:

  • content_type (Symbol) (defaults to: :primary) —

    type of content.

  • locale (String, nil) (defaults to: nil) —

    locale to check; defaults to the model's
    locale_for_embedding.

Returns:

  • (Boolean) —

    true when an embedding exists.



230
231
232
233
234
 
# File 'app/concerns/models/embeddable.rb', line 230

def has_embedding?(content_type = :primary, locale: nil)
  locale ||= locale_for_embedding
  find_content_embedding(content_type, locale:).present? ||
    find_chunk_embeddings(content_type, locale:).exists?
end

#locale_for_embedding ⇒ String

Override in model to specify the locale for embedding content. This
determines which locale's content is embedded and enables
locale-filtered searches.

Examples:

SiteMap with locale column

def locale_for_embedding
  locale.to_s.split('-').first # 'en-US' -> 'en'
end

Item with publication_locales array

def locale_for_embedding
  publication_locales&.first || 'en'
end

Returns:

  • (String) —

    locale code (e.g. 'en', 'fr', 'en-US').



153
154
155
 
# File 'app/concerns/models/embeddable.rb', line 153

def locale_for_embedding
  'en' # Default to English
end

#needs_chunking?(content_type = :primary) ⇒ Boolean

Check whether content for content_type would need chunking before
embedding (i.e. exceeds the token limit).

Parameters:

  • content_type (Symbol) (defaults to: :primary) —

    type of content to evaluate.

Returns:

  • (Boolean) —

    true when content exceeds the token limit.



421
422
423
424
 
# File 'app/concerns/models/embeddable.rb', line 421

def needs_chunking?(content_type = :primary)
  content = locale_aware_content_for_embedding(content_type, locale_for_embedding).to_s
  Embedding::ContentChunker.new(content).needs_chunking?
end