Class: UploadMarkdownExtractionWorker

Inherits:
Object
  • Object
show all
Includes:
Sidekiq::Job
Defined in:
app/workers/upload_markdown_extraction_worker.rb

Overview

Extracts an Upload's attachment to Markdown via the docling-serve service
and caches it on +uploads.extracted_markdown+ (see Upload#extract_markdown!).

Enqueued by Publication::MarkdownExtractionHandler whenever a publication's
PDF is linked or replaced, and directly by anything that needs a document
readable unattended (retailer-onboarding workbooks, datasheets):

When the upload is a publication's literature, this worker is the async DRIVER
of that publication's searchable text: it mirrors the extraction into
+items.search_text+ (the lexical-search half of hybrid retrieval + the
embeddability gate) and regenerates the embedding, so the verbatim document
text + spec tables join the vision descriptions in the vector. This replaced
the former synchronous PDF::Reader path.

Examples:

Extract a workbook for a retailer-onboarding flow

UploadMarkdownExtractionWorker.perform_async(upload.id)
# later: upload.reload.extracted_markdown

Constant Summary collapse

DEFAULT_POPULATION_LIMIT =

Max population limit.

25
MAX_POPULATION_LIMIT =

Registered sources.

500
REGISTERED_SOURCES =

Constant.

{
  'technical_support_replay_evidence' => lambda {
    Assistant::TechnicalSupport::CaseEvidenceUploadQuery.new.call
  }
}.freeze

Instance Method Summary collapse

Instance Method Details

#perform(upload_id, options = {}) ⇒ Object

Parameters:

  • upload_id (Integer, nil) —

    the Upload to extract, or nil to populate a registered source

  • options (Hash) (defaults to: {}) —

    extraction options

Options Hash (options):

  • :force (Boolean) —

    re-extract even when a cached extraction exists

  • :source_key (String) —

    registered missing-extraction scope when upload_id is nil

  • :limit (Integer) —

    maximum source records to enqueue



41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
 
# File 'app/workers/upload_markdown_extraction_worker.rb', line 41

def perform(upload_id, options = {})
  return enqueue_missing_extractions(options) if upload_id.nil?

  upload = Upload.find_by(id: upload_id)
  return Rails.logger.info("[UploadMarkdownExtractionWorker] Upload #{upload_id} not found, skipping") unless upload
  return Rails.logger.info("[UploadMarkdownExtractionWorker] Upload #{upload_id} has no stored attachment, skipping") unless upload.attachment_stored?
  return Rails.logger.info("[UploadMarkdownExtractionWorker] Upload #{upload_id} (#{upload.attachment_name}) is not an extractable format, skipping") unless DoclingClient.supported_file?(upload.attachment_name)
  return Rails.logger.info("[UploadMarkdownExtractionWorker] Upload #{upload_id} backs only discontinued publications, skipping") if only_discontinued_publications?(upload)

  begin
    markdown = upload.extract_markdown!(force: options[:force].to_b)
  rescue DoclingClient::Timeout => e
    # A doc that can't convert within docling's 300s sync ceiling is permanent:
    # retrying re-wedges the single CPU accessory for another 300s and still
    # fails. Record it as a non-paging warning keyed only by the upload ID
    # and cache an empty terminal result so the nightly missing-extraction
    # sweep does not wedge on the same file forever. A manual force pass can
    # retry it later; an existing cached extraction is never overwritten.
    # Returning here means no re-raise → no Sidekiq retry storm.
    # ponytail: soft-skip beats a 4x retry; async task API is the fix if this
    # gets common.
    ErrorReporting.warning(e, source: :background, upload_id: upload_id)
    Upload.where(id: upload_id, extracted_markdown: nil).update_all(extracted_markdown: '')
    return
  end
  Rails.logger.info("[UploadMarkdownExtractionWorker] Upload #{upload_id} extracted: #{markdown.length} chars")

  sync_publication_search_text!(upload, markdown)
end