Class: Heatwave::Crawler::Extract::MainContent

Inherits:

Object

• Object
• Heatwave::Crawler::Extract::MainContent

Defined in:

app/services/heatwave/crawler/extract/main_content.rb

Overview

Main-content extraction: runs arc90 readability (via ruby-readability) to
isolate the page's primary content block, then converts it to GitHub-
flavoured Markdown (via reverse_markdown) so headings, lists, links and
tables survive as structure an LLM can reason over.

Why this exists: the plain-text extractor dumps every text node that
survives a tag-based strip, so boilerplate (cookie banners, related-links
blocks, comment threads, non-semantic sidebars) competes with the real
content for the caller's character budget. Readability scores content
blocks and keeps the main one — the same 60K budget then covers actual
content (documented incident: a 198K-char video-transcript page truncated
mid-transcript).

Falls back to PlainText when readability
output is suspiciously short (readability occasionally picks the wrong
block — comment sections — or strips wanted content such as spec tables)
or raises on hostile markup.

Constant Summary

MIN_MARKDOWN_CHARS =

Below this many characters, readability output is treated as a failed
extraction and the plain-text fallback is used instead.

200

READABILITY_OPTIONS =

ruby-readability sanitises .content against a tag whitelist that
defaults to %w[div p] — everything else is flattened to text. Without an
explicit whitelist, headings/lists/tables would be destroyed before
Markdown conversion, so keep every structural tag GFM can render.

{
  tags: %w[
    div p h1 h2 h3 h4 h5 h6 ul ol li dl dt dd blockquote pre code
    table thead tbody tfoot tr td th a img figure figcaption
    strong em b i sup sub hr span section article
  ].freeze,
  attributes: %w[href src alt title].freeze,
  remove_empty_nodes: true
}.freeze

Class Method Summary

• .call ⇒ String

  (Explicit method rather than delegate :call, to: :new — YARD's DSL handler crashes on that delegate form during the docs build.).

Instance Method Summary

• #call(html) ⇒ String

  Markdown main content, or plain-text fallback.

Class Method Details

.call ⇒ String

(Explicit method rather than delegate :call, to: :new — YARD's DSL
handler crashes on that delegate form during the docs build.)

Forwards the raw HTML string (UTF-8) to #call.

Returns:

• (String) —

  Markdown main content, or plain-text fallback

# File 'app/services/heatwave/crawler/extract/main_content.rb', line 48

def call(...) = new.call(...)

Instance Method Details

#call(html) ⇒ String

Returns Markdown main content, or plain-text fallback.

Parameters:

• html (String) —

  raw HTML (UTF-8)

Returns:

• (String) —

  Markdown main content, or plain-text fallback

# File 'app/services/heatwave/crawler/extract/main_content.rb', line 53

def call(html)
  markdown = readable_markdown(html)
  return markdown if markdown.present? && markdown.length >= MIN_MARKDOWN_CHARS

  Heatwave::Crawler::Extract::PlainText.call(html)
rescue StandardError => e
  Rails.logger.debug { "[Crawler::Extract::MainContent] readability failed (#{e.class}: #{e.message}); falling back to plain text" }
  Heatwave::Crawler::Extract::PlainText.call(html)
end