Class: Heatwave::Crawler::Extract::PlainText

Inherits:

Object

• Object
• Heatwave::Crawler::Extract::PlainText

Defined in:

app/services/heatwave/crawler/extract/plain_text.rb

Overview

Naive whole-document text extraction: strips non-content landmarks and
collapses whitespace. This is the pre-crawler html_to_text behaviour,
kept as the density fallback when readability extraction comes back thin
(small pages, spec sheets, hostile markup readability scores poorly).

Constant Summary

REMOVE_SELECTOR =

'script, style, nav, footer, header, aside, noscript, ' \
'[role="navigation"], [role="banner"], [role="complementary"]'

Class Method Summary

• .call ⇒ String

  (Explicit method rather than delegate :call, to: :new — YARD's DSL handler crashes on that delegate form during the docs build.).

Instance Method Summary

• #call(html) ⇒ String

  Whitespace-normalised plain text.

Class Method Details

.call ⇒ String

(Explicit method rather than delegate :call, to: :new — YARD's DSL
handler crashes on that delegate form during the docs build.)

Forwards the raw HTML string to #call.

Returns:

• (String) —

  whitespace-normalised plain text

# File 'app/services/heatwave/crawler/extract/plain_text.rb', line 17

def call(...) = new.call(...)

Instance Method Details

#call(html) ⇒ String

Returns whitespace-normalised plain text.

Parameters:

• html (String) —

  raw HTML

Returns:

• (String) —

  whitespace-normalised plain text

# File 'app/services/heatwave/crawler/extract/plain_text.rb', line 22

def call(html)
  doc = Nokogiri::HTML(html)
  doc.search(REMOVE_SELECTOR).remove
  doc.text
     .gsub(/[ \t]+/, ' ')
     .gsub(/\n{3,}/, "\n\n")
     .strip
end