Documentation
¶
Overview ¶
Package docconv converts common document formats (PDF, DOCX, XLSX, PPTX, EPUB, MOBI, HTML, plain text) into clean, LLM-ready markdown.
Backend selection:
- PDF, EPUB, MOBI use MuPDF via go-fitz (CGO).
- DOCX uses fumiama/go-docx for structured parsing; go-fitz acts as a fallback when the native parse fails.
- PPTX uses a hand-rolled stdlib-xml walker over ppt/slides/slide*.xml for structured parsing; go-fitz acts as a fallback.
- XLSX uses excelize.
- HTML goes through go-readability plus html-to-markdown.
Under the "nofitz" build tag, DOCX/PPTX/XLSX/HTML/text all still work; PDF/EPUB/MOBI return ErrFitzRequired.
The simplest entry point is Extract:
md, err := docconv.Extract("report.pdf", nil)
Callers that want image description in the output can plug in an ImageDescriber (typically a vision LLM adapter). With OCRFallback set, the same interface is also used to transcribe pages of scanned documents that have no extractable text.
Index ¶
- Constants
- Variables
- func Extract(path string, opts *Options) (string, error)
- func ExtractReader(r io.Reader, format Format, opts *Options) (string, error)
- func ExtractReaderContext(ctx context.Context, r io.Reader, format Format, opts *Options) (string, error)
- func Supports(mime string) bool
- type Format
- type ImageDescriber
- type Metadata
- type Options
Constants ¶
const ( // DefaultDescribePrompt is a flat prompt preserved for back-compat with // callers and CLI wrappers that want a complete prompt string rather // than the context-aware template. New callers should prefer leaving // Options.DescribePrompt empty so the library applies // DefaultDescribePromptTemplate with surrounding document context. DefaultDescribePrompt = "Describe this image for a reader who cannot see it. Keep it under two sentences." // DefaultOCRPrompt is used when OCRPrompt is empty. DefaultOCRPrompt = "Transcribe all text in this image verbatim. Preserve headings, lists, and tables as markdown. Do not add commentary." // DefaultOCRDPI is the page-render resolution used for OCR fallback. DefaultOCRDPI = 200.0 // DecorativeMarker is the sentinel an ImageDescriber may return to // signal that an image carries no informational content (logo, rule, // ornament). docconv drops such images from output rather than // emitting noisy placeholder alt-text. // // Honoured only when DescribePrompt is empty (library-owned prompt // path); caller-supplied prompts bypass DECORATIVE handling because // the caller's prompt may use the word for other purposes. DecorativeMarker = "DECORATIVE" )
Default prompts and constants used by the library when the corresponding Options fields are zero-valued. Exported so CLI wrappers and tests can reference them.
const DefaultDescribePromptTemplate = `` /* 1062-byte string literal not displayed */
DefaultDescribePromptTemplate is the prompt docconv applies when Options.DescribePrompt is empty. Apply with fmt.Sprintf and exactly two %s substitutions in order: surrounding document text (may be empty), and author-supplied alt-text (may be empty).
Designed to be register-agnostic: surrounding text carries the tone (technical datasheet vs. narrative prose), so the prompt itself does not attempt to classify document type.
Variables ¶
var ( // ErrUnsupportedFormat is returned when the input format cannot be // handled by this build (for example, an image passed to Extract). ErrUnsupportedFormat = errors.New("unsupported document format") // ErrNoText is returned when a document has no extractable text and // OCRFallback is disabled. The document may be a scan or a document // with only embedded images. ErrNoText = errors.New("no extractable text (may be scanned/image)") // ErrCorruptDocument is returned when the document cannot be parsed. ErrCorruptDocument = errors.New("document is corrupt or unreadable") // ErrPasswordProtected is returned when the document requires a // password that has not been supplied. ErrPasswordProtected = errors.New("document is password protected") // ErrFitzRequired is returned by the "nofitz" build when a caller // requests a format that requires the go-fitz / MuPDF backend. ErrFitzRequired = errors.New("go-fitz required for this format (rebuild without -tags nofitz)") )
Typed errors returned by the package. Callers can route behaviour by comparing with errors.Is.
Functions ¶
func Extract ¶
Extract returns markdown-formatted text from any supported document on disk. A nil *Options is treated as the zero value.
func ExtractReader ¶
ExtractReader extracts from an io.Reader with an explicit format hint. Pass FormatAuto to request magic-byte detection when reading from a stream where the file name is unavailable.
Types ¶
type Format ¶
type Format int
Format represents a document type.
const ( // FormatAuto asks the library to detect the format from extension or // magic bytes. Never returned by Detect. FormatAuto Format = iota // FormatPDF is an Adobe PDF document. FormatPDF // FormatDOCX is a Microsoft Word OOXML document. FormatDOCX // FormatXLSX is a Microsoft Excel OOXML spreadsheet. FormatXLSX // FormatPPTX is a Microsoft PowerPoint OOXML presentation. FormatPPTX // FormatEPUB is an EPUB ebook. FormatEPUB // FormatMOBI is a Mobipocket ebook. FormatMOBI // FormatHTML is an HTML document. FormatHTML // FormatText is plain text. FormatText // FormatImage is a standalone image. Extract returns // ErrUnsupportedFormat; callers should feed images to a vision // model directly rather than through this package. FormatImage )
func Detect ¶
Detect returns the format of a file based on extension and magic bytes. Magic-byte sniffing takes precedence over the extension when both are available. Returns ErrUnsupportedFormat when neither identifies a known format.
func DetectReader ¶
DetectReader returns the format from the first few KiB of a reader. The reader is wrapped with bufio.Reader and the peek is returned via the new reader, so callers should use the returned reader for subsequent work.
func FromMIME ¶
FromMIME returns the Format matching a MIME type, or FormatAuto if the MIME is not one docconv can extract. Parameters (anything after ';') are ignored so "text/plain; charset=utf-8" resolves the same as "text/plain". Matching is case-insensitive. Never panics on malformed input.
Images (image/*) are intentionally unmapped: callers should send images directly to a vision model rather than through Extract, which returns ErrUnsupportedFormat for image inputs.
type ImageDescriber ¶
type ImageDescriber interface {
// DescribeImage returns a text description of an image.
//
// img is raw encoded image bytes (PNG, JPEG, ...). mimeType is the image
// MIME type, useful for APIs that require explicit typing. prompt is the
// caller-chosen (or library-default) instruction sent to the vision
// model.
DescribeImage(ctx context.Context, img []byte, mimeType string, prompt string) (string, error)
}
ImageDescriber describes images for LLM context. Consumers implement this with whatever vision model or chain they choose; docconv never imports any specific LLM SDK.
The same interface serves two roles:
- Inline image description when IncludeImages is true.
- Page-level transcription (OCR) when OCRFallback is true and a document has no extractable text.
The library supplies a distinct prompt for each role and the implementation may branch on that prompt if it wants different behaviour, or simply pass the prompt through to its vision model of choice.
type Metadata ¶
type Metadata struct {
Title string
Author string
Subject string
Keywords string
Created string
Modified string
Pages int
Format string
Extra map[string]string
}
Metadata is what backends report alongside the extracted markdown. Fields not populated by the backend are left blank and suppressed in the rendered YAML front-matter.
type Options ¶
type Options struct {
// LLMClient, when non-nil, is called to describe embedded images
// (see IncludeImages) and to transcribe pages when OCRFallback is set.
LLMClient ImageDescriber
// IncludeImages controls whether embedded images are represented in
// the markdown output:
//
// - true + LLMClient != nil: images are described inline via
// DescribePrompt.
// - true + LLMClient == nil: images are referenced with placeholder
// alt-text.
// - false: images are stripped entirely.
IncludeImages bool
// ImageDir is an optional directory where extracted images are
// written. When empty, images live only in memory.
ImageDir string
// IncludeMetadata prepends a YAML front-matter block with document
// metadata (title, author, page count, created-at) where the source
// document exposes it.
IncludeMetadata bool
// OCRFallback enables LLM-based transcription of documents that have
// no extractable text. Requires LLMClient != nil. Default false; the
// library returns ErrNoText in that case.
//
// When enabled, the library renders each page via go-fitz and feeds
// the image to LLMClient.DescribeImage with OCRPrompt. This is slow
// and costs API credits, so it is strictly opt-in.
OCRFallback bool
// OCRDPI is the render resolution used for OCR fallback pages. Zero
// means use the default (200).
OCRDPI float64
// DescribePrompt overrides the default prompt used for embedded-image
// description.
//
// When empty (the default), the library applies DefaultDescribePromptTemplate
// with the surrounding document text and any author-supplied alt-text
// spliced in; it also honours the DecorativeMarker sentinel by
// suppressing images the describer flags as DECORATIVE.
//
// When set to a non-empty string, the caller's prompt is used verbatim
// for every image — surrounding document text and author alt-text are
// NOT injected, and DecorativeMarker handling is disabled. Callers who
// want to own prompt semantics take the whole prompt.
DescribePrompt string
// OCRPrompt overrides the default prompt used for OCR fallback pages.
// Empty means use the library default.
OCRPrompt string
// IncludeComments, when true, surfaces reviewer comments (currently
// DOCX word/comments.xml) inline as HTML comments at each anchor
// position: "<!-- comment by AUTHOR (DATE): TEXT -->". HTML comments
// survive markdown conversion untouched — invisible in rendered
// markdown, present in the byte stream that LLMs tokenise.
//
// Off by default because reviewer chatter degrades the "what does this
// document say" signal LLM-agent consumers expect (unresolved
// disputes can be treated as document claims). Turn on for audit,
// legal review, or version-archaeology workflows where reviewer
// context is itself the content of interest.
IncludeComments bool
}
Options configures extraction behaviour. A nil *Options is treated as the zero value: no image extraction, no metadata, no OCR fallback.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
docx
Package docx walks the fumiama/go-docx AST and emits semantic HTML.
|
Package docx walks the fumiama/go-docx AST and emits semantic HTML. |
|
mdconv
Package mdconv wraps html-to-markdown/v2 with the option set this project prefers.
|
Package mdconv wraps html-to-markdown/v2 with the option set this project prefers. |
|
ooxml
Package ooxml extracts embedded media from OOXML files (DOCX, XLSX, PPTX).
|
Package ooxml extracts embedded media from OOXML files (DOCX, XLSX, PPTX). |
|
pptx
Package pptx walks a single PPTX slide XML and emits semantic HTML.
|
Package pptx walks a single PPTX slide XML and emits semantic HTML. |