Documentation
¶
Index ¶
Constants ¶
const (
ImageDir = "images" // Subdirectory name within siteOutputDir for images
)
Variables ¶
This section is empty.
Functions ¶
func ExtractHeadings ¶
ExtractHeadings returns all ATX heading texts in markdown in document order. Does not parse Setext-style headings or detect headings inside fenced code blocks — acceptable for the inputs we feed it (HTML-to-markdown conversion output, where Setext headings do not occur).
Types ¶
type Chunk ¶
type Chunk struct {
Content string // The chunk content (includes heading context when HeadingHierarchy is enabled)
HeadingHierarchy []string // Extracted heading hierarchy from the chunk
}
Chunk represents a single chunk of content with its metadata.
func ChunkMarkdown ¶
func ChunkMarkdown(markdown string, cfg ChunkerConfig) ([]Chunk, error)
ChunkMarkdown splits markdown content into chunks using a hybrid strategy:
- Primary: split at ATX markdown headers so each chunk carries an identifiable heading context.
- Fallback for any section that exceeds MaxChunkSize tokens: recursive separator-based packing (paragraph, line, sentence, word, then rune) with ChunkOverlap tokens of overlap between consecutive sub-chunks.
Token counts here are estimated via a "1 token per 4 characters" heuristic. This is intentionally coarse: real tokenizers vary by model, and the only publicly available choices (cl100k_base / o200k_base) are GPT-family encodings that misrepresent Claude and most non-OpenAI consumers. If you need precise counts for a specific model, tokenize the output yourself with that model's tokenizer.
type ChunkerConfig ¶
type ChunkerConfig struct {
MaxChunkSize int // Maximum chunk size in tokens (triggers recursive split if exceeded)
ChunkOverlap int // Overlap between chunks in tokens (for recursive fallback)
}
ChunkerConfig holds configuration for the chunker.
func DefaultChunkerConfig ¶
func DefaultChunkerConfig() ChunkerConfig
DefaultChunkerConfig returns sensible defaults for RAG chunking.
type ContentProcessor ¶
type ContentProcessor struct {
// contains filtered or unexported fields
}
ContentProcessor handles extracting, cleaning, processing (images, links), converting to Markdown, and saving of page content
func NewContentProcessor ¶
func NewContentProcessor(imgProcessor *ImageProcessor, appCfg *config.AppConfig, log *logrus.Entry) *ContentProcessor
NewContentProcessor creates a ContentProcessor
func (*ContentProcessor) ExtractProcessAndSaveContent ¶
func (cp *ContentProcessor) ExtractProcessAndSaveContent( doc *goquery.Document, finalURL *url.URL, siteCfg *config.SiteConfig, siteOutputDir string, taskLog *logrus.Entry, ctx context.Context, ) (pageTitle string, savedFilePath string, markdownBytes []byte, imageCount int, err error)
ExtractProcessAndSaveContent extracts content using the configured selector, processes images and internal links, converts to Markdown, and saves to a path derived from finalURL and siteOutputDir. Returns the extracted page title and any critical error encountered during processing or saving.
type ImageDownloadTask ¶
type ImageDownloadTask struct {
AbsImgURL string
NormImgURL string
BaseImgURL *url.URL // Parsed absolute URL
ImgHost string
ExtractedCaption string
ImgLogEntry *logrus.Entry // Logger with image-specific context
Ctx context.Context // Context for this specific task
}
ImageDownloadTask holds information needed for an image worker to process one image
type ImageProcessor ¶
type ImageProcessor struct {
// contains filtered or unexported fields
}
ImageProcessor handles the orchestration of image downloading and processing
func NewImageProcessor ¶
func NewImageProcessor( store storage.ImageStore, fetcher fetch.HTTPFetcher, robotsHandler *fetch.RobotsHandler, rateLimiter *fetch.RateLimiter, globalSemaphore *semaphore.Weighted, hostSemPool *fetch.HostSemaphorePool, resolved *config.ResolvedSiteConfig, appCfg *config.AppConfig, log *logrus.Entry, ) *ImageProcessor
NewImageProcessor creates a new ImageProcessor
func (*ImageProcessor) ProcessImages ¶
func (ip *ImageProcessor) ProcessImages( mainContent *goquery.Selection, finalURL *url.URL, siteCfg *config.SiteConfig, siteOutputDir string, taskLog *logrus.Entry, ctx context.Context, ) (imageMap map[string]models.ImageData, imageErrs []error)
ProcessImages finds images within the main content, checks status, dispatches downloads to a worker pool, and returns a map of successfully processed images and any errors It modifies the 'data-crawl-status' attribute on img tags in the selection
type LinkProcessor ¶
type LinkProcessor struct {
// contains filtered or unexported fields
}
LinkProcessor handles extracting and queueing links found on a page
func NewLinkProcessor ¶
func NewLinkProcessor( store storage.PageStore, pq *queue.ThreadSafePriorityQueue, compiledDisallowedPatterns []*regexp.Regexp, log *logrus.Entry, ) *LinkProcessor
NewLinkProcessor creates a LinkProcessor
func (*LinkProcessor) ExtractAndQueueLinks ¶
func (lp *LinkProcessor) ExtractAndQueueLinks( originalDoc *goquery.Document, finalURL *url.URL, currentDepth int, siteCfg *config.SiteConfig, wg *sync.WaitGroup, taskLog *logrus.Entry, ) (queuedCount int, err error)
ExtractAndQueueLinks finds crawlable links within the specified selectors of a document, filters them based on scope and rules, and adds new ones to the priority queue It takes the *original* document to ensure all potential links are considered, before the content might be modified by Markdown conversion etc
Source Files