Documentation Chunking and Summarization

Overview

This pipeline stage is responsible for chunking documentation pages and generating summary chunks for downstream consumption, such as embedding, retrieval, and frontend display. It operates on rendered documentation files (typically .mdx), extracting concise, structured summaries per page. These summaries are used to power search, chat, and context-aware features in DeepDoc.

• Inputs: Rendered documentation files, site generation plan (DocPlan), configuration dictionary, and optional slug filters.
• Outputs: List of ChunkRecord objects, each representing a summary chunk with metadata (title, path, URL, trust score, etc.).

This stage is a critical part of the Pipeline & Generation Engine, bridging the gap between raw documentation and downstream embedding/indexing.

Input / Output Schema

Input

Output

Each summary chunk is a ChunkRecord (deepdoc/chatbot/types.py), with fields:

Summary chunk text format:

Document: {title}
Path: {relative_path}
URL: {url}

{snippet}

Processing Logic

High-Level Data Flow

Detailed Algorithm

Function Mapping

• Page Filtering: build_doc_summary_chunks() (deepdoc/chatbot/docs_summary.py:15)
• Path Resolution: _doc_path_for_page() (deepdoc/chatbot/docs_summary.py:105)
• File Reading: Standard Path.read_text()
• Frontmatter Stripping: _strip_frontmatter() (deepdoc/chatbot/docs_summary.py:95)
• Summary Extraction: _extract_summary() (deepdoc/chatbot/docs_summary.py:70)
  • Removes code blocks (regex), HTML tags, and splits by section.
  • Extracts title from first # heading.
  • Truncates each section to max_doc_summary_chars.
• URL Resolution: _doc_url() (deepdoc/chatbot/docs_summary.py:112)
• Chunk Assembly: Inline in build_doc_summary_chunks()
• Hashing: hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
• ChunkRecord Construction: ChunkRecord() (deepdoc/chatbot/types.py)

Edge Cases & Business Logic

• Slug Filtering: If slugs is provided, only pages matching those slugs are processed.
• Missing Files: If the resolved doc file does not exist, the page is skipped.
• Frontmatter: If frontmatter is malformed, _strip_frontmatter() falls back to original content.
• Section Splitting: If no ## sections are found, the entire cleaned text is used as a single snippet.
• Title Extraction: If no # heading is present, falls back to page.title.
• Chunk Limits: Only up to max_doc_summary_chunks_per_page chunks per page, each truncated to max_doc_summary_chars.

Error Handling & Recovery

• Missing Files: Pages with missing .mdx files are silently skipped. No chunk is produced.
• Malformed Frontmatter: _strip_frontmatter() handles ValueError gracefully, returning original content.
• Encoding Errors: Files are read with errors="replace", so unreadable bytes are replaced, not fatal.
• Empty Content: If after cleaning, content is empty, chunk falls back to page title as snippet.
• Regex Failures: Regex operations are robust; if code blocks or HTML tags are not found, content is left unchanged.

Configuration

All chunking parameters are loaded via get_chatbot_cfg() (deepdoc/chatbot/settings.py):

Example config:

chatbot_cfg = get_chatbot_cfg(cfg)
max_chunks = chatbot_cfg["chunking"]["max_doc_summary_chunks_per_page"]
max_chars = chatbot_cfg["chunking"]["max_doc_summary_chars"]

Monitoring & Observability

• Metrics: No explicit metrics are emitted by this stage.
• Logging: No logging is performed for skipped files or errors.
• Debugging: To debug chunking failures:
  • Check existence of .mdx files in output_dir.
  • Validate config values for chunk limits.
  • Inspect chunk hashes for determinism.
• Chunk Hashing: Each chunk is hashed for reproducibility and deduplication.

Chunking Algorithm Diagram

See Also

Usage Patterns

• Batch Chunking: Run build_doc_summary_chunks() after docs are rendered, before embedding/indexing.
• Selective Chunking: Use slugs to chunk only specific pages (e.g., for incremental builds).
• Config Tuning: Adjust max_doc_summary_chunks_per_page and max_doc_summary_chars for optimal chunk size.
• Downstream Consumption: Summary chunks are fed into embedding APIs, search indexes, and chat context providers.

from deepdoc.chatbot.docs_summary import build_doc_summary_chunks
chunks = build_doc_summary_chunks(
    output_dir=Path("site_output"),
    plan=doc_plan,
    cfg=chatbot_cfg,
    has_openapi=True,
    slugs=["architecture-overview", "setup-getting-started"],
)

deepdoc generate --chunk-summaries --output-dir site_output