Pipeline & Generation Engine
This page documents the Pipeline & Generation Engine for DeepDoc, covering the document generation pipeline: how evidence is assembled, how documentation plans are created, and how final output is generated and validated. This is the core engine that transforms repository source code and metadata into developer-focused documentation.
Overview
The Pipeline & Generation Engine is responsible for the third phase of DeepDoc's documentation workflow. It takes the output of the planning phase (a set of documentation "buckets" and their assigned files) and produces validated, richly cross-linked documentation pages. This stage is critical for ensuring that documentation is accurate, grounded in source evidence, and meets all structural and business requirements.
Inputs:
- A
DocPlanobject (deepdoc/planner_v2.py) containing a list ofDocBuckets, each representing a documentation page with assigned files, symbols, and metadata. - A
RepoScanobject (deepdoc/planner_v2.py) with parsed source code, file summaries, endpoint data, and integration signals. - Configuration dict (
cfg) controlling thresholds, budgets, and feature flags.
Outputs:
- Markdown documentation pages, one per bucket, with required sections, diagrams, and file references.
- Validation results for each page, including missing sections, hallucinated paths, and coverage warnings.
- Generation manifest for incremental updates and staleness tracking.
This engine is orchestrated by the BucketGenerationEngine class (deepdoc/generator_v2.py), which coordinates evidence assembly, LLM-based page generation, validation, and post-processing.
Input / Output Schema
Input: DocPlan & RepoScan
| Field | Type | Description |
|---|---|---|
DocPlan | Object | Contains buckets (list of DocBucket), navigation structure, and metadata |
DocBucket | Object | Represents a documentation page: type, title, slug, owned files, symbols, etc. |
RepoScan | Object | Full repo scan: file tree, parsed files, endpoints, integrations, etc. |
cfg | dict | Configuration: budgets, thresholds, feature flags |
Output: Generated Documentation
| Field | Type | Description |
|---|---|---|
| Markdown pages | string | One per bucket, with required sections and diagrams |
| Validation results | object | For each page: missing sections, file refs, hallucinated paths, warnings |
| Generation manifest | object | Tracks file hashes and staleness for incremental regeneration |
Processing Logic
The pipeline consists of several orchestrated steps, each handled by a dedicated class or function. Below is a high-level flowchart and a detailed breakdown of each stage.
Step-by-Step Breakdown
Evidence Assembly
The EvidenceAssembler class (deepdoc/generator_v2.py:90) gathers all relevant evidence for a single bucket:
- Full or partial source code for tracked files (
_build_source_context(),deepdoc/generator_v2.py:201) - Compressed evidence cards for large files (
_build_file_evidence_card(),deepdoc/generator_v2.py:603) - Endpoint, integration, artifact, and database context as needed
- Helper function bodies if relevant (
_build_helper_context(),deepdoc/generator_v2.py:311) - Cross-references to other buckets/pages
Page Generation
The PageGenerator class (deepdoc/generator_v2.py:1444) uses the assembled evidence and a prompt template (selected via get_prompt_for_bucket() in deepdoc/prompts_v2.py:1429) to generate a Markdown page via an LLM call.
- The prompt includes all required sections, diagrams, and file references.
- The LLM is instructed to follow strict rules for file references, cross-linking, and diagram inclusion.
Validation
The PageValidator class (deepdoc/generator_v2.py:1581) checks the generated page for:
- Presence of all required sections (
_check_sections(),deepdoc/generator_v2.py:1641) - File references for all owned files (
_check_file_refs(),deepdoc/generator_v2.py:1668) - No hallucinated file paths (
_check_hallucinated_paths(),deepdoc/generator_v2.py:1693) - Evidence-backed references and route claims
- Mermaid diagram count and structure
- Contract compliance (e.g., required sibling links)
Post-processing & Fixes If validation fails, the engine attempts to fix common issues:
- Adds missing sections as stubs
- Fixes Mermaid diagram syntax (
fix_mermaid_diagrams(),deepdoc/generator_v2.py:1890) - Removes hallucinated file references (
fix_file_references(),deepdoc/generator_v2.py:2051) - Escapes MDX hazards for safe rendering
If all else fails, a stub page is generated (
_generate_stub_page(),deepdoc/generator_v2.py:2789).
Parallel Generation & Manifest Update
The BucketGenerationEngine (deepdoc/generator_v2.py:2407) orchestrates parallel generation for all buckets, tracks results, and updates the manifest for incremental builds.
Sequence Diagram: End-to-End Generation
Error Handling & Recovery
The pipeline is designed for robustness and graceful degradation:
- Missing or Malformed Evidence: If a file is missing or cannot be parsed, it is skipped with a warning. Evidence cards are generated for files not included as raw source.
- LLM Failures: The
_call_with_retry()method (deepdoc/generator_v2.py:2705) retries transient LLM errors with exponential backoff and jitter. - Validation Failures: If required sections or file references are missing, the engine attempts to fix the output. If it cannot be fixed, a stub page is generated.
- Dead Letter Handling: Buckets that repeatedly fail generation are reported in the summary and can be retried manually.
- Partial Coverage: If evidence is sparse (e.g., giant files, missing symbols), the engine compresses context and emits warnings in the output.
If a bucket's source files change after generation, the manifest marks it as stale and triggers regeneration.
Configuration
The engine exposes several tunable parameters via the cfg dict:
| Parameter | Default | Description |
|---|---|---|
source_context_budget | 200,000 | Max characters of raw source per page |
large_file_lines | 500 | Line threshold for full vs. compressed source |
giant_file_lines | 2000 | Line threshold for giant-file clustering |
BATCH_SIZE | 8 | Number of pages generated in parallel |
MAX_PARALLEL_WORKERS | 8 | Max thread pool workers for generation |
RATE_LIMIT_PAUSE | 2.0 | Pause (seconds) between LLM calls |
MAX_RETRIES | 4 | Max retries for transient LLM errors |
Feature flags and thresholds can be set in the config dict passed to the engine.
Monitoring & Observability
- Progress Reporting: Uses
rich.progressto show live progress bars and status updates during generation. - Validation Warnings: All validation failures and warnings are aggregated and reported in the generation summary.
- Manifest Tracking: The manifest records file hashes and generation timestamps for each bucket, enabling incremental builds and staleness detection.
- Debug Output: Intermediate evidence, validation results, and post-processing actions are logged for troubleshooting.
Use the generation summary to quickly identify buckets with missing coverage, validation failures, or evidence gaps.
See Also
Parsing & Source Analysis
Explains the parsing logic, source code analysis, and type inference mechanisms that feed into the planning and generation pipeline.
DeepDoc Architecture & System Overview
High-level architecture, core concepts, and system boundaries for DeepDoc.
Setup & Getting Started
Installation, configuration, and environment setup for DeepDoc.
Database Models & Schema
Describes the database schema, ORM models, and relationships.
Cross-References
- DeepDoc Architecture & System Overview — for the overall system context and how the pipeline fits into the broader architecture.
- Parsing & Source Analysis — for details on how source files are parsed and analyzed before planning and generation.
- Setup & Getting Started — for configuration and environment setup relevant to running the pipeline.
Reference:
EvidenceAssembler(deepdoc/generator_v2.py:90)assemble()(deepdoc/generator_v2.py:134)PageGenerator(deepdoc/generator_v2.py:1444)generate()(deepdoc/generator_v2.py:1452)PageValidator(deepdoc/generator_v2.py:1581)validate()(deepdoc/generator_v2.py:1594)BucketGenerationEngine(deepdoc/generator_v2.py:2407)plan_docs()(deepdoc/planner_v2.py:1415)
For more on the planning phase and how buckets are constructed, see DeepDoc Architecture & System Overview.