Testing & Continuous Integration

DeepDoc employs a robust, multi-layered testing and CI strategy to ensure high reliability, maintainability, and rapid feedback for contributors. This guide details the test structure, CI/CD pipeline logic, and quality assurance practices, referencing concrete code and configuration patterns. For a high-level system context, see DeepDoc Architecture & System Overview.

Overview

Testing and continuous integration in DeepDoc serve several critical purposes:

• Regression prevention: Automated tests catch breaking changes in parsing, CLI, embedding, and site generation logic.
• Cross-framework validation: Tests cover Django, Express, Fastify, Falcon, and Vue parsing, ensuring framework-agnostic correctness.
• CI/CD enforcement: The pipeline ensures that only passing code is merged and deployed.
• Quality gates: Linting, artifact checks, and config validation are enforced before documentation is built or published.

Tests are organized by concern:

• Framework support and parsing: tests/test_framework_support.py
• CLI and environment handling: tests/test_cli_generate.py
• Embedding and LLM integration: tests/test_chatbot_providers.py
• Site generation and MDX normalization: tests/test_fumadocs_builder.py

For details on how parsing and planning fit into the overall architecture, see Parsing & Source Analysis and Pipeline & Generation Engine.

How It Works

The testing and CI flow is as follows:

• Test execution: All tests in the tests/ directory are run using pytest. This includes framework parsing, CLI, embedding, and site generation tests.
• Linting & coverage: Static analysis and coverage checks are enforced before site build.
• Site build: The documentation site is built using Fumadocs (Site Generation & Frontend Integration Overview), with tests verifying correct MDX normalization and OpenAPI asset staging.
• Deployment: Only if all checks pass are artifacts published or deployed.

Key Components

1. Framework Parsing Tests

test_django_detects_class_views_and_drf_router_actions()

File: tests/test_framework_support.py:13

• Purpose: Verifies that Django class-based views, DRF router actions, and function views are correctly detected by the endpoint parser.
• Inputs: Simulated Django URL config as a string.
• Outputs: Asserts that all expected endpoints are found by detect_endpoints() (deepdoc/parser/api_detector.py).
• Example:

  endpoints = detect_endpoints(Path("urls.py"), content, "python")
  method_paths = {(ep.method, ep.path, ep.handler) for ep in endpoints}
  assert ("GET", "/health", "health") in method_paths

• Related: Parsing & Source Analysis

test_scan_repo_resolves_express_mounts_across_files()

File: tests/test_framework_support.py:247

• Purpose: Ensures Express routers and controllers are resolved across multiple files, including nested mounts.
• Inputs: Temporary file structure simulating Express app.
• Outputs: Asserts that endpoint paths and handler files are correctly resolved.
• Example:

  scan = scan_repo(repo_root, deepcopy(DEFAULT_CONFIG))
  endpoint = next(ep for ep in scan.api_endpoints if ep["handler"] == "webhookController.handleOrderStatus")
  assert endpoint["path"] == "/api/v1/webhook/orderstatus"

• Related: Webhook Integrations

2. CLI & Environment Handling

test_cli_autoloads_repo_env_file()

File: tests/test_cli_generate.py:10

• Purpose: Ensures that .env files are loaded into the environment by the CLI, but do not override existing exports.
• Inputs: .env file and environment variable state.
• Outputs: Asserts correct environment variable precedence.
• Example:

  result = CliRunner().invoke(cli.main, ["clean", "--yes"])
  assert cli.os.environ["DEEPDOC_SAMPLE_KEY"] == "from-dotenv"

• Related: CLI Commands & Tooling, Setup & Getting Started

test_clean_removes_deepdoc_artifacts_and_config()

File: tests/test_cli_generate.py:32

• Purpose: Validates that the clean CLI command removes all DeepDoc-generated artifacts but preserves unrelated files.
• Inputs: Simulated repo structure with DeepDoc artifacts.
• Outputs: Asserts that only DeepDoc files are deleted.
• Example:

  assert not output_dir.exists()
  assert (repo_root / "keep.txt").exists()

3. Embedding & LLM Integration

test_embedding_client_splits_batches_on_context_window_error()

File: tests/test_chatbot_providers.py:12

• Purpose: Ensures that the embedding client splits batches when the provider raises a context window error (e.g., Azure).
• Inputs: Fake embedding client, monkeypatched to simulate errors.
• Outputs: Asserts that batch splitting and retry logic work as intended.
• Example:

  vectors = client.embed(["alpha", "beta", "gamma"])
  assert vectors == [[5.0], [4.0], [5.0]]

• Related: DeepDoc Embedding API Integration, Anthropic Integration

test_embedding_client_trims_single_oversized_text_on_retry()

File: tests/test_chatbot_providers.py:35

• Purpose: Verifies that the embedding client trims input text when a single item exceeds the context window.
• Outputs: Asserts that the retried text is shorter and ends with a truncation marker.
• Example:

  assert calls[-1].endswith("... [truncated for embedding]")

4. Site Generation & MDX Normalization

test_build_fumadocs_from_plan_creates_site_scaffold()

File: tests/test_fumadocs_builder.py:21

• Purpose: Ensures that the site builder creates the correct Fumadocs scaffold from a documentation plan.
• Inputs: Plan with buckets for overview, auth, and endpoint reference.
• Outputs: Asserts that index and section files are generated.
• Example:

  build_fumadocs_from_plan(repo_root, output_dir, {...}, plan, has_openapi=True)
  assert (output_dir / "index.mdx").exists()

• Related: Site Builder Workflow and Frontend Integration

test_escape_mdx_route_params_avoids_runtime_expressions()

File: tests/test_fumadocs_builder.py:644

• Purpose: Ensures that route parameters in MDX are escaped to avoid runtime expressions.
• Outputs: Asserts that curly braces are replaced with HTML entities.
• Example:

  assert "/reports/{slug}" in escaped

Configuration

Several environment variables and config keys affect test and CI behavior:

• .env loading: The CLI will load .env files in the repo root, but will NOT override existing environment variables. See test_cli_autoloads_repo_env_file() and test_cli_repo_env_does_not_override_existing_exports() in tests/test_cli_generate.py.
• Config files: .deepdoc.yaml and other DeepDoc config files are created and cleaned up by CLI commands and tested for correct behavior.

For environment setup and configuration, see Setup & Getting Started.

Common Patterns

Running Tests

All tests are written for pytest. To run the full suite:

pytest tests/

CLI Testing

Use click.testing.CliRunner to invoke CLI commands in tests, as shown in tests/test_cli_generate.py:

from click.testing import CliRunner
result = CliRunner().invoke(cli.main, ["clean", "--yes"])
assert result.exit_code == 0

Temporary File Structures

Many tests use tmp_path to create isolated file trees for parsing and site generation:

def test_scan_repo_resolves_express_mounts_across_files(tmp_path):
    repo_root = tmp_path / "sync-app"
    # ... create files and directories ...
    scan = scan_repo(repo_root, deepcopy(DEFAULT_CONFIG))

Monkeypatching for Isolation

Tests that simulate external dependencies (like LLM providers) use monkeypatch to inject fake implementations:

monkeypatch.setattr("deepdoc.chatbot.providers.prepare_litellm", lambda: _FakeLiteLLM())

Site Generation Verification

Site builder tests check for the presence or absence of generated files:

assert (output_dir / "index.mdx").exists()
assert not (repo_root / "site" / "components" / "api-page.tsx").exists()

Gotchas & Edge Cases

See Also