Webhook Integrations

What This Integration Does

Webhook integrations in DeepDoc provide a mechanism for the system to react to external events or synchronize with third-party systems in real time. Webhooks are used to trigger actions such as corpus refreshes, incremental updates, or background jobs when changes occur in connected systems (e.g., code repositories, artifact stores, or external APIs). This enables DeepDoc to maintain up-to-date documentation and chatbot corpora without manual intervention.

Webhooks are a key part of DeepDoc's extensibility, allowing for integration with CI/CD pipelines, external LLM providers, and other automation tools. They ensure that documentation and chatbot knowledge bases reflect the latest source-of-truth data.

Where It Enters the Codebase

Webhook handling and integration logic are distributed across several modules. The main entry points are:

• Webhook Handler Patterns:
  • Webhook handler patterns are referenced in deepdoc/chatbot/indexer.py (for corpus refresh triggers) and in the integration context for generic webhook handler detection.
• Indexer & Corpus Refresh:
  • ChatbotIndexer.sync_full() and ChatbotIndexer.sync_incremental() in deepdoc/chatbot/indexer.py:35,60 — These methods are invoked when a webhook signals that new data is available or changes have occurred.
  • chatbot_index_needs_refresh() in deepdoc/chatbot/indexer.py:194 — Used to determine if a webhook-triggered refresh is required.
• CLI Integration:
  • The CLI commands in deepdoc/cli.py (notably generate, update, and serve) can be triggered as a result of webhook events, either directly or via automation.
• Configuration & Environment:
  • Webhook-related configuration is handled in deepdoc/chatbot/settings.py (see functions like get_chatbot_cfg() and environment variable resolution).
• Provider Wrappers:
  • While not direct webhook handlers, provider wrappers in deepdoc/chatbot/providers.py can be invoked as a result of webhook-triggered jobs (e.g., to re-embed new content).

Participating Features & Endpoints

Webhook integrations are leveraged by several DeepDoc features and endpoints:

• Chatbot Corpus Refresh:
  • When a webhook signals a change (e.g., new code pushed, artifact updated), DeepDoc triggers a corpus refresh via ChatbotIndexer.sync_full() or sync_incremental() (deepdoc/chatbot/indexer.py).
• Incremental Documentation Updates:
  • The update command in deepdoc/cli.py can be triggered by a webhook to incrementally update documentation after source changes.
• Background Jobs:
  • Webhook events may enqueue background jobs for chunking, embedding, or summarization (see Documentation Chunking and Summarization).
• Frontend Integration:
  • Webhook-driven updates ensure that the Site Generation & Frontend Integration Overview remains in sync with the latest data.

Request/Response or Message Flow

Below is a typical webhook-triggered flow for a corpus refresh:

Example Webhook Payload

While the exact payload shape depends on the external system, a typical webhook POST might look like:

{
  "event": "repo_push",
  "changed_files": ["src/foo.py", "docs/README.md"],
  "deleted_files": [],
  "timestamp": "2024-06-01T12:34:56Z"
}

Handler Logic

• The webhook handler parses the incoming payload and extracts relevant fields (changed_files, deleted_files).
• It invokes ChatbotIndexer.sync_incremental() (deepdoc/chatbot/indexer.py:60) with these lists.
• The indexer determines which corpora need to be rebuilt or merged (see _corpus_needs_rebuild() in deepdoc/chatbot/indexer.py:179).
• Embedding and chunking jobs are dispatched as needed (see build_embedding_client() in deepdoc/chatbot/providers.py:136).
• On success, a 200 OK is returned to the webhook sender.

Auth & Configuration

Webhook endpoints may be protected by secret tokens, IP allowlists, or other authentication mechanisms, depending on deployment.

Environment Variables

Configuration

• Webhook endpoints and allowed origins are configured in the chatbot config (deepdoc/chatbot/settings.py:13).
• Allowed origins can be extended at runtime if DEEPDOC_CHATBOT_PREVIEW_PORT is set.
• API keys for downstream providers (e.g., embedding or chat APIs) are resolved via resolve_service_api_key() (deepdoc/chatbot/settings.py:140).

Retry, Reconciliation & Failure Handling

Failure Handling

• Embedding/Chunking Errors:

  • If embedding or chunking fails (e.g., due to provider downtime), exceptions are raised with detailed error messages. See error handling in LiteLLMEmbeddingClient.embed() (deepdoc/chatbot/providers.py:60).
  • Context window errors are detected and retried with smaller batches or truncated text (_is_context_window_error() and _trim_text_for_retry() in deepdoc/chatbot/providers.py:109,117).

• Corpus Consistency:

  • The indexer checks for missing or inconsistent corpora using _corpus_needs_rebuild() (deepdoc/chatbot/indexer.py:179).
  • If a previous sync failed, the next webhook-triggered sync will attempt a full rebuild for any missing corpus.

• Webhook Delivery:

  • Webhook handlers are stateless and safe for repeated delivery (idempotent). If a webhook is delivered multiple times, only changed or missing corpora are rebuilt.

No Built-in Retry for Incoming Webhooks

• DeepDoc does not implement retry logic for incoming webhook delivery failures. It is the responsibility of the external system to retry failed webhook POSTs.

Operational Gotchas

• CORS & Allowed Origins:
  • If running locally, ensure DEEPDOC_CHATBOT_PREVIEW_PORT is set so the frontend can communicate with the backend during preview (deepdoc/chatbot/settings.py:126).
• Batch Size Limits:
  • Embedding batch sizes are provider-specific and can be configured. Azure defaults to batch size 1; others default to 24 (deepdoc/chatbot/providers.py:54).
• Context Window Errors:
  • Large payloads may exceed the provider's context window. The embedding client will attempt to split or truncate text, but repeated failures may require manual intervention.
• Webhook Security:
  • If deploying public webhook endpoints, ensure you implement authentication (e.g., secret tokens or IP allowlists) at the API gateway or application layer.
• Version Compatibility:
  • Changes to payload structure or event types in external systems may require updates to the webhook handler logic.

Diagrams

Webhook Integration Flow

See Also

• Runtime Services & Chatbot Engine
• CLI Commands & Tooling
• Site Generation & Frontend Integration Overview
• Site Builder Workflow and Frontend Integration
• Anthropic Integration
• Documentation Chunking and Summarization
• DeepDoc Chat API Integration
• DeepDoc Embedding API Integration