Runtime Services & Chatbot Engine

Overview

The Runtime Services & Chatbot Engine feature powers DeepDoc's interactive, context-grounded Q&A experience for developers. It orchestrates the retrieval, ranking, and synthesis of code, documentation, and artifact evidence to answer user questions with precise, source-linked responses. This engine is responsible for:

• Serving real-time chatbot queries via a FastAPI backend.
• Indexing and refreshing code, docs, and relationships for retrieval.
• Integrating with LLM providers for query expansion, reranking, and embedding.
• Managing service orchestration and error handling for robust, developer-facing APIs.

This feature is central to DeepDoc's mission of providing exhaustive, trustworthy, and actionable answers grounded in your actual codebase. For a high-level system context, see DeepDoc Architecture & System Overview.

Files Covered

Main Workflows

The main workflow is the chatbot query lifecycle, which processes a user question and returns a grounded, source-linked answer. This involves:

1. Receiving a Query: The FastAPI app (create_fastapi_app() in deepdoc/chatbot/service.py) exposes a /query endpoint that accepts a QueryRequest.
2. Query Expansion: The question is expanded into alternative queries using LLMs for better recall (_expand_query()).
3. Embedding: All query variants are embedded in a batch using the embedding provider (build_embedding_client()).
4. Similarity Search: Each corpus (code, artifact, docs, relationships) is searched for relevant chunks (_multi_query_search()).
5. Chain Retrieval: Relationship hits are used to pull in related code chunks (_chain_retrieve()).
6. Reranking: Candidate chunks are reranked using an LLM for precision (_rerank()).
7. Prompt Assembly: The final prompt is built, including code, artifacts, docs, and relationships (_build_prompt()).
8. LLM Answer Generation: The answer is generated using the chat provider.
9. Response Formatting: The answer, citations, doc links, and usage stats are returned.

Main Chatbot Query Flow

• Integration points: LLM providers (LLM Client), embedding APIs (DeepDoc Embedding API Integration), and the indexer (Documentation Chunking and Summarization).

Participating Endpoints

• Both endpoints are defined in create_fastapi_app() (deepdoc/chatbot/service.py:514).

Core Helpers & Business Rules

ChatbotQueryService (deepdoc/chatbot/service.py:25)

Handles the full lifecycle of a chatbot query. Key methods:

query() (deepdoc/chatbot/service.py:53)

• Purpose: Orchestrates query expansion, embedding, retrieval, reranking, and response formatting.
• Parameters:
  • question: str
  • history: list[dict[str, str]] | None
• Returns: dict[str, Any] with answer, citations, doc links, and usage stats.

Branching & Business Rules:

• Query Expansion: If retrieval_cfg["query_expansion"] is true, uses _expand_query() to generate variants.
• Corpus Search: Searches code, artifact, doc, and relationship corpora, each with their own top_k limits.
• Chain Retrieval: If relationship chunks reference files not in code hits, pulls in their top code chunks.
• Reranking: If retrieval_cfg["rerank"] is true, uses LLM to rerank candidates; otherwise, sorts by score and evidence priority.
• Prompt Limits: Applies max_prompt_code_chunks, max_prompt_artifact_chunks, etc., to limit included context.
• No Context: If no evidence is found, returns a fallback message via _no_context_result().

Example Usage:

service = ChatbotQueryService(repo_root, cfg)
response = service.query("How does authentication work?", history=[])

_expand_query() (deepdoc/chatbot/service.py:164)

• Purpose: Uses LLM to generate alternative search queries for better recall.
• Branching:
  • If retrieval_cfg.get("query_expansion", False) is false, returns [question] only.
  • On LLM failure, returns only the original question.
• Returns: List of query strings.

_multi_query_search() (deepdoc/chatbot/service.py:186)

• Purpose: Searches a corpus with multiple query vectors, merges results by max score per chunk.
• Branching:
  • If records is empty, returns [].
• Returns: Top-k RetrievedChunk objects.

_chain_retrieve() (deepdoc/chatbot/service.py:217)

• Purpose: Uses relationship chunks (import graphs) to pull in code from related files not already in code hits.
• Branching:
  • Only adds code chunks from files referenced in relationship hits and not already present.
  • Limits to first 2 code chunks per related file.
• Returns: Original code hits + chain-retrieved extras.

_rerank() (deepdoc/chatbot/service.py:260)

• Purpose: Uses LLM to rerank candidate chunks for better answer precision.
• Branching:
  • If retrieval_cfg.get("rerank", False) is false, falls back to _sort_hits().
  • If no candidates, returns original hits.
  • On LLM failure, falls back to original order.
• Returns: Tuple of reranked code, artifact, and doc hits.

_no_context_result() (deepdoc/chatbot/service.py:150)

• Purpose: Returns a fallback answer when no evidence is found.
• Returns: Dict with a user-facing message and empty citations.

Provider Helpers

• build_chat_client() (deepdoc/chatbot/providers.py:132): Returns a LiteLLMChatClient for chat completions.
• build_embedding_client() (deepdoc/chatbot/providers.py:136): Returns a LiteLLMEmbeddingClient for embeddings.

Provider Classes

• LiteLLMChatClient (deepdoc/chatbot/providers.py:12): Wraps LLM chat completions. Handles model selection, API key resolution, and error handling.
• LiteLLMEmbeddingClient (deepdoc/chatbot/providers.py:51): Wraps LLM embedding calls. Handles batching, context window errors, and retry logic.

Indexer

• ChatbotIndexer (deepdoc/chatbot/indexer.py:25): Builds and refreshes all chatbot corpora (code, artifact, doc_summary, relationship).
  • sync_full(): Full rebuild.
  • sync_incremental(): Incremental update based on changed/deleted files.

State Transitions

The main state transitions involve the lifecycle of indexed corpora and the query/response flow.

• Corpus state: Managed by ChatbotIndexer — transitions from "stale" to "fresh" on full/incremental sync.

Integrations Involved

• LLM Providers: Used for query expansion, reranking, and answer generation. See LLM Client.
• Embedding Providers: Used for vectorizing queries and evidence. See DeepDoc Embedding API Integration.
• Webhook Integrations: Indexer can be triggered by webhooks for incremental sync. See Webhook Integrations.

Configuration & Environment

Config Flags

• chatbot.enabled: Enables/disables the chatbot backend.
• chatbot.retrieval.query_expansion: Enables LLM-based query expansion.
• chatbot.retrieval.rerank: Enables LLM-based reranking.
• chatbot.retrieval.top_k_code, top_k_artifact, top_k_docs, top_k_relationship: Limits for retrieval.
• chatbot.retrieval.max_prompt_code_chunks, etc.: Prompt context limits.

Environment Variables

Edge Cases & Failure Modes

• No Evidence Found: If no code, artifact, or doc chunks are retrieved, a fallback message is returned (_no_context_result()).
• LLM Provider Errors: If the LLM provider fails (e.g., API key missing, quota exceeded), a RuntimeError is raised and a 500 error is returned.
• Embedding Errors: If embedding fails due to context window or batch size, the system retries with smaller batches or truncated text.
• Startup Failure: If the backend cannot initialize (e.g., config missing), the scaffolded app returns a 503 error with details.
• CORS Misconfiguration: If allowed origins are not set, frontend requests may be blocked.

Diagrams

Sequence Diagram: Chatbot Query Lifecycle

Component Diagram: Runtime Service Orchestration

Quick Reference

Constants, Enums & Status Values

No explicit enums or status constants are defined in the evidence for this feature. All status and config values are handled via config dicts and runtime logic.

See Also

• DeepDoc Architecture & System Overview
• Public API Endpoints
• Webhook Integrations
• DeepDoc Chat API Integration
• DeepDoc Embedding API Integration
• Documentation Chunking and Summarization

service_lifecycle

TODO: This section (service_lifecycle) needs to be filled in with details from the source files listed above.

main_workflows

TODO: This section (main_workflows) needs to be filled in with details from the source files listed above.

error_handling

TODO: This section (error_handling) needs to be filled in with details from the source files listed above.

extensibility

TODO: This section (extensibility) needs to be filled in with details from the source files listed above.