context_harness/
lib.rs

1// `async_trait` generates `#[must_use]` futures whose `Result` outputs are
2// already `#[must_use]`; Rust 1.99's Clippy reports the generated overlap.
3#![allow(clippy::double_must_use)]
4
5//! # Context Harness
6//!
7//! **A local-first context ingestion and retrieval framework for AI tools.**
8//!
9//! Context Harness provides a connector-driven pipeline for ingesting documents
10//! from multiple sources (filesystem, Git repositories, S3 buckets, Lua scripts), chunking
11//! and embedding them, and exposing hybrid search (keyword + semantic) via a
12//! CLI and MCP-compatible HTTP server.
13//!
14//! ## Architecture
15//!
16//! ```text
17//! ┌─────────────┐   ┌─────────────┐   ┌──────────┐
18//! │ Connectors  │──▶│  Pipeline    │──▶│  SQLite   │
19//! │ FS/Git/S3   │   │ Chunk+Embed │   │ FTS5+Vec  │
20//! └─────────────┘   └─────────────┘   └────┬─────┘
21//!                                          │
22//!                      ┌───────────────────┤
23//!                      ▼                   ▼
24//!                 ┌──────────┐       ┌──────────┐
25//!                 │   CLI    │       │   HTTP   │
26//!                 │  (ctx)   │       │  (MCP)   │
27//!                 └──────────┘       └──────────┘
28//! ```
29//!
30//! ## Data Flow
31//!
32//! 1. **Connectors** scan external sources and produce [`models::SourceItem`]s.
33//! 2. The **ingestion pipeline** ([`ingest`]) normalizes items into [`models::Document`]s,
34//!    computes deduplication hashes, and upserts them into SQLite.
35//! 3. Documents are split into [`models::Chunk`]s by the paragraph-boundary
36//!    chunker ([`chunk`]).
37//! 4. Chunks are indexed in **FTS5** for keyword search and optionally
38//!    embedded via the **embedding provider** ([`embedding`]) for vector search.
39//! 5. The **query engine** ([`search`]) supports keyword, semantic, and hybrid
40//!    retrieval with min-max normalized scoring.
41//! 6. Results are exposed via the **CLI** (`ctx`) and the **MCP HTTP server** ([`server`]).
42//!
43//! ## Quick Start
44//!
45//! ```bash
46//! ctx init                      # create database
47//! ctx sync all                  # ingest all configured sources (parallel)
48//! ctx sync git:platform         # ingest a specific git connector
49//! ctx embed pending             # generate embeddings
50//! ctx search "deployment" --mode hybrid
51//! ctx serve mcp                 # start HTTP server
52//! ```
53//!
54//! ## Connectors
55//!
56//! | Connector | Source | Module |
57//! |-----------|--------|--------|
58//! | Filesystem | Local directories | [`connector_fs`] |
59//! | Git | Any Git repository (local or remote) | [`connector_git`] |
60//! | S3 | Amazon S3 / S3-compatible buckets | [`connector_s3`] |
61//! | Lua Script | Any source via custom Lua scripts | [`connector_script`] |
62//!
63//! ## Search Modes
64//!
65//! | Mode | Engine | Requires Embeddings |
66//! |------|--------|---------------------|
67//! | `keyword` | SQLite FTS5 (BM25) | No |
68//! | `semantic` | Cosine similarity over vectors | Yes |
69//! | `hybrid` | Weighted merge (configurable α) | Yes |
70//!
71//! ## Modules
72//!
73//! | Module | Purpose |
74//! |--------|---------|
75//! | [`config`] | TOML configuration parsing and validation |
76//! | [`models`] | Core data types: `SourceItem`, `Document`, `Chunk`, `SearchResult` |
77//! | [`connector_fs`] | Filesystem connector: walk local directories |
78//! | [`connector_git`] | Git connector: clone/pull repos with per-file metadata |
79//! | [`connector_s3`] | S3 connector: list and download objects with SigV4 signing |
80//! | [`connector_script`] | Lua scripted connectors: custom data sources via Lua 5.4 scripts |
81//! | [`lua_runtime`] | Shared Lua 5.4 VM runtime: sandboxing, host APIs, value conversions |
82//! | [`tool_script`] | Lua MCP tool extensions: load, validate, execute Lua tool scripts |
83//! | [`traits`] | Extension traits: `Connector`, `Tool`, `ToolContext`, registries |
84//! | [`profiles`] | Reusable prompt profiles: `Profile`, `ProfilePrompt`, `ProfileRegistry`, `TomlProfile` |
85//! | [`profile_script`] | Lua scripted profiles: load, resolve, scaffold, test |
86//! | [`agents`] | Deprecated compatibility aliases for the former prompt-agent API |
87//! | [`chunk`] | Paragraph-boundary text chunker |
88//! | [`embedding`] | Embedding provider trait, OpenAI implementation, vector utilities |
89//! | [`embed_cmd`] | Embedding CLI commands: `pending` and `rebuild` |
90//! | [`export`] | JSON export for static site search (`ctx export`) |
91//! | [`stats`] | Database statistics: document, chunk, and embedding counts |
92//! | [`ingest`] | Ingestion pipeline: connector → normalize → chunk → embed → store |
93//! | [`search`] | Keyword, semantic, and hybrid search with score normalization |
94//! | [`get`] | Document retrieval by UUID |
95//! | [`sources`] | Connector health and status listing |
96//! | [`server`] | MCP-compatible HTTP server (Axum) with CORS |
97//! | [`db`] | SQLite connection pool with WAL mode |
98//! | [`migrate`] | Database schema migrations (idempotent) |
99//!
100//! ## Configuration
101//!
102//! Context Harness is configured via a TOML file (default: `config/ctx.toml`).
103//! See [`config`] for all available options and [`config::load_config`] for
104//! validation rules.
105
106pub mod agent_host;
107pub mod agent_model;
108pub mod agent_resource;
109pub mod agent_runtime;
110pub mod agent_script;
111pub mod agent_store;
112pub mod agent_task_store;
113pub mod agents;
114pub mod app_store;
115pub mod chunk;
116pub mod config;
117pub mod connector_fs;
118pub mod connector_git;
119pub mod connector_s3;
120pub mod connector_script;
121pub mod ctx_dirs;
122pub mod db;
123mod document_ingestor;
124pub mod embed_cmd;
125pub mod embedding;
126pub mod export;
127pub mod extract;
128pub mod get;
129pub mod ingest;
130pub mod lua_runtime;
131pub mod lua_tool_binding;
132pub mod mcp;
133pub mod migrate;
134pub mod models;
135pub mod profile_script;
136pub mod profiles;
137pub mod progress;
138pub mod redact;
139pub mod registry;
140pub mod search;
141pub mod server;
142pub mod sources;
143pub mod sqlite_store;
144pub mod stats;
145pub mod targeted_refresh;
146pub mod tool_binding;
147pub mod tool_script;
148pub mod traits;
149pub mod vector_index;
150pub mod workspace;
151
152pub use profiles::{Profile, ProfilePrompt, ProfileRegistry, TomlProfile};
153
154#[allow(deprecated)]
155pub use agents::{Agent, AgentPrompt, AgentRegistry, TomlAgent};
156pub use context_harness_core::store;
157pub use models::SourceItem;
158pub use traits::{
159    Connector, ConnectorRegistry, GetTool, SearchTool, SourcesTool, Tool, ToolContext,
160    ToolExecutionError, ToolExecutionErrorClass, ToolRegistry,
161};
context_harness/lib.rs

context_harness/
lib.rs
