Semantic Search & Graph Retrieval

View Repository Source

An embedded property graph and retrieval pipeline indexing 143 sessions of AI coding history with BM25 FTS and semantic search.

Searching through transcripts of AI agent sessions is fundamentally different from searching standard documentation. Agent sessions are mostly machine output, not conversation, and naive transcript searches quickly drown in directory listings and code dumps.

This project is an indexing and retrieval pipeline built over my own AI coding-agent history: 143 sessions, 14,924 content blocks (one per prose run, tool call or tool result), 27,443 chunks, and 155 distinct tools across 45 projects. These are counts from my own corpus as of the August 2026 rebuild.

Intelligent Discrimination and Chunking

To make the data searchable, I implemented a content_kind discriminator that separates human prose from tool invocations and results. A default query searches the 4,960 prose chunks, while error-hunting queries can target specific tool outputs deliberately.

Turns are sub-chunked to ~250 tokens with sequential linkage. This prevents embedding models from silently truncating oversized inputs—a subtle defect that degrades retrieval quality invisibly.

Traversable Graph Architecture

Instead of a flat index, sessions are modeled as a traversable graph. Projects, chronological sessions, chunks, and tool invocations act as first-class nodes. Retrieval can be rigorously scoped by project, working directory, Git branch, or even which specific tool a turn called.

Practical Utility: Credential Auditing

A searchable archive of agent transcripts is also a searchable archive of everything those transcripts leaked. I wrote a credential auditor over this corpus. Its latest scan covered 188 files and surfaced 81 potential exposures across seven secret classes, a concrete worklist for credential hygiene.

The Public Library

The pipeline above is private and runs over my own history. The linked repository is a different, smaller thing: the generic library extracted from it, with document parsers, chunking, pluggable embedding providers and a graph storage backend. It does not include the session ingestion, the content_kind discrimination, the full-text index or the credential auditor.