bijux-rag Documentation¶

bijux-rag¶

At a glance: index → retrieve → ask • offline CI profile • reproducible chunk IDs + index fingerprints • CLI + FastAPI boundaries • OpenAPI drift-gated
Quality: make/tox gates green (tests, lint, types, docs strict, security, SBOM, REUSE, hygiene). Everything writes to artifacts/. No telemetry.

Docs: https://bijux.github.io/bijux-rag/
PyPI: https://pypi.org/project/bijux-rag/
Issues: https://github.com/bijux/bijux-rag/issues
Changelog: https://bijux.github.io/bijux-rag/changelog/

bijux-rag is a standalone Retrieval-Augmented Generation (RAG) toolkit for Python, emphasizing a functional core with pure transformations for document processing, chunking, and retrieval. It isolates I/O through explicit adapters and effect descriptions, enabling composable, testable pipelines without dependency on external frameworks. The toolkit supports both synchronous and asynchronous operations, with a focus on resilience, type safety, and interoperability.

All quality gates—enforced via Tox and Make—remain green: comprehensive tests (unit, integration, end-to-end), static analysis (linting, typing with MyPy/Pyright/Pytype), security audits (Bandit, Pip-Audit), and builds. Coverage is gated at 90%+ on the pinned eval suite; the codebase adheres to REUSE licensing standards and ships full MkDocs documentation.

At a Glance¶

Core Philosophy: Functional programming principles for RAG—pure functions, immutable data structures (e.g., document trees), and explicit effects via IOPlan/AsyncPlan—to ensure determinism and ease of testing.
Key Components: Primitives for chunking (fixed-size, recursive), embedding pipelines, result folding (fail-fast, error collection), and streaming (bounded concurrency, rate limiting).
Resilience Features: Policy-driven retries (with exponential backoff and jitter), timeouts, transactions, and fakes for testing (clocks, sleepers).
Interfaces: CLI for batch processing, HTTP API via FastAPI for serving, and Python API for custom pipelines.
Dependencies: Minimal runtime (Pydantic, NumPy, FastAPI, Uvicorn); dev extras for testing (Pytest, Hypothesis) and docs (MkDocs).
Version & Compatibility: v0.1.0; Python 3.11–3.13; MIT-licensed.
Quality Metrics: 100% coverage; strict typing; security-scanned; REUSE-compliant.

↑ Back to Top

↑ Back to Top

Features¶

bijux-rag prioritizes modularity and purity, allowing users to build RAG systems from composable building blocks while maintaining control over effects and dependencies.

Functional Primitives: Pure functions for document tree manipulation (flattening, folding), result handling (Result[T, ErrInfo] monad with folds like fail-fast or error-capped), and iterator-based pipelines.
Effect Management: Deferred I/O via IOPlan (sync) and AsyncPlan (async), supporting retries, transactions, backpressure, and rate limiting as configurable policies.
Resilience and Testing: Built-in policies for transient error handling; test utilities like fake clocks and sleepers ensure reliable unit testing without mocks.
Adapters and Interop: Storage options (file, in-memory); compatibility with NumPy for vectors, Pydantic for validation, and standard libraries (e.g., itertools, functools).
Streaming Capabilities: Lazy async streams with bounded mapping, fair merging, and chunking policies for high-throughput scenarios.
Tooling Integration: Comprehensive setup with Ruff for style, multiple type checkers, Hypothesis for property-based tests, and MkDocs for documentation.

↑ Back to Top

Installation¶

Requires Python 3.11 or later.

pip install bijux-rag

For development (includes testing, documentation, and linting tools):

pip install bijux-rag[dev]

From source:

git clone https://github.com/bijux/bijux-rag.git
cd bijux-rag
make bootstrap  # Sets up virtualenv and installs in editable mode

Dependencies are minimal and security-audited; refer to pyproject.toml for details.

↑ Back to Top

Quick Start¶

Process documents via CLI:

bijux-rag process --input docs.csv --output embeddings.msgpack

This command reads CSV documents, applies functional chunking, performs embedding (via configured adapter), and outputs MessagePack results.

Programmatic equivalent:

from bijux_rag.core.rag_types import RawDoc
from bijux_rag.pipelines.embedding import embed_docs
from bijux_rag.infra.adapters.memory_storage import InMemoryStorage

docs = [RawDoc(doc_id="1", title="Example", abstract="Sample text.")]
storage = InMemoryStorage()
results = list(embed_docs(docs, storage))  # Composable iterator pipeline

↑ Back to Top

Usage¶

bijux-rag offers multiple entry points: CLI for scripting, HTTP API for services, and Python API for integration.

CLI¶

Access help:

bijux-rag --help

Example with custom parameters:

bijux-rag process --input input.csv --chunk-size 512 --embedder default

Note: Embedder options depend on configured adapters; defaults to basic implementations.

HTTP API¶

Launch the server:

bijux-rag serve --port 8000

Interact via endpoints like /embed (POST documents for processing) or /retrieve (query-based retrieval). Explore via OpenAPI at /docs.

Python API¶

Focuses on composability:

Documents: Use RawDoc and Chunk types; build trees with make_chunk.
Pipelines: Chain functions, e.g., read_docs | fixed_size_chunk | embed_docs.
Effects: Wrap I/O in IOPlan for sync or AsyncPlan for async; apply policies like retry_idempotent.
Streaming: Leverage AsyncGen for lazy processing, e.g., async_gen_bounded_map for concurrency control.

Synchronous retry example:

from bijux_rag.domain.effects import retry_idempotent, RetryPolicy
from bijux_rag.policies.chunking import fixed_size_chunk
from bijux_rag.result import fold_results_fail_fast

policy = RetryPolicy(max_attempts=3)
safe_read = retry_idempotent(policy)(storage.read_docs("input.csv"))
docs_results = list(safe_read("input.csv"))
chunks = list(fold_results_fail_fast(docs_results, [], fixed_size_chunk))

Asynchronous streaming example:

from bijux_rag.domain.effects.async_ import async_gen_map, resilient_mapper

mapper = resilient_mapper(embed_fn, RetryPolicy(max_attempts=3))
stream = async_gen_map(source_stream, mapper)
async for result in stream():
    # Handle result

Consult the API reference in documentation for complete details.

↑ Back to Top

Architecture¶

Adopts a hexagonal (ports and adapters) design with a functional core:

Boundaries: CLI and HTTP shells interpret inputs and delegate to domain logic.
Core: Pure, deterministic functions for RAG operations (e.g., tree folding, result monads).
Domain: Effect descriptions (IOPlan, AsyncPlan), policies (chunking, retry), and types.
Infra: Pluggable adapters for storage (file, memory) and other I/O.
Interop/Policies: Helpers for stdlib FP and reusable behaviors.

This structure facilitates adapter swaps (e.g., local to cloud storage) without altering core code. Review Architecture Documentation for decision records (ADRs) and overviews.

↑ Back to Top

Testing and Quality¶

Execute tests:

make test

Other targets:

make lint: Enforces style (Ruff) and types (MyPy, Pyright, Pytype).
make security: Runs Bandit and dependency audits.
make docs: Builds and serves MkDocs.
make all: Comprehensive run (clean, install, test, lint, build).

CI ensures all gates pass on every commit.

↑ Back to Top

Contributing¶

Report issues or suggest features via GitHub Issues. Pull requests must maintain green gates. Setup instructions:

make bootstrap

Follow guidelines in CONTRIBUTING.md.

↑ Back to Top

License¶

MIT License—see LICENSE. The project is fully REUSE-compliant for copyright and licensing metadata.

↑ Back to Top

Acknowledgments¶

Draws inspiration from functional programming paradigms (e.g., monads, immutability) and RAG literature. Gratitude to open-source tools like Ruff, Hypothesis, and MkDocs that support the project's quality standards.

↑ Back to Top