Home
Multimodal ingestion that does not dictate your stack
OmniIngest is a multimodal ingestion framework for building reusable knowledge pipelines. It turns documents, web pages, images, and audio into validated knowledge items without the faff of stitching together one-off preprocessing scripts.
Documentation: omniingest.onrender.com
Why use OmniIngest
- Multimodal processing. PDF, DOCX, HTML, images, audio, and plain text use one pipeline model.
- Multiple model providers. Every model-backed step can select its own Pydantic AI model. Chat and embedding models are configured independently.
- Storage-independent pipelines. YAML describes transformation. CLI or Python caller chooses JSON, vector store, graph store, or another output.
- Structured when needed. JSON Schema output, jq transforms, and step validation turn model results into predictable data.
- Built for sizeable jobs. fsspec resources, concurrent stages, progress events, and stage checkpoints keep long ingestion runs manageable.
Quick start
Install command-line tool with uv:
Create a pipeline which extracts text and makes sentence-aware chunks:
school_notes.yaml
pipeline_id: school_notes
domain_profile: education
steps:
- agent: text
- agent: chunking
config:
chunk_size: 900
overlap: 120
Run it against a PDF:
YAML stops at transformation. Pick storage at run time:
omni-ingest school_notes.yaml --input lessons/soil-erosion.pdf --output - | jq '.items | length'
omni-ingest retrieval.yaml --input handbooks/*.pdf --output vector:default
Learn more
- Command-line usage covers files, remote resources, stdin, outputs, and resume.
- Pipeline playbook builds from simple YAML to structured output, validation, components, and model tools.
- Working with event stream covers progress and live application integration.
- API reference documents Python interfaces.