FastIngest
FastIngest is a high-throughput, constant-memory bulk ingestion pipeline for .NET (CSV, XLSX, and NDJSON/JSONL to PostgreSQL, SQL Server, MySQL, SQLite, MongoDB, Cosmos DB, and Elasticsearch). Designed for enterprise workloads processing millions of rows without memory spikes, FastIngest leverages zero-allocation streaming readers, concurrent producer-consumer bounded channels, fluent validation, and native database bulk protocols (such as PostgreSQL binary
COPY, SQL Server SqlBulkCopy, and MongoDB unordered BulkWriteAsync).Architecture Overview
FastIngest processes incoming tabular and line-delimited data using a decoupled producer-consumer pipeline that keeps memory usage constant O(1) regardless of file size:
Key Features
Concurrent Producer-Consumer Pipelining: Decouples CPU parsing/validation from database I/O socket operations using bounded
System.Threading.Channels with backpressure.Constant-Memory Streaming: Stream arbitrarily large files (gigabytes to tens of gigabytes) with strict $O(1)$ memory guarantees.
Line-Delimited JSON (NDJSON / JSONL): High-speed streaming parser over
System.IO.Pipelines.PipeReader and System.Text.Json.Utf8JsonReader with automatic format heuristics (.jsonl, .ndjson, or { content peeking).7 Native Database Sinks: Direct bulk protocol integrations for PostgreSQL (
COPY), SQL Server (SqlBulkCopy), MySQL (MySqlBulkCopy), SQLite (WAL), MongoDB (BulkWriteAsync), Azure Cosmos DB, and Elasticsearch.Validation Strategies:
FailFast: Immediately halts ingestion on the first invalid record.CollectAndContinue: Collects invalid row details and exports an error report CSV while allowing valid records to proceed.Fluent Pipeline API: Composable, chainable pipeline configuration with channel capacity tuning (
WithChannelCapacity), custom JSON serializer options (WithJsonOptions), and progress tracking.SemVer 2.0 Driven by Git Tags: Automated versioning via MinVer and seamless CI/CD publishing.
Performance Benchmarks
Official BenchmarkDotNet suite results comparing FastIngest streaming binary COPY against Entity Framework Core 9 (
net9.0, Apple M4, PostgreSQL 16 Alpine via Testcontainers):
Run the benchmarks yourself with
./benchmarks/run-benchmarks.sh. See full analysis in docs/benchmarks/performance.md.LLM & AI Discoverability
FastIngest provides machine-readable documentation endpoints adhering to the
llms.txt standard for AI coding assistants (such as Cursor, Windsurf, Claude Code, GitHub Copilot, and ChatGPT):Documentation Index:
https://fastingest.aebibtech.com/llms.txtFull Consolidated Context:
https://fastingest.aebibtech.com/llms-full.txtLicense
This project is licensed under the MIT License - see the LICENSE file for details.
Developed & Maintained by Paul Abib Camano (Aebibtech).
0 likes · 3 views · Oct 1, 2026
Developed FastIngest, a high-throughput data ingestion pipeline for .NET with database integrations.
Timeline
Sep 23, 2026 - Ongoing