FastIngest is a high-throughput, constant-memory bulk ingestion pipeline for .NET (CSV, XLSX, and NDJSON/JSONL to PostgreSQL, SQL Server, MySQL, SQLite, MongoDB, Cosmos DB, and Elasticsearch). Designed for enterprise workloads processing millions of rows without memory spikes, FastIngest leverages zero-allocation streaming readers, concurrent producer-consumer bounded channels, fluent validation, and native database bulk protocols (such as PostgreSQL binary COPY, SQL Server SqlBulkCopy, and MongoDB unordered BulkWriteAsync).

FastIngest processes incoming tabular and line-delimited data using a decoupled producer-consumer pipeline that keeps memory usage constant O(1) regardless of file size:

Concurrent Producer-Consumer Pipelining: Decouples CPU parsing/validation from database I/O socket operations using bounded System.Threading.Channels with backpressure.

Constant-Memory Streaming: Stream arbitrarily large files (gigabytes to tens of gigabytes) with strict $O(1)$ memory guarantees.

Line-Delimited JSON (NDJSON / JSONL): High-speed streaming parser over System.IO.Pipelines.PipeReader and System.Text.Json.Utf8JsonReader with automatic format heuristics (.jsonl, .ndjson, or { content peeking).

7 Native Database Sinks: Direct bulk protocol integrations for PostgreSQL (COPY), SQL Server (SqlBulkCopy), MySQL (MySqlBulkCopy), SQLite (WAL), MongoDB (BulkWriteAsync), Azure Cosmos DB, and Elasticsearch.

Validation Strategies:

Fluent Pipeline API: Composable, chainable pipeline configuration with channel capacity tuning (WithChannelCapacity), custom JSON serializer options (WithJsonOptions), and progress tracking.

SemVer 2.0 Driven by Git Tags: Automated versioning via MinVer and seamless CI/CD publishing.

Official BenchmarkDotNet suite results comparing FastIngest streaming binary COPY against Entity Framework Core 9 (net9.0, Apple M4, PostgreSQL 16 Alpine via Testcontainers):

FastIngest provides machine-readable documentation endpoints adhering to the llms.txt standard for AI coding assistants (such as Cursor, Windsurf, Claude Code, GitHub Copilot, and ChatGPT):

Documentation Index: https://fastingest.aebibtech.com/llms.txt

Full Consolidated Context: https://fastingest.aebibtech.com/llms-full.txt

This project is licensed under the MIT License - see the LICENSE file for details.

Developed & Maintained by Paul Abib Camano (Aebibtech).