An independent AI engineering project by Ugo Chukwu · Etherlabs Role: Dataset design, model training, evaluation and release engineering Stack: Python, Llama 3.1 8B, Unsloth, QLoRA, PEFT, Kaggle, Hugging Face Status: Educational baseline with public code and model artifacts

I built and released a domain-specific adapter for Llama 3.1 8B, then compared it with a pinned GPT-4.1 model on the same domain tasks. On 200 balanced simulated transaction cases, the adapter achieved 89.66% classification F1 versus 63.80% for GPT-4.1. The same evaluation exposed weaknesses in explanation quality and unfamiliar regulatory questions.

The result is Brain #1 of my IntelligenceOS learning programme: a model component with documented training decisions, inspectable evidence and explicit limits. It is an educational release, not a production risk engine.

A general-purpose model can produce a convincing risk assessment without consistently following a domain's categories, evidence requirements or response format. I wanted to find out what a small amount of targeted training could improve—and what still needed to be handled outside the model.

I scoped the work to three tasks: classify simulated transactions, assess risk from supplied facts, and answer bounded regulatory questions. Success meant being able to trace a result back to the data, training configuration, model output and scoring decision. A completed training job would not be enough.

I prepared a 1,900-row training set: 800 classification examples, 940 authored risk examples and 160 regulatory instruction examples. The classification inputs used simulated transaction data; the risk cases were authored, and regulatory material had recorded source provenance. This was not a deployment using real customer decisions.

I fine-tuned a Llama 3.1 8B Instruct base using a four-bit QLoRA adapter. The selected run used rank 16, a learning rate of 0.0001, batch size 2, gradient accumulation of 8 and 238 optimizer steps. I selected the final-step adapter rather than searching checkpoints for the best test score.

The training protocol fixed the prompt format, answer supervision and end-of-sequence handling. I checked tokenization, finite gradients and changed adapter weights. A saved Kaggle version preserved the run; the release includes its configuration, training receipt and recorded loss curve.

An earlier saved version failed during setup. I retained that failure in the project history, corrected the setup and completed Version 2. Successful execution and capability qualification remained separate decisions throughout.

I ran a matched comparison against gpt-4.1-2025-04-14 across 380 cases: 200 transaction classifications, 100 risk assessments, 40 taught-section regulatory questions and 40 questions from unfamiliar sections.

Both models received the same task information and comparable output requirements, without retrieval or tools. The comparison retained raw generations and scoring records. Semantic review used mixed, model-hidden packets and internal AI-assisted review. It was not an independent professional assessment.

The cases were already part of the project evaluation. This was a retrospective comparison on a bounded benchmark, not a new external validation set. Correlated variants, authored examples and format-sensitive scoring limit how far its results generalize.

Measure Domain adapter GPT-4.1 Classification F1 — 200 cases 89.66% 63.80% Risk-tier accuracy — 100 cases 96% 99% Supported risk rationale — 100 cases 66% 99% Taught-section regulatory joint correctness — 40 cases 97.5% 15% Unseen-section citation accuracy — 40 cases 0% 80% Unseen-section regulatory joint correctness — 40 cases 0% 7.5%

The classification advantage was meaningful within this benchmark: 25.85 percentage points in F1. It supports a specific statement about specialization on these simulated cases. It does not establish superiority across fraud work or over every GPT model.

The regulatory result needs more care. Joint correctness required the expected citation and substantive answer under the project's rubric. Performance was strong on taught sections and weak on unfamiliar sections. That contrast is evidence of limited transfer, not evidence that the adapter can reliably interpret regulations.

The adapter also passed the full risk-answer checklist more often, 61% versus 8%. I do not present that as a reasoning win: the checklist rewarded completeness against reference evidence fields, and GPT often omitted fields beyond the prompt's minimum. GPT's supported-rationale score was substantially higher.

The model's original strict capability qualification failed. I kept that result in the release.

Instead of continuing to train indefinitely or changing the failed result into a pass, I closed this phase as an educational baseline. The purpose was to demonstrate the complete process of domain adaptation, evaluation and evidence-based iteration. Production qualification remains future work.

This decision sharpened the next system boundary. A valid response format does not prove a sound explanation. A correct risk tier does not make every supporting claim true. And a citation-shaped answer does not establish that the cited rule supports it.

The next phase will put verification around the model: preserve the supplied facts and raw answer, check support for claims, keep uncertainty visible and require review before consequential actions. That verification layer is planned work, not a capability claimed for this release.

The public GitHub release contains the frozen runtime, inputs, saved outputs, review records, training evidence and checksums. A CPU replay reproduces the reported scores without starting a GPU or making API calls. It replays recorded evidence; it is not live inference.

The adapter is hosted separately on Hugging Face under the Etherlabs namespace, with its model card and Llama license materials. Running it requires the compatible base model and CUDA environment described in the release instructions. The adapter is reusable within the applicable license; it is not ownership of the underlying Llama base model.

This project demonstrates the engineering work behind a credible model experiment: defining the task, controlling the training contract, measuring a matched baseline, investigating failures and shipping a result that another person can inspect.