A pipeline can return clean, well-formed, confident output and still be wrong, because it only saw part of the input.

I wrote up a postmortem on a contract-extraction tool built around that failure. A language model pulled renewal dates and notice periods from vendor contracts. A temporary character limit, added to keep test runs cheap, quietly cut off long documents before they reached the model. An amendment near the end of one contract changed the notice period, and the tool never saw it. No errors, no warnings. The model did what it was asked with the text it was given.

The part worth taking from it: most evaluation asks “is the extracted value correct?” Far fewer setups ask “did the system see the whole document?” Those are different questions, and a healthy-looking dashboard only answers the first.

Cheap guards that help:

- Fail loudly when input exceeds the limit instead of slicing it

- Log how much of each document was sent, next to its full length

- Make the model cite the passage each field came from

- Put the hardest cases, long documents with the key clause at the end, in the test set

Full postmortem here: https://medium.com/@nagatomopedro05/postmortem-the-contracts-pipeline-that-never-read-the-last-page-cf84251671c8