Building an impossible API

First, a bit of background. During my time at Waterlily, our core mission was predicting the long-term care future of a patient based on their survey answers using AI. We would then connect them with advisors who review their long-term plans and provide actionable feedback.

To cover those long-term care costs, patients often need to buy Long-Term Care (LTC) insurance. There are a lot of insurance providers out there, and they all have different APIs. At the time, we successfully had three of them integrated into our platform. My task was to integrate a fourth—an external insurance provider that presented a unique challenge.

This new provider had no public API for us to connect with. Usually, insurance companies will provide API access after we land a B2B contract with them. But we faced a classic Catch-22: we needed to show them a working integration on our platform to pitch them and land the contract in the first place.

If we couldn't integrate with them somehow, we wouldn't land the B2B contract, which meant we'd never get the API.

The insurance company did have a public-facing portal where users could manually input different variables and get a desired policy as an output. I opened the network tab, hoping to find some kind of hidden endpoint I could use. Nothing. The application was built with Java Maven, and all the heavy lifting was done server-side.

I even tried to reverse-engineer their calculations, but they had complex internal logic that prevented us from accurately recreating it. Even if we had asked them for the logic, building that complex calculator from scratch would have taken a massive amount of time and deviated from our core roadmap.

We only had one option left: web scraping.

The idea was to use a headless browser to simulate a user interacting with their portal and extract the desired policy data. I knew it would be fragile if they changed their UI, our integration would break. Furthermore, we would be limited to 100 policy extractions per day due to account rate limits.

I communicated these risks to the CEO, and we agreed that it was a necessary "today" solution. It was a calculated risk that would allow us to at least implement the feature, prove our concept, and get the conversation started.

I got to work. I initialized a new AWS Lambda instance coupled with a queue so the requests could be handled asynchronously. For the headless browser library, I chose Playwright and implemented the authentication logic.

The flow was designed so our internal API would receive the policy inputs, pass them to Playwright, and Playwright would mirror those exact inputs into the provider's portal.

However, the initial results were wrong. Even though we were passing the same inputs to our API that a user would put into the portal, the outputs didn't match. I dug deep into the Playwright traces and logs to figure out why.

I discovered that some inputs were failing to register due to timing issues on the provider's end. For example, when you selected a specific field, the portal required a brief rendering pause before the subsequent dependent fields were ready to accept input. I spent a lot of time brute-forcing the flow, implementing dependent timing logic and dynamic waits to ensure every single input was correctly registered. Eventually, it worked perfectly.

The endpoint was producing the correct outputs, but we had a new problem: it was incredibly slow. Because the Lambda was running the authentication logic on every single request, it was taking 1.5 minutes just to extract a single policy.

The provider's authentication actually stayed valid for roughly a day, so logging in on every request was highly inefficient. I consulted with a senior engineer, Rahul, and we brainstormed optimization strategies.

Playwright allows you to save the authentication and browser state to a file. The challenge was figuring out how to persist that state across different Lambda invocations. We also needed to ensure that parallel requests wouldn't break the session if the provider's portal logged out the first session when a second one authenticated, our system would collapse. If that were the case, we'd have to juggle multiple portal accounts.

After thorough testing, we confirmed that a single portal account could support parallel sessions. We decided to cache the Playwright authentication state in an AWS S3 bucket. Any new Lambda invocation could pull that state file and bypass the login screen entirely. We also added Playwright-level verifications to check if the loaded state was still valid; if the session had expired, it would invalidate the S3 cache and perform a fresh login.

To ensure the system never stalled during business hours, I set up a background automation job to refresh the session proactively based on the 24-hour expiration window.

This optimization slashed our processing time by 30 seconds, bringing the total time down to 1 minute for a fully automated, headless extraction.

The feature passed QA and we shipped it to production. The users were happy to have the new integration, the provider was thrilled to see their service working seamlessly on our platform, and ultimately, this "impossible" integration unblocked a massive 8-figure B2B deal for the company!