An AI assistant was developed capable of intelligently solving problems and answering questions posed by users. An agent approach was taken and the assistant was given access to an API. Through careful description of each API endpoint combined with prompt engineering, the assistant was able to call sequences of API endpoints in order to successfully answer difficult questions.

The assistant was deployed as a websocket API to facilitate fluid two-way communication between client and server.

The project went through multiple iterations over the course of a year as AI technology developed and requirements grew.

Agent approach allowed the assistant to iterate through the think -> act -> evaluate loop in order to solve more complex problems by using several tools (e.g. API endpoints) in sequence.

Retrieval Augmented Generation (RAG) was used for smaller pieces of text that provided basic information and context. This technique could not be used for larger datasets up to 100 GB.

The assistant handled errors and unanswerable questions gracefully and fluidly from the user perspective.

Streaming was used to return the agent response as it was generated by the LLM and improve UX.

Earlier versions included semantic search over text embeddings stored in a Pinecone vector DB.

OpenAI - OpenAI models and API were used.

LangChain - Library used to implement all the AI assistant logic and behavior.

LangSmith - LLM visualization tool used for debugging.

* Image source: Alex Honchar - (article)