Airtable Powered AI Chatbot
Asif Farhan Khan
Like this project
Posted Jul 29, 2024
This is how you turn a boring airtable data into an engaing and interactie chatbot! Interactions on chatbot saved Edward numerous hours of data crunching!
An immensely promising addition to our website. This AI Chatbot is turning a boring Airtable data into an Airtable on Steroid! Loved his collaboratory approach to work!
Client's Objective
I am looking for a chatbot to use our data in airtable and provide answers to clients based on it. For example we have: a list of companies and the category they belong, so chat can be what companies are in category x or what category does company y belong. We have a large list 10k or so rows.
we have a correlation table between different companies. So they can ask who is correlated to company x and the bot shows the answer by listing the top 5 that are correlated. The data is also in airtable.
This is the sort of thing I am looking at using a bot for. Saves us have lots of cross tabs and pulldowns. The visitor can just type and get the specific answer they are after.
Chatbot Use Cases:
what e-commerce platform is most popular for ecom companies?
what CRM system is most used by e-commerce companies?
what crm system do non e-commerce companies use?
what are the 3 biggest differences between those that use e-commerce and those that don't?
what are the most important factors when selecting a new digital tool for those that use e-com?
how is this different to those that do not use e-commerce?
What is the main challenges when working with suppliers such as ad agencies or system integrators?
how do those that use e-com rate their e-commerce system?
are those using e-commerce more likely to be in business longer than the rest of companies in the sample?
what industry are those using e-commerce more likely to be in?
Technical Requirements
Airtable API Key
Airtable Base ID
Airtable Table ID
Solution Design
A pop-up style AI chatbot that is trained on the Airtable's survey data and fetched using client provided access with API key to answer queries related to the table data in a conversational manner using OpenAI's ChatGPT-4o, with intelligent research capablities designed and embeded to client's custom website using a simple HTML iframe.
Data Auditing & Challenges
As a marketing company, The client has carried out a survey to understand the needs of various companies utilising social media also comparing responses of ecommerce and non ecommerce background.
The data encompassed of a survey questions such as:
Challenges
The training data for the chatbot is about 1 GB of data, making it harder for the bot to process it efficiently [Solved]
The data on the airtable was very unorganised and had multiple rows for a single survey question, each row pertaining for one option which has to be cleaned and standardised [Solved]
Some of the text contained underscores to as a question option seperator and some used ' -', the challenge was that a lot of these symbols were used even as a part of the sentence [Solved]
As the rating question (1-10) didn't have any options these had to be reformatted to adjust to the existing data [Solved]
There was an issue of hallucination where the AI confused several parts of the data to be something else like considered "Offers" as an industry. [Solved]
Deliverables
Airtable to be used as the knowledge base for the AI Chatbot
Chatbot would ideally be embeded on client's website (built with softr)
Once the chatbot is setup, It may have two version; an open version and an authentication based
Development Stages & ETA
Preprocessing Data - 2 Days
Engine development - 3 Days
Fine Tuning the Model - 2 Days
Interface development - 1 Day
Cloud deployment - 2 Days
Final Delivery : End to end in 10 Days
Preprocessing Data
This is the backend stage where I audited the data both manually and programitically to understand how we can transform it to standardise it for the usage, removing any garbage values, or redundant data.
Engine development
Once the Engine development is done, the most basic functionalities can be used and will be good for testing. However this will contain a number of issues and a thorough testing can determine how much improvement is required in the data as well as the model.
Fine Tuning the Model
Based on the testing on the above stage all shortcoming of the data and the program will be handled to make it efficient and error free. this will ensure robustness of the product in terms of security (data leak), chat bot manipulation etc
Interface development
Then we move towards interface development which will determine, how we would have the chatbot to interact with users, and develop chat box, output style, additional functionalities like buttons, tabs etc
Cloud deployment
Finally we will move ahead with deployment of the product and with this it will be hosted as a web app with a dedicated URL and a server which will allow anyone to access it on the web. We can then go ahead and expand this if you require authentication and advanced features.
Ideas to Implement
Turn the Survey data into a dashboard
Have the chatbot display graphs and charts
Development Process
The first process involved data wrangling where the raw unsorted survey data were represented as a unique row for each otption to the survey question, This had to be sorted, A nested python list was used to contain all the unique_question after stipping poorly formatted suffix and garbage values. Within each element of the list, was a dictionary of optiona name as key and the polling percentage as yet another dictionary :
Discraded all rows that did not carry any value or had missing data so as to increase the output accuracy and combat hallucination from the chatbot, The data that were rating based had to be reformatted to a specific structure for the bot to be able to distinguish
A chatbot engine was designed that would internally handle all the data pre-processing, training, input, analysing data and response.
A pop-up style chatbot was designed connected to the chatbot engine with user friendly and easy to manage interface
Finally the client was delivered with a clean, smart and intelligent chatbot that answered user's query in a conversational form easy to comprehend. All delivered just as an iFrame that can easily be embedded to any website like wordpress, wix or custom webapp!
Features
The AI chatbot is powered by cutting edge capablities of OpenAI's ChatGPT-4o
The knowledge base is solely hosted on Airtable
The data is not static and is refreshed every 6 Hrs
Rapid Delivery within 10 Days
Weekly summary of chat responses automated to be sent to the client for improving data