Airtable Powered AI Chatbot

Asif Farhan Khan

AI Chatbot Developer

AI Application Developer

AI Developer

Airtable

ChatGPT

Python

Artificial Intelligence

An immensely promising addition to our website. This AI Chatbot is turning a boring Airtable data into an Airtable on Steroid! Loved his collaboratory approach to work!

Watch on YouTube

Client's Objective

Chatbot Use Cases:

Technical Requirements

Solution Design

Data Auditing & Challenges

Challenges

Deliverables

Development Stages & ETA

Preprocessing Data

Engine development

Fine Tuning the Model

Interface development

Cloud deployment

Ideas to Implement

Development Process

Features

Client's Objective

I am looking for a chatbot to use our data in airtable and provide answers to clients based on it. For example we have: a list of companies and the category they belong, so chat can be what companies are in category x or what category does company y belong. We have a large list 10k or so rows.

we have a correlation table between different companies. So they can ask who is correlated to company x and the bot shows the answer by listing the top 5 that are correlated. The data is also in airtable.

This is the sort of thing I am looking at using a bot for. Saves us have lots of cross tabs and pulldowns. The visitor can just type and get the specific answer they are after.

Chatbot Use Cases:

what e-commerce platform is most popular for ecom companies?

what CRM system is most used by e-commerce companies?

what crm system do non e-commerce companies use?

what are the 3 biggest differences between those that use e-commerce and those that don't?

what are the most important factors when selecting a new digital tool for those that use e-com?

how is this different to those that do not use e-commerce?

What is the main challenges when working with suppliers such as ad agencies or system integrators?

how do those that use e-com rate their e-commerce system?

are those using e-commerce more likely to be in business longer than the rest of companies in the sample?

what industry are those using e-commerce more likely to be in?

Technical Requirements

Airtable API Key

Airtable Base ID

Airtable Table ID

Solution Design

A pop-up style AI chatbot that is trained on the Airtable's survey data and fetched using client provided access with API key to answer queries related to the table data in a conversational manner using OpenAI's ChatGPT-4o, with intelligent research capablities designed and embeded to client's custom website using a simple HTML iframe.

Data Auditing & Challenges

As a marketing company, The client has carried out a survey to understand the needs of various companies utilising social media also comparing responses of ecommerce and non ecommerce background. The data encompassed of a survey questions such as:

Challenges

The training data for the chatbot is about 1 GB of data, making it harder for the bot to process it efficiently [Solved]

The data on the airtable was very unorganised and had multiple rows for a single survey question, each row pertaining for one option which has to be cleaned and standardised [Solved]

Some of the text contained underscores to as a question option seperator and some used ' -', the challenge was that a lot of these symbols were used even as a part of the sentence [Solved]

As the rating question (1-10) didn't have any options these had to be reformatted to adjust to the existing data [Solved]

There was an issue of hallucination where the AI confused several parts of the data to be something else like considered "Offers" as an industry. [Solved]

Deliverables

Airtable to be used as the knowledge base for the AI Chatbot

Chatbot would ideally be embeded on client's website (built with softr)

Once the chatbot is setup, It may have two version; an open version and an authentication based

Development Stages & ETA

Preprocessing Data - 2 Days

Engine development - 3 Days

Fine Tuning the Model - 2 Days

Interface development - 1 Day

Cloud deployment - 2 Days

Final Delivery : End to end in 10 Days

Preprocessing Data

This is the backend stage where I audited the data both manually and programitically to understand how we can transform it to standardise it for the usage, removing any garbage values, or redundant data.

Engine development

Once the Engine development is done, the most basic functionalities can be used and will be good for testing. However this will contain a number of issues and a thorough testing can determine how much improvement is required in the data as well as the model.

Fine Tuning the Model

Based on the testing on the above stage all shortcoming of the data and the program will be handled to make it efficient and error free. this will ensure robustness of the product in terms of security (data leak), chat bot manipulation etc

Interface development

Then we move towards interface development which will determine, how we would have the chatbot to interact with users, and develop chat box, output style, additional functionalities like buttons, tabs etc

Cloud deployment

Finally we will move ahead with deployment of the product and with this it will be hosted as a web app with a dedicated URL and a server which will allow anyone to access it on the web. We can then go ahead and expand this if you require authentication and advanced features.

Ideas to Implement

Turn the Survey data into a dashboard

Have the chatbot display graphs and charts

Development Process

The first process involved data wrangling where the raw unsorted survey data were represented as a unique row for each otption to the survey question, This had to be sorted, A nested python list was used to contain all the unique_question after stipping poorly formatted suffix and garbage values. Within each element of the list, was a dictionary of optiona name as key and the polling percentage as yet another dictionary :

[
{...},
{"How much of the rest of your digital tools are integrated with the centre?": [
      {
            "80% or more": {
                "% of total sample using Ecom": 0.55,
                "% of those using Ecom": 0.125,
                "% of those not using Ecom": 0.08035714285714286,
                "E-com Vs non-Ecom. + more likely - less likely": 0.04464285714285714
            }
        },
        {
            "less than 20%": {
                "% of total sample using Ecom": 0.22448979591836735,
                "% of those using Ecom": 0.125,
                "% of those not using Ecom": 0.3392857142857143,
                "E-com Vs non-Ecom. + more likely - less likely": -0.2142857142857143
            }
        }
    ]
},    
{...}
]```

Discraded all rows that did not carry any value or had missing data so as to increase the output accuracy and combat hallucination from the chatbot, The data that were rating based had to be reformatted to a specific structure for the bot to be able to distinguish

A chatbot engine was designed that would internally handle all the data pre-processing, training, input, analysing data and response.

A pop-up style chatbot was designed connected to the chatbot engine with user friendly and easy to manage interface

Finally the client was delivered with a clean, smart and intelligent chatbot that answered user's query in a conversational form easy to comprehend. All delivered just as an iFrame that can easily be embedded to any website like wordpress, wix or custom webapp!

Features

The AI chatbot is powered by cutting edge capablities of OpenAI's ChatGPT-4o

The knowledge base is solely hosted on Airtable

The data is not static and is refreshed every 6 Hrs

Rapid Delivery within 10 Days

Weekly summary of chat responses automated to be sent to the client for improving data

Like this project

Posted Jul 29, 2024

This is how you turn a boring airtable data into an engaing and interactie chatbot! Interactions on chatbot saved Edward numerous hours of data crunching!

Likes

Views