For USAID I created a product spec for a narrative and sentiment-based prototype that allowed non-technical stakeholders explore both Instagram and Youtube comments based on freeform queries.

Consulting

Data acquisition and data lake requirements

Exploratory data analysis and modeling of data

Translation of Arabic content to English using Cohere and OpenAI APIs

Insights presentations to key stakeholders

Processing of comment data to vectorized format for retrieval automated generation (RAG) with agent workflows using OpenAI

Development of prototype application that dynamically built data visualization modules based on input of user

Summarization features of comments and allowing users to query / ask questions about subsets of data

I authored several documents including vendor evaluation from which to source our data from. After meeting with 5+ vendors, we decided that it was more cost effective to build our own data acquisiton around select providers within RapidAPI ecosystem.

Throughly evaluating API usage costs was an essential part of scoping this project before taking on any data acquisition. This not only helped define accurate budget projections but also about how much engineering time would be needed and data processing time based on these assessments.

Since most content was in Arabic, we had to be focused on not only the technology but how to translate 1.6M+ comments within a 2 week period given the rate limits of OpenAI. During the process, I ran into some downgraded performance unexpectedly with the clients API key. This prompted a last minute evaluation and review of many different AI solutions. Eventually this lead me to Cohere which affordably provided excellent translations with 10x the throughput in API limits.

Upon acquiring our initial data samples, I outlined and preseted key challenges that impacted the project but also being mindful of where most value could be had with more focus and explaining tradeoffs of executing on them.

We looked at seasonal patterns, like vs comment distributions, but also low level details about when people most posted comments which provided recommendations later on. While I can't elaborate on the specific insights, these types of reviews are essential for setting baseline assumptions and talking with stakeholders.

Using a vector database, ChromaDB, makes it possible to search comments in plain language. This type of database enabled the searching of custom data (social media comments) in context of any language model request so the end user can query and perform tasks against any of the data in question.

Search IG and Youtube for specific phrases or keywords

Produce data visualizations related to the data to quickly understand key characteristics

Summarize the data succinctly with AI

Users can ask / query the subset of data further such as "What is this 'EUD' people are talking about?" (in this case, Eud is a holiday in Jordan and the AI could explain what it was and how it was being used)