4 min read

·

Apr 7, 2025

--

Share

In our previous article, we introduced the Hushh PKM. If you haven’t read it, we highly recommend you to go through it

As we embarked on the journey to bring Hushh PKM to life, we explored numerous technical approaches to turn our vision into reality. Some of these ideas proved pivotal, while others were refined or set aside as we iterated towards the most effective solutions.

In our previous discussions, we touched on the concept and benefits of a Personal Knowledge Model. Now, it’s time to dive deeper into the technical heart of our project. This blog post will take you behind the scenes, exploring the prototypes we’ve tested, the approaches we’re considering, and the technical challenges we’ve overcome. Join us as we delve into the intricacies of building a cutting-edge PKM that seamlessly integrates with your digital world, ensuring privacy, efficiency, and innovation at every step.

Now, to begin with, at hushh, we have a combination of structured data in our relational database and unstructured data in the form of documents, texts, images etc.

Press enter or click to view image in full size

We explored multiple architectural approaches, each with its own trade-offs.

Prototype 1: Direct Retrieval-Augmented Generation (RAG)

Our first instinct was to apply RAG directly over our dataset. Initially, it worked well. But as the volume and complexity of our data grew, cracks began to show. Our dataset wasn’t just text-heavy — it was filled with dense numerical information and complex relationships between data points. RAG, in its vanilla form, struggled with this type of data and also lost important context when it broke information into chunks.

Additionally, PKM is being explored to allow others to interact with a user’s PKM, to know their basic preferences etc.For this, we needed to ensure fine control over who can access what.

(You’ve obviously got your close circle and then the not-so-close ones, so it should be up to the user to decide who gets what level of access.)

RAG wasn’t flexible enough to handle this kind of granular access management.

Press enter or click to view image in full size

Prototype 2: Next, we thought about using LLMs to generate GraphQL queries based on the user’s query. Here’s how it works:

The user asks a question.

The LLM, with its understanding of the database, figures out the category of data and picks the right table.

It then generates a GraphQL query based on the user’s request.

The data is fetched from the database using that query.

Finally, the LLM processes the data and provides an answer to the user’s original question.

Example:

Question: What all did I purchase last week ?

graphql_Response:

{receipt_radar_structured_data_duplicateCollection(

filter: {and: [{user_id:{eq:$hushh_id}},{Date: {gte: “27–09–2024”}}]}){

edges{

node{

purchase_category

} } } }

This graphql query is then used to retrieve data from database and the final answer can look something like:

Hey ! Based on the data we have, it looks like you purchased a black quilted sling bag and a pair of blue heels from Zara last week. Let me know if you would like to know anything else.

This approach worked pretty well. It retrieved accurate data and only the necessary information, without the extra complexity of managing a vector database. But there was one issue: categorization. In some cases, it’s not possible to answer a question using just one category of data.

Ps: (I experimented with a lot of different LLMs for the generation of graphQL queries. Surprisingly, even large models were not able to generate accurate queries without a lot of examples — more on that later.)

Now, we were on the lookout for something else. A few other challenges were bothering us:

Context loss over time: Once a conversation or query exceeds a certain length, important context can be lost. Since we’re dealing with personal user data, some of it might remain relevant even after 10 years.

Data deletion in vector databases: When managing vector databases, selectively deleting data, based on numerous factors, becomes really tricky.

Granular consent issues: It’s also difficult to implement granular consent in vector databases, which is crucial for our use case.

Before we dive into the approaches we’re currently considering, we’d love to hear your thoughts. What approaches would you take? Feel free to reach out to me on LinkedIn (here) if you’d like to discuss this further or contribute to the project.