Slack & Discord Conversational Knowledge Platform

Turning Discord & Slack conversations into searchable AI-powered knowledge.

Role: Backend Engineer (Sole Developer)

Stack: TypeScript • Node.js • Express • MongoDB • Elasticsearch • OpenAI • Stripe • Docker

Knowledge Sweeper automatically connects to Slack and Discord workspaces, ingests conversations, enriches them using AI, and transforms thousands of chat messages into searchable knowledge, daily digests, action items, and team insights.

I built the backend from scratch, covering everything from authentication and third-party integrations to AI processing, search infrastructure, scheduled jobs, billing, and the API powering the entire platform.

Teams constantly lose context because:

Important decisions disappear in long threads.

Valuable links become impossible to find.

Action items never make it into project tools.

New employees struggle to discover historical knowledge.

The platform needed to automatically organize years of conversations while remaining reliable even when external APIs or AI services became unavailable.

I designed a backend that continuously syncs Slack and Discord conversations, processes them asynchronously, enriches them with AI, indexes them for search, and generates intelligent summaries without slowing down the user experience.

Instead of processing AI requests during API calls, messages enter an enrichment pipeline where NLP and LLM processing happens in the background. This keeps the application responsive regardless of workspace size.

The backend follows a layered architecture separating routes, controllers, services, scheduled jobs, and external integrations.

Key characteristics include:

Modular service architecture

AI processing pipeline

Elasticsearch-powered search

MongoDB as the source of truth

Scheduled synchronization jobs

Fault-tolerant external integrations

Stripe subscription management

One implementation detail I paid particular attention to was graceful degradation. If OpenAI or Elasticsearch becomes unavailable, the platform continues functioning using local NLP analysis and MongoDB text search instead of failing entirely.

The onboarding flow is enforced entirely by the backend.

Users:

Register

Verify OTP

Connect Slack or Discord

Select channels

Configure digest preferences

Rather than trusting frontend state, every protected endpoint validates onboarding progress server-side and redirects users back to the exact step they left.

This prevents bypassing required setup while creating a seamless resume experience.

One of the biggest engineering decisions was moving AI work completely off the request path.

Instead of waiting for multiple LLM requests during an API call:

Messages are stored immediately.

They enter an indexed processing queue.

Background jobs process batches.

Results are merged with local NLP analysis.

Search indexes update automatically.

This allows thousands of imported messages to be processed without impacting response times.

The pipeline combines:

Local NLP

Sentiment analysis

Entity extraction

AI summaries

Decision detection

Action item extraction

Semantic embeddings

If any AI request fails, processing continues using local analysis instead of stopping the entire batch.

Search uses a layered strategy.

Primary search is powered by Elasticsearch, providing:

Full-text search

Highlighting

Faceted filters

Similar content recommendations

If Elasticsearch is unavailable, requests automatically fall back to MongoDB's text search while keeping the API response identical, allowing the frontend to continue working without changes.

The platform generates daily and weekly knowledge digests summarizing the most important conversations across connected workspaces.

Rather than making one AI request for every message, summaries are generated once per user using only the highest-value conversations.

This dramatically reduces AI costs while producing concise reports highlighting:

Key decisions

Action items

Important discussions

Shared files

Useful links

If I continued developing the platform, my next priorities would be:

Timezone-aware digest scheduling

Webhook-based Slack synchronization for lower latency

ObjectId normalization across collections

Dedicated queue infrastructure for higher processing volumes

The system processes thousands of imported messages asynchronously while keeping API response times fast. It integrates 5 external services (Slack, Discord, OpenAI, Elasticsearch, Stripe) with graceful fallbacks for each, so users always get useful results instead of errors.

The backend was designed around one principle: never let an external service failure become a user-facing failure. If OpenAI is down, local NLP picks up. If Elasticsearch is down, MongoDB text search takes over. The platform keeps working.