Voice-Enabled AI Assistant for Fastn.ai

Umar Farooq

Software Engineer

AI Engineer

Fullstack Engineer

React

TypeScript

Artificial Intelligence

Fastn Voice Force

A voice-enabled AI assistant powered by the OpenAI Realtime API and OpenAI Agents SDK, featuring specialized agents for Fastn.ai tool integration and documentation assistance.

About the OpenAI Agents SDK

This project uses the OpenAI Agents SDK, a toolkit for building, managing, and deploying advanced AI agents. The SDK provides:

A unified interface for defining agent behaviors and tool integrations.

Built-in support for agent orchestration, state management, and event handling.

Easy integration with the OpenAI Realtime API for low-latency, streaming interactions.

Extensible patterns for multi-agent collaboration, handoffs, tool use, and guardrails.

For full documentation, guides, and API references, see the official OpenAI Agents SDK Documentation.

Features

This project includes two specialized AI agents:

Fastn Tools Agent: Dynamically fetches and executes Fastn.ai tools for various integrations including Google Workspace, Slack, and other productivity platforms.

Fastn Docs Agent: Provides comprehensive assistance with Fastn documentation, best practices, and getting started guidance.

Setup

Prerequisites

Node.js (v16 or higher)

OpenAI API key

Fastn.ai API credentials

Installation

Install Dependencies

npm install

Environment Configuration

Copy the sample environment file and configure your API keys:

cp .env.sample .env

Update .env with your credentials:

# OpenAI API Key - Required for voice agent functionality
OPENAI_API_KEY=your_openai_api_key_here

# Fastn Tools Agent - Get these from https://ucl.dev/
FASTN_API_KEY=your_fastn_tools_api_key
FASTN_SPACE_ID=your_fastn_tools_space_id

Note: The Fastn Docs Agent uses pre-configured credentials and doesn't require additional setup.

Getting API Credentials

OpenAI API Key: Get your API key from OpenAI Platform

Fastn Tools API Credentials: Visit Fastn UCL Platform to:

Create your account and workspace

Generate API keys for your tools integration

Get your Space ID from your workspace settings

The Fastn Docs Agent comes pre-configured with documentation access and doesn't require separate API credentials.

Running the Application

Start Development Server

npm run dev

Open Application

Navigate to http://localhost:3000

The app defaults to the Fastn Tools Agent

Switch between agents using the "Agent Set" dropdown in the bottom toolbar

Agent Architecture

Fastn Tools Agent

The Fastn Tools Agent dynamically fetches and executes tools from the Fastn.ai platform. It can:

Access Google Workspace (Docs, Sheets, Calendar)

Integrate with Slack for messaging and notifications

Handle various productivity and workflow automation tasks

Provide intelligent tool selection based on user requests

Fastn Docs Agent

The Fastn Docs Agent specializes in providing assistance with Fastn documentation and knowledge base. It can:

Answer questions about Fastn features and capabilities

Provide getting started guidance and tutorials

Explain best practices and usage patterns

Help troubleshoot common issues

Offer examples and code snippets

Benefits

Voice-First Interface: Natural conversation experience with low-latency responses

Dynamic Tool Access: Automatically discovers and uses the latest Fastn.ai tools

Specialized Knowledge: Dedicated documentation agent for comprehensive Fastn guidance

Seamless Integration: Direct connection to Fastn.ai platform and third-party services

Modular Design: Easy to extend with additional agents for specific use cases

Intelligent Routing: Automatically selects the right tools based on user intent

Configuration

The application is configured with two main agent sets:

Fastn Tools Agent (src/app/agentConfigs/fastnAgent.ts)

Dynamically fetches available tools from Fastn.ai platform

Handles complex tool execution workflows

Provides intelligent parameter mapping and validation

Fastn Docs Agent (src/app/agentConfigs/fastnDocsAgent.ts)

Connects to Fastn serviceAgent API for documentation queries

Provides contextual help and examples

Assists with onboarding and troubleshooting

Adding Custom Agents

To add your own agent:

Create a new agent configuration in src/app/agentConfigs/

Add the agent to the allAgentSets mapping in src/app/agentConfigs/index.ts

Update the UI selector options in src/app/components/BottomToolbar.tsx

API Integration

The application integrates with Fastn.ai through two main endpoints:

Tools API: https://live.fastn.ai/api/ucl/getTools - Retrieves available tools

Execution API: https://live.fastn.ai/api/ucl/executeTool - Executes tools with parameters

ServiceAgent API: https://live.fastn.ai/api/v1/serviceAgent - Queries documentation

Usage

Interacting with Fastn Tools Agent

"Create a new Google Doc called 'Meeting Notes'"

"Send a Slack message to the team channel"

"Schedule a meeting for tomorrow at 2pm"

"Add a new row to my project tracking sheet"

Interacting with Fastn Docs Agent

"How do I get started with Fastn?"

"What integrations does Fastn support?"

"Show me examples of workflow automation"

"Help me troubleshoot my API connection"

Output Guardrails

Assistant messages are checked for safety and compliance before they are shown in the UI. The guardrail call now lives directly inside src/app/App.tsx: when a response.text.delta stream starts we mark the message as IN_PROGRESS, and once the server emits guardrail_tripped or response.done we mark the message as FAIL or PASS respectively. If you want to change how moderation is triggered or displayed, search for guardrail_tripped inside App.tsx and tweak the logic there.

Navigating the UI

You can switch between agent sets using the "Agent Set" dropdown in the bottom toolbar.

The conversation transcript is on the left, including tool calls, tool call responses, and agent actions. Click to expand non-message elements.

The event log is on the right, showing both client and server events. Click to see the full payload.

On the bottom toolbar, you can disconnect, toggle between automated voice-activity detection or push-to-talk, turn off audio playback, toggle logs, and select different agents.

Resources

Fastn Platform

Fastn UCL Platform - Get your API keys and manage your Fastn workspace

Fastn Documentation - Complete guide to Fastn features and integrations

Developer

Umar Farooq LinkedIn - Connect with the developer

Umar Farooq Portfolio - View portfolio and projects

OpenAI

OpenAI Platform - Manage your OpenAI API keys

Realtime API Documentation - Learn about the Realtime API

Contributing

This project demonstrates voice-enabled AI agents for Fastn.ai integration. Feel free to extend it with additional agents, tools, or integrations that enhance the Fastn workflow automation experience.

For questions, suggestions, or collaboration opportunities, reach out via LinkedIn or check out the developer's portfolio.

Like this project

Posted Oct 6, 2025

Developed voice-enabled AI assistant for Fastn.ai using OpenAI SDK.

Likes

Views