DocsStation All docs

AI Voice Agent

By csestudy247 · Updated November 20, 2026 · 7 min read

Complete Project Documentation: Real-Time AI Voice Agent

Repository: GitHub – AI Voice Agent

Python 3.12Pipecat AIGemini Live APIWebRTCSilero VADSQLite

1. Project Overview

The CSE Study247 AI Voice Agent is a real-time, bidirectional conversational voice assistant. Instead of using a traditional multi-step voice-bot pipeline—speech-to-text, LLM response generation, and a separate text-to-speech system—it uses Google’s Gemini Live API (models/gemini-2.5-flash-native-audio-preview) through the Pipecat framework.

Gemini Live processes incoming speech audio and generates streaming audio responses directly. This approach is designed to provide a natural, low-latency voice conversation in the browser.

What the Voice Agent Does

📚 Study & Website Support

Answers questions about subjects, notes, lectures, and resources including DSA, Python, Java, OS, DBMS, and Networks available on CSE Study247.

🎓 Exam Guidance

Provides syllabus information and PYQ references for GATE CS/DA, ISRO, DRDO, UGC NET, and SBTE Bihar Polytechnic Diploma examinations.

🗓️ Mentorship Booking

Checks available time slots and books free 20-minute, one-to-one career or study guidance sessions in SQLite.

🗣️ Barge-in / Interruptions

Users can speak while the assistant is talking; the assistant stops its response and listens to the new input.

2. High-Level Architecture

The flow connects the browser microphone, WebRTC transport, Pipecat processing pipeline, local tools, and Google Gemini Live.

[ User Microphone (Browser) ]
              │ (WebRTC Audio Stream)
              ▼
   [ Pipecat WebRTC Transport ]
              │
              ▼
    [ AudioTrace / VAD Layer ]  <--- Silero VAD (speech boundaries)
              │
              ▼
  [ LLMContext / Frame Aggregator ]
              │
              ├──────────► [ Google Gemini Live API ]
              │              (Native Audio In → Native Audio Out)
              │                     │
              │ (Function Calling) │ (Streaming Audio Frames)
              ▼                    ▼
[ Local Tools & Knowledge Base ] [ Assistant Audio Output ]
 - search_knowledge_base_tool            │
 - get_available_guidance_slots          ▼
 - book_free_guidance_session     [ Browser Speaker Output ]

3. Key Features & Capabilities

FeatureDescription
Native Speech-to-SpeechGemini Live preview models provide direct audio interaction without separate STT and TTS models.
WebRTC TransportUses WebRTC for browser-based, low-latency audio communication.
Local Knowledge RetrievalText search over local Markdown files to retrieve platform information.
Automated Slot BookingSQLite-backed booking system with validation and collision prevention.
Turn Detection & Barge-InSilero VAD detects speech turns and supports interruptions.
Spoken Filler FeedbackThe agent can provide short conversational fillers while a tool or database query is running.

4. Deep-Dive: How the Code Works

A. Orchestrator — main.py

main.py is the command-line entry point:

  • python main.py starts the Pipecat development runner on port 7860.
  • python main.py bookings prints a CLI table of booked guidance sessions.
  • python main.py moss-index builds an optional offline semantic vector index.

B. Agent Pipeline — src/csestudy247_voice_agent/agent.py

  1. GeminiLiveLLMService connects using the Google AI Studio API key, configures the Gemini Live model, and selects the voice Puck.
  2. LLMContext stores the system instruction and available tools.
  3. LLMContextAggregatorPair maintains conversation context by aggregating user and assistant messages.
  4. run_pipeline starts the asynchronous audio-processing pipeline.

C. Function Calling & Tools — src/csestudy247_voice_agent/tools.py

The LLM can call these Python functions:

  1. search_knowledge_base_tool(query) searches Markdown knowledge files for verified exam, syllabus, or notes information.
  2. get_available_guidance_slots(time_preference) generates available morning, afternoon, or evening slots over the next seven days, excluding booked slots.
  3. book_free_guidance_session(caller_name, email, learning_goal, preferred_slot) validates booking details and stores the reservation in data/guidance_bookings.db with a unique booking ID.

D. Knowledge Base Engine — src/csestudy247_voice_agent/knowledge_base.py

The knowledge-base engine uses token-overlap and phrase matching to search files under data/knowledge_base/:

  • csestudy247_overview.md — site overview, contact emails, mobile-app information, and exam resources.
  • csestudy247_courses_and_roadmaps.md — roadmaps for DSA, AI/ML, Voice Agents, Web Development, and FAQ answers.

5. Prerequisites

  • Operating System: Windows 10/11, macOS, or Linux.
  • Python: Version 3.12. Verify with python --version.
  • Google AI Studio API Key: Obtain a key from Google AI Studio.
  • Microphone and Browser: Chrome, Edge, Brave, or Firefox.

6. Step-by-Step Installation & Setup

Step 1: Clone the Repository

Open PowerShell, Command Prompt, or a terminal and clone the repository:

git clone https://github.com/sanjit507/AI-Voice-Agent.git
cd AI-Voice-Agent/AI_Voice

Step 2: Create a Python 3.12 Virtual Environment

Windows:

python -m venv .venv
.venvScriptsactivate

Linux / macOS:

python3 -m venv .venv
source .venv/bin/activate

Step 3: Install Dependencies

Install the project in editable mode with development dependencies:

pip install -e ".[dev]"

Alternatively, if using uv:

uv sync --dev

Step 4: Configure Environment Variables

Copy .env.template to .env.

Windows:

copy .env.template .env

Linux / macOS:

cp .env.template .env

Open the .env file and add your API key and configuration:

GOOGLE_API_KEY=your_actual_google_api_key_here
GOOGLE_GEMINI_MODEL=models/gemini-2.5-flash-native-audio-preview-12-2025
GOOGLE_GEMINI_VOICE=Puck
🔐 Security: Never commit your .env file to GitHub. The repository’s .gitignore is intended to exclude .env and .venv/.

7. Running the Voice Agent

Option 1: Start the Browser Voice Bot

Start the server:

python main.py

Or with uv:

uv run python main.py
  1. Wait for the terminal to show the local development URL, such as http://localhost:7860.
  2. Open http://localhost:7860 in your browser.
  3. Click the connect/start button and allow microphone access when prompted.
  4. The assistant should greet you: “Hello! Welcome to CSE Study247. How can I help you today?”

Option 2: View Guidance Bookings in the Terminal

To view sessions booked through the voice agent:

python main.py bookings

8. Testing & Sample Voice Prompts

Speak clearly into your microphone and try the following sample requests.

IntentSample PromptExpected Action
About Platform“What is CSE Study247 and what do you offer?”Summarizes notes, exams, mobile app, and platform features from the knowledge base.
Notes & Subjects“Do you provide handwritten notes for Data Structures and OS?”Checks available subjects and may mention Love Babbar / Striver sheets.
Exam Questions“Can I get syllabus and question papers for GATE and ISRO?”Explains relevant syllabus and question-paper resources.
Bihar Polytechnic“Do you have notes for SBTE Bihar Computer Science diploma?”Explains semester syllabus, lab programs, and notes resources.
Modern Roadmaps“Can I learn how to build AI Voice Agents here?”Describes tutorials covering Pipecat, Gemini Live, and LangChain.
Book Session“I want to book a free guidance session.”Starts a multi-turn flow to collect learning goal, preferred slot, name, and email.
Contact Support“How can I contact the support team?”Provides csestudy247help@gmail.com and the website contact form.

9. Project Directory & File Guide

AI_Voice/
├── main.py                           # Primary CLI controller
├── pyproject.toml                    # Package definition & dependencies
├── .env.template                    # Safe credentials template
├── data/
│   ├── guidance_bookings.db         # SQLite booking database
│   └── knowledge_base/              # Markdown files searched by the bot
│       ├── csestudy247_overview.md
│       ├── csestudy247_courses_and_roadmaps.md
│       └── dsa_python_course_full_curriculum.md
├── src/
│   └── csestudy247_voice_agent/
│       ├── agent.py                 # Gemini Live speech pipeline
│       ├── tools.py                 # Search & booking tools
│       ├── knowledge_base.py        # Local text search
│       ├── pipeline.py              # WebRTC transport & turn strategies
│       ├── bookings.py              # SQLite booking CLI
│       ├── moss.py                  # Optional MOSS semantic retrieval
│       ├── moss_index.py             # Optional MOSS indexer
│       └── config.py                # Paths & environment configuration
└── stages/                           # 11 educational pipeline scripts

10. Troubleshooting & FAQ

Error: Missing environment variable GOOGLE_API_KEY

Cause: The .env file is missing or does not contain the API key.

Solution: Create the .env file inside the AI_Voice/ directory and add the key from Google AI Studio.

Browser shows “Connection Failed” or no audio

Cause: Microphone permissions may be blocked, or WebRTC may fail to connect to localhost.

Solution: Allow microphone access in the browser’s site settings. Chrome or Edge are recommended for testing.

Can the agent speak languages other than English?

Gemini Live supports multilingual conversation. The documentation indicates that users can converse in Hindi or Hinglish and receive conversational responses.

How can I add more study notes to the knowledge base?

Create a new Markdown (.md) file inside AI_Voice/data/knowledge_base/ containing the information. The search tool can retrieve content from knowledge-base files on subsequent queries.

Project: CSE Study247 AI Voice Agent · Built with Pipecat and Google Gemini Live.
View the GitHub repository

Scroll to Top