Complete Project Documentation: Real-Time AI Voice Agent
Repository: GitHub – AI Voice Agent
1. Project Overview
The CSE Study247 AI Voice Agent is a real-time, bidirectional conversational voice assistant. Instead of using a traditional multi-step voice-bot pipeline—speech-to-text, LLM response generation, and a separate text-to-speech system—it uses Google’s Gemini Live API (models/gemini-2.5-flash-native-audio-preview) through the Pipecat framework.
Gemini Live processes incoming speech audio and generates streaming audio responses directly. This approach is designed to provide a natural, low-latency voice conversation in the browser.
What the Voice Agent Does
📚 Study & Website Support
Answers questions about subjects, notes, lectures, and resources including DSA, Python, Java, OS, DBMS, and Networks available on CSE Study247.
🎓 Exam Guidance
Provides syllabus information and PYQ references for GATE CS/DA, ISRO, DRDO, UGC NET, and SBTE Bihar Polytechnic Diploma examinations.
🗓️ Mentorship Booking
Checks available time slots and books free 20-minute, one-to-one career or study guidance sessions in SQLite.
🗣️ Barge-in / Interruptions
Users can speak while the assistant is talking; the assistant stops its response and listens to the new input.
2. High-Level Architecture
The flow connects the browser microphone, WebRTC transport, Pipecat processing pipeline, local tools, and Google Gemini Live.
[ User Microphone (Browser) ]
│ (WebRTC Audio Stream)
▼
[ Pipecat WebRTC Transport ]
│
▼
[ AudioTrace / VAD Layer ] <--- Silero VAD (speech boundaries)
│
▼
[ LLMContext / Frame Aggregator ]
│
├──────────► [ Google Gemini Live API ]
│ (Native Audio In → Native Audio Out)
│ │
│ (Function Calling) │ (Streaming Audio Frames)
▼ ▼
[ Local Tools & Knowledge Base ] [ Assistant Audio Output ]
- search_knowledge_base_tool │
- get_available_guidance_slots ▼
- book_free_guidance_session [ Browser Speaker Output ]
3. Key Features & Capabilities
| Feature | Description |
|---|---|
| Native Speech-to-Speech | Gemini Live preview models provide direct audio interaction without separate STT and TTS models. |
| WebRTC Transport | Uses WebRTC for browser-based, low-latency audio communication. |
| Local Knowledge Retrieval | Text search over local Markdown files to retrieve platform information. |
| Automated Slot Booking | SQLite-backed booking system with validation and collision prevention. |
| Turn Detection & Barge-In | Silero VAD detects speech turns and supports interruptions. |
| Spoken Filler Feedback | The agent can provide short conversational fillers while a tool or database query is running. |
4. Deep-Dive: How the Code Works
A. Orchestrator — main.py
main.py is the command-line entry point:
python main.pystarts the Pipecat development runner on port7860.python main.py bookingsprints a CLI table of booked guidance sessions.python main.py moss-indexbuilds an optional offline semantic vector index.
B. Agent Pipeline — src/csestudy247_voice_agent/agent.py
GeminiLiveLLMServiceconnects using the Google AI Studio API key, configures the Gemini Live model, and selects the voicePuck.LLMContextstores the system instruction and available tools.LLMContextAggregatorPairmaintains conversation context by aggregating user and assistant messages.run_pipelinestarts the asynchronous audio-processing pipeline.
C. Function Calling & Tools — src/csestudy247_voice_agent/tools.py
The LLM can call these Python functions:
search_knowledge_base_tool(query)searches Markdown knowledge files for verified exam, syllabus, or notes information.get_available_guidance_slots(time_preference)generates available morning, afternoon, or evening slots over the next seven days, excluding booked slots.book_free_guidance_session(caller_name, email, learning_goal, preferred_slot)validates booking details and stores the reservation indata/guidance_bookings.dbwith a unique booking ID.
D. Knowledge Base Engine — src/csestudy247_voice_agent/knowledge_base.py
The knowledge-base engine uses token-overlap and phrase matching to search files under data/knowledge_base/:
csestudy247_overview.md— site overview, contact emails, mobile-app information, and exam resources.csestudy247_courses_and_roadmaps.md— roadmaps for DSA, AI/ML, Voice Agents, Web Development, and FAQ answers.
5. Prerequisites
- Operating System: Windows 10/11, macOS, or Linux.
- Python: Version 3.12. Verify with
python --version. - Google AI Studio API Key: Obtain a key from Google AI Studio.
- Microphone and Browser: Chrome, Edge, Brave, or Firefox.
6. Step-by-Step Installation & Setup
Step 1: Clone the Repository
Open PowerShell, Command Prompt, or a terminal and clone the repository:
git clone https://github.com/sanjit507/AI-Voice-Agent.git
cd AI-Voice-Agent/AI_VoiceStep 2: Create a Python 3.12 Virtual Environment
Windows:
python -m venv .venv
.venvScriptsactivateLinux / macOS:
python3 -m venv .venv
source .venv/bin/activateStep 3: Install Dependencies
Install the project in editable mode with development dependencies:
pip install -e ".[dev]"Alternatively, if using uv:
uv sync --devStep 4: Configure Environment Variables
Copy .env.template to .env.
Windows:
copy .env.template .envLinux / macOS:
cp .env.template .envOpen the .env file and add your API key and configuration:
GOOGLE_API_KEY=your_actual_google_api_key_here
GOOGLE_GEMINI_MODEL=models/gemini-2.5-flash-native-audio-preview-12-2025
GOOGLE_GEMINI_VOICE=Puck.env file to GitHub. The repository’s .gitignore is intended to exclude .env and .venv/.7. Running the Voice Agent
Option 1: Start the Browser Voice Bot
Start the server:
python main.pyOr with uv:
uv run python main.py
- Wait for the terminal to show the local development URL, such as
http://localhost:7860. - Open
http://localhost:7860in your browser. - Click the connect/start button and allow microphone access when prompted.
- The assistant should greet you: “Hello! Welcome to CSE Study247. How can I help you today?”
Option 2: View Guidance Bookings in the Terminal
To view sessions booked through the voice agent:
python main.py bookings
8. Testing & Sample Voice Prompts
Speak clearly into your microphone and try the following sample requests.
| Intent | Sample Prompt | Expected Action |
|---|---|---|
| About Platform | “What is CSE Study247 and what do you offer?” | Summarizes notes, exams, mobile app, and platform features from the knowledge base. |
| Notes & Subjects | “Do you provide handwritten notes for Data Structures and OS?” | Checks available subjects and may mention Love Babbar / Striver sheets. |
| Exam Questions | “Can I get syllabus and question papers for GATE and ISRO?” | Explains relevant syllabus and question-paper resources. |
| Bihar Polytechnic | “Do you have notes for SBTE Bihar Computer Science diploma?” | Explains semester syllabus, lab programs, and notes resources. |
| Modern Roadmaps | “Can I learn how to build AI Voice Agents here?” | Describes tutorials covering Pipecat, Gemini Live, and LangChain. |
| Book Session | “I want to book a free guidance session.” | Starts a multi-turn flow to collect learning goal, preferred slot, name, and email. |
| Contact Support | “How can I contact the support team?” | Provides csestudy247help@gmail.com and the website contact form. |
9. Project Directory & File Guide
AI_Voice/
├── main.py # Primary CLI controller
├── pyproject.toml # Package definition & dependencies
├── .env.template # Safe credentials template
├── data/
│ ├── guidance_bookings.db # SQLite booking database
│ └── knowledge_base/ # Markdown files searched by the bot
│ ├── csestudy247_overview.md
│ ├── csestudy247_courses_and_roadmaps.md
│ └── dsa_python_course_full_curriculum.md
├── src/
│ └── csestudy247_voice_agent/
│ ├── agent.py # Gemini Live speech pipeline
│ ├── tools.py # Search & booking tools
│ ├── knowledge_base.py # Local text search
│ ├── pipeline.py # WebRTC transport & turn strategies
│ ├── bookings.py # SQLite booking CLI
│ ├── moss.py # Optional MOSS semantic retrieval
│ ├── moss_index.py # Optional MOSS indexer
│ └── config.py # Paths & environment configuration
└── stages/ # 11 educational pipeline scripts
10. Troubleshooting & FAQ
Error: Missing environment variable GOOGLE_API_KEY
Cause: The .env file is missing or does not contain the API key.
Solution: Create the .env file inside the AI_Voice/ directory and add the key from Google AI Studio.
Browser shows “Connection Failed” or no audio
Cause: Microphone permissions may be blocked, or WebRTC may fail to connect to localhost.
Solution: Allow microphone access in the browser’s site settings. Chrome or Edge are recommended for testing.
Can the agent speak languages other than English?
Gemini Live supports multilingual conversation. The documentation indicates that users can converse in Hindi or Hinglish and receive conversational responses.
How can I add more study notes to the knowledge base?
Create a new Markdown (.md) file inside AI_Voice/data/knowledge_base/ containing the information. The search tool can retrieve content from knowledge-base files on subsequent queries.
View the GitHub repository