--- title: LLM Memory Architecture description: "Large Language Models (LLMs) are often perceived as having human-like memory because they can hold coherent conversations. However, technically, **LLMs are stateless mathematical functions**. This blog explores how developers bridge the gap between this inherent statelessness and the complex memory…" url: "https://csestudy247.com/docs/llm-memory-architecture/" updated: "2026-09-24" category: LangGraph --- # LLM Memory Architecture Large Language Models (LLMs) are often perceived as having human-like memory because they can hold coherent conversations. However, technically, **LLMs are stateless mathematical functions**. This blog explores how developers bridge the gap between this inherent statelessness and the complex memory systems required for production-grade AI agents using frameworks like LangGraph. ## 1. The Core Paradox: Why LLMs are Stateless At its fundamental level, an LLM at inference is a parameterized mathematical function: *y*=*f*(*θ*,*x*). - *θ*: Billions of fixed parameters (the model's "brain" trained on the internet). - *x*: The input tokens (your prompt). - *y*: The output tokens. Because *θ* is fixed after training and the function only looks at the current *x*, it has no "internal" memory of past interactions. If you tell an LLM your name in one call and ask for it in the next, it will fail unless that context is provided again ## 2. Short-Term Memory (STM): The Conversation Buffer To make an LLM "remember," we create a **Conversation Buffer**. Instead of sending just the new message (*x*2​), we concatenate the entire history (*x*1​,*y*1​,*x*2​) and send it as the new input. LangGraph Implementation: Checkpointers and Threads In LangGraph, STM is managed through **Checkpointers** and **Thread IDs**. - **Checkpointer**: Saves the state of the graph at every "superstep". - **Thread ID**: A unique identifier for a specific conversation session. Memory is typically **thread-scoped**, meaning it exists only within that specific conversation boundary. **Diagram: STM Workflow** %%{init: {'theme':'base', 'themeVariables': { 'fontSize':'15px', 'fontFamily':'Arial'}}}%% graph TD A["💬 User Message"]:::stepNode B["📝 Append to Thread History"]:::stepNode C["🤖 LLM Processing"]:::stepNode D["💾 Save to Checkpointer"]:::stepNode E["✨ Generate Response"]:::stepNode F["📤 Return to User"]:::stepNode A --> B B --> C C --> D D --> E E --> F classDef stepNode fill:#e6e6fa,color:#000,stroke:#6f42c1,stroke-width:2px,font-size:15px ```javascript # ========================================================== # Example 1: LangGraph WITHOUT Memory # ========================================================== from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langgraph.graph import StateGraph, START, MessagesState # Load OpenAI API key from .env file load_dotenv() # Create the LLM model = ChatOpenAI() # ---------------------------------------------------------- # Node Function # ---------------------------------------------------------- # This node receives the current state (messages), # sends it to the LLM, and returns the AI response. def call_model(state: MessagesState): response = model.invoke(state["messages"]) return {"messages": [response]} # ---------------------------------------------------------- # Build the Graph # ---------------------------------------------------------- builder = StateGraph(MessagesState) # Add one node builder.add_node("call_model", call_model) # Connect START → call_model builder.add_edge(START, "call_model") # Compile graph graph = builder.compile() # ---------------------------------------------------------- # First Conversation # ---------------------------------------------------------- graph.invoke( { "messages": [ { "role": "user", "content": "Hi! My name is Sanjit" } ] } ) """ Output Human : Hi! My name is Sanjit AI : Nice to meet you, Sanjit! """ # ---------------------------------------------------------- # Second Conversation # ---------------------------------------------------------- graph.invoke( { "messages": [ { "role": "user", "content": "What is my name?" } ] } ) """ Output Human : What is my name? AI : I don't know your name. Reason: Every invoke() starts a NEW conversation. The previous message ("My name is Sanjit") was NOT stored anywhere. ``` --- ```python # ========================================================== # Example 2: LangGraph WITH Memory # ========================================================== from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langgraph.graph import StateGraph, START, MessagesState from langgraph.checkpoint.memory import InMemorySaver # Load environment variables load_dotenv() # Create LLM model = ChatOpenAI() # ---------------------------------------------------------- # Node Function # ---------------------------------------------------------- def call_model(state: MessagesState): """ Receives all conversation messages, sends them to the LLM, returns the AI response. """ response = model.invoke(state["messages"]) return {"messages": [response]} # ---------------------------------------------------------- # Build Graph # ---------------------------------------------------------- builder = StateGraph(MessagesState) builder.add_node("call_model", call_model) builder.add_edge(START, "call_model") # ---------------------------------------------------------- # Create Checkpointer # ---------------------------------------------------------- # InMemorySaver stores conversation history # in RAM using a thread_id. checkpointer = InMemorySaver() # Compile graph with memory graph = builder.compile(checkpointer=checkpointer) # ---------------------------------------------------------- # Thread Configurations # ---------------------------------------------------------- config1 = { "configurable": { "thread_id": "thread-1" } } config2 = { "configurable": { "thread_id": "thread-2" } } # ---------------------------------------------------------- # First Message (Thread 1) # ---------------------------------------------------------- graph.invoke( { "messages": [ { "role": "user", "content": "Hi! My name is Sanjit." } ] }, config=config1, ) """ Output Human : Hi! My name is Sanjit. AI : Nice to meet you Sanjit. """ # ---------------------------------------------------------- # Ask Again in SAME Thread # ---------------------------------------------------------- graph.invoke( { "messages": [ { "role": "user", "content": "What is my name?" } ] }, config=config1, ) """ Output Human : What is my name? AI : Your name is Sanjit. """ # ---------------------------------------------------------- # Ask in a DIFFERENT Thread # ---------------------------------------------------------- graph.invoke( { "messages": [ { "role": "user", "content": "What is my name?" } ] }, config=config2, ) """ Output Human : What is my name? AI : I don't know your name. """ ``` ## 3. Solving the Context Window Crisis LLMs have a **Context Window**—a limit on the number of tokens they can process at once. As conversations grow, the buffer can exceed this limit, leading to hallucinations or errors. **Technique A: Trimming** We set a `max_token_limit` and keep only the most recent messages that fit within it, discarding the oldest ones. - **Pros**: Simple to implement using functions like `trim_messages`. - **Cons**: The LLM completely loses the context of early parts of the conversation. ```python # ========================================================== # LangGraph Memory Trimming Example # ========================================================== # This example demonstrates how to limit the conversation # history sent to the LLM using trim_messages(). # # Even though the complete conversation is stored in memory, # only the most recent messages that fit within MAX_TOKENS # are sent to the model. # ========================================================== from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langgraph.graph import StateGraph, START, MessagesState from langgraph.checkpoint.memory import InMemorySaver from langchain_core.messages.utils import ( trim_messages, count_tokens_approximately, ) # ---------------------------------------------------------- # Load OpenAI API Key # ---------------------------------------------------------- load_dotenv() # Create LLM model = ChatOpenAI() # ---------------------------------------------------------- # Maximum Tokens Allowed # ---------------------------------------------------------- # Only messages that fit within this budget # will be sent to the LLM. MAX_TOKENS = 150 # ---------------------------------------------------------- # Graph Node # ---------------------------------------------------------- def call_model(state: MessagesState): # Trim the conversation. # Strategy = "last" # Keep the latest messages until MAX_TOKENS is reached. messages = trim_messages( state["messages"], strategy="last", token_counter=count_tokens_approximately, max_tokens=MAX_TOKENS, ) # Print token count after trimming print("nCurrent Token Count ->", count_tokens_approximately(messages)) # Print messages actually sent to LLM print("nMessages Sent To LLMn") for message in messages: print(message.content) print("-" * 60) # Send trimmed conversation response = model.invoke(messages) return {"messages": [response]} # ---------------------------------------------------------- # Build Graph # ---------------------------------------------------------- builder = StateGraph(MessagesState) builder.add_node("call_model", call_model) builder.add_edge(START, "call_model") # ---------------------------------------------------------- # Add Memory # ---------------------------------------------------------- checkpointer = InMemorySaver() graph = builder.compile(checkpointer=checkpointer) # ---------------------------------------------------------- # Thread Configuration # ---------------------------------------------------------- config = { "configurable": { "thread_id": "chat-1" } } # ========================================================== # Conversation 1 # ========================================================== result = graph.invoke( { "messages": [ { "role": "user", "content": "Hi, my name is Sanjit." } ] }, config, ) print(result["messages"][-1].content) """ Output Current Token Count -> 10 Messages Sent To LLM Hi, my name is Sanjit. ------------------------------------------------------------ AI: Hello Sanjit! Nice to meet you. """ # ========================================================== # Conversation 2 # ========================================================== result = graph.invoke( { "messages": [ { "role": "user", "content": "I am learning LangGraph." } ] }, config, ) print(result["messages"][-1].content) """ Output Current Token Count -> 40 Messages Sent To LLM Hi, my name is Sanjit. AI: Hello Sanjit! I am learning LangGraph. ------------------------------------------------------------ AI: That's great! """ # ========================================================== # Conversation 3 # ========================================================== result = graph.invoke( { "messages": [ { "role": "user", "content": "Can you explain short-term memory?" } ] }, config, ) print(result["messages"][-1].content) """ Output Current Token Count -> 108 Messages Sent To LLM Hi, my name is Sanjit. AI: Hello Sanjit! I am learning LangGraph. AI: That's great! Can you explain short-term memory? ------------------------------------------------------------ AI: Short-term memory is ... """ # ========================================================== # Conversation 4 # ========================================================== result = graph.invoke( { "messages": [ { "role": "user", "content": "What is my name?" } ] }, config, ) print(result["messages"][-1].content) """ Output Current Token Count -> 8 Messages Sent To LLM What is my name? ------------------------------------------------------------ AI: I'm sorry, I don't know your name. """ # ========================================================== # View Complete Stored Memory # ========================================================== snapshot = graph.get_state(config) print("nComplete Conversation Stored In Memoryn") for message in snapshot.values["messages"]: print(message.content) print("-" * 80) ``` **Technique B: Summarization & Deletion** Instead of deleting old messages, we send them to another LLM to generate a **Summary**. We then pass this summary plus the most recent messages to the main LLM. This preserves the "essence" of the past without bloating the token count. ```python # ========================================================== # LangGraph Conversation Summarization Memory Example # ========================================================== # # This example demonstrates how LangGraph automatically # summarizes long conversations. # # Instead of sending the entire chat history to the LLM, # it: # # 1. Stores all messages. # 2. When conversation becomes long (>6 messages), # creates a summary. # 3. Deletes old messages. # 4. Keeps only the latest two messages. # 5. Uses the summary in future conversations. # # ========================================================== from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, RemoveMessage from langgraph.graph import MessagesState, StateGraph, START from langgraph.checkpoint.memory import InMemorySaver # ---------------------------------------------------------- # Load OpenAI API Key # ---------------------------------------------------------- load_dotenv() # Create LLM model = ChatOpenAI() # ========================================================== # Custom State # ========================================================== # # MessagesState already contains: # # messages # # We extend it by adding # # summary # # Final State # # { # "messages": [...], # "summary": "..." # } # # ========================================================== class ChatState(MessagesState): summary: str # ========================================================== # Summarization Node # ========================================================== # # This node runs only when the conversation # becomes longer than 6 messages. # # It creates/updates the summary and # removes old messages. # # ========================================================== def summarize_conversation(state: ChatState): # Previous summary existing_summary = state["summary"] # ------------------------------------- # Build Prompt # ------------------------------------- if existing_summary: prompt = ( f"Existing summary:n{existing_summary}nn" "Extend the summary using the new conversation above." ) else: prompt = "Summarize the conversation above." # Conversation + Summarization Prompt messages_for_summary = state["messages"] + [ HumanMessage(content=prompt) ] # Ask LLM to summarize response = model.invoke(messages_for_summary) # ------------------------------------- # Delete old messages # Keep last 2 messages # ------------------------------------- messages_to_delete = state["messages"][:-2] return { # Store summary "summary": response.content, # Remove old conversation "messages": [ RemoveMessage(id=m.id) for m in messages_to_delete ], } # ========================================================== # Chat Node # ========================================================== def chat_node(state: ChatState): messages = [] # If summary exists, # inject it as System Message if state["summary"]: messages.append({ "role": "system", "content": f"Conversation Summary:n{state['summary']}" }) # Add latest conversation messages.extend(state["messages"]) print("nMessages Sent To LLMn") for message in messages: print(message) print("-" * 80) # Call LLM response = model.invoke(messages) return { "messages": [response] } # ========================================================== # Decide whether summarization is required # ========================================================== def should_summarize(state: ChatState): return len(state["messages"]) > 6 # ========================================================== # Build Graph # ========================================================== builder = StateGraph(ChatState) builder.add_node("chat", chat_node) builder.add_node( "summarize", summarize_conversation ) builder.add_edge( START, "chat" ) builder.add_conditional_edges( "chat", should_summarize, { True: "summarize", False: "__end__" } ) builder.add_edge( "summarize", "__end__" ) # ========================================================== # Memory # ========================================================== checkpointer = InMemorySaver() graph = builder.compile( checkpointer=checkpointer ) # ========================================================== # Thread # ========================================================== config = { "configurable": { "thread_id": "t1" } } # ========================================================== # Helper Function # ========================================================== def run_turn(text): return graph.invoke( { "messages": [ HumanMessage(content=text) ], "summary": "" }, config=config ) # ========================================================== # Show Current Memory # ========================================================== def show_state(): snapshot = graph.get_state(config) values = snapshot.values print("n========== CURRENT STATE ==========") print("nSummaryn") print(values.get("summary", "")) print("nStored Messages:", len(values["messages"])) print("nConversationn") for message in values["messages"]: print(type(message).__name__) print(message.content) print("-" * 80) # ========================================================== # Conversation 1 # ========================================================== run_turn("Quantum Physics") show_state() """ OUTPUT Summary (empty) Stored Messages 2 Human: Quantum Physics AI: Quantum Physics is ... """ # ========================================================== # Conversation 2 # ========================================================== run_turn("How is Albert Einstein related?") show_state() """ OUTPUT Summary (empty) Stored Messages 4 Human: Quantum Physics AI: ... Human: How is Albert Einstein related? AI: ... """ # ========================================================== # Conversation 3 # ========================================================== run_turn("What are some of Einstein's famous works?") show_state() """ OUTPUT Summary (empty) Stored Messages 6 No summarization yet. """ # ========================================================== # Conversation 4 # ========================================================== run_turn("Explain Special Theory of Relativity") show_state() """ OUTPUT Conversation becomes 8 Messages ↓ Condition 8 > 6 ↓ Summarize Node Executes ↓ Summary Generated ↓ Old Messages Deleted ↓ Final State Summary The conversation discussed: • Quantum Physics • Albert Einstein • Photoelectric Effect • Special Relativity • General Relativity • E = mc² • Brownian Motion • Quantum Mechanics Stored Messages 2 Human: Explain Special Theory of Relativity AI: Special Theory of Relativity is... """ ``` **Summarization Logic** %%{init: {'theme':'base','themeVariables':{'fontSize':'14px','fontFamily':'Arial'}}}%% graph LR A["💬 Old Messages"] B["📝 Summarizer"] C["📦 Summary"] D["💬 Recent Messages"] E["🤖 Main LLM"] F["✅ Final Response"] A --> B --> C --> E D --> E E --> F style A fill:#f5f5f5,stroke:#777 style B fill:#fff3e0,stroke:#ff9800,stroke-width:2px style C fill:#e3f2fd,stroke:#1976d2,stroke-width:2px style D fill:#f5f5f5,stroke:#777 style E fill:#ede7f6,stroke:#673ab7,stroke-width:2px style F fill:#e8f5e9,stroke:#28a745,stroke-width:2px ## 4. Long-Term Memory In the world of AI agents, memory is what transforms a generic chatbot into a personalized assistant. While **short-term memory** handles the immediate conversation flow, **long-term memory (LTM)** allows an agent to remember user preferences, names, and past projects across different sessions and threads. %%{init: {'theme':'base', 'themeVariables': { 'fontSize':'13px', 'fontFamily':'Arial'}}}%% graph TD A["💬 User Message 'I prefer Python'"]:::inputNode A --> B["1️⃣ Remember Node"]:::rememberNode B --> B1["🧠 Extractor LLM"]:::extractNode B1 --> B2["📋 MemoryDecision should_write=True memories=[{text:'Prefers Python', is_new:True}]"]:::pydanticNode B2 --> B3["💾 store.put() namespace=(users, u1) key=uuid"]:::storeNode A --> C["2️⃣ Chat Node"]:::chatNode C --> C1["🔍 store.search() query='preferences'"]:::searchNode C1 --> C2["📝 Inject into System Prompt 'User prefers Python'"]:::injectNode C2 --> C3["🤖 LLM Generate"]:::llmNode C3 --> C4["✨ 'Here's Python code...'"]:::outputNode B3 -.-> D["🗄️ Persistent Store"]:::dbNode C1 -.-> D D -.-> E["🔄 Next Session"]:::nextNode E -.-> F["✅ Remembers Python"]:::successNode classDef inputNode fill:#6f42c1,color:#fff,stroke:#333,stroke-width:3px,font-size:14px classDef rememberNode fill:#ff9800,color:#fff,stroke:#333,stroke-width:3px,font-size:14px classDef chatNode fill:#005cbf,color:#fff,stroke:#333,stroke-width:3px,font-size:14px classDef extractNode fill:#e91e63,color:#fff,stroke:#333,stroke-width:2px,font-size:12px classDef pydanticNode fill:#9c27b0,color:#fff,stroke:#333,stroke-width:2px,font-size:12px classDef storeNode fill:#28a745,color:#fff,stroke:#333,stroke-width:2px,font-size:12px classDef searchNode fill:#00bcd4,color:#000,stroke:#333,stroke-width:2px,font-size:12px classDef injectNode fill:#795548,color:#fff,stroke:#333,stroke-width:2px,font-size:12px classDef llmNode fill:#28a745,color:#fff,stroke:#333,stroke-width:2px,font-size:12px classDef outputNode fill:#4caf50,color:#fff,stroke:#333,stroke-width:3px,font-size:13px classDef dbNode fill:#6f42c1,color:#fff,stroke:#333,stroke-width:3px,font-size:14px classDef nextNode fill:#005cbf,color:#fff,stroke:#333,stroke-width:2px,font-size:13px classDef successNode fill:#4caf50,color:#fff,stroke:#333,stroke-width:4px,font-size:14px This blog post explores how to implement LTM in LangGraph, moving from basic concepts to production-grade persistent storage. --- ### 1. What is Long-Term Memory? Unlike short-term memory which exists within a single thread, LTM is **persistent**. It acts as a "Memory Store" (like a database) where the agent extracts useful information from conversations and stores it for future use. For example, if a user mentions they prefer Python, the agent can store this in LTM and always provide code examples in Python, even in a completely new chat thread. ### 2. The Store Architecture LangGraph implements memory through a class hierarchy based on an abstract class called `BaseStore`. This class defines standard activities like creating, searching, editing, and deleting memories. - **InMemoryStore**: Stores data in RAM. It is excellent for quick prototyping but loses all data when the program stops. - **PostgresStore****/****RedisStore**: These are production-grade implementations that store memories in a persistent database. ### 3. Name Spaces: The Organization Logic To keep memories organized, LangGraph uses **Name Spaces**, which function like folders in a directory. A common structure is `('users', 'user_id', 'sub_folder')`. This ensures that User A’s preferences never leak into User B’s sessions. --- #### **Step 1: Basic Memory Operations** You can interact with a store using three primary methods: `put` (create/update), `get` (retrieve by key), and `search` (retrieve multiple). ```python from langgraph.store.memory import InMemoryStore # Initialize the store store = InMemoryStore() # Define a namespace for a specific user namespace = ("users", "u1") # 1. Put (Create) memory store.put(namespace, "pref_1", {"data": "User likes Python"}) store.put(namespace, "pref_2", {"data": "User prefers dark mode"}) # 2. Get (Retrieve specific) memory = store.get(namespace, "pref_1") print(memory.value) # Output: User likes Python # 3. Search (Retrieve all in namespace) items = store.search(namespace) for item in items: print(item.value) ``` --- #### **Step 2: Advanced Semantic Search** In production, you might have hundreds of memories. Instead of loading them all, you can use **Semantic Search** to fetch only the memories relevant to the current query. This is done by passing an embedding model to the store. ```python # Initialize store with an embedding model for semantic search store = InMemoryStore( index={ "embed": embedding_model, # e.g., OpenAI text-embedding-3-small "dims": 1536 } ) # Search based on meaning, not just keys results = store.search(namespace, query="What are the user's coding preferences?", limit=1) ``` [GitHub Full Code](https://github.com/campusx-official/memory-in-llms/blob/main/6_ltm_basics.ipynb) --- #### **Step 3: Implementing the "Remember" and "Chat" Workflow** A complete LTM-enabled agent typically follows a two-node workflow: 1. **Remember Node**: Analyzes the user's message to see if there is anything worth saving to the long-term store. 2. **Chat Node**: Fetches existing memories from the store and injects them into the system prompt to personalize the response. **Extracting Memories with Pydantic** To ensure the LLM extracts memories accurately, we use a Pydantic model to structure the output. ```python from pydantic import BaseModel, Field from typing import List class MemoryItem(BaseModel): text: str is_new: bool # Used for de-duplication class MemoryDecision(BaseModel): should_write: bool memories: List[MemoryItem] ``` **The "Remember" Node Logic** This node uses an "Extractor LLM" to decide what to save. To avoid cluttering the store, a **de-duplication strategy** is used: the LLM compares the new message against existing memories and only marks truly new information for storage. ```python def remember_node(state, config, store): user_id = config['configurable']['user_id'] namespace = ("users", user_id, "details") # Get last message last_msg = state['messages'][-1].content # LLM extracts new memories (using structured output) decision = extractor_llm.with_structured_output(MemoryDecision).invoke(...) if decision.should_write: for item in decision.memories: if item.is_new: store.put(namespace, str(uuid.uuid4()), {"data": item.text}) return {"messages": [AIMessage(content="Memory updated.")]} ``` [GitHub Full Code](https://github.com/campusx-official/memory-in-llms/blob/main/7_ltm_implementation.ipynb) --- #### **Step 4: Moving to Production with PostgreSQL** For a production system, you must switch from `InMemoryStore` to a persistent database like **PostgreSQL**. This ensures memories survive server restarts. 1. **Set up Postgres**: Use Docker to run a PostgreSQL container. 2. **Configure the Store**: Connect LangGraph to the database using a connection string. ```python from langgraph.store.postgres import PostgresStore # Database URL for a local Docker setup DB_URL = "postgresql://postgres:postgres@localhost:5432/langgraph" # Using a context manager to handle the connection with PostgresStore.from_conn_string(DB_URL) as store: # Setup the tables (only needed once) store.setup() # Compile graph with the persistent store app = graph.compile(store=store) ``` [GitHub Full Code](https://github.com/campusx-official/memory-in-llms/blob/main/8_ltm_postgres.ipynb) ## Conclusion By implementing **Long-Term Memory**, you allow your LangGraph agents to build a "relationship" with the user over time. Using a combination of **Semantic Search** for relevance and **PostgreSQL** for persistence, you can build production-ready systems that offer deeply personalized experiences. ## STM vs LTM %%{init: {'theme':'base','themeVariables':{'fontSize':'14px','fontFamily':'Arial'}}}%% graph TD T["🧠 STM vs 🗄️ LTM"] T --> A["Scope"] A --> A1["🧠 STM: One Thread"] A --> A2["🗄️ LTM: All Threads"] A2 --> B["Duration"] B --> B1["🧠 STM: Temporary"] B --> B2["🗄️ LTM: Persistent"] B2 --> C["Storage"] C --> C1["🧠 STM: Memory"] C --> C2["🗄️ LTM: Database"] C2 --> D["Restart"] D --> D1["🧠 STM: Lost ❌"] D --> D2["🗄️ LTM: Saved ✅"] D2 --> E["Speed"] E --> E1["🧠 STM: Fast ⚡"] E --> E2["🗄️ LTM: Slower"] style T fill:#6f42c1,color:#fff,stroke:#333,stroke-width:2px style A fill:#00bcd4,color:#000 style B fill:#00bcd4,color:#000 style C fill:#00bcd4,color:#000 style D fill:#00bcd4,color:#000 style E fill:#00bcd4,color:#000 style A1 fill:#ff9800,color:#fff style B1 fill:#ff9800,color:#fff style C1 fill:#ff9800,color:#fff style D1 fill:#dc3545,color:#fff style E1 fill:#ff9800,color:#fff style A2 fill:#1976d2,color:#fff style B2 fill:#1976d2,color:#fff style C2 fill:#1976d2,color:#fff style D2 fill:#28a745,color:#fff style E2 fill:#1976d2,color:#fff ## LLM Memory Architecture Quiz Q.1 In the mathematical representation of an LLM at inference, y=f(θ,x), what does the component θ represent? - The fixed weights or parameters learned during training. - The temporary memory buffer of the current conversation. - The input tokens provided by the user. - The output tokens generated by the model. Consider the part of the model that represents its intrinsic knowledge and doesn't change when you ask it a question. [Discuss it](#) Explanation θ (theta) represents the model's learned parameters (weights and biases). These parameters are fixed during inference and contain the knowledge acquired during training. Q.2 What is the primary reason why LLMs are described as stateless systems? - They can only process one token at a time. - The model's parameters are updated after every user interaction. - Each invocation of the model is independent and lacks intrinsic memory of past interactions. - They do not have a context window for processing text. Think about whether the model remembers your name in a completely new session. [Discuss it](#) Explanation LLMs are stateless because each invocation is independent. Without external memory, they do not remember previous conversations or user interactions. Q.3 How does Short-Term Memory (STM) typically function in a conversation buffer setup? - By permanently updating the model's θ parameters with new facts. - By storing the user's name in a global variable across all model versions. - By using a camera lens to capture the physical state of the user. - By concatenating past conversation history with the current user message into the prompt. Focus on how the history of the chat is presented to the LLM when you send a new message. [Discuss it](#) Explanation Short-Term Memory works by including previous conversation history in the prompt so the LLM can use it as context during inference. Q.4 In LangGraph, what component is used to save the state of a graph at every superstep to enable memory? - The Thread ID - A Checkpointer - The BaseStore - The MessageState Think of the tool that marks the current progress so you can return to it later. [Discuss it](#) Explanation A Checkpointer saves the graph state after each superstep, enabling persistence, recovery, and conversational memory. Q.5 When dealing with the Context Overflow problem, how does Trimming differ from Summarization? - Trimming deletes the state, while summarization saves it to a database. - Trimming increases the context window size, while summarization reduces the token count. - Trimming keeps the last n messages and ignores the rest, while summarization condenses old messages into a brief overview. - Trimming keeps the oldest messages, while summarization keeps the newest ones. One method is like cutting off the end of a long rope, while the other is like folding the rope into a smaller bundle. [Discuss it](#) Explanation Trimming removes older messages while keeping the most recent ones. Summarization compresses older conversations into a concise summary so important information is retained. Q.6 Which type of Long-Term Memory (LTM) is specifically concerned with how things are done, such as preferred workflows or rules? - Procedural Memory - Episodic Memory - Semantic Memory - In-Memory Store Think of the memory that remembers a recipe or a coding workflow rather than a standalone fact. [Discuss it](#) Explanation Procedural Memory stores processes, workflows, habits, and instructions describing how tasks should be performed. Q.7 In the LangGraph Store implementation, what is a Namespace? - A function that generates embeddings for semantic search. - The name of the database where the memory is stored. - A way to organize memories into hierarchical structures, similar to folders in a file system. - A unique identifier for a single message. Imagine organizing files into folders for different users or projects. [Discuss it](#) Explanation A Namespace organizes memories into logical groups, making it easier to separate user memories, application data, or different domains. Q.8 Which LangGraph component is primarily responsible for storing Long-Term Memory across multiple conversations? - Checkpointer - BaseStore (Store) - MessageState - Thread ID Think about which component persists information beyond a single conversation thread. [Discuss it](#) Explanation The LangGraph Store (BaseStore) is designed for Long-Term Memory, allowing information to persist across multiple conversations and sessions. Q.9 In the four-step workflow of Long-Term Memory, what happens during the Injection stage? - The model's weights are fine-tuned with the user's preferences. - New information is extracted from the user's message. - The retrieved long-term memory is placed into the current conversation buffer (Short-Term Memory). - Data is permanently saved to a Postgres database. Consider how the LLM actually sees previously stored memories. [Discuss it](#) Explanation During Injection, retrieved Long-Term Memory is inserted into the current prompt or Short-Term Memory so the LLM can use it when generating a response. Q.10 What is a major challenge when implementing a Remember node to extract memories in real-time? - Ensuring the user is always asked for permission before a memory is saved. - Preventing the creation of redundant or duplicate memories in the store. - The Postgres database cannot handle high-frequency writes. - LLMs cannot generate JSON or structured output. Think about what happens if a user tells the bot 'My name is Nitish' in five different chat sessions. [Discuss it](#) Explanation A Remember node must avoid storing duplicate or redundant memories; otherwise, the memory store becomes cluttered and retrieval quality decreases. Q.11 What is the 'Context Window' of a Large Language Model? - The total number of parameters the model was trained on. - The amount of text an LLM can read and remember at one time before answering. - The physical screen size where the chat interface is displayed. - The time limit the model has to generate a response. Think of the camera lens analogy mentioned in the sources. [Discuss it](#) Explanation The context window is the maximum amount of text/tokens an LLM can process in a single invocation. Q.12 How is 'Short-Term Memory' technically achieved in a stateless LLM? - By updating the model's weights in real-time. - By storing information in the model's intrinsic RAM. - By concatenating the conversation history into the input prompt for each new call. - LLMs are not actually stateless; they have built-in memory. Consider how a 'Conversation Buffer' works. [Discuss it](#) Explanation Since LLMs are stateless, developers maintain a conversation buffer by appending previous messages to the current input so the model has the context. Q.13 What does it mean for short-term memory to be 'thread-scoped'? - Memory is shared across all users in the system. - Memory is deleted immediately after a single word is generated. - Memory exists only within the boundary of a specific conversation or session. - Memory is stored in the global weights of the LLM. Think about how individual chat sessions in ChatGPT are independent. [Discuss it](#) Explanation Short-term memory is typically restricted to a single 'thread' or conversation ID, meaning context from one chat does not leak into another. Q.14 What is a significant drawback of using 'trimming' to solve the context window problem? - It requires expensive vector databases to function. - The LLM completely loses the context of the oldest parts of the conversation. - It makes the LLM significantly slower at generating text. - It causes the LLM to forget the user's name even if it was just mentioned. Trimming involves discarding messages that exceed a token limit. [Discuss it](#) Explanation Trimming removes the oldest messages once a token limit is reached, which can lead to a 'breaking point' in the conversation if early context was important. Q.15 Why is 'summarization' often more effective than 'trimming' for long conversations? - It is cheaper because it uses fewer API calls. - It removes the need for any persistent database. - It preserves the essence of the entire past conversation while staying within token limits. - It allows the LLM to remember every single word ever spoken. Think about how a summary represents many pages of text. [Discuss it](#) Explanation Summarization uses an LLM to condense old messages into a brief summary, which is then passed alongside recent messages to maintain context without exceeding the window. Q.16 Which type of Long-Term Memory is responsible for remembering 'facts' about a user, such as their job or language preference? - Episodic Memory - Semantic Memory - Volatile Memory This type of memory deals with 'what is true'. [Discuss it](#) Explanation Semantic memory consists of facts about the user or the system, such as a user's preference for Python or their profession. Q.17 In LangGraph, why is 'PostgresStore' preferred over 'InMemoryStore' for production? - PostgresStore is faster for processing tokens. - InMemoryStore loses all data when the program or server restarts. - PostgresStore allows the LLM to run without any parameters. - InMemoryStore is only compatible with Gemini models. RAM is volatile, whereas a database is persistent. [Discuss it](#) Explanation InMemoryStore saves state in RAM, which is lost upon restart; PostgresStore provides a durable, persistent layer for production-grade applications. Q.18 What is the purpose of 'Namespaces' in LangGraph memory stores? - To give the AI agent a specific personality name. - To define the programming language used for the graph. - To organize memories into logical 'folders' like ('users', 'user_id', 'preferences'). - To set the maximum number of tokens allowed per thread. Think about how files are organized on Google Drive. [Discuss it](#) Explanation Namespaces (represented as tuples) allow developers to organize and categorize memories within a store, making them easier to manage and retrieve. Q.19 What is required to enable 'semantic search' in a LangGraph memory store? - An embedding model and vector indexing. - A higher token limit in the system prompt. - Deleting all old messages periodically. - Using only the GPT-4o model. This involves searching by meaning rather than exact keywords. [Discuss it](#) Explanation To perform semantic search (searching by meaning), the store must be initialized with an embedding model to convert memories and queries into vectors for comparison. Q.20 What are the four high-level steps of the Long-Term Memory (LTM) workflow? - Input, Output, Tokenization, Training. - Trimming, Summarization, Deletion, Resetting. - Creation, Storage, Retrieval, Injection. - Login, Query, Response, Logout. The steps move from identifying a memory to putting it back into a prompt. [Discuss it](#) Explanation The LTM workflow involves creating/extracting a memory, storing it in a database, retrieving it based on relevance, and injecting it into the current prompt. Q.21 In the mathematical equation y = f(θ, x), what does the 'θ' (theta) represent in an LLM? - The input tokens provided by the user. - The output tokens generated by the model. - The billions of fixed parameters/weights trained into the model. - The time it takes to process a single request. This is the 'parametric knowledge' of the model. [Discuss it](#) Explanation Theta represents the billions of parameters that define the LLM's knowledge, which are fixed after the training phase. Q.22 What is a 'Checkpointer' in the context of LangGraph? - A tool that checks if the LLM is generating hallucinations. - A mechanism that saves the state of the graph at every superstep. - A database specifically for storing image files. - A function that resets the conversation history to zero. Think about saving progress in a video game. [Discuss it](#) Explanation A checkpointer is a concept in LangGraph used to implement short-term memory by saving the state of the graph at every superstep. Q.23 Which phenomenon allows an LLM to answer questions about a private PDF that wasn't in its training data? - Parametric Retrieval - In-Context Learning - Gradient Descent - Supervised Fine-Tuning The LLM uses information present in the prompt itself. [Discuss it](#) Explanation In-context learning is an emergent ability where the LLM uses information and patterns present in the current prompt to generate an answer. Q.24 Why is 'In-Memory Store' (InMemoryStore) unsuitable for production-grade applications? - It is too slow to handle user queries. - It can only store up to 10 messages at a time. - It is volatile and loses all stored data if the server restarts. - It requires a specialized Docker container to function. Think about the difference between RAM and a hard drive. [Discuss it](#) Explanation InMemoryStore saves data in RAM; once the program or server restarts, all the memories are lost, making it unsuitable for production. Q.25 In LangGraph's Long-Term Memory, what is a 'Namespace'? - The legal name of the AI agent. - A way to organize memories into logical 'folders' using tuples. - A specific programming language used to build the graph. - The total number of tokens allowed in a thread. The source uses the analogy of Google Drive folders. [Discuss it](#) Explanation Namespaces, typically created as tuples like ('users', 'user_id'), are used to organize and categorize memories within a memory store. Q.26 Which type of memory specifically remembers 'how to do things' or user-preferred strategies? - Episodic Memory - Semantic Memory - Procedural Memory - Parametric Memory This memory stores rules and learned behaviors. [Discuss it](#) Explanation Procedural memory stores strategies, rules, and learned behaviors, such as a user's preference for step-by-step explanations. Q.27 What must be provided to a LangGraph Store to enable 'Semantic Search' functionality? - A list of forbidden keywords. - An embedding model (like text-embedding-3-small). - A high-speed internet connection. - A specific Thread ID for every user. This is needed to turn text into vectors for comparison. [Discuss it](#) Explanation To conduct semantic search (searching by meaning), you must pass an embedding model when creating the memory store. Q.28 In the Long-Term Memory workflow, what does the 'Injection' step involve? - Saving a new fact into the Postgres database. - Filtering out noise from the user's chat history. - Pulling relevant long-term memories into the current short-term context/prompt. - Deleting old messages from the conversation buffer. It's the final step before the LLM receives the input. [Discuss it](#) Explanation Injection is the step where retrieved long-term memories are added to the short-term memory (prompt) so the LLM can use them. Q.29 What is a 'Remember Node' in an agentic workflow? - A node that reminds the user of their appointments. - A node that extracts useful information from the user's message to save in LTM. - A node that counts how many tokens the user has consumed. - A node that deletes all history to save database space. It 'extracts' stable facts from a conversation. [Discuss it](#) Explanation A 'Remember' node is used to identify and extract stable, user-specific information from a message and store it in the long-term memory. Q.30 To prevent duplicate memories in a store, what strategy is suggested in the sources? - Deleting the entire database every 24 hours. - Using an LLM to compare the new message against existing memories before saving. - Only allowing the user to save one memory per day. - Limiting memories to 10 words or less. The LLM acts as a 'deduplicator' by checking current store contents. [Discuss it](#) Explanation To avoid redundancy, the system can send existing memories to the 'extractor' LLM so it can decide if a new message contains a truly new fact or just a duplicate of what is already stored. [LLMs Don’t Have Memory — So How Do They Remember?](https://www.youtube.com/embed/DcPKJrOF9Wo?list=PLKnIA16_RmvYsvB8qkUQuJmJNuiCUJFPL) [How To Implement Short Term Memory Using LangGraph](https://www.youtube.com/embed/FSBkTI1QuvY?list=PLKnIA16_RmvYsvB8qkUQuJmJNuiCUJFPL) [Long Term Memory in LangGraph](https://www.youtube.com/embed/KrXBcokM3Tc?list=PLKnIA16_RmvYsvB8qkUQuJmJNuiCUJFPL)