LLM Memory Architecture

Large Language Models (LLMs) are often perceived as having human-like memory because they can hold coherent conversations. However, technically, LLMs are stateless mathematical functions. This blog explores how developers bridge the gap between this inherent statelessness and the complex memory systems required for production-grade AI agents using frameworks like LangGraph.

1. The Core Paradox: Why LLMs are Stateless #

At its fundamental level, an LLM at inference is a parameterized mathematical function: y=f(θ,x).

• θ: Billions of fixed parameters (the model’s “brain” trained on the internet).
• x: The input tokens (your prompt).
• y: The output tokens.

Because θ is fixed after training and the function only looks at the current x, it has no “internal” memory of past interactions. If you tell an LLM your name in one call and ask for it in the next, it will fail unless that context is provided again

2. Short-Term Memory (STM): The Conversation Buffer #

To make an LLM “remember,” we create a Conversation Buffer. Instead of sending just the new message (x2​), we concatenate the entire history (x1​,y1​,x2​) and send it as the new input.

LangGraph Implementation: Checkpointers and Threads

In LangGraph, STM is managed through Checkpointers and Thread IDs.

• Checkpointer: Saves the state of the graph at every “superstep”.
• Thread ID: A unique identifier for a specific conversation session. Memory is typically thread-scoped, meaning it exists only within that specific conversation boundary.

Diagram: STM Workflow

Code Example (In-Memory STM):

# ==========================================================
# Example 1: LangGraph WITHOUT Memory
# ==========================================================

from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langgraph.graph import StateGraph, START, MessagesState

# Load OpenAI API key from .env file
load_dotenv()

# Create the LLM
model = ChatOpenAI()


# ----------------------------------------------------------
# Node Function
# ----------------------------------------------------------
# This node receives the current state (messages),
# sends it to the LLM, and returns the AI response.
def call_model(state: MessagesState):
    response = model.invoke(state["messages"])
    return {"messages": [response]}


# ----------------------------------------------------------
# Build the Graph
# ----------------------------------------------------------
builder = StateGraph(MessagesState)

# Add one node
builder.add_node("call_model", call_model)

# Connect START → call_model
builder.add_edge(START, "call_model")

# Compile graph
graph = builder.compile()


# ----------------------------------------------------------
# First Conversation
# ----------------------------------------------------------
graph.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "Hi! My name is Sanjit"
            }
        ]
    }
)

"""
Output

Human : Hi! My name is Sanjit
AI    : Nice to meet you, Sanjit!
"""


# ----------------------------------------------------------
# Second Conversation
# ----------------------------------------------------------
graph.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "What is my name?"
            }
        ]
    }
)

"""
Output

Human : What is my name?
AI    : I don't know your name.

Reason:
Every invoke() starts a NEW conversation.
The previous message ("My name is Sanjit")
was NOT stored anywhere.
"""

---

# ==========================================================
# Example 2: LangGraph WITH Memory
# ==========================================================

from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langgraph.graph import StateGraph, START, MessagesState
from langgraph.checkpoint.memory import InMemorySaver

# Load environment variables
load_dotenv()

# Create LLM
model = ChatOpenAI()


# ----------------------------------------------------------
# Node Function
# ----------------------------------------------------------
def call_model(state: MessagesState):
    """
    Receives all conversation messages,
    sends them to the LLM,
    returns the AI response.
    """
    response = model.invoke(state["messages"])
    return {"messages": [response]}


# ----------------------------------------------------------
# Build Graph
# ----------------------------------------------------------
builder = StateGraph(MessagesState)

builder.add_node("call_model", call_model)

builder.add_edge(START, "call_model")


# ----------------------------------------------------------
# Create Checkpointer
# ----------------------------------------------------------
# InMemorySaver stores conversation history
# in RAM using a thread_id.
checkpointer = InMemorySaver()


# Compile graph with memory
graph = builder.compile(checkpointer=checkpointer)


# ----------------------------------------------------------
# Thread Configurations
# ----------------------------------------------------------
config1 = {
    "configurable": {
        "thread_id": "thread-1"
    }
}

config2 = {
    "configurable": {
        "thread_id": "thread-2"
    }
}


# ----------------------------------------------------------
# First Message (Thread 1)
# ----------------------------------------------------------
graph.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "Hi! My name is Sanjit."
            }
        ]
    },
    config=config1,
)

"""
Output

Human : Hi! My name is Sanjit.
AI    : Nice to meet you Sanjit.
"""


# ----------------------------------------------------------
# Ask Again in SAME Thread
# ----------------------------------------------------------
graph.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "What is my name?"
            }
        ]
    },
    config=config1,
)

"""
Output

Human : What is my name?
AI    : Your name is Sanjit.
"""


# ----------------------------------------------------------
# Ask in a DIFFERENT Thread
# ----------------------------------------------------------
graph.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "What is my name?"
            }
        ]
    },
    config=config2,
)

"""
Output

Human : What is my name?
AI    : I don't know your name.
"""

3. Solving the Context Window Crisis #

LLMs have a Context Window—a limit on the number of tokens they can process at once. As conversations grow, the buffer can exceed this limit, leading to hallucinations or errors.

Technique A: Trimming

We set a max_token_limit and keep only the most recent messages that fit within it, discarding the oldest ones.

• Pros: Simple to implement using functions like trim_messages.
• Cons: The LLM completely loses the context of early parts of the conversation.

# ==========================================================
# LangGraph Memory Trimming Example
# ==========================================================
# This example demonstrates how to limit the conversation
# history sent to the LLM using trim_messages().
#
# Even though the complete conversation is stored in memory,
# only the most recent messages that fit within MAX_TOKENS
# are sent to the model.
# ==========================================================

from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langgraph.graph import StateGraph, START, MessagesState
from langgraph.checkpoint.memory import InMemorySaver
from langchain_core.messages.utils import (
    trim_messages,
    count_tokens_approximately,
)

# ----------------------------------------------------------
# Load OpenAI API Key
# ----------------------------------------------------------
load_dotenv()

# Create LLM
model = ChatOpenAI()

# ----------------------------------------------------------
# Maximum Tokens Allowed
# ----------------------------------------------------------
# Only messages that fit within this budget
# will be sent to the LLM.
MAX_TOKENS = 150


# ----------------------------------------------------------
# Graph Node
# ----------------------------------------------------------
def call_model(state: MessagesState):

    # Trim the conversation.
    # Strategy = "last"
    # Keep the latest messages until MAX_TOKENS is reached.
    messages = trim_messages(
        state["messages"],
        strategy="last",
        token_counter=count_tokens_approximately,
        max_tokens=MAX_TOKENS,
    )

    # Print token count after trimming
    print("\nCurrent Token Count ->",
          count_tokens_approximately(messages))

    # Print messages actually sent to LLM
    print("\nMessages Sent To LLM\n")

    for message in messages:
        print(message.content)
        print("-" * 60)

    # Send trimmed conversation
    response = model.invoke(messages)

    return {"messages": [response]}


# ----------------------------------------------------------
# Build Graph
# ----------------------------------------------------------
builder = StateGraph(MessagesState)

builder.add_node("call_model", call_model)

builder.add_edge(START, "call_model")


# ----------------------------------------------------------
# Add Memory
# ----------------------------------------------------------
checkpointer = InMemorySaver()

graph = builder.compile(checkpointer=checkpointer)


# ----------------------------------------------------------
# Thread Configuration
# ----------------------------------------------------------
config = {
    "configurable": {
        "thread_id": "chat-1"
    }
}


# ==========================================================
# Conversation 1
# ==========================================================

result = graph.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "Hi, my name is Sanjit."
            }
        ]
    },
    config,
)

print(result["messages"][-1].content)


"""
Output

Current Token Count -> 10

Messages Sent To LLM

Hi, my name is Sanjit.

------------------------------------------------------------

AI:
Hello Sanjit! Nice to meet you.
"""


# ==========================================================
# Conversation 2
# ==========================================================

result = graph.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "I am learning LangGraph."
            }
        ]
    },
    config,
)

print(result["messages"][-1].content)


"""
Output

Current Token Count -> 40

Messages Sent To LLM

Hi, my name is Sanjit.

AI:
Hello Sanjit!

I am learning LangGraph.

------------------------------------------------------------

AI:
That's great!
"""


# ==========================================================
# Conversation 3
# ==========================================================

result = graph.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "Can you explain short-term memory?"
            }
        ]
    },
    config,
)

print(result["messages"][-1].content)


"""
Output

Current Token Count -> 108

Messages Sent To LLM

Hi, my name is Sanjit.

AI:
Hello Sanjit!

I am learning LangGraph.

AI:
That's great!

Can you explain short-term memory?

------------------------------------------------------------

AI:
Short-term memory is ...
"""


# ==========================================================
# Conversation 4
# ==========================================================

result = graph.invoke(
    {
        "messages": [
            {
                "role": "user",
                "content": "What is my name?"
            }
        ]
    },
    config,
)

print(result["messages"][-1].content)


"""
Output

Current Token Count -> 8

Messages Sent To LLM

What is my name?

------------------------------------------------------------

AI:
I'm sorry, I don't know your name.
"""


# ==========================================================
# View Complete Stored Memory
# ==========================================================

snapshot = graph.get_state(config)

print("\nComplete Conversation Stored In Memory\n")

for message in snapshot.values["messages"]:
    print(message.content)
    print("-" * 80)

Technique B: Summarization & Deletion

Instead of deleting old messages, we send them to another LLM to generate a Summary. We then pass this summary plus the most recent messages to the main LLM. This preserves the “essence” of the past without bloating the token count.

# ==========================================================
# LangGraph Conversation Summarization Memory Example
# ==========================================================
#
# This example demonstrates how LangGraph automatically
# summarizes long conversations.
#
# Instead of sending the entire chat history to the LLM,
# it:
#
# 1. Stores all messages.
# 2. When conversation becomes long (>6 messages),
#    creates a summary.
# 3. Deletes old messages.
# 4. Keeps only the latest two messages.
# 5. Uses the summary in future conversations.
#
# ==========================================================

from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, RemoveMessage
from langgraph.graph import MessagesState, StateGraph, START
from langgraph.checkpoint.memory import InMemorySaver

# ----------------------------------------------------------
# Load OpenAI API Key
# ----------------------------------------------------------
load_dotenv()

# Create LLM
model = ChatOpenAI()


# ==========================================================
# Custom State
# ==========================================================
#
# MessagesState already contains:
#
# messages
#
# We extend it by adding
#
# summary
#
# Final State
#
# {
#     "messages": [...],
#     "summary": "..."
# }
#
# ==========================================================

class ChatState(MessagesState):
    summary: str


# ==========================================================
# Summarization Node
# ==========================================================
#
# This node runs only when the conversation
# becomes longer than 6 messages.
#
# It creates/updates the summary and
# removes old messages.
#
# ==========================================================

def summarize_conversation(state: ChatState):

    # Previous summary
    existing_summary = state["summary"]

    # -------------------------------------
    # Build Prompt
    # -------------------------------------
    if existing_summary:

        prompt = (
            f"Existing summary:\n{existing_summary}\n\n"
            "Extend the summary using the new conversation above."
        )

    else:

        prompt = "Summarize the conversation above."

    # Conversation + Summarization Prompt
    messages_for_summary = state["messages"] + [
        HumanMessage(content=prompt)
    ]

    # Ask LLM to summarize
    response = model.invoke(messages_for_summary)

    # -------------------------------------
    # Delete old messages
    # Keep last 2 messages
    # -------------------------------------
    messages_to_delete = state["messages"][:-2]

    return {

        # Store summary
        "summary": response.content,

        # Remove old conversation
        "messages": [
            RemoveMessage(id=m.id)
            for m in messages_to_delete
        ],
    }


# ==========================================================
# Chat Node
# ==========================================================

def chat_node(state: ChatState):

    messages = []

    # If summary exists,
    # inject it as System Message
    if state["summary"]:

        messages.append({
            "role": "system",
            "content":
            f"Conversation Summary:\n{state['summary']}"
        })

    # Add latest conversation
    messages.extend(state["messages"])

    print("\nMessages Sent To LLM\n")

    for message in messages:
        print(message)

    print("-" * 80)

    # Call LLM
    response = model.invoke(messages)

    return {
        "messages": [response]
    }


# ==========================================================
# Decide whether summarization is required
# ==========================================================

def should_summarize(state: ChatState):

    return len(state["messages"]) > 6


# ==========================================================
# Build Graph
# ==========================================================

builder = StateGraph(ChatState)

builder.add_node("chat", chat_node)

builder.add_node(
    "summarize",
    summarize_conversation
)

builder.add_edge(
    START,
    "chat"
)

builder.add_conditional_edges(

    "chat",

    should_summarize,

    {
        True: "summarize",
        False: "__end__"
    }

)

builder.add_edge(
    "summarize",
    "__end__"
)


# ==========================================================
# Memory
# ==========================================================

checkpointer = InMemorySaver()

graph = builder.compile(
    checkpointer=checkpointer
)


# ==========================================================
# Thread
# ==========================================================

config = {
    "configurable": {
        "thread_id": "t1"
    }
}


# ==========================================================
# Helper Function
# ==========================================================

def run_turn(text):

    return graph.invoke(

        {
            "messages": [
                HumanMessage(content=text)
            ],
            "summary": ""
        },

        config=config

    )


# ==========================================================
# Show Current Memory
# ==========================================================

def show_state():

    snapshot = graph.get_state(config)

    values = snapshot.values

    print("\n========== CURRENT STATE ==========")

    print("\nSummary\n")

    print(values.get("summary", ""))

    print("\nStored Messages:", len(values["messages"]))

    print("\nConversation\n")

    for message in values["messages"]:

        print(type(message).__name__)
        print(message.content)
        print("-" * 80)


# ==========================================================
# Conversation 1
# ==========================================================

run_turn("Quantum Physics")

show_state()

"""
OUTPUT

Summary

(empty)

Stored Messages

2

Human:
Quantum Physics

AI:
Quantum Physics is ...
"""


# ==========================================================
# Conversation 2
# ==========================================================

run_turn("How is Albert Einstein related?")

show_state()

"""
OUTPUT

Summary

(empty)

Stored Messages

4

Human:
Quantum Physics

AI:
...

Human:
How is Albert Einstein related?

AI:
...
"""


# ==========================================================
# Conversation 3
# ==========================================================

run_turn("What are some of Einstein's famous works?")

show_state()

"""
OUTPUT

Summary

(empty)

Stored Messages

6

No summarization yet.
"""


# ==========================================================
# Conversation 4
# ==========================================================

run_turn("Explain Special Theory of Relativity")

show_state()

"""
OUTPUT

Conversation becomes

8 Messages

↓

Condition

8 > 6

↓

Summarize Node Executes

↓

Summary Generated

↓

Old Messages Deleted

↓

Final State


Summary

The conversation discussed:

• Quantum Physics

• Albert Einstein

• Photoelectric Effect

• Special Relativity

• General Relativity

• E = mc²

• Brownian Motion

• Quantum Mechanics

Stored Messages

2

Human:
Explain Special Theory of Relativity

AI:
Special Theory of Relativity is...
"""

Summarization Logic

4. Long-Term Memory #

In the world of AI agents, memory is what transforms a generic chatbot into a personalized assistant. While short-term memory handles the immediate conversation flow, long-term memory (LTM) allows an agent to remember user preferences, names, and past projects across different sessions and threads.

This blog post explores how to implement LTM in LangGraph, moving from basic concepts to production-grade persistent storage.

---

1. What is Long-Term Memory? #

Unlike short-term memory which exists within a single thread, LTM is persistent. It acts as a “Memory Store” (like a database) where the agent extracts useful information from conversations and stores it for future use. For example, if a user mentions they prefer Python, the agent can store this in LTM and always provide code examples in Python, even in a completely new chat thread.

2. The Store Architecture #

LangGraph implements memory through a class hierarchy based on an abstract class called BaseStore. This class defines standard activities like creating, searching, editing, and deleting memories.

• InMemoryStore: Stores data in RAM. It is excellent for quick prototyping but loses all data when the program stops.
• PostgresStore / RedisStore: These are production-grade implementations that store memories in a persistent database.

3. Name Spaces: The Organization Logic #

To keep memories organized, LangGraph uses Name Spaces, which function like folders in a directory. A common structure is ('users', 'user_id', 'sub_folder'). This ensures that User A’s preferences never leak into User B’s sessions.

---

Step 1: Basic Memory Operations #

You can interact with a store using three primary methods: put (create/update), get (retrieve by key), and search (retrieve multiple).

from langgraph.store.memory import InMemoryStore

# Initialize the store
store = InMemoryStore()

# Define a namespace for a specific user
namespace = ("users", "u1")

# 1. Put (Create) memory
store.put(namespace, "pref_1", {"data": "User likes Python"})
store.put(namespace, "pref_2", {"data": "User prefers dark mode"})

# 2. Get (Retrieve specific)
memory = store.get(namespace, "pref_1")
print(memory.value) # Output: User likes Python

# 3. Search (Retrieve all in namespace)
items = store.search(namespace)
for item in items:
    print(item.value)

---

Step 2: Advanced Semantic Search #

In production, you might have hundreds of memories. Instead of loading them all, you can use Semantic Search to fetch only the memories relevant to the current query. This is done by passing an embedding model to the store.

# Initialize store with an embedding model for semantic search
store = InMemoryStore(
    index={
        "embed": embedding_model, # e.g., OpenAI text-embedding-3-small
        "dims": 1536
    }
)

# Search based on meaning, not just keys
results = store.search(namespace, query="What are the user's coding preferences?", limit=1)

---

Step 3: Implementing the “Remember” and “Chat” Workflow #

A complete LTM-enabled agent typically follows a two-node workflow:

1. Remember Node: Analyzes the user’s message to see if there is anything worth saving to the long-term store.
2. Chat Node: Fetches existing memories from the store and injects them into the system prompt to personalize the response.

Extracting Memories with Pydantic

To ensure the LLM extracts memories accurately, we use a Pydantic model to structure the output.

from pydantic import BaseModel, Field
from typing import List

class MemoryItem(BaseModel):
    text: str
    is_new: bool # Used for de-duplication

class MemoryDecision(BaseModel):
    should_write: bool
    memories: List[MemoryItem]

The “Remember” Node Logic

This node uses an “Extractor LLM” to decide what to save. To avoid cluttering the store, a de-duplication strategy is used: the LLM compares the new message against existing memories and only marks truly new information for storage.

def remember_node(state, config, store):
    user_id = config['configurable']['user_id']
    namespace = ("users", user_id, "details")
    
    # Get last message
    last_msg = state['messages'][-1].content
    
    # LLM extracts new memories (using structured output)
    decision = extractor_llm.with_structured_output(MemoryDecision).invoke(...)
    
    if decision.should_write:
        for item in decision.memories:
            if item.is_new:
                store.put(namespace, str(uuid.uuid4()), {"data": item.text})
    
    return {"messages": [AIMessage(content="Memory updated.")]}

---

Step 4: Moving to Production with PostgreSQL #

For a production system, you must switch from InMemoryStore to a persistent database like PostgreSQL. This ensures memories survive server restarts.

1. Set up Postgres: Use Docker to run a PostgreSQL container.
2. Configure the Store: Connect LangGraph to the database using a connection string.

from langgraph.store.postgres import PostgresStore

# Database URL for a local Docker setup
DB_URL = "postgresql://postgres:postgres@localhost:5432/langgraph"

# Using a context manager to handle the connection
with PostgresStore.from_conn_string(DB_URL) as store:
    # Setup the tables (only needed once)
    store.setup()
    
    # Compile graph with the persistent store
    app = graph.compile(store=store)

Conclusion #

By implementing Long-Term Memory, you allow your LangGraph agents to build a “relationship” with the user over time. Using a combination of Semantic Search for relevance and PostgreSQL for persistence, you can build production-ready systems that offer deeply personalized experiences.

LLM Memory Architecture Quiz #