Memory

The full example for this section can be found in the GitHub repo.

What is Memory?

Agentic memory (or just “memory”) in AI applications refers to the system’s ability to retain and utilize information from previous interactions. Without memory, each conversation with an LLM starts from scratch—the model has no awareness of what was discussed moments ago. Memory allows your AI application to maintain context across multiple exchanges, creating more coherent and personalized experiences.

The Use Case for Memory

Consider a customer support chatbot. Without memory, a user asking “What’s the status of my order?” followed by “Can you cancel it?” would leave the AI confused—it wouldn’t know which order to cancel. With memory, the conversation flows naturally, just as it would with a human agent.

Memory becomes critical in several scenarios:

• Multi-turn conversations where context builds over time
• Personalized interactions that adapt based on user preferences
• Task-oriented dialogues where the AI needs to track goals and progress
• Long-running sessions where conversations span multiple topics

The following diagram can be used to describe an agentic loop that uses memory. It attempts to retrieve relevant memories (with retrieval/filtering often in the same step), assembles context for our agent based on the memories and then finally once the full request is assembled the agent will then send the prompt to the LLM. Once a response has been retrieved, it might then check if memories need updating (and update accordingly).

graph TD
    A[User Input] --> B[Agent Processing]
    B --> C{Memory Operations}
    
    C --> D[Short-term Memory]
    C --> E[Long-term Memory]
    C --> F[Episodic Memory]
    
    D --> G[Current Context<br/>Active conversation]
    E --> H[Semantic Knowledge<br/>Facts & concepts]
    F --> I[Past Interactions<br/>Event sequences]
    
    G --> J[Memory Retrieval]
    H --> J
    I --> J
    
    J --> K[Relevance Filtering]
    K --> L[Context Assembly]
    
    L --> M[Response Generation]
    M --> N[Action/Output]
    
    N --> O[Memory Update]
    O --> D
    O --> E
    O --> F
    
    style D fill:#e1f5ff,color:#000
    style E fill:#fff4e1,color:#000
    style F fill:#f0e1ff,color:#000
    style J fill:#e1ffe1,color:#000
    style M fill:#ffe1e1,color:#000

Basic Memory Management in Rig

In Rig, conversation history is currently decoupled from the library - which is to say that you need to implement it yourself. The simplest form of memory management is storing a conversation history as a Vec<T>. Each exchange between the user and the assistant is stored as a Message object:

#![allow(unused)]
fn main() {
use rig::completion::Message;

    let mut conversation_history: Vec<Message> = Vec::new();
}

As the conversation progresses, you append new messages to this vector:

#![allow(unused)]
fn main() {
    // Add a user message
    conversation_history.push(Message::User {
        content: OneOrMany::one(UserContent::text(
            "Do you know what the weather is like today?",
        )),
    });

    // Add the assistant's response
    conversation_history.push(Message::Assistant {
        id: None,
        content: OneOrMany::one(AssistantContent::text(
            "I don't have access to real-time weather data...",
        )),
    });
}

When making subsequent requests to the LLM, you include this history to maintain context. However, this approach has a fundamental limitation: conversation history grows indefinitely, eventually exceeding the model’s context window and increasing costs.

Managing Ephemeral Conversation Memory

To handle growing conversation histories, you’ll need a more sophisticated approach. Let’s create a ConversationMemory struct that can both manage messages and compact them when needed.

First, define the basic structure:

#![allow(unused)]
fn main() {
use rig::completion::Message;

pub struct ConversationMemory {
    messages: Vec<Message>,
    max_messages: usize,
    summary: Option<String>,
}

impl ConversationMemory {
    pub fn with_max_messages(max_messages: usize) -> Self {
        Self {
            messages: Vec::new(),
            max_messages,
            summary: None,
        }
    }
}
}

Add methods for basic message management:

#![allow(unused)]
fn main() {
impl ConversationMemory {
    pub fn add_user_message(&mut self, input: &str) {
        let message = Message::User {
            content: OneOrMany::one(UserContent::text(input)),
        };

        self.messages.push(message);
    }

    pub fn add_assistant_message(&mut self, input: &str) {
        let message = Message::Assistant {
            content: OneOrMany::one(AssistantContent::text(input)),
            id: None,
        };

        self.messages.push(message);
    }

    pub fn get_messages(&self) -> &[Message] {
        &self.messages
    }

    pub fn clear(&mut self) {
        self.messages.clear();
    }
}
}

Although being able to clear and fetch/add messages is quite useful, will also need a way to compact the messages by generating a summary.

Typically there are a number of ways this can be done, but for the sake of simplicity we will hold a variable that has a number of maximum messages. If the message length passes the threshold, we clear the message list and ask the LLM to generate a summary.

#![allow(unused)]
fn main() {
use rig::agent::Text;
use rig::completion::CompletionModel;
use rig::message::{AssistantContent, UserContent};

impl ConversationMemory {
    pub async fn compact<T>(&mut self, model: &T) -> Result<(), Box<dyn std::error::Error>>
    where
        T: CompletionModel,
    {
        if self.messages.len() <= self.max_messages {
            return Ok(());
        }

        // Create a prompt asking the LLM to summarize the conversation
        let summary_prompt = format!(
            "Please provide a concise summary of the following conversation, \
             capturing key points, decisions, and context:\n\n{}",
            self.format_messages_for_summary()
        );

        // Request the summary from the LLM
        let response = model.completion_request(&summary_prompt).send().await?;

        let AssistantContent::Text(Text { text, .. }) = response.choice.first() else {
            return Err("Model returned non-text response".into());
        };

        self.summary = Some(text);
        self.messages.clear();

        Ok(())
    }

    fn format_messages_for_summary(&self) -> String {
        self.messages
            .iter()
            .map(|msg| match msg {
                Message::User { content } => {
                    let text_content = content
                        .iter()
                        .filter_map(|x| {
                            if let UserContent::Text(Text { text, .. }) = x {
                                Some(text.to_owned())
                            } else {
                                None
                            }
                        })
                        .collect::<Vec<String>>()
                        .join("\n");

                    format!("User: {text_content}")
                }
                Message::Assistant { content, .. } => {
                    let text_content = content
                        .iter()
                        .filter_map(|x| {
                            if let AssistantContent::Text(Text { text, .. }) = x {
                                Some(text.to_owned())
                            } else {
                                None
                            }
                        })
                        .collect::<Vec<String>>()
                        .join("\n");

                    format!("Assistant: {text_content}")
                }
                _ => String::new(),
            })
            .collect::<Vec<_>>()
            .join("\n")
    }
}
}

You can inject the summary back into the conversation in two ways. The first approach adds it to the system prompt:

#![allow(unused)]
fn main() {
impl ConversationMemory {
    pub fn build_system_prompt(&self, base_prompt: &str) -> String {
        match &self.summary {
            Some(summary) => {
                format!(
                    "{}\n\nPrevious conversation summary:\n{}",
                    base_prompt, summary
                )
            }
            None => base_prompt.to_string(),
        }
    }
}
}

Alternatively, you can add the summary as a user message at the start of the conversation:

#![allow(unused)]
fn main() {
impl ConversationMemory {
    pub fn get_message_summary(&self) -> Vec<Message> {
        let mut messages = Vec::new();

        if let Some(summary) = &self.summary {
            messages.push(Message::User {
                content: OneOrMany::one(UserContent::text(format!(
                    "Context from previous conversation:\n{summary}"
                ))),
            });
        }

        messages
    }
}
}

In practice, this is how you’d use it (the call_agent_with_chat_history helper prompts an agent with the stored history and appends the new turn to it):

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut mem = ConversationMemory::with_max_messages(2);

    let prompt1 = "What is the Rust programming language?";
    let _ = call_agent_with_chat_history(prompt1, &mut mem.messages).await?;

    let prompt2 = "Do you think Rust is a good language for AI development? I want to build an AI agent with Rust.";
    let _ = call_agent_with_chat_history(prompt2, &mut mem.messages).await?;

    println!("Message history: {:?}", mem.messages);

    let model = Client::from_env()?.completion_model("gpt-5.2");

    println!("Attempting to compact conversation and create a summary...");
    mem.compact(&model).await?;

    // SAFETY: We can guarantee that the summary exists here, barring any provider errors
    // which will cause fn main to return early
    let summary = mem.summary.unwrap();
    println!("Conversation summary: {summary}");

    let summary_message_plus_prompt =
        format!("Previous conversation summary:\n{summary}\n\nSorry what did we just talk about?",);

    let _ = call_agent_with_chat_history(&summary_message_plus_prompt, &mut mem.messages).await?;

    Ok(())
}

Note that as of Rig 0.38, the with_history method on a prompt request no longer appends the new conversation turn to the message list you pass in - you record the new user message and assistant response yourself:

#![allow(unused)]
fn main() {
async fn call_agent_with_chat_history(
    prompt: &str,
    history: &mut Vec<Message>,
) -> Result<String, Box<dyn std::error::Error>> {
    let openai_client = Client::from_env()?;

    // Reasoning with OpenAI requires a verified org which may break this example with GPT-5
    // however, using *no* reasoning is enabled with GPT-5.1 and GPT-5.2
    // hence adding the additional parameters here
    let additional_params = AdditionalParameters {
        reasoning: Some(Reasoning {
            effort: Some(ReasoningEffort::None),
            summary: None,
        }),
        ..Default::default()
    }
    .to_json();

    let agent = openai_client
        .agent("gpt-5.2")
        .preamble("You are a helpful assistant. Be concise.")
        .name("Bob") // used in logging
        .additional_params(additional_params)
        .build();

    println!("User: {prompt}");

    let response_text = agent.prompt(prompt).with_history(history.iter()).await?;
    println!("Assistant: {response_text}");

    // As of Rig 0.38, `with_history` no longer appends the new turn to the
    // caller's message list, so we record it ourselves
    history.push(Message::user(prompt));
    history.push(Message::assistant(&response_text));

    Ok(response_text)
}
}

Although you can see here that the compaction is manual, there’s a lot of ways you can build around it: you can check to see if the window needs compacting after every message, you could compact it based on token limit (although this would need a tokenizer to count tokens).

Strategies for Long-Term Memory

While conversation compaction handles ephemeral memory, many applications need to retain information across sessions. Here are three strategies for managing long-term memory:

1. Conversation Observations

Conversation observations capture insights about specific exchanges. These might include:

• Important decisions made during the conversation
• Questions that remain unanswered
• Topics the user expressed strong interest in

Implementation approach:

#![allow(unused)]
fn main() {
    pub struct ConversationObservation {
        pub timestamp: DateTime<Utc>,
        pub topic: String,
        pub insight: String,
        pub importance: f32,
    }
}

After each significant exchange, use an LLM to extract observations:

#![allow(unused)]
fn main() {
let extraction_prompt = format!(
    "Extract key observations from this conversation exchange. \
     Focus on decisions, preferences, and important context:\n\n{}",
    recent_messages
);
}

Store these observations in a database or vector store, retrieving the most relevant ones when starting new conversations.

2. User Observations

User observations track persistent information about the user themselves:

• Stated preferences (“I’m vegetarian”)
• Personal context (“I live in Seattle”)
• Communication style (“I prefer concise answers”)
• Long-term goals or projects

These observations should be:

• Maintained separately from conversation history
• Updated incrementally as new information emerges
• Verified before use to ensure they’re still accurate

Consider using a structured format:

#![allow(unused)]
fn main() {
    pub struct UserProfile {
        pub preferences: HashMap<String, String>,
        pub context: Vec<String>,
        pub communication_style: Option<String>,
        pub last_updated: DateTime<Utc>,
    }
}

Periodically ask the LLM to extract user observations from recent conversations:

#![allow(unused)]
fn main() {
let profile_prompt = 
    "Based on the recent conversations, extract any new information about \
     the user's preferences, context, or communication style. Return only \
     new or updated information.";
}

3. Grounded Facts

Grounded facts are verifiable pieces of information that emerged during conversations:

• External information retrieved during the session
• Calculations or analyses performed
• File contents or data processed
• API responses or database queries

These differ from observations because they’re objectively verifiable and often come from external sources rather than the conversation itself.

Store grounded facts with their source and timestamp:

#![allow(unused)]
fn main() {
    pub struct GroundedFact {
        pub fact: String,
        pub source: String,
        pub timestamp: DateTime<Utc>,
        pub confidence: f32,
        pub conversation_id: String,
    }
}

When starting a new conversation, retrieve relevant facts:

// Retrieve facts related to the current conversation context
// (fetch_facts_by_relevance is pseudo-code for your own retrieval function)
let relevant_facts = fetch_facts_by_relevance(current_topic, 5)?;

// Include them in the system prompt or as initial messages
let context = format!(
    "Relevant information from previous interactions:\n{}",
    relevant_facts.join("\n")
);

Caching

Generally speaking, effective memory systems don’t just have a single data source that it pulls from. The most production-capable systems often have multiple tiers of memory storage to accommodate for different needs. Typically, one of these should allow for fast fetching of the most commonly fetched items - using a caching layer. Whether it’s in memory, through Redis or Memcache (or perhaps another service dedicating to caching), caching is highly versatile and agentic memory is no exception.

How to create caches and use them is already a very well-trodden topic, and as such a full implementation will not be provided here. However, there are some highly useful crates you can use to help create your own memory cache:

• lru - an implementation of a Least Recently Used cache
• slotmap - A data structure with low overhead for storing, inserting and deleting items

Combining Memory Strategies

The most effective AI applications combine multiple memory strategies:

1. Ephemeral memory maintains immediate context through the conversation history and compaction
2. Observations capture insights about the conversation and user over time
3. Grounded facts preserve verified information that might be needed again

When building a new conversation context, you might:

• Load the user’s profile to personalize responses
• Retrieve relevant observations from past conversations
• Fetch grounded facts related to the current topic
• Maintain the current conversation in ephemeral memory
• Compact the conversation when it grows too large

This layered approach ensures your AI application has access to the right information at the right time, without overwhelming the model’s context window or your budget.