How we built Agent Builder’s memory system

We launched LangSmith Agent Builder last month as a no-code way to build agents. A key part of Agent Builder is its memory system. In this article we cover our rationale for prioritizing a memory system, technical details of how we built it, learnings from building the memory system, what the memory system enables, and discuss future work.

What is LangSmith Agent Builder

LangSmith Agent Builder is a no-code agent builder. It’s built on top of the Deep Agents harness. It is a hosted web solution targeted at technically lite citizen developers. In LangSmith Agent Builder, builders will create an agent to automate a particular workflow or part of their day. Examples include an email assistant, a documentation helper, etc.

Early on we made a conscious choice to prioritize memory as a part of the platform. This was not an obvious choice – most AI products launch initially without any form of memory, and even adding it in hasn’t yet transformed products like some may expect. The reason we prioritized it was due to the usage patterns of our users.

Unlike ChatGPT or Claude or Cursor, LangSmith Agent Builder is not a general purpose agent. Rather, it is specifically designed to let builders customize agents for particular tasks. In general purpose agents, you are doing a wide variety of tasks that may be completely unrelated, so learnings from one session with the agent may not be relevant for the next. When a LangSmith Agent is doing a task, it is doing the same task over and over again. Lessons from one session translate to the next at a much higher rate. In fact, it would be a bad user experience if memory is not present – that would mean you would have to repeat yourself over and over to the agent in different sessions.

When thinking about what exactly memory would even mean for LangSmith Agents, we turned to a third party definition of memory. The COALA paper defines memory for agents in three categories:

Procedural: the set of rules that can be applied to working memory to determine the agent’s behavior Semantic: facts about the world Episodic: sequences of the agent’s past behavior

How we built our memory system

We represent memory in Agent Builder as a set of files. This is an intentional choice to take advantage of the fact that models are good at using filesystems. In this way, we could easily let the agent read and modify its memory without having to give it specialized tools - we just give it access to the filesystem!

When possible, we try to use industry standards. We use AGENTS.md to define the core instruction set for the agent. We use agent skills to give the agents particular specialized instructions for specific tasks. There is no subagent standard, but we use a similar format to Claude Code. For MCP access, we use a custom tools.json file. The reason we use a custom tools.json file and not the standard mcp.json is that we want to allow users to give the agent only a subset of the tools in an MCP server to avoid context overflow.

We actually do not use a real filesystem to store these files. Rather, we store them in Postgres and expose them to the agent in the shape of a filesystem. We do this because LLMs are great at working with filesystems, but from an infrastructure perspective it is easier and more efficient to use a database. This “virtual filesystem” is natively supported by DeepAgents - and is completely pluggable so you could bring any storage layer you want (S3, MySQL, etc).

We also allow users (and agents themselves) to write other files to an agent’s memory folder. These files can contain arbitrary knowledge as well, that the agent can reference as it runs. The agent would edit these files as it’s working, “in the hot path”.

The reason it is possible to build complicated agents without any code or any domain specific language (DSL) is that we use a generic agent harness like Deep Agents under the hood. Deep Agents abstracts away a lot of complex context engineering (like summarization, tool call offloading, and planning) and lets you steer your agent with relatively simple configuration.

These files map nicely on to the memory types defined in the COALA paper. Procedural memory – what drives the core agent directive – is AGENTS.md and tools.json. Semantic memory is agent skills and other knowledge files. The only type of memory missing is episodic memory, which we didn’t think was as important for these types of agents as the other two.

What agent memory in a file system looks like

We can look at a real agent we’ve been using internally – a LinkedIn recruiter – built on LangSmith Agent Builder.

• AGENTS.md: defines the core agents instructions
• subagents/: defines only one subagent linkedin_search_worker: after the main agent is calibrated on a search, it will kick off this agent to source ~50 candidates.
• tools.json: defines an MCP server with access to a LinkedIn search tool
• There are also currently 3 other files in the memory, representing JDs for different candidates. As we’ve worked with the agent on these searches, it has updated and maintained those JDs.

How memory editing works: a concrete example

To make it more concrete how memory works, we can walk through an illustrative example.

Start: You start with a simple AGENTS.md: Summarize meeting notes.

Week 1: The agent produces paragraph summaries. You correct it: "Use bullet points instead." The agent edits AGENTS.md to include formatting preferences.

Week 2: You ask the agent to summarize a different meeting. It reads its memory and uses bullet points automatically. During this session, you ask it to: "Extract action items separately at the end." Memory updates again.

Month 3: The agent's memory includes formatting preferences, domain-specific terminology, and roles of frequent participants. The AGENTS.md built itself through corrections, not through upfront documentation.

Learnings from building this memory system

The hardest part is prompting: In almost all cases where the agent was not performing well, the solution was to improve the prompt. We had to ensure the agent remembers when it should and doesn't when it shouldn't.

Validate file types: Several files have specific schemas that must be maintained for the system to function correctly.