Context engineering and memory

Story

Hide story

It is around eleven and the crew is changing over. The logbook is full: forty pages of delays, swaps and one cracked window in car three. The incoming guard has ten minutes and a cup of tea going cold.

The outgoing guard reads out what he remembers, which is mostly the last hour. “The rest is in there somewhere,” he says, and yawns.

Which of the forty pages belongs on the first one?

Concept#

Context engineeringContext engineeringDeciding what goes into the context window at each step. One common grouping (LangChain) names four moves: write, select, compress and isolate.Open the glossary entryClose is deciding what the model sees at each step: system instructions, tool definitions, retrieved documents, conversation history, memory and intermediate results. Anthropic describes it as curating and maintaining the best set of tokens during inference (Anthropic, 2025-09-29). Output quality, cost and latency depend on this choice, often more than on clever wording of one instruction.

Why It Matters#

Long-running agents and chat products often fail for context reasons, not because the model is “not smart enough”. The window fills up, irrelevant text distracts the model, a summary drops a rule, one tool dumps a huge result, or an old fact is treated as current. Each step also re-sends the context, so waste is paid for again and again (see Agentic patterns: ReAct, planning and memory).

A bigger window does not remove the problem. Cost and latency still grow with the tokens you send. Accuracy can also fall as input grows: Liu et al. (2023) found that models used information in the middle of a long input less reliably than information at the start or end, and Chroma’s 2025 “context rot” report found that performance became less reliable as input length grew across the 18 models it tested. Treat tokens as a budget to spend on purpose.

Mental Model#

The context window is a small desk. You decide which papers lie on it right now, which go in a filing cabinet (memory) and which a colleague (a sub-agent) handles on their own desk. A tidy desk is faster to work at, and nothing on it can be quietly lost when you clear it, because the important pages were filed first.

Technical Deep Dive#

The four moves#

The LangChain team groups the techniques into four moves (LangChain blog, 2025-07-02). Anthropic’s guidance describes the same ideas under other names, such as compaction, note-taking and sub-agents.

Anthropic calls clearing old tool results one of the safest, lightest forms of compaction, because a result the agent has already used rarely needs to stay in the window verbatim.

Why more context is not better#

• Cost and latency grow with tokens. Every agent step processes the history again.
• Accuracy can fall. Distracting or badly placed text hurts (see “lost in the middle” in Sampling and the model landscape and the context-rot report above). The size of the effect depends on the model and the task, so measure it on your own data.
• More text is more attack surface. Every document, tool result and memory is text an attacker may have written (see Model Context Protocol for how tool results carry injected instructions).

Layout for caching#

Providers can reuse the work already done for a prompt prefix, so a repeated prefix costs less and returns faster. The rule is simple: stable content first, variable content last. Put the system prompt, tool definitions and reference documents at the top, then the conversation, then the new user message.

Anthropic’s documentation states the order as tools, then system, then messages, and says that a change at one level invalidates that level and everything after it (docs checked 2026-10-09). So editing the tool list invalidates the whole cache, and putting a timestamp or the user’s name at the top of the system prompt invalidates the system prompt and everything after it on every request (the tools block can still hit). Caching has details that differ by provider: there is a minimum prompt length below which nothing is cached, and the cache lives for a limited time (5 minutes by default in Anthropic’s API). Read your provider’s current rules before you design around them.

Caching saves money and time. It does not fix context rot: a cached prompt that is too long and noisy is still too long and noisy.

Memory types#

MemoryAgent memoryState kept by an agent. The course splits it into four types: working, episodic, semantic and procedural. Short-term memory is the current context and scratchpad; long-term memory is stored outside and retrieved when relevant.Open the glossary entryClose is how an agent keeps information beyond one window. A common split, borrowed from cognitive science and used in the CoALA paper (Sumers et al., 2023), has four kinds:

CoALA describes procedural memory as knowledge in the model’s weights and in the agent’s code. In many products the practical form is instruction and skill files the agent loads when it needs them. This is a teaching split, not a standard every framework uses.

Designing memory: write, recall, forget#

Pick a type first, then answer five design questions.

1. What to write. Store what is likely to matter again (a stated preference, a decision, a completed task), not every message. Writing everything forever buys noise, cost and privacy risk.
2. When to write. At the end of a turn is simple. A background job after the session is cheaper for the user’s latency.
3. How to recall. Rank memories by more than similarity. The Generative Agents paper (Park et al., 2023) scores each memory as a weighted combination of recency, importance and relevance, then fits the top ones into the prompt. That is one design, not the only one.
4. How to update and forget. Handle conflicts (the user moved city), expiry (a price from last year) and deletion on request. A deletion has to reach the raw records, any summaries built from them, and the embeddings. Check your legal obligations with the right people. This course does not give legal advice.
5. Privacy. Decide what may be stored, keep one user’s memory isolated from another’s, and tell users what is remembered.

Treat recalled memory as data, not as instructions. If a memory was written from a web page or an email, it may contain an instruction an attacker planted. Recalling it later puts that text in front of the model with the authority of “something I knew”. This is memory poisoning, and the defence is the same as for any untrusted text: label it, limit what tools it can trigger, and require approval for risky actions.

Four ways context fails#

When an agent works for a while and then degrades, one of four things has usually happened. These four names are this course’s working taxonomy, not an industry standard. For each, pair a remedy with a number you can track.

Worked example. A refund agent was told “never refund more than 200 without a manager”. After 30 steps it approves a 350 refund. Its history was summarised twice, and the second summary reads “the agent processes refunds”. The constraint was lost in compression, which is drift. The fix is to keep the rule in the system prompt or a pinned note that compaction never rewrites, then add a regression test that asks the agent about the rule after a compaction.

Anti-patterns#

• Dumping the whole history into every call.
• Loading every tool definition when the task needs three.
• Passing raw tool output straight into the window.
• Storing everything forever.
• Trusting recalled memory as if it were instructions.

Diagram#

Diagram: The window holds a stable cached prefix, recalled memories, retrieved documents, compressed history and the current task.

Real-World Application#

A research agent keeps a plan file and notes outside the window, hands each web-research thread to a sub-agent that returns a short brief, and recalls user preferences from a profile store. Anthropic’s write-up of its multi-agent research system describes a lead agent that saves its plan to memory, because its context could be truncated once it grew past a limit (Anthropic, 2025-06-13). That is the write move used as a safety net.

Interview Perspective#

Expect “how do you handle long conversations”, “design memory for an assistant” and “the agent works for 10 steps and then degrades”. A strong answer names the failure (in this course’s terms: overflow, distraction, drift or bloat), the move that fixes it, and how you would measure the fix. Common mistakes: answering only “use a bigger context window”, having no forgetting or privacy story, and trusting recalled memory as instructions.

Practice Questions#

These are generated practice questions, not reported from any company.

• Basic: What are the four context moves, and which one is “summarise old turns”?
• Intermediate: How do you order a prompt so caching works, and what ruins it?
• Advanced: Summarisation drift made the agent forget a constraint. Fix it and say how you would test the fix.
• Product-Company: Design per-user memory for a customer-support assistant that supports deletion on request.

Activity#

Take a 40-turn support chat transcript (or write a short one with an invented customer). Mark each turn as keep verbatim, summarise, store as memory, or drop. For each memory you store, name its type and write one forget rule. Then pick one rule from the conversation and write the probe question that would show a summary has lost it.

Homework#

Spec a memory service for a support assistant. Include the schema, the write policy, a recall score (similarity, recency, importance, with your weights and why), update and forget rules, privacy controls, and an eval for “remembers right, forgets right”. State which numbers are your choices rather than facts.

Quick Revision#

• Four moves: write, select, compress, isolate.
• Stable prefix first for caching; an early edit invalidates everything after it.
• Memory: working, episodic, semantic, procedural. Define write, recall, forget and privacy for each.
• Failures: overflow, distraction, drift, tool-output bloat. Each has a remedy and a metric.
• More context is not better. Recalled memory is untrusted data.

Check yourself#

• Checkpoint question: An agent degrades after about 30 steps. Walk through the four failures, the evidence that tells them apart, and the fix for each.
• Independent re-test: Design memory that can safely forget a user’s data, including summaries and embeddings.
• Common misconception to avoid: “Just use the largest context window available.”

Related#

• Agentic patterns: ReAct, planning and memory
• Cost and latency optimization
• Multi-agent systems: when to use them and what they cost