How AI Agents Actually Work: Loops, Tools, and Memory

“Agent” has become one of those words that gets stuck on everything. A chatbot with a search button is an agent now. So is a script that sends three emails. The word has been stretched so far that it's hard to tell what anyone means by it, which is a shame, because there's a real idea underneath, and it's simpler than the marketing suggests.

An agent is a language model running in a loop, with tools it can call and a way of keeping track of what has happened so far. This post goes through each of those parts using one running example: a support agent for an online shop that receives the message “My mug arrived broken. Order 4412. Can I get a refund?” By the end you should be able to sketch the whole thing on a napkin, and know where it tends to break.

Agent or workflow?

Before the mechanics, one distinction that saves a lot of wasted effort. A workflow is a fixed path you wrote in advance. Step one asks the model to classify the email, step two looks up the order, step three drafts a reply. The model does real work at each step, but your code decides what happens next.

An agent flips that. The model decides what happens next, one step at a time, based on what it just saw.

Workflows are more predictable, cheaper, and easier to debug. Agents cope with situations you couldn't map out ahead of time. If you can draw the whole flowchart of a task before you start, build a workflow. Reach for an agent when the number or order of steps depends on what turns up along the way: debugging a failing test, researching a question across a dozen sources, handling a customer whose problem doesn't fit any of your categories. Plenty of things sold as agents would work better, and cost less, as workflows.

The loop

A language model on its own does one thing: text in, text out. It can't look up an order or send a refund. An agent wraps that single step in a cycle. The model reads everything so far and decides on the next action. Your code carries it out, and the result goes back in for the model to read again. This repeats until the model replies without asking for anything.

flowchart TD
    A[Task from the user] --> B[Model reads everything so far]
    B --> C{Needs a tool?}
    C -- Yes --> D[Your code runs the tool]
    D --> E[Result added to the conversation]
    E --> B
    C -- No --> F[Final answer]

Here's about as small as a real agent gets, in rough Python:

messages = [{'role': 'user', 'content': task}]

while True:
    reply = model(messages, tools=tools)
    messages.append(reply)

    if not reply.tool_calls:
        break  # nothing left to do, so we're done

    for call in reply.tool_calls:
        result = run_tool(call.name, call.arguments)
        messages.append({'role': 'tool', 'content': result})

There's no planner module and no hidden reasoning engine. The model replies, your code checks whether it asked for a tool, runs it, and feeds the result back in. A reply with no tool request ends the loop. Everything that makes an agent product look sophisticated is built around those few lines: better prompts, safer tools, smarter handling of long tasks.

Here is what the loop does with the broken mug:

Nobody wrote “check the order, then check the policy, then refund.” The model chose that sequence itself, and it would have chosen differently if the order lookup had shown something odd. That freedom is the whole point of an agent, and also the source of most of its problems.

Plan first, or just go?

Some agents write a plan before they act, then work through it and revise as results come in. Others take one step at a time with no stated plan. Planning helps on tasks with many dependent steps, because it keeps the agent from wandering. Going step by step is cheaper, and fine when the next move is usually obvious from the last result. Many setups sit in between: a short plan at the start, with permission to change it.

Tools

The model can only produce text, so how does it do anything? It asks. A tool is four things: a name, a plain-language description, a schema for its parameters, and a real function in your code. You send the model the first three with every request. When it wants to use a tool, it writes out a structured call instead of a normal reply, and your code does the rest.

{
  "name": "issue_refund",
  "description": "Refund an order. Only for orders delivered in the last 30 days. Amount must not exceed the order total.",
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": {"type": "integer"},
      "amount": {"type": "number", "description": "Refund amount in dollars"}
    },
    "required": ["order_id", "amount"]
  }
}

The exact format differs between providers, but the shape is always like this. The model sees the description and the schema. It never sees your code and never touches your database. It writes a request, your code runs the function, and the output goes back to the model as text.

That gap between asking and doing is the most useful thing to understand about agents, because it's where the safety lives. The function that actually runs can say no:

def issue_refund(order_id, amount):
    order = db.get_order(order_id)
    if amount > order.total:
        return f'Error: refund exceeds the order total of ${order.total:.2f}'
    if amount > 50:
        return 'Error: refunds over $50 need human approval'
    queue_refund(order_id, amount)
    return 'Refund queued'

Notice that it returns an error message instead of raising an exception that crashes the loop. The model reads that message on its next pass and adapts, perhaps by handing the case to a person. Returning failures as ordinary results lets the agent recover from mistakes that would otherwise end the run.

It helps to think of permissions in tiers. Reading things, like looking up an order, can usually run freely. Reversible writes, like drafting a reply, can run with logging. Anything irreversible or expensive, like sending money, deleting data, or emailing a customer, gets a limit or a human approval step in your code, no matter how confidently the model asks. You don't have to trust the model to behave. You only have to make sure the tools can't do anything you wouldn't allow.

A few mistakes show up again and again:

• Vague descriptions. The description is the only thing the model knows about when to use a tool. “Gets data” is useless. “Looks up one order by ID and returns the item, total, and delivery status” is useful.

• Overlapping tools. Give an agent twelve tools where three do nearly the same thing and it will sometimes pick the wrong one. Five distinct tools usually beat a big drawer of options.

• Huge outputs. A tool that returns a 5,000-line file or a 300-row query result floods the context. Return what's needed and let the model ask for more.

• Actions that aren't safe to repeat. Models sometimes call a tool twice. Make writes idempotent, for example by checking whether a refund for that order already exists.

Most models can also request several tools in one turn, and standards like MCP exist so you don't hand-write every integration. None of that changes the loop.

Memory

Here's the part that surprises most people: the model doesn't remember anything. On every trip around the loop, the entire conversation so far is sent in again from the top. What looks like the agent remembering what it did two steps ago is just it re-reading that list. So everything about memory comes down to one question: what text should be in front of the model right now?

There are three layers, and they get mixed up constantly.

A bigger window doesn't solve it

It's tempting to think that a million-token context window means you can stop worrying. In practice, more isn't free. You pay for every token on every step, and models tend to use information less reliably when the context is long and cluttered. A smaller context holding the right things usually beats a huge one holding everything.

What people actually do

• Trim tool results. Once the agent has read a long file or query result, replace it with a short note on what mattered in it.

• Compact the history. When the conversation gets long, have the model summarize the older part into a few lines and carry on from the summary.

• Keep a scratchpad. A notes file the agent updates (“tried X, failed because of Y, next try Z”) survives trimming, and it lets the agent pick up after an interruption.

• Hand off to sub-agents. For a messy side task, like reading forty pages to find one number, start a separate agent with its own fresh context and have it return only the answer. The main agent's context stays clean.

Long-term memory is the same trick stretched across time. At the end of a run the agent writes something down, and a later run searches for what's relevant and pastes it into the context. When a product says its agent “remembers you,” this is almost always what's happening: saved notes, retrieved and put back in front of the model. The hard parts are deciding what's worth saving, and keeping stale notes from confidently steering the agent wrong.

Where agents go wrong

Once you know the loop, the failures are predictable. Most of them come from the same place: each step builds on the one before it, so small problems get carried forward.

That last row deserves some numbers. Say your prompt starts at 2,000 tokens and each step adds about 1,000 tokens of tool output and reply. Step 20 sends around 21,000 tokens, and the whole run processes roughly 230,000 input tokens. Twenty independent calls of the starting size would be 40,000. Prompt caching softens this a lot, because the repeated beginning of the conversation is billed at a discount, but the growth is still worth watching.

Building one, or judging one you're being sold

If you're building an agent, a sensible order of work looks like this:

1. Start with the smallest loop that could work: one model, two or three tools, and a hard step limit.

2. Log every model reply and every tool call. When an agent does something strange, the log is the only way to find out why.

3. Write a small set of test tasks with known good outcomes and run them after every change. Include some that should fail safely, like a refund request for an order that doesn't exist.

4. Put the guardrails in your code, not in the prompt. A prompt asks the model to behave. Code makes it.

5. Begin with boring tasks, and hand over risky tools only once the boring ones are reliable.

If you're on the buying side, three questions cut through most pitches. What tools can it actually call? What happens when it gets something wrong? What does it remember between runs?

Underneath the branding, an agent is a loop, a handful of tools, and some saved text. That's enough to be genuinely useful, as long as you're honest about what it is.