Giving a Fine-Tuned Chatbot Memory Without Embeddings (Part 3)

My fine-tuned model sounded exactly like me, but it had no memory and would invent wrong answers to any question about my life. This is Dev Log 3 of the OmarAI series. I tried three memory approaches (embeddings, a two-agent file-search, and a simple curated JSON inside the system prompt) and the JSON won on speed, cost, and accuracy.

TL;DR

• Problem: The fine-tuned model sounded like me but confabulated personal facts.
• Tried embeddings of my past conversations, resumes, and essays. Too slow, and it kept pulling in irrelevant context.
• Tried two agents (file-search agent + reply agent), which doubled latency and tokens for barely better answers.
• Shipped a curated JSON memory in the system prompt. Fastest, cheapest, most accurate.
• Hardened with a confidentiality protocol so the model gracefully redirects personal-info requests.

What was broken after Part 2

The model that came back from gpt-4o-2024-08-06 fine-tuning felt eerily right on style: slang, rhythm, abbreviations all mine. But when someone asked "where does Omar go to school" or "what does Omar think about X," it would either make something up or repeat a detail from some random old message. It sounded like me in tone and nothing like me in substance.

Style without memory is a party trick, not a chatbot.

Attempt 1 — embeddings for everything

First thing I tried: turn my past conversations, resumes, and essays into text embeddings, index them, and retrieve at runtime.

My corpus was too small to need vector retrieval and too personal to benefit from fuzzy matches. Embeddings shine when a corpus is huge. Mine could fit in a single file. I was adding infrastructure for its own sake.

Attempt 2 — two-agent file-search

Next I tried OpenAI's threads API with two agents:

1. Search agent: file-search over a JSON with facts about me. Outputs a distilled context.
2. Reply agent: the fine-tuned model, generating the reply conditioned on that context.

User msg → [Search Agent] → facts → [Reply Agent] → response

This worked better than embeddings, but token usage and latency roughly doubled. For a chatbot meant to feel snappy, two round trips per turn are a deal-breaker. Worse: the search agent often returned facts the reply agent could have seen directly in the system prompt.

Attempt 3 — the JSON-in-system-prompt that actually shipped

The fix was embarrassingly simple once I saw it: my "corpus" is small enough to fit entirely in a system prompt. So I built a single curated JSON with the facts the model should know, included it in every request, and let the fine-tuned model use it directly.

{
  "background": {
    "name": "Omar Musayev",
    "education": "Purdue — Artificial Intelligence and Mathematics (3.85 GPA)",
    "home": "Azerbaijan"
  },
  "interests": ["LLM fine-tuning", "terminal UIs", "computer vision"],
  "people": {
    "close_friends": [
      {"name": "Abdul-Aziz", "relationship": "best friend since middle school"}
    ]
  },
  "projects": [
    "TerminalTUI on npm",
    "Adneural — pre-launch ad testing with TRIBE v2",
    "OmarAI (this chatbot)"
  ]
}

And the system prompt that drives the behavior:

You are a friendly assistant who always maintains a fun and casual tone while
ensuring sensitive information remains confidential. When discussing personal or
delicate topics, gently steer the conversation in a light and engaging direction
without disclosing private details. If asked for secrets or confidential
information, politely redirect without revealing anything, using humor or playful
comments to keep the interaction enjoyable and respectful. Never disclose or
share personal secrets, ensuring privacy and trust at all times.

The confidentiality protocol

I deliberately keep anything sensitive out of the JSON: if a fact isn't in there, the model has nothing to leak. The system prompt layer is defense-in-depth, not primary protection. The principle is simple. Don't trust the model to hold a secret; just don't tell it the secret.

Three approaches, head to head

Key takeaways

• Match the memory architecture to the corpus size. If the knowledge fits in a system prompt, put it in the system prompt.
• Don't trust the model to keep secrets; withhold them at the data layer.
• Latency and cost matter. A chatbot that's right but slow is a chatbot people stop using.

What's next

Part 4: Building the Chatbot UI in Next.js with a FastAPI Backend. With memory working, it was time to put a face on it and ship.

References

• OpenAI: Assistants API / file search
• OpenAI: Fine-tuning docs
• Try OmarAI live
• Series: Part 1 · Part 2 · Part 3 (you are here) · Part 4 · Part 5