Should I train an AI model on my own code?
A short learning video. A renovation company owner whose coding agent keeps using old code, a friend’s advice to “train a model”, and a software engineer who shows the simpler fix.
What this video shows
Nora’s booking app has its own mail code. Her coding agent still sent emails the old way, and a friend told her to train a model on her code. Sam’s answer is usually no. A trained model learns a picture of the code from the day it was trained, and the code keeps changing. Today’s coding agents search your files before they write, so the agent already sees today’s code. What it did not have was the one rule that says which way is hers. One line in her rules file fixed it (how to write that file: “How do I make my coding agent follow my rules?”), and a test now keeps it that way.
Training does help in a few cases. A company with its own programming language, a giant like Google or Meta with its own model team, or a small fast model that only finishes lines can gain from it. The studies are not all on one side either: a Tencent study found that search beat training on its own, and that both together scored best, and an IBM study found training ahead for code completion on two company code bases. For most teams, the cheaper steps come first. More on how agents look things up: RAG in daily coding.
What you will learn
- The difference between training a model and letting the agent read your code.
- Why a model trained on your code goes out of date, and can get worse at other things.
- Which fix fits which problem: search, a rules file or today’s docs.
- The rare cases where training is the right choice. To pick the agent itself, see “Claude Code, Cursor or Codex: which coding agent for which job in 2026?”
Further reading
- OpenAI docs: “Model optimization” (prompting, context and when to fine-tune). developers.openai.com
- Google Cloud docs: “Gemini Code Assist code customization overview”. docs.cloud.google.com
- A Tencent team, arXiv 2505.15179, 21 May 2025: “RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry”. arxiv.org
- Boris Cherny on the Latent Space podcast (2025): why Claude Code uses agentic search instead of a code index. a write-up with his words
If you want your team’s coding agents set up to know your code and keep your rules, see how I can help or tell me about your project.
Transcript
0:03 Nora: Should I train an AI model on my own code?
0:07 Sam: Usually, no. You don’t need to change the model. Give it the right things to read. Your code, a short rules file, and today’s docs.
0:18 Nora: My coding agent doesn’t know my app. Yesterday it sent emails the old way, not with our own mail code. A friend told me: train a model on your code.
0:28 Sam: There are two ways to teach it. Training changes the model. It needs many examples and time, and you must do it again when your code changes. Letting it read changes only what it sees. It works at once, and it always sees today’s code.
0:48 Sam: Your code changes every day. A trained model learns a picture of last month’s code. It can also forget other things. In one study, a small model trained on one company’s code got much worse at general coding.
Nora: So it would know my old app.
1:06 Sam: Today’s coding agents read your code themselves. They search the files before they write, like a new developer would. Even Google’s coding assistant searches your private code. It does not train on it.
1:20 Nora: Then why did it use the old mail code?
Sam: Because nothing told it which way is yours. Write one line in your rules file: send every email with our mail code.
1:32 Sam: So match the fix to the problem. It doesn’t know your code? Let it search. It breaks your rules? Write them in the rules file. It uses an old version of a library? Give it today’s docs.
1:48 Nora: Is training ever the right choice?
Sam: Sometimes. When your company has its own programming language that no public model has seen. When you are as big as Google or Meta, with a team that trains models. Or for a small, fast model that only finishes the line you are typing.
2:07 Sam: And it is getting harder. OpenAI is closing its fine-tuning service. GitHub stopped its custom models trained on company code. And a trained model stops working when the model it was built on retires.
2:22 Nora: One line in my rules file. Today the agent used our mail code.
Sam: Good. Now add a test that checks it, so it stays that way.
2:32 Sam: Usually, no. Let the agent read your code. Add a short rules file, and give it today’s docs.
Have an app idea, a stuck app, or a daily job you want an AI agent to do?
Write three lines. I reply within one working day with a plain answer.
