How to Train or Prepare Your Own AI Model: Fine-Tuning vs RAG vs From Scratch

Founders ask to “train our own AI” when they mean several different jobs: clean the data, teach a model house style, retrieve from private documents, or actually run gradient descent on a network they own. Those jobs have different costs, risks, and odds of finishing. This guide is the decision path we use with CTOs in the US, UK, Europe and Australia before anyone rents a pile of GPUs.

DayDecoder implements applied AI on products — often RAG and fine-tuning against APIs, rarely a foundation model from scratch. We are a software company in Ahmedabad, not a GPU cloud. If your problem is “answer from our docs”, start with RAG, not with a training cluster.

Three different projects people mix up

You can combine them: retrieve, then a small tuned model to emit JSON. You almost never start by pretraining GPT-class models. The electricity bill is the least of the reasons.

Data prep is the product

1. Right to use. Employment contracts, customer terms, copyright. “It was in Slack” is not a licence.
2. PII and secrets. Strip keys, dump production passwords, and think about names. EU/UK retention and purpose limitation apply if those laws cover you. US sector rules may too.
3. Tenancy. A training set that mixes customers is how you leak one into another at inference, even if you later filter RAG.
4. Labels. Classification and extraction need agreed labels and a second annotator on a sample. Disagreement is data.
5. Splits. Time-based for anything with drift. Random splits leak the future in support tickets and trading (if you were even considering that).
6. Freeze and version. Dataset id on every eval run. If you cannot name the data, you cannot reproduce the model.

For RAG, “prep” is chunking, metadata, and deletion. For fine-tunes, it is pairs or traces you would be proud to show a customer. For from-scratch, it is a corpus and a legal memo.

Fine-tuning without a science fair

Use a provider’s fine-tune or an adapter (LoRA-style) on a model you already evaluate. Freeze an eval set first. If the base model plus good prompts plus RAG already hits the bar, stop. Fine-tunes shine for: stubborn JSON, classification with house labels, tone that prompting cannot hold. They do not magically teach last Tuesday’s price list.

Keep the adapter version next to the prompt version in logs. Roll back. Do not fine-tune weekly on yesterday’s tickets without a holdout; you will overfit the mood of the queue.

From-scratch and “open weights”

Running an open-weight model you did not train is not from-scratch. It is hosting. Hosting can be right for data residency. It is still ops: GPUs or a vendor, scaling, patching, and the same evals.

From-scratch (or heavy continued pretrain) is for when you have a domain a public model cannot cover and a team that has done it before. We will say no more often than yes. If you still want it, budget for people who live in experiment trackers, not for a single contractor and a dream.

Tooling we actually see

• Storage and a warehouse or bucket you control.
• A labelling tool or a boring spreadsheet for v1.
• Prompt and dataset git, or a registry.
• Eval harness (even a pytest over JSON fixtures).
• For hosting: vLLM, a managed endpoint, or a vendor fine-tune API.
• For RAG: the stack in the RAG article, not a new model.

Laravel/Node still own the product: flags, tenancy, queues. Training jobs are batch workers, not request handlers. See integrating AI into an existing app.

Indicative cost (not a quote)

People dominate. A clean dataset and eval for a focused classifier or extraction model is often a five-figure professional services slice. Provider fine-tunes add usage and training fees that are usually small next to engineering if the dataset is small. Hosting an open-weight 7B–70B-class model is a monthly GPU or vendor bill that you should get from current list prices, not from this page; it moves. Pretraining anything large is a different order of magnitude (compute plus specialists) and is out of band for most SMBs we speak to. These are typical/indicative observations, not a rate card. Token usage in production is the envelope we already described in chatbot cost.

If a vendor quotes a single number “to train your AI”, ask which of the three projects they mean, and what eval you get to keep.

A sequence we recommend

1. Write the job and the eval set. No GPU yet.
2. Prompt + RAG or a classifier API. Measure.
3. Fine-tune only if the eval is stuck on format or tone.
4. Host your own weights only for residency or cost at scale you have measured.
5. From-scratch only with a written reason the above cannot work.

DayDecoder: 10+ years, 103 customers. We will talk you out of a training cluster when a pipeline will do. contact@daydecoder.com.

Evaluation is the only scoreboard

Pick metrics that match the job: exact JSON validity, citation rate, human preference on a frozen set, calibration of “I don’t know”. Accuracy on a chatty demo is not a metric. For classifiers, look at the costly error (false 86 of a dish, false all-clear on a fraud hint) more than the average. For RAG, include questions that must refuse.

Automate the set in CI when it is cheap (JSON schema). Keep a human pass when the quality is subjective, on a schedule, not only when a VP is angry. When a provider model changes under you, the eval is how you notice. That is also why you pin versions in production.

People, vendors and who owns the artefact

You should own the dataset documentation, the eval set, and the adapter files. A vendor who fine-tunes but will not give you the artefact is renting you a mood. Prefer exports. If you use a closed fine-tune API, you still own the examples and the scores. Put that in the contract, next to the NDA.

We work in your cloud when the data is sensitive. Training jobs should not copy production databases to a laptop. Snapshots, minimised extracts, access logs. Ordinary security, applied to a trendier pipeline.

When we will recommend “do not train”

When the corpus is a dozen PDFs (use RAG). When labels disagree more than they agree (fix the process). When the sponsor wants a foundation model for a support widget (read the cost article). When the only data is other people’s copyrighted books. When nobody will own eval after week two. Saying no is part of the job. You can still hire us to build the retrieval product that solves the actual user problem.

A note on agents and tools

Tool-using agents are not a training problem first. They are an authorization problem. A small model with tight tools and a large model with a browser and your admin cookie are different risk classes. Train or not, the policy layer stays. See the integration article for tool allowlists. Do not fine-tune your way out of a missing can('refund') check.

Licensing of base models and generated weights

Open-weight does not mean “do what you like”. Read the model licence: attribution, acceptable use, whether you may offer the model as a service, whether you may use outputs to train another. Fine-tunes can inherit restrictions. We are not your counsel, but we will not ignore a licence we can read. Closed APIs have their own terms about whether they train on your data; for EU/UK customers that is often the first procurement question, and it should be.

Your own eval set and labels are an asset. Keep them. If a contractor walks away with only a Slack export, you do not have a model programme. You have a memory of one.

If you need a commercial arrangement for data labelling, write quality SLAs (agreement rate, gold questions) the way you would for any BPO. Cheap labels that disagree are a tax on engineering, not a saving. Gold questions you already know the answer to should be sprinkled into every batch so you can see quality without waiting for a post-mortem.

Version the labelling guide when you change a definition (“refund” vs “partial refund”). Old labels keep their guide id. Mixing definitions is how F1 scores go up while the product gets confused. This is dull. It is also the difference between a model programme and a pile of CSVs named final_final_v3.

Handover checklist

A model programme, even a small one, hands over: dataset id and licence notes, labelling guide version, eval set, adapter or prompt versions, where weights live, who can promote, and how to roll back to RAG-only. Include the “do not train” reasons you already accepted so a future sponsor cannot reopen GPU theatre without reading them. Include PII handling and deletion. If deletion does not reach the vector store and the fine-tune set, you do not have deletion. You have a blog post about deletion. We will not sign off until the path is real or explicitly out of scope with a date to return. Put that date on a calendar, not in a chat thread that will be lost when someone leaves. If the date slips, write why; silent slips are how GPU theatre returns wearing a new vendor logo.

What “good enough” looks like before anyone rents GPUs

Write a one-page scorecard before procurement: the job in a sentence, the frozen eval size, the error that would embarrass you in production, the human who will review a sample every week, and the rollback (prompt-only, RAG-only, or a previous adapter). If you cannot fill that page, you are not ready to train. You are ready to collect examples and argue about labels, which is cheaper and more valuable.

A useful threshold is boring: the model plus retrieval beats a senior teammate on the frozen set for the narrow job, with citations or structured output you can check, and it does not invent policy. “It sounded confident in the demo” is not a threshold. Neither is a vendor leaderboard. Your eval, on your documents, with your refuse cases, is the only number that should move a budget line.

Keep a shadow period: the new adapter or hosted model answers in logs, the old path still serves users, and a person compares. Promote when the scorecard holds for a stretch you named in advance, not when a stakeholder is excited on a Thursday. That is the same discipline as a feature flag on any other Laravel or Node release. Training does not get a special pass to skip staging.

FAQ

Is fine-tuning the same as “the model learned our business”?

No. It learned patterns in the examples. Facts still belong in retrieval or a database. Treating a fine-tune as a knowledge base is how you ship confident stale answers.

Can we use customer tickets as training data?

Only with the right to use them, PII handling, and tenancy. Even then, a holdout and an eval that includes “refuse” cases. Tickets also contain prompt-injection-like text. Treat it as untrusted.

Should we train on our codebase?

Sometimes for internal assistants, with repo access rules and no secrets in the set. For a customer-facing product, RAG on docs plus tools into git is usually safer than a trained-in snapshot of last quarter’s code.

Which base model?

The one that wins your frozen eval and meets residency and procurement. Keep a provider layer. We will not name a winner here that will be wrong in six months.

Will you keep a copy of our training data?

Not as a souvenir. Data stays in your cloud when we work that way, with deletion you can audit. Spell it in the NDA and the proposal. That is the same rule as the rest of our custom software work.