Fine-Tuning 101: Adapting LLMs for Your Test Standards

The Problem With Prompting Alone

A general language model knows how to write tests. It has seen millions of them. What it has not seen is your test framework, your naming conventions, your helper utilities, or the specific class signatures in your codebase. When it generates a test that calls a method that does not exist, or produces assertions in a style your team has explicitly moved away from, the issue is not the model’s reasoning. It is the gap between what the model was trained on and what your project actually looks like.

There are three ways to close that gap: add more context to each prompt, retrieve relevant examples from your codebase at generation time, or train the model to internalize your patterns. Each handles a different part of the problem, and understanding what each does well is what lets you pick the right one.

What Fine Tuning Actually Does

Fine tuning continues training on new examples after the base model has already learned general language and code patterns. For a QA team, the new examples are tests from your own suite: a focal method paired with its corresponding test, assertion patterns your team uses, setup and teardown conventions specific to your framework.

The goal is not to teach the model to write tests from scratch. It already knows that. The goal is to shift the model’s output distribution so that its default output matches your standards rather than the statistical average across all the public code it was trained on.

A December 2024 empirical study by Siddiq et al. that evaluated 37 models across three testing tasks found that adapted models outperformed the prior state of the art on assertion generation and test evolution, with CodeLlama-7b reaching 71.42% exact match on assertion generation against a baseline of 31.42% from ATLAS. On test generation, the picture was more complicated: the best adapted model, DeepSeek Coder-6b, reached 33.68% correct on Defects4J, while GPT-3.5 using prompting reached 49.16%. Adaptation alone was not sufficient to beat a large model with prompting on test generation (Siddiq et al., “A Large-scale Empirical Study on Fine-tuning Large Language Models for Unit Testing,” arXiv:2412.16620, 2024).

The finding matters for QA teams: model adaptation is not a universal win. It excels at tasks where consistency with a known pattern matters most, such as assertion style and test structure. For generating complete novel tests, retrieval and prompting still have an edge.

Three Approaches: Full Training, LoRA, and RAG

You do not need to update every weight in a model to achieve behavior specific to your codebase. Methods like LoRA (Low-Rank Adaptation) update a small fraction of the parameters while freezing the rest. A 2024 study by Li et al. that tested ten code generation models found that LoRA delivered performance comparable to full training in several cases: on the HumanEval-X Java benchmark, LoRA on StarCoder2-3b produced a pass@1 of 0.3675 versus 0.3494 for the full approach, with 99.35% instruction coverage and 87.06% branch coverage on tests that compiled (Li et al., “Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study,” arXiv:2411.02462, 2024).

The practical advantage is compute. Full training of StarCoderBase (15.5 billion parameters) required six 40GB A100 GPUs. LoRA trained the same model on a single 80GB A100. For a team without a large GPU cluster, LoRA is the viable path to a model trained on your own codebase.

Retrieval-augmented generation (RAG) takes a different approach: instead of training the model, it retrieves relevant code from your repository at generation time and adds it to the prompt. A 2025 study from Tencent WXG that compared RAG and model adaptation on a 120,000-file C++ codebase found that RAG with BM25 retrieval reached an average exact match of 53.76, while the trained approach reached 44.20. Combining both improved exact match further to 57.43. Importantly, training on the company codebase caused average pass@1 to drop 24.1% on HumanEval and 11.7% on MBPP, suggesting the model lost some general capability in exchange for domain alignment (Tencent WXG study, arXiv:2505.15179, 2025).

The takeaway for QA teams: RAG preserves general capability and scales better as codebases grow. Model adaptation gives stronger domain alignment but requires dataset curation and carries the risk of degrading the model’s broader usefulness. Used together, they cover both concerns.

Building a Training Dataset From Your Test Suite

For both full training and LoRA, the dataset is where most of the quality work happens. A model trained on inconsistent or low quality examples will produce inconsistent outputs. The dataset must represent what you actually want the model to do.

For test generation, each training example is a focal method paired with a corresponding test. For assertion generation, each example is a test method with the assertion replaced by a mask, paired with the correct assertion. For a custom test style, examples should cover the patterns you use most: your assertion library, your mock setup, your naming convention for test methods.

A 2026 study on test code generation at Ericsson used an 8x7b mixture of experts model trained on a custom prompt template that included dependent class signatures, public method lists, and two retrieved exemplar tests per example. Training improved the model by about 8% on average over its base version on the method conformity metric, reaching performance comparable to a much larger 8x22b model. Without the retrieved context in the training prompt, the base model consistently invented functions and signatures that did not exist in the codebase (Arakelyan et al., “Test code generation at Ericsson using Program Analysis Augmented Fine Tuned LLMs,” arXiv:2506.11006, 2025).

The Ericsson work illustrates a pattern worth adopting: include the actual signatures your tests will call in the training prompt. A model that has seen these signatures during training will reference them correctly at generation time, rather than generating plausible but incorrect alternatives.

Where to Start

The practical sequence for a QA team is straightforward.

Start with RAG. Extract your test suite’s most representative examples, build a retrieval index, and use it to inject context at prompt time. This requires no training run and produces immediate improvements on tasks where your codebase has unique identifiers or patterns the base model has not seen.

If RAG is in place and you still see systematic style or structure problems, a LoRA run on a curated dataset of 5,000 to 10,000 test pairs is the next step. Measure pass@1 and coverage on a reserved test set before and after to confirm the model improved rather than just shifted.

Avoid full training unless you have the GPU budget and a dedicated evaluation pipeline. The compute cost is substantial, and the risk of losing general capability is real.

For practical implementation of these patterns using Python and LangChain, the LangChain for QA: Build RAG, Agentic RAG and AI Chatbots course covers retrieval setup, vector indexing, and how to combine retrieval with generation in a QA workflow.

Teams building the broader AI testing skill set will find the architecture and evaluation patterns in the Generative AI and AI Agents for QA SDET Masterclass, which covers transformer fundamentals, agent design, and how to evaluate model output quality in a test automation context.