A rehearsal project: a fine-tuned generator proposes several candidate solutions to a grade-school math problem, a calculator pass checks the arithmetic, and a fine-tuned verifier picks the one it trusts most.
Measured on a 50-question held-out split of GSM8K (95% interval roughly 58–83%). On this sample both scored 36 out of 50, so the verifier gave no measurable gain over greedy decoding.
Phi-3.5-mini-instruct (Unsloth’s 4-bit build) fine-tuned with QLoRA directly on GSM8K question/answer pairs, teaching the base model GSM8K's reasoning and answer-annotation style.
Retrained from the base model on the full GSM8K set plus the model’s own completions that reached the correct answer, sampled several per question — reinforcing the reasoning patterns that actually worked.
Sampled candidates were labeled correct/incorrect per question to build preference pairs (chosen vs. rejected), and the RFT model was tuned with DPO to directly prefer its own correct completions over its incorrect ones for the same problem.
At inference, every candidate's arithmetic is independently recomputed: each <<expr=result>> annotation the model writes is parsed and evaluated with a restricted AST-based evaluator — only +, −, ×, ÷ and parentheses, no eval() — then compared against the model's stated result within a small tolerance. Any candidate with a wrong step is vetoed before the verifier ever sees it; if every candidate fails, the full set is passed through rather than discarding a genuine attempt.
A separate meta-llama/Llama-3.2-1B-Instruct was fine-tuned as a binary correctness classifier on question+candidate pairs labeled from that same data — used at inference time to score the calculator-valid candidates and pick the one it's most confident is correct.
The fine-tuned generator alone reaches 72% on this held-out sample. The calculator check and the verifier did not add measurable accuracy on top of that, and 50 questions is too few to detect a small gain. The verifier was trained on a small dataset (about 800 examples), so it is best read as a proof of concept for the generate → check → select pipeline.