AWS shows SFT fixes GPT-OSS-20B's English-only reasoning, GRPO adds less
GPT-OSS-20B tends to reason in English even when told to use French, German, Spanish or Italian, and AWS says supervised fine-tuning on Amazon SageMaker HyperPod fixed that. In a walkthrough on its machine learning blog, AWS reports that reasoning-language accuracy rose from 22 percent to 100 percent on a 50-case test.
- Base GPT-OSS-20B reasoned in the correct language only 22 percent of the time on a 50-case test, with 0 percent for French, Spanish and Italian.
- After SFT with LoRA for 1,000 steps, reasoning-language accuracy reached 100 percent (50/50), according to AWS.
- Answer-language accuracy after SFT was 62 percent (31/50), with Spanish lowest at 30 percent.
- AWS says SFT teaches missing behavior while GRPO optimizes behavior the model can already sometimes produce.
- The setup ran on Amazon SageMaker HyperPod with Amazon EKS, and the code is in the awsome-distributed-ai repository.
The problem: a reasoning model that thinks in English
GPT-OSS-20B is an open-weight reasoning model. According to AWS, when asked to reason in French, German, Spanish or Italian, it acknowledges the instruction and then produces its chain-of-thought in English anyway. On a 50-example multilingual reasoning test, the base model reasoned in the correct target language only 22 percent of the time (11 of 50).
That score came almost entirely from the English examples. Per language, the base model reached English 100 percent, French 0 percent, German 10 percent, Spanish 0 percent and Italian 0 percent. AWS argues this matters for teams serving non-English users, since a person who asks a question in French expects the model to reason and answer in French, not switch to English mid-thought.
The post uses the case to address a question teams face when customizing open-weight models: whether to use supervised fine-tuning (SFT), reinforcement learning with Group Relative Policy Optimization (GRPO), or both. AWS summarizes its answer this way: SFT does most of the work, GRPO refines when there is headroom to refine, and matching the technique to the capability gap decides the outcome.
Setup: dataset, evaluation and training stack
The team used the HuggingFaceH4/Multilingual-Thinking dataset, a chain-of-thought reasoning set translated into French, Spanish, German and Italian alongside English. It has roughly 949 training and 50 validation examples. The model receives a system-prompt instruction naming a reasoning language and must both reason and answer in that language.
The evaluation covers 50 test cases: 10 math and logic problems across five languages. Two metrics are tracked: whether the chain-of-thought is in the requested language, and whether the final answer is in the requested language.
AWS frames the two techniques with a simple rule. SFT teaches a behavior the model does not have, while GRPO optimizes a behavior the model can already sometimes produce. GRPO samples a group of responses per prompt and pushes the policy toward the above-average ones, so if the desired behavior has zero probability in the base distribution, there is very little to reinforce.
The runs used Amazon SageMaker HyperPod with Amazon EKS orchestration. The initial SFT and GRPO runs on GPT-OSS-20B used the PyTorch TRL library. For a control experiment on Qwen2.5-7B across two tasks with different capability gaps, AWS switched to veRL, with Ray, KubeRay and vLLM. Checkpoints and datasets sat on Amazon FSx for Lustre, the inter-node fabric used Elastic Fabric Adapter, and all fine-tuning used Low-Rank Adaptation (LoRA).
SFT results: reasoning fixed, answers lag
The SFT run used LoRA with r=8 and α=16, trained for 1,000 steps on 4× ml.g6e.12xlarge instances (16× L40S 48GB) with Fully Sharded Data Parallel, bf16 precision and gradient checkpointing. The job ran as an elastic HyperPodPyTorchJob managed by Kueue.
After SFT, the model reasoned in the correct language 100 percent of the time (50 of 50) across all five languages. According to AWS, learning happened in two stages. Around step 200 the model learned to reason in the target language. Around step 1000 it additionally learned to give the answer in the target language.
- Reasoning-language correct: 100 percent (50/50)
- Answer-language correct: 62 percent (31/50)
- Both correct: 62 percent (31/50)
- Per-language answer accuracy: FR 60 percent, DE 60 percent, ES 30 percent, IT 60 percent, EN 100 percent
The answer-language score leaves room for improvement, particularly in Spanish at 30 percent. The post says SFT is not optional here, because the base model cannot follow the reasoning-language instruction. The source text provided ends before the GRPO results and the full Qwen control findings.
What it means for teams customizing open-weight models
For teams adapting open-weight models to new languages or behaviors, the AWS post offers a practical order of operations. If the model cannot yet produce the behavior at all, start with SFT. GRPO is positioned as a refinement step for cases where the model already produces the behavior at least some of the time.
The post also publishes its commands, manifests and scripts in the awsome-distributed-ai GitHub repository, so teams with a SageMaker HyperPod cluster using Amazon EKS can reproduce the GPT-OSS-20B setup. AWS lists prerequisites including EFA-enabled instances, an FSx for Lustre file system and a HuggingFace access token for the model and dataset.
