The controversy around Chinese AI models begins with a practical question: how can a smaller model answer like a larger one? Distillation is a legitimate method for model compression and transfer, but collecting a teacher model’s outputs at scale also raises API-terms and data-rights questions. Our earlier Kimi K3 distillation controversy examined that boundary, while the related AI model distillation article covers disclosed model derivatives. This article focuses on how an AI distillation technique is implemented through data and code.
The API examples below assume permission from the teacher-model provider, a valid contract, and a lawful basis for the data. They do not cover account evasion, abnormal automation, rate-limit bypass, or theft of private outputs. Use them only with an authorized API, an open model, or a model you control.
1. The foundation: what is transferred?
Knowledge distillation, formalized by Hinton, Vinyals, and Dean in 2015, transfers knowledge from a large teacher network to a smaller student network. The teacher supplies more than one correct label: it supplies a probability distribution over alternatives, a “soft target.” The student learns both the hard label and the teacher’s relationships among alternatives. The original paper made this a foundation for model compression.
LLM distillation has two broad paths. White-box distillation can access weights and logits and match intermediate representations, logits, or attention. Black-box distillation can access only an API, so it creates prompts and stores the teacher’s answers, preferences, code, or reasoning traces. The 2024 survey on LLM distillation separates these paths and discusses API-based in-context, chain-of-thought, and instruction-following distillation.
| Type | Teacher signal | Student objective | Strength | Limitation |
|---|---|---|---|---|
| Logit distillation | Token logits and probabilities | Temperature-scaled KL divergence | Information-rich and stable | Requires teacher weights or logits |
| Feature distillation | Intermediate features and attention | Representation alignment and feature loss | Transfers internal representations | Difficult across different architectures |
| Response distillation | Text answers, preferences, code | SFT, DPO, or preference optimization | Works through an API | Copies teacher errors, style, and bias |
| Reasoning-trace distillation | Stepwise solutions and verification | Reasoning SFT or RL | Can improve small-model problem solving | A trace is not necessarily authentic reasoning |
The basic loss can be written as follows. T is the temperature. A higher temperature smooths the teacher distribution so the student can learn which wrong answers are less implausible.
In API distillation, the student learns answers rather than logits. “More distillation data is always better” is therefore wrong. The student may also copy hallucinations, arithmetic mistakes, refusals, and verbosity. Provenance, verification, difficulty, duplication, and personal data matter before model size does.
2. Collecting distillation data: generate questions and preserve the record
2.1 A minimum schema
Distillation data collection is not simply sending prompts to a teacher and appending answers to a file. The lineage of each request and response must be stored so that the run can be reproduced, deleted, and audited later.
First check whether the teacher’s terms permit reuse for training. If user inputs are reused, define a policy for consent, personal-data removal, and retention. Repeating the same question increases cost while reducing diversity. Sample language, length, domain, and difficulty in balance.
2.2 A rate-limited collector for an authorized API
The following assumes an OpenAI-compatible API. It reads the key only from the environment, records spacing, retries, hashes, and metadata, and must be adapted to the provider’s documented endpoint and response shape.
This is not a bypass tool for mass scraping. In production, use the provider’s official Batch API, quotas, retention settings, and spending cap. Restrict access to raw responses because they may be sensitive, and decide whether the originals should be deleted after training.
3. Cleaning: turn responses into learnable conversations
Raw JSONL is not automatically a fine-tuning dataset. Remove empty responses, duplicates, excessively long samples, personal information, and samples without a verifiable answer. Mark only samples that pass a domain validator—equation execution, code tests, or label checks—as verified; split train, validation, and test sets by question ID.
The goal is not to copy every teacher mannerism. Define the target capability first. Add executable tests for a code model, a checker for mathematics, or policy and refusal criteria for support. If the prompt, verification result, and license metadata are discarded along with the teacher answer, the model’s provenance becomes impossible to explain.
4. Select a base model and fine-tune the student
Choose a base model for the task and its license, not simply the largest available model. Check whether its tokenizer represents the target language and code well, whether commercial use, derivatives, and distillation are allowed, and whether context length and GPU memory fit the job. A license conflict can make a technically successful training run impossible to distribute.
Transformers provides model-specific chat templates. Do not invent role tokens; use apply_chat_template. The Transformers chat-template guide explains that models use different control tokens such as [INST] and <|user|>, and that duplicated special tokens can damage performance. The TRL SFTTrainer documentation defines preprocessing parameters such as dataset_text_field and max_length.
Start with LoRA/PEFT instead of updating every weight when GPU memory and experiment cost are limited. This model fine-tuning stage should be treated as an experiment with a held-out test set, not as proof of lineage. Check the license and merge procedure again when an adapter is merged for distribution. A falling training loss does not prove successful distillation: compare unseen questions, alternate phrasing, refusal behavior, long context, and executable code.
5. Two recent papers point to the next step
DA-KD: stop spending teacher budget on easy samples
DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models is an ICML 2025 paper. It argues that existing methods ignore difficulty differences and spend the same distillation cost on easy samples. Its difficulty-aware framework dynamically adjusts the dataset. The practical lesson is to spend teacher budget on what the student still cannot solve rather than simply growing the dataset.
RLKD: transfer structure beyond a flat reasoning trace
Distilling the Implicit Multi-Branch Structure in LLMs’ Reasoning via Reinforcement Learning is a 2025 study called RLKD. It argues that SFT on a single teacher reasoning trace can collapse the hidden branching of problem selection and solving into flat token imitation. It proposes a Generative Structure Reward Model to score structural alignment and uses RL to optimize it. Its results should be read within the authors’ experiments, not as a universal rule that RL distillation always beats SFT.
The papers identify different bottlenecks. DA-KD focuses on which data deserves teacher budget; RLKD focuses on the difference between copying an answer and learning a problem-solving structure. A production pipeline should design those two decisions separately.
6. How to calculate API distillation cost
API distillation cost is not the number of requests alone. Add input tokens, output tokens, cache and batch discounts, retries, teacher calls for verification, and failed requests.
The screenshot’s estimate assumes 500 input tokens and 1,000 output tokens per sample. At 200,000 samples, that is 100 million input tokens plus 200 million output tokens. Using the screenshot’s Luna example ($1 input and $6 output per million tokens), the standard API calculation is 100×$1 + 200×$6 = $1,300 (about ₩1.82 million), and Batch is $650 (about ₩0.91 million). Adding a 30–50% operating reserve gives roughly $845–$975 (about ₩1.18–1.37 million) for Batch. The $220 shown in the screenshot uses 20 million output tokens by mistake; under the stated 200,000-sample and 1,000-output-token assumption, $1,300 is the consistent result.
Actual prices vary by model, date, provider, and account. Using official prices checked on July 31, 2026, the same 200,000-sample workload is approximately as follows. The conversion uses $1 = ₩1,400 for illustration only; exchange rates, tax, and account discounts are excluded.
| Model and price per 1M tokens (input/output) | Standard API | Batch (50%) |
|---|---|---|
| OpenAI GPT-5.6 Sol $5/$30 | $6,500 (about ₩9.10M) | $3,250 (about ₩4.55M) |
| OpenAI GPT-5.6 Terra $2/$12 | $2,600 (about ₩3.64M) | $1,300 (about ₩1.82M) |
| Gemini 3.5 Flash $1.50/$9 | $1,950 (about ₩2.73M) | $975 (about ₩1.37M) |
OpenAI lists model-specific input and output rates in its model comparison, and documents 24-hour processing with a 50% discount for Batch. Google lists separate standard and Batch rates for Gemini 3.5 Flash in its official pricing table. The screenshot’s $650 is therefore a valid low-cost-teacher scenario, not a universal price for current frontier models.
Budget for failed retries (5–10%), teacher regeneration or multiple candidates (10–30%), LLM judging and verification (10–30%), and evaluation-set generation (3–5%). For example, if Terra Batch costs $1,300 and retry plus verification overhead is assumed to be 40%, the working budget is about $1,820 (about ₩2.55 million). Use a cheaper verifier, filter easy samples before teacher calls, and apply caching and Batch where latency permits. These are approximate budgets derived from public rates and assumed token counts, not guaranteed invoices: actual charges depend on input/output/reasoning tokens, cache hits, failed requests, exchange rates, and taxes.
7. Why frontier companies cannot identify every distillation attempt
Large providers cannot block every attempt because the attacker and provider have asymmetric visibility.
- Legitimate use and collection can look identical at the API level. Education, evaluation, and customer support can also send many sequential questions.
- An attacker can distribute work across accounts, regions, and applications and mix collection prompts with ordinary queries. The provider must avoid blocking legitimate users through false positives.
- Outputs do not carry an owner’s definitive secret key. Style, answers, and refusal patterns are clues; public data and similar prompts can produce them too.
- Providers may not retain every prompt and answer for long periods because of privacy, security, and contracts. The logs available for analysis are limited.
- Several teachers may contribute to one student. Public data, synthetic data, human editing, and multiple models can be mixed, making attribution difficult.
Detection therefore combines abnormal query distributions, account graphs, timing, error rates, output reuse, and behavioral similarity rather than relying on one sentence watermark. Those signals can support terms enforcement and security actions, but they do not automatically prove distillation in court. Providers need rate limits, account verification, output provenance, and contractual training restrictions together; researchers need to record permission and data lineage.
Conclusion: distillation is a data contract before it is a code snippet
The heart of an AI distillation technique is not one line in SFTTrainer. It is a reproducible lineage showing which teacher received which questions, who verified the answers, and what the student was trained to learn. Distillation with permitted open-model outputs, self-generated data, and clearly licensed evaluation sets can reduce cost, latency, and deployment size. The same algorithm becomes industrially corrosive when private-service outputs are collected by evasion and their provenance is hidden.
The next competition will likely ask not who scraped the most, but who built the most generalizing student with the fewest teacher calls and the most auditable data. Distillation is neither a magic button for stealing intelligence nor a simple compression switch. It is a systems-engineering problem involving permission, cost, data quality, evaluation, and audit.
