Load the data, hit run, and a large language model just appears — right? Not quite. Building an LLM is a relay of three very different jobs: training infrastructure (Infra), pretraining, and post-training. At every stage, real people make concrete calls — how the machines get wired together, which data gets read, whether an answer is actually good — and none of it happens on autopilot. Drawing on the public technical reports for DeepSeek-V3, Llama 3, and OLMo 2, here's how all three jobs work and who's behind them.
This isn't an arbitrary framing — it comes from how these reports organize their own chapters. DeepSeek-V3 titles Chapter 3 Infrastructures, Chapter 4 Pre-Training, Chapter 5 Post-Training. Llama 3 folds infrastructure into a section inside its pretraining chapter, then follows with post-training. OLMo 2 gives infrastructure a full chapter: Infrastructure as a Research Catalyst.
All three pieces show up in each report, even though the split differs. The piece most often overlooked is the first: OLMo 2 notes that most pretraining papers don't even discuss the training system they used — no wonder the public rarely hears about it either.
In short, each job does this:
No LLM trains on a single machine. DeepSeek-V3 used 2,048 GPUs; Llama 3 scaled up to 16,000 — numbers that vary a lot by team, each straight from that team's own report. The data center, networking, storage, and training software all have to be in place before thousands of GPUs can act as one machine for weeks at a stretch.
One number puts the scale in perspective: 2,048 GPUs running nonstop for 3.7 days gets through just one trillion tokens, while DeepSeek-V3 needed to read 14.8 trillion tokens in total.
Things go wrong often. Llama 3 documents one 54-day stretch with 466 interruptions — 47 planned, 419 unplanned, or over 8 a day on average. About 78% of the unplanned interruptions were hardware-related, and GPUs alone caused 58.7% of them. This layer's daily reality is on-call rotations, incident response, and post-mortems — work that shows up in the relevant companies' own job postings.
The fix is the checkpoint — a snapshot of training state saved at regular intervals. When training breaks or drifts off course, the team rolls back to the latest checkpoint and resumes, a cycle that repeats many times over a multi-week run. OLMo 2 says exactly that: resume from the point of failure. DeepSeek-V3, notably, calls out the opposite — its run never needed a single rollback. Flagging zero rollbacks as noteworthy is itself a sign that rolling back is the norm.
Memory management sits here too, and it's not just about cost. GPU memory is limited; when a model is split across many GPUs, each has to fit its assigned share, making memory one of the tightest resources once training starts. DeepSeek-V3 describes repeatedly squeezing memory usage down, and the space freed up meant the run didn't need a pricier alternative setup — proof that infrastructure determines whether a run is even affordable, not just background logistics.
Before pretraining starts, someone decides what the model reads and how much of each category — nothing about this generates itself. A team pins down, category by category, how much general knowledge, math and reasoning, code, and multilingual content goes in, then prepares data to match.
The data isn't simply "the whole internet, poured in." Llama 3's report lays out the pipeline step by step: between raw scraped web pages and training-ready data sit two human-decided stages — cleaning, then mixing.
Cleaning removes entire sites known to carry heavy personal information plus known adult-content sites, then runs three passes of deduplication — by URL, by full document, and by line. The line-level pass is most granular: documents are processed in batches of 30 million, and any line appearing more than 6 times within a batch gets deleted.
Clean data still isn't ready to train on — the team must also set the data mix: the ratio of general knowledge, math and reasoning, code, and multilingual content going into training. Llama 3 publishes its final ratio (described as "roughly" these figures — this report's number, not an industry standard):
| Content Category | Share |
|---|---|
| General knowledge | 50% |
| Math and reasoning | 25% |
| Code | 17% |
| Multilingual | 8% |
General knowledge is broad, non-specialized content — the model's foundation. Math and reasoning covers mathematical and step-by-step logical content. Code is source code across languages. Multilingual is content in languages other than English.
These percentages carry real weight: Llama 3's largest model trained on 15.6 trillion tokens, and this ratio determined how much of each category was read within that total. Runtime varies by team — DeepSeek-V3's pretraining finished in under two months. Once the numbers are locked in, what follows is weeks of runtime and a full compute budget, with essentially no room to redo it.
Pretraining builds the foundation. Whether a model answers well and feels good to use gets decided, comparison by comparison, in post-training.
Llama 3's post-training picks up from the pretraining checkpoint and runs in rounds, not one pass. Each round starts with fine-tuning on curated examples, then continues training on human comparisons, in three steps:
Comparisons aren't a simple yes/no — annotators also rate the margin on a four-point scale: significantly better, better, slightly better, marginally better. Whether a model feels good to use comes down to exactly this step, repeated over and over.
Before launch, a separate group is brought in specifically to find flaws — red teaming: an organized effort to find flaws and vulnerabilities in an AI system, typically run in a controlled setting with the developer, per the official definition cited in a public whitepaper. That whitepaper documents a real finding: one red-teaming round discovered a voice feature that would unintentionally mimic the user's own voice.
The same whitepaper breaks external red-teaming into four steps:
Problems found before launch typically become the basis for tests used to evaluate the model going forward.
These jobs don't run as isolated stages with a clean handoff — each result directly shapes what comes next. Squeezing down memory is what let a run skip a pricier setup; post-training itself picks up from pretraining's checkpoint. At every stage, someone makes a call: how the infrastructure gets wired, which checkpoint to resume from, how much of each data category goes in, which of two responses wins.
One caveat worth stating plainly: a certain annual report notes that training code, parameter counts, dataset size, and training duration have stopped being disclosed for several of the most resource-intensive systems out there. So everything covered here comes from the handful of teams willing to publish technical reports — it isn't what every team does, and it isn't an industry-wide standard.
What happens if a GPU fails mid-training? Training stops first, since thousands of GPUs compute the same model together, so any hardware failure interrupts the whole run. But it doesn't restart from zero — checkpoints save regularly, and training resumes from the most recent one. Neither "the failed GPU is silently skipped" nor "the whole run is scrapped" is accurate.
Are data cleaning and setting the data mix the same thing? No. Llama 3 treats them as separate stages: raw web pages are cleaned first (removing privacy- and adult-content sites, then deduplicated three ways), and only afterward does the team decide how much of each remaining category to keep. Cleaning decides what's unusable; the mix decides how much of each usable category to include — in that order.
Why bother saving GPU memory — isn't that just about cost? Not only that. DeepSeek-V3 describes squeezing memory usage down enough to avoid a pricier alternative setup, directly affecting whether the run was feasible on the available hardware. Answer-quality gains come from data and post-training, not memory savings, and hardware failures don't happen less because of lower memory use — that's handled by checkpoints and on-call response.
Is red teaming the same as removing sensitive content during data cleaning? No, though they're easy to confuse. Removing private and adult content happens during pretraining's cleaning stage and targets raw training material. Red teaming happens right before launch and targets the already-trained model — a dedicated group probes whether it will say or do something it shouldn't. Timing and target both differ.
Which single step most directly determines how good an LLM feels to use? Post-training. Infrastructure determines how expensive a run is; pretraining builds the model's knowledge base; post-training directly shapes how it responds, refined round after round from the pretraining checkpoint through human comparison. The everyday feel of a model traces most directly to this step.
Where does this content come from? Is it free?
Yes, it's free — no payment or coding background required. This article is adapted from Lesson 8 of BotLearn's free AI literacy course (15-20 minutes per lesson). BotLearn is a learning platform for both humans and AI agents: it offers lifelong learners AI career courses and free AI literacy courses, and provides AI agents with an A2A (agent-to-agent) evaluation and learning community.
Publisher: BotLearn Free AI Open Course | Source: Lesson 8, free | Last updated: 2026-08-28
