AI fundamentals: a path through the noise

[Updated on 2026-07-30]

The same question keeps coming back: what is worth learning in AI when the field moves this fast? The answer is the reasons, not the releases. Model names and frameworks turn over every year; the underlying ideas have barely moved in a decade. The skills that carry over from one generation of models to the next are the fundamentals. Everything below is those fundamentals.

Scaling and the compute budget

Scaling laws are the only part of this field that behaves like a real law, with the caveat that even this one did not last. In 2020, Kaplan and colleagues found that if compute is held fixed and the model is made larger, test loss falls along a smooth, predictable curve [1]. Two years later that result was overturned. DeepMind re-derived it with explicit assumptions about how much data the model is trained on and got roughly the opposite answer: at fixed compute, the optimum is a smaller model trained on more tokens, about twenty tokens per parameter [2]. Under that rule, the flagship models of 2020–2022 were something like ten times too large for the data they had seen.

The correction is informative because it shows what scaling laws are. They are not physical laws; they are fits to a specific recipe, and the recipe changes over time. Chinchilla moved the target and the field retrained at the new point for about a year, and then something subtler happened. When the cost of serving a model replaced the cost of training it, the calculation changed. A smaller model trained slightly past the optimum costs more to train but less to run, and a model is run constantly. The industry therefore over-trains smaller models and accepts the extra training cost, because scaling is now measured against inference cost rather than training cost [27]. The models in actual use are not at the point Chinchilla would pick.

One genuinely open question remains: do abilities appear smoothly as models grow, or is some of what is called “emergence” an artifact of the metric chosen [36][39]? The distinction may sound pedantic, but it determines how the next generation of models is expected to behave.

The transformer

Attention matters for one reason: parallelism. Recurrent networks read a sentence one token at a time, and step three cannot start before step two finishes. That serial dependency is what capped how big models could get. The transformer replaced the loop with a single operation, one in which every token attends to every other token at once [3]. Training becomes one large matrix multiply spread across thousands of chips, because there is no hidden state forcing an order. That one change is the entire reason scale was possible.

The decoder-only variant reduces to a simple loop: predict the next token, append it, predict again. Everything said about models being confident, or hallucinating, or having a “temperature” follows from the fact that this loop samples from a probability distribution over the next token. There is no separate subsystem deciding to be wrong. A confident hallucination is just a sample from a peaked but incorrect part of the distribution.

The remaining components, including the QKV projections, the residual stream, and layer normalization, repeat across every model, so debugging gradients through a transformer is about the most transferable skill available [4]. For the language side, Jurafsky and Martin cover the whole pipeline from n-grams to RLHF, still the best single book on it [5]. For the historical arc from bag-of-words to the next-token paradigm, the companion note covers the same ground in full NLP: from word counts to world models .

Data

Scaling laws describe a recipe, and data is the ingredient that turned out to be the constraint. Chinchilla’s twenty-tokens-per-parameter rule [2] has a corollary that matters more than the number itself: for a fixed compute budget the limiting reagent is data, and the 2020–2022 generation of models was trained on far less than the optimum. When the field corrected course, the bottleneck moved from compute to high-quality text.

Raw web text is not ready to train on. Production pipelines deduplicate, filter by quality signals, and reweight domains, and the size of the resulting curated corpus, not the size of the crawl, is what enters the scaling law. FineWeb is the canonical open demonstration of how much filtering buys [47].

Two newer facts reshape the picture. First, synthetic data now feeds the pipeline. Reasoning models generate their own training data through rejection sampling and verifiable rewards [23][34], and post-training has always been built on data the pre-trained model never saw. Second, synthetic data has a failure mode: training on model-generated data erodes the tails of the real distribution, the “curse of recursion” [38]. High-quality data is finite, and that finiteness is the current binding constraint on scale.

There is a data problem inside evaluation too. Benchmark data that leaks into training turns a capability measurement into a memorization measurement. Contamination is a data curation failure as much as an evaluation failure, and it is one of the reasons the evaluation section insists on held-out, task-level measurement.

Mixture of experts

Almost no frontier model is dense anymore, and the reason is practical. Sparse Mixture-of-Experts replaces each feed-forward layer with a pool of experts and routes every token to a handful of them. This breaks a common assumption: that a model’s parameter count is what it costs to run. It is not. DeepSeek-V3 has 671B parameters but runs at roughly the cost of a 37B dense model, because only a small fraction activate per token [24].

That decoupling has two consequences. First, parameters and compute buy different things. Experts appear to buy memorized knowledge, since sparsity lets the model remember more, while reasoning tracks active compute. Push sparsity too far and reasoning plateaus even as knowledge keeps improving [29]. Second, scaling laws had to be redrawn. With total size, active size, expert count, and data all as free variables, the single loss curve becomes a family of curves, and naive Chinchilla-style allocation stops applying [28].

The efficiency gains compound. The “Densing Law,” a measure of capability per parameter, has been doubling roughly every 3.5 months since 2023, driven mostly by sparsity and quantization [27]. That number reflects engineering: the field keeps finding ways to spend less on each parameter.

Why inference got cheap

Inference cost is not a hardware problem that faster GPUs will solve; it is a design problem, and the decisions that determine it are made during training. The two flagship demonstrations are DeepSeek-V3 and Zhipu’s GLM-4.6. The recipe has five parts.

First, decouple parameters from compute, as the MoE section describes. Knowledge is bought with total parameters while running cost is paid on active parameters only. DeepSeek-V3 runs 671B total parameters with 37B active per token; GLM-4.6 runs 357B with about 32B active [43]. The measured effect is large: an 8-expert MoE consumes about the same energy per token as a dense 8B model while delivering quality near a dense 56B model [44].

Second, compress the KV cache at the architecture level. Decode throughput is limited by how many concurrent sequences fit in memory, and the KV cache is the dominant per-token memory consumer; the KV caching note builds the mechanism and its memory accounting from first principles. DeepSeek-V3 uses Multi-head Latent Attention, which projects all attention heads into a single latent vector cached per token: about 70 KB per token, versus 516 KB for Llama-3.1-405B and 327 KB for Qwen-2.5-72B with grouped-query attention [42]. A sevenfold reduction in KV memory translates almost directly into more concurrent sequences and therefore more tokens per second per GPU. Grouped-query attention, where query heads share keys and values, is the weaker version of the same idea and is what GLM-4.6 uses at a 200K context window [43].

Third, train in the precision that is served in. DeepSeek-V3 was the first open model trained natively in FP8, which roughly halved training cost and, more importantly, meant the shipped weights are already FP8. FP8 serving of such a model is exact rather than a lossy retrofit. The next step is FP4, available on Blackwell, and GLM-4.6 demonstrates mixed FP8/Int4 quantization running on domestic silicon [43]. The training recipe and the serving recipe are the same document; that is the meta-lesson.

Fourth, make the model produce fewer tokens. This is easy to miss, because it is not a serving optimization at all. GLM-4.6’s headline result is that it completes tasks with about 15% fewer tokens than GLM-4.5, roughly 651K tokens per CC-Bench trajectory versus 745K [43]. A caveat: GLM-4.6 was post-trained with verifiable-reward RL on agentic coding and tool use, the same task class CC-Bench measures, so the figure partly reflects harness alignment. Fewer tokens per task means less compute, less memory traffic, less energy, and a lower bill at any price. It is an objective trained into the model, and it compounds with every serving technique.

Fifth, use the model itself for speculative decoding. DeepSeek-V3’s multi-token prediction head and GLM-4.6’s efficiency training both let the model draft several tokens that are validated in a single forward pass, turning serial generation into parallel verification [31][42].

The runtime half of the recipe operates within these decisions. PagedAttention allocates the KV cache like virtual memory pages, cutting fragmentation and lifting serving throughput by 2–4× [30]. Continuous batching keeps GPUs busy by interleaving sequences, and prefill and decode can be disaggregated into separate pools because the two phases bind on different resources. Prefix caching reuses attention state across requests that share a prompt.

The remaining structural limits are not software. Compute now outpaces interconnect bandwidth, so as models spread across more GPUs, the communication between them rivals the time spent computing. The split between compute-bound prefill and memory-bound decode pushes toward specialized, heterogeneous hardware. At the other end of the scale, small local models are becoming competitive on everyday queries, and routing easy queries to a local device avoids the serving cost entirely. The practical future is a split: cheap local models for easy queries, large models for hard ones, and a router deciding which is which.

Post-training

By around 2023, the accurate description changed: pre-training is largely commoditized, and the interesting work moved to what happens after it. A raw pretrained model predicts the next token well but is not directly usable. It does not follow instructions, it does not stop when asked, and it does not do what is meant. Post-training is the difference between that base model and something worth handing to a user.

The landmark result is InstructGPT: a pretrained model can be aligned to follow instructions without touching its architecture, by fine-tuning on human preference data [7]. For a long time that meant the full RL loop: learn a reward model from human comparisons, then optimize the policy against it [6]. Then DPO showed that almost all of that machinery can be skipped. The learned reward model and the RL optimization fold into a single step that works directly on pairs of outputs [8]. The modern recipes, including GRPO and its relatives, are refinements of the same idea, tuned to be cheaper and stabler [34].

These techniques are best learned as a family. The question underneath is always the same: how to get the model to do what is useful, given a way to tell which outputs are better. For the practical discipline, the Tuning Playbook is worth one careful read, because it is about experimental hygiene, separating systematic work from lucky runs, a discipline that is easy to underestimate [9]. For the theory beneath all of it, Sutton and Barto plus David Silver’s lectures remain the definitive place [10]. For a survey of the research landscape, Lilian Weng’s blog covers more ground with more clarity than most published surveys [11].

Reasoning and test-time compute

From 2024 on, reasoning became the product, and the central finding is that reasoning quality improves with test-time compute [25]. Given a hard problem, the model can think: generate intermediate steps, reconsider, backtrack. The more compute it is allowed to spend thinking, the better it does. o1 established the finding, and DeepSeek-R1 showed the recipe was reproducible in the open [23].

The recipe is elegant and surprisingly small. Train with reinforcement learning where the reward is verifiable, such as a compiler or an answer key, something that can check correctness without a learned reward model. Use GRPO, an update that needs no separate value network, which keeps the whole thing cheap [34]. Two practical tricks make it work: cold-start data seeds the RL with examples of reasoning, and rejection sampling keeps only the correct samples. The simplest replicas show how little the formula needs. s1 matches o1-preview-level math with a thousand curated reasoning traces and a “budget forcing” heuristic that simply tells the model to keep thinking [26].

What remains genuinely open is what that thinking is. Is it search, exploring possibilities, or recall, retrieving patterns it memorized? The MoE evidence leans toward recall: parameters buy memorization, not reasoning [29]. The question deserves to stay open, because the answer sets expectations for what scale can buy.

Building systems and agents

Agents are where models stop answering and start acting. A tool-using agent turns a token generator into something that can call a function, run code, query a database, and act on the result. The capability moved from research demo to product requirement by 2025, and the standardization that made it viable is the Model Context Protocol, a universal interface between an agent and external tools and data [32]. One integration works across an ecosystem instead of requiring a separate integration per vendor.

The engineering lessons from production agents are consistent. The agent architectures that actually work are simple and composable: a loop that observes, decides, calls a tool, and repeats, with the model deciding when to stop. Elaborate frameworks add state and indirection that usually cost more than they save [18]. The design space of planning, memory, and tool use is large and still being mapped; Lilian Weng’s agent survey remains the best single map [19].

The training side caught up with the engineering side. The RLVR recipe that produced reasoning models, reinforcement learning with verifiable rewards, is being applied directly to tool use, where the reward is whether the tool call succeeded and the task completed. Early work is asking whether agents and world models follow the same scaling laws as language models [33].

Building with these models is a different skill from training them, and the gap shows up everywhere [17]. Papers describe what works in ideal conditions; production is what survives real constraints: latency, cost, and a thousand edge cases no benchmark captured. Two techniques carry most of the weight. RAG grounds the model in external knowledge so it stops guessing things it could look up. And the recurring engineering concerns are latency budgets, cost per request, and graceful failure: a production system needs a fallback when the model is wrong, slow, or down. Patterns for building LLM systems have consolidated around these concerns [17].

Agent evaluation is harder than language-model evaluation, because a task is not a single answer but a trajectory of tool calls, observations, and corrections. SWE-bench measures real task completion, and suites like CC-Bench publish full trajectories so that failures can be inspected rather than averaged away [43].

The gap between a demo and a deployment is large, and closing it requires a different set of skills than training a better model. Most professional work in this area is spent on exactly that.

Evaluation and benchmarks

A benchmark score is a claim about a model, and reading it correctly requires knowing what kind of claim it is. Four failure modes account for most misreadings.

Contamination. If benchmark data appears in the training data, the score measures memorization, not capability. The only defense is held-out evaluation on data that was provably not seen, and that defense weakens every time a model trains on more of the web.

Saturation and gaming. Once a benchmark is solved, it stops distinguishing models. A solved benchmark says more about the benchmark authors than about the model. New benchmarks arrive, saturate within a year or two, and the cycle repeats.

The metric choice decides the story. Whether abilities appear smoothly or “emerge” can be an artifact of how performance is measured [36][39]. Small changes in the aggregation rule change whether a jump looks smooth or sudden.

Missing denominators. The most common error in comparing models is comparing accuracy without normalizing by cost. Two models with the same score are not equivalent if one costs a fraction of the price and uses 15% fewer tokens per task [43]. The metric that matters for production is cost per solved task, not score per benchmark.

Task-level evaluation is the current response to these failures. SWE-bench Verified measures whether code tasks actually complete. Zhipu’s CC-Bench evaluates models inside isolated containers doing multi-turn real-world tasks and publishes the full trajectories, including per-task token counts, so results can be audited [43]. A caveat applies to these numbers as to any benchmark: the published CC-Bench figures come from GLM-4.6, which was RL-post-trained on the same class of agentic coding and tool-use tasks the benchmark measures, so part of the score is alignment to the harness rather than general capability. Reproducible, task-level, cost-aware evaluation is a skill in its own right, and it is where the field’s evaluation practice is heading.

Interpretability

Mechanistic interpretability is the most intellectually urgent open problem in the field, because models that are not understood internally are being deployed at scale, and it is not clear whether that is acceptable or a risk.

There has been real progress. Transformer Circuits showed that language models learn interpretable features, compose them into circuits, and pack many features into the same neurons, a phenomenon called “superposition.” That is why the internals look like a mess when individual units are inspected [12][35]. Sparse autoencoders extract these features at scale: train a network to reconstruct activations using only a few active units, and the units that emerge tend to correspond to human-legible concepts [13]. Neel Nanda’s writing is the best practical starting point for going deeper [14].

The unresolved question is whether this line of work ever reaches causal understanding, knowing why the model does what it does, or stays descriptive, a very detailed natural history of the internals. That distinction determines whether these systems can ever be trusted with real responsibility. It is not settled, and the honest position is that the answer is not yet known.

Multimodality and world models

Text is a small fraction of the data, and the same transformer machinery that consumes text tokens consumes image, audio, and video tokens. The frontier has moved accordingly: the leading closed models are multimodal by default, and their system cards describe safety evaluations across modalities [49]. Beyond perception, the research frontier is moving toward world models, systems that learn the dynamics of an environment rather than a distribution over text, and early work asks whether such models follow the same scaling laws as language models [33].

For a learner the practical consequence is to avoid assuming text-only. The fundamentals in this guide apply to any token modality; the differences are in data pipelines and evaluation.

Energy and the cost of tokens

Token count is the field’s currency, and energy is what a token costs physically. The relationship is simple enough for a back-of-the-envelope model, and it explains why every optimization in this guide is also an energy optimization.

At the GPU level, the energy of a request is the sum of a compute term and a memory term:

where $C$ is the tensor-core FLOPs, $Q$ is the bits moved through memory, and the calibrated H100 coefficients are $\alpha \approx 0.52$ pJ/FLOP and $e \approx 11.7$ pJ/bit [45]. The two phases of generation bind on different terms. Prefill, which processes the prompt, is compute-bound: $C_{\text{pre}} \approx 2N\,T_{\text{in}}$ FLOPs for a model with $N$ parameters and $T_{\text{in}}$ input tokens. Decode is memory-bound: each generated token needs about $2N$ FLOPs but also requires streaming all $N$ weights from high-bandwidth memory, so its energy is dominated by $N$ times the bits per weight. This is why every lever in the inference recipe shows up in the formula:

• Fewer bits per weight: FP8 halves the memory term, FP4 quarters it. Measured on a 405B model, FP8 cuts energy per token by roughly 30% [44].
• Fewer active parameters: MoE reduces the effective N for both terms. An 8-expert MoE uses about the same energy per token as a dense 8B while delivering near-56B quality [44].
• Less KV memory per token: Multi-head Latent Attention and grouped-query attention cut the memory traffic of long-context decode [42].
• Fewer tokens per task: the GLM-4.6 result of 15% fewer tokens is a direct 15% cut in energy at fixed quality [43].
• Better utilization: a GPU serving a single request idles near 43% of thermal design power. Raising batch size from 32 to 256 drops energy per token by about a quarter, because the fixed cost of the machine is spread over more tokens [44].

One structural law completes the picture. Tokens per watt halves every time the serving context window doubles, because a larger window leaves room for fewer concurrent sequences while the GPU draws about the same power [46]. The context window, not the GPU, is the dominant energy lever of a deployment, and routing short requests to small-context pools is cheaper than buying newer hardware.

This is an active research area, not a settled number. Benchmarks for inference energy now report joules per token explicitly, and analytical models predict it from model size, context, and precision [44][45]. Any claim about the environmental or economic cost of a model can be checked against this decomposition.

What’s still broken

The field also has well-known failure modes.

Models hallucinate. A model samples from a learned distribution over text, and where the distribution is wrong, it samples confidently from the wrong place. This is a property of sampling, not a bug. The “stochastic parrot” critique from 2021 named it early [40]. The related problem is calibration: confidence does not track correctness, so a model that is wrong will often report that it is sure [37].

And the input side has hit a wall. The internet’s high-quality text is finite, synthetic data now feeds the pipeline, and training on model-generated data erodes the tails of the real distribution [38]. The next scaling will have to come from somewhere else.

Alignment remains unfinished in ways that scale. As reward signals move toward verifiable objectives, the failure modes move with them: reward hacking, where a model satisfies the measured objective without doing the intended thing, and the open problem of overseeing models on tasks beyond human competence. Constitutional AI and related methods attempt to encode constraints without an explicit reward model, but the research frontier is far from settled [48].

The learning path

The shortest path, in order:

Build. Karpathy’s Zero to Hero goes from nothing to a working language model in code, and the experience of having built one teaches more than reading any number of paper titles [20]. Then the math: linear algebra and probability. Goodfellow is the standard reference, Murphy the deeper alternative, and Shalev-Shwartz and Ben-David the theory of why models generalize at all [15][16][21]. Then implement a transformer end to end; a single implementation teaches more than reading papers [22]. Then alignment [7][8]. Then the reasoning pipeline [23][34]. Then inference: quantization, KV caches, batching [30], because an unaffordable model is not usable; the local inference note covers the hardware math and the tools. Then evaluation: reading a benchmark correctly is a skill most practitioners acquire only after it costs them. Then build an agent with tools, because the loop of observe, decide, act is where the current product work happens [19][32].

For the frontier, the multimodality section explains why text is only part of the input; the research frontier is the world-models line. The Hugging Face course covers the practical tooling: tokenizers, fine-tuning, deployment [41].

The depth required varies by role. A researcher improving frontier models needs the full picture: the derivations, the training recipes, the architecture. A research engineer, who implements and runs the experiments, needs functional depth plus systems skill. An application engineer working on inference and fine-tuning needs the functional model: the loop, sampling, the KV cache, quantization, tokenizers. The derivations are required only on the research track. First-principles understanding transfers across roles and models; model-specific knowledge does not.

Everything else, the new architectures, frameworks, and releases, is detail on top of this foundation. Depth in fundamentals beats breadth in releases, and the open problems are where that depth pays off.

References

[1] Kaplan et al., “Scaling Laws for Neural Language Models,” 2020. https://arxiv.org/abs/2001.08361
[2] Hoffmann et al., “Training Compute-Optimal Large Language Models,” 2022. https://arxiv.org/abs/2203.15556
[3] Vaswani et al., “Attention Is All You Need,” 2017. https://arxiv.org/abs/1706.03762
[4] CS231n, http://cs231n.github.io/; 3Blue1Brown, https://www.youtube.com/c/3blue1brown; Jay Alammar, https://jalammar.github.io/
[5] Jurafsky & Martin, “Speech and Language Processing,” 3rd ed. https://web.stanford.edu/~jurafsky/slp3/
[6] Christiano et al., “Deep Reinforcement Learning from Human Preferences,” 2017. https://arxiv.org/abs/1706.03741
[7] Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback,” 2022. https://arxiv.org/abs/2203.02155
[8] Rafailov et al., “Direct Preference Optimization,” 2023. https://arxiv.org/abs/2305.18290
[9] Google Research, “Deep Learning Tuning Playbook.” https://github.com/google-research/tuning_playbook
[10] Sutton & Barto, “Reinforcement Learning: An Introduction,” 2nd ed. http://incompleteideas.net/book/the-book-2nd.html; David Silver, UCL lectures. https://www.youtube.com/playlist?list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ
[11] Lilian Weng, “Lil’Log.” https://lilianweng.github.io/
[12] Elhage et al., “A Mathematical Framework for Transformer Circuits,” 2021. https://transformer-circuits.pub/
[13] Bricken et al., “Towards Monosemanticity,” 2023. https://transformer-circuits.pub/2023/monosemantic-features
[14] Neel Nanda, “Mechanistic Interpretability.” https://www.neelnanda.io/mechanistic-interpretability
[15] Murphy, “Probabilistic Machine Learning.” https://probml.github.io/pml-book/
[16] Shalev-Shwartz & Ben-David, “Understanding Machine Learning.” https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/
[17] Eugene Yan, https://eugeneyan.com/; “Patterns for Building LLM-based Systems & Products,” 2023. https://eugeneyan.com/writing/llm-patterns/
[18] Anthropic, “Building Effective Agents,” 2024. https://www.anthropic.com/engineering/building-effective-agents
[19] Lilian Weng, “LLM-powered Autonomous Agents,” 2023. https://lilianweng.github.io/posts/2023-06-23-agent/
[20] Karpathy, “Neural Networks: Zero to Hero.” https://karpathy.ai/zero-to-hero.html
[21] Goodfellow, Bengio, Courville, “Deep Learning,” 2016. https://www.deeplearningbook.org/
[22] Stanford CS336, “Language Modeling from Scratch.” https://github.com/stanford-cs336/assignment1-basics
[23] DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” 2025. https://arxiv.org/abs/2501.12948
[24] DeepSeek-AI, “DeepSeek-V3 Technical Report,” 2024. https://arxiv.org/abs/2412.19437
[25] OpenAI, “OpenAI o1 System Card,” 2024. https://arxiv.org/abs/2412.16720
[26] Muennighoff et al., “s1: Simple Test-Time Scaling,” 2025. https://arxiv.org/abs/2501.19393
[27] Xiao et al., “Densing Law of LLMs,” Nature Machine Intelligence, 2025. https://www.nature.com/articles/s42256-025-01137-0
[28] Abnar et al., “Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models,” 2025. https://arxiv.org/abs/2501.12370
[29] Jelassi et al., “Mixture of Parrots: Experts Improve Memorization More Than Reasoning,” 2024. https://arxiv.org/abs/2410.19034
[30] Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” 2023. https://arxiv.org/abs/2309.06180
[31] Leviathan, Kalman, and Matias, “Fast Inference from Transformers via Speculative Decoding,” 2023. https://arxiv.org/abs/2211.17192
[32] Model Context Protocol, specification. https://modelcontextprotocol.io/
[33] Pearce et al., “Scaling Laws for Pre-training Agents and World Models,” 2024. https://arxiv.org/abs/2411.04434
[34] Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” 2024. https://arxiv.org/abs/2402.03300
[35] Elhage et al., “Toy Models of Superposition,” 2022. https://transformer-circuits.pub/2022/toy_model/index.html
[36] Wei et al., “Emergent Abilities of Large Language Models,” 2022. https://arxiv.org/abs/2206.07682
[37] Huang et al., “A Survey on Hallucination in Large Language Models,” 2023. https://arxiv.org/abs/2311.05232
[38] Shumailov et al., “The Curse of Recursion: Training on Generated Data Makes Models Forget,” 2023. https://arxiv.org/abs/2305.17493
[39] Schaeffer et al., “Are Emergent Abilities of Large Language Models a Mirage?” NeurIPS 2023. https://arxiv.org/abs/2304.15004
[40] Bender et al., “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” FAccT 2021. https://doi.org/10.1145/3442188.3445922
[41] Hugging Face, “The Hugging Face Course.” https://huggingface.co/learn/nlp-course
[42] DeepSeek-AI, “DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model,” 2024. https://arxiv.org/abs/2405.04434
[43] Zhipu AI, “GLM-4.6: Advanced Agentic, Reasoning and Coding Capabilities,” 2025. https://z.ai/blog/glm-4.6
[44] Niu et al., “Benchmarking the Power Consumption of LLM Inference,” AAAI 2025. https://ojs.aaai.org/index.php/AAAI/article/view/40535
[45] “From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs,” 2026. https://arxiv.org/abs/2607.26571
[46] Chen et al., “The 1/W Law: Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency,” 2026. https://arxiv.org/abs/2603.17280
[47] Penedo et al., “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale,” 2024. https://arxiv.org/abs/2406.17557
[48] Bai et al., “Constitutional AI: Harmlessness from AI Feedback,” 2022. https://arxiv.org/abs/2212.08073
[49] OpenAI, “GPT-4o System Card,” 2024. https://arxiv.org/abs/2410.21276