Haytham ElFadeel
My name is Haytham ElFadeel. I’m a founder, engineering executive, self-taught machine learning scientist, and former alpine climber and cyclist.
I’ve been documenting some of my research, papers, learning, and thoughts on this website since 2019.
Weight Decay Is Not About Small Weights
Every deep learning practitioner carries around a set tradition, today we open up one of them ‘Weight Norm and Weight Decay’. The essay explore what does weight decay actually do, and it relationship with learning rate, Grokking, effective learning rate, Muon, Spectral Spheres and Maximal Update Parametrization.
Essay , 
2026
On Doing Useful Work
Being busy doesn’t tell us whether we’re working on something worthwhile. Drawing on lessons from Richard Hamming, Jim Keller, Andy Grove, and others, this post explores how to choose better problems, understand why things work, and recognize when an approach is running out of room. It also looks at developing research taste, improving judgment, and building teams that can make good decisions without waiting for us.
Post , 2026
Adaptive Policy Optimization
GRPO and RLOO offer a memory-efficient alternative to PPO for training large language models by eliminating the need for a learned value function. However, we identify four sources of bias and instability in GRPO—length bias, difficulty normalization bias, finite-sampling bias, and unbounded advantage scaling—that harm training stability and generalization. We propose Adaptive Policy Optimization (APO), an unbiased and adaptive method that corrects these issues. Experiments on mathematical reasoning benchmarks show that APO improves training stability and token efficiency, and outperforming GRPO by up to 8%.
Paper , 2025
Progress-Aware Reward Model
RLVR improves LLM reasoning but suffers from severe credit assignment issues due to sparse, end-of-trajectory outcome rewards. Process reward models partially address this but rely on costly human supervision or noisy Monte Carlo estimates. We propose a hierarchical credit assignment framework that decomposes feedback across outcome, step, and token levels. Our method introduces a progress reward measuring changes in the probability of reaching a correct answer after each reasoning step, and an entropy-weighted advantage that modulates token-level updates based on model uncertainty.
Paper , 2025
On Offline Reinforcement Learning
Part 1 What is Offline RL, How does it compare to behavior cloning, and does it work.
Part 2 Why Offline RL is not widely used - Challenges of offline RL.
Part 3 Recent advancement that could make offline RL more feasible.
Essay , 2024, and 2025
Mitigations for Reward Hacking
Reward is central to reinforcement learning, but often hard to define, engineer, and easy hacked.
Essay , 2024
Towards Tokenization-Free Language Models: Research Trends
Tokenization is useful because it significantly decreases the inference and training costs. However, tokenization has several disadvantages: vulnerability to adversarial attacks, worse performance for number and character, biases, and additional complexity.
Essay , 2024, updated in 2025
On the Credit Assignment Problem
Unlike behavior cloning, which copies the average demonstration as-is, reinforcement learning aims to reinforce good actions and discourage bad ones. To do that effectively, we need to know which actions were good, which were bad, and which were irrelevant. This essentially is the credit assignment problem (CAP). This essay examines three dimensions of uncertainty—depth, breadth, and density—and presents a taxonomy of methods for addressing them.
Paper , 2024
Is scaling law is slowing?
Recent discussions suggest that Scaling Laws in AI might be slowing. Is that true, and does this mean innovation is hitting a ceiling?
Post , 2024
Preference RL Is Hard: Whose Preference?
Reinforcement Learning from Human Feedback is the dominate paradigm for aligning large language models to human preferences and values. At its core lies a simple idea: learn what humans prefer by having them compare outputs, then train a model to maximize those preferences. But beneath this simple idea is a fundamental problem—whose preferences are we actually capturing?
Essay, 2023, updated in 2024
Representational Efficiency in Deep Learning
Massive amounts of effort and money have gone into scaling (e.g. compute, data, parameters), and some architectural and efficiency tweaks. All of which are valuable. But they also raise an uncomfortable question:
If our deep neural networks are already “universal approximators,” Why does it still feel like we’re brute-forcing representation—sometimes learning good performance without learning good structure? why do LLMs need all the web text to learn how to do basic math? Why it takes millions of images to generate a five-fingered hand image?
This essay argues that the bottleneck is representational efficiency, more than anything else. The headline “universal approximation” is correct but incomplete.
Essay , 2023
Decoupled Transformer
How much Attention is really needed?
Paper , 2021
Knowledge Distillation - Can a student exceed their teacher performance
KD transfers information from teacher models to a student, and is often used in a same-capacity setting to improve a model by distilling dark knowledge. We formalize same-size logit distillation with temperature scaling, analyze why naïve same-size KD is frequently teacher-bounded, and propose Reliability-Weighted Knowledge Distillation (WKD), a per-example weighting scheme that downweights incorrect teachers while preserving logit scale. Experiments on SQuAD v2.0 show that WKD improves F1 and EM over standard KD.
Paper , 2021
ROaD-Electra Robustly Optimized and Distilled Electra
ROaD-Electra Robustly Optimized and Distilled Electra is state of the art encoder-only foundtional language model trained using multi-task improves the transformer performance, generalization, and robustness.
Part 1 Introducing ROaD-Electra.
Part 2 Improving ROaD-Electra with Multitask Pre-training and Knowledge distillation.
Paper , 2020
Gaussian Label Smoothing
Label Smoothing regularizes models by replacing one-hot targets with soft labels, reducing over-confidence and improving generalization. However, uniform smoothing assumes all incorrect classes are equally likely, which is poorly suited to sequence tasks with positional labels. Gaussian Label Smoothing (GLS) addresses this by replacing the uniform noise distribution with a Gaussian centered on the gold position, assigning higher probability to nearby tokens and better reflecting task structure.
Paper , 2020
How Academic Division of AI, Could Limits AGI Progress
A recurring failure mode in AI research is task isolation: we optimize narrowly scoped problems (either: language, vision, planning, or control) as if these faculties were separable in the real world. This produces systems that are statistically competent on benchmarks yet brittle under distribution shift, weak at commonsense inference, and poor at connecting perception to action.
Essay , 2019
What is Understanding - AI Perspective
What does it actually mean to understand something? I argue that understanding is a form of knowledge compression: building a compact model that captures the underlying structure of a problem rather than merely memorizing inputs and outputs. I then connect this view with two others views of understanding.
Essay , 2014, updated 2019