Reliable language models
Evaluating when models trust unreliable context, why reasoning does not always prevent hallucination, and how to make model outputs more dependable.
Anat Kleiman
I am a final-year PhD candidate at Harvard, advised by Sham Kakade and Jonathan Frankle.
I am funded by the Kempner Graduate Fellowship.
My work spans reliable language models, continual learning, and machine unlearning. I am broadly interested in the empirical science of deep learning: identifying consequential failure modes, understanding why they arise, and designing practical methods that make models more dependable.
Research
Continual learning
Measuring and mitigating catastrophic forgetting as models learn successive tasks, without retaining past data or adding heavy computational overhead.
Machine unlearning
Understanding which training points meaningfully influence model behavior and using that structure to make data removal substantially more efficient.
Selected work
Publications
A complete, current record is available on Google Scholar .
When Unlearning Is Free: Leveraging Low Influence Points to Reduce Computational Costs
Identifies training examples with negligible model influence and uses them to reduce unlearning costs by up to roughly 50% in real-world settings.
The SMeL Test: A Simple Benchmark for Media Literacy in Language Models
Tests whether language models can distinguish reliable from untrustworthy sources—and finds that even strong reasoning models often cannot.
Soup to Go: Mitigating Forgetting During Continual Learning with Model Averaging
Introduces Sequential Fine-tuning with Averaging, retaining prior knowledge across domains without replaying old data.
Transcendence: Generative Models Can Outperform the Experts That Train Them
Shows when a generative model trained on expert demonstrations can exceed every expert in its training set.
Earlier work
2023
Predicting Task Forgetting in Large Language Models
Anat Kleiman, Jonathan Frankle, Sham M. Kakade, Mansheej Paul
ICML Workshop on Challenges in Deployable Generative AI
2023
Soft Prompting Might Be a Bug, Not a Feature
Luke Bailey*, Gustaf Ahdritz*, Anat Kleiman*, Siddharth Swaroop, Finale Doshi-Velez, Weiwei Pan
ICML Workshop on Challenges in Deployable Generative AI
2022
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets
Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, Olga Russakovsky
International Journal of Computer Vision
2020
Buffer Pool Aware Query Scheduling via Deep Reinforcement Learning
Chi Zhang, Ryan Marcus, Anat Kleiman, Olga Papaemmanouil
AIDB Workshop at VLDB
Background
Experience
2026–present
basys.ai
Machine Learning Consultant · Advising AI startups on ML for healthcare
Summer 2024
Apple
Machine Learning Research Intern · Machine unlearning
Summer 2022
Netflix
Machine Learning Intern · Video automation
2019–2020
Uber ATG
Software Engineering Intern · Self-driving vehicles
Education
Harvard University
PhD in Computer Science
Advised by Sham Kakade and Jonathan Frankle
Princeton University
MSE in Computer Science
Worked with Ryan Adams