Publications

Publications in reversed chronological order. For complete list, see Google Scholar.

2024

  1. A New Perspective on Shampoo's Preconditioner
    Depen Morwani, Itai Shapira, Nikhil Vyas, Eran Malach, Sham Kakade, Lucas Janson
    Under submission at ICLR 2025
    arXiv
  1. Deconstructing What Makes a Good Optimizer for Language Models
    Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, Sham Kakade
    Under submission at ICLR 2025
    arXiv
  1. SOAP: Improving and Stabilizing Shampoo using Adam
    Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, Sham Kakade
    Under submission at ICLR 2025
    arXiv

2023

  1. Feature emergence via margin maximization: case studies in algebraic tasks
    Depen Morwani, Benjamin L. Edelman, Costin-Andrei Oncescu, Rosie Zhao, Sham Kakade
    Accepted as Spotlight (Acceptance Rate 5%) to ICLR 2024
    arXiv
  1. Simplicity Bias in 1-Hidden Layer Neural Networks
    Depen Morwani, Jatin Batra, Prateek Jain, Praneeth Netrapalli
    Accepted to NeurIPS 2023
    arXiv
  1. Feature-Learning Networks Are Consistent Across Widths At Realistic Scales
    Nikhil Vyas, Alex Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, Cengiz Pehlevan
    Accepted to NeurIPS 2023
    arXiv
  1. Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning
    Nikhil Vyas, Depen Morwani, Rosie Zhao, Gal Kaplun, Sham Kakade, Boaz Barak
    Accepted as Spotlight (Acceptance Rate 3.5%) to ICML 2024
    arXiv

2022

  1. Inductive bias of gradient descent for weight normalized smooth homogeneous neural nets
    Depen Morwani, Harish G. Ramaswamy
    Accepted to ALT 2022
    arXiv