Shangwen Sun

I am a research assistant at New York University, working on deep learning under the guidance of Prof. Alfredo Canziani and Prof. Yann LeCun. My research interests include developing a deeper understanding of LLMs with the goal of improving their efficiency and reliability, and bringing general intelligence into the physical world.

Previously, I received a B.S. in Mathematics and a B.S. in Economics as a double degree from Peking University and an M.S. in Financial Engineering from Baruch College, CUNY. Before transitioning to AI research, I worked in the hedge fund industry for over two years.

Email: shangwen.sun@nyu.edu | Resume | Github

Experience

Publications

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

Shangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu

[code] [arXiv] Accepted ICML 2026

We provide a mechanistic account of why massive activations and attention sinks co-emerge in pre-trained LLMs. We show that normalization links the two by converting massive activations into near-constant hidden states that can induce sinks. We find that attention-space geometry and training context lengths shape sink behavior, biasing some heads toward short-range structure. Since the two phenomena can be reduced independently without hurting modeling performance, their overlap appears incidental rather than functionally necessary.

Miscellaneous

I believe we are what we repeatedly do, and that small efforts, repeated consistently, lead us to the goals.