Graduate students
Cheongwoong Kang
PhD Student (21.03 ~ )
3 notes
Research
2025
Non-autonomous Neural ODE Transformers
Continuous-depth parameterization and spectral analysis reveal transformer dynamics while preserving flexible fine-tuning.
2025
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
Recent advancements in large language models (LLMs) based on transformer architectures have sparked significant interest in understanding their inner workings. In this paper, we introduce a novel approach to modeling transformer architectures using highly flexible non-autonomous neural ordinary differential equations (ODEs). Our proposed model parameterizes all weights of attention and feedforward blocks through neural networks, expressing these weights as functions of a continuous layer index. Through spectral analysis of the model’s dynamics, we uncover an increase in eigenvalue magnitude that challenges the weightsharing assumption prevalent in existing theoretical studies. We also leverage the Lyapunov exponent to examine token-level sensitivity, enhancing model interpretability. Our neural ODE transformer demonstrates performance comparable to or better than vanilla transformers across various configurations and datasets, while offering flexible fine-tuning capabilities that can adapt to different architectural constraints.
2023
Impact of Co-occurrence on Factual Knowledge of Large Language Models
We hypothesize that relying heavily on simple co-occurrence statistics of the pre-training corpora is one of the main factors that cause factual errors. Our results reveal that LLMs are vulnerable to the co-occurrence bias, defined as preferring frequently co-occurred words over the correct answer. Consequently, LLMs struggle to recall facts whose subject and object rarely co-occur in the pre-training dataset although they are seen during finetuning.
7 papers
Publications
2026
2025
2023
- —
멀티 에이전트 강화학습에서의 게이트 기반 메시지 교환
- [80]
Impact of Co-occurrence on Factual Knowledge of Large Language Models