Xuezhe Ma | Home

Bio

I have been a Research Lead in Information Sciences Institute and a Research Assistant Professor in Department of Computer Science, at University of Southern California since Fall 2020.
I got my Ph.D degree in Language Technologies Institute at Carnegie Mellon University, where I was working with Prof. Eduard Hovy. Before coming to CMU, I was a master student of Center for Brain-like Computing and Machine Intelligence (BCMI), Shanghai Jiao Tong University, Shanghai, China. I received my Bachelor degree in Computer Science from Shanghai Jiao Tong University, where I was a member of ACM Class, now part of Zhiyuan College in SJTU.

I believe that representation learning techniques based on deep learning methods can fundamentally transform the conventional feature designing paradigm. Representation learning can, in principle, automatically learn representations that are mathematically and computationally convenient to process. My research focuses on fulfilling this transformation to enhance the effectiveness, efficiency, interpretablility and robustness of representation learning, by developing and analyzing deep learning techniques . The key contributions of my research are as follows:

Efficiency in Multi-Modal Large Language Models (LLMs): We developed efficient unified neural architectures and learning algorithms to learn a universal semantic space for vairous data modalities.
Efficient and Robust Long-Context Modeling in LLMs: We developed efficient and robust architectures and methods for modeling long range dependencies in LLMs.
Applications and evaluation methods of multi-modal LLMs for long sequential data: We apply long-context LLMs to real-world tasks and develop reliable metrics for evaluation.

Publications

Most recent publications on Google Scholar.

Selected Papers
All

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length ArXiv Code

Xuezhe Ma*, Xiaomeng Yang*, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, Chunting Zhou*

Preprint

Look-back Decoding for Open-Ended Text Generation PDF Bib

Nan Xu, Chunting Zhou, Asli Celikyilmaz, Xuezhe Ma

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023) (Outstanding Paper Award)

Challenges in Context-Aware Neural Machine Translation PDF Bib Video

Linghao Jin*, Jacqueline He*, Jonathan May, Xuezhe Ma

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)

LIMA: Less is more for alignment PDF Bib ArXiv

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy

Advances in Neural Information Processing Systems (NeurIPS-2023)

Mega: moving average equipped gated attention PDF Bib ArXiv Code

Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, Luke Zettlemoyer

Proceedings of 11th International Conference on Learning Representations (ICLR 2023)

Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization ArXiv Code

Xuezhe Ma

Preprint

Towards a Unified View of Parameter-Efficient Transfer Learning PDF Bib ArXiv Code

Junxian He*, Chunting Zhou*, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig

Proceedings of the 10th International Conference on Learning Representations (ICLR-2022)

Luna: Linear Unified Nested Attention PDF Bib ArXiv Video Code

Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, Luke Zettlemoyer

Advances in Neural Information Processing Systems (NeurIPS-2021)

AESOP: Paraphrase Generation with Adaptive Syntactic Control PDF Bib Video Code

Jiao Sun, Xuezhe Ma*, Nanyun Peng

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021)

Decoupling Global and Local Representations via Invertible Generative Flows PDF Bib ArXiv Code

Xuezhe Ma, Xiang Kong, Shanghang Zhang, Eduard Hovy

Proceedings of 9th International Conference on Learning Representations (ICLR 2021)

Examining and Combating Spurious Features under Distribution Shift PDF Bib ArXiv Code

Chunting Zhou, Xuezhe Ma, Paul Michel, Graham Neubig

Proceedings of the 38th International Conference on Machine Learning (ICML 2021)

COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences ArXiv

Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormolabashi, Te-Lin Wu, Xuezhe Ma, Nanyun Peng

Findings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL 2021)

DiSCoL: Toward Engaging Dialogue Systems through Conversational Line Guided Response Generation PDF ArXiv Code

Sarik Ghazarian, Zixi Liu, Tuhin Chakrabarty, Xuezhe Ma, Aram Galstyan, Nanyun Peng

Proceedings of 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2021): Demo

A Two-Step Approach for Implicit Event Argument Detection PDF Code

Zhisong Zhang, Xiang Kong, Zhengzhong Liu, Xuezhe Ma, Eduard Hovy

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)

MaCow: Masked Convolutional Generative Flow PDF Bib ArXiv Code

Xuezhe Ma, Xiang Kong, Shanghang Zhang, Eduard Hovy

Proceddings of Thirty-third Conference on Neural Information Processing Systems (NeurIPS 2019)

FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow PDF Bib ArXiv Code

Xuezhe Ma*, Chunting Zhou*, Xian Li, Graham Neubig, Eduard Hovy

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019)

MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders PDF Bib ArXiv Code

Xuezhe Ma, Chunting Zhou, Eduard Hovy

Proceedings of 7th International Conference on Learning Representations (ICLR 2019) (Oral)

An Empirical Investigation of Structured Output Modeling for Graph-based Neural Dependency Parsing PDF Bib Code

Zhisong Zhang, Xuezhe Ma, Eduard Hovy

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019)

Density Matching for Bilingual Word Embedding PDF Bib ArXiv Video Code

Chunting Zhou, Xuezhe Ma, Di Wang, Graham Neubig

Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2019)

On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing PDF Bib ArXiv Code

Wasi Uddin Ahmad*, Zhisong Zhang*, Xuezhe Ma, Eduard Hovy, Kai-Wei Chang, Nanyun Peng

Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2019)

Stack-Pointer Networks for Dependency Parsing PDF Bib ArXiv Slides Code

Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, Eduard Hovy

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018) (Oral)

Softmax Q-Distribution Estimation for Structured Prediction: A Theoretical Interpretation for RAML ArXiv

Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, Graham Neubig, Eduard Hovy

ArXiv Preprint

Dropout with Expectation-Linear Regularization PDF Bib ArXiv

Xuezhe Ma, Yingkai Gao, Zhiting Hu, Yaoliang Yu, Yuntian Deng, Eduard Hovy

Proceedings of 5th International Conference on Learning Representations (ICLR 2017)

Neural Probabilistic Model for Non-projective MST Parsing PDF Bib ArXiv Code

Xuezhe Ma, Eduard Hovy

Proceedings of 5th International Joint Conference on Natural Language Processing (IJCNLP 2017)

An Interpretable Knowledge Transfer Model for Knowledge Base Completion PDF Bib ArXiv Video

Qizhe Xie, Xuezhe Ma, Zihang Dai, Eduard Hovy

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017) (Oral)

End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF PDF Bib ArXiv Code

Xuezhe Ma, Eduard Hovy

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016)

Harnessing Deep Neural Networks with Logic Rules PDF Bib ArXiv

Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, Eric P. Xing

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016) (Outstanding Paper Award)

Unsupervised Ranking Model for Entity Coreference Resolution PDF Bib ArXiv

Xuezhe Ma, Zhengzhong Liu, Eduard Hovy

Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2016) (Oral)

Efficient Inner-to-outer Greedy Algorithm for Higher-order Labeled Dependency Parsing PDF Bib

Xuezhe Ma, Eduard Hovy

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015)

Unsupervised Dependency Parsing with Transferring Distribution via Parallel Guidance and Entropy Regularization PDF Bib

Xuezhe Ma and Fei Xia

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL 2014) (Oral)

Fourth-Order Dependency Parsing PDF Bib

Xuezhe Ma and Hai Zhao

Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012)

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length ArXiv Code

Xuezhe Ma*, Xiaomeng Yang*, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, Chunting Zhou*

Preprint

Look-back Decoding for Open-Ended Text Generation PDF Bib

Nan Xu, Chunting Zhou, Asli Celikyilmaz, Xuezhe Ma

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023) (Outstanding Paper Award)

Challenges in Context-Aware Neural Machine Translation PDF Bib Video

Linghao Jin*, Jacqueline He*, Jonathan May, Xuezhe Ma

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)

Evaluating Large Language Models on Controlled Generation Tasks PDF Bib

Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, Xuezhe Ma