About
I build multimodal foundation models, from pre-training data and recipes through SFT and RL, and the evaluations that tell us whether they work.
I am a Senior Staff Software Engineer at Google in Seattle. I work on Google's agentic harness capabilities, and on evaluation, training data, and environments for training and evaluating frontier models.
Before Google, I was a Principal Applied Scientist at Amazon AGI. There I was the technical lead for multimodal training, data, and evaluation across two generations of Amazon Nova models: Nova Lite, Nova Pro, Nova Omni, and Nova Multimodal Embeddings. I led a team of 60+ scientists and engineers across pre-training, supervised fine-tuning, and reinforcement learning. Earlier at AWS I worked on Amazon Titan multimodal models and on large-scale and few-shot object detection for Amazon Rekognition.
Alongside model work, I contribute to evaluation research, including benchmarks that test whether models ground their answers in the images, video, and audio they are given.
I received my Ph.D. from Nanyang Technological University, Singapore, in 2016, advised by Prof. Jianxin Wu and Prof. Jianfei Cai, and my B.Sc. from Shanghai Jiao Tong University in 2011.
News
- ANCHOR, an audio-visual grounded reasoning benchmark, accepted to the NeurIPS 2026 Evaluations and Datasets Track.
- MKP-Adapter, backward-compatible multimodal embeddings, accepted to ECCV 2026.
- Joined Google in Seattle to work on agentic harnesses and on evaluation and training data for frontier models.
- Co-organized the BEAM 2 and ILG workshops at ECCV 2026.
- ToolsRL, tool-supervised RL for visual reasoning, accepted to CVPR 2026 Findings.
- MAVERIX, an audio-visual evaluation benchmark, published at AAAI 2026.
- Amazon Nova 2 and Nova Multimodal Embeddings released, with technical reports.
- Co-organized the BEAM workshop at CVPR 2025 and the ILG workshop at ICCV 2025.
Research interests
Multimodal foundation models
Pre-training, SFT, and RL for models that read documents, images, video, and audio.
Agents
Agentic harnesses, computer and browser use, training and evaluation environments.
Benchmarks and data
Grounded and faithful reasoning, hallucination, data curation, and training mixtures.
Experience
2026–now
Senior Staff Software Engineer (ML) · Seattle
- Agentic harness capabilities, and evaluation, training data, and environments for frontier model training and evaluation.
2023–2026
Amazon AGI
Principal Applied Scientist · Seattle
- Amazon Nova 2 (2025): multimodal pre-training, SFT, and RL for Nova 2 Lite, Pro, and Omni, including document understanding, video and audio understanding, and UI grounding. Omni is a single model that takes text, images, video, and audio and generates text and images.
- Amazon Nova (2024): multimodal capabilities for Nova Pro and Lite, the first models on Amazon Bedrock with video understanding, evaluated on agentic UI benchmarks such as GroundUI-1K, VisualWebBench, and Mind2Web.
- Nova Multimodal Embeddings (2025): one embedding space for text, images, documents, video, and audio.
- The models' vision and multimodal capabilities underpin Amazon Nova Act, Amazon's browser agent SDK.
2017–2023
Amazon Web Services
Senior Applied Scientist, Applied Scientist · Seattle
- Amazon Titan (2023): multimodal understanding track, including Titan Multimodal Embeddings and Titan Image Generator.
- Rekognition Custom Labels: few-shot, semi-supervised, and continual learning for detection and classification.
- Rekognition Labels v2: large-scale classification and detection, including detecting 11K+ classes without fine-grained boxes.
Publications
Benchmarks & evaluation
- NeurIPS 2026Evaluations & DatasetsANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning BenchmarkJoel Julin, Souraja Kundu, Liza Dahiya, George Z. Wei, Liuyue Xie, Jingru Yi, Xu Zhang, Hao Yang, László A. Jeni
- AAAI 2026MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeXLiuyue Xie, Avik Kuthiala, George Z. Wei, Ce Zheng, Ananya Bal, Mosam Dabhi, Liting Wen, Taru Rustagi, Ethan Lai, Sushil Khyalia, Rohan Choudhury, Morteza Ziyadi, Xu Zhang, Hao Yang, László A. Jeni
- CVPR 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language ModelsPrannay Kaul, Zhizhong Li, Hao Yang, Yonatan Dukler, Ashwin Swaminathan, C. J. Taylor, Stefano Soatto
- ECCV 2022Rethinking Few-Shot Object Detection on a Multi-Domain BenchmarkKibok Lee, Hao Yang, Satyaki Chakraborty, Zhaowei Cai, Gurumurthy Swaminathan, Avinash Ravichandran, Onkar Dabeer
- PreprintUnder reviewVisAtom: Do Multimodal Models Truly See Before They Think?Mingye Lu, Zheyu Aqa Zhang, David Wu, Ji Li, Shengcao Cao, Xu Zhang, Hao Yang, Yang Zou, Liangyan Gui, Yu-Xiong Wang
Foundation models
- Tech report2025Amazon Nova 2: Multimodal Reasoning and Generation ModelsAmazon AGI
- Tech report2025Amazon Nova Multimodal Embeddings: Technical Report and Model CardAmazon AGI
- Tech report2024The Amazon Nova Family of Models: Technical Report and Model CardAmazon AGI
- ECCV 2026Multi-modal Knowledge Preserving Adapter for Embedding Backward CompatibilityJaeseok Byun, Gukyeong Kwon, Han-Kai Hsu, Meher Gitika Karumuri, Zhikang Zhang, Hao Yang, Davide Modolo
- CVPR 2026FindingsVisual Reasoning through Tool-supervised Reinforcement LearningQihua Dong, Gozde Sahin, Pei Wang, Zhaowei Cai, Robik Singh Shrestha, Hao Yang, Davide Modolo
- WACV 2026Learning Compact Video Representations for Efficient Long-Form Video Understanding in Large Multimodal ModelsYuxiao Chen, Jue Wang, Zhikang Zhang, Jingru Yi, Xu Zhang, Yang Zou, Zhaowei Cai, Steve Yuan, Xinyu Li, Hao Yang, Davide Modolo
- ICCV 2025Enhancing Numerical Prediction of MLLMs with Soft LabelingPei Wang, Zhaowei Cai, Hao Yang, Davide Modolo, Ashwin Swaminathan
- CVPR 2025Scaling up Image Segmentation across Data and TasksPei Wang, Zhaowei Cai, Hao Yang, Ashwin Swaminathan, R. Manmatha, Stefano Soatto
- NeurIPS 2023Your Representations Are in the Network: Composable and Parallel Adaptation for Large Scale ModelsYonatan Dukler, Alessandro Achille, Hao Yang, Varsha Vivek, Luca Zancato, Benjamin Bowman, Avinash Ravichandran, Charless Fowlkes, Ashwin Swaminathan, Stefano Soatto
- CVPR 2023A Meta-Learning Approach to Predicting Performance and Data RequirementsAchin Jain, Gurumurthy Swaminathan, Paolo Favaro, Hao Yang, Avinash Ravichandran, Hrayr Harutyunyan, Alessandro Achille, Onkar Dabeer, Bernt Schiele, Ashwin Swaminathan, Stefano Soatto
- CVPR 2023Guided Recommendation for Model Fine-TuningHao Li, Charless Fowlkes, Hao Yang, Onkar Dabeer, Zhuowen Tu, Stefano Soatto
- ICLR 2020Rethinking the Hyperparameters for Fine-tuningHao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, Stefano Soatto
Detection, recognition & large-scale learning
- CVPR 2022Omni-DETR: Omni-Supervised Object Detection with TransformersPei Wang, Zhaowei Cai, Hao Yang, Gurumurthy Swaminathan, Nuno Vasconcelos, Bernt Schiele, Stefano Soatto
- ECCV 2020Workshop, Task-CVMulti-task Incremental Learning for Object DetectionXialei Liu, Hao Yang, Avinash Ravichandran, Rahul Bhotika, Stefano Soatto
- ICCV 2019Detecting 11K Classes: Large Scale Object Detection without Fine-Grained Bounding BoxesHao Yang, Hao Wu, Hao Chen
- ECCV 2018Zero-Annotation Object Detection with Web Knowledge TransferQingyi Tao, Hao Yang, Jianfei Cai
- TMM 2018IEEE Trans. MultimediaExploiting Web Images for Weakly Supervised Object DetectionQingyi Tao, Hao Yang, Jianfei Cai
- AAAI 2018SC2Net: Sparse LSTMs for Sparse Coding OralJoey Tianyi Zhou, Jiawei Du, Kai Di, Xi Peng, Hao Yang, Sinno Jialin Pan, Ivor Tsang, Yong Liu, Zheng Qin, Rick Siow Mong Goh
- CVPR 2017MIML-FCN+: Multi-instance Multi-label Learning via Fully Convolutional Networks with Privileged InformationHao Yang, Joey Tianyi Zhou, Jianfei Cai, Yew Soon Ong
- ECCV 2016Improving Multi-label Learning with Missing Labels by Structured Semantic Correlations OralHao Yang, Joey Tianyi Zhou, Jianfei Cai
- CVPR 2016Exploit Bounding Box Annotations for Multi-label Object RecognitionHao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao, Jianxin Wu, Jianfei Cai
- TNNLS 2015IEEE Trans. NNLSLinear Regression Based Efficient SVM Learning for Large Scale ClassificationJianxin Wu, Hao Yang
- IJCAI 2013Reduced Heteroscedasticity Linear Regression for Nyström ApproximationHao Yang, Jianxin Wu
- ACML 2012Practical Large Scale Classification with Additive KernelsHao Yang, Jianxin Wu
- ACCV 2012Object Templates for Visual Place CategorizationHao Yang, Jianxin Wu
Patents
- Mitigating Bias in Multimodal Models via Query Transformation.
- Automated Model Selection for Network-Based Image Recognition Service.
Service
- Workshop organizer: BEAM at CVPR 2025 and BEAM 2 at ECCV 2026; ILG at ECCV 2024, ICCV 2025, and ECCV 2026.
- Program committee: CVPR, ICCV, ECCV, ICLR, NeurIPS, AAAI, IJCAI, WACV.
- Journal reviewer: IEEE TPAMI, IEEE TIP, IEEE TBD, Pattern Recognition, Pattern Recognition Letters.
Beyond research
Science fiction and fantasy (Tolkien, Martin, Philip K. Dick, Gaiman, Stephenson), auteur films, street photography, basketball, diving, and music from classical and film scores to 60s–90s rock and J-pop.