Lei Ke

Postdoc at CMU

I am a Postdoctoral Research Associate at Computer Science of Carnegie Mellon University, working with Katerina Fragkiadaki. Previsouly, I was a postdoc researcher at Computer Vision Lab of ETH Zurich, with Martin Danelljan and Fisher Yu. I obtained my Ph.D. degree from CSE Department at HKUST, supervised by Chi-Keung Tang and Yu-Wing Tai. During the PhD journey, I also spent two years as a visiting scholar at ETH Zurich. My research goal is to enable machine to achieve 4D and multi-modality scene understanding from videos/images. I received my B.E. degree from the school of computer science at Wuhan University.

More info: [Email],[Google Scholar], [GitHub] where my leading opensource projects obtains over 6.5K+ GitHub stars.


Recent Publications

DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos
arXiv 2024
Wen-Hsuan Chu*, Lei Ke*, Katerina Fragkiadaki
DreamScene4D for generating 3D dynamic scenes of multiple objects from monocular videos.
[Paper] [Project] [Web]
Gaussian Grouping: Segment and Edit Anything in 3D Scenes
arXiv 2023
Mingqiao Ye, Martin Danelljan, Fisher Yu, Lei Ke
( denotes Project Lead)
Gaussian Grouping for open-world 3D Anything reconstruction, segmentation and editing.
[Paper] [Project] [Web]
Matching Anything By Segmenting Anything
CVPR 2024
Siyuan Li, Lei Ke, Martin Danelljan, Luigi Piccinelli, Mattia Segu, Luc Van Gool, Fisher Yu
MASA provides a universal instance appearance model for matching any objects in any domain.
Highlight (~3% acceptance rate)
Segment Anything in High Quality
NeurIPS 2023
Lei Ke*, Mingqiao Ye*, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu
(* denotes equal contribution)
We propose HQ-SAM to upgrade SAM for high-quality zero-shot segmentation.
HQ-SAM receives 2000+ Github stars in one month.
[Paper] [Project] [Pdf]
Segment Anything Meets Point Tracking
Frano Rajič, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, Fisher Yu
We propose SAM-PT to extend SAM to zero-shot video segmentation with point-based tracking.
SAM-PT receives 500+ Github stars in one week.
BiMatting: Efficient Video Matting via Binarization
NeurIPS 2023
Haotong Qin*, Lei Ke*, Xudong Ma, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Xianglong Liu, Fisher Yu
(* denotes equal contribution)
An accurate and efficient video matting model using binarization.
Cascade-DETR: Delving into High-Quality Universal Object Detection
ICCV 2023
Mingqiao Ye*, Lei Ke*, Siyuan Li, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, Fisher Yu
Promoting DETR's detection accuracy in universal domains via cascade attention.
Mask-Free Video Instance Segmentation
CVPR 2023
Lei Ke, Martin Danelljan, Henghui Ding, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu
Removing video and image mask annotation necessity for highly accurate VIS.
OVTrack: Open-Vocabulary Multiple Object Tracking
CVPR 2023
Siyuan Li, Tobias Fischer, Lei Ke, Henghui Ding, Martin Danelljan, Fisher Yu
First method and benchmark for open vocabulary tracking.
Video Mask Transfiner for High-Quality Video Instance Segmentation
ECCV 2022
Lei Ke, Henghui Ding, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu
A new HQ-YTVIS benchmark and transformer-based method for highly accurate VIS.
Mask Transfiner for High-Quality Instance Segmentation
CVPR 2022
Lei Ke, Martin Danelljan, Xia Li, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu
An efficient transformer-based method for highly accurate instance segmentation.
Transfiner receives 300+ Github stars in 3 months.
Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation
NeurIPS 2021
Lei Ke, Xia Li, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu
Efficient cross-attention on space-time memory for video instance segmentation.
Spotlight (3% acceptance rate). PCAN receives 200+ Github stars in one month.
Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers
CVPR 2021 & TPAMI 2023
Lei Ke, Yu-Wing Tai, Chi-Keung Tang
Instance segmentation with bilayer decoupling structure for occluder & occludee.
BCNet receives 300+ Github stars in 6 months.
Occlusion-Aware Video Object Inpainting
ICCV 2021
Lei Ke, Yu-Wing Tai, Chi-Keung Tang
Commonality-Parsing Network across Shape and Appearance for Partially Supervised Instance Segmentation
ECCV 2020
Qi Fan*, Lei Ke*, Wenjie Pei, Chi-Keung Tang, Yu-Wing Tai
(* denotes equal contribution)
GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervision
ECCV 2020
Lei Ke, Shichao Li, Yanan Sun, Yu-Wing Tai, Chi-Keung Tang
Cascaded Deep Monocular 3D Human Pose Estimation with Evolutionary Training Data
CVPR 2020
Shichao Li, Lei Ke, Kevin Pratama, Yu-Wing Tai, Chi-Keung Tang, Kwang-Ting Cheng
Oral Presentation, 5.03% acceptance rate
Reflective Decoding Network for Image Captioning
ICCV 2019
Lei Ke, Wenjie Pei, Ruiyu Li, Xiaoyong Shen, Yu-Wing Tai
Memory-Attended Recurrent Network for Video Captioning
CVPR 2019
Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen and Yu-Wing Tai
[Paper] [arXiv]


2024.04—Now: Postdoc at MLD, CMU

2023.07—2024.03: Postdoc at CVL, ETHz

2021.01—2023.03: Visiting PhD student at CVL, ETHz

2019.05—2023.05: HKUST Computer Vision Research Assistant

2017.11—2019.11: Tencent Youtu X-lab Computer Vision Research Intern, worked closely with Wenjie Pei.

2017.05—2017.10: Alibaba Engineering Intern

2016.05—2017.02: Undergraduate Research Assistant at Wuhan University


2023, Received CVPR Doctoral Consortium Awards(13% success rate), supervised by Michael Maire.
2022, Most Popular Speakers in TechBeat
2022, Research Travel Grant, ETH Zürich
2019, Research Travel Grant, HKUST
2019 - Present, Postgraduate Studentship, HKUST
2017, COMAP's Mathematical Contest in Modeling, Honorable prize
2015 - 2017, Excellent Student Scholarship, Wuhan University
2016, National Software Design Competition, Second Prize
2016, National Inspirational Scholarship, Wuhan University
2015, National College Students' Mathematics Competition, Third Prize

Professional Activities