Search Results for author: Tianyu He

Found 31 papers, 6 papers with code

Is Vanilla MLP in Neural Radiance Field Enough for Few-shot View Synthesis?

no code implementations10 Mar 2024 Hanxin Zhu, Tianyu He, Xin Li, Bingchen Li, Zhibo Chen

Neural Radiance Field (NeRF) has achieved superior performance for novel view synthesis by modeling the scene with a Multi-Layer Perception (MLP) and a volume rendering procedure, however, when fewer known views are given (i. e., few-shot view synthesis), the model is prone to overfit the given views.

Novel View Synthesis

CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency

no code implementations26 Feb 2024 Hanxin Zhu, Tianyu He, Zhibo Chen

Furthermore, to regularize the unseen target views, we constrain the rendered colors and depths from different input views to be the same.

Novel View Synthesis

UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing

no code implementations20 Feb 2024 Jianhong Bai, Tianyu He, Yuchi Wang, Junliang Guo, Haoji Hu, Zuozhu Liu, Jiang Bian

Recent advances in text-guided video editing have showcased promising results in appearance editing (e. g., stylization).

Video Editing

Towards a Psychological Generalist AI: A Survey of Current Applications of Large Language Models and Future Prospects

no code implementations1 Dec 2023 Tianyu He, Guanghui Fu, Yijing Yu, Fan Wang, Jianqiang Li, Qing Zhao, Changwei Song, Hongzhi Qi, Dan Luo, Huijing Zou, Bing Xiang Yang

The complexity of psychological principles underscore a significant societal challenge, given the vast social implications of psychological problems.

GAIA: Zero-shot Talking Avatar Generation

no code implementations26 Nov 2023 Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, Han Hu, HsiangTao Wu, Sheng Zhao, Jiang Bian

Zero-shot talking avatar generation aims at synthesizing natural talking videos from speech and a single portrait image.

Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos

no code implementations3 Nov 2023 Dayal Singh Kalra, Tianyu He, Maissam Barkeshli

In gradient descent dynamics of neural networks, the top eigenvalue of the Hessian of the loss (sharpness) displays a variety of robust phenomena throughout training.

Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape

no code implementations31 Oct 2023 Wei Zhao, Yijun Wang, Tianyu He, Lianying Yin, Jianxin Lin, Xin Jin

To augment the richness of 3D facial animation, we construct a new 3D dataset with detailed shapes and learn to synthesize facial details in line with speech content.

To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets

1 code implementation19 Oct 2023 Darshil Doshi, Aritra Das, Tianyu He, Andrey Gromov

Robust generalization is a major challenge in deep learning, particularly when the number of trainable parameters is very large.

Memorization

MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models

1 code implementation18 Oct 2023 Dingyao Yu, Kaitao Song, Peiling Lu, Tianyu He, Xu Tan, Wei Ye, Shikun Zhang, Jiang Bian

For developers and amateurs, it is very difficult to grasp all of these task to satisfy their requirements in music processing, especially considering the huge differences in the representations of music data and the model applicability across platforms among various tasks.

Music Classification

EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model

no code implementations20 Jun 2023 Lianying Yin, Yijun Wang, Tianyu He, Jinming Liu, Wei Zhao, Bohan Li, Xin Jin, Jianxin Lin

In this paper, we present a novel framework (EMoG) to tackle the above challenges with denoising diffusion models: 1) To alleviate the one-to-many problem, we incorporate emotion clues to guide the generation process, making the generation much easier; 2) To model joint correlation, we propose to decompose the difficult gesture generation into two sub-problems: joint correlation modeling and temporal dynamics modeling.

Denoising Gesture Generation

Memories are One-to-Many Mapping Alleviators in Talking Face Generation

no code implementations9 Dec 2022 Anni Tang, Tianyu He, Xu Tan, Jun Ling, Li Song

More specifically, the implicit memory is employed in the audio-to-expression model to capture high-level semantics in the audio-expression shared space, while the explicit memory is employed in the neural-rendering model to help synthesize pixel-level details.

Neural Rendering Talking Face Generation

Fed-TDA: Federated Tabular Data Augmentation on Non-IID Data

1 code implementation22 Nov 2022 Shaoming Duan, Chuanyi Liu, Peiyi Han, Tianyu He, Yifeng Xu, Qiyuan Deng

Non-independent and identically distributed (non-IID) data is a key challenge in federated learning (FL), which usually hampers the optimization convergence and the performance of FL.

Data Augmentation Federated Learning

Image Coding for Machines with Omnipotent Feature Learning

no code implementations5 Jul 2022 Ruoyu Feng, Xin Jin, Zongyu Guo, Runsen Feng, Yixin Gao, Tianyu He, Zhizheng Zhang, Simeng Sun, Zhibo Chen

Learning a kind of feature that is both general (for AI tasks) and compact (for compression) is pivotal for its success.

Self-Supervised Learning

AutoInit: Automatic Initialization via Jacobian Tuning

no code implementations27 Jun 2022 Tianyu He, Darshil Doshi, Andrey Gromov

Good initialization is essential for training Deep Neural Networks (DNNs).

Semantically Video Coding: Instill Static-Dynamic Clues into Structured Bitstream for AI Tasks

no code implementations25 Jan 2022 Xin Jin, Ruoyu Feng, Simeng Sun, Runsen Feng, Tianyu He, Zhibo Chen

Traditional media coding schemes typically encode image/video into a semantic-unknown binary stream, which fails to directly support downstream intelligent tasks at the bitstream level.

Action Recognition Object +8

Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications

no code implementations23 Nov 2021 Darshil Doshi, Tianyu He, Andrey Gromov

We derive recurrence relations for the norms of partial Jacobians and utilize these relations to analyze criticality of deep fully connected neural networks with LayerNorm and/or residual connections.

Meta Clustering Learning for Large-scale Unsupervised Person Re-identification

no code implementations19 Nov 2021 Xin Jin, Tianyu He, Xu Shen, Tongliang Liu, Xinchao Wang, Jianqiang Huang, Zhibo Chen, Xian-Sheng Hua

Unsupervised Person Re-identification (U-ReID) with pseudo labeling recently reaches a competitive performance compared to fully-supervised ReID methods based on modern clustering algorithms.

Clustering Unsupervised Person Re-Identification

Enhance Images as You Like with Unpaired Learning

1 code implementation4 Oct 2021 Xiaopeng Sun, Muxingzi Li, Tianyu He, Lubin Fan

Low-light image enhancement exhibits an ill-posed nature, as a given image may have many enhanced versions, yet recent studies focus on building a deterministic mapping from input to an enhanced version.

Generative Adversarial Network Low-Light Image Enhancement

Unleash the Potential of Adaptation Models via Dynamic Domain Labels

no code implementations29 Sep 2021 Xin Jin, Tianyu He, Xu Shen, Songhua Wu, Tongliang Liu, Xinchao Wang, Jianqiang Huang, Zhibo Chen, Xian-Sheng Hua

In this paper, we propose an embarrassing simple yet highly effective adversarial domain adaptation (ADA) method for effectively training models for alignment.

Domain Adaptation Memorization

Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularization

1 code implementation CVPR 2022 Xin Jin, Tianyu He, Kecheng Zheng, Zhiheng Yin, Xu Shen, Zhen Huang, Ruoyu Feng, Jianqiang Huang, Xian-Sheng Hua, Zhibo Chen

Specifically, we introduce Gait recognition as an auxiliary task to drive the Image ReID model to learn cloth-agnostic representations by leveraging personal unique and cloth-independent gait information, we name this framework as GI-ReID.

Cloth-Changing Person Re-Identification Computational Efficiency +1

Dense Interaction Learning for Video-based Person Re-identification

no code implementations ICCV 2021 Tianyu He, Xin Jin, Xu Shen, Jianqiang Huang, Zhibo Chen, Xian-Sheng Hua

The CNN encoder is responsible for efficiently extracting discriminative spatial features while the DI decoder is designed to densely model spatial-temporal inherent interaction across frames.

Video-Based Person Re-Identification

Generative Memorize-Then-Recall framework for low bit-rate Surveillance Video Compression

no code implementations30 Dec 2019 Yaojun Wu, Tianyu He, Zhibo Chen

In this paper, we figure out this issue by disentangling surveillance video into the structure of a global spatio-temporal feature (memory) for Group of Picture (GoP) and skeleton for each frame (clue).

Generative Adversarial Network Motion Compensation +1

Language Graph Distillation for Low-Resource Machine Translation

no code implementations17 Aug 2019 Tianyu He, Jiale Chen, Xu Tan, Tao Qin

Neural machine translation on low-resource language is challenging due to the lack of bilingual sentence pairs.

Knowledge Distillation Machine Translation +3

Learning to Transfer: Unsupervised Meta Domain Translation

1 code implementation1 Jun 2019 Jianxin Lin, Yijun Wang, Tianyu He, Zhibo Chen

Unsupervised domain translation has recently achieved impressive performance with Generative Adversarial Network (GAN) and sufficient (unpaired) training data.

Generative Adversarial Network Meta-Learning +1

Multi-Agent Dual Learning

no code implementations ICLR 2019 Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, Tie-Yan Liu

Dual learning has attracted much attention in machine learning, computer vision and natural language processing communities.

Machine Translation Translation

Learning based Facial Image Compression with Semantic Fidelity Metric

no code implementations25 Dec 2018 Zhibo Chen, Tianyu He

The experimental results verify the framework's efficiency by demonstrating performance improvement of 71. 41%, 48. 28% and 52. 67% bitrate saving separately over JPEG2000, WebP and neural network-based codecs under the same face verification accuracy distortion metric.

Face Recognition Face Verification +2

Learning for Video Compression

no code implementations26 Apr 2018 Zhibo Chen, Tianyu He, Xin Jin, Feng Wu

One key challenge to learning-based video compression is that motion predictive coding, a very effective tool for video compression, can hardly be trained into a neural network.

Multimedia Image and Video Processing

Cannot find the paper you are looking for? You can Submit a new open access paper.