Search Results for author:
Found 24 papers, 15 papers with code
Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs
Next, we extend our analysis to pretrained LLMs, including Llama and OLMo, showing that many attention heads exhibit a similar active-dormant mechanism as in the BB task, and that the mutual reinforcement mechanism also governs the emergence of extreme-token phenomena during LLM pretraining.
Cross-video Identity Correlating for Person Re-identification Pre-training
For example, compared with the previous state-of-the-art~\cite{ISR}, CION with the same ResNet50-IBN achieves higher mAP of 93. 3\% and 74. 3\% on Market1501 and MSMT17, while only utilizing 8\% training samples.
CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models
(3) Financial Practice: whether LLMs can fulfill the practical financial jobs, such as tax consultant, junior accountant and securities analyst.
SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentation
These strategies are designed to extract the most accurate masks from SAM's output, thus guiding the training of the student model with enhanced precision.
Collaborative Heterogeneous Causal Inference Beyond Meta-analysis
Instead of adjusting the distribution shift separately, we use weighted propensity score models to collaboratively adjust for the distribution shift.
DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models
Large language models (LLMs) face a daunting challenge due to the excessive computational and memory requirements of the commonly used Transformer architecture.
PanGu-$π$: Enhancing Language Model Architectures via Nonlinearity Compensation
We then demonstrate that the proposed approach is significantly effective for enhancing the model nonlinearity through carefully designed ablations; thus, we present a new efficient model architecture for establishing modern, namely, PanGu-$\pi$.
UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity
Firstly, we construct a new \textbf{dataset} named UFine6926.
LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models
Moreover, based on the observation that the accuracy of CLIP model does not increase correspondingly as the parameters of text encoder increase, an extra objective of masked language modeling (MLM) is leveraged for maximizing the potential of the shortened text encoder.
Towards Higher Ranks via Adversarial Weight Pruning
To this end, we propose a Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an adversarial manner.
Data-Free Distillation of Language Model by Text-to-Text Transfer
Data-Free Knowledge Distillation (DFKD) plays a vital role in compressing the model when original training data is unavailable.
NICE: Improving Panoptic Narrative Detection and Segmentation with Cascading Collaborative Learning
To address this, we introduce two cascading modules based on the barycenter of the mask, which are Coordinate Guided Aggregation (CGA) and Barycenter Driven Localization (BDL), responsible for segmentation and detection, respectively.
How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations
Through extensive probing and a new pasting experiment, we further reveal several mechanisms within the trained transformers, such as concrete copying behaviors on both the inputs and the representations, linear ICL capability of the upper layers alone, and a post-ICL representation selection mechanism in a harder mixture setting.
Cross-Model Cross-Stream Learning for Self-Supervised Human Action Recognition
Inspired by SkeletonBYOL, this paper further presents a Cross-Model and Cross-Stream (CMCS) framework.
FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation
Existing skeleton-based action recognition methods typically follow a centralized learning paradigm, which can pose privacy concerns when exposing human-related videos.
Contrastive Learning from Spatio-Temporal Mixed Skeleton Sequences for Self-Supervised Skeleton-Based Action Recognition
First, SkeleMix utilizes the topological information of skeleton data to mix two skeleton sequences by randomly combing the cropped skeleton fragments (the trimmed view) with the remaining skeleton sequences (the truncated view).
GraphMLP: A Graph MLP-Like Architecture for 3D Human Pose Estimation
To the best of our knowledge, this is the first MLP-Like architecture for 3D human pose estimation in a single frame and a video sequence.
Ranked #61 on
3D Human Pose Estimation
on Human3.6M
Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition
In this paper, to make better use of the movement patterns introduced by extreme augmentations, a Contrastive Learning framework utilizing Abundant Information Mining for self-supervised action Representation (AimCLR) is proposed.
Contrastive Learning
Few-Shot Skeleton-Based Action Recognition
+5
Pose-guided Feature Disentangling for Occluded Person Re-identification Based on Transformer
Therefore, we propose a transformer-based Pose-guided Feature Disentangling (PFD) method by utilizing pose information to clearly disentangle semantic components (e. g. human body or joint parts) and selectively match non-occluded parts correspondingly.
Learning Student Networks in the Wild
Experiments on various datasets demonstrate that the student networks learned by the proposed method can achieve comparable performance with those using the original dataset.
Pre-Trained Image Processing Transformer
To maximally excavate the capability of transformer, we present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs.
Ranked #1 on
Single Image Deraining
on Rain100L
(using extra training data)
On Positive-Unlabeled Classification in GAN
In contrast, it is more reasonable to treat the generated data as unlabeled, which could be positive or negative according to their quality.
Learning from Bad Data via Generation
A worst-case formulation can be developed over this distribution set, and then be interpreted as a generation task in an adversarial manner.
Robust Student Network Learning
In this way, a portable student network with significantly fewer parameters can achieve a considerable accuracy which is comparable to that of teacher network.
