Search Results for author:
Found 43 papers, 26 papers with code
Towards Adaptive Residual Network Training: A Neural-ODE Perspective
Serving as a crucial factor, the depth of residual networks balances model capacity, performance, and training efficiency.
MamKPD: A Simple Mamba Baseline for Real-Time 2D Keypoint Detection
Moreover, MamKPD achieves state-of-the-art results on the MPII dataset and competitive results on the AP-10K dataset while saving 85% of the parameters compared to ViTPose.
STOP: Spatiotemporal Orthogonal Propagation for Weight-Threshold-Leakage Synergistic Training of Deep Spiking Neural Networks
The prevailing of artificial intelligence-of-things calls for higher energy-efficient edge computing paradigms, such as neuromorphic agents leveraging brain-inspired spiking neural network (SNN) models based on spatiotemporally sparse binary spikes.
Vector-ICL: In-context Learning with Continuous Vector Representations
Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities on textual data.
LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy
Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages, which requires extensive parameter tuning thus unsuitable for pre-trained LLMs; (2) KV cache compression at test time, primarily through token eviction policies, which often overlook inter-layer dependencies and can be task-specific.
Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steering
However, they often struggle to fully comprehend and effectively utilize their input contexts, resulting in responses that are unfaithful or hallucinated.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
We introduce phi-3-mini, a 3. 8 billion parameter language model trained on 3. 3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3. 5 (e. g., phi-3-mini achieves 69% on MMLU and 8. 38 on MT-bench), despite being small enough to be deployed on a phone.
Ranked #5 on
MMR total
on MRR-Benchmark
(using extra training data)
DHRNet: A Dual-Path Hierarchical Relation Network for Multi-Person Pose Estimation
Multi-person pose estimation (MPPE) presents a formidable yet crucial challenge in computer vision.
Learning a Decision Tree Algorithm with Transformers
Decision trees are renowned for their ability to achieve high predictive performance while remaining interpretable, especially on tabular data.
Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs
In human-written articles, we often leverage the subtleties of text style, such as bold and italics, to guide the attention of readers.
Fast-ELECTRA for Efficient Pre-training
Although ELECTRA offers a significant boost in efficiency, its potential is constrained by the training cost brought by the auxiliary model.
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
In this study, we introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs).
Sparse Backpropagation for MoE Training
One defining characteristic of Mixture-of-Expert (MoE) models is their capacity for conducting sparse computation via expert routing, leading to remarkable scalability.
Toward Student-Oriented Teacher Network Training For Knowledge Distillation
How to conduct teacher training for knowledge distillation is still an open problem.
P4E: Few-Shot Event Detection as Prompt-Guided Identification and Localization
Our framework decomposes event detection into an identification task and a localization task.
Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfitting
We show that label noise exists in adversarial training.
Perturbation Deterioration: The Other Side of Catastrophic Overfitting
While this phenomenon is commonly explained as overfitting, we observe that it is a twin process: not only does the model catastrophic overfits to one type of perturbation, but also the perturbation deteriorates into random noise.
Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training
In this paper, we propose collaborative adversarial training to improve the data utilization, which coordinates virtual adversarial training (VAT) and adversarial training (AT) at different levels.
Multi-head or Single-head? An Empirical Comparison for Transformer Training
Multi-head attention plays a crucial role in the recent success of Transformer models, which leads to consistent performance improvements over conventional attention in various applications.
UCPhrase: Unsupervised Context-aware Quality Phrase Tagging
Training a conventional neural tagger based on silver labels usually faces the risk of overfitting phrase surface names.
Ranked #1 on
Phrase Tagging
on KPTimes
Data Quality Matters For Adversarial Training: An Empirical Study
Specifically, we first propose a strategy to measure the data quality based on the learning behaviors of the data during adversarial training and find that low-quality data may not be useful and even detrimental to the adversarial robustness.
On the Transformer Growth for Progressive BERT Training
Due to the excessive cost of large-scale language model pre-training, considerable efforts have been made to train BERT progressively -- start from an inferior but low-cost model and gradually grow the model to increase the computational complexity.
Overfitting or Underfitting? Understand Robustness Drop in Adversarial Training
Our goal is to understand why the robustness drops after conducting adversarial training for too long.
Very Deep Transformers for Neural Machine Translation
We explore the application of very deep Transformer models for Neural Machine Translation (NMT).
Ranked #1 on
Machine Translation
on WMT2014 English-French
(using extra training data)
Partially-Typed NER Datasets Integration: Connecting Practice to Theory
While typical named entity recognition (NER) models require the training set to be annotated with all target types, each available datasets may only cover a part of them.
Understanding the Difficulty of Training Transformers
Transformers have proved effective in many NLP tasks.
Ranked #5 on
Machine Translation
on WMT2014 English-French
TBC-Net: A real-time detector for infrared small target detection using semantic constraint
Meanwhile, we propose a joint loss function and a training method.
Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling
Its performance is largely influenced by the annotation quality and quantity in supervised learning scenarios, and obtaining ground truth labels is often costly.
CrossWeigh: Training Named Entity Tagger from Imperfect Annotations
Therefore, we manually correct these label mistakes and form a cleaner test set.
Ranked #6 on
Named Entity Recognition (NER)
on CoNLL++
(using extra training data)
Facet-Aware Evaluation for Extractive Summarization
In this paper, we present a facet-aware evaluation setup for better assessment of the information coverage in extracted summaries.
Raw-to-End Name Entity Recognition in Social Media
Our model neither requires the conversion from character sequences to word sequences, nor assumes tokenizer can correctly detect all word boundaries.
On the Variance of the Adaptive Learning Rate and Beyond
The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam.
Arabic Named Entity Recognition: What Works and What's Next
This paper presents the winning solution to the Arabic Named Entity Recognition challenge run by Topcoder. com.
Reliability-aware Dynamic Feature Composition for Name Tagging
We design a set of word frequency-based reliability signals to indicate the quality of each word embedding.
Looking Beyond Label Noise: Shifted Label Distribution Matters in Distantly Supervised Relation Extraction
In this paper, we study the problem what limits the performance of DS-trained neural models, conduct thorough analyses, and identify a factor that can influence the performance greatly, shifted label distribution.
Cross-relation Cross-bag Attention for Distantly-supervised Relation Extraction
Distant supervision leverages knowledge bases to automatically label instances, thus allowing us to train relation extractor without human annotations.
Learning Named Entity Tagger using Domain-Specific Dictionary
Recent advances in deep neural models allow us to build reliable named entity recognition (NER) systems without handcrafting features.
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
Many efforts have been made to facilitate natural language processing tasks with pre-trained language models (LMs), and brought significant improvements to various applications.
Ranked #49 on
Named Entity Recognition (NER)
on CoNLL 2003 (English)
Expert Finding in Heterogeneous Bibliographic Networks with Locally-trained Embeddings
We study the task of expert finding in heterogeneous bibliographical networks based on two aspects: textual content analysis and authority ranking.
Graph Clustering with Dynamic Embedding
Unlike most existing embedding methods that are task-agnostic, we simultaneously solve for the underlying node representations and the optimal clustering assignments in an end-to-end manner.
Social and Information Networks Physics and Society
Empower Sequence Labeling with Task-Aware Neural Language Model
In this study, we develop a novel neural framework to extract abundant knowledge hidden in raw texts to empower the sequence labeling task.
Ranked #13 on
Part-Of-Speech Tagging
on Penn Treebank
Heterogeneous Supervision for Relation Extraction: A Representation Learning Approach
These annotations, referred as heterogeneous supervision, often conflict with each other, which brings a new challenge to the original relation extraction task: how to infer the true label from noisy labels for a given instance.
