Search Results for author:
Found 18 papers, 13 papers with code
Spatial-Temporal Mixture-of-Graph-Experts for Multi-Type Crime Prediction
As various types of crime continue to threaten public safety and economic development, predicting the occurrence of multiple types of crimes becomes increasingly vital for effective prevention measures.
A Survey of Foundation Models for Music Understanding
Music is essential in daily life, fulfilling emotional and entertainment needs, and connecting us personally, socially, and culturally.
LLoCO: Learning Long Contexts Offline
Processing long contexts remains a challenge for large language models (LLMs) due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation.
Masked Completion via Structured Diffusion with White-Box Transformers
We do this by exploiting a fundamental connection between diffusion, compression, and (masked) completion, deriving a deep transformer-like masked autoencoder architecture, called CRATE-MAE, in which the role of each layer is mathematically fully interpretable: they transform the data distribution to and from a structured representation.
When Do We Not Need Larger Vision Models?
Our results show that a multi-scale smaller model has comparable learning capacity to a larger model, and pre-training smaller models with S$^2$ can match or even exceed the advantage of larger models.
White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?
This leads to a family of white-box transformer-like deep network architectures, named CRATE, which are mathematically fully interpretable.
Emergence of Segmentation with Minimalistic White-Box Transformers
Transformer-like models for vision tasks have recently proven effective for a wide range of downstream applications such as segmentation and detection.
White-Box Transformers via Sparse Rate Reduction
Particularly, we show that the standard transformer block can be derived from alternating optimization on complementary parts of this objective: the multi-head self-attention operator can be viewed as a gradient descent step to compress the token sets by minimizing their lossy coding rate, and the subsequent multi-layer perceptron can be viewed as attempting to sparsify the representation of the tokens.
Efficient Maximal Coding Rate Reduction by Variational Forms
The principle of Maximal Coding Rate Reduction (MCR$^2$) has recently been proposed as a training objective for learning discriminative low-dimensional structures intrinsic to high-dimensional data to allow for more robust training than standard approaches, such as cross-entropy minimization.
Incremental Learning of Structured Memory via Closed-Loop Transcription
Our method is simpler than existing approaches for incremental learning, and more efficient in terms of model size, storage, and computation: it requires only a single, fixed-capacity autoencoding network with a feature space that is used for both discriminative and generative purposes.
Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction
In particular, we propose to learn a closed-loop transcription between a multi-class multi-dimensional data distribution and a linear discriminative representation (LDR) in the feature space that consists of multiple independent multi-dimensional linear subspaces.
Can We Characterize Tasks Without Labels or Features?
The problem of expert model selection deals with choosing the appropriate pretrained network ("expert") to transfer to a target task.
How Low Can We Go: Trading Memory for Error in Low-Precision Training
Low-precision arithmetic trains deep learning models using less energy, less memory and less time.
TenIPS: Inverse Propensity Sampling for Tensor Completion
Tensors are widely used to represent multiway arrays of data.
Incremental Learning via Rate Reduction
Current deep learning architectures suffer from catastrophic forgetting, a failure to retain knowledge of previously learned classes when incrementally trained on new classes.
Efficient AutoML Pipeline Search with Matrix and Tensor Factorization
Data scientists seeking a good supervised learning model on a new dataset have many choices to make: they must preprocess the data, select features, possibly reduce the dimension, select an estimation algorithm, and choose hyperparameters for each of these pipeline components.
PARN: Position-Aware Relation Networks for Few-Shot Learning
However, due to the inherent local connectivity of CNN, the CNN-based relation network (RN) can be sensitive to the spatial position relationship of semantic objects in two compared images.
WIDER Face and Pedestrian Challenge 2018: Methods and Results
This paper presents a review of the 2018 WIDER Challenge on Face and Pedestrian.
