Papers with Code Newsletter #20

👋 Welcome to the 20th issue of the Papers with Code newsletter. This week, we highlight:

several advances in machine learning for science,
state-of-the-art results on zero-shot image classification,
our partnership with ACL to bring ease of access to code and datasets,
several new research datasets and tools,
... and much more

Machine Learning for Science 🧬

In this special edition of the newsletter, we highlight a few papers that propose the use of machine learning systems and algorithms for accelerating scientific discovery.

Figure source: https://github.com/deepmind/alphafold

ML for Protein Structure Prediction

In the past few years, we have witnessed lots of progress in machine learning methods for computational biology. One of the most significant breakthroughs in this space came from DeepMind with the introduction of AlphaFold, an AI system that attains high accuracy on protein structure prediction. AlphaFold is a computational approach based on a novel neural network architecture which is capable of protein structure prediction to near experimental accuracy.

AlphaFold leverages several key ideas in the field, including: an architecture that jointly embeds multiple sequence alignments and pairwise features, a new output representation and loss that improves end-to-end structure prediction, an equivariant attention architecture, a mechanism to iteratively refine predictions using intermediate losses, masked MSA loss to jointly train with structure, self-distillation to learn from unlabelled protein sequences, and self-estimates of accuracy. In a whole, AlphaFold incorporates physical and biological knowledge about protein structure and achieves competitive results on the CASP14 assessment. DeepMind recently released the code for AlphaFold (find below) and published an accompanying article.

🔗 Paper & Code

Architecture overview of InfoCal used for generating extractive rationales. Figure source: Taylor et al. (2021)

ML for Predicting Hospital Readmission

Predictive systems are now being deployed in all areas of the health industry. In some difficult tasks, like ophthalmology and radiology, ML systems are now able to make better predictions that lead to more precise decision making. In the clinical setting, NLP systems are now being applied to electronic health records to more efficiently process and understand clinical free-text. Such systems can be trained to achieve accurate predictions on important tasks such as predicting the likelihood of a patient being readmitted to hospital.

In a recent paper, Taylor et al. (2021) apply InfoCal, a state-of-the-art model to generate extractive rationales for its prediction to support clinical decision-making. This is applied to the real-world task of predicting hospital readmission using hospital discharge notes. The model is able to produce competitive performance compared to transformer-based models. In addition, this study looks more closely at the interpretability of the model, with findings that suggest the importance of clinical language domain expertise critical to performance and interpretability.

🔗 Paper

Overview of method used to balance initiation interval to improve system throughput and reducing latency. Figure source: Que et al. (2021)

ML for Detecting Gravitational Waves

More recently, there has been huge interest in applying ML such as advanced neural networks to Physics. For instance, detecting gravitational waves is an important problem as it provides a unique way to study fundamental physics and other related problems. However, the current analysis approaches for estimating physical properties of the gravitational waves (e.g.,Markov-chain Monte Carlo) are inefficient at scale. To address some of these challenges, there exists several ML-based approaches (e.g, CNNs, RNNs, generative models) that target different aspects related to the detection of gravitational waves.

Latency is a big challenge in the efficient analysis of gravitational waves. Due to this, there are several works that focus on improving hardware efficiency and ways to accelerate algorithms for inferencing. To address some of these issues, Que et al. (2021) recently proposed a reconfigurable architecture for reducing the latency of RNNs used for detecting gravitational waves. The proposed architecture accelerates RNN inference for analyzing time-series data from LIGO detectors. The core idea is to optimize initiation intervals in a multi-layer LSTM by identifying factors that could be reused for each layer. Unbalanced initiation interval can stall the system and results in hardware inefficiency. Therefore, the proposed solution is to balance it to enable fast data analysis in gravitational wave experiments. These ideas are further used to generate low-latency FPGA designs with efficient resource utilization.

🔗 Paper & Code

More Works Doing ML for Science

Machine learning for science has become a common theme and there are new works emerging at a rapid pace. Below are just a few examples of some of the different types of work happening:

🔴 ML for molecule properties prediction - Choukroun and Wolf (2021)

🧬 ML for learning biological properties - Rives et al. (2020)

➕ ML for charged particle tracking - DeZoort et al. (2021)

🟢 ML for classifying unseen cell types - Wang et al. (2021)

🔬 ML applications for COVID-19 - Shorten et al. (2021)

🌧 ML for clouds and climate - Beucler et al. (2021)

⚕️ ML for improving real-time streaming tomography - Liu et al. (2019)

🌤 ML for tackling climate change - Rolnick et al. (2019)

🦠 ML for biological image synthesis - Osokin et al. (2017)

🌌 ML for cosmological reconstructions - Gómez-Vargas et al. (2021)

New Results on Papers with Code 📈

In this new segment of the newsletter we will provide quick highlights on some of the latest state-of-the-art results from Papers with Code.

🔥 Locked-Image Text Tuning (LiT): This new paper proposes a new contrastive tuning strategy, LiT-tuning, for state-of-the-art zero-shot transfer image classification. It outperforms models like CLIP and ALIGN on several benchmarks like ImageNet and ReaL.

💪🏼 Are Transformers More Robust than CNNs?: Proposes fair comparisons between Transformers and CNNs by focusing on robustness evaluation. Results suggest that CNNs can be as robust as Transformers when defending against adversarial attacks.

📺 CLIP2TV: Presents a simple new CLIP-based method, CLIP2TV, that achieves state-of-the-art results on the task of video-text retrieval on the MSR-VTT dataset.

💬 Novel Open-Domain QA: Introduces a novel four-stage open-domain QA pipeline with competitive performance on open-domain QA datasets like NaturalQuestions, TriviaQA, and EfficientQA.

Browse all new state-of-the-art results here.

Trending Research Datasets and Tools 🔬

Datasets

Natural Adversarial Objects: a new dataset to evaluate the robustness of object detection models. NAO contains 7,934 images and 9,943 objects that are unmodified and representative of real-world scenarios, but cause SoTA detection models to misclassify.

Graph Robustness Benchmark: a new benchmark that provides scalable, unified, modular, and reproducible evaluation on the adversarial robustness of graph machine learning models.

DeepNets-1M: a large-scale dataset of diverse computational graphs of neural network architectures. It's used to train models that perform parameter prediction on CIFAR-10 and ImageNet.

Tools

OpenPrompt: a new open-source unified framework for prompt-learning.

JaMIE: an open-access NLP tool for Japanese medical information extraction.

jaxdf: a new JAX-based research framework for writing differentiable PDE discretizations.

Community Highlights ✍️

We would like to thank:

@burrsettles for several contributions, including addition of several Duolingo datasets such as the Duolingo SLAM Shared Task.
@nishant for contributing a new dataset for subjective summary extraction called SubSumE.
@ermshaua for contributing TSSB, a new time series segmentation benchmark.
@gmftbyGMFTBY for contributing to multiple benchmarks including many additions to the Conversational Response Selection on Douban leaderboard.
@heinrichreimer for several contributions, including addition of a new task, Key Point Matching.

Special thanks to all our contributors for their ongoing contributions to Papers with Code.