Epoch 2️⃣1️⃣: This week in ML (+ Bioinformatics 🧬 and Astronomy 🌌)
Scaling Vision Transformers, SimGAN, Meta-genomic classification, Multi-StyleGAN, SCONE, and more ...
Attention-based neural networks such as the Vision Transformer (ViT) have recently attained state-of-the-art results on many computer vision benchmarks. Scale is a primary ingredient in attaining excellent results, therefore, understanding a model's scaling properties is a key to designing future generations effectively. While the laws for scaling Transformer language models have been studied, it is unknown how Vision Transformers scale. To address this, the authors scaled ViT models and data, both up and down, and characterize the relationships between error rate, data, and compute. Along the way, they refine the architecture and training of ViT, reducing memory consumption and increasing accuracy the resulting models.
Excerpt from the Google AI Blog
The authors propose to treat the physics simulator as a learnable component that is trained by Deep Reinforcement Learning with a special reward function that penalizes discrepancies between the trajectories (i.e., the movement of the robots over time) generated in simulation and a small number of trajectories that are collected on real robots. They use generative adversarial networks (GANs) to provide such a reward, and formulate a hybrid simulator that combines learnable neural networks and analytical physics equations, to balance model expressiveness and physical correctness. On robotic locomotion tasks, their method outperforms multiple strong baselines, including domain randomization.
ML in Bioinformatics 🧬
MetaCache-GPU: Ultra-Fast Metagenomic Classification
The cost of DNA sequencing has dropped exponentially over the past decade, making genomic data accessible to a growing number of scientists. In bioinformatics, localization of short DNA sequences (reads) within large genomic sequences is commonly facilitated by constructing index data structures which allow for efficient querying of substrings. In this paper, the introduce MetaCache-GPU -- an ultra-fast metagenomic short read classifier specifically tailored to fit the characteristics of CUDA-enabled accelerators. Our approach employs a novel hash table variant featuring efficient minhash fingerprinting of reads for locality-sensitive hashing and their rapid insertion using warp-aggregated operations. Link to the code.
Multi-StyleGAN: Towards Image-Based Simulation of Time-Lapse Live-Cell Microscopy
Time-lapse fluorescent microscopy (TLFM) combined with predictive mathematical modelling is a powerful tool to study the inherently dynamic processes of life on the single-cell level. Such experiments are costly, complex and labour intensive. A complimentary approach and a step towards completely in silico experiments, is to synthesise the imagery itself. Here, the authors propose Multi-StyleGAN as a descriptive approach to simulate time-lapse fluorescence microscopy imagery of living cells, based on a past experiment. This novel generative adversarial network synthesises a multi-domain sequence of consecutive timesteps. Code to be made available soon at this link.
Astroinformatics 🌌
Detecting Pulsars with Neural Networks: A Proof of Concept
Pulsar searches are computationally demanding efforts to discover dispersed periodic signals in time- and frequency-resolved data from radio telescopes. The complexity and computational expense of simultaneously determining the frequency-dependent delay (dispersion) and the periodicity of the signal is further exacerbated by the presence of various types of radio-frequency interference (RFI) and observing-system effects. The present a novel approach to the analysis of pulsar search data. Specifically, they present a neural-network-based pipeline that efficiently suppresses a wide range of RFI signals and instrumental instabilities and furthermore corrects for (a priori unknown) interstellar dispersion. Link to the code.
SCONE: Supernova Classification with a Convolutional Neural Network
The authors present a novel method of classifying Type Ia supernovae using convolutional neural networks, trained on photometric information only, eliminating the need for accurate redshift data. Photometric data is pre-processed via 2D Gaussian process regression into two-dimensional images created from flux values at each location in wavelength-time space. These "flux heat-maps" of each supernova detection, along with "uncertainty heat-maps" of the Gaussian process uncertainty, constitute the dataset for their model. This preprocessing step not only smooths over irregular sampling rates between filters but also allows SCONE to be independent of the filter set on which it was trained. Link to the code.
Explainable, Interpretable, Bias and Ethics in AI
Interesting Events and News 📰
Machine learning is booming in medicine. It’s also facing a credibility crisis
New AI supercomputer will help create the largest-ever 3D map of the universe
We’re Facing a Fake Science Crisis, and AI Is Making It Worse
Articles and Resources 📃 I liked
Prospects for ML in astronomy (Slides)
Model Monitoring Enables Robust Machine Learning Applications
Automate your data science project structure in three easy steps
GML Express: keynotes at ICLR, topics at ICML 2021, and new GNN tutorials.
Recommended Podcasts 🎧
Gradient Dissent | Clément Delangue, CEO of Hugging Face, on the power of the open source community
The Real Python Podcast | Detecting Deforestation With Python & Using GraphQL With Django and Vue
The Robot Brains Podcast | Alison Gopnik on the different (and similar) ways robots and children learn
Lex Fridman Podcast | #190 – Jordan Ellenberg: Mathematics of High-Dimensional Shapes and Geometries