Epoch 1️⃣9️⃣: This week in ML (+ Bioinformatics 🧬 and Astronomy 🌌)
Decision Transformer, MixerGAN, paratope-epitope prediction, H01 Dataset, Dark-Matter maps, NNPhD, and more ...
The authors present a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows them to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, they present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Link to the code.
While attention-based transformer networks achieve unparalleled success in nearly all language tasks, the large number of tokens coupled with the quadratic activation memory usage makes them prohibitive for visual tasks. As such, while language-to-language translation has been revolutionized by the transformer model, convolutional networks remain the de facto solution for image-to-image translation. The recently proposed MLP-Mixer architecture alleviates some of the speed and memory issues associated with attention-based networks while still retaining the long-range connections that make transformer models desirable. Leveraging this efficient alternative to self-attention, the authors propose a new unpaired image-to-image translation model called MixerGAN: a simpler MLP-based architecture that considers long-distance relationships between pixels without the need for expensive attention mechanisms. Quantitative and qualitative analysis shows that MixerGAN achieves competitive results when compared to prior convolutional-based methods.
ML in Bioinformatics 🧬
Neural message passing for joint paratope-epitope prediction
Antibodies are proteins in the immune system which bind to antigens to detect and neutralise them. The binding sites in an antibody-antigen interaction are known as the paratope and epitope, respectively, and the prediction of these regions is key to vaccine and synthetic antibody development. Contrary to prior art, the authors argue that paratope and epitope predictors require asymmetric treatment, and propose distinct neural message passing architectures that are geared towards the specific aspects of paratope and epitope prediction, respectively. They obtain significant improvements on both tasks, setting the new state-of-the-art and recovering favourable qualitative predictions on antigens of relevance to COVID-19.
(H01 Dataset) A connectomic study of a petascale fragment of human cerebral cortex
The H01 dataset is a 1.4 petabyte rendering of a small sample of human brain tissue, released by a collaboration between the Lichtman Laboratory at Harvard University and Google. The dataset comprises imaging data that covers roughly one cubic millimeter of brain tissue, and includes tens of thousands of reconstructed neurons, millions of neuron fragments, 130 million annotated synapses, 104 proofread cells, and many additional subcellular annotations and structures. H01 is thus far the largest sample of brain tissue imaged and reconstructed in this level of detail, in any species, and the first large-scale study of synaptic connectivity in the human cortex that spans multiple cell types across all layers of the cortex. The primary goals of this project are to produce a novel resource for studying the human brain and to improve and scale the underlying connectomics technologies.
Astroinformatics 🌌
Revealing the Local Cosmic Web from Galaxies by Deep Learning
80% of the matter in the Universe is in the form of dark matter that comprises the skeleton of the large-scale structure called the Cosmic Web. As the Cosmic Web dictates the motion of all matter in galaxies and inter-galactic media through gravity, knowing the distribution of dark matter is essential for studying the large-scale structure. However, the Cosmic Web’s detailed structure is unknown because it is dominated by dark matter and warm-hot inter-galactic media, both of which are hard to trace. The authors find that the mapping between the position and velocity of galaxies and the Cosmic Web using the results of the state-of-the-art cosmological galaxy simulations, Illustris-TNG. Using the local galaxy sample from Cosmicflows-3, they find the dark-matter map in the local Universe.
Machine-Learning Non-Conservative Dynamics for New-Physics Detection
Energy conservation is a basic physics principle, the breakdown of which often implies new physics. This paper presents a method for data-driven "new physics" discovery. Specifically, given a trajectory governed by unknown forces, our Neural New-Physics Detector (NNPhD) aims to detect new physics by decomposing the force field into conservative and non-conservative components, which are represented by a Lagrangian Neural Network (LNN) and a universal approximator network (UAN), respectively, trained to minimize the force recovery error plus a constant λ times the magnitude of the predicted non-conservative force. The authors demonstrate that NNPhD successfully discovers new physics in toy numerical experiments, rediscovering friction (1493) from a damped double pendulum, Neptune from Uranus' orbit (1846) and gravitational waves (2017) from an inspiraling orbit.
Explainable, Interpretable, Bias and Ethics in AI
Interesting Events and News 📰
How ISI Kolkata researchers developed Deep Learning tool for COVID-19 screening
PyTorch builds the future of AI and machine learning at Facebook
New Practical Data Science Specialization by deeplearning.ai and AWS
Chinese AI lab challenges Google, OpenAI with a model of 1.75 trillion parameters
China's Advanced 'Artificial Sun' Fusion Reactor Just Broke a New World Record
In Surprise Discovery, Astronomers Find Huge Star Cluster in Our Cosmic Backyard
We're Going Back to Venus! NASA Announces Two New Missions by 2030
Articles and Resources 📃 I liked
JAX learning resources? [Reddit Thread 🧵]
NeuralMagic | ResNet-50 on CPUs: Sparsifying for Better Performance on CPUs
Recommended Podcasts 🎧
Talk Python To Me | #318: Measuring your ML impact with CodeCarbon
The AI Health Podcast | The FDA's Bakul Patel on Regulating AI Medical Devices
Gradient Dissent | Graphcore's Phil Brown on how IPUs are advancing machine intelligence
Towards Data Science | 85. Brian Christian - The Alignment Problem