Sara Hooker [Archive] - Forums - Portal of Robotics and Artificial Intelligence

Airicist2

16th June 2022, 18:55

https://youtu.be/sQFxbQ7ade0

Sara Hooker - The Hardware Lottery, Sparsity and Fairness

Oct 21, 2020

Dr. Tim Scarfe, Yannic Kilcher and Sayak Paul chat with Sara Hooker from the Google Brain team! We discuss her recent hardware lottery paper, pruning / sparsity, bias mitigation and intepretability.

The hardware lottery -- what causes inertia or friction in the marketplace of ideas? Is there a meritocracy of ideas or do the previous decisions we have made enslave us? Sara Hooker calls this a lottery because she feels that machine learning progress is entirely beholden to the hardware and software landscape. Ideas succeed if they are compatible with the hardware and software at the time and also the existing inventions. The machine learning community is exceptional because the pace of innovation is fast and we operate largely in the open, this is largely because we don't build anything physical which is expensive, slow and the cost of being scooped is high. We get stuck in basins of attraction based on our technology decisions and it's expensive to jump outside of these basins. So is this story unique to hardware and AI algorithms or is it really just the story of all innovation? Every great innovation must wait for the right stepping stone to be in place before it can really happen. We are excited to bring you Sara Hooker to give her take.

00:00:00 Tim Intro - Hardware Lottery
00:03:15 Tim Intro - Cultural divide in machine learning
00:04:47 Tim Intro - Pruning
00:06:47 Tim Intro - Bias Mitigation
00:09:46 Tim Intro - Intepretability
00:11:05 Sara joines
00:11:51 Show introduction with everyone on the call
00:14:45 Elevator pitch on hardware lottery
00:16:08 Whats so special about hardware and tooling
00:17:56 Connectionist approaches losing out and now being stuck with them
00:20:58 GPU to TPU
00:26:08 Isn't this just a story of stepping stones and innovation in general (Kenneth Stanley reference)
00:29:27 We have a missing counterfactual of what hardware could exist.
00:30:37 Capsule networks - we have converged on one "global update" paradigm of NNs
00:32:49 Compression / Structured / Unstructured sparsity
00:35:46 As you prune, what does a model forget? Longtail of model parameters encode low frequency information
00:39:14 Cultural Divide In Machine Learning (Welling vs Sutton)
00:42:28 Our own intelligence is based on local updates
00:44:33 Max Welling, Priors and the focussing on the long tail, lets not treat the data equally!
00:47:07 Sparsity training the future? (model compression research/gradient flow)
00:51:42 Is it a resource allocation problem? Too much exploration would spread us too thin
00:56:50 Isn't it just being at the right place at the right time? Is it a lottery?
01:00:08 National strategy to combat the hardware lottery
01:03:00 Ironic if DL created the next generation of hardware
01:03:28 Maybe we are in the AI winter now though? Maybe we need to go symbolic :)
01:07:13 Periods of feast and famine in ML research (Bayesian causal symbolic DL etc)
01:08:49 Characterising and mitigating bias in compact models.
01:12:18 Bias - dataset vs algorithm - how do we gety rid of it protected attributes
01:16:54 Are sparse networks more intepretable single image intepretations saliency methods
01:21:56 Protected attributes
01:25:19 How do you differentiate between something that is underrepresented vs more challenging
01:28:16 Anna Karenina principle and Sara's eccentric style of paper writing