All Segments
Unfavorable Incentives in Machine Learning Research - John Schulman (OpenAI Co-founder)
4 segments available
Segments Timeline
The State of Machine Learning Literature
John Schulman discusses the quality of machine learning (ML) papers compared to other fields, particularly social sciences. He highlights the practicality of ML research, emphasizing the importance of reproducibility and the tendency for researchers to open source their methods. Schulman notes that while there are complaints about the literature, the field remains relatively healthy due to its focus on practical implementation.
"when you look at the average ml paper does it feel like a really solid piece of literature or does it feel often like it's the equivalent of what packing is in the social sciences everyone has their c..."
The State of Machine Learning Literature
John Schulman discusses the quality of machine learning (ML) papers compared to other fields, particularly social sciences. He highlights the practicality of ML research, emphasizing the importance of reimplementing methods and the tendency for complex methods to be overlooked. Schulman notes that while there are unfavorable incentives in the field, such as making baseline methods appear worse, he believes the field is progressing overall.
"when you look at the average ml paper does it feel like a really solid piece of literature or does it feel often like it's the equivalent of what packing is in the social sciences everyone has their c..."
Unfavorable Incentives in ML Research
In this segment, Schulman addresses the unfavorable incentives present in machine learning research. He points out that researchers may intentionally make baseline methods appear worse and may focus on complex mathematical formulations rather than genuine scientific understanding. Schulman expresses a desire for more scientific inquiry in the field, rather than merely optimizing benchmarks.
"lot I guess there's also various unfavorable incentives like people are incentivized to make the Baseline methods like the methods they comparing to worse and there are other um mild pathologies like ..."
Incentives and Pathologies in ML Research
In this segment, Schulman elaborates on the unfavorable incentives present in machine learning research. He points out issues like the tendency to make comparison methods look worse and the inclination to present methods as mathematically sophisticated. Despite these challenges, he expresses a desire for more scientific understanding rather than merely optimizing benchmarks, indicating a need for deeper exploration in the field.
"actually try to open source their work a lot I guess there's also various unfavorable incentives like people are incentivized to make the Baseline methods like the methods they comparing to worse and ..."