In machine learning, you don't get credit for publishing rigorous papers. You get credits for publishing papers that show improved performance:
One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.
Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.
I guess people look at statistical machine learning and deep learning, see all the formulae and hear all the calculus terminology and think - "oh, wow, that's a really rigorous field! Look at all the formalisms!".
But it's not. It's an extremely, almost exclusively, empirical field. The mathiness and the formulae are just unfortunate attempts to pass off the whole endeavour as something that it's not- some kind of careful science that uncovers deep truths about intelligence and cognition. In truth, it's all just about beating other peoples' systems in very specific benchmarks.
If it wasn't for this culture of pretensions to higher science, machine learning papers would most likely be written with much more clarity than they are now and mistakes like the one described in the above article would be rare.
I've been quite disappointed to see how many papers from "cutting edge" research groups are chasing small improvements in well-known benchmarks by finding new techniques that happen to work, while there is a lot less effort put into finding out why. I guess Geoff Hinton's explanation about what gets published today explains it.
One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.
Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.
Geoff Hinton interview:
https://www.wired.com/story/googles-ai-guru-computers-think-...
I guess people look at statistical machine learning and deep learning, see all the formulae and hear all the calculus terminology and think - "oh, wow, that's a really rigorous field! Look at all the formalisms!".
But it's not. It's an extremely, almost exclusively, empirical field. The mathiness and the formulae are just unfortunate attempts to pass off the whole endeavour as something that it's not- some kind of careful science that uncovers deep truths about intelligence and cognition. In truth, it's all just about beating other peoples' systems in very specific benchmarks.
If it wasn't for this culture of pretensions to higher science, machine learning papers would most likely be written with much more clarity than they are now and mistakes like the one described in the above article would be rare.