As Machine Learning captures our imagination, it’s important to separate the material from the hype. Here are 5 simple questions you should ask to help reduce AI hype:
- “How much training data is required?”
- “Can this work unsupervised (= without labelling the examples)?”
- “Can the system predict out of vocabulary names?” (i.e. Imagine if I said “My friend Rudinyard was mean to me” – many AI systems would never be able to answer “Who was mean to me?” as Rudinyard is out of its vocabulary)
- “How much does the accuracy fall as the input story gets longer?”
- “How stable is the model’s performance over time?”
source: Stephen Merity of Salesforce
My commentary on Stephen’s article. Stephen is placing the bar for ML high. AFAIK, Google DeepMind’s AlphaGo fulfills these five requirements, but Google Photos does not, since Google photo uses human crowdsourced labeling of what a cat is to determine cat videos. The fifth requirement, “How stable is the model’s performance over time?”, seems to be the least interesting and the most interesting at the same time. It’s the least interesting requirement because the evolution of a cat will indeed eventually change the way the cat looks and untrain even the most sophisticated ML model, unless the model adapts as the cat evolves. It’s the most interesting requirement because I can imagine how a model would fail to understand meme’s such as “Bye Felicia”, “On Fleek”, “idk my BFF Jill” or another human-created that seems like gibberish to machines or people not in the know.