Microsoft’s Steve Guggenheimer has revealed Microsoft’s leading position on four AI tests. The CVP of Microsoft’s developer platform says its AI is now first on the WikiSQL, ARC, SQuAD challenges. The leader of these challenges changes constantly, but it’s significant that Microsoft researchers top four simultaneously. The WikiSQL challenge is of Salesforce’s making and tests natural language interfaces for relational databases. Guggenheimer puts Microsoft’s WikiSQL success down to a new approach called IncSQL, which led to a 5.7% improvement to 87.1%. This is significantly higher than second place at 83.8%. ARC stands for AI Reasoning Challenge and is of Stanford’s creation. It asks an AI close to 8,000 grade-school multiple choice question to test different answering approaches. Microsoft Dynamic 365’s AI research team worked with the University of San Diego to pioneer the ET-RR approach. With it, Microsoft scored 36.36, versus the 33.87 of second place.
The SQuAD Tests
Stanford has two Question Answer Dataset tests, which test reading comprehension. Microsoft found that it’s AI was able to answer questions as well as a human in version 1.1. However, the university has since created a second version of the test. SQuAD 2.0 encourages the creation of AI’s that know their comprehension shortcomings, which can be incredibly valuable. As you’d expect, Microsoft researchers top 2.0 too, though they’re yet to reach human pairty. Stanford puts human’s EM value at 86.8 and F1 at 89.4. Microsoft Research Asia’s nlet sits at 74.2 and 77, respectively. Obviously, these tests aren’t the sole indicator of how well a company will do in the market. Microsoft links the two more than some, though, and will be implementing many of its breakthroughs in Dynamics + AI in October. “These results show the breadth of MRC challenges our teams are researching and the rapid pace of innovation and collaboration in the industry,” said Guggenheimer. “Combining researchers with engineering to tackle product challenges while participating in industry research challenges is shaping up to be a beneficial way to advance AI research and bring AI-based solutions to customers more quickly.”
