IBM will help broadcast the event live to television and its mobile app, using IBM Watson Media to watch for match highlights as they happen. It will take into account crowd noises, emotional player reactions, and other factors to determine the best highlight of a match. IBM has been a partner with the United States Tennis Association for 28 years, and it is renewing that partnership in a new five-year deal.
IBM has already done a test run of automated highlights at The Masters golf tournament and the Wimbledon tennis tournament. Now it is rolling out the technology more publicly at the USTA event.
“We have focused on cognitive and AI to advance the events and bring new value to tennis,” said Noah Syken, IBM vice president of sports and entertainment partnerships, in an interview with VentureBeat.
The U.S. Open will use one of the first solutions available through IBM Watson Media, called Cognitive Highlights. Developed at IBM Research with IBM iX, Cognitive Highlights can identify a match’s most important moments by analyzing statistical tennis data, sounds from the crowd, and player reactions using both action and facial expression recognition. The system then ranks the shots from seven U.S. Open courts and auto-curates the highlights, which simplifies the video production process and ultimately positions the USTA team to scale and accelerate the creation of cognitive highlight packages.
Each day, the USTA will post a Highlight of the Day, as ranked by Watson, on its Facebook page. People who “favorite” players on the U.S. Open apps will receive real-time push notification alerts about those players’ highlights. Fans on iOS 10 can play the highlights within the lock screen. On the player bio page, video highlights will be available across all of the USTA’s digital platforms. IBM will also show the highlights onsite, in the player’s lounge and in the fan-facing IBM Watson Experience on the plaza near Court 9.
“We’ve taken the USTA through the whole journey since before there was digital programming,” Syken said.
IBM Watson Media will offer refined searches of content down to the scene level based on rich metadata that identifies items and context. It can, for instance, automatically identify faces of celebrities in the audience.
Recommendation uplift offers more informed suggestions of new content for viewers based on a deeper understanding of video content. Intelligent closed captioning uses automated speech-to-text transcription. And the IBM system has spotlighting, or identification of content — such as violence or adult language — that may require further screening and labeling.
IBM Watson Media will provide a deep analysis of objects, people, and words, as well as concepts, emotional overtones, and character personalities. This deep understanding of video can help companies deliver more relevant content to consumers, provide advertisers more effective matches for their products, and help production teams find specific parts of videos. Available in September through the IBM Cloud on the IBM Marketplace, the IBM Watson Media platform will provide a packaged product that can be scaled and used across a company’s video assets.
At the U.S. Open, IBM hopes to elevate the fan experience with a Cognitive Concierge in the event app’s Guest Info feature, which will answer a variety of questions from people attending the tournament.
The app will provide information about scoring, schedules, transportation, and dining options featuring a natural language interface — meaning attendees can just type their questions and Watson will respond. Depending on the question, Watson may also plot the response on an interactive map of the venue, which will also indicate an attendee’s current location, where available.
“An afternoon or evening at the U.S. Open, especially for first time visitors, has really never been easier to enjoy,” said Kirsten Corio, USTA executive, in a statement. “Watson’s capabilities within the US Open app will help our fans plan ahead, move around the grounds at their own pace, and always know where they are and where they want to go.”
IBM is also using SlamTracker to analyze player and ball position data, which enables analysis of pace of play and proximity to the baseline, and it evaluates what kind of style a player is using in relation to past performance. IBM can look at 11 years’ worth of slam data, including matches where the same players have played before, and make predictions about the match or determine if a player has changed play styles.
John Kent, technology lead for IBM Global Sponsorships, said in an interview that the combination of human interaction and AI can create more compelling content for the audience.
“Human editors used to create the highlights,” said Kent. “That’s good, but it doesn’t scale well. If you take AI, and have Watson watch and listen to the match, you can determine which shot was the highlight shot. We can send a push notification with the best result.”
Last year, 11 million users visited IBM’s USTA digital platforms 42 million times within a two-week span.