'GrokNet', the AI behind Facebook Shops, looks for body type, skin tone, location, socioeconomic class in photos

• Yay, Clearview AI but for shopping!

Facebook Mark Zuckerberg today announced the launch of Facebook Shops, an e-commerce feature to allows business users to list and sell products on Facebook and Instagram.

Here’s the Facebook blog post launching the new shop feature.

Here’s more at CNBC about Facebook Shops.

Separately, Kyle Wiggers at VentureBeat reports on some of the very creepy personal details examined for the shopping experience by Facebook’s artificial intelligence:

Facebook says its AI-powered shopping systems segment, detect, and classify images to know where products appear and deliver shopping suggestions. One of those systems — GrokNet — was trained on seven data sets containing images of products that millions of users post, buy, and sell in dozens of categories, ranging from SUVs to stiletto heels to side tables. Another creates 3D views from 2D videos of products, even those obscured by dim or overly bright lighting, while a third spotlights apparel like scarfs, ties, and more that might be partially obscured by their surroundings.

Facebook says that GrokNet, which can detect exact, similar (via related attributes), and co-occurring products across billions of photos, performs searches and filtering on Marketplace at least twice as accurately than the algorithm it replaced. For instance, it’s able to identify 90% of home and garden listings compared with Facebook’s text-based attribution systems, which can only identify 33%. In addition to generating tags for colors and materials from images before Marketplace sellers list an item, as part of a limited test, it’s tagging products on Facebook Pages when Page admins upload a photo.

In the course of training GrokNet, Facebook says it used real-world seller photos with “challenging” angles along with catalog-style spreads. To make it as inclusive as possible for all countries, languages, ages, sizes, and cultures, it sampled examples of different body types, skin tones, locations, socioeconomic classes, ages, and poses.

Rather than manually annotate each image with product identifiers, which would have taken ages — there are 3 million possible identifiers — Facebook developed a technique to automatically generate additional identifiers using GrokNet as a feedback loop. Leveraging an object detector, the approach identifies boxes in images surrounding likely products, after which it matches the boxes against a list of known products to keep matches within a similarity threshold. The resulting matches are added to the training set.

Well, that’s not creepy at all.

Read more at venturebeat.com:
Facebook details the AI behind its shopping experiences
[Kyle Wiggers, May 19, 2020, via techmeme.com]