apparently the plan was to use humans to train an ML system but it never ended up getting good enough to take over
not to defend amzn but technically they were mostly reviewing footage after the fact and correcting mistakes the system made. the system was still responsible for your checkout.
all of this is for the same reason they bought whole foods and immediately installed similar camera systems. they were building a corpus of video tagged with the correct result so they could train/test their models. at whole foods they used check out as the source of truth and with amazon go/fresh they used manual tagging. in the algo world well tagged data is worth its weight in gold (how much do bits weigh). the still still absolutely bonkers part is that they tried for six years.
what i’ve heard through the grapevine was that the biggest issue was that the compute cost required to accurately—in realtime—track individuals picking up all sorts of objects (even tagged with rfid) was nigh impossible to bring down enough to be worth it or scalable. i had an amzn employee drunkenly mention that at one point each checkout required over 10 minutes of gpu compute per minute of an individual shopping. it might’ve been even higher—i want to say 20, but that seems too ridiculous even for amzn.
add on top of that the fact that they tuned it to really avoid false charges so it consistently didn’t charge for items and blamo, huge money sink.
no wonder they failed to sell it to other stores:
- every customer in your store costs you money
- it misses a bunch
- it was silly easy to trick
- inventory is also a nightmare
- risk management probably just said “no chance”
