AI Agents Can't Act on a Building They Can't Read
- Last Updated: November 19, 2026
Mickey Balter
- Last Updated: November 19, 2026



At Groceryshop this year, almost every conversation I had came back to agentic AI. Agents that reorder stock. Agents that rebalance labor. Agents that catch a pricing error before a customer does. The ambition is real, and so is a lot of the software.
What struck me was where nearly every example took place: on a screen. The agent reads a dashboard, makes a decision, and updates a system. That works when the job is digital from start to finish. Most of what a store needs done is not.
A store's work happens on the floor. A shelf gets refilled. An online order gets picked. A display gets reset. A spill gets cleaned up before someone slips on it. For an agent to run any of that, its decision has to leave the software and land at a specific spot in a building. That handoff is where most agentic plans for physical retail break down.
Ask how an agent knows where things are in a store today. The honest answer is that it reads a document: a planogram (the diagram of what goes on each shelf), a floor plan, or an aisle-and-bay code in an inventory system.
Those documents are plans. They describe the store as it was meant to look on the day someone drew it, and the real store drifts from that plan every day. Products end up faced in the wrong bay. Overflow lands on a display at the end of an aisle. Seasonal resets move whole categories. A fixture gets swapped, and the bay numbering never gets updated.
Store teams absorb this drift without thinking about it. An associate who has been there ten years knows the batteries are near the registers this month, whatever the system says. An agent has no such knowledge. It acts on the description, and when the description is wrong, the agent is confidently wrong in every store at once.
This is the gap I keep coming back to. Retailers have rich data on what they sell, what it costs, and how much of it is on hand. Very few have reliable data on where any of it physically is. To software, most stores are close to unreadable.
An agent that assigns and manages physical work needs three things the building almost never provides today.
The agent needs where the product, pallet, or piece of equipment actually sits, not the aisle number on the plan. Precision matters here. Knowing an item is "in grocery" is useless to someone holding a scanner. Knowing it is within a meter of a specific bay gives them a task they can finish.
The agent has to route each task to the associate or robot best placed to handle it. In a large-format store, sending a restock to someone at the far end of the building is how a labor plan falls apart by mid-morning.
This is the part most plans skip. An agent that issues a task and never confirms the outcome is guessing. Did the picker reach the right bay? Did the reset land where the plan said it would? Location data closes that loop, and closing it is how the agent improves. Without it, the system issues a guess with every task and never finds out whether the guess was right.
All three are location questions. A better model cannot answer any of them.
Most retailers I speak with agree with this in principle. The hesitation starts when they price it.
The traditional route to a readable building runs through hardware: devices on the ceiling, gateways, power, a maintenance contract, and a rollout schedule measured in quarters per region. That can work for a flagship. It does not work for an agent strategy that has to run across a whole fleet, because an agent is only as useful as the least readable store it operates in. A pilot in ten instrumented stores tells you very little about the other two thousand.
That is why I believe the location layer has to be software-only, running on the phones and devices already in the building. When a store can become readable without anyone climbing a ladder, location stops being a capital project and becomes something you switch on alongside the rest of the stack. We have activated more than 300 stores in a single week this way. An agent rollout needs that pace.
For anyone scoping agentic AI for physical operations, I would test the plan against three questions:
The model is not the hard part of agentic AI in physical spaces. The hard part is giving it a building that can answer basic questions. Where is this? Who can reach it? Did it get done?
Retail will hit this wall first, because it runs the largest and most frequently rearranged indoor spaces in commercial use. Hospitals, campuses, and warehouses are a few years behind on the same curve. The organizations that get real returns from agents will be the ones that made their buildings readable before they handed agents the work.
Mickey Balter is Co-Founder and CEO of Oriient, an indoor positioning platform that operates entirely in software with no in-store hardware.
The Most Comprehensive IoT Newsletter for Enterprises
Showcasing the highest-quality content, resources, news, and insights from the world of the Internet of Things. Subscribe to remain informed and up-to-date.
New Podcast Episode

Related Articles