The Kiwi cut ties with Jason Hamilton after finishing 51st at the LPGA Championship following a hugely fruitful partnership since 2014.
Since turning professional in 2013, the 19-year-old has won 14 times on the LPGA Tour, but has struggled in recent months.
Her fellow New Zealander Williams made his name carrying the bags of the likes of Greg Norman, Tiger Woods and Adam Scott, and has watched her game unravel from afar.
He now believes that Ko's decision is a sign of panic and has accused her of making the decision prematurely.
"It's very surprising news," Williams told Fairfax Media. "This is the first form slump of any kind she's had as a professional, and it's only a slight form dip.
"It doesn't seem like a reason to release her caddie. Maybe they've hit the panic button a bit early."