Goals from Harry Kane and Andy Keogh fired the Lions to a first home victory in 2012 and ensured the Tigers dropped to 10th place, six points adrift of the top six.
"We've given ourselves a mountain to climb," Barmby said. "But I won't tolerate anyone throwing in the towel. We're in a rut, but we need to roll up our sleeves and get a result and a performance worthy of Hull City.
"It was frustrating, we had most of the game but didn't create enough. I've got no complaints with the result and I'm not particularly happy about our performance.
"Obviously we knew the day it was and the respect the Millwall legend had, but it's always difficult to come here and we didn't get to grips with it."