The Shrimpers have not won away from home since October and goals from Liam Trotter and Tom Soares condemned Brown’s side to a fifth league defeat in six matches.
It was a chastening afternoon for the visitors, who were second best for most of the contest, and a furious Brown believes his players have let the whole club down.
“I spoke to them after the game, pointed fingers at one or two people. They don’t like it but at the end of the day it doesn’t matter,” Brown said.
“This is about people’s livelihoods, about me and my backroom staff keeping our jobs, that’s the important part. When a player crosses the white line, if he plays poorly he doesn’t lose his job. But there are people in my backroom staff, and me, who are fearing that.
“They defended properly and we didn’t. Two poor goals we conceded, and it took until the second goal for us to play how we wanted to play.
“What I’d give for a clean sheet now is everything. And I mean everything. If I get a clean sheet, you start off with a point, it’s a foundation, something to build on. But as a group we don’t look like keeping a clean sheet because of the mentality.
“We’ve got great travelling support week in week out, no matter where we are, and we’re letting them down, it’s as simple as that. Me, my backroom staff, the players, we’re letting everybody down as a football club.”
Wimbledon boss Neal Ardley paid tribute to his side for putting in the hard yards over the Christmas period.
The Dons have taken seven points from their last four matches, leaving them a point behind 20th-placed Oldham.
“These boys have hardly had a day off over Christmas, they’ve had no Christmas,” Ardley said. “They’ve gone hard and given it everything, and seen it through, and I can’t ask anything more than that.
“We had a target, we wanted two wins and six points out of the Christmas period. Wins are important when you’re down there, so we’ve tried to win every game, not just look for a point and to get seven is fantastic.
“We changed our shape compared to what we have been lately, we mixed everything up, we had a gameplan and I thought for the first 35 minutes we executed it perfectly.
“Then, just for 10 minutes, we got a little bit carried away and sloppy, but second half we came out and did what we had to do.”
Burton midfielder Martin Samuelsen will use the international break to recover from illness after pulling out of Norway U21's games.
Jonas Lossl is staying 'optimistic' over Huddersfield's chances of survival this season and has urged his team-mates to 'go for it'.