Midfielder Charlie Barnett was sent off for a challenge on Vale left-back Dan Jones in just the third minute.
But Accrington, who have now won just one of their last 14 games, were the better team until home debutants Lee Hughes and Darren Purse scored early in the second-half.
Veteran Hughes completed the scoring with a third but Richardson was focusing on the controversial red.
“From my angle it looked like Charlie had made a tackle and I didn’t think it was a free-kick, never mind a sending off,” he said. “But I’ll have to take another look at the DVD before giving a final verdict.
“It was very frustrating because to go down to 10 men so soon at a place like this makes it extremely difficult and changes everything.
“Everything we’ve worked on during the week goes out of the window and to be a man down against a very good team was always going to be tough.
“If we had kept 11 men on the pitch I would have really fancied us to get something. We put in a very solid performance to draw at Plymouth last week and I was confident we could build on that.
“We coped very well in the first-half and I can’t fault the players for the effort they put in. All of them worked very hard and kept going right to the end.
“But after Vale got their first goal it changed the game and we were really up against it.”
Millwall's unbeaten run of games stretched to eight in the Sky Bet Championship as they comfortably defeated Barnsley 2-0 at Oakwell.