Hylton took his tally in this season's competition to seven with a first-half double and a third in the 62nd minute before Lee Frecklington pulled one back from the penalty spot after 71 minutes.
Kari Arnason picked up a second yellow card after 83 minutes to kill off the Millers' fightback only for Hylton to ruin his perfect day with a straight red card a minute later.
Evans said: "It doesn't matter if you're playing a league club or non-league club, if you make bad individual errors you will get punished.
"We've got (Daniel) Nardiello committing himself to a tackle and they get across and score, then the second one there's two mistakes from Ben Pringle and Mark Bradley.
"I think second half we dominated the game but they get a goal which is two yards offside. I'm not blaming the referee but how the assistant referee does not see that, he's the only one in Hampshire who doesn't see it. But there's no accountability for the officials.
"We get a goal from the penalty, we then hit the post when we should have scored and if that had gone in we might have still been in the cup.
"As I said to the boys, you don't get league games to come back and make up for mistakes in the cup. We're taking nothing away from Aldershot, we wish them every success and we wish them luck in the draw."