The 20-year-old played a key role for Huddersfield on loan from Chelsea last season as Wagner's side defied the odds to reach the Premier League and was expected to seal a permanent summer move to the West Yorkshire club.
But the Terriers could not reach agreement with Chelsea over a permanent switch and Brown, who joined Brighton on another season-long loan instead, will now be bidding to inflict a fifth straight defeat on his former club.
"There are no secrets, everybody knows we tried everything to keep Izzy here," Wagner said.
"Unfortunately we were not successful in signing him. The circumstances were not good, it was nobody's fault and we had to accept it.
"Chelsea are the club who had to make the decisions and it's something we respect and accept and now we meet him again.
"Izzy has not had the best start at Brighton, he was injured, but he has started the last couple of games. We hope we can keep him quiet."