Brown impressed while on loan in West Yorkshire and he helped the Terriers earn promotion to the Premier League for the first time.
He scored five goals after moving down to the Championship before moving back to parent club Chelsea in the summer.
Seagulls boss Chris Hughton soon snapped him up but Brown has had to bide his time for a first-team place.
He played in the defeat to Liverpool on Saturday and the 20-year-old is excited to see some familiar faces this weekend when they take on Town.
He told the club's official website: "That’s a big one for me from a personal perspective, and the team as well.
"The fans still think fondly of me, but my focus is here now – I had a good half of a season there and the promotion at Wembley was a great day for me and my family.
"We know it’s big because of our tough matches over recent weeks, but we’ll be going there looking to win and get the three points. We’ve been in the top half for a while now which surpassed expectations and now we want to get back there as soon as we can."