The Terriers managed to battle their way through the playoffs last term and beat Reading in a penalty shootout to secure their place in the top flight.
With both teams struggling for form and desperate for points, Britton knows his Swansea side will be in for a tough afternoon when they host Huddersfield at the Liberty Stadium.
He told the club's official website: "They did really well last season to get promoted and they have shown already that they can compete at this level.
"We have to focus on producing a performance this weekend that will give us a chance of getting a result, then we can think about Leicester next week.
"We know we have to be better at home, and it’s down to us players to make that happen."