Swansea escaped relegation from the Premier League last season when Clement oversaw a remarkable finish with 13 points taken from the final five games.
But Swansea's struggles have continued this season and a last-gasp 1-0 defeat at West Ham before the international break dropped them into the relegation zone.
Only bottom-placed Crystal Palace have scored fewer than Swansea, whose goal attempts and shots on target are the lowest in the division.
But head coach Clement, speaking ahead of Saturday's home game against Huddersfield, said: "It is much too early to say we are in a relegation battle.
"When I first arrived at the club (in January) they were writing us off, and when there were five games to go and it (relegation) was between us and Hull the pundits were saying 'Swansea are down'.
"But it made no difference to us, we knew what we had to do then and we know what we have to do now.
"A couple of good results - a win and a draw or back-to-back wins - and everything looks totally different."