Boss Paul Clement has been looking to add to his side and was closing in on the defender but it looks as though he has lost out because of geography as Yiadom would not have to upsticks if he joins the Terriers.
The right-back, who previously played for Barnet, is close to completing a sensational ascent from non-league to Premier League following the two Yorkshire clubs agreeing a deal.
A fee has been agreed, believed to be around £1.5 million, for the 25-year-old to join Huddersfield and be their 11th new addition of the summer.
Yiadom has international experience for Ghana and was called up to their squad for the Africa Cup of Nations earlier this year.
It is another set back for the Swans as they appear to have also missed out on another target in Hull City's Sam Clucas as he appears to be on his way to Burnley.