Tries from Stuart Hogg and Huw Jones plus a brace from Stuart McInally put the Scots firmly in control on Saturday.
However, Townsend's troops then conceded four second-half tries at Murrayfield and an upset appeared a real possibility.
Scotland eventually held on to win 44-38 but they welcome the All Blacks to Edinburgh next weekend and Townsend admits his team need to improve their all-round game.
"We have got to make sure we don't give up possession in our half as cheaply and defend much better," he said.
"Credit to Samoa, they played really good rugby. The first 15 minutes, they kept a lot of ball, they attacked either through the forwards or out wide, and when we gave them opportunities off our restart or errors within our half, they made the most of them.
"They took their chances and their forwards were strong carriers.
"We know how great a side [the All Blacks] are in attack and defence, so this first game is history now, we can move on to things we need to be better at next week to play at our best against the All Blacks."