Striker Kaid Mohamed headed home the Robins' equaliser in the 49th minute that cancelled out Hereford's first-half strike from Delroy Facey.
Hereford forced three corners before taking the lead in the 12th minute through striker Facey. Frenchman Benoit Dalibard swept a high ball over to Facey, who headed past goalkeeper Scott Brown from close range.
Midfielder Nicky Featherstone went close to putting Hereford further ahead four minutes later, but his low shot was cleared off the line by Luke Garbutt. Winger Joe Colbeck kept the pressure on the Robins with a dipping cross from the right but Tom Barkhuizen failed to connect on the left flank.
Cheltenham rallied, however, to level four minutes into the second half when Mohamed nodded home a cross from Jermaine McGlashan into the right hand corner of the net.
Teenager Luke Taylor went close to restoring Hereford's lead but Brown pushed his shot around the post. But Cheltenham stepped up a gear, and only a last-ditch clearance by Ben Purkiss stopped Mohamed snatching a winner in the 80th minute.