Defender Andy Cannon opened the scoring for the visitors with a fine curling effort from the edge of the box after 27 minutes - his first goal for the club since August 2016.
In a tight game, the hosts had efforts from Scott Wagstaff, Mark Byrne and leading scorer Tom Eaves well saved by Dale goalkeeper Josh Lillis.
Rochdale striker Ian Henderson tested Tomas Holy with a curled attempt shortly before Cannon's opener, and then fired inches wide with a chipped shot from an acute angle.
The Gills equalised on the stroke of half-time when left-back Bradley Garmston fired home from 20 yards.
Henderson's attempted lob from distance seven minutes after the break saw him shoot onto the roof of the net, while home captain Lee Martin met substitute Elliott List's deflected cross to volley in the winner for Steve Lovell's side shortly after the hour.
Former Gillingham midfielder Billy Knott volleyed over from 10 yards on his Dale debut two minutes into stoppage time as his side went home with nothing and remain second bottom.