I'm not sure if you have bronze or aluminum sidedots, or of the age of your instrument. Bronze dots do oxidize kind of a dark greenish color, and clean up with a light sanding or scotch brite. There's no finish on the edge of the board, so it's an easy bit to perform. Aluminum oxidizes a blackish color that can be usually be wiped away with your fingers.
The sidedots are glued in place with cyanoacrylate, which cures hard. They are in deep, about a quarter of an inch, so there's a good surface area for the glue to hold in place. I've never heard of one falling out, or of adhesive leaking.
The only rubbery thing I can think of is perhaps buffing compound, which would wipe off as you described. What color was the rubbery substance?