Chances are it will be okay for a while. If the whole shaft is too easy to pull out, you will need to replace the pot. If it's working now, you can limp along until you have to get it replaced.
You might want to consider replacing the entire module yourself when it's time: there isn't any soldering and since you know how to take the knobs off, there won't be a command performance of recent events.
I've heard of people ripping the shafts out on the collet knobs, but never on the set screw style - that's a first.
