I think replacing scales is much easier. I just cut the pins of with a flush cutter. A tiny one sold for trimming solder leads was enough, the pins are soft. Probably some copper nickel alloy.

I have mill bits just like you describe for this. I’ve used them to mill aluminium before, so I’m thinking they would work well on the pins.