Q: or is this a bad idea? and why?
A: Even Sharpies will wear off over time. As you use your controller, if you are touching where you drew, the marker will end up on your hands. If you have to mod it, use a sticker or temp tatoo... you can remove those if you get tired of them, and they shouldn't rub off on your hands.