It actually wasn't a problem for me (I wasn't even aware that I had the fragments) until this January, when I started experiencing some nasty pain. It wasn't debilitating, but the doctor said it was guaranteed to end in some nasty early-age arthritis unless I got them removed, so I did. :3
Awwww, thanks :D