This question is about to whom it may concern.
No, to whom it may concern is not rude. It is the proper address to use when you're uncertain who it is you're addressing.
However, if you know the person you are addressing, using the phrase to whom it may concern is inappropriate and may be considered rude.