This question is about thank you for your time.
No, saying, "Thank you for your time," is not rude. Thanking someone for taking the time to interview you for a job, give you a referral, or help you in some other way is a courteous thing to do.
Taking the time to follow up with someone to thank them in a handwritten note or an email demonstrates your gratitude and will help solidify your relationship or good impression.
Of course, you can make anything sound rude with the tone of your voice, so make sure you're genuinely thanking the person when you say, "Thank you for your time." If you have even a hint of sarcasm or passive-aggressiveness in your written or spoken voice, people will be able to pick up on it, and your thank you will do more harm than good.
When you are genuinely trying to express your thanks, being more specific in what you're thanking the person for than just their time is an even better option. For example, you could say, "Thank you for the opportunity to interview," "Thank you for your consideration," or "Thank you for taking the time to speak with me today."
If in doubt, though, you can't go wrong with the classic "Thank you for your time." It also works if the interview or meeting went horribly for some reason and you can't think of anything specific to say thank you for. Again, just make sure it's genuine so that it doesn't come across as passive-aggressive.