= new reply since forum marked as read
Though the relatives every weekend is pretty horrid too. 
A guy at work invited all the truck drivers there to a cookout/party at his place the first summer I worked there. That sounded like fun to me. I loaded up a cooler with a twelve pack and roped my wife into coming along and being my sober driver. I do not drink and drive. If my wife had not have gone, I would have stayed home.
I like to drink beer and I like to have fun. There were about 15 drivers there, I'd say, several of them with their wives, and a dispatcher with his wife. So I'm pouring down the beers, eating pork sandwiches, and socializing. I drank 9 beers while I was there and you know what? I wasn't the only one who was keeping track of what I was drinking. I am a big guy and I can hold that kind of alcohol without getting stupid and showing my ass. I also do not drink anything if I have to go into work the next day. Indeed, if I imbibe it's usually just on Saturday night, and this party was on Saturday. Nobody had to be back to work until Monday.
I came into work on Monday morning and this dispatcher walks in and gets right up on me and starts sniffing me. I didn't put two and two together at the time. No matter what the reason was that's still very odd behavior and this guy's a jackass to begin with so I didn't think much of it. Later that week, I had some minor thing on my truck break and had to go to the company weld shop to get it fixed. So these guys are working there and gossiping and they tell me they heard some stuff about me. These guys don't work with truckers most of the time and they weren't at the party. They said they heard I drink like a fish. 
So, I'm pretty sure the whole damned company had this idea of me being a raging alcoholic. I'm surprised they didn't drug test me. I'm pretty sure the dispatcher wanted to. You'd think being a bunch of rough-ass working Joes nobody would have given two shits.
Anyway, joke's on them. I would go on for the next 2.5 years working over 50 hours a week on average and taking three classes a semester at school to finish my degree. I'll be out of there soon. You don't do that sort of thing if you spend your spare time drinking.
I like it so much, I'll use it if need be.
What I like about it is that it's both permanent yet doesn't get personal. In this case, it's about the speaker and won't wound the pride of the person asking.
Cher
Simple. Direct. And Honest.
It's not so hard. Just do it. 
The answer to your question ("Is there anyway to politely get someone to back off without hurting their feelings?"
is "not really."
I "politely" backed off from social invitation after social invitation after social invitation and eventually became known as the guy who was "too good to hang out with us."
Now, in the early days of my work career, the people I worked with were a lot like me...basically music-loving hippies...and I DID hang out with them.
But by the time I moved up the ladder a little and started wearing a tie to work, I felt exactly like you..."Work and the rest of my life is two separate realms."
Stand firm on that, politely. There's a 50-50 chance that you may get a little blowback from that, but so what? As long as you are doing your job and are not causing any problems, you will simply be known as the person who is doing their job but really doesn;t want to hang out after work.
