If my selfish acts bother you, that's acceptable. If your selfish acts bother me, that's not acceptable. The name for this is Special Pleading.
Libertarianism is built on the basic assumption that it is wrong to give an damn about anyone except yourself. The High Priestess of libertarianism, Ayn Rand, wrote a book, The Virtue of Selfishness, that argues that point. I started reading it. On the first or second page, she has a "definition" of altruism that no actual altruist would accept -- acts which benefit others are good, acts which benefit the altruist are bad. The first part of this definition is so simplistic as to be laughable -- if my good would harm others, then it is not good. The second is also simplistic, since any altruist would say that an act which benefits the altruist and does not harm others is at worst morally neutral and is probably good.
Since she began her argument with the Straw Man fallacy, I saw no reason to continue.
Her novel, The Fountainhead, says that personal pique is an acceptable reason to destroy other people's property. Of course, her fans refuse to admit this, but self-deception lies at the heart of her ideology.
Adam Smith wrote, "All for ourselves, and nothing for other people, seems, in every age of the world, to have been the vile maxim of the masters of mankind". Substitute the world "libertarians" for "the masters of mankind" and you have the libertarian ethos in a nutshell. Or, to put that in other words, the libertarian ethos is "I've got mine, screw you."