This is the recipe that my hubby grew up with in Texas, and they usually added either prepared mustard or mustard powder. It gives it a little zing without adding too much sourness that you get from vinegar.
I can see the EVOO angle, but yeah, we adore the oil-cured olives for this purpose instead. And you don't have to use the Moroccan dried cured olives. A nice big meaty green olive or those lovely oily red olives also work very well. It really depends on what you want for your flavour profile. You may want to adjust your herbs, too, to complement the olives more.
I may have glossed over it quickly, but we did add a fair amount of salt into the potato water while they were boiling