There is simply no excuse for the existence of spatter. And I enforce its removal in virtually every application of which I influence. Having said that it is predominantly for aesthetic reasons and professionalism, though the exceptions posted duly noted.
However, I would like to argue perhaps to stimulate, that if spatter can cause untempered martensite, it can just as easily cause acicular ferrite which can actually increase toughness. Martensite results from higher alloy content and rapid cooling rates. Acicular ferrite results from lesser alloy content (often microalloying) and rapid cooling rates. Not to mention that a rapid cooling rate (rapid enough to be considered quench cooling) can reduce the grain size of a large grain material which can improve toughness as well. The very reason that HAZ's in A36 for example can demonstrate better toughness than the unnaffected base metal. Those who have dealt with ASME Section III are very familiar with this phenomena.
As for stainless steel I might argue that if the corrosive medium is on the inside and the spatter is on the outside you are again dealing simply with professionalism and aesthetics.
Bottom line, know your alloy.