If you are sometimes NOT using a stainless brush / wire wheel then the base metal most defiantly will rust. Usually stainless will rust from contamination from an outside source, e.g. carbon brush, stainless brush used on carbon, file, etc. Excessive heat input can also lower the corrosion resistance of the base metal, but I don't think you'd see the rust show up as quick as it sounds like it is showing up for you. Unless you are running this pipe in a high caustic area or something. Depending on the application welds can / are required to be passivated (or areas of the base metal) post welding (or contamination of the base metal) to mitigate corrosion issues. Avesta makes a pretty good selection of pickeling pastes to passivate stainless, but if you decide to go that route make sure you match up the correct paste with the correct base metal.
My thoughts are you are probably using a contaminated piece of gear though. The way to avoid it is to isolate what is causing the contamination. That means the SS wire brush or wire wheel you use ONLY gets used on stainless. You do not set the pipe down on carbon pipe stands without some type of poison pad on the head. The situation doesn't have to be clean room sterile, but you need to do your due diligence not to mix gear you use on carbon with gear you use on stainless. The same is true for other high metallurgy pipe like 6-7% Moly's and the like. I don't believe I have ever had the case where it wasn't pretty evident where the rust was coming from on stainless welds. General rule of thumb is, IF IT TOUCHES CARBON IT DOES NOT TOUCH THE STAINLESS! Also, the type of cleaner or marking utensils you are using can cause issues with corrosion / intergranular cracking. You ALWAYS want to use low chloride cleaners and marking utensils on stainless. (aka Don't use your sharpie on stainless pipe unless it is the LOW CHLORIDE KIND!) If you're bored the Nickel Institute has a good write up on keeping your stainless fabrication environment clean and happy.
It is usually pretty easy to tell if the stainless is rusting from using a bad brush because it rusts exactly where the brush strokes are. The look on a foreman's face when we walked around for me to buy off on the piping and he realized that one of his bargain basement fitters had decided to hit a couple dozen completed joints with a carbon wire wheel was priceless ...