Someone straighten me out please...

I'm polishing my HSR plenum and am totally frustrated. Here's what I have done, question is coming.

Sanded with 150/320/320 wet/400/400wet.
Got out the 6" Sisal buff wheel with Emery and polished for a few hours. Then went to a Yellow Treated Spiral Sewn wheel with Emery and polished for many more hours. It looked pretty good, almost mirror like.

I next went to a Cotton Buff Sewn with Tripoli and started scratching the hell out of my previous efforts. I have now stopped and am questioning if I have my procedural steps out of whack. I bought the wheels a while back and had good results on my valve covers. The yellow sewn feels rougher than the Cotton Buff Sewn but I cant believe the amount of tiny scratches I am now putting back into the finish. Step 3 below is way shinier than step 4 below.

Am I out of order here? It sure looks as if I'm working backwards. Thanks.

A recap:
1. Sand until you cant sand no more
2. 6" Sisal with Emery (Eastwood PN 13059)
3. 6" Yellow Sewn with Emery (Eastwood PN 13234)
4. 6" Cotton Buff Sewn with Tripoli (Eastwood PN 13034)
5. 6" Canton Flannel with White Rouge (Eastwood PN 13057)

 

Not sure if thats a typo, but you say you used the sisal wheel with tripoli and then in the recap you say you used a sewn wheel.

 

Yeah, thats a typo. I'll edit the post.
But, does that sound right, going from step three to four causing moderate scratches to my previous shine?