Righty Tighty, Lefty Loosy?
Spray some penetrating oil in the best you can, from underneath as well as at the head, though I have doubts that much will get in where you need it from the top.
I pull the spacer out the day I get the bike home, so haven't had that piece in since. I don't recall the exact fasteners, just the top plate three bolts that are re-used when re-installing it. Those are 6mm.