[squirrel_hunter]

A picture would confirm the bolt, however I think it this sound like an Alternator Cover bolt. If it turns but does not tighten then I would suggest that the thread has been stripped. Take the bolt out and see if the thread on the bolt is intact or if you have some pieces of metal on it. I would imagine its taken the thread from the engine crank case which is why its not tightening, why it is not sealing the cover to the case, why the gasket is not sealing, and why your engine is leaking oil.

My suggested fix would be to fit a new thread, be it Helicoil or other means. If this were on my bike I would have to take it to a dealer to remedy.