The yakuza rule the streets.
If anybody is stupid enough to get in the hairs of the yakuza in Osaka, he just has to get what's coming to him.
Gaijins are way too quick to pick a fight in Japan. And there is a perception (perhaps justified) that all Westerners think Japanese are pussies. So there are plenty of yakuza out there who are itching to prove them wrong. This yearning is so strong that, in many places, the cops can't handle it.
In the more violent places in Japan, namely Osaka, Kagoshima, Kochi and Kumamoto (for starters) do not get in a fight unless you are absolutely sure you can win six to one. (In my experience, Fukuoka is pretty violent but the cops are tough enough to handle them.)
The good news is that the yakuza are there to maintain order (or some semblance thereof). So as long as you don't excessively stand up to your "rights" (which in most cases are privileges) you won't be harmed. The bad news is, once they start beating you up, they won't stop when they should.