I have read about this, but I found my source to be somewhat confusing. If I understood it correctly, he first fought with the Toyotomi clan. Afterwards, he decided that he did not like being the embodiment of his father's quality of battle, so he broke free then joined the other side, that of the Tokugawa. I really think I misunderstand my reading, nonetheless, this is my take on the question.