Once you start the Knights of the Nine you can't get anymore infamy or you'll have to start it all over again.
Once I started the Nine with a infamy of 40+.But I didn't sway from the path until I was done with the Nine quest. The good thing about Nine=once your done it wipes your slate clean and start back off with 0 infamy no matter how high it was.If you already had 0 before starting it you get this great spell.