It seems fighting some of the gladiators makes certain factions hostile. I'll eventually add a list of the possible culprits.
"The Pitter Priest"
"Jehan of the Marquis"
"Walter the Dweebish"
"One Shot Lucky Chop Lucy"
"Crimmard, Underbutler of Sunpurse"
"Torgund the Skald"