Particular Conjuration focus was bought from red guild merchant, but I'm p.sure it happens with all of them.
If you select a Summoning Reagent, you cannot clear that selection at all. Tomes can be cleared just fine, but reagents won't be.
To clear a summon selection you activate the focus on itself.