I've seen this done before, and I found it to be a really cool mechanic: to put it simply into words, the bonus slots of a specialization wizard should only be available for that specific school. This would make it so that specialization wizards feel slightly more like true specialists. When you see a fighter with a Greatsword specialization, they're going to be using a Greatsword all the time. But when you see a wizard with an Illusion specialization, it's not always the case that they use illusion spells so often.
To sum up my idea in a single sentence: if you prepare spells that don't belong to your specialization in say, your bonus third circle spell slot granted by the specialization, you should lose that slot.
===================================================
EXAMPLE:
if you are an Evocation Specialization wizard, and you have, including the bonus spell slot given by specializing:
Circle 1: 4 Slots
Circle 2: 3 Slots
And you prepare:
Circle 1: 4 Endure Elements
Circle 2: 3 Invisibility
You should, after resting, lose 1 slot of each spell, because the BONUS slots are not being used on Evocation school. This would result in only 3 Endure Elements and 2 Invisibility casts in total.
However, if you'd prepare, for example:
Circle 1: 3 Endure Elements, 1 Burning Hands
Circle 2: 2 Invisibility, 1 Combust
Then you don't lose any slots, and you get to use them all.
To sum up my idea in a single sentence: if you prepare spells that don't belong to your specialization in say, your bonus third circle spell slot granted by the specialization, you should lose that slot.
===================================================
EXAMPLE:
if you are an Evocation Specialization wizard, and you have, including the bonus spell slot given by specializing:
Circle 1: 4 Slots
Circle 2: 3 Slots
And you prepare:
Circle 1: 4 Endure Elements
Circle 2: 3 Invisibility
You should, after resting, lose 1 slot of each spell, because the BONUS slots are not being used on Evocation school. This would result in only 3 Endure Elements and 2 Invisibility casts in total.
However, if you'd prepare, for example:
Circle 1: 3 Endure Elements, 1 Burning Hands
Circle 2: 2 Invisibility, 1 Combust
Then you don't lose any slots, and you get to use them all.