Modifiers (generally) cost mana too, so you have to add that mana cost to each cast of the wand. If you have a 5 mana cost spell and a 30 mana modifier, you end up with 35 mana per cast. If your wand has a total capacity of 100 mana and a recharge rate of 100MPS, you'll get 2 casts with the modifier (70 total mana) and one cast without until you regen to a capacity total of 35 mana (or greater) to cast with the modifier again. If you have a dual or tri cast spell and put the modifier before that, it will apply the modifier to the whole cast block adding the modifier mana requirement to each spell in that block. If at any point that exceeds your capacity, it won't add the modifier at all.
Right, so if you have 3 spells on a shuffle wand, and 1 modifier in front of one, than the modifier will only affect the cast "block" of that one spell. Since it is a shuffle wand it can choose to cast any of those three which gives you a 30% chance of casting a modified spell. This is why non-shuffle is so important. You can always tell if its a shuffle or non-shuffle wand by whether it has a gemstone on the wand. If it does have one, it will be NON-shuffle, if it does NOT have a gem, it will be shuffle.
A non-shuffle gives you consistency and the ability to make those spell "blocks"
That's part of what makes the game tough is how you only get to tinker with the wands in those mid-way points, unless you get a perk that says otherwise. But a pro-tip, your second wand, the bomb wand, you can switch your main wand's spell to that and it will usually work in a pinch to give you a rapid fire shot with low mana regent. Not optimal, bit functional when no other options exist.
The pentagon one, IIRC, if used with a single spell will just shoot around you in 5 directions. But if you have a trigger spell you place it like so [Trigger spell] [Pentagon] [damage spell]. I believe pentagon has a pretty high mana consumption so that may have prevented you from having the total capacity needed to successfully cast it. Therefore you would have just casted an unmodified spell.