One mechanic to fix RPG spellcasting

(Or, well... it's an untested work-in-progress idea for fixing the things that usually irk me about it, personally.)

Magic has to be constrained, obviously, to avoid just having an instant win button that any wizard can press at any time.

Fantasy game designers usually accomplish this by giving spellcasting certain factors:

• Limited uses (in the form of spell slots, power points, consumable resources, etc). Book-keeping, yuk. You also need to stretch to find a good in-world reason for the limits to exist and the character to be aware of them, so immersion doesn't break.
• Spells of different fixed power level. Usually these have to be unlocked. If only one high-level spell has some effect, then you'll never have a lesser magician who has a knack for that one spell, or can even use it. Different fixed power levels pairs with limited uses to have several consequences:
• Even more book-keeping (bleh).
• Spells often just succeed, or have a single opportunity to resist, because it sucks to have a valuable resource just do nothing.
  
• Strategic choices usually become obvious (and therefore boring). You might straight up stop using your weakest spells.
  
• If spells have different power levels and are only rate-limited then a spell can be exactly as powerful as you can bear if a wizard wants to cast it their allotted N times per day, every day, including in downtime. Get it wrong and some spells are too weak to bother with; others break the game. Usually both, and it gets worse the wider the range of power levels.
• Jarring styles of play. The wizards have expendable resources and an incentive to use them in overwhelming bursts and/or to solve every problem, as much as possible, and then switch to 'spell recovery mode' (usually rest). This doesn't apply to other characters (or it gets awkwardly levered in by giving them inexplicably rate-limited mundane powers).
  
• Quick casting. It's bad enough in D&D-likes, where most spells take six seconds to cast. That's on the shorter side of what I'd prefer. In more granular games like GURPS, they're often cast even faster.
  
• Not much ritual. If there's a big slow evil magical spell happening, it's almost always a plot point rather than a mechanically underpinned option, let alone the primary method of spellcasting. In a lot of the genre fiction, spells are cast over long periods, either building and building in power/effect, or building towards a moment where they take effect! I really like the feeling of this, and almost never see it in a RPG.
  

I think these four aspects are worth changing for their own sake. It's worth noting that they also lead to a 'best practise' which I dislike:

Bum-rush the wizard

The RPG spellcaster in combat is unarmoured or otherwise fragile, and makes up for this with a limited number of extra-powerful hits which they deliver quickly and at range without cost, and can usually snipe a specific target and/or affect swathes of foes at once.

That adds up to a nearly universally optimal strategy: bum-rush the enemy wizard. Expend resources or open yourself to risk in order to take out the spellcaster early; it'll be worth it.

This is made worse by the usual mechanical parity of NPC/PC spellcasters (a satisfying and useful part of most games which I'd like to keep). The strategy is also optimal when it's creatures fighting a PC party with spellcasters. Enemies either bum-rush the PC wizard or are complete idiots. The best you can do is switch up those two options, sometimes making it unpleasant for the PC wizard player and sometimes boring the whole group with stupid enemies.

Here's my solution so far.

 

One unified mechanic which solves all my problems

A spell has fairly minor base effects which always apply, and a buffet of advanced effects.

If you're not already casting a spell, you can always try to cast any spell you know. You must continue casting it if you want the effect to continue (no check) or build (requires check).

When you cast a spell, its base effects take place. When you continue casting a spell, you either maintain it, or you try to make it more powerful by adding an advanced effect. You can do this each round, for most reasonable definitions of 'round'. Each time you want to add an advanced effect to the spell you are casting, you must make an appropriate check, which becomes more difficult with each effect added.

If you spend a round winding down a spell you're casting to end it, there's no risk at all. If you choose to stop casting it abruptly, there's a small risk. If you're forced to stop casting by some interruption or injury, there's a moderate risk. And if you bite off more than you can chew with advanced effects and fail the spellcasting check, there's a high risk.

The more risk, the worse the potential outcomes when the spell ends, and the higher the weighting towards the worse outcomes. The spell might just fizzle or have some cosmetic effect, but then again it might lash out at or turn on the caster, or run horribly rampant. These effects could be generic, tuned to the spell, or both.

The more skilled at wizardry you are, the more likely you are to succeed on your checks, and therefore reduce the risks you face by casting a more sophisticated spell.

Examples

The mage's change spell only makes her target look like a frog, at first. As she adds more and more advanced effects, they become the size of a frog, and then become a frog, and then start thinking like a frog. By adding other advanced effects, she can make the spell last – first an hour, then a week, then permanently – even after she stops casting it.

The wizard raises a wall from the ground at the heel of his staff. As he walks, it follows in his tracks. He puts more effort in, and it changes from packed earth to sandstone, and then to granite. If he risks enough advanced effects, he can make it higher and thicker, or join it up and add features to turn it into a building, or make it last well beyond the time he spends casting. A skilled wizard with patience and a tolerance for repeated small risks could make a whole stone fortress, a little section at a time, and only have to renew it (say) once per year.

The warlock starts casting a spell to call lightning down on the enemy camp. Clouds gather, roil, and darken as she casts. If she releases it immediately, it will have no effect, but as she continues casting, an ominous glow signals more and more charge is gathering. The choice of when to actually bring the lightning down is a calculated risk. Wait too long, and it might lash out at her!

The evil cultist starts conjuring blood eels directly from the bodies of the victims. Over time, more and more of them exsanguinate, and the blood eels grow larger, more numerous, more self-directed, and roam further from the caster. If he doesn't want to push his luck, he can then maintain his swarm of eels for as long as he keeps casting.

Outcomes

So how about those desiderata from the start of this post.

Limited uses? Nope.

Spells of different fixed power level? Gone.

Quick casting? Not if you want to accomplish much!

Not much ritual? Effects build as the chanting continues, which is the key part; see also Possible Extensions, below.

The "bum-rush the wizard" strategy? Still useful, for PCs and NPCs alike, but not a no-brainer. When the wizard starts casting, he's not particularly dangerous. If you let him keep on casting, he eventually becomes very dangerous indeed ...unless he overreaches and destroys himself! We have arrived at "actual decision-making", the backbone of TTRPGs!

Corollaries:

• Casting a spell is a risk, so spells can be powerful and yet not rate-limited. Just play out the spell each time, to see whether something goes horribly wrong. The more powerful the effects you're invoking (and thus the more potential to 'break the game' when done over and over), the more likely you are to suffer catastrophe.
• Note that this requires negative outcomes to have actual major impacts, not just be slaps on the wrist. Otherwise you end up with a long, slow cycle of spend-resources-to-make-resources-to-fix-the-harm-which-occurred-while-making-free-resources-etc. It's going to work best in games where healing is neither easy nor quick, and where character death is a possibility.
• Character improvement for magicians is easy. You can take whatever approach you want with learning/acquiring spells or specific advanced effects, specialising in spells, etc. The main improvement is just in spellcasting skill: a better caster can reliably get more spell effects before something horrible happens. This is skill-system-independent, with one exception: casters shouldn't get so powerful that they can do magic without risk (except possibly for the weak base effects of the spell, which might be safely freely repeatable if that won't damage the setting).
• Certain styles of spell, with quickly-achievable instant fixed effects, wouldn't work very well with this. Your words of command, snap featherfalls, reflexive shields, wizard knocks/locks cast while fleeing, etc. But I'm confident that every interesting spell can be made more interesting with effects that build and layer.
• I expect this to work very well with progressive status conditions, Pathfinder-style. Binary save-or-suck is often a problem with magic systems. "Spells get more powerful as the casting continues" fixed that pretty directly.
  

There are games that have elements of this mechanic – the GLOG's wizards, which I like; games with spells you pump up over time like GURPS Dungeon Fantasy; that one Conan game with open-ended / modular casting; any game with spell mishaps. I haven't seen anything exactly like this, but if I'm reinventing the wheel, please let me know.

Possible Extensions

Off the cuff, ideas for advanced rules:

1. Capacity for the caster to change the spell they're casting on the fly by relinquishing specific advanced effects they added previously, lowering the check difficulty.
2. Extensions for counterspelling / duels of wizardry.
3. In some styles of game, advanced effects might be improvised rather than listed for the player.
4. Rituals involving long timespans and more casters: Increase the time requirement for each 'stage' of the spell, multiplying by some factor or by the number of casters. Get a benefit like reduced risk or easier checks for adding advanced effects.
5. Maybe everyone can do magic; the specialised wizards are just the ones who are good enough to do it with advanced effects and lower risk.
   
6. If you really wanted, you could add complexity back in by giving each advanced spell effect a numeric power level, and a corresponding increase to check difficulty.