Reviving the village witch (Or: Worlds Where The State Can't Hoard All The Magic)

Here's another extra-long post, following on from the recent one (https://periaptgames.blogspot.com/2023/07/how-is-there-village-witch-or-states.html) which laid out in eyeball-blistering detail the following argument.

• When something is available, usable, useful, and repeatable, that's a technology, and people will use it to make their lives better.
• Pre-industrial states try to control the levers of power (organised violence, useful military technologies, useful people, wealth, etc). None are perfect at this but the ones that are bad at it will have a competitive disadvantage compared to the ones that are good at it (and inter-state competition is the default state of the world).
• Historical states use positive incentives, coercion, and cultural programming - or Benefits, Blackmail, and Brainwashing (BBBs) - to control the human component of the levers of power.
• In classic fantasy, magic-users have somewhat predictable/consistent magic, and notable benefits to using it in at least some circumstances; therefore magic is a useful technology. Usually it's a very useful technology with major military, economic, and administrative applications.
• We generally prefer TTRPG fantasy settings with the following elements: verisimilitude; magic is a modestly scarce resource; the state doesn't control all (or even most) magic; magic-users themselves don't (necessarily) run everything.
• The premises seem to preclude the preferred setting details.
• States poured enormous amounts of investment into useful military technologies like 'mounted knights' or 'foreign mercenaries', which you needed a bunch of to help win a battle. If one magic-user can turn the tide of a battle via a localised rain shower, a conjured barrier, better military signals, subverting the enemy command, flying over a castle's walls, or whatever, then the state's best strategy is to switch from training men-at-arms to training mages even if the cost of that is very high. Analogous arguments apply for economic use of magic, medical use of magic, infrastructure-developing use of magic, etc.
• If magic-users are powerful enough to avoid being compelled by the state's various methods, then almost by definition, magic-users are powerful enough to become the state, and that being the norm is not necessarily what we want in a fantasy milieu.
• If they're not powerful enough to avoid the state, then we have these weird worlds where each village witch, reclusive warlock, or adventuring wizard represents a failure of the state to carry through with what should be one of its main objectives. The same applies to magic-users specialising in kinds of magic that aren't useful for the state, magic-using outlaws (implicitly threatening the state), magical training not regulated by the state, etc. Again, not what we usually want in a fantasy milieu.

I wanted to add a couple of things after writing the original post, based on the idea that we're mainly talking about TTRPG settings.

THING ONE

If you think this isn't a problem because you model people in the past as doing certain things differently, or you fundamentally disagree with my characterisation of pre-industrial states, remember that TTRPGs are played by modern people. If the NPC rulers for some reason aren't amassing magical power and control over magic-users, the players are likely to notice there's an unexploited power vacuum and start trying it, at least in games where their characters can reach that level of power/influence. What are you going to tell them? 'No, actually, your character doesn't have the idea to do that thing which will significantly further their interests'?

THING TWO

Something I didn't specifically call out is that it just feels 'off' for magic to be exploitable in the way that mundane technologies are, for most fantasy settings. It feels like the argument that Superman can do maximal good by spending all his time forcing heads of state to pass better laws for the good of humanity, or by generating clean electricity for the world. It's inappropriate to the tenor of the main source texts.

But once something is useful and repeatable, it seems inevitable that people will start treating it as a technology, even if it is originally spooky and mysterious. They'll start looking at their records and doing calculations like: "I usually make ten florins on the days that I prepare all plant growth and weather spells, and I usually only make five or six florins on the days that I also prepare levitation and transport spells. Preparing fireball spells has never paid off for me. I should look into specialising further in spells for improving crops." That kind of thinking is just how people work; it's not a modern thing.

This is all the more obvious in a setting running on TTRPG-friendly rules, where magic comes pre-systematised (so it can easily be resolved at the table) and is seldom really dangerous for its wielder (so players can be wizards). The problem of retaining a sense of the mysterious/fantastic in magic for TTRPG play is a separate one and demands a series of posts of its own.

ENOUGH OF ALL THAT

On to the possible solutions! I thought of a dozen of them, although some are variations of one idea. Thank you to the various commenters and conversationalists who helped sculpt my thinking on them.

1. THE MOST OBVIOUS SOLUTION: THERE'S A SWEET SPOT

Take the question 'How powerful and useful is magic?' and the question 'How likely is it that a successful state spends resources trying to maximise control of magic?'. The former implies a continuum of possibilities which fairly directly answer the latter.

At one end of the continuum, we have the answer 'Magic is not powerful or useful at all'. The answer to the second question is 'no successful states wastes resources maximising control of magic'. States will only care about magic to the degree that pre-industrial states are largely indistinguishable from their individual rulers, and some of those rulers may be curious about this little meaningless part of the world.

At the other end of the continuum, we have the answer 'Magic is more powerful and useful than any other technology / instrument of control'. Now the answer to the second question is 'all successful states put everything they have into maximising their control of magic, vying for control over the literal only way for them to maintain themselves'. This is the extreme case where, say, a wizard can snap her fingers and remove people from existence halfway across the world.

In the vast middle of the continuum, it seems axiomatic that the more powerful and useful magic is, the more important it is to the state.

The solution, then, is to say that there's a 'sweet spot' in the continuum, a point at which magic is powerful and useful enough to be worthwhile learning for some people, but not powerful and useful enough for the state to invest much in it. That is, it looks like this:



And not like this:



Note that as the continuum progresses to the right, we expect the state to become increasingly desperate for magic, and then at some point far enough along, instead of 'the state' there is 'the magocracy' (and then possibly just 'anarchic groups of loosely affiliated all-powerful wizards' all the way at the right end). But in the first diagram, there's a point where magic is powerful enough to serve some use in individual people's lives, but it's challenging enough for the state to get hold of that the payoff isn't always worth the effort.

I think most of the people who were unconvinced by my original post will have already had something like this solution in their heads: they may conceive of that 'sweet spot' as being very large indeed. If that's the case, the implication is only that some states will seek to control some magical power instead of other kinds of power, which matches what we usually want in a fantasy setting.

Verdict:

Your mileage may vary. To me personally it still indicates a magic which is far weaker / costlier / less useful / less exploitable than we usually see in TTRPG magic systems.

2. VARIOUS BAD SOLUTIONS (ABANDON SOME OF THE ARGUMENT'S AXIOMS)

It's possible to dissolve the problem by getting really weird with your fantasy. Any of the following will do the trick:

• "People are different to people in our world: the nice, altruistic ones outnumber the ruthless ones so overwhelmingly that the latter never amass power (and we're willing to apply this to player characters too)."
• "People are different to people in our world: they are completely irrational and are unable to notice and adopt useful technologies (and we're willing to apply this to player characters too)."
• "People can't do magic (even player characters)."
• "All the effects of magic are completely unpredictable. Maybe sometimes people do magic just to see what will happen but it's an absolute roll of the dice every time, and not even in an exploitable way where the state can radicalise magic-users and have them creep into the enemy camp and create a bunch of random spell effects as a kamikaze attack."
• "Magic can only accomplish the same things as mundane feats, and it's always the same cost (or costlier) on every axis as doing things the mundane way."
• "States are just different to the states we're familiar with from pre-modern history. There's no monopoly on violence, or they aren't mostly just individually powerful wealthy people, or they don't maximise making things nicer for themselves. And if a state like that arises, it doesn't outcompete other states even though it should."
• "The world has no scarcity of magic, actually. Everyone can be a magic-user if they want to."
• "The world doesn't have to make sense, and for things like this it doesn't."

Verdict:

Yes, these solve the problem, but not really for the kinds of fantasy worlds that we're most interested in (except as intellectual exercises! I'd actually love to see some of those written up - I just wouldn't want to run a game in them). e.g., if you replace people with 'people' who don't work like the way we know people work, then your world is going to feel like science fiction with fantasy trappings.

I do want to call out that last one in the list, "The world doesn't have to make sense", as an odd duck. It's certainly possible to just not have a self-consistent, understandable, plausible world, and for a Friday night chips-and-beers tabletop game, it might be completely fine. I don't want to yuck anyone's yum, here. Personally, I find it unworkable, for settings that players are going to spend a lot of time in (especially if you are lucky enough to have clever, curious, really engaged players). So as solutions goes, it amounts to "I don't think this will be a problem for me and my players". And that's fine if it's true.

3. RUINOUS COSTS: DANGER, POLLUTION, DISFIGUREMENT

Now this is an interesting attempt at a solution! For certain game systems, especially OSR ones like GLOG but also some others like (some classes in) WFRP, magic is a troubling, dangerous, unstable power. The implication is that magic can be so dangerous or disfiguring to the caster, or ruinous or polluting to the surrounding area, that it's not worth the state getting involved with it. Usually it's a risk (chance of catastrophe) rather than a cost (the following nasty things consequently happen), although the latter is possible.

I think this is a variation of the 'sweet spot' solution that is actually a really difficult needle to thread. It's just saying that magic has both upsides and downsides. If it didn't have any upsides, then we're at the 'nobody uses magic / unsuitable for TTRPG' end of the spectrum again. If magic has meagre upsides and massive downsides, then it will only be used in times of desperation or not at all, unless the magic-user is deranged. By the time we're talking about magic having useful upsides (coupled with those downsides), then we're in the realm of the state wanting the upsides and finding workarounds for the downsides.

And I think the default position is that the state will find workarounds. For a start, it's possible (albeit more difficult) to exert control over people at a distance. The state can get control of (and direct the labour of) magic-users at arm's length, via proxies and patsies and hierarchies, so that those who bear the brunt of the downsides are the magic-user and the people near them who the state doesn't care about. To put it another way, if every magician is dangerous in the same way a suicide bomber is dangerous, that doesn't actually change the state's military interest in sending bombs against its enemies. It just changes the methodology.

In practice I think that in a well-thought-out world the 'ruinous danger' angle might have the opposite effect from the one we want, actually increasing the ratio of magic in the hands of the state compared to 'private magic-users'. There are two reasons:

1. The state can use BBBs to get control over magic-users and then require them only use their magic when directed. Assuming this kind of leash makes a mage less likely to cast a spell and therefore less dangerous to themselves, then it's trivially true that magic-users outside of state control will destroy themselves at a higher rate than magic-users under state control. We could even speculate that if this is publicly visible, magic-users might desire to come under the control of the state under the false impression that 'official mages' always suffer fewer catastrophes.
2. If it's at all possible to combat, ameliorate, or prevent the 'ruinous danger' in whatever form it takes, then it is the people in power who can amass the resources to do so. Imagine the case of the private country mage accidentally summoning up a horde of fire imps. His chance of survival is going to be lower than that of the mage in the king's service who does her magic in a flooded, lead-lined, well-mortared stone room with armed guards and a priest on standby (having read the book of 'best practices' that the state has slowly developed through trial and error).

Verdict:

It's hard to make this workable on its own; I think it works best in tandem with other solutions. It's a flavourful extra reason for states to usually look at magic and say "yeah, let's just stick to having the one court magician for emergencies".

4. RUINOUS COSTS, VARIANT (i): CRITICAL MASS

One notable variant of 'magic is too dangerous' is that having too many magic-users in close proximity is automatically dangerous in some way (e.g., rifts in reality). This leads to a natural distribution of wizards over small towns and lonely towers, and we can imagine it sparking local modulating action (persecution, execution, exile, etc) if there get to be too many in one area.

Verdict:

I like this variant of 'ruinous costs' a lot.

• It makes it somewhat harder for states to acquire magical power.
• It makes it a lot harder for the state to consolidate magical power geographically, which is pretty key in settings without mass transit, portals, or such.
• It makes it impossible for the state to replace armies with massed magic-users, which was one of the big concerns

But I think it's still best used in conjunction with other solutions, otherwise it doesn't stop the state being the main employer of wizards.

5. RUINOUS COSTS, VARIANT (ii): METAPHYSICAL CHAIN REACTIONS

The main problem with 'magic is ruinously dangerous' is that the people in charge can find ways to avoid the brunt of the danger. This solution asks: what if that's not true?

Magic is disfiguring, polluting, potentially catastrophic, commits you to the bad afterlife, or whatever, and the negative consequences can and do reach up the chain of command!

Let's say a magic-user (and by this metaphysical transitive property, their liege lord) has a 1/1000 chance of imploding per spell cast. That's fine for individual mages using spells rarely, and works well with magic-users being superstitious, ignorant, risk-takers, etc. But if the king is in the habit of sending out brigades of 100 mages to lay siege to castles, and they each cast a couple of spells in the process, then it's not very many castles captured before the king's odds of survival aren't looking so good. The outcome doesn't have to be self-implosion, but sufficiently nasty consequences make for a big disincentive for rulers to take advantage of magic.

It does raise questions for the GM or game designer about what it means for something to be done at someone's command, i.e., what's sufficient for ruinous consequences to get passed up the chain.

• What if the queen gives the order to her seneschal who gives the order to his commander who gives the order to her mage soldiers?
• What if the king pays a proxy 100 florins to "do what's best for the kingdom" and that proxy pays 10 florins each to nine suspected mages to "do what's best for the kingdom" and never follows up?
• What if the emperor is known to often give enormous rewards to wizards who do things that the state likes, such as assassinate imperial enemies and perform medical miracles and make ships safe at sea, but the emperor never promises anything and sometimes delays the reward or never gives it and always refuses to reward a wizard who approaches expectantly?
• What if there's a 'master of mages' position in the royal court with a high turnover rate, and the position operates completely autonomously?

Verdict:

This solution works (once you've delineated the mechanical consequences), but I personally find it unsatisfying/contrived without a really good in-setting metaphysical reason for why the spooky action at a distance happens.

One outcome that I do like is that this variant of 'ruinous costs' doesn't strictly remove the incentive to have a magic-using servant or two, and they don't have to be kept at arm's reach. It makes some sense for, say, a risk-taking monarch to have one seldom-consulted court magician, and that's a useful conceit.

6. MAGIC STAYS AWAY FROM THE STATE

This is like the last solution, except that instead of negative consequences, magic just stops working properly when used to benefit powerful people.

One way to implement this would be to say that spells which are used on behalf/command of someone who has amassed power (in terms of wealth, soldiers, legal protections, other kinds of magic, etc) become weak in proportion to the amount of power amassed. In practice, adjusting magical effects on a continuum sounds like a headache.

Another way would be to say that when a magic-user becomes powerful enough, or serves (or agrees to serve) a powerful enough person, all the magic leaves their body forever (and maybe goes and turns some random commoner or newborn into a mage). This is very direct and to the point, but has enormous setting implications, and isn't well-suited to TTRPG play. It could also create a new kind of caste hierarchy amongst people not quite powerful enough to trigger the 'magic escapes' outcome, but wealthy enough to pay for magic. And you need that clause that prevents magic-users themselves becoming too powerful, otherwise you risk a constant cycle of magic-users overthrowing the state, becoming the state, and losing their power.

In both cases the GM or game designer would also have to make a lot of decisions about what different kinds of power are 'worth', how much an individual really 'has', to what degree direct and indirect control of things matters, etc.

I'd still want a good in-universe reason for this being the way the metaphysics works.

Verdict:

This has all of the problems of 'magic is ruinously dangerous to state control' with few of the upsides. So, pass. Except...

7. MAGIC STAYS AWAY FROM THE STATE, VARIANT (i): LESS METAPHYSICS

In this variant, the affordances of the magic system - the crunchy bits that people interact with, like spells and magic items - are themselves unusually oriented towards escaping state control.

That is, there is an inherent, self-perpetuating disparity, based on the nature of magic itself, between powerful people's ability to acquire more magic and magic-users' ability to escape being acquired.

I can imagine a magic system that's chock-full of readily available effects along the lines of 'hide from authority', 'become rational', 'teleport to home', 'escape notice', 'remove brainwashing', 'change focus of target's attention', etc. It would need to have no effects like 'find mage', 'subjugate person', 'detect magic', 'persuade other', etc (or makes those effects unusually high-tier), because those could be used to enhance BBBs. With this kind of magic system, the state would still want control over magic, but would be limited in its ability to use coercion and indoctrination to do it, leaving it with just positive incentives.

A stronger way to implement this would be to say that when a person learns to use magic, part of the process is becoming psychologically or emotionally affected by it, in a way that they are made extremely averse to authority and powerful people, always resist coercion, are paranoid of positive incentives, etc. This is more impactful, but is tricky, because it has major roleplaying implications and you'd have to be careful to make it sufficiently difficult that the state can't find clever workarounds (getting control of magic through proxies, regulation, reverse psychology, market forces, other kinds of magic that override the mental effect, etc).

Verdict:

This does put additional constraints on worldbuilding and/or roleplaying, precluding certain kinds of fantasy setting. Apart from that, seems workable.

7. MAGIC STAYS AWAY FROM THE STATE, VARIANT (ii): MORE METAPHYSICS

Suppose magic is sophisticated and autonomous enough to supervise its own use according to some coherent ideology/aesthetic. Or equivalently, magic is directly overseen by super-powerful force(s) with a particular ideology/aesthetic.

And suppose that this ideology/aesthetic amounts to, basically, "shape the world into the kind of classic fantasy world that we like".

Let's set aside the meta-fictional interpretation of this metaphysics where the force accomplishing this is the GM. The obvious textual interpretation is that this fantasy world has interventionist gods or godlike entities, and those omnipotent meddlers hate it when states achieve huge consolidated control of spellcasters, magical armies, magical production lines, and so on. The tastes of these entities run to hedge wizards and travelling thaumaturges, village witches and hermit warlocks, heirloom magic and dabbling thaumaturgy. So that's what there is in the world.

Different mechanisms would accomplish this. If you want it to be something that feels really 'present' in the world, then the gods prevent magic working when used by or for the powerful, and maybe enact some huge flashy punishment for those who try. If you want it to be something subtle in the background, then the gods make innumerable tiny interventions to invisibly change the course of history, and you have to puzzle out in-setting why it is that states haven't dominated the magic business.

Or it can be done mostly through mortal instruments - powerful clerics who understand and agree with the ideology/aesthetic of their deities, and mostly use their magic to implement it; hordes of druids who stand in opposition to the very idea of states and seek to destroy them.

Verdict:

It's sort of tautologically true that you can get exactly what you want out of a setting by invoking all-powerful superbeings who also want that... Unless what you want in your setting is 'no all-powerful superbeings'. I am probably further in that direction than most GMs, my personal taste being towards massively inconsistent and varied religious practices, 'silent' gods, and little to no magical evidence of the divine.

Still, of all the solutions I've looked at, I have to admit this one is amongst the best, since it's very easy to explain ("there are interventionist gods and they have tastes"), very easy to implement (just do whatever you were going to do anyway), and exactly gets you whichever fantasy setting you want (aside from that one superbeings requirement).

8. THE CAMERA LENS IS NARROWLY FOCUSED

This one expands on someone's comment on the first article. Suppose that the argument so far holds: states which amass control over magic do tend to outcompete states which don't, and this is obvious within the world, so most states in most places and times are trying to control as much magic as they can.

Even if this is the case, by massaging the details we can come up with a workable section of the setting to play our game in.

For example. There's a reason the classic witch or wizard is so old: it's because magic-users almost all serve 'the usual' twenty to thirty years in the army or bureaucracy or similar apparatus (compelled by the state's BBBs) and then the survivors are rewarded for their service with a plot of land out in the country where they can't cause trouble. This is analogous to the Roman Legion approach, where together some positive incentives, strong cultural expectations, and a mild element of coercion result in the amassing of a useful resource (fit young men) where the state wants (the army).

So now assume that the actual tabletop gameplay all takes place in the hinterlands and dangerous wildernesses, away from the locuses of power where mass use of magic is going on at the state's behest. That means any mages the adventuring party encounters are mostly the retired ones - old, moderately powerful, fairly wealthy, in possession of a plot of land to build their farmhouse/tower/lair/bakery/inn/manor, and finally able to specialise in whatever strange and fantastical kind of magic they're interested in which the state doesn't care for.

Verdict:

In some ways this is a variant of the 'sweet spot' solution. It definitely seems workable for many games and many settings, and it also seems like it would work well in tandem with some of the other solutions. There are still a few questions to resolve, like 'what about PC spellcasters?' and 'what about when they visit the big city?' and 'what about when the state is losing a war and goes to muster its veterans and stragglers?', but on the whole I really like this one. Thank you anonymous commenter.

9. MAGIC IS ACQUIRED SUDDENLY AND SPECIFICALLY

Say magic comes about like this: people are born with, or develop, a knack for a certain type of wizardry that can't be transferred to a different type which would be more useful to the state. You can't learn, say, volcano magic by studying or going to volcanoes (things that the state could control and optimise around); it just happens to a random person.

The outcome is that there's less pressure from the people in power to create specific kinds of magic-user (although no less pressure to control the ones who do exist). So there's magic-users in all the nooks and crannies of the world - just ones whose powers aren't too useful to the state. A thousand confection-mages can plausibly be spread out over the kingdom, in small market towns or on the road or inexplicably delving a dungeon, with only a few working for the state (serving manorial estates, creating wealth in big cities, etc).

Unfortunately I don't think this works very well, for two reasons.

• Most kinds of magic are still desired by the state, because the state is just the people in charge, who want most of the things magic can offer. It's just that some are at the 'we want this' level and others are at the 'we really want this' level. Even little party trick stuff is still going to be desirable for individual rulers, just not a high priority. TTRPG settings tend to have lots of the high-priority kind which can produce wealth or military advantages or longevity or whatever.
  
• Players are usually going to want their PCs to be the kind of magic-user with sweeping combat and utility powers, i.e., the kinds the state really wants to get control of.
  

Verdict:

This is at best a partial solution. It constrains the kind of magic in the setting quite a lot, too: If there's this spontaneously acquired magic but also other kinds of magic that are, say, acquired by study in a particular sphere, the state would just be able to invest in the latter.

10. MENTAL GYMNASTICS

Suppose to learn (or otherwise acquire) magic, you need some set of mental characteristics. Let's say, wisdom, compassion, a distaste for institutions, a preference for local over regional interests, an elevation of the abstract above the mundane, and maybe - let's just come right out and say it - a lack of interest in statecraft. Then, as long as magic itself is really powerful, states will fail to get control over magic-users most of the time.

Verdict:

This is really just an instance of solution 7(i) dressed up nicely. It has the same issues - roleplaying implications and potential workarounds - and also requires individual (or small collectives of) magic-users to largely be more powerful than states (otherwise the state can still compel its use through coercion).

Overall, it demands too much of characters for me to use it. You could perhaps apply it to certain kinds of magic (wise clerics, non-worldly hermits) as part of a patchwork solution.

11. RESCUED BY INSTITUTIONS

Fantasy milieus commonly have at least a pretense that society is organised in a classic high medieval power structure, i.e.,

Peasants < A local lord (knight or baron) < His liege, an aristocrat < Possibly a higher ranking aristocrat < The monarch

Separate from this is the "First Estate", ordained clergymen, who had a hierarchy of their own. And in urban settlements with a charter (most cities, some towns), the power structure tends to cut out the aristocrats.

So maybe we can have institutions other than states which shelter magic practitioners and allow them to flourish outside the state's purview? It seems like there's three good candidates for that:

1. Guilds
2. The First Estate (church or comparable religious institution)
3. A Fourth Estate (something unique to magic-users)

RESCUED BY INSTITUTIONS, ATTEMPT 1: GUILDS

I think there's a misconception that guilds were somewhere between modern labour unions and MMORPG 'guilds' for collaborating on activities. Not really. At least in Western Europe between the Crusades and industrialisation (a pretty common touchstone for fantasy worlds), guilds tended to have some quite specific characteristics:

• They tried to keep prices high, to their own benefit, via various means (principally squashing competition from outside the guild via political action).
• They tried to keep quality high, or at least high enough not to ruin the guild members' reputation; to do so they met regularly, policed themselves, stamped marks on their work, and often regulated teaching.
• They were almost always small, selective bodies. By deliberately keeping their membership low with a "dead men's shoes" type system, each individual member could get more trade, charge more, have more prestige, and so on. Plenty of historical people diligently apprenticed in a trade and never got to practice it as a guild-accredited artisan - usually because guild members' children were preferentially elevated into the guild!
• Guilds usually were officially associated with specific religious, moral, or charitable causes - this was a deeply pious time, and often the guild would have a specific religious purpose, a patron saint, would collectively give to the church, sponsor the building of new churches, etc.
• Guilds sometimes had an additional civic purpose - they might sponsor the construction of civil infrastructure like bridges, organise festivals, etc. They might be formal divisions of the town militia, and the guild would bear the responsibility of arming and organising their members.
• Guilds often acted as an informal social safety net for widows and children of deceased guild members.

Successful guilds also tended to amass wealth and be politically important. Whereas outside cities, society was often arranged in the hierarchy of vassalage mentioned above, inside towns, the power structure might look more like:

Townsfolk < Wealthy and successful burghers < Aldermen/mayor < The monarch

In many cases, guilds would weigh heavily in the two middle tiers.

All of this could apply to magical arts. If magic did develop through a system of powerful guilds, would that be enough to keep it safe from grasping states?

• Guilds flourish under a system where cities can have municipal charters, so their townsfolk are 'free'... but 'free' mostly means 'free from the tyranny of aristocrats', not 'free from the tyranny of the monarch', which is to say, city charters don't prevent the state from meddling.
• Having a close-knit collective of magic-users definitely helps prevent the "if magic-users are too powerful to be controlled by the state, magic-users will overthrow and become the state" problem. It means that individual mages can be quite weak compared to the state's knights/assassins/armies/etc, which is desirable for most settings. It only works as long as the guild has strong bonds and weak ambitions, though; you need to find the sweet spot where magic guilds stop their members defecting to the state, but also don't grow to become de facto state powers.
• Mage guilds need to have an extremely high rate of membership and a tight grasp on teaching magic, otherwise the state can still bypass it by spending resources to create and/or hire its own magic-users and slowly consolidate power.

Verdict:

I think this is borderline workable, as long as you don't mind the constraints it places on the setting. It still allows for a small number of court magicians and war wizards, and princes complaining about the ruinous rates the Guild charges, which are nice situations to have in the pocket. Unfortunately there's a weird implication that most of those small-town witches / dungeon-delving sorcerers / wandering illusionists / reclusive sages / etc still have deep guild affiliations (otherwise the state would be finagling them into its service).

RESCUED BY INSTITUTIONS, ATTEMPT 2: THE FIRST ESTATE

In this version we're saying that something directly comparable to the Medieval West European Christian Church is in charge of magic. It's a big, important, hierarchical, highly respected institution with major (but not complete) autonomy and a lot of accrued wealth.

A large part of the reason the Church survives (and mostly thrives) outside the secular system of fiefdoms and vassalage is that everybody of the era is highly religious, so acting against the Church gets "you are violating God's will" added to the normal list of reasons it's a bad idea to act against powerful institutions.

There's two issues with this solution.

First, why is the First Estate in charge of magic? It's extremely restrictive to world-building to have all magic be divine magic, or be non-divine magic that is nevertheless sheltered by or structured around the Church.

Second, we have historical counterexamples to the Church's ability to protect its own interests. The infamous one is Henry VIII of England, married to Catherine and wanting to marry Anne instead. He destroys the institutions of the First Estate, gets excommunicated, divorces Catherine, marries Anne, starts looting the wreckage of the Catholic Church in much of England, and replaces the local religion with his own new version. This was all done in pursuit of a personal goal, and frankly a minor one compared to some of the benefits that magic can usually offer.

Verdict:

This solution's plausibility really depends on setting details. If it's the kind of fantasy milieu where interventionist gods literally exist, then no earthly ruler is going to succeed in replacing the First Estate a la King Henry. But in that case you're probably better off just going with Solution 7 (ii).

RESCUED BY INSTITUTIONS, ATTEMPT 3: A FOURTH ESTATE

Not 'Fourth Estate' like the modern media. Magicians.

The three 'Estates' were, in their historical context, viewed (by the people who put stock in the framework) as deeply different things. It's possible that if magic actually existed, it would be thought of as so fundamentally different that magic-users would end up structured as another Estate, separate from those who ruled, those who toiled, and those who prayed. This might be especially true if magic is of a weird and otherworldly kind which changes the person who wields it.

But I think this probably oversells how weird magic is if you grew up in a world where it existed. Imagine saying 'if metallurgy actually existed, it would be thought of as so fundamentally different that blacksmiths would end up structured as another Estate'. Or 'if literacy actually existed, it would be thought of as so fundamentally different that scribes would end up structured as another Estate'.

Verdict:

Maybe a long, ancient, ingrained tradition of "mages are just a different class of folk, and we let them govern their own" is doable. But it's not like rulers didn't try to meddle in the affairs of the Church, a different class of folk who were meant to govern their own.

And, as I've said before, all you need is for one state to find a path to controlling a bunch of magic (in this case by busting down the traditional setting-aside of the Fourth Estate). Then it starts to outcompete its neighbours, its successors learn from its success, and soon there's a new normal.

12. SOME STATES, SOME OF THE TIME

One of the rejoinders I heard to the original argument was that it makes its own case too strongly. In particular, we do have what amount to counterexamples from our own history - cases where states had the ability to seize additional power and failed to do so.

For example, someone mentioned Genoese engineers in the late medieval/early modern period who built cannons for anyone with the cash. From Wikipedia, we get examples like "Orban [...] was an iron founder and engineer [...] in the Kingdom of Hungary [...] who cast large-calibre artillery for the Ottoman siege of [Byzantine] Constantinople in 1453. [...] He had offered his services to the Byzantines in 1452, a year before the Ottomans attacked the city, but the Byzantine emperor Constantine XI could not afford Orban's high salary nor did the Byzantines possess the materials necessary for constructing such a large siege cannon. Orban then left Constantinople and approached the Ottoman sultan Mehmed II, who was preparing to besiege the city. [...] Orban was given abundant funds and materials by the sultan."

The implication is that in some cases a skilled magic-user might just exist outside the state system, and offer their services to anyone rich enough... and some states say "yeah, that's fine. We'll call you back if we get money."

Of course, it's not a perfect analogy. For a start, magical power tends to be both more wide-ranging and more tightly linked to the individual than engineering power. We also usually want settings where spellcasters of various capabilities and specialisations might be wandering around in every little corner of the world; I get the impression that this was not historically the case for cannon-founders.

(Also part of the Wikipedia entry points out that the Byzantines didn't actually have the materials to make a large siege cannon, which in the analogy is like a state with no access to parchment failing to put effort into getting a maker of magical scrolls under its control. Still, a sufficiently ruthless state might have imprisoned or killed Orban just in case he took an offer to their enemies, as he ended up doing.)

To generalise, though - yes, I think it's fair to say that there are forces and incentives opposing the ones which should drive states to invest heavily in magic power. Systems are complicated, people are motivated by things other than perfect rationality, information is obscure (especially before the Information Age), circumstances can change, outside factors can intervene, and winning strategies are seldom obvious before the competition takes place.

And maybe all this can lead to a 'fantasy mess' milieu where different states have different levels of, and methods of, investment in getting the useful technology of magic under their control - and even when they try, sometimes fail due to all sorts of messy reasons.

Verdict:

This is again quite similar to the 'sweet spot' solution, implying that yes, some states, some of the time, should be investing heavily in magic, but for many states much of the time, there are good reasons for them not to. Things often are messy and inefficient. I think it's a fair view, enough so that it actually makes me feel that the problem is less of a sticking point for verisimilitude.

So... sure. From time to time, states can emerge which hate magic, or overvalue military power, or rely on a different kind of power base they've built up, or accidentally drive all their citizen mages into hiding, or whatever. If you have a sufficiently cynical view of people's basic competence, this could even be the norm. Expect that if there's a well-run, thoughtful, wealthy state, it will be grabbing up a disproportionate numbers of magic-users - but nothing like all of them.

FINAL VERDICT

So, what are the winners?

The obvious 'out' is [#7 variant (ii), 'superpowers keep things the way they like it']. As long as you have all-powerful interventionist gods in your setting, you can directly import this rationale and pretty much never worry about verisimilitude again.

I think my own preference, at the end of all this, is to combine [#1, 'sweet spot'] with [#12, 'some states, some of the time'] and [#8, 'narrowly focused camera lens']. In conjunction, you get a glorious mess where yes there are lots of sorcerous mercenaries and manor mages and court magicians and so on, in some states more than others, but whichever state(s) the game is taking place in are the ones that happen to be lower on the magic-investment end of things, so that there are plenty of magic-users choosing their own paths as individuals. On reflection, I think that works okay.

And my personal favourite, which I'd throw into the mix just for flavour? The partial solution where mages tear apart reality if you stack them too tightly in one place: [#4 variant (i), 'critical mass'].

ANY OTHER SOLUTIONS?

I'm excited to hear whether other people thought up other ways to address this world-building problem, or picked a hole in one of these solutions. Comment if you did!