But like you've said, the main enemies and have already been driven out or eliminated in one way or another, and there was no real opposition. Were they afraid that the opposition will inevitably re-emerge unless they act decisively?
I think there was probably a bit of that, after all Zinoviev had last been kicked out for receiving a copy of the Ryutin platform. If you work from the approach of Stalin having a unique role, there is also the suggestion that physical destruction of his enemies, for revenge, was a deep seated need of his.
But they couldn't have done it without being given the mandate from the top, and the purpose of that still looks vague.
I mean, what seems so strange is that as far as I know, most dictatorships out there can remain in power without ever having to resort to such massive violence, particularly against their own party members.
I think this is why the "Stalin" explanation is so attractive. Anomalies can be easily explained by pointing to an exceptional individual. On the other hand we could work on the assumption that Stalin's formative political years shaped this mentality, and thus others might have worked on the same principles. Perhaps the constant bloodless purges, and the reason/propaganda surrounding them, the seemingly constant re-emergence of opposition, lead some to the 'logical' conclusion of a blood purge.
This only seems to have happened in the Stalinist/Maoist countries - the Cultural revolution in China for example, appears to be a similar phenomenon.
The Cultural Revolution was different in a number of ways. One, Mao had almost been sidelined from power before he warmed up the 'machine', while we find Stalin almost at the peak of his powers. Second the Cultural Revolution had a broader, communal aspect. Not only were party members to be targetted, or at least people accused of having political motives, but the Cultural Revolution targetted things like the 'Four Olds' (IIRC), which while couched in political terms targetted decided unpolitical activity at times. So the Cultural Revolution had a deliberate transformative aspect, while the Stalinist purges don't appear to have been intended to change anything, only 'secure' the status quo. In terms of execution, the Red Guards were also fairly novel. The Soviets relied on their own security apparatus, but Mao essentially raised his own political militia from the youth.
Even Hitler never had to deal with anything like it to my knowledge. The closest event was probably the "Rohm putsch", but that was quick, waay less bloody, and most of the people targeted were actual enemies or threats in one way or another.
The SA and other individuals collected during the Night of Long Knives tended to be more obvious opposition figures. Also motive/function plays a role again, with Hitler trying to surpress a chaotic element while it seems Stalin wanted to harness a 'chaotic' element in encouraging accusations throughout society.
Off the top of my head Saddam Hussein had a fairly big blood purge when he took official power. Still more focused than the Stalinist purges, it appears Saddam himself was the only one making accusations. The Stalinist approach of accusations from anyone and everyone suggests a social aspect... and perhaps its end result and unique nature endorse the idea that this was a run-away machine.
As far as I know it wasn't that much of a failure though, was it? And even if they fell short of the plan, you know they could have still done the usual thing and praised the achievements and make it seem as if everything was going great. Unless the country was facing an imminent economic crisis and the problems were undeniable, which wasn't the case, there was no particular reason to go down the route of "the situation is bad because there are enemies among us."
Oh they certainly praised their limited successes, and generally the problems were not so serious they endangered the whole economy. Though some of the reshuffling for the second plan highlights some concerns. I think the goal was to say on one hand "we are doing the right thing, it works" while saying that anything that did go wrong "wasn't our fault, in fact it is the 'enemys' fault". So one can complement the other.
AFIAK it wasn't just that it switched, it was also the scale. More people have been executed in the period of 1936-1938 than at any point since the end of the civil war.
I don't have the numbers in front of me, but things like the peasant uprisings in the 1920-1921 period tended to be very bloody affairs. Decossackisation was also pretty big in scale, if not directly as bloody.