Since 90s John Carmack wrote games for hardware planning two years in the future. During his planning and development stage, those games were barely playable on standard gaming machines of the time. But he correctly knew that by the time these games will be released, hardware will get two to four times faster.
Thus for long he was the fist to come up with games that demonstrated the full capabilities of new hardwares. This formula worked until around mid 2000s when suddenly processors weren't getting the yearly speed boost as usual. Doom3 was it's first casualty. It was designed for a hardware that never appeared in market. Game designers had to trim down on-screen characters to make the game playable and released it late.
And rest is what we see now. That's my understanding of the situation.
While I would agree that Doom3 was a casualty of the effect you outline, most of the stuff Carmack is working on now is very much within the realm of current & future hardware.
The megatexture tech is a clever use of 'clipmaps' or virtual mipmaps and runs on current-gen hardware. They are currently looking at a sparse voxel octree technique to create a megatexture-like (theoretically)infinite LOD system for geometry. This is also intended to be used on current (or near future) hardware.
A lot of the graphics techniques used by id aren't invented by Carmack, his genius is making them run in realtime on consumer computers. And yes it can be a guessing game :)
Thus for long he was the fist to come up with games that demonstrated the full capabilities of new hardwares. This formula worked until around mid 2000s when suddenly processors weren't getting the yearly speed boost as usual. Doom3 was it's first casualty. It was designed for a hardware that never appeared in market. Game designers had to trim down on-screen characters to make the game playable and released it late.
And rest is what we see now. That's my understanding of the situation.