When you're doing tiles, you need to simplify to highest resolution actually discernible in that tile. So, for each zoom level, generate bbox tile bounds, and then simplify or cluster everything in each tile for the pixel resolution you want. That way you will never have 1.5GB tile. After all, all that data is wasted space if it can't be seen.

I've done this for GeoJSON tiles for a long time, before the MVT tiles. The MVT tiles pack way more data though, but are not a solution to the O(n2) problem.