Many cloud providers now offer serverless SQL and Spark capacities (serverless=no set up for you). This is the magnitude change for me.
With pandas you can maybe process 10 million rows, with polars maybe 50 million. But with a distributed service maybe 100 times more?
Many cloud providers now offer serverless SQL and Spark capacities (serverless=no set up for you). This is the magnitude change for me.
With pandas you can maybe process 10 million rows, with polars maybe 50 million. But with a distributed service maybe 100 times more?