I don't think people really understand how big of a wall outside of GPU/TPUs the CPU, ram and even flash market has hit. We're paying as much if not more for the same stuff we were buying 4-5 years ago.
I do think we're more efficient and matrix operations are better (again, GPU/TPUs), but by and large, the computer hardware world has stopped exponential growth.
The period from 1990 to 2005 was amazing. Both in transistor counts and clock speed, it seemed like we were nearly *doubling* performance with every new generation. Memory and disk space had similar gains.
Explain this though. The code is deterministic, even if it relies on pseudo random number generation. It doesn't just happen, someone has to make a conscious decision to force a different code path (or model) if the system is loaded.
Its not deterministic. Any individual floating point mul/add is deterministic, but in a GPU these are all happening in parallel and the accumulation is in the order they happen to complete.
When you add A then B then C, you get a different answer than C then A then B, because floating point, approximation error, subnormals etc.
It can be made deterministic. It's not trivial and can slow it down a bit (not much) but there are environment variables you can set to make your GPU computations bitwise reproducible. I have done this in training models with Pytorch.
For all practical purposes any code reliant on the output of a PRNG is non-deterministic in all but the most pedantic senses... And if the LLM temperature isn't set to 0 LLMs are sampling from a distribution.
If you're going to call a PRNG deterministic then the outcome of a complicated concurrent system with no guaranteed ordering is going to be deterministic too!
No, this isn't right. There are totally legitimate use cases for PRNGs as sources of random number sequences following a certain probability distribution where freezing the seed and getting reproducibility is actually required.
How is this related to overloading? The nondeterminism should not be a function of overloading. It should just time out or reply slower. It will only be dumber if it gets rerouted to a dumber, faster model eg quantized.
Just to make sure I got this right. They serve millions of requests a day & somehow catastrophic error accumulation is what is causing the 10% degradation & no one at Anthropic is noticing it. Is that the theory?
There's a million algorithms to make LLM inference more efficient as a tradeoff for performance, like using a smaller model, using quantized models, using speculative decoding with a more permissive rejection threshold, etc etc
Trains are not an efficient use of time for travel within the US.
The US is huge. If you were take a 300mph (nearing 500kph) train (which would make it the fastest train in the world), it would be OVER an 8 hour trip from New York to LA. (Again, about 2500 miles or 4000k)
Even in some of the densest areas, the trip times end up being pretty long due to distances: dc to New York? 600 kilometers or almost 400 miles.
People aren't taking trains from Madrid to Tallinn, either.
The proper point of comparison here is more medium length trips. There's no reason not to have a high speed train for Portland - Seattle - Vancouver, for example.
This is irrelevant, though, since the size of the country isn’t what determines where people go. It’s not like trains got less practical when Alaska got admitted to the union.
Sprawling, low density, single use zoning, combined with parking minimums, have much more to do with it.
The question was what the train network is like outside the cities. And the answer is we don’t use trains because it is not efficient for the scale of the country. This is correct.
Most people ARE interested in coast to coast travel. It is called flyover country for a reason.
There are a few exceptions like the Baltimore corridor, or the San Francisco peninsula, and these are in fact serviced by good trains.
Train travel from LA to NY wouldn't be efficient, but there are plenty of population poles like LA to SF where train travel would make sense and a network of those could make cross country travel feasible if not in a hurry. But as the GP pointed out, it is not that useful if you can arrive to LA by train, if then when you arrive you need to rent a car before you've even left the station.
It is always shocking to me when I land at an airport in the US and there no public transport available.
It is common for conversations about good local public transport to have a retort in some sub-thread about how big the US is, as if the feasibility of long distance travel affected the feasibility of other modes of local travel.
You mention the SF peninsula. When I first moved there, I lived in the westside of SF and had friends living in Sunnyvale. On a weekend, it took 4 hs to get to Mountain View (~40miles, at the time, checking now it seems like Caltrain weekend service might have improved, so the same trip would take about 2hs), and then had to be picked up by car to finish the rest of the trip. It was faster (~3:30hs, if more expensive) to go from Paris to Amsterdam (>300miles) by train.
I see your point, but consider this: getting to and through a major airport is a huge pain the ass. Trains also tend to take you to city centers more often than airports, which almost always need to be a significant distance from anything interesting due to the noise.
Let's take a hypothetical scenario:
- 5 hours flight time (average for NY and LA), 2 hours on each side to get to and from the airport to the actual city. Total is 9 hours.
- 10 hours train time and 1 hour on each end (which is generous given the proximity of train stations to city centers), 12 hours.
The difference is not that much, and a train ride is so much less faff than a flight that it's not even funny. Little to no security theater, you don't get fondled by security agents, you don't have to stand hours in line with silly passport controls and luggage checkins/pickups. And the list goes on.
A good train infrastructure can be vastly more pleasant than a good air infrastructure. Where air wins out is intercontinental flights where trains are truly not an option anymore.
Eh, you're overselling it. Even in a hypothetical world with a 12 hour train trip, it still loses to an 8 hour plane trip.
You're losing an entire day on the train. You still have to deal with luggage pains, now you're eating on the transportation which will be inferior, and will have similar problems with last mile transportation.
Flying is currently not a great 8 hour experience, but it beats losing an entire day. I can do LA to NY for a weekend trip. (I personally wouldn't but there are some that would for sure)
Trains can and do make sense even in the US, and we do ourselves great harm by underinvesting in them, but there will always be a place for plane travel.
How long would it take from New York to Philadelphia, or Boston to DC? How long would it take between San Diego and SF? What about a train between Chicago and Detroit?
We're building a fast train from Toronto to Quebec city in Canada. It's going to be a lot more comfortable and way faster than driving. A MP in my family takes the train from Montreal to Ottawa very frequently, they don't want to bother with parking in the capital and they can work on the train.
I 100% agree trains might be underfunded in the US. The LA to NY flight will stay preferred to a hypothetical high speed train due to time. It is unlikely ever to be less than 10 hours.
For train rides under 4 hours, and if you can get trains running smoothly (less stops), the time spent on the train and the overall integration of trains is a lot better as a mode of transportation.
NY to Philadelphia is relatively fast. There was a faster train on that route even before Acela (Metroliner). Boston to DC is a fairly long day by train and, while I've done it, no business person who is really on a schedule is going to vs. taking a pretty short flight. (And, if you're really going city to city, both cities have close-in airport options.)
Why did the fed raise interest rates? To soak up some of that cash. It was too slow, but this is exactly the sound money policy that everyone expects. You loosen cash (what you mistakenly call printing money), when you need investment, and tighten cash when inflation and risk taking is out of control.
a sound response to some of the worst fed decision making in US history. they essentially ruined the housing market, priced out a generation of younger buyers, which is now crushing fertility rates, savings, and more
Investment real estate ruined the housing market. All of a sudden housing prices are expected to grow year over year as an investment. As more and more growth expectations were applied to housing, public policy (including zoning) changed to protect those expectations. Is it any surprise that there came a point at which it became too expensive?
Once problem we need to solve is how to unwind housing prices without financially ruining honeowners whose house is their primary/only wealth. Of course this problem is even more severe in areas of the country that are becoming uninhabitable due to changes in climate as it drives down demand.
The article makes a claim that it is no better in Canada and proceeds to remark about financial conditions. The mail is being delivered just fine, just not profitably.
I've not had issues with the USPS, but I don't doubt that it is getting worse. Private delivery (Amazon and the like) has been pretty much flawless. Order from McMaster, and it almost invariably arrives within two days (continental US)
I just don't experience what the author is getting at.
The USPS is in year 4 of a 10 year overhaul of its infrastructure. New facilities with new equipment optimized for the current mail mix. Magazines and catalogs used to be huge. Now they are a fraction of their volume. First class letter volume is crashing as everyone goes paperless.
Where there use to be separate facilities for processing first class, bulk mail, and packages; the new facilities deal with everything. And where the old system of 50 NDC (National Distribution Centers) are being consolidated to ~30 RPDC (Regional Processing and Distribution Centers) leading to a whole new strategy of how mail moves East to West and West to East. And mail sorting for delivery used to happen by mail carriers at local Post Offices, is now happening at LSDC (Local Sorting and Delivery Centers) set up service all mail carriers in a 50 - 70 mile radius.
And all of these changes are happening while still having to deliver mail (It never stops Jerry. It just keeps coming and coming)
So if you’re in the Midwest like Chicago, stuff coming from the Midwest or Eastcoast has been getting stuck in Indianapolis-taking 10 to 15 days. Stuff coming from West Coast gets here in 5.
There are 42,000 active zip codes and 640,000 employees. Making changes to that organization is hard and takes time.
What’s really cool is the work going on with Amazon, Walmart, Target and others for them to deliver packages directly to LSDC for same day delivery. Once you get away from the cities, no one can compare with the USPS for last mile delivery.
TLDR USPS is changing. Things may get worse before they get better.
Even simple "changes" (like cutting a single truck) cause massive mayhem in their wake.
The simple "elimination" of nightly mail pickup from rural post offices has resulted in multi-day backlogs just getting anything out of your local Post Office. If I walk into any of my local POs in the county this morning to ship a Priority Mail Express letter overnight, they will not guarantee that it will be received tomorrow. The envelope will sit for nearly 24 hours locally for tomorrow morning's mail truck before it is brought back to a processing center tomorrow night to be introduced into the mailstream and finally processed outbound.
Basically anything being shipped cross-country is going to be snared in Rural Delay Hell and it absolutely sucks. Combine this with the recent postmarking changes and we're in for an absolute fun treat when it comes for mail-in-voting time.
Yes. The only reason bug shippings are getting same day is because they are bypassing most of the postal network and dropping it off in time for the days final sortation for delivery.
This is an article written by a company/llm trying to justify huge increases to the pricing structure.
Oh! Yknow that thing we were charging you $200 a month for now? We're going to start charging you for the value we provide, and it will now be $5,000 a month.
Meanwhile, the metrics for "value" are completely gamed.
The price will be what you are willing to pay. No justification required, excepting for fairness (info asymmetry and what else?). It is written by me. Unfunded bootstrapped !!call it dire straits.
At the same time, I actually wouldn’t mind a world in which AI agents cost $5000 a month if that’s what companies want to charge.
I feel like at some level that would remove the possibility of making a “just as good as humans but basically free” arguments and move discussion in the direction that feels more productive: discussing real benefits and shortcomings of both. Eg, loss of context with agents vs HR costs with humans, etc…
I hate it when blanket statements like this creep in.
Which Covid? The initial version was definitely more deadly than later versions.
What about future covids? Are you willing to guarantee every version of covid from here on out will be less deadly? It is the general case to be true, but it is not some sort of law.
I do think we're more efficient and matrix operations are better (again, GPU/TPUs), but by and large, the computer hardware world has stopped exponential growth.
The period from 1990 to 2005 was amazing. Both in transistor counts and clock speed, it seemed like we were nearly *doubling* performance with every new generation. Memory and disk space had similar gains.
reply