I strongly suspect the same thing - that they weren't using agents at all in the reports we've seen, let alone agents with instructions on how to verify a viable attack, a threat model, etc.
Presumably they are saying that you'd end up using a lot of `unsafe`. Of course, that's still much better than C, but I assume that their point isn't "You can't do it in Rust" it's "You can't translate directly to safe rust from C".
Most mitigations just flat out do not attempt to help against "arbitrary read/write". The LLM didn't just find "a vuln" and then work through the mitigations, it found the most powerful possible vulnerability.
Lots of vulnerabilites get stopped dead by these mitigations. You almost always need multiple vulnerabilities tied together, which relies on a level of vulnerability density that's tractable. This is not just busywork.
> There are a couple of interesting implications from this outcome, should it hold. The first of those is that, as Rust code reaches more deeply into the core kernel, its code for concurrent access to shared data will look significantly different from the equivalent C code, even though the code on both sides may be working with the same data. Understanding lockless data access is challenging enough when dealing with one API; developers may now have to understand two APIs, which will not make the task easier.
The thing is, it'll be far less challenging for the Rust code, which will actually define the ordering semantics explicitly. That's the point of rejecting the READ_ONCE/WRITE_ONCE approach - it's unclear what the goal is when using those, what guarantee you actually want.
I suspect that if Rust continues forward with this approach it will basically end up as the code where someone goes to read the actual semantics to determine what the C code should do.
In my experience, in practice, it usually isn't that hard to figure out what people meant by a READ/WRITE_ONCE().
Most common cases I see are:
1. I'm sharing data between concurrent contexts but they are all on the same CPU (classic is sharing a percpu variable between IRQ and task).
2. I'm reading some isolated piece of data that I know can change any time, but it doesn't form part of a data structure or anything, it can't be "in an inconsistent state" as long as I can avoid load-tearing (classic case: a performance knob that gets mutated via sysfs). I just wanna READ it ONCE into a local variable, so I can do two things with it and know they both operate with the same value.
I actually don't think C++ or Rust have existing semantics that satisfy this kinda thing? So will be interesting to see what they come up with.
> I suspect that if Rust continues forward with this approach it will basically end up as the code where someone goes to read the actual semantics to determine what the C code should do.
That will also put it on the unfortunate position of being the place that breaks every time somebody adds a bug to the C code.
Anyway, given the cultures involved, it's probably inevitable.
> That will also put it on the unfortunate position of being the place that breaks every time somebody adds a bug to the C code.
Can someone explain charitably what the poster is getting at? To me, the above makes zero sense. If the Rust code is what is implemented correctly, and has the well-defined semantics, then, when the C code breaks, it's obviously the C code's problem?
I think a charitable interpretation is that given that the Rust code will be less forgiving, it will "break" C code and patterns that "used to work", albeit with latent UB or other nonobvious correctness issues. Now, obviously this is ultimately a good thing, and no developer worth their salt would seriously argue that latent bugs should stay latent, but as we've already seen, people have egos and aren't always exceedingly rational.
It's not exactly that the Rust implementation will be less forgiving. But developers will have a clear picture of the operations semantics, while C developers will have a harder time understanding the operations, and consequently avoid them.
Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability.
Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just to put my metadata into Redis?
It would be nice to see benchmarks with another metadata store.
We developed Object Mount (formerly cunoFS) (https://www.storj.io/object-mount?hn=1) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead.
This approach isn't right for every use case and juice might be a better fit for this sort of 'direct block store', but wanted to include it here for folks that might want something like Juice but without having to maintain a metadata store.
(Disclosure: I work at Storj that develops Object Mount)
I am currently looking for a way to take a legacy application that uses the filesystem as it's database and needs to support locking (flock) on FreeBSD and scale it horizontally (right now we only scale vertically and rebuilding from a corrupted FS and/or re-pulling our, backup, data from S3 takes too long if we lose a machine). We investigated NFS but the FreeBSD NFS performance was 1/10th or worse than on Linux and switching to Linux is not in the cards for us right now.
Does Object Mount support file locks (flock specifically) and FreeBSD? I see some mention of FreeBSD but I can't find anything on locking.
For context, we are working with a large number of small (<10KB if not <4KB) files normally.
> The magic is: JuiceFS also supports using PostgreSQL as both metadata and object data storage backend! This means you only need to change JuiceFS’s backend to an existing PostgreSQL instance to get a database-based “filesystem.”
Sounds ideal for your kind of situation where a filesystem is being abused as a database :-)
Thanks, good to see these. Looks like these only show metadata operations though. But I guess that's probably enough to extrapolate that the system is going to be roughly 2-4x slower with other metadata stores.
Redis is as reliable as the storage you persist it to. If you're running Redis right, it's very reliable. Not S3 reliable, though. But if you need S3 reliable, you would turn to something else.
I expect that most folks looking at this are doing it because it means:
1. Effectively unbounded storage
2. It's fast
3. It's pretty darn cheap
4. You can scale it horizontally in a way that's challenging to scale other filesystems
5. All the components are pretty easy to set up. Many folks are probably already running S3 and Redis.
Redis using the WAL for durable data can only be as fast as you can fsync on every write, just like every other database. The time to fsync is going to dominate the write path for Redis, and you'll sacrifice almost all of Redis's speed to achieve it. If you're running Redis in always fsync mode it I'd suggest there are better storage systems to use.
> Where do you think Redis persists its data to
This is sort of the point, Redis isn't for persistence. I mean it can, you can use the WAL, it's handy for not having to recompute when you relaunch Redis etc, but Redis is fundamentally an in-memory system and should be treated as such. It is designed for use-cases that necessitate in-memory performance, and that don't require durability.
With the wal redis runs with perfectly reasonable performance. Of course you're not going to have the performance of an in-memory only DB if you're flushing to disk on every write.
There's no vacuuming, there's no need for indexing. You can see the time complexity of most operations. Key-value operations are mostly O(1). You'll never get that kind of performance with other databases because they intentionally don't give you that granularity.
The metadata of the filesystem isn't the performance bottleneck in most cases.
> With the wal redis runs with perfectly reasonable performance. Of course you're not going to have the performance of an in-memory only DB if you're flushing to disk on every write.
But that's not what gets benchmarked against.
> You'll never get that kind of performance with other databases because they intentionally don't give you that granularity.
Okay but you also won't get horizontal scaling while maintaining consistency in Redis.
> The metadata of the filesystem isn't the performance bottleneck in most cases.
Okay, but they do show that other metadata stores are 2-4x slower than Redis, so it would be great to see whole-system benchmarks that use those.
> 4. You can scale it horizontally in a way that's challenging to scale other filesystems
Easy to scale on RDS, along with everything else. But there’s no Kubernetes operator. Is there a better measure of “easy” or “challenging?” IMO no. Perhaps I am spoiled by CNPG.
But how would RocksDB work with S3? It needs support for append that generic S3 buckets do not provide and for checkpoints and backup it assumes the support for hardlinks that S3 does not have at all.
Just like every package manager already does? This issue predates LLMs and people have never cared enough to pressure dev tooling into caring. LLMs have seemingly created a world where people are finally trying to solve the long existing "oh shit there's code execution everywhere in my dev environment where I have insane levels of access to prod etc" problem.
reply