I worked with the Be guys when they were bought by Palm. Never in my career have I experienced such an obsession with solving problems no one has. To me its kind of sad so much obviously raw talent wasted on such frivolous effort.
What problems are they solving that no one has? And what effort is frivolous?
The file system stuff they were doing was incredible and highly useful. I'd really like to have a file system where I have indexed metadata committed directly with my file writes about now. Alas, since BeFS is dead, I pretty much have to roll my own with SQLite indexes alongside flat files. At the time, sure, it may look like no one has the problem, but if it would have become viable I think that we'd now be wondering how we lived without it.
I dunno... I've never seen a system in which having substantial amounts of non-recreatable data stored "outside" of the file ended up being seamless and seemed like a net gain to usability [1]. I'm thinking about years of annoyance at OS/2's extended attributes, Classic Mac OS's resource forks, and I think WinFS would have had some similar external metadata too. In practice, it seems to me like they end up being fragile and frustrating even in a world of single-user computers (they sometimes get lost when moved to a non-native file system or when the file is manipulated by a program that doesn't respect them [2]), and become easily more trouble than they're worth in a networked world where you want to share documents and other files with users on other systems/platforms. A UNIX-style unstructured-array-of-bytes model seems to be the baseline on the Internet and any extra structure is iffy at best.
Also, I note that when Giampaolo himself was implementing the subsequent metadata filing system that he worked on (Mac OS X 10.4+'s Spotlight feature), he went with a system-level metadata harvesting/indexing model where essentially all data (except for user-added Spotlight Comments, I think?) is wholly derived from the file itself by importers that are invoked by the system to populate its external indexed cache of metadata. That way, even if that central metadata database is blown away, it can be recreated at any time. This also allows interchange processes to only rely on what nearly always works across systems: the data itself and very basic metadata such as filename and creation/modified dates. And programs never have to worry about updating metadata aside from that which is internal to the file format.
[1] doesn't mean they don't exist (perhaps even BeOS, which I haven't used much) and I'd really like to hear about them, just talking about my own experience
[2] in this case, it can sometimes be even worse than a program wrongly blowing away the "resource fork" - a badly behaved program can update the file without updating its metadata, causing the metadata to be wrong!
What I gathered from discussions on the Haiku mailing lists was that -- sometime in the far far future -- file metadata will stay but the indexes will move out of the filesystem into an external location. This makes sense, since it allows you to index complex filetypes such as PDFs and web pages without affecting file system performance. I once worked on an index_server for Haiku that worked somewhat like Spotlight's daemon. A new and much improved version of index_server is already in the Haiku repos, but it's disabled for now. This will form the basis for moving indexes out of the file system.
On losing metadata: every modern filesystem in use supports extended attributes, so this shouldn't be a problem when copying files between BFS and NTFS, EXT4, btrfs, HFS+ or ZFS. The only filesystem that will cause problems is FAT32, which most people still use with USB drives. I don't know how that will be handled in Haiku.
I don't quite follow: when you say an "external location" do you mean in file(s) in the file system but not in the file system's internal structure itself?
With regard to most file systems supporting extended attributes: that's true (though FAT32 not supporting them is a huge caveat, and I also wouldn't be at all shocked to see some issues during conversion across FSes... I don't have personal experience though), but email, HTTP downloads, peer-to-peer, etc don't tend to support them, essentially guaranteeing you'll lose them if you transfer them across the Internet.
Also, it's really nice to not have to worry about, say, an EA-unaware POSIX app losing metadata... how many old-style Mac resource forks were lost during a simple file rename before mv was made resource fork-aware!
> I don't quite follow: when you say an "external location" do you mean in file(s) in the file system but not in the file system's internal structure itself?
That's exactly what I mean. My version of index_server stored the index in a directory called "index" in the root of every volume it indexed. I'm not sure what the new one does.
Note that moving the index out of BFS was merely an idea that was bounced around the ML.
Maintaining consistency is an issue. In many cases, though, the external attributes are generally extractable from the file content itself, so you can do a rebuild if the data gets corrupted.
The BeOS mail application used the extended attributes extensively. Its mail store was basically a bunch of flat files containing the messages (perhaps in folders corresponding to the mail folders), and the extended attributes were used to index the date, sender, etc. These indices were then used to generate mail views, respond to searches, etc. - the mail app could effectively be a custom file system browser that used the extended attributes to speed itself up and quickly access and search mail metadata.
