Have you looked into cross-compiling those arm targets using Docker and Github Actions? I'm not familiar with the toolchain you have to use but other projects do that.
I've tried it! For Rust SQLite extensions, I'm able to use some cross compilers for some extra ARM targets on linux[0], I just haven't had the time to do it for all extensions yet. For MacOS arm builds cargo has the `aarch64-apple-darwin` target which works, but is a bit slower than expected at runtime. I've also tried zig for cross compiling C extensions on different platforms, which seemed to work ok but I've heard of some perf issues cropping up with those.
