>Use the python multiprocessing module. If you've already written it with the multithreading module, it is a drop in replacement. Your data structure will live in shared memory
Only if it can be immutable. So it can't be shared and changed by multiple processes as needed (with synchronization).
And even if you can have it mostly immutable, if you need to refresh it (e.g. after some time read a newer large file from disk to load into your data structure), you can't without restarting the whole server and processes.
So, it could work for this case, but it's hardly a general solution for the problem.
For this use case it would be better to put the data in a shared SQLite database than relying on multiprocessing CoW.
Even accessing objects from the shared memory would cause the reference counter to increment and the data would be copied, causing a memory usage explosion.
Only if it can be immutable. So it can't be shared and changed by multiple processes as needed (with synchronization).
And even if you can have it mostly immutable, if you need to refresh it (e.g. after some time read a newer large file from disk to load into your data structure), you can't without restarting the whole server and processes.
So, it could work for this case, but it's hardly a general solution for the problem.