There are many interesting tools that achieve this, like Outlines[0] and jsonformer[1]. I haven't tried them myself but they look very promising.

[0]: https://github.com/outlines-dev/outlines [1]: https://github.com/1rgs/jsonformer