This is very good advice BUT, and this isn't a dig against OP, just a heads-up. A lot of AI companies will either straight up ignore robots.txt or will fake the user-agent of their crawling bot to bypass any blocking you might do on the server-side.
This isn't the only source but is one I could easily find about this specific issue:
I wish there was a silver bullet for that stuff but alas.