Just delete the .bib file. arXiv only needs the .bbl file. What is confusing is even if you have both the .bib and .bbl, arXiv doesn't ignore the former, it raises an error anyway. Bad error reporting.
Another tip if you're using overleaf: make sure your bibliography related warnings that overleaf points out, are gone too. Overleaf is resilient to these but arXiv is not.
I think the bbl file is what I meant. It's created from the source and bib file, and arXiv refuses to accept the one that is created by overleaf. I really dislike arXiv for demanding the full source and refusing pdfs, but then insisting you also provide a file that is generated from that source, but not accepting that file unless it's generated with specific builders.
Oh that's surprising. I uploaded content generated by overleaf (they provide a arXiv compatible dump of your project) a month or so ago and it worked smoothly (after uploading to arXiv I deleted the bib file of course).
I tried about a year ago and about half a year ago, respectively. Both times, the exported bbl file hat a different version than the one accepted by arXiv. First time, the bbl version was too old. Second time, the bbl version was too new. First time, I ended up printing the latex generated pdf file to a new pdf, so that arXiv wouldn't recognize it as a Latex produced file. That kinda sucked because in the printed pdf, you couldn't select text or access references. Second time that didn't work anymore because arXiv added a check to prevent users from uploading printed pdfs. I ended up simply hosting it at our institute page and abandoning arXiv.
Another tip if you're using overleaf: make sure your bibliography related warnings that overleaf points out, are gone too. Overleaf is resilient to these but arXiv is not.