With PDF, it's the browser's own built-in PDF viewer that displays the "download" button, so it's hard to avoid. I did some search, and apparently you can append #toolbar=0 to the PDF url, and it will remove the entire toolbar (including the download button).
With and without toolbar (works in Chrome browser at least):
https://demo.files.gallery/content/files/document.pdf
https://demo.files.gallery/content/files/document.pdf#toolbar=0
In my opinion, all this is kinda useless anyway. The user can just right-click the opened PDF document and choose "save as...", or just remove #toolbar=0 in url and refresh page, or just click the "share" button in browser and select "save page as ...". So many easy ways to access and download the file. As long as you are making files public visible on a website, it's pretty much impossible to block them from being downloaded.