In theory you could delete all contacts from your address book, except the ones that you would like to chat with on Whatsapp, then later re-add the ones you deleted, but doing it manually would be too much effort.

My dirty fix for this is to synchronize the contacts with a CardDAV server (owncloud/nextcloud, radicale, baikal,..) and to use an app that lets you synchronize multiple address books at the tap of a button.

The trick is to add a second address book to your server, to which you only add the names and phone numbers of people who use Whatsapp, remove your regular address book from your phone, synchronize the “Whatsapp address book”, grant Whatsapp the contacts permission, add your contacts, remove the contacts permission for Whatsapp, and synchronize your regular address book again.