At first I thought that the catchall targeted mails would all be greylisted or junked aggressively but I checked the mail server logs in that regard and they didn’t even show up. Sending myself a couple of test email, quickly revealed that the server is rejecting all unknown email addresses. So the “easy way” of setting up catchall in server app version 4 clearly didn’t work anymore.
A little googling later I found this excellent post from Charles Gutjahr on adding a catchall to OS X Server Mail. It works like a charm. So if you run into similar troubles, give the little tutorial a try. 01/11/2016