I can't speak for the person you're quoting, but I think there is a valid argument to be made that should people wish to carry out activities that are likely to cause substantial noise, it would be tactful of them, if they are able, to do so at a time when fewer people are likely to be trying to enjoy their gardens peacefully.
A 'good' time would therefore be when you would expect most people to be either out or engaged in activities indoors - mid-morning, mid week might be a good shout. Of course not everyone is home then , largely because they have the aforementioned other engagements - which is where the argument falls down.