I don't believe landlords should be forced to have tenants indefinitely, and I don't believe clogging up the courts with yet more cases just to evict awkward tenants is a good thing.
The Courts will be no more clogged up than with s.21. If someone doesn't leave after the s.21 notice period then Court action is still required, first to obtain the Possession Order and then separately to obtain a Warrant of Execution to have bailiffs complete the eviction. The only thing the s.21 does is not give the Court any discretion to suspend or refuse to grant a Possession Order (providing the s.21 notice has been served correctly). But s.21 isn't the only route where this applies- ground 8 (rent arrears of two months or more) is also a mandatory ground for possession.
It's worth noting that if someone is convicted of a major criminal offence (and the offences this includes are remarkably broad, and includes offences like having a bladed article in public) or breaches things such as a noise abatement order then they are also mandatory grounds. Any sort of ASB or failing to comply with the terms of a tenancy is also a discretionary ground.
The change in law will also make the landlord wishing to make genuine efforts to sell the property a mandatory ground for eviction.
I'm unsure why you are against the idea of a landlord having to have some evidence before applying to the Court to make someone homeless. There should be that check and balance in any system. If a tenant is genuinely anti-social then this should be relatively easy to prove to the civil burden of proof; the neighbours may not wish to testify but there will be evidence of, say, a council noise abatement order being breached or evidence of repeated police attendance at the property.
Of course the sad truth is that a significant minority of landlords use s.21 in a retaliatory fashion- an "awkward tenant" is, to them, someone who asks for repairs to be carried out promptly.
No but there is behaviour which falls just short of what a court might consider antisocial behaviour for example; or people who regularly pay their rent late but don't cross the 2 months threshold the courts require for evicting for non-payment; etc.
There are discretionary grounds 10 and 11, which are either that the tenant has any level of rent arrears or that the tenant is persistently late in paying their rent. There are also discretionary grounds for breaching the tenancy agreement or for failing to keep the property in a good state of repair.
None of it was deemed a serious enough noise complaint by the council nor anti-social enough by the police.
If it's not serious enough for the council or the police to consider it as ASB then it's not serious enough to make someone homeless, is it?
If someone is punching holes in the walls and smashing up furniture then it's likely that would meet the discretionary ground for eviction. I'm not sure many courts would refuse possession in that situation; they may make it a suspended possession order (i.e. do it again and you'll be evicted) but they're unlikely, in my experience, to shrug it off.