The fourth film by the British director is a deeply disturbing and suffocating look at complicity, loosely based on Martin Amis' novel