Four people are confined to a seaside house in the German filmmaker's quietly apocalyptic mystery about artistic anxiety