Joel Edgerton stars in the writer-director's third "man in a room" movie in a row, a character study that both confounds and absorbs