Rebecca Miller's new film is a strangely engaging but tonally inconsistent ensemble piece about a washed-up New York composer