From quirky documentaries to unclassifiable dramas, we highlight the films that might have slipped beneath your radar this past year